Enabling OpenVINO

[sharronliu]

This PR enable OpenVINO™ technology for GQCNN. The original GQCNN models are converted to OpenVINO™ models, and deployed across various Intel devices for inference.

Evaluated OpenVINO™ models are: GQCNN-4.0-SUCTION, GQCNN-4.0-PJ, GQ-Suction, GQ-Bin-Picking-Eps90, GQ-Image-Wise. The evaluation is done in the same way as mentioned in the replication scripts under scripts/policies/.

I tested with Intel NUC6i7KYK at CPU and GPU. A list of supported device types are available here.

To replicate my results, the GQCNN OpenVINO™ models are available here for test purpose. These models were converted from the original pre-trained GQCNN models. Untar these models from gqcnn root: tar -zxvf gqcnn.models.OpenVINO.20191119.tgz. Or you may convert the GQCNN models by yourself, following the tutorials in this PR.

[visatish]

Hi @sharronliu,

Glad to hear that you are using the GQ-CNN, and thanks so much for porting it over to the Intel OpenVINO framework!

I left a few comments on the PR, and it would be great if you could take a look at them. In my comments I reference https://github.com/BerkeleyAutomation/gqcnn/tree/ncs, which was used in the past to deploy the FC-GQ-CNN on the Intel Movidius Myriad X with the NCSDK.

I'm also curious as to whether or not this works for the FC-GQ-CNN or only for the GQ-CNN. One thing we encountered earlier was that certain operations in the pair-wise softmax used by the FC-GQ-CNN were not yet supported by the NCSDK. It was either the split or concat or both. I'm not sure if OpenVINO supports these. The workaround was to implement them in NumPy post-inference.

Thanks,
Vishal

[visatish]

Would it be possible to automate this as part of gqcnn/model/openvino/network_openvino.py? I just don't like the user having to edit the source. See https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L251 for an example. There when the user invokes compile(), which executes the compilation shell commands, the first step is to generate the protobuf.

[sharronliu]

Yes I think the freezing codes could be put into a separate function (e.g. freeze()), like what you did for ncs_fc_network_tf.py.
An open question is where to invoke the freeze(). I noticed compile() is invoked each time when the FCGQCNN model is loaded.
While freeze() is necessary only for OpenVINO model conversion, which is usually done once the pre-trained model are updated. Would you mind if freeze() is invoked each time loading the GQCNN model?

Another open question is, network_openvino.py may not be the right place to do freezing. Then OpenVINO model is an intermediate representation of the original model. It's converted, no more depends on Tensorflow, nor needs to load the original TF model. While freezing is expected from the original TF model.

[visatish]

I didn't have these issues here: https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L242, so I'm curious as to what the difference is. One is that I use self._graph.as_graph_def() instead of self._sess.graph_def, however I don't think that's it. Another is that the method in which the protobufs are written is a bit different. Finally I guess you are using OpenVINO whereas I was using the NCSDK, but I would think that they would support similar operations, right?

I also think it's a bit weird that the frozen graph would require RandomUniform because that should be strictly for training. If I recall correctly, only operations that the output directly depends on (i.e. things for inference) should be kept by: https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L245. I don't exactly remember when/where Floor is used.

This might be worth looking into until your PR for the RandomUniform and Floor operations is approved. Even if you could just copy my code and try it out, that would be super helpful. Maybe something changed and it will still break.

[sharronliu]

A little bit surprised hearing from you that RandomUniform or Floor is not used...

The OpenVINO Model Optimizer tool detects the exist of these two layers. Besides, the tensorboard visualization tool also illustrates these two layers, for pre-trained GQCNN 2.0, 2.1, 3.0, 4.0-pj, 4.0-suc downloaded by download_models.sh. I'm attaching the screen snapshot from tensorboard client, which might help you to recall something... :)

The usage of RandomUniform or Floor should not be introduced by OpenVINO or NCSDK. Both tools are inference stacks. While these two layers are observed in the original GQCNN graphs.

I also noticed these two layers were not introduced in the pre-trained models in Aug. 2018, when I started some works on GQCNN acceleration. My initial guessing is they were added later.

I won't think it related to self._graph.as_graph_def() or self._sess.graph_def, these two are equal if check == while freezing.

As you mentioned RandomUniform should be strictly for training. Would you suggest a right place to freeze the model? I'm thinking whether network_tf.py is the proper place, since it loads the GQCNNTF model for training. If the inference model is loaded in somewhere else, I can invoke the freezing there.

[sharronliu]

RandomUniform and Floor shown in the tensorboard visualization.

[visatish]

@sharronliu Okay, I think I know what's going on and you're right. So the RandomUniform and Floor ops are coming from the dropout layers. Our earlier experimentation with the NCS was using the FC-GQ-CNN, in which all of the layers are convolutions (the fully-connected ones are converted at inference time), which don't have a dropout applied afterwards. We never actually tried to port the original GQ-CNN and thus never experienced this issue.

Like you mentioned, the correct solution to this seems to be to remove the dropout layers in the inference graph. One option is to never include them entirely. The other is to post-process the graph definition and bypass those layers. I'm leaning towards the latter, however I'm a bit busy at the moment and will probably only get to it later this weekend. I'll let you know how it goes!

Thanks,
Vishal

[sharronliu]

Vishal, thanks for your time and quick reply. I just added feedbacks to your previous comment on having an automated freeze().

Then some explanation on NCSDK, now also part of OpenVINO. Which implies, the installation of OpenVINO will support deployment on all devices (CPU, GPU, MYRIAD, FPGA). This #93 was tested with MYRIADX also (with batch size forced to 1). BTW, I didn't mention FPGA in the yaml configure file, since I don't have the device to test it.
For people who has been ever working on NCAPI, it would be very easy to understand the OpenVINO Inference Engine API. And more python example codes here.

Then regarding your question on FCGQCNN. I originally planned to support FCGQCNN also. While I met some issue when freezing the graph. Before getting a frozen FCGQCNN, this PR is tested for GQCNN so far.

[sharronliu]

@visatish , I'm doing some minor changes for this PR based on the latest OpenVINO release version 2020.2. In this version of OpenVINO, the Floor layer is supported, while RandomUniform is not supported due to this layer is considered not present in a deployment network. I will try to drop this layer for a frozen network.