ONNX export compatible with OpenCV #462

dkurt · 2020-03-28T16:43:50Z

This PR has changes which let to export trained model to ONNX format compatible with OpenCV (PR opencv/opencv#16925 is required)

Convert to ONNX

python3 export_to_onnx.py --model ssd300_mAP_77.43_v2.pth

Run with OpenCV

import numpy as np
import cv2 as cv

net = cv.dnn.readNet('ssd.onnx')

img = cv.imread('example.jpg')
rows, cols = img.shape[0:2]
inp = cv.dnn.blobFromImage(img, 1.0, (300, 300), mean=(123, 117, 104), swapRB=True)

net.setInput(inp)
out = net.forward()

for detection in out[0,0,:,:]:
    score = float(detection[2])
    if score > 0.5:
        xmin, ymin, xmax, ymax = [int(v) for v in detection[3:] * [cols, rows, cols, rows]]
        cv.rectangle(img, (xmin, ymin), (xmax, ymax), (23, 230, 210), thickness=2)

cv.imshow('Object detection', img)
cv.waitKey()

Or run with OpenVINO

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_onnx.py \
  --input_model ssd.onnx \
  --mean_values [123,117,104] \
  --reverse_input_channels \
  --input_shape [1,3,300,300]

luoduo21 · 2020-03-30T04:00:09Z

@dkurt Hi, do you know how to export the model with prior box node. If I add the prior box node, the error will appear:
output of traced region did not have observable data dependence with trace inputs; this probably indicates your program cannot be understood by the tracer.

dkurt · 2020-03-30T05:59:31Z

Hi, @luoduo21, can you please show the exact source branch version to reproduce? In this PR priors are stored in the constant node.

luoduo21 · 2020-03-31T17:24:50Z

@dkurt
self.priors = self.priorbox.forward(), volatile=True)
output = ( self.priors, loc.view(loc.size(0), -1, 4), # loc preds self.softmax(conf.view(-1, self.num_classes)), # conf preds )

the export code:

torch.onnx.export(net, example, model_path, verbose=True, input_names=['input'], output_names=['priors','boxes', 'scores'],export_params=True)

Maybe onnx doesn't support exporting constant node?

dkurt · 2020-04-01T09:18:56Z

@luoduo21, Can you please share why you need to export it as output? I mean do you want to run SSD and proposed branch not works for your model? You don't have to export raw priors+boxes+scores but you can export post-processed outputs from Detect layer (like in this PR).

luoduo21 · 2020-04-02T01:27:07Z

@dkurt I want to convert the onnx model to the ncnn model. I tried to add a Detect layer to export the post-processing directly,Although it can be successful，there are some problems in the process of onnx - > ncnn. So I want to export the three nodes directly。

LucasVandroux · 2020-06-09T17:02:52Z

@dkurt Thanks a lot for providing the code to convert the SSD model to the ONNX format. Unfortunately, I am not able to run your script export_to_onnx.py. Could you please help me understand what I am missing out?

Environment

Ubuntu (WSL): 18.04
PyTorch: 1.5.0+cpu
ONNX: 1.7.0

Issues

Impossible to find the coco_labels.txt file. Fixed using Corrected path to coco_labels #400
ONNX: RuntimeError: No Op registered for DetectionOutput with domain_version of 9 (see full output below). I didn't manage to solve that one but it looks that it is coming from the changes you did in ./layers/functions/detection.py in the symbolic function.

Traceback (most recent call last):
  File "export_to_onnx.py", line 43, in <module>
    save_model(net, input, args.output)
  File "export_to_onnx.py", line 24, in save_model
    onnx_model_pb = export_to_string(model, input)
  File "export_to_onnx.py", line 19, in export_to_string
    torch.onnx.export(model, inputs, f, export_params=True, opset_version=version)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/utils.py", line 515, in _export
    _check_onnx_proto(proto)
RuntimeError: No Op registered for DetectionOutput with domain_version of 9

==> Context: Bad node spec: input: "262" input: "281" input: "282" output: "283" name: "DetectionOutput_177" op_type: "DetectionOutput" attribute { name: "background_label_id" i: 0 type: INT } attribute { name: "code_type" s: "CENTER_SIZE" type: STRING } attribute { name: "confidence_threshold" f: 0.01 type: FLOAT } attribute { name: "keep_top_k" i: 200 type: INT } attribute { name: "nms_threshold" f: 0.45 type: FLOAT } attribute { name: "num_classes" i: 21 type: INT } attribute { name: "share_location" i: 1 type: INT } attribute { name: "top_k" i: 200 type: INT } attribute { name: "variance_encoded_in_target" i: 0 type: INT }

dkurt · 2020-06-09T18:01:36Z

@LucasVandroux, thanks for feedback! That looks very strage. Can you please try ONNX of version 1.6.0?

LucasVandroux · 2020-06-09T19:53:10Z

@dkurt Thanks for your fast answer! I have the same problem with ONNX 1.6.0. Here is the full output:

python export_to_onnx.py --model ../models/ssd300_mAP_77.43_v2.pth

/mnt/c/Users/lucas/Documents/ssd.pytorch/ssd.py:34: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  self.priors = Variable(self.priorbox.forward(), volatile=True)
/mnt/c/Users/lucas/Documents/ssd.pytorch/layers/functions/detection.py:41: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  prior_data = prior_data[:prior_data.shape[0] // 2]
/mnt/c/Users/lucas/Documents/ssd.pytorch/layers/functions/detection.py:50: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  for i in range(num):
/mnt/c/Users/lucas/Documents/ssd.pytorch/layers/functions/detection.py:58: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if scores.size(0) == 0:
Traceback (most recent call last):
  File "export_to_onnx.py", line 43, in <module>
    save_model(net, input, args.output)
  File "export_to_onnx.py", line 24, in save_model
    onnx_model_pb = export_to_string(model, input)
  File "export_to_onnx.py", line 19, in export_to_string
    torch.onnx.export(model, inputs, f, export_params=True, opset_version=version)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/mnt/c/Users/lucas/Documents/ssd.pytorch/.env/lib/python3.6/site-packages/torch/onnx/utils.py", line 515, in _export
    _check_onnx_proto(proto)
RuntimeError: No Op registered for DetectionOutput with domain_version of 9

==> Context: Bad node spec: input: "262" input: "281" input: "282" output: "283" name: "DetectionOutput_177" op_type: "DetectionOutput" attribute { name: "background_label_id" i: 0 type: INT } attribute { name: "code_type" s: "CENTER_SIZE" type: STRING } attribute { name: "confidence_threshold" f: 0.01 type: FLOAT } attribute { name: "keep_top_k" i: 200 type: INT } attribute { name: "nms_threshold" f: 0.45 type: FLOAT } attribute { name: "num_classes" i: 21 type: INT } attribute { name: "share_location" i: 1 type: INT } attribute { name: "top_k" i: 200 type: INT } attribute { name: "variance_encoded_in_target" i: 0 type: INT }

The full list of python packages I am using is as follow:

pip list

future (0.18.2)
numpy (1.18.5)
onnx (1.6.0)
opencv-python (4.2.0.34)
Pillow (7.1.2)
pip (9.0.1)
pkg-resources (0.0.0)
protobuf (3.12.2)
setuptools (39.0.1)
six (1.15.0)
torch (1.5.0+cpu)
torchvision (0.6.0+cpu)
typing-extensions (3.7.4.2)

And here the end of my git log:

git log

commit 1c02c5f68a325a3f8e5ec27341c778b4cf0d59af (HEAD -> pr/462)
Author: Dmitry Kurtaev <dmitry.kurtaev+github@gmail.com>
Date:   Sat Mar 28 18:06:52 2020 +0300

    ONNX export compatible with OpenCV

commit 5b0b77faa955c1917b0c710d770739ba8fbff9b7 (origin/master, origin/HEAD, master)
Author: Degroot <alexdg@amazon.com>
Date:   Mon Mar 11 14:43:06 2019 -0700

    Update .dim() to .size(0) in eval.py for compatibility with latest .dim() functionality

commit ad98ca4f83ebd7e690825b8bb067419030bda992
Author: Miguel Morales <mimoralea@gmail.com>
Date:   Mon Mar 11 15:35:33 2019 -0600

    Update README.md (#235)

    Link to webyclip.com is broken

Would some other information help you as well?

dkurt · 2020-06-09T20:07:46Z

So, I tried the following steps:

git clone https://github.com/dkurt/ssd.pytorch --branch opencv_support --depth 1
cd ssd.pytorch
wget https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth
python3 export_to_onnx.py --model ssd300_mAP_77.43_v2.pth

with torch==1.4.0 and onnx==1.6.0

It reports some warnings but produces .onnx file

Also tried torch=1.5.0 and can confirm that it throws

RuntimeError: No Op registered for DetectionOutput with domain_version of 9

==> Context: Bad node spec: input: "262" input: "281" input: "282" output: "283" name: "DetectionOutput_177" op_type: "DetectionOutput" attribute { name: "background_label_id" i: 0 type: INT } attribute { name: "code_type" s: "CENTER_SIZE" type: STRING } attribute { name: "confidence_threshold" f: 0.01 type: FLOAT } attribute { name: "keep_top_k" i: 200 type: INT } attribute { name: "nms_threshold" f: 0.45 type: FLOAT } attribute { name: "num_classes" i: 21 type: INT } attribute { name: "share_location" i: 1 type: INT } attribute { name: "top_k" i: 200 type: INT } attribute { name: "variance_encoded_in_target" i: 0 type: INT }

Thanks for note! So, for current state or PR you may try torch 1.4.0 only

LucasVandroux · 2020-06-10T12:45:51Z

@dkurt thank you, the problem is indeed coming from the pytorch version and one needs to use torch==1.4.0. I also needed to apply the fix from #400 .

LucasVandroux · 2020-06-10T16:48:13Z

@dkurt thanks again for providing the code to convert the ONNX model to OpenVINO. I used it, and it worked perfectly with openvino_2020.2.120.

For future developers who would like to use this model in their application, please note that the output label index corresponds to VOC and not to COCO.

LucasVandroux · 2020-06-12T11:09:57Z

For those who might be interested in getting directly the ONNX file and the OpenVINO files, I uploaded them here: link.

TamarVolcani · 2020-12-20T10:01:01Z

where is the file export_to_onnx.py? I don't see it in the repository...

LucasVandroux · 2020-12-20T13:26:54Z

where is the file export_to_onnx.py? I don't see it in the repository...

@TamarVolcani as this merge request hasn't yet been merged into the original repository, you will not find it there but in the fork of the repository: https://github.com/dkurt/ssd.pytorch/blob/opencv_support/export_to_onnx.py

shiyuetianqiang · 2021-01-07T16:09:06Z

So, I tried the following steps:

git clone https://github.com/dkurt/ssd.pytorch --branch opencv_support --depth 1
cd ssd.pytorch
wget https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth
python3 export_to_onnx.py --model ssd300_mAP_77.43_v2.pth

with torch==1.4.0 and onnx==1.6.0

It reports some warnings but produces .onnx file

Also tried torch=1.5.0 and can confirm that it throws

RuntimeError: No Op registered for DetectionOutput with domain_version of 9

==> Context: Bad node spec: input: "262" input: "281" input: "282" output: "283" name: "DetectionOutput_177" op_type: "DetectionOutput" attribute { name: "background_label_id" i: 0 type: INT } attribute { name: "code_type" s: "CENTER_SIZE" type: STRING } attribute { name: "confidence_threshold" f: 0.01 type: FLOAT } attribute { name: "keep_top_k" i: 200 type: INT } attribute { name: "nms_threshold" f: 0.45 type: FLOAT } attribute { name: "num_classes" i: 21 type: INT } attribute { name: "share_location" i: 1 type: INT } attribute { name: "top_k" i: 200 type: INT } attribute { name: "variance_encoded_in_target" i: 0 type: INT }

Thanks for note! So, for current state or PR you may try torch 1.4.0 only

Hi, thanks for your work for converting the SSD to onnx model.
Follow your suggestions, I also adjust my corresponding softwares version, and it does work for converting proposed model like ssd300_mAP_77.43_v2.pth.
But when I employ your repo "https://github.com/dkurt/ssd.pytorch/tree/opencv_support" to train a new ssd model, and employ the convert file, it fails.
The error is shown as follows:
" File "/home/llq/Desktop/ssd/2ssd_another/layers/box_utils.py", line 187, in nms
keep = scores.new(scores.size(0)).zero_().long()
TypeError: expected Float (got Long)"
I have tried to fix it, however, I failed, could you please provide some advice?

abuvaneswari · 2021-01-13T03:35:53Z

Hello,

Thank you for the script to convert the model and the instructions. I am able to do the conversion to ONNX (opsets 9, 10 and 11) as well as to Intel IR formats.

However, I am not able to inference the converted ONNX models (opsets 9, 10, 11) with onnxruntime. I tried both 1.4.0 and 1.6.0 versions of onnxruntime.

Here is the error message that I get:
File "/home/buvana/base_ort.py", line 92, in
m = ONNXNet(model)
File "/home/buvana/base_ort.py", line 10, in init
self.ort_session_f = ort.InferenceSession(onnx_path_f)
File "/home/buvana/venv/lib64/python3.6/site-packages/onnxruntime/capi/session.py", line 158, in init
self._load_model(providers or [])
File "/home/buvana/venv/lib64/python3.6/site-packages/onnxruntime/capi/session.py", line 177, in _load_model
self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node: : No Op registered for DetectionOutput with domain_version of 10

Please provide your suggestions.

dkurt · 2021-01-13T06:42:28Z

@abuvaneswari, Proposed solution won't work with ONNXRuntime. It's only for OpenCV and OpenVINO. Both of them can import ONNX model directly.

mohadese-yousefi · 2021-04-19T08:44:48Z

So, I tried the following steps:
git clone https://github.com/dkurt/ssd.pytorch --branch opencv_support --depth 1
cd ssd.pytorch
wget https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth
python3 export_to_onnx.py --model ssd300_mAP_77.43_v2.pth
with torch==1.4.0 and onnx==1.6.0
It reports some warnings but produces .onnx file
Also tried torch=1.5.0 and can confirm that it throws
RuntimeError: No Op registered for DetectionOutput with domain_version of 9

==> Context: Bad node spec: input: "262" input: "281" input: "282" output: "283" name: "DetectionOutput_177" op_type: "DetectionOutput" attribute { name: "background_label_id" i: 0 type: INT } attribute { name: "code_type" s: "CENTER_SIZE" type: STRING } attribute { name: "confidence_threshold" f: 0.01 type: FLOAT } attribute { name: "keep_top_k" i: 200 type: INT } attribute { name: "nms_threshold" f: 0.45 type: FLOAT } attribute { name: "num_classes" i: 21 type: INT } attribute { name: "share_location" i: 1 type: INT } attribute { name: "top_k" i: 200 type: INT } attribute { name: "variance_encoded_in_target" i: 0 type: INT }
Thanks for note! So, for current state or PR you may try torch 1.4.0 only
Hi, thanks for your work for converting the SSD to onnx model.
Follow your suggestions, I also adjust my corresponding softwares version, and it does work for converting proposed model like ssd300_mAP_77.43_v2.pth.
But when I employ your repo "https://github.com/dkurt/ssd.pytorch/tree/opencv_support" to train a new ssd model, and employ the convert file, it fails.
The error is shown as follows:
" File "/home/llq/Desktop/ssd/2ssd_another/layers/box_utils.py", line 187, in nms
keep = scores.new(scores.size(0)).zero_().long()
TypeError: expected Float (got Long)"
I have tried to fix it, however, I failed, could you please provide some advice?

I converted this line:
keep = scores.new(scores.size(0)).zero_().long()
to
keep = torch.zeros(scores.size(), dtype=torch.int64)
and solved.

ONNX export compatible with OpenCV

1c02c5f

dkurt mentioned this pull request Mar 28, 2020

Enable ONNX SSD from https://github.com/amdegroot/ssd.pytorch opencv/opencv#16925

Merged

dkurt mentioned this pull request Aug 27, 2020

readNetFromONNX: Cant create layer of type 'Exp' in getLayerInstance opencv/opencv#18088

Closed

dkurt mentioned this pull request Nov 27, 2020

SSD DNN - PriorBox layers are not working opencv/opencv#18950

Closed

4 tasks

fengyuentau mentioned this pull request Mar 25, 2021

Any plans to officially support the 'NonMaxSuppression' operator from ONNX? opencv/opencv#19779

Open

4 tasks

dkurt closed this Apr 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ONNX export compatible with OpenCV #462

ONNX export compatible with OpenCV #462

dkurt commented Mar 28, 2020

luoduo21 commented Mar 30, 2020

dkurt commented Mar 30, 2020

luoduo21 commented Mar 31, 2020 •

edited

dkurt commented Apr 1, 2020

luoduo21 commented Apr 2, 2020

LucasVandroux commented Jun 9, 2020

dkurt commented Jun 9, 2020

LucasVandroux commented Jun 9, 2020 •

edited

dkurt commented Jun 9, 2020

LucasVandroux commented Jun 10, 2020

LucasVandroux commented Jun 10, 2020

LucasVandroux commented Jun 12, 2020

TamarVolcani commented Dec 20, 2020

LucasVandroux commented Dec 20, 2020 •

edited

shiyuetianqiang commented Jan 7, 2021

abuvaneswari commented Jan 13, 2021

dkurt commented Jan 13, 2021 •

edited

mohadese-yousefi commented Apr 19, 2021

ONNX export compatible with OpenCV #462

ONNX export compatible with OpenCV #462

Conversation

dkurt commented Mar 28, 2020

Convert to ONNX

Run with OpenCV

Or run with OpenVINO

luoduo21 commented Mar 30, 2020

dkurt commented Mar 30, 2020

luoduo21 commented Mar 31, 2020 • edited

dkurt commented Apr 1, 2020

luoduo21 commented Apr 2, 2020

LucasVandroux commented Jun 9, 2020

dkurt commented Jun 9, 2020

LucasVandroux commented Jun 9, 2020 • edited

dkurt commented Jun 9, 2020

LucasVandroux commented Jun 10, 2020

LucasVandroux commented Jun 10, 2020

LucasVandroux commented Jun 12, 2020

TamarVolcani commented Dec 20, 2020

LucasVandroux commented Dec 20, 2020 • edited

shiyuetianqiang commented Jan 7, 2021

abuvaneswari commented Jan 13, 2021

dkurt commented Jan 13, 2021 • edited

mohadese-yousefi commented Apr 19, 2021

luoduo21 commented Mar 31, 2020 •

edited

LucasVandroux commented Jun 9, 2020 •

edited

LucasVandroux commented Dec 20, 2020 •

edited

dkurt commented Jan 13, 2021 •

edited