[test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 #3886

ksghaier · 2022-08-18T15:05:39Z

This commit adds a compiled tvm model and two gst test commands

Sample model (mobilenet_v1_0.75_224_{arch}.so) is compiled from public
tflite model as follow:

Get the model from:
http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_0.75_224_quant.tgz
Follow the steps described in:
https://tvm.apache.org/docs/how_to/compile_models/from_tflite.html

Signed-off-by: Aymen Sghaier aymen.sghaier@nxp.com

[Template] PR Description

In general, github system automatically copies your commit message for your convenience.
Please remove unused part of the template after writing your own PR description with this template.

$ git commit -s filename1 filename2 ... [enter]

Summarize changes in around 50 characters or less

More detailed explanatory text, if necessary. Wrap it to about 72
characters or so. In some contexts, the first line is treated as the
subject of the commit and the rest of the text as the body. The
blank line separating the summary from the body is critical;
various tools like `log`, `shortlog` and `rebase` can get confused 
if you run the two together.

Further paragraphs come after blank lines.

**Changes proposed in this PR:**
- Bullet points are okay, too
- Typically a hyphen or asterisk is used for the bullet, preceded
  by a single space, with blank lines in between, but conventions vary here.

Resolves: #123
See also: #456, #789

**Self evaluation:**
1. Build test: [ ]Passed [ ]Failed [*]Skipped
2. Run test: [ ]Passed [ ]Failed [*]Skipped

**How to evaluate:**
1. Describe how to evaluate in order to be reproduced by reviewer(s).

Add signed-off message automatically by running **$git commit -s ...** command.

$ git push origin <your_branch_name>

taos-ci · 2022-08-18T15:05:42Z

📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #3886. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

myungjoo · 2022-08-19T00:43:31Z

tests/nnstreamer_filter_tvm/runTest.sh

+PATH_TO_IMAGE="../test_models/data/orange.png"
+PATH_TO_CLASS="class.out.log"
+
+gstTest "--gst-plugin-path=${PATH_TO_PLUGIN} filesrc location=${PATH_TO_IMAGE} ! pngdec ! videoscale ! imagefreeze ! videoconvert ! video/x-raw,format=RGB,framerate=0/1 ! tensor_converter ! tensor_filter framework=tvm model=${PATH_TO_MODEL} custom=device=CPU,num_input_tensors:1 ! tensor_decoder mode=image_labeling option1=${PATH_TO_LABEL} ! filesink location=${PATH_TO_CLASS}" 1 0 0 $PERFORMANCE


custom=device=CPU,...
==>
custom=device:CPU...

myungjoo · 2022-08-19T00:43:37Z

tests/nnstreamer_filter_tvm/runTest.sh

+testResult $? 1 "Golden test comparison" 0 1
+
+# Negative test: wrong input type float32 instead of uint8
+gstTest "--gst-plugin-path=${PATH_TO_PLUGIN} filesrc location=${PATH_TO_IMAGE} ! pngdec ! videoscale ! imagefreeze ! videoconvert ! tensor_converter ! tensor_transform mode=arithmetic option=typecast:float32,div:127.5,add:-1.0 ! tensor_filter framework=tvm model=${PATH_TO_MODEL} custom=device=CPU,num_input_tensors:1 ! tensor_decoder mode=image_labeling option1=${PATH_TO_LABEL} ! filesink location=${PATH_TO_CLASS} " 1_n 0 1 $PERFORMANCE


custom=device=CPU,...
==>
custom=device:CPU...

updated, thanks

tests/nnstreamer_filter_tvm/runTest.sh

myungjoo · 2022-08-19T00:49:07Z

If you can supply lighter models for testing, it'd be great.
However, if you are going to keep them, please ignore TAOS/pr-prebuild-filesize — Oooops. File size checker is failed at mobilenet_v1_0.75_224_quant_aarch64.

myungjoo · 2022-08-19T01:13:30Z

[  354s] [Starting] nnstreamer_filter_tvm
[  354s] /home/abuild/rpmbuild/BUILD/nnstreamer-2.1.1/tests/nnstreamer_filter_tvm/runTest.sh: line 67: arch: command not found
[  354s] ==================================================
[  354s]     Test Group nnstreamer_filter_tvm Starts.
[  354s] [FAILED][Critical] 1:gst-launch of case 1
[  354s] [FAILED][Critical] 1:Golden test comparison
[  354s] [PASSED] 1_n:gst-launch of case 1_n
[  354s] ==================================================
[  354s] [FAILED] Test Group nnstreamer_filter_tvm has failed cases (2)

Looks like $(arch) is often not supported in a few Linux distro.
Please find a more universal way to identify the arch.

taos-ci · 2022-08-19T01:19:53Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202208190947010.022424936294556-8d6f2826f3597edcaa391cd0d6ed438faae5da22/.

ksghaier · 2022-08-19T07:37:15Z

S/pr-prebuild-filesi

unfortunately I have no smaller compiled model than this one right now

myungjoo

LGTM

taos-ci · 2022-08-19T08:05:34Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202208191633460.31454110145569-2ad5767436a5feab65f700255a8ec9a79ef57191/.

myungjoo · 2022-08-22T05:47:33Z

Unfortunately, there are failed unit tests (TVM tests. Both in x64 arch. Maybe the x64 binary has some problems?)

[  264s] [Starting] nnstreamer_filter_tvm
[  264s] ==================================================
[  264s]     Test Group nnstreamer_filter_tvm Starts.
[  264s] [FAILED][Critical] 1:gst-launch of case 1
[  264s] [FAILED][Critical] 1:Golden test comparison
[  264s] [PASSED] 1_n:gst-launch of case 1_n
[  264s] ==================================================
[  264s] [FAILED] Test Group nnstreamer_filter_tvm has failed cases (2)
[  264s]

anyj0527 · 2022-08-23T08:27:06Z

Unfortunately, there are failed unit tests (TVM tests. Both in x64 arch. Maybe the x64 binary has some problems?)

May check this: #3412 (comment)
AFAIK, the existing test models (tvm_add_one_arch.so_) are created with it.

taos-ci · 2022-08-26T01:05:25Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202208260935430.74763894081116-2ad5767436a5feab65f700255a8ec9a79ef57191/.

taos-ci · 2022-10-10T15:34:34Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202210110001490.90890288352966-1a2488fa92bf4464814cc89c1a0f8d9baa3fc6f6/.

This commit adds a compiled tvm model and two gst test commands Sample model (mobilenet_v1_0.75_224_{arch}.so) is compiled from public tflite model as follow: - Get the model from: http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_0.75_224_quant.tgz - Follow the steps described in: https://tvm.apache.org/docs/how_to/compile_models/from_tflite.html Supported architectures are armv7l, aarch64 and x86_64 Signed-off-by: Aymen Sghaier <aymen.sghaier@nxp.com>

taos-ci · 2022-10-11T10:24:29Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202210111852150.17939901351929-da5dd81548bf3e930b254a527a5a3465f058b2da/.

taos-ci · 2022-10-12T05:32:56Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202210121359540.48101806640625-da5dd81548bf3e930b254a527a5a3465f058b2da/.

anyj0527 · 2022-10-13T06:18:27Z

@ksghaier @myungjoo
The added testcase get runtime error with current Tizen/Debian tvm package (it's v0.7.0).
However, with the recent tvm release (v0.9.0) and slight modification in our source, I verified those TCs are passed in x86_64 ubuntu local env.
I think after the new TIZEN release in this month, tvm packages should be upgraded. Then this PR could be merged.

myungjoo · 2022-11-16T02:28:56Z

@ksghaier @myungjoo The added testcase get runtime error with current Tizen/Debian tvm package (it's v0.7.0). However, with the recent tvm release (v0.9.0) and slight modification in our source, I verified those TCs are passed in x86_64 ubuntu local env. I think after the new TIZEN release in this month, tvm packages should be upgraded. Then this PR could be merged.

@anyj0527 PTAL if it's now ok.

taos-ci · 2022-11-16T02:47:25Z

cibot: @ksghaier, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nnstreamer/ci/repo-workers/pr-checker/3886-202211161128370.079869031906128-da5dd81548bf3e930b254a527a5a3465f058b2da/.

anyj0527 · 2022-11-16T04:54:03Z

@ksghaier @myungjoo The added testcase get runtime error with current Tizen/Debian tvm package (it's v0.7.0). However, with the recent tvm release (v0.9.0) and slight modification in our source, I verified those TCs are passed in x86_64 ubuntu local env. I think after the new TIZEN release in this month, tvm packages should be upgraded. Then this PR could be merged.

@anyj0527 PTAL if it's now ok.

Tizen tvm package need upgrade (0.7.0 -> 0.9+). I'll make a new issue for this.

ksghaier requested review from a team, myungjoo, jijoongmoon, again4you, jaeyun-jung, leemgs, wooksong, helloahn, kparichay, gichan-jang, anyj0527, lhs8928, songgot, jihochu and niley7464 as code owners August 18, 2022 15:05

myungjoo reviewed Aug 19, 2022

View reviewed changes

tests/nnstreamer_filter_tvm/runTest.sh Show resolved Hide resolved

myungjoo changed the title ~~[test/tvm] Add tvm ssat test for image labeling~~ [test/tvm] Add tvm ssat test for image labeling @open sesame 8/19 09:46 Aug 19, 2022

ksghaier force-pushed the add_tvm_test branch from 8d6f282 to 2ad5767 Compare August 19, 2022 07:33

myungjoo approved these changes Aug 19, 2022

View reviewed changes

myungjoo changed the title ~~[test/tvm] Add tvm ssat test for image labeling @open sesame 8/19 09:46~~ [test/tvm] Add tvm ssat test for image labeling @open sesame 8/26 09:35 Aug 26, 2022

myungjoo added for-next Will not be merged in "main" branch until the next release. and removed for-next Will not be merged in "main" branch until the next release. labels Sep 27, 2022

ksghaier force-pushed the add_tvm_test branch from 2ad5767 to 1a2488f Compare October 10, 2022 15:02

ksghaier force-pushed the add_tvm_test branch from 1a2488f to da5dd81 Compare October 11, 2022 09:52

anyj0527 changed the title ~~[test/tvm] Add tvm ssat test for image labeling @open sesame 8/26 09:35~~ [test/tvm] Add tvm ssat test for image labeling @open sesame 10/12 14:00 Oct 12, 2022

myungjoo changed the title ~~[test/tvm] Add tvm ssat test for image labeling @open sesame 10/12 14:00~~ [test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 #3886

[test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 #3886

ksghaier commented Aug 18, 2022

taos-ci commented Aug 18, 2022

myungjoo Aug 19, 2022

ksghaier Aug 19, 2022

myungjoo Aug 19, 2022

ksghaier Aug 19, 2022

myungjoo commented Aug 19, 2022

myungjoo commented Aug 19, 2022 •

edited

taos-ci commented Aug 19, 2022

ksghaier commented Aug 19, 2022

myungjoo left a comment

taos-ci commented Aug 19, 2022

myungjoo commented Aug 22, 2022 •

edited

anyj0527 commented Aug 23, 2022

taos-ci commented Aug 26, 2022

taos-ci commented Oct 10, 2022

taos-ci commented Oct 11, 2022

taos-ci commented Oct 12, 2022

anyj0527 commented Oct 13, 2022

myungjoo commented Nov 16, 2022

taos-ci commented Nov 16, 2022

anyj0527 commented Nov 16, 2022 •

edited

[test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 #3886

Are you sure you want to change the base?

[test/tvm] Add tvm ssat test for image labeling @open sesame 11/16 11:00 #3886

Conversation

ksghaier commented Aug 18, 2022

[Template] PR Description

taos-ci commented Aug 18, 2022

myungjoo Aug 19, 2022

Choose a reason for hiding this comment

ksghaier Aug 19, 2022

Choose a reason for hiding this comment

myungjoo Aug 19, 2022

Choose a reason for hiding this comment

ksghaier Aug 19, 2022

Choose a reason for hiding this comment

myungjoo commented Aug 19, 2022

myungjoo commented Aug 19, 2022 • edited

taos-ci commented Aug 19, 2022

ksghaier commented Aug 19, 2022

myungjoo left a comment

Choose a reason for hiding this comment

taos-ci commented Aug 19, 2022

myungjoo commented Aug 22, 2022 • edited

anyj0527 commented Aug 23, 2022

taos-ci commented Aug 26, 2022

taos-ci commented Oct 10, 2022

taos-ci commented Oct 11, 2022

taos-ci commented Oct 12, 2022

anyj0527 commented Oct 13, 2022

myungjoo commented Nov 16, 2022

taos-ci commented Nov 16, 2022

anyj0527 commented Nov 16, 2022 • edited

myungjoo commented Aug 19, 2022 •

edited

myungjoo commented Aug 22, 2022 •

edited

anyj0527 commented Nov 16, 2022 •

edited