fix not-working ort-nightly backend-test #52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR changes the intention of the development version on the Scoreboard. If there is no dev version of ONNX Runtime, then we should probably just remove it from the Scoreboard, rather than testing a stable version of ORT against a development version of ONNX. |
I tried to use the ort-nightly from https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly |
|
I changed from stable ort to ort-nightly. Should we use onnx-stable here again? |
p-wysocki
approved these changes
May 25, 2023
| RUN pip3 install onnx | ||
| RUN pip3 install ort-nightly | ||
| RUN pip3 install onnx-weekly | ||
| RUN pip3 install ort-nightly --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ |
There was a problem hiding this comment.
Should this link be inaccessible? It says PublicPackages in it, but I'm getting The user <user> is not authorized to access when trying to go to that link. Otherwise, as long as it's being updated with latest nightly releases, LGTM.
There was a problem hiding this comment.
What do you mean with PublicPackages?
executing "pip3 install ort-nightly --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple" at the cli worked me
There was a problem hiding this comment.
As we now have results at http://onnx.ai/backend-scoreboard/index_dev.html I think its ready to merge... I'm only a bit surprised as onnx-weekly is not working, but ort-nightly from https://aiinfra.pkgs.visualstudio.com is used
There was a problem hiding this comment.
What do you mean with PublicPackages?
Nevermind, it's okay.
I'm only a bit surprised as onnx-weekly is not working
It seems it's to be expected: onnx/onnx#5233. I also think this can be merged.
There was a problem hiding this comment.
the packages from testpypi are removed. they are uploaed to pypi directly.
There was a problem hiding this comment.
can we merge? Who has the right's?
postrational
requested changes
Jul 24, 2023
There was a problem hiding this comment.
Please run the following commands to test your PR:
docker build -t scoreboard/ort-dev -f runtimes/onnx-runtime/development/Dockerfile .
docker run --env-file setup/env.list scoreboard/ort-dev
At this stage I'm not able to build the image and I'm getting the following error:
#13 724.2 ERROR: Cannot install ort-nightly==1.12.1.dev20220801005, ort-nightly==1.13.0.dev20220802003, ort-nightly==1.13.0.dev20220803001, ort-nightly==1.13.0.dev20220803005, ort-nightly==1.13.0.dev20220804003, ort-nightly==1.13.0.dev20220805003, ort-nightly==1.13.0.dev20220806001, ort-nightly==1.13.0.dev20220808001, ort-nightly==1.13.0.dev20220808004, ort-nightly==1.13.0.dev20220809003, ort-nightly==1.13.0.dev20220811001, ort-nightly==1.13.0.dev20220811002, ort-nightly==1.13.0.dev20220812001, ort-nightly==1.13.0.dev20220812004, ort-nightly==1.13.0.dev20220815003, ort-nightly==1.13.0.dev20220816002, ort-nightly==1.13.0.dev20220817002, ort-nightly==1.13.0.dev20220818007, ort-nightly==1.13.0.dev20220819004, ort-nightly==1.13.0.dev20220822001, ort-nightly==1.13.0.dev20220822004, ort-nightly==1.13.0.dev20220823003, ort-nightly==1.13.0.dev20220830001, ort-nightly==1.13.0.dev20220830004, ort-nightly==1.13.0.dev20220831002, ort-nightly==1.13.0.dev20220901005, ort-nightly==1.13.0.dev20220903001, ort-nightly==1.13.0.dev20220905001, ort-nightly==1.13.0.dev20220906001, ort-nightly==1.13.0.dev20220906004, ort-nightly==1.13.0.dev20220908001, ort-nightly==1.13.0.dev20220913011, ort-nightly==1.13.0.dev20220915001, ort-nightly==1.13.0.dev20220915006, ort-nightly==1.13.0.dev20220916010, ort-nightly==1.13.0.dev20220917005, ort-nightly==1.13.0.dev20220919005, ort-nightly==1.13.0.dev20220922014, ort-nightly==1.13.0.dev20220922015, ort-nightly==1.13.0.dev20220923001, ort-nightly==1.13.0.dev20220923002, ort-nightly==1.13.0.dev20220924004, ort-nightly==1.13.0.dev20220926002, ort-nightly==1.13.0.dev20220927003, ort-nightly==1.13.0.dev20220930006, ort-nightly==1.13.0.dev20221002001, ort-nightly==1.13.0.dev20221003004, ort-nightly==1.13.0.dev20221004002, ort-nightly==1.13.0.dev20221005002, ort-nightly==1.13.0.dev20221005006, ort-nightly==1.13.0.dev20221006001, ort-nightly==1.13.0.dev20221007003, ort-nightly==1.13.0.dev20221008001, ort-nightly==1.13.0.dev20221011001, ort-nightly==1.13.0.dev20221012005, ort-nightly==1.13.0.dev20221013004, ort-nightly==1.13.0.dev20221014003, ort-nightly==1.13.0.dev20221015001, ort-nightly==1.13.0.dev20221016001, ort-nightly==1.13.0.dev20221017001, ort-nightly==1.13.0.dev20221018002, ort-nightly==1.13.0.dev20221019004, ort-nightly==1.13.0.dev20221021002, ort-nightly==1.13.0.dev20221021004, ort-nightly==1.14.0.dev20221022001, ort-nightly==1.14.0.dev20221025002, ort-nightly==1.14.0.dev20221028003, ort-nightly==1.14.0.dev20221028006, ort-nightly==1.14.0.dev20221102003, ort-nightly==1.14.0.dev20221104001, ort-nightly==1.14.0.dev20221104009, ort-nightly==1.14.0.dev20221107010, ort-nightly==1.14.0.dev20221108004, ort-nightly==1.14.0.dev20221109001, ort-nightly==1.14.0.dev20221109003, ort-nightly==1.14.0.dev20221110006, ort-nightly==1.14.0.dev20221111004, ort-nightly==1.14.0.dev20221114003, ort-nightly==1.14.0.dev20221115006, ort-nightly==1.14.0.dev20221116003, ort-nightly==1.14.0.dev20221117004, ort-nightly==1.14.0.dev20221118001, ort-nightly==1.14.0.dev20221122007, ort-nightly==1.14.0.dev20221123005, ort-nightly==1.14.0.dev20221123008, ort-nightly==1.14.0.dev20221127001, ort-nightly==1.14.0.dev20221128003, ort-nightly==1.14.0.dev20221129005, ort-nightly==1.14.0.dev20221130005, ort-nightly==1.14.0.dev20221201004, ort-nightly==1.14.0.dev20221202005, ort-nightly==1.14.0.dev20221203001, ort-nightly==1.14.0.dev20221204002, ort-nightly==1.14.0.dev20221205004, ort-nightly==1.14.0.dev20221206004, ort-nightly==1.14.0.dev20221207005, ort-nightly==1.14.0.dev20221208004, ort-nightly==1.14.0.dev20221209003, ort-nightly==1.14.0.dev20221210001, ort-nightly==1.14.0.dev20221212002, ort-nightly==1.14.0.dev20221213016, ort-nightly==1.14.0.dev20221214001, ort-nightly==1.14.0.dev20230117005, ort-nightly==1.14.0.dev20230118004, ort-nightly==1.14.0.dev20230120003, ort-nightly==1.14.0.dev20230120005, ort-nightly==1.14.0.dev20230121001, ort-nightly==1.14.0.dev20230123004, ort-nightly==1.14.0.dev20230124001, ort-nightly==1.14.0.dev20230125002, ort-nightly==1.15.0.dev20230126004, ort-nightly==1.15.0.dev20230131001, ort-nightly==1.15.0.dev20230202005, ort-nightly==1.15.0.dev20230203006, ort-nightly==1.15.0.dev20230204001, ort-nightly==1.15.0.dev20230205001, ort-nightly==1.15.0.dev20230207001, ort-nightly==1.15.0.dev20230208003, ort-nightly==1.15.0.dev20230210002, ort-nightly==1.15.0.dev20230210003, ort-nightly==1.15.0.dev20230213001, ort-nightly==1.15.0.dev20230214005, ort-nightly==1.15.0.dev20230214008, ort-nightly==1.15.0.dev20230222003, ort-nightly==1.15.0.dev20230228004, ort-nightly==1.15.0.dev20230301002, ort-nightly==1.15.0.dev20230301008, ort-nightly==1.15.0.dev20230302002, ort-nightly==1.15.0.dev20230306002, ort-nightly==1.15.0.dev20230307001, ort-nightly==1.15.0.dev20230308002, ort-nightly==1.15.0.dev20230309002, ort-nightly==1.15.0.dev20230310003, ort-nightly==1.15.0.dev20230310005, ort-nightly==1.15.0.dev20230311001, ort-nightly==1.15.0.dev20230313003, ort-nightly==1.15.0.dev20230314003, ort-nightly==1.15.0.dev20230315002, ort-nightly==1.15.0.dev20230317002, ort-nightly==1.15.0.dev20230318003, ort-nightly==1.15.0.dev20230319001, ort-nightly==1.15.0.dev20230320005, ort-nightly==1.15.0.dev20230324001, ort-nightly==1.15.0.dev20230324006, ort-nightly==1.15.0.dev20230325001, ort-nightly==1.15.0.dev20230327003, ort-nightly==1.15.0.dev20230328003, ort-nightly==1.15.0.dev20230329005, ort-nightly==1.15.0.dev20230330002, ort-nightly==1.15.0.dev20230331007, ort-nightly==1.15.0.dev20230331013, ort-nightly==1.15.0.dev20230403002, ort-nightly==1.15.0.dev20230404003, ort-nightly==1.15.0.dev20230404004, ort-nightly==1.15.0.dev20230405002, ort-nightly==1.15.0.dev20230406007, ort-nightly==1.15.0.dev20230407004, ort-nightly==1.15.0.dev20230408001, ort-nightly==1.15.0.dev20230410004, ort-nightly==1.15.0.dev20230412001, ort-nightly==1.15.0.dev20230425004, ort-nightly==1.15.0.dev20230426002, ort-nightly==1.15.0.dev20230427001, ort-nightly==1.15.0.dev20230428009, ort-nightly==1.15.0.dev20230429003, ort-nightly==1.15.0.dev20230502002, ort-nightly==1.15.0.dev20230502003, ort-nightly==1.15.0.dev20230502004, ort-nightly==1.15.0.dev20230512004, ort-nightly==1.16.0.dev20230504003, ort-nightly==1.16.0.dev20230505007, ort-nightly==1.16.0.dev20230507001, ort-nightly==1.16.0.dev20230508004, ort-nightly==1.16.0.dev20230509001, ort-nightly==1.16.0.dev20230510010, ort-nightly==1.16.0.dev20230511007, ort-nightly==1.16.0.dev20230512006, ort-nightly==1.16.0.dev20230514001, ort-nightly==1.16.0.dev20230515004, ort-nightly==1.16.0.dev20230516003, ort-nightly==1.16.0.dev20230525010, ort-nightly==1.16.0.dev20230526008, ort-nightly==1.16.0.dev20230528001, ort-nightly==1.16.0.dev20230530005, ort-nightly==1.16.0.dev20230531003, ort-nightly==1.16.0.dev20230601002, ort-nightly==1.16.0.dev20230605005, ort-nightly==1.16.0.dev20230607005, ort-nightly==1.16.0.dev20230608006, ort-nightly==1.16.0.dev20230609006, ort-nightly==1.16.0.dev20230610002, ort-nightly==1.16.0.dev20230612006, ort-nightly==1.16.0.dev20230613002, ort-nightly==1.16.0.dev20230614003, ort-nightly==1.16.0.dev20230623004, ort-nightly==1.16.0.dev20230625001, ort-nightly==1.16.0.dev20230626003, ort-nightly==1.16.0.dev20230627004, ort-nightly==1.16.0.dev20230628004, ort-nightly==1.16.0.dev20230629002, ort-nightly==1.16.0.dev20230630005, ort-nightly==1.16.0.dev20230701001, ort-nightly==1.16.0.dev20230703003, ort-nightly==1.16.0.dev20230704003, ort-nightly==1.16.0.dev20230705005, ort-nightly==1.16.0.dev20230706004, ort-nightly==1.16.0.dev20230707004, ort-nightly==1.16.0.dev20230708001, ort-nightly==1.16.0.dev20230710005, ort-nightly==1.16.0.dev20230711005, ort-nightly==1.16.0.dev20230720006, ort-nightly==1.16.0.dev20230721005 and ort-nightly==1.16.0.dev20230722001 because these package versions have conflicting dependencies.
#13 724.2
#13 724.2 The conflict is caused by:
#13 724.2 ort-nightly 1.16.0.dev20230722001 depends on coloredlogs
| @@ -27,8 +27,8 @@ ENV ONNX_BACKEND="onnxruntime.backend.backend" | |||
| RUN locale-gen en_US.UTF-8 && update-locale LANG=en_US.UTF-8 | |||
|
|
|||
| # Install dependencies | |||
| RUN pip3 install onnx | |||
| RUN pip3 install ort-nightly | |||
| RUN pip3 install onnx-weekly | |||
There was a problem hiding this comment.
We should be testing the development version of ONNX Runtime against the stable version of ONNX. We don't want the ONNX tests to be changing before a release.
Suggested change
| RUN pip3 install onnx-weekly | |
| RUN pip3 install onnx |
There was a problem hiding this comment.
updated the docker file according to the comments and tested it.
For me it runs now again:
"======== 144 failed, 1173 passed, 1317 deselected, 1 warnings in 27.82s ========"
Signed-off-by: andife <fehlner@arcor.de>
Signed-off-by: andife <fehlner@arcor.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ort-nightly is not updated any more