-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rough draft of scene qa script #1971
Draft
jrreyna
wants to merge
31
commits into
main
Choose a base branch
from
qa-scenes
base: main
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
+1,485
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
Do not delete this pull request or issue due to inactivity.
label
Dec 20, 2022
jturner65
force-pushed
the
qa-scenes
branch
3 times, most recently
from
March 23, 2023 13:37
a0a3104
to
2d9499b
Compare
jturner65
force-pushed
the
qa-scenes
branch
4 times, most recently
from
April 3, 2023 15:44
457d833
to
d1b3789
Compare
jturner65
force-pushed
the
qa-scenes
branch
3 times, most recently
from
April 5, 2023 15:49
91e4d2a
to
f7be4e2
Compare
jturner65
force-pushed
the
qa-scenes
branch
2 times, most recently
from
April 13, 2023 13:25
0ece367
to
c915125
Compare
…ead of specifying on the command line when running the qa_scene.py script. Also added colored print statements to make the output easier to read. Also added a qa_scene_utils.py file to separate the important processing code from helper functions
…Application so that we can test how fast the rendering is
…abstracted some big chunks of scene/stage processing code into functions, and created a new qa scene json config file
…ate unique filenames for output files
…ofile_scene() from inside the iteratively_test_all_scenes() function. Added ability to specify initial transform of agent. Started writing code to generate a default scene if none exists in the dataset config file
…u specify in the config file. You can press 'c' to watch the collision test occur, or if you don't run the app with the viewer on, it will run all the tests specified in the config file you want to test. Refactoring some of the functions in qa_scenes.py, and abstracting some code snippets into separate methods bc some of the functions were getting long and difficult to read. Also added code to time how long the collision test takes at each position
…er times to the comprehensive json
…over a given number of seconds. It saves the results in the comprehensive json file
…object sleep test
…aries for each scene
…till work when some of the tests aren't run, e.g. if we are running the render test, but not the collision nor sleep tests
…Now runs on both stages and scenes
…nd path before it
…some config files and updated the old ones
…better crash tracking
…ig options for reduced metric output. Minor cleanup.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Introduces a tool for automated SceneDataset performance metric analysis.
NOTE : Requires colorama library
To use the tool:
data/
directory.tools/qa_scenes/configs/
overriding default values fromtools/qa_scenes/qa_scene_settings.py
python tools/qa_scenes/qa_scenes.py --config_file_path path/to/my.qa_scene_config.json
Results will print to terminal and images/json/csv will output to
tools/qa_scenes/qa_scenes_output/
Features:
tools/qa_scenes/qa_scene_settings.py
How Has This Been Tested
Locally
Types of changes
Checklist