New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Systematic testing enhancements #4161
base: main
Are you sure you want to change the base?
Conversation
" Use a specific seed value for systematic testing.\n" \ | ||
" --ponysystematictestingverbosity\n" \ | ||
" Control verbosity of systematic testing output.\n" \ | ||
" 0 is least verbose; greater is more details.\n" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to mention here what the highest meaningful value this could be is.
Is it 3? 5? 9? 99? 999?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now the max value used is 10
(for no particular reason) but i haven't figured out how the levels of details should work and how far they should go (any ideas?). i expect that the levels of detail is something that will eventually get well defined and stabilized over time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have opinions about how high the levels of verbosity should go - whatever you have now (e.g. 10) is probably fine.
I just think that this documentation string should mention that maximum number so that I (as a user seeing this documentation string) don't need to search through the whole codebase to find out what the highest meaningful number is.
44c3b22
to
c237129
Compare
rebased to fix merge conflicts |
Update SYSTEMATIC_TESTING_PRINTF to take a level argument. Add cli argument for controlling output level of systematic testing printf's. Add SYSTEMATIC_TESTING_YIELD and SYSTEMATIC_TESTING_PRINTF in a lot more places.
c237129
to
7f9a122
Compare
rebased to fix merge conflicts |
Once @jemc's comment about max value is addressed, I'm ok with these changes especially given that this is a new feature that I haven't used and I trust that @dipinhora is doing the right thing. |
Update SYSTEMATIC_TESTING_PRINTF to take a level argument.
Add cli argument for controlling output level of systematic testing printf's.
Add SYSTEMATIC_TESTING_YIELD and SYSTEMATIC_TESTING_PRINTF in a lot more places.