New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add flaky test diagnostic script and fix flaky test #734
Conversation
samples/snippets/helpers.py
Outdated
|
||
def flaky_test_diagnostic(file_name, test_name, N=20): | ||
|
||
import pytest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the only non-standard-library dependency for this helper function, so I am hiding the import in here so that pytest does not become a dependency for the whole module. I can also inject the module as an arg, or not worry about pytest as a dependency of the module at put it top-level.
@@ -34,6 +34,9 @@ def teardown(teardown_training_pipeline): | |||
|
|||
def test_ucaip_generated_create_training_pipeline_sample(capsys, shared_state): | |||
|
|||
# The return of the cancellation can be flaky; max of 20 runs was 215 sec. | |||
shared_state["cancel_batch_prediction_job_timeout"] = 300 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Max execution time was 215 seconds out of 20 runs xref: #580 (comment)
26679e9
to
c5afe2c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good - thanks Nick!
Fixes ##580