Skip to content
This repository has been archived by the owner on Feb 23, 2024. It is now read-only.

Fix: #254 by increasing timeout, use backoff module instead of flaky #271

Merged
merged 5 commits into from Oct 11, 2021

Conversation

nicain
Copy link
Contributor

@nicain nicain commented Oct 7, 2021

Here I have moved away from "flaky" in favor of "backoff", which was already a dependency of the library.

This repository currently uses the "flaky" module/plugin to pytest to allow retries on tests. As I was trying to quantify this flakiness, I quickly realized that under-the-hood, this module uses runtime modification (monkey-patching) of the usual pytest callbacks, which breaks the interface contracts that the pytest ecosystem subscribes to and does not let me aggregate the timing statistics I need for quantifying flakiness.

The "backoff" module is already in this repo. In this PR I modify the "usual" way we set up fixtures to use the callback system from "backoff".

Fixes #254 by increasing timeout via backoff with 4 tries: [100, 250, 300, 500]

The worst execution time was 234 seconds, so the max allowed is now 500:
Flakiness report:

SUCCESS
       when    duration outcome  trial
0     setup    1.524859  passed      0
1      call  141.551338  passed      0
2  teardown    0.000545  passed      0
       when    duration outcome  trial
0     setup    1.418313  passed      1
1      call  234.604837  passed      1
2  teardown    0.000532  passed      1
       when    duration outcome  trial
0     setup    1.519378  passed      2
1      call  124.333620  passed      2
2  teardown    0.000748  passed      2
       when    duration outcome  trial
0     setup    1.569923  passed      3
1      call  129.542894  passed      3
2  teardown    0.000723  passed      3
       when    duration outcome  trial
0     setup    1.459673  passed      4
1      call  125.115383  passed      4
2  teardown    0.000780  passed      4
       when   duration outcome  trial
0     setup   1.515023  passed      5
1      call  79.950361  passed      5
2  teardown   0.000626  passed      5
       when   duration outcome  trial
0     setup   1.526374  passed      6
1      call  82.278016  passed      6
2  teardown   0.001218  passed      6
       when    duration outcome  trial
0     setup    1.533906  passed      7
1      call  107.097216  passed      7
2  teardown    0.000794  passed      7
       when    duration outcome  trial
0     setup    1.417850  passed      8
1      call  136.093182  passed      8
2  teardown    0.000509  passed      8
       when    duration outcome  trial
0     setup    1.433857  passed      9
1      call  134.896833  passed      9
2  teardown    0.000841  passed      9

@product-auto-label product-auto-label bot added the api: translate Issues related to the googleapis/python-translate API. label Oct 7, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Oct 7, 2021
@nicain nicain closed this Oct 7, 2021
@nicain nicain reopened this Oct 7, 2021
@nicain nicain marked this pull request as ready for review October 7, 2021 20:03
@nicain nicain requested review from telpirion and a team as code owners October 7, 2021 20:03
@nicain nicain requested a review from leahecole October 7, 2021 20:03
@nicain nicain changed the title feat: use backoff module instead of flaky Fix: #254 by increasing timeout, use backoff module instead of flaky Oct 8, 2021
@nicain nicain requested a review from leahecole October 8, 2021 19:07
@nicain nicain added the automerge Merge the pull request once unit tests and other checks pass. label Oct 11, 2021
@nicain
Copy link
Contributor Author

nicain commented Oct 11, 2021

@aribray Ping, PTAL?

Copy link
Contributor

@aribray aribray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@gcf-merge-on-green
Copy link
Contributor

Merge-on-green attempted to merge your PR for 6 hours, but it was not mergeable because either one of your required status checks failed, one of your required reviews was not approved, or there is a do not merge label. Learn more about your required status checks here: https://help.github.com/en/github/administering-a-repository/enabling-required-status-checks. You can remove and reapply the label to re-run the bot.

@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Oct 11, 2021
@leahecole leahecole merged commit 0cff0f2 into main Oct 11, 2021
@leahecole leahecole deleted the backoff branch October 11, 2021 22:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api: translate Issues related to the googleapis/python-translate API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

translate_v3_batch_translate_text_with_glossary_test: test_batch_translate_text_with_glossary failed
4 participants