tests: add retry conformance test framework and test always idempotent operations #450
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
product-auto-label
bot
added
the
api: storage
google-cla
bot
added
the
cla: yes
…ge into retry-conf-tests
tritone
reviewed
Jun 24, 2021
andrewsg
requested changes
Jul 16, 2021
| notification, | ||
| hmac_key, | ||
| ) | ||
| except Exception as e: |
There was a problem hiding this comment.
Why catch and log this instead of just letting the exception bubble up and have pytest deal with it?
There was a problem hiding this comment.
Catching the exception here helps verify whether or not the test ran successfully, with a variable success_results. success_results is compared against expect_success from the schema.
For test scenarios 3 and 4, the tests are expected to fail, hence why we're catching the exception.
There was a problem hiding this comment.
I see-- would the test logs be filled with all these exceptions?
There was a problem hiding this comment.
Yep the exceptions here are caught and logged.
tritone
reviewed
Aug 4, 2021
| notification, | ||
| hmac_key, | ||
| ) | ||
| except Exception as e: |
There was a problem hiding this comment.
I see-- would the test logs be filled with all these exceptions?
tritone
approved these changes
Aug 18, 2021
andrewsg
approved these changes
Aug 19, 2021
cojenco
changed the title
cojenco
added a commit
to cojenco/python-storage
that referenced
this pull request
Oct 13, 2021
…t operations (googleapis#450) * initial tests for retry strategy conformance tests implementation * revise helper methods to interact with Retry Test API * add helper to delete retry test * handle exceptions in helper methods * remove try except * change payload key to instructions to align emulator change * rename and revise to loop through each case and method * wip populate resources to conf tests * add logic to populate fixture resources * add helper method to populate fixture hmacy key * revise endpoints and 2 clients * add assertions to testdata scenarios * add logic for preconditions wip * add mapping scenarios and formatting for readability * add library methods to method invocation mapping and json * lint * refactor using **kwargs * remove unused module and fix lint * handle misused arguments following style guide * handle unused arguments * wip: add library methods to mapping and json * add client_options to resource populating client * log warnings and revise try except blocks * update schema for S1 and S2 * fix lint and mark error * move retry conformance tests to separate folder * update noxfile * relocate conformance tests * add S1 S2 tests and delete json file * lint and clean comments * add S2 object library methods * address comments * change test parametrization to separate test cases and address comments * add assertion message and display library method name * add multiple lib methods and revise assertion message * add S2 entry library methods after emulator fix * address comments * revise test case structure using python globals * revise library methods naming to start with class name * unify library method signatures * cleanup code * use pytest fixtures to populate resources * address comments and update docstrings * address comments. change to use anonymous credentials * update descriptions
cojenco
added a commit
to cojenco/python-storage
that referenced
this pull request
Oct 13, 2021
…t operations (googleapis#450) * initial tests for retry strategy conformance tests implementation * revise helper methods to interact with Retry Test API * add helper to delete retry test * handle exceptions in helper methods * remove try except * change payload key to instructions to align emulator change * rename and revise to loop through each case and method * wip populate resources to conf tests * add logic to populate fixture resources * add helper method to populate fixture hmacy key * revise endpoints and 2 clients * add assertions to testdata scenarios * add logic for preconditions wip * add mapping scenarios and formatting for readability * add library methods to method invocation mapping and json * lint * refactor using **kwargs * remove unused module and fix lint * handle misused arguments following style guide * handle unused arguments * wip: add library methods to mapping and json * add client_options to resource populating client * log warnings and revise try except blocks * update schema for S1 and S2 * fix lint and mark error * move retry conformance tests to separate folder * update noxfile * relocate conformance tests * add S1 S2 tests and delete json file * lint and clean comments * add S2 object library methods * address comments * change test parametrization to separate test cases and address comments * add assertion message and display library method name * add multiple lib methods and revise assertion message * add S2 entry library methods after emulator fix * address comments * revise test case structure using python globals * revise library methods naming to start with class name * unify library method signatures * cleanup code * use pytest fixtures to populate resources * address comments and update docstrings * address comments. change to use anonymous credentials * update descriptions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Retry strategies are included in the client library when it's safe to do so. These retry strategies are complex and need automated integration tests to ensure they work and continue to work in the future.
This adds retry strategy conformance tests that are expected to run against a local GCS API emulator and validate correct behavior around retryable errors, idempotency considerations and preconditions. This PR includes the tests and schema json for:
Tests are loaded by the json directly. An individual test case is generated for each scenario ID, instruction set, method, and method_invocation_mapping_library_method). Test names are displayed as
test-S{scenario_id}-{method}-{client-library-method-name}, for example,test-S1-storage.buckets.insert-bucket_create