Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: media operation retries can be configured using the same interface as with non-media operation #447

Merged
merged 9 commits into from Jun 11, 2021
41 changes: 41 additions & 0 deletions google/cloud/storage/_helpers.py
Expand Up @@ -23,6 +23,7 @@
import os

from six.moves.urllib.parse import urlsplit
from google import resumable_media
from google.cloud.storage.constants import _DEFAULT_TIMEOUT
from google.cloud.storage.retry import DEFAULT_RETRY
from google.cloud.storage.retry import DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED
Expand All @@ -45,6 +46,12 @@
("if_source_metageneration_not_match", "ifSourceMetagenerationNotMatch"),
)

_NUM_RETRIES_MESSAGE = (
"`num_retries` has been deprecated and will be removed in a future "
"release. Use the `retry` argument with a Retry or ConditionalRetryPolicy "
"object, or None, instead."
)


def _get_storage_host():
return os.environ.get(STORAGE_EMULATOR_ENV_VAR, _DEFAULT_STORAGE_HOST)
Expand Down Expand Up @@ -563,3 +570,37 @@ def _bucket_bound_hostname_url(host, scheme=None):
return host

return "{scheme}://{host}/".format(scheme=scheme, host=host)


def _api_core_retry_to_resumable_media_retry(retry, num_retries=None):
"""Convert google.api.core.Retry to google.resumable_media.RetryStrategy.

Custom predicates are not translated.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this clear to the end user?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have this warning in every single public method docstring, so with that it should be clear.


:type retry: google.api_core.Retry
:param retry: (Optional) The google.api_core.Retry object to translate.

:type num_retries: int
:param num_retries: (Optional) The number of retries desired. This is
supported for backwards compatibility and is mutually exclusive with
`retry`.

:rtype: google.resumable_media.RetryStrategy
:returns: A RetryStrategy with all applicable attributes copied from input,
or a RetryStrategy with max_retries set to 0 if None was input.
"""

if retry is not None and num_retries is not None:
raise ValueError("num_retries and retry arguments are mutually exclusive")

elif retry is not None:
return resumable_media.RetryStrategy(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like error types are not translatable in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean custom predicates? Yes, those are not translatable. It'll be documented wherever possible.

max_sleep=retry._maximum,
max_cumulative_retry=retry._deadline,
initial_delay=retry._initial,
multiplier=retry._multiplier,
)
elif num_retries is not None:
return resumable_media.RetryStrategy(max_retries=num_retries)
else:
return resumable_media.RetryStrategy(max_retries=0)