Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(storage): anonymous credentials for private bucket #107

Merged
merged 12 commits into from Apr 25, 2020
Merged

fix(storage): anonymous credentials for private bucket #107

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3204edc
fix(storage): anonymous credentials for private bucket
HemangChothani Apr 15, 2020
134066e
fix(storage): use API_BASE_URL for hostname
HemangChothani Apr 16, 2020
85729b7
Merge branch 'master' of https://github.com/googleapis/python-storage…
HemangChothani Apr 16, 2020
dbd74fa
fix(storage): nit
HemangChothani Apr 17, 2020
1d1ae94
Merge branch 'master' of https://github.com/googleapis/python-storage…
HemangChothani Apr 17, 2020
ec45809
fix(storage): fix system test
HemangChothani Apr 20, 2020
95f12c4
Merge branch 'master' into storage_issue_102
frankyn Apr 20, 2020
6cc573b
Merge branch 'master' into storage_issue_102
frankyn Apr 22, 2020
c293e2f
Merge branch 'master' of https://github.com/googleapis/python-storage…
HemangChothani Apr 23, 2020
0588395
Merge branch 'storage_issue_102' of https://github.com/q-logic/python…
HemangChothani Apr 23, 2020
6e8213b
fix(storage): revert changes
HemangChothani Apr 24, 2020
dc32e92
Merge branch 'master' into storage_issue_102
frankyn Apr 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 14 additions & 0 deletions google/cloud/storage/client.py
View file
Open in desktop
Expand Up @@ -40,6 +40,8 @@
from google.cloud.storage.batch import Batch
from google.cloud.storage.bucket import Bucket
from google.cloud.storage.blob import Blob
from google.cloud.storage import blob
from google.cloud.storage import bucket
from google.cloud.storage.hmac_key import HMACKeyMetadata
from google.cloud.storage.acl import BucketACL
from google.cloud.storage.acl import DefaultObjectACL
Expand Down Expand Up @@ -122,6 +124,7 @@ def __init__(
if client_options.api_endpoint:
api_endpoint = client_options.api_endpoint
kw_args["api_endpoint"] = api_endpoint
_override_private_url_variables(api_endpoint)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. I think the code needs to be refactored for maintainability but the this is a good start.

The URLs in URL Templates could be refactored to be pulled in from the client which is passed around the library.

For example:

python-storage/google/cloud/storage/blob.py

Line 1020 in a50cdd1

base_url = _MULTIPART_URL_TEMPLATE.format(bucket_path=self.bucket.path)

Could be updated to:

_BASE_UPLOAD_TEMPLATE =  u"{hostname}/upload/storage/v1{bucket_path}/o?uploadType="
_MULTIPART_URL_TEMPLATE = _BASE_UPLOAD_TEMPLATE + u"multipart"
# truncated....
base_url = _MULTIPART_URL_TEMPLATE.format(hostname=storage_client._connection.API_BASE_URL,bucket_path=self.bucket.path)

WDYT?


if no_project:
self.project = None
Expand Down Expand Up @@ -1056,3 +1059,14 @@ def _item_to_hmac_key_metadata(iterator, item):
metadata = HMACKeyMetadata(iterator.client)
metadata._properties = item
return metadata


def _override_private_url_variables(api_endpoint):
blob._DOWNLOAD_URL_TEMPLATE = api_endpoint + u"/download/storage/v1{path}?alt=media"
blob._BASE_UPLOAD_TEMPLATE = (
api_endpoint + u"/upload/storage/v1{bucket_path}/o?uploadType="
)
blob._MULTIPART_URL_TEMPLATE = blob._BASE_UPLOAD_TEMPLATE + u"multipart"
blob._RESUMABLE_URL_TEMPLATE = blob._BASE_UPLOAD_TEMPLATE + u"resumable"
blob._API_ACCESS_ENDPOINT = api_endpoint
bucket._API_ACCESS_ENDPOINT = api_endpoint
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These URLs are used for Signature based requests like Signed URLs. These can be left alone because the method has a way to update the hostname because they don't use the same API (Download and Upload use JSON while Signature requests use XML API).

14 changes: 12 additions & 2 deletions tests/unit/test_client.py
View file
Open in desktop
Expand Up @@ -137,17 +137,27 @@ def test_ctor_w_empty_client_options(self):
)

def test_ctor_w_client_options_dict(self):
from google.cloud.storage import blob

PROJECT = "PROJECT"
credentials = _make_credentials()
client_options = {"api_endpoint": "https://www.foo-googleapis.com"}
api_endpoint = "https://www.foo-googleapis.com"
client_options = {"api_endpoint": api_endpoint}

client = self._make_one(
project=PROJECT, credentials=credentials, client_options=client_options
)

self.assertEqual(client._connection.API_BASE_URL, api_endpoint)

self.assertEqual(blob._API_ACCESS_ENDPOINT, api_endpoint)
self.assertEqual(
client._connection.API_BASE_URL, "https://www.foo-googleapis.com"
blob._DOWNLOAD_URL_TEMPLATE,
api_endpoint + u"/download/storage/v1{path}?alt=media",
)
self.assertEqual(
blob._BASE_UPLOAD_TEMPLATE,
api_endpoint + u"/upload/storage/v1{bucket_path}/o?uploadType=",
)

def test_ctor_w_client_options_object(self):
Expand Down