Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete GCP namespace support #7836

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

Neon-White
Copy link
Contributor

@Neon-White Neon-White commented Feb 20, 2024

Explain the changes

  1. This PR implements several functions (namely multipart uploads and tag support) required for proper namespace support over Google Cloud Storage.
    I opted to use AWS SDK v3 for posterity, and went with barebones commands to speed things up a bit,
  2. Since GCP only supports multipart uploads via the S3-compatible XML API, it is necessary to generate an HMAC key to use an S3 client with GCP. The PR also takes care of generating the HMAC key, as well as storing it in our DB.

Issues: Fixed #xxx / Gap #xxx

  1. Up until now, our GCP interface supported resumable uploads but not multipart uploads - this PR addresses that gap

Testing Instructions:

  1. Use multipart upload to upload an object with tags to a GCP namespace bucket
  2. Read the uploaded object and compare checksums to verify both write and read were successful
  3. Read the object's tags and compare them to the ones from before the upload
  4. Add additional tags to the object
  5. Make sure the additional tags are present as well

I tested the PR locally with multipart uploads, as well as addition and removal of object metadata

  • Doc added/updated
  • Tests added

@Neon-White Neon-White marked this pull request as ready for review February 28, 2024 10:04
@Neon-White Neon-White force-pushed the implement-gcp-ns branch 2 times, most recently from a89dccb to 4522b53 Compare March 4, 2024 16:36
src/sdk/namespace_gcp.js Outdated Show resolved Hide resolved
src/sdk/namespace_gcp.js Show resolved Hide resolved
src/sdk/namespace_gcp.js Show resolved Hide resolved
src/sdk/namespace_gcp.js Outdated Show resolved Hide resolved
src/sdk/namespace_gcp.js Show resolved Hide resolved
src/sdk/namespace_gcp.js Show resolved Hide resolved
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
- Fix invalid call to stats collector

Signed-off-by: Ben <belimele@redhat.com>
- Explain why an S3 client is used

Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
Signed-off-by: Ben <belimele@redhat.com>
- Remove `else` branch by adding an early exit

Signed-off-by: Ben <belimele@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants