Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement callbacks to Alpakka's GCS client whenever a chunk is uploaded using multipart upload #201

Open
1 task
mdedetrich opened this issue Apr 14, 2022 · 0 comments
Labels
gcs Specifically related to Google Cloud Storage backend upstream Issues that require changes/co-operation with upstream libraries/projects

Comments

@mdedetrich
Copy link
Collaborator

What is currently missing?

At some point we would like to support GCS in the same way we currently support S3 as a backup storage.

How could this be improved?

Similar to what was done with alpakka's s3 client, functionality needs to be upstreamed to Alpakka GCS client that allows us to both get the uploaded parts (aka chunks) of a currently incompletely multipart upload ergo akka/alpakka#2730 and also the ability to resume the upload of a file while providing context (which in our case is Kafka cursors) ergo akka/alpakka#2770

One thing to consider is that GCS appears to have 2 API's. Their original JSON based API which is what the current Alpakka' GCS client implements and also an S3 compatible API which they added later on. It needs to be investigated whether GCS's JSON based API even supports the functionality we need. If not the changes required would be more extensive as we would need to either re-implement the current GCS client using S3's API or whether we should make a completely new GCS client with the current on in tact (also need to consider if we should share code between Alpakka's S3 client and GCS's S3 client).

Is this a feature you would work on yourself?

  • I plan to open a pull request for this feature
@mdedetrich mdedetrich added the upstream Issues that require changes/co-operation with upstream libraries/projects label Apr 14, 2022
@mdedetrich mdedetrich added the gcs Specifically related to Google Cloud Storage backend label May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gcs Specifically related to Google Cloud Storage backend upstream Issues that require changes/co-operation with upstream libraries/projects
Projects
None yet
Development

No branches or pull requests

1 participant