Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delegate pushing the file to S3 #4889

Open
imsdu opened this issue Apr 22, 2024 · 2 comments
Open

Delegate pushing the file to S3 #4889

imsdu opened this issue Apr 22, 2024 · 2 comments
Assignees

Comments

@imsdu
Copy link
Contributor

imsdu commented Apr 22, 2024

Motivation
Some files may be large and can not be uploaded directly to S3 through the upload endpoint and has to be delegated to the client for performance and reliability reasons.

One example are jobs in the parallel cluster which will rely on S3-DRA to push the file to S3

Acceptance criteria

  1. The user can provide optional metadata
  2. If the user has no write access to this storage (permission/deprecated project) if the metadata/id is invalid, an error is returned
  3. If the provided input is valid, a temporary he necessary information to push the file (bucket / object key / id) is returned to the user
  4. This payload is signed to make sure it is not modified by the client
  5. Once the file is pushed to the bucket, the client calls Delta again the signed payload, if the payload can be verified and has not reached an expiration date, the file is created in Nexus
  6. The default storage must be a S3 storage

Implementation notes
https://connect2id.com/products/nimbus-jose-jwt/examples/jws-json-multiple-signatures

Not in scope

  1. The client having write access to the location in the given bucket
  2. Using presigned urls for other cases
  3. Providing a storage id which is not the default one
  4. Providing a file id
  5. Updating the file
@dantb dantb self-assigned this May 10, 2024
@dantb
Copy link
Contributor

dantb commented May 13, 2024

@imsdu "If the provided input is valid" what is the provided input? Is it only the metadata? What about filename, media type, etc? Can they also provide a tag?

I'm also not clear on where the client includes the Id for the second call?

@imsdu
Copy link
Contributor Author

imsdu commented May 15, 2024

@dantb This operation is meant to replace the legacy link operation so it would be the metadata + the content type.
And other controls must be performed related to access, project / storage deprecation, ...

Don't hesitate to do 2 PRs if you feel like it helps, one to generate the signed payload, the second to submit it again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants