Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimum chunk size not enforced #1080

Open
ianhomer opened this issue Feb 21, 2023 · 1 comment
Open

Minimum chunk size not enforced #1080

ianhomer opened this issue Feb 21, 2023 · 1 comment

Comments

@ianhomer
Copy link

ianhomer commented Feb 21, 2023

We see a different behaviour on fake GCS server when chunk size is below the minimum of 262144 - https://cloud.google.com/storage/docs/performing-resumable-uploads

On fake GCS

curl -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

accepts the partial upload and responds status code 308 with

{"kind":"storage#object","name":"my-upload-1.txt",... }

On real GCS

curl --http1.1 -v -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

we get a 400 with response

Invalid request.  The number of bytes uploaded is required to be equal or greater than 262144, 
except for the final request (it's recommended to be the exact multiple of 262144).  
The received request contained 500 bytes, which does not meet this requirement.

When chunk size is multiple of 262144 both behave in a similar manner.

The small chunk size is handy for some test scenarios, however since it does not match the behaviour of real GCS, it masks a failure scenario that would be good to test with fake gcs.

Should fake GCS mirror this real GCS behaviour where a minimum chunk size is enforced?

@fsouza
Copy link
Owner

fsouza commented Feb 21, 2023

Hey @ianhomer, thanks for reporting. We should definitely mirror GCS' behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants