Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StorageException: Resumable upload is already complete. #666

Closed
n12c opened this issue Dec 16, 2020 · 6 comments · Fixed by #677
Closed

StorageException: Resumable upload is already complete. #666

n12c opened this issue Dec 16, 2020 · 6 comments · Fixed by #677
Assignees
Labels
api: storage Issues related to the googleapis/java-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@n12c
Copy link

n12c commented Dec 16, 2020

Following #647, we no longer sometimes see:

java.lang.NullPointerException: null
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.getCurrentUploadOffset(HttpStorageRpc.java:783)
	at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:125)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:70)
	at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
	at com.google.cloud.storage.StorageImpl.createFrom(StorageImpl.java:265)
...snip...

This appeared in 1.113.4 but no longer appears in 1.113.6. Thanks for the fix.

Instead (in 1.113.6) we now sometimes see:

com.google.cloud.storage.StorageException: Resumable upload is already complete.
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.getCurrentUploadOffset(HttpStorageRpc.java:769)
	at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:125)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:70)
	at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
	at com.google.cloud.storage.StorageImpl.createFrom(StorageImpl.java:265)
...snip...

What do you think? More tweaks to #604 required?

Thanks for your time.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/java-storage API. label Dec 16, 2020
@frankyn frankyn added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Dec 16, 2020
@frankyn frankyn self-assigned this Dec 16, 2020
@frankyn frankyn added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Dec 16, 2020
@frankyn
Copy link
Member

frankyn commented Dec 16, 2020

Hi @n12c!

Thanks for filing the issue. Do you have an example use case that causes this issue?

@n12c
Copy link
Author

n12c commented Dec 16, 2020

I don't have a self-contained reproducer; this seems quite intermittent. Generally we just call com.google.cloud.storage.Storage#createFrom(com.google.cloud.storage.BlobInfo, java.io.InputStream, int, com.google.cloud.storage.Storage.BlobWriteOption...) with content in the range of 100 bytes to a few MiB in length. I added retry logic to my own code, which repeats this call unmodified and without delay (with a fresh java.io.InputStream having the exact same content); this always seems to succeed on the second try. We always use BlobWriteOption#disableGzipContent and sometimes use BlobWriteOption#encryptionKey(java.security.Key). The only instances I have seen have all occurred while CSEK was in use, but that doesn't necessarily prove it doesn't also happen without CSEK.

I can try to write a reproducer but it would basically consist of uploading random objects using this API in a tight loop until (hopefully) encountering the same issue.

@frankyn
Copy link
Member

frankyn commented Dec 16, 2020

Thanks! I'll try to reproduce the issue, it might be a retry an accidental retry when there shouldn't be one.

@n12c
Copy link
Author

n12c commented Dec 16, 2020

Update: I just saw this happen without CSEK (no encryption of any kind, just the default Google managed keys). So, CSEK is not a factor.

@richiesgr
Copy link

Hi
Any update regarding what cause this ?

@frankyn
Copy link
Member

frankyn commented Jan 11, 2021

Hi @richiesgr I'm working to release a fix today. Apologies for the delay.

gcf-merge-on-green bot pushed a commit that referenced this issue Jan 12, 2021
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [x] Ensure the tests and linter pass
- [x] Appropriate docs were updated (if necessary)

Fixes #666 ☕️
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants