Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spanner-client: Retry PDML on "Received unexpected EOS on DATA frame from server" #1197

Closed
thiagotnunes opened this issue Jul 29, 2020 · 0 comments · Fixed by #1205
Closed

spanner-client: Retry PDML on "Received unexpected EOS on DATA frame from server" #1197

thiagotnunes opened this issue Jul 29, 2020 · 0 comments · Fixed by #1205
Assignees
@olavloite
Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@thiagotnunes
Copy link
Contributor

This bug is related to the Spanner client library.

For long lived transactions (>= 30 minutes), in the case of large PDML changes, it is possible that the gRPC connection is terminated with an error "Received unexpected EOS on DATA frame from server".

In this case, we need to retry the transaction either with the received resume token obtained on reading the stream or from scratch. This will ensure that the PDML transaction continues to execute until it is successful or a hard timeout is reached.

We have already implemented such change in the Java client library, for more information see this PR: googleapis/java-spanner#360.

In order to test the fix, we can use a large spanner database. Please speak to @thiagotnunes for more details.
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/nodejs-spanner API. label Jul 29, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Jul 29, 2020
@olavloite olavloite added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Aug 4, 2020
@olavloite olavloite self-assigned this Aug 4, 2020
olavloite added a commit to olavloite/nodejs-spanner that referenced this issue Aug 4, 2020
@olavloite
fix: retry PDML on Aborted and Internal errors
dcddd75 
PDML statements should be retried on the following errors:
* ABORTED: Retry using a new transaction
* INTERNAL(Received unexpected EOS on DATA frame from server):
    Retry using the same transaction and any resume token that was received.

Fixes googleapis#1197
@olavloite olavloite mentioned this issue Aug 4, 2020
Merged
thiagotnunes pushed a commit that referenced this issue Aug 10, 2020
@olavloite
fix: retry PDML on Aborted and Internal errors (#1205)
2b97bac 
PDML statements should be retried on the following errors:
* ABORTED: Retry using a new transaction
* INTERNAL(Received unexpected EOS on DATA frame from server): 
  Retry using the same transaction and any resume token that was received.
* INTERNAL (RST_STREAM): 
  Retry using the same transaction and any resume token that was received.

Fixes #1197
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@olavloite olavloite

Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: retry PDML on Aborted and Internal errors olavloite/nodejs-spanner
3 participants
@thiagotnunes @olavloite @yoshi-automation