bulk: import jobs often fail with server sent GOAWAY and closed the connection #65926
Assignees
Labels
A-disaster-recovery
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
T-disaster-recovery
Projects
Comments
adityamaru
added
C-bug
|
cc @stevendanna who has been thinking a bit about retry policies for cdc sinks (one of which is gcs). |
|
We're waiting on the google SDK to internally retry on seeing a GOAWAY error - googleapis/google-cloud-go#4226. We should then bump the SDK version to include this fix. |
|
Reopening this as a tracking issue. The GOAWAY fix cannot be used yet, see #68158. |
craig bot
pushed a commit
that referenced
this issue
Jul 28, 2021
68158: Revert "go.mod: bump cloud.google.com/go/storage to v1.16.0" r=tbg,erikgrinaker a=knz Reverts #67806 Informs #68154 Informs #65926 Fixes #68150 Fixes #68152 Fixes #68148 Fixes #68145 Fixes #68144 Fixes #68143 This reverts commit e52ddc8. This is because that commit contains an upgrade to gRPC 1.39 which contains a race condition. Release note: None Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
|
Should be closed by #68650. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Time and again we have seen our roachtests fail with:
gs://cockroach-fixtures/tpce-csv/customers=2000000/746/NewsItem.txt?AUTH=implicit: http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug="server_shutting_down"While this is an infra flake and the only solution is to retry the import, maybe we should be retrying internally so as to not fail the job. This retry could either be at the job resumer level or could be marked as a retriable error in our external storage resuming reader implementations. Either way, the focus of this issue should be to find what error type is bubbled up in such scenarios so that we can intercept and consider it retriable.
Epic: CRDB-2556
The text was updated successfully, but these errors were encountered: