Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: NPE during write to GCS #65611

Closed
dt opened this issue May 24, 2021 · 3 comments
Closed

backupccl: NPE during write to GCS #65611

dt opened this issue May 24, 2021 · 3 comments
Assignees
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery

Comments

@dt
Copy link
Member

dt commented May 24, 2021

Created a scheduled backup to gcs. Two nodes crashed with NPE:

panic(0x46894c0, 0x7f3f210)
        /usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc00483fe00 sp=0xc00483fd38 pc=0x48b725
runtime.panicmem(...)
        /usr/local/go/src/runtime/panic.go:212
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:742 +0x413 fp=0xc00483fe30 sp=0xc00483fe00 pc=0x4a2473
cloud.google.com/go/storage.(*Writer).open.func1(0xc01fcc4b40, 0xc015571880, 0xc04cc15278, 0xc01bb4ca60, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/vendor/cloud.google.com/go/storage/writer.go:131 +0xfa fp=0xc00483ffb0 sp=0xc00483fe30 pc=0x375c95a
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00483ffb8 sp=0xc00483ffb0 pc=0x4c4ae1
created by cloud.google.com/go/storage.(*Writer).open
        /go/src/github.com/cockroachdb/cockroach/vendor/cloud.google.com/go/storage/writer.go:118 +0x412```
@dt dt created this issue from a note in Disaster Recovery Backlog (Triage) May 24, 2021
@blathers-crl
Copy link

blathers-crl bot commented May 24, 2021

Hi @dt, please add a C-ategory label to your issue. Check out the label system docs.

While you're here, please consider adding an A- label to help keep our repository tidy.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@mwang1026 mwang1026 moved this from Triage to Bug in Disaster Recovery Backlog May 24, 2021
@mwang1026 mwang1026 added A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels May 24, 2021
@dt dt self-assigned this May 24, 2021
@dt
Copy link
Member Author

dt commented May 25, 2021

I think this is an upstream bug in the SDK, since some println debugging revealed that when we hit this crash, we've already closed both the writer and the client that created it. Filed googleapis/google-cloud-go#4167

@dt
Copy link
Member Author

dt commented May 25, 2021

#65660

@dt dt closed this as completed May 25, 2021
Disaster Recovery Backlog automation moved this from Bug to Done May 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery
Development

No branches or pull requests

2 participants