spanner: TestIntegration_TransactionRunner failed

[flaky-bot]

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

---

commit: 891bf76
buildURL: Build Status, Sponge
status: failed

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

commit: 891bf76
buildURL: Build Status, Sponge
status: passed

[hengfengli]

The first error is shown as follows:

--- FAIL: TestIntegration_BatchQuery (6.68s)
    integration_test.go:2657: cannot create testing DB projects/dulcet-port-762/instances/gotest-1586907429049834785-01/databases/gotest_1586907429049721886_27: rpc error: code = Unavailable desc = transport is closing
=== CONT  TestIntegration_ReadErrors
=== CONT  TestIntegration_InvalidDatabase
--- FAIL: TestIntegration_TransactionRunner (7.49s)
    integration_test.go:2657: cannot create testing DB projects/dulcet-port-762/instances/gotest-1586907429049834785-01/databases/gotest_1586907429049721886_26: rpc error: code = Unavailable desc = transport is closing

This is a similar case as #1922.

@tbpg I checked the second error log and this is not related to spanner: TestIntegration_TransactionRunner. Please assign it to the pubsub team.

=== RUN   TestPublishScheduler_DoesntRaceWithPublisher
    TestPublishScheduler_DoesntRaceWithPublisher: publish_scheduler_test.go:174: some_key_3: expected key some_key_3 - item 2 to be handled but never was
panic: test timed out after 45m0s

goroutine 63 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1460 +0x11c
created by time.goFunc
	/usr/local/go/src/time/sleep.go:168 +0x52

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00014c000, 0x615c1f, 0x2c, 0x6176a0, 0x1)
	/usr/local/go/src/testing/testing.go:1044 +0x699
testing.runTests.func1(0xc00014c000)
	/usr/local/go/src/testing/testing.go:1285 +0xa7
testing.tRunner(0xc00014c000, 0xc00012bd50)
	/usr/local/go/src/testing/testing.go:992 +0x1ec
testing.runTests(0xc000134020, 0x763f60, 0x7, 0x7, 0x0)
	/usr/local/go/src/testing/testing.go:1283 +0x528
testing.(*M).Run(0xc000148000, 0x0)
	/usr/local/go/src/testing/testing.go:1200 +0x300
main.main()
	_testmain.go:66 +0x224

goroutine 44 [semacquire]:
sync.runtime_Semacquire(0xc0001d62c8)
	/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0001d62c0)
	/usr/local/go/src/sync/waitgroup.go:130 +0xd4
google.golang.org/api/support/bundler.(*Bundler).Flush(0xc00018c630)
	/go/pkg/mod/google.golang.org/api@v0.21.0/support/bundler/bundler.go:400 +0x184
cloud.google.com/go/pubsub/internal/scheduler.(*PublishScheduler).FlushAndStop(0xc00019e100)
	/tmpfs/src/google-cloud-go/pubsub/internal/scheduler/publish_scheduler.go:156 +0x10d
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:634 +0x104
testing.(*common).FailNow(0xc0001b8120)
	/usr/local/go/src/testing/testing.go:657 +0x5b
testing.(*common).Fatalf(0xc0001b8120, 0x616c9e, 0x39, 0xc000095dc8, 0x3, 0x3)
	/usr/local/go/src/testing/testing.go:724 +0x9e
cloud.google.com/go/pubsub/internal/scheduler_test.TestPublishScheduler_DoesntRaceWithPublisher(0xc0001b8120)
	/tmpfs/src/google-cloud-go/pubsub/internal/scheduler/publish_scheduler_test.go:174 +0x923
testing.tRunner(0xc0001b8120, 0x6176a0)
	/usr/local/go/src/testing/testing.go:992 +0x1ec
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1043 +0x661
FAIL	cloud.google.com/go/pubsub/internal/scheduler	2700.014s

[tbpg]

The test is flaky. A failure log is linked in the first comment/issue body:

--- FAIL: TestIntegration_TransactionRunner (7.49s)
    integration_test.go:2657: cannot create testing DB projects/dulcet-port-762/instances/gotest-1586907429049834785-01/databases/gotest_1586907429049721886_26: rpc error: code = Unavailable desc = transport is closing

The second comment saying it's flaky is an example where the test passed for the same commit it failed earlier.

[olavloite]

This issue is the same as #1922 and occurs when a CreateDatabase RPC fails with an UNAVAILABLE error. CreateDatabase is non-idempotent and is therefore not automatically retried by the gapic client. We implemented custom retry logic for this and other non-idempotent RPCs for Java, and we'll do that for Go as well.

[hengfengli]

I think this issue can be closed.

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 3bcbd5d
buildURL: Build Status, Sponge
status: failed

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 3bcbd5d
buildURL: Build Status, Sponge
status: failed

[hengfengli]

I think this is related to googleapis/go-genproto#366.