Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123355

Closed
cockroach-teamcity opened this issue May 1, 2024 · 14 comments · Fixed by #123519
Closed

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123355

cockroach-teamcity opened this issue May 1, 2024 · 14 comments · Fixed by #123519
Assignees
@msbutler
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 1, 2024 •
edited by cockroach-jira-scripts

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ af3173a899e16af30d2ca9489d8c2c926c410121:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m46s)

Stack:

goroutine 361679 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* runtime/pprof.Do({0x7f3a4a8?, 0xc352800?}, {{0xc000e0b1a0?, 0x7f3a7f8?, 0xc00ce09ce0?}}, 0xc0020007c0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.openCleanupManager.func1()
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:97 +0x56
* created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 14136
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:96 +0x265
* 
* goroutine 14196 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 14140 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc0086c3040)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:163 +0xe5
* github.com/cockroachdb/pebble.openCleanupManager.func1.1({0x7f3a7f8?, 0xc00cd5a510?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:98 +0x17
* runtime/pprof.Do({0x7f3a4a8?, 0xc352800?}, {{0xc000e0b1a0?, 0xc00114ffd0?, 0x4a77bd?}}, 0xc00114ffc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.openCleanupManager.func1()
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:97 +0x56
* created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 14136
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:96 +0x265
* 
* goroutine 14173 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 14136
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 14194 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 14177 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 14136
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 14172 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
*

Parameters:

  • attempt=1
  • run=9
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38316
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels May 1, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 1, 2024
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ af3173a899e16af30d2ca9489d8c2c926c410121:

Fatal error:

panic: test timed out after 53m17s
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10342
running tests:
	TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 338432 [running]:
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
testing.(*M).startAlarm.func1()
Log preceding fatal error 


goroutine 10456 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc00ba2d7a0, {0xdcc7798, 0xc00a5eb4d0}, {0x0?, 0x0?, 0x0?}, 0xc00c6cf608)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:224 +0x330
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1.1(0xc00c6cf608)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:286 +0xc5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc009827620, 0xc0120c8f80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:539 +0x325
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0xdcc7798, 0xc00a5eb4d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:284 +0x3f8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10368
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 10147 [chan receive, 53 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0xdcc7798?, 0xc00f50c630?})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:417 +0xe5
runtime/pprof.Do({0xdcc7448, 0x133be680}, {{0xc0000d9420?, 0xc0057ecc08?, 0xc00588efd0?}}, 0xc00588efa0)
	GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc008cc2000)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:415 +0x92
created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 10125
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:407 +0x19d

goroutine 10458 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc008768a50, 0x24c86)
	GOROOT/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc008768a40)
	GOROOT/src/sync/cond.go:70 +0x75
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00eb69260, {0xdcc7798, 0xc00a5eb5c0}, {0xdcad7b0, 0xc00be01808}, 0xc00be00c08)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x3ec
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xdcc7798, 0xc00a5eb5c0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x95
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10342
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 10408 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0087689d0, 0x23d45)
	GOROOT/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc0087689c0)
	GOROOT/src/sync/cond.go:70 +0x75
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00eb691f0, {0xdcc7798, 0xc00f1d9320}, {0xdcad7b0, 0xc00be01808}, 0xc00be00c08)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x3ec
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xdcc7798, 0xc00f1d9320})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x95
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ b74d6dbacaeac1b6f274cdd54513d1ebc754ac5b:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 272411 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 15075
* 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:890 +0x150
* 
* goroutine 15069 [IO wait, 2 minutes]:
* internal/poll.runtime_pollWait(0xffff75fa6728, 0x72)
* 	GOROOT/src/runtime/netpoll.go:345 +0xa0
* internal/poll.(*pollDesc).wait(0xe?, 0x4233f8?, 0x0)
* 	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x28
* internal/poll.(*pollDesc).waitRead(...)
* 	GOROOT/src/internal/poll/fd_poll_runtime.go:89
* internal/poll.(*FD).Accept(0x400ce32900)
* 	GOROOT/src/internal/poll/fd_unix.go:611 +0x250
* net.(*netFD).accept(0x400ce32900)
* 	GOROOT/src/net/fd_unix.go:172 +0x28
* net.(*TCPListener).accept(0x4001c777e0)
* 	GOROOT/src/net/tcpsock_posix.go:159 +0x28
* net.(*TCPListener).Accept(0x4001c777e0)
* 	GOROOT/src/net/tcpsock.go:327 +0x2c
* github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith(0x400fb37428, {0x778ad90, 0x400f2d13b0}, {0x7777c30, 0x4001c777e0}, 0x40109177a0)
* 	github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:166 +0x5c
* github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1({0x778ad90, 0x400f2d13b0})
* 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1867 +0xc8
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11176
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 25285 [semacquire, 2 minutes]:
* sync.runtime_Semacquire(0x3b23c80?)
* 	GOROOT/src/runtime/sema.go:62 +0x2c
* sync.(*WaitGroup).Wait(0x4015957440)
* 	GOROOT/src/sync/waitgroup.go:116 +0x74
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0x400bfec210, {0x778ad58, 0x400880cc80}, 0x400325c808, 0x4001e92150, {0x0, 0x4, {0x1, {0x5eba7d3, 0x3}, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0x77c
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0x400bfec210, {0x778ad58, 0x400880cc80}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0x7b8
* github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0x4007618d18, {0x778ad90, 0x40154f7f50}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0xf8
* github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0x4008169dc0, {0x778ad90, 0x40154f7f50}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x248
* github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0x778ad90?, 0x40154f7f20?}, {0x77d7918, 0x4004bf3f20})
* 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x17c
* github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0x778ad90, 0x40154f7f20})
* 	github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0xd8
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 15496
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ 6210c2cd725ecb3875cad722020ea7c1d9042cec:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m47s)

Stack:

goroutine 278407 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18930
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 60070 [select]:
* github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x778b1b0, 0x400aaa5cb0})
* 	github.com/cockroachdb/cockroach/pkg/sql/stats/pkg/sql/stats/automatic_stats.go:468 +0x13c
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18527
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 29362 [sync.Cond.Wait, 2 minutes]:
* sync.runtime_notifyListWait(0x40104900a8, 0x29)
* 	GOROOT/src/runtime/sema.go:569 +0x154
* sync.(*Cond).Wait(0x4010490098)
* 	GOROOT/src/sync/cond.go:70 +0xcc
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x4010490000, {0x54eb420?, 0xb6ecd80?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628
* runtime/pprof.Do({0x778ae18?, 0xb6ecd80?}, {{0x40009c96e0?, 0x4010b4e840?, 0x400b9fd768?}}, 0x400b9fd7b8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x80
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 15293
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c
* 
* goroutine 31062 [semacquire, 2 minutes]:
* sync.runtime_Semacquire(0x4007913a28?)
* 	GOROOT/src/runtime/sema.go:62 +0x2c
* sync.(*WaitGroup).Wait(0x400d046208)
* 	GOROOT/src/sync/waitgroup.go:116 +0x74
* golang.org/x/sync/errgroup.(*Group).Wait(0x400d046200)
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x2c
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0x400d046200?, {0x778b178?, 0x400d01e640?}})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x48
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x778b1b0?, 0x400e187ad0?}, {0x4007913cd8, 0x2, 0x55547a0?})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0xf8
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0x4009547080, {0x778b1b0, 0x400e187ad0}, {0x77e0920, 0x400e187710})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x6ec
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x778b1b0, 0x400e187a40})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x2d0
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 30864
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ f8b33a7b2175201b5e87440649da550cc67781b2:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 310937 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 25197
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 146868 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x9c
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13393
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x64
* 
* goroutine 73647 [sync.Cond.Wait, 2 minutes]:
* sync.runtime_notifyListWait(0x4004f080a8, 0xdff)
* 	GOROOT/src/runtime/sema.go:569 +0x154
* sync.(*Cond).Wait(0x4004f08098)
* 	GOROOT/src/sync/cond.go:70 +0xcc
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x4004f08000, {0x54ebac0?, 0xb6ece00?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628
* runtime/pprof.Do({0x778b938?, 0xb6ece00?}, {{0x40009443a0?, 0x4010e2e2d0?, 0x400df3cf68?}}, 0x400df3cfb8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x80
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 13484
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c
* 
* goroutine 146869 [sync.Cond.Wait]:
* sync.runtime_notifyListWait(0x400c1300a8, 0xef)
* 	GOROOT/src/runtime/sema.go:569 +0x154
* sync.(*Cond).Wait(0x400c130098)
* 	GOROOT/src/sync/cond.go:70 +0xcc
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x400c130000, {0x54ebac0?, 0xb6ece00?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628
* runtime/pprof.Do({0x778b938?, 0xb6ece00?}, {{0x40009443a0?, 0x40168201b0?, 0x4002767f68?}}, 0x4002767fb8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x80
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 13393
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c
* 
* goroutine 73646 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x9c
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13484
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x64
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 80c77c6651754d31dfb2c852de419d032e0d412b:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m49s)

Stack:

goroutine 350955 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* 
* goroutine 11844 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 11843 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 11848 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 11825 [select]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSenderCircuitBreakers).probeStallLoop(0xc005c86508, {0x7f43d78, 0xc00d598210})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_circuit_breaker.go:296 +0x186
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 11824 [select]:
* github.com/cockroachdb/cockroach/pkg/ccl/utilccl.UpdateMetricOnLicenseChange.func2({0x7f43d78, 0xc00d61bb30})
* 	github.com/cockroachdb/cockroach/pkg/ccl/utilccl/license_check.go:166 +0x12e
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 11850 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 11886 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/security.(*ClientCertExpirationCache).startPurgePastExpirations.func1({0xc00faaf008?, 0x0?})
* 	github.com/cockroachdb/cockroach/pkg/security/cert_expiry_cache.go:196 +0x17c
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
*

Parameters:

  • attempt=1
  • run=21
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 2, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 528a46247efd9d872b19ad750eecaedcb0dd0fce:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 250261 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc00f49d308, {0xdcd73b8, 0xc00ca0fec0}, {{0xdcffa60, 0xc008f47270}, {0x0, 0x0, 0x0}, {0xc0036f8027, 0x34}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0xbbe
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc00f49d308, {0xdcd73b8, 0xc00ca0fec0}, {{0xc0036f806b, 0x0}, 0xc00bfa9320, {0xc00d54ee80, 0x2, 0x2}, {0xc00192a45e, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:251 +0x745
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2({{0xc0036f80d8, 0x0}, 0x0, {0xc184f7a73a08cdea, 0x9459bc5cc, 0x0}, 0x1}, 0xc00f49d308, 0xc005793500, 0xc0057934f0, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2402 +0x117a
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00f49d308)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2404 +0xe3f
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00f49d308, {0xdcd73b8, 0xc00b53f350}, 0xc0093bb4a0, 0xc00e11d7e8, 0xc008f4e110)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2219 +0x3eb
* github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc007608008, {0xdcd73b8, 0xc00b53f350}, {0xc007608008?}, 0xc00e11d7e8, 0xc008f4e110)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:967 +0x217
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0xc006e03c08, {0xdcd73b8, 0xc00b53f350}, {0x0, 0x4, {0x1, {0x9f624f3, 0x3}, {0xc00b089af1, 0xf}, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:247 +0x5b3
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func3()
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:992 +0x21d
* created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 21473
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:989 +0xa50
* 
* goroutine 60865 [select]:
* github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0xdcd73b8, 0xc0133f00c0})
* 	github.com/cockroachdb/cockroach/pkg/sql/stats/pkg/sql/stats/automatic_stats.go:468 +0x337
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13242
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 21473 [semacquire, 1 minutes]:
* sync.runtime_Semacquire(0xc00b089b18?)
* 	GOROOT/src/runtime/sema.go:62 +0x25
* sync.(*WaitGroup).Wait(0xc00b089b10)
* 	GOROOT/src/sync/waitgroup.go:116 +0xa5
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0xc0093bf980, {0xdcd7380, 0xc004becb40}, 0xc006e03c08, 0xc00e11d7e8, {0x0, 0x4, {0x1, {0x9f624f3, 0x3}, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0xfb7
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0093bf980, {0xdcd7380, 0xc004becb40}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0xe05
* github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0xc00887a858, {0xdcd73b8, 0xc00bfbe750}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0x1d8
* github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0xc0093d8c30, {0xdcd73b8, 0xc00bfbe750}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x562
* github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0xdcd73b8, 0xc00bfbe720}, {0xdd23538, 0xc00b0d1e40})
* 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x28c
* github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0xdcd73b8, 0xc00bfbe720})
* 	github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0x1c7
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13599
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented May 2, 2024

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ dc356793a1d152cc61fce9dd11ddbb8bedeee429:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m47s)

Stack:

goroutine 371800 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 12901
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 12993 [chan receive, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func4({0xc0062bac08?, 0xc002098fb0?})
* 	github.com/cockroachdb/cockroach/pkg/server/start_listen.go:145 +0x65
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12901
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 12992 [chan receive, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x7f44ad8, 0xc00a757bc0})
* 	github.com/cockroachdb/cockroach/pkg/server/start_listen.go:84 +0x45
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12901
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 159100 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13128
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x58
* 
* goroutine 26404 [semacquire, 2 minutes]:
* sync.runtime_Semacquire(0xbb1d6a0?)
* 	GOROOT/src/runtime/sema.go:62 +0x25
* sync.(*WaitGroup).Wait(0x7ef40c0?)
* 	GOROOT/src/sync/waitgroup.go:116 +0x48
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0xc0082fb600, {0x7f44aa0, 0xc00aa1a960}, 0xc0027a8808, 0xc00dae9c20, {0x0, 0x4, {0x1, {0x666316f, 0x3}, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0xa7d
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0082fb600, {0x7f44aa0, 0xc00aa1a960}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0xad6
* github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0xc005eed250, {0x7f44ad8, 0xc010b2c3f0}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0x131
* github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0xc007831500, {0x7f44ad8, 0xc010b2c3f0}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...})
* 	github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x339
* github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0x7f44ad8?, 0xc010b2c3c0?}, {0x7f91178, 0xc00c6e9bf0})
* 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x20c
* github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0x7f44ad8, 0xc010b2c3c0})
* 	github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0x102
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 16507
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
*

Parameters:

  • attempt=1
  • run=1
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@msbutler msbutler self-assigned this May 2, 2024
@msbutler
Copy link
Collaborator

msbutler commented May 2, 2024

i can immediately repro this under stress. The timeout disappears if I comment out metadata replication added in #123225

@msbutler msbutler mentioned this issue May 2, 2024
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue May 2, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
7181308 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in cockroachdb#123001.

Fixes cockroachdb#123355
Fixes cockroachdb#123397

Release note: none
msbutler added a commit to msbutler/cockroach that referenced this issue May 2, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
d5dab48 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in cockroachdb#123001.

Fixes cockroachdb#123355
Fixes cockroachdb#123397

Release note: none
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ 5b1a3c18174302d23dd3c2f3616fa86c7ac00396:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m51s)

Stack:

goroutine 340939 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0x400973f900, {0x77a3430, 0x4001413290}, 0x400ac04b58, 0x400befc4e0, 0x400ac04688, 0x400c85b408, 0x0)
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707 +0x178
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0x400ac04008, {0x77a3430, 0x4001413290}, 0x400ac04688, 0x3?, {0xffff3b8cff10?, 0x400296ff30?}, 0x0, 0x400bfc6818, {0x0, ...})
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2490 +0x360
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0x400ac04008, {0x77a33f8, 0x400bbf5680}, 0x400ac04688, {0xffff3b8cff10, 0x400296ff30})
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2020 +0xdfc
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0x400ac04008, {0x77a33f8, 0x400bbf5680}, {{0x77cbca0, 0x400bbf54f0}, {0x0, 0x0, 0x0}, {0x4002add027, 0x34}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1166 +0x3d58
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x77a3430?, 0x4001412a20?})
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:146 +0x8c
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x77a3430?, {0x77a3430?, 0x4001412a20?}, {0x77cbca0?, 0x400bbf54f0?}, 0x439bd0?, 0x400428cd78?)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3499 +0x25c
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0x400ac04008, {0x77a3430, 0x4001412a20}, {{0x77cbca0, 0x400bbf54f0}, {0x0, 0x0, 0x0}, {0x4002add027, 0x34}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x534
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0x400ac04008, {0x77a3430, 0x4001412a20}, {{0x4002add06b, 0x0}, 0x400f2938c0, {0x400d4a9780, 0x2, 0x2}, {0x4009001114, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:251 +0x35c
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2({{0x4002add0d8, 0x0}, 0x0, {0xc18521fdc78e3025, 0x305760fb9, 0x0}, 0x1}, 0x400ac04008, 0x400428d950, 0x400428d940, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2402 +0x7fc
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0x400ac04008)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2404 +0x5c4
* github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0x400ac04008, {0x77a3430, 0x400d56f500}, 0x400f6cd680?, 0x400c623cb8?, 0x400bb6e060?)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2219 +0x168
* github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0x4003286be0?, {0x77a3430?, 0x400d56f500?}, {0x40029fd03e?}, 0x4?, 0x400d56f380?)
* 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:967 +0xb4
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0x400296f808, {0x77a3430, 0x400d56f500}, {0x0, 0x4, {0x1, {0x5ecd8d3, 0x3}, {0x40042ac480, 0xf}, ...}, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:247 +0x2d4
* github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func3()
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:992 +0xcc
* created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 23097
* 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:989 +0x45c
* 
* goroutine 23011 [chan send, 3 minutes]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sendMetadata(...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:525
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed(0x400b143220, {0x77a33f8, 0x400c772870}, {{0x40009bbc00, 0x3, 0x8}, {0x400dcbcb30, 0x3, 0x8}}, {0x40084f2318?, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:240 +0x344
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideSpanOnRangeBoundaries({0x77a33f8, 0x400c772870}, 0x8?, {{0x40084f2310, 0x2, 0x8}, {0x40084f2318, 0x2, 0x8}}, {0x5ed7d8?, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:559 +0x244
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideAllSpansOnRangeBoundaries({0x77a33f8, 0x400c772870}, {0x400dcb87c0, 0x1, 0x1}, 0x400f4d3f28, 0x400c6b1208)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:329 +0x170
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed.func2({0x77a33f8?, 0x400c772870?})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:98 +0x50
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed.Group.GoCtx.func3()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x2c
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x58
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22977
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x98
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 3, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 906605a3df4466e8d2572407d5a71168c88bcc4a:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (52m59s)

Stack:

goroutine 332503 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* github.com/cockroachdb/cmux.muxListener.Accept(...)
* 	github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:184
* google.golang.org/grpc.(*Server).Serve(0xc00a80e1e0, {0xdcddb00, 0xc002aa87c8})
* 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:823 +0x684
* github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.1({0xc00b389c08?, 0x0?})
* 	github.com/cockroachdb/cockroach/pkg/server/start_listen.go:179 +0x56
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 11092 [IO wait, 4 minutes]:
* internal/poll.runtime_pollWait(0x7f8bb09c8a10, 0x72)
* 	GOROOT/src/runtime/netpoll.go:345 +0x85
* internal/poll.(*pollDesc).wait(0xc00abf6820, 0x72, 0x0)
* 	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0xb1
* internal/poll.(*pollDesc).waitRead(...)
* 	GOROOT/src/internal/poll/fd_poll_runtime.go:89
* internal/poll.(*FD).Accept(0xc00abf6800)
* 	GOROOT/src/internal/poll/fd_unix.go:611 +0x507
* net.(*netFD).accept(0xc00abf6800)
* 	GOROOT/src/net/fd_unix.go:172 +0x45
* net.(*TCPListener).accept(0xc0096fab20)
* 	GOROOT/src/net/tcpsock_posix.go:159 +0x3e
* net.(*TCPListener).Accept(0xc0096fab20)
* 	GOROOT/src/net/tcpsock.go:327 +0x65
* github.com/cockroachdb/cmux.(*cMux).Serve(0xc0089fcf40)
* 	github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:124 +0x150
* github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.3.1()
* 	github.com/cockroachdb/cockroach/pkg/server/start_listen.go:187 +0x45
* sync.(*Once).doSlow(0xc00cb430d4, 0xc00b255e50)
* 	GOROOT/src/sync/once.go:74 +0xf1
* sync.(*Once).Do(0xc00cb430d4, 0xc00b255e50)
* 	GOROOT/src/sync/once.go:65 +0x45
* github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.3({0xc00b760408?, 0x0?})
* 	github.com/cockroachdb/cockroach/pkg/server/start_listen.go:186 +0x89
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 11094 [select]:
* github.com/cockroachdb/cockroach/pkg/util/schedulerlatency.StartSampler.func1({0xdcf1698, 0xc0083d1050})
* 	github.com/cockroachdb/cockroach/pkg/util/schedulerlatency/sampler.go:144 +0x87f
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ fc9f95366aae294dc007e2aeff6db03fbff88feb:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m45s)

Stack:

goroutine 303844 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0x4007f38780, {0x77a20b0, 0x40131769f0}, {0x77f7860, 0x4013176690})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x6ec
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x77a20b0, 0x4013176990})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x2d0
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 29312
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 14170 [select, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x4006539c08?, 0x4002874f60?})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 14178 [select, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x40033fd408?, 0x40094a9b30?})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 14172 [select, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x400634c408?, 0x485455412c5d3001?})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 54535 [sync.Cond.Wait]:
* sync.runtime_notifyListWait(0x400d85a0a8, 0x263)
* 	GOROOT/src/runtime/sema.go:569 +0x154
* sync.(*Cond).Wait(0x400d85a098)
* 	GOROOT/src/sync/cond.go:70 +0xcc
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x400d85a000, {0x54fb980?, 0xb70d9c0?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628
* runtime/pprof.Do({0x77a1d18?, 0xb70d9c0?}, {{0x40001f5780?, 0x400161af30?, 0x400abea768?}}, 0x400abea7b8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x80
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 14593
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue May 3, 2024
@msbutler
Merge #123519
b768e8b 
123519: rangefeed: fix sendMetadata deadlock r=dt a=msbutler

If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none

Co-authored-by: Michael Butler <butler@cockroachlabs.com>
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 24m57s
running tests:
	TestTenantStreamingJobRetryReset (24m40s)

Stack:

goroutine 139696941 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65d6c40?, 0xc005685408?}, {0x7f95050?, 0xc01243a660?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:850 +0x7a
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc0041f07e0, 0x4, 0x4}, {0x65d6c40, 0xc005685408}, {0x7f95050, 0xc01243a660}, 0xbb6b3b0, 0xc003000660)
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:852 +0x11e
* github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed.func3()
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1179 +0xf4
* created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed in goroutine 3351746
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1164 +0x365
* 
* goroutine 3733473 [select, 2 minutes]:
* database/sql.(*DB).connectionOpener(0xc00fafb520, {0x7f5b360, 0xc0082e5950})
* 	GOROOT/src/database/sql/sql.go:1246 +0x87
* created by database/sql.OpenDB in goroutine 3001221
* 	GOROOT/src/database/sql/sql.go:824 +0x14c
* 
* goroutine 3026822 [chan receive]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*tokenAllocationTicker).tick(...)
* 	github.com/cockroachdb/cockroach/pkg/util/admission/io_load_listener.go:425
* github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:115 +0xa5
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 3001221
* 	github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:109 +0x2f9
* 
* goroutine 6836015 [semacquire, 2 minutes]:
* sync.runtime_Semacquire(0xc00b75a0d8?)
* 	GOROOT/src/runtime/sema.go:62 +0x25
* sync.(*WaitGroup).Wait(0x4a8d60?)
* 	GOROOT/src/sync/waitgroup.go:116 +0x48
* golang.org/x/sync/errgroup.(*Group).Wait(0xc00e6eca40)
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x25
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00e6eca40?, {0x7f5b360?, 0xc01093fa40?}})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x47
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x7f5b398?, 0xc001d1af90?}, {0xc002a27cf0, 0x2, 0x5d15160?})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0xff
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc010043500, {0x7f5b398, 0xc001d1af90}, {0x7fb06e0, 0xc02101a5d0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x886
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x7f5b398, 0xc001d1aed0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x37b
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 6821301
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 19343041 [runnable]:
* github.com/sasha-s/go-deadlock.lock.func1()
* 	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:182
* created by github.com/sasha-s/go-deadlock.lock in goroutine 19343106
* 	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x14d
* 
*

Parameters:

  • attempt=1
  • deadlock=true
  • run=2
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 349093 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* sync.(*WaitGroup).Wait(0xc009c72a08)
* 	GOROOT/src/sync/waitgroup.go:116 +0xa5
* golang.org/x/sync/errgroup.(*Group).Wait(0xc009c72a00)
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x38
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc009c72a00?, {0xdcf3fe0?, 0xc001c211d0?}})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x65
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed({0xdcf4018, 0xc005e236b0}, {0x0, 0x0, 0x0, 0x0, 0x1, 0x0, {0x0, 0x0, ...}}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:101 +0x8ef
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans(0xc00a038c08, {0xdcf3fe0, 0xc0079c7e00}, {0xc009c72980, 0x1, 0x1}, 0xc00b3a4a80, {0xc004863c80, 0x1, 0x1})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:246 +0xc4f
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00a038c08, {0xdcf3fe0, 0xc0079c7e00}, {0xc00e9573e0, 0x1, 0x4ed109?}, {0xc006609e50?, 0x4fdf59?}, 0xc00b3a4a80, {0xc004863c80, ...})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:197 +0x2e5
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0035cc360, {0xdcf3fe0, 0xc0079c7e00}, {0xc00e9573e0, 0x1, 0x1}, {0xc006609ea0?, 0x4abf09?}, 0xc00b3a4a80, {0xc004863c80, ...})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:79 +0xed
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0xdcf3fe0, 0xc0079c7e00})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:337 +0x186
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 18769
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125
* 
* goroutine 10538 [chan receive, 5 minutes]:
* github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc0049ff808?, 0xc004a7db30?})
* 	github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/server.go:388 +0x78
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 96441 [select]:
* github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0047dca50, {0xdcf3fe0, 0xc012ade4b0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:340 +0x3ad
* github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0047dca50, {0xdcf4018, 0xc0164eb020}, 0xc00c5dee70?)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:365 +0x129
* github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1({0xdcf4018, 0xc0164eb020})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/scheduled_processor.go:348 +0x68
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10545
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 63762 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10744
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0xad
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=3
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 338579 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* internal/poll.(*pollDesc).waitRead(...)
* 	GOROOT/src/internal/poll/fd_poll_runtime.go:89
* internal/poll.(*FD).Accept(0xc00b085480)
* 	GOROOT/src/internal/poll/fd_unix.go:611 +0x507
* net.(*netFD).accept(0xc00b085480)
* 	GOROOT/src/net/fd_unix.go:172 +0x45
* net.(*TCPListener).accept(0xc008a1e680)
* 	GOROOT/src/net/tcpsock_posix.go:159 +0x3e
* net.(*TCPListener).Accept(0xc008a1e680)
* 	GOROOT/src/net/tcpsock.go:327 +0x65
* github.com/cockroachdb/cmux.(*cMux).Serve(0xc004868b00)
* 	github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:124 +0x150
* github.com/cockroachdb/cockroach/pkg/server.startHTTPService.func2({0xc007b87808?, 0xc003d13d80?})
* 	github.com/cockroachdb/cockroach/pkg/server/server_http.go:312 +0x45
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 39278 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10706
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0xad
* 
* goroutine 10487 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xdcf4018, 0xc00b9da810})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0x145
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 39279 [sync.Cond.Wait, 2 minutes]:
* sync.runtime_notifyListWait(0xc00c2940a8, 0xb97)
* 	GOROOT/src/runtime/sema.go:569 +0x159
* sync.(*Cond).Wait(0xc00c294098)
* 	GOROOT/src/sync/cond.go:70 +0x75
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc00c294000, {0xc00bc6a6f0?, 0x959b360?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0xb79
* runtime/pprof.Do({0xdcf3cc8, 0x133f42c0}, {{0xc001200100?, 0xc00458a7a0?, 0xc001a83900?}}, 0xc0094e9fc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x118
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x92
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 10706
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x8e5
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=3
  • shard=5
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue May 6, 2024
@msbutler
Merge #123519
a7b52e0 
123519: rangefeed: fix sendMetadata deadlock r=msbutler a=msbutler

If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none

Co-authored-by: Michael Butler <butler@cockroachlabs.com>
@craig craig bot closed this as completed in 22c698a May 6, 2024
Disaster Recovery Backlog automation moved this from Backlog to Done May 6, 2024
blathers-crl bot pushed a commit that referenced this issue May 6, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
74b8e5a 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none
@blathers-crl blathers-crl bot mentioned this issue May 6, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue May 6, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
0c8a1cf 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none
@blathers-crl blathers-crl bot mentioned this issue May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

rangefeed: fix sendMetadata deadlock msbutler/cockroach
2 participants
@cockroach-teamcity @msbutler