Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/multitenantccl/tenantcostclient: TestScheduledJobsConsumption failed #122336

Closed
cockroach-teamcity opened this issue Apr 13, 2024 · 9 comments · Fixed by #123233
Closed

ccl/multitenantccl/tenantcostclient: TestScheduledJobsConsumption failed #122336

cockroach-teamcity opened this issue Apr 13, 2024 · 9 comments · Fixed by #123233
Assignees
@yuzefovich
Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-server-and-security DB Server & Security
Projects
DB Server & Security
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 13, 2024 •
edited by cockroach-jira-scripts

ccl/multitenantccl/tenantcostclient.TestScheduledJobsConsumption failed on release-24.1 @ 6e3e9e6012dc04dabc64333a87e77a80d9f9d46b:

=== RUN   TestScheduledJobsConsumption
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestScheduledJobsConsumption1000854501
    test_log_scope.go:81: use -show-logs to present logs inline
    tenant_side_test.go:1148: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go:1148
        	Error:      	"12" is not less than or equal to "3"
        	Test:       	TestScheduledJobsConsumption
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestScheduledJobsConsumption1000854501
--- FAIL: TestScheduledJobsConsumption (20.52s)

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-37810
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 13, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 13, 2024
@blathers-crl blathers-crl bot added the T-server-and-security DB Server & Security label Apr 13, 2024
This was referenced Apr 14, 2024
Closed
Closed
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Apr 17, 2024

cc @cockroachdb/disaster-recovery

@blathers-crl blathers-crl bot added this to Backlog in Disaster Recovery Backlog Apr 17, 2024
@cockroach-teamcity
Copy link
Member Author

ccl/multitenantccl/tenantcostclient.TestScheduledJobsConsumption failed on release-24.1 @ 273a97e181d634e5d5d51ac7d3e05f3f4f31887a:

=== RUN   TestScheduledJobsConsumption
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestScheduledJobsConsumption700134005
    test_log_scope.go:81: use -show-logs to present logs inline
    tenant_side_test.go:1148: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go:1148
        	Error:      	"5" is not less than or equal to "3"
        	Test:       	TestScheduledJobsConsumption
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestScheduledJobsConsumption700134005
--- FAIL: TestScheduledJobsConsumption (12.39s)

Parameters:

  • attempt=1
  • deadlock=true
  • run=3
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 19, 2024
Closed
@msbutler
Copy link
Collaborator

Flake around expected number read batches during server startup. This doesn't need to block the release.

@msbutler msbutler removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 19, 2024
This was referenced Apr 20, 2024
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/multitenantccl/tenantcostclient.TestScheduledJobsConsumption failed on release-24.1 @ 88c9d88cb0093d35f2d630137fd178518aa48569:

=== RUN   TestScheduledJobsConsumption
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestScheduledJobsConsumption3454015885
    test_log_scope.go:81: use -show-logs to present logs inline
    tenant_side_test.go:1148: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go:1148
        	Error:      	"21" is not less than or equal to "3"
        	Test:       	TestScheduledJobsConsumption
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestScheduledJobsConsumption3454015885
--- FAIL: TestScheduledJobsConsumption (20.21s)

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss removed the T-server-and-security DB Server & Security label Apr 25, 2024
@yuzefovich
Copy link
Member

This test is now failing / flaking as a result of #121861, cc @rafiss

@rafiss
Copy link
Collaborator

rafiss commented Apr 25, 2024

Did you determine that with a bisect?

It looks like the test disables auto-stats collection:

cockroach/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go

Line 1078 in 9d04b47

stats.AutomaticStatisticsClusterMode.Override(ctx, &st.SV, false)

Does EnsureAllTables still get called in that case? Also, I see that these failures are only under deadlock or race. I'm not familiar with this test, but could it be skipped in those configs?

@yuzefovich
Copy link
Member

No, I didn't do a bisect, but I'm fairly confident that #121861 is what triggered these flakes (the test starting flaking on all branches after the backport but not on master where we have an additional commit). Yes, we do call ensureAllTables even when the auto stats is disabled since they might be enabled on a per-table basis (i.e. we issue a different query).

Not sure whether we just want to skip this test in some configs. I wonder whether we should introduce a testing knob that disables "initial table collection" in stats.Refresher?

@cockroach-teamcity
Copy link
Member Author

ccl/multitenantccl/tenantcostclient.TestScheduledJobsConsumption failed on release-24.1 @ 3f3127b8b592b9c63df76e47cb9f0a3b47357f62:

=== RUN   TestScheduledJobsConsumption
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestScheduledJobsConsumption732571998
    test_log_scope.go:81: use -show-logs to present logs inline
    tenant_side_test.go:1148: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go:1148
        	Error:      	"8" is not less than or equal to "3"
        	Test:       	TestScheduledJobsConsumption
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestScheduledJobsConsumption732571998
--- FAIL: TestScheduledJobsConsumption (21.71s)

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@dt dt removed this from Backlog in Disaster Recovery Backlog Apr 29, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Apr 29, 2024

cc @cockroachdb/disaster-recovery

@blathers-crl blathers-crl bot added this to Backlog in Disaster Recovery Backlog Apr 29, 2024
@exalate-issue-sync exalate-issue-sync bot added T-server-and-security DB Server & Security and removed T-disaster-recovery labels Apr 29, 2024
@yuzefovich yuzefovich removed this from Backlog in Disaster Recovery Backlog Apr 29, 2024
@yuzefovich yuzefovich self-assigned this Apr 29, 2024
This was referenced Apr 29, 2024
Merged
Merged
This was referenced May 3, 2024
Merged
Merged
@craig craig bot closed this as completed in 23452f7 May 3, 2024
DB Server & Security automation moved this from To do to Done 21.2 May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-server-and-security DB Server & Security
Projects
DB Server & Security
  
Done 21.2
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

stats: add a testing knob to disable initial table collection yuzefovich/cockroach
6 participants
@dt @benbardin @rafiss @yuzefovich @cockroach-teamcity @msbutler