Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/changefeedccl: unskip TestChangefeedOnlyInitialScanCSV for pulsar sink #119289

Closed
cockroach-teamcity opened this issue Feb 16, 2024 · 4 comments · Fixed by #123228
Closed
Assignees
Labels
A-cdc Change Data Capture A-testing Testing tools and infrastructure branch-master Failures on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-cdc
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 16, 2024

ccl/changefeedccl.TestChangefeedOnlyInitialScanCSV failed on master @ 617bf347978dcc0d711399b1a76402d7f88de958:

=== RUN   TestChangefeedOnlyInitialScanCSV
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestChangefeedOnlyInitialScanCSV2394812205
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:7530: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedOnlyInitialScanCSV2394812205
--- FAIL: TestChangefeedOnlyInitialScanCSV (1.54s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar
    helpers_test.go:867: making server as system tenant
    helpers_test.go:943: making pulsar feed factory
    --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar (1.53s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar/initial_scan_only_with_csv
    changefeed_test.go:7513: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7513
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7525
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestChangefeedOnlyInitialScanCSV/pulsar/initial_scan_only_with_csv
        --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar/initial_scan_only_with_csv (0.32s)

Parameters:

  • attempt=1
  • run=7
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-36130

@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Feb 16, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Feb 16, 2024
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Feb 16, 2024
@jayshrivastava jayshrivastava self-assigned this Feb 16, 2024
@cockroach-teamcity
Copy link
Member Author

ccl/changefeedccl.TestChangefeedOnlyInitialScanCSV failed on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

=== RUN   TestChangefeedOnlyInitialScanCSV
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestChangefeedOnlyInitialScanCSV809831575
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:7530: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedOnlyInitialScanCSV809831575
--- FAIL: TestChangefeedOnlyInitialScanCSV (1.53s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar
    helpers_test.go:867: making server as system tenant
    helpers_test.go:943: making pulsar feed factory
    --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar (1.52s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv_multiple_tables
    changefeed_test.go:7513: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7513
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7525
        	Error:      	Not equal: 
        	            	expected: 6
        	            	actual  : 5
        	Test:       	TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv_multiple_tables
        --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv_multiple_tables (0.30s)

Parameters:

  • attempt=1
  • run=2
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/changefeedccl.TestChangefeedOnlyInitialScanCSV failed on master @ 7bb52a7d1c75d5adfdfa53e5fcff6f5e6497408f:

=== RUN   TestChangefeedOnlyInitialScanCSV
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestChangefeedOnlyInitialScanCSV311463542
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:7530: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedOnlyInitialScanCSV311463542
--- FAIL: TestChangefeedOnlyInitialScanCSV (1.63s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar
    helpers_test.go:867: making server as system tenant
    helpers_test.go:943: making pulsar feed factory
    --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar (1.62s)
=== RUN   TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv
    changefeed_test.go:7513: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7513
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:7525
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv
        --- FAIL: TestChangefeedOnlyInitialScanCSV/pulsar/initial_backfill_only_with_csv (0.28s)

Parameters:

  • attempt=1
  • run=18
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@jayshrivastava jayshrivastava removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Feb 27, 2024
@jayshrivastava
Copy link
Contributor

Will skip this test

jayshrivastava added a commit to jayshrivastava/cockroach that referenced this issue Feb 28, 2024
craig bot pushed a commit that referenced this issue Feb 28, 2024
119535: roachtest: fail tests if monitor encounters an error r=srosenberg a=renatolabs

This commit updates the roachprod and roachtest monitors to 1) send an
event when the monitor is abruptly terminated (i.e., reader stream
sees an EOF when the associated context is *not* canceled); and 2)
return any errors encountered by the roachprod monitor in roachtest,
causing the currently running test to fail. The error has TestEng
ownership so that teams are not be pinged on these kinds of flakes.

The main purpose of this change is for the monitor to fail in
situations where the monitored node is preempted by the cloud
provider. Previously, these events would be ignored, leading to a test
timeout, wasting resources and leading to confusing test failures
being reported on GitHub.

Fixes: #118563.

Release note: None

119725: changefeedccl: disable TestChangefeedOnlyInitialScanCSV with pulsar r=rharding6373 a=jayshrivastava

Informs: #119289
Release note: None
Epic: None

119732: master: Update pkg/testutils/release/cockroach_releases.yaml r=rail a=github-actions[bot]

Update pkg/testutils/release/cockroach_releases.yaml with recent values.

Epic: None
Release note: None
Release justification: test-only updates

Co-authored-by: Renato Costa <renato@cockroachlabs.com>
Co-authored-by: Jayant Shrivastava <jayants@cockroachlabs.com>
Co-authored-by: CRL Release bot <teamcity@cockroachlabs.com>
@rharding6373 rharding6373 changed the title ccl/changefeedccl: TestChangefeedOnlyInitialScanCSV failed ccl/changefeedccl: unskip TestChangefeedOnlyInitialScanCSV for pulsar sink Mar 4, 2024
@rharding6373 rharding6373 added the P-2 Issues/test failures with a fix SLA of 3 months label Mar 20, 2024
@rharding6373
Copy link
Collaborator

We should fix and unskip this test before making pulsar available for beta.

rharding6373 added a commit to rharding6373/cockroach that referenced this issue Apr 29, 2024
Although the comments in mock pulsar sink say that emitting messages and
flushing are synchronous, the use of a buffered channel to transmit
messages from the mock producer to the test feed makes it asynchronous.
This leads to problems where the consumer (the test) may not have stored
the data in a "durable" way before the changefeed job completes. As a
result, tests can be flaky as it does not always look like all rows have
been emitted.

This PR makes the mock pulsar sink synchronous by using an unbuffered
channel.

Fixes: cockroachdb#119289
Informs: cockroachdb#118899

Epic: CRDB-9180

Release note: None
craig bot pushed a commit that referenced this issue May 6, 2024
123228: changefeedccl: make mock pulsar sink synchronous r=asg0451 a=rharding6373

Although the comments in mock pulsar sink say that emitting messages and flushing are synchronous, the use of a buffered channel to transmit messages from the mock producer to the test feed makes it asynchronous. This leads to problems where the consumer (the test) may not have stored the data in a "durable" way before the changefeed job completes. As a result, tests can be flaky as it does not always look like all rows have been emitted.

This PR makes the mock pulsar sink synchronous by using an unbuffered channel.

Fixes: #119289
Informs: #118899

Epic: CRDB-9180

Release note: None

Co-authored-by: rharding6373 <harding@cockroachlabs.com>
@craig craig bot closed this as completed in 97dcb3d May 6, 2024
@rharding6373 rharding6373 added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture A-testing Testing tools and infrastructure branch-master Failures on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants