New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[changelog] Let bootstrapping changelog consumer consumes after-image #966
Conversation
Is this change still necessary as we are planning to deprecate after-image support? |
This is the intended direction to switch from before/after to after-only image for bootstrapping changelog consumer. |
clients/da-vinci-client/src/main/java/com/linkedin/davinci/consumer/VeniceChangeCoordinate.java
Outdated
Show resolved
Hide resolved
...in/java/com/linkedin/davinci/consumer/InternalLocalBootstrappingVeniceChangelogConsumer.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This overall looks good, just some nits
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
[changelog] Let bootstrapping changelog consumer consumes after-image
Make bootstrapping consumer also consume after-image data from version topic, instead of before-after image data from cc topic. Fixed a few small bugs during the journey, and clean up the integration tests.
Bugs:
How was this PR tested?
Added new integration tests and clean up old ones. Now we have 1 integration tests that cover ingestion bootstrapping consumer with/without bootstrap and 1/3 parallel consumers, which replace a few integration tests that serve the same purpose but with a lot of duplication in code.
For the existing TestChangelogConsumer#testAAwithStoreView(), I fixed the flakiness in the test, but the test is still too big and messy, consider refactoring it in the future.
Does this PR introduce any user-facing changes?
Bootstrapping changelog consumer will only consume after-image change events now.