Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

To refactor KafkaSink to use more workers for message production. #11019

Open
1 task
asddongmen opened this issue Apr 30, 2024 · 0 comments
Open
1 task

To refactor KafkaSink to use more workers for message production. #11019

asddongmen opened this issue Apr 30, 2024 · 0 comments
Labels
area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. type/enhancement This is a enhancement PR

Comments

@asddongmen
Copy link
Contributor

asddongmen commented Apr 30, 2024

Is your feature request related to a problem?

To ensure all data from a table are sent sequentially downstream to Kafka, the KafkaSink uses only one worker to produce messages of a changefeed (in a processor).
This means that only one TCP connection is established between CDC and the Kafka server. Therefore, in case of high network latency, the throughput of KafkaSink can significantly deteriorate. (See #10775)

Describe the feature you'd like

  • To refactor KafkaSink to use more workers for message production of different table, which should be helpful in high network latency case.

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

No response

@asddongmen asddongmen added component/sink Sink component. type/feature Issues about a new feature area/ticdc Issues or PRs related to TiCDC. type/enhancement This is a enhancement PR and removed type/feature Issues about a new feature labels Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. type/enhancement This is a enhancement PR
Projects
None yet
Development

No branches or pull requests

1 participant