Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd (ticdc): a better Kafka consumer #10990

Open
wants to merge 77 commits into
base: master
Choose a base branch
from
Open

cmd (ticdc): a better Kafka consumer #10990

wants to merge 77 commits into from
+924 −861

Conversation

wk989898
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #10989

What is changed and how it works?

Rewrite Kafka consumer

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

none

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. contribution Indicates that the PR was contributed by an external member. needs-ok-to-test labels Apr 26, 2024
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 26, 2024

Hi @wk989898. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 26, 2024
@wk989898
Copy link
Contributor Author

/test all

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 26, 2024

@wk989898: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@asddongmen
Copy link
Contributor

/ok-to-test

@asddongmen asddongmen marked this pull request as draft April 28, 2024 01:23
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 28, 2024
@asddongmen
Copy link
Contributor

/test cdc-integration-kafka-test

@asddongmen
Copy link
Contributor

Keep this PR in draft state until it is ready for review and merge. In the meantime, you can use /test cdc-integration-kafka-test to manually trigger the Kafka integration test.

@asddongmen asddongmen changed the title Rewrite Kafka consumer cmd (ticdc): a better Kafka consumer Apr 28, 2024
@asddongmen
Copy link
Contributor

It would be extremely helpful if you could provide a design documentation for this new Kafka consumer. The documentation can includes the drawbacks of the current Kafka consumer, suggestions for improvements, how the new design solves these problems, and why it is better than the old one.

asddongmen
asddongmen reviewed Apr 28, 2024
View reviewed changes
cmd/kafka-consumer/kafkago_consumer.go Outdated
"go.uber.org/zap"
)

func kafkaGoGetPartitionNum(address []string, topic string) (int32, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, the prefix kafkaGo is verbose.

@3AceShowHand
Copy link
Contributor

/retest

@3AceShowHand
Copy link
Contributor

/retest

@3AceShowHand
Copy link
Contributor

/retest

@3AceShowHand
Copy link
Contributor

/test dm-integration-test

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 30, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 30, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-05-30 07:28:15.546031179 +0000 UTC m=+2934249.303166752: ☑️ agreed by 3AceShowHand.

@ti-chi-bot ti-chi-bot bot added the approved label May 30, 2024
@3AceShowHand
Copy link
Contributor

/test cdc-integration-storage-test

@3AceShowHand
Copy link
Contributor

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@CharlesCheung96 CharlesCheung96 Awaiting requested review from CharlesCheung96

@asddongmen asddongmen Awaiting requested review from asddongmen

Assignees
No one assigned
Labels
approved contribution Indicates that the PR was contributed by an external member. needs-1-more-lgtm needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enhance Kafka consumer performance
4 participants
@wk989898 @asddongmen @3AceShowHand @CharlesCheung96