Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: adjust sorter options to avoid Seek CPU usage exploding #11099

Merged
merged 3 commits into from
May 17, 2024
Merged

cdc: adjust sorter options to avoid Seek CPU usage exploding #11099

merged 3 commits into from
May 17, 2024

Conversation

hicqu
Copy link
Contributor

@hicqu hicqu commented May 15, 2024

What problem does this PR solve?

Issue Number: close #11078

What is changed and how it works?

  • trigger compaction more frequently
  • use a better hash function for table IDs

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

None

@hicqu
fix
e764902 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed labels May 15, 2024
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels May 15, 2024
@hicqu
fix
6929539 
Signed-off-by: qupeng <qupeng@pingcap.com>
@codecov Codecov
Copy link

codecov bot commented May 15, 2024 •
edited

Codecov Report

Attention: Patch coverage is 66.66667% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 57.7109%. Comparing base (52b4301) to head (d2621b6).
Report is 10 commits behind head on master.

Additional details and impacted files
Components Coverage Δ
cdc 62.1972% <66.6666%> (+0.3170%) ⬆️
dm 51.3116% <0.0000%> (+0.0564%) ⬆️
engine 63.4585% <ø> (+0.0141%) ⬆️
Flag Coverage Δ
unit 57.7109% <66.6666%> (+0.1968%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11099        +/-   ##
================================================
+ Coverage   57.5141%   57.7109%   +0.1968%     
================================================
  Files           851        854         +3     
  Lines        125524     126470       +946     
================================================
+ Hits          72194      72987       +793     
- Misses        47968      48095       +127     
- Partials       5362       5388        +26

@hicqu
Copy link
Contributor Author

hicqu commented May 15, 2024

/retest

@hicqu
fix
d2621b6 
Signed-off-by: qupeng <qupeng@pingcap.com>
@flowbehappy flowbehappy self-requested a review May 16, 2024 06:19
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 16, 2024

@flowbehappy: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels May 16, 2024
@hicqu
Copy link
Contributor Author

hicqu commented May 17, 2024

/merge

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 17, 2024

@hicqu: We have migrated to builtin LGTM and approve plugins for reviewing.

Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@hicqu
Copy link
Contributor Author

hicqu commented May 17, 2024

/approve

@hicqu hicqu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT2 Indicates that a PR has LGTM 2. labels May 17, 2024
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 17, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen, flowbehappy, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,asddongmen,hicqu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 17, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-05-16 13:08:48.89531572 +0000 UTC m=+1745082.652451292: ☑️ agreed by asddongmen.
  • 2024-05-17 05:35:01.623110909 +0000 UTC m=+1804255.380246482: ☑️ agreed by 3AceShowHand.

@hicqu
Copy link
Contributor Author

hicqu commented May 17, 2024

/retest

@hicqu hicqu merged commit 08aec53 into pingcap:master May 17, 2024
28 checks passed
@hicqu hicqu deleted the adjust-sorter branch May 17, 2024 06:33
@hicqu
Copy link
Contributor Author

hicqu commented May 17, 2024

/cherry-pick

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #11132.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 17, 2024
@hicqu @ti-chi-bot
This is an automated cherry-pick of pingcap#11099
ea4a2d9 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request May 17, 2024
Open
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 17, 2024
@hicqu @ti-chi-bot
This is an automated cherry-pick of pingcap#11099
b2b7e38 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #11133.

@ti-chi-bot ti-chi-bot mentioned this pull request May 17, 2024
Open
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #11134.

@ti-chi-bot ti-chi-bot mentioned this pull request May 17, 2024
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.1: #11135.

@ti-chi-bot ti-chi-bot mentioned this pull request May 17, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this pull request May 17, 2024
@ti-chi-bot
cdc: adjust sorter options to avoid Seek CPU usage exploding (#11099) (…
5a2fbdd 
…#11135)

close #11078
hicqu added a commit that referenced this pull request May 23, 2024
@ti-chi-bot @hicqu
cdc: adjust sorter options to avoid Seek CPU usage exploding (#11099) (…
3badce1 
…#11134)

Signed-off-by: qupeng <qupeng@pingcap.com>
Co-authored-by: qupeng <qupeng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@asddongmen asddongmen asddongmen approved these changes

@sdojjy sdojjy Awaiting requested review from sdojjy

Assignees
No one assigned
Labels
affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CDC consumes too much cpu, and lag goes up from 2s to 4s every few hours
5 participants
@hicqu @ti-chi-bot @flowbehappy @3AceShowHand @asddongmen