Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDC consumes too much cpu, and lag goes up from 2s to 4s every few hours #11078

Closed
fubinzh opened this issue May 11, 2024 · 3 comments · Fixed by #11099
Closed

CDC consumes too much cpu, and lag goes up from 2s to 4s every few hours #11078

fubinzh opened this issue May 11, 2024 · 3 comments · Fixed by #11099
Assignees
@hicqu
Labels
affects-6.5 affects-7.1 affects-7.5 affects-8.1 area/ticdc Issues or PRs related to TiCDC. severity/major This is a major bug. type/bug This is a bug.
Projects
Question and Bug Reports

Comments

@fubinzh
Copy link

fubinzh commented May 11, 2024

What did you do?

  1. TiDB cluster with 2 CDC nodes (24c32g)
    cdc config:
 config: |
      log-file = "/var/lib/ticdc/log/ticdc.log"

      [debug]
        [debug.kv-client]
          frontier-concurrent = 32

TiKV cdc.min-ts-interval = "50ms"
2. CDC simiple protocol changefeed create to sync 4k tables (throughput <20k, rowwidth<10k)

What did you expect to see?

CDC lag should be stable

What did you see instead?

CDC lag goes up from 2s to 4s every few hours

image
image
image
image

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

[root@upstream-tikv-0 /]# /tikv-server -V
TiKV
Release Version:   8.1.0
Edition:           Community
Git Commit Hash:   6ebf7ef3a1627a59062ddfbe32b33006df881950

TiCDC version (execute cdc version):

[root@upstream-ticdc-0 /]# /cdc version
Release Version: v8.1.0-lock-test
Git Commit Hash: 3afce276e3a79788be4e294bd4f322d844cffea7
Git Branch: heads/refs/tags/v8.1.0-lock-test
UTC Build Time: 2024-05-10 03:24:05
Go Version: go version go1.21.6 linux/amd64
Failpoint Build: false
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug This is a bug. labels May 11, 2024
@github-actions github-actions bot added this to Need Triage in Question and Bug Reports May 11, 2024
@flowbehappy flowbehappy assigned lidezhu and hicqu and unassigned lidezhu May 11, 2024
@fubinzh
Copy link
Author

fubinzh commented May 13, 2024

/severity major

@flowbehappy flowbehappy changed the title CDC lag goes up from 2s to 4s every few hours CDC consumes too much cpu, and lag goes up from 2s to 4s every few hours May 13, 2024
@hicqu
Copy link
Contributor

hicqu commented May 14, 2024

图片

lag increasings always keep about 20 minutes, and intervals are always about 6h. I think there are some period changes.

@hicqu
Copy link
Contributor

hicqu commented May 15, 2024

I find some doubts:
图片
图片
图片

Maybe we can adjust some parameters to fix it.

@hicqu hicqu mentioned this issue May 15, 2024
Merged
Question and Bug Reports automation moved this from Need Triage to Done May 17, 2024
This was referenced May 17, 2024
Open
Open
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 17, 2024
@ti-chi-bot
cdc: adjust sorter options to avoid Seek CPU usage exploding (#11099) (…
5a2fbdd 
…#11135)

close #11078
@asddongmen asddongmen mentioned this issue May 21, 2024
Closed
@flowbehappy flowbehappy mentioned this issue May 23, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hicqu hicqu

Labels
affects-6.5 affects-7.1 affects-7.5 affects-8.1 area/ticdc Issues or PRs related to TiCDC. severity/major This is a major bug. type/bug This is a bug.
Projects
Question and Bug Reports
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cdc: adjust sorter options to avoid Seek CPU usage exploding hicqu/ticdc
4 participants
@flowbehappy @fubinzh @hicqu @lidezhu