Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv: add raft.commands.pending metric #123259

Closed
nvanbenschoten opened this issue Apr 29, 2024 · 0 comments · Fixed by #123264
Closed

kv: add raft.commands.pending metric #123259

nvanbenschoten opened this issue Apr 29, 2024 · 0 comments · Fixed by #123264
Assignees
@nvanbenschoten
Labels
A-kv Anything in KV that doesn't belong in a more specific category. A-kv-observability C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-escalation-improvement Having this feature would have made an escalation easier E-quick-win Likely to be a quick win for someone experienced. O-support Originated from a customer P-2 Issues/test failures with a fix SLA of 3 months

Comments

@nvanbenschoten
Copy link
Member

nvanbenschoten commented Apr 29, 2024 •
edited by cockroach-jira-scripts

We should add a metric called raft.commands.pending which measures the total number of pending commands in all of the Replica.mu.proposals maps on a node. This would give us better insight into the source of raft.commands.reproposed.unchanged reproposals.

Jira issue: CRDB-38261
@nvanbenschoten nvanbenschoten added O-support Originated from a customer A-kv Anything in KV that doesn't belong in a more specific category. A-kv-observability E-quick-win Likely to be a quick win for someone experienced. P-2 Issues/test failures with a fix SLA of 3 months labels Apr 29, 2024
@blathers-crl blathers-crl bot added the X-blathers-untriaged blathers was unable to find an owner label Apr 29, 2024
@nvanbenschoten nvanbenschoten added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-escalation-improvement Having this feature would have made an escalation easier labels Apr 29, 2024
@cockroachdb cockroachdb deleted a comment from blathers-crl bot Apr 29, 2024
@nvanbenschoten nvanbenschoten self-assigned this Apr 29, 2024
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 29, 2024
@nvanbenschoten
kv: add raft.commands.pending metric
60ec815 
Fixes cockroachdb#123259.

This commit adds a new metric called `raft.commands.pending`. This
metric tracks the number of Raft commands that the leaseholders on a
node are tracking as in-flight. These commands will be periodically
reproposed until they are applied or until they fail, either
unequivocally or ambiguously.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Apr 29, 2024
Merged
@yuzefovich yuzefovich removed the X-blathers-untriaged blathers was unable to find an owner label Apr 30, 2024
craig bot pushed a commit that referenced this issue May 2, 2024
@nvanbenschoten
Merge #123264
f2459fb 
123264: kv: add `raft.commands.pending` metric r=kvoli,miraradeva a=nvanbenschoten

Fixes #123259.

This commit adds a new metric called `raft.commands.pending`. This metric tracks the number of Raft commands that the leaseholders on a node are tracking as in-flight. These commands will be periodically reproposed until they are applied or until they fail, either unequivocally or ambiguously.

Demonstration that this is hooked up properly while running `cockroach workload run kv --init --concurrency=512`:

<img width="1007" alt="Screenshot 2024-04-29 at 5 55 44 PM" src="https://github.com/cockroachdb/cockroach/assets/5438456/6cb7d3c9-d0d1-46d7-97f3-6a806e279aba">

Release note: None

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in a687bb1 May 2, 2024
blathers-crl bot pushed a commit that referenced this issue May 2, 2024
@nvanbenschoten
kv: add raft.commands.pending metric
d26780f 
Fixes #123259.

This commit adds a new metric called `raft.commands.pending`. This
metric tracks the number of Raft commands that the leaseholders on a
node are tracking as in-flight. These commands will be periodically
reproposed until they are applied or until they fail, either
unequivocally or ambiguously.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue May 2, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue May 2, 2024
@nvanbenschoten
kv: add raft.commands.pending metric
416e764 
Fixes #123259.

This commit adds a new metric called `raft.commands.pending`. This
metric tracks the number of Raft commands that the leaseholders on a
node are tracking as in-flight. These commands will be periodically
reproposed until they are applied or until they fail, either
unequivocally or ambiguously.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue May 2, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-kv Anything in KV that doesn't belong in a more specific category. A-kv-observability C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) C-escalation-improvement Having this feature would have made an escalation easier E-quick-win Likely to be a quick win for someone experienced. O-support Originated from a customer P-2 Issues/test failures with a fix SLA of 3 months
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv: add raft.commands.pending metric nvanbenschoten/cockroach
2 participants
@yuzefovich @nvanbenschoten