kv: add raft.commands.pending metric

[nvanbenschoten]

Fixes #123259.

This commit adds a new metric called raft.commands.pending. This metric tracks the number of Raft commands that the leaseholders on a node are tracking as in-flight. These commands will be periodically reproposed until they are applied or until they fail, either unequivocally or ambiguously.

Demonstration that this is hooked up properly while running cockroach workload run kv --init --concurrency=512:

Release note: None

[cockroach-teamcity]

This change is 

[kvoli]

Reviewed 6 of 6 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @miraradeva)

[miraradeva]

Is it true that RaftCommandsProposed = RaftCommandsPending + RaftCommandsReproposed + RaftCommandsReproposedLAI?

Reviewed 6 of 6 files at r1, all commit messages.
Reviewable status: complete! 2 of 0 LGTMs obtained (waiting on @nvanbenschoten)

---

pkg/kv/kvserver/replica_metrics.go line 56 at r1 (raw file):

	BehindCount           int64
	PausedFollowerCount   int64
	RaftProposalCount     int64

It's a little confusing that there is another metric here called RaftCommandsProposed. Should this one be RaftCommandsPending for consistency?

[nvanbenschoten]

TFTRs!

bors r=kvoli,miraradeva

Is it true that RaftCommandsProposed = RaftCommandsPending + RaftCommandsReproposed + RaftCommandsReproposedLAI?

Not quite. The other three are counters, not gauges, so they'll continue to tick up even as RaftCommandsPending bounces off of 0.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 2 stale) (waiting on @kvoli and @miraradeva)

---

pkg/kv/kvserver/replica_metrics.go line 56 at r1 (raw file):

Previously, miraradeva (Mira Radeva) wrote…

It's a little confusing that there is another metric here called RaftCommandsProposed. Should this one be RaftCommandsPending for consistency?

Good point, done.

[craig]

Build succeeded:

• examples_orms
• lint
• macos_arm64_build
• check_generated_code
• linux_amd64_fips_build
• docker_image_amd64
• unit_tests
• linux_arm64_build
• windows_build
• acceptance
• macos_amd64_build
• linux_amd64_build
• local_roachtest
• local_roachtest_fips

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from a687bb1 to blathers/backport-release-23.1-123264: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}