Added new commit statistics metrics #10993

oleg68 · 2023-10-18T16:20:06Z

Problem statement

Now it is hard to tune a fdb cluster for a write-intensive workload.

Description

While tuning a fdb cluster with a write-intensive application often the bottleneck is the commit latency: when trying to parallel degree of transactions payload, the commit latency grows and prevents increasing the transaction throughput.

There are lots of conditions and knobs influencing the commit latency: number of commit proxies, number of resolvers, number of tlog processes, commit batching knobs: MAX_COMMIT_BATCH_INTERVAL, COMMIT_TRANSACTION_BATCH_INTERVAL_MAX, COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA and others.

But for now, there is no any information, where is the root cause of the high commit latency, so it is unclear, what is to be changed.

Proposal

To collect and to log latency statistics from parts of commit workflow:
- Waiting for a batch
- Preresolution (allocating a commit version)
- Resolution
- Postresolution
- Pushing to TLog
- Replying
To collect and log the batch size statistics
- The number of transactions in one batch
- The total bytes in the transaction batch

PR content

This PR implements this proposal: the following new metrics are logged and exposed in status json:

CommitBatchTransactions (commit_batch_transactions) - the number of transactions in one batch
CommitBatchBytes (commit_batch_bytes) - the total size of one commit batch in bytes
CommitBatchingWaiting (commit_batching_waiting) - the time while the transaction is waiting for the batch becomes ready
CommitPreresolutionLatency (commit_preresolution_latency) - the time of the Preresolution phase
CommitResolutionLatency (commit_resolution_latency) - the time of the Resolution phase
CommitPostResolutionLatency (commit_resolution_latency) - the time of the Postresolution phase
CommitTLogLoggingLatency (commit_tlog_logging_latency) - the time of the TlogLogging phase
CommitReplyLatency (commit_reply_latency) - the time of Reply phase

The sum of the added *_latency_mean metrics should be equal to the commit_latency_mean metrics that already exists

foundationdb-ci · 2023-10-18T16:32:57Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 03cf4dc
Duration 0:12:41
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-18T16:36:19Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 03cf4dc
Duration 0:16:01
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-18T16:37:55Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 03cf4dc
Duration 0:17:40
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-18T16:38:04Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 03cf4dc
Duration 0:17:48
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-10-18T16:38:30Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 03cf4dc
Duration 0:18:14
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-18T16:42:54Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 03cf4dc
Duration 0:22:37
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

(cherry picked from commit ed145d9b9b0b72aa1b25acee2fc8b5dc135d8014) (cherry picked from commit 54996a8) (cherry picked from commit ae8aef7) # Conflicts: # fdbserver/CommitProxyServer.actor.cpp

foundationdb-ci · 2023-10-19T07:41:23Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 8ae2ad1
Duration 0:20:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-19T07:52:14Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 8ae2ad1
Duration 0:31:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-19T08:06:55Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 8ae2ad1
Duration 0:46:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-19T08:27:28Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 8ae2ad1
Duration 1:06:50
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-19T08:27:42Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 8ae2ad1
Duration 1:07:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-19T08:45:07Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 8ae2ad1
Duration 1:24:26
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

oleg68 · 2023-10-19T11:25:43Z

Seems two first tests faled due an environment problems:

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

jzhou77 · 2023-10-27T20:11:49Z

fdbserver/Status.actor.cpp

+
+			TraceEventFields const& commitBatchTransactions = metrics.at("CommitBatchTransactions");
+			if (commitBatchTransactions.size()) {
+				obj["commit_batch_transactions"] = addLatencyStatistics(commitBatchTransactions);


Since this modified status json, you need to update documentation/sphinx/source/mr-status-json-schemas.rst.inc and fdbclient/Schemas.cpp as well.

I added new elements to fdbclient/Schemas.cpp, but now there is no cluster.processes..roles objetcs in documentation/sphinx/source/mr-status-json-schemas.rst.inc at all.

I can add the description of the new metrics, but I cannot describe all other elements of roles.

(cherry picked from commit 25fe53b) (cherry picked from commit 5b54ebf)

foundationdb-ci · 2023-10-31T12:13:45Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: a6b8f54
Duration 0:21:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-31T12:25:49Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: a6b8f54
Duration 0:33:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-31T12:39:20Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: a6b8f54
Duration 0:47:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-31T13:09:33Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: a6b8f54
Duration 1:17:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-31T13:15:37Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: a6b8f54
Duration 1:22:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-10-31T13:22:15Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: a6b8f54
Duration 1:29:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77

@sbodagala can you run a correctness for this PR before merging?

sbodagala · 2023-11-03T16:23:43Z

@sbodagala can you run a correctness for this PR before merging?

Ran a correctness test (with 100000 simulation tests). The test run stopped after doing 99994 test runs, at which point 10 simulation tests have failed (not all of these failures may have been caused by this change set though).

ended=99994 fail=10 fail_fast=10 max_runs=100000

Majority of the failures are on tests in "tests/restarting/from_7.3.0/" directory:

SourceVersion="9cbaacab35d8c1e1230497afd6fec9fbd177cf51"

WillRestart="0", RandomSeed="1062210747", BuggifyEnabled="1", TestFile="tests/fast/RandomUnitTests.toml"

WillRestart="1", RandomSeed="1380708830", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-1.txt"

WillRestart="0",  RandomSeed="1380708831", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-2.txt"

WillRestart="1", RandomSeed="2128522034", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-1.txt"

WillRestart="0", RandomSeed="2128522035", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-2.txt"

 WillRestart="0", RandomSeed="398579556", BuggifyEnabled="0", TestFile="tests/fast/BlobGranuleVerifySmall.toml"

 WillRestart="1", RandomSeed="110402234", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-1.toml"

WillRestart="0", RandomSeed="110402235", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-2.toml"

WillRestart="1", RandomSeed="3174333056", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-1.toml"

WillRestart="0", RandomSeed="3174333057", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-2.toml"

WillRestart="1", RandomSeed="869995033", BuggifyEnabled="1", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-1.toml"

WillRestart="0", RandomSeed="869995034", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/DrUpgradeRestart-2.toml"

WillRestart="1", RandomSeed="630498846", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/SnapTestRestart-1.txt"

WillRestart="0", RandomSeed="630498847", BuggifyEnabled="0", TestFile="tests/restarting/from_7.3.0/SnapTestRestart-2.txt"

Oleg Samarin added 2 commits October 19, 2023 10:13

Added new commit metrics

a314b1c

(cherry picked from commit ed145d9b9b0b72aa1b25acee2fc8b5dc135d8014) (cherry picked from commit 54996a8) (cherry picked from commit ae8aef7) # Conflicts: # fdbserver/CommitProxyServer.actor.cpp

Renamed a variable

8ae2ad1

oleg68 force-pushed the os-main-commit-metrics branch from 03cf4dc to 8ae2ad1 Compare October 19, 2023 07:20

oleg68 mentioned this pull request Oct 19, 2023

XDB-218 Added new commit statistics metrics owtech/foundationdb#104

Merged

jzhou77 requested a review from sbodagala October 26, 2023 20:20

sbodagala previously approved these changes Oct 26, 2023

View reviewed changes

jzhou77 requested changes Oct 27, 2023

View reviewed changes

Added new metrics to the json schema

a6b8f54

(cherry picked from commit 25fe53b) (cherry picked from commit 5b54ebf)

oleg68 dismissed sbodagala’s stale review via a6b8f54 October 31, 2023 11:52

oleg68 requested a review from jzhou77 October 31, 2023 12:05

jzhou77 approved these changes Oct 31, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new commit statistics metrics #10993

Added new commit statistics metrics #10993

oleg68 commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 18, 2023

foundationdb-ci commented Oct 19, 2023

foundationdb-ci commented Oct 19, 2023

foundationdb-ci commented Oct 19, 2023

foundationdb-ci commented Oct 19, 2023

foundationdb-ci commented Oct 19, 2023

foundationdb-ci commented Oct 19, 2023

oleg68 commented Oct 19, 2023

jzhou77 Oct 27, 2023

oleg68 Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

foundationdb-ci commented Oct 31, 2023

jzhou77 left a comment

sbodagala commented Nov 3, 2023

Added new commit statistics metrics #10993

Are you sure you want to change the base?

Added new commit statistics metrics #10993

Conversation

oleg68 commented Oct 18, 2023

Problem statement

Description

Proposal

PR content

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Oct 18, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Oct 19, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

oleg68 commented Oct 19, 2023

jzhou77 Oct 27, 2023

Choose a reason for hiding this comment

oleg68 Oct 31, 2023

Choose a reason for hiding this comment

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Oct 31, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

jzhou77 left a comment

Choose a reason for hiding this comment

sbodagala commented Nov 3, 2023