-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
khepri_machine: Add command deduplication mechanism #250
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #250 +/- ##
==========================================
- Coverage 89.70% 89.64% -0.06%
==========================================
Files 20 21 +1
Lines 3002 3091 +89
==========================================
+ Hits 2693 2771 +78
- Misses 309 320 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
25798e6
to
1196cb6
Compare
1196cb6
to
5936457
Compare
I need to bump the state machine version and convert the state. |
d1ee73c
to
16394e5
Compare
e29bf66
to
584a733
Compare
The pull request is ready for review. The machine version is taken into account, so if the machine is too old, the public API won’t use the dedup mechanism. I tested this patch successfully with |
584a733
to
ac9dbbc
Compare
ac9dbbc
to
0296adb
Compare
5d1bb39
to
e76d572
Compare
e76d572
to
04b2767
Compare
[Why] When `ra:process_command()` returns an error like `{error, shutdown}`, it doesn’t mean the command was nod appended to the Ra log. It’s to be expected with networks, but something I forgot in Khepri. The retry mechanism we have will call `ra:process_command()` again after specific errors. Therefore, a command may be appended and thus applied twice. We observed exactly this in a testcase of the rabbitmq-shovel plugin. To help us work on a solution, we need a smaller testcase. [How] This new testcase reproduces the problem by spamming a Khepri store with transactions that bump a counter, while the cluster is changed to trigger an election. At the end, the counter should be equal to the number of times the transaction was executed.
[Why] The "client" side of `khepri_machine` implemented in `process_sync_command/3` have a retry mechanism if `ra:process_command/3` returns an error such as `noproc`, `nodedown` or `shutdown`. However, this retry mechanism can't tell if the state machine already received the command and just couldn't reply, for instance because there is a node stopping or a change of leadership. Therefore, it's possible that the same command is submitted twice and thus processed twice. That's ok for idempotent commands, but it may not be alright for all transactions for example. That's why we need a deduplication mechanism that ensures the same command is not applied multiple times. [How] Two new commands are introduced to implement the deduplication system: * #dedup{} which is used to wrap the command to protect and assign a unique reference to it * #dedup_ack{} which is used at the end of the retry loop to let the state machine know that the "client" side received the reply When the state machine receives a command wrapped into a #dedup{} command, it will remember the reply for the initial processing of that command. For any subsequent copies of the same #dedup{} (based on the unique reference), the state machine will not apply the wrapped command and will simply returned the reply it remembered from the first application. Later when the state machine receives a #dedup_ack{}, it will drop the cached reply for that reference. Just in case the client never sends a #dedup_ack{}, the state machine will drop any expired cached entries. The expiration time is based on the command timeout. If it's infinity, it defaults to 15 minutes. This whole deduplication mechanism can be enabled or disabled through the new `protect_against_dups` command option which takes a boolean. This option is off by default, except for R/W transactions. Thus if the caller knows the transation is idempotent, it can decide to turn the dedup mechanism off. Because the state machine's state grows with a new field and handles two new commandes, we bump the machine version from 0 to 1. V2: We now use the `effective_machine_version` counter provided by `ra_counters:counters/2` if it is available as it is faster than querying the Ra server. If the counter is unavailable, we fall back to the query. The new counter is added by rabbitmq/ra#426 and will be used once a Ra release contains this change.
04b2767
to
c6465c2
Compare
Why
The "client" side of
khepri_machine
implemented inprocess_sync_command/3
have a retry mechanism ifra:process_command/3
returns an error such asnoproc
,nodedown
orshutdown
.However, this retry mechanism can't tell if the state machine already received the command and just couldn't reply, for instance because there is a node stopping or a change of leadership.
Therefore, it's possible that the same command is submitted twice and thus processed twice.
That's ok for idempotent commands, but it may not be alright for all transactions for example. That's why we need a deduplication mechanism that ensures the same command is not applied multiple times.
How
Two new commands are introduced to implement the deduplication system:
#dedup{}
which is used to wrap the command to protect and assign a unique reference to it#dedup_ack{}
which is used at the end of the retry loop to let the state machine know that the "client" side received the replyWhen the state machine receives a command wrapped into a
#dedup{}
command, it will remember the reply for the initial processing of that command. For any subsequent copies of the same#dedup{}
(based on the unique reference), the state machine will not apply the wrapped command and will simply returned the reply it remembered from the first application.Later when the state machine receives a
#dedup_ack{}
, it will drop the cached reply for that reference.Just in case the client never sends a
#dedup_ack{}
, the state machine will drop any expired cached entries. The expiration time is based on the command timeout. If it's infinity, it defaults to 15 minutes.This whole deduplication mechanism can be enabled or disabled through the new
protect_against_dups
command option which takes a boolean. This option is off by default, except for R/W transactions.Thus if the caller knows the transation is idempotent, it can decide to turn the dedup mechanism off.
V2: We now use the
effective_machine_version
counter provided byra_counters:counters/2
if it is available as it is faster than querying the Ra server. If the counter is unavailable, we fall back to the query. The new counter is added by rabbitmq/ra#426 and will be used once a Ra release contains this change.