Proposal for multi-writer consistency and CRDT for directories #531
Conversation
…roposal detailing the underlying consistency model.
Codecov Report
@@ Coverage Diff @@
## master #531 +/- ##
==========================================
- Coverage 85.29% 85.28% -0.01%
==========================================
Files 152 152
Lines 28110 28110
Branches 4019 4019
==========================================
- Hits 23977 23975 -2
- Misses 3444 3446 +2
Partials 689 689
Continue to review full report at Codecov.
|
| ---------- | ||
|
|
||
| Currently Tahoe-LAFS doesn't guarantee any consistency on writes to mutable | ||
| files or directories by multiple writer nodes. This is an obstacle to |
There was a problem hiding this comment.
This comment is mostly about me convincing myself I know what's going on here but perhaps it will be useful to others as well.
https://tahoe-lafs.readthedocs.io/en/tahoe-lafs-1.12.1/write_coordination.html is where this non-guarantee is documented.
Unfortunately it doesn't give any details about why this is the case. My understanding is that there are two aspects to the issue:
- Two different versions of the ciphertext shares that can be used to reconstruct the plaintext become mixed together when considered across an entire "grid" (for n and k zfec parameters and 0 < m < k, A writes shares [0..m) vN on storage servers [0..m), B writes share [0..k) vN+1 on servers [0..k), A writes shares [m..k) on storage servers [m..k); if k < 2n then there will be fewer than n shares of either vN or vN+1 on the grid and all data is lost; if k >= 2n then there maybe be a enough shares to reconstruct both vN and vN+1 and the data a client reads will depend on which storage servers they end up ("randomly") talking to).
- The plaintext data that represents (eg) a directory can be accidentally rolled back if there are uncoordinated writers (A reads, B reads, B updates, A updates).
The former of these is the more "basic" or "low-level" because the mutable directory update code is based on being able reconstruct the plaintext. The former problem will interfere with that ability. If you do happen to avoid problems at that layer, then you may encounter further problems at the "higher level" of the directory update logic - which tries to implement a kind of test-then-write logic along with a loop and an conflict resolution policy (roughly "add both" but read the code if you want to have a complete understanding, I guess).
So, that is all to say, yes, Tahoe-LAFS indeed makes no guarantees in this space, even for directory updates, even though it contains some code to try to make concurrent directory updates not break as often.
There was a problem hiding this comment.
But actually I'm not sure about the former point. The way server selection works for mutable share updates might be different than what I expect, I should probably go look at it.
The document is still work in progress but it should be enough in shape to have a meaningful discussion on. So let's get everyone's opinions in here.
This change is