Replies: 3 comments 8 replies
-
|
@tiptop96 Ra does not dictate whether your state machine state is stored in memory or not. In RabbitMQ, there are examples of machine that keep a subset in memory (quorum queues), keep virtually everything on disk (streams) and both on disk and in memory (Khepri). The examples keep everything in memory because they are, well, basic examples that demonstrate Ra machine concepts such as state transition applications and effects. Ra will keep a certain amount of Raft state in memory but that does not include your machine's data. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the swift response! That is great. So there is no harm in writing to external storage i Does that mean the statement from this talk about no side effects inside |
Beta Was this translation helpful? Give feedback.
-
|
@tiptop96 where specifically did I mention that I recommend writing to durable storage in External state modifications is what the effects are for. How those should be reconciled with the external storage is a good question, since what Raft guarantees is the safety of the state machine transition log given a quorum of nodes online. I suggest that you take a look at rabbitmq/osiris and rabbitmq/khepri, as well as how quorum queues in RabbitMQ use Ra. |
Beta Was this translation helpful? Give feedback.
-
|
It was an assumption on my end. I am struggling to follow along. If the log is safe and I have a callback which will be called in the same order for each message in the log it seems like it would also guarantee that operations executed inside that callback would result in the external storage being consistent across nodes (unless an error occurs when writing)? However the way I understand it that is not the case for effects as they come with very limited guarantees:
I will take a look at both of them, thanks for the advice! If you feel these questions are to general for the scope of your responsibilities I understand. 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Replying state machine transitions is safe. It is side effects that are not necessarily safe to replay. So if you plan on using them, you must carefully consider whether the effects-driven operations can be made idempotent, for example.
|
Beta Was this translation helpful? Give feedback.
-
|
Oh, and we have a much smaller K/V store (mostly for demonstration and Jepsen/chaos testing purposes since Ra itself is a library). However, I don't think it really tries to store everything on disk but it can be useful as an example that's not a "simple machine" that adds together numbers but also not as involved as RabbitMQ's quorum queues or Khepri's Ra machine (that does not try to store everything on disk only for its first/current public release). |
Beta Was this translation helpful? Give feedback.
-
|
I see, that makes sense. I looked at the examples, it will probably take some time to wrap my head around |
Beta Was this translation helpful? Give feedback.
-
Practically the answer is yes, Ra (and Raft) was designed to keep the state machine state in memory. That said writing mutable/on-disk state machines is possible but it is very, very hard to do right and easy to overlook some subtle interaction that may threaten the safety properties of Raft. In addition to this you have quite high write amplification as you'd write to the Ra log then to your on disk state for all entries. Less than ideal for sure. khepri, quorum queues and any other Ra use in RabbitMQ is all in memory. Quorum queues are a special case as they delay the truncation of the log such that messages can be re-read the from the log when needed rather than kept in memory but the state machine itself is kept in memory. So yes the "rule" of no side effects inside |
Beta Was this translation helpful? Give feedback.
-
|
Awesome! I want to note that I am not planning to apply this in practice right now, I am just exploring and trying to learn, so subtle and challenging is what I am aiming for! I didn't know Raft was designed to keep the state machine in memory. Is this an assumption due to the fact that disk-based is a lot more challenging and requires precautions that are not be defined by the algorithm, or is it explicitly stated somewhere? The Quorum queues case is really intriguing. I imagine that I could use the state machine as a sort of index to find relevant log entries for queries, this would keep memory footprint low and avoid all the nasty problems of persisting the state machine. Is that a reasonable approach? As previously mentioned, if any question is to general for this discussion I very much understand. Huge thanks for answering my questions! |
Beta Was this translation helpful? Give feedback.
-
|
Raft was designed to keep the state of a replicated state machine log in memory, which is not the same as keeping everything related to the application in memory. In fact, Raft libraries do not really know or dictate what you use them for. Ra just knows it has a state machine that must be replicated, not much else. Storing data on disk is a perfectly fine case as long as you have answers to a few things
and so on. There is no shortage of data stores and data services that use Raft in the industry. Often they delegate on disk storage unrelated to Raft state to a local store (such as LevelDB/RocksDB but it can be anything). Raft is used then for a few specific things:
Everything else is not related to Raft but must fit the Raft approach to distributed replicated state. For example, having hundreds of cluster nodes will not work very well with Raft. Having multiple clusters (e.g. a quorum queue is a Raft cluster in RabbitMQ) and 7-11 replicas in a cluster of, say, 101 node should work reasonably well. Raft is not a database developer's toolkit, it only solves a narrow set of problems well (and with certain well known limitations). Node-local storage, indexing, backups, restore, what querying language you support — all of those decisions Raft or Ra won't help you with. |
Beta Was this translation helpful? Give feedback.
-
|
That is a super in-depth answer, thanks! The distinction between what Raft is and what you could do with Raft, the limitations of it, and the list of things to keep in mind when working with persistent storage are all really helpful. Really appreciate it! 🙌 |
Beta Was this translation helpful? Give feedback.
-
This assumption might be wrong, but from what I gather
rakeeps the whole state in memory. If this is not true please correct me. I am interested in usingrato update some external storage on each node in order to not have the size of my state be constrained by available RAM.The first thing that comes to mind is to have the state machine always return an empty state and just write to some external store whenever we receive a write or do a restore. However this would be a side effect, which I understand is not advisable.
Is there a way to not keep the complete state in memory?
Maybe this
rawasn't designed for this use case, perhaps it isn't even a valid use case. 😅Beta Was this translation helpful? Give feedback.
All reactions