How does percistence work? What is written to disk and when?

[hpvd]

I was reading around for some time now, but I couldn't really understand how persistence works in detail.
Other in-memory databases use some kind of snapshot technology (RAM->disk) and/or a writeaheadlock (WAL on disk)...
So for terminusdb:

1. what is written to disk and when?

2. what are the implications from this?
   a. when is the data accessible for read access?
   b. how much memory (RAM) is needed (factor x to data)?

3. and from terminusdbs special feature "git for data":
   a. which part of data is typically accessible directly from RAM (fast)? Only the latest state or the complete history?
   b. how big is a "typical" (yes I know there is nothing like this ;-) history when a DB grows to e.g. 1GB (latest state)?
   c. how much do I have to pay (CPU/Latency) for the git functionality with latest version of terminus DB?
   - on read access?
   - on write?
   d. can I simply pay with more CPU cores and reach (nearly) the same latency (good parallelization)?

   Sorry for so many questions, but they may also help others to get the complete picture of terminusdb (->trust->users...)

[GavinMendelGleason]

1. Database writes are tiered - there are three immutable graphs (actually the schema is also a graph but typically will be substituted for the instance graph during updates) and one label
   a. Instance Graph
   b. Repository Graph
   c. Database Graph
   d. Label

These graphs are written to disk with the database graph being absolutely last. The database graph contains reference to the repository graph which in turn knows which branch is associated with the latest update in the instance graph. The label is atomically updated with a revision number only after a successful write of the database graph. This can be either to a label server or to a file depending on the setup.

All of this ensures that all transactions are ACID.

2. This means that:
   a. The data is immediately available if the server which committed the write is the same as the read server. Otherwise, if we are using a different server on the same store it is available after load of the layer from disk.
   b. The ram needed as compares to the transaction depends on what sort of data is being processed, but for JSON documents we generally need around 10x as much ram in a transaction as the total JSON document list size.

Note, since everything is immutable, we can share any number of readers over the same store, and get data distribution using NFS.

3. In terms of Git for Data
   a. The part accessible from RAM is whatever is queried. We keep an LRU of layers and so it depends on which layers are being used. If you are using the complete history all the time and have enough RAM, it might all be in RAM.
   b. History can be hundreds or thousands of commits. Commits on our TerminusCMS server which track new users logins, data products etc. are extremely long.
   c. This depends on optimisations. If you run an optimisation on a commit then you'll end up with a new pseudo-layer which is a bit like a squash called a "delta-rollup" which represents the combined state of the updates. You will retain the historical information but will have an optimised version which is fast to read. Queries will happen against this pseudo-layer unless you ask specific questions about what changed in that commit, or what its parent was.
   d. You can run loads of readers against a given store easily and this will improve performance for queries a lot. For writing we are more bottlenecked and doing parallel writes requires recombinding from seperate transaction domains

   Currently our transaction domains are per data product. We know how to make them per branch but we haven't gotten to it yet unforunately, and it's even possible to get finer grained than that, but pushing further is significantly more complex. We'll try to get to the branch level transaction domain soon.

[hpvd]

sounds great! Many thanks for your work! Looking forward to this!

[hpvd]

just a question:
regarding "We keep an LRU of layers and so it depends on which layers are being used"
-> A) does this "least recently used" is also about write or only read?

Im asking, because of being interested in read performance of the latest added (newest) values.
(most simple use case: a website with many changes but only few visitor (readers), so nearly every visit is a visit (read) of a new version of the site)

If not,
-> B) is there an easy way to always pre-warm the cache (disk-> RAM) to make even the very first read fast?

Within the discussion topic
Outcome of ingest speed improvement discussion
https://github.com/orgs/terminusdb/discussions/994
there is a method described in paragraph
Build in-memory and move to disk only when we decide to keep
which look like it may also (additional to its original intention) help in this direction,
-> C) would it?

[GavinMendelGleason]

A) If read servers are round robin, it will likely mean loading the latest layer from disk. If the changes are small this probably isn't so bad, and the rest (older layers) will already be loaded.

B) You can just open the latest layer to pre-warm it. We are intending to add pub sub on commits so updated layers could tell all readers to pre-warm once we get pub sub implemented.

[hpvd]

many thanks for getting back that fast!
B) great idea to use this for pre-warming (did already read about it in #1200)

[hpvd]

for anyone looking into this, there is now a plugin to subscribe on any change: #1200 (comment)