As predicted in the last update, adding support for files and hard links to our directory tree CRDT went smoothly, and we achieved convergence in our randomized tests on Monday. Because hard links make it possible for the same file to appear in multiple locations, many code paths needed to be updated to work in terms of references rather than files. Happily, we had already anticipated hard links by allowing a file to be associated with multiple parent refs, so the path was mostly paved. Once we add support for file contents and confirm that everything works in an end-to-end test, we plan to post an in-depth write-up on the directory tree CRDT and do a documentation pass on the timeline module.
The file support added last week assumes that all files are empty. To allow files to be associated with editable content, we're adapting the buffer module from xray_core to work with Memo's new B-tree. The primary difference between the previous B-tree implementation and the new one is support for storing the tree's nodes in a database. This will allow us to store a file's entire history without loading old fragments into memory, but it also means that many methods now have the potential to perform I/O with the database and encounter I/O errors.
We'll need to adjust the Buffer APIs slightly to account for this potential. For example, we can no longer return an iterator that implements the Iterator trait, since next would need to return a Result type. We're also dropping some of the previous buffer's support for Xray's RPC system because we anticipate dealing with network interactions differently in Memo. We don't have complete clarity on our plans for dealing with networking just yet, but it makes sense to keep our assumptions minimal at this stage.
Once we get buffers implemented against our new B-tree, we'll need to integrate them into our timeline. We plan to maintain a mapping between file ids and the buffers that contain their contents, but the details will become clearer once we get into it. Buffers will need to be integrated with up to three distinct sources of I/O: the file system for reading/saving contents, the network for collaboration, and the database for history persistence. It should be a fun design problem to give them a convenient API while addressing all of those concerns.