Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Compared to other filesystems, littlefs's handling of open files may come across as a bit odd, especially when you open the same file with multiple file handles. This commit addresses this by forcing all open readable file handles to be consistent if any open writable file handle is synced though either lfs_file_sync or lfs_file_close. This means open readable file handles always mirror the state of the filesystem on disk. To do this we again rely on the internal linked-list of open file handles, marking files as clean, copying over the written file cache, and synchronizing any custom attributes attached to the file handles. Note, this still needs cleanup and tests --- Why was the previous behavior? One of the nifty mechanism in littlefs is the ability to have multiple device-side copies of a file that share copy-on-write blocks of data. This is very useful for staging any amount of changes, which may live either in RAM caches or allocated-but-not-committed blocks on disk, that can be atomically updated in a single commit. After this change, littlefs still uses this update mechanism to track open files, meaning if you lose power, the entire file will revert to what was written at the last lfs_file_sync. Because this mechanism already exists, it was easy enough to rely on this to handle multiple open file handles gracefully. Each file handle gets its own copy-on-write copy of the contents at time of open, and and writes are managed independently of other open files. This behavior was idiosyncratic, but consistent, though after some time enough users raised feedback that this behavior needed to be reassessed. Now multiple open files should conform to what's found in other filesystem APIs, at a small code cost to manage syncing open files.
… dirty
This is a bit of a complicated area for the custom-attribute API without
much precedent. littlefs allows users to provide custom attributes in
the lfs_file_config struct, which get written along with other file
metadata.
Sounds great on paper, but the devil is in the details. When does the
metadata actually get written?
What about this case?
lfs_file_opencfg(lfs, file, "path", LFS_O_WRONLY, cfg_with_attrs);
lfs_file_close(lfs, file); // does not write metadata
This normally doesn't write out metadata! We've opened the file for
writing, but made no changes, so normally littlefs doesn't bother to
commit anything to disk.
Before, as a courtesy, littlefs marked the file as dirty if it noticed
the file was opened for writing with custom attributes, but this is
inaccurate could to leave to problems after a file is synced:
lfs_file_opencfg(lfs, file, "path", LFS_O_WRONLY, cfg_with_attrs);
lfs_file_sync(lfs, file);
change_attrs();
lfs_file_close(lfs, file); // does not write metadata
Unfortunately, it isn't easy to know when metadata needs to be written.
Custom attributes are provided as read-only pointers to buffers which
may be updated without additional filesystem calls, this means we don't
know if custom attributes have actually changed on the device side. If
they haven't changed, writing out metadata on every sync would be
wasteful.
Another solution would be to compare our device-side attributes with
the disk-side attributes every sync, but that would be even more
expensive.
---
So for now, the simpliest and most efficient solution wins. Custom
attributes attached to open files, are not written unless the file data
itself changes.
Note that explicit calls to lfs_setattr always update on-disk
attributes, and opening a file with LFS_O_CREATE | LFS_O_TRUNC will also
always update the on-disk attributes (though not with just LFS_O_CREAT!).
There are a few ways we could provide an API that manually forces a write
of custom attributes, such as lfs_file_setattr, though without dynamic
memory, providing these APIs gets a bit complicated. So for now we will
see if users run into issues with the current scheme.
This was provided as a courtesy to hopefully make custom attributes more easy to use, however the zeroing turned out to be a bit complicated when syncing custom attributes across multiple open files. Implicitly zeroing trailing buffer space is also inconsistent with the other APIs in the filesystem, such as lfs_file_read, so this commit removes the behavior. If you need to handle differently sized custom attributes, you can either pre-zero the custom attribute buffers, or use lfs_getattr to find the on-disk size of custom attributes explicitly.
Related to changes to custom attribute and open file syncing
LFS_O_SNAPSHOT brings back some of littlefs's idiosyncratic behavior removed in the changes to open file syncing in a form that may be more useful for users. LFS_O_SNAPSHOT allows you to open a "snapshot" of a file. This is a cheap, local copy of a file who's changes are not reflected on disk. Internally, snapshot files use the same mechanism as pending writes. A separate, copy-on-write CTZ skip-list is created, with read-only references to the existing data blocks until a write occurs. The difference is that snapshot files are not enrolled in the mlist, meaning they won't get updates from open file syncs, and during close their contents are simply discarded. As an extra benefit, LFS_O_CREAT | LFS_O_SNAPSHOT is equivalent to Linux's O_TMPFILE, making it easy to create temporary, unnamed files. This may be useful for embedded development, where unnamed flash-backed buffers may provide a slower, but larger, alternative to RAM-backed buffers.
geky
added
the
needs major version
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If #491 is merged, it means breaking changes to the API, and a major version bump of some form (but not disk-breaking). This means it's a good time to look at open issues we couldn't fix early without breaking the API.
Cross-file syncing
These changes originally intended to address the issues raised around the behavior of multiple open file handles referencing the same underlying file. Specifically what happens to the other open file handles when you write to one of them?
This issue is explained in more detail by @BrianPugh in #483 and joltwallet/esp_littlefs#18
To address this, and make littlefs behave more like other filesystems:
On
lfs_file_syncorlfs_file_close, littlefs now updates all file handles (lfs_file_ts) open for reading.This means readable file handles should always reflect the state of the file on disk.
Some caveats:
lfs_file_syncorlfs_file_close. Just like the disk, the state of the file at open is preserved unless another file handle commits new data through eitherlfs_file_syncorlfs_file_close.constin read-only memory.Custom attribute tweaks
While working on cross-file syncing, I ran into some related issues (#183 (comment)) on syncing custom attributes. This highlighted some subtle inconsistencies with custom attributes, so I ended up modifying their behavior in a couple ways:
In either an explicit
lfs_setattr, orlfs_file_sync/lfs_file_closewith custom attributes, littlefs now updates the custom attributes attached to any file handles open for reading.Just like the file's data, the file's custom attributes should now reflect the state on disk.
Reduced when custom attributes are written to strictly when a file is dirty.
This means if you open a file for writing with attached custom attributes, and then close the file without modification, the custom attributes will not be written out. Previously littlefs would always write out custom attributes on the first
lfs_file_sync, but this was potentially misleading (if you expectlfs_file_syncto always write custom attributes) and ran the risk of creating unnecessary commits.Note that files open with
LFS_O_CREAT | LFS_O_TRUNCare still guaranteed to write out attached custom attributes.This is explained more in b19a51c.
Trailing space in custom attribute buffers are no longer zeroed.
This previously was a courtesy to help handle custom attributes that may have a different size on disk since
lfs_file_opencfghas no way to indicate the on-disk file size. However this was a bit difficult to maintain with open file syncing, and was a bit inconsistent.If you need to handle differently sized custom attributes, you can either pre-zero the custom attribute buffers, or use
lfs_getattrto find the on-disk size of custom attributes explicitly.LFS_O_SNAPSHOT
Instead of completely removing littlefs's idiosyncratic behavior, I'm considering moving the more useful bits into the LFS_O_SNAPSHOT flag for
lfs_file_open.What is LFS_O_SNAPSHOT?
LFS_O_SNAPSHOT allows you to open a "snapshot" of a file. This is a cheap, local copy of a file who's changes are not reflected on disk.
Internally, snapshot files use the same mechanism as pending writes. A separate, copy-on-write CTZ skip-list is created, with read-only references to the existing data blocks until a write occurs. The difference is that snapshot files are not enrolled in the mlist, meaning they won't get updates from open file syncs, and during close their contents are simply discarded.
As an extra benefit,
LFS_O_CREAT | LFS_O_SNAPSHOTis equivalent to Linux's O_TMPFILE, making it easy to create temporary, unnamed files.This may be extra useful for embedded development, where unnamed flash-backed buffers may provide a slower, but larger, alternative to RAM-backed buffers.
I'm still on the fence on if LFS_O_SNAPSHOT is a worthwhile addition, so I'd be happy to receive any feedback. A quick build showed a ~50 byte code cost for the feature, but I will update with a better measurement if I can.
Initial issues with open-syncing raised by @BrianPugh
Related to #483, #183