Proposal: a persistent Virtual FileSystem #1973
Replies: 3 comments 9 replies
-
@WebReflection can you elaborate on
? Wouldn't the underlying interpreters provide that out of the box? |
Beta Was this translation helpful? Give feedback.
-
MicroPython has a VFS which is transparent to the user if they don't want to think about it, but also allows mounting arbitrary filesystems (even ones defined in Python). By default on the BUT, it's all synchronous, because it has to match the normal CPython API, eg the For microcontrollers an async filesystem would be useful but we haven't made any progress in that direction. I would suggest that PyScript investigate what an async filesystem would look like, IMO that is more future proof than a synchronous filesystem. |
Beta Was this translation helpful? Give feedback.
-
I've been thinking a lot about this. I'd love some clarifications/comments:
These questions are offered in a spirit of technical exploration. As always, remember I have a Winnie-the-Pooh small brain so likely missing something obvious. 😁 As always, comments, suggestions, constructive critique and ideas are most welcome. 🚀 |
Beta Was this translation helpful? Give feedback.
-
In more than an occasion our users asked how to persist data in PyScript or how to share the FileSystem between main and workers.
There are, in my opinion, at least 2 ways to go here:
Case 1: We provide the API
Possibly the most user-friendly way to go, we could use this project of mine to actually expose a
pyfs
module or apyscript.fs
module that brings in most common primitives to read, write, crawl, or stat a filesystem. We might discuss if it makes sense to have the API always async or if we want to expose it only in workers and have it fully sync thanks to Atomics.Workers will incrementally be then the way to provide super-powers to PyScript but arguably, as we do have
async/await
ability in python, even top-level when needed, I feel like we might be good with an always async approach so that on main, one could land MicroPython thanks to its faster bootstrap and on workers either MicroPython or Pyodide and these can share the same persistent storage.On the other hand, this approach is manual and it might bring some confusion because it won't be fully integrated with the already provided VFS from either Pyodide or MicroPython ...
Case 2: We override the default mount point
This option should work out of the box (but I haven't tried yet) and it should make any FS related operation persistent on every further run of the same project.
My only concern here is that all interpreters will use a library a part to perform
syncfs()
operations and it might not be easy to reason about the need of async/await between operations, although this is theoretical.In this scenario we'll have a
persistent_vfs = true
flag that, if enabled, automatically un-mount the MEMFS and mount the IDBFS one and we do callsyncfs()
on writes ... but I wonder if there are possible gotchas I am not considering in here.@hoodmane @dpgeorge @ntoll @antocuni @fpliger what are your thoughts? Any preference? Any blocker? Any caveat to consider? 🤔
Beta Was this translation helpful? Give feedback.
All reactions