Filesystem refactoring

[pwmarcz]

This describes the current state of my filesystem refactoring project.

Legend:

✔️ Done (merged to master)
🚧 In progress (usually has a PR open)
⭐ Next (usually will be unlocked by current "in progress")

Bug fixes and new features

✔️ gramineproject/graphene#952 Listing a directory doesn't show mountpoints
✔️ lseek on directories (PR gramineproject/graphene#2406)
✔️ Fix /proc code (will fix gramineproject/graphene#948, gramineproject/graphene#1387; PR gramineproject/graphene#2453)
✔️ Fix lseek overflow on big offsets (PR gramineproject/graphene#2478)
✔️ File locking (fcntl) (will fix gramineproject/graphene#437) (PR gramineproject/graphene#2481)
✔️ Fix poll on pseudo-handles (PR gramineproject/graphene#2498) (found in gramineproject/graphene#2419)
✔️ Fix crashes after mknod on path reuse (PR gramineproject/graphene#2499) (found in gramineproject/graphene#2419)
🚧 Investigate the master list of issues (gramineproject/graphene#1803): probably will lead to more minor bug fixes.
Symlinks (will fix gramineproject/graphene#516): as in-memory files emulated by Graphene

Detailed list

Multi-process synchronization

✔️ Initial design and discussion (gramineproject/graphene#2158)
✔️ Prototype in Python to clarify the API (https://github.com/pwmarcz/fs-demo)
✔️ Proof of concept: FD position sync (PR gramineproject/graphene#2264, gramineproject/graphene#2267)
✔️ File locking (fcntl) (will fix gramineproject/graphene#437) (PR gramineproject/graphene#281, gramineproject/graphene#2522)
✔️ Fix mknod crash (PR gramineproject/graphene#2499)
Synchronize dentry cache

Dentry/low level filesystem cleanup

✔️ Initial design and discussion (issue gramineproject/graphene#2321)
✔️ Rewrite low-level dentry functions (PR gramineproject/graphene#2324)
✔️ Remove ino (PR gramineproject/graphene#2374)
✔️ Make mode and type functional (PR gramineproject/graphene#2379)
✔️ Simplify readdir (PR gramineproject/graphene#2383)
✔️ Change treatment of synthetic files (pipes, sockets) - needed for changing mount semantics (PR gramineproject/graphene#2402)
✔️ Change mount semantics (don't overwrite existing dentries by mount) (PR gramineproject/graphene#2370)
✔️ Rewrite pseudo-filesystems (PR gramineproject/graphene#2453)
✔️ Remove qstr (PR gramineproject/graphene#2585, #267 )
✔️ Handle unlink and rename correctly
✔️ Migrate to inodes: this will take several steps, but should unblock further changes (PR gramineproject/graphene#2646, #5, issue #279)
Make sure dentry stores stat data; remove mode() and stat() operations
Remove lseek() operation (don't pass it through to underlying filesystem)
Clear semantics for remaining filesystem operations (issue #279)

Path lookup

✔️ Rewrite path handling (PR gramineproject/graphene#2342)
✔️ Rewrite path lookup logic (PR gramineproject/graphene#2333, gramineproject/graphene#2354)

[dimakuv]

@pwmarcz Is this meta-issue completely done? Can it be closed?

[pwmarcz]

I think so. I will not be continuing this project, and the remaining items are either very hard (such as synchronization) or not urgent (symlinks).