Allow hooking into the filesystem and providing a persistent backend (like absurd-sql)

[jlongster]

This is (finally) a follow-up to my absurd-sql project I talked about here: #447 (comment)

These are the changes so far required to make it work. It's not a lot of changes, but there are several small things:

• We need to stabilize the FS exports to projects can hook into the filesystem. Long-term, I'd like to figure out a better way to do this. Emscripten has plans to change their FS implementation a lot, so it will likely change. For now, this lets us hook in the FS that's already there. We do that by specifying some exports so closure doesn't minify them.
• We add a filename to the Database constructor which is a boolean specifying that you've passed in a filename, not a buffer. Open to a better API here.
• Don't delete the DB on close (if it's not a buffer), haha. If Obviously now that it persists we don't want that.
• Perhaps the ugliest part, we add a little bit of C code to customize the default unix vfs. One reason to do this is to customize blockDeviceCharacteristics which might improve performance characteristics. The bigger reason is to hook into the lock/unlock methods of the vfs and override them. Unfortunately, this requires some ugly code in api.js to map the internal sqlite file to a filename and then call into our FS to handle them.

Normally, we could handle the fcntl system call at the FS layer. Unfortunately, right now emscripten doesn't provide a way to do that. If it did, we wouldn't need the last point. We might still want to override blockDeviceCharacteristics, but that could be an optional stop to improve performance. Right now it's critical so we get locking.

I've filed an issue with emscripten: emscripten-core/emscripten#15070. This PR could be simplified once we have that. Not sure if you want to wait on that to land this PR or now.

Anyway, I'm open to how you want to proceed. While absurd-sql is robust and ready for production (I'm using it in prod), the API and implementation here is somewhat proof of concept. Let me know what you think.

[Taytay]

Nice @jlongster

What do you think @lovasoa? I'm excited about this PR on principle.

[lovasoa]

Hello everyone!
I've been wanting to review this PR for a long time, but never found the time to get to it.
I realize that sql.js is stagnating mainly because of me, and I would like to add a new maintainer. @jlongster, you have been doing really awesome work around this project. Do you think you would have the time and energy to be a maintainer?

[arsnyder16]

@lovasoa Any luck getting a new maintainer or a chance to review this? Would be a great to get these changes in so absurd sql could be used without using a forked version of sql.js

[quolpr]

@lovasoa could you please review it? This change will make a big move in using of sql.js for the web 🙏

[tantaman]

Hey @lovasoa - is there any way I can help here?

I can do a cursory review but I'm not deeply familiar with the internals of sql.js yet -- I hope to be there eventually.

I'm interested in the future of sql.js as the developer and maintainer of https://aphrodite.sh/ (multi-platform / local-first ORM)

[jimmywarting]

considering that we now have https://github.com/whatwg/fs then i think indexeddb is rather 👎

[mfulton26]

@jimmywarting any idea on when StorageManager.getDirectory() will become available on Chrome for Android?

This sounds extremely promising but I need mobile support. I couldn't find any mention of this in Chrome Platform Status or Issues - chromium.

StorageManager API: getDirectory | Can I use... Support tables for HTML5, CSS3, etc

[mfulton26]

1352738 - StorageManager.getDirectory on Chrome for Android - chromium

[jimmywarting]

i don't know.
technically there isn't any mobile specific UI issues about picking files etc, whatwg fs is purely about the api and adding a sandboxed fs
The "File system access" on the other hand is for a mean to request access to a user selected file or folder and requires UI for prompting access. The android issue are here: https://crbug.com/1011535

safari is actively working on whatwg fs behind a expe flag, but it sucks pretty much right now. You can only create folders and empty files. you can't read or write data to/from files atm.

i haven't heard anything from FF yet.

Anyway. i just learned that navigator.storage.getDirectory() is just an alternative new api for webkitRequestFileSystem(TEMPORARY, 0, success, fail)
modifying files in webkitRequestFileSystem will make an effect on navigator.storage.getDirectory and also the other way around

the benefit of using navigator.storage.getDirectory in web worker are that you can have both a async and sync variant

[jimmywarting]

1352738 - StorageManager.getDirectory on Chrome for Android - chromium

you probably want to close this as dupl of https://crbug.com/1011535

[jimmywarting]

also have a look at https://github.com/WICG/file-system-access/issues

[mfulton26]

I considered closing it as a duplicate but then I realized that as the Origin Private File System requires no user interactions nor permission granting that it might be best to track it separately. I had not found that other crbug though. Thank you for all the links!

[rhashimoto]

I've built SQLite filesystems with both Origin Private File System access handles and IndexedDB. In my benchmarks OPFS was somewhat slower. This may in part be because I used non-synchronous access handles for reading database files (and synchronous handles for writing) to allow concurrent read access from multiple database connections, but is also because atomic commit and optional durability settings provide some implementation advantages for IndexedDB.

That said, unless performance or browser support is the overriding priority, I would much rather write code for OPFS than IndexedDB.

[mrmurphy]

The file system standard looks like, but as far as I can tell from reading around the web, Firefox has no plans to implement support. For the moment, anyway, IDB seems to be the most compatible backend for SQL.js. Is this PR held up on review because of the file system access standard?

[rhashimoto]

Firefox has objections to the full File System Access API proposal, but has been more encouraging on the parts used for Origin Private File System, including the AccessHandle API (which is what Safari has implemented). Their implementation activity has stepped up in the last few weeks so it looks like they are actively working on at least trial support.

[mrmurphy]

Well that's very encouraging!

[biels]

Any news on this? https://caniuse.com/?search=opfs Apparently OPFS is avaliable on most browsers now and one intended use case is actually as storage for sqlite-wasm databases in web workers

[rhashimoto]

Both IndexedDB and OPFS access handles can be used for persistent SQLite storage in the browser. At the moment it appears that neither is clearly better than the other for this purpose; it depends on your application requirements.

• IndexedDB
  • Lower write transaction overhead.
  • Allows concurrent reads.
  • Allows trading durability for performance.
• OPFS access handles
  • Faster raw reads and writes.
  • Allows synchronous VFS implementation (i.e. without Asyncify or SharedArrayBuffer).

[jimmywarting]

or SharedArrayBuffer

Oh, that reminds me. I just recently wanted to port something NodeJS-ish that used sync node:crypto stuff to handle things to make it more cross env. friendly for Deno and web. so i wanted to switch it out to web crypto instead now that NodeJS has it too.

But it was to annoying to switch the entire codebase to be async-ified and breaking all the existing tests so i built await-sync to make the process a bit easier to use.

i think you @rhashimoto would maybe find this interesting and perhaps useful to transform async code to sync more easily.
...It uses SharedArrayBuffer behind the scenes. so it only works in backends and also web worker.

(a bit off topic, but i wanted to share it...)

[krisgrm]

Following this one