ports/webassembly: import of a preloaded Emscripten FS fails on Node

[hoihu]

Checks

• I agree to follow the MicroPython Code of Conduct to ensure a safe and respectful space for everyone.

• I've searched for existing issues matching this bug, and didn't find any.

Port, board and/or hardware

webassembly

MicroPython version

MicroPython v1.23.0-preview.324.g6634fea23.dirty on 2024-04-22; JS with Emscripten

(I've added more modules to the manifest)

Reproduction

• build webassembly port pyscript -> OK
• open repl via node build-pyscript/micropython.mjs -> OK
• micropython.mjs can be used on HTML pages as described in the readme file -> OK

so far so good - however... I want to "freeze" existing micropython files in an emscripten FS. For this I'm using the --preload-file argument in the makefile like this:

$(Q)emcc $(LDFLAGS) -o $@ $(OBJ) $(JSFLAGS) --preload-file <directory_with_python_files>

this works as expected in the browser and I can successfully import it via mp.pyimport(xxxx)

But I can't run the node repl with that micropython.mjs anymore. When trying to start the REPL I'm getting the following error:

Expected behaviour

When building with --preload-file I'd expect that the REPL still starts and I can import the preloaded files

Observed behaviour

The above error appears (no REPL)

Additional Information

No, I've provided everything above.

[dpgeorge]

Thanks for the report, I can reproduce the issues.

I think this is a bug in Emscripten: it's not possible to use -s MODULARIZE and --preload-file. That's because modularize outputs a .mjs file (micropython.mjs in our case) and when node loads a .mjs file the require() function is not available (instead one should use import).

It looks like this may have been fixed very recently in Emscripten by: emscripten-core/emscripten#21721 (see previous emscripten-core/emscripten#18317 which shows the same error as reported here).

[hoihu]

Thanks for the feedback!

I updated emscripten to 3.1.59 and using the latest micropython master. The above error disappears, but now I get the following:

(venv) fischmar@MACNP7L4XQC6C webassembly % node build-pyscript/micropython.mjs
package error: [Error: ENOENT: no such file or directory, open 'micropython.data'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'micropython.data'
}
still waiting on run dependencies:
dependency: fp /cloud/test.py
dependency: datafile_build-pyscript/micropython.data
(end of list)

It looks like it misses the micropython.data file, but it's present in the build folder...

(I preloaded a folder cloud - that's why it shows up in the missed dependencies)

[hoihu]

I'm sorry I have misread how to use the --preload-file argument (see https://emscripten.org/docs/tools_reference/emcc.html#emcc-preload-file).

This argument can only be used if the resulting mjs file is used within a browser (and that works).
If using embedded files within node the --embed-file argument must be used.

Using the call like that (in the makefile):
$(Q)emcc $(LDFLAGS) -o $@ $(OBJ) $(JSFLAGS) --embed-file <directory_with_python_files>

and using the repl afterwards:

(venv) fischmar@MACNP7L4XQC6C webassembly % node build-pyscript/micropython.mjs

MicroPython v1.23.0-preview.360.gc3301da17.dirty on 2024-05-11; JS with Emscripten
Type "help()" for more information.
>>> import cloud

correctly imports me the given files. So that's great!

Since that's a functionality that may be quite often used, perhaps could be added as option to the makefile call (like e.g. make VARIANT=pyscript EMBED_FILES=<path_to_python_files> ?)

[dpgeorge]

Since that's a functionality that may be quite often used, perhaps could be added as option to the makefile call

Yes, that sounds reasonable.

Although a better solution for the end user would be a way to do it that doesn't require compiling the webassembly port. Eg use something like this:

const { loadMicroPython } = await import(`${base}/micropython.mjs`);
const mp = await loadMicroPython({ url: `${base}/micropython.wasm` });
mp.FS.mkdir("/lib");
mp.FS.writeFile("/lib/cloud.py", await (await fetch("cloud.py")).text());
mp.FS.writeFile("/lib/utils.py", await (await fetch("utils.py")).text());

That would download each py file independently, so not be efficient with HTTP(S) traffic. But you could bundle all your files into a big JSON and fetch that, then iterate it and create the filesystem.

(Actually, pyscript provides a nice interface to do this kind of thing, if you want to use that.)