Support for esm.ts using node --loader ts-node/esm

[monyarm]

I use ts for my gulpfile, but some of the plugins i use (gulp-imagemin specifically), are esm.
So having a way to use both would be useful, as otherwise when one tries to run gulp, they get something like this:

[18:15:57] Requiring external module ts-node/register
TypeError: Unknown file extension ".ts" for /home/monyarm/Documents/gulp-gameoptimizer/gulpfile.ts
    at new NodeError (node:internal/errors:371:5)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:71:15)
    at Loader.getFormat (node:internal/modules/esm/loader:105:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:243:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Loader.import (node:internal/modules/esm/loader:177:17)
    at importModuleDynamicallyWrapper (node:internal/vm/module:437:15) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

[phated]

That sucks.

This is a complex feature to add and is on our roadmap for rechoir, but we probably won't have time to work on it for awhile.

I'll move this over to the rechoir library, as it needs to be implemented in the loader.

[phated]

I just spent hours digging into this, and my general sense is that esm loaders in nodejs are an absolute shitshow right now. They've recently changed their loader API, so a lot of stuff has to shim their internal loader logic, but then I noticed that ts-node had to wholesale copy-paste node internals into their codebase to support their module loader. Check this out: https://github.com/TypeStrong/ts-node/blob/main/dist-raw/README.md

Anyway, I dove down that rabbit hole because the most efficient way for rechoir to support esm loaders would be to bootstrap our own loader on startup that proxies through to everyone else's loaders. However, once I discovered that we'd have to copy all the internals from node to do this "right", I decided to hack on a much worse performing solution and came up with something that works:

const { spawn } = require('child_process');

const argv = process.argv

async function main() {
  try {
    require('ts-node/register')
    return require('./bar.ts');
  } catch (err) {
    if (err.code === 'ERR_REQUIRE_ESM') {
      try {
        return await import('./bar.ts');
        console.log(mod);
      } catch (err) {
        if (err.code === 'ERR_UNKNOWN_FILE_EXTENSION') {
          var child = spawn(argv[0], [
            '--loader', 'ts-node/esm',
            ...argv.slice(1)
          ], { stdio: 'inherit' });
          child.on('exit', function (code, signal) {
            process.on('exit', function () {
              /* istanbul ignore if */
              if (signal) {
                process.kill(process.pid, signal);
              } else {
                process.exit(code);
              }
            });
          });
        } else {
          throw err
        }
      }
    } else {
      throw err
    }
  }
}

main()
  .then(mod => console.log(mod))
  .catch(err => {
    process.nextTick(() => { throw err })
  })

Essentially what is happening here is that we are assuming require.extensions is the default loading situation (since it is the most battle tested, and you can't do this in reverse from ESM) and load the ts-node/register module for commonjs modules. Then we try to load the file, but if it errors with the ERR_REQUIRE_ESM code, we know that it tried to resolve as an ESM module. In that scenario, we try to load the file with a dynamic import() and if that fails with the code ERR_UNKNOWN_FILE_EXTENSION it means that there was no loader registered for it and we need to reboot the entire node instance with the loader and start the whole process over.

The reason this is pretty insane is that it'll be double registering loaders (for commonjs and esm) for every esm loader you try to use. Additionally, multiple ESM loaders will cause multiple child processes to be spawned (which then reloads all modules from the start).

[phated]

I'm not a huge fan of this, but it solves the problem and makes the actual loaders other people's problem. I'll think on it a bit more.

[phated]

For reference, I used concepts from gulp-cli, liftoff and https://github.com/lukeed/loadr for the above solution.

[JusticHentai]

I have the same problem, did you solve it?

[phated]

@JusticHentai No, and this will be punted until post-v5 of gulp because nodejs support of loaders is so shitty.

[cspotcode]

Just commenting to say that I'm following along with issues like this across the ecosystem, and am always open to discussing ways that tooling and loader authors can shim our way around node's shortcomings.

I have this idea that we might devise a pass-through loader which does nothing on its own, but which allows tools such as yours to install hooks at runtime as necessary.

Roughly, it would look like this:

Node is invoked like this:

node --loader @cspotcode/multiloader ./whatever-tool-or-entrypoint.js

If a tool realizes at runtime that it wants to install ts-node's hooks:

await process[Symbol.for('multiloader-api')].add('ts-node/esm');
await import('./some-ts-file.ts')

Theoretically someone could add NODE_OPTIONS='--loader /abs/path/to/@cspotcode/multiloader' to their shell profile, and then node would have this runtime loader-installation API. I say theoretically, because I would not recommend that users do this, but it's a nice thought experiment, and I think it makes the case that node should support this natively.

[phated]

@cspotcode I believe something like a "multiloader" would make the most sense sense, but your current example only works for dynamic imports. We'd want to be able to prepare the multiloader at launch with flags. Maybe that would be something like:

node --loader multiloader --require ts-node/esm` 

[cspotcode]

That is actually possible today, with, for example: node --loader @cspotcode/multiloader/compose?yaml-loader,ts-node/esm

The runtime API I described in my previous comment is not yet implemented, but composing multiple loaders at startup is implemented today.

https://github.com/cspotcode/multiloader

The end-goal is a future where tools such as yours do not need to attempt to spawn a child process at all. When they decide that a loader is necessary, they can install it at runtime, the same way we do for CJS loaders.

EDIT: fixed a typo in the invocation syntax above

[phated]

For gulp, I believe we'll always need to respawn because node doesn't have good APIs for allowing us to pass other nodejs flags through, but I understand that having the programmatic API would be useful for other applications.

[cspotcode]

Hmm, I'm looking at your code above (#43 (comment)) and assuming that you'd replace the child_process.spawn() with await process.loaderApi.add(); is that not the case?

As far as I know, when you do require('ts-node/register'), you expect things to "just work" after that point, which is how it was with CJS. With the runtime loader API, they would still "just work." Registering ts-node's ESM loader also installs the CJS hooks, so you'd have one-stop shopping for injecting TS support into the node runtime.

Does gulp do things differently?

[phated]

@cspotcode sorry if I was unclear, I'm wasn't talking about the loaders for respawn. We still need to respawn for the usage of gulp --some-v8-flag

[cspotcode]

Ah I see. I think it still might have benefits for this situation:

Additionally, multiple ESM loaders will cause multiple child processes to be spawned (which then reloads all modules from the start).

I think a multiloader solution can limit the number of spawns to, at most, 1.

The first time you realize that you need to spawn, you can eagerly register multiloader into the child process. From then on, you know you can install as many CJS and ESM loaders into the process as necessary, without any further respawning. Since the multiloader API is exposed on the process object, you can feature-detect for its presence, avoiding unnecessary respawns when it's available.

When it's available, you can even skip require('ts-node/register') and go straight for adding the ts-node/esm loader, since that will also install the CJS hooks.

[phated]

@cspotcode I've been thinking about this a bunch and I think your original description is ideal. I'd want to specify --loader multiloader when we boot up the CLI tool (this will cause a respawn due to the new node flag) and then we want a programatic API to register loaders, like you showed:

// Would that have to be async?
process[Symbol.for('multiloader-api')].register('ts-node/esm');

[cspotcode]

Nice, yeah if we want to collaborate on such a runtime API, I think that'd be great. I was toying with adding this to my multiloader thing but I don't remember where I got to. I'll see if I can dig up the code -- maybe it's my work laptop? -- and push it tomorrow.

https://github.com/cspotcode/multiloader/