Module Interop (CommonJS resources from ESModules)

[MylesBorins]

Kicking off this thread to discuss interop, specifically getting CommonJS (cjs) resources from inside of an ESModule(esm)

We currently have a transparent interop. .js is reserved for cjs and .mjs is reserved for esm. When you import, statically or dynamically, the file extension is used to determine the resource type.

There is a pull request to introduce modes which would change the behavior regarding file extensions depending on meta data in the package.json

There are two other proposals for how to get cjs resources that are non transparent, and would be necessary to if .js were to be aliased to esm

• import.meta.require
• import {require} from 'nodejs'

There has also been a suggestion of offering scoped variables in esm, similar to what we currently do with cjs.

What do people think?

[weswigham]

There has also been a suggestion of offering scoped variables in esm, similar to what we currently do with cjs.

As I mentioned in the other issue, I think this is critical. Nobody migrating older code wants to do a big bang migration where you need to replace every reference to require, exports, __dirname, and __filename in their codebase all at once. A userspace commonjs package, today, can provide shims for these using a global require and getters on the global object for the others, by inspecting the stack of an error thrown within them - not good for perf, but good for interop. It's bad enough that you may have to change your file extension and change all of your exports all at once - I think it's appropriate to make as many commonjs features available in a module scope as possible. Module "purity" is, in my opinion, not as importing as lowering adoption barriers as much as possible. And again, as I mentioned in the other thread - a userspace package can offer this today, so we should own the experience in node proper, instead, to make sure that behaviors with respect to loaders and hooks are respected, and that the way it behaves with respect to eval and vm contexts is well-documented.

[TimothyGu]

See a part of my response here: #7 (comment)

When I was referring to vm module, I meant more about vm.runInThisContext.

[weswigham]

Nobody migrating older code wants to do a big bang migration where you need to replace every reference to require, exports, __dirname, and __filename in their codebase all at once.

And to expand on this, most projects that I've met with that adopt new features do so very gradually - if a file is changed for a business reason, then some modernization will happen at that time as a matter of applying best-practices where possible, otherwise it will remain untouched (as there is no reason to change it). Huge migrations to the "latest and greatest" are difficult to justify from this perspective.

[TimothyGu]

There has also been a suggestion of offering scoped variables in esm, similar to what we currently do with cjs.

This is infinitely better than having a require hanging off of global, but as of right now it is not technically feasible, either in V8 or by spec.

[weswigham]

And we (Node.js core) wouldn’t have to deal with the issue of tricky edge cases with eval or new Function that may result from doing that.

If we think those edge cases are so tricky, we shouldn't leave it up to a userland package to solve them! Because what's in core becomes written in stone, it is up to us to solve the hard problems adequately. Punting because it's difficult or unclear just pushes the problem into the ecosystem, and then people have to rely on word of mouth like "oh, you use package XXX, it doesn't handle Y like you'd expect, maybe use package ZZZ instead - but it has the following caveats: ...", and you have it wait years for a "winner" to shake out, and in the meantime everyone is using a bunch of potentially conflicting options (especially in this case, since these are globals - conflicts are super likely).

[TimothyGu]

most projects that I've met with that adopt new features do so very gradually

Indeed, Node.js’ current support for CJS interop is designed for such (unlike npm’s proposal). The key difference between the status quo and your proposal is that we encourage converting an entire file at once, while you seem to want the ability to convert one specific feature in all files.

IMO interop for converting an entire file at once is sufficient, but do let me know if you think differently.

[zackschuster]

+1 on import.meta.require, if only because feature detection would be easier: if ((const { require } = import.meta) != null) { }, for instance. afaict, dedicated modules would need to rely on exception handing to accomplish the same thing.

wildly incorrect

[zackschuster]

@weswigham i'm confused, if the primary audience for your concern doesn't touch working code & the default behavior accomodates them, is there really a problem? i mean, most people don't change working code, period, even if one way is "strictly better", and QoL affordances won't change that.

i think a bigger concern should be the massive amount of confusion coming if all the scoped variables are moved to import.meta without a suitably "magic" equivalent cropping up (like injected imports, tho idk if that's even technically possible).

[weswigham]

@weswigham i'm confused, if the primary audience for your concern doesn't touch working code & the default behavior accomodates them, is there really a problem? i mean, most people don't change working code, period, even if one way is "strictly better", and QoL affordances won't change that.

From people I've interviewed/sat with, a lot of "upgrades" on large codebases are highly incremental (down to the logical unit, usually, not even file). So say I need to go update some business logic in some function to fix a bug - when that change goes through CI/CD, those lines (and those only) get checked for new style issues - including issues related to modern best practices (like adopting more es6 features). Then I need to go through those lines and update them (putting in a little extra effort on top of my bugfix).

As it stands today, "a little extra effort" here means rewriting a large group of identifiers and all imports and exports in an entire file - that's not a small burden. Many larger projects also accidentally end up with larger files with too much functionality in them (grown naturally over time), which would need a lot of effort to disentangle. Migrating a 2000-line+ file with conditional requires or file-specific identifier references is going to be nearly impossible without refactoring the entire project, or at least the entire file as it currently stands.

Minimizing the minimal work required to start adopting es6 modules should be a goal.

[devsnek]

this argument keeps coming up because of people who are willing to leave certain things they don't believe are needed to users in the dust. I did very much the same thing with my pr to make cjs modules get named exports, and it took a few weeks and tc39 to unravel that mess. in the end I think bmeck put it best with his phrase "the worst thing about mjs is that it works". that phrase I think also applies to this situation very well. the worst thing about the current interop is that it works, with all these other behaviours somehow leaving some functionality behind or broken. imo meta.require is a nice utility and would be cool to have but it is nonessential and I wouldn't feel that node is lacking without it.

[MylesBorins]

So one thing that I personally optimize for, perhaps to a fault, is Node + Browser interop. It is dawning on me that we may need to align on a design / architecture philosophy before we can really dig into the meat of some of these implementation details.

I've opened #11 to discuss the matter

[zackschuster]

@weswigham thanks for explaining! 😄 i hadn't heard of conditional linting like that. sounds like performing a minor fix could require a whole-hog rewrite of a huge chunk of the program. totally understandable why your position is that way!

[giltayar]

@weswigham - I'm guessing that people that have legacy codebases would probably leave most of the files in them in CJS land. And if they decide to move a file or two (or ten) to ESM land, they would anyway need to change all the module.exports to their ESM equivalent (otherwise why make the change?). Making them also change __dirname to some form of import.meta.url and all require-s to import.meta.require isn't that big of a deal once they've choosen to do some amount of work.

[giltayar]

@devsnek - the mjs proposal definitely works, and moving incrementally from CJS to ESM is aan exercise in technical refactoring that could easily be helped by a codemod, except for one place that needs real code refactoring: if some code in a CJS module dynamically require-ed another CJS module, then moving that importing CJS module to ESM means transforming the require code that is synchronous to import code that is now async, and that may mean big refactorings in other places in the code.

I believe import.meta.require will alleviate that pain by not requiring that move to async. I'm still ambivalent about this proposal, because the pain alleviation is temporary—once the imported CJS module becomes ESM, the move from sync require code to async import code needs to happen anyway.