Web Modules for Node

[zenparsing]

Over the past week or so I've been experimenting with a non-transparent interop solution I'm calling Web Modules for Node. The basic idea is that instead of trying to create a single overarching module design that encompasses the strenths of both CommonJS and ES modules, we instead reframe the effort in terms of adding support for "browser" modules, similar to how node has been adding support for other traditionally browser-based features. Adding support for "web modules" may prove to be a more tractable problem.

Highlights of the MVP:

• All web modules are identified by URL.
• No file extension searching.
• No directory "index" support.
• Package.json export maps for mapping package imports to files.
• import.meta.require (or equivalent option) provides interop capabilities.
• --module/-m for loading the main file as a web module.

At the very least, it might be good to consider as a fallback in the event that various forms of transparent interop do not work out.

Looking forward to your thoughts and feedback.

Repository and prototype

[GeoffreyBooth]

Thanks for working on this! Do you mind summarizing some differences between the new ecmascript-modules implementation and yours? They both have some of those same bullet points.

[ljharb]

Using URLs seems like a big security and usability issue; such concerns have been raised before with deno’s focus in them. How would those work in an offline environment, since node can’t rely on anything but the filesystem?

[bmeck]

@ljharb the ESM loader in both upstream and fork is currently using URLs, you just whitelist the protocols allowed. Any unlocking of potentially hazardous URL types needs to be done with care.

[bmeck]

@zenparsing this is a pretty all encompassing proposal that doesn't discuss the reasoning for the bullet points or the relation to previous discussions and concerns for them. Could we get a more thorough explanation for some things like:

• All web modules are identified by URL. What do you mean by URL here. URL string format for cache key / resolution, any arbitrary URL?... etc.
• What happens to files without extensions / hashbangs since they cannot use flags
• "package.json export maps for mapping package imports to files." , all of this
• What happens for import() from other contexts (indirect eval Script / Function / CJS / etc.)?
• Why use import.meta.require vs the alternatives? +discussing previous concerns about it.
• How are formats determined/through what system can that be modified and/or virtualized? (asking because this seems to be a waterfall style attempt at a complete proposal of all features)

I think this issue would ideally be split up and made as separate issues against our minimal kernel because waterfall style aggregation of ideas doesn't seem to give us momentum or iterative feedback.

[zenparsing]

@bmeck Thanks for your (really good) questions. I'll need a day or so to follow up on each bullet point.

[Fishrock123]

similar to how node has been adding support for other traditionally browser-based features.

Yes, but, ideally when they make sense for node.
(This has not always been followed much to my and some others' dismay.)

Node.js isn't a web browser and, in my opinion, adding something that is purely due to limitations in a not-node environment should not need to be part of node.

That's my 2c, anyways.

[ljharb]

I actually don't mind adding a "web modules" feature; but imo that should be separate from, and shipped after, a node-first ESM system.

[weswigham]

A "web modules" compat filter is honestly just a lint rule applied on top of current node conveniences, IMO. Any number of tools or flags can do that lint, but there's no need for it to affect the default UX of node, IMO.

[hybrist]

A "web modules" compat filter is honestly just a lint rule applied on top of current node conveniences, IMO. Any number of tools or flags can do that lint, but there's no need for it to affect the default UX of node, IMO.

That is true as long as you never run 3rd party code. If you do, that changes. No amount of import maps will allow you to use that utility module that contains the line import findMatches from './find-matches';. Because that relative import will just plain fail if you're not running a special server or do bundling.

Tbh - if the end result is that node encourages publishing packages that won't work as-is in the browser even when they don't use node APIs, why not keep CommonJS? There doesn't seem to be a fundamental difference between "ESM that only works in node" and "CommonJS that only works in node". We can talk about "it's your choice!" all we want but if npm is full of modules with relative import statements that won't work in the browser, my choice doesn't matter.

[GeoffreyBooth]

There doesn't seem to be a fundamental difference between "ESM that only works in node" and "CommonJS that only works in node". We can talk about "it's your choice!" all we want but if npm is full of modules with relative import statements that won't work in the browser, my choice doesn't matter.

☝️ This. I thought the point of supporting ES modules in Node, of becoming spec compliant with regard to modules, was to make Node more similar and more interoperable with other environments. That's why "browser equivalence" is our no. 2 top priority just after spec compliance. Yes in some ways that makes Node worse, but people want to remove barriers between environments. There's some convenience lost if we ship without extension searching, but there's also convenience gained by most ESM npm packages being usable in browsers without transpilation. It's a trade-off. I understand many of the people in this group only code for server or Node environments, but the browser environment is very important to a lot of JavaScript developers, and what Node does affects that too.

[ljharb]

@jkrems due to the specification for ES Module specifiers, using import is a platform-specific API, because "how to resolve specifiers" is specified to be engine-determined. Separately, import maps can be produced - by npm or by node - that will allow your ESM node code to work in the browser without a build process.

@GeoffreyBooth one of the points is to implement the specification, of which some parts are implementation-defined. Browser equivalence is one of our priorities, but I think it's deceptive to call it "number 2" when we didn't agree on them in any particular order. Also, if removing a barrier makes node worse, then the barrier is a good thing.

[devsnek]

@GeoffreyBooth you can write all your code in your application to be browser compliant, but code in node_modules always and forever has to perform node specific resolution because of package.json and cjs and {list goes on}. So if you try to bundle node_modules without following all the rules that node follows, your server will not work. end of story. allowing ./file_without_ext doesn't make node less browser compatible because it's already impossible for it to be browser compatible. the only place you can affirm browser compatibility is your own application's code because you own the resolution of your own code.

[zenparsing]

Something else to ponder here: we've been thinking about this in terms of replacing CJS, but actually CJS is pretty great in different ways. I wonder: do we really need to replace all use cases for CJS?

To take an example: shebang scripts. I personally don't have a ton of experience with shebang scripts. I've mainly used them as very small entry points into a larger program (for instance, requiring in a module that implements a CLI program and lauching it). Do I need to write that shebang script with "web modules"? Can I just have a script that uses dynamic import?

#!/usr/bin/env node
import('./path-to-my-cli-program.js');

Just something to think about.

[GeoffreyBooth]

Do I need to write that shebang script with “web modules”?

This came up when we were working out the edge cases for the import file specifier proposal. Specifically the case was I think running make lint on the Node repo itself, which runs tools/lint-js.js if I remember correctly. There’s no package.json in either the root of the repo or in the tools folder, so tools/lint-js.js runs outside of any package scope. An early implementation for IRP had such files running as ESM, which broke the Node build, so we changed these “loose” .js files to be defined as running as CommonJS (which I think was what the spec had said to do, but actually hitting the bug made us very aware of it). The part of the spec concerning symlinks, and making sure the module format to run the symlink path is based on the symlink target, came out of this case (and the real world example of /usr/local/bin/npm, which links to a npm-cli.js file). Your example of using import() could just as easily be achieved via a symlink, and the symlink has the advantage that it works in older versions of Node.

But yes, agreed, ESM doesn’t need to support every use case that CommonJS supports; though since we added createRequireFromPath, many of CommonJS’ use cases (like importing JSON or .node files) can be handled via that. I’m sure there are still a few edge cases that can’t be achieved in ESM, and depending on how significant they are it might be fine to have some things where we just tell people “yeah, you need to keep using CommonJS for that.”

[hybrist]

Separately, import maps can be produced - by npm or by node - that will allow your ESM node code to work in the browser without a build process.

Let's please not keep repeating this, we have established further up that this isn't true. Unless I'm missing something important in the current proposal, it will not affect relative imports. So ./foo will not work unless the exact match exists, independent of any kind of import map. Any code in your dependency tree using that format will break in the browser, just like CommonJS files would. It is, in effect, just as (non-)portable as CommonJS.