Proposal for a iteration plan and endgame

[thecodingdude]

Hi guys - as a regular NodeJS user and somebody interested in ESM in Node, I applaud the work that has so far been done in attempting to get ESM into Node. This has discussed as far back as 2012 so it's great to see some traction in getting this to Node stable, although I still have some concerns.

@MylesBorins in the last meeting said "as a group we have a lot of conversation and are being very thorough. But - how are we moving forward? We need to reach consensus." and it's something that I've been thinking about over the last few weeks.

Issue #142 currently has 112 comments, which I am sure is very valid, but make no mistake, since 2012 there has been so many discussions and what I don't see is a concrete plan and proposal in actually getting this implemented and working in Node.

The problem

The main issue for me right now as a spectator, is understanding what has happened so far. What's been decided? What progress has been made? What decisions have been finalised? Where are we in terms of getting this unflagged? Looking through this repo, with all due respect, it's difficult to see where ESM was from Jan 2018 until today.

Using the below proposal it will make it easier to track each individual item, which has a cutoff date in which a decision will be finalised and where the code can then be written to match that.

Proposal

I'd like to propose both an iteration plan and an endgame for this team. These are lifted directly from VSCode and will easily allow the community to see the exact progress being made. This comes with the caveat that someone somewhere has to make a decision on how ESM will proceed in Node. And that will most likely dictate the code that is then written and will one day be unflagged.

The general idea is to put everything required for ESM into the plan, then the endgame dictates when and how each part is formulated. This will then help ensure that this tweet in 2016 does not become a reality.

Also, on a final note, we can discuss this ad nauseam but we can be sure of one thing: there is no perfect solution here, whatever is decided will probably in some ways be suboptimal, right now, getting something out there is better than nothing, and whilst allowing everyone a voice is appreciated, somebody is going to have to be the one to say "this is what we're doing", perhaps PEP 572 might inspire you (controversial, but accepted none the less, perhaps this team could use a similar system)

[bmeck]

@thecodingdude I have been thinking about this for a while, I think a minimal implementation is rather difficult on its own right due to so many pieces being controversial since this basically touches the entire ecosystem. I'd note that a truly minimal implementation without any controversial bits might be much less usable to the point of it not being a suitable target for a variety of use cases. We need to be sure that this minimalism is designed with reserved design space such that features could be added without conflicting with the minimal implementation. In particular what I think this would look like:

• import can import ONLY ESM and only .mjs files, not even JSON/C++/etc.
• in order to load CJS you must use something like createRequireFunction instead of import.meta.require
  • I'd note that we might want to prevent loading core builtins which would make this problematic. I think loading core modules is sane in all future scenarios though, so this should be fine.
• no loader mechanisms are shipped
• no "mode" field in package.json
• no package-name-map integration

This would be the minimal "kernel" we could build upon; it isn't pretty and it isn't feature complete, but it appears to be something that could be argued doesn't cause forwards compatibility conflicts and is the absolute minimal interaction between the systems.

I'm not sure that we could agree on something so minimal however, because of how hard it is to use such a thing. However, I would like to see proposals have a base from which they could all build upon.

[weswigham]

getting something out there is better than nothing

Something that can't see wide use because it ignores the existing ecosystem is as bad as not shipping it or forking the runtime - especially if it's introduction will cause painful questions and scenarios for the foreseeable future. Avoiding that is, IMO, higher priority.

I'd like to propose both an iteration plan and an endgame for this team

I'm not sure of those process elements help here. The first assumes we have a backlog of tasks from which we're choosing where to focus on (and a BDFL to do a final task prioritization and assignment) - we don't. We're still trying to come to consensus on what parts, exactly, we need to ship to be successful. The latter is about managing release activities, which, as of right now, we have none (and unless we design some kind of staged release/upgrade plan, won't have any from what I can guess).

[thecodingdude]

I'm not sure of those process elements help here.

Organisation is never a bad thing - coming into this repo it's impossible to tell what exactly is going on here. I've been following for months and I have absolutely no idea what has been achieved because it appears nobody is keeping track, this 'random' comment is the only thing I've found that actually lists everything that's relevant to this project.

We're still trying to come to consensus on what parts, exactly, we need to ship to be successful.

Right, and without some sort of tracking document, how on earth will you come to that consensus? Again, trying to follow along is really difficult because there are so many moving parts and so many comments (and, no disrespect, but there's noise in there too) and it doesn't appear to be the case that these comments are resulting in concrete decisions. People have been bickering too and fro about this for over 6 years now. I'd rather see @MylesBorins say "we're using MJS, that is final, that is our decision, and that's how it's going to be", otherwise we'll end up still debating this into the next decade and not making progress.

Also, having a plan just lets everyone know "hey - here's everything that's relevant, what we're working on, what we've discussed so far, what decisions we've made, and how it's going to work' which is basically still non-existent.

@bmeck I love the idea of a minimal kernel to build upon, #141 is a good start by @MylesBorins to get some code out into the wild and let people test/hack upon it. Seems to be the ideal place to build up as features are decided.

[weswigham]

We were in the process of producing such a document when some members decided it would be easier to put their opinions down into an implementation example rather than hash out such a document without example. We'd only recently finished enumerating and elaborating all of the usecases that had come forward (which there is an issue and document tracking).

[Janpot]

a minimal solution might not be usable for many projects right away. But it might help in showing which issues early adopters have the most trouble with in attempting to migrate their stuff over. This in turn could help in prioritizing or preferring certain features over others. Maybe it could serve as a metric in the debate?

[weswigham]

a minimal solution might not be usable for many projects right away. But it might help in showing which issues early adopters have the most trouble with in attempting to migrate their stuff over.

That might be true if the volume of work was high, but that's honestly not the issue (implementations are many right now). The slowdown is because of a bunch of apparently mutually exclusive features (mostly around cjs usage) and the process of selecting which among them are most important - that's what takes so long to reach consensus.

[demurgos]

@Janpot
I agree with @weswigham's answer. I just want to add that these implementations can actually be tried if you really want to, Node stable even ships with a feature-gated implementation. They actually helped to gain some insight. But they are experimental and you should not rely on them. The fact that "--experimental-modules" is available does not mean that it will continue to work this way (discussed in one of the last two meetings).
The danger of providing a partial solution is that some libraries may start to use it. A solution preventing the transition may cause a split and correcting errors may cause breaking change or even be impossible.

[SMotaal]

I wonder if something like nodex can help mitigate expectations, if you use nodex your code is expected to at least break if you don't continuously keep up with upstream updates, your performance is expected to fluctuate, your uptime is more like sometimes-up-but-usually-down-time, your dependencies are more like what-the-fork... it's unlike the wow factor it's just an x factor.

Sometime separation might actually be a good strategy. It might be a separate binary, or simply a special npm package which has unique blessings to unlock certain aspects, which leaves the ultimate choice to the user without affecting the mainstream users.

[ljharb]

That’s not how the ecosystem works in practice tho - some packages will require the new binary, and some users will then be forced to use it (this happened already with things like the harmony generators flag, for example).

[Janpot]

Not that I'm necessarily in favour, but with that mindset, isn't it the case that you can't release anything experimental and advertised it as such?

[ljharb]

Yes, that mindset definitely limits what can be released even under a flag - and it’s a crucial check and balance against the mindset that anything and everything should be shipped early and often.

[Janpot]

That mindset not only "limits what can be released even under a flag", it downright excludes anything. And I would argue there's definitely a middle ground between that and "anything and everything should be shipped early and often". Only a Sith deals in absolutes 😄

[jaydenseric]

This issue was raised Jul 17, it is now Aug 20, and there is no concrete response, just another rabbit hole of debate and tangents; a demonstration of the concerns raised in the issue.

I've been enthusiastically experimenting with --experimental-modules and PR'd a lot of packages for .mjs support, and published several packages with dual CJS/ESM support. I'm doing all I can from the outside to make ESM in Node.js a reality.

Please don't shoot the messenger, I don't say this lightly. At this point, a lot of the talented developers I know personally are losing faith Node.js can succeed in the long term. We want to see a pragmatic alternative unburdened by legacy technical debt (CJS) and private centralisation (npm) and bureaucracy (implementation freezes, working groups, etc.). The future might be something like Deno:

• No package.json / npm. Just ESM imports:

   import { test } from 'https://unpkg.com/deno_testing@0.0.5/testing.ts'

• Aims to be browser compatible regarding imports, fetch API, etc. fetch and FormData polyfills in Node.js are poor DX and an isomorphic hassle.
• Can run TypeScript. Not big for me, but a lot of people care about this.

I could prepare all my packages for such an environment pretty quick since I already write their source in ESM. Delete package.json, strip out Babel, and find-and-replace the imports for dependencies. We will have to reimplement popular dependencies, but frankly a great CJS -> ESM refactor needs to happen for all packages at some point anyway. Dev tools like ESLint might be the tricky part, as package.json is used by editor plugins to use locally installed dev deps.

The ability for the industry to pivot to a technically superior solution shouldn't be underestimated. Look how quick Node.js took off in the first place, and that was before JS was respected as a backend language.

[devsnek]

@jaydenseric we're doing a lot of stuff in other issues in the repo, and we have bi-weekly conferences as well. we are working on getting esm done in node, but we're getting it done correctly. deno can do whatever it wants because it explicitly opts out of the node ecosystem. we can't move fast and break millions of ecosystem packages.

your feature requests about typescript and fetch and formdata aren't really relevant to this issue/repo, you can bring them up on the nodejs/node repo though. there are already issues open for some of them.

i'd also advise you not to ship libraries supporting the experimental modules implementation. it actually hurts our ability to develop it if the ecosystem already uses it before we finish it. flagged features are best used locally and in non-release branches.