Use Case #42 Discussion

[weswigham]

At today's meeting, some people expressed interested in a more complete description of the context and background around use case 42. Here's the usecase, for reference:

Rachel is a TypeScript user who is importing some JavaScript code that uses CommonJS. She uses declaration files that were written on DefinitelyTyped, but were authored as ES module top-level exports as so:

export function foo() {
}
export function bar() {
}

When she imports them from TypeScript, she gets code-completion on the namespace import for foo and bar

import * as thing from "some-package";
thing./* completions here*/

When she compiles her code to run on either the 'commonjs' or 'esnext' module targets, she expects both to run correctly.

So, first, some background. TypeScript has these things called "declaration files". They're additional metadata about a .js file that includes additional type information for a module (written in files with a .d.ts extension); this is how vscode can provide good completions for things like lodash and jquery. They usually look something like this:

// Type definitions for abs 1.3
// Project: https://github.com/IonicaBizau/node-abs
// Definitions by: Aya Morisawa <https://github.com/AyaMorisawa>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
/**
 * Compute the absolute path of an input.
 * @param input The input path.
 */
declare function Abs(input: string): string;
export = Abs;

or this:

// ... some more definitions
export * from "./createBrowserHistory";
export * from "./createHashHistory";
export * from "./createMemoryHistory";
export { createLocation, locationsAreEqual } from "./LocationUtils";
export { parsePath, createPath } from "./PathUtils";
export { createBrowserHistory, createHashHistory, createMemoryHistory };

that is to say, normal es6 syntax with the addition of export= (to describe the commonjs only pattern of replacing the namespace object) and type annotations, without any expressions or blocks. Someone went and put in the effort to write these type definitions at some point in the past, and there's now a community of people authoring these and keeping them up-to-date. The hub of that community is DefinitelyTyped - every definition file published there is automatically published to the @types npm namespace under the same name as the package it corresponds with - this means that, for example, jquery has types available via @types/jquery. It's a kind of crowd-sourced documentation/metadata store.

So, the dilemma. The typescript compiler (and by extension the vscode js language service, as it is really just the typescript compiler behind a thin facade) follows node's module reolution scheme to find js and/or ts files. In addition, it will also look for an adjacent .d.ts file to provide type information for a .js file, and, failing that, an @types/packagename package with a declaration file to provide the types. (Failing either of those in some configurations it will fall back to the actual JS, if it is able to, but this is costly - there's a lot of JS and it needs to be processed a lot to get good type data from it, which is why declarations are preferred.) We have two unique issues to deal with in the esm transition, both of which come into play here in this use-case. The simpler one is emit - providing a node-esm emit target that interoperates decently. The more complicated one is typechecking.

To start with typechecking (for both js files and ts ones): You'll note in my description of declaration files above, I didn't mention anything about any encoding of the library's available module format(s). This is important - we expect that no matter if you're targeting cjs or amd or esnext that the same declaration file will be able to accurately represent the module. This is critical, as it turns out, because some of our consumers will target esnext with us, but then actually transpile to cjs using another bundling tool, like rollup or webpack (retaining the es6 style imports for dead code elimination). We (strongly) operate under the assumption that interop between various module formats is invisible to the end-user - this carries into the js editing experience, where we assume that weather you wrote import * as x from "lodash" or const x = require("lodash") it will produce roughly the same module[1], and have the same members when you look for completions on x. Now, clearly we're capable of changing this assumption (likely behind a flag, but w/e), but this would (will?) fracture our ecosystem of existing type definition files; anything already written would need to only be interpreted as a cjs package, and we'd have to introduce a marker (file extension, pragma, or otherwise) to flag a declaration file as node-esm so that we can reject the old one and only accept the other for resolution depending on the exact interop scheme. It's not exactly pretty, and goes about as far away from a single "universal" declaration file as you can get (and, naturally, starts to require extra maintenance work to maintain the doubled APIs). Compound that with the fact that nobody usually bothers to tell their editor anything about the files they're working with (ie, will this random .js file be targeting node-esm, esm, or cjs? - at least at first), and we might really have to start arbitrarily guessing about what types an import should actually map to on disk, depending on any exact interop scheme, which is no good from a type safety perspective.

Our emit issues are more clear, and mostly center around exactly how interop might work. The typescript compiler, being a transpiler with multiple supported output formats, allows you to write the same es6-style input code and transpile it to either cjs or esm module formats (or amd or umd or systemjs). It will also auto-generate a declaration file for you. Generally, it is expected that your code will function the same way when targeting any of these module runtimes and present the same interface to consumers who can understand the format (and the same declaration file is currently produced for all of them). Some constructs (like export namespace assignment) aren't supported on some emit targets (ie, esnext), but otherwise interop is generally expected (after all, that's a big part of a transpiler's job). Node's interop scheme, if not fully transparent, would probably require us to emit helpers/perform transforms mapping from the more transparent interop we support today to any more explicit form of module system interop supported by the platform, thus requiring a new, independent module target, different from normal un-transformed esnext modules. Failing that, it would require a flag that at least alters our checking and resolution to only allow any stricter platform interop scheme, which would, naturally, not be able to be the default so as to not break long time users.

We also have relatively strong compatibility needs, since our service needs to keep working on code that was written 1, 2, 3 years ago, as nobody wants to launch their editor on their legacy codebase and just be greeted with a bunch of confused warnings and incorrect types, which necessitates a lot of changes be non-default flags. Our stance is, typically, we only intentionally introduce "breaks" if said break is identifying real problems in your code (ie, you were accessing a property which could never actually exist, but we didn't catch before). And then we'll usually have a flag to turn off the new check. Even for the 3.0 release that we have a milestone up for now, we don't have any really major breaks in the backlog - just larger features (it's more of a pragmatic answer to "what comes after 2.9 when incremented by .1" than a semver major release).

[1]We have a flag for placing the module on an es6-style import's default and disallowing calling namespace objects (esModuleInterop), to align our typecheck and emit with babel's emit, however this isn't currently our default for fear of breaking longtime users.

cc @DanielRosenwasser I hope I've explained your concerns, but you should feel free to chime in.

[guybedford]

Here are my suggestions on the above. I hope this makes some sense and please let me know if I'm missing anything at all here. I'm an avid user and huge fan of TypeScript and use it in all my work, and I I would love to see these interops working smoothly.

As you say a flag of sorts is likely necessary to change from the full freedoms in use today to something closer to the NodeJS interop.

Perhaps strictNodeInterop: true or similar, enabling a workflow where:

import { x } from 'cjs'; // fails
import x from 'cjs'; // provides the CommonJS
import { x } from 'es-module'; // works, because the resolver informs the type checker this is an ES module

The nice thing about such a mode is that it simplifies the emission a lot - the emission is guaranteed to work out. As you say, otherwise you need to do some fancy destructuring stuff to continue to make things work in the emission (import __cjs from 'cjs'; const { x } = __cjs;) which I would still expect to happen by default when not using the strictNodeInterop flag.

Now it may well still turn out that we magically get TC39 to allow module namespaces to define named exports post-execution. And if this happens, then the above is no longer a problem for TypeScript. Note though that semantically distinguishing CommonJS and ES modules as having different values for import * and require('x') is probably still important here in that import * for CommonJS must be a copy of the named exports, with a default added for good measure, while the require('x') value is the live module.exports binding.

Emission for legacy module formats pretty much remains the same here entirely in that the Babel interop applies (__esModule checking etc), and that will all continue to work out well since NodeJS commonJS modules likely won't be able to import ES modules to create any second-order double-interop problems.

The rules for the type checker should probably follow similarly:

1. Ideally fix up the ability to support import x from 'cjs' alongside import { x } from 'cjs' by having the resolver know the format of the module loaded. This way users can have proper interop coexist side-by-side with the named exports sugar (which is right now my biggest complaint and pain with TypeScript actually). We split into two cases: CJS and ES modules, where the exports for CJS modules are taken to have the form interface CJSModule { default: CJSModuleDefinitelyTypedDefinition, ...CJSModuleDefinitelyTypedDefinition } where the named exports are extended over the namespace instead of treated as the namespace and default is never overridden. This I think would be preferable to the treatment of synthetic default imports today which seems to switch to one mode or the other instead of having both coexist.
2. Enable a strictNodeInterop mode on the above that then just cuts out the named exports sugar. Or maybe we are in luck with the spec, and don't even need to do that!

I know there's a lot more nuance here to the typechecker, and I'm more than glad to delve into the details here further. Please also let me know if I've missed any concerns here too. And thanks for the great work you guys do.

[guybedford]

Also similarly here, now that nodejs/node#20403 has landed, it is possible in NodeJS to support:

x.mjs:

import fs, { readFile } from 'fs';

So ideally TypeScript should be able to support this as well now - again this would be enabled by the same thing described in (1) above - supporting CommonJS modules as having their iterable properties extended over a module namespace object, with the default as the definition itself.

[weswigham]

@guybedford The contention isn't that it wasn't possible to wire up a mode that works with node-esm (it's always possible with enough flagging and elbow grease), it's that it'd be a break and would require behavioral changes we couldn't make as defaults, and on top of that would require invalidating the entire existing ecosystem in some interop cases, which makes the editing experience very bad. Though in these same cases where we'd need to flag things and invalidate old declarations as esm, the node ecosystem also fractures (as esm won't be substitutable for cjs to retain API compatibility), so it's almost expected (just not good for... almost anyone) 🤷‍♀️

[weswigham]

Also similarly here, now that nodejs/node#20403 has landed

And that actually complicates things even more, since its' cjs but doesn't act like other cjs (since normal cjs can't produce named exports rn), so is effectively a 3rd kind of module that needs to be worked into resolution and declarations somehow.

[guybedford]

@weswigham my suggestion is exactly to enable by default an interop that is both backwards and forwards-compatible supporting default and named exports for CommonJS side-by-side. This provides a non-breaking upgrade path, where the strictNodeInterop flag then only acts as the final "deprecation" step to align with Node interop.

It's really important to enable the support in TypeScript today for users to write forward-compatible code by default (using the default :P), as stopping that will continue hold the entire ecosystem back.

[devsnek]

it seems like the summary of this is that people want an api to resolve and get the named exports of a file by node's loader's rules?

[weswigham]

@weswigham my suggestion is exactly to enable by default an interop that is both backwards and forwards-compatible supporting default and named exports for CommonJS side-by-side. This provides a non-breaking upgrade path, where the strictNodeInterop flag then only acts as the final "deprecation" step to align with Node interop.

That's what our esModuleInterop flag already does for babel's interop; but here's the thing - it's flagged because enabling that behavior has the potential to introduce types that won't exist in a user's compilation (and thus, confusion), eg, if some one were to import d from "foo", where foo is either a cjs or esm module (because some people are advocating for "dual mode" packages), depending on how the remainder of their build pipeline is setup, we have no idea which it'll be fulfilled by (especially if both are represented by the same declaration file, but its still ambiguous even if they're separate); and for one of those choices the import is likely invalid (esm), causing us to fail to issue an error on the bad import when we should have, and allowing a runtime error to occur (or the reverse is true - we forbid the import, but it would have worked at runtime under the config they were using and they're frustrated that they can't do what their docs say is right). Which is bad - when the typechecker is uncertain or fails to meet expectations, developers lose trust in it.

The point I've been trying to hammer home is that while, with explicit intent described for all inputs, yes, you can make any module system interop scheme your heart desires work out in the end; but absent intent (aka configuration), even the current light interop mechanism leads to ambiguity in the ecosystem. It would be far, far less error-prone if tools could consider esm and cjs as roughly interchangeable.

it seems like the summary of this is that people want an api to resolve and get the named exports of a file by node's loader's rules?

I'd find that useful for other reasons (reducing code duplication in the ecosystem by relying on a platform primitive), but isn't really a core point here.

[devsnek]

but isn't really a core point here.

the only other thing i see here is ts devs trying to figure out some ts-specific detail of how to represent an cjs module wrapped with esm? maybe i'm wrong because these comments are so long and spaghetti but i can't seem to find any other point here. for those of us that don't use ts, etc, can someone here explain concisely how this relates to node's module implementation so we can help out?

[guybedford]

eg, if some one were to import d from "foo", where foo is either a cjs or esm module (because some people are advocating for "dual mode" packages), depending on how the remainder of their build pipeline is setup, we have no idea which it'll be fulfilled by

I think for this reason it is important for the resolver to inform both the type checker and the emission what the module format is - by knowing if the dependency module is CommonJS or ESM it can do the right thing here. If it doesn't know anything about the dependency module, then I guess it could fall back to some default behaviour.

For example, already TypeScript can know that a .mjs file shouldn't do that special "default" esModuleInterop mode, but that a .js file in a CommonJS package should have an interop default.

The point I've been trying to hammer home is that while, with explicit intent described for all inputs, yes, you can make any module system interop scheme your heart desires work out in the end; but absent intent (aka configuration), even the current light interop mechanism leads to ambiguity in the ecosystem.

Completely agreed - and this is why I'm a firm believer in having the NodeJS interop as recommended by this group become that exact intent that tools can assume by default, then perhaps still have those configurations options to use other intentions for more loose interpretations.

It would be far, far less error-prone if tools could consider esm and cjs as roughly interchangeable.

The closest to interchangeable we've yet come through a proposal to the implementation that I know of would be for named exports iterated off of CommonJS modules available on the namespace. Even with that, ES modules wouldn't be able to be treated as their "default" export as CommonJS modules can be, an example of a difference that isn't possible to reconcile.

[DanielRosenwasser]

Let's back up. My understanding is that right now we are trying to understand the use-cases, not solve them. It sounds like not everyone completely understands the scenario, so let me try to summarize the use-case so we can get everyone on the same page.

Let's say you have a CommonJS file like so:

// ./some-cjs.js
module.exports.foo = function() { /*...*/ };
module.exports.bar = function() { /*...*/ };

Babel and TypeScript users are currently able to import that file as follows:

import { foo, bar } from "./some-cjs";

foo();
bar();

TypeScript, being a JavaScript type-checker, also tries to ensure that users are correctly importing foo and bar.

However, rather than scanning the contents of the .js file, TypeScript will always prefer to look at what's called a .d.ts file, which describes the shape of a specific .js file. This is significantly cheaper, as no code needs to be checked, no inference needs to take place, and type metadata is immediately present.

Because of the ability to import using named imports, users will likely have authored the .d.ts file as follows:

// ./some-cjs.d.ts
export declare function foo(): void;
export declare function bar(): void;

Notice that the .d.ts file uses top-level ES exports. This implies that the JavaScript implementation could have been written as follows:

// ./hypothetical-module-shape.js
export function foo() {}
export function bar() {}

If the user in our original use-case (Rachel) switches her emit to target ES modules directly and rely on Node's ES module interop, her expectation is for her code to continue to function since the type-checker has given her some assurance.

[devsnek]

@DanielRosenwasser thank you for keeping it concise 👍 however i'm still unsure of where the issue for node comes into play. is it the .js extension? i assume ts can output transpiled files with any extension right?

and more to my issue with this issue, why is a specific superset of javascript being considered for use cases? i would assume we want to make things best for people actually using node esm, as they actually have to deal with it. people who transpile can put any sort of magic between them and the system without worrying about it.

in my mind the above use case can be simplified to not include typescript or babel at all, but rather be the following:

Rachel is used to seeing the keys of the commonjs module.exports be used as named exports throughout the ecosystem. She wants to continue doing this with node's ESM loader.

[DanielRosenwasser]

in my mind the above use case can be simplified to not include typescript or babel at all, but rather be the following:

Rachel is used to seeing the keys of the commonjs module.exports be used as named exports throughout the ecosystem. She wants to continue doing this with node's ESM loader.

Yes, I agree, that's probably the most concise way to approach it. I think I was getting not just at the expectation, but why our user has that expectation. Existing tooling, infrastructure, and community has guided this user to write their code in a specific way with a strong degree of confidence. Maybe that wasn't necessary.

and more to my issue with this issue, why is a specific superset of javascript being considered for use cases? i would assume we want to make things best for people actually using node esm, as they actually have to deal with it. people who transpile can put any sort of magic between them and the system without worrying about it.

That's actually the thing - there's a limitation to the sort of magic you actually can achieve in this scenario. Throwing more tooling from the TypeScript/Babel/Flow side which requires users to reconfigure their builds isn't desirable either. But I guess we can circle back on this at a later date.

[devsnek]

Throwing more tooling from the TypeScript side which requires users to reconfigure their builds isn't desirable either.

well they brought that on themselves didn't they? i'm sorry to be harsh and unpleasant but i think its rude to users of node.js to compromise their experience because typescript/babel users don't want another line of config. i firmly believe that node.js exists for everyone to use, but those who choose to place a layer of abstraction between it and them shouldn't be given priority.