Add new moduleResolution option: yarn-pnp

[darthtrevino]

Search Terms

yarn pnp plug'n'play plug

Suggestion

Yarn has released a new feature for using modules without having a node_modules directory present: plug'n'play. Some community tooling is available for customizing the TypeScript compilerHost so that it can use plug'n'play modules, but this does not work for users of tsc.

My suggestion is to add a new "moduleResolution": "yarnpnp" option.

Use Cases / Examples

I want to be able to use yarn plug-n-play in my typescript projects. My projects are typically yarn monorepos where common libraries are basic node packages using tsc for compilation. This would allow the basic projects to continue using tsc, but with an alternative module resolution scheme.

Some community tooling has been created here for augmenting the CompileHost with a slightly updated moduleResolution strategy, but using it in basic projects would probably require a forked tsc.

https://github.com/arcanis/ts-pnp

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript / JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. new expression-level syntax)

[weswigham]

I think we'd prefer if yarn and npm settled on a common format - since npm is working on tink which does the same thing with a .package-map.json. Regardless, we have a conventional stance right now to not execute external JS as part of a build (maybe that'll change in the future, but right now that's our standing philosophy), so something like this would, minimally, be blocked on yarnpkg/yarn#6388 for us.

Daniel has a much more detailed comment on the pnp rfc here

[darthtrevino]

That sounds perfectly reasonable, thanks @weswigham

[arcanis]

yarnpkg/yarn#6388 will likely not be acted upon (edit: it did). People wishing to generate the dependency tree in a specific format can easily do so using a postinstall step, as shown in the sample app. I'd advise against relying on that for tooling integration, though, since it puts an unnecessary burden on the user.

The most generic way for you to do this is to simply expose resolveModuleName within the tsconfig files (#18896), just like we did for ts-loader (TypeStrong/ts-loader#862). It would be a useful feature that would not only unlock PnP for tsc users, but also other use cases for other tools authors. I believe it also wouldn't conflict with your default policy of not executing external code, since it would require an explicit acknowledgment from the user.

[weswigham]

As I said - we've currently got no plans for a tsconfig.js (and I bring it up often) or a tsconfig managed plugin system. We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor). I definitely know how I'd like to expose the ability to do something like this, were we to accept it, since I designed an extensive plugin model for the compiler in the past (before it was declined).

Not to say we won't revisit it at some point, but similar suggestions have definitely been declined repeatedly in the past. A static format is much easier to deal with.

[arcanis]

I understand, I was simply informing you that waiting on yarnpkg/yarn#6388 would likely not be an option 🙂

Another thing I forgot to mention - in a world where tsc would have built-in support, the resolution name should be pnp, not yarn-pnp. Plug'n'Play is a generic specification designed in such a way that it can be implemented by different vendors, not only by Yarn.

[Phoenixmatrix]

I'd just love if resolveModuleNames was easy to override in tools such as tsc or VS Code, since yarn pnp isn't the only use case for this, and module bundlers being infinitely customizable means the only thing holding usage back is really the editor support (and if you want compatibility with the CLI compiler). The feature is basically "right there" but is hard to use without straight up forking.

There's always going to be a demand for custom ways of resolving modules. The node ecosystem is someone rigid in this regard (because of the node_modules convention being "untouchable"), but the community has done work around it (eg: Webpack custom resolvers and aliases). Now there's also pnp and tink. At HubSpot we have our own bpm. Pretty much all tools make this reasonably easy to work with (Jest, Webpack, stuff like eslint-plugin-imports, and so on. I think Flow now has that feature too?). The TS language service has resolveModuleNames, as mentioned above. If I understand well, there's even a custom resolution hook for ES6 modules in node now?

So the last bastions to make things friction-free are the language service consumers such as tsc and VSCode. If we have that, we're golden.

Another thing that comes to mind: right now because the rest of the toolchain allows customizations but the typescript ecosystem does not, there's additional friction in keeping stuff in sync: whatever I do in my webpack config, I have to replicate in my tsconfig. With better integration of custom resolvers, we could remove this friction by making them behave the same. I could have a custom resolution algorithm and have webpack and VSCode and Jest honor it without changing all the configs one by one. That would be huge. Awkwardly, this is trivially easy to do by forking the pieces involved, but that's not very scalable.

[sebinsua]

We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor).

I understand what you're saying but it would be nice if the TypeScript team could take a step back and look at the bigger picture here. It makes sense that you care more about the performance of TypeScript, but many of your users test their applications using CI systems and it matters to them how slow the installation of their application is just as much as the speed of the type-checking/compile process.

For example, installation of the application I'm currently consulting on takes ~2 minutes on CI. I would like to get that number down.

Is there anyway that TypeScript can support pnp without you supporting custom resolvers? If as @arcanis says, it's a "generic specification", I don't see why it shouldn't just be supported out-of-the-box.

I hope we can get this feature some way or other, otherwise it just seems like we are going to be permanently doomed to slower installation times.

[weswigham]

We can't support it directly as-is, but as @arcanis points out elsewhere, you can probably write a short postinstall script that converts the dynamic .pnp.js file into, say, the paths tsconfig option. I haven't attempted it myself yet, but I can't think of why that shouldn't work.

[Phoenixmatrix]

I've gone the dynamically generating tsconfig route for some of our stuff (we actually made a VS Code extension for this for our home grown resolution mechanism).

There's some awkward limitations there, such as nested dependencies when you have multiple version of the same package. (eg: package Foo depends on Bar@1 and Baz depends on Bar@2, and your app depends on Foo and Baz, you're out of luck as far as I know. I dont think tsconfig aliases can express this).

[arcanis]

@weswigham I'm wondering something - you might be aware that the Node project is working towards standardizing loaders. Doesn't it make your approach impracticable on the medium-long term?

Something to mention regarding the perfs - we actually noticed slightly better perfs in Flow when it uses PnP, which makes sense given that the PnP resolution removes a lot of stat calls.

[jkrems]

Some of the current thoughts around loaders involve having ways to properly sandbox them, e.g. not allowing completely arbitrary code but have clear boundaries and potentially allowing to run the hooks out of the main thread. Not saying that will happen but we are trying to take the security implications seriously.

[TobiasMelen]

I worked a bit on a script to run at postinstall and write dependency paths to a tsconfig as mentioned by @weswigham . There are issues making this solution very sub-optimal, even if Typescripts resolution is used for the type checker only.

1. The entire dependency tree needs to be flattened to a path list in tsconfig. With only direct dependencies, typescript won't be able to resolve types in nested dependencies. Since there's no way to tell what package is relevant for typescript this list gets very long and seems (I haven't benchmarked in any way, sorry) to slow down type checking in vs-code severely.

2. The tooling experience gets really wonky. Typescript will know, and hint, about all kinds of packages that pnp.js will not let you import since they're not direct dependencies. When auto-importing, typescript/vs-code will also translate module names to Yarn cache paths, which is not wanted.

3. @types/ packages have to be merged down in tsconfig/paths to their "real" package counterparts.
   I don't know what implications this has for the compilation process, but it sure feels like something that could cause problems.

Typescript would probably need a config property for overloading parts of the module tree in "node" resolution mode, in a way that can describe nested dependencies, to make a solution like this work reliably.

[arcanis]

@weswigham would you be open to a middle ground solution?

I want to keep the .pnp.js file turing-complete, because it provides a lot of modularity that I suspect we will need. Simultaneously, I understand some of the concerns you have, so let me summarize them to make sure I properly captured the essence of the problem:

• You don't want to provide code hooks into the tsc compiler because any slowdown (that you wouldn't be able to control, by design) would look bad on the tsc team rather than on the people writing a code hook.

• Simultaneously, you don't want to provide code hooks into the tsc compiler because, since it's used by vscode, it could lead to some kind of code execution as soon as a TS file is opened in vscode.

While I don't think those premises are as bad as you think, I don't necessary want to challenge them - it's clear they are a general philosophy of your teams and I can respect this.

Considering your requirements and mines, what would you think about having a package (let's say tsc-pnp), authored by Yarn, that would contain the hook without the data tables? Then this package would read the .pnp.js file (without evaluating it), extract the data tables, and setup the hook using those as source.

Basically, something like this:

.pnp.js

const staticTable = /*table-start*/{
  "lodash": "./cache/lodash-1.0.0",
}/*table-end*/;

exports.resolveName = name => staticTable[name];

tsc-pnp

const pnpJs = readFileSync(pnpJsPath, `utf8`);
const [,table] = pnpJs.match(/\/\*table-start\*\/(.*)\/\*table-end\*\//);
const staticTable = JSON.parse(table);

exports.resolveName = name => staticTable[name];

This would fix the problems mentioned above:

• You would keep control on the resolution performances, and would be able to ensure that we don't put a huge while(true) somewhere by mistake that would affect your teams metrics.

• VSCode wouldn't execute untrusted code by default - you would have the control on when and why you want to upgrade the tsc-pnp package to new versions, and it would go through the classic PR processes, making it traceable.

Now, you might wonder "what's the difference with simply using a json file?" - the difference is that in this case Yarn keeps control over the layout of the .pnp.js file, treating it a bit as an opaque resource. That would give us the latitude we want/need to make changes. I admit even so I find the situation slightly unfortunate (it won't work with different PnP implementations), but it seems the best way to move forward on this.

Would your team be open to discussion on such a solution? I expect some refinement to be needed, of course 🙂

[arcanis]

My best idea yet, I think (particularly with regard to VSCode's context): starting from the v2 we'll be able to make bundles with different set of features (since everything will become a plugin on top of the API). In particular, this means we'll be able to make a version of Yarn that just reads a lockfile and generates a .pnp.js file.

So a solution would be, if you don't trust the .pnp.js file in the repository, to simply regenerate it from within VSCode (in this case it would ship with a small "Yarn microkernel"). This would be fast (zero network because it would just consume the existing lockfile data, zero I/O, because with our v2 model we advocate for the package archives to be stored within in the repository), and secure (no untrusted code would run when opening a file).

What do you think? Yarn is starting to move to TS, but we would really like to get TS support for PnP as it makes the things slightly awkward otherwise. Flow has had support for PnP since even before the release 🙂

[csvn]

I thought PnP looked very interesting, and wanted to try it out before I realized VSCode and Typescript would not be able to find d.ts files correctly. Typescript is such an important tool, that I think trying out PnP will be blocked until it's supported in a reasonable manner by TS. 😅

[Phoenixmatrix]

Not only does it block yarn-pnp, but with TypeScript picking up a lot of steam, it's blocking any kind of innovation in the entire ecosystem unless its Microsoft approved, which is really not great.

[arcanis]

For what it's worth the PnP runtime is now decoupled into its own package, which should solve one of the main concerns the TS team had (they can now simply read the PnP data without executing any third-party code). I'd be more than happy to finally meet and discuss how to make this happen, but I don't feel like the ball ever was in my camp.

[jeysal]

Sorry if this is kind of a "+1" comment, but it looks like there are already solutions proposed and this is in a state where all that's needed is commitment from the TypeScript team.
We've tried to use Yarn PnP in the Jest repo before and post migration from Flow to TypeScript, and this is currently the biggest blocker, the other somewhat big one being ESLint's import rules.
I think this is really essential to helping PnP gain traction, because it enables other large projects that depend on TypeScript for their type checking / build process to use PnP, helping it mature and become more visible as a practical alternative to node_modules 🚀

[NullVoxPopuli]

I just ran in to this. I'm not even trying pnp -- I needed it due to duplicate dependencies in nested_modules directories in a monorepo....

Any movement on this issue would be very much appreciated.

[ark120202]

I have created tsc-pnp - a drop-in replacement for tsc and a VSCode extension that adds PnP support to TypeScript language service. Would love to hear feedback!

[arcanis]

Oh nice! Fwiw @Vlasenko has been working on a similar tool called pnpify that we'll be shipping along with Yarn 2. It will bring native support for tsc, VSCode, and other similar tools that lack builtin support (we emulate a virtual node_modules directory, which is often good enough to fool such tools).

Of course builtin support would still be beneficial to everyone (from a developer experience point of view of course, but because it would enable new features that aren't possible today), but at least we now have some control on the situation. Feel free to ping us on Discord to discuss your extension, @ark120202! Joining our efforts would likely be beneficial to everyone 😃

[thetutlage]

I very much believe in the principles of Typescript for not opening up the API for integrating plugins and that what makes Typescript tooling way more reliable than many other build tools.

As @weswigham said, Node and package managers should settle on a single standard. Yes, the innovation is great, but the pace of adoption cannot be the same and makes the entire ecosystem such a mess

[pauldraper]

We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof

@weswigham, tsc may not intend to call any hooks, but it does run under whatever Node.js runtime and libraries the user has.

For example, the PnP project could produce a Node.js executable that supports PnP natively by doing something like shimming the entire fs module.

This is in fact exactly how tink exec works. Is that deep (and still arbitrary) manipulation preferable? I would think the answer is "no".

(especially when in an editor)

That's a lost battle entirely. There are commonly so many moving parts that slow things down: other plugins, remote file protocols, etc. Even beyond that, again, TypeScript is merely the guest of whatever IDE the user has chosen, which again could be doing tink-level intrusion.

Personally, I'd look at PnP like that: akin to a file system; a user-chosen, low-level, cross-cutting concern that may impact performance. (But PnP is not literally file system calls because that's a poor way to abstract/implement module resolution. RIP tink.)

Node and package managers should settle on a single standard.

Agreed. Hot take: the only (good) innovation in the Node.js package managment space comes from Yarn.

---

Policy question aside, @arcanis I believe there is only thing lacked by the PnP API for TypeScript: inferred imports.

When declaration files are generated, the type inference can produce types that are not named in the source file. Formerly, these would simply fail.

Then TypeScript came up an algorithm to attempt to automatically produce a import for these. (Short version: 1. Try node_modules 2. Try relative 3. Fail if the relative path contains "node_modules" in it.)

import { a } from 'a';
export const b = a;

import { a } from 'a';
declare export const b: import('some/module').SomeType;

Either the behavior would go back to what it was before (a regression, but not necessarily a deal breaker), or the PnP API needs a reverseResolve that calculates an import from one file to another (if one exists).

[0alpha]

What's blocking this right now?

[pauldraper]

1. The last update from the TS team on this issue is Nov 2018 by @weswigham stressing the desire to not run external hooks, for stability and performance.

2. PnP can improve performance due to lower number of stats calls.

3. PnP has introduced a static file format (though the primary interface is JS).

4. The similar npm effort tink has been dead for over a year.

5. pnpify tsc works, by hooking the fs module.

6. PR Native support for PnP #35206 was filed in Nov 2019 for native support (currently open).

7. @yarnpkg/plugin-compat automatically applies that patch.

[luxalpa]

It's three years later and still no solution in sight for such a trivial issue. Honestly, everyone is completely overblowing this. I don't see why typescript can't have a special flag for one of the most popular package managers out there. Yes, a common standard would be nice, but seriously it's not a big deal. Typescript also supports various old JavaScript ideas (such as experimental decorators) and react's JSX. People are afraid of adding a feature because they can't remove it later, but honestly it's still way better to add a feature temporarily and break everyone's build in the future than not having the feature at all and thus breaking everyones builds for all times.
The typescript team claims to care about the languages performance, reliability and security impact, yet with their decision they effectively encourage and force other people to hack and circumvent the typescript compiler in all kinds of ways. This turns typescript usage from a pleasant experience into an insecure mess.
And if you're so worried, where is the problem with just asking the user this program is executing scripts, are you sure you want to allow it? in tsc or VSCode? It's like 10 minutes of extra work.

On the other hand I also don't understand why yarn must so desperately need to have this .js file. Sure it's neat, but why not simply give the user the option to provide a typescript specific .json file? I don't see why it must be .js at this current time - especially since many typescript user only care about the typings anyway.

I'm probably not understanding the entirety of this issue, but I find this discussion really weird. I don't know what else to say, but it should have been solved by now, in fact I'm sure this issue could have been solved on a single afternoon and does not need to spend 3 years with no progress. I'd much rather spend my time creating applications than fixing my tools.

[arcanis]

On the other hand I also don't understand why yarn must so desperately need to have this .js file. Sure it's neat, but why not simply give the user the option to provide a typescript specific .json file? I don't see why it must be .js at this current time - especially since many typescript user only care about the typings anyway.

It's already been possible to generate a .json file for more two years (and we did it exclusively for TS, no one else really need it), but it hasn't helped a lot so far - the last message from the TS team (here) didn't seem aware of that, and my correction was seemingly lost in notifications.

Generally I can say that the Yarn team is quite open to make compromises in order to find a solution that would be satisfying to us, the TS team, and our users. What we're lacking is someone on their side to pick up the phone and make that an actual discussion, rather than us begging for their attention.

[thetutlage]

I think the TypeScript team has a valid point here. The tooling providers need to settle on a standard. No one can say, I will follow my own set of standards and then the entire ecosystem should add a flag for it.

[darthtrevino]

Yarn 2+ is extremely well built tooling, and @arcanis is doing heroic work making it compatible across the JS ecosystem.

[thetutlage]

I am not saying Yarn 2 is bad. If you zoom out a little bit, then it is so hard to know which trend to follow and which one to ignore.

Let's imagine, the TypeScript team adds a flag to support YarnPnP. Then another tool comes and makes the same argument (the we are better or faster) and then TypeScript cannot say no, as they have already committed in the direction of adding flags.

I have seen this trend a lot less in other more mature ecosystems in comparison to JavaScript

---

Also, the arguments should not be considered against something or someone. Yarn is a fantastic tool, created by great people. I am just talking about the idea of adding flags in general

[darthtrevino]

Sure, but there should be support for common module systems, or some path forward for them. I’d be glad to facilitate a call if the TS team is open to it. Plug n Play is not a fly by night system, it’s been around for years and it’s a dramatic improvement in safety and correctness over node_modules.

On Thu, Sep 2, 2021 at 9:01 PM Harminder Virk ***@***.***> wrote: I am not saying Yarn 2 is bad. If you zoom out a little bit, then it is so hard to know which trend to follow and which one to ignore. Let's imagine, the TypeScript team adds a flag to support YarnPnP. Then another tool comes and makes the same argument (the we are better or faster) and then TypeScript cannot say no, as they have already committed in the direction of adding flags. I have seen this trend a lot less in other more mature ecosystems in comparison to JavaScript — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3XCESR5V54O4UTF4V253UABCADANCNFSM4GBBNH5Q> .

-- Sent from Gmail Mobile

[luxalpa]

I think the TypeScript team has a valid point here. The tooling providers need to settle on a standard. No one can say, I will follow my own set of standards and then the entire ecosystem should add a flag for it.

The entire point of competing standards is to give developers choices. Typescript exists exactly because of this; else any developer could say "why should we care about typescript, there's 100s of other programming languages". This is why the other guy earlier said something about typescript being a blocker here. Progress always means that some things are going to be different than others. For example even though there are Vue.js, Ember.js, Svelte and Angular as other frontend frameworks, the typescript team decided to primarily follow React with their JSX philosophy. We could put the same argument here that Typescript should not be supporting TSX at all until all the JavaScript frameworks decided on a templating language, but we also all know this will never happen.

The reason why Yarn's word here has importance whereas some random developer / app doesn't is because Yarn is an extremely widely used tool and yes, we don't want to support every tiny unimportant personal toolset. But we do want to support everything that's popular because we don't want to block progress. It's time to wake up from the dream land where everyone uses the same tools and protocols and instead accept the reality.

[Phoenixmatrix]

The tooling providers need to settle on a standard

TypeScript is literally the only major part of the entire toolchain that doesn't let you implement a custom resolver. Hell, the language service does, it's just the wrappers (tsc, VSCode) that don't expose it.

Webpack, ESLint, Jest, you name it. They all let you. TypeScript is the problem here.

[0rvar]

Now that Yarn PnP can produce a static JSON file, there should be no objections left to implementing this.

I am very impressed with Typescript, and it has made frontend development 100x better in my eyes.
On the other hand, I am not happy about the attitude towards PnP from the Typescript team.
Please re-evaluate this proposal with fresh eyes!

[pauldraper]

The typescript team claims to care about the languages performance, reliability and security impact, yet with their decision they effectively encourage and force other people to hack and circumvent the typescript compiler in all kinds of ways.

Exactly. And it's not like TypeScript even ships with it's own JS runtime. It's always depending on some arbitrary code.

It's Node.js, or browser, or Nashorn, or Deno or whatever.

Indeed right now, users hack the runtime to get PnP to work.

AFAIK TypeScript remains the last common Node.js tool not to support PnP in a reasonable way.

[pauldraper]

@arcanis if you read this thread, can you merge arcanis/ts-pnp#8 ?

Without it, ts-pnp doesn't work for scoped typings packages.

[Luiz-Monad]

As I said - we've currently got no plans for a tsconfig.js (and I bring it up often) or a tsconfig managed plugin system. We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor). I definitely know how I'd like to expose the ability to do something like this, were we to accept it, since I designed an extensive plugin model for the compiler in the past (before it was declined).

Not to say we won't revisit it at some point, but similar suggestions have definitely been declined repeatedly in the past. A static format is much easier to deal with.

F# has Type providers, they are definitively a extension to the compiler type system, but you are clearly including them on User code, its very obvious when the compiler takes a performance hit. Its the same as including any library, like Angular which is super big and makes the compiler shuggle. I don't get this argument, "but we are using third party code on the compiler"

http://blog.mavnn.co.uk/type-providers-from-the-ground-up/

At this moment, TypeProviders does run inside the F# compiler, but nothing would preclude the infrastructure to just emit "fsi" (the equivalent of ".d.ts") files dynamically and feed it into the compiler. Nothing would also preclude other tools from doing similar things to the Typescript compiler, emitting in memory ".d.ts" from the "zip" archives on ts-pnp and feeding them on the compiler. If you can't take a hit of just reading a definition file... no matter where it comes from.

This feature would make the static compiler for a dynamic environment so much more flexible.
Meanwhile, perhaps I should write a converter to port TypeScript to F#. (despite the impedance between Row Types and GADTs, and the Typescript typesystem being unsound, but we can always use dynamic as escape hatch on F#/C#, or ignore all types and use a H&M inference algorithm)