Add support to transpile modules inside node_modules

[arunoda]

Now some of us ships NPM packages (specially components) written in ES2015 without transpiling them.

That's a pretty good thing specially if they are gonna used in a project like Next.js or CRA (which does transpiling). They offer benefits like:

• No need to transpile before shipping to NPM
• Get the benefit of Webpack 2's tree-shaking

But we can't do this now we exclude everything inside node_modules from babel transpiling.

So, here's the proposed solution.

We have an entry in next.config.js to include modules which needs to go through babel. See:

module.exports = {
   transpileModules: [
     "my-component",
     "redux/src"
   ]
}

[rauchg]

• Why is this not doable with webpack() extension?
• transpileModules sounds better
• Will we accept regexps?

[arunoda]

Why is this not doable with webpack() extension?

I hope you mean custom webpack config. That's doable. But this sits in a existing loader. Getting that is somewhat harder.

transpileModules sounds better

Awesome. I'll update.

Will we accept regexps?
Yep. sure.

[slorber]

Hey,

As I'm starting my own website I'm trying new techs like Lerna/Next/Styled... and would be happy to provide early feedback on this.

I've opened a duplicate issue where I tried to import/transpile a CRA-based module in my NextJs module but didn't know how to make the transpilation happen (note that I'd like to keep my module runnable as a standalone)

I've also noticed that Babel, also based on Lerna, is transpiling each modules before exposing them, but it looks to me better to do like @arunoda suggest and let the client app do the transpiling.

I'd like to have a single babel config for my client and share that config with all my decoupled modules. That's probably not so easy if I want to keep ability to run my modules as standalone, outside of Next runner

My current test project is here: https://github.com/slorber/playground/ I'll try to upgrade it as soon as there's a fork/PR. @arunoda are you working on it?

[arunoda]

@slorber currently we are focusing on 2.0 release and we are fine tuning stuff and finding bugs as possible as we can.

I haven't started work on this but we can do this just after 2.0.

[slorber]

Ok so I'll make a fork. I'm already running against 2.0.0 beta because I'm not building a critical website and I don't think webpack 1.13 resolve jsnext:main/module field.

I'm not a bundler expert but I think I'd rather use "module" field of package.json no? "main" seems for already transpiled code as far as I know. But as the webpack config allows to include/exclude transpilation I'm not sure it's relevant. Any recommendation on which of the 3 fields I'd rather use?

[arunoda]

@slorber I think webpack only supports main just like NPM. You can use that.
We can check for the filepath in the exclude function in our next.conf.js

[slorber]

Hmm according to what I've seen in practice against Next 2.x I've seen module works (but fails later at runtime because not transpiled) while jsnext:main did not work (as far as I remember). But it's supposed to be supported.

Anyway, jsnext:main or module does not seem to be the solution to this problem so for company-internal modules just enabling transpilation is probably enough

[rauchg]

The community has not agreed on one approach right? For example, I was able to use react-youtube the other out of the box with no problems. I'm assuming a big number of modules transpile before publish?

Ref: https://github.com/rauchg/blog/blob/master/components/post/youtube.js

[slorber]

Yes that makes sense to always transpile before publish because you don't know who/how the module will be consumed and you don't want to force the client to setup appropriate babel settings for your lib. That's what Rollup suggest: to publish the module transpiled in different ways so that bundler can decide which to use.

But for company internal packages, the transpilation settings might be the same across several project (like a babel preset) and it makes sense to me to let the client bundler to transpile all the company dependencies

[philcockfield]

Very much agree @slorber - this would be very handy for internal modules if you're breaking your project up and isolating things as much as possible.

And @rauchg / @arunoda supporting RegExp's would be really nice, so you could have one entry that catches all the company internal modules, using say the NPM org namespace:

// next.config.js
module.exports = {
  transpileModules: [
    /^\@my-npm-org\/.*/
  ]
}

[rauchg]

Beautiful suggestion @philcockfield

[slorber]

Hey maybe it could be worth offering some presets. It looks to me most tools (Lerna/npm link...) rely on symlinks so why not something as simple as:

module.exports = {
  transpileModules: ["symlinks"]
}

[philcockfield]

The more I use next.js in earnest, and build out a rich library of modules around it, the more this feature becomes important. It's becoming a real PITA replicating the babel compilation step in my internal modules.

🚀🤖

[arunoda]

I'm working on this today :)

[arunoda]

@philcockfield give this a try: #749

[MoOx]

Also next-transpile-modules doesn't work with folder that don't have package.json main field. And when working with ReScript, @rescript/std is never consumed directly (only files directly since the code is generated by rescript-compiler) and having a main doesn't make any sense.
An built-in solution is more than welcome several years after discussions about this :)

I am willing to pay if that can help as next-transpile-modules is officially a big hack (that doesn't even work in all case as explained above)

[macrozone]

Also next-transpile-modules doesn't work with folder that don't have package.json main field. And when working with ReScript, @rescript/std is never consumed directly (only files directly since the code is generated by rescript-compiler) and having a main doesn't make any sense.
An built-in solution is more than welcome several years after discussions about this :)

I am willing to pay if that can help as next-transpile-modules is officially a big hack (that doesn't even work in all case as explained above)

i agree with that, next-transpile-modules just broke with next 10.0.8 and I think its time to rethink that

[belgattitude]

@macrozone

next-transpile-modules just broke with next 10.0.8

Any specific problem ? Asking cause I'm using it lot and upgrade regularly (even on 10.0.8, see https://github.com/belgattitude/nextjs-monorepo-example... but in private repos too).

next-transpile-modules is officially a big hack

node_modules seems a big hack 😄 , I feel like latest version of next-transpile-modules are pretty well done...

That said a built-in would be welcome.

PS: for monorepo world, I feel making the setup in next.config might lead to a situation where we have to repeat the setup in other apps (nestjs...)... I really liked the approach from @jeantil : #15327... worth considering.

[MoOx]

next-transpile-modules is officially a big hack

First line of the code of next-transpile-modules

/**
 * disclaimer:
 *
 * THIS PLUGIN IS A F*CKING BIG HACK.
 *
 * don't even try to reason about the quality of the following lines of code.
 */

[martpie]

👋 😅

I cannot speak in the name of Vercel, but I have talked a little bit with Tim a while ago. They are aware of this problem and they understand there is a real need for this. I offered my help to work on an official solution.

It's getting higher and higher in the backlog, but it's just not there yet. It will come (in time...)! in the meantime, I'll keep maintaining next-transpile-modules the best I can. 🙇

We all just need to be patient.

Sidenote: if your usecase is IE11 support, I cannot encourage you enough to convince your n+1 to drop support for it. There are plenty of good reasons for it, but the one they (managers) will listen to is probably money. So if you can convince them it costs you more money to support IE11 than it brings, you may be able to succeed in your "negotiations" :p

[macrozone]

@macrozone

next-transpile-modules just broke with next 10.0.8

Any specific problem ? Asking cause I'm using it lot and upgrade regularly (even on 10.0.8, see https://github.com/belgattitude/nextjs-monorepo-example... but in private repos too).

tbh, i don't remember, it throw some webpack error (unrecognized option), but i decided to remove next-transpile-modules, because i just used it for lodash-es and switched to normal lodash (and babel-plugin-lodash to help with tree shaking)

we also dropped support for IE in most of our projects (🥰), so "modern" packages are less of a problem now than it was

Sidenote: if your usecase is IE11 support, I cannot encourage you enough to convince your n+1 to drop support for it. There are plenty of good reasons for it, but the one they (managers) will listen to is probably money. So if you can convince them it costs you more money to support IE11 than it brings, you may be able to succeed in your "negotiations" :p

totally agree on that, try it. They will say that companies still use ie11, but thats more than often a lie. Most companies that i know had at least one other, modern browser installed and ie11 only for compatibilty reasons. Its just that their users are uneducated and/or installations are poorly managed.

Also supporting IE means keeping it alive. So you invest that the problem persists.

[O4epegb]

I have not seen any errors with next-transpile-modules and next@10.0.8, it works fine. And I think it is better solution than having babel plugin (because then you need to have babel config and this is another can of worms).

But I agree that it would be nice to have native nextjs solution for this, that could handle es imports at very least, this is the main case for me to use next-transpile-modules, because more and more libraries are using untranspiled esm imports.

[ghost]

@rauchg any updates around this? This issue has been around for sometime, and packaging dependencies is actually a more complex situation now than in it was in 2017. It'd be great to know where this landed.

[diegogallovich]

Hey Everyone, this package right here is a great, I would say stable solution for this issue while the guys at Vercel close this.

https://www.npmjs.com/package/next-transpile-modules

In my case, next was having issues transpiling this package: https://github.com/tbleckert/react-select-search

'next-transpile-modules' really solved it for me in a non corrosive, or invading way. Just transpiles the packages you specify (The ones you know are giving you a headache)

[TomasHubelbauer]

In Next 12 with create-next-app, this error happens when importing next/server in API routes. I wanted to import it to get the types for request and response in the API route function parameters and ran into this. I ended up using untyped parameters instead for now, hope this will get resolved soon.

[kirkstrobeck]

This works for me – I revised this solution to non-mutate, posting for others

const path = require('path');

module.exports = () => ({
  webpack: (config, { defaultLoaders: { babel } }) => ({
    ...config,

    module: {
      ...config.module,

      rules: [
        ...config.module.rules,

        {
          include: [path.resolve(__dirname, '../')],
          test: /\.(js|jsx|ts|tsx)$/,
          use: [babel],
        },
      ],
    },
    
    resolve: {
      ...config.resolve,

      alias: {
        ...config.resolve.alias,

        root: path.resolve(__dirname, '../'),
      },
    },
  }),
});

[karol-f]

Yeah, but still, it won't help you with things that next-transpile-modules handles well, e.g. Global CSS - martpie/next-transpile-modules#146

[kirkstrobeck]

Interesting

[bcowell-riis]

Any update on this?

[alec-chernicki]

Want to bump this as we're leveraging more monorepo tooling at Disney Streaming and starting to lean into unbuilt internal packages similar to this blog post: https://turborepo.com/posts/you-might-not-need-typescript-project-references. It would be amazing to have a first-class way to tell the compiler to transform files within node_modules if they match a regex (we'd make it match our internal package scope).

[danoc]

Does anyone know if externalDir will be "graduating" from experimental in the near future? We'd like to use it, but we're not comfortable using it until the functionality is stable.

[TxHawks]

I have no idea, but I think it's safe to use even if it doesn't eventually graduate.

I say this because before it was introduced, we had a custom Webpack config extending Next's doing the same, which worked just fine.

It might need maintenance from time to time with changes in Next internals, but shouldn't be too much of a Hussle. externalDir is nicer though, so I hope it does end up being officially supported.

[wmira]

amazing that this is still not supported.. :(

[sheerun]

Does https://vitejs.dev/ support it?

[belgattitude]

@AlecOrtega

Just saw your message by scrolling up, this might interest you: #32223 (reply in thread).

About the status experimental I wouldn't worry too much... Before it landed I used something in the lines of

const nextConfig = {
  webpack: (config, { defaultLoaders }) => {
    const resolvedBaseUrl = path.resolve(config.context, "../../");
    config.module.rules = [
      ...config.module.rules,
      {
        test: /\.(tsx|ts|js|jsx|json)$/,
        include: [resolvedBaseUrl],
        use: defaultLoaders.babel,
        exclude: (excludePath) => {
          return /node_modules/.test(excludePath);
        },
      },
    ];
    return config;
  },
};

Which is relatively simple in case the feature is dropped.

[mikestopcontinues]

This FR is an anti-pattern. Treat your packages like packages. If you want to speed up your dev workflow, use your workspace manager to build watch all of the packages you're working on.

[Pauan]

@mikestopcontinues which I do think is a bad direction since it encourages treating packages inside node_modules in a non-standard fashion.

ES6 modules are standard and official. CommonJS is not official, and it is being deprecated and phased out. Every bundler (except Next) supports ES6 modules in node_modules.

They should behave like node_modules do, as if you were depending on external deps.

Like I said, external npm packages (like node-fetch and chalk) are already being shipped with pure ES6 modules. That means that ES6 packages inside of node_modules need to be supported.

This being the JS ecosystem, we're never going to all agree on how things should be done.

No, we already have an agreement. ES6 modules are official. They are supported in the same way by every bundler (Webpack, Rollup, Snowpack, Parcel, etc.)

Next is the only bundler which is not following the already established conventions.

[belgattitude]

@Pauan

Like I said, external npm packages (like node-fetch and chalk) are already being shipped with pure ES6 modules. That means that ES6 packages inside of node_modules need to be supported.

I'm not sure when but es modules are already supported by nextjs, my best guess: https://nextjs.org/blog/next-11-1#es-modules-support.

I used them under an experimental flag since long. But I don't think you have to set it up nowadays

It was something in the lines of

module.exports = {
  // Prefer loading of ES Modules over CommonJS
  experimental: { esmExternals: true },
};

The only problem I had with esm is still when setting up jest (even the latest 29 seems difficult to work with). FWIW I switched to vitest.

[Pauan]

@belgattitude Unfortunately esmExternals doesn't always work, because it requires the package to have "type": "module".

But there are many packages that use ES6 modules (with the appropriate "module" field in package.json) that do not use "type": "module", because they are browser packages, not Node packages.

So although Next does have partial support for ES6 modules in node_modules, it doesn't match what the other bundlers do, and so many npm packages are still broken.

[mikestopcontinues]

@Pauan I think you're advocating for the wrong thing. Instead of trying to complicate matters further by transpiling modules within node_modules, let's simply advocate for Next to support ESM properly, so we can bundle our sub-packages in only one format. (@kylemh, this is why it's an anti-pattern.)

[belgattitude]

@Pauan @mikestopcontinues I'm genuinely interested, could you provide libraries or use cases that don't work related to es modules and nextjs ? Cause I can't figure out by myself

I would gladly create reproductions (or with remix, vite-react-ssr), I feel this is the way to move forward and make it actionable

[leerob]

ICYMI: Next.js 13.1 included built-in transpilation:

/** @type {import('next').NextConfig} */
const nextConfig = {
  transpilePackages: ['@acme/ui', 'lodash-es'],
};
 
module.exports = nextConfig;

https://nextjs.org/blog/next-13-1#built-in-module-transpilation-stable