Add support to transpile modules inside node_modules

[arunoda]

Now some of us ships NPM packages (specially components) written in ES2015 without transpiling them.

That's a pretty good thing specially if they are gonna used in a project like Next.js or CRA (which does transpiling). They offer benefits like:

• No need to transpile before shipping to NPM
• Get the benefit of Webpack 2's tree-shaking

But we can't do this now we exclude everything inside node_modules from babel transpiling.

So, here's the proposed solution.

We have an entry in next.config.js to include modules which needs to go through babel. See:

module.exports = {
   transpileModules: [
     "my-component",
     "redux/src"
   ]
}

[rauchg]

• Why is this not doable with webpack() extension?
• transpileModules sounds better
• Will we accept regexps?

[arunoda]

Why is this not doable with webpack() extension?

I hope you mean custom webpack config. That's doable. But this sits in a existing loader. Getting that is somewhat harder.

transpileModules sounds better

Awesome. I'll update.

Will we accept regexps?
Yep. sure.

[slorber]

Hey,

As I'm starting my own website I'm trying new techs like Lerna/Next/Styled... and would be happy to provide early feedback on this.

I've opened a duplicate issue where I tried to import/transpile a CRA-based module in my NextJs module but didn't know how to make the transpilation happen (note that I'd like to keep my module runnable as a standalone)

I've also noticed that Babel, also based on Lerna, is transpiling each modules before exposing them, but it looks to me better to do like @arunoda suggest and let the client app do the transpiling.

I'd like to have a single babel config for my client and share that config with all my decoupled modules. That's probably not so easy if I want to keep ability to run my modules as standalone, outside of Next runner

My current test project is here: https://github.com/slorber/playground/ I'll try to upgrade it as soon as there's a fork/PR. @arunoda are you working on it?

[arunoda]

@slorber currently we are focusing on 2.0 release and we are fine tuning stuff and finding bugs as possible as we can.

I haven't started work on this but we can do this just after 2.0.

[slorber]

Ok so I'll make a fork. I'm already running against 2.0.0 beta because I'm not building a critical website and I don't think webpack 1.13 resolve jsnext:main/module field.

I'm not a bundler expert but I think I'd rather use "module" field of package.json no? "main" seems for already transpiled code as far as I know. But as the webpack config allows to include/exclude transpilation I'm not sure it's relevant. Any recommendation on which of the 3 fields I'd rather use?

[arunoda]

@slorber I think webpack only supports main just like NPM. You can use that.
We can check for the filepath in the exclude function in our next.conf.js

[slorber]

Hmm according to what I've seen in practice against Next 2.x I've seen module works (but fails later at runtime because not transpiled) while jsnext:main did not work (as far as I remember). But it's supposed to be supported.

Anyway, jsnext:main or module does not seem to be the solution to this problem so for company-internal modules just enabling transpilation is probably enough

[rauchg]

The community has not agreed on one approach right? For example, I was able to use react-youtube the other out of the box with no problems. I'm assuming a big number of modules transpile before publish?

Ref: https://github.com/rauchg/blog/blob/master/components/post/youtube.js

[slorber]

Yes that makes sense to always transpile before publish because you don't know who/how the module will be consumed and you don't want to force the client to setup appropriate babel settings for your lib. That's what Rollup suggest: to publish the module transpiled in different ways so that bundler can decide which to use.

But for company internal packages, the transpilation settings might be the same across several project (like a babel preset) and it makes sense to me to let the client bundler to transpile all the company dependencies

[philcockfield]

Very much agree @slorber - this would be very handy for internal modules if you're breaking your project up and isolating things as much as possible.

And @rauchg / @arunoda supporting RegExp's would be really nice, so you could have one entry that catches all the company internal modules, using say the NPM org namespace:

// next.config.js
module.exports = {
  transpileModules: [
    /^\@my-npm-org\/.*/
  ]
}

[rauchg]

Beautiful suggestion @philcockfield

[slorber]

Hey maybe it could be worth offering some presets. It looks to me most tools (Lerna/npm link...) rely on symlinks so why not something as simple as:

module.exports = {
  transpileModules: ["symlinks"]
}

[philcockfield]

The more I use next.js in earnest, and build out a rich library of modules around it, the more this feature becomes important. It's becoming a real PITA replicating the babel compilation step in my internal modules.

🚀🤖

[arunoda]

I'm working on this today :)

[arunoda]

@philcockfield give this a try: #749

[slorber]

thanks @arunoda

So as commented on your PR if this does not support symlinks the feature will be a bit limited because it won't work with npm link or Lerna, but only for npm modules that are not transpiled (right? I don't see any other usecase unless you commit modules inside /node_modules)

Why not supporting symlinks? is it harder to support?

Also I wanted to test your branch on my app, but I'm not sure what's the best way to do that. Is there any known procedure so we can easily test a branch and it's not too painful for the tester? I've tried some stuff like:

• npm install https://github.com/arunoda/next.js.git#add-706 : fails because the next /bin folder is empty on github repo
• git clone of the fork inside /node_modules: not much success (but it might be because of my specific Lerna settings)

What's the best way to test a fork currently?

[lorensr]

If you're looking at doing this with next.config.js: module.exports = { webpack: (config, then config.module.rules has a few things, looks like you need to change one of these rules, or add one?:

  { loader: 'babel-loader',
    include: '/Users/me/gh/guide/node_modules/next/dist/pages',
    options: 
     { babelrc: false,
       cacheDirectory: true,
       sourceMaps: 'both',
       plugins: [Object] } },
  { test: /\.js(\?[^?]*)?$/,
    loader: 'babel-loader',
    include: 
     [ '/Users/me/gh/guide',
       '/Users/me/gh/guide/node_modules/next/dist/pages' ],
    exclude: [Function: exclude],
    query: 
     { babelrc: true,
       cacheDirectory: true,
       sourceMaps: 'both',
       presets: [] } } ]

Looking forward to the simpler syntax suggested.

[andrewmclagan]

Sorry for my ignorance I cant see what the resolution of this issue is? We would love to be importing es6 into our codebase, we need the tree-shaking .

Is there a PR on this?

[slorber]

@andrewmclagan This issue is still open and has a related PR that probably won't satisfy all (like LernaJS users)

[Yuripetusko]

What's the status of this? Are there any other ways to make next's webpack to transpile files imported from node_modules ?

[andrewmclagan]

@slorber i will take a look at the PR. Contribute our use-case.

[muhajirdev]

I am facing kind of similiar problem. Trying to use get-urls package. Works find with dev but when i compile it. I got error from uglify

...
{ Error: commons.js from UglifyJs
...

Is there any workaround for this please?

[MoOx]

Also next-transpile-modules doesn't work with folder that don't have package.json main field. And when working with ReScript, @rescript/std is never consumed directly (only files directly since the code is generated by rescript-compiler) and having a main doesn't make any sense.
An built-in solution is more than welcome several years after discussions about this :)

I am willing to pay if that can help as next-transpile-modules is officially a big hack (that doesn't even work in all case as explained above)

[macrozone]

Also next-transpile-modules doesn't work with folder that don't have package.json main field. And when working with ReScript, @rescript/std is never consumed directly (only files directly since the code is generated by rescript-compiler) and having a main doesn't make any sense.
An built-in solution is more than welcome several years after discussions about this :)

I am willing to pay if that can help as next-transpile-modules is officially a big hack (that doesn't even work in all case as explained above)

i agree with that, next-transpile-modules just broke with next 10.0.8 and I think its time to rethink that

[belgattitude]

@macrozone

next-transpile-modules just broke with next 10.0.8

Any specific problem ? Asking cause I'm using it lot and upgrade regularly (even on 10.0.8, see https://github.com/belgattitude/nextjs-monorepo-example... but in private repos too).

next-transpile-modules is officially a big hack

node_modules seems a big hack 😄 , I feel like latest version of next-transpile-modules are pretty well done...

That said a built-in would be welcome.

PS: for monorepo world, I feel making the setup in next.config might lead to a situation where we have to repeat the setup in other apps (nestjs...)... I really liked the approach from @jeantil : #15327... worth considering.

[MoOx]

next-transpile-modules is officially a big hack

First line of the code of next-transpile-modules

/**
 * disclaimer:
 *
 * THIS PLUGIN IS A F*CKING BIG HACK.
 *
 * don't even try to reason about the quality of the following lines of code.
 */

[martpie]

👋 😅

I cannot speak in the name of Vercel, but I have talked a little bit with Tim a while ago. They are aware of this problem and they understand there is a real need for this. I offered my help to work on an official solution.

It's getting higher and higher in the backlog, but it's just not there yet. It will come (in time...)! in the meantime, I'll keep maintaining next-transpile-modules the best I can. 🙇

We all just need to be patient.

Sidenote: if your usecase is IE11 support, I cannot encourage you enough to convince your n+1 to drop support for it. There are plenty of good reasons for it, but the one they (managers) will listen to is probably money. So if you can convince them it costs you more money to support IE11 than it brings, you may be able to succeed in your "negotiations" :p

[macrozone]

@macrozone

next-transpile-modules just broke with next 10.0.8

Any specific problem ? Asking cause I'm using it lot and upgrade regularly (even on 10.0.8, see https://github.com/belgattitude/nextjs-monorepo-example... but in private repos too).

tbh, i don't remember, it throw some webpack error (unrecognized option), but i decided to remove next-transpile-modules, because i just used it for lodash-es and switched to normal lodash (and babel-plugin-lodash to help with tree shaking)

we also dropped support for IE in most of our projects (🥰), so "modern" packages are less of a problem now than it was

Sidenote: if your usecase is IE11 support, I cannot encourage you enough to convince your n+1 to drop support for it. There are plenty of good reasons for it, but the one they (managers) will listen to is probably money. So if you can convince them it costs you more money to support IE11 than it brings, you may be able to succeed in your "negotiations" :p

totally agree on that, try it. They will say that companies still use ie11, but thats more than often a lie. Most companies that i know had at least one other, modern browser installed and ie11 only for compatibilty reasons. Its just that their users are uneducated and/or installations are poorly managed.

Also supporting IE means keeping it alive. So you invest that the problem persists.

[O4epegb]

I have not seen any errors with next-transpile-modules and next@10.0.8, it works fine. And I think it is better solution than having babel plugin (because then you need to have babel config and this is another can of worms).

But I agree that it would be nice to have native nextjs solution for this, that could handle es imports at very least, this is the main case for me to use next-transpile-modules, because more and more libraries are using untranspiled esm imports.

[ghost]

@rauchg any updates around this? This issue has been around for sometime, and packaging dependencies is actually a more complex situation now than in it was in 2017. It'd be great to know where this landed.

[diegogallovich]

Hey Everyone, this package right here is a great, I would say stable solution for this issue while the guys at Vercel close this.

https://www.npmjs.com/package/next-transpile-modules

In my case, next was having issues transpiling this package: https://github.com/tbleckert/react-select-search

'next-transpile-modules' really solved it for me in a non corrosive, or invading way. Just transpiles the packages you specify (The ones you know are giving you a headache)

[TomasHubelbauer]

In Next 12 with create-next-app, this error happens when importing next/server in API routes. I wanted to import it to get the types for request and response in the API route function parameters and ran into this. I ended up using untyped parameters instead for now, hope this will get resolved soon.