npm@5.3: npm prune --production fails

[michaelkrone]

I'm opening this issue because:

• npm is producing an incorrect install.
• npm is crashing.
• npm is doing something I don't understand.
• Other (see below for feature requests):

What's going wrong?

npm prune --production fails with npm ERR! May not delete: /path/node_modules/.bin
NODE_ENV=production npm prune fails with npm ERR! May not delete: /path/node_modules/.bin

How can the CLI team reproduce the problem?

Run NODE_ENV=production npm prune or npm prune --production in any project which has a node_modules/.bin folder with npm@5.3.0

npm@5.2.0 is working as expected.

supporting information:

• npm -v prints: 5.3.0
• node -v prints: v8.1.4
• npm config get registry prints: https://registry.npmjs.org/
• Windows, OS X/macOS, or Linux?: Linux 3.13.0-123-generic

[etremblay]

We have the same issue with npm 5.3.0

[fasterthanlime]

A few details on how it affects us: we're shipping an Electron app for Linux/macOS/Windows.

We ship a PKGBUILD for ArchLinux - the Arch package for npm has already been upgraded to @5.3.0 - there's no way to depend on 5.2.0 specifically, so if the user already has 5.3.0 installed, the PKGBUILD will just fail on the npm prune --production part.

(Electron apps typically run npm prune --production as part of their packaging because the devDeps just take up disk space and aren't used in production).

I've tried working around it by wiping node_modules/.bin before calling npm prune but that didn't seem to work either.

I think I'm out of workaround until npm@5.4.0 ships ☹️

---

Here's an npm debug log showing the failure: https://gist.github.com/fasterthanlime/d72dabfb7467f8909b95aa6e1f3e1f40

---

Another npm debug log right after running rm -rf node_modules: https://gist.github.com/fasterthanlime/6992fb80d4efb1235d2106c65b1f38bc

---

Reading the npm source, unbuild.js calls gentlyRm on npm.bin here then gentlyRm throws a gentle reminder that removing npm.bin is not allowed, here.

Neither of these files have been touched between 5.2.0 and 5.3.0 though.

---

Confirming that: this happens

• (at least) on Linux & Windows (haven't tried macOS)
• whenever the project in question depends on a package that installs something to .bin - whether or not that path actually exists on disk
  • I'm guessing that applies only to dev packages, otherwise npm prune --production wouldn't be trying to remove stuff

[robjtede]

Can confirm this happens on MacOS High Sierra (10.13) too.

[fasterthanlime]

I still haven't been able to find a workaround: this prevents us from releasing a new version of our app on some platforms. I hate to look impatient but npm is a pretty critical piece of infrastructure 😄

Any kind of status update would be super appreciated! ❤️

[robjtede]

@fasterthanlime I was able to roll back with npm i -g npm@5.2

[fasterthanlime]

@fasterthanlime I was able to roll back with npm i -g npm@5.2

Thanks! That's not applicable when releasing an install script for Arch Linux though: https://github.com/itchio/itch/blob/master/release/templates/PKGBUILD.in - if I depend on npm>=5.2, it'll install npm@5.3.0, and if it installs 5.3.0, the whole process will fail when it attempts to prune.

[GianlucaBortoli]

I'm having this issue too, both on widows 10 x64 and on ubuntu 14.04.
I confirm that this problem does not happen with npm versions <= 5.2.

[jamesjessian]

@GianlucaBortoli Do you mean the problem happens with npm > 5.2 ?
The problem goes away for me (win 10 x64 - trying to get electron-packager working) if I roll back with npm i -g npm@5.2. Thanks @robjtede.

[GianlucaBortoli]

@bishbashbosh123 by bad, I just fixed the comment to stress that the broken npm versions are > 5.2

[sholladay]

I would think that a viable workaround for this would just be to run:

rm -rf node_modules package-lock.json && npm install --production

[fasterthanlime]

rm -rf node_modules package-lock.json && npm install --production

I realize you're trying to help, but:

• removing package-lock.json is certainly not desirable (very happy that npm@5 ships something better than shrinkwrap for that)
• if you have tools like electron-rebuild as devDependencies, it's not a viable workaround
  • first, you need to install all deps (including devs)
  • the postinstall hook triggers, rebuilding native dependencies against, for example, electron - using electron-rebuild (a devDependency)
  • you can then prune, removing electron-rebuild and other devDependencies

I'd like to highlight that this isn't just "some dude's weird setup", it's the way most electron apps are packaged for production - this bug breaks electron-packager for all npm@5.3.0 users, for example: electron/packager#686

[sholladay]

happy that npm@5 ships something better than shrinkwrap for that

The package-lock and shrinkwrap are interchangeable (you can literally rename them to one another). Except, the former cannot be published, which makes it quite useless in my opinion as it is not included in the distributable tarball. I think what you are happy about, perhaps without realizing it, is the new lockfileVersion that includes subresource integrity and a few other niceties. The new format can (and should) be used with shrinkwraps. I imagine you are also happy that it is generated automatically at install time rather than via a separate command. But anyhow, you can still use my workaround without deleting package-lock.json if you want, it will still work. I included it because it is what I personally recommend for the common case of a full reinstall. Of course there are situations when that is not desirable.

if you have tools like electron-rebuild as devDependencies, it's not a viable workaround

That does look problematic. I don't know much about that tool, but I would expect there to be a mechanism to bypass the postinstall hooks. In fact, you can do this in a blunt force way with npm's --ignore-scripts. So perhaps rm -rf node_modules && npm install --production --ignore-scripts would work for you? Ideally you would be able to bypass only the postinstall script for that specific project, though (e.g. via an environment variable).

[fasterthanlime]

The package-lock and shrinkwrap are interchangeable

Right! shrinkwrap is dead, long live shrinkwrap. (I didn't realize it was a "backwards-compatible upgrade" rather than a different format, I'm not sure many npm@5 users do either!)

That does look problematic. I don't know much about that tool, but I would expect there to be a mechanism to bypass the postinstall hooks.

Ah but the thing is, we very much want to run the postinstall hooks.

Folks who develop electron apps have two versions of v8:

• whatever lives at $(which node) (probably 8.x at the time of this writing)
• whatever their target electron version ships (which actually contains two copies of v8 - one for the main process, which is actually patched node, and one for the renderer process, which is libchromiumcontent)

the two versions of v8 that electron ships are always synchronized, for compatibility reason (they interoperate quite a bit by default, and you can make them interoperate even more with the electron IPC api, remote requires, etc.)

However, there's a good chance that the version of v8 electron ships and the version of v8 used by $(which node) differ. When you install a node.js module with native code, npm builds it against your CLI node's v8 version. electron-rebuild rebuilds the native modules against (your target) electron's v8 version, so that you can actually require() them from your electron app.

I suppose a working workaround based on yours would be:

rm -rf node_modules
npm install --production --ignore-scripts
npm install --no-save electron-rebuild --ignore-scripts
node_modules/.bin/electron-rebuild
npm remove electron-rebuild --ignore-scripts

Although, it's starting to be a lot of trouble for what looks like a simple regression 😕

---

That's not to say that there isn't other, more complicated workarounds - some electron apps have a nested "app" folder that contains a second package.json - with only their production dependencies. I don't think it's realistic suggesting that as a temporary workaround for this npm@5.3.0 bug though, it's a pretty heavy change to make and comes with other problems challenges 😉

[sholladay]

For unrelated reasons, I just published fresh-cli, which does the reinstall trick I mentioned above. Sounds like it won't fix the electron-rebuild use case as-is, but maybe it will help someone.

[fasterthanlime]

Any news on this? It's starting to impact our users of our older versions.

[fasterthanlime]

Super easy reproduction steps ✨

mkdir prune-test && cd prune-test
npm init -y
npm install -D yes
npm prune --production

Output:

 prune-test ツ npm prune --production
npm WARN prune-test@1.0.0 No description
npm WARN prune-test@1.0.0 No repository field.

npm ERR! May not delete: C:\msys64\home\amwenger\Dev\prune-test\node_modules\.bin

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\amwenger\AppData\Roaming\npm-cache\_logs\2017-07-25T10_20_15_408Z-debug.log

[leoj3n]

I downgraded, was able to get past this error for one build, but now getting same error with 5.2.0 on second+ runs.

[fasterthanlime]

@leoj3n npm@5.4.0 is out, and fixes this particular bug - downgrading is no longer recommended.

If you do encounter problems with npm@5.4.0, please open separate issues - but first, make sure it's not a duplicate of #18277 for example.

[leoj3n]

@fasterthanlime Thanks, yes that works. I had npm installed as a local dependency as well as global, and it seems both were getting used at some point, so I had to update my project local npm dep as well as global dep (this is an app that requires npm to run npm commands programmatically).

[hostenvoy]

@fasterthanlime It works now with 5.4.0. Thanks for the info.

[imerkle]

wasted 1 day . npm -i -g npm@5.4 works

[Metrakit]

@dsslimshaddy Thanks you save my day !! The right cmd is npm i -g npm@5.4 😄

[ananthfrancis]

I fixed this issue by executing below command.
npm i -g npm@5.2 .

[robjtede]

@ananthfrancis this specific issue has been resolved in the latest npm release 5.4. Downgrading is no longer recommended.

You may still have issues with native add-ons though in which case you should follow discussion on #18277.

[wayofthefuture]

@robjtede Good luck getting npm 5.4 installed if you are using nvm-windows. Npm is non-upgradable, the perfect paradox. We are downgrading to 8.1 we have no choice.

coreybutler/nvm-windows#300

[gregbarcza]

With npm 5.4 I got the following error:

Error: Cannot find module 'npmlog'
    at Function.Module._resolveFilename (module.js:527:15)
    at Function.Module._load (module.js:476:23)
    at Module.require (module.js:568:17)
    at require (internal/module.js:11:18)
    at /usr/local/lib/node_modules/npm/bin/npm-cli.js:22:13
    at Object.<anonymous> (/usr/local/lib/node_modules/npm/bin/npm-cli.js:92:3)
    at Module._compile (module.js:624:30)
    at Object.Module._extensions..js (module.js:635:10)
    at Module.load (module.js:545:32)
    at tryModuleLoad (module.js:508:12)

[wgrhstf]

The Problem now occurs on npm 5.6.0 Before updating everything worked on 5.4 but now with 5.6.0 electron-packager breaks with

Error: Command failed: npm prune --production
npm ERR! path C:\Users\xxx\AppData\Local\Temp\electron-packager\linux-ia32\AppName-linux-ia32\resources\app\node_modules\npm\node_modules\dezalgo
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename 'C:\Users\xxx\AppData\Local\Temp\electron-packager\linux-ia32\AppName-linux-ia32\resources\app\node_modules\npm\node_modules\dezalgo' -> 'C:\Users\xxx\AppData\Local\Temp\electron-packager\linux-ia32\AppName-linux-ia32\resources\app\node_modules\npm\node_modules\.dezalgo.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

[BMalaichik]

Have the same issue running npm prune --production

• node 8.9.4
• npm@5.6.0

Refusing to delete ~/.../built/node_modules/.bin/tsc: is outside ~/.../built/node_modules/typescript