Cannot install packages from GitHub when mounting directory (node:12-slim)

[scooper91]

When trying to install packages from GitHub using node:12-slim, we're seeing the following error:

npm ERR! Command failed: git clone --mirror -q git://github.com/mouse0270/bootstrap-notify.git /root/.npm/_cacache/tmp/git-clone-0decf327/.git                                                                    
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-0decf327/.git'

This first broke on the morning of the 5th October. It worked fine on the morning of the 4th October.

This only happens when we're mounting a directory - if we just mount the package.json, it works fine.

I have not tried other node images.

To re-create, create a package.json in an empty directory:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "bootstrap-notify": "mouse0270/bootstrap-notify#b8d0eb0"
  }
}

Create a Dockerfile alongside the package.json:

FROM node:12-slim
RUN apt-get update -qq && apt-get -qqy --no-install-recommends install git

From the directory, run:

docker build . -t test
docker run --rm -v $PWD:/blah -w /blah test npm i

That errors when trying to create leading directories:

➜ docker run --rm -v $PWD:/blah -w /blah test npm i
npm WARN test@1.0.0 No description
npm WARN test@1.0.0 No repository field.

npm ERR! code 128
npm ERR! Command failed: git clone --mirror -q git://github.com/mouse0270/bootstrap-notify.git /root/.npm/_cacache/tmp/git-clone-11c521e0/.git                                                                    
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-11c521e0/.git'
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-10-09T16_49_11_513Z-debug.log

The debug log file shows the following:

0 info it worked if it ends with ok                        
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'i' ]    
2 info using npm@6.11.3                                                      
3 info using node@v12.11.1                           
4 verbose npm-session a6eeaf780c678ebd                  
5 silly install runPreinstallTopLevelLifecycles
6 silly preinstall test@1.0.0    
7 info lifecycle test@1.0.0~preinstall: test@1.0.0            
8 silly install loadCurrentTree
9 silly install readLocalPackageData
10 timing stage:loadCurrentTree Completed in 5ms
11 silly install loadIdealTree                                                                                                                                                                                    
12 silly install cloneCurrentTreeToIdealTree
13 timing stage:loadIdealTree:cloneCurrentTree Completed in 0ms
14 silly install loadShrinkwrap
15 timing stage:loadIdealTree:loadShrinkwrap Completed in 1ms
16 silly install loadAllDepsIntoIdealTree
17 info lifecycle bootstrap-notify@3.1.7~prepack: bootstrap-notify@3.1.7
18 info lifecycle bootstrap-notify@3.1.7~postpack: bootstrap-notify@3.1.7
19 silly pacote git manifest for bootstrap-notify@github:mouse0270/bootstrap-notify#b8d0eb0 fetched in 1048ms
20 silly resolveWithNewModule bootstrap-notify@3.1.7 checking installable status
21 timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 1054ms
22 timing stage:loadIdealTree Completed in 1056ms
23 silly currentTree test@1.0.0
24 silly idealTree test@1.0.0
24 silly idealTree `-- bootstrap-notify@3.1.7
25 silly install generateActionsToTake
26 timing stage:generateActionsToTake Completed in 6ms
27 silly diffTrees action count 1
28 silly diffTrees add bootstrap-notify@3.1.7
29 silly decomposeActions action count 8
30 silly decomposeActions fetch bootstrap-notify@3.1.7
31 silly decomposeActions extract bootstrap-notify@3.1.7
32 silly decomposeActions preinstall bootstrap-notify@3.1.7
33 silly decomposeActions build bootstrap-notify@3.1.7
34 silly decomposeActions install bootstrap-notify@3.1.7
35 silly decomposeActions postinstall bootstrap-notify@3.1.7
36 silly decomposeActions finalize bootstrap-notify@3.1.7
37 silly decomposeActions refresh-package-json bootstrap-notify@3.1.7
38 silly install executeActions
39 silly doSerial global-install 8
40 verbose correctMkdir /root/.npm/_locks correctMkdir not in flight; initializing
41 verbose lock using /root/.npm/_locks/staging-89c05d72a2dc0b12.lock for /blah/node_modules/.staging
42 silly doParallel extract 1
43 silly extract bootstrap-notify@3.1.7
44 silly tarball no local data for bootstrap-notify@github:mouse0270/bootstrap-notify#b8d0eb0. Extracting by manifest.
45 timing audit submit Completed in 322ms
46 http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/audits/quick 321ms
47 timing audit body Completed in 1ms
48 timing action:extract Completed in 1024ms
49 verbose unlock done using /root/.npm/_locks/staging-89c05d72a2dc0b12.lock for /blah/node_modules/.staging
50 timing stage:rollbackFailedOptional Completed in 0ms
51 timing stage:runTopLevelLifecycles Completed in 2137ms
52 silly saveTree test@1.0.0
52 silly saveTree `-- bootstrap-notify@3.1.7
53 warn test@1.0.0 No description
54 warn test@1.0.0 No repository field.
55 verbose stack Error: Command failed: git clone --mirror -q git://github.com/mouse0270/bootstrap-notify.git /root/.npm/_cacache/tmp/git-clone-ee38da5f/.git
55 verbose stack fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-ee38da5f/.git'
55 verbose stack
55 verbose stack     at ChildProcess.exithandler (child_process.js:295:12)
55 verbose stack     at ChildProcess.emit (events.js:210:5)
55 verbose stack     at maybeClose (internal/child_process.js:1021:16)
55 verbose stack     at Socket.<anonymous> (internal/child_process.js:430:11)
55 verbose stack     at Socket.emit (events.js:210:5)
55 verbose stack     at Pipe.<anonymous> (net.js:658:12)
56 verbose cwd /blah
57 verbose Linux 5.0.0-29-generic
58 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "i"
59 verbose node v12.11.1
60 verbose npm  v6.11.3
61 error code 128
62 error Command failed: git clone --mirror -q git://github.com/mouse0270/bootstrap-notify.git /root/.npm/_cacache/tmp/git-clone-ee38da5f/.git
62 error fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-ee38da5f/.git'
63 verbose exit [ 1, true ]

When mounting just the package.json, it works fine, e.g:

➜ docker run --rm -v $PWD/package.json:/blah/package.json -w /blah test npm i
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN test@1.0.0 No description
npm WARN test@1.0.0 No repository field.

added 1 package from 1 contributor and audited 1 package in 1.949s
found 0 vulnerabilities

Let me know if you need any more information.

Thanks!

[LaurentGoderre]

Trying to do npm install as root in a mounted folder caused a lot of problems. Can you check what thw permissions of your $PWD are?

[scooper91]

Interesting - when I run npm i in the container as the node user, the package installs fine.

The files are owned by the node user (in the container) regardless of whether I run the container as root or node:

We haven't changed anything from our side, and we didn't used to have any issues. I am unsure what the root cause of the problem is, and why we're seeing this error (especially the difference between mounting a file vs directory).

Is running npm i as root something which we shouldn't be able to do?

[LaurentGoderre]

You can do it but it does lead to all sort of permission mismatch which makes debugging these types of issue very hard.

[LaurentGoderre]

I should also point out that it's especially problematic with mounted volumes from the host system.

[hcomnetworkers]

I've encountered the same problem when upgrading our docker-images from node 11 to 12.

First I tried 12.13.0-alpine, then 12.13.0-buster-slim and 12.11.1-buster-slim, all with the same error.

12.9.1-buster-slim finally worked again, and so did 12.9.1-alpine. I've not checked versions in between, but somewhere there must be the problem.

[dazwin]

I have what looks like the same issue which started sometime between 12/13 and 12/27 - using node+npm from deb.nodesource.com. It didn't matter which version of node I used (still using 8 for some builds), but it starts in npm 6.11.0 (6.10.3 works fine).

[MaxMotovilov]

Confirming the issue. Currently using this as a workaround (thanx to @hcomnetworkers !)

FROM node:12.9.1-alpine
....
RUN npm i
....
FROM node:12-alpine
....
COPY --from=0 ...

(assumes that all node-gyp output will be compatible between different minor versions of the node container)

[kinghrothgar]

I believe y'all are running into this issue npm/cli#624

A fix is in the latest release of npm https://github.com/npm/cli/releases/tag/v6.13.6

[pawel-smth]

I believe y'all are running into this issue npm/cli#624

A fix is in the latest release of npm https://github.com/npm/cli/releases/tag/v6.13.6

Installing npm that includes v6.13.6 solved the issue for me. Thank you!

Just in case - for those who are on alpinelinux-based containers (comes with official docker image of node-red):
as of today, container is built on alpinelinux v3.10.
the fix is available only in npm v12.14.1-r0 that is not in v3.10
To get required version, one need to append "@edge http://nl.alpinelinux.org/alpine/edge/main" to /etc/apk/repositories and then install "apk add npm@edge"

[ValterJunior]

Guys, I was having the same problem and maybe/probably because I am too dumb on docker I couldn't figure out how to make npm@edge version work on my docker image. Finally, I've noticed the ending sentence of @hcomnetworkers which made my day. ❤️

So, if anyone is struggling with that, simply downgrade the version of your Linux alpine image to 12.9.1-alpine as suggested above, as a workaround, and it should work at least until the image edge version changes are moved to the latest release.

[younglinuxer]

chown root:root -R xxx/bin/npm or chown root:root -R /xxxx