npm 2 can't install optional dependencies in shrinkwrap generated with npm 3

[djanowski]

Steps to reproduce:

• Upgrade to npm 3 locally (Darwin).
• Install a package that has a platform-specific optional dependency (eg Babel depends on Chokidar depends on fsevents).
• npm shrinkwrap
• Commit and deploy to production (Linux).
• npm 2.12 fails because of platform mismatch.

remote:        Installing node modules (package.json + shrinkwrap)
remote:        npm ERR! Linux 3.13.0-49-generic
remote:        npm ERR! argv "node" "/tmp/build_86a7a9012a4974c00a15a52510209bfe/.heroku/node/bin/npm" "install" "--unsafe-perm" "--userconfig" "/tmp/build_86a7a9012a4974c00a15a52510209bfe/.npmrc"
remote:        npm ERR! node v0.12.5
remote:        npm ERR! npm  v2.12.0
remote:        npm ERR! code EBADPLATFORM
remote:        
remote:        npm ERR! notsup Unsupported
remote:        npm ERR! notsup Not compatible with your operating system or architecture: fsevents@0.3.6
remote:        npm ERR! notsup Valid OS:    darwin
remote:        npm ERR! notsup Valid Arch:  any
remote:        npm ERR! notsup Actual OS:   linux
remote:        npm ERR! notsup Actual Arch: x64

[iarna]

@djanowski: Isn't this the case with a shrinkwrap from the same project in npm@2?

[iarna]

That is, does it actually matter what version you created the shrinkwrap with? My impression was that npm@2 would shrinkwrap optional deps if they installed ok, and so would create this same scenario.

[thefotios]

I hit this same bug and wanted to expand upon it. The same thing will happen when you run the entire process in Linux for that same package. It appears that optional dependencies (even ones that don't get installed) are hoisted into the shrinkwrap file.

$ npm install -g npm@3
$ npm install --save gulp-watch    
> fsevents@0.3.6 install /home/flindiakos/tmp/npm_test/node_modules/fsevents
> node-gyp rebuild

make: Entering directory '/home/flindiakos/tmp/npm_test/node_modules/fsevents/build'
  SOLINK_MODULE(target) Release/obj.target/.node
  COPY Release/.node
…

Build succeeds. There is no indication that the optional dependency did not install. There's even a node_modules/fsevents folder.

$ npm shrinkwrap
wrote npm-shrinkwrap.json

Shinkwrap file now has fsevents even though it's an optional dep that can't be installed (the package only applies to OSX).

{
    “dependencies”: {
        "fsevents": {
            "version": "0.3.6",
            "from": "fsevents@>=0.3.1 <0.4.0",
            "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-0.3.6.tgz"
        }
    }
}

Now that it's installed, switch back to npm@2.

$ npm install -g npm@latest
$ npm version
{ npm_test: '1.0.0',
  npm: '2.13.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }```

$ rm -rf node_modules
$ npm install
npm ERR! Linux 4.0.7-200.fc21.x86_64
npm ERR! argv "node" "/home/flindiakos/.n/bin/npm" "install"
npm ERR! node v0.12.6
npm ERR! npm  v2.13.1
npm ERR! code EBADPLATFORM

npm ERR! notsup Unsupported
npm ERR! notsup Not compatible with your operating system or architecture: fsevents@0.3.6
npm ERR! notsup Valid OS:    darwin
npm ERR! notsup Valid Arch:  any
npm ERR! notsup Actual OS:   linux
npm ERR! notsup Actual Arch: x64

[iarna]

I'm 90% sure that this is actually caused by this bug:

#8921

Where npm@3 is not checking the os & arch

[mblakele]

I'm seeing a similar EBADPLATFORM with fsevents on linux, when using npm v2.13.4 with shrinkwrap generated by npm v3.2.2.

npm ERR! Linux 3.13.0-51-generic
npm ERR! argv "/home/teamcity/.nvm/versions/io.js/v2.3.3/bin/iojs" "/home/teamcity/.nvm/versions/io.js/v2.3.3/bin/npm" "install"
npm ERR! node v2.3.3
npm ERR! npm  v2.13.4
npm ERR! code EBADPLATFORM

npm ERR! notsup Unsupported
npm ERR! notsup Not compatible with your operating system or architecture: fsevents@0.3.8
npm ERR! notsup Valid OS:    darwin
npm ERR! notsup Valid Arch:  any
npm ERR! notsup Actual OS:   linux
npm ERR! notsup Actual Arch: x64

The shrinkwrap block generated by npm3 for fsevents is:

    "fsevents": {
      "version": "0.3.8",
      "from": "fsevents@>=0.3.1 <0.4.0",
      "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-0.3.8.tgz"
    },

But npm 2.x doesn't seem to like that.

[othiym23]

@mblakele This appears to be an area where the shrinkwrap file doesn't translate well across platforms. I'm pretty sure that this bug exists in npm@2 as well as @iarna indicates above: you're generating the shrinkwrap on darwin, but trying to install on linux. I'm not sure what the ideal behavior here is (aside from not, you know, blowing up, like it does now), but I'm also pretty sure that this isn't a new problem with npm@3.

[mblakele]

I agree: it seems like an npm2 bug, at least if npm2 is meant to handle shrinkwrap generated by npm3. That's what I'd like it to handle, so that I can use 3 in dev and 2 in CI and prod.

[othiym23]

I think it's entirely possible to reproduce this failure using only npm@2, both to generate and consume the shrinkwrap. The issue is moving between OSes with a shrinkwrap plus optional dependencies, not the npm version.

[iarna]

Perhaps there should be some way to exclude optional deps from a shrinkwrap– npm would still install them when they're available, but they would stop exploding things when they aren't.

[iarna]

So I put together a minimal test case for this and the shrinkwrap files produced by npm@2 and npm@3 are byte-for-byte identical. Obviously this wouldn't be the case for real projects where flattening will produce different trees, but it does show that the behavior around optional deps and shrinkwraps is the same in both versions.

Folks who are seeing this, can you give me a real world example of a package.json for your failure case?

[mblakele]

@iarna I found a little time to create a single-package test case using chokidar.

• package.json: https://gist.github.com/mblakele/0d0cd3147b82892bb0a3
• npm-shrinkwrap.json @2.13.5: https://gist.github.com/mblakele/e4331ec19bf85fef6133
• npm-shrinkwrap.json @3.0.0: https://gist.github.com/mblakele/0837ef66daeae99fc4e4

Failure:

$ npm -v
2.11.3
$ npm install
npm ERR! Linux 3.13.0-53-generic
npm ERR! argv "/home/ubuntu/.nvm/versions/io.js/v2.3.3/bin/iojs" "/home/ubuntu/.nvm/versions/io.js/v2.3.3/bin/npm" "install"
npm ERR! node v2.3.3
npm ERR! npm  v2.11.3
npm ERR! code EBADPLATFORM

npm ERR! notsup Unsupported
npm ERR! notsup Not compatible with your operating system or architecture: fsevents@0.3.8
npm ERR! notsup Valid OS:    darwin
npm ERR! notsup Valid Arch:  any
npm ERR! notsup Actual OS:   linux
npm ERR! notsup Actual Arch: x64

npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/npm3-npm2-compat-test/npm-debug.log

debug log: https://gist.github.com/mblakele/a5877913ca944c8c4601

[LeeDr]

Any suggestions for a work-around when someone (a newbie) is having this problem? Can I do some clean-up and resolve the issue?
I'm on Windows 10;
$ npm --version
2.11.3
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! code EBADPLATFORM

npm ERR! notsup Unsupported
npm ERR! notsup Not compatible with your operating system or architecture: fsevents@0.3.8
npm ERR! notsup Valid OS: darwin
npm ERR! notsup Valid Arch: any
npm ERR! notsup Actual OS: win32
npm ERR! notsup Actual Arch: x64

Thanks!

[paraplexed]

Upgrading to npm 3.3.5 fixed this for me. (likely fixed in #8921 like iarna suggested)

[mblakele]

My test case https://gist.github.com/mblakele/0d0cd3147b82892bb0a3 still fails when installing on linux with npm@2.14.6 using shrinkwrap generated by osx with npm@3.3.6.

[cletusw]

See also #2679

[iarna]

Thank you for digging that issue number up @cletusw. Closing this as a duplicate of #2679