shrinkwrap and optionalDependencies

[seanmonstar]

We have to shrinkwrap, since some of our dependencies are loose, and that's no good for production. Still, some of the dependencies are optional, depending on the OS.

If we include a shrinkwrap file, then npm doesn't even try to install optional dependencies. And if we shrinkwrap after the optional has been installed, it's no longer optional.

[jbergstroem]

Same applies to devDependencies. Kinda defeats the purpose of npm install --production as well.

[tadeuszwojcik]

Exactly, in current form npm-shrinkwrap is not optimal as using it installs dev dependencies when building deployment package what is far from optimal, also when hosting on azure or heroku shrinkwrapped dev deps are also being installed

[dmcinnes]

+1 -- we just ran into this problem. It loads dev dependencies in production which do not work.."precommit-hook" module for instance looks for the .git directory which doesn't exist and the whole thing fails.

We've had to stop using shrinkwrap altogether.

The different dependency groups should be separated in the shrinkwrap file so we can still install in production mode

[gabrielf]

The devDependencies issue seems to be solved, they are no longer included by default, but it creates another problem similar to the optional dependency problem described above.

If we don't include devDependencies in npm shrinkwrap by using npm shrinkwrap --dev then our build server cannot run the tests since the dev dependencies are not included. If we include the devDependencies in the shrinkwrap then the dev dependencies end up in production which we don't want.

Would this be solved by the npm-shrinkwrap file having the same sections as the package.json? Instead of:

{
"name": "...",
"version": "0.0.0",
"dependencies": {...}
}

It could mimic the package.json:

{
"name": "...",
"version": "0.0.0",
"dependencies": {...},
"devDependencies": {...}
}

Then it would be up to npm install and whether NODE_ENV is set to production if the devDependencies are actually installed or not.

But the use case for optional dependencies is perhaps more complex then this since it's not an all or nothing deal?

[walling]

We ran into this issue as well. We basically want to shrinkwrap the production modules, and leave devDependencies and optionalDependencies in a loose state. There are a few issues:

• npm shrinkwrap always locks optionalDependencies, if they are installed (no option to disable this behaviour).
• npm install --dev is impractical as it installs all devDependencies recursively. This takes forever and makes node_modules directory huge.
• npm install does not install devDepencies, if it wasn't locked before with npm shrinkwrap --dev (but we don't want to lock those).
• The normal behaviour of npm install is to install devDependencies in package.json, but not recursively. This behaviour is different with a shrinkwrap file.

I haven't tested with npm install --production. I expect that it should not install devDependencies, but I fear that it just installs everything in the shrinkwrap file.

Is anyone working on this? I think it's quite important, now that we are supporting multiple platforms (Linux, OS X, and Windows) for development and production. We don't want to add node_modules to the git repository. npm shrinkwrap could be a perfect solution, but this issue makes it more difficult than it should be.

[luk-]

No, I don't think anyone is working on it (to answer your question). It's
not a high priority issue. Having devDependencies on your server will not
cause you any problems.

On Wednesday, October 2, 2013, Bjarke Walling wrote:

We ran into this issue as well. We basically want to shrinkwrap the
production modules, and leave devDependencies and optionalDependencies in a
loose state. There are a few issues:

• npm shrinkwrap always locks optionalDependencies, if they are
  installed (no option to disable this behaviour).
• npm install --dev is impractical as it installs all devDependencies
  recursively. This takes forever and makes node_modules directory huge.
• npm install does not install devDepencies, if it wasn't locked
  before with npm shrinkwrap --dev (but we don't want to lock those).
• The normal behaviour of npm install is to install devDependencies in
  package.json, but not recursively. This behaviour is different with a
  shrinkwrap file.

I haven't tested with npm install --production. I expect that it should
not install devDependencies, but I fear that it just installs everything in
the shrinkwrap file.

Is anyone working on this? I think it's quite important, now that we are
supporting multiple platforms (Linux, OS X, and Windows) for development
and production. We don't want to add node_modules to the git repository. npm
shrinkwrap could be a perfect solution, but this issue makes it more
difficult than it should be.

—
Reply to this email directly or view it on GitHubhttps://github.com/isaacs/npm/issues/2679#issuecomment-25523878
.

[aseemk]

I believe some of this was implemented a while back:

• npm shrinkwrap ignores dev dependencies now by default (only at the top level though).
• npm install will install dev dependencies now by default after the shrinkwrap dependencies.

Try upgrading to the latest npm.

[IgorMinar]

the issue with optional dependencies still exists.

if one of my dependencies contains:

  "optionalDependencies": {
   "fsevents": "0.1.6",
   "recursive-readdir": "0.0.2"
  },

and I run npm shrinkwrap it produces npm-shrinkwrap.json that contains:

        "chokidar": {
          "version": "0.8.1",
          "from": "https://registry.npmjs.org/chokidar/-/chokidar-0.8.1.tgz",
          "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-0.8.1.tgz",
          "dependencies": {
            "fsevents": {
              "version": "0.1.6",
              "from": "fsevents@0.1.6",
              "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-0.1.6.tgz"
            },
            "recursive-readdir": {
              "version": "0.0.2",
              "from": "recursive-readdir@0.0.2",
              "resolved": "https://registry.npmjs.org/recursive-readdir/-/recursive-readdir-0.0.2.tgz"
            }
          }
        },

during the subsequent npm install these optional dependencies are attempted to be installed as normal dependencies, which breaks the install if the target platform is not supported by these dependencies (in my case fsevents is mac-only package, but after shrinkwrap it is being installed on linux as well where it fails)

[jumoel]

+1

This is still a problem.

[sarith]

+1

We just ran into this as well.
Any workarounds?

[othiym23]

This is now tagged onto the npm road map. I can't promise that this is something we'll get to immediately, but the npm CLI team does plan to address this within the medium term (6-12 months). It will probably require making some hard choices about how to handle this case in a robust way, and may end up being a semver-major change (in that shrinkwraps that fix this may end up not being interoperable with older versions of npm).

Also, this is at least partially a duplicate of #6801.

[jackwanders]

Hi @othiym23, my organization has also been suffering from this issue. I've read through the v3 source involved in installing and shrink-wrapping dependencies, and from a technical angle, it seems feasible to create a shrinkwrap process that can account for the installation failure of optional dependencies.

The two scenarios that seem to require resolution are:

1. Properly allowing a shrinkwrapped optional dependency to fail on unsupported platforms
2. Properly installing an optional dependency of a shrinkwrapped dependency that itself was not installed at the time of shrinkwrapping (i.e. the shrinkwrap occurred on the unsupported platform)

For the first scenario, would it be possible to simply check if the package's parent has it listed in its optionalDependencies, and if so, adding an optional failure rollback to its installSteps?

The second scenario seems a bit more complicated. It would seem that, during the shrinkwrap process, you would want to iterate through all optional dependencies, generating an optionalDependencies block within npm-shrinkwrap.json, including data for all such packages, regardless of whether they are currently installed. Essentially, we want to provide as much data as possible about what WOULD have been installed on the system that's doing the shrinkwrap, assuming the installation did not fail. This would both generate exact specs of non-installed optional dependencies and allow future installations to attempt their own installation of these dependencies, rolling back on failure.

I'm curious to know if there is anything glaringly infeasible about this approach. Given the impact this is having on my team, I'm invested in at least discovering if a resolution is possible, and contributing to the cause if it is.

[lxe]

The workaround I've been doing is:

1. Remove the entry of your optional dependency from your npm-shrinkwrap.json tree
2. Add it as an optionaDependencies entry in the package.json of your shrinkwrapped project.

Not ideal, but easy-ish to automate, and not too difficult to manage.

[mikemaccana]

Thanks @lxe I've been doing the same and it works.

[caseyhoward]

If this issue isn't going to get fixed, can we get something added to the caveats section of the documentation (https://docs.npmjs.com/cli/shrinkwrap#caveats) that npm shrinkwrap isn't meant to be used between different environments? The optionalDependencies documentation (https://docs.npmjs.com/files/package.json#optionaldependencies) should also be updated to say that it's not compatible with "npm shrinkwrap". It could save people a lot of time. I don't know how many this issue has bitten me.

[Jessidhia]

I really wish this wasn't just closed as "won't fix, the bug is now intended behavior".

Documenting it as a caveat is fine, but not that it is not meant to be used between different environments, just that it isn't currently supported.

At least I'm pretty sure the overwhelming majority of Mac developers don't use Mac servers for their production hosts. You would also have to get all your developers to run the same OS. Systems like Vagrant and Docker help, but requiring developing in a VM to be able to deal with shrinkwraps is not very nice.

[palmerj3]

@JamieMason apparently this makes no difference, i've learned.

[felixfbecker]

This should be a way higher priority to fix. yarn has shown that people need lockfiles, and npm's lockfile system (shrinkwrap) is obviously broken

[thom-nic]

FWIW, yarn has also had issues with optional dependencies:

• Errors from optionalDependency installation scripts should be ignored yarnpkg/yarn#1059 (marked closed but users are still reporting issues)
• [BUG] Yarn --ignore-optional flag is not working yarnpkg/yarn#705
• Yarn doesn't install non-optional sub-dependencies when --ignore-optional is used and one or more other packages mark it as optional yarnpkg/yarn#1233

For my part, I have had good luck with --production --no-optional using npm@4.2.0.

[ntwb]

Per the release of npm v5: http://blog.npmjs.org/post/161081169345/v500

"package locks no longer exclude optionalDependencies that failed to build. This means package-lock.json and npm-shrinkwrap.json should now be cross-platform. (#15900)"

Does npm v5 fix this optionalDependencies issues, it sounds like it to me 🤔

[ntwb]

Arghhhh... I just created a package-lock.json on Mac where fsevents was installed and included in the package-lock.json file, I then created a PR and Travis CI failed the build.

I immediately thought of this issue which I thought was resolved as part of npm 5 (per my own above comment FWIW), turns out it's not.

Once I manually removed fsevents from package-lock.json Travis CI was happy once more 😢

Here's the PR where this occurred: stylelint/stylelint#2670

I'm going to close that PR and add package-lock.json to our .gitignore until this issue is sorted once and for all.

[kyrios]

So package-lock.json/shrinkwrap is a dead concept?