[FEATURE] Do not remove node_modules on npm ci

[Zenuka]

What / Why

I would really like to see a flag like npm ci --keep to do an incremental update on our buildserver as this would speed up our deployments a lot. Like suggested before on github and on the community. The last update was on the 7th of October that this was being reviewed by the cli team. Could someone post an update on this? :-)

[DanielRuf]

This is not what ci / cleaninstall is meant to do. The current behaviour is correct. What you want to use is npm shrinkwrap.

[claudiahdz]

We added an update to avoid deleting the node_modules folder but not its contents (as originally requested on that post). The npm ci command purpose is to delete everything to start from a clean slate. If you want to keep your old node_modules what you need is npm i.

[Zenuka]

Thanks for your replies! Sorry for my late reply. I've looked at npm shrinkwrap but is this intended to run on our build server for continuous integration? When running this command it renames my package-lock.json to npm-shrinkwrap.json but what should I then run during CI? Just npm install to have an incremental update? Or should I run npm ci but that will delete all packages again :-( What I'm looking for is a command that does an incremental update but will install exactly what is in our package-lock.json

@claudiahdz; My understanding is that running npm install during CI will update the package-lock.json and that could mean that running the same build a couple of weeks later would install different packages. Is that incorrect?

P.s. I thought npm ci was short for Continuous Integration

[khalwat]

As referenced here: npm/npm#20104 (comment)

The current behavior is problematic if you're using npm ci inside of a Docker container (which is quite common for Continuous Integration) and you have a bind mount on node_modules

It causes the following error:

webpack_1   | npm ERR! path /var/www/project/docker-config/webpack-dev-devmode/node_modules
webpack_1   | npm ERR! code EBUSY
webpack_1   | npm ERR! errno -16
webpack_1   | npm ERR! syscall rmdir
webpack_1   | npm ERR! EBUSY: resource busy or locked, rmdir '/var/www/project/docker-config/webpack-dev-devmode/node_modules'

which then results in aborting the Docker container.

It'd be lovely to have a --no-delete flag or if npm ci could delete the contents of node_modules but not the directory itself.

[DanielRuf]

ci = clean install

This is expected. Why don't you use the normal npm i with a lockfile?

[DanielRuf]

It'd be lovely to have a --no-delete flag or if npm ci could delete the contents of node_modules but not the directory itself.

rm -rf node_modules/* && npm i

[khalwat]

ci = clean install

This is expected. Why don't you use the normal npm i with a lockfile?

...because: https://docs.npmjs.com/cli/ci.html

This command is similar to npm-install, except it’s meant to be used in automated environments such as test platforms, continuous integration, and deployment – or any situation where you want to make sure you’re doing a clean install of your dependencies. It can be significantly faster than a regular npm install by skipping certain user-oriented features. It is also more strict than a regular install, which can help catch errors or inconsistencies caused by the incrementally-installed local environments of most npm users.

The faster installs and clean slate approach make this ideal for CI environments such as the one I mentioned above.

[khalwat]

rm -rf node_modules/* && npm i

This is what I do now, but see above for the desire to use npm ci

[ljharb]

It seems reasonable to me to file an RFC asking for a config flag that makes npm ci remove the contents of node_modules and not the dir itself. This is also an issue for me, in that i've set up Dropbox to selectively ignore a node_modules dir, but if i delete it, that selective setting goes away, and the next time node_modules is created, it syncs.

[DanielRuf]

It seems reasonable to me to file an RFC asking for a config flag that makes npm ci remove the contents of node_modules and not the dir itself. This is also an issue for me, in that i've set up Dropbox to selectively ignore a node_modules dir, but if i delete it, that selective setting goes away, and the next time node_modules is created, it syncs.

Isn't this also what another issue described, to allow npm to create files to ignore the dir (for OSX spotlight and others)? I think there were also others who need this feature.

[Zenuka]

ci = clean install

This is expected. Why don't you use the normal npm i with a lockfile?

npm i would be a great but only if it wouldn't change the lock file. I've seen the package-lock.json been updated during an npm i or should that not happen?

[DefiCake]

I support this feature. As stated, npm i modifies package-lock.json. A flag would be the ideal solution.

[xenoterracide]

same, a flag would be great

[DanielRuf]

I support this feature. As stated, npm i modifies package-lock.json. A flag would be the ideal solution.

Why not add a flag for npm i then? Because this would make not much sense for ci = clean install in my sense.

[khalwat]

What part of "clean install" is incompatible with keeping the parent node_modules/ directory intact (while doing a clean install of the actual contents)?

I realize that CI doesn't stand for Continuous Integration in this case; but a Clean Install is often quite useful in a Continuous Integration environment, as the documentation makes clear.

This command is similar to npm-install, except it’s meant to be used in automated environments such as test platforms, continuous integration, and deployment – or any situation where you want to make sure you’re doing a clean install of your dependencies. It can be significantly faster than a regular npm install by skipping certain user-oriented features. It is also more strict than a regular install, which can help catch errors or inconsistencies caused by the incrementally-installed local environments of most npm users.

npm ci is specifically mean to be used in automated environments, many times this means a Docker-based setup.

The behavior of deleting the node_module/ directory is troublesome in a Docker-based setup, for the reasons mention in this thread.

So we're asking for an option that will make this command useful for its intended purpose and environment.

[xenoterracide]

I support this feature. As stated, npm i modifies package-lock.json. A flag would be the ideal solution.

Why not add a flag for npm i then? Because this would make not much sense for ci = clean install in my sense.

I have to ask this question are their any other differences between npm install and npm ci if not then why aren't both options available in npm install maybe ci needs to become some alias like npm install --no-update-package-lock --clean-node-modules

[DanielRuf]

The behavior of deleting the node_module/ directory is troublesome in a Docker-based setup, for the reasons mention in this thread.

In my opinion this should only happen once when the image is built. After that npm i should be used during development.

[DanielRuf]

maybe ci needs to become some alias like npm install --no-update-package-lock --clean-node-modules

Personally that makes more sense to me, additional flags for the normal npm i command.

[xenoterracide]

I'm indifferent to which, and honestly too much of a n00b with js land to have a concrete argument that it must be ci, all I know is that it should not update the package-lock.json and should not remove node_modules

[DanielRuf]

npm cidoes not update the lockfile, it installs from the lockfile. This was introduced to do a clean install because prior this people were advised to rm -rf node_modulesand run npm i again. And afaik people wanted that it does not change the lockfile but installs from it.

So npm ci was born. And it also skips some things like the list of the installed packages and the tree and a few more things.

See https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable

It covers a specific use case.

For other use cases we should add new flags to npm iwith which we can also emulate npm ciwhich is a more flexible and better solution than flags for npm ci which should still cover only the current use case imho. What users request here is a bit similar to yarn install --frozen-lockfile or yarn --frozen-lockfile.

[DanielRuf]

Otherwise flags are spread over npm ci, npm iand so on which makes it a bit more difficult (documentation, code, ...). At least this is what I think. Let's put it to npm i t have more powerful and flexible ways to configure its behavior.

[Zenuka]

For other use cases we should add new flags to npm iwith which we can also emulate npm ciwhich is a more flexible and better solution than flags for npm ci which should still cover only the current use case imho. What users request here is a bit similar to yarn install --frozen-lockfile or yarn --frozen-lockfile.

I'd be very happy if the feature was added to npm i. Should I update the original post?

[ilan-schemoul]

npm cidoes not update the lockfile, it installs from the lockfile. This was introduced to do a clean install because prior this people were advised to rm -rf node_modulesand run npm i again. And afaik people wanted that it does not change the lockfile but installs from it.

So npm ci was born. And it also skips some things like the list of the installed packages and the tree and a few more things.

See https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable

It covers a specific use case.

For other use cases we should add new flags to npm iwith which we can also emulate npm ciwhich is a more flexible and better solution than flags for npm ci which should still cover only the current use case imho. What users request here is a bit similar to yarn install --frozen-lockfile or yarn --frozen-lockfile.

How is rm -rf node_modules/* not qualifying as "cleaning" ? The feature asked here is very similar to the one present in npm ci. In my opinion it makes more sense to add a flag to npm ci so it uses rm -rf node_modules/* instead of rm -rf node_modules instead of importing the entire behavior of npm ci into npm i.

BTW this issue should get more attention and maintainers should voice their opinions and plans about that, using docker is basically always used in CI (continuous integration) which is one of the main use case of npm ci !

[ljharb]

Please open an RFC for this change, rather than an issue in this repo.

[ljharb]

To avoid confusion, I’d rename this issue as “empty instead of remove node_modules dir in npm CI”

[Zenuka]

My intention of this issue was never to delete the node_modules folder or only it's contents. It was always to preserve the contents of node_modules but make sure it's up to date and in sync with package-lock.json. So an incremental update which adheres to the package-lock.json.

Maybe I'm wrong but I feel there are two issues here. Maybe someone could start another issue or RFC about deleting only contents of node_modules instead of deleting the folder completely? Or am I missing something?

[ljharb]

@Zenuka the entire reason npm CI is fast, and exists, is because it ignores the existing node_modules dir, so it’s pretty unlikely that will change.

[DanielRuf]

@nmm-shumway you can manually upgrade npm and this is also recommended in most cases: npm i -g npm@latest

[nmm-shumway]

Yeah, honestly that's probably what I'll recommend doing at a work. We would always have needed to update our internal npm version for any patch anyway, so there's really no reason for us not to be using v7 and migrating off of lockfile version 1 already.

This advice does raise a question though: if it's recommended to use v7 with the current LTS version of Node, is there a reason why the website and nvm still ships the LTS with v6? I guess it's a different conversation to have with different people on a different issue tracker, but would it be possible to get those updated so they ship with the correct dependency out of the box?

It still feels odd to me that the default version of npm that people are downloading won't have correct behavior. Or at least I assume it's a bug and not intended behavior. Just purely for clarification -- in the above scenario I bring up, am I correct in assuming that the v7 result is the intended behavior, not the v6 result?

[ljharb]

@nmm-shumway v6 to v7 is a breaking change, so a node major that ships v6 will always do so. LTS status has nothing to do with it.

[nmm-shumway]

Should this bugfix be backported to v6 then? I still don't understand whether v6 ignoring the package-lock.json when new versions of libraries get released is expected behavior or not.

[ljharb]

@nmm-shumway id hope so, but my guess is that it’s unlikely since v7 is completely rewritten. However users are expected to upgrade npm to the latest that works on their node version anyways, so if you’re on node 10+ you should be on npm 7+.

[nmm-shumway]

Meh, this is fine. My org can upgrade the npm version we're using, so this doesn't really matter to me beyond curiosity.

It seems a little bit weird to me that users are expected to upgrade across versions in an LTS if those versions have breaking changes, and if the breaking changes aren't a problem, it seems a little weird to me that they're being treated as a blocking issue preventing shipping inside the LTS distributions. But, again, I don't actually need a resolution on that, from what I can see in my own testing, v7 fixes the majority of the issues I'm having, so at least for my org the answer is definitely just to upgrade.

I appreciate the responses, swapping to v7 should make our dev/testing builds a bit more reliable, so I'm pretty happy about that.