npm prune equivalent behavior

[bouk]

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

Currently yarn doesn't seem to be removing any packages that aren't needed. With npm you can run npm prune, but yarn doesn't have that and doesn't do the same operation in other cases.

What is the expected behavior?

yarn init
npm install react
yarn clean
yarn install # I expect one of these two commands to remove node_modules/react

Please mention your node.js, yarn and operating system version.

Node v5.7.0
Yarn v0.15.1
macOS 10.12

[sebmck]

Thanks for the report! yarn install --force should do what you're looking for. When generating integrity hashes (what we use to check validity of the current install) we should also do an fs.readdir of all module folders.

[akshat-ilen]

yarn remove <package-name> removes the package... @kittens can we also add instruction for yarn prune for the user as we are adding the instruction for the yarn -i?

[torifat]

@kittens running yarn prune says:

error The prune command isn't necessary. `yarn install` will prune extraneous packages.

[jljorgenson18]

@torifat Yea that looks to be working, even when you cut out devDependencies by using --production

[thomaswmanion]

When I run yarn, then npm prune --production, then have to run yarn again, It doesn't reinstall the dev dependencies. Force seems like overkill. How else would you get around this?

[develar]

My expectation that yarn will not require additional and explicit prune or dedupe. And my node_modules will be always clean and small. Currently, removing dependency from package.json manually (e.g. @types/semver) and then yarn, yarn doesn't remove files. npm prune is still required :(

[rarkins]

On our CI platform, we currently do this:

npm install
tests
npm prune --production
deploy

i.e. after testing, but before deploying, we remove devDependencies. They get added back again at the start of the next build/test. Is there an equivalent way with yarn that doesn't bypass the cache or require npm?

[davglass]

I agree with @rarkins. Do y'all have an example of how you produce an app suitable for deployment that removes all of the devDependencies? But still allows for the lock file to make sure that all the deps are solid? This is the biggest issue that we face with npm's shrinkwrap. We want production deps to be locked but devDependencies to be free floating so that they are updated more easily as they tend to change more rapidly.

[erulabs]

yarn install --production is close, but it does a few things prune doesn't. It's a little annoying to have to build native modules twice in a CI pipeline just to prune. You can prevent rebuilding packages that already exist in cache with the --ignore-scripts flag. Additionally, prune doesn't check npm registry, so add --prefer-offline.

So, yarn install --production --ignore-scripts --prefer-offline == npm prune --production

[chetanddesai]

@erulabs what order would you run those things in?

$ yarn install
$ <run tests>
$ <remove node_modules?>
$ yarn install --production --ignore-scripts --prefer-offline
$ deploy?

I'm having trouble getting the equivalent to npm prune --production. The use-case is ripping out any build time node_modules so they don't get deployed out with the running service that only needs the runtime dependencies.

[erulabs]

@chetanddesai You don't need to wipe out the node_modules directory (yarn more or less does that already) - just:

$ yarn
$ <build phase and/or test phase>
$ yarn install --production --ignore-scripts --prefer-offline

Remember that if you're trying to do this stuff in a container - yarn keeps its cache outside the node_modules dir, so you'll need to yarn cache clean if you want to really cleanup everything in the environment.

[Diokuz]

yarn install --production

It is not very intuitive. I do not want to (re)install production packages, I want to remove dev packages. Is it hard to implement an alias yarn prune --production?

[abhiaiyer91]

Does Yarn not remove files when using --force. For the prune suggestions above, I went ahead and installed some extraneous dependencies. Then ran yarn install --force, everything was fine in both package.json and my yarn.lock, but unfortunately the files in node_modules were still present. This could potentially be an issue if you're using a mono-repo and your extraneous dependency is an incorrect parent repository dependency.

[sdbondi]

As @abhiaiyer91 says yarn install --production --ignore-scripts --prefer-offline still leaves the node_module packages. I'm just using npm prune --production --loglevel=warn

[pxwise]

Running this script gets me close to npm prune --production for docker builds.

const exec = require('child_process').exec;
const devDependencies = Object.keys(require('./package.json').devDependencies).join(' ');
const command = 'yarn remove ' + devDependencies;

const child = exec(command, (err, stdout, stderr) => {
  if (err) throw err;
  console.log(`stdout: \n${stdout}`);
  console.log(`stderr: \n${stderr}`);
});

[tech4him1]

For me yarn install --production --ignore-scripts --prefer-offline doesn't clean up all the dependencies. Is there a particular reason that you don't want to add a prune command, or do you just need someone to step up and add it?

[rockymadden]

@tech4him1 I found the same, try adding --force and it does clean them up on 0.22.0.

[arcanis]

I can't reproduce this bug. If you still experience it on the latest Yarn release, please update the script below to make it a viable reproduction, and reopen the issue (or ping me if you're not the OP).

#!/usr/bin/env bash

YARN=${YARN:-yarn}

TEMP="$(mktemp -d)"
trap 'rm -rf "$TEMP"' EXIT

echo $TEMP
cd "$TEMP"

printf '{ "name": "main", "version": "1.0.0", "license": "MIT", "dependencies": {} }' > package.json

$YARN add --dev babel-core
$YARN add left-pad

$YARN install
$YARN install --production

ls -l node_modules node_modules/*

[Diokuz]

Another issue here: some packages are private, so docker cannot install them. I want to copy all the stuff to docker context, then just remove devDeps. No reinstall – it fails.

[vladholubiev]

@Diokuz Docker can install private packages if you provide NPM_TOKEN environment variable in Dockerfile

[Diokuz]

@vladgolubev in my case it was not possible because our private npm server (not an npm.org) was physically unreachable from docker.

[poupryc]

No news here ?

[loopmode]

Running this script gets me close to npm prune --production for docker builds.

const exec = require('child_process').exec;
const devDependencies = Object.keys(require('./package.json').devDependencies).join(' ');
const command = 'yarn remove ' + devDependencies;

const child = exec(command, (err, stdout, stderr) => {
  if (err) throw err;
  console.log(`stdout: \n${stdout}`);
  console.log(`stderr: \n${stderr}`);
});

Of course this becomes more complicated when you use yarn workspaces.
One needs to get a list of package.json files e.g. via some sort of glob, then run the "delete devDependencies" trick over them.

Here's a helper for that. It does have a dependency on globby and needs a recent node because it uses async/await.
But chances are you already have it in your project, as many popular packages depend on globby (e.g create-react-app, lerna, eslint, webpack-dev-server - just try yarn why globby).
And since we're talking Docker cleanup, a recent Node.js version can be assumed.

https://gist.github.com/loopmode/318e881454dc0498874a4e764d3dce55

[b6pzeusbc54tvhw5jgpyw8pwz2x6gs]

My jq usage solution is,

$ yarn remove $(cat package.json | jq -r '.devDependencies | keys | join(" ")')

[lucaspiller]

For anyone still desiring this feature, all of the current workarounds require the yarn command to work online. In my case I have private packages from GitHub, and don't want the access credentials to be accessible in the final image.

With npm prune this can be done, as I can use a multi-stage Docker build. The first image has access to the credentials and installs dependencies and devDependencies (for compilation). Then in the second image which doesn't have access to the credentials, I can copy the node_modules folder from the first image and prune it.

None of the current solutions work with the --offline flag (which could be a separate bug, as the dependencies are up to date according to yarn install), so I can't use yarn for this.

[patarapolw]

My solution, using jq for Docker is

COPY package.json yarn.lock ./
RUN echo $(cat package.json | jq 'del(.devDependencies)') > package.json
RUN yarn --frozen-lockfile

[loopmode]

@patarapolw isn't that equivalent to yarn install --production --frozen-lockfile? What's the benefit?

[gaggle]

@patarapolw isn't that equivalent to yarn install --production --frozen-lockfile? What's the benefit?

I can't speak for @patarapolw but when I run yarn install --production --frozen-lockfile I get all the dev-dependencies as well. There may be some subtle differences but as far as I can conclude --prod does nothing 🤷

(This is for a workspaces enabled project if that makes a difference)

[sebmellen]

@gaggle running into the same issue now in a workspaces-enabled project. I've resorted to using this fix from @b6pzeusbc54tvhw5jgpyw8pwz2x6gs, which works.

Note, however, that I'm installing all the dependencies in a Docker image after using package-json-merge to create one concatenated fullpackage.json from the many package.json files in the submodules of our workspace. These submodules are all added in a /packages folder (structured as /packages/<NAME_OF_SUBMODULE>/).

So first, I have a script in my top-level package.json that uses package-json-merge and runs before the Docker container gets any build context: "merge-package.json-files": "package-json-merge ./packages/**/package*.json > fullpackage.json && sleep 5",.

Then in the Dockerfile, I:

1. Copy over the fullpackage.json as package.json (COPY fullpackage.json package.json), and then
2. Copy over the yarn.lock file: COPY yarn.lock yarn.lock, and then
3. Run the command linked above: RUN yarn remove $(cat package.json | jq -r '.devDependencies | keys | join(" ")'), and then I
4. Run the following: RUN yarn install --production --ignore-scripts --prefer-offline --force --frozen-lockfile.

I hope this helps anyone using Docker & workspaces together. It works but it's a little messy.

I'm interested to know if you have found a cleaner setup since posting your comment?

[merceyz]

The easiest way to have a setup like that would be Yarn 2 /w node_modules and this plugin https://gitlab.com/Larry1123/yarn-contrib/-/tree/master/packages/plugin-production-install.

https://yarnpkg.com/getting-started/migration

[aprilmintacpineda]

I think a separate script to prune dev dependencies should be supported by yarn. This is our use case:

• yarn which will install all dependencies
• Build javascript
• Upload build together with node_modules

It would be ideal if we can do:

• yarn -- which will install all dependencies
• Build javascript
• yarn prune --production -- remove dev dependencies
• Upload build together with node_modules

So that the files to be uploaded will be fewer taking less time to upload.

yarn --production is not applicable because it will not achieve the effect we want to get.

[TakeshiDaveau]

I aggre with @aprilmintacpineda for typescript project we can't use yarn install --production before it's transpiled, and after the transpilation we don't need typescript anymore

[benjamin-richardson-circular]

Running my Dockerfile like this seems to achieve the result I want. My node_modules drops from 190MB to 150MB and all Yarn has to do is prune rather than re-install everything.

FROM node:14-alpine

WORKDIR /usr/src/app

COPY package.json .
COPY yarn.lock .

RUN yarn install

COPY . .

RUN yarn build

RUN rm -rf app/

# Prune off the dev dependencies after build step
RUN yarn install --production

CMD [ "yarn", "start" ]

[vcarl]

I haven't been able to get yarn2 to respect the cache during my docker build, it reliably fetches if I don't manually specify cache preservation. And because the cache is in a separate folder, it's being included in the final build. Currently migrating away from yarn (much to my chagrin! I prefer it over npm!) because I can't argue with 15% smaller bundles, see reactiflux/reactibot#224

I spent about 8 hours trying to replicate npm prune with yarn, but I wasn't able to get the image size benefits no matter what combination I tried. And in any case, every solution here involved a more complex implementation and a deeper understanding of the tool — with npm install; npm prune I don't have to wonder what caches are involved or do any optimizing, it's just optimal without any futzing