Pruning use case is slow for Heroku users

[jmorrell]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
We recently switched from installing only dependencies to a flow where devDependencies are also installed, the build steps run, and then the devDependencies are pruned out for users on Heroku.

The commands we use for Yarn are here: https://github.com/heroku/heroku-buildpack-nodejs/blob/master/lib/dependencies.sh#L88-L115

but the short version is that we install dependencies with this command:

yarn install --production=$production --frozen-lockfile --ignore-engines 2>&1

and then prune dependencies with this command, which is based on this issue

yarn install --frozen-lockfile --ignore-engines --ignore-scripts --prefer-offline 2>&1

My understanding is that we must ignore scripts in this case, because many of our users define their build step in a postinstall script, and this would be re-run.

However this additional flag also invalidates the integrity check, which causes a lot of extra work including rebuilds of native dependencies. We can see in our metrics that pruning with Yarn takes several times as long as pruning with npm:

If the current behavior is a bug, please provide the steps to reproduce.

@edmorley dug into this in this comment

Even when NODE_ENV=production for both steps, instead of a no-op, we get a rebuild.

STR:

1. git clone https://github.com/mozilla/treeherder && cd treeherder
2. export NODE_ENV=production
3. yarn install --production=true --frozen-lockfile --ignore-engines (this is what's run when YARN_PRODUCTION=true)
4. Repeat step 3 (to confirm it's now a no-op)
5. cp node_modules/.yarn-integrity ./.yarn-integrity-original
6. yarn install --frozen-lockfile --ignore-engines --ignore-scripts --prefer-offline (what's run for the prune)
7. diff -U3 .yarn-integrity-original node_modules/.yarn-integrity

Expected:

• Step 6 is a no-op like step 4.
• The yarn integrity file is identical at step 7.

Actual:

• Step 6 is not a no-op.
• Step 7 shows the integrity file has changed (which causes yarn to re-check dependencies):

--- .yarn-integrity-original
+++ node_modules/.yarn-integrity
@@ -4,6 +4,7 @@
     "node_modules"
   ],
   "flags": [
+    "ignoreScripts",
     "production"
   ],
   "linkedModules": [],

What is the expected behavior?
In the above repro, I would expect the second invocation to be a noop.

In the case where devDependencies must be removed I would expect Yarn to not re-build native dependencies.

Please mention your node.js, yarn and operating system version.

This is running on Heroku, which is a modern version of Ubuntu, currently either 14.04 (legacy) or 16.04, with a new version coming in a few months.

The Node version depends on what the customer specifies. Most of our users do not specify a version of Yarn and automatically get the latest version.

Likely related to #932

[jmorrell]

Of course, if I'm doing something wrong here I would appreciate being enlightened :)