DevEx: use yarn instead of npm for installing dependencies

[samouri]

Use Yarn

We've considered switching to yarn in the past, but held off due to major improvements coming to npm. npm 5 is here, and was undoubtably much better than npm 3 and 4. So the question has become: is npm5 better than yarn.

Here are the features yarn supports that make it a no-brainer for me:

• yarn can work offline with the command yarn install --offline. this is huge
• We never need to manually update our shrinkwrap. As of now, updating the shrinkwrap requires deleting and reinstalling node_modules which is slow and forces us to halt working while it runs. yarn handles its lockfile automatically
• yarn install runs faster than npm install for the most common cases
• yarn will recognize if nothing in package.json has changed, and can early-return out of the install.

performance comparison of yarn install vs. npm install

	yarn	npm
no cache, no node_modules	114s	65s
yes cache, no node_modules	25s	31s
yes cache, yes node_modules	1s	23s
update lockfile	n/a	60s

To test

# install yarn
npm install -g yarn

# start calypso. npm start still technically works
yarn start

Note: fixes #23348

[matticbot]

Test live: https://calypso.live/?branch=try/use-yarn

[gwwar]

As a general note yarn does not recommend installing from npm.

Note: Installation of Yarn via npm is generally not recommended. When installing Yarn with Node-based package managers, the package is not signed, and the only integrity check performed is a basic SHA1 hash, which is a security risk when installing system-wide apps. For these reasons, it is highly recommended that you install Yarn through the installation method best suited to your operating system.

https://yarnpkg.com/lang/en/docs/install/

Likely for OSX

brew install yarn --without-node

Or for anyone with apt-get:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install --no-install-recommends yarn

[gwwar]

Did we also end up testing against npm ci ? That's likely to be available to Node LTS soon.

[gwwar]

Took a quick look at the bug list, and I think yarn looks stable enough for prod 👍. Only questionable thing we might want to verify is yarnpkg/yarn#3202

npm is steadily improving but if enough folks would like this in for the perf boost, I would also recommend that we outline our recommended procedures for standard operations, like installing yarn, keeping yarn up to date, installing/updating packages, etc and add a readme.

cc @Automattic/jetpack if folks had any gotchas/insights

[samouri]

Did we also end up testing against npm ci ? That's likely to be available to Node LTS soon.

I personally couldn't get the new version of npm to work locally. It kept crashing

[samouri]

To do:

• yarn engine to make it so that everyone is on the same version of yarn
• restrict it so that npm install fails with an error to use yarn
• replace any documentation we have about how to use npm with equivalent yarn instructions

[sirreal]

Let's tie in this relevant discussion "Exploration: Drop yarn in favour of npm": Automattic/jetpack#8718 (thanks @simison)

[simison]

Offline support

Both yarn and npm5 have offline support.

npm i --offline (without ./node_modules) fails on Calypso due using direct git-dependencies instead of using a registry. Works great if you already have ./node_modules though.

Lockfile

Since Calypso is using npm shrinkwrap (updated manually), comparisons would be fairer if we'd assume a move to npm lockfile (updated automatically). See #23566 (comment)
More about these two: https://docs.npmjs.com/files/package-locks

Here is a nice read about differences between yarn.lock and npm package-lock.json: https://yarnpkg.com/blog/2017/05/31/determinism/

In short:

• devex is the same: both get updated when adding/removing/updating modules

• yarn lockfile is more compact, less likely causing merge conflicts. It relies on lockfile+package.json

  → I don't have much experience with yarn lockfiles, but I sure remember trying to solve some nasty npm-lockfile merge conflicts. Might be this can be avoided by using exact version requirements in package.json (i.e. avoiding ~/^). Seems like npm 5.7 has a helper tool to resolve these, too.

• npm lockfile is more likely guaranteeing you expected versions of installs. It doesn't rely on package.json — though reading about "npm ci" makes me think I've understood something wrong.

  → Again using strict version requirements in package.json might make yarn lockfile behave the same?

Speed

There used to be a huge difference in speed, but it seems like the differences are getting smaller.

@Viper007Bond has a nice speed comparison script you can run: https://github.com/Viper007Bond/npm-yarn-benchmark

Noting that NPM shrinkwrap and npm lockfile are equal concerning output so speed comparisons using Calypso should still be valid.

Getting started -devex

Arguably, yarn is one more dependency to install, update and learn to use. Consider that installing it might not be very straightforward if you just want to contribute or test Calypso quickly.

You could also keep using npm commands and use yarn as a dependency under the hood. ;-)

---

Aight, hope this info helps to decide! :-)

[DanReyLop]

Since Calypso is using npm shrinkwrap (updated manually), comparisons would be fairer if we'd assume a move to npm lockfile (updated automatically).

I'd love to see this misconception die eventually. package-lock.json is exactly the same as npm-shrinkwrap.json. Just because it's "the new thing", it doesn't make it better. See the first section of this document. The only difference between the 2 of them is that package-lock.json is supposed to be packaged when you upload your project to npm and npm-shrinkwrap.json is not (maybe I'm paraphrasing). In short, shrinkwrap is for apps, package-lock is for libraries, and wp-calypso is an app. Edit: I had them switched up a bit. This is a more comprehensive explanation.

In theory, when you have a shrinkwrap (or package-lock) and do npm install foo --save (I think the --save flag is the default now, but I digress), it will add foo to the package.json's dependencies and autonatically update the shrinkwrap/package-lock. Same with any other command that changes package.json, such as npm remove foo. The reasons why we have that convoluted npm run update-deps command are:

• That doesn't always work. Specially with older versions of npm, npm install foo --save would install the package, update package.json, but fail regenerating the shrinkwrap. Then, the only options would be to manually edit the shrinkwrap, or nuking node_modules and starting again from scratch.
• Updating transitive dependencies. Suppose we depend on foo 1.2.3, and foo depends on bar 1.0.x. When we generated the shrinkwrap, the latest version of bar available was 1.0.0. A day later, bar 1.0.1 is released with a critical fix. Running npm run update-deps, since it reinstalls everything from scratch, will pick all the latest versions of the sub-dependencies that match the semver patterns, so it would pick bar 1.0.1.

All that said, I don't know if we still have to nuke the shrinkwrap every time we install a package, I thought this had improved in npm 5. If yarn improves that, I'd say that's reason enough to switch.

[simison]

Ah, thanks @DanReyLop! I stand corrected:

Both npm-shrinkwrap and package-lock indeed get updated for operations modifying ./node_modules.

[gwwar]

The only difference between the 2 of them is that package-lock.json is supposed to be packaged when you upload your project to npm and npm-shrinkwrap.json

Yeah, almost! package-lock.json is never published, but a shrinkwrap will be. Libraries tend to use the package-lock.json to not force consumers into using exact dependencies, whereas a shrinkwrap is desirable for applications for its repeatability.

See docs: https://docs.npmjs.com/files/package-lock.json

One key detail about package-lock.json is that it cannot be published, and it will be ignored if found in any place other than the toplevel package.

[gwwar]

All that said, I don't know if we still have to nuke the shrinkwrap every time we install a package

We don't need to if the npm install save works, but it does tend to fail a decent amount for us in wp-calypso. Folks also tend not to notice/understand the error message so it was safer to instruct folks to always use the our update-deps script.

[samouri]

whereas a shrinkwrap is desirable for applications for repeatibility.

I don't think I understand that point. How could it be desirable for applications considering the difference only matters once published to npm? Applications don't get published to npm, they get deployed to servers

[gwwar]

Applications don't get published to npm

We don't with wp-calypso but other folks might. Imagine, instead if our docker file did a npm install wp-calypso@x.x.x instead of a clone from master.

The recommended use-case for npm-shrinkwrap.json is applications deployed through the publishing process on the registry: for example, daemons and command-line tools intended as global installs or devDependencies. It's strongly discouraged for library authors to publish this file, since that would prevent end users from having control over transitive dependency updates.

https://docs.npmjs.com/files/shrinkwrap.json

This doesn't mean we can't move to a lockfile though :) since we don't do this. It just used to be that we could only lockdown deps previously with a shrinkwrap in older versions of npm.

[DanReyLop]

One key detail about package-lock.json is that it cannot be published, and it will be ignored if found in any place other than the toplevel package.

Ok I remember now. Since we use wp-calypso as a dependency in woocommerce-services, we prefer that we keep npm-shrinkwrap.json so all the wp-calypso's transitive dependencies will be locked down. If Calypso moved to package-lock.json, woocommere-services would have it a bit harder to "pin" versions of the wp-calypso dependency, since both the wp-calypso commit SHA and the moment you changed it would influence the final dependency tree. It's not a major thing and it only affects a small team, but since there's absolutely no other difference between the 2 files that applies to wp-calypso, please keep the shrinkwrap if you stay with npm :)

[samouri]

@DanReyLop: if woocommerce-services is using Calypso as a dependency and adhering to its shrinkwrap, doesn't that introduce bloat into it's node_modules?

[DanReyLop]

if woocommerce-services is using Calypso as a dependency and adhering to its shrinkwrap, doesn't that introduce bloat into it's node_modules?

It does, but we live with it. woocommerce-services/node_modules is basically 90% Calypso, 10% everything else, but everything we aren't using isn't included in the final JS bundle so we're OK with that. Also, now our extension is "hybrid", with most of the code living in wp-calypso/extensions/woocommerce/woocommerce-services, so it's too late to change it to dops-components (which is deprecated anyways) or splitting parts of Calypso using lerna.

[samouri]

closing this due to a lack interest