Migrate to yarn

[scinos]

This issue is used to keep track of the migration from npm to yarn.

Summary

We move wp-calypso repo from npm to yarn (work already started by @blowery and @jameslnewell in #39121).

The biggest risk with this transition is accidentally altering the dependency tree. This could introduce production bugs at the same time we are changing our tools, making everything extra harder to diagnose and fix.

To ensure a clean transition, we first need to have a clean npm dependency tree.

Preparation

These can be merged to master

• Develop a tool for computing the effective tree (https://www.npmjs.com/package/@automattic/effective-module-tree)
• Update @automattic/color-studio to 2.2.1 (Update @automattic/color-studio to 2.2.1 #40167)
• Adds missing dependencies (Adds missing dependencies #40165, Adds missing dependency 'cookie' #40057)
• Link all sub-packages in main package.json (Links all subpackages to the parent package.json #39955)
• Move devDependencies to dependencies (workaround for Failed to install dependencies in workspace: expected workspace package to exist yarnpkg/yarn#7807) (Moves existing devDependencies to dependencies #40880)
• Change scripts to ensure they are run with the correct package manager (assert npm is used if package-lock.json is present, yarn if yarn.lock is present) (Assert the correct package manager is used for run commands #40893)
• Make dependency of node-sass explicit (Make node-sass dependency explicit #41073)

Migration to yarn

To keep things easy to review, we'll have a long-run feature branch update/npm-to-yarn-main. All these changes will be merged into that branch.

• Change documentation from npm to yarn (Update docs to use yarn in examples #40916)
• Use yarn policy to set a version (Fixes yarn version to 1.22.4 using yarn version policies #40902)
• Change all scripts to use yarn commands (Replace npm commands with yarn equivalents #40921)
• Enable yarn workspaces (Migrate package-lock.json to yarn.lock #40923)

Once all the above have been reviewed and merged toupdate/npm-to-yarn-main, we'll do a final merge to master.

[scinos]

@nerrad @simison @nb @blowery @dmsnell @tyxla @sirbrillig @zinigor @sirreal @griffbrad @p-jackson @ChaosExAnima @brbrr @Stojdza @senadir @creativecoder

Hey folks! In the past, you have shown some interest in Calypso migrating to yarn. In case you are still interested, this is the issue where I'm capturing the state of the project. Please, feel free to check the open PRs and the overall plan and let me know if you have any feedback or directly comment on the relevant PRs.

[nb]

Thanks for all the work on that, @scinos (and before @blowery and @jameslnewell)! This is exciting, looking forward to eliminating a whole class of tricky problems.

A couple of (hopefully not too high-level) questions:

• What is our criteria for being ready to complete the migration? How do we know it's stable enough? Do we expect that the nom and yarn trees are the same?
• What are ways in which we can reduce the risk for production bugs once we deploy?

[scinos]

@nb There are two main criteria for the migration:

1. The changes in the tree must be contained, small, expected and explainable). We know npm can generate a tree that can't be represented in yarn with 100% fidelity, so there will be some changes.

2. Calypso bundles must not change significatively when comparing yarn vs npm. As above, there might be some minor changes, but they must be fully explainable and expected.

I think that if we can guarantee the above two points, we will be in a good spot and the migration won't introduce any new bug.

[nb]

Thanks, @scinos.

“Contained, small, expected and explainable” change to the tree makes perfect sense.

As for the bundles, I am curious to see whether we will be able to reason about the changes we see.

Do you think it's worth trying to test on a smaller audience? Like wordpress.com staging (seen by proxied Automatticians)? Or it will be too much effort?

[scinos]

For the record, the complete list of changes in the tree is listed in #40923 (comment). The changes in the bundles are listed in #40923 (comment)

At this point, I think we are pretty safe. The only bit that slightly concerns me is the duplication of acorn (see comment in the second link) because it adds 27kb to 3 async chunks. I assume those chunks are only loaded in some specific use cases, so most users won't be impacted. At any rate, deduplicating acorn to mitigate it should be pretty high in our priority list once yarn lands in master.

As for testing in a subset of users... in this specific case, I think it is too much effort. Keeping npm and yarn alive for a few days/weeks means we have to keep yarn in sync with every change in the dependency tree in npm. That process is partially manual, and therefore error prone. Which means we can introduce more risks, but specially, it means that the day we press the button to merge it "for good", we will potentially be merging a dep tree different than the tree we would have been testing in staging.

[scinos]

This project has been merged in master. The final list of dependency changes are listed in #41140 (comment).