[rush] Split ProjectChangeAnalyzer, fix build cache hashes #4476

dmichon-msft · 2024-01-09T22:33:58Z

Summary

Makes #3994 more feasible.

Details

Splits the functionality of ProjectChangeAnalyzer into two components:

ProjectChangeAnalyzer#_tryGetSnapshotAsync - Performs all the async I/O operations to determine the current state of the repository, including taking a snapshot of process.env so that per-operation hashes can be lazily computed in a stable manner.
InputSnapshot - Synchronous data structure representing the state of the repository, which can be queried for operation-level state hashes, or tracked file lists. Performs the evaluation of input/output collisions as part of evaluating said queries. Includes evaluation of incrementalBuildIgnoredGlobs and dependsOnEnvVars, but does not include the hashes from dependencies, since the Operation graph is not provided to this API.

Moves build cache hash computation to the createOperations tap in CacheableOperationPlugin and updates the computation to factor in the runtime operation graph, rather than only the npm dependencies.

For a specific example of how this alters the build cache, consider a repository with one project and two phases in rush build:
Project A
Phases _phase:first and _phase:second, with _phase:second depends on self: ['_phase:first']

Define a command line parameter --some-flag-for-first that is forwarded only to _phase:first.

Before this change, running rush build and rush build --some-flag-for-first would produce different cache IDs for A (_phase:first), but the same cache id for A (_phase:second).

With this change the cache IDs for A (_phase:second) are also impacted by the presence of the --some-flag-for-first, since it can change the output of A (_phase:first).

How it was tested

For the splitting, added unit tests for InputSnapshot and ProjectChangeAnalyzer#_tryGetSnapshotAsync.

For the build cache IDs, temporarily removed applicability of the --production CLI parameter from _phase:test and validated that running node ./apps/rush/lib/start-dev.js test and node ./apps/rush/lib/start-dev.js test --production still produced different cache keys for _phase:test.

For comparison, validated that the command-line.json alternation and existing rush test and rush test --production had a collision on the cache keys for _phase:test.

Impacted documentation

Any documentation about how build cache keys are computed.

libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts

D4N14L · 2024-01-09T22:56:04Z

libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts

+        additionalFilesFromRushProjectConfiguration.add(match);
+        additionalFilesForOperation.add(match);
+      } else {
+        const filePathForGit: string = path.posix.join(project.projectRelativeFolder, match);


This will be from the root of the rush.json folder. Should verify if that is the correct path root for the git hash for files method.

Any reason to not just concatenate the path parts together with a slash?

Yes. It is valid for match to start with ../.

libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts

libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts

libraries/rush-lib/src/logic/snapshots/InputSnapshot.ts

D4N14L

We should do a pre-publish to verify that everything that is changing here doesn't break our flow, since we're very likely the largest customer on this tool.

common/changes/@microsoft/rush/split-project-change-analyzer_2024-01-09-22-11.json

common/changes/@microsoft/rush/split-project-change-analyzer_2024-01-09-22-06.json

common/reviews/api/rush-lib.api.md

libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts

libraries/rush-lib/src/logic/buildCache/ProjectBuildCache.ts

libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts

libraries/rush-lib/src/logic/snapshots/InputSnapshot.ts

iclanton · 2024-01-18T21:49:14Z

libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts

+        additionalFilesFromRushProjectConfiguration.add(match);
+        additionalFilesForOperation.add(match);
+      } else {
+        const filePathForGit: string = path.posix.join(project.projectRelativeFolder, match);


Any reason to not just concatenate the path parts together with a slash?

dmichon-msft requested review from iclanton, octogonz, apostolisms and D4N14L as code owners January 9, 2024 22:33