Required packages are missing when optional=false

[TravisJRyan]

Verify latest release

• I verified that the issue exists in the latest pnpm release

pnpm version

9.1.0

Which area(s) of pnpm are affected? (leave empty if unsure)

Dependencies resolver

Link to the code that reproduces this issue or a replay of the bug

No response

Reproduction steps

Use below package.json:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@babel/cli": "^7.16.0",
    "del": "^6.0.0"
  }
}

Then run pnpm i --no-optional. You'll see that fill-range is not in node_modules/.pnpm. But if you run the same install again, the package is present.

In the above example, babel-cli has an optional dependency on chokidar (ref) which requires braces whose dependency is fill-range (ref). This seems to be why fill-range is not installed. But braces should be present anyway based on the below:

dependencies:
del 6.1.1
└─┬ globby 11.1.0
  └─┬ fast-glob 3.3.2
    └─┬ micromatch 4.0.5
      └── braces 3.0.2

We'd expect fill-range to appear in node_modules/.pnpm but it is not present

Further debugging details below:

We found that this is due to a missing hard link step. See below details where we run the same install twice, and the link only occurs during the second install.

First install:

{…,"level":"debug","name":"pnpm:_dependency_resolved","resolution":[fill-range@7.0.1](mailto:fill-range@7.0.1),"wanted":{"dependentId":[braces@3.0.2](mailto:braces@3.0.2),"name":"fill-range","rawSpec":"^7.0.1"}}
{…,"level":"debug","name":"pnpm:progress","packageId":[fill-range@7.0.1](mailto:fill-range@7.0.1),"requester":"/PATH_OMITTED/test-rel/final-lockfile","status":"resolved"}
{…,"level":"debug","name":"pnpm:progress","packageId":[fill-range@7.0.1](mailto:fill-range@7.0.1),"requester":"/PATH_OMITTED/test-rel/final-lockfile","status":"found_in_store"}
{…,"level":"debug","name":"pnpm:_dependency_resolved","resolution":[to-regex-range@5.0.1](mailto:to-regex-range@5.0.1),"wanted":{"dependentId":[fill-range@7.0.1](mailto:fill-range@7.0.1),"name":"to-regex-range","rawSpec":"^5.0.1"}}
{…,"level":"debug","name":"pnpm:link","target":"/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/node_modules/fill-range","link":[/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range](mailto:/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range)}

(At this point trying to perform some action with the project throws fill-range not found.)

Second install:

{…,"level":"debug","name":"pnpm:_broken_node_modules","missing":[/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range](mailto:/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range)}
{…,"level":"debug","name":"pnpm:_broken_node_modules","missing":[/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range](mailto:/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range)}
{…,"level":"debug","name":"pnpm:progress","packageId":[fill-range@7.0.1](mailto:fill-range@7.0.1),"requester":"/PATH_OMITTED/test-rel/final-lockfile","status":"resolved"}
{…,"level":"debug","name":"pnpm:progress","packageId":[fill-range@7.0.1](mailto:fill-range@7.0.1),"requester":"/PATH_OMITTED/test-rel/final-lockfile","status":"found_in_store"}
{…,"level":"debug","name":"pnpm:progress","method":"hardlink","requester":"/PATH_OMITTED/test-rel/final-lockfile","status":"imported","to":[/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range](mailto:/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range)}
{…,"level":"debug","name":"pnpm:link","target":"/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/node_modules/fill-range","link":[/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range](mailto:/PATH_OMITTED/test-rel/final-lockfile/node_modules/.pnpm/fill-range@7.0.1/node_modules/fill-range)}

(Everything now works.)

Speculating a bit here, but it seems the sequence is:

1. pnpm decides that it should resolve chokidar (optional) and all its dependencies, which includes braces -> fill-range
2. pnpm then sees that fill-range is needed elsewhere for del (which is non-optional) -> globby -> ... braces -> fill-range and decides it will use this same install from earlier
3. pnpm then does some cleanup for non-optionals and removes the needed link to fill-range?

Describe the Bug

When using optional=false with pnpm 9, some packages are not installed correctly, causing errors. This is when they are subdependencies of optional dependencies but are also actual implied dependencies that should be getting installed.

This is clearly a bug as the second install produces a different node_modules than the first.

Expected Behavior

On the first install, all packages should be installed correctly. This is causing a number of issues in our CI.

Which Node.js version are you using?

20.11.0

Which operating systems have you used?

• macOS
• Windows
• Linux

If your OS is a Linux based, which one it is? (Include the version if relevant)

No response

[zkochan]

I could reproduce the issue. I will look into it after #8072