Use parser from pip to handle the command-line arguments gracefully?

[SomeoneSerge]

Currently, requirements.txt with pip CLI arguments cause a parsing error, thus need to be somehow patched before being handed to mach-nix. In particular this means e.g. that with an unmodified requirements.txt the process won't get to checking the overrides (which may be overriding the package of interest, thus eliminating the actual need to handle the custom arguments). Handling un-patched requirements.txt is desirable because then one might get by without ever checking out the repo

Some cases where CLI arguments do appear:

• -e in git dependencies in requirements.txt #183
• --no-binary in https://github.com/zju3dv/LoFTR/blob/348ad897d4960108a1d59877c46edd22d3374385/requirements.txt#L2

A tolerably hack-ish way might be to just re-use pip._internal.commands.install.InstallCommand:

>>> from pip._internal.commands.install import InstallCommand
>>> ic = InstallCommand("install", "fake install command")
>>> options, requirements = ic.parser.parse_args(["albumentations==0.5.1", "--no-binary=imgaug,albumentations", "-e", "git+https://github.com/user/repo.git#egg=repo"])
>>> requirements
['albumentations==0.5.1']
>>> options.editables
['git+https://github.com/user/repo.git#egg=repo']

It might still be worth displaying a warning when custom options are provided, because there's no actual guarantee that an override and the CLI arguments are in sync

P.S. I'm opening this issue not as a "feature request" but more as a request for validation and comments. Thanks

[DavHau]

We could parse the requirements via nix language, transform special commands to buildPythonPackage calls and add those packages automatically via packagesExtra. Only the remaining normal requirement lines would then be passed to the python code.

Since the git references do usually not contain checksums, we could allow the user to pass those via an argument.
It could look like this for example:

mkPython {
  requirements = "...";
  extraHashes."https://github.com/user/repo.git" = "abc...";
}

mach-nix could then use those hashes inside the fetchGit/fetchTarball calls.

[SomeoneSerge]

Is there an advantage to implementing (and maintaining) a parser in nix opposed to using an existing one?

[DavHau]

In my opinion, in a perfect world, mach-nix would be written in 100% nix language. Currently it would be quite difficult to implement dependency resolution in nix. But I think everything that can be done in nix, should be done in nix. Maybe one day we can get rid of the IFD python code completely.

Also I'd like to reduce the amount of nix code generation as much as possible. I plan to get rid of code generation completely and instead let the resolver dump json data which is then interpreted by nix.

We can also do the parsing in python if it makes things easier, but then again we would have python output that needs to be interpreted by nix. So does that make things easier?

[SomeoneSerge]

One disadvantage this approach might have is more code to maintain, which is a big deal as mach-nix appears to be a voluntarily maintained project (which I'm grateful for). Nowadays pip has its own dependency solver, supports different build back-ends, those in they turn may have their own notion of dependencies, etc. Re-using what pip has already implemented and tested just seems like a more sustainable way. It could ensure that we stay consistent with pip's CLI behaviour (including its bugs sometimes, I guess) as well as support poetry-like back-ends in addition to requirements.txt

Btw, these pip install --dry-run/pip resolve proposals could be relevant:

• Add pip install --dry-run or similar, to get resolution result pypa/pip#53
• A pip resolve command to convert to transitive == requirements very fast by scanning wheels for static dependency info (WORKING PROTOTYPE!) pypa/pip#7819

[DavHau]

One disadvantage this approach might have is more code to maintain, which is a big deal as mach-nix appears to be a voluntarily maintained project (which I'm grateful for)

We could implement a separate IFD based python based requirements parser which is separated from the mach-nix resolver/code-generator. The main point is that I would not want to increase the code generator code base.

Nowadays pip has its own dependency solver, supports different build back-ends, those in they turn may have their own notion of dependencies, etc. Re-using what pip has already implemented and tested just seems like a more sustainable way.

We cannot just use pip directly for resolving since it does make arbitrary API request to a non-reproducible / non-pinable online DB. But what we already do is to use pip's resolvlib library. That's the closest we can get to pip. But I'm not even sure if trying to stay close to pip brings us any benefit. Pip's whole model of depedency resolution is very constrained by not having an offline DB which is cheap to query. We don't have that limitation. see #278

Which alternate dependency notion other than requirements.txt does pip support?

Btw, these pip install --dry-run/pip resolve proposals could be relevant: ...

Thanks for looking into this. I had a quick look and them and fail to see how those could help us. Even if pip had a --dry-run, we couldn't use it in nix since it still requires downloading files from pypi. By design it cannot work without downloading files. Pypi doesn't even have a proper dependency DB that could be downloaded etc. Even if you pre-compute all your requirements perfectly, you would still need online access again whenever you change a requirement.

For a solution which is completely based on pip and works with nix, maybe have a look at this: NixOS/nixpkgs#121425

In general I agree that we should make use of existing library code wherever we can. But from an architecture point of view mach-nix has huge advantages compared to pip and I think we should use them. Trying to be closer to pip would probably require giving up some of these advantages.