Supporting adding extra annotations to the exported requirements file

[aignas]

Is your feature/enhancement proposal related to a problem? Please describe.

I am using https://github.com/bazelbuild/rules_python to manage my Python project and I noticed that the pdm is a really nice way to generate a cross-platform lock file pdm.lock. Moreover, it also has a feature to export the lock file to requirements.txt, which is cross-platform, i.e. it has annotations on what should be installed on which platform. Having a single lock file for multiple platforms would be great, but unfortunately it does not work at the moment with rules_python because the extras are removed as per #741.

At least for now, the latest rules_python version relies on those extras annotations to be present in the requirements file and I would love pdm to be able to provide those too.

Describe the solution you'd like

A CLI flag for the pdm export --format requirements command to add --extras.

[frostming]

Can you explain more on how the extra info is used?

Given this requirements

pydantic==1.8.2
pydantic[email]==1.8.2

Note all other transitive dependencies are also exported but omitted. Why does rules-python need the second entry?

Because from the perspective of an installer, there is not any difference between whether you use the extras tag, or specify the extra dependencies manually.

[aignas]

Right now it only needs the second entry, but we can handle a requirements file with both entries.

To be concrete, rules_python is parsing each entry in the requirements.txt and is generating a BUILD.bazel file with contents similar to below:

filegroup(name="whl", src=["pydantic-...-any.whl"])
py_library(
    name = "pkg",
    srcs = [
        # All of the extracted whl contents
    ],
    deps = [
        # deps for the python sources. Only dependencies for the requested extras are included
        # which means that `pydantic==1.8.2` would yield a subset of dependencies and
        # `pydantic[email]==1.8.2` would yield the correct set of deps
    ]
)

The deps generation for the example above is done by:

1. Extracting the requested deps from the requirements.txt entry (i.e. parse the pydantic[email,foo]==1.8.2 --hashes=sha256:... line to get ["email", "foo"]
2. Parsing the whl METADATA file to get the Requires-Dist and generating the applicable dependencies for the target platform.

In the ideal case, we should consume pdm.lock directly, but I am interested in this feature to act as a stop-gap solution, where the rules_python does not yet support pdm.lock file but the users can start using pdm with rules_python via exporting requirements.txt file using pdm export --format requirements --dev --with-extras requirement_lock.txt.

Let me know if you'd like to know more details or if the text above is a little too detailed/confusing.

[frostming]

2. Parsing the whl METADATA file to get the Requires-Dist and generating the applicable dependencies for the target platform.

But exported requirements should contain all transitive dependencies, so even it parses pydantic==1.8.2, the extra dependencies are still pinned. Or does it only consider the dependencies in pyproject.toml?

[aignas]

rules_python cannot read dependencies from pyproject.toml yet and we depend on all of the transitive dependencies being in the requirements.txt file with their hashes. Something similar to this.

Ideal file would contain only pyjwt[crypto] and the linked example is just a test to see that we can handle a requirements.txt file similar to the one output by poetry export command.

The Requires-Dist parsing I mentioned previously only considers the package names and environment markers, i.e. it does not do transitive dependency pinning.

[frostming]

I still can't get why it is necessary.

Let's take pydantic[email]==1.8.2 as an example. Here is what pdm export --no-hashes outputs:

dnspython==2.4.2  pydantic[email]
email-validator==2.1.0.post1  # from pydantic[email]
idna==3.6  # pydantic[email]
pydantic==1.8.2
typing-extensions==4.9.0

All dependencies are pinned, does rules-python pick up all dependencies from the above list? what is missing and what else pydantic[email] can provide?

[aignas]

Just to clarify, rules_python currently needs:

• All of the transitive dependencies, because we are downloading each package separately from PyPI and other sources. Bazel has its own sandbox and that is how it currently consumes Python dependencies.
• Hashes for all of the dependencies. Bazel needs to have integrity checksums for each source that it consumes so that if a dependency within the transitive dependency closure changes, it can refetch the required dependencies and rerun tests.
• The required extras for each of the requirement, so that each dependency that is fetched can have the correct runtime dependency lists.

I am not sure if there is a CLI option combination. I understand that supporting this use case might be out of scope of the tool, so let me know if supporting this aligns with the future of pdm you have in mind.

[bitnom]

I still can't get why it is necessary.

Let's take pydantic[email]==1.8.2 as an example. Here is what pdm export --no-hashes outputs:

dnspython==2.4.2  pydantic[email]
email-validator==2.1.0.post1  # from pydantic[email]
idna==3.6  # pydantic[email]
pydantic==1.8.2
typing-extensions==4.9.0

All dependencies are pinned, does rules-python pick up all dependencies from the above list? what is missing and what else pydantic[email] can provide?

Is that behavior part of a PEP? It seems redundant and counterintuitive (If I'm not misunderstanding it).