Source Distribution Format spec requires a Metadata-Version forbidden by PyPI

[bgilbert]

Problem

PyPI does not allow upload of source distributions or wheels with 2.2 or 2.3 metadata. (The 2.2 bug has been open since June 2021 and the 2.3 bug since June 2022.) twine upload fails with:

ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         '2.2' is an invalid value for Metadata-Version. Error: Use a known     
         metadata version. See                                                  
         https://packaging.python.org/specifications/core-metadata for more     
         information.                                                           

However, the Source Distribution Format spec requires 2.2 or newer metadata:

A .tar.gz source distribution (sdist) contains [...] a PKG-INFO file containing metadata in the format described in the Core metadata specifications specification. The metadata MUST conform to at least version 2.2 of the metadata specification.

The spec scopes itself this way:

The current standard format of source distribution format is identified by the presence of a pyproject.toml file in the distribution archive.

There are many sdists on PyPI with pyproject.toml files, all of which are therefore violating the spec. It seems unrealistic for the spec to insist otherwise, so perhaps its requirements should be updated to allow metadata older than 2.2.

History

The relevant language was added in #785, apparently based on PEP 643, both by the same author (cc @pfmoore). However, the MUST language isn't supported by the PEP, which only has a SHOULD requirement:

Source distributions SHOULD use the latest version of the core metadata specification that was available when they were created.

The PEP also explicitly asserts backward-compatibility:

As this proposal increments the core metadata version, it is compatible with existing source distributions, which will use an older metadata version. Tools can determine whether a source distribution conforms to this PEP by checking the metadata version.

So the MUST language may have been added accidentally when reformatting the material for PyPUG.

[chrysle]

Have a look at the introduction of the specification. It is simply a guideline for source distributions which actually have metadata version 2.2 and higher, and doesn't require source distributions to have:

This document does not attempt to standardise this format, except to note that if a legacy source distribution contains a PKG-INFO file using metadata version 2.2 or later, then it MUST follow the rules applicable to source distributions defined in the metadata specification.

Therefore I think the MUST operator is necessary.

[bgilbert]

The current text doesn't support that reading, though. First, if the entire document were only meant to apply to 2.2 or above, this sentence would not need to be phrased as a restriction on behavior:

The metadata MUST conform to at least version 2.2 of the metadata specification.

And second, the section you quoted refers to something different. A fuller quote:

The current standard format of source distribution format is identified by the presence of a pyproject.toml file in the distribution archive. The layout of such a distribution was originally specified in PEP 517 and is formally documented here.

There is also the legacy source distribution format, implicitly defined by the behaviour of distutils module in the standard library, when executing setup.py sdist. This document does not attempt to standardise this format, except to note that if a legacy source distribution contains a PKG-INFO file using metadata version 2.2 or later, then it MUST follow the rules applicable to source distributions defined in the metadata specification.

So, if a source distribution doesn't have a pyproject.toml, and does have 2.2+ metadata, then it's required to follow the spec nevertheless. That seems reasonable enough, though notably it doesn't affect any source distribution currently uploaded to PyPI. If the source distribution does have a pyproject.toml, the first paragraph indicates that the spec always applies, leading to the problem described in this issue.