Upload of 'google-cloud-bigquery' wheel fails with 400

[tseaver]

Describe the bug
Automated upload of 1.27.1 release fails with:

 0%|          | 0.00/180k [00:00<?, ?B/s]
  4%|▍         | 8.00k/180k [00:00<00:04, 36.8kB/s]
 53%|█████▎    | 96.0k/180k [00:00<00:01, 51.4kB/s]
100%|██████████| 180k/180k [00:00<00:00, 223kB/s]  
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Invalid requirement: 'pyarrow (<0.17.0) ; ( platform_system != "Windows":python_version < "3.0") and extra == \'pyarrow\''.

1.27.0 release also failed, using a variant on the pyarrow dependency in use for a long while:

Uploading google_cloud_bigquery-1.27.0-py2.py3-none-any.whl

  0%|          | 0.00/180k [00:00<?, ?B/s]
  4%|▍         | 8.00k/180k [00:00<00:04, 36.6kB/s]
 53%|█████▎    | 96.0k/180k [00:00<00:01, 51.2kB/s]
100%|██████████| 180k/180k [00:00<00:00, 222kB/s]  
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Invalid requirement: 'pyarrow (<0.17.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version < "3.0") and extra == \'pyarrow\''.

python setup.py check --restructuredtext --strict returns no errors for either one with latest setuptools / pip installed.

Expected behavior
Upload should succeed.

To Reproduce

$ git clone git@github.com:googleapis/python-bigquery
$ cd python-bigquery
$ git checkout v1.27.1  # or v1.27.0
$ python3.8 setup.py check --restructuredtext --strict
$ python3.8 setup.py sdist bdist_wheel
$ twine upload dist/*

[crwilcox]

Diff of previous release: googleapis/python-bigquery@v1.27.0...v1.27.1

[tseaver]

The last release to upload successfully was 1.26.1, so this diff may be more germane

[di]

I think the issue is this change:

-    'pyarrow: platform_system != "Windows" or python_version >= "3.4"': [
-        # Bad Linux release for 0.14.0.
-        # https://issues.apache.org/jira/browse/ARROW-5868
-        "pyarrow>=0.4.1, != 0.14.0"
+    'pyarrow: platform_system != "Windows" or python_version >= "3.5"': [
+        "pyarrow>=1.0.0, <2.0dev; python_version>='3.4'",
+        # Pyarrow >= 0.17.0 is not compatible with Python 2 anymore.
+        "pyarrow < 0.17.0; python_version < '3.0'",
+    ],

This is definitely not a valid PEP 508 dependency string:

>>> from packaging.requirements import Requirement
>>> Requirement('pyarrow (<0.17.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version < "3.0") and extra == \'pyarrow\'')
Traceback (most recent call last):
  File "/Users/dustiningram/git/pypa/packaging/packaging/requirements.py", line 98, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/dustiningram/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pyparsing.py", line 1955, in parseString
    raise exc
  File "/Users/dustiningram/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected stringEnd, found ';'  (at char 18), (line:1, col:19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dustiningram/git/pypa/packaging/packaging/requirements.py", line 100, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'; ( plat'": Expected stringEnd
>>>

Whereas this would be:

>>> Requirement('pyarrow (<0.17.0) ; ( platform_system != "Windows" or python_version >= "3.5") and extra == \'pyarrow\'')
<Requirement('pyarrow<0.17.0; (platform_system != "Windows" or python_version >= "3.5") and extra == "pyarrow"')>

It seems like maybe this is a bug with how setuptools handles a python_version in the extra name and in the dependency specification? Still looking into it.

[di]

Here's the metadata that gets written to the wheel with this extra:

Requires-Dist: pyarrow (<0.17.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version < "3.0") and extra == 'pyarrow'
Requires-Dist: pyarrow (<2.0dev,>=1.0.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version >= "3.4") and extra == 'pyarrow'

What is the actual intended behavior? It seems like this could be simplified, but as I'm trying to unpack what this would do if it's working, it doesn't totally make sense to me.

Furthermore, it looks like https://pypi.org/project/pyarrow/0.17.0/ correctly specifies Requires: Python >=3.5, so is this even necessary? It seems like the resolver should handle most of this logic.

[tseaver]

I have pushed a change to BigQuery which moves all platform constraints for extras into the "key", rather than mixing them, which I believe is the issue.

From the 1.27.0 release:

extras = {
...
    # Exclude PyArrow dependency from Windows Python 2.7.
    'pyarrow: platform_system != "Windows" or python_version >= "3.5"': [
        "pyarrow>=1.0.0, <2.0dev; python_version>='3.4'",
        # Pyarrow >= 0.17.0 is not compatible with Python 2 anymore.
        "pyarrow < 0.17.0; python_version < '3.0'",
    ],
...
}

which yielded these broken Requires-Dist entries:

Requires-Dist: pyarrow (<0.17.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version < "3.0") and extra == 'pyarrow'
Requires-Dist: pyarrow (<2.0dev,>=1.0.0) ; ( platform_system != "Windows" or python_version >= "3.5":python_version >= "3.4") and extra == 'pyarrow'

From the 1.27.1 version:

extras = {
...
    # Exclude PyArrow dependency from Windows Python 2.7.
    'pyarrow: platform_system == "Windows"': [
        "pyarrow>=1.0.0, <2.0dev; python_version>='3.5'",
    ],
    'pyarrow: platform_system != "Windows"': [
        "pyarrow>=1.0.0, <2.0dev; python_version>='3.5'",
        # Pyarrow >= 0.17.0 is not compatible with Python 2 anymore.
        "pyarrow < 0.17.0; python_version < '3.0'",
    ],
...
}

which yielded these broken Requires-Dist entries:

Requires-Dist: pyarrow (<2.0dev,>=1.0.0) ; ( platform_system != "Windows":python_version >= "3.5") and extra == 'pyarrow'
Requires-Dist: pyarrow (<2.0dev,>=1.0.0) ; ( platform_system == "Windows":python_version >= "3.5") and extra == 'pyarrow'

From the PR:

extras = {
...
    "pyarrow: python_version >= '3.5'": ["pyarrow >= 1.0.0, < 2.0dev"],
    # Exclude PyArrow dependency from Windows Python 2.7.
    "pyarrow: platform_system != 'Windows' and python_version < '3.0'": [
        "pyarrow < 0.17.0",
    ],
...
}

which yielded these working Requires-Dist entries:

Requires-Dist: pyarrow (<0.17.0) ; ( platform_system != 'Windows' and python_version < '3.0') and extra == 'pyarrow'
Requires-Dist: pyarrow (<2.0dev,>=1.0.0) ; ( python_version >= '3.5') and extra == 'pyarrow'

[tseaver]

I'm presuming that the next upload will work. Since this isn't actually a Warehouse bug, I'm closing it.