ouster-sdk package triggers InvalidVersion exception (PEP 440) #1491

kgreenek · 2023-10-13T19:05:21Z

🐞 bug report

Affected Rule

whl_library
pip_parse

Is this a regression?

Yes. This issue does not appear in 0.25.0, but it is present in 0.26.0.

Description

When installing the ouster-sdk package from pypi, it triggers an InvalidVersion exception (see the Exception section below).

I originally filed an issue with the ouster-sdk maintainers. That issue can be found here:
ouster-lidar/ouster_example#560

There is some valuable discussion there. Notably, the ouster-sdk version is 0.9.0, which should be PEP 440 compliant. It is unclear where the version string that is causing the InvalidVersion exception is coming from. It does appear to be something specific about the ouster-sdk wheel though, since I do not have problems with other packages.

🔬 Minimal Reproduction

Minimal reproduction with instructions for running:
https://github.com/kgreenek/ouster_sdk_pep440

🔥 Exception or Error


ERROR: /home/kevin/src/kgreenek/ouster_sdk_pep440/ouster_pcap_record/BUILD.bazel:4:10: //ouster_pcap_record:ouster_pcap_record depends on @pip_deps_ouster_sdk//:pkg in repository @pip_deps_ouster_sdk which failed to fetch. no such package '@pip_deps_ouster_sdk//': whl_library pip_deps_ouster_sdk failed: Collecting ouster-sdk==0.9.0 (from -r /tmp/tmp7wuogsxy (line 1))
  Using cached ouster_sdk-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Saved ./ouster_sdk-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 (Traceback (most recent call last):
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/python_3_10_6_x86_64-unknown-linux-gnu/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/python_3_10_6_x86_64-unknown-linux-gnu/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 200, in 
    main()
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 192, in main
    _extract_wheel(
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 132, in _extract_wheel
    whl_deps = sorted(whl.dependencies(extras_requested) - self_edge_dep)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/rules_python/python/pip_install/tools/wheel_installer/wheel.py", line 79, in dependencies
    if req.marker is None or any(
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/rules_python/python/pip_install/tools/wheel_installer/wheel.py", line 80, in 
    req.marker.evaluate({"extra": extra})
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/markers.py", line 252, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/markers.py", line 158, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/markers.py", line 116, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/specifiers.py", line 568, in contains
    normalized_item = _coerce_version(item)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
  File "/home/kevin/.cache/bazel/_bazel_kevin/0bfc23b9cb417e948c9bb9daad8abbda/external/pypi__setuptools/pkg_resources/_vendor/packaging/version.py", line 198, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '#34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2'
) error code: '1'

🌍 Your Environment

Operating System:

  
Ubuntu 22.04

Output of bazel version:

  
Bazelisk version: v1.18.0
Build label: 6.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jun 2 16:59:58 2023 (1685725198)
Build timestamp: 1685725198
Build timestamp as int: 1685725198

Rules_python version:

  
0.26.0

The text was updated successfully, but these errors were encountered:

groodt · 2023-10-13T21:49:39Z

It’s coming from the upgrade to packaging somewhere. There must be an extra in one of the transitive deps that has an invalid specifier.

twslankard · 2023-10-13T22:19:36Z

While investigating this issue, I determined that removing platform_version >= "21.0.0" from the dependency spec for ouster-sdk's scipy dependency fixes the issue. As far as I'm aware, this is still a valid dependency spec, but perhaps not?

groodt · 2023-10-13T22:47:34Z

Definitely seems like you’ve identified the problematic specifier.

Try putting a space after the semicolon before platform_system?

groodt · 2023-10-15T10:57:33Z

Looks like I can reproduce the issue. Seems to be something going on between modern setuptools and modern packaging.

(.venv) ⋊> /tmp python3                                                                                                                                                                                                                      21:53:47
Python 3.9.16 (main, Oct 15 2023, 20:39:25)
[Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.__version__
'65.7.0'
>>> import pkg_resources
>>> req = pkg_resources.Requirement('scipy >=1.7, <2; platform_version >= "21.0.0"')
>>> req.marker.evaluate()
False
>>> import packaging
>>> packaging.__version__
'23.1'
>>> from packaging.markers import Marker
>>> marker = Marker("platform_version >= '21.0.0'")
>>> marker.evaluate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 252, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 158, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 116, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/specifiers.py", line 568, in contains
    normalized_item = _coerce_version(item)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/version.py", line 198, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000'

If I install setuptools>66 then req.marker.evaluate() is also failing. I think they switched to a newer version of packaging with newer setuptools.

groodt · 2023-10-15T11:20:34Z

I think I've got to the bottom of this.

The 21.0.0 is triggering PEP 440 version comparison (not string comparison). It does seem to indicate that platform_version >= "21.0.0" is strictly invalid with new versions of packaging.

>>> marker = Marker("platform_version >= '21.0.0'")
>>> marker.evaluate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 252, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 158, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/markers.py", line 116, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/specifiers.py", line 568, in contains
    normalized_item = _coerce_version(item)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
  File "/private/tmp/.venv/lib/python3.9/site-packages/packaging/version.py", line 198, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000'

To have this work, I think you could use:

>>> marker = Marker("platform_release >= '21.0.0'")
>>> marker.evaluate()

groodt · 2023-10-16T22:26:14Z

There's a fix proposed in the problematic package: ouster-lidar/ouster_example#560 (comment)

I'm going to close this issue for now. I do think we're probably going to see a few more problematic packages because we are using a more modern version of packaging and setuptools than pip at the moment. However, pip are also likely to prevent installation of problematic packages in a future version (slated to be the first version in 2024). See pypa/pip#12063

If it becomes too problematic, we can consider what a rollback to older versions of packaging and/or setuptools might look like or reopen this issue.

kgreenek · 2023-10-17T06:08:00Z

Sounds good to me. Thank you for taking time to help get to the bottom of the issue. I really appreciate it.

twslankard · 2024-05-24T02:13:43Z

I set aside this problem for a while and only started looking at it again today.

What I'm realizing is that platform_release is also not necessarily a PEP 440 version, since it could be something weird like 6.5.0-752-oracle.

This is making me wonder whether rules_python's wheel installer is inappropriately requiring these environment markers, such as platform_release (which ultimately comes from Python's own platform.release() method,) to be conformant.

I don't know how to override the value of platform_release to normalize it, so I suspect the only way around this is to change the behavior of rules_python through configuration or a patch of some sort.

groodt mentioned this issue Oct 15, 2023

ouster-sdk python package raises InvalidVersion error with upcoming version of pip ouster-lidar/ouster_example#560

Open

groodt closed this as completed Oct 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ouster-sdk package triggers InvalidVersion exception (PEP 440) #1491

ouster-sdk package triggers InvalidVersion exception (PEP 440) #1491

kgreenek commented Oct 13, 2023

groodt commented Oct 13, 2023

twslankard commented Oct 13, 2023

groodt commented Oct 13, 2023

groodt commented Oct 15, 2023

groodt commented Oct 15, 2023

groodt commented Oct 16, 2023 •

edited

kgreenek commented Oct 17, 2023

twslankard commented May 24, 2024 •

edited

ouster-sdk package triggers InvalidVersion exception (PEP 440) #1491

ouster-sdk package triggers InvalidVersion exception (PEP 440) #1491

Comments

kgreenek commented Oct 13, 2023

🐞 bug report

Affected Rule

Is this a regression?

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

groodt commented Oct 13, 2023

twslankard commented Oct 13, 2023

groodt commented Oct 13, 2023

groodt commented Oct 15, 2023

groodt commented Oct 15, 2023

groodt commented Oct 16, 2023 • edited

kgreenek commented Oct 17, 2023

twslankard commented May 24, 2024 • edited

groodt commented Oct 16, 2023 •

edited

twslankard commented May 24, 2024 •

edited