Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing distutils with pipx on 3.12 #1088

Closed
clemisch opened this issue Apr 30, 2024 · 6 comments · Fixed by #1096
Closed

Missing distutils with pipx on 3.12 #1088

clemisch opened this issue Apr 30, 2024 · 6 comments · Fixed by #1096
Labels
Bug Label for all kind of bugs.

Comments

@clemisch
Copy link

clemisch commented Apr 30, 2024 •
edited

Describe the bug

There appears to be no distutils package in Python 3.12.

$ pyinfra clem-es1 exec -- echo "hello world"
Traceback (most recent call last):
  File "/home/clem/.local/bin/pyinfra", line 5, in <module>
    from pyinfra_cli.__main__ import execute_pyinfra
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra_cli/__main__.py", line 8, in <module>
    import pyinfra
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/__init__.py", line 21, in <module>
    init_base_classes()
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/context.py", line 122, in init_base_classes
    from pyinfra.api import Config, Host, Inventory, State
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/__init__.py", line 11, in <module>
    from .deploy import deploy  # noqa: F401 # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/deploy.py", line 16, in <module>
    from .host import Host
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/host.py", line 79, in <module>
    class Host:
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/host.py", line 120, in Host
    executor=get_execution_connector("ssh"),
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/connectors.py", line 46, in get_execution_connector
    return get_execution_connectors()[name]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/connectors.py", line 40, in get_execution_connectors
    for connector, connector_mod in get_all_connectors().items()
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/connectors.py", line 32, in get_all_connectors
    entrypoint.name: _load_connector(entrypoint)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/api/connectors.py", line 24, in _load_connector
    connector = entrypoint.load()
                ^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/shared/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2482, in load
    return self.resolve()
           ^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/shared/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2488, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/connectors/chroot.py", line 14, in <module>
    from .local import run_shell_command as run_local_shell_command
  File "/home/clem/.local/pipx/venvs/pyinfra/lib64/python3.12/site-packages/pyinfra/connectors/local.py", line 6, in <module>
    from distutils.spawn import find_executable
ModuleNotFoundError: No module named 'distutils'

To Reproduce

Install via pipx and run anything.

Expected behavior

Either Python <3.12 should be enforced, and/or missing distutils circumvented.

Meta

  • Include output of pyinfra --support : gives same error as above, missing distutils.
  • Installed with pipx install pyinfra on Fedora 40 and Python 3.12.

FWIW, forcing 3.11 via pipx install pyinfra --python /usr/bin/python3.11 leads to a working install.
@clemisch clemisch added the Bug Label for all kind of bugs. label Apr 30, 2024
@belthesar
Copy link

As a heads up, distutils has been deprecated since 3.10. The replacement is setuptools. In the short term, the setuptools package can be used to provide a compatible distutils interface while full migration to setuptools or another build backend happens later.

@diazona
Copy link
Contributor

diazona commented May 1, 2024 •
edited

Same here, I also experience this problem with pipx. Oddly enough, when installing pyinfra in a separate manually-created virtualenv it works fine:

Output 
$ pwd
/home/diazona/tmp/pyinfra
$ python3.12 -m venv venv
$ source venv/bin/activate
$ python -m pip install pyinfra
[...]
$ which pyinfra
/home/diazona/tmp/pyinfra/venv/bin/pyinfra
$ pyinfra --support
--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-6.7.5-gentoo-x86_64-AMD_Ryzen_7_2700X_Eight-Core_Processor-with-glibc2.39
      Release: 6.7.5-gentoo
      Machine: x86_64
    pyinfra: v2.9.2
    Executable: /home/diazona/tmp/pyinfra/venv/bin/pyinfra
    Python: 3.12.3 (CPython, GCC 12.2.0)

I'm not sure why it doesn't happen with a regular virtualenv (I guess setuptools is getting pulled in somewhere), but in any case, having dealt with this problem in other packages, I think the fix should be simple: in pyinfra/api/connectors.py, replace pkg_resources.iter_entry_points("pyinfra.connectors") with importlib.metadata.entry_points(group="pyinfra.connectors") and adjust imports accordingly. I'd be happy to contribute a PR that addresses this and the couple other uses of pkg_resources in the code base, if desired. (I just don't have time to prepare a PR at the moment.)

@diazona diazona mentioned this issue May 6, 2024
Merged
@jmgilman
Copy link

Is there a workaround until the PR gets reviewed?

@clemisch
Copy link
Author

Quoting comment from diazona above:

when installing pyinfra in a separate manually-created virtualenv it works fine

So the workaround is: don't use pipx ;-)

@jmgilman
Copy link

For others who run into this issue: the latest release (v2.9.2) also doesn't work with Python 3.12.x, even when using poetry. I had to downgrade to Python 3.11.x before I could get it to successfully work.

@diazona
Copy link
Contributor

diazona commented May 12, 2024 •
edited

It works fine for me...

$ python3.12 -m venv venv
$ source venv/bin/activate
$ python -m pip install pyinfra==2.9.2
[...]
$ pyinfra --support
--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-6.7.5-gentoo-x86_64-AMD_Ryzen_7_2700X_Eight-Core_Processor-with-glibc2.39
      Release: 6.7.5-gentoo
      Machine: x86_64
    pyinfra: v2.9.2
    Executable: /home/diazona/tmp/pyinfra-test/venv/bin/pyinfra
    Python: 3.12.3 (CPython, GCC 12.2.0)

I've also found that in a Docker container created from the python3.12 image, I can install pipx and use it to install pyinfra (2.9.2 or 3.0b1) and it works fine there, although that seems to be because setuptools/distutils/pkg_resources are already installed in the system.

So I think all we can really say in general is that pyinfra sometimes doesn't work with Python 3.12. "Sometimes" meaning "if distutils/pkg_resources aren't available", but it seems like there are various circumstances in which that's the case and it's hard to give a simple general rule to identify those circumstances.

It does seem to work in all cases with Python 3.11 or lower, so my preferred workaround is to just use Python 3.11. (The convenience of pipx is too much to give up 😄 and to be fair, this isn't pipx's fault.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Label for all kind of bugs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove uses of pkg_resources and distutils diazona/pyinfra
4 participants
@diazona @belthesar @jmgilman @clemisch