Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

20b1: "RuntimeError: failed to detect cpython3.6.8-64|..." #1515

Closed
asottile opened this issue Jan 28, 2020 · 19 comments
Closed

20b1: "RuntimeError: failed to detect cpython3.6.8-64|..." #1515

asottile opened this issue Jan 28, 2020 · 19 comments
Assignees
@gaborbernat

Comments

@asottile
Copy link
Contributor

this is breaking some tests for pre-commit

this is similar to #1514 but involves absolute paths

here's a minimal reproduction:

set -euxo pipefail
rm -rf venv1 venv2
virtualenv venv1
venv1/bin/pip install virtualenv --upgrade --pre
"$PWD/venv1/bin/python" -mvirtualenv venv2 -p "$PWD/venv1/bin/python"
$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /home/asottile/workspace/pre-commit/venv1/bin/python3
Also creating executable in /home/asottile/workspace/pre-commit/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install virtualenv --upgrade --pre
Collecting virtualenv
  Using cached virtualenv-20.0.0b1-py2.py3-none-any.whl (4.6 MB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.4.0-py2.py3-none-any.whl (29 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Installing collected packages: appdirs, filelock, zipp, importlib-metadata, importlib-resources, six, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.4.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-20.0.0b1 zipp-2.1.0
+ /home/asottile/workspace/pre-commit/venv1/bin/python -mvirtualenv venv2 -p /home/asottile/workspace/pre-commit/venv1/bin/python
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 203, in system_executable
    result = self.find_exe_based_of(inside_folder=env_prefix)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 225, in find_exe_based_of
    "failed to detect {} in {}".format(what, os.pathsep.join(possible_folders))
RuntimeError: failed to detect cpython3.6.8-64|cpython3.6.8|cpython3.6-64|cpython3.6|cpython3-64|cpython3|cpython-64|cpython|CPython3.6.8-64|CPython3.6.8|CPython3.6-64|CPython3.6|CPython3-64|CPython3|CPython-64|CPython|CPYTHON3.6.8-64|CPYTHON3.6.8|CPYTHON3.6-64|CPYTHON3.6|CPYTHON3-64|CPYTHON3|CPYTHON-64|CPYTHON|python3.6.8-64|python3.6.8|python3.6-64|python3.6|python3-64|python3|python-64|python|PYTHON3.6.8-64|PYTHON3.6.8|PYTHON3.6-64|PYTHON3.6|PYTHON3-64|PYTHON3|PYTHON-64|PYTHON in /usr/bin:/usr
$ uname -a
Linux asottile-MacBookPro 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
$ venv1/bin/python --version --version
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0]
$ venv1/bin/pip freeze --all
appdirs==1.4.3
filelock==3.0.12
importlib-metadata==1.4.0
importlib-resources==1.0.2
pip==20.0.2
setuptools==45.1.0
six==1.14.0
virtualenv==20.0.0b1
wheel==0.34.1
zipp==2.1.0
@gaborbernat
Copy link
Contributor

I think this will be #1512, cab you try rewrite branch?

@asottile
Copy link
Contributor Author

still broken -- here's with rewrite and -vv:

set -euxo pipefail
rm -rf venv1 venv2
virtualenv venv1
venv1/bin/pip install git+https://github.com/pypa/virtualenv@rewrite
"$PWD/venv1/bin/python" -mvirtualenv venv2 -p "$PWD/venv1/bin/python" -vv
$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /home/asottile/workspace/pre-commit/venv1/bin/python3
Also creating executable in /home/asottile/workspace/pre-commit/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install git+https://github.com/pypa/virtualenv@rewrite
Collecting git+https://github.com/pypa/virtualenv@rewrite
  Cloning https://github.com/pypa/virtualenv (to revision rewrite) to /tmp/pip-req-build-bqa968pu
  Running command git clone -q https://github.com/pypa/virtualenv /tmp/pip-req-build-bqa968pu
  Running command git checkout -b rewrite --track origin/rewrite
  Switched to a new branch 'rewrite'
  Branch 'rewrite' set up to track remote branch 'rewrite' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.4.0-py2.py3-none-any.whl (29 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-0.1.dev49+g2e3f6c8-py2.py3-none-any.whl size=4594112 sha256=4da2780e918b2a72faa8b86313567d31d3d83219d4edb0fc91496df0b18c01cd
  Stored in directory: /tmp/pip-ephem-wheel-cache-u7ms6z7n/wheels/79/5c/82/576f088287bb37f13bf7bff00083b8da8570ef25e9ccd9a856
Successfully built virtualenv
Installing collected packages: importlib-resources, six, zipp, importlib-metadata, filelock, appdirs, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.4.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-0.1.dev49+g2e3f6c8 zipp-2.1.0
+ /home/asottile/workspace/pre-commit/venv1/bin/python -mvirtualenv venv2 -p /home/asottile/workspace/pre-commit/venv1/bin/python -vv
79 setup logging to DEBUG [DEBUG report:43]
85 find interpreter for spec PythonSpec(path=/home/asottile/workspace/pre-commit/venv1/bin/python) [INFO builtin:44]
85 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/home/asottile/workspace/pre-commit/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:48]
85 accepted target interpreter PythonInfo(spec=CPython3.6.8.final.0-64, exe=/home/asottile/workspace/pre-commit/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:50]
86 filesystem is case-sensitive [DEBUG info:28]
91 Attempting to acquire lock 139987952676704 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
92 Lock 139987952676704 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
92 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:70]
93 Attempting to release lock 139987952676704 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
94 Lock 139987952676704 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
94 Attempting to acquire lock 139987952676704 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
95 Lock 139987952676704 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
95 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python [DEBUG cached_py_info:70]
96 Attempting to release lock 139987952676704 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
96 Lock 139987952676704 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev49+g2e3f6c8/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
97 done in 19ms [INFO __main__:23]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 203, in system_executable
    result = self.find_exe_based_of(inside_folder=env_prefix)
  File "/home/asottile/workspace/pre-commit/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 225, in find_exe_based_of
    "failed to detect {} in {}".format(what, os.pathsep.join(possible_folders))
RuntimeError: failed to detect cpython3.6.8-64|cpython3.6.8|cpython3.6-64|cpython3.6|cpython3-64|cpython3|cpython-64|cpython|CPython3.6.8-64|CPython3.6.8|CPython3.6-64|CPython3.6|CPython3-64|CPython3|CPython-64|CPython|CPYTHON3.6.8-64|CPYTHON3.6.8|CPYTHON3.6-64|CPYTHON3.6|CPYTHON3-64|CPYTHON3|CPYTHON-64|CPYTHON|python3.6.8-64|python3.6.8|python3.6-64|python3.6|python3-64|python3|python-64|python|PYTHON3.6.8-64|PYTHON3.6.8|PYTHON3.6-64|PYTHON3.6|PYTHON3-64|PYTHON3|PYTHON-64|PYTHON in /usr/bin:/usr

@gaborbernat
Copy link
Contributor

So according to the host prefix the executable should be either under usr or usr bin, where is it?

@asottile
Copy link
Contributor Author 

$ which -a python3.6
/usr/bin/python3.6

@gaborbernat
Copy link
Contributor

Can't seem to reproduce myself. @asottile can you please run with #1522, and report why the interpreter was refused?

@gaborbernat gaborbernat mentioned this issue Jan 29, 2020
Merged
@asottile asottile mentioned this issue Jan 29, 2020
Closed
@asottile
Copy link
Contributor Author 

set -euxo pipefail
rm -rf venv1 venv2
virtualenv venv1
venv1/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
"$PWD/venv1/bin/python" -mvirtualenv venv2 -p "$PWD/venv1/bin/python" -vv
$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /tmp/x/venv1/bin/python3
Also creating executable in /tmp/x/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
Collecting git+https://github.com/gaborbernat/virtualenv@1515
  Cloning https://github.com/gaborbernat/virtualenv (to revision 1515) to /tmp/pip-req-build-jlkokynr
  Running command git clone -q https://github.com/gaborbernat/virtualenv /tmp/pip-req-build-jlkokynr
  Running command git checkout -b 1515 --track origin/1515
  Switched to a new branch '1515'
  Branch '1515' set up to track remote branch '1515' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-0.1.dev53+g7a7236d-py2.py3-none-any.whl size=4594556 sha256=fa07557f274d74576e0e8c9282cd3d563bd8d74663d343e63278da5567d64941
  Stored in directory: /tmp/pip-ephem-wheel-cache-h3ivsba5/wheels/ed/48/58/db8714ec3a4c7a2283f34228da5a94b90f0df836be6abec2fd
Successfully built virtualenv
Installing collected packages: six, appdirs, zipp, importlib-metadata, filelock, importlib-resources, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-0.1.dev53+g7a7236d zipp-2.1.0
+ /tmp/x/venv1/bin/python -mvirtualenv venv2 -p /tmp/x/venv1/bin/python -vv
49 setup logging to DEBUG [DEBUG report:43]
53 find interpreter for spec PythonSpec(path=/tmp/x/venv1/bin/python) [INFO builtin:44]
53 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:48]
53 accepted target interpreter PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:50]
54 filesystem is case-sensitive [DEBUG info:28]
59 create folder /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d [DEBUG _sync:22]
59 Attempting to acquire lock 140392693168616 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
59 Lock 140392693168616 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
59 get interpreter info via cmd: /usr/bin/python3.6 -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
111 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:85]
112 Attempting to release lock 140392693168616 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
112 Lock 140392693168616 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
112 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
112 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
112 Attempting to acquire lock 140392693168616 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
112 Lock 140392693168616 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
112 get interpreter info via cmd: /usr/bin/python -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
141 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python [DEBUG cached_py_info:85]
141 Attempting to release lock 140392693168616 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
141 Lock 140392693168616 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev53+g7a7236d/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
141 refused interpreter because version_info differs VersionInfo(major=2, minor=7, micro=17, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
142 done in 93ms [INFO __main__:23]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 204, in system_executable
    result = self.find_exe_based_of(inside_folder=env_prefix)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 232, in find_exe_based_of
    "failed to detect {} in {}".format(what, os.pathsep.join(possible_folders))
RuntimeError: failed to detect cpython3.6.8-64|cpython3.6.8|cpython3.6-64|cpython3.6|cpython3-64|cpython3|cpython-64|cpython|CPython3.6.8-64|CPython3.6.8|CPython3.6-64|CPython3.6|CPython3-64|CPython3|CPython-64|CPython|CPYTHON3.6.8-64|CPYTHON3.6.8|CPYTHON3.6-64|CPYTHON3.6|CPYTHON3-64|CPYTHON3|CPYTHON-64|CPYTHON|python3.6.8-64|python3.6.8|python3.6-64|python3.6|python3-64|python3|python-64|python|PYTHON3.6.8-64|PYTHON3.6.8|PYTHON3.6-64|PYTHON3.6|PYTHON3-64|PYTHON3|PYTHON-64|PYTHON in /usr/bin:/usr

my hunch is that the virtualenv I used to make the first virtualenv is installed to an older virtualenv which was created before ubuntu released a patch release -- it differs from the system python:

$ ~/opt/venv/bin/python --version --version
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0]
$ python3.6 --version --version
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]

this is a fairly common situation (especially on releases which roll more often) and would be a regression from old-virtualenv

@gaborbernat
Copy link
Contributor

gaborbernat commented Jan 29, 2020 •
edited

Indeed. Seems the host python for the ~/opt/venv/bin/python is with 3.6.8... however /usr/bin/python3.6 is 3.6.9...

Now I wonder how can ~/opt/venv/bin/python still be 3.6.8? Did you copy the executable? (as I'd expect to be 3.6.9 in case of a symlink). If so, we're in position, because other than the executable everything else (libs, standard library, etc) will be python 3.6.9, not 3.6.8. Should the 3.6.8 executable still work with libs/standard libraries of 3.6.9?

virtualenv here does the sensible thing and fails because the executable you point to is 3.6.8, so it wants to create an interpreter that has the same spec (same version, implementation, architecture). Nevertheless, virtual environments are a reference to the host, it cannot create a virtual environment from another virtual environment. It needs a host Python. So when it goes to find the host interpreter, it's no longer there to be found; only 3.6.9... and that's not the same.

As for a solution, here ughhh; dunno, maybe it's okay to differ on minor versions; though in this case not that we'll actually create a virtual environment of a minor patch higher; not the same you pointed at. Note though, would you leave it more open with the -p specification, for example as -p 3.6 we'd happily find the one under /usr/bin/python3.6 and all would work. However here you don't say I want any 3.6; you say by pointing it to another interpreter that I want one of the same type as that... Which we cannot satisfy because we can't find an interpreter having such specification (due to the minor version difference).

I'm tempted to mark this working as intended... However, before I do that I'd ask for your input, and my fellow maintainers @pfmoore @pradyunsg.

@asottile
Copy link
Contributor Author

~/opt/venv was created using virtualenv 16.x which copies the executable in

this would be really bad if this is not fixed, basically every ubuntu update would break my entire system (which runs entirely out of virtualenvs -- no system installs)

@gaborbernat
Copy link
Contributor

Ok, added a fallback system to accommodate for system upgrades. I think this should fix your use case now.

@asottile
Copy link
Contributor Author 

$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /tmp/x/venv1/bin/python3
Also creating executable in /tmp/x/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
Collecting git+https://github.com/gaborbernat/virtualenv@1515
  Cloning https://github.com/gaborbernat/virtualenv (to revision 1515) to /tmp/pip-req-build-fdhjnc61
  Running command git clone -q https://github.com/gaborbernat/virtualenv /tmp/pip-req-build-fdhjnc61
  Running command git checkout -b 1515 --track origin/1515
  Switched to a new branch '1515'
  Branch '1515' set up to track remote branch '1515' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-0.1.dev56+g6f742e5-py2.py3-none-any.whl size=4595182 sha256=7d2da3b2e6b125f7d79e4ae61879c56ac5a718ca74fff170b071e67dcea9335f
  Stored in directory: /tmp/pip-ephem-wheel-cache-womd99l5/wheels/ed/48/58/db8714ec3a4c7a2283f34228da5a94b90f0df836be6abec2fd
Successfully built virtualenv
Installing collected packages: importlib-resources, filelock, zipp, importlib-metadata, six, appdirs, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-0.1.dev56+g6f742e5 zipp-2.1.0
+ /tmp/x/venv1/bin/python -mvirtualenv venv2 -p /tmp/x/venv1/bin/python -vv
55 setup logging to DEBUG [DEBUG report:43]
60 find interpreter for spec PythonSpec(path=/tmp/x/venv1/bin/python) [INFO builtin:44]
60 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:48]
60 accepted target interpreter PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:50]
61 filesystem is case-sensitive [DEBUG info:28]
65 Attempting to acquire lock 140088126041112 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
66 Lock 140088126041112 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
66 get interpreter info via cmd: /usr/bin/python3.6 -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
122 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:85]
122 Attempting to release lock 140088126041112 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
122 Lock 140088126041112 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
122 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
122 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
122 Attempting to acquire lock 140088126041112 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
123 Lock 140088126041112 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
123 get interpreter info via cmd: /usr/bin/python -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
152 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python [DEBUG cached_py_info:85]
152 Attempting to release lock 140088126041112 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
152 Lock 140088126041112 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev56+g6f742e5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
153 refused interpreter because version_info differs VersionInfo(major=2, minor=7, micro=17, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
153 done in 98ms [INFO __main__:23]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 204, in system_executable
    result = self._find_exe_based_of(inside_folder=env_prefix)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 232, in _find_exe_based_of
    most_likely = self._select_most_likely(discovered, self)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 263, in _select_most_likely
    sorted_discovered = sorted(discovered, key=sort_by, reverse=True)  # sort by priority in decreasing order
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py", line 256, in sort_by
    item.version_info.patch == target.version_info.patch,
AttributeError: 'VersionInfo' object has no attribute 'patch'

looks like there's a typo hehe -- should be micro I believe instead of patch

@gaborbernat
Copy link
Contributor

So sorry for that, you're right, push a fixed commit.

@asottile
Copy link
Contributor Author

getting there :)

$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /tmp/x/venv1/bin/python3
Also creating executable in /tmp/x/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
Collecting git+https://github.com/gaborbernat/virtualenv@1515
  Cloning https://github.com/gaborbernat/virtualenv (to revision 1515) to /tmp/pip-req-build-k_kxyesz
  Running command git clone -q https://github.com/gaborbernat/virtualenv /tmp/pip-req-build-k_kxyesz
  Running command git checkout -b 1515 --track origin/1515
  Switched to a new branch '1515'
  Branch '1515' set up to track remote branch '1515' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-0.1.dev58+g2bc34d2-py2.py3-none-any.whl size=4595183 sha256=80b0bab8fb03f7c6ec8e812e4a3eaaf2aa2b04fe979afbd1bf4a2353f5e77b3c
  Stored in directory: /tmp/pip-ephem-wheel-cache-pk3dj2eg/wheels/ed/48/58/db8714ec3a4c7a2283f34228da5a94b90f0df836be6abec2fd
Successfully built virtualenv
Installing collected packages: six, zipp, importlib-metadata, filelock, importlib-resources, appdirs, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-0.1.dev58+g2bc34d2 zipp-2.1.0
+ /tmp/x/venv1/bin/python -mvirtualenv venv2 -p /tmp/x/venv1/bin/python -vv
50 setup logging to DEBUG [DEBUG report:43]
54 find interpreter for spec PythonSpec(path=/tmp/x/venv1/bin/python) [INFO builtin:44]
54 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:48]
54 accepted target interpreter PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:50]
55 filesystem is case-sensitive [DEBUG info:28]
60 Attempting to acquire lock 139750969670512 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
60 Lock 139750969670512 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
61 get interpreter info via cmd: /usr/bin/python3.6 -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
114 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:85]
114 Attempting to release lock 139750969670512 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
114 Lock 139750969670512 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
115 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
115 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
115 Attempting to acquire lock 139750969670512 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
115 Lock 139750969670512 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
115 get interpreter info via cmd: /usr/bin/python -s /tmp/x/venv1/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:104]
140 write PythonInfo to /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python [DEBUG cached_py_info:85]
140 Attempting to release lock 139750969670512 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
140 Lock 139750969670512 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g2bc34d2/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
140 refused interpreter because version_info differs VersionInfo(major=2, minor=7, micro=17, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
141 no exact match found, chosen most similar PythonInfo(spec=CPython3.6.9.final.0-64, exe=/usr/bin/python3, original=/usr/bin/python3.6, platform=linux, version='3.6.9 (default, Nov  7 2019, 10:44:02) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) within base folders /usr/bin:/usr [DEBUG py_info:236]
141 done in 92ms [INFO __main__:23]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/usr/lib/python3.6/pathlib.py", line 1001, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.6/pathlib.py", line 656, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.6/pathlib.py", line 640, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not PythonInfo

@gaborbernat
Copy link
Contributor

Can you try again?

@asottile
Copy link
Contributor Author

similar (same?) trace

$ bash t.sh 
+ rm -rf venv1 venv2
+ virtualenv venv1
Using real prefix '/usr'
Path not in prefix '/home/asottile/opt/venv/include/python3.6m' '/usr'
New python executable in /tmp/x/venv1/bin/python3
Also creating executable in /tmp/x/venv1/bin/python
Installing setuptools, pip, wheel...
done.
+ venv1/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
Collecting git+https://github.com/gaborbernat/virtualenv@1515
  Cloning https://github.com/gaborbernat/virtualenv (to revision 1515) to /tmp/pip-req-build-3m9oy6u1
  Running command git clone -q https://github.com/gaborbernat/virtualenv /tmp/pip-req-build-3m9oy6u1
  Running command git checkout -b 1515 --track origin/1515
  Switched to a new branch '1515'
  Branch '1515' set up to track remote branch '1515' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Using cached importlib_resources-1.0.2-py2.py3-none-any.whl (32 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting six<2,>=1.12.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Using cached zipp-2.1.0-py3-none-any.whl (4.6 kB)
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-0.1.dev58+g4fde4cd-py2.py3-none-any.whl size=4595205 sha256=df7b19b0733c90b5e49198951da4e7ee80c738d4dbe5dd96a42cd68c4337fc59
  Stored in directory: /tmp/pip-ephem-wheel-cache-qiev4jtf/wheels/ed/48/58/db8714ec3a4c7a2283f34228da5a94b90f0df836be6abec2fd
Successfully built virtualenv
Installing collected packages: zipp, importlib-metadata, importlib-resources, filelock, appdirs, six, virtualenv
Successfully installed appdirs-1.4.3 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 six-1.14.0 virtualenv-0.1.dev58+g4fde4cd zipp-2.1.0
+ /tmp/x/venv1/bin/python -mvirtualenv venv2 -p /tmp/x/venv1/bin/python -vv
53 setup logging to DEBUG [DEBUG report:43]
57 find interpreter for spec PythonSpec(path=/tmp/x/venv1/bin/python) [INFO builtin:44]
57 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:48]
57 accepted target interpreter PythonInfo(spec=CPython3.6.8.final.0-64, exe=/tmp/x/venv1/bin/python, platform=linux, version='3.6.8 (default, Oct  7 2019, 12:59:55) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:50]
58 filesystem is case-sensitive [DEBUG info:28]
62 Attempting to acquire lock 140647458176864 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
62 Lock 140647458176864 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
62 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:70]
63 Attempting to release lock 140647458176864 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
63 Lock 140647458176864 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
63 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
63 refused interpreter because version_info differs VersionInfo(major=3, minor=6, micro=9, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
63 Attempting to acquire lock 140647458176640 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
63 Lock 140647458176640 acquired on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
63 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python [DEBUG cached_py_info:70]
64 Attempting to release lock 140647458176640 on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
64 Lock 140647458176640 released on /home/asottile/.local/share/virtualenv/py-info/0.1.dev58+g4fde4cd/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
64 refused interpreter because version_info differs VersionInfo(major=2, minor=7, micro=17, releaselevel='final', serial=0) != VersionInfo(major=3, minor=6, micro=8, releaselevel='final', serial=0) [DEBUG py_info:226]
64 no exact match found, chosen most similar PythonInfo(spec=CPython3.6.9.final.0-64, exe=/usr/bin/python3, original=/usr/bin/python3.6, platform=linux, version='3.6.9 (default, Nov  7 2019, 10:44:02) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) of /usr/bin/python3.6 within base folders /usr/bin:/usr [DEBUG py_info:237]
64 done in 12ms [INFO __main__:23]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 27, in <module>
    run()
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/__main__.py", line 16, in run
    run_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    CreatorSelector(interpreter, parser),
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 26, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 33, in can_create
    for src in cls.sources(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 54, in sources
    for host_exe, targets in cls._executables(interpreter):
  File "/tmp/x/venv1/lib/python3.6/site-packages/virtualenv/create/via_global_ref/builtin/cpython/common.py", line 31, in _executables
    host_exe = Path(interpreter.system_executable)
  File "/usr/lib/python3.6/pathlib.py", line 1001, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.6/pathlib.py", line 656, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.6/pathlib.py", line 640, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not PythonInfo

@gaborbernat
Copy link
Contributor

I'll be back tomorrow with better tests 😅

@asottile
Copy link
Contributor Author

<3 thanks for this, I'll see if I can make a docker image in the meantime

@asottile
Copy link
Contributor Author

asottile commented Jan 29, 2020 •
edited

here's a Dockerfile:

$ tail -n999 Dockerfile t.sh 
==> Dockerfile <==
FROM ubuntu:bionic
RUN : \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        git-core \
        python3-distutils \
        python3.6=3.6.8-1~18.04.3 \
        python3.6-minimal=3.6.8-1~18.04.3 \
        libpython3.6-minimal=3.6.8-1~18.04.3 \
        libpython3.6-stdlib=3.6.8-1~18.04.3 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
RUN curl https://asottile.github.io/get-virtualenv.py | python3.6 - /venv
RUN : \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3.6 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
COPY t.sh /t.sh

==> t.sh <==
#!/usr/bin/env bash
set -euxo pipefail
rm -rf venv2
/venv/bin/pip install git+https://github.com/gaborbernat/virtualenv@1515
/venv/bin/python -mvirtualenv venv2 -vv

t.sh is chmod +x

@gaborbernat
Copy link
Contributor

This requires some major refactor, as we effectively should be resolving to the system python instead of the virtual environments... and this messes up our current caching. It must be done though, just will take a day or two more FYI.

@gaborbernat gaborbernat self-assigned this Jan 30, 2020
gaborbernat added a commit that referenced this issue Jan 30, 2020
@gaborbernat
Better handling of base executable not found #1515
9d37eb3 
Improve the base executable discovery mechanism:

- print at debug level why we refuse some candidates,
- when no candidates match exactly, instead of hard failing fallback to the closest match where the priority of matching attributes is python implementation, major version, minor version, architecture, patch version, release level and serial (this is to facilitate things to still work when the OS upgrade replace/upgrades the system python with a newer version than what the virtualenv host python was created with),
- always resolve system_executable information during the interpreter discovery, and the discovered environment is the system interpreter instead of the venv/virtualenv (this happened before lazily the first time we accessed and caused reporting that the created virtual environment is of type of the virtualenv host python version, instead of the system pythons version - these two can differ if the OS upgraded the system python underneath and the virtualenv host was created via copy),
@asottile
Copy link
Contributor Author

via #1522

@DRMacIver DRMacIver mentioned this issue Feb 12, 2020
Closed
@benjeffery benjeffery mentioned this issue Mar 25, 2020
Closed
@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@gaborbernat gaborbernat

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gaborbernat @asottile