2018.11.26: pipenv emits pythons errors when creating a virtualenv

[command-tab]

Issue description

Possibly related to #3229, pipenv continues to throw errors when creating a virtualenv:

$ pipenv --three
['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess\n    combine_stderr=combine_stderr)\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess\n    return subprocess.Popen(cmd, **options)\n', '  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__\n    restore_signals, start_new_session)\n', '  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\n', "FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/pythonz': '/usr/local/bin/pythonz'\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/contextmanagers.py", line 150, in spinner\n    yield _spinner\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run\n    write_to_stdout=True\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess\n    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))\n', 'TypeError: write() takes exactly one argument (3 given)\n']
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess
    combine_stderr=combine_stderr)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess
    return subprocess.Popen(cmd, **options)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/pythonz': '/usr/local/bin/pythonz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 208, in cli
    clear=state.clear,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 574, in ensure_project
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 516, in ensure_virtualenv
    ensure_python(three=three, python=python)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 397, in ensure_python
    path_to_python = find_a_system_python(python)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 360, in find_a_system_python
    python_entry = finder.find_python_version(line)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 114, in find_python_version
    major=major, minor=minor, patch=patch, pre=pre, dev=dev, arch=arch, name=name
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 396, in find_python_version
    ver = next(iter(self.get_pythons(sub_finder)), None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 279, in get_pythons
    reverse=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 277, in <genexpr>
    (p for p in self._filter_paths(finder) if p.is_python),
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 264, in <genexpr>
    pth for pth in unnest(finder(p) for p in self.path_entries if p is not None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 251, in unnest
    for el in target:
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 264, in <genexpr>
    pth for pth in unnest(finder(p) for p in self.path_entries if p is not None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/mixins.py", line 121, in find_python_version
    for child in unnest(self.pythons.values())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 531, in pythons
    for path, entry in self.children.items():
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/cached_property.py", line 35, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 501, in children
    for child_key, child_val in self._gen_children():
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 493, in _gen_children
    entry = PathEntry.create(path=child, **pass_args)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 575, in create
    _new = cls(**creation_args)
  File "<attrs generated init b90d7581ea07925e94241736776cf96c889eb52c>", line 16, in __init__
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 518, in get_py_version
    py_version = PythonVersion.from_path(path=self, name=self.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 395, in from_path
    py_version = get_python_version(path.path.absolute().as_posix())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 68, in get_python_version
    combine_stderr=False, write_to_stdout=False)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run
    write_to_stdout=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess
    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))
TypeError: write() takes exactly one argument (3 given)

Expected result

Expected no error logging while simply creating a virtualenv.

Actual result

virtualenv creation fails.

Steps to replicate

$ pipenv --three

---

pipenv --support also emits these errors and fails to collect support information!

[frostming]

It seems /usr/local/bin/pythonz is no longer a valid path.

[command-tab]

Hmm. Should it ever have been? I've never used pythonz directly. I thought it was just a pipenv dependency that it managed when I installed pipenv.

For kicks, I just uninstalled and re-installed pipenv 2018.11.26 and am still having the issue.

[techalchemy]

Can you provide the output from pipenv --support? And to confirm, no it shouldn’t have been

[techalchemy]

Oh nvm just saw your message. I just want to confirm there is no rogue version of pipenv on your path somewhere. Can you check which pipenv and python -m pipenv --version

[command-tab]

$ which pipenv
/usr/local/bin/pipenv

python -m pipenv --version returned No module named pipenv because I have both Python 2 and Python 3 installed via Homebrew, and python runs Python 2.7. Per a prior recommendation here, I have pipenv installed under Python 3. So I substituted in python3 and got:

$ python3 -m pipenv --version
pipenv, version 2018.11.26

Thanks!

[techalchemy]

Ah ok. And yeah the bug makes sense. I’ll just tag it upstream and make sure it gets fixed

[jasonrhaas]

Any word on when a fix will be released for this? I am having this problem its preventing me from installing the pipenv requirements. I have tried @commandtab 's workaround with no such luck.

[jarshwah]

I can not get pipenv to work at all with the current version.

$ which pipenv
/Users/josh/.pyenv/shims/pipenv

$ python -m pipenv --version
pipenv, version 2018.11.26

I tried installing with python3.7 from homebrew, and same issue.

It seems like there's no urgency here, or that it's not a big problem. Am I missing something, or is pipenv completely broken with the current version?

[command-tab]

The current release sure seems hosed to me. Another temporary workaround I saw in a past issue was to specifically pass in a python path like so:

pipenv --three --python=`which python3`

I've been doing this with success for the time being 😬

[jarshwah]

That workaround wasn't working for me. I had to downgrade to 2018.10.13 and then use the workaround on that thread :

pipenv install -d --python=$HOME/.pyenv/versions/3.7.1/bin/python

As far as I can tell, pipenv has been completely broken since October.

[command-tab]

@techalchemy Any word on an upstream tag fix? pipenv virtualenv creation still requires workarounds for invocation.

Thank you ❤️

[techalchemy]

This was a bit rough with all the other fixes that needed to be integrated but I think CI will pass now and this will be fixed with #3330 -- sorry for the trouble

[pavithrarajs]

Ah ok. And yeah the bug makes sense. I’ll just tag it upstream and make sure it gets fixed

Still issue persists, any idea when this fix will be released.

[Alan-Solitar]

Same issue here.

Can confirm @command-tab's workaround still works.

[gormster]

This definitely still happens with homebrew pipenv and python… kind of surprised there isn't a test to cover it seeing how long this issue has been around…

[pcolmer]

I can also confirm that this problem is still happening, despite this GitHub issue being closed.

Do the maintainers need a new issue to be created?

[dahifi]

Same issue with Ubuntu 18 LTS installed in Windows Subsystem for Linux.

Workaround works for me.

[pcolmer]

@techalchemy this issue still seems to exist. Can you please either reopen it or make it clear that you want a new issue to be created?

Thanks.

[AlphaWong]

Same issue with Ubuntu 18 LTS installed in Windows Subsystem for Linux.

same here

[sanderbrauwers]

@techalchemy please take a look again

[Varkal]

Same issue here on macOs Mojave and python 3.7

[edsu]

Me too:

• Catalina
• Python 3.7.

Everything works fine once I comment out this line in my Pipfile:

[requires]

#python_version = "3.7.4"

[JarredStanford]

Me too:

• Catalina
• Python 3.7.

Everything works fine once I comment out this line in my Pipfile:

[requires]

#python_version = "3.7.4"

ahhhh this worked for me, thank you!

[josh-dix]

@JarredStanford @edsu
This worked for me as well on WSL in VSCode (I didn't try an outside shell as it worked fine for a different project).

@command-tab
Adding "--python=which python3" also worked. I did not try adding "--three" and it seems not necessary for my use case / version.

After getting both workarounds and I was able to get it to work, I removed them and tried a third time to ensure the environment was still causing an issue (narrator: it was).

@techalchemy
I'm thinking this may not be "fully fixed" in 3330? It could be a tangential issue causing the same symptoms. In my case I'm using WSL and wondering if 'python3' and 'python3.exe' both being in my WSL path seems suspicious.

Would you think it best I open a new issue, would you like me to run additional diagnostics, or something else?

[lencshu]

I m using wsl2 with arclinux (manjaro)
It seems pipenv is calling the python in host env windows to install
so we need to define where to find the python with:
pipenv --python=<PATH_TO_PYTHON>
or
pipenv --python=which python3

works for me.

[wbierman]

Can confirm this happens under WSL with Ubuntu 18.04 LTS. It seems like pipenv is going in search of a python interpreter rather than using the one that it is being executed under, and in-so-doing is traversing PATH in backwards order. I have multiple python environments installed, and whenever I manually remove one from PATH, pipenv will find a different one and complain about that one instead.

[brunogirin]

I'm seeing the same issue under WSL with Ubuntu 18.04 LTS. pipenv install --python $(which python3) works but when I subsequently do pipenv shell, it cannot find the dependencies it should have downloaded.

[johansmolinski]

Also I am affected, using WSL 1 and Ubuntu 18.04. Workaround --python $(which python) works.

[mr-smidge]

The reason for the last part of the exception, i.e. this bit:

  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 68, in get_python_version
    combine_stderr=False, write_to_stdout=False)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run
    write_to_stdout=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess
    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))
TypeError: write() takes exactly one argument (3 given)

... is because there is a bug in the call to print an error when handling the first exception. This bug appears to have already been fixed in master back in Jan 2019 - 574fe73#diff-a59595db75020aeb9f688d6a0b4818e6L162. But since the latest version of pipenv is 2018.11.26, I imagine most don't have a fixed version.

One can patch it manually. Open /usr/local/lib/<your-python-version>/site-packages/pipenv/vendor/vistir/misc.py, navigate to line 162, and change this:

sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))

... to this:

sys.stderr.write(f"Error {exc} while executing command " + " ".join(cmd._parts))

You will now get a more accurate exception report. In my case, the issue is to do with WSL Linux-based pipenv detecting Windows-based Python installations and having trouble running them, which is kind of obvious:

Error [Errno 8] Exec format error: '/mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe' while executing command /mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe -c import sys; print(['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/contextmanagers.py", line 150, in spinner\n    yield _spinner\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 314, in run\n    write_to_stdout=True\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess\n    combine_stderr=combine_stderr)\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess\n    return subprocess.Popen(cmd, **options)\n', '  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__\n    restore_signals, start_new_session)\n', '  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\n', "OSError: [Errno 8] Exec format error: '/mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe'\n"]

For WSL users, I believe the issue can be alleviated by ensuring that any requirements on your Python version in Pipfile are exactly matched by a Linux-based Python installation, e.g.

[requires]
python_version = "3.6.8"  # Make sure this exactly matches an installed version, or remove it.

Hope this helps.

[paramkpr]

Not sure how much this helps, but deleting the Pipfile, running pipenv lock, and then putting the contents of the old Pipfile back in and running pipenv install works for me permanently.