pipenv install XYZ command sometimes hangs indefinitely at "Locking [packages] dependencies"
#1816
Comments
|
Not that this should happen in any case, but what hardware are you running this on? |
|
@uranusjr Please find the attached hardware information report. |
|
Does this happen consistently? Try |
|
@techalchemy, I ran It remained hung at "Locking [packages] dependencies". Any luck reproducing this on your machine? With tensorflow in particular, it is happening every time for me. EDIT: I tested this out on a friend's computer. He's running Ubuntu 17.10, and he did not experience the problem. Both computers are on the same network. One difference between his computer and mine is that he only installed pipenv rather than pipenv and virtualenvwrapper. I'll spin up a virtual machine to see if this is somehow connected to virtualenvwrapper and/or the environment variables I have set that are shared across both (i.e. $WORKON_HOME). |
|
I'm currently seeing the same problem; my |
|
I see you installed Pipenv against the system Python. Do you happen to have pyopenssl (or something else SSL-related, honestly I can’t remember, it’s part of |
|
I've installed What's weirder is that if I run that command directly ( |
|
Yes, it appears as though I do, but only for my python 2, not for python 3 (and my pipenv instance was installed through the python 2 version of pip): I'll go pop over to my other laptop and into some virtual environments where I can't reproduce this to see if they do not have pyOpenSSL. EDIT: Deleted a couple of paragraphs about another issue I was seeing. That was my error--I was actually in a subdirectory but didn't realize it because without my normal zsh configuration disabled for testing purposes, I no longer had a visual indicator in my prompt of my cwd. As @uranusjr correctly stated "but Pipenv does a lot of environment detection, and is known to work weirdly in an improperly-configured shell."... I indeed had momentarily incorrectly configured my shell. @uranusjr $ python -m pipenv.help output (from the virtual machine)Pipenv version: Pipenv location: Python location: Other Python installations in
PEP 508 Information: System environment variables:
Pipenv–specific environment variables: Debug–specific environment variables:
|
|
@MarenstainBear Which Python do you install Pipenv in? Or do you install Pipenv in another Python, or inside a virtualenv? (Not your project, but Pipenv itself.) The behaviour you mentioned below is indeed curious, but Pipenv does a lot of environment detection, and is known to work weirdly in an improperly-configured shell. I wouldn’t read it too deeply; it’s probably choking on something else. |
|
@roddds Try setting |
|
+1 I'm seeing the same issue trying to install numpy on MacOS. Verified I don't see this issue installing boto3. |
|
@uranusjr this is what I get when checking Resolver output$ PIPENV_PACKAGES=='django -i https://pypi.python.org/simple\ngraphene -i https://pypi.python.org/simple\ngraphene-django -i https://pypi.python.org/simple\npsycopg2 -i https://pypi.python.org/simple\npsycopg2-binary -i https://pypi.python.org/simple\nipython -i https://pypi.python.org/simple\ndjango-manifold==0.1.6 -i https://pypi.python.org/simple\nscipy -i https://pypi.python.org/simple\npython-dateutil -i https://pypi.python.org/simple\ndjango-extensions -i https://pypi.python.org/simple' python /home/rodrigo/.pyenv/versions/3.6.4/envs/val/lib/python3.6/site-packages/pipenv/resolver.py My Pipfile[[source]] url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi" |
|
@roddds You have a superfluous But you had The second |
|
I thought I was seeing the same problem on a fresh install of a Mac (macOS 10.12, Homebrew Python 3), and found several issues related here. It's a beefy machine, I have good Internet access – not sure what causes it to take so long, but in the end the lockfile was written after about three minutes. |
|
Same here. Every time I run |
|
Locking simply takes a long time folks. Because of python dependency management we have to download each package and run I’m going to close this issue for now until someone can actually show me that their process is hung or not doing anything. We are considering ways to improve this process |
|
Would it be feasible to display the package that's currently being locked? I think that would at least help to stop people thinking that Pipenv is hanging. |
I'd suggest printing a message that informs the user that |
|
@slhck I would be interested to know why this has to take so long to begin with. What kind of work does |
|
@anowlcalledjosh I like this suggestion, and have probably wondered what was being locked myself. Not sure why we never thought to print it. @Victor-Savu as I mentioned. compiling something like numpy or tensorflow. |
|
@techalchemy My process, as far as I can tell, is actually hanging. I've let it go for five hours before killing it. Looking at As you can see in this capture of my terminal, it took my system 12.25 seconds to spin up a new python3 virtualenv and use pip to install tensorflow using the command If I instead run the command You can see from this image that after about 59 seconds of actively doing something, the subprocess related to resolver.py stops actively using system resources and, as far as I can tell, will never complete. When I took this screenshot, the process had been running for 16 minutes.
|
|
@MarenstainBear Pipenv spawns a subprocess inside the virtualenv to perform locking. Would you be able to verify if the subprocess (the virtualenv (even more ideally you can try finding out if the resolver reaches its end) |
|
@MarenstainBear That’s good(?) to hear, at least it means it’s debuggable. How far did it manage to get inside |
|
Yeah and for how long was it running? To me that still sounds like what I said before — resolving the dependency graph which is slow |
|
@techalchemy @uranusjr I'm going to dig into what's going on in the resolve_deps function this weekend. In the meanwhile, here are the results of the process I started yesterday. After 24 hours, it had not resolved, and it really wasn't using any system resources, so I think it was not trying to resolve dependencies and instead just hanging. (I killed the process in order to see the output of the Command exited with non-zero status 1 |
|
@techalchemy what does it mean to "heat up the cache"? |
|
@MarenstainBear thanks for doing all that. Your issue is definitely a bug. Others in the thread I am not yet convinced. Can you install from master and see if we have this fixed? |
|
@techalchemy I installed from master at commit 8a67a21 (dated Sat Mar 31 01:13:26 2018 -0400). My issue was not resolved. I saw the same behavior here in version 11.9.1 as I was seeing in 11.9.0. I did briefly try a different way to debug this problem using strace. The command |
|
Next release should help with this a bit. Should have it in a few hours. |
|
I was experiencing the same problem when installing pandas. As I had faced problems with pandas and the latest version of pip, so I decided to downgrade pip to 9.0.3 and the issue with pipenv also got solved. |
|
pipenv install scrapy
|
|
I had the same issue. Here's where it was hanging for me: The problem was something to do with pip trying to write something to it's cache. I was able to resolve it by clearing my pip cache - which for Mac users is Thanks to all the contributers to this issue! |
|
Clearing |
|
Same stuff here, pip env hanging on Mac OS. I've tried to create a simple Building wheels for collected packages: itsdangerous, MarkupSafe
Running setup.py bdist_wheel for itsdangerous: started
Running setup.py bdist_wheel for itsdangerous: finished with status 'done'
Stored in directory: /Users/renzo/Library/Caches/pipenv/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5
Running setup.py bdist_wheel for MarkupSafe: started
Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
Stored in directory: /Users/renzo/Library/Caches/pipenv/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46So what worked for me was rm -rf ~/Library/Caches/pipenvReally strange behaviour. Found this behaviour on after upgrading python from 3.6.3 to 3.7.0 |
|
|
|
Error: no such option: --clear |
|
Upgrade to the latest version first. edit unless this isn’t released and is only in master. In that case wait until I cut a release probably this week, or use the version available in master :D can’t remember if this actually made the cutoff... |
|
@techalchemy |
|
For this particular problem, |
|
|
Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here pypa/pipenv#2284 here pypa/pipenv#1816 and here pypa/pipenv#356 making it extremely unusuable For that reason instead of switching to an alternative solution, I have decided to pin all the dependencies so that this project doesn't start failing with breaking changes in the future I will probably come back here and generate lock and use semver minors/patches only once pipenv have matured enough, good luck to its developers!
Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here pypa/pipenv#2284 here pypa/pipenv#1816 and here pypa/pipenv#356 making it extremely unusuable For that reason instead of switching to an alternative solution, I have decided to pin all the dependencies so that this project doesn't start failing with breaking changes in the future I will probably come back here and generate lock and use semver minors/patches only once pipenv have matured enough, good luck to its developers!
Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here pypa/pipenv#2284 here pypa/pipenv#1816 and here pypa/pipenv#356 making it extremely unusuable For that reason instead of switching to an alternative solution, I have decided to pin all the dependencies so that this project doesn't start failing with breaking changes in the future I will probably come back here and generate lock and use semver minors/patches only once pipenv have matured enough, good luck to its developers!
|
still NOT work for me for macOS High Sierra 10.13.6 Hope fix this |
|
Happening in this alpine linux based Dockerfile (child of this): With this Pipfile: |
hangs forever. Pipfile: |
|
Lockfiles which only contain stuff that is on your local machine are just as unhelpful as not providing any information. Unless you are offering new debugging information or a new thing that we can take and reproduce, please be mindful about issue tracker noise. |
|
Those are simple dependency projects that I can easily install manually running |
|
Maybe this could shed some light: Waiting forever... Suppose it's a network issue, now why this works then? : |
|
After a lot of attempts (installing dependencies one by one, avoiding local dependencies, avoiding editable dependencies, installing with skip lock then locking, etc) I was unable to find out any pattern. Sometimes it works in a sensible amount of time, sometimes it aborts after a few minutes with a timeout error, sometimes it hangs "forever" (which definition is contingent on my diminishing patience). I assume my problem is related to the many issues reporting some variant of "locking is too slow". |
|
Again, please stop taking random guesses and posting random snippets of your output. If you want help with a specific problem, fill out an issue with the entire issue template. We are not going on a wild goose chase with anyone in this issue while they selectively post output that is mostly unrelated to pipenv. We have seen what the screen looks like when it hangs. Pasting that to us is not useful. |
|
Ok, if you think failing to install numpy in a clean environment is a
random problem I know I'm taking my chances when using pipenv, thanks for
the heads up and I'll shut my mouth up.
…On Wed, Jul 25, 2018 at 6:27 PM, Dan Ryan ***@***.***> wrote:
Again, please stop taking random guesses and posting random snippets of
your output. If you want help with a specific problem, fill out an issue
with the entire issue template. We are not going on a wild goose chase with
anyone in this issue while they selectively post output that is mostly
unrelated to pipenv. We have seen what the screen looks like when it hangs.
Pasting that to us is not useful.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1816 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ACtq-WfYHCgtRnbIu_yljQV7m0JvwxA7ks5uKOKqgaJpZM4S0kt->
.
|
|
@memeplex it's not about it being a random problem, it's about the output not being helpful for troubleshooting. In no place did I call your problem random -- I told you specifically what you can do if you are interested in having a productive conversation about the issue you're facing, since it's impossible to troubleshoot or diagnose a system with selected fragments of terminal output which includes only messages we ourselves wrote and which don't actually include any errors or any way for us to reproduce it. So far we know:
We triage a considerable number of issues, which is why we use issue templates to help us gather this information quickly to help assess what might be happening. When people simply paste '[locking]...' and tell us 'this takes forever', we feel for you, but it's kind of an early indication that you aren't actually here to be productive. I try to give the benefit of the doubt and urge people toward filing an issue with the issue template and a reproducible test case, but if you can't do that I can only assume your intention wasn't actually to troubleshoot and that you only wanted to be negative. That is not really welcome on the issue tracker, there are plenty of other places you can do that. This tracker is used for identifying and solving problems with the codebase, not for piling on anytime you see something that you experienced. The rule is essentially that if you can't or aren't willing to contribute anything beyond restating something and telling us that we need to fix it pronto, you should probably refrain from posting. We are volunteers and we do this in our free time; if you are coming to the tracker you are directly asking for us to spend our time on you. We are usually able to do that, but we ask in return that you follow our process to help us triage efficiently. |
This comment has been minimized.
This comment has been minimized.
|
If you feel you have something new to add on this topic please create a new issue and fill out the issue template in full. We are aware of the problems and their various manifestations, and we have several fixes in process |
and others
Similar to these issues:
Summary
When using
pipenv install XYZ, the process hangs indefinitely at "Locking [packages] dependencies".This doesn't happen every time, and may be related to which package is being installed. This morning it is happening every time I try to install tensorflow. Last night it was happening for numpy as well, but today numpy is fine.
The hanged response seems to last indefinitely (though I did manually terminate the process after an hour). The Pipfile.lock is not written, but the Pipfile is written and the expected packages are installed and listed when I run
pipenv run pip freeze.I was able to reproduce this using various permutations of:
pipenv --threeandpipenv --twoI am also able to reproduce it by simply running
pipenv lockorpipenv updateif I have tensorflow (or whatever package is currently causing me trouble) listed in my Pipfile.$ python -m pipenv.help output
Pipenv version:
'11.8.3'Pipenv location:
'/home/mary/.local/lib/python2.7/site-packages/pipenv'Python location:
'/usr/bin/python'Other Python installations in
PATH:2.7:/usr/bin/python2.72.7:/usr/bin/python2.73.5:/usr/bin/python3.5m3.5:/usr/bin/python3.52.7.12:/usr/bin/python2.7.12:/usr/bin/python23.5.2:/usr/bin/python3PEP 508 Information:
System environment variables:
MANDATORY_PATH_LXSESSION_PIDXDG_GREETER_DATA_DIRPROJECT_HOMELC_CTYPEPYTHONDONTWRITEBYTECODEXDG_CURRENT_DESKTOPXDG_SESSION_TYPELOGNAMEXDG_SEATPATHXDG_VTNRQT_PLATFORM_PLUGINPYTHONUNBUFFEREDVIRTUALENVWRAPPER_SCRIPTZSHDISPLAYSSH_AGENT_PIDLANGTERMSHELLXDG_SESSION_PATHXAUTHORITYLANGUAGESHLVLQT_LINUX_ACCESSIBILITY_ALWAYS_ONQT_QPA_PLATFORMTHEMESESSION_FOLDERQT_ACCESSIBILITYWINDOWIDLIBVIRT_DEFAULT_URIHOMEXDG_SESSION_DESKTOPSAL_USE_VCLPLUGINXDG_RUNTIME_DIRWORKON_HOMESSH_AUTH_SOCKVTE_VERSIONGDMSESSIONVIRTUALENVWRAPPER_WORKON_CDXDG_SEAT_PATHPIP_PYTHON_PATHXDG_SESSION_IDDBUS_SESSION_BUS_ADDRESS_VIRTUALENVWRAPPER_HOOK_DIRVIRTUALENVWRAPPER_PROJECT_FILENAMEDESKTOP_SESSIONLSCOLORSXDG_CONFIG_DIRSDEFAULTS_PATHXDG_CONFIG_HOMEOLDPWDLS_COLORSGDM_LANGXDG_DATA_DIRSPWDXDG_MENU_PREFIXLESSPAGERUSERPipenv–specific environment variables:
Debug–specific environment variables:
PATH:/home/mary/bin:/home/mary/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/gamesSHELL:/usr/bin/zshLANG:en_US.UTF-8PWD:/home/mary/Development/Projects/poor_mans_smart_cameraContents of
Pipfile('/home/mary/Development/Projects/poor_mans_smart_camera/Pipfile'):Expected result
I expect the step in which the Pipfile.lock is generated to take at most a minute.
I have good hardware with lots of free memory and processing power, so I don't think this is a resource constraint. My internet speed is 60 Mbps, so I doubt it is a networking issue when requesting packages.
Actual result
verbose_output.txt
While this screenshot doesn't show verbose output like the file above, it does visually demonstrate what I am experiencing.
Steps to replicate
pipenv --threeorpipenv --two.pipenv install pylint.pipenv install tensorflow.Pipfile
The text was updated successfully, but these errors were encountered: