Lock updating is very slow #1914
Comments
|
We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896 npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass Closing to track in the other issues. |
|
Same here im trying it out right now and installing django is already at 10 minutes and going |
|
seems that if you try and install a couple at a time its slow but if you do one at a time works a bit faster |
|
If you can provide a Pipfile that reproduces the slow behavior it would be helpful -- thanks |
Is this still the case when the package has its own |
No. Pipenv is an application depedency resolution tool. When a dependency is used as a Python package by another application, it is no longer an application. Its dependencies are determined by setup.py (or setup.cfg or whatever you use to upload to PyPI). Loading dependencies from a lock file is a sure route to dependency hell, we will likely never ever do that. |
|
It's still super slow |
|
@iddan Thanks for the reminder, Captain Obvious! |
|
Sorry, as an OSS maintainer I know how sometimes issues can get dismissed because of age. Just wanted to state that even the issue received discussion it’s still relevant |
|
@techalchemy note of |
|
@techalchemy what if it takes 20 minutes with my brand new mac pro? |
As far as I gather, the overwhelming benefit of pipenv is the assurance dependencies will play nicely together — not just for you, but for anyone later dealing in your code. The product of that assurance, the lock file, absolutely takes more time than anyone expects or desires, including the devs — see #2200. However, I think you can also understand the opportunity pipenv has to shepherd well-meaning devs in the Python community at large toward a workflow imposing less head-scratching on future contributors — who might've only been visitors had they given up during the "figure out how to setup the dev environment" stage; and less hands thrown up by future maintainers — who might've only been drive-by PR authors had they given up during the "seriously screwing around with deep project internals" stage. If Better to leave it available only as an env var, or some other method whose application rests squarely in the "your user-specific local config, your fault" territory, allowing pipenv to overcome the passing phase of lockfile generation slowness without giving up the truly beneficial cultural shift toward explicitness over implicitness in package management. Python's incredibly vast standard library is an enormous asset, whose history has undergone many eras of imposing consistency. That most standard packages play nicely together is an enormous feat involving consideration over many years by many people. One day, that play-nice-ability will extend to most Python projects encountered on the web — far, far from the stdlib, and with far, far fewer PEPs required (and far, far fewer BDFLs vacating in frustration). The impact of such a unilaterally buttery experience is hard to measure, but some current languages did refuse to compromise conceptual integrity for immediate convenience... and oh, the places they'll Go. So yes, generating the lockfile is slow, and yes, it's frustrating when you only wanted Lockfile generation is slow only because it's making explicit what we've all taken for granted. But because it hurts, we will adjust things so it doesn't. We broke our arm because we pushed ourselves doing things we believed in. Sure, we can avoid the pain by never using that arm again— or we can put it in a cast while it heals. I'd be the last person to tell you not to make pipenv convenient for yourself today (otherwise I'd be a shitty developer — though, the jury's still out), but I implore you to see the frustration of lockfile generation as a growing pain while the Python community develops into a strong, healthy body with more fully-functioning limbs than one really expected when removing a cast. We made it to Python 3. Let's make it to dependency management in the stdlib. |
|
This took 38 minutes on my machine to create the lock file. Running Windows 10 and using Python 3.7 |
|
Only numpy and pillow were already installed and it took <1 seconds to install numba and 25 minutes to lock it. Does pipenv forcibly compile every lib on lock or how does this work? |
|
FYI persistent skip lock is just waiting on someone to flip the switch for |
|
TLDR; Typical Pandas-datareader install fails on first attempt, possibly cause of Using version 2018.11.26 $ pipenv --version
pipenv, version 2018.11.26Contents of Typical Results: 144.82 real 33.68 user 5.77 sys $ time pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (0fdb67) out of date, updating to (a65489)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (0fdb67)!
Installing dependencies from Pipfile.lock (0fdb67)…
🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/15 — 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
144.82 real 33.68 user 5.77 sysInvoking w\ Results: 4.54 real 5.33 user 0.87 sys $ time pipenv install -r requirements.txt --skip-lock
Requirements file provided! Importing into Pipfile…
Installing dependencies from Pipfile…
🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
4.54 real 5.33 user 0.87 sys |
|
I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too. Although |
|
It won't even finish on my machine: |
|
@black-snow I'd recommend trying it in a different shell. Without diving too deep into things, it seems like pexpect (a library for programmatically interfacing with interactive CLI programs) is used to detect the shell pipenv is being executed from, and that might be stalling. pexpect kinda seems overkill for such a thing, but I'm not aware of the whole context. |
|
@theY4Kman thanks for the advice. pipenv is working fine on another pc with same ubuntu and bash version ... |
|
Read this while waiting for pipfile to lock...:) Would be great if there was a solution. |
|
Seems like we need some kind of API to parse and cache Python packages deps and distribute it in a machine-friendly format. So we will no longer need to download entire packages and parse them. |
|
I believe bundler and ruby gems maintains and uses somthing like that. |
|
Java also has POM files (XML) which contain package dependencies and other information about the package. They are uploaded separately from the compiled JARs. |
|
Every newer package manager has some separate meta files (npm/yarn, composer, gradle/maven, cargo, ruby gems/bundler, ...). |
|
related issue: |
|
You can get dependency information from PyPi without downloading the entire bundle (see PEP 566, which superseded PEP 426). |
|
@techalchemy did you see the comment above? |
|
This is pretty consistently happening, essentially Understand the I'd at least like to know if it's me doing something wrong, or just a "feature" of pipenv. |
|
In our project |
|
I'm trying to install PyTorch and it took 20 minutes to lock and then it pulls the following error The error is unreadle, no idea what went wrong. Installing with pip in the environment work fine! This is really a show stopper. Going back to requirements.txt... |
|
This is the workaround I use for now: Then |
|
@awhillas Pretty sure the last line says all you need:
|
|
Locking dependencies is important so I don't think "skip lock" is a lasting solution. At the same time, I simply don't buy that "locking dependencies" (whatever that could possibly entail under the hood) is maximally optimized as it is now and functionally requires many minutes or hours to complete. Indeed, my pipenv lock ran for several minutes on a Pipfile that has a laughable 5 dependencies before failing—stack attached at bottom—and during that time used only 10–15% of available CPU and a little sip of memory. Can we at least put forth some group effort to profile and determine the bottlenecks? I have a feeling there are just some silly low hanging fruit in there just waiting to take this process into reasonable runtime.
For Pipfile: |
|
As an addendum to previous comment, running |
|
FYI I found tensorflow is the culprit of the slow/timing out lock, if that helps profile pipenv! (Still consider this a pipenv problem...) |
|
Tensorflow is one of many packages that can cause pipenv to basically become a useless tool. I like the suggestion of group profiling, though. I think that's an excellent idea to start tacking this problem. Just to re-iterate, PEP 566 enabled enumerating dependence (via pypi) without loading the entire source, which may be helpful: #1914 (comment) |
|
@brandonrobertz from what I see, downloading all packages of the dependencies is where most time is spent. This also has been confirmed before. How to verify this:
Here's the Dockerfile I used to test this: FROM python:3.6
ENV WORKDIR=/work/
WORKDIR /work/
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install pipenv
RUN PIPENV_SKIP_LOCK=true pipenv install scipy
RUN date
RUN pipenv lock
RUN date
RUN rm Pipfile.lock
RUN pipenv lock
RUN date |
|
@shtratos Yeah, that makes sense and others have suggested that in this issue thread. Downloading and parsing dependencies is expensive. Some of those steps seem like they can be eliminated by pulling from the pypi dependency API. For some libraries this will probably not work, due to poor quality and bad practices (not using setup.py or requirements.txt). But since some of the major offenders seem to be very popular libraries (tensorflow, numpy), implementing this with a fallback to the super slow process might be a good path forward. |
|
Can you point me in the direction of where to find that code? I could take a stab at parallelizing it in a fork. |
I doi not think so, it works fine on my machine, the problem seems to be more general than that |
|
In my case the problem seems to be pylint. It always hangs on locking when simply running
|
3 out of the 4 other issues are now closed and the remaining one hasn't seen any activity since 2018. This issue still persists, so maybe re-opening it would be a good idea? |
I don't think that is still true for wheels, which should be the majority of packages now? |
|
I know I at least have to build the wheel for dlib every time, which is horrendous. The process of resolving dependencies should be cached somewhere on a per-package-version basis, even if it requires another remote lookup on the client each time to get the tree (it shouldn't, you could just cache it locally too after the fact). For example, package repos could easily store resolved dep trees. The additional network hit for the cached resolved dep tree would always be faster than downloading an entire package and computing. |
|
FWIW I've removed pipenv from all my projects (this being the one main reason, there are others).
|
|
Please reopen this issue. |
|
It's still super slow. |
|
Could we signal that this is still a problem by re-opening this issue? |
|
@lyle-nel I don't think re-opening this specific issue would help since the content of it is dated, for example,
This is no longer true about |
|
For what it's worth, it still feels too slow. Is there a different issue where this is tracked? |
|
@truekonrads and others -- I am locking conversation on this issue because the originally issue from 2018 of why it was slow has been resolved. For all of the people that come here to comment in the modern day that |
and others
Updating a lockfile can be very slow. A standard
pipenv lockeasily takes well over a minute for me:This means every time I need to install, uninstall or upgrade a package I need to take a 2-minute break to wait for pipenv to finish updating its lockfile. I'm not sure why that is; yarn and npm seem to perform a similar task but only take seconds, even for projects that have many many more dependencies.
The text was updated successfully, but these errors were encountered: