pipenv install is very slow

[Diggsey]

Running pipenv install after changing one dependency takes about ~5 minutes for me, on a windows 10 machine with an SSD.

The vast majority of that time is spent inside Locking [packages] dependencies...

It seems like there might be some quadratic-or-worse complexity in this step?

I've included most of our pipfile below, but I had to remove some of our private repo dependencies:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
alembic = "==0.8.4"
amqp = "==1.4.7"
analytics-python = "==1.2.5"
anyjson = "==0.3.3"
billiard = "==3.3.0.20"
braintree = "==3.20.0"
celery = "==3.1.18"
coverage = "==4.0.3"
docopt = "==0.4.0"
eventlet = "==0.19.0"
flake8 = "==3.0.4"
Flask-Cors = "==2.1.2"
Flask-Login = "==0.3.2"
Flask = "==0.12.1"
funcsigs = "==0.4"
fuzzywuzzy = "==0.12.0"
gcloud = "==0.14.0"
html2text = "==2016.9.19"
itsdangerous = "==0.24"
Jinja2 = "==2.8"
jsonpatch = "==1.15"
jsonschema = "==2.5.1"
PyJWT = "==1.4.2"
kombu = "==3.0.30"
LayerClient = "==0.1.9"
MarkupSafe = "==0.23"
mixpanel = "==4.3.0"
mock = "==1.3.0"
nose-exclude = "==0.4.1"
nose = "==1.3.7"
numpy = "==1.12.1"
pdfrw = "==0.3"
Pillow = "==4.1.0"
pusher = "==1.6"
pycountry = "==1.20"
pycryptodome = "==3.4.5"
pymongo = "==3.2"
PyMySQL = "==0.7.4"
python-dateutil = "<=2.5.1"
python-Levenshtein = "==0.12.0"
python-magic = "==0.4.6"
python-coveralls = "==2.9.0"
pytz = "==2015.6"
raygun4py = "==3.1.2"
"repoze.retry" = "==1.3"
requests = "==2.8.1"
sendgrid = "==2.2.1"
slacker = "==0.7.3"
SQLAlchemy-Enum34 = "==1.0.1"
SQLAlchemy-Utils = "==0.31.6"
SQLAlchemy = "==1.1.9"
typing = "==3.5.2.2"
twilio = "==5.6.0"
Unidecode = "==0.4.19"
voluptuous = "==0.8.11"
Wand = "==0.4.4"
watchdog = "==0.8.3"
Werkzeug = "==0.12.1"
wheel = "==0.24.0"
WTForms = "==2.0.2"
xmltodict = "==0.9.2"
zeep = "==0.24.0"

[nateprewitt]

Hey again @Diggsey, this is due to the way we're writing changes right now. I have these changes ready to merge but they're breaking for the projects.py API so we're going to hold off until the next major release. Hopefully we'll have this up and ready in the next few weeks. I'll leave this open to track the issue for now.

[kennethreitz]

We sprinted on this together at PyCon. It'll be faster soon.

[NicolasWebDev]

Right now for me it is not slow, it is freezing...

A pipenv install my_package or a simple pipenv install does not give me any output, after 20 minutes.

EDIT: Confirmation, still nothing after a few hours. Is it the same problem? Usually it was slow, but it ended after 5 to 10 minutes.

[nateprewitt]

Hey @NicolasWebDev, what version of pipenv are you using? Also do you have delegator.py installed on your system separately? If so, what version is that at? This was an issue that should have been resolved in v3.6.0.

If everything above is up to date, could you please provide the contents of your Pipfile? Thanks!

[NicolasWebDev]

Hi @nateprewitt, you were right, I was on v3.5.x. Updating to 4.1.1 resolved the freezing issue. Now it is still taking 5 minutes, but at least it is usable!

Sorry for the noise, I always forget that packages installed through pip are not automatically updated.
Thanks!

[nateprewitt]

Glad you got things resolve @NicolasWebDev! We're working on getting this sped up more, hopefully #373 will be a step closer in the next release.

[nateprewitt]

@Diggsey @NicolasWebDev, I've just released 4.1.2 which should have these speed improvements added. There's still some work to do here, but this will at least expedite the initial bootstrap time for pipenv.

[Diggsey]

@nateprewitt Thanks for the update, pipenv does seem faster for me now, but it still takes several minutes just to run pipenv lock, even when nothing has changed - it still waits in Locking [packages] dependencies... for the vast majority of that time.

[nateprewitt]

@Diggsey, a lot of that time is because you're downloading a huge number of files in that Pipfile. It seems like you're pinning all of your dependencies too. Are you directly importing all of these into your project or are some dependency requirements of the others?

[Diggsey]

@nateprewitt We could remove some of them, but the majority are direct dependencies - why does it need to download all of the dependencies every time it generates the lock file?

[nateprewitt]

We need to determine everything that it installs as dependencies. In order to get that, we download each package and determine what an installation looks like at lock time. This allows us to appropriately pin everything in the Pipfile.lock. Without downloading, there isn't a reliable way of checking sub-dependencies and resolving range dependency declarations.

[Diggsey]

Given that most of the packages are going to stay the same over time, would it be possible to cache the downloaded packages?

[kennethreitz]

@Diggsey want to look into that for us?

[uranusjr]

This might be a silly question, but doesn’t Pip already do package caching?

[kennethreitz]

I'm under the impression that it does.

[marctc]

Can pipenv use the pip cache system or it has to be implemented from the scratch?

[kennethreitz]

Pipenv just runs pip, so the cache should automatically be used.

[kennethreitz]

fixed! lock is wicked fast now.

[tilgovi]

Oh, thank you. I think that was the only thing holding my back from pushing everyone over to pipenv at work.

[Diggsey]

Wow, nice, that was literally more than a 100x speedup for me, and it also caught a dependency conflict that the previous version didn't catch!

What would be useful is a verbose flag for pipenv lock - I was only able to diagnose the dependency conflict by editing piptools/logging.py to enable verbose logging, but once I did that it gave a very clear indication of what was going on.

[tricoder42]

I'm probably missing something :) Where is it fixed? Latest release is 4 days ago, so I installed latest version from master. However, pipenv install is still slow.

I tried:

• install pipenv the fancy way ⚡️ 🍰 ⚡️
• use both latest published version of pipenv and latest version from master
• install single package

Using latest stable version (5.3.5.), it takes 3:40 to install one package:

∙ time pipenv install --dev raven
Installing raven...
Collecting raven
  Using cached raven-6.1.0-py2.py3-none-any.whl
Collecting contextlib2 (from raven)
  Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Installing collected packages: contextlib2, raven
Successfully installed contextlib2-0.5.5 raven-6.1.0

Adding raven to Pipfile's [dev-packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
pipenv install --dev raven  10,11s user 2,77s system 5% cpu 3:40,04 total

Using version from master, it still hangs (one package, +10 minutes)

EDIT: It has just finished:

pipenv install graphene_django  8,03s user 1,28s system 1% cpu 11:23,11 total

Any ideas? Thanks a lot!

[kennethreitz]

Sometimes dependencies take a while to install, especially if they have c compilations. Want to share your Pipfile?

[tricoder42]

I understand sometimes it takes a while, but it was slow from the beginning. Just curious if it's a problem at my side.

Here's my Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
pytest-django = "*"
pytest-testmon = "*"
pytest-watch = "*"
django-debug-toolbar = "*"
raven = "*"

[packages]
dj-database-url = "*"
Django = "*"
djangorestframework = "*"
gunicorn = "*"
newrelic = "*"
psycopg2 = "*"
requests = "*"
whitenoise = "*"
graphene-django = "*"

[kennethreitz]

psycopg2 might be taking a while, as it may be compiling from source. Everything else should be nice and fast. Try removing it and see how much your speed increases.

[kennethreitz]

$ pipenv install raven just took like 1s for me.

[tricoder42]

$ pipenv install raven just took like 1s for me.

That's what I would expect! Can I turn on verbose output somehow?

I tried remove psycopg2, but it doesn't affect much. Running pipenv install raven hangs for a while.

I have:

• Python 3.6.2
• macOS 10.12.6

[kennethreitz]

i see no reason why raven shouldn't be instantaneous.

do $ pip install raven inside of $ pipenv shell and tell me if it's slow there too.

[jkp]

@techalchemy thanks for your response. The packaging.version.parse() find seems like a good lead. Could you put some more colour on this statement:

Regarding the JSON api, it's not actually enabled to hit directly in the current release, and I am planning to disable it in the codebase again before we release.

I didn't understand why you are planning to disable it?

[uranusjr]

@jkp Regarding the JSON API, let’s just say it is not the best designed thing out there. The simple API is much more suitable for us. By disabling it we’re using the simple API, not using no APIs altogether.

[keshavkaul]

It's taking too long for me to install Pyspark.
My Pipfile -

[[source]]
name = "pypi"
verify_ssl = true
url = "https://pypi.org/simple"

[dev-packages]
pylint = "*"
pyspark = "*"

[requires]
python_version = "3.5"

shell output -

Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...

Process is stuck on the last line above.
Finishes after 15-20 mins

pipenv, version 2018.7.1

[uranusjr]

@keshavkaul PySpark is very large, and can take quite some time just to download. Give it some time, it will be much better afterwards (because Pipenv caches the result).

(Or you can urge the developers to release a wheel distribution. That would help a bit.)

[uranusjr]

Note for future visitors: Please refrain from posting your slow installation result. We know it can be slow. We know why it is slow. Your result does not add anything to this topic.

[alexandre-paroissien]

Would it be possible to have some information or progress bar like apt-get or wget (download speed, size downloaded, total size) during the libraries downloads?
I guess this is the issue here, pipenv seemed slow for me but it was just the library download, I had to open a system monitor to understand that pipenv was downloading the files and how much was already downloaded, what speed etc

[crifan]

hava same issue: Locking [packages] dependencies... hang forever
my environment:

• macOS High Sierra 10.13.6
• Python: Python 3.6.4
• pipenv: version 2018.7.1

[techalchemy]

@crifan there is no need to post the same message on every issue open or closed which mentions locking speed. We will see your comment no matter how many times you say the same thing. If you want to be helpful, you will need to provide a reproducible example case. Chiming in to say “me too” simply doesn’t add anything besides extra traffic on the issue tracker. Please be mindful of that.

[ScarletMcLearn]

Same here. Pipenv very slow.
Takes an hour to lock and install.

[whyboris]

I think this issue has been answered well here: #1914 (comment)

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.

• by @techalchemy

[meshy]

Would it be possible to get the list of hashes from the PyPI API, rather than compute them ourselves?

[imhazige]

pipenv is awesome, but this issue seems to be still existing. will be glad to see any progress. --skip-lock did not work.

[slyang-git]

pipenv is awesome, but this issue seems to be still existing. will be glad to see any progress. --skip-lock did not work.

Worked for me

[BramVanroy]

I found that using Git Bash on Windows was very slow compared to Powershell. I don't have any statistics or data on this, but PS seemed faster. So if you are using Git Bash you may want try the native PS for pipenv'ing.

[sinscary]

I know this issue is closed, But for me while installing pandas taking lot of time. The verbose output is this

pipenv install pandas --verbose
Installing pandas…
⠋ Installing...Installing 'pandas'
$ ['/Users/sinscary/.local/share/virtualenvs/signzy-MSzur11z/bin/pip', 'install', '--verbose', '--upgrade', 'pandas', '-i', 'https://pypi.org/simple']
Adding pandas to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
⠦ Locking...

It's stuck at Locking for more than 30 minutes. I am using python 3.7.0, macos mojave. Any help with that.

[black-snow]

Why are all the issues to this topic closed? I can't pipenv install a single thing due to the lock-step hang.

[feluxe]

The following docker image takes more than 30 minutes to build on my laptop (i7/16Gb), the pipenv install ... command runs for ages...

Dockerfile

FROM python:3.7-alpine

# Update package list.
RUN set -ex && apk update

# Install apk dependencies.
RUN set -ex && apk add --no-cache musl-dev gcc libffi-dev openssl-dev make

# Install Pipenv.
RUN set -ex && pip install pipenv --upgrade

# Copy Pipfiles.
RUN mkdir /website
COPY Pipfile /website

# Install Pipenv dependencies.
WORKDIR /website
RUN set -ex && pipenv install --system --skip-lock --verbose

Pipfile

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[requires]
python_version = "3.7"

[packages]
sanic = "*"
jinja2 = "*"
asyncpg = "*"
uvloop = "*"
munch = "*"

[dev-packages]

[pipenv]
allow_prereleases = true

Is this normal? Can someone reproduce?

Update: Be careful with Alpine Linux

I realized that the issue is not on pipenv's side...

I replaced the Alpine base docker-image with one that is built on Debian-Slim and now pipenv install finishes within seconds.

The issue in my example is that Alpine Linux will always try to build packages which contain cython-extension or c-extensions from source, which can take forever in a Docker container, whereas Debian Linux installs them using the wheel format, which happens A LOT faster (within seconds).

More on this: https://stackoverflow.com/questions/49037742/why-does-it-take-ages-to-install-pandas-on-alpine-linux

[rajkumarGosavi]

I have long left pipenv and whenever i need to create virtual env in use "venv" or other options.

[LucasSymons]

Having a weird slowness issue as well, only 2 modules for some scripting I am doing.

boto3
click

Took 15/20 min to install, internet tests at over 60Mbps down and running on an up to date MacBook Pro 2019 (not my choice of hardware).

[rajkumarGosavi]

Please use virtualenv for time being. until better solution available

[stuaxo]

99% of the time I do this, the dependencies will resolve to the same one in my lock file, this is because it's part of my dev pipeline.

In the case where there are no new upstream packages since the last run, surely the process could be skipped ?

[matteius]

We have done some performance enhancements to pipenv in recent times, including a big install optimization which released 2022.8.31. For an independent comparison of benchmarks, please have a look at: lincolnloop.github.io/python-package-manager-shootout