pip install of a directory is super slow

[msabramo]

See #2195 (comment), for a summary of this issue.

---

I am dubious of why pip needs 17 seconds to process a local directory that is not on NFS (in fact, it's on an SSD drive) for pip, which has no dependencies, since everything is vendored.

$ time pip install --no-install ~/dev/git-repos/pip
DEPRECATION: --no-install and --no-download are deprecated. See https://github.com/pypa/pip/issues/906.
Processing /Users/marca/dev/git-repos/pip
  Requirement already satisfied (use --upgrade to upgrade): pip==6.0.dev1 from file:///Users/marca/dev/git-repos/pip in /Users/marca/dev/git-repos/pip
pip install --no-install ~/dev/git-repos/pip  2.80s user 5.86s system 50% cpu 17.205 total

It should probably at least be logging whatever is taking that long, but maybe it shouldn't even be doing whatever it's doing.

Note that the "Processing" line appears right away and pretty much the whole delay seems to be between that line and the next one.

[tomprince]

It is making a copy of the entire directory, including .git. It probably shouldn't be doing that, no.

[msabramo]

$ du -sh pip
263M    pip

$ du -sk * .cache .git .tox .travis | sort -nr | head -n 5
181860  .tox
34836   tests
31700   .git
9212    pip
2852    build

[msabramo]

I tried passing 3 -v's (time pip install -vvv --no-install ~/dev/git-repos/pip) -- that didn't yield any more info.

[msabramo]

Stepping through it with pdb, things slow down when I get to:

> /Users/marca/dev/git-repos/pip/pip/req/req_set.py(365)prepare_files()
-> unpack_url(

[msabramo]

And yep, @tomprince is right - it slows down when it does a copy of the whole tree:

> /Users/marca/dev/git-repos/pip/pip/download.py(635)unpack_file_url()
-> shutil.copytree(link_path, location, symlinks=True)

[msabramo]

$ time pip install --no-install ~/dev/git-repos/pip
DEPRECATION: --no-install and --no-download are deprecated. See https://github.com/pypa/pip/issues/906.
Processing /Users/marca/dev/git-repos/pip
  2014-12-15 15:23:34.630794: Copying tree; link_path = '/Users/marca/dev/git-repos/pip'; location = '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/pip-D6etc4-build'
  2014-12-15 15:23:57.418679: DONE copying tree; link_path = '/Users/marca/dev/git-repos/pip'; location = '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/pip-D6etc4-build'
  Requirement already satisfied (use --upgrade to upgrade): pip==6.0.dev1 from file:///Users/marca/dev/git-repos/pip in /Users/marca/dev/git-repos/pip
pip install --no-install ~/dev/git-repos/pip  2.75s user 5.03s system 32% cpu 24.168 total
>>> elapsed time 24s

[msabramo]

Some discussion in #2196

[msabramo]

It's much faster now that #2196 is merged.

[msabramo]

This should be reopened since #2196 was reverted. I'd like to come with an alternative PR that builds an sdist instead of using heuristics to figure out what to copy. See the comments on that PR for details.

[msabramo]

$ time pip install --no-install ~/dev/git-repos/pip
DEPRECATION: --no-install and --no-download are deprecated. See https://github.com/pypa/pip/issues/906.
Processing /Users/marca/dev/git-repos/pip
  Requirement already satisfied (use --upgrade to upgrade): pip==6.1.0.dev0 from file:///Users/marca/dev/git-repos/pip in /Users/marca/dev/git-repos/pip
pip install --no-install ~/dev/git-repos/pip  3.67s user 8.12s system 7% cpu 2:45.83 total
>>> elapsed time 2m46s

Yikes, almost 3 minutes.

Probably mostly due to this:

$ du -sh .tox
177M    .tox

The .tox directory is 177M out of a total 270M for my whole pip directory.

[msabramo]

Please see #2535, which speeds up unpack_file_url by building an sdist and unpacking it.

[rgommers]

This issue should be reopened, because the merged PR did nothing (see gh-3219).

[sbidoul]

This should be resolved by #7882 (build local directories in place).

[pradyunsg]

We have now (per #7951) published a beta release of pip, pip 20.1b1. This release includes #7882, which implemented a solution for this issue.

I hope participants in this issue will help us by testing the beta and checking for new bugs. We'd like to identify and iron out any potential issues before the main 20.1 release on Tuesday.

I also welcome positive feedback along the lines of "yay, it works better now!" as well, since the issue tracker is usually full of "issues". :)

[PythonCoderAS]

I will say that it is considerably better.

Old: noglob pip3 install . 3.76s user 2.51s system 12% cpu 50.245 total

New: noglob pip3 install . 3.40s user 0.70s system 42% cpu 9.764 total

[astrofrog]

Works great/faster for me! 👍

[klamann]

» pip --version
pip 20.0.2 
» time pip install .
noglob pip install .  8.03s user 18.47s system 25% cpu 1:44.84 total

» pip --version
pip 20.1b1 
» time pip install .
noglob pip install .  3.69s user 0.31s system 92% cpu 4.307 total

down from ~2 minutes to 4 seconds, thank you so much!

[pradyunsg]

Thank you for the positive reports @PythonCoderAS @astrofrog @klamann! :)

Unfortunately, there have been a number of issues with the implementation of in-place builds (which are being tracked under #7555) which means that for now, we need to revert #7882. As a result, this issue will become a problem again, and we'll therefore be reopening it. Longer-term, we hope to have a solution that addresses the issues that in-place builds solved, but without the impact on other workflows that the current solution had.

Sorry for the disruption that this will cause.

[rgommers]

Unfortunately, there have been a number of issues with the implementation of in-place builds

@pradyunsg thanks for the update. Some feedback on terminology (please feel free to ignore, just FYI): this sentence, as well as gh-7555, confused me because pip does not do in-place builds. What in-place builds has always meant is python setup.py build_ext --inplace (or python setup.py develop).

Here you changed the meaning to: "build without copying to a tmpdir". Extension modules still don't end up in-place, they end up in a build/ dir that's usually easily cleaned up. It would be nice to be a little more explicit in for example gh-7555.

[pfmoore]

That was originally my wording. Sorry for any confusion, I wasn't aware that setuptools used the term "in place" to mean something different (and I'm still not really sure how that terminology applies outside of setuptools). We'll see if we can find a more neutral term in future (although offhand, I'm not sure what - suggestions gratefully accepted 😉)

[rgommers]

No worries at all, thanks @pfmoore. I just thought I'd point it out, since confusion about terminology can sometimes result in talking past each other.

and I'm still not really sure how that terminology applies outside of setuptools

For tools like CMake and scikit-build I think it means the same thing: actually in-place, binaries land next to sources.

"editable installs" on the other hand is (I believe) invented here, and kinda means "in-place that pip is aware of".

although offhand, I'm not sure what - suggestions gratefully accepted

maybe just "local build" (vs. the current "copy to tmpdir and build")?

[gaborbernat]

"editable installs" on the other hand is (I believe) invented here, and kinda means "in-place that pip is aware of".

We recently had a long discussion on what editable install means, and I think we actually landed in a place that is more along the lines of machine local as far as pip goes. But pip is unaware of where and how on the local machine and is the build backends job to define and handle that.

[merwok]

Could try «in-tree build» (similar to «in-tree PEP 517 backend») or «build in source dir»

[PythonCoderAS]

My question is, why can't the feature be optional, so it does not cause problems but can be enabled by an argument or something similar?

[salotz]

I'm trying to wrap my head around the workarounds for this, where an editable install isn't an option. Is there any?

[merwok]

A workaround could be to build a wheel (using your build backend directly) then point pip to install it

[pradyunsg]

why can't the feature be optional, so it does not cause problems but can be enabled by an argument or something similar?

It can. The reason for reverting the change was that we didn't have any opt-outs or a period for getting feedback on the change. We do have new flags to help facilitate that (--use-feature and --deprecated-feature), but someone has to reimplement/reintroduce the functionality in this context now.

Broadly, I think what we want to do here is:

• Add a --use-feature=in-tree-build as an opt in.
• Switch the default in a later release w/ a --deprecated-feature=out-of-tree-build as an opt out + pushing users of --use-feature=in-tree-build to drop it.
• Drop both of the options in a subsequent release.

[salotz]

A workaround could be to build a wheel (using your build backend directly) then point pip to install it

I was thinking without an extra build step. But I guess I should have never thought Python could get away without Makefile equivalents from the beginning.

[dkbarn]

FYI for those who are running into this issue -- A workaround is to replace pip install . with:

python setup.py bdist_wheel
pip install dist/*.whl

[uranusjr]

Now that in-tree-build is available, should we close this?

[pradyunsg]

Yes.