Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install -U upgrades already satisfied dependencies #304

Closed
ejucovy opened this issue Jun 9, 2011 · 23 comments
Closed

pip install -U upgrades already satisfied dependencies #304

ejucovy opened this issue Jun 9, 2011 · 23 comments
Labels
auto-locked Outdated issues that have been locked by automation C: upgrade The logic of upgrading packages type: bug A confirmed bug or unintended behavior

Comments

@ejucovy
Copy link

ejucovy commented Jun 9, 2011

If I pip install -U foo, I would expect that the latest version of foo will be installed, and foo's dependencies will only be reinstalled if they're not already satisfied. But in fact the dependencies all get reinstalled even if I already have identical versions installed:


$ pip install -U django-supervisor
Downloading/unpacking django-supervisor
  Downloading django-supervisor-0.2.0.tar.gz
  Running setup.py egg_info for package django-supervisor
Downloading/unpacking supervisor (from django-supervisor)
  Downloading supervisor-3.0a10.tar.gz (438Kb): 438Kb downloaded
  Running setup.py egg_info for package supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
Downloading/unpacking meld3>=0.6.5 (from supervisor->django-supervisor)
  Downloading meld3-0.6.7.tar.gz
  Running setup.py egg_info for package meld3
Installing collected packages: django-supervisor, supervisor, meld3
  Found existing installation: django-supervisor 0.1.1
    Uninstalling django-supervisor:
      Successfully uninstalled django-supervisor
  Running setup.py install for django-supervisor
  Found existing installation: supervisor 3.0a10
    Uninstalling supervisor:
      Successfully uninstalled supervisor
  Running setup.py install for supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
    Skipping installation of /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor/__init__.py (namespace package)
    Installing /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor-3.0a10-py2.6-nspkg.pth
    Installing echo_supervisord_conf script to /usr/local/ejucovy/django/bin
    Installing pidproxy script to /usr/local/ejucovy/django/bin
    Installing supervisorctl script to /usr/local/ejucovy/django/bin
    Installing supervisord script to /usr/local/ejucovy/django/bin
  Found existing installation: meld3 0.6.7
    Uninstalling meld3:
      Successfully uninstalled meld3
  Running setup.py install for meld3
Successfully installed django-supervisor supervisor meld3
Cleaning up...

My "existing installations" of supervisor-3.0a10 and meld3-0.6.7 are both "successfully uninstalled", and then identical versions are installed.

@hltbra
Copy link
Contributor

hltbra commented Jun 9, 2011

In my judgement it is a known behavior, don't know if it is really a bug - I guess easy_install has the same behavior.

I would like to see other's opinions.

PS.: There was a question in StackOverflow related to this: http://stackoverflow.com/questions/5937756/why-is-pip-looking-for-download-cache-if-the-same-exact-package-is-already-instal

@hltbra
Copy link
Contributor

hltbra commented Jun 9, 2011

Related to issue #49

@carljm
Copy link
Contributor

carljm commented Jun 9, 2011

It is a bug, and it is a duplicate of #49.

@carljm carljm closed this as completed Jun 9, 2011
@ejucovy
Copy link
Author

ejucovy commented Jun 9, 2011

I don't think it's a duplicate of #49. I read #49 as saying that install -U foo shouldn't reinstall foo itself if it's already at the latest version -- which is different from whether or not it should reinstall foo's already-satisfied dependencies.

This distinction matters for hard-to-build libraries which have frequent releases but fairly stable APIs -- for the most part, one installation is good enough -- I'd only want to reinstall it if my dependencies start to use newer features from those nested dependencies (i.e. if the requirement is no longer satisfied) -- for example:

  • foo 0.1 depends on lxml>=2.3.0
  • foo 0.2 is released, and depends on lxml>=2.3.0 (same dependency)
  • lxml 2.4.0 is released

If I've already installed foo 0.1 and lxml 2.3.0, and I pip install -U foo, I wouldn't want it to install lxml 2.4.0. It should only install lxml 2.4.0 when foo starts to depend on lxml>=2.4.0.

@carljm carljm reopened this Jun 9, 2011
@carljm
Copy link
Contributor

carljm commented Jun 9, 2011

Ah, yes, that is a bit different. Even if #49 is fixed, there would be some additional code needed to achieve the behavior you're wanting when the dependency is not at its latest version, but still satisfies the dependency requirements.

I think if we made this change, some people would find it surprising. I can see why it's desirable in certain cases -- but I can also see cases where the current behavior (minus #49) is preferable. So I'm on the fence on this one.

@ejucovy
Copy link
Author

ejucovy commented Jul 16, 2011

Would an extra command-line option be appropriate? pip install foo --upgrade vs pip install foo --upgrade-recursive? (Or --upgrade-nonrecursive if preserving the current behavior backwards-compatibly is important)

@nisavid
Copy link

nisavid commented Jul 29, 2011

making upgrades non-recursive by default would provide consistency with other package managers, such as APT and Portage. and i think there is a good reason for such behavior, which is that it avoids unintended side effects--if i want to upgrade a package P, then i would like to enter a command along the lines of upgrade P, not upgrade P --but-not-other-things.

on the other hand, i think that an "upgrade all" command (see #59) should be recursive by default, since "all" should really mean "all" by default. in this case, non-recursive behavior would mean "upgrade all packages installed directly, but not any dependencies that weren't installed directly" (like Portage's emerge --update @world without --deep).

@cls
Copy link

cls commented Sep 13, 2011

A related issue is that if the package being upgraded has a dependency on another package which is already installed but unavailable from a repository, Pip will fail and is unable to upgrade the requested package, despite its dependencies being fulfilled.

@ex-nerd
Copy link

ex-nerd commented Jan 11, 2012

This appears to happen with -I as well as -U.

@carljm
Copy link
Contributor

carljm commented Jan 11, 2012

Since -I stands for --ignore-installed, and is intended to make pip operate as if nothing were currently installed, reinstalling everything is the correct behavior for -I. Thus the behavior here is only a bug for -U, not for -I.

@jvanasco
Copy link

easy_install does not have the same behavior. easy_install will not re-install dependencies if they are already met.

this isn't a feature or behavior, this is a bug.

this is also really annoying -- testing PIP distribution for a framework's package , or updating a single framework add-on, means needing to re-install the entire framework and all of it's dependencies. these unnecessary downloads and installs are a waste of time and resources.

@qwcode
Copy link
Contributor

qwcode commented Jul 9, 2012

for those who just want a way to do this now, short of a behavior/code change to "-U"

I think this achieves the desired result, right?

upgrade top-level requirements only:

  • pip install -U --no-deps REQS //upgrades only top-level
  • pip install REQS //this 2nd pass will install any new dependencies from the upgrade

@fdintino
Copy link

fdintino commented Jul 9, 2012

For the simplest case that suffices, but I don't think that works for a requirements.txt file, or if there are dependencies which have updates to their install_requires. We have a complicated script that does a diff of our requirements.txt and more-or-less does what you describe, but it doesn't handle upgrade depth > 1.

What complicates matters to an extent is that many django packages comment out or remove their install_requires lines because of this bug; otherwise they end up with some alpha version of django installed (that has been our experience, and I've seen it in the github issues of many prominent django packages.

@qwcode
Copy link
Contributor

qwcode commented Jul 9, 2012

Hey @fdintino , I rigged up a basic test with requirements, -U, and --no-deps and it seemed to work, i.e. the items in the requirements file were upgraded, but no dependencies. that's consistent with my understanding of what the code is doing. it's tedious in there though, so there could be a failing with this idea.

for the case where a top-level requirement has a new "install_requires" dependency, that's why I mentioned the 2nd pass command to install those.

for the case where there are "are dependencies which have updates to their install_requires", I'm not sure I follow you there. you would only discover and want to act on that, if you were wanting to do a full recursive upgrade, right?

@fdintino
Copy link

fdintino commented Jul 9, 2012

Only if you no longer met the requirements. So, for example, if I updated django-sentry, and django-sentry required django-celery>=2.5.4,django-celery<3.0where it previously required django-celery>=2.5.3,django-celery<3.0 (perhaps because of a bug fix or a new feature), and I currently have django-celery==2.5.3, then I would expect it to update django-celery to satisfy the requirement, the way my patch does. However, if I happened to have django-celery==2.5.4 I would not expect it to update celery. In the case of celery it's not a big deal, but when packages have Django>1.2,Django<=1.4, and projects are often written to target Django 1.2, 1.3, or 1.4, an unexpected upgrade of django can be a huge headache.

@qwcode
Copy link
Contributor

qwcode commented Jul 9, 2012

thanks @fdintino . I follow now. you don't want to cut off all recursive behavior (that will do necessary upgrades to fulfill requirements), just want to stop recursive "forced" upgrades.

btw, your comment is cropped due to formatting. might want to fix that for others.

@qwcode
Copy link
Contributor

qwcode commented Jul 11, 2012

I posted a gist that breakdowns achieving the desired behavior using the 2 commands in sequence mentioned above.
Not that it invalidates the desire for this ticket or the pull, but it was helpful to me to confirm how
it works currently and what's possible currently.
(hint: in the example, "b" is the parallel to django that was mentioned as a concern)

https://gist.github.com/3088149

comment appreciated on the gist if this is not the scenario.

@mverleg
Copy link

mverleg commented Jan 4, 2017

I notice this has been open a long time.

I found that my version of pip now has the option to do

pip install --upgrade --upgrade-strategy=only-if-needed package

This seems to be the desired behaviour, although rather verbose. Personally I think it'd have been great if this were the default, but maybe it's too late to change now.

If it's too late to change the default, then I think this can be closed?

@dstufft
Copy link
Member

dstufft commented Jan 4, 2017

In #3871 (comment) I mentioned what I believe to be the way we're going to move forward on this, reproducing here:

Circling back to this now. Here's what I think we should do:

  1. Add --upgrade-strategy=[eager/non-eager] in pip vX which defaults to eager, and allow people to opt-in to the non-eager strategy. This will allow us to get some real world testing from people without forcing a change on everyone at once.
  2. Once we've addressed any feedback from 1., switch the default of --upgrade-strategy to non-eager in pip vX+1. This will get us a lot of real world use by forcing a change on everyone, but gives an escape hatch for people who, for whatever reason are broken by the change.
  3. Once we've addressed any feedback from 2., look at deprecating --upgrade-strategy in pip vx+2 (to be removed following our normal deprecation policy).

This will have a longer lead time before non-eager is the default, but it allows us to phase it in slower to make sure there's not some use case that's going to go kaboom from it. Ideally I think we'd want to eventually get rid of the --upgrade-strategy flag. That kind of flag feels like it's just asking for trouble with things breaking because it essentially requires us to duplicate our upgrade testing to actually fully test things. I do think it's a good flag to add for now though, to allow the phase in and dealing with any breakages.

@mverleg
Copy link

mverleg commented Jan 4, 2017

Okay sounds good, I'll wait for step 2 to complete. Seems that other issue is a duplicate, but already closed.

@pradyunsg
Copy link
Member

@xavfernandez @dstufft Can this be closed or will we wait till the default switches?

@dstufft
Copy link
Member

dstufft commented May 19, 2017

We'll probably wait till the default switches.

@xavfernandez
Copy link
Member

Closing, since #4500 was merged

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 3, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation C: upgrade The logic of upgrading packages type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

No branches or pull requests