Add "upgrade" and "upgrade-all" commands

[vbabiy]

(Updated to reflect reality in April 2020)

This issue was initially scoped for changing the behavior of pip install --upgrade, by adding a new upgrade command with a different behavior when upgrading packages. The older recursive "eager" default caused grief for many (#304) However, after a lot of discussion on this, the approach taken was to add the --upgrade-strategy flag. The flag initially had a default of "eager" which was changed to a better default in #4500.

The tracking issue for "upgrade all packages" functionality is #4551.

[vbabiy]

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.

---

Original Comment By: Carl Meyer

[vbabiy]

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.

---

Original Comment By: Jannis Leidel

[vbabiy]

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in <sys.prefix>/lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.

---

Original Comment By: Ian Bicking

[vbabiy]

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.

---

Original Comment By: Carl Meyer

[vbabiy]

+1 for this bug

---

Original Comment By: smyrman

[vbabiy]

Issue #167 was marked as a duplicate of this issue.

---

Original Comment By: Carl Meyer

[vbabiy]

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).

---

Original Comment By: Tomasz Elendt

[vbabiy]

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?

---

Original Comment By: Anonymous

[vbabiy]

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's

---

Original Comment By: Kelsey Hightower

[vbabiy]

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!

---

Original Comment By: Carl Meyer

[vbabiy]

Carl: Cool, I will get started and update this ticket with the results.

---

Original Comment By: Kelsey Hightower

[vbabiy]

While working on the upgrade command the following questions came up:

• What methods should pip upgrade support to specify which packages to
  upgrade? Should we support a requirements file?
• How should pip upgrade handle packages that are not already installed?
  Should we install missing packages?

pip upgrade use cases and how to handle them:

# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?

---

Original Comment By: Kelsey Hightower

[vbabiy]

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.

---

Original Comment By: Carl Meyer

[vbabiy]

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.

---

Original Comment By: Kelsey Hightower

[vbabiy]

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

• Tests
• Filter requirements file; add non-editables to list of packages to
  upgrade.

Running pip using the upgrade command

# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

---

Original Comment By: Kelsey Hightower

[vbabiy]

Last set of questions (I hope):

• Should pip upgrade parse the requirements file and filter out
  editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

• Should pip upgrade search the indexes for a later version to install?
  (This is what I am doing now). If not how should the upgrade command determine
  if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

• The package is not installed
• An upgrade is available (later version from the indexes), and no explicit
  version was requested
• The requested version is different from the installed one (Version miss
  match).
• I am deferring the requirements file to the install command until I filter
  out the non-editable requirements.

---

Original Comment By: Kelsey Hightower

[vbabiy]

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.

---

Original Comment By: Kelsey Hightower

[vbabiy]

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.

---

Original Comment By: Kelsey Hightower

[vbabiy]

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.

---

Original Comment By: Carl Meyer

[vbabiy]

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c

---

Original Comment By: Kelsey Hightower

[vbabiy]

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command

# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

---

Original Comment By: Kelsey Hightower

[vbabiy]

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.

---

Original Comment By: Carl Meyer

[vbabiy]

Thanks Carl, I will wrap this up and start looking at #13

---

Original Comment By: Kelsey Hightower

[vbabiy]

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py

---

Original Comment By: Kelsey Hightower

[jedie]

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

[nisavid]

@jedie i think you meant to direct your comment at @KHightower. @vbabiy migrated her comment to here but did not write the upgrade command.

[gimmi]

+1

[piotr-dobrogost]

@kelseyhightower Any progress?

[fradeve]

+1

[januz]

+1!

[Liso77]

On 9/15/16, Nick Coghlan ***@***.***> wrote: @RonnyPfannschmidt Windows users still outnumber Linux users ~18 to 1, and they don't have anything comparable to a distro package management community to fall back on at this point (while the core technology is there in recent versions, the packaging and curation communities aren't). That means they're a lot more reliant on user level tools like pip and conda to pick up the slack.

Isn't then conda update --all good enough?

[pradyunsg]

@Liso77
Well, No. It is not.

Conda and pip are tools with different goals. This is a great read on this topic. The 3rd point is most relevant.

---

If the discussion for upgrade-all surfaces again (hopefully in a new issue) my vote is for spelling it as follows:

pip install --upgrade :all:

[WhyNotHugo]

pip install --upgrade :all: is extremely weird. Let's stick to POSIX semantics, which basically everyone is doing nowadays: pip install --upgrade --all

[FichteFoll]

What about just pip install --upgrade without any package names or specifiers? To easy to run accidentally?

pip-tools does it like this for pip-compile -P.

[pradyunsg]

Maybe we should bike shed once we have some sort of working implementation... :)

…

On Sun, Feb 12, 2017, 20:55 FichteFoll ***@***.***> wrote: What about just pip install --upgrade without any package names or specifiers? To easy to run accidentally? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SfnSBflH8rK3nFLw1hvYBaovjbcGks5rbyRUgaJpZM4AJ4Py> .

[defjaf]

How about a version of pip list --outdated that produces its list in a format that can be directly (i.e., without sed, cut, etc.) ingested by pip install --upgrade (e.g., pip list --outdated --format install | xargs pip install --upgrade or something similar with backticks)?

[SMH17]

whatever syntax will be used the most important thing is introducing this command, It's unbelievable that is still missing

in meanwhile I suggest you to try
https://github.com/jgonggrijp/pip-review
with pip-review --local --interactive ask you package by package if you want to update, not very good but better than nothing

[ncoghlan]

@SMH17 It's entirely believable that it's still missing, as there are exactly zero commercial Python vendors formally providing funded development time to work on Python packaging usability improvements on behalf of their customers.

So if you'd like to see the situation improve, it's likely that the most helpful thing you can personally do is to either encourage your Python support vendor to invest developer time in improving the tools you use, or if you don't have a support vendor yet, encourage your employer to pay one.

[ncoghlan]

As an additional piece of context regarding the lack of urgency around this issue, it's worth keeping in mind that the general recommendations are to

• keep your working environment definitions under source control to improve reproducibility on other systems (using something like https://github.com/jazzband/pip-tools or https://github.com/kennethreitz/pipenv to keep those definitions up to date)
• aim to routinely upgrade to new releases of dependencies in order to minimise exposure windows to unknown or undisclosed security vulnerabilities

That doesn't mean the commands proposed here aren't useful, they're just markedly less valuable if the current working environment is already being maintained through pip-compile + pip-sync, or pipenv lock + pipenv install.

[piotr-dobrogost]

It would be valuable to update the original description as I guess there have been some changes made to pip install since the update made by @qwcode.

[sebma]

Hi everyone.

I have broken some of my python package dependencies because the following command:

pip install --upgrade packageName has upgraded packages recursively.

Why not change the default behaviour of the --upgrade option, that is to uninstall and reinstall ONLY the given package from the command line ?

How about this ?

[aaossa]

@sebma I think that the default behaviour should not be changed. Maybe you could try using the -no-dependencies flag the next time. It should work 👍

[FichteFoll]

@sebma, @aaossa, I'll have you both know that it has pretty much been decided already that the default upgrade strategy will change in the future (ref: #3871 (comment)). The necessary feature (i.e. the --upgrade-strategy argument) has been added in #3972.

As @pradyunsg mentioned earlier, this issue is kind of a left-over. The first part is sort of handled by now (see my first paragraph) and the second part is the only reason why this package is still open, it seems. I don't know if a separate "upgrade-all" issue has been created since.

[simion]

I've released a nice interactive upgrader for requirements file: https://github.com/simion/pip-upgrader

[pradyunsg]

Shift the default upgrade strategy to only-if-needed.

#4500 did this.

Add "upgrade the world" functionality that depends on #988.

#4551 for discussion on this.

---

Addressing the points made in the current top-post:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here.

It has been decided against adding an upgrade command or making pip install upgrade already installed packages. pip now does have non-recursive upgrades by default, with the recursive behavior available behind --upgrade-strategy eager.

pip upgrade-all would upgrade all installed packages.

#4551 exists and it would be nice to have a fresh discussion on this; when #988 is done.

---

@dstufft @xavfernandez @pfmoore Do any of you think this issue should be closed?

_{Edit (05-18-2017): Punctuation + minor text added}

[dstufft]

Seems reasonable.

[serafeimgr]

Hello all,
I made a simple script/gist that does the job.

https://gist.github.com/serafeimgr/b4ca5d0de63950cc5349d4802d22f3f0

[qoheniac]

Why not simply do this?

pip install --upgrade $(pip list --outdated | awk '{print $1}' | tr '\n' ' ')

[FichteFoll]

Because it's not that easy in reality since you may install versions that don't satisfy some of your other packages' dependencies.

[defjaf]

Based on and with thanks to @serafeimgr's gist, I've written a possibly-useful command-line tool, pip_upgrade_outdated; source on github. Feedback welcome.

(See also this issue: Yes, the parallel execution is particularly dangerous, and, yes, this may break things. Nonetheless many people run something like this by hand all the time, so might find it useful.)

[serafeimgr]

Thank you for taking the time to build a complete solution.
Even though my recommendation would be to find a way to push this feature to pip.

I think pipenv & pipfile is going to replace pip/requirements.txt anyway.
Maybe @kennethreitz knows more about the roadmap and the --upgrade all feature.

[marcelocantos]

@qoheniac | tr ... is redundant.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.