[2020-resolver] No longer can get list of available versions.

[vityas]

What did you want to do?

I would like to list all the available versions of a package. In the past we would use this:

pip install pip==asdf  # any gibberish here :)

Output

ERROR: Could not find a version that satisfies the requirement pip==asdf
ERROR: No matching distribution found for pip==asdf

expected output (pip==20.2.4)

ERROR: Could not find a version that satisfies the requirement pip==asdf (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 
0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 
1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 
6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 
8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 
19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 
20.1, 20.1.1, 20.2b1, 20.2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1)

Additional information

• Using verbose flag: pip install -v pip==asdf we can sorta see that versions. (300 line output, too long to post here)
  Seems like no extra requests needed. I wasn't 100% sure if it was the case.
• This is useful when debugging failed installation of a requirement.txt, pipenv or other 3rd party package managers

[brainwane]

@vityas thank you for this bug report, which I've just discussed with pip maintainer @pradyunsg.

I understand what you're concerned about. It used to be the case that if you ran that particular command, pip would tell you about all the versions of that package available in the index you were hitting (such as PyPI). Now pip doesn't print that info about versions by default, with the new resolver.

This was not a useful error message for people who UNINTENTIONALLY got it, but it was the only way to get (on the command line) a list of all the available versions of a package on PyPI or on another index.

If you're hitting PyPI, you can go the release history page for a project, such as https://pypi.org/project/pip/#history , but with the new resolver behavior, currently, there's no easy way to get this info in the CLI directly and in a concise format. pip hits an index and gets the list, but now doesn't present it by default. And if you turn up verbosity, you get the info but in a hard-to-use format, as you said.

Right now, in order to expedite the release of pip 20.3 this week. I'm marking it as post-release work.

For now, we advise you to (per the deprecation timeline) opt-out and choose the old resolver behavior for now, using the flag --use-deprecated=legacy-resolver.

But also: you're bringing further attention to two problems with pip's functionality more generally. pip has a few functions, such as pip search, which offer a little bit of discoverability to help users see what is available in package indexes. Pradyun wants to consolidate functionality and design for these features. And we know that pip's output, especially its --verbose output, needs work. So I hope you will please talk to our UX people, @nlhkabu @ei8fdb and @georgiamoon, by signing up for the user experience studies, to help them work on the larger project of making our verbose output and our package discoverability functionality better.

Thank you!

[RafalSkolasinski]

I would really love to see this back - it seemed like the most streamlined way to find out what versions one can install. See this highly voted stack answer here.

[pfmoore]

It's definitely not (and never was) a supported way of getting that information. But I can imagine the possibility that we might implement a supported way of getting the same data. Although to be honest, as long as you just want to get the information from PyPI (and don't need to support extra indexes, or local package repositories) it's not that hard to write something yourself that gets the same information, using the PyPI JSON interface, plus packaging.tags for doing the wheel compatibility checks.

Maybe someone could publish a "PyPI query" package that did things like this? It might be an interesting project for someone who wants to get into working with the packaging ecosystem.

[RafalSkolasinski]

I believe that at this point this is just a behaviour that many people may have relied on for years. Sometimes things that were not meant to be present but were just become part of what is expected.

[brainwane]

@RafalSkolasinski Thanks for sharing your experience! Indeed, we're getting a lot of demonstrations of Hyrum's Law as we engage in this work this year. :-)

As you saw in a comment above, when prioritizing what to address and what to leave for later, I decided to leave this particular issue for later. We've had to be incredibly limited in what we build and fix with the funding we got to work on pip this year. It runs out at the end of December. One of the things we did with that funding was set up a more systematic way to hear from our users about how you use pip, what features you use, and so on, to help us make better decisions -- user experience research and testing. I hope you will sign up for the user experience surveys and share your opinion in more depth that way. (For example, you could share whether you always use PyPI or whether this functionality is important to you when installing from other sources as well.)

And the pip team has no reliable future funding, and can do a lot more if companies that depend on Python packaging tools chip in (here's the packaging sponsorship program).

[NoahGorny]

Hello folks!
I had a PR #8978 which proposes a new way to look at the available versions (pip search <packages> --list-all-versions)

If you support this idea, please let folks here know so we can move forward with this offer 😄

cc @ei8fdb

[wimglenn]

This was not a useful error message for people who UNINTENTIONALLY got it

@brainwane I disagree with that, actually it's still useful even when UNINTENTIONALLY getting it. If I did pip install distlib==0.29.post0 and saw:

ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0
(from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 
0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.9.post0, 0.3.0, 0.3.1)

That tells me I just had a silly typo and I wanted 0.2.9.post0. But if I saw

ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0 (from versions: none)

It tells me something else, probably that I have misconfigured my index urls and I should check the pip config. In 20.3.x

ERROR: Could not find a version that satisfies the requirement

This is less helpful again. Is my index misconfigured? Could it find any version at all? Do I just need to wait a moment or clear pip's cache - maybe I just published this package recently?

pip search afaik is broken and slated to be removed (#5216). At least, I could not find any way to use it reliably (and couldn't get it working at all with a devpi-server index).

Thank you for mentioning the --use-deprecated=legacy-resolver option! For the purposes of keeping the project johnnydep functioning on pip 20.3.x, this helped for now. Currently it scrapes pip's CLI output in a way that needs careful tweaking and testing each time a new version of pip is released. So here's a bit of UX feedback, it would be really nice if there was a less fragile way to get:

• download urls
• package dependencies
• versions available

It's definitely not (and never was) a supported way of getting that information.

@pfmoore So how do we get that information? A while ago pip moved all package finding code under _internal and made it explicit that a CLI subprocess was the only supported way to use pip from your program. This is arguably an important part of the CLI.

it's not that hard to write something yourself that gets the same information, using the PyPI JSON

I think that is oversimplifying the problem. It might be true if pip was only a PyPI client, but it's a lot more than that isn't it? Pip's the de facto tool for working with Python packages, whether from PyPI or not, since we don't really have one in the stdlib.

To repeat again the points in #1884: the PyPI JSON is not in general "the same information" because where pip looks is highly configurable, e.g. we may have another --index-url and then an --extra-index-url and maybe that index is just a simple repository which is supporting a JSON api at all. There are a lot of little details here and doing it in a way that accurately produces the same packages that pip would see would mean reproducing a significant amount of package finder code found within pip._internal.

[ddelange]

@brainwane thanks for your explanation on why this feature (I'm gonna call it a feature as we rely on it in johnnydep, pipgrip, probably more) was deprecated in the new resolver.

Like mentioned above, it would be quite cumbersome and most likely introducing bugs to move away from using this feature in pip, so I've dropped support for pip 21 (hoping to loosen that constraint again though!), and will keep an eye out for this issue.

Thanks for all your hard work and responses throughout these exciting times! 🙏

[earonesty]

imo: removing output that helped with diagnosis (a version typo), seems to go against a basic unix philosophy "Write programs which fail in a way that is easy to diagnose".

[Mahlzeit2000]

imo this feature should be brought back in or at least outsourced to a new option thats bit less unwieldy than sudo pip3 --use-deprecated=legacy-resolver install xx==. The new behavoir can be misleading and less useful for the many reasons already stated.

[thephet]

This feature must come back. It is terrible design to remove it. It was very useful to navigate between different package versions. Basically, when an upgrade of a package breaks something, to be able to roll back to the previous version.

[MrMino]

This has been a canonical feature of Pip for many of its users for at least 6 years now. I've given this tip out numerous times at work, I'm probably not the only one. There's this SO answer - +950 rep on a 10 year-old, very specific question that's been viewed 280k times. Tooling that uses pip has been scraping this data for years.

If you wish to remove this output, then to appease users you'll have to add a feature that makes it possible to list it out. And one that is as succinct and streamlined as appending ==. By the looks of it, there's nothing else possible that comes close.

Listing versions by appending == is, simply put, a good feature.

We've come to enjoy it due to a happy accident related to how pip handles its arguments. But that doesn't mean it's not valuable and shouldn't be there.

If you're not going to add == usage back, then the discussion is probably a waste of time. A longer flag or a subcommand will feel like a chore to type when a version is mistyped. At that point it indeed would be a good idea to create a separate tool, if only it wasn't for the fact that it would have to clone large chunks of code from Pip in order for the outputs to be relevant for the issue at hand.

UX-wise you have to remember you are operating in a regime where each keystroke is a considerable cost. Anyone who uses == a lot is familiar with the following typing-scheme:

pip install package== ⏎ → ↑ <desired_version>⏎

or

pip install package==<wrong_version> ⏎ → ↑ ←BP ←BP ←BP ... <desired_version>⏎

The convenience factor of this is huge. I doubt anyone will be able to come up with anything better.

Trying to fix the lack of that by adding --list-versions as an available flag is misguided. Not to bash on @NoahGorny's idea - pip should have an explicit, user-facing feature that makes it possible to obtain the list. But #8978 should be discussed separately to this issue.

IMO pip search is, to put it lightly - irrelevant. It produces unreliable and overly verbose results, uses different index from pip install, not to mention the XMLRPC API has been temporarily disabled of PyPI and #5216.

[miraculixx]

Following the workaround noted by @brainwane I'm using this little helper

# in .bashrc
function pip-versions {
  pip install --use-deprecated=legacy-resolver $1==
}

Then e.g.

$ pip-versions six
ERROR: Could not find a version that satisfies the requirement six== (from versions: 0.9.0, 0.9.1, 0.9.2, 1.0b1, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.8.0, 1.9.0, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0)
ERROR: No matching distribution found for six==

Not nice, good enough for now

[MrMino]

There. It's even better than the previous thing, at least for me. WARNING is probably wrong, but the yellow text is cool™. This could still use some sorting - currently it uses set() to exclude duplicates, which means that ordering might not be correct, for I'm too lazy to do it properly. Not sure if I should use list(OrderedDict.fromkeys(...)) to simply preserve it, or distutils.LooseVersion to sort explicitly.

Some dupes still go through for some reason, but I'm too tired this year to find out why (edit: actually, I just forgot to reinstall pip after patching this, its fixed already).

This is how it looks right now. In case where the library is nowhere to be found (also, when cache is not used):

In case the version is wrong:

When you have multiple indexes specified, the diagnostic will only mention the ones in which the package version/s were found. Having multiple indexes logged looks something like this (some info redacted because I'm NDA-scared):

See you next year.

[andreasnuesslein]

just for completeness sake, this affects saltstack as well: saltstack/salt#59100

[brainwane]

As I mentioned in early December, in a comment above:

We've had to be incredibly limited in what we build and fix with the funding we got to work on pip this year. It runs out at the end of December.

And the pip team has no reliable future funding, and can do a lot more if companies that depend on Python packaging tools chip in (here's the packaging sponsorship program).

FYI, that funding has now run out. As far as I know, no one is currently paid to work on pip (including code review). It's all volunteers.

[MrMino]

@brainwane As far as I can tell it's been reviewed already: #9405.

[Asday]

pip has a few functions, such as pip search,

Posted Nov 2020. I wonder if that's what lead to the DDoS(?) of the XMLRPC endpoint.

[MrMino]

@Asday has that increased usage of it started in November? I thought it was earlier?

[ddelange]

The suggested workaround (--use-deprecated=legacy-resolver) was supposed to be deprecated in pip v21, however it wasn't.

What is the new deprecation timeline?

refs
#9139 (comment)
#9187 (comment)

[pfmoore]

@ddelange I think the honest answer is "as soon as one of the developers gets round to it"... (one of the consequences of the funded period having completed is that "tidy up" activities like this are back to being purely volunteer-based). The fact that the removal was delayed may have given users a little bit longer, but it doesn't really alter the message that people should have found alternatives to relying on the old resolver, and should no longer be using it as of the 21.0 release. Prudence suggests that you should assume it'll go without further warning in 21.1 (April).

The only reason I don't offer to just remove it now, for the sake of being definite, is that there's a lot of non-user-visible work needed to remove all of the tests that check the old resolver's behaviour, and I don't feel like spending my time on that.

[MrMino]

work needed to remove all of the tests that check the old resolver's behaviour, and I don't feel like spending my time on that.

@pfmoore is there an issue made for that? Maybe we can help?

[pfmoore]

@MrMino A PR removing the legacy resolver would be great (or even just one that removed the option and fixed the tests, and left actual removal of the old resolver code for later).

I can't promise I'll have a lot of time to review such a PR (my review time is almost as limited as my coding time at the moment) but maybe other @pypa/pip-committers could help out there.

[Asday]

A PR removing the legacy resolver would be great

Unless I'm missing something, no it wouldn't. That would remove the ability to get the list of versions entirely.

[MrMino]

@Asday Are you sure you're not missing #9405 ? ;)

[Asday]

Sure aren't - that's not merged yet. I also have a feeling that "remove deprecated thing" would be merged faster than "add new feature".

[fenchu]

please get this meged (21.0.1 does not have it), when you work on alpha and beta modules this is a complete pain, I do not remember --use-deprecated=legacy-resolver