Error with import pip in pip 9.0.2

[Miserlou]

Some users are now experiencing an import error when calling import pip in a function with pip 9.0.2. Downgrading to 9.0.1 fixes the issue.

Trace: https://user-images.githubusercontent.com/2273226/37558391-5e41fc94-2a24-11e8-9fdc-5884445e829a.png

More details here:
Miserlou/Zappa#1446

[nx-jackal]

I can confirm this as well. Just about to file same issue.

[alex]

This is a dupe of #5079 -- importing pip is not a supported use case (and never has been).

[Miserlou]

Come on.. making a huge breaking change in a single x.x.PATCH version bump? For a top level method that's called "main"? I think that's pretty poor form..

[FlipperPA]

This also broke Anaconda and I came up with the same solution as @Miserlou:

ContinuumIO/anaconda-issues#8911

Errors are being reported across a lot of projects.

[RayJ414]

This error also bothered me when I was trying to use pip to update my anaconda-navigator.

[Miserlou]

Is there any chance of us getting a 9.0.3 that fixes this - ideally quite soon?

This is already affecting lots of major projects (Anaconda, SpaCy, Zappa), and there are more than 850k+ uses of this on GitHub alone that are now broken by this supposed "patch" version update.

Can you at least revert this massive change in 9.0.3 and then announce to downstreams your intention to change this behavior for a future 10.x.x or at least 9.x.x release?

[RemiCardona]

We don't want to use an older version either, but that's exactly what we ended up doing. 9.0.2 does not exist inside our CI environment, at least for now.

[stmcginnis]

Also seeing this hit in some OpenStack projects.

[pfmoore]

Pip 10 is due out in about a month. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage.

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

Proposals for reintroducing some level of API support are certainly an option (see #5080 for example) but at this stage, it's too late for any such changes to make it into pip 10.

[stmcginnis]

With no warnings being raised by the code for those using the old way, nothing denoting this as "internal" with starting with _, and this just being a bugfix bump, I actually think it's quite easy to see this as a sudden, unexpected breakage.

[Miserlou]

Yes, this is the type of change that people could expect from a MAJOR version update. That would be fine.

Unfortunately, this massive change came in a PATCH update, which is supposed to fix things, not break them. This is the exact opposite of semantic versioning. And, as a result, we are seeing massive damage across the Python ecosystem.

You should revert this change ASAP in another PATCH update, then have the major breaking change come with the MAJOR version update. Now that you've already broken everything for everybody prematurely, I think they will be more aware that the actual major change is coming.

I also think it's a cop out to say that this was documented and announced already, considering that it was not documented in the stable documentation, and that the announcement said that the break would happen for the major version, not for in a patch a month prior.

Please, just save everybody a massive headache, revert this problem and start actually adhering to semantic versioning.

[pfmoore]

@Miserlou OK, I take your point - we targeted the internal renaming change for pip 10 because it's a major version. I don't really know the drivers for the patch release - @dstufft pinged me about it and it's apparently to avoid significant breakage for Mac OS users when an imminent deadline for TLS support hits us, which is before the pip 10 release. We expected the problems to be significant enough to warrant a short-term patch release. There was certainly no intention of breaking existing usage.

I have to leave the decision of a follow-up 9.0.3 to @dstufft - I don't have the details of what went into 9.0.2 or know whether it's even possible to identify how to fix this issue. And I can't judge whether pulling 9.0.2 altogether will be an overall benefit, as I have no idea how many people are affected by the Mac OS issue. I understand that (at least for some people) pinning to 9.0.1 is a solution, so that may end up being the least bad option.

[alex]

The macOS TLS issue will affect everyone using a system Python on macOS<10.13

[pradyunsg]

I have to leave the decision of a follow-up 9.0.3 to @dstufft

I'm of the same position as @pfmoore.

[ViktorHaag]

Workaround, if you have it available to you, is to check the order of imports and test moving importing pip above other package imports (specifically, importing pip before importing requests seems to solve some cases). (Note that this still implies use of pip's internals, which isn't officially supported.)

[anx-ckreuzberger]

Same issue here with pip 9.0.2 in a gitlab-ci with docker python 3.4: KeyError

  File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'

[anx-ckreuzberger]

FYI, 9.0.2 was released with broken builts:

Travis-CI Reference: https://travis-ci.org/pypa/pip/builds/354616774?utm_source=github_status&utm_medium=notification

Granted the errors might be unrelated, though this just smells like a "rushed release"...

[anx-ckreuzberger]

@pfmoore

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

[RonnyPfannschmidt]

pip always was documented as NOT having a consumable python api, i'm disappointed in people that even at that point try to turn around the "told you so since quite some years" karma

[diginc]

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

[Miserlou]

Hooray! Thanks for this!

I'd also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

[dstufft]

pip list --format=json?

[drunkwcodes]

Firstly thank you all contributing to pip people for it totally covers all of my use cases with a few user-friendly options and arguments.

Due to this "surprisingly widely adopted and useful undefined behavior," do we need to make a piplib as libgit2 for git? Please note that libgit2 is not sharing any code with git, and is maintained by other group from git's. And it is good for git ecosystem. Maybe piplib will cover some interesting use cases which we didn't expect.

Here is a similar story: https://public-inbox.org/git/1267957655.3759.29.camel@mattotaupa/T/

[sruggier]

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there's no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it's probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don't go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

[drunkwcodes]

@sruggier The packages metioned are all good, and WheelFile.install() also needs more promotion. But the one-stop service pip.main() provided is irreplaceable with all of the above. It's still worth trying.

The most important is that I hope these needs are held by another project, and pip10 will arrive smoothly without extra guarantees. The deprecation and minimizing the code base are the whole points for a major release.

Concrete implementation details for a permanent infrastructure "software" is ridiculous. You can't judge maintainers by the documented abuse case and hold back the wheel of pip.

If you insist to use pip as a lib, a lot of assumptions will need to be reconsidered.

[pfmoore]

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn't promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it's not hard to implement in a 3rd party module.

[honnibal]

I appreciate the work you all do on pip, and know it's not easy. But for the record:

I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10

spaCy has moved away from import pip. But some people are still using spaCy 1, which did import pip --- and for obvious reasons, didn't pin pip down to a patch release. If the change had come in at v10, our old versions wouldn't have been affected. We've just issued a hotfix to address this.

[kalefranz]

proper instruction on how to programmatically inspect python environments in a pip10 world

What is PyPA's position on distlib? Pip is obviously using it in some capacity, but so too is it using packaging, which distlib purports to deprecate.

If there isn't an official position, then at least current thoughts and opinions from pip's core maintainers would be much appreciated. If there are relevant discussions on this topic elsewhere, a simple pointer to other discussions is good enough for me.

[pfmoore]

I'm not aware that distlib deprecates packaging. It mentions "distutils2/packaging" but distutils2 was something different.

My personal view is that both distlib and packaging are perfectly reasonable things to use. You should evaluate them yourself to confirm they meet your needs, obviously, just like any other dependency you rely on.

[kalefranz]

Maybe deprecates is too strong of word then. The source of my current understanding:

https://distlib.readthedocs.io/en/latest/overview.html#distlib-evolved-out-of-packaging

[pfmoore]

Ah, that's a different "packaging" - it was the proposed stdlib module that was intended to replace distutils. It's completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

[tuukkamustonen]

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that :) See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.