After pip 10 upgrade on pyenv "ImportError: cannot import name 'main'"

[KleinerNull]

Maintainer note: Anyone that still gets this issue please see #5599.

---

• Pip version: 10.0
• Python version: 3.6.2
• Operating system: Ubuntu 16.04

Description:

After upgrading pip from 9.03 to 10.0 via pip install pip --user --upgrade in a pyenv environment pip refueses to start and raise this instead:

Traceback (most recent call last):
  File "/home/kleinernull/.pyenv/versions/3.6.2/bin/pip", line 7, in <module>
    from pip import main
ImportError: cannot import name 'main'
Traceback (most recent call last):
  File "/home/kleinernull/.pyenv/versions/3.6.2/bin/pip", line 7, in <module>
    from pip import main
ImportError: cannot import name 'main'

The content of all three different pip files is the same:

~ ⟩ cat .pyenv/versions/3.6.2/bin/pip                                                                            ~
#!/home/kleinernull/.pyenv/versions/3.6.2/bin/python3.6

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main as _main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(_main())

~ ⟩ cat .pyenv/versions/3.6.2/bin/pip3                                                                           ~
#!/home/kleinernull/.pyenv/versions/3.6.2/bin/python3.6


# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

~ ⟩ cat .pyenv/versions/3.6.2/bin/pip3.6                                                                         ~
#!/home/kleinernull/.pyenv/versions/3.6.2/bin/python3.6

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The same happend with my 3.6.1 environment too.

Temporaly fix

According to the code of the master branch the import should be that:

#!/home/kleinernull/.pyenv/versions/3.6.2/bin/python3.6

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main as _main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(_main())

And this resolves the issue. I have no clue if this has something to do with the upgrade itself or with pyenv as environment.

[pradyunsg]

Hey @KleinerNull!

I have no clue if this has something to do with the upgrade itself or with pyenv as environment.

I think that this is an environment issue. I suggest you open an issue over at pyenv as I think the folks there would be in a better position to comment on this/fix it.

[KleinerNull]

@pradyunsg

I opened a issue at the pyenv repo.

[JohnVonNeumann]

We are suffering from this one as well, it's broken our pipeline. Operation being run:

 11 #upgrade pip and install uwsgi
 12 pip install --user --upgrade pip
 13 pip install uwsgi

Ubuntu 16.04
Python3.6

[HayaoSuzuki]

We are encountering the same trouble.

// in host
$ docker pull ubuntu:xenial
$ docker run --name pip-test --rm -it ubuntu:xenial bash

// in container
# apt update
# apt install -y python-dev python-pip
# pip install --upgrade pip
Collecting pip
  Downloading pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |################################| 1.3MB 865kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
# pip install requests
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

[maingoh]

Same here .. Exact same output as @HayaoSuzuki and we don't use pyenv

[pfmoore]

If you're getting this outside of pyenv I suspect it's more likely to be related to #5221

[PhML]

Just to mention that pip install --user --upgrade pip on Ubuntu 16.04 also breaks.

[ghost]

Interestingly easy installing pip installs 10.0.0 but does not exhibit the issue

// in host
$ docker pull ubuntu:xenial
$ docker run --name pip-test --rm -it ubuntu:xenial bash

// in container
# apt update
# apt install -y python-setuptools 
# easy_install pip
Installed /usr/local/lib/python2.7/dist-packages/pip-10.0.0-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

# pip install requests
Collecting requests
 Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
   100% |################################| 92kB 2.9MB/s 
Collecting certifi>=2017.4.17 (from requests)
 Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
   100% |################################| 153kB 4.4MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests)
 Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
   100% |################################| 143kB 4.5MB/s 
Collecting idna<2.7,>=2.5 (from requests)
 Downloading idna-2.6-py2.py3-none-any.whl (56kB)
   100% |################################| 61kB 7.4MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests)
 Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
   100% |################################| 133kB 4.4MB/s 
Installing collected packages: certifi, chardet, idna, urllib3, requests
Successfully installed certifi-2018.1.18 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

[RonnyPfannschmidt]

which is not surprising, since that pip is egg-installed in a different place thus no longer affecting the global pip - it also means you now have a very interesting workingset for python

[davidjlloyd]

While I appreciate that the underlying issue reported here and in #5221 is the environment, the cause is primarily that the import from pip import main was broken as the package pip.main was moved to pip._internal.main. It would be trivial to add a link from pip.main to pip._internal.main to fix this (whereas fixing the environment is a lot of work across many locations for many people). Is there a good reason to not do this?

[KleinerNull]

@davidjlloyd

This comment gives some informations about not to do this, but I am not entirely sure if this is also important for the pip script itself. However, it solved the problem for me.

[pfmoore]

@davidjlloyd Because from pip import main was never supported, basically. And while it's easy to say "yes, but it's a simple API and it just worked", it really didn't - we've had multiple bug reports from people expecting certain behaviours from it, that we simply don't cater for (running pip.main in multiple threads, expecting pip.main to not change the configuration of the logging system, ...)

Rather than keep explaining to people that they shouldn't do this, and continually dealing with the fact that people are assuming "if I can find a function to call, it's supported" we moved everything to the _internal namespace to make it abundantly clear that you're not supposed to call it.

The bulk of the complaints have come from people using pip.main - which is ironic, as it's so easy to call pip in the supported command line way via subprocess. So of all the possible breakages, this is the easiest to fix - and yet people still haven't fixed it, even after months of warnings. (Although to be fair to the Linux distributions, they don't support people upgrading their system packages using pip, so reports like #5221 of cases where distro-supplied scripts break is fundamentally user error, not failure of the distros to address the pip 10 changes - something I'm sure they are handling perfectly well).

[peteflorence]

I can also confirm that this issue is absolutely destroying my previously very stable process of a building a docker image, here are example minimal things I'm doing inside my docker image build process:

+ pip install -U pip setuptools
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl (569kB)
Installing collected packages: pip, setuptools
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0 setuptools-39.0.1

...

+ pip install jupyter opencv-python plyfile pandas
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

[peteflorence]

Fix for us was pinning to pip 9.03, so:

pip install --upgrade pip==9.0.3

instead of

pip install -U pip

obvious fix but in case it helps somebody else!

[pfmoore]

@peteflorence So presumably when running docker, you're creating a base image, then running pip install -U pip setuptools as root in that image? Is there a reason you need the latest pip/setuptools if all you're doing is installing other packages with them? Could you not simply upgrade to the latest pip/setuptools available as a distro package?

I appreciate that this is a problem for you - it seems to be common for docker builds to be more inclined to do stuff as root than if you were in a normal OS (probably because a docker image is isolated). But it's still not a good idea to do this. The problem is that pip doesn't manage /usr/bin/pip, so we have no way of "fixing" that to work with pip 10.

What you could do is switch from using /usr/bin/pip to using python -m pip. It's still not supported, and may hit other issues (I don't know what changes your base OS vendor may have made to the system pip) but it will avoid the issue in /usr/bin/pip while you sort out a longer-term solution to your issue.

[pfmoore]

Pinning to pip 9 is also a solution, but this begs the question, why upgrade your OS pip at all if pip 9 is OK? Does your vendor not offer a packaged version of pip 9?

[davidjlloyd]

Rather than keep explaining to people that they shouldn't do this, and continually dealing with the fact that people are assuming "if I can find a function to call, it's supported" we moved everything to the _internal namespace to make it abundantly clear that you're not supposed to call it.

I'm not sure that aggressively breaking existing uses of an unsupported feature instead of deprecating the feature for a couple of major releases is the best approach. Our initial reaction was to pin pip back to 9.0.3, and more lazy developers would probably just call it a day at that point. This would leave a lot of users stubbornly clinging to old releases, which I doubt anyone wants. However your motivation makes sense, and the eventual result is the same.

For any users hitting this issue, I think the nicest solution for replacing system or pyenv installations was kindly provided by @standag here: #5221 (comment)

This solution should work for anyone building in Docker such as @peteflorence without pinning to stale releases or anything horrible like that.

[jackton1]

Temporary Fix upgrade pip using.

curl https://bootstrap.pypa.io/get-pip.py | python3

Instead of pip install -U pip

For pip2 pip2 install --upgrade pip

[bkbncn]

I upgraded with pyenv, both python2 and python3, now pip2 not work and pip3 works
After comparing pip2 and pip3, the difference is the import line

pip2
from pip import main

pip3
from pip._internal import main

After substitute pip2 import line with pip3 version, it works

[brandonshough]

Having the same issue, heh.

[fabyc]

Same issue, but solved with solution: #5240 (comment)

[ernestm]

Same issue:

+ pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl
 (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
+ pip install awscli requests simplejson
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

Now it's weird, it installs pip 10 to /usr/local/bin, and that is first in the PATH before /usr/bin, but it doesn't use it, not until you go to a new shell. I saw this happen when I went to this box to try to install a newer awscli by hand...

$ python --version
Python 2.7.12
$ pip --version
pip 10.0.0 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
$ sudo pip install --upgrade awscli 
 <blah blah>
$ aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1049-aws botocore/1.4.70
$ /usr/local/bin/aws --version
aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-1049-aws botocore/1.10.4
$ which aws
/usr/local/bin/aws
$ echo $PATH
/home/ec2-user/bin:/home/ec2-user/.local/bin:/opt/bamboo-elastic-agent/bin:/opt/jdk-8/bin:/opt/maven-2.1/bin:/opt/maven-1.0.2/bin:/opt/ant-1.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/bin:/bin:/opt/puppetlabs/bin

[ikreymer]

All those issues seem to stem from the same problem: upgrading pip, but keeping on using an old launcher that still use the old entrypoint. A good way to avoid this kind of issue is to use python -m pip.

Yep, I think that seems to be the root of the issue, and happens across all different platforms mentioned in this thread, including Windows.

In case this helps anyone else, I can confirm that switching appveyor.yml pip upgrade from:

pip install --disable-pip-version-check --user --upgrade pip

to:

python -m pip install --upgrade pip

does fix the issue. Now to update several more repos!

[ben-albrecht]

I encountered the same symptom discussed here in a different situation.

I was trying to use a locally installed pip, on a Ubuntu 16.0.4 system:

curl -O https://bootstrap.pypa.io/get-pip.py
export PYTHONUSERBASE=$(pwd)
python ./get-pip.py --user
export PYTHONPATH=$(pwd)/lib/python2.7/site-packages

python ./bin/pip --version

Traceback (most recent call last):
  File "/path/to/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal

It turned out that Ubuntu prepends a pip-specific path to sys.path via site.py, which was overriding my PYTHONPATH, shown below:

import sys
print(sys.path)
['', '/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg', '/path/to/lib/python2.7/site-packages`, ...]

I tried avoiding the prepend with -S flag for python, documented in the man page as:

-S Disable the import of the module site and the site-dependent manipulations of sys.path that it entails.

and it worked:

python -S ./bin/pip --version

pip 10.0.1 from path/to/bin/pip (python 2.7)

[jputrino]

Sharing in case this might help others - In my docker image (base is ubuntu:xenial) I received the following error:

Step 8/12 : RUN pip install -U pip  && pip install -r /tmp/requirements.txt
 ---> Running in e4ff51b013f0
Collecting pip
  Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.1
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

I changed pip install -U pip && pip install -r /tmp/requirements.txt to pip2 install -U pip && pip2 install -r /tmp/requirements.txt. This resolved the issue.

[EricCousineau-TRI]

I'm not sure if I saw an answer / response to @davidjlloyd's comment:

I'm not sure that aggressively breaking existing uses of an unsupported feature instead of deprecating the feature for a couple of major releases is the best approach.

Can I please ask why there was not a deprecation process in place for this?
It seems like it would have been quite easy, on the import of pip, check sys.argv[0]; if it's not pip or pipX[.Y], spew some DeprecationWarnings that this will fail in release X+, with a link to what you mentioned: https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

Is there a process in place to try and ensure this kind of thing is hopefully avoided in the future?

[pfmoore]

I'm not sure if I saw an answer / response to @davidjlloyd's comment:

It's been answered repeatedly, Maybe not on this specific issue, but a search of the issue tracker should find plenty of discussion on the matter.

Can I please ask why there was not a deprecation process in place for this?

Because it was never supported. Why would we warn that we're desupporting something we never supported? People assumed it was OK to read pip's source code and use functions they found in there in their own code. It never was. We said it could break, and in pip 10 it did.

It seems like it would have been quite easy, on the import of pip, check sys.argv[0]; if it's not pip or pipX[.Y], spew some DeprecationWarnings

I'm not sure it's as easy as you think. And I say that from the perspective of someone who has had to deal with issues where warnings added in pip 10 were being triggered when we hadn't expected them to be...

And again, there's no need to deprecate something that's not supported in the first place.

Is there a process in place to try and ensure this kind of thing is hopefully avoided in the future?

What sort of thing? Breakage caused by us changing things that we don't guarantee backward compatibility for? No. There's no need for us to avoid that. Although in fact, we do try to manage the process, as a courtesy to our users (not an obligation!). In this case, we publicised the change 6 months in advance, offered suggestions for people who needed to change their code, and have been spending a lot of time since the release helping users who have had problems because software they rely on hasn't heeded those warnings. That's a lot of work that a very small volunteer group put into trying to mitigate a situation caused by people expecting support that was never offered or promised. You're welcome.

We do have processes (deprecation warnings, etc) in place for changing things that we do guarantee compatibility for - but importing pip into your own program isn't one of them.

[OneLogicalMyth]

I had a bash script that installed flask and requests the upgrade broke my script, but I do understand the reasons stated above. My work around to keep my script working was to simply launch a new bash process, maybe that is wrong but it worked for me. Hopefully it might help others with similar scripts.

pip install --upgrade pip
echo "pip install Flask" | bash
echo "pip install requests" | bash

[austinbutler]

@OneLogicalMyth what you are looking for may be hash -d pip, see #5221 (comment).

[OneLogicalMyth]

completely missed that even after reading it carefully, thank you @austinbutler this has resolved my issue.

[pradyunsg]

The only reason this issue was kept open and not closed as a duplicate immediately was because I was concerned if pyenv does something with shims and if pip would need to help out in some way. That wasn't the case so, this issue is closed now.

I've listed the workarounds before (all of them trivial) for almost all the these problems mentioned here - take a look at #5221 (comment). Hopefully, that addresses all the concerns raised in this issue. If it doesn't, please open a new issue.

@benoit-pierre I added your suggestion of using python -m pip to the linked comment. Thanks for that. :)

---

Installing pip using easy_install could well have issues, and if it does, we won't be able to support you, because (as @pradyunsg said) easy_install is old, not actively maintained, and missing recent features.

Thanks for clarifying my position @pfmoore. That's what I was talking about.

I'm getting increasingly confused as to what the various problems being discussed are.

Me too.

---

Just some general tips (take it or leave it), about things I saw in this issue:

• If the pip command doesn't work for some reason, try running python -m pip. Chances are python -m pip will work for you even if your wrapper scripts break. If it doesn't work, open an new issue.
• Don't run pip as root or do sudo pip. You'll likely break your system and if that's not bad enough, you're doing remote code execution as root.
• Don't use easy_install for the reasons are quoted at top of this comment. Here's more context (that's slightly out of date).
• Please please don't pin to pip 9. I strongly believe that doing that is actively acting to slow down the progress of the Python Packaging in general.
  • pip 9 is never going to support PEP 517/518 and staying on pip 9 means you'll have to take the pains to switch over later, when a package breaks last minute because they switch to the newer packaging standards.

---

To summarize, this is likely not a single project/person's fault and everyone is well reasoned for taking their actions. Yes, your environment broke. We understand. Do not throw blame at people who're volunteering their free time, to help you out right now and to develop pip.

Want to help us? https://donate.pypi.org is a thing and if not that's not your type of thing, there's lot of open issues in this very issue tracker that you can help us out with.

I'm gonna step back from this issue now.

[pradyunsg]

Okay, one last thing, if someone still wants to discuss regarding our decision to move all of our codebase to pip._internal, open a new issue and gimme a mention. I'll spend some time to write up something with links and all, about why pip did this. That way, @pypa/pip-committers would have a single place to people link to on this topic.

[freckletonj]

For the time being, this worked on python3 / pip3, to roll it back

python3 -m pip install --user --upgrade pip==9.0.3

[pradyunsg]

@freckletonj Please do not tell people to revert to pip 9. That is not how you should resolve this issue. There are much better ways.

I have linked to a comment just above yours, listing out ways to resolve this issue.

[freckletonj]

sorry @pradyunsg , but that still doesn't work:

$ pip3 install --upgrade --user pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.1
$ pip3
Traceback (most recent call last):
  File "/usr/local/bin/pip3", line 7, in <module>
    from pip import main
ImportError: cannot import name 'main'

[pradyunsg]

From #5221 (comment), which I've linked to above:

hash -r pip # or hash -d pip

[pradyunsg]

If anyone has any other queries, please open a new issue.

[pradyunsg]

#5599 provides information and provides a single location to seek help toward resolving this issue for end users.

The comments section of that issue are open for users to discuss specific problems and solutions. :)