ImportError: cannot import name 'SourceDistribution' from 'pip._internal.distributions.source'

[peterjc]

Environment

CircleCI using Conda, eg https://circleci.com/gh/peterjc/thapbi-pict/1073

• pip version: pip-19.2.3 (from conda) updated to 19.3 (via pip install --upgrade pip setuptools)
• Python version: python-3.7.3
• OS: Linux

I have made only minor changes to my code, but today the CircleCI builds broke during the build/install stage of my Python tool - multiple branches affected. The timing fits the release of pip 19.3 on PyPI.

Description

It appears a recent release of pip moved some internal files about, my guess is #6830.

Expected behavior

How to Reproduce

Full output on https://circleci.com/gh/peterjc/thapbi-pict/1073 which is running https://github.com/peterjc/thapbi-pict/blob/9fcec2da60e6e6ae5cf7ee6ad4b53dcc3a40cfe7/.circleci/config.yml#L56

...
python setup.py sdist --formats=gztar
python setup.py bdist_wheel
pip install dist/thapbi_pict-*.whl

Output

Traceback (most recent call last):
  File "/opt/conda/bin/pip", line 7, in <module>
    from pip._internal import main
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/__init__.py", line 9, in <module>
    from pip._internal.commands.download import DownloadCommand
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 10, in <module>
    from pip._internal.operations.prepare import RequirementPreparer
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 9, in <module>
    from pip._internal.distributions import (
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/__init__.py", line 1, in <module>
    from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution' from 'pip._internal.distributions.source' (/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/source/__init__.py)
Exited with code 1

[pfmoore]

It would be really helpful to know what in the CircleCI (or conda?) setup is using those internal APIs - the reorganisation shouldn't have affected any external code. The main issue we know of so far is the reorganisation of main, which impacts wrapper scripts, but this one looks a bit different. Any insight you can give would be very useful, thanks.

[peterjc]

Confirmed that removing pip install --upgrade pip setuptools fixed things,

peterjc/thapbi-pict@4060404
https://circleci.com/gh/peterjc/thapbi-pict/1074

i.e. Something broke in how I updated from pip 19.2.3 to 19.3

I am not intentionally using the CircleCI provided Python, rather the Conda provided Python - is there value in switching from pip ... to python -m pip ... just to remove the possibility of this being due to mixing Pythons?

[peterjc]

Working on a branch now, added a little more logging and reverted back to the broken version:

https://circleci.com/gh/peterjc/thapbi-pict/1076

This confirms the traceback is trigged by pip install dist/thapbi_pict-*.whl (wildcard to match the single freshly built wheel).

[peterjc]

OK, using python -m pip ... rather than pip ... does not help:

https://circleci.com/gh/peterjc/thapbi-pict/1077
peterjc/thapbi-pict@c168fbb

[pfmoore]

For what it's worth, I've just confirmed that from pip._internal import main works on a simple upgrade of pip (on Windows, so not really a good comparison, but anyway...). So I suspect there's something mismatched in your environment, although I can't see from the traceback what it is.

[peterjc]

CircleCI has the capability to re-run a job with SSH access to the VM (not that I've ever used it), so in theory this could be used for debugging, but I don't really know where to start.

I have a work around for now, and at some point conda-forge will have pip 19.3 packaged too, so unless other people see the same exception, it seems a low priority?

[pfmoore]

OK, cool. If you have a workaround then I'm fine leaving it there for now. If others add comments reporting it's happening elsewhere, we can do some more digging then - maybe they will have extra information that will help.

[maulin-amwater]

I am facing a similar issue on macOS, python: 3.6.9. pip command on my terminal doesn't work anymore. Any suggestions?

[nshmura]

I have same problem on CircleCI since 1 hours ago.
Version changing of docker image from python:3.7 to python:3.7.4 fixes this issue. (just workaround) . like this:

version: 2
...
jobs:
  build:
    docker:
      - image: python:3.7.4
...

[bgkelly]

Are any of you guys using caching? I think it's possible some pip internals changed and when you restore the cache of the old python version pip breaks. Suggest you invalidate your circle CI caches by changing the keys and see what happens ... (worked for me!)

[ddobson]

As suggested by @bgkelly clearing the cache also worked for me!

[dlrust]

Yep - same for me - changing the cache key has solved this for our team's projects on circleci. Thanks @bgkelly

[dedoussis]

Changing the cache key did not solve it for me.
Adding a comment in requirements.txt did it though, but that's not a desired solution.

[pradyunsg]

Alright, so this seems like a caching issue for CircleCI.

Could someone please reach out to CircleCI folks (via their support channels)? It'd be great if they could look into this and possibly chime in here.

[wusala01]

Hey people,

I really don't know if this helps you in any way, but I got the same Error Message on my MSYS setup trying to install WeasyPrint.

After uninstalling python3-pip and python3-setuptools i deleted all pip folders in /usr/lib/python3.7/site-packages/ and reinstalled both.. solved that specific error.

pacman -R python3-pip python3-setuptools
rm -R /usr/lib/python3.7/site-packages/pip
rm -R /usr/lib/python3.7/site-packages/pip-19.3.1.dist-info
pacman -S python3-pip python3-setuptools

To be fair, I now ran into compiler issues... but maybe i could help....

[sbellem]

Just in case this may help, I have encountered the error in a different context: when copying a virtual environment from one docker image to another, in a multistage setup.

The source image contains a less recent version of pip whereas the target image contains an upgraded version of pip. After copying the virtual environment directory of the source image into the target image, there are 2 different directories (pip-<version>.dist-info) under the virtual environment site-packages and pip does not work, giving the same error as reported in this issue.

I did not spend time troubleshooting the issue as I am somewhat skeptical of this approach involving copying the virtual environment into an environment where there are already packages installed.

In any case, it appears to be a package collision problem.

Hope this helps! 😄

[torbenfox]

Hi
I believe we see the same issue on our internal ReadTheDocs server. It runs the following command when building a document:

var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/bin/python -m pip install --upgrade --cache-dir /var/lib/readthedocs/repo/user_builds/some_doc/.cache/pip pip

If I clear the build cache via the ReadTheDocs GUI interface, the build is successful - but only until next time a build is started.

Here is the full log:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/commands/__init__.py", line 9, in <module>
    from pip._internal.commands.download import DownloadCommand
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 10, in <module>
    from pip._internal.operations.prepare import RequirementPreparer
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 9, in <module>
    from pip._internal.distributions import (
  File "/var/lib/readthedocs/repo/user_builds/some_doc/envs/latest/lib/python3.6/site-packages/pip/_internal/distributions/__init__.py", line 1, in <module>
    from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution'

[jiangge]

Hi,

I've encountered the problem too.

OS: Debian GNU/Linux 10 (Linux 4.19.0-6-amd64)
pip: 19.3.1
conda: 4.7.12
Python:3.7.4

$ pip
Traceback (most recent call last):
File "/home/jiang/anaconda3/bin/pip", line 7, in
from pip._internal import main
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/init.py", line 40, in
from pip._internal.cli.autocompletion import autocomplete
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 8, in
from pip._internal.cli.main_parser import create_main_parser
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 11, in
from pip._internal.commands import (
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/commands/init.py", line 9, in
from pip._internal.commands.download import DownloadCommand
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 10, in
from pip._internal.operations.prepare import RequirementPreparer
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 9, in
from pip._internal.distributions import (
File "/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/distributions/init.py", line 1, in
from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution' from 'pip._internal.distributions.source' (/home/jiang/anaconda3/lib/python3.7/site-packages/pip/_internal/distributions/source/init.py)

[pfmoore]

@pradyunsg Hats off for the fast fix, we really appreciate it <3

Agreed, fantastic turnaround on the fix here!

[michel-rodrigues]

Thanks @pradyunsg

[browarsoftware]

Same error as above.
Python version: python-3.7.6
OS: Windows

Works after moving to PIP 19.3.1

[danys94]

Works great now, thanks.

[ssbarnea]

Considering the nature of this bug, can you please remove 20.0 release from PYPA? https://pypi.org/project/pip/#history

Another question would what we could do to avoid a similar regression in the future.

[pradyunsg]

Sorry everyone for the breakage, and thanks for working with us here to fix this breakage!

@xavfernandez figured out why this happened and we'll be making changes to our release process to prevent similar failures in the future: #7624

[MarkBird]

great work, thanks for the lightning fast turnaround!

[CharrierCoop]

Hello, I'm stuck in 20.0
All the commands that I tried return this error about 'SourceDistribution'.

pip install --upgrade pip
pip install pip "<20.0"
pip install --upgrade "pip==19.3.1"

I tired to uninstall too :

pip uninstall pip
python -m pip uninstall pip setuptools

Anyone have another solution to uninstall pip?

Ubuntu 18.04.3 LTS
Python 3.6.9
run in virtualenv

[pradyunsg]

Hi @CharrierCoop! You should be able to download get-pip.py and install pip 20.0.1 using that.

See https://pip.pypa.io/en/stable/installing/

[CharrierCoop]

Perfect !!
thank you @pradyunsg

[pradyunsg]

can you please remove 20.0 release from PYPA?

No. We're not going to be deleting it. We have a standard-backed feature for "yanking" broken releases. Once PyPI adds support for that, we'll yank this release (if anyone cares by then).

See https://www.python.org/dev/peps/pep-0592/#motivation for more details.

[Lotus-Blue]

sudo python get-pip.py pip==19.3.1"
fix this issue

thank you very much

[abhiTronix]

sudo python get-pip.py pip==19.3.1"
fix this issue

No need this issue was fixed by @pradyunsg

[rohansh-tty]

Hi @CharrierCoop! You should be able to download get-pip.py and install pip 20.0.1 using that.

See https://pip.pypa.io/en/stable/installing/

I tried doing the same on Raspberry Pi 4( Python 3.7.3, pip version - 20.0 and OS - Raspbian Buster Lite), but it wasn't successful.
ERROR: Could not find a version that satisfies the requirement pi p
ERROR: No matching distribution found for pip

Any solution?

[pradyunsg]

@Gilf641 Please file a new issue -- it'll be easier to manage for the maintainers.

[pradyunsg]

I'm gonna go ahead and close this issue now, since it seems like things are working for most users.

If you're still facing issues, please look for duplicates filed in earlier (since Jan 21, for pip 20.0) and if there are none, file a new issue.