pip install on python 2.7 attempting to install numpy 1.17, contrary to NEP 14

[IPetrik]

pip install numpy on python 2.7 attempts to install numpy 1.17, which dropped python 2.7 support. This is contrary to the functionality described in NEP 14, which states:

... pip install numpy on Python 2 will continue to give the last working release in perpetuity...

Numpy/Python version information:

pip version: 19.2.2
python: 2.7.15 |Anaconda, Inc.| (default, Oct 23 2018, 18:31:10) \n[GCC 7.3.0]

[seberg]

There seems to be a bug in pip or setuptools which causes this, can you try upgrading pip? (see also gh-13911 which I expect is a duplicate). You can probably upgrade pip with pip itself: python -m pip install --upgrade pip and python -m pip install --upgrade setuptools?

[glsscnnn]

the real question is why you're using python 2.7

[rgommers]

the real question is why you're using python 2.7

there's lots of valid reasons to still run 2.7, please don't start that debate here.

[IPetrik]

There seems to be a bug in pip or setuptools which causes this, can you try upgrading pip? (see also gh-13911 which I expect is a duplicate). You can probably upgrade pip with pip itself: python -m pip install --upgrade pip and python -m pip install --upgrade setuptools?

I am running the latest versions of both - pip 19.2.2 and setuptools 41.1.0.

the real question is why you're using python 2.7

there's lots of valid reasons to still run 2.7, please don't start that debate here.

Indeed there are. In my case I am trying to run an academically developed scientific analysis software that has not been ported to py3.

But the point here isn't whether or not I should be using python 2.7, it is that numpy made a formal statement of expected functionality, and it is not the true functionality. That, in my mind, is the bug.

[charris]

I noticed yesterday that pip would not install pytest 5.0.1 for Python 2.7 because 2.7 was not supported, so pip is working correctly in some cases. I wonder if there is some information somewhere that needs updating for 1.17?

[charris]

You might try python2.7 -mpip to make sure pip knows what version is running. It also looks like you are using Anaconda, which can be a complication when mixed with pip.

[charris]

What does which pip show?

[seberg]

Hmmm, on my machine, pip 18.1 and setuptools 41.0.1 work fine (both binary and no binary mode). Upgrading to 19.2.2 and 41.1.0 does not change anything for me. The python version is 2.7.16+ (debian).

As Chuck notes, I assume you make sure to use python2.7 -mpip instead of pip to be certain that you get the newest installed version (in case you are using --user or have another pip version around for some reason.)?

It would be really good to understand what exactly is causing this.

[rgommers]

@IPetrik would you be able to put together a completely reproducible example? You're using Anaconda, so something like:

conda create -n py27-empty
conda activate py27-empty
conda install pip
# Some diagnostics
which pip
pip --version
conda list setuptools

pip install numpy

I tried several variations of this, but cannot reproduce your issue.

[IPetrik]

After trying to prepare a minimal example, I figured out the issue: we use a local devpi PyPI cache server that was in my pip.conf as an extra-index-url. If I remove that from my pip.conf, everything works right... oops.

[charris]

Hmm, it seems that pip should handle that. Is there something special about the cache and pip.conf? I'm unfamiliar with those things. @njsmith Any ideas?

[IPetrik]

I wonder if the devpi server is not forwarding the metainformation about the available versions for a given package properly.

[rgommers]

Found one reason that a new pip can attempt to install 1.17.0 for Python 2.7. This happens when a package uses setup_requires='numpy'. This is because that will invoke easy_install unless that package has a pyproject.toml (see pypa/pip#6945 (comment)).

There's a workaround: explicitly do pip install numpy before pip install package-with-setup-requires.

And the fix is: that package should add a pyproject.toml, that will make this issue go away for a new release of that package.

[artob]

Found one reason that a new pip can attempt to install 1.17.0 for Python 2.7. This happens when a package uses setup_requires='numpy'. This is because that will invoke easy_install unless that package has a pyproject.toml (see pypa/pip#6945 (comment)).

There's a workaround: explicitly do pip install numpy before pip install package-with-setup-requires.

A concrete case of that being pip install pycuda, which fails like so:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-LYaxup/pycuda/setup.py", line 251, in <module>
    main()
  ...
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 44, in _execfile
    code = compile(script, filename, 'exec')
  File "/tmp/easy_install-mp9NYd/numpy-1.19.0rc1/setup.py", line 182
    raise ValueError(f'Submodule {p} missing')
                                            ^
SyntaxError: invalid syntax

The workaround is simple, as @rgommers noted: first execute pip install numpy==1.16.6 (for the last Python 2.7-compatible NumPy release) and then re-try pip install pycuda.

[rgommers]

That's a pip bug, and a very serious one it looks like. I can reproduce that with pip 20.1.1. pip-2.7 install numpy correctly installs 1.16.6, so the metadata for numpy on PyPI is correct. pip-2.7 install pycuda does try to install latest numpy instead of 1.16.6.

Likely that's already reported, I have no time to look into it the coming days. @artob if you look if there's a Pip issue for this and open one otherwise, that'd be much appreciated.