NumPy version written out incorrectly in egg-info filename

[larsoner]

NumPy prefers using distutils to setuptools for the install step and thus creation of the egg-info filename. This means that the egg-info filename gets the + in the version replaced with _ and written out as numpy-1.11.0.dev0_2329eae.egg-info, despite the fact that the version is numpy-1.11.0.dev0+2329eae.

This causes a problem for satisfying requirements for other packages, since setuptools sees this as a LegacyVersion (after replacing the _ with a - in the filename), and thus you can get errors like this:

pkg_resources.VersionConflict: (numpy 1.11.0.dev0-2329eae (/home/larsoner/.local/lib/python2.7/site-packages), Requirement.parse('numpy>=1.6.1'))

One solution that might work is using setuptools for the install step, because setuptools creates the name with the + in it and it thus gets parsed properly. However, given the lengths taken to triage distutils vs setuptools in that step, I assume there is a reason why distutils is preferred. And it looks like back in 2013 this was addressed:

scipy/scipy@a4e93fb#diff-2eeaed663bd0d25b7e608891384b7298L185

Maybe setuptools would be okay to use now...?

Related setuptools ticket:

https://bitbucket.org/pypa/setuptools/issues/419/setuptools-pkg_resources-fails-to-detect

Any ideas for how to solve this problem?

[rgommers]

Sigh. At some point the time will come where we have to switch to setuptools (and/or install_requires). Not sure if that's now. Whenever that time comes, it won't be because the Python packaging developers have fixed any of the issues that have stopped us in the past, but because the cost of accepting what they break becomes higher than the cost of switching.

Besides the executable bit issue there's things like:

• numpy installation thru install_requires directive issue (Trac #1841) #2434 (comment)
• MAINT: add install_requires to setup.py scipy/scipy#465
• MAINT: setup.py cleanups and additions scipy/scipy#454 (comment)
• setup.py sys calls break in a pip install scipy/scipy#4474 (comment)
• setuptools incorrect but never fixed PYTHONPATH handling
• (there's more, I should have kept a list somewhere but didn't)

I don't want to keep spending mental effort on this nor stand in the way of progress, so I'm going to change my position on using setuptools (but not install_requires) from "no" to neutral. The only thing I'd like to see is that if the change to setuptools is made, it is done only if one or more people commit to fixing the resulting issues in docs, numpy.distutils and/or setuptools itself.

EDIT: note that I'm far from the only skeptic, so it will need discussing on the mailing list in any case.

[rgommers]

@Eric89GXL I want to make it clear that I'm not asking/inviting you to volunteer for this (it's not going to be fun). Although of coarse you're free to do so. I'd much rather see you do cool signal processing stuff:)

[njsmith]

Hi Ralf,
Doesn't really help in the short term, but you might be interested in
https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html .
Discussion is still in the mutual incomprehension phase, but I have hopes
:-)
.
Regarding the horribleness of 'pip install -U', my understanding is that
the pip developers have agreed that if someone shows up with a patch
implementing a non-recursive 'pip upgrade' command and deprecating 'install
-U', then they'll merge it. I've been pointing this out to people, but no
one has taken me up on it yet; still, if you know someone who might feel
inspired...
On Oct 9, 2015 5:16 PM, "Ralf Gommers" notifications@github.com wrote:

Sigh. At some point the time will come where we have to switch to
setuptools (and/or install_requires). Not sure if that's now. Whenever
that time comes, it won't be because the Python packaging developers have
fixed any of the issues that have stopped us in the past, but because the
cost of accepting what they break becomes higher than the cost of switching.

Besides the executable bit issue there's things like:

• numpy installation thru install_requires directive issue (Trac #1841) #2434 (comment)
  numpy installation thru install_requires directive issue (Trac #1841) #2434 (comment)
• MAINT: add install_requires to setup.py scipy/scipy#465 MAINT: add install_requires to setup.py scipy/scipy#465
• MAINT: setup.py cleanups and additions scipy/scipy#454 (comment)
  MAINT: setup.py cleanups and additions scipy/scipy#454 (comment)
• setup.py sys calls break in a pip install scipy/scipy#4474 (comment)
  setup.py sys calls break in a pip install scipy/scipy#4474 (comment)
• setuptools incorrect but never fixed PYTHONPATH handling
• (there's more, I should have kept a list somewhere but didn't)

I don't want to keep spending mental effort on this nor stand in the way
of progress, so I'm going to change my position on using setuptools (but
not install_requires) from "no" to neutral. The only thing I'd like to
see is that if the change to setuptools is made, it is done only if one
or more people commit to fixing the resulting issues in docs,
numpy.distutils and/or setuptools itself.

—
Reply to this email directly or view it on GitHub
#6257 (comment).

[rgommers]

Wow, you guys have been busy! I subscribe to distutils-sig a couple times per year but always unsubscribe quickly (for obvious reasons). This discussion is a good reason to re-subscribe once more....

[rgommers]

Regarding the horribleness of 'pip install -U', my understanding is that the pip developers have agreed that if someone shows up with a patch implementing a non-recursive 'pip upgrade' command and deprecating 'install -U', then they'll merge it

Last time I checked that was contingent on someone also writing a brand new dependency solver: pypa/pip#59 (comment) (which I find quite unreasonable, but well...)

[njsmith]

AFAICT the consensus in that bug thread is:

1. we want upgrade and upgrade-all
2. upgrade-all requires a shiny new dependency solver (which seems
   reasonable, because globally solving a few hundred simultaneous upgrades is
   genuinely non-trivial)
3. therefore we have to wait before adding upgrade and upgrade-all.
   .
   Notice the logic gap between steps (2) and (3) :-). AFAICT this is just
   people (random passerbys) being confused; no one has actually suggested any
   reason why we have to wait before adding 'upgrade' as a shorthand for 'pip
   install PKG=LATEST'.
   On Oct 10, 2015 12:02 PM, "Ralf Gommers" notifications@github.com wrote:

Regarding the horribleness of 'pip install -U', my understanding is that
the pip developers have agreed that if someone shows up with a patch
implementing a non-recursive 'pip upgrade' command and deprecating 'install
-U', then they'll merge it

Last time I checked that was contingent on someone also writing a brand
new dependency solver: pypa/pip#59 (comment)
pypa/pip#59 (comment) (which I
find quite unreasonable, but well...)

—
Reply to this email directly or view it on GitHub
#6257 (comment).

[rgommers]

That was a painful read. Good luck with that call (I'll watch it later but can't join).....

Some points that come to mind after reading the whole thing at once:

• might be worth pointing out that numpy isn't unique in terms of build complexity and dependencies, which is what the pypa crowd seems to think (each time an example is given they look at only that one and even call that hypothetical (pyopenblas)). Other projects like scipy / scikits / pytables / etc. are just as complex, and many people in the community also have their own packages with nontrivial build requirements.

• 95% of the discussion is about metadata/dependencies, but an actual interface for pip and other install tools to call a build tool it doesn't know about is at least as important. I think you can decouple those two topics more than you did, which would make it a lot easier to make progress.

  • Someone (Marcus Smith?) proposed letting the pip authors make a draft of this interface, but no follow-up on that one.

[rgommers]

Notice the logic gap between steps (2) and (3)

I admit that I'm then one of those passers-by. Note that the comment I linked to says "wait ..before releasing any shiny new commands...". On an issue that is about 2 new commands it clearly implies that they're both on hold, flawed logic or not.

[rgommers]

After looking at https://github.com/pypa/interoperability-peps I'm going to suggest that completely decoupling metadata (a minefield with lots of PEPs) from the build interface (currently zero PEPs) and writing a PEP only for the latter (which should just be fine) would be much more productive.

[njsmith]

Oh, I totally agree that that's what people have said in that bug thread,
my point is that if someone showed up with a fix for just the upgrade part
then when you squint at the thread you find few arguments against it.
On Oct 10, 2015 3:01 PM, "Ralf Gommers" notifications@github.com wrote:

Notice the logic gap between steps (2) and (3)

I admit that I'm then one of those passers-by. Note that the comment I
linked to says "wait ..before releasing any shiny new commands...". On
an issue that is about 2 new commands it clearly implies that they're both
on hold, flawed logic or not.

—
Reply to this email directly or view it on GitHub
#6257 (comment).

[njsmith]

And the problem with decoupling the build interface from metadata is that that's pretty much what the distutils delenda est PEP draft does, and the main blocker is that @dstufft says he's opposed to any new format for distributing sources that doesn't solve the static metadata problem. Maybe I'm not understanding what you have in mind as a "build interface". Anyway, discussion continues - I think we mostly are all learning things in that discussion still.

[rgommers]

Build interface: https://mail.python.org/pipermail/distutils-sig/2015-October/026991.html

with decoupling the build interface from metadata is that that's pretty much what the distutils delenda est PEP draft does

I'm not sure about that. Even if you said absolutely nothing about metadata, you could get to a point where pip doesn't call its own vendored setuptools at all but has a clean way to invoke any build tool on current sdists on PyPi. It wouldn't solve every last issue we have with packaging but be an important step forward.

and the main blocker is that @dstufft says he's opposed to any new format for distributing sources that doesn't solve the static metadata problem

I just wanted to give my impression after reading the whole thing at once and not drag @dstufft in a discussion here. That part is still people misunderstanding each others language and use cases - a call will hopefully help. Oscar Benjamin explained it pretty well imho.

Anyway, I'm subscribed to distutils-sig once again:)

[dstufft]

I haven't read the whole discussion on the upgrade thing, but I don't see any reason that pip upgrade (non-recursive, only if needed) couldn't be implemented before a real dependency solver was added. I'd think we'd need it before pip upgrade-all though.

FTR pip doesn't use a bundled copy of setuptools for building, we only have a bundled copy of pkg_resources for inspecting sys.path for installed distributions. We use whatever setuptools is installed (and error out if you try to install a sdist without setuptools installed). You don't need setuptools installed at all to install from wheels.

Currently the build interface is: executes setup.py with some arguments, because the sdist "standard" is basically, "produces a thing that looks similar to what distutils produces". It's not really a standard, but it's what we got right now. Other than the fact pip goes through some shenanigans to ensure that setuptools is imported (and thus, has monkeypatched distutils) to make all distutils using setup.py files use setuptools, there's nothing that says those setup.py files need to use setuptools, it's just the most expedient since the interface isn't well defined. I don't see any reason that we couldn't just formalize (or semi formalize at least) the current standard so that people can rely on that particular interface.

Of course, the other problem is that pip won't ensure that your build time dependencies are installed (and it doesn't now even for setup_requires, it assumes that the setup.py will handle any build requirements. I also don't see any reason we couldn't layer on a small additional standard to setup.cfg or something to support build time dependencies.

Those are the smaller, more incremental changes we could make to the current system. Of course, we can also solve it by making a whole new format, but if we're going to do that I want to make sure it solves the other problems too.

[rgommers]

I haven't read the whole discussion on the upgrade thing, but I don't see any reason that pip upgrade (non-recursive, only if needed) couldn't be implemented before a real dependency solver was added. I'd think we'd need it before pip upgrade-all though.

Thanks for the clarification @dstufft, that helps. Imho pip upgrade is way more important than upgrade-all.

[dstufft]

I should note, that I'm not the final word on pip, if Marcus is against it there might have to be some discussion, but I don't see any reason why it needs to be dependent.

[rgommers]

Yeah, I know. Before I even think about spending effort on that I'll summarize the plan on the relevant pip issue.

[rgommers]

Currently the build interface is: executes setup.py with some arguments, because the sdist "standard" is basically, "produces a thing that looks similar to what distutils produces". It's not really a standard, but it's what we got right now. Other than the fact pip goes through some shenanigans to ensure that setuptools is imported (and thus, has monkeypatched distutils) to make all distutils using setup.py files use setuptools, there's nothing that says those setup.py files need to use setuptools, it's just the most expedient since the interface isn't well defined. I don't see any reason that we couldn't just formalize (or semi formalize at least) the current standard so that people can rely on that particular interface.

Well, there's a reason I said "clean way":) Bento actually does implement part of the setuptools interface so setup.py install and setup.py sdist work: http://cournape.github.io/Bento/html/transition.html#adding-bento-based-setup-py-for-compatibility-with-pip-etc.

But the implementation isn't pretty and it's not exactly easy to implement the whole thing as it works currently and there are far too many commands, so documenting the whole interface as is would be counterproductive.

I think the basic commands could be (semi-)formalized, but others should go away or be renamed. The proposal of Daniel Holth seems like a good way forward here.

[rgommers]

Build-time dependencies are another issue indeed. Fixing that would not be a small step forward I'd think; it would be a major one.

[larsoner]

Issue in setuptools has been fixed so it detects distutils-based names properly now.

Should I leave this issue open and change the title to cover moving to setuptools, or close it since the original bug should now be fixed?

[rgommers]

Cool, thanks Eric!

Let's close this, since this issue is quite specific. There is gh-6599 that has "move to setuptools" as one of the points on the list. Unfortunately the combined plan of moving also to pip install . is running into a few delays, but still we should be able to make progress on the rest of the packaging/install wishlist.