Unprivileged installations onto OS X don't support wheels even if a newer setuptools is installed

[glyph]

Mac OS X comes with an old version of setuptools pre-installed.

Unfortunately through some combination of sys.path-management shenanigans, this older version of setuptools tends to take precedence when importing, even if a user has explicitly done pip install --user --upgrade setuptools. This means that pip wheel doesn't work.

This is a problem if a user does not have privileged access to a system but wants to do some Python development there; or, if they do have privileged access to a system but take the wise precaution of not automatically using sudo when a person from the internet tells them to.

If I add install_requires=['setuptools>=0.8'] to pip's setup.py, and then install it, then this problem goes away. However, now it hard depends on the new version of setuptools. I think maybe this is just the way things should be, because interoperating with the older version of setuptools doesn't seem to cause anything but heartache.

pkg_resources has some affordance for dealing with multiple versions of software installed on a system, of course, but it doesn't seem to be exposed in an obvious way. I can't seem to just call pkg_resources.require("a sensible version of setuptools"); I have to set __requires__ to the sensible version of setuptools in __main__ before even importing pkg_resources.

[Ivoz]

Can you say what versions of setuptools OS Xs come with?

[glyph]

0.6c12dev-r88846

[glyph]

Oddly enough, I've discovered that if I use the included easy_install to bootstrap the new setuptools explicitly, by setting --upgrade (and prefix and install_lib options in a setup.cfg), then it writes an easy-install.pth which references the new setuptools egg in my home directory, and it works.

[qwcode]

pip 1.5.1 (to be released soon) will solve this, because it won't require 'setuptools>=0.8' for it's wheel support, so it will allow the older/global setuptools to be used.

as for the sys.path modification you're seeing, that's the effect of easy_install. easy_install'd global packages override the user site. pip can't undo it, or fix it IMO. the solution is for OSX to not use easy_install the global setuptools.

[glyph]

@qwcode The global setuptools does not appear to be easy_installed because there's no easy-install.pth in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python where setuptools lives.

[glyph]

@qwcode In fact,

$ find /System/Library/Frameworks/Python.framework/Versions/2.7/ -name '*.pth'
/System/Library/Frameworks/Python.framework/Versions/2.7//Extras/lib/python/PyObjC/pyobjc_framework_Cocoa-2.3.2a0-py2.7-nspkg.pth
/System/Library/Frameworks/Python.framework/Versions/2.7//Extras/lib/python/PyObjC/pyobjc_framework_ExceptionHandling-2.3.2a0-py2.7-nspkg.pth
/System/Library/Frameworks/Python.framework/Versions/2.7//Extras/lib/python/PyObjC.pth
$ 

[qwcode]

what "sys.path-management shenanigans" are you referring to then?

[glyph]

@qwcode I don't know! By the time pdb gets imported things are already set up, so I literally can't debug this.

[qwcode]

paste your sys.path output with a new setuptools in the user site, if you won't mind

[qwcode]

If it's not easy_install, then all I can imagine would be a custom site.py that's altering things.

[glyph]

Default:

['',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages']

After installing pip with easy_install and then installing setuptools with pip install --user --upgrade setuptools:

['',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages/pip-1.5-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages']

Going back to the beginning (deleting ~/Library/Python entirely), and then upgrading setuptools with easy_install and then installing pip with the new easy_install:

['',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages/setuptools-2.1-py2.7.egg',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages/pip-1.5-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/glyph/Library/Python/2.7/lib/python/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages']

[qwcode]

actually, I mispoke, 1.5.1 drops the setuptools>=0.8 requirement for installing wheels, but not for pip wheel

[dstufft]

I've always assumed this was some Apple shenanigans shoving their own stuff first.

[qwcode]

I can't tell from the output where the global setuptools is at. I assume it's not in /Library/Python/2.7/site-packages
if OSX has setuptools placed in a path dir that has precedence over the user site, then there's not much we can do.

[glyph]

@qwcode Apparently there is, because easy_install is doing it :)

[glyph]

Would it be helpful to you folks for me to get you access to a Mac?

[qwcode]

@qwcode Apparently there is, because easy_install is doing it :)

don't follow what you mean. specifically in your first paste (#1 out of the 3), I'm saying I don't know where setuptools is at, and it's not installed with easy_install per your earlier comments.
setuptools.__file__ will show where it's at.

[glyph]

setuptools is installed as /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools.

[qwcode]

Apparently there is, because easy_install is doing it :)

ok, I guess your point is that upgrading setuptools globally with easy_install (your #3 case) is a solution.
when you did the upgrade, setuptools in /System remained correct?
and if you tried the upgrade with pip, pip would want to remove it, and you'd need the root privileges.

[qwcode]

btw, I've added a stub for this case into the "Packaging User Guide" (https://python-packaging-user-guide.readthedocs.org/en/latest/platforms.html#installing-on-osx).

we need OSX people to help fill out that section with cases like this.

[qwcode]

Apparently there is, because easy_install is doing it :)
ok, I guess your point is that upgrading setuptools globally with easy_install (your #3 case) is a solution.

to be clear, the easy_install upgrade is working due to the way it modifies the sys.path with it's easy-install.pth.
pip specifically and intentionally does not do sys.path modifications.
so, it's worthy of documenting this in the "Packaging User Guide", but there's no bug or enhancement in pip to be made that I can see.

[dstufft]

As far as I can tell this is fundamentally a problem with Apple's system python. The question then is do we want to provide a work around for this case or do we want to simply document the fact that Apple's system Python is installed like this?

[qwcode]

Assuming the problem we're talking about is that the user install of setuptools (via pip) has lower precedence than the default system install of setuptools, then I can't imagine what a workaround would be. If OSX has the user site lower in precedence to other system-managed paths (which contains setuptools), then what are we going to do? I can only imagine resorting to pth hacks like easy_install, but I don't think that's something we want to get into.

[glyph]

@qwcode Why wouldn't you want to get into that? Writing out a single .pth file seems like a highly worthwhile alternative to "manually install a non-system python yourself" or "you just have to do everything as root".

[glyph]

Note that I'm not saying you shouldn't report the bug with Apple and eventually get the issue fixed for real too, but there's a big "meanwhile" to be concerned about here.

[glyph]

@qwcode Regarding an earlier comment you made; I'm upgrading setuptools "globally" in the sense that it's not in a virtualenv, but it's functionally equivalent to a --user install. (It's only not literally a --user install because easy_install 0.6 doesn't have that command-line option.)

[qwcode]

ok, I pulled my macbook off the shelf, and I confirm everything you're reporting, except your 3rd case, when you say you upgraded setuptools with easy_install and you reported your sys.path as then containing:

'/Users/glyph/Library/Python/2.7/lib/python/site-packages/setuptools-2.1-py2.7.egg'

that's the user site. If I try the upgrade of setuptools with easy_install, it wants to write to /Library/Python/2.7 with root permissions, which is what I would expect. I would not expect easy_install to install to the user site by default.

[qwcode]

@qwcode Why wouldn't you want to get into that? Writing out a single .pth

so the logic would be something like this on OSX:

• IF pip install --user PKG AND PKG is already located in a sys.path with higher precedence
• THEN add pth file into the user site that alters sys.path and changes the order around

that's invasive, and it's hard think through what side-effects might arise.

[glyph]

@qwcode This doesn't necessarily need to be fixed in pip proper; it could be something in get-pip.py, for example.

[glyph]

@qwcode Also, yes, I specified the appropriate install-dir for easy_install to put it in my user site. As I said, it's only not --user because it doesn't have that option in that version :)

[qwcode]

I don't see how we could fix this in get-pip.py. this is logic that applies to the install of a package, unless you're implying you only want to deal with this for pip and setuptools.

[glyph]

I'm not actually too familiar with get-pip.py's mode of operation. I guess it still doesn't install setuptools for you at all, so the user's on their own here anyway? Arguably this should just be fixed in setuptools so that however you install setuptools it can fix itself rather than requiring external stuff to mess with it.

[dstufft]

No, get-pip.py will now install setuptools for you.

[glyph]

How does it presently install setuptools?

[dstufft]

Using normal pip means

get-pip.py just has pip 1.5.1 bundled inside of it, it unzips to a temp directory and does a pip install of setuptools and pip.

[glyph]

It seems like the issue is fixed with a regular setup.py install or an easy_install. get-pip.py could work around this by having a little bit of user-interaction around unprivileged installations.

[glyph]

I've worked around this in pip2014.com by dropping a pip-bootstrap.pth into ~/Library/Python/2.7/lib/python/site-packages with the following contents:

import sys; sys.path.insert(0, sitedir)

[dstufft]

I'm not sure that is a suitable fix, but it's at least something to point in a decent direction.

[glyph]

Why isn't it suitable?

[dstufft]

Won't that put the sitedir before "."?

[glyph]

Apparently not, at least looking at my own Python install.

[glyph]

Keep in mind this gets processed during the addition of sitedirs, not at the end of sys.path construction.

[glyph]

This may not be practically relevant any more, since OS X now ships with setuptools 1.1.6, which is recent enough to allow for bootstrapping into something good.

[jaraco]

I agree it may not be precisely relevant, but I'm finding issues upgrading Setuptools in Mac OS X, such as reported in pypa/setuptools#738. Although 1.1.6 is light years better than 0.6, there are still applications that rely on Setuptools > 12 (setuptools_scm) and >= 18.5 (html5lib). I'm going to see what the packaging guide has to say about upgrading Setuptools on Mac... and decide if these issues are relevant to this issue or to another. Depending on what I find I'll probably open a new ticket and reference this one.

[jaraco]

btw, I've added a stub for this case into the "Packaging User Guide" (https://python-packaging-user-guide.readthedocs.org/en/latest/platforms.html#installing-on-osx).

Best I can tell, this guidance is no longer present, so I'm going to start from scratch.

[dstufft]

This is no longer an issue in macOS Sierra, normal installations into --user will now correctly override the system provided packages. Since there is not a clean way to handle this pre-Sierra I'm going to just close this and let people get it fixed by upgrading their OS.