improve index_url passing for setup_requires

[jonathanultis]

4 tests fail in master before and after this patch.

This patch makes setuptools respect the index_url for setup_requires entries. They still won't respect the versions list. But this is a little better.

It fixes an issue with cliff and stevedore, which use the pbr library to share setup logic.

[gotcha]

Can you add a test so that we understand what you are trying to achieve ?

[jonathanultis]

Not easily. Test setup requires a local pypiserver, and the ability to prevent access to pypi.python.org in some way. I accomplished that by setting a bogus entry for pypi.python.org in my /etc/hosts file for a while.

With that setup:

$ cat buildout.cfg
[buildout]
# point at the local pypiserver
index = localhost:8080
parts = scripts
develop = .

[scripts]
recipe = zc.recipe.egg:scripts
eggs =
    setuprequires
interpreter = python

$ cat setup.py
from setuptools import setup
setup(
    name = "setuprequires",
    version = "1.0",
    description = ("verify that setup_requires eggs use the proper index"),
    install_requires = ["setuptools", "cliff", "stevedore"]
)

First, bootstrap with access to pypi.python.org open. Then, remove access to pypi.python.org. Then buildout.

If pypi.python.org is inaccessible, the build will fail because of the setup_requires entry for "pbr" here

Unfortunately, buildout can't get the list of setup_requires eggs without triggering the download and install of those eggs. So, setup_requires dodge buildout's index url and version pinning, right now.

This patch improves the situation a little bit by asking the egg to use the correct index url for setup_requires eggs.

Internally, we use a bootstrap that pulls ez_setup.py from a bandersnatch mirror instead of bootstrap.pypi.io, and an ez_setup.py that pulls setuptools from the local bandersnatch mirror as well. Our goal is to keep our builds entirely isolated from pypi.python.org reliability issues.

[reinout]

The idea looks good in principle and I can see what you're trying to achieve.

But: why cannot buildout figure out the setup_requires? It can figure out the regular install_requires. Perhaps there's something we could fix to get that working? Or is there some internal setuptools reason why it is impossible for buildout?

If so, perhaps setuptools could be changed to allow it. That would get the version pinning working, too.

[reinout]

See also my comment #244 (comment)

[idgserpro]

See also my comment #244 (comment)

@reinout as stated in the setuptools documentation, setup_requires isn't the dependency on the package that declares it but rather the dependency on the installation script.

So I don't think it necessary to buildout storage these eggs and nor to include in scritps that they are installing. In fact, this seems wrong. See the documentation that easy_install doesn't include these eggs in the python site-package.

But it would be a great idea the easy_install use the same buildout index. This avoids errors in an environment where the access of pypi is blocked and the installation of the packages is done through a local mirror.

@jonathanultis 's idea is great!