pip sometimes builds install_requires dep AFTER project, which forces cython .pxd imports to be either setup_requires (breaks cython) or pep517 (breaks wheels and therefore tox)

Ok, so I tried to use tox, and it runs the following command:

cmdargs: '/home/jonas/Develop/myproject/.tox/py37/bin/python -m pip install --no-deps -U \'/home/jonas/Develop/myproject/.tox/dist/myproject-0.6.zip\''

myproject has an install_requires dependency on wobblui, from which it will attempt to .pxd import things. (Cython's cimport)

Now what pip does is build the wheels in this order:

Building wheels for collected packages: myproject, wobblui, nettools
  Building wheel for myproject (setup.py): started

Please note this builds myproject BEFORE it's install_requires dependency. As a result, unsurprisingly, the build fails.

Now this is somewhat problematic because:

• why would pip even need to build in this order? Does this have any upside?

• setup_requires not only slows down pep517 when analyzing for deps (because get_requires_for_build_wheel can analyze deps without any of the install_requires/actually building the wheel, but will need to install all of setup_requires first) but also sometimes breaks Cython(!) so is simply not an option for Cython projects: Question: odd error, disappears on second run: AssertionError: PyTypeTest on non extension type cython/cython#2730

• pep518/pyproject.toml's build-system.requires doesn't allow building wheels sometimes, and if that is used in a dependency that itself is build-system.requires in tox then it just breaks down tox making it also not a good option: some more details here Please consider undeprecating setup_requires setuptools#1742

Therefore, I think pip should just not do that. Please make it build things always such that install_requires is build first, it brings a host of other troubles for .pxd cross-package cimports/Cython if it just jumbles the order around

[benoit-pierre]

There's not guarantee with respect to installation order when it comes to install requirements, you should not rely on it.

[benoit-pierre]

Even if a dependency is installed before, there's no guarantee it will be available during the build phase (as the build may be isolated).

Hm the isolation remark is interesting, but I'm fairly sure build isolation wasn't disabled and it never was an issue. (not sure why)

Well my suggestion is to make a guarantee, so I can rely on it. That is why I made this ticket. Or is there any advantage in the current behavior of not keeping the order?

As I wrote above especially the time consumption when using setup_requires or build-system.requires can be massive, 5 minutes or longer with a larger Cython dep, and this can lead to 10 minutes or more increases in build time in cross-compilation tools like https://github.com/kivy/python-for-android if they need to analyze the deps. (And there is AFAIK inherently no way to speed this up, since hooks.get_requires_for_build_wheel per design needs to install both setup_requires and "build-system".requires fully before it can analyze the deps, even though in 99% of real-world cases this won't be necessary) So this is a significant inherent problem, because for modern setuptools and hooks.get_requires_for_build_wheel you would actually not need to build the wheel and then wouldn't also need these deps, but the format of setup_requires/"build-system".requires just isn't smart enough right now to understand this

Sorry, to be more clear: the exact different use cases are 1. deps are needed for build_ext/fully building the wheel/running it (which is the case for .pxd cimport targets) and 2. deps are needed to just run pep517's hooks.get_requires_for_build_wheel on modern setuptools which does not need to build the wheel in that case (which is not the case for .pxd cimport targets). And currently there appears to be no way to differentiate this when specifying package deps

So there's a lack of "this is needed for build_ext but not for any of the other basic stuff you might want to do with the package" option, and install_requires could fill that gap (and in practice often does in my tests) if pip just obeyed build order (which it often, but sadly not always will) and kept site-packages access to the previous install_requires-built stuff around (which it seems to do)

[benoit-pierre]

What basic stuff? If it's a build time dependency, it needs to be build before being installed. If you want to speed up things, build wheels, so those build time dependency don't have to be build from source.

[benoit-pierre]

It really would be easier to understand what's the exact issue if you push an example or the offending project to Github and provide the exact command being used.

How simply you just put it simply doesn't reflect the actual pep517-landscape.

There's not just "build time dependency" and "needed at runtime dependency". There is also "needed just to run basic setuptools and examine package" and currently that's lumped in together with "build time dependency"

@ example: that really depends on what you want to see. If you want to see the slow-down, you need to write code using pep517 examining a package using hooks.get_requires_for_build_wheel. That is the only current way in python packaging to get the dependencies of an arbitrary python package without actually building the wheel. (which is very important for basic packaging analysis e.g. how we need it for https://github.com/kivy/python-for-android without wasting lots of time building the wheels)

So the problem here really is in the nuances of package analysis of packages that are not installed without actually installing them, which historically I think was impossible but with pep517 it now is, but there slamming unnecessary stuff into setup_requires/build-system.requires really tanks the analysis time in an extremely undesirable manner

[benoit-pierre]

First, there's PEP 518 and PEP 517. PEP 518 is the replacement for setup_requires. PEP 517 is for supporting multiple build systems (as in for setuptools replacements). setuptools does provide a PEP 517 compatible backend.

Just for clarity, I did actually mean pep517 interacting with build-system.requires/pep 518. We use pep517 for packaging analysis in python-for-android to map the deps to cross-compile patches when packaging for android

Edit: but this affects really any tool that wants to be able to visualize dependencies without actually installing the package. I realize that's not super common but I'd say it shouldn't be treated as super arcane either

[benoit-pierre]

Build wheels, than analyze those?

pep517 combined with new setuptools has hooks.get_requires_for_build_wheel where you can get the metadata without building the wheel. I cannot understate how important that is because building the wheels takes really really long for some projects, so this makes a massive (like 90% time decrease or more) difference in analysis time. So if for any .pxd imports I need to put them into the basic setup_requires or build-system.requires I will tank this analysis time massively although for that step it wouldn't even be needed. That is why it would be so much better if it worked just specifying these things via install_requires - which again, already does almost work in practice, since pip usually obeys install order. The problem really just arises when it doesn't

Sorry if I express all of this poorly, I got into this mostly by necessity. (I work on Cython libraries and python-for-android components that deal with this) I'm really not a packaging expert

[benoit-pierre]

Because things are not declarative, there's no way to inspect source packages without involving the build systems and build dependencies because you never know when they are going to be needed (like an import in setup.py to provide custom commands)... ¯\(ツ)/¯

In theory, a valid source distribution PKG-INFO could provide all the necessary information.