manylinux-pypy settings?

[henryiii]

Rework (at least discuss) CIBW_MANYLINUX_PYPY_* options as mentioned in #671 (comment) and #671 (comment) before 2

Currently, we have two separately defined images; pypy and non-pypy. Setting one does not affect the other. So, for example:

[tool.cibuildwheel]
manylinux-x86_64_image = "manylinux2014"
manylinux-i686-image = "manylinux2014"
# Still default
#  manylinux-pypy_x86_64-image = "manylinux2010"
#  manylinux-pypy_i686-image = "manylinux2010"

Since the manylinux images come by default with pypy now, we should consider if we want to change the behavior in 2.0. Our options:

1. Drop pypy variants; they just use the same thing as CPython. Makes it harder to statically define these, though - for example, setting manylinux1 will make it really hard to also build for PyPy. Not fond of this option.
2. Default to the regular value if undefined, but take the "override" if given. So that would make the pypy wheels 2014 in the example above, but you could still override pypy separately.
3. Keep the current behavior.

If we were designing from scratch, I'd probably go with 2. But don't think it's important enough to cause issues over? Especially since manylinux1 is is still very common (over 50% of our users), and setting manylinux1 for pypy is a mistake.

[henryiii]

CC @YannickJadoul and @mayeut, who likely have input on this. Maybe @mattip too. My vote is to remain on option 3, I think.

[mattip]

manylinux1, based on CentOS5, is way past end-of-life and should be dropped as soon as possible. The only reason to keep it is for CPython3.6, since by default even the latest (also end-of-life) 3.6 version ships with a pip that is too old to support manylinux2010 (CentOS6).

maynlinxu2010, based on CentOS6, went end-of-life in Nov 2020. Again, the reason to keep manylinux2010 is for older pip versions that still do not support manylinux2014 (CentOS7 - which is supported until 2024).

The minimum manylinux usable by PyPy is manylinux2010, CentOS5 is too old to run PyPy. So I would suggest:

• drop manylinux1 across the board. People who still want it can use older versions of cibuildwheel
• use the same image for all python variants including PyPy, unless the user overrides them

I wonder if there is data available from PyPI on manylinux1 downloads?

[henryiii]

There's a long discussion here: pypa/manylinux#994

Basically there's good reason that over 50% of our users use manylinux1 even though we default to manylinux2010: pip 9 is the default pip on Ubuntu 18.04, even for the optional Python 3.8, and RHEL/CentOS 7 and even 8. The EOL for CentOS 7 is 2024, but the plan is to drop manylinux1 at the end of the year; cibuildwheel will likely follow that. The user experience by default is really poor if you ask for a package from pip 9 and it doesn't provide manylinux1 wheels. CMake tried it and had to revert to shipping manylinux1.

IMO, pip 9 is special because pip 10 yanked all public api, including pip.main, so no package manager wanted to upgrade it. And manually upgrading the package manager pip is not supported and could break your system, too. Even the pip entrypoint script from pip 8 imports main! And if you upgrade pip manually, every venv starts with the hacked-in pip from the package manager - including Python 3.8 on Ubuntu. This was a mess to handle in Build.

Manylinux_2 finally fixed this for the future by letting pip load a future release too if the system allows it. But that's a very new pip today.

[henryiii]

Also, we really can't limit what manylinux people use - you can specify any image.

[henryiii]

We do default to 2010 everwhere, the question is should setting a value for the image change pypy too, or should they be only individually overridable?

CPython 3.6 hits EOL in December 2021, so I expect PyPA projects to likely start dropping support mid next year, following the 3.5 pattern this year. But unlike 3.5, 3.6 was wildly popular, with CentOS 7 and 8, Ubuntu 18.04, and more relying on it. I still see 3.6 pop up a lot, though it has been abandoned by the scientific community at least.

[mattip]

If dropping manylinux1 is not an option, then I would go with option 3.

[henryiii]

Are there plans to add the other arch's for pypy?

[joerick]

While I agree that we probably wouldn't have pypy-specific options if we built the feature now, they're harmless enough in the current incarnation. I also think my preference is also for (3), it seems like (1) might cause issues with older images as mentioned above, and (2) adds a bit of cleverness that would benefit very few users, surprise a few others, while requiring extra documentation.

[henryiii]

Let's say we are going with (3) then, status quo.

[mayeut]

Let's say we are going with (3) then, status quo.

Works for me for all the reasons already mentioned. I'll close the issue.

I wonder if there is data available from PyPI on manylinux1 downloads?

@mattip, following the long discussion in pypa/manylinux#994 mentioned by @henryiii, I keep a cron task updating https://mayeut.github.io/manylinux-timeline/ with both packager side and consumer side stats. The manylinux1 downloads stats are per python version in the consumer statistics section.

[mattip]

Very cool. I am surprised that glibc2.17 (centos7) and before is used in less than 8% of clients downloading packages from PyPI. Maybe because the data misses all the machines behind firewalls that download once and then redistribute internally.

[joerick]

I keep a cron task updating https://mayeut.github.io/manylinux-timeline/ with both packager side and consumer side stats

Niiice. I forgot how cool this website was!

[mattip]

I am commenting here since opening a new issue would loose all the history. PyPy is intending on building its "official" tarballs with manylinux2014 docker images going forward. Is there a possibility that cibuildwheel could limit pypy wheels to manlinux2014+?

[henryiii]

Change the default or limit? I think the defaults simply match whatever upstream recommends, so if you say it should be 2014, we can change it. For "limit", we don't really limit anything; it simply doesn't work if manylinux doesn't have it. So if it's removed from the 2010 official image, it won't work here anymore either.

[henryiii]

I really see no reason for PyPy to ship 2010, pip needs to be pretty new for PyPy, so then it's only CentOS/RHEL 6 that's affected, and usage of that is really low, AFAIK, especially for PyPy.