Travis credits

[henryiii]

I've released CMake 3.20.5, which worked beautifully on the GitHub Actions backend, which the exception of a missnamed deployment secret (so I was just able to download the artifacts and push those to PyPI manually, fantastic! I've added a new key with the proper name - in the future, we can change the name in the workflow to one of the other two options now).

However, the PowerPC job on Travis failed due to networking issues, and I restarted it several times, and now have an "out of credits" message and can't build the PowerPC arch for 3.20.5 (so if you are reading this and need PowerPC, use 3.20.4 for the moment). Here are some ideas:

• Move Scikit-build to GHA, to avoid extra Travis credit usage for things that are easy to run elsewhere
• Maybe use emulation on GHA? Much slower, but could be released all at once and only if all jobs succeed, which is nice.
• I could run the emulation build locally via cibuildwheel 2 and upload it, probably. Edit: I don't know how to do run emulation on Docker on macOS.
• Maybe a small fund for Travis credits could be included in our NSF CSSI proposal?
• When does the allowance reset? I would have thought on the first.

[mayeut]

There's a race condition between travis-ci & GHA I did not think about when crafting #139 :

Because building cmake python package requires cmake, the following line exists (required for manylinux1 only on linux):

cmake-python-distributions/pyproject.toml

Line 24 in be7c3ff

"pipx install cmake",

We could pin its version or require it only for x86/x64.
I will open a PR pinning it in a first time.
The dependency on a previous version of cmake will only become an issue when adding a new platform and can be addressed at this point (c.f. also #33 which is relevant for this issue)

I will also open a PR for with GHA emulation just to see how much time it takes.

[mayeut]

I could run the emulation build locally via cibuildwheel 2 and upload it, probably. Edit: I don't know how to do run emulation on Docker on macOS.

@henryiii,
on macOS (Intel) with Docker Desktop, running CIBW_ARCHS="ppc64le" cibuildwheel --platform linux on the #161 branch just works (qemu is bundled in Docker Desktop, at least macOS Intel).

[mayeut]

Maybe use emulation on GHA? Much slower, but could be released all at once and only if all jobs succeed, which is nice.

The GHA workflow takes a bit more than 2 hours with emulation. IMHO, this takes too long for PR but should be OK for releases.

Here's what I'm thinking:

• keep Travis-CI for PR and push to master (could cost 0 credits if moving arm64 to the lxd partner queue rather than using graviton2)
• Have cron/manual dispatch to test GHA emulated
• Use GHA emulated for releases as it avoids partial releases. c.f. also Draft release feature on main archive to allow testing a release before it goes live pypi/warehouse#726

Given almost everything happens in a docker container, it should not matter wether we're using GHA emulated or Travis-CI and I think there's very little risk with what's proposed. The cron job would ensure that we're not hitting a qemu issue now & then.

[mayeut]

@jcfr, @henryiii,

I've built the missing wheel for 3.20.5 but can't push it to PyPI (lacking rights on PyPI for this).
If rights are granted, I can push it, otherwise, I can share it. Another option is to push a release once #160 and/or #161 land.

[henryiii]

I'm just a maintainer on the PyPI package, so can't add you there. You can send it to me and I can upload it, or @jcfr or @thewtex could add you.

That plan sounds good. The "only upload if all wheels are built" part, especially. Adding three parallel 2 hour jobs isn't awful for release-only. We also are reducing our unnecessary usage of Travis.

I did try to build locally on my mac, but had an illegal instruction (even just trying to open that ppc manylinux image).

[mayeut]

@henryiii,

You can send it to me and I can upload it

I sent you a google drive link along with the sha256sum
I tested it briefly on python:latest (& cibuildwheel tested it) via qemu.

I did try to build locally on my mac, but had an illegal instruction (even just trying to open that ppc manylinux image).

that's something I've certainly seen before and one of the quirks of using qemu.

That plan sounds good. The "only upload if all wheels are built" part, especially. Adding three parallel 2 hour jobs isn't awful for release-only. We also are reducing our unnecessary usage of Travis.

I will update my PR for this.

[thewtex]

Here are some ideas:

These are all great ideas.

One more idea: cross-compile on amd64 via dockcross. @mayeut how possible is it to support dockcross in cibuildwheel?

[jcfr]

To streamline releases:

• @henryiii : You have been added as a maintainer to https://pypi.org/manage/project/ninja (you were already a maintainer of cmake)
• @mayeut : you have been added as a maintainer to both https://pypi.org/manage/project/cmake and https://pypi.org/manage/project/ninja

[mayeut]

you have been added as a maintainer to both https://pypi.org/manage/project/cmake and https://pypi.org/manage/project/ninja

@jcfr, thanks. Unfortunately, I was away for a long week-end and the link to join those projects expired before I could accept. Can you send those again please ?

[jcfr]

Can you send those again please ?

Of course, I just sent new invites :)

[mayeut]

@jcfr, thanks, I could upload the missing wheel to PyPI.

One more idea: cross-compile on amd64 via dockcross.

@thewtex,
That's yet another option but cross-compilation is not supported for linux in cibuildwheel.
It would also require for auditwheel to support cross-repairing (probably before cibuildwheel can be used for that).