WIP: Build and upload wheels on CI #110
Conversation
.github/workflows/wheels.yml
Outdated
| - name: Fake metadata # Probably a bad idea | ||
| run: touch pyproject.toml |
There was a problem hiding this comment.
Terrible hack to get things going, otherwise the compilation doesn't even start.
|
@astrojuanlu Thanks for taking the lead on this! The error is a bit baffling, could you tell me which compiler version is being used to compile pagmo? |
|
@astrojuanlu The tbb version is also probably very relevant. |
|
@astrojuanlu also, using yum to satisfy the dependencies is gonna end up using very old versions of the packages |
|
Thanks for the quick response @bluescarni! From the logs, the versions seem to be
Yes. But, to my knowledge, to produce manylinux-compatible wheels, one has to stick to a given version of libc. I could potentially recompile newer versions of the dependencies, but for example using conda (as you do in your CI) is not an option. |
I don't think it's possible that GCC 4.8 is being used. pagmo uses C++17, which is supported since GCC 7. As far as I remember, in the manylinux 2014 image, an updated version of GCC is pre-installed.
Yes, when we were supporting pip we used to build all the dependencies on our own via the manylinux toolchain. @darioizzo created a new docker image based on the manylinux one with the dependencies pre-compiled in order to reduce the load on the CI, but as far as I know this custom image has not been maintained for years and it is likely unusable at this time. |
|
I was making a mistake: the yum-installed gcc is indeed 4.8.5, but the one in the path is 10.2.1: I tried compiling pagmo2 locally inside a (pasting screenshot for coloring)
|
|
(Happy to move this conversation to https://github.com/esa/pagmo2/issues if it helps) |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@astrojuanlu I think this error is due to the old TBB version in use. Can you try to install a new version manually in your environment? |
|
Fails: oneapi-src/oneTBB#950 getting deeper into the rabbit hole, it seems... |
|
@astrojuanlu sorry posted on the TBB issue while I meant to post here... Copying the message below. Can you try perhaps with a stable version rather than the git head? The conda package for pagmo compiled fine with this TBB version: (TBB version 2021.6.0) |
|
@astrojuanlu you may still find use for our old docker file https://github.com/esa/manylinux_x86_64_with_deps/blob/master/Dockerfile2010 With the specific instructions to compile all necessary deps (as of a few years back) |
|
Looks like
|
|
Nice, it compiled pagmo2! 🎉 Now it failed because the headers of Python 3 are not available, but this is promising I think. It needed half an hour though, which is a good indication that we should create a Docker image with the precompiled dependencies. I'll continue with this in the coming days. |
|
I think if you want to create a docker image with deps and upload it to the pagmo docker you could just add it to the repo linked above via a new dockerfime (and remove the old ones)? |
Nice progress! Regarding the precompiled docker image, I am ok with it if someone volunteers to keep it up to date going forward (meaning essentially that the docker image should regularly be rebuilt in sync with upstream manylinux 2014). Otherwise, it's better IMO to just use the vanilla manylinux image and accept the runtime cost of rebuilding all dependencies each time. Better this than having a custom image that bitrots over time. |
|
After a few iterations I didn't manage to get a helpful answer for oneapi-src/oneTBB#950, so for now until I have a better answer I'll keep the The current blocker is that CMake is not finding Python 3 in the image. |
|
I think I understood that the Docker image should only contain the dependencies, but I'm hitting a wall trying to understand how to fit the bespoke pygmo2 build process into cibuildwheel. I opened a discussion upstream seeking for help: pypa/cibuildwheel#1368 |
|
@astrojuanlu have you looked into https://scikit-build.readthedocs.io/en/latest/skbuild.html ? This is a bridge between CMake and the Python |
|
Indeed, I was aware of it but was hoping that I could avoid rewriting pygmo2 build system. Looks like it's the most sensible path forward though according to the feedback I was given, so I will give it a try. |
But the point of using skbuild is precisely that CMake is used as the build system and you need a handful (<20 IME) of lines in setup.py to make it work... |
astrojuanlu
force-pushed
the
build-wheels-ci
branch
from
3cbd260
to
a7019bd
Compare
astrojuanlu
force-pushed
the
build-wheels-ci
branch
from
efb84a3
to
ebc7738
Compare
The Development component has two sub-components: Development.Module and Development.Embed. The Development.Embed component was missing in some interpreters of the manylinux image, whereas the Development.Module one seems to be present. The CMake documentation does not make very clear what is the intent of each. Hopefully someone more knowledgeable can justify or amend this decision.
|
I managed to build manylinux wheels locally for Python 3.6, 3.7, 3.8, 3.9, and 3.10 using This is the command I used: As you can see, I'm using a Docker image There are still CI failures because gh-113 is incomplete. That one should be finished first, although I found a few obscure test failures that blocked me. Honestly I'm in full "I hate CMake" mode at this point - the biggest hurdle has been getting it to find the stuff. The last barrier I was hitting was with which, interestingly, was succeeding for Python 3.6, but failing for everything else with this error: The documentation was not very helpful (what are diff --git a/CMakeLists.txt b/CMakeLists.txt
index e80329e..9b06181 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -137,7 +137,7 @@ if(${pagmo_VERSION} VERSION_LESS ${_PYGMO_MIN_PAGMO_VERSION})
endif()
# python.
-find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
+find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module)
message(STATUS "Python3 interpreter: ${Python3_EXECUTABLE}")
message(STATUS "Python3 installation directory: ${Python3_SITEARCH}")
and now it seems to be working, but it would be nice that somebody more knowledgeable has a look at the implications of this. I'm staying away from this at least for a few days to catch up with other stuff. |
|
Closing this as its now solved in #117. |
This uses cibuildwheel to build Python wheels and source distributions, and upload them to PyPI if proper credentials are given. cibuildwheel is the standard for building complicated wheels on CI for Python packages, and the resulting wheels are manylinux compliant. I'm using the
manylinux2014standard because it has the newest images, to minimize the amount of packages I need to compile.I got pretty far, but now the compilation of Pagmo fails on
island.cpp.o:If you could lend a hand with this, I'm happy to keep pushing for the PR, at least until we find some other bottleneck.
Notice that I didn't try to make the code pretty or reusable: essentially I took the instructions from
tools/and rewrote them for CIBW usingyuminstead of conda.Fix gh-102.