Github Action to automatically build wheels that include Cbc libraries for all platforms

[tkralphs]

Thanks to the generous help of @mkoeppe, we have just finished implementing a Github Action for CyLP that automatically creates binary wheels for Linux and MacOS (Windows is coming) that include the Cbc libraries using the amazing cibuildwheels package. See the YAML for the action here:

https://github.com/coin-or/CyLP/blob/master/.github/workflows/cibuildwheel.yml

The action first builds the Cbc libraries from source and then uses the cibuildwheels to package the libraries the "right" way, doing all of the heavy lifting of setting the rpaths for the libraries and etc. using the audithweel (Linux) and delocate (MacOS) packages. Once it's up and running, this would be a super easy and completely automated way to distribute the latest Cbc with each release of python-mip. In my testing so far, it "just works". Check out the variety of wheels that are now available for CyLP:

https://pypi.org/project/cylp/0.91.5/#files

Note that in addition to the cibuildwheel.yml file, this also requires a pyproject.toml, which is a good thing to have anyway. It should be relatively easy to copy what we did for CyLP. We've already worked out the kinks.

[sebheger]

@tkralphs Sounds great. We will prove it when Windows and new m1 architectures are available. I have seen issues are already been created.

There are two related things I am concerned about. Maybe they vanish if I look into the details, but just mention them:

I would prefer to include the latest Cbc Release, not the latest Master Commit, to make bug tracking easier. And @h-g-s told me, that the complete C wrapper which is used at Python-MIP is only part of the CBC Master, but Cbc releases are not built from master at the moment.

I have seen that the newer Cbc release already offers some build resources, but not in the way it would be suitable for Python-MIP. Have you thought of also building the .so / .dll files already on the Cbc side? As I am not a c/c++ dev, the question might be a kind of „stupid“?

I would appreciate your thoughts.

[tkralphs]

Yes, it would be ideal for python-mip to depend on release versions of Cbc. For that, we need to get the current master out into release. There is some work to do to make that happen, but I've been getting almost no community support for this. That is a problem and illustrates the fragility of the Cbc ecosystem on which python-mip depends.

It is true that with the old build system used in version 2.10, one cannot produce DLLs for Windows. The ability do this is one of the main improvements in the move to the newer build system used in the master version.

[h-g-s]

Hi @tkralphs . Having automated binary builds would be great. It is currently very manual and complicated to build in a way that a good level of compatibility is achieved. I have a repository which builds binaries for Cbc using github workflow https://github.com/h-g-s/cbc-binaries, but is has some limitations. The biggest one is that is uses the available virtual machines for github, (e.g. Ubuntu 18.04 for linux). To have a good degree of compatiility in linux, for example, you need to build in a distribution with an old glib. In python wheels they have the "manylinux" virtual machine (which runs on docker over github, from what I understood) which has an "old enough" glibc.

@sebheger Yes, currently python-mip depends on the master version of Cbc, which was not released as stable yet. Agree with @tkralphs that moving the current Cbc master to stable would be important. I'll check what me and @samuelsbrito can do to help in this path.

[h-g-s]

Update: it seems that there are still some fixes required to run python-mip in the new master (after the merge of the refactor branch). It seems that a processing limit (max_nodes) is not being respected. I'll investigate it, maybe other parameters are also not working well in the C interface after the merge.

[tkralphs]

In python wheels they have the "manylinux" virtual machine (which runs on docker over github, from what I understood) which has an "old enough" glibc.

Yes, this is exactly what the cibuildwheels package does for you---it automates the whole process of fetching the correct manylinux docker images, building everything, and then finally exporting the wheels. You can even easily run this on your local machine if you have docker installed, once it's all working. The process is a little different for MacOS and Windows. To get this working, the setup should be almost identical to CyLP.

[sebheger]

I have done a draft #266 already a few days ago. But the workflow fails. Had not have the time to do more investigations...