Proposal to Simplify Dependency Management: Transitioning from "contrib" Submodule to vcpkg and System Libraries

[timosachsenberg]

Background:
Our project has been utilizing the "contrib" submodule to manage third-party libraries. While this approach has served us adequately, it has also introduced complexities in terms of project setup, updates, and compatibility across different development environments. As we continue to grow and expand the project, addressing these challenges becomes crucial to ensure a smoother development process.

Proposal:
We propose that we consider migrating to a combination of vcpkg and system libraries for managing our project dependencies. Here are some key benefits of this approach:

The Pros:

Simplified Dependency Management:
Why system libraries:

• Development libraries, installed via system package managers, can be leveraged directly. It usually doesn't get faster and easier to install dependencies.

Why vcpkg:

• vcpkg seems currently to be the best supported dependency management system for C++ libraries
• vcpkg provides a centralized and well-maintained repository of C++ libraries, making it easier for us to discover, install, and manage dependencies.
• Ease of Updates: vcpkg simplifies the process of updating libraries to newer versions, allowing us to stay up-to-date with the latest features, bug fixes, and security patches more efficiently.
• Community Support: The vcpkg community is active and can provide valuable support and insights, making it easier for our team to troubleshoot issues and stay informed about best practices.

Technical Details to Consider for vcpkg:

• if we want to use Qt in CI we can't compile it from scratch. (Optionally) excluding Qt from vcpkg and using systems packages (Added port file to use qt from system libraries, fixed bug in LPWrapper cmake files (thanks Julianus) #7236) is done quite frequently and worked for me.

Cons:

• if a project is not well-supported in vcpkg (e.g., the original project came with super crappy cmake files and not all build options are supported) we need to request or provide a patch to the vcpkg repository.

The general question is: can we get this to work on all platforms? And can we still cache it?
For Ubuntu we know it works:

Ubuntu:

• system packages
• vcpkg

but what about the others? Can you fill some from your memory? I saw you added something to brew

manylinux2014

• [?] system packages
• [?] vcpkg

Mac Intel

• system packages / brew
• vcpkg

Mac Silicon

• system packages / brew
• vcpkg

Windows

• [?] system packages / choco?
• vcpkg

[jpfeuffer]

Actually vcpkg coin-or should work. I think I mistook it for conda. I updated the list.

[jpfeuffer]

The problem with vcpkg is its missing/rudimentary support for binary artifacts and Qt is just too slow. Other libs are not very quick either. Yes, you can overwrite ports to use e.g. an install from a package manager but then you cannot guarantee the same dependencies anymore (specifically zlib, bzip, etc.). Might be possible by also overwriting their ports but still sounds very unstable to me.

[jpfeuffer]

Also this: microsoft/vcpkg#26991 (comment)

[timosachsenberg]

Ok good point. I wasn't so concerned about zlib as there are also binary compatible zlib-ng and others because the ABI seems so stable. But I see the point that there are other dependencies. So we would basically need our own registry and binary artefacts stored somewhere on our local infrastructure?

[jpfeuffer]

I think a cache would be sufficient, or we rely on system packages or conda for CI. For users, I think building Qt is fine once reduced to the minimum.
It will also be much better if we split GUI from lib and only install GUI parts if needed.

[timosachsenberg]

For CI I think system packages would be great - just because it seems to be reliable and fast. If not supported (manylinux ? - not sure…) we could try to resort to vcpkg in CI as well.

In wonder how reliable caching works if we use e.g. https://github.com/marketplace/actions/run-vcpkg but I guess this is something we need to try

[jpfeuffer]

especially since our caches are pretty full and get evicted pretty quickly due to compiler caches already.

[radusuciu]

Question about this as is relates to the Dockerfile migration. @jpfeuffer, you said you want the contrib image to be built separately, will the Dockerfile live in https://github.com/OpenMS/contrib? Is this temporary or will that also be absorbed into the main repo eventually?

[jpfeuffer]

As long as our own contrib exists, everything related to it should stay there. The updating logic and the monitoring from a different repository via a submodule just becomes too complex as I mentioned in the PR.
I'd rather have PRs to OpenMS fail until people update it according to a new contrib version than having a complex, elongated CI actions that build different things based on which part of the OpenMS repo was updated.

This doesn't mean that you can keep a "dependency" container in the multistage docking file. As long as the dependency container can be replaced with an existing contrib container.

[radusuciu]

Got it, thank you!

In the current PR I've added a build-arg that points to the contrib image as requested, but I've also removed the stage.. I think it's probably better to avoid having two different sources for contrib images. We can revisit when I've taken care of other known issues.