CI cost breakdown

[fsimonis]

Please describe the problem you are trying to solve.

The CI is holding us back whenever we have an activity spike in the preCICE repositories, mostly around our project meetings.
We should try to reduce the overall runtime of the CI as much as possible.

CI overview

Based on the upgraded CI of #1962 and this workflow, omitting MacOS runners.

Image	Compiler	Setup	Config	Duration
Arch	Clang	MPIPetsc	Debug	14m 17s
Arch	Clang	MPIPetsc	Release	14m 26s
Arch	g++	MPIPetsc	Debug	16m 43s
Arch	g++	MPIPetsc	Release	17m 12s
Fedora	Clang	MPIPetsc	Debug	13m 51s
Fedora	Clang	MPIPetsc	Release	13m 57s
Ubuntu 24.04	g++	MPIPetscGinkgo	Debug	18m 11s
Ubuntu 22.04	g++	Bare + Cov	Debug	8m 6s
Ubuntu 22.04	g++	Bare	Release	6m 19s
Ubuntu 22.04	g++	MPI + Cov	Debug	19m 35s
Ubuntu 22.04	g++	MPI	Release	13m 53s
Ubuntu 22.04	g++	MPIPetsc + Cov	Debug	20m 18s
Ubuntu 22.04	g++	MPIPetsc	Release	14m 27s

Total time 185m + 375s = 11475s = 3h 11m 15s

Cost breakdown

Cost breakdown of worst case Ubuntu 22.04 LTS MPI PETsc + Coverage report:

Section	Duration	Percentage
CCache restore	13s	1%
Venv setup	5s	<1%
Configuration	13s	1%
Compilation	12m 26s	64%
Testing	2m 10s	11%
Coverage report	4m 11s	21%
cpack + lintian	12s	1%
Total	1170s

Note: saving a ccache as artifact of ~500MB can take up to 2m.

Potential improvements

From most to least relevant

Compilation

Known to work:

• Larger runner with more cores GitHub-hosted / self-hosted
• Faster runner with higher boost clocks etc.
• Reduce header bloat from Eigen / fmtlib / boost Debug build analysis with clang 16 on arch
• User faster parallel linker mold (scales with larger runners, ineffective with current runners Improve CI speed and portability #1969 )

Experimental:

• Pre-compiled headers
• Unity builds
• ccache that actually works (requires a lot of tinkering and requires cache handling)

Coverage report

Known to work:

• Use a faster coverage implementation like fastcov (scales with larger runners)

Testing

Known to work:

• Convert more test from integration tests to unit tests
• Run tests in parallel using CTest

Experimental:

• Register individual tests in CTest and run them in parallel (May converge towards time of MPI_Init, but scales with runner size)