Optimise CI - reduce flakyness, unify workflows #4516
Labels
CI
iroha2-dev
The re-implementation of a BFT hyperledger in RUST
question
Further information is requested
I encounter this time and time again: you open a PR, wait for checks for 10/20/30 minutes, and then you see that same workflows fail on the same tests. During that time I get distracted on something else and forget to re-start checks again just in time. Sometimes it takes 5-6 restarts to get them work. This way, the moment of all green lights might delay for days. And it delays development in my case.
From my observation, the following workflows are flaky:
I2::Dev::Tests
>with_coverage
,integration
,unstable
And these are particular flaky tests:
integration::extra_functional::offline_peers::genesis_block_is_committed_with_some_offline_peers
integration::extra_functional::unstable_network::soft_fork
Are these tests worth it?
My another concern is that I don't see the rationale behind having so many workflows:
I2::Dev::Static
I2::Tests::UI
I2::Dev::Tests
(there are some others too)
These workflows all run Cargo and compile more or less the same stuff. Yes, there are variations in features presets, but Cargo handles it for us. It can granularly reuse compilation artifacts depending on the context (apart from cases with different RUSTC flags, I suppose).
So, I guess that it is worth trying to combine all these workflows into a single one, and build it in a way so that it can report as many useful information as possible in a single run. I wonder how much more/less it would be efficient.
Another useful implication of this would be a shorter feedback on some early errors. For example, a certain change in PR introduces something and Iroha cannot even compile. Currently, all 8+ workflows will run and fail on the same error. In the case of a unified CI, there will be less work repetition.
Proposals
The text was updated successfully, but these errors were encountered: