Mutation Testing - Implementing Selective Build and Test for Efficient CI Workflow

[ASuciuX]

The aim is to identify functions that are associated with incomplete or ineffective tests. The system flags these functions to alert developers about potential issues in the test coverage.

PR-Differences Feature:

When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR. It checks the new and altered functions through mutation testing. Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes.

Optimization Considerations:

1. Selective Build and Test Execution:

• Context: The project is extensive, and running a complete build and test (using cargo build and cargo test) for every mutation can be very time-consuming. Since a single function can lead to numerous mutations, the total build and test time can be excessively long in a Continuous Integration (CI) workflow.
• Strategy: To optimize, build and test processes are confined only to the specific package where the mutated function resides. This targeted approach significantly reduces the time and resources needed, as it avoids running these processes for the entire repository for each mutation.

2. Performance Benchmark Reference:

• Here, there will be included tables showing benchmarks from recent PRs. These benchmarks provide valuable insights into the performance and efficiency of the CI process, helping to identify areas for further optimization (shards/tests/etc.).

3. Incompletely Tested Functions Reporting:

• At the end of the process, the system generates a report listing functions with incomplete test coverage. This feature is not currently automated but could be enhanced by saving the results in a cache for future reference and analysis.

4. Logging Process:

• The logging process maintains a continuous record of functions with incomplete tests, utilizing a cache system. This cache is not a static artifact as it needs to be dynamically updated across workflows.
• Post-merge actions include running mutation tests similar to the Filter-PR process. The results are then appended to the existing log cache, ensuring an up-to-date status.
• Suggested Approach: The data from the Filter-PR process could be cached separately. Upon merging, this data is directly appended to the main log cache.
  • Caution: Care must be taken to avoid adding new functions with incomplete tests to the cache, as it could lead to sequential PR issues.
  • Important Note: Direct commits to primary branches (like develop/master/next) will not trigger this process, but one trigger to perform them on the spot could be added.

• Alternative Approach: After merging, the system could rerun the mutations, but this is less efficient. It requires waiting for the mutation process to complete before another merge can occur, potentially leading to delays if the mutation process is lengthy.
  

[ASuciuX]

I've updated the information regarding the performance of shards. An issue has been raised with cargo-mutants to investigate potential bugs that might be causing the observed poor performance. You can find the issue here: GitHub Issue.

Here's a comparison of the results with and without shards:

First PR:

• Without Shards https://github.com/ASuciuX/stacks-core/actions/runs/7269581208/job/19807417329?pr=27 - 3 hours 18 minutes stacks-node + stackslib 26 mutants - missed, unviable
• With Shards https://github.com/ASuciuX/stacks-core/actions/runs/7302059398/job/19899965731?pr=35 - job 2 2 hours 46 minutes ( total run time in parallel is around 10 hours combined)

Second PR:

• Without Shards https://github.com/ASuciuX/stacks-core/actions/runs/7269581403/job/19807417955?pr=28 - 5 hours 21 minutes stacks-node 243 mutants - missed, unviable
• With Shards https://github.com/ASuciuX/stacks-core/actions/runs/7302062845/job/19899978677?pr=36job 0 failing cargo-test (shouldn’t happen and i suspect it being a problem related to shards) job 5 3 hours 26 minutes ( total run time in parallel is around 15 hours 30 minutes combined)

[ASuciuX]

PR Mutants Flow based on the mutants run when opening PRs