Test-case sampling

[enriozuni]

With the fix of issue #240, it was seen that there is a need for test-case sampling in CryptoAnalysis. Currently, there are around 280 JUnit test cases in the repository and the CI build of the repository requires a lot of memory. Moreover, it momentarily takes ~25 minutes for a successful build to complete. This means that a small commit would trigger the CI build which runs all different suites with test cases.

An alternative solution to this problem would be test-case sampling, where the following ideas could be implemented:

• For each commit, run only the subset of test cases that correspond with the code changes of the commit.
• Whenever master branch is built, run the whole test cases like in a normal build.
• Setup a normal build on a fixed time period (e.g. two weeks) in the develop branch, so bugs are detected early.

Research on this topic is made in these papers:

• Change-Based Test Selection in the Presence of Developer Tests

• Change-based test selection: an empirical evaluation

• 15 Years of Software Regression Testing Techniques -- A Survey

[kruegers]

The following papers also seem worth checking out:
https://ieeexplore.ieee.org/document/8115710
http://mir.cs.illinois.edu/awshi2/publications/ISSRE2019.pdf
https://uu.diva-portal.org/smash/get/diva2:1371200/FULLTEXT01.pdf
https://link.springer.com/chapter/10.1007/978-3-030-29509-7_1

[enriozuni]

After a Skype discussion, the following proposals were drawn:

1. Weekly / Bi weekly full testing
2. Change based testing for every push
   2.1 Alternative: Run "important" tests for every push
3. Ability to force a full build (on demand)