Migration to ctest

[LecrisUT]

I had a look on the do_regtest.py, and I would suggest to convert this script to native ctest, because:

• path to the appropriate executable is automatically resolved regardless of the build environment
• scheduling parallel tests are handled automatically after setting the PROCESSORS test property, i.e. it can account for ctest, mpi, and openmp parallelization
• ctest can reschedule the test execution order based on the previous test runtime cost
• scheduling based on gpu availability is also available (may require coding the gpu selection based on CMake environment variable)
• Allows the tests to be run both within the build environment, as well as after installation

Proposed migration procedure:

1. Migrate the unit tests to be called as add_test
2. Create a regression test wrapper (cp2k_add_test) that calls cp2k and sets appropriate regex evaluations: PASS_REGULAR_EXPRESSION SKIP_REGULAR_EXPRESSION FAIL_REGULAR_EXPRESSION
3. Add cp2k_add_test regression tests based on the folder structure
4. Add relevant test labels, e.g. unit, regression, smoke, etc.
5. Add the PROCESSORS property for parallelizing tests
   5b. (optional) Add handling of CTEST_RESOURCE_GROUP_* to efficiently parallelize GPU tests as well
6. Phase out do_regtest.py depending on CMake migration

---

Another use-case for do_regtest.py is to perform individual test evaluation, i.e. passing the full executable path, test name and additional input variables, the script would run the test and evaluate the stdout and everything else.

[oschuett]

Thanks for the suggestion. I haven't really looked at ctest yet. As we're migrating to CMake we should presumably also integrate with ctest.

That being said, our do_regtests script has been refined for over 20 years and contains many bells and whistles that we have come to rely on. So, I don't think we can retire it anytime soon.

[LecrisUT]

That being said, our do_regtests script has been refined for over 20 years and contains many bells and whistles that we have come to rely on. So, I don't think we can retire it anytime soon.

I don't suggest to completely retire it, just to gradually remove some functionalities out of it, as things are being covered by ctest, particularly the scheduling. I think it's still a perfectly valid choice to have do_regtests to wrap the cp2k execution and tests, but I think it should be considered more carefully on what is included in the python script beyond the basic functionality of performing regression tests (re: #2996)

[LecrisUT]

PS: Feel free to check my template about a design for the ctest testsuite. It includes unit testing, regression testing, packaging tests, and example testing. If you want to discuss it in more details, let me know.

[LecrisUT]

Just saw that CP2K_ENABLE_REGTESTS makes it write files in the source folder. That is really bad and error prone in multi-build environments like when packaging and trying to build for multiple MPI environments. I understand that it was done to have minimal effect on the do_regtest.py, but still that creates dangerous setups for the end-user in order to accommodate the limitations of the test scripts.