Compile list of testing frameworks and tools

[verolero86]

Please add your comments here on which tools your HPC center is using for regression or acceptance testing.

[verolero86]

NERSC: ReFrame for regression testing. Also use Kibana and Grafana.

[amaji]

Purdue: Testpilot

[novosirj]

Purdue: Testpilot

It appears as if the framework that provided pass/fail and single command running was Testpilot. I'd like to view the HUST17 presentation that was given by Purdue, but it seems to be missing from the HUST17 site. @amaji, do you think you can come up with that/maybe upload it there (it's supposed to be at https://hust17.github.io/2017_presentations/test-pilot.pdf -- github is here: https://github.com/hust17/hust17.github.io, and I see that @chrisbpawsey uploaded the rest).

Also, how does one acquire TestPilot?

[amaji]

@novosirj Thanks for pointing out. I never realized that our presentation was not uploaded. Will make sure that it is in HUST17 site.

Regarding releasing Testpilot: We are currently porting it to Slurm. I'll see if we can put it in Github early next year.

[novosirj]

That’s great news, and I also don’t have to feel bad for taking two years to figure out what software it was/implement it. 😅

Let me know if we can help (testing, code changes, etc.).

[shahzebsiddiqui]

buildtest
Github: https://github.com/HPC-buildtest/buildtest-framework
Docs: http://buildtest.rtfd.io/
Slides from HUST 19: https://hust-workshop.github.io/2019_presentations/1-buildtest.pdf

[tkphd]

Some tools mentioned at the SC21 BOF:

• https://github.com/eth-cscs/reframe
• https://github.com/hpc/pavilion2
• https://github.com/olcf/olcf-test-harness

Some published test suite invocations:

• https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-tests
• https://eessi.github.io/docs/software_testing/
• https://github.com/stackhpc/hpc-tests

[gweodoo]

Test orchestrator:

• https://pcvs.hpcframework.com/
• https://github.com/cea-hpc/pcvs

Benchmark collection:

• https://github.com/cea-hpc/pcvs-benchmarks
• https://inria.hal.science/hal-04156064v1/ (mpicheck)