Test suite

Built-in test suite

As with any big software software libraries, developing IFEM is a complex process. In particular, IFEM has many application areas, and it is not always easy to know all the consequences of a change you do.

For this reason, IFEM includes a number of helpers to facilitate proper testing. These are all integrated with the build system for simple utilization.

Basic test

The main library, as well as all applications developed at SINTEF Digital using IFEM, has a built-in regression and unit tests. This will compile and run all tests defined in the 'Test' subfolder of the application. You run this target through

$ make check

Note that the default 'test' target will only execute tests, but due to limitations in cmake this will not build the tests. The 'check' target was added to builds tests then execute them.

Testing a patch series

Sometimes, when developing a huge change, you split your changes into a set of patches for better readability and revision control. When doing this, it is easy to unintentionally break the build at some step in the process, something that is highly dis-encouraged since it would make things like bisecting the tree much harder. To aid in avoiding this pitfall, the IFEM build system has a built-in 'make check-commits' target. This will run through the patchset, compile it at each level, and run the regression tests.

Using this requires that you use GIT as your revision control system.

By default all commits between the head of the 'trunk' branch and the currently selected branch will be tested. You can control this by the two environment variables

• 'CHECK_BASE' for the base commit
• 'CHECK_HEAD' for the head commit.

To execute this command, you run e.g. run

$ make check-commits CHECK_BASE=HEAD~1

This will check only the last commit (bit useless, you could as-well use the 'make test' target).

On the top level, you also have a target which will checks all application at each commit

$ make check-commits-all

You can control the number of threads used by the CHECK_THREADS environment variable (defaults to 3).

You can do a compile-only test by setting the environment variable CHECK_MODE to 'buildonly'.

Adding a regression test

IFEM has built-in regression testing, using a custom script and the CTest framework.

Adding a regression test is fairly simple. It consists of two steps:

• Create a .reg file. A .reg file is a simple text file on the format

|                                  |
|----------------------------------|
| params to program
 <blank line>
 output from program to check for  |

• Add it to the CTest tests. This is done through CMakeLists.txt, example entry

ifem_add_test(${TESTFILE} ${APPLICATION})

Adding a unit test

IFEM has built-in unit testing, using a combination of the gtest and CTest frameworks.

To add a unit test, simply create a .C file in the Test subdirectory of your application. Make sure to add the unit tests in cmake using

include(IFEMTesting)
add_check_target()

at the end of your CMakeLists.txt.

Adding a VTF regression test

You can add a test for verifying that the contents of a VTF file is correct. First you need to install a comparison tool on your machine, https://github.com/TheBB/vtfls

Adding a VTF regression test is fairly simple. It consists of two steps:

• Create a .vreg file. A .vreg file is a simple text file on the format

|                                  |
|----------------------------------|
| params to program
 <blank line>
 output from vtfls program to check for  |

• Add it to the CTest tests. This is done through CMakeLists.txt, example entry

ifem_add_vtf_test(${TESTFILE} ${APPLICATION})

Static code analysis

The IFEM buildsystem (including the applications) has built-in support to execute static code analysis on the sources.

Two analyzers are supported:

• cppcheck - if it is installed you are ready to go.
• clang-check - needs to be installed, and you need to configure with

cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DIFEM_USE_OPENMP=0

You can execute these tests in a build-directory through

ctest -C [analyze|cppcheck|clang-check] -V

Analyze executes both tools if available, while the other configurations only executes the tests with the named tool. Note: Due to the way CTest configurations work, the usual regression and unit tests will also be executed.