unit.tutorial

Running a test

While the macros and the basic structure of the test should seem familiar to everyone who ever wrote a text, the way it was launched may not.

Our test-library is meant to be used with our gdb-runner. This tool calls gdb, parses the output and executes a test with fitting plugins for breakpoints. This is the most straight-forward way to execute a test on a remote device, such as a microcontroller. This library provides a plugin for the gdb-runner, named metal-test-backend; how to use it is described in detail in the gdb-runner documentation. A test launched with the runner will be called hosted in this document.

For small tests, especially when integrated into a CI-System, the gdb-runner might be too slow and all the generated information not needed. To do this, we also provide a standalone mode, with the bare minimum set of features. It does not generate any output, nor does it have any flow-control. It just will count the errors and return a value != 0 in the main, so the build-system sees an indicated error. In this document this will be the called the standalone mode.

You can add coverage support in the hosted environment by using the metal-newlib plugin, so the compile gcov code can write into files.

Test Levels and Macros

metal.unit provides two test levels, error and warning expressed in the code as

• ASSERT
• EXPECT

Both will be written into the console, but only errors will cause the test to fail, i.e. return a value != 0, that is.

All tests are implemented through macros, so parameter names and location can be obtained easily. That way the backend can provide detailed information of the code. The usual format for the macros looks like this:

    METAL_``['`level`]``_``['test-name]``(``['arguments]``);

We will use a few of them in examples, though to see all, please refer to the reference.

As an example, we use the equality check:

METAL_ASSERT_EQUAL(x, y);
METAL_EXPECT_EQUAL(i, j);

In standalone mode, there will not be any output for this example. However if we use it the hosted mode the gdb-runner will not only output the names of the parameters and the location of the test, but also provide information about the actual value. Since the debugger is used for this, we can not only output types with a defined serialization operator, but everything.

*The format of the output depends on the gdb version.] *

For the given example, the output could look like this.

    test.cpp(10) assertion succeeded [equality]: x == y; [3 == 3]
    test.cpp(11) expectation failed [equality]: i == j; [42 == -1]

Static Tests

Tests are static when they do not need to executed, which requires that they are constant-expressions. An example could look like this:

    METAL_STATIC_ASSERT_EQUAL(sizeof(int), 4u);

This currently uses static_assertin C++ and a simulation of this behaviour in C, which means that a test cannot be compiled when it fails. This means that no static_warning is provided. This behaviour may change in a future version.

Bitwise

Some tests provide an alternative called bitwise. This will take the corresponding operation and apply it for every single bit. It also provides bitwise output.

   METAL_ASSERT_EQUAL_BITWISE(c, u);

This could produce output like this.

    test.cpp(33) assertion succeeded [bitwise equality]: l == k; [0b11111111 == 0b11111111]

• The logical connection between the bits still differs; a not equal will fail if any of the bits is different, while an equal will fail if any of the bits are different. Please see the reference for details.*

Since the shift operator is used, the behaviour is implementation-defined for signed types.

Ranged Tests

For many tests this library also provide ranged tests. This makes test writing easier and cleans up the output.

This behaves differently between C++ and C - C++ expects two iterators, while C expects a pointer and size.

An example in C++ could look like this.

    std::array<int,   3> arr1 = {-1,0,1};
    std::array<short, 4> arr2 = {-1,0,1,2};

    METAL_ASSERT_EQUAL_RANGED(arr1.begin(), arr1.end(), arr2.begin(), arr2.end());

Which would produce a similar output.

    test.cpp(21) error entering ranged test [{[arr1.begin(), arr1.end()], [arr2.begin(), arr2.end()]}] with mismatch: 4 != 3
    test.cpp(21) assertion succeeded [equality]: **range**[0]; [-1 == -1]
    test.cpp(21) assertion succeeded [equality]: **range**[1]; [0 == 0]
    test.cpp(21) assertion succeeded [equality]: **range**[2]; [1 == 1]
    test.cpp(21) exiting ranged test: { executed : 3, warnings : 0, errors : 0}

In C an example could look like this.

   int arr1[3] = {-1,0,1};
   int arr1[4] = {-1,0,1};
   METAL_EXPECT_EQUAL_RANGED(arr1, 3, arr2, 4);

Which would produce similar output.

This test start has a mismatch, which will produce an error. This will however not appear in the list of the erros of the ranged test.

Test Structure

Test cases]

A test can be written directly into the main function, but we also provide a solution to seperate test cases using METAL_CALL. In hosted mode these entering and leaving test cases will be documented. This allows grouping tests and controlling the test flow. The same tests

Tests not written inside cases are called free tests.

void test_cases_1()
{
    METAL_ASSERT_EQUAL(42, 42);
}

void test_cases_2()
{
    METAL_ASSERT_NOT_EQUAL(42, 42);
}

int main(int argc, char *argv[])
{
    METAL_EXPECT_EQUAL(42,0);
    METAL_CALL(&test_cases_1, "Test Case 1");
    METAL_CALL(&test_cases_2, "Test Case 2");
    return METAL_REPORT();
}

Which would provide output like this.

    starting test execution
    cases.cpp(16) expectation failed [equality]: 42 == 0; [42 == 0]
    cases.cpp(17) entering test case [Test Case 1]
    cases.cpp(6) assertion succeeded [equality]: 42 == 42; [42 == 42]
    leaving test case [Test Case 1]: { executed : 1, warnings : 0, errors : 0}
    cases.cpp(18) entering test case [Test Case 2]
    cases.cpp(11) assertion failed [equality]: 42 != 42; [42 != 42]
    leaving test case [Test Case 1]: { executed : 1, warnings : 0, errors : 1}
    free tests : { executed : 1, warnings : 1, errors : 0}
    full test report: { executed : 2, warnings : 1, errors : 1}

Currently METAL_CALL only accepts function pointers with the signature void().

Critical tests

Critical tests are test, that will cause the test to chancel if failing. This of course only works in hosted mode, but the METAL_ERROR allows you to implement a custom handler for that.

void test_case()
{
    METAL_CRITICAL(METAL_ASSERT(0));
}

int main(int argc, char * argv[])
{
    METAL_CALL(func, "my test case");
    METAL_EXPECT(0);
    return METAL_REPORT();
}

Which might have this output.

    starting test execution
    cancel_case.cpp(11) entering test case [my test case]
    cancel_case.cpp(6) critical assertion failed [expression]: 0
    canceling test case [my test case]: { executed : 1, warnings : 0, errors : 1}
    cancel_case.cpp(12) expectation failed [expression]: 0
    free tests : { executed : 1, warnings : 1, errors : 0}
    full test report: { executed : 2, warnings : 1, errors : 1}

A failing critical exception will not result in an error, but still stop execution.

Header files

There are three ways to include the macros for the test.

Header	Language	Comment
<metal/unit>	C/C++	Automatically selects the flavour
<metal/unit.h>	C	Macros in C style
<metal/unit.hpp>	C++	Macros in C++ style
<metal/unit.ipp>	C/C++	The function actually used for the breakpoint. It's included by the other headers unless METAL_NO_IMPLEMENT_INCLUDE is defined.