Improve math test environment and vectors

[aolofsson]

The current test environment is quite primitive. It works with a set of golden vectors in a tabular text format.

Example:
pal/src/math/test/p_log10_f32.dat

Currently some of the functions are missing test vectors, and certainly all functions need more exhaustive testing...open to suggestions with respect to framework. (I am sure there is a lot out there).

I have had great success with plain text based unit testing in the past. (at least as an intermediate format).

The current pal/src/test_main.c is VERY primitive. What I would prefer not having is a personalized test function for each function that is copy pasted or auto generated by some other program (been there done that...) Since all of the functions are quite similar and math oriented, it seems that a common single test framework with input data and expected data is the way to go...

[wejn]

Would it be within the realm of possibility to use some generator in higher-level language to tackle the tests?

I know you wrote you'd prefer not to have it generated by some other program... but honestly, that tests/math/Makefile.am stuff is making my head spin (guess because I'm not that skilled with automake). Not to mention the exact way some macros in tests/math/simple.c work is also beyond my comprehension. Not "newbie" friendly. ;-)

Also, the csv format, as it stands, doesn't really lend itself to functions that take two vectors (p_sort, for instance). It could be extended but then the sed "parser" probably won't cut it.

In any case, would you be open to a simpler test suite in, say, Ruby?

One last thing -- I've had a rather nice experience with rantly gem which is QuickCheck's poor cousin written in Ruby. Property based testing is, in my opinion, far more robust than a few hardcoded tests. I think it might be beneficial to actually write property-based tests in favor of hardcoded values. There are property-based checkers for many languages (including C) which might make adoption easier.

The question with property-based testing, of course, is how to check the implementation (properties) against reality. Would it make sense to compare against libc (etc) where possible?

[aolofsson]

The testing environment currently leaves a lot to be desired. Agree 100%! Open to suggestions and contributions.

The only thing I would ask is that there is an intermediate text format with some pre determined format that can be fed as an input/output to the function so that folks can bypass the test generation infrastructure. (for example feeding in from matlab and other tools). This has saved me many times in hardware design, but perhaps this is over constraining things for software development? Open to suggestions/feedback..

[olajep]

The only thing I would ask is that there is an intermediate text format with some pre determined format that can be fed as an input/output to the function so that folks can bypass the test generation infrastructure.

Added reading from CSV file for math tests back here:
"tests:math:simple: Support passing gold file on command line"
fcb69c4

[mateunho]

I've been thinking about (and developing in hiding 😃) expansion of current test framework to cover all cases of I/O to functions. There's only one problem I got so far. Test data files are converted to C struct's at compile time by a shell script (within Makefile). This script assumes that all data passed to given function has to be float type, so it appends f to every value.
Would it be a big issue if I just remove the f?
Is it essential for test performance or may be sacrificed in the name of full coverage?

[lfochamon]

@olajep I see you've been rolling out your own test infrastructure. Check really isn't suited for embedded environments. Have you checked out https://github.com/ThrowTheSwitch/Unity? I've used it in past embedded devs and it's really good. It's all pure ANSI C (I've compiled it for ATMEL and MSP430), fairly small (14KB, which granted is more than the 3KB you've been getting), and flexible (it only requires some sort of putchar that you can even define yourself). It might be a nice fit here (I thought of using it on the dsp functions as an example). Or at least it could give you some inspiration for uTest. Hope it helps!