Reduce verbosity of python version?

[cHemingway]

Hi,

I've come back to this after working on some other stuff, and I have now ported a lot of my tests over to use metal.test, and I like how quick and easy it makes the edit/test cycle. However, is there any way to turn off printing every comparison made? If I am running a comparison in a loop, the screen rapidly fills with lines. I had a quick look in metal-unit.py, but couldn't see a parameter for this.

I suspect this also slows test execution down, as a breakpoint is hit every time a METAL_ASSERT macro is used? If the breakpoint could be disabled unless an error occured, I think this would speed things up.

[klemens-morgenstern]

There isn't currently, I guess there are two things here:

1. Don't trigger the breakpoints when it passes. (Compile option)
2. Don't print passing tests (Runtime option)

Will implement that soon, sounds useful in general.

Did you try one of my ranged test.macros? They are there to avoid that clutter

[cHemingway]

Of those options, 2 sounds good to me!
I suspect it could also implement 1 at runtime, if __metal_impl branched differently depending on pass/fail?

I did try using the METAL_RANGED macro with METAL_ASSERT and a custom condition, but the problem is that METAL_ASSERT still output for me, which slowed things down too much.
I am doing a seperate test on each element to avoid generating an array of the "correct" data, as its a bit big, though this is probably a bit of premature optimisation on my part.

[klemens-morgenstern]

Added (2), you can `set parameter metal-test-print-level " to all/warning/error to reduce the amount of output generated by gdb. This will still hit the breakpoint everytime, so your time savings might not be as awesome as you'd hope.

[klemens-morgenstern]

Added conditions for METAL_UNIT_ONLY_ERRORS and METAL_UNIT_ONLY_WARNING

If you define either this will not trigger the breakpoint. Note however, that this will mess up the statistics at this point, i.e. the tests will also not show up as "executed". I will hopefulle change that later on, hence keeping the issue open.

[cHemingway]

That was quick, thanks!

Adding METAL_UNIT_ONLY_WARNINGS before #include "metal/unit.h" supresses all the assertion warnings for me, but I still get all the messages from METAL_CALL.
I have opened a pull request which I think fixes this?

It would be nicer if this could be enabled/disabled at runtime though. Maybe if __metal_unit_break looked like this?

void METAL_NO_INLINE __metal_unit_break(__metal_level lvl,
               __metal_oper oper,
               int condition,
               int bitwise,
               const char* str1,
               const char* str2,
               const char* str3,
               const char* file,
               int line)
{
 if (condition) {
    if (__metal_level_assert == lvl)
        __metal_errored |= !condition;
        
    __metal_status = condition; // Break here if METAL_ONLY_WARNINGS, METAL_ONLY_ERRORS
 } else {
   // GDB adds 2nd break here if *not* METAL_ONLY_WARNINGS etc is set, to log test has executed.
 }

[klemens-morgenstern]

I tried to use the conditions provided by GDB but they didn't work. I don't want to base by plugin on line numbers though, that is so easily broken, it's a maintenance nightmare.

I don't get why you want to disable the METAL_CALL - if you don't want it just write your tests into main.

[cHemingway]

Ah makes sense, I didn't think properly about line numbers being an issue, but it totally would be easily broken. I guess it would need to be split into two functions then which would add some overhead, and I can understand why you would not want to add that complication.

I feel that disabling METAL_CALL when METAL_UNIT_ONLY_WARNINGS is set is more consistent? As then if I set METAL_UNIT_ONLY_WARNINGS, I will only see warnings and errors in the test output, not "entering test case" etc.

Regarding my tests, if I put them all in main, then to disable a group of tests (e.g, dealing with a particular peripheral/feature), I would need to comment out a bunch of lines or wrap the whole group in an if() statement or macro. Putting related tests in a function makes it easier to enable/disable them.

[klemens-morgenstern]

You can just put your tests in plain functions. The only reason to use METAL_CALL is to have them grouped for statistics, so what you want to achieve is literally that, just calling the function directly.

[cHemingway]

Understood, I have tried set parameter metal-test-print-level warning, and this nicely works showing only warnings generated.

Should I close the pull request #9 ?

Would you accept a different pull request that changed the python scripts in GDB instead, e.g. adding a parameter metal-test-print-cases that is default true, but can be set to false to hide the printing? The statistics in the JSON file would still work then.

[klemens-morgenstern]

Sure would. Though I think moving the stats to the target might be a better idea in the long run. I won't get around to doing that soon though.