Feat: add optional configurable tests summary

[pmalek]

Use case

I have a set of pretty long integration tests and I want to retain the output from them in pretty much the standard-verbose fashion (that's what I use for now on CI).

Since the output is so long and I intend to keep it, it's hard to go through the log and find each individual PASS: TestXXX (12.34s) entry.

Proposed solution

It would be really nice if gotestsum could add a toggle (opt-in probably) to print a summary of all tests with their run times.

This way we could use standard-verbose format but also get the summary at the end.

Considered alternatives

Parse --jsonfile and use --post-run-command with a custom written script that will do the above

[dnephin]

Thanks for suggesting this! Using --jsonfile and --post-run-command to do this is a really interesting approach. It should be possible without a custom script:

gotestsum \
    --jsonfile tmp.json.log \
    --hide-summary all \
    --post-run-command "gotestsum --format testname --raw-command -- cat tmp.json.log"

Unfortunately, that does not currently work because the jsonfile is closed after --post-run-command runs.

I'd like to fix that bug. Once fixed the command above should solve that use case, although it is still quite verbose. It might be interesting to add a new flag to enable that behaviour.

Another approach would be to run gotestsum --format testname --jsonfile tests.log. Print only the test names and failure output, but keep a copy of the full verbose log in a file. From that --jsonfile you have a few options:

• print the standard verbose format with gotestsum --format=standard-verbose --raw-command -- cat tests.log, possibly as a separate build step
• save either the standard verbose output or the tests.log as a CI artifact, so that it can be downloaded later if needed for debugging

[pmalek]

gotestsum --format=standard-verbose --raw-command -- cat tests.log

This is pretty neat! Thanks, I wasn't aware of this. The small problem with this is that I'll get all the output from e.g. TestMain and coverage, which will bloat the output.

---

As for the workarounds: this is what would work for me:

gotestsum \
    --jsonfile tmp.json.log \
    ... more flags ...
jq -c 'select(.Action | match("pass|skip|fail"))' < tmp.json.log > tmp2.json.log
$(GOTESTSUM) --format testname --raw-command -- cat tmp2.json.log

but a problem with this is that I'm running that from a Makefile and then when a test fails I won't get the summary. I can run a target like so: @$(MAKE) --ignore-errors ... but then I won't get the CI to fail 🙄

Any advice 😅 ?

[dnephin]

If I understand correctly, I think fixing the bug I mentioned above (writing the jsonfile before calling post-run-command) should fix the issue with errors. The post-run-command is run even on error, so you don't have to run it as a separate command.

In another issue (#258 (comment)) there was a proposal to add something like --jsonfile-timing (or something along those lines), which would be similar to --jsonfile except that it would omit all of the output. The file would only include the events for tests starting and tests exiting.

It sounds like that flag would remove the need for the jq filter. Does that sound right?

My hope is that fixing the jsonfile bug, and adding a way to print only the non-output events would get this working the way you want.

[pmalek]

In another issue (#258 (comment)) there was a proposal to add something like --jsonfile-timing (or something along those lines), which would be similar to --jsonfile except that it would omit all of the output. The file would only include the events for tests starting and tests exiting.

It sounds like that flag would remove the need for the jq filter. Does that sound right?

That would be something I'm looking for. Being able to get top n slowest tests would be a cherry on top but that can also be achieved by some jq filter as well or custom script.

[dnephin]

Being able to get top n slowest tests would be a cherry on top

gotestsum tool slowest should be able to do that for you. It reads the same jsonfile.
https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests

Currently it does "all tests slower than threshold", rather than "slowest n tests". It shouldn't be too difficult to add a new flag to output "n slowest" instead.

[dnephin]

With #306 adding the new --json-timing-events flag, and #318 adding the gotestsum too slowest --num flag I think it should be possible to do what you want. The full command would be this:

gotestsum \
  --jsonfile-timing-events tmp.json.log \
  --format standard-verbose --hide-summary all \
  --post-run-command "bash -c '
    echo; echo Slowest tests;
    gotestsum tool slowest --num 10 --jsonfile tmp.json.log'"

I'll look to do a release soon with those changes.

[pmalek]

That's cool! Thanks @dnephin.

Would it be possible to integrate the slowest tests report printing within the gotestsum test invocation step? So that instead of the above command, one could use:

gotestsum \
  --format standard-verbose --hide-summary all \
  --print-slowest-tests 10

The name of the flag (print-slowest-tests) would be up for debate but this way users don't need to deal with temporary files themselves. If they use --jsonfile-timing-events then surely they want to utilize that file somewhere after the test run but if someone is only interested in the slowest tests there's no need to specify that and a temporary file could be used under the covers. WDYT?

[dnephin]

Ya, I think it's worth still considering a new flag. I'll re-open this issue to see if others are interested. I'm not sure how common it is to want to see the slowest tests on every run. Do you want to use this for local development or in CI?

[pmalek]

I believe both. CI to keep a history of what were the N slowest tests and locally to verify a fix that one might work on