Format option proposal: pkgname-coverage-graph

[kushuh]

I like to work with coverage, and I think it would be a great addition to have an output format, that would help identify which packages are badly covered, at a glance.

The format would be based on the existing pkgname one. Here is an example of what it could look like:

Thoughts about the global idea ? I drafted the presentation and colors so it might be improved. I like the more graphical aspect for coverage. It could also work easily when coverage is not enabled:

The end of the input could look like pkgname. A line could also show the global test coverage.

[dnephin]

This is really cool, thanks for opening this issue!

#293 was recently merged, which I think is trying to accomplish a similar thing, but there's always room for more improvements.

My initial thoughts on these new formats:

I'd prefer to avoid the "0 failed" output for packages that have not failed. Making it a dark grey helps make it less prominent, but I think we can omit it entirely since it should be implied by the "package passed all tests" indicator (the check mark and color). Printing the number of failed tests for a package that failed is a nice addition.

Should we also print the number of skipped tests, if we are printing the number of failed ones?

Printing the number of tests in the package also seems like a great addition. When I first wrote gotestsum there were some strange inconsistencies in the test2json input where some times tests for a package would finish after the package end statement. It's been 5+ years, so I suspect those inconsistencies are now fixed, but it's something we'll have to confirm, otherwise the test counts for the package would often be incorrect.

I've got mixed feelings about coloring the entire line, and using a different color for the elapsed time. It definitely makes the failures more obvious, but I think it becomes a lot harder to make it work generally. There are lots of people out there that have different color schemes for their terminals (I generally use something like solarized-dark). When text is a different color from the default it can be difficult to read. One example of this is hashicorp/go-hclog#108. hclog used to color the entire line, but it was difficult to read, so they changed it to only colorize the log level (like most other Go logging libraries). If we make this a separate format or a format option (--format-color-full ?) that should let users opt-in when it works for them.

I don't use coverage much myself, but I like the idea of making the percentage more visually obvious in the output. My main concern with the bars is that it may be difficult to determine the right width for the bars. A bar that is too wide is going to look terrible if it wraps on a very narrow terminal. A very short bar might not look great when the packages names are much longer than the bar. The bars might work, but I expect they will take some experimentation.

Color coding the coverage value based on the percentage seems like a really great start to making the values more obvious. We could do something like this:

• over 90% -> regular color , with a star
• 50-90% -> yellow
• under 50% -> red

I'm not sure if 50% and 90% are the right places to put the thresholds. I imagine there's some prior art for this kind of thing in other tools.

Thoughts? Do you already have a branch that implements these changes? The screenshots make it look like it's already all working, which is cool!

[kushuh]

Thanks ! I agree that the color scheme is aggressive, softer schemes can be experimented, to accommodate most terminals. The --format-color-full could be a solution, maybe as a later option.

Also yeah, skipped tests is more relevant than printing 0 failed test, in case of success.

For the bars, I went for 10% = 4bars, because then 100% = 80 bars which is a standard size in code. When I looked at the source code, there is already a logic that seems to fetch the current terminal width (the termWidth here), so maybe this can be used to determine an optimal bar length.

My initial idea was to leave bars the same color whatever the coverage is, and turn them red when the test failed to make it more obvious. But after thinking about the color discussion above, maybe leaving the text monochromatic and keeping colors for different coverage ranges makes more sense. And yes, the range limits are totally arbitrary (50, 90), but it should be easy to experiment with.

I haven't made a functional branch with those changes yet (the screenshots are mocked, based on the gif example in the Readme): I first wanted to ensure the idea was interesting. But I can for sure work on this during next week !

[fgimian]

One thing that I haven't found in the Go ecosystem so far is coverage reporting which tells you which lines were missed from each file too, similar to the coverage.py output below:

$ coverage report -m
Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
my_program.py                20      4    80%   33-35, 39
my_other_module.py           56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%

Do you think this is something that could also be incorporated here somehow?

Huge thanks
Fotis

[dnephin]

go tool cover (https://pkg.go.dev/cmd/cover) is a similar idea. It can produce 2 reports: coverage percentage per function, and a detailed HTML report. The HTML report effectively has this information, but it's not presented this way.

It seems like it wouldn't be too difficult to copy and modify https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/cmd/cover/html.go to make a new report in the format you want. It is possible there is a tool out there that has already done the work, I haven't looked. If you want to build it, it could work as gotestsum tool cover-report or something similar. A new issue for this would be great.

golang/go#31519 is the only issue I could find about it in the go issue tracker. It's possible I missed others, I did not look at closed issues. Sometimes comments on the go issue tracker can link to external tools.

[fgimian]

go tool cover (https://pkg.go.dev/cmd/cover) is a similar idea. It can produce 2 reports: coverage percentage per function, and a detailed HTML report. The HTML report effectively has this information, but it's not presented this way.

It seems like it wouldn't be too difficult to copy and modify https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/cmd/cover/html.go to make a new report in the format you want. It is possible there is a tool out there that has already done the work, I haven't looked. If you want to build it, it could work as gotestsum tool cover-report or something similar. A new issue for this would be great.

golang/go#31519 is the only issue I could find about it in the go issue tracker. It's possible I missed others, I did not look at closed issues. Sometimes comments on the go issue tracker can link to external tools.

Thank you so much for the reply. I'll do some more hunting too and open a new issue if I don't find anything suitable. 😄

Cheers
Fotis