test: add cases for output functions #937
Conversation
G-Rath
force-pushed
the
test-outputs
branch
4 times, most recently
from
1b960fb
to
debe3ad
Compare
|
This is ready for review but #938 should be merged first. Note that beyond the usual review of the technical implementation, reviewers should also review the test cases to make sure they're correct for the real world and are complete, and also review the snapshots for each outputs for any oddentities that we might want to change (for which dedicated issues should be created - they won't be addressed in this PR) |
G-Rath
force-pushed
the
test-outputs
branch
2 times, most recently
from
3d16a73
to
75150a5
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #937 +/- ##
==========================================
+ Coverage 63.71% 63.94% +0.22%
==========================================
Files 146 146
Lines 11958 11958
==========================================
+ Hits 7619 7646 +27
+ Misses 3875 3854 -21
+ Partials 464 458 -6 ☔ View full report in Codecov by Sentry. |
This stabilizes the order in the SARIF output so that it is deterministic, as #937 proved it was not.
|
fwiw the coverage output for this looks even better after #961 is landed, as it means we've got 100% coverage across most files in the package 😄 |
This introduces a set of crafted scanner results that each supported
outputformat is run through to showcase how they look across all the different results possible from a scanner run - it originally started life as the tests for #889 but I realised they could base used more generally for testing and reviewing all the outputters, so here we are.It looks like this has also revealed the SARIF output is unstable in its ordering, which I'll aim to address in a dedicated PR