You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some of our projects have upwards of 2000 test methods. When we run force:apex:test:run -w 50 -r human in our CI/CI scripts and there is a single failed test method, we see 1999 lines of "Pass" and we have to painfully try to hunt down the single test method with the Fail status.
In a prior version of the CLI, it would only report test failures which meant that for a 100% successful test run, it would only print the summary block with the outcome of the test run -- there's no need to print 2000 lines of noise in the log file telling me that each test method was successful
Something changed at some point within the past few months (basically at the point where passing -r human was required in order to see the detail of the test failures)
Solution 1 - silence test methods that Pass by default
Pros: This is the way it should be and the way just about every other test reporter I've ever seen in my life has worked. Users that want all Passing test methods to be enumerated can use the --verbose existing command line argument to enable.
Cons: This will change existing behavior
Solution 2 - Add a command line argument (eg. --not-so-verbose) to silence Pass test methods from the output
Pros: Preserves existing behavior
Cons: Requires updating existing CI/CD scripts to pass this new flag. Also, documenting and naming the flag to make it somehow compatible with the --verbose documentation could be challenging.
Additional context
Also, it should be noted that in the Summary table, the "Pass rate" is a whole number (presumably using half-up rounding logic) which results in incongruous outcome:
=== Test Summary
NAME VALUE
─────────────────── ──────────────────────────
Outcome Failed
Tests Ran 870
Pass Rate 100%
Fail Rate 0%
Skip Rate 0%
Test Run Id 7071F0000279qGm
Test Execution Time 280082 ms
Org Id 00D1F000000aI3mUAE
Username robot.1631036532038@pt.com
Which is it? A failure or 100% pass? What actually happened is one test method failed but that still resulted in the "99.9%" getting rounded up to 100%
Some of our projects have upwards of 2000 test methods. When we run
force:apex:test:run -w 50 -r human
in our CI/CI scripts and there is a single failed test method, we see 1999 lines of "Pass" and we have to painfully try to hunt down the single test method with the Fail status.In a prior version of the CLI, it would only report test failures which meant that for a 100% successful test run, it would only print the summary block with the outcome of the test run -- there's no need to print 2000 lines of noise in the log file telling me that each test method was successful
Something changed at some point within the past few months (basically at the point where passing
-r human
was required in order to see the detail of the test failures)Solution 1 - silence test methods that Pass by default
Pros: This is the way it should be and the way just about every other test reporter I've ever seen in my life has worked. Users that want all Passing test methods to be enumerated can use the
--verbose
existing command line argument to enable.Cons: This will change existing behavior
Solution 2 - Add a command line argument (eg.
--not-so-verbose
) to silence Pass test methods from the outputPros: Preserves existing behavior
Cons: Requires updating existing CI/CD scripts to pass this new flag. Also, documenting and naming the flag to make it somehow compatible with the
--verbose
documentation could be challenging.Additional context
Also, it should be noted that in the Summary table, the "Pass rate" is a whole number (presumably using half-up rounding logic) which results in incongruous outcome:
Which is it? A failure or 100% pass? What actually happened is one test method failed but that still resulted in the "99.9%" getting rounded up to 100%
Version that I'm running
The text was updated successfully, but these errors were encountered: