Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Human formatted Apex test report shouldn't display individual PASSING test methods #243

Open
daveespo opened this issue Sep 7, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@daveespo
Copy link

daveespo commented Sep 7, 2021

Some of our projects have upwards of 2000 test methods. When we run force:apex:test:run -w 50 -r human in our CI/CI scripts and there is a single failed test method, we see 1999 lines of "Pass" and we have to painfully try to hunt down the single test method with the Fail status.

In a prior version of the CLI, it would only report test failures which meant that for a 100% successful test run, it would only print the summary block with the outcome of the test run -- there's no need to print 2000 lines of noise in the log file telling me that each test method was successful

Something changed at some point within the past few months (basically at the point where passing -r human was required in order to see the detail of the test failures)

Solution 1 - silence test methods that Pass by default

Pros: This is the way it should be and the way just about every other test reporter I've ever seen in my life has worked. Users that want all Passing test methods to be enumerated can use the --verbose existing command line argument to enable.

Cons: This will change existing behavior

Solution 2 - Add a command line argument (eg. --not-so-verbose) to silence Pass test methods from the output

Pros: Preserves existing behavior

Cons: Requires updating existing CI/CD scripts to pass this new flag. Also, documenting and naming the flag to make it somehow compatible with the --verbose documentation could be challenging.

Additional context
Also, it should be noted that in the Summary table, the "Pass rate" is a whole number (presumably using half-up rounding logic) which results in incongruous outcome:

=== Test Summary
NAME                 VALUE
───────────────────  ──────────────────────────
Outcome              Failed
Tests Ran            870
Pass Rate            100%
Fail Rate            0%
Skip Rate            0%
Test Run Id          7071F0000279qGm
Test Execution Time  280082 ms
Org Id               00D1F000000aI3mUAE
Username             robot.1631036532038@pt.com

Which is it? A failure or 100% pass? What actually happened is one test method failed but that still resulted in the "99.9%" getting rounded up to 100%

Version that I'm running

@oclif/plugin-autocomplete 0.3.0 (core)
@oclif/plugin-commands 1.3.0 (core)
@oclif/plugin-help 3.2.2 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-plugins 1.10.1 (core)
@oclif/plugin-update 1.4.0-3 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
@salesforce/sfdx-trust 3.6.0 (core)
alias 1.1.10 (core)
apex 0.2.8 (core)
auth 1.7.1 (core)
config 1.2.24 (core)
custom-metadata 1.0.12 (core)
data 0.6.1 (core)
etcopydata 0.6.4-Beta (beta)
generator 1.1.7 (core)
limits 1.2.1 (core)
org 1.7.0 (core)
salesforce-alm 52.3.1 (core)
schema 1.0.8 (core)
sfdx-cli 7.116.2 (core)
shane-sfdx-plugins 4.43.0
├─ @mshanemc/plugin-streaming 1.1.7
└─ @mshanemc/sfdx-sosl 1.1.0
source 1.0.12 (core)
telemetry 1.2.3 (core)
templates 52.1.0 (core)
user 1.4.0 (core)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants