Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SkipRemainingOnFailure reports skipped on filtered and explicitly skipped tests #2440

Open
3 tasks done
fflaten opened this issue Apr 29, 2024 · 0 comments · May be fixed by #2442
Open
3 tasks done

SkipRemainingOnFailure reports skipped on filtered and explicitly skipped tests #2440

fflaten opened this issue Apr 29, 2024 · 0 comments · May be fixed by #2442
Labels

Comments

@fflaten
Copy link
Collaborator

fflaten commented Apr 29, 2024

Checklist

What is the issue?

Run.SkipRemainingOnFailure processes tests that are both excluded by filter and/or skipped using -Skip. They are included in the summary

In the repro below, 3 tests were included by filter but one was skipped. The summary says Remaining tests skipped after first failure: 3 when only 1 test was actually skipped by the plugin.

Pester v5.6.0-beta1

Starting discovery in 1 files.
Discovery found 4 tests in 18ms.
Filter 'Tag' set to ('Demo').
Filters selected 3 tests to run.
Running tests.
Describing A
This is test 2
  [-] Test 1 11ms (4ms|7ms)
   Expected 2, but got 1.
   at 1 | Should -Be 2, untitled:Untitled-7:5
   at <ScriptBlock>, untitled:Untitled-7:5
 Describing B
   [!] Test 3 6ms (0ms|6ms)
   [!] Test 4 0ms (0ms|0ms)
Tests completed in 67ms
Tests Passed: 0, Failed: 1, Skipped: 2, Inconclusive: 0, NotRun: 1
Remaining tests skipped after first failure: 3

Expected Behavior

Don't mark excluded tests as Skip.
Update plugin summary to only count skipped tests that it was responsible for (exclude -Skip tests).

Steps To Reproduce

$sb = {
    Describe 'A' {
        It 'Test 1' -Tag 'Demo' {
            Write-Host 'This is test 2'
            1 | Should -Be 2
        }
        It 'Test 2' {
            Write-Host 'This is test 3'
        }
        Describe 'B' {
            It 'Test 3' -Tag 'Demo' {
                Write-Host 'This is test 2'
            }
            It 'Test 4' -Tag 'Demo' -Skip {
                Write-Host 'This is test 3'
            }
        }
    }
}

$conf = New-PesterConfiguration
$conf.Output.Verbosity = 'Detailed'
$conf.Run.ScriptBlock = $sb
$conf.Run.SkipRemainingOnFailure = 'Block'
$conf.Filter.Tag = 'Demo'

Invoke-Pester -Configuration $conf

Describe your environment

Pester version : 5.6.0-beta1 /workspaces/Pester/bin/Pester.psm1
PowerShell version : 7.4.1
OS version : Unix 5.15.133.1

Possible Solution?

Update criteria from if (-not $test.Executed) to if ($test.ShouldRun -and -not $test.Skip -and -not $test.Executed).

@fflaten fflaten added the Bug label Apr 29, 2024
@fflaten fflaten changed the title SkipRemainingOnFailure reports skipped on filtered and skipped tests SkipRemainingOnFailure reports skipped on filtered and manually skipped tests Apr 29, 2024
@fflaten fflaten changed the title SkipRemainingOnFailure reports skipped on filtered and manually skipped tests SkipRemainingOnFailure reports skipped on filtered and explicitly skipped tests Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
1 participant