Templates in test names do not expand variables when skipped

[GSDragoon]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

When using templates for data driven tests, the template values do not expand if the test is skipped. It correctly discovers and displays a skipped test for each value in the array, but does not expand any of the values. <_> or <Name> is in the test name instead.

Expected Behavior

I would expect the templates to be expanded for skipped tests too. That way each skipped test is able to be identified by a meaningful test name.

Steps To Reproduce

Example.Tests.ps1
Includes both skipped and non-skipped tests as well as arrays and hashtables to see the behavior in each scenario.

Describe 'Test names with templates using ForEach' {
  It 'Should display array values <_>' -ForEach @('foo', 'bar', 'test') {
    $_ | Should -Not -BeNullOrEmpty
  }

  It 'Should display array values, even when skipping <_>' -Skip:$true -ForEach @('foo', 'bar', 'test') {
    $_ | Should -Not -BeNullOrEmpty
  }

  It 'Should display hashtable values <Name> <Value>' -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ){
    $Name | Should -Not -BeNullOrEmpty
    $Value | Should -Not -BeNullOrEmpty
  }

  It 'Should display hashtable values, even when skipping <Name> <Value>' -Skip:$true -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ) {
    $Name | Should -Not -BeNullOrEmpty
    $Value | Should -Not -BeNullOrEmpty
  }
}

Then run Invoke-Pester -Configuration (New-PesterConfiguration @{ Output = @{ Verbosity = 'Detailed' } }) to execute the tests and see the names of each.

The output will look like this. Notice the skipped tests do not have the expanded template values.

Pester v5.5.0

Starting discovery in 1 files.
Discovery found 12 tests in 11ms.
Running tests.

Running tests from 'Example.Tests.ps1'
Describing Test names with templates using ForEach
  [+] Should display array values foo 9ms (2ms|7ms)
  [+] Should display array values bar 3ms (2ms|1ms)
  [+] Should display array values test 9ms (4ms|5ms)
  [!] Should display array values, even when skipping <_> 2ms (0ms|2ms)
  [!] Should display array values, even when skipping <_> 1ms (0ms|1ms)
  [!] Should display array values, even when skipping <_> 2ms (0ms|2ms)
  [+] Should display hashtable values foo test 1 8ms (3ms|5ms)
  [+] Should display hashtable values bar test 2 5ms (4ms|2ms)
  [+] Should display hashtable values foobar test 3 5ms (3ms|2ms)
  [!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
  [!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
  [!] Should display hashtable values, even when skipping <Name> <Value> 2ms (0ms|2ms)
Tests completed in 199ms
Tests Passed: 6, Failed: 0, Skipped: 6 NotRun: 0

Describe your environment

Pester version : 5.5.0 C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1
PowerShell version : 5.1.19041.4170
OS version : Microsoft Windows NT 10.0.19045.0

Possible Solution?

No response

[fflaten]

Thanks for the detailed report. The current behavior is by design.

Pester expands variables in the name between test setup and execution to support variables set in BeforeAll/BeforeEach. Skipping a test will skip all of this.

Related issue #2220

[csandfeld]

@fflaten while it may be by design, from a user perspective I would expect names to always be expanded no matter what state the test ends up in.

[nohwnd]

This would be hard for us to do and not end up with empty values for all the template names, offering about as much value as not expanding them at all. This is because the names are expanded by evaluating the string in the current execution contex of the test. So any variable or simple subexpression will be expanded correctly.

[csandfeld]

I am sure there are good reasons for the current implementation, but I did notice that when skipping tests using Set-ItResult like this

$sb = {
    Describe 'Test names with templates using ForEach' {
        It 'Displays array values when Set-ItResult -Skipped <_>' -ForEach @('foo', 'bar', 'test') {
            Set-ItResult -Skipped
            $_ | Should -Not -BeNullOrEmpty
        }

        It 'Displays hashtable values when Set-ItResult -Skipped <Name> <Value>' -ForEach @(
            @{ Name = 'foo'; Value = 'test 1' }
            @{ Name = 'bar'; Value = 'test 2' }
            @{ Name = 'foobar'; Value = 'test 3' }
        ) {
            Set-ItResult -Skipped
            $Name | Should -Not -BeNullOrEmpty
            $Value | Should -Not -BeNullOrEmpty
        }
    }
}

$c = New-PesterConfiguration
$c.Run.Container = New-PesterContainer -ScriptBlock $sb
$c.Output.Verbosity = 'Detailed'
Invoke-Pester -Configuration $c

template names are expanded successfully

Pester v5.5.0

Starting discovery in 1 files.
Discovery found 6 tests in 80ms.
Running tests.
Describing Test names with templates using ForEach
  [!] Displays array values when Set-ItResult -Skipped foo is skipped 45ms (41ms|4ms)
  [!] Displays array values when Set-ItResult -Skipped bar is skipped 4ms (2ms|2ms)
  [!] Displays array values when Set-ItResult -Skipped test is skipped 5ms (2ms|3ms)
  [!] Displays hashtable values when Set-ItResult -Skipped foo test 1 is skipped 45ms (43ms|2ms)       
  [!] Displays hashtable values when Set-ItResult -Skipped bar test 2 is skipped 4ms (2ms|2ms)
  [!] Displays hashtable values when Set-ItResult -Skipped foobar test 3 is skipped 14ms (12ms|2ms)    
Tests completed in 288ms
Tests Passed: 0, Failed: 0, Skipped: 6 NotRun: 0

I guess that is because the skipping is performed from inside the execution context. Is there a performance concern here, or could the same not be done when a test is skipped by using the It -Skip parameter?

[nohwnd]

We cannot know what will happen inside of the test execution. Setting the test result using the set-itresult command is a special type of test failure. It doesn’t really have to happen inside of the test body. It can happen in any method that’s been called, and even by any custom exception if it has the right shape. For that reason, we cannot do static analysis, which would anyway be more expensive than just expanding the test name and runing the test.

The second aspect of this is that we have to run the test anyway, because we have to run the code that preceeds the set-it result command.

[csandfeld]

Thank you for the insights @nohwnd, appreciate you taking the time to explain.

[GSDragoon]

As csandfeld alluded to, using Set-ItResult is a workaround. It's not as clean, but works.

Instead of -Skip on the It:

Describe 'Test names with templates using ForEach' {
  BeforeDiscovery {
    $skipCondition = ...
  }
  It 'Should display hashtable values, even when skipping <Name> <Value>' -Skip:$skipCondition -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ) {
    $Name | Should -Not -BeNullOrEmpty
    $Value | Should -Not -BeNullOrEmpty
  }
}

[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)

Use Set-ItResult inside the body of the test:

Describe 'Test names with templates using ForEach' {
  BeforeAll {
    $skipCondition = ...
  }
  It 'Should display hashtable values when skipping using Set-ItResult <Name> <Value>' -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ) {
    if ($skipCondition) {
      Set-ItResult -Skipped -Because 'Skipping this test'
    }
    else {
      $Name | Should -Not -BeNullOrEmpty
      $Value | Should -Not -BeNullOrEmpty
    }
  }r
}

[!] Should display hashtable values when skipping using Set-ItResult foo test 1 is skipped, because Skipping this test 6ms (2ms|4ms)
[!] Should display hashtable values when skipping using Set-ItResult bar test 2 is skipped, because Skipping this test 4ms (2ms|2ms)
[!] Should display hashtable values when skipping using Set-ItResult foobar test 3 is skipped, because Skipping this test 4ms (2ms|2ms)

The catch is that the variable(s) used in the skip condition inside the test body need to be in BeforeAll instead of of BeforeDiscovery. In my particular case, I would need to duplicate that logic as multiple tests rely on the skip condition. Some can use -Skip and some would use Set-ItResult.

[fflaten]

In my particular case, I would need to duplicate that logic as multiple tests rely on the skip condition.

You can pass the skipcondition-value from Discovery to Run-phase through -ForEach. Add it as a key in your testcases, or as ... -ForEach @{ skipcondition = $skipCondition } on a parent Context/Describe. Will be available as $skipCondition in the test.