Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pester throws an exception when run inside runspace #2383

Closed
3 tasks done
kborowinski opened this issue Jul 24, 2023 · 13 comments · Fixed by #2468
Closed
3 tasks done

Pester throws an exception when run inside runspace #2383

kborowinski opened this issue Jul 24, 2023 · 13 comments · Fixed by #2468
Labels

Comments

@kborowinski
Copy link
Contributor

kborowinski commented Jul 24, 2023

Checklist

What is the issue?

(This issue was moved from here)

Environment:

  1. PowerShell 7.3.6 on up-to-date Windows 10 22H2 x64
  2. Pester 5.5.0
  3. ThreadJob 2.0.3 and Microsoft.PowerShell.ThreadJob 2.1.0

Explanation:
I use Pester for advanced infrastructure testing. Due to the long execution time of some tests I need to run them in parallel using Start-ThreadJob cmdlet from ThreadJob module. The tests are independent of each other. Sporadically, during initialization, Pester throws an exception:
Cannot index into a null array.

pester_threadjob_error_0

The line 13998 in Pester.psm1 module where exceptions happens:

image

Expected Behavior

Pester should not throw when running in an runspace.

Steps To Reproduce

  1. Start new session:
    pwsh -nop

  2. Run following scriptblock:

$cpuCount = [Environment]::ProcessorCount

$splatStartThreadJob = @{
    ThrottleLimit = $cpuCount
    StreamingHost = $Host
    ErrorAction = 'Stop'
    InitializationScript = {Import-Module Pester -Force -ErrorAction Stop}
}

$threadJobs = 1..($cpuCount * 2) | ForEach-Object {
    Start-ThreadJob -Name $_ -ScriptBlock {
        $pc = New-PesterConfiguration
        $pc.Run.Container = New-PesterContainer -ScriptBlock {Describe 'd' { It 'i' { 1 | Should -Be 1 }}}
        $pc.Output.Verbosity = 'None'
        $pc.Run.PassThru = $true
        $pc.Run.SkipRemainingOnFailure ='Block'
        [PSCustomObject]@{
            Test    = $Using:_
            Result  = Invoke-Pester -Configuration $pc
        }
    } @splatStartThreadJob
}

Wait-Job -Job $threadJobs | Receive-Job
  1. If there was no error on first try, then start new PowerShell session, do not use existing session

Output:
image

Describe your environment

Pester version : 5.5.0 C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1
PowerShell version : 7.3.6
OS version : Microsoft Windows NT 10.0.19045.0

Possible Solution?

Do not use the -StreamingHost parameter in Start-ThreadJob when calling Invoke-Pester

OR:

Replace line 13998 in Pester.psm1 with the following:

 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {
@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 24, 2023

BTW, just for fun, repro code refactored to use ForEach-Object -Parallel works:

$cpuCount = [Environment]::ProcessorCount
$results = 1..($cpuCount * 2) | ForEach-Object -Parallel {
    Import-Module Pester -Force -ErrorAction Stop
    $pc = New-PesterConfiguration
    $pc.Run.Container = New-PesterContainer -ScriptBlock {Describe 'd' { It 'i' { 1 | Should -Be 1 }}}
    $pc.Output.Verbosity = 'None'
    $pc.Run.PassThru = $true
    $pc.Run.SkipRemainingOnFailure ='Block'
    [PSCustomObject]@{
        Test    = $_
        Result  = Invoke-Pester -Configuration $pc
    }
} -ThrottleLimit $cpuCount
$results

Output:
image

The issue is - this is for PowerShell 7+ only. I need parallelization on PowerShell 5.1. Also, @PaulHigin it looks like the $Host.UI.psobject.Properties behavior is ThreadJob bug or limitation?

@fflaten
Copy link
Collaborator

fflaten commented Jul 24, 2023

Did you check if UI is null? psobject.properties might work silently because of member enumeration. Not in front of a computer today so can't test myself

Does it work without -StreamingHost?

@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 24, 2023

It does work without StreamingHost 👀:

image

$Host.UI is not null but $Host.UI.psobject.Properties are:

image

image

This has to be ThreadJob bug or limitation that I didn't know about 🤔

@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 24, 2023

This is very strange:

  1. I've changed the Pester.psm1 module to wait for debugger when $Host.UI.posobject.Properties are $null (and store the value in the $p variable):

pester

  1. Then I've attached to the process and entered the runspace for debugging:

runspace

And as you can see in the right window the $p variable is really $null but by the time the debugger was attached to the runspace the $Host.UI.psobject.Properties has all the values and is not null anymore 👀👀👀

@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 24, 2023

OK, got it! Finally! The $Host.UI is kind of lazy loaded! (possible race condition)

I have modified the Pester.psm1 once more with simple loop that waits until the $Host.UI.psobject.Properties are available:

image

And it works now - this is the output of the repro script now:

lazy

Really strange behavior. I would love if @SeeminglyScience could have a look at it for some thorough explanation.

@kborowinski
Copy link
Contributor Author

@fflaten I was able to chat with @SeeminglyScience yesterday on Discord, and for him it looks like race condition bug (I hope he doesn't mind me posting the conversation screenshot here):

image

I will investigate some more if that's PowerShell or ThreadJob module bug or something else. In the meantime, I'll not use StreamingHost in my parallel testing module since, in that case, I do not need it anyways.

@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 25, 2023

This is yet simplest repro case (the number of jobs has to be quite high though to get the repro):

1..500 | % {Start-ThreadJob {1 | Should -BeExactly 1; $supportsVT = $Host.UI.psobject.Properties['SupportsVirtualTerminal'].Value} -StreamingHost $Host -InitializationScript {Import-Module Pester -Force}} | Wait-Job | Receive-Job | Remove-Job -Force

Output:
image

Error:
image

@fflaten
Copy link
Collaborator

fflaten commented Jul 25, 2023

Thank you for the thorough troubleshooting! This is a weird one and a race-condition sounds right.

I was able to reproduce a couple of times then suddenly not, so wasn't able to see if we could make the repro simpler. I doubt module import and calling Should is relevant, so was hoping to reproduce with sleep in scriptblock and/or initializationscript ++.

Either way I support the suggested fix with a few conditions:

  • Bug report is created in repo for ThreadJob or PowerShell
  • Issue is referenced in comment above the fix

Reason being that checking psobject.properties and not just $host.UI is unexpected and would likely be removed in a future cleanup/refactor PR + hopefully they're able to fix the root cause which might help others as well 🙂

Do you want to provide the PR once the bug report is created by you or Patrick?

@kborowinski
Copy link
Contributor Author

kborowinski commented Jul 25, 2023

I was able to reproduce the issue without InitializationScript but I still needed to call Should. I was unable to reproduce the issue otherwise.

I also think that since this is race condition then checking only psobject.Properties is not the right solution, because the exception could happen on $Host.UI for someone else, therefore I would refrain from implementing my current workaround.

I think that for the moment there are 2 options:

  1. I could suggest in the workaround section to not use StreamingHost when calling Pester from Start-ThreadJob
  2. PR with a full $Host.UI check:
 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {

What do you think @fflaten?

Ps. I will create an issue in the ThreadJob module repo though.

@kilasuit
Copy link

kilasuit commented Jul 26, 2023

@kborowinski - In your original testing when using Start-ThreadJob I see that you are using the older of the two ThreadJob modules. In my testing in 7.3.6 & 5.1 on Microsoft Windows 10.0.25905 - aka Win 11 23H2 via the Insiders Dev ring with the newer Microsoft.PowerShell.ThreadJob module v2.1.0 I did not get it throwing that error, using this to test 1..500 | % {Start-ThreadJob {1 | Should -BeExactly 1; $supportsVT = $Host.UI.psobject.Properties['SupportsVirtualTerminal'].Value} } | Wait-Job | Receive-Job

So this may be fixed for you using that module (which due to command ordering preferences is being autoloaded over the ThreadJob Module for me) so can you please test this out for me?

Also this wasn't tested using Windows Terminal & I am just about to retest there too.

Additional background info
ThreadJob (v2.0.3) as is what is currently included in 7.3.6 however in future (somepoint soon as I have a PR for this) Microsoft.PowerShell.ThreadJob (v2.1.0 - and is the successor to ThreadJob going forward - joys of module renames eh!!) will also be included. I also noted that you mention that you need this to work on PowerShell 5.1 as well as in 7.x.x - does testing with Microsoft.PowerShell.ThreadJob (v2.1.0) on both repro for you too?
FYI Paul Higin has recently retired so may not answer the question you posted to him above & also thanks for rasing this issue in the PowerShell Org ThreadJob Repo too though as above I think this possibly may have been fixed.

@kborowinski
Copy link
Contributor Author

@kilasuit Unfortunately the same issue occurs in Microsoft.PowerShell.ThreadJob v2.1.0

@fflaten fflaten added the Bug label Apr 8, 2024
@fflaten
Copy link
Collaborator

fflaten commented May 18, 2024

  1. PR with a full $Host.UI check:
 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {

I'm fine with this workaround incl. a comment with link to this issue since I doubt we'll be able to cover this with a test consistently.

Thanks for really troubleshooting this and sorry for the delayed response 🙂 . Would you still like to provide a PR?

@kborowinski
Copy link
Contributor Author

No problem, I will provide a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants