ForEach-Object -Parallel / Start-ThreadJob don't honor unsilencing of silent-by-default streams via common parameters

[JohnLBevan]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

This is a new copy of #13816. That was closed due to inactivity, though the issue is still valid, so opening a new issue per this thread

---

Inside a ForEach-Object -Parallel / Start-ThreadJob script block, passing common parameters -Verbose, -Debug, -InformationAction Continue, does not make the respective stream output visible to the caller / -WhatIf is not honored.

Workaround: set the corresponding preference variables in the caller's scope.

Steps to reproduce

# Note: The same applies to Start-ThreadJob
1 | ForEach-Object -Parallel {
  Write-Host "host"
  Write-Output "Output"
  Write-Error "error"       
  Write-Warning "Warning"
  Write-Debug "debug" -Debug
  Write-Verbose "verbose" -Verbose
  Write-Information "information" -InformationAction Continue
}

Expected behavior

All streams should produce output, as would be the case with ForEach-Object without -Parallel as well as with Start-Job:

host
Output
Write-Error: error
WARNING: Warning
DEBUG: debug
VERBOSE: verbose
information

Actual behavior

The verbose, debug, and information streams unexpectedly produce no output

host
Output
Write-Error: error
WARNING: Warning

Error details

n/a

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

[jhoneill]

This is expected.

Each thread has its own runspace

When "stuff" comes back from the runspaces the receiver (foreach -parallel in this case) sees items going to the different channels, so "the runspace wrote this to the error channel, it wrote that to the information channel" and decides "for an error I do this / for information I do that" It's more like something running as full job in its own process than like the normal process block of a foreach. The runspace doesn't have a console to write to, so continue / silentlycontinue don't change much.

[JohnLBevan]

Thanks @jhoneill; that makes sense on reflection, & confirmed the behaviour we see exactly matches:

That said, should this be the desired behaviour, as it feels like a gotcha. Looking at those on the original thread several very experienced PowerShell folk (active here & on SO for PWSH posts) have fallen into this trap, and simply adding a switch doesn't immediately make you think "this is creating a new runspace, therefore output streams and action preferences will now behave differently".

I don't have a strong opinion here as I realise such things are subjective / now when I inevitably fall into this trap again I'll probably recall your comments and pick it up; but I doubt I'd intuit it first time. Will leave this issue open for a bit longer incase others in the community have strong opinions; otherwise we can close as "working as intended".

[jhoneill]

@JohnLBevan - No, I don't think it is what most people would wish and yes it is a gotcha.

I've did my own equivalent for Windows PowerShell about 10 years ago, and I'm still using it with a client who won't move off in-the-box PowerShell to downloaded PowerShell , and there and with thread jobs its clear that we're starting things letting them run asynchronously on their own thread and when they complete they all get pulled back together.

There was (IIRC) some debate about should this be a new cmdlet or should it be discoverable extending foreach-object. The latter won, but it leads to some people (possibly experienced in other aspects of PowerShell) making wrong assumption that -parallel is making a script multi-threaded. Variables aren't thread safe, so having multiple threads [reliably] isn't practical, but we can have fan-out fan-back-in but you just add -parallel to all existing loops, because running a block with sufficient isolation to be safe, mean means cutting it off from from things you would like it to access.

[JohnLBevan]

That makes sense; thanks @jhoneill.

I guess a clean solution (though a change to how PowerShell currently works) would be for any new runtime to inherit the context of the caller - i.e. for the new instance to be started with the preference variables set from the caller.

That wouldn't help in the original example (i.e. where -Verbose is added within the new runtime), but would make some code more predictable; e.g. in the case of the below example... And that's likely to capture the scariest scenarios (e.g. where the WhatIf preference isn't being inheritted).

$verbosePreference = [System.Management.Automation.ActionPreference]::Continue
1 | ForEach-Object -Parallel {Write-Verbose 'hello'}

That said, this then begs the question "Which global variables should be set in the new instance; i.e. should PowerShell's own variables have some special status vs user defined global variables"... So a can of subjective worms...

Maybe the simple answer's for everyone to install CoPilot and have it point out to them when they make this mistake ;).

[jhoneill]

@JohnLBevan
Yes, and then again, no :-)

If runspaces worked more like scopes, then it would solve some of things that hinder some of the people some of the time.
But when assign things in a scope it makes a new one, when we change the properties of something we've inherited it changes the original without making a new one. So we're back with the problem of multiple threads changing the same non-thread safe variable. If we serialize and create a read-only duplicate, someone will complain that they can't call methods on the inherited objects, or that they aren't read write.

With the streams, things are written to the stream and there all streams are silent in the run space, but when what's written comes back to the caller the value there decides what happens - hopefully this makes that clearer

In the second case "SilentlyContinue" was sent to the information stream. Inside the runspace where it is running the host can't print it but everything going to that stream comes back to the caller, where the host can print it and has been told to do so.

You can put $whatIfPrefernence = $using:whatifPreference in a scriptblock , but it won't inherit. But what it says has no way of being captured and sent back (a common issue with what if)

Confirm becomes a problem because of the host not having its own console to read from or write to

It's 2 separate things really, one is the code being self contained and variables not going back and forth, the other is that code running "headless" . And like I said both those form traps which people don't automatically anticipate (hence they are gotchas) , but no one has found a good solution to get to something we'd like better .

[mklement0]

Thanks for reopening, @JohnLBevan.

From a UX perspective, there is no justification for the selectively different behavior for some streams.
I'm not familiar with the implementation, but I'm not sure there's a technical reason either, given that all streams can be relayed via an out-of-process runspace (though that involves serialization); e.g.:

# OK: All streams are surfaced, as well as the -WhatIf output.
Start-Job {
  Write-Host "host"
  Write-Output "Output"
  Write-Error "error"
  Write-Warning "Warning"
  Write-Debug "debug" -Debug
  Write-Verbose "verbose" -Verbose
  Write-Information "information" -InformationAction Continue
  $tempFile = New-TemporaryFile
  Remove-Item $tempFile -WhatIf
  Remove-Item $tempFile
} | Receive-Job -Wait -AutoRemoveJob

Note that Write-Information is special in that the information stream is apparently unconditionally written to - whether or not unsilencing has been request; this contrasts with Write-Verbose and Write-Debug, but the premise of the issue is the unexpected behavior when unsilencing is requested at the source (in the parallel runspace).

The following demonstrates that ForEach-Object -Parallel does see the information-stream output:

# !! No output.
% -parallel { Write-Information -InformationAction Continue hi } 
# Output, forced via *>&1
# Even works if you *omit* -InformationAction Continue
# That is, ForEach-Object -Parallel sees the stream output, and can choose whether or not to surface it.
% -parallel { Write-Information -InformationAction Continue hi } *>&1

---

The inverse problem - not discussed as such in the original issue - is whether and how to propagate common parameters such as -WhatIf and -Verbose applied to ForEach-Object -Parallel itself to parallel runspaces.

I guess a clean solution (though a change to how PowerShell currently works) would be for any new runtime to inherit the context of the caller - i.e. for the new instance to be started with the preference variables set from the caller.

Such a change - on an opt-in basis - is being considered, but the discussion has stalled:

• Provide option in ForEach-Object -parallel to transfer current runspace state #12240

Common parameters are currently not propagated (though those that operated on the caller side, such as -OutVariable, do work as expected).

While that may just be surprising in cases such as:

% -Parallel { Write-Verbose "Never prints." } -Verbose

... it is dangerous with -WhatIf and -Confirm:

# !! Unconditionally removes the file, because -WhatIf is in effect ignored.
# !! Ditto with -Confirm
% -Parallel { Remove-Item foo.txt } -WhatIf

ForEach-Object -Parallel runspaces are considered non-interactive, so if you tried to use -Confirm inside a parallel runspace, you'd (sensibly) get an error: "A command that prompts the user failed because the host program or the command type does not support user interaction"

There are two possible solutions for those common parameters that need to be set in each parallel runspace to take effect.:

• Propagate the caller-side common parameters to the parallel runspaces by setting equivalent preference variables in each parallel runspace: e.g., -WhatIf would translate to a runspace-local $WhatIfPreference = $true. In fact, this is the mechanism that is already used for propagating common parameters to advanced functions (in-runspace).

  • The backward-compatibility impact would have to be assessed.

• Prevent use of such common parameters, by emitting an error (note that this is appropriate for -Confirm either way)

In fact, such restrictions are already in place for most (non-stream-collecting) common parameters:

# !! Error: 
# !! "ForEach-Object: The following common parameters are not currently supported in the Parallel parameter set: 
# !! ErrorAction, WarningAction, InformationAction, PipelineVariable"
% -Parallel {   } -InformationAction Continue

Therefore, it sounds that, at least for now (given the wording of "currently"), the solution is to also add -WhatIf, -Confirm, -Verbose, -Debug (possibly others?) to the list of common parameters that trigger this error.
In short: prevent use of all common parameters that require being seen by each parallel runspace in order to have any effect.