Strange behavior in PS7 with replacing specific bytes in While-clause

[Northman-de]

For no other reason than to challenge myself, I tried to make one of my scripts to use as less bytes as possible.

I need to find all double closing brackets and separate them:

while($l.Contains("))")){$l=$l.Replace("))",")`t)")}
# ))  -> )`t)
# ))) -> )`t)`t)

To describe this here I use a TAB to have a visible character. In my code I pasted a [char]7 (that will never be in the input text).
This works in 5.1, 6.2 and 7.1

To make my code shorter I used this:

while($l-ne($l=$l.Replace("..",".`t"))){}

This also works in 5, 6 and 7 if you use a TAB (pasted, not `t).
With [char]7 it's OK in Powershell 5 and 6, but fails in 7.

Actual behavior in PS7

$code = '$l = "#####" ; while($l-ne($l=$l.Replace("##","#Q#"))){} ; $l.Split("Q") -join "-"'
1..31 | %{"{0:00} - {1}" -f $_, (Invoke-Expression $code.Replace("Q",[char]$_))}

 1 - #-##-##
 2 - #-##-##
 3 - #-##-##
 4 - #-##-##
 5 - #-##-##
 6 - #-##-##
 7 - #-##-##
 8 - #-##-##
 9 - #-#-#-#-#
10 - #-#-#-#-#
11 - #-#-#-#-#
12 - #-#-#-#-#
13 - #-#-#-#-#
14 - #-##-##
15 - #-##-##
16 - #-##-##
17 - #-##-##
18 - #-##-##
19 - #-##-##
20 - #-##-##
21 - #-##-##
22 - #-##-##
23 - #-##-##
24 - #-##-##
25 - #-##-##
26 - #-##-##
27 - #-##-##
28 - #-##-##
29 - #-##-##
30 - #-##-##
31 - #-##-##

The results of bytes 9 to 13 are as expected. In PS5 or PS6 all bytes result in the correct output.

Why does this behave different for specific values?

Environment data

Name                           Value
----                           -----
PSVersion                      7.1.2
PSEdition                      Core
GitCommitId                    7.1.2
OS                             Microsoft Windows 10.0.18363
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[mklement0]

The short of it:

• PowerShell is not to blame here: the change in behavior is the result of a change in the underlying .NET runtime.

• Whether that change was intentional or should be reported as a bug there, I don't know.

---

The problem boils down to this:

• With string operands, PowerShell's -eq operator (and its inverse, -ne), performs case-insensitive, culture-invariant string comparison (StringComparison.InvariantCultureIgnoreCase).

• As of .NET 5.0.3, the runtime underlying PowerShell 7.1.2, this method appears to ignore non-printing control characters, which explains your symptom. Earlier versions, including up to (at least) 3.1.2 (underlying PowerShell 7.0.5) and the .NET Framework version underlying Windows PowerShell do not.

For instance:

# Escape sequence "`a" is [char] 7
'Food' -eq "Foo`ad"   # !! $true in .NET 5+ - the non-printing control character is *ignored*

// C# equivalent
"Food".Equals("Foo\ad", StringComparison.InvariantCultureIgnoreCase)

[vexx32]

That's an interesting change to be made. Do you happen to know if that was documented anywhere? Seems like it should have been considered a breaking change and doc'd somewhere as a result.

[mklement0]

I agree, @vexx32 - a cursory web search didn't provide any clues.

Searching the issues at https://github.com/dotnet/runtime/issues I did find this, which may explain the change (I haven't dug deeper):

dotnet/runtime#43736 (comment)

in .NET 5.0 we have switched to using ICU instead of NLS. You can look at https://docs.microsoft.com/en-us/dotnet/standard/globalization-localization/globalization-icu for more details.

GitHub

dotnet/runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps. - dotnet/runtime

Globalization and ICU

Learn more about: .NET globalization and ICU

[daxian-dbw]

WG-Engine:

We reviewed this issue and agreed to document the behavior changes to -eq, -ne, -in, -notin, -contains, -notcontains, and even -gt, -ge, -lt, -le due to the breaking change in .NET 5+.

@sdwheeler Would it be OK to document this in https://docs.microsoft.com/en-us/powershell/scripting/whats-new/what-s-new-in-powershell-72?view=powershell-7.2#breaking-changes-and-improvements?

What's New in PowerShell 7.2 - PowerShell

New features and changes released in PowerShell 7.2

[daxian-dbw]

The document update PR is out: MicrosoftDocs/PowerShell-Docs#8672
Thanks @sdwheeler!