Default and OEM character encodings in the Core edition should be Windows-1252, not ISO-8859-1

[mklement0]

ISO-8859-1 is currently (alpha16) the default character encoding, as well as when explicit encoding specifiers Default and OEM are used - see here.

This choice is problematic, because ISO-8859-1 is a subset of the commonly used Windows-1252 encoding.
(The two encodings are often conflated, but they are not the same.)

Specifically, using ISO-8859-1 makes the following characters - the printable characters in the codepoint range 0x80 - 0x9F - unavailable:

€ ‚ ƒ „ … † ‡ ˆ ‰ Š ‹ Œ Ž ‘ ’ “ ” • – — ˜ ™ š › œ ž Ÿ

Note that the € character is part of that list.

You can verify the problematic behavior as follows:

> '€' | Set-Content tmp.txt; Get-Content tmp.txt
?

Because € cannot be represented in ISO-8859-1, it was quietly converted to a literal ?.

Contrast this with use of Windows-1252:

> $enc1252 = [System.Text.CodePagesEncodingProvider]::Instance.GetEncoding(1252); [IO.File]::WriteAllText('tmp.txt', '€', $enc1252); [IO.File]::ReadAllText('tmp.txt', $enc1252)
€

The € char. - codepoint 0x80 in Windows-1252 (but not ISO-8859-1) - was correctly preserved.

---

Also, please note that in order to fully emulate Windows PowerShell behavior, using a fixed encoding in Core is not sufficient.

Instead, the encoding would have to be locale-dependent, as on Windows:
Unix locales would have to be mapped to the Windows legacy codepages - see here.

[SteveL-MSFT]

Looks like the Content cmdlets used ISO-8859-1 as default to align with HTTP1.1, but HTML5 now uses Windows-1252 as equivalent due to mislabeling of sites using ISO-8859-1. Seems like using Windows-1252 would be ideal.

[SteveL-MSFT]

Unfortunately just changing

PowerShell/src/System.Management.Automation/utils/ClrFacade.cs

Line 385 in fc30ae1

s_defaultEncoding = Encoding.GetEncoding(28591);

causes PowerShell to stop working

[mklement0]

@SteveL-MSFT

Thanks for that. To make it even more explicit:

Mistaking what was actually Windows-1252 for ISO-8559-1 became so commonplace that the HTML5 specification, which links to this page about encoding, decided to treat label "iso-8859-1" as an alias for windows-1252 (the link lists all chars. with the high bit set, i.e., starting at 0x80), which is also reflected in this living WHATWG document.

[mklement0]

@SteveL-MSFT: I suspect that the reason it breaks is that the Windows-1252 code page is not available by default in Core, but you can load it via the System.Text.Encoding.CodePages NuGet package, as demonstrated in this Stack Overflow answer.

Curiously, from PowerShell Core itself, that package (assembly) is available by default, which is what the PowerShell snippet in the initial post takes advantage of ([System.Text.CodePagesEncodingProvider]::Instance.GetEncoding(1252)).

[SteveL-MSFT]

@mklement0 thanks!

[iSazonov]

@SteveL-MSFT

Looks like the Content cmdlets used ISO-8859-1 as default to align with HTTP1.1, but HTML5 now uses Windows-1252 as equivalent due to mislabeling of sites using ISO-8859-1. Seems like using Windows-1252 would be ideal.

Defaults for Content cmdlets and Web cmdlets is different things. I believe the issue is for Content cmdlets and it is duplicate #3248. Default for Web Cmdlet I suggest better discuss in #3267.