New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143
Comments
I'm delighted that @JamesWTruher implemented #14692! I hope we will not stop half way and finally close this story by fully implementing the proposal. |
As it turns out, there's another accommodation worth making for batch files with respect to exit codes (now implemented in v1.3.1 of the
# Create a (temporary) batch file that provokes an error with an unsupported whoami.exe option,
# and exits with `exit /b` *without an explicit argument* with the intent to *pass whoami.exe's exit code through*.
'@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd
# Invoke the batch file and report the exit code.
PS> .\test.cmd; $LASTEXITCODE
0 # !! whoami.exe's exit code, 1, was NOT passed through.
Note that inside a To be clear: The problem lies with |
The following:
|
@SteveL-MSFT, on a meta note similar to the one in #14025 (comment): Two separate requests to discuss the issue at hand in the April community call were made (one by @iSazonov, referring here directly, and by @JustinGrote, referring here indirectly, via #1995 (comment)). In the community call you glossed over these requests by saying that @JamesWTruher's presentation had already covered the topic by his presentation on the new, experimental This despite the fact that the very point of the issue at hand is to point out problems with the feature as currently implemented - and that is what needed to be discussed. That the discussion was simply brushed aside again does not instill confidence in the concerns of the community being taken seriously. |
On Windows, there is basic inconsistency between the way a batch file arguments and Windows executable arguments are parsed. with a batch file, the
from PowerShell, the experience is as follows:
I didn't try a VB script. |
Thanks for engaging, @JamesWTruher.
It's not improper. It's an invaluable accommodation that relieves users of (most of) the burden of having to account for the anarchy of argument-passing on Windows. Ensuring that arguments are ultimately passed as-is - based on PowerShell's parsing rules alone - to external programs as-is is a core duty of a shell. While the limitations of Windows prevent a fully robust solution, we can provide a solution that covers most use cases, based on straightforward rules - and the ones laid out for the class of batch files (as opposed to hard-coding exceptions for specific executables) achieve that.
No. passing something like
Good point - I hadn't considered VBScript - I'll investigate. |
Quick update, @JamesWTruher: |
As for VBScript (WSH): VBScript's command-line argument parsing is provided by the WSH (Windows Script Host) CLIs,
WSH supports neither WSH parses a command-line token such as By contrast, To demonstrate the difference, assuming the following ' Save with Windows-1252 encoding for the guillemets («») to render properly.
Wscript.Echo CStr(WScript.Arguments.Count) + " argument(s) received:" + vbLf
i = 0
for each arg in WScript.Arguments
i = i + 1
WScript.Echo " «" + arg + "»"
next
WScript.Echo
PS> cscript.exe .\test.vbs 'Andre "The Hawk" Dawson' 'another argument'
3 argument(s) received:
«Andre \The»
«Hawk\ Dawson»
«another argument»
Note the broken argument partitioning and the literally retained
PS> ie cscript.exe .\test.vbs 'Andre "The Hawk" Dawson' 'another argument'
2 argument(s) received:
«Andre The Hawk Dawson»
«another argument»
While the behavior is partially broken - what were meant to be embedded |
@JamesWTruher, with respect to batch files:
Specifically, unquoted PS> dbea -UseBatchFile 'a b', 'a,b', 'a;b', 'a=b'
7 argument(s) received (enclosed in «...» for delineation):
«"a b"»
«a»
«b»
«a»
«b»
«a»
«b»
Command line (without executable; the value of %*):
"a b" a,b a;b a=b Note how PowerShell implicitly double-quoted With the proposed Accommodation B (see above), even such space-less arguments will be double-quoted in batch-file calls, to ensure that with the exception of # Call via the `ie` function, to activate the accommodations:
PS> dbea -ie -UseBatchFile 'a b', 'a,b', 'a;b', 'a=b' 'a&b' 'a|b' 'a<b', 'a>b' 'a^b'
10 argument(s) received (enclosed in «...» for delineation):
«"a b"»
«"a,b"»
«"a;b"»
«a»
«b»
«"a&b"»
«"a|b"»
«"a<b"»
«"a>b"»
«"a^b"»
Command line (without executable; the value of %*):
"a b" "a,b" "a;b" a=b "a&b" "a|b" "a<b" "a>b" "a^b" Note that arguments The reason for the Fortunately, passing |
The initial post details the proposed accommodations and their benefits and this follow-up comment provides examples of what we would gain. Perhaps it is helpful to complement that with examples of what won't work, unless these accommodations are implemented:
The examples show the current behavior with Unsupported Scenario A: Inability to pass arguments with embedded "@echo off`necho [%1]`n" > temp.cmd; .\temp.cmd 'Luke "Aches & Pains" Appling'; Remove-Item temp.cmd
["Luke \"Aches]
The system cannot find the path specified. As you can see, not only was the argument not recognized as a whole, the command broke, because - due to the embedded Unsupported Scenario B: Inability to pass space-less arguments that contain Using Azure's # Note the '&count=10' part of the URL
PS> az.cmd rest --method get --url 'https://example.org/resources?api-version=2019-07-01&count=10'
<Azure-specific error message>
'count' is not recognized as an internal or external command,
operable program or batch file. Since the URL by definition contains no spaces, PowerShell passes it unquoted, so that the unquoted In other words: with direct invocation, it is impossible to pass a URL that contains Unsupported Scenario C: Inability to pass # FAILS due to invalid syntax.
# The fact that the CLI help dialog pops up implies that.
# (A syntactically correct call would result in a quiet no-op, due to use of /quiet)
PS> $dir='c:\program files\foo'; msiexec /quiet /i foo.msi INSTALLDIR=$dir Because PowerShell passes Unsupported Scenario D: Inability to reliably report a batch file's exit code: PS> '@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd; .\test.cmd; $LASTEXITCODE
0 That is, even though Unsupported Scenario E: Inability to call WSH (VBScript, JScript) scripts with arguments with embedded # To work around #15289, cscript.exe is explicitly used for invocation, but the behavior would be the same without it.
PS> ' WScript.Echo WScript.Arguments(0)' > temp.vbs; cscript .\temp.vbs 'Luke "Aches & Pains" Appling'; Remove-Item temp.vbs
Luke \Aches Note how the argument wasn't recognized as a whole, and a Unsupported Scenario F: Inability to call PS> cmd /c 'C:\Program Files\PowerShell\7\pwsh' -noprofile -c "'hi there'"
'C:\Program' is not recognized as an internal or external command, operable program or batch file. |
I don't know if the behavior of scenario F should be changed. This is simply the behavior of cmd.exe. To make it work just use
This form of the command works in Windows Powershell 5.1, it works in the current powershell preview with I'm just not a huge fan of detecting a specific exe file and behaving differently. Detecting a specific file type ( |
Accommodation F is definitely the least important accommodation, and the failure is unequivocally But I still think it's worth doing:
By contrast, the
I'm not a fan of that either, and I wish we didn't have to do it (a kingdom for Unix-style argument passing!), but if we want to be a predictable shell on Windows that doesn't constantly and in perpetuity frustrate users with quoting headaches we have no other choice. The accommodations above, which our previous conversations helped shape, relate exclusively to:
These rules are:
|
P.S., @TSlivede: Special-casing
PS> & { $PSNativeCommandArgumentPassing='Legacy'; cmd /c "echo Honey, I'm `"$HOME`"." }
Honey, I'm "C:\Users\jdoe". |
I agree, that
and making Alternatives to
An environment variable could in some cases be missing, so I would want to avoid that.
|
I don't feel strongly about Accommodation F, given that not all
I previously suggested offering such a command in addition to the proposed accommodations (possibly without F), namely as a cmdlet I've called E.g.: # `ins` is the alias of `Invoke-NativeShell`
PS> ins "echo Honey, I'm `"$HOME`"."
Honey, I'm "C:\Users\jdoe".
|
You're confirming @KirkMunro's point. The sad thing about this decision is that in this case there is no need to choose between serving favored groups and quality-of-life improvements for the community at large. Introducing another setting will just create more confusion while providing no benefit at all; on the contrary. Only two settings are needed:
What you call "magic" is the very opposite from the user's perspective: It enables users to focus on PowerShell's syntax alone, trusting the shell to pass arguments on as specified solely by its own rules - that is a core mandate of a shell. That the act of passing arguments to external programs requires additional work behind the scenes is (a) unfortunate historical baggage on Windows that cannot be avoided and (b) should be an implementation detail that the user is shielded from. By not providing a single setting that combines In the absence of a proper solution, the As part of my ongoing withdrawal (in both senses of the word): I've said all that I have to say, and I'm unsubscribing from this thread. |
to reflect the decision re PowerShell/PowerShell#15143 and the continued need for ie on Windows.
I don't quite get this. There was a lot of resistance to "fixing" argument passing by "special casing" this feature for certain executables. But in the end, the team opts to "special case" the feature for these executables anyway - except not to fix the issues but drop back to the old (legacy) parsing mode. That's a head scratcher. This new approach feels more like a "Hybrid" mode where some executables get the old parsing treatment. Is that list configurable? What if we discover another native Windows app that doesn't behave well with Standard mode? Seems like this feature should stay experimental until the issues with cmd.exe, msiexec and c/wscript can be worked out. BTW @mklement0, thanks for the One last note, the recent loss of community support should be concerning to the team. Something is clearly not working right. Don't get me wrong, I don't think the team should merge community PRs willy / nilly. There needs to be a high quality bar to ensure new PowerShell releases are high quality and avoid breaking changes as much as possible. But @mklement0 has made some in-depth and well researched suggestions here that appear to be falling on deaf ears. :-( |
The saga continues at #15408 (comment) |
Whatever you do, please don't special case cmd implicitly (e.g. by sniffing the command name or its file extension). I often |
Pretty sure doing that was one of the main premises @JamesWTruher was working from. I'd agree that's probably going to cause more confusion than it helps ultimately, though. 🤔 |
There are two things I can think of which might inform some of this, but I yet haven't fully explored...
(I've always used Other things I can think of, which may or may not be relevant...
|
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes. |
Summary of the new feature/enhancement
A Guide to this Proposal:
The rest of this initial post details the motivation and the proposed implementation.
Easy-to-grasp examples of the proposed accommodations are in this comment.
Complementary examples of what won't work unless we implement these accommodations are in another comment.
As requested by @TravisEz13 in #14692 (comment), following a suggestion from @iSazonov in #14692 (comment):
The following is adapted from #14747 (comment), which contains some additional information about native argument-passing on Windows.
PR #14692 introduces experimental feature
PSNativeCommandArgumentPassing
that will address parameter-passing woes when calling native programs with respect to embedded quoting and empty-string arguments, taking advantage ofSystem.Diagnostics.ProcessStartInfo.ArgumentList
, which:on Unix-like platforms: fully solves all problems.
on Windows: solves the problem only for those programs that adhere to the quoting and escaping conventions used by Microsoft's C/C++ runtime.
While this is a great step in the right direction, it leaves out many Windows CLIs that do not play by these rules:
The most prominent exception is
cmd.exe
- and therefore calls to batch files: they accept only""
as an escaped"
, not the\"
required by the C/C++ convention); while Microsoft compiler-generated executables also support""
, there are third-party programs that support only\"
)An additional problem is that batch files unfortunately and inappropriately parse their arguments as if they had been passed from inside
cmd.exe
, which causes something like.\foo.cmd http://example.org?foo&bar
to break due to&
being misinterpreted as a statement separator. Using"http://example.org?foo&bar"
, i.e. quoting from PowerShell doesn't help, because PowerShell - justifiably - omits the quotes when it rebuilds the process command line behind the scenes, given that value contains neither spaces nor embedded"
chars.az.cmd
for Azure, and the wrapper batch files thatnpm
(Node.js's package manager) creates for (Java)script-based utilities that come with packages) use batch files as their CLI entry points, so that something likeaz ... 'http://example.org?foo&bar'
predictably fails.Calling
cmd.exe /c "<command-line>"
orcmd.exe /k "<command-line>"
directly with a single-argument command line to be executed through a happy accident actually currently works as intended, without a workaround - and that behavior must be retained.Many programs are particular about partial quoting of arguments, notably
msiexec.exe
with property arguments such asPROP="VALUE WITH SPACES"
; purely syntactically,"PROP=VALUE WITH SPACES"
(which is what PowerShell currently sends) should be equivalent (and if you let the C/C++ runtime / CLR parse it, is - the resulting verbatim string isPROP=VALUE WITH SPACES
in both cases), but in practice it is not.PROP=$someValuePossiblyWithSpaces
), and users generally shouldn't have to worry about such intricacies - see below.Finally, calls to the WSH (Windows Script Host) CLIs
cscript.exe
(console) andwscript.exe
- either directly or via associated script file types, notably.vbs
(VBScript) and.js
(JScript), behave poorly with\"
-escaped embedded"
characters; while the problem cannot be fully solved,""
-escaping results in better behavior: see below for details.It's impossible for PowerShell to fully solve these problems, but it makes sense to make accommodations for these exceptions, as long as they are based on general rules (rather than individual exceptions) that are easy to conceptualize and document.
I believe it is vital to make these accommodations as part of the
PSNativeCommandArgumentPassing
experimental feature implemented in PR #14692 in order to solve the vast majority of quoting headaches once and for all.They are detailed below.
For the remaining, edge cases there is:
--%
for console applications, or, preferably, because it has fewer limitations and enables use of PowerShell variable values and expressions via string interpolation,cmd /c "<cmd.exe command line>"
.Start-Process
for GUI-subsystem applications with a CLI such asmsiexec
, which allows you to fully control the process command line by passing a single string to-ArgumentList
(in a pinch you can also use it with console applications, but you lose stream integration).Proposed technical implementation details
After PowerShell's own parsing, once the array of verbatim arguments - stripped of
$null
s - to pass on is available:On Unix-like platform:
.ArgumentList
- that is all that is ever needed.On Windows:
Except for the cases detailed below, also pass that array to
.ArgumentList
- behind the scenes; .NET then performs the necessary re-encoding based on the C/C++ conventions for us, and any conventional CLI should interpret the result correctly.The following exceptions may apply independently or in combination, and they require manual re-encoding by PowerShell (with assignment to
.Arguments
, as currently):A current behavior that must be retained - i.e. no escaping of embedded
"
must be performed - is the very specific case ofcmd.exe
being called directly, with either the/c
or the/k
option followed by a single argument (with spaces) representing acmd.exe
command line in full./c
or/k
as multiple arguments, by transforming it into a single-argument, double-quoted-overall form.If the target command is a batch file:
""
(rather than\"
) to escape embedded verbatim"
(and ensure enclosure in syntactic"..."
, even if the value has no spaces)"..."
-enclose any argument that contain no spaces (such arguments are normally not quoted) but contain any of the followingcmd.exe
metacharacters:& | < > ^ , ;
(while,
and;
have no impact on arguments pass-through with%*
, they serve as argument separators in intra-batch file argument parsing; this also applies to=
, but, unfortunately, passing something likeFOO=bar
as"FOO=bar"
conflicts with the accommodation formsiexec
-style CLIs below).cmd /c "<batch-file> ... & exit"
rather than directly; see below for the detailed rationale.cmd.exe
the executable./c "<batch-file> ... & exit"
, where<batch-file>
path may need to be double-quoted and...
represents the space-joined list of the arguments quoted based on the rules above. Again, no escaping of any"
characters ending up in the overall"..."
string passed to/c
need or must be performed.Irrespective of the target executable, if any of the arguments have the form of a
misexec
-style partial-quoting argument, apply double-quoting only to the "value" part (the part after the separator):^([/-]\w+[=:]|\w+=)(.+)$
, and (b) the part after=
or:
requires double-quoting (either due to containing spaces and/or an embedded"
and/or, in the case of a batch file containingcmd.exe
metacharacters), leave the part up to and including=
or:
unquoted, and double-quote only the remaining part.FOO='bar none'
,-foo:$value
(with$value
containing verbatimbar none
),/foo:bar` none
(and even quoted-in-full variants'FOO=bar none'
, ....).Arguments
command line as follows:FOO="bar none"
,-foo:"bar none"
,/foo:"bar none"
If the target executable is a WSH CLI -
cscript.exe
orwscript.exe
- or the filename extension is one of the following WSH-associated extensions listed by default in$env:PATHEXT
(which makes them directly executable):.vbs .vbe .js .jse .wsf .wsh
, use""
rather than\"
to escape"
characters embedded in arguments; again see below for details.Again, these are reasonable accommodations to make, which:
I invite everyone to scrutinize these accommodations to see if they're complete, overzealous, ...
This is a chance to finally cure all native quoting / argument-passing headaches - even if only by opt-in.
(To experiment with the proposed behaviors up front (based on my personal implementation that sits on top of the current behavior), you can use
Install-Module Native
and prependie
to command lines; if the proposed changes are implemented, such a stopgap will no longer be necessary, although it can still help on earlier versions.)The text was updated successfully, but these errors were encountered: