Native invocation using ArgumentList

[JamesWTruher]

This adds a new behavior for native executables which uses ArgumentList as the command arguments for the native executable.

PR Summary

The experimental feature PSNativeCommandArgumentPassing when enabled will use the ArgumentList property of the StartProcessInfo object rather than our current mechanism of reconstructing a string when invoking a native executable.
$PSNativeCommandArgumentPassing is a new automatic variable which allows for runtime selection of behavior. $PSNativeCommandArgumentPassing may be set to either Legacy or Standard, Legacy is the historic behavior. The default when the experimental feature is enabled is the new Standard behavior.

This new behavior is a breaking change from current behavior. This may break scripts and automation which work-around the various issues when invoking native applications. Historically, quotes must be escaped and it is not possible to provide empty arguments to a native application. This PR attempts to address these issues.

new behaviors made available by this change

• literal or expandable strings with embedded quotes the quotes are now preserved

PS > $a = 'a" "b'
PS > $PSNativeCommandArgumentPassing = "Legacy"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS > $PSNativeCommandArgumentPassing = "Standard"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

• empty strings as arguments are now preserved:

PS>  $PSNativeCommandArgumentPassing = "Legacy"
PS> testexe -echoargs '' a b ''           
Arg 0 is <a>
Arg 1 is <b>
PS> $PSNativeCommandArgumentPassing = "Standard"
PS> testexe -echoargs '' a b ''           
Arg 0 is <>
Arg 1 is <a>
Arg 2 is <b>
Arg 3 is <>

the new behavior does not change invocations which look like this:

PS> $PSNativeCommandArgumentPassing = "Legacy"                                            
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect          
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>
PS> $PSNativeCommandArgumentPassing = "Standard"                                  
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>

Additionally, parameter tracing is now provided so Trace-Command will provide useful information for debugging

PS> $PSNativeCommandArgumentPassing = "Legacy"                                           
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:19:53.6438 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:19:53.6440 ParameterBinding Information: 0 :     BIND argument [-echoargs a" "b a" "b "a b"]
DEBUG: 2021-02-01 17:19:53.6522 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS> $PSNativeCommandArgumentPassing = "Standard"                                            
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 :     BIND cmd line arg [-echoargs] to position [0]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [1]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [2]
DEBUG: 2021-02-01 17:20:01.9831 ParameterBinding Information: 0 :     BIND cmd line arg [a b] to position [3]
DEBUG: 2021-02-01 17:20:01.9908 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

PR Context

Invoking native applications is sometimes very problematic because of how the application is invoked. A new property is available on the ProcessStartInfo object which hopes to improve the experience. This PR takes advantage of this new property.

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
• Breaking changes
  • None
  • OR
  • Experimental feature(s) needed
    • Experimental feature name(s): PSNativeCommandArgumentPassing
• User-facing changes
  • Not Applicable
  • OR
  • Documentation needed
    • Issue filed: Call native operator #13068
• Testing - New and feature
  • N/A or can only be tested interactively
  • OR
  • Make sure you've added a new test if existing tests do not effectively test the code changed
• Tooling
  • I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
  • OR
  • I have considered the user experience from a tooling perspective and enumerated concerns in the summary. This may include:
    • Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode (which runs in a different PS Host).
    • Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
    • Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
    • Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

The impact on completions may be affected if those completers are calling native applications.

[iSazonov]

/cc @mklement0 Please look the PR.

[mklement0]

• While switching to .ArgumentList is definitely called for and fully solves the problem on Unix-like platforms, on Windows there are also vital accommodations that we need to make for common use cases, notably calling batch files and misexec-style CLIs, as detailed in this comment [update: see Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143].

  • If there's disagreement about these accommodations, let's have a discussion, but none has happened yet (except with other community members), so, yes, perhaps an RFC is necessary.

• Both improvements should be part of the same PR, to once and for all solve 100% of all argument-passing problems on Unix, and 99% of all problems on Windows (the remaining problems cannot be fixed generically, and must be addressed by users via --%).

• That is, only one (opt-in) breaking change should be made, and it should be controlled by a single preference variable (which is the next best solution if the breaking change is deemed unacceptable).

• As for the preference variable:

  • First, the name $PSNativeApplicationUsesArgumentList is too technical to begin with (end users who expect argument-passing in their shell to "just work" cannot be expected not know about plumbing details such as .Arguments vs. .ArgumentList, which would make discovery difficult); also, the variable should be [bool]-typed (while assigning 1 works through implicit conversion, we should show $true / $false in the docs).

  • Second, with the PR encompassing the Windows-only accommodations too, a broader name is called for. I'm struggling to come up with a good suggestion. $PSFixNativeArgumentPassing?

[SteveL-MSFT]

For the variable, I prefer $PSNativeCommandArgumentParsing as an enum with values ArgumentList, and String (can't think of something better...).

[mklement0]

@SteveL-MSFT:

I prefer $PSNativeCommandArgumentParsing

Not sure if it's just a typo, but to be clear: this isn't about argument parsing, it's about argument passing (which, on Windows only, also involves encoding, as a single string - which .ArgumentList will do for us, but the suggested accommodations will have to be coded manually).

an enum with values ArgumentList, and String (can't think of something better...).

I can see why you would want to avoid the word "fix" in the variable name - even though that's what it comes down to.

Again, what matters to the user is only the following - not technical characterizations based on plumbing users may not even know of:

• Is the old behavior in effect, where I need to apply workarounds?
• Or is the new behavior in effect, where things work as expected?

Thus, if you do want an enum, I suggest a $PSNativeCommandArgumentPassing variable with the following enumeration values, corresponding to the above:

• Legacy
• Standard (or similar)

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, we agreed on automatic variable $PSNativeCommandArgumentPassing as an enum with values Legacy and Standard (default). This feature needs to be an ExperimentalFeature.

[SteveL-MSFT]

LGTM

[TravisEz13]

@daxian-dbw Can you update your review?

[iSazonov]

@mklement0 Do you want to review the PR?

[TravisEz13]

@PoshChan Please remind me in 1 hour

[PoshChan]

@TravisEz13, this is the reminder you requested 1 hour ago

[iSazonov]

Since this PR introduces a breaking change anyway and is being implemented as an experimental feature, I would like to understand why don't we implement mklement0's proposal in full as an experimental feature and close the long history?
See:

• Native invocation using ArgumentList #14692 (comment)
• Arguments for external executables aren't correctly escaped #1995 (comment)
• Arguments for external executables aren't correctly escaped #1995 (comment)

Also I asked previously and still wonder why @mklement0 is not in a PowerShell Committee. I believe MSFT team would have saved a lot of time and effort in this case.

[TravisEz13]

@iSazonov can you open a new issue w.r.t. #14692 (comment) and mark it for committee review?

[TravisEz13]

@PoshChan Please remind me in 1 Hour

[mklement0]

The (hopefully) final proposal is now in #14747 (comment)

[PoshChan]

@TravisEz13, this is the reminder you requested 1 Hour ago

[ghost]

🎉v7.2.0-preview.5 has been released which incorporates this pull request.:tada:

Handy links:

• Release Notes