Introduce a new cmdlet for calls to native (external) programs

[mklement0]

Summary of the new feature / enhancement

This supersedes the since-closed #18961, following a discussion with @jborean93 - see his summary.

Calls to native (external) programs are a recurring pain point:

• There's the longstanding problem of broken handling of embedded double quotes, which the $PSNativeCommandArgumentPassing preference variable is meant to address.

  • Even the new Standard mode cannot handle all edge cases, however, and sometimes direct control of the process command line is inevitable.

  • Also, passing a full command line to the platform-native (legacy) shell is currently non-obvious and requires knowledge of the respective shell's CLI syntax.

• There's the lack of integration with PowerShell's error handling, which the (still-experimental in v7.3.1) PSNativeCommandUseErrorActionPreference preference variable is meant to address.

  • Also, the ability to collect stderr output (which may or may not indicate actual error conditions) in memory is currently absent.

• There's the problem of individual CLIs on Windows using character encoding that isn't aligned with the current code page / [Console]::OutputEncoding, such as python (ANSI encoding) and node.exe (UTF-8).

While all these issues do have solutions, they are cumbersome, and especially ill-suited to per-call overrides of behavior configured via the preference variables.

A new cmdlet named, say, Invoke-NativeCommand (alias inc), could be introduced to ease that pain - see below.

Related issues:

• Make Start-Process support execution of full, no-shell command lines (on Windows) #14347
• Call native operator #13068 (comment)
• Add msbuild to NativeCommandProcessor (s_legacyCommands) in PWSH 7.3 #18660 (comment)
• $OutputEncoding and [Console]::InputEncoding are not aligned -Windows Only #14945

---

Proposed technical implementation details (optional)

Invoke-NativeCommand / inc should support the following:

• Fundamental Invocation forms with respect to passing the command to execute:

# With PowerShell's own syntax, via a script block,
# Note: Only useful if combined with parameters that modify the default behavior.
# Parameter -ScriptBlock  implied.
Invoke-NativeCommand { choice.exe /d y /t 0 /m '3" of snow?' } -ArgumentPassing Standard

# Passing a *no-shell* command line (-CommandLine implied), primarily useful on *Windows*
# Similar to Start-Process with a single -ArgumentList value, but including the executable.
# See also: https://github.com/PowerShell/PowerShell/issues/14347
# This allows full control over the process command line.
# On Unix, -UseShell could be implied.
# Parameter -CommandLine  implied.
Invoke-NativeCommand 'choice.exe /d y /t 0 /m "3\" of snow?"'

# Passing a *shell* command line (-CommandLine implied)
# Calls via `cmd /c` (or, better, via a *temporary batch file*) on Windows, and via `sh -c` on Unix.
Invoke-NativeCommand -UseShell 'ver && choice.exe /d y /t 0 /m "3\" of snow?" & echo ignore me 2>NUL'

• Behavior-modifying parameters:

  • -Encoding <encoding:

    • Ad-hoc override for $OutputEncoding and [Console]::OutputEncoding with the specified encoding.

  • -UseShell - see above.

    • Only meaningful with -CommandLine, optionally combined with -ArgumentList / remaining arguments.
    • As noted, this switch could be implied when using -CommandLine (a single string) on Unix-like platforms.

  • -ArgumentPassing <mode>

    • Only useful with -ScriptBlock
    • Ad-hoc override for $PSNativeCommandArgumentPassing, where <mode> may notably be Legacy or Standard

  • -IgnoreExitCode / -IgnoreExitCode:$false

    • Ad-hoc override for $PSNativeCommandUseErrorActionPreference
    • Useful for nonstandard CLIs that use nonzero exit codes (too) to communicate success, such as robocopy.exe
    • Alternative consideration: use the common -ErrorAction parameter to override $PSNativeCommandUseErrorActionPreference, with Continue ignoring the exit code, and Stop throwing a script-terminating error in case of a nonzero exit code; the question then becomes whether SilentlyContinue and Ignore should act differently from Continue and actually suppress stderr output (too), even though it is not the absence or presence of stderr output that constitutes an error (only the exit code does).

  • -StandardErrorVariable <var-name>

    • Analogous to the common -ErrorVariable parameter, but exclusively applied to stderr output.
    • Analogously to how -ErrorVariable works, stderr output would still be passed through; 2>$null would silence it.
    • Alternative consideration: use the common -ErrorVariable parameter to collect stderr output - even though stderr output isn't actually error output from PowerShell's perspective.

[Andrew74L]

• There's the longstanding problem of broken handling of embedded double quotes, which the $PSNativeCommandArgumentPassing preference variable is meant to address.

  • Even the new Standard mode cannot handle all edge cases, however, and sometimes direct control of the process command line is inevitable.
  • Also, passing a full command line to the platform-native (legacy) shell is currently non-obvious and requires knowledge of the respective shell's CLI syntax.

I don't really understand the technicalities of this issue, but would the situation be helped if this were possible:
Write-Output -LiteralString some.exe arg1 arg2 | Out-File $env:TEMP\native.cmd
So that this command could be run:
cmd /C $env:TEMP\native.cmd

If the arguments require variables, they could be 'fed' to cmd via 'HKCU\Software\Microsoft\Command Processor\AutoRun'.
Presumably some or all of this could be packaged in a function.

[yecril71pl]

1. I do not think inc is a good alias, as it normally means increment, as attested by Pascal and various assemblers. I would use native instead.
2. What about input encoding? It seems the following options are currently valid for text I/O: UTF–8 or (if tty then OEM else ANSI); however, there are a lot of narrow-character programs that blindly assume ANSI regardless of tty, which causes all sorts of problems.

[Andrew74L]

1. I do not think inc is a good alias, as it normally means increment, as attested by Pascal and various assemblers. I would use native instead.

I'd prefer aliases to be in the format v-n or v-nn. So i-nc in this case.

[yecril71pl]

I'd prefer aliases to be in the format v-n or v-nn. So i-nc in this case.

Aliases were designed to mimic usual OS commands. There are no OS commands in that style.

[mklement0]

@Andrew74L, you can directly call cmd.exe's CLI, via cmd /c; e.g. cmd /c 'ver & echo hi'(analogously, you can call the native shell on Unix-like platforms via /bin/sh -c) - no strict need for a batch file.

As such, if -UseShell did only that behind the scenes, we wouldn't gain all that much, but the functionality can be improved upon in a manner that does require a temporary batch file on Windows. The improvements, modeled on /bin/sh's features, include being able to pipe code to cmd.exe, submitting multiline commands, and submitting commands that can accept arguments (in other words: analogous to /bin/sh, you could then pass an in-memory batch file).

This is what the Invoke-NativeShell command from (my) Native module already does (though it uses /bin/bash rather than /bin/sh, due to the former's ubiquity - that is worth considering here too).

[mklement0]

As for the alias-name discussion:

Aliases were designed to mimic usual OS commands.

In my view, such aliases - named for a different shell's internal commands (e.g., dir) or a different platforms standard utilities (e.g. ls) - should never have been introduced, and we should certainly avoid introducing new ones.

They are to be avoided for two reasons: given PowerShell's fundamentally different syntax, the user expectation that something like dir works the way they're used to is frustrated in all but the simplest use cases. As for a different platform's utilities: the same argument applies there too, but, more importantly, now that PowerShell is cross-platform, we do not want to shadow standard utilities with PowerShell aliases; in fact, such aliases - e.g. ls - were removed from PowerShell Core precisely for that reason (albeit unfortunately only on Unix-like platforms).

I'd prefer aliases to be in the format v-n or v-nn. So i-nc in this case.

This leaves us with PowerShell-idiomatic aliases, such as gc for Get-Content: they do not contain -, and are composed of a standardized alias prefix associated with each approved verb (Run Get-Verb Invoke, for instance). The noun part - by definition open-ended - is not standardized.
Thus, the alias name must be punctuation-free, start with i, followed by at least one letter identifying the noun.

Any new aliases can conflict with existing utilities - the best we can do is avoid conflicts with standard utilities and well-known, widely used utilities.

I don't think inc is problematic, as the context is different. For instance, nobody seems to have a problem with gc, even though that name is used to refer to the garbage collector in .NET.

Of course, one option is not to ship with an alias at all and let users define their own, if desired.

[Andrew74L]

@Andrew74L, you can directly call cmd.exe's CLI, via cmd /c; e.g. cmd /c 'ver & echo hi'(analogously, you can call the native shell on Unix-like platforms via /bin/sh -c) - no strict need for a batch file.

@mklement0, yes, but I was referring to your comment in the initial post...

There's the longstanding problem of broken handling of embedded double quotes, which the $PSNativeCommandArgumentPassing preference variable is meant to address.

More on this below.

As such, if -UseShell did only that behind the scenes, we wouldn't gain all that much, but the functionality can be improved upon in a manner that does require a temporary batch file on Windows. The improvements, modeled on /bin/sh's features, include being able to pipe code to cmd.exe, submitting multiline commands, and submitting commands that can accept arguments (in other words: analogous to /bin/sh, you could then pass an in-memory batch file).

Interesting. Would that require any modifications to cmd.exe?

This is what the Invoke-NativeShell command from (my) Native module already does (though it uses /bin/bash rather than /bin/sh, due to the former's ubiquity - that is worth considering here too).

I tried this...

Invoke-NativeShell 'for /F "skip=1 tokens=2" %I in ('quser ^| find /V ">"') do @echo %I'
Access denied - >
WARNING: You are passing arguments to the -CommandLine argument, but the latter doesn't reference them. Did you mean to make them part of the -CommandLine string? E.g. `ins 'echo hi'` rather than `ins echo hi`
I was unexpected at this time.

Then this...

Invoke-NativeShell "for /F ""skip=1 tokens=2"" %%I in ('quser ^| find /V "">""') do @echo %%I"
2

Very good! Although passing the command line 'as is', obviously didn't work, and this also works...

cmd /C "for /F ""skip=1 tokens=2"" %I in ('quser ^| find /V "">""') do @echo %I"
2

Why isn't something like the following possible...

Write-Output -LiteralString for /F "skip=1 tokens=2" %I in ('quser ^| find /V ">"') do @echo %I | Out-File $env:TEMP\native.cmd
cmd /C $env:TEMP\native.cmd

Or is it?

[mklement0]

Your first attempt is a syntax error: you've neglected to escape the inner ' as ''.

The second attempt is correct and the doubling of % is necessary, because a batch file is used behind the scenes.
That interactively something like %i is required, but in batch files %%i is a prime example of what makes cmd.exe so ... uh ... special.

Leaving aside that Write-Output has no -LiteralString parameter and would be included verbatim in the output, and that that the unescaped ( and ) and @ wouldn't work as intended, your attempt to save to a batch file would fail for the same reason - you're using %i, not %%i.

Also note that you don't need cmd /c to invoke batch files - just invoke them directly.

In short: users will need to be aware that batch-file syntax is necessary, which affects for loops - which aren't typically submitted at the command prompt anyway - and the ability to escape % chars. as %% (something that can not be done at the command prompt, except with workarounds).

[Andrew74L]

Your first attempt is a syntax error: you've neglected to escape the inner ' as ''.

The second attempt is correct and the doubling of % is necessary, because a batch file is used behind the scenes. That interactively something like %i is required, but in batch files %%i is a prime example of what makes cmd.exe so ... uh ... special.

For the first attempt I wasn't 'trying'. The point was to show that Invoke-NativeShell can't be regarded as equivalent to sitting at a command prompt. Command lines still have to be 'tweaked'.

Leaving aside that Write-Output has no -LiteralString parameter and would be included verbatim in the output, and that that the unescaped ( and ) and @ wouldn't work as intended, your attempt to save to a batch file would fail for the same reason - you're using %i, not %%i.

Write-Output -LiteralString is a hypothetical switch on that cmdlet (perhaps that's not the right cmdlet to have chosen). Yes, I should have used %%i, not i%. The idea is that -LiteralString should work sort of like --%, and send everything that follows down the pipeline, verbatim. Am I right to suppose that that is impossible?

Also note that you don't need cmd /c to invoke batch files - just invoke them directly.

Ah, yes.

In short: users will need to be aware that batch-file syntax is necessary, which affects for loops - which aren't typically submitted at the command prompt anyway - and the ability to escape % chars. as %% (something that can not be done at the command prompt, except with workarounds).

Requiring the escaping of % chars. when used interactively, suggests that the cmdlet should be thought of as Invoke-Native_Script_. I associate 'shell' more with 'prompt' than 'script'. Aside from that pedantic note, I accept your points.

[mklement0]

You're passing a shell command line and that command line is a string, which means that you have to satisfy PowerShell's string-literal syntax rules; using a here-string can ease that pain:

Invoke-NativeShell @'
for /F "skip=1 tokens=2" %%I in ('quser ^| find  ">"') do @echo %%I
'@

You should never expect to be able to use a different shell's syntax as-is, unquoted in PowerShell.

Any attempt to make this happen inevitably comes with severe limitations and/or obscure behavior.

In fact, an attempt was made in v3+: --%, the stop-parsing token, but it precisely resulted in said problems, summarized in this Stack Overflow answer.

The desire for such an inherently impossible solution - as understandable as that desire may be - gave rise to #13068. My summary of why I think such a pursuit is ultimately misguided is in #13068 (comment) (also linked to from the initial post).

the cmdlet should be thought of as Invoke-Native_Script_.

I see your point, but (a) that a distinction needs to be made at all is a testament to cmd.exe's ... uh ... quirkiness and (b) it can - fortunately - be bypassed altogether here, given that we're talking about a command named

Invoke-NativeCommand

here, with a -UseShell switch - and all that is needed for the docs to make it clear that on Windows batch-file syntax applies.

Again, pragmatically speaking: For most made-for-cmd.exe command lines intended for interactive use, the distinction won't matter. For complex commands better suited to use from inside a batch file, chances are that (a) the source of such commands makes that clear, and (b) the docs for the - for now hypothetical - Invoke-NativeCommand will have to make it clear that -UseShell indeed implies batch-file syntax.

[Andrew74L]

You're passing a shell command line and that command line is a string, which means that you have to satisfy PowerShell's string-literal syntax rules; using a here-string can ease that pain:

Invoke-NativeShell @'
for /F "skip=1 tokens=2" %%I in ('quser ^| find  ">"') do @echo %%I
'@

For some reason I forgot about using a here-string. The problem remains as to how to get non-environmental variables into the string, as is the case with --%. To quote your SO post:

Other than %...% environment-variable references, you cannot embed any other dynamic elements in the command; that is, you cannot embed regular PowerShell variable references or expressions.

This is where I think HKEY_CURRENT_USER\Software\Microsoft\Command Processor\AutoRun would be useful. The proposed new command could use the following sort of logic to 'pipe' (using that term loosely) variables to cmd.exe

@'here-string'@ > $env:TEMP\native.txt
$hash = @{foo='bar';bar='foo'} # HT from command line
$setcmds = foreach ($_ in $hash.GetEnumerator()) {[string]::Concat('set ',$_.key,'=',$_.value)}
$autorun = [string]::Join('&', $SetCmds)
$original = Get-ItemPropertyValue 'HKCU:\Software\Microsoft\Command Processor\' -Name AutoRun
Set-ItemProperty 'HKCU:\Software\Microsoft\Command Processor\' -Name AutoRun -Value $autorun
# Run proposed command using native.txt content as arguments
Set-ItemProperty 'HKCU:\Software\Microsoft\Command Processor\' -Name AutoRun -Value $original

Some details are omitted but you get the idea.

[mklement0]

No reason to use environment variables - just use an expandable, i.e. double-quoted version of a [here-string] literal to embed PowerShell variable and/or expression values.

Invoke-NativeShell @"
  @echo "`$PSHOME is: $PSHOME; `$(2 + 2) equals $(2 + 2)"
"@

[Andrew74L]

That is more concise, but it also means mixing syntax and having to use escape characters.
At this stage I think I'd prefer that the here-string "does one thing well". I envisage a command something like this...

Invoke-NativeCommand -FilePath cmd -ArgumentsHereString @'/C echo %bar%%foo%'@ -Variables {foo='bar';bar='foo'}
foobar

[mklement0]

Pragmatically speaking, $ is rarely used verbatim in shell commands, so in the typical case it's just a matter of placing$var references in the string.

That said, Invoke-NativeShell does have an argument-passing mechanism, but it doesn't use environment variables it use the shell's own way of receiving arguments (which is one of the reasons why a batch file must be used behind the scenes, namely positional argument-passing that the in-memory batch file can reference as %1, %2, ... (or %* as a whole):

Invoke-NativeShell 'echo %1 %2' one two

This would translate to (-ArgumentList can only be used in combination with -UseShell) :

Invoke-NativeComand -UseShell 'echo %1 %2' -ArgumentList  one, two
# With -ArgumentList declared with ValueFromRemainingArguments
Invoke-NativeComand -UseShell 'echo %1 %2' one two

Tangentially related:

• There is no inline form of here-strings - thought it would be convenient:
  • See Introduce a new, single-line raw string literal (single-line here-string alternative) #13204
• Child-process-scoped environment variables would be convenient too, which not only shell command lines would benefit from (analogous to, say, FOO=bar BAZ=quux sh -c 'echo $FOO-$BAZ' in POSIX-compatible shells)
  • See Provide simple one-time setting of environment variable #3316

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.