Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call native operator #13068

Closed
SteveL-MSFT opened this issue Jun 30, 2020 · 209 comments
Closed

Call native operator #13068

SteveL-MSFT opened this issue Jun 30, 2020 · 209 comments
Labels
Committee-Reviewed PS-Committee has reviewed this and made a decision Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif Issue-Enhancement the issue is more of a feature request than a bug Resolution-No Activity Issue has had no activity for 6 months or more
Projects

Comments

@SteveL-MSFT
Copy link
Member

SteveL-MSFT commented Jun 30, 2020

Problem Statement

Currently, there are cases where cutting and pasting a native command line fails to run as expected in PowerShell. This may be due to incorrect parsing of quotes meant to be passed to the native command or use of PowerShell syntax that is not meant to be interpreted as PowerShell. PowerShell has a --% special argument when used with native commands treats the rest of the arguments as literals passed to the native command, but has several issues:

  1. It is not discoverable, users need to know about this special parameter ahead of time
  2. |, &&, and || take precedence, so: wsl --% ls | less would execute wsl ls and pipe the results to less running in PowerShell rather than less running in wsl
  3. If you cut and paste a command line, you would need to edit the line to insert --% towards the beginning
  4. On Unix systems, the args after --% are passed verbatim w/o globbing where native commands on Unix expect the shell to perform globbing

Proposed technical implementation details

Proposal is to introduce a new --% (Call Native) operator.

Any text content after this operator will call into the "default shell" of the OS to execute. On Windows, this would be cmd.exe and on Unix-based systems this would be /bin/sh. This resolves the globbing issue on Unix-based systems, and also allow %variable% expansion on Windows. Unlike --% switch, this also means that |, &&, and || are treated as part of the native command line.

This means that these two are functionally the same:

wsl --% ls $foo `&`& echo $PWD
--% wsl ls $foo && echo $PWD

where $foo and $PWD is evaluated by the shell within WSL. Note that in the first example, you would have to know to escape && to have it execute within WSL instead of within PowerShell.

To pipe output from such execution back into PowerShell, the user is required to store the results into a variable first:

$out = --% ls *.txt
$out | select-string hello

Note that unlike the current & call operator, you cannot use any PowerShell syntax, so:

--% $commandline

would not resolve $commandline as a variable first by PowerShell, but instead pass $commandline to the default shell to process unresolved.

The cut & paste problem is solved by simply pasting after --% is typed.

The above example for wsl would look like:

--% wsl ls | less

where the intent is to have that whole line execute within the WSL Linux instance.

Discoverability

Users already familiar with --% as a switch may easily transition to using this new operator where it makes sense. For new users, --% is unique so that search engines find it easily related to PowerShell.

Alternate Considerations

&! and &n were proposed as the sigil, but there was push back because &! is a valid operator in some languages making a web search for documentation more difficult. There was also a concern whether visual similar to & call operator would be confusing to users.

There is question about supporting line-continuation when using this new operator. I would suggest that we do not support it initially.

A cmdlet solution instead of an operator solution was proposed. We believe this doesn't solve the "cut & paste" problem as now you need to know to put the pipeline in single quotes and/or escape special characters. We do believe a cmdlet as in Invoke-NativeCommand (noun to be determined) would be useful as an additional option instead of replacing the need for an operator.

Related issues

This should also solve these issues:

#12491
#1761

@SteveL-MSFT SteveL-MSFT added Issue-Enhancement the issue is more of a feature request than a bug Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif labels Jun 30, 2020
@SteveL-MSFT SteveL-MSFT added this to the 7.1-Consider milestone Jun 30, 2020
@SteveL-MSFT SteveL-MSFT added this to To do in Shell via automation Jun 30, 2020
@SteveL-MSFT SteveL-MSFT changed the title Native command mode Call native operator Jun 30, 2020
@essentialexch
Copy link

I like the idea and find it useful, but think it would be far more useful (I'm thinking specifically in the case of a DSL) if I can get a single string expansion. So if I've built a cmd.exe compatible string in my PS, I have a way to string verbatim to cmd.exe. Perhaps

&n -command $string

That would potentially also let me specify my interpreter:

&n -shell /bin/sh -command $string
or
&n -shell cmd.exe -command $string

in terms of the name/operator, is "&!" in use? That has similarity to shbang( "#!" ).

@SteveL-MSFT
Copy link
Member Author

$! is a good suggestion!

The shell can be specified by simply specifying it as the command:

&! /bin/sh -c blah

The $var expansion creates a problem in that $ may need to be literal to be passed to the native command/shell. One of the problems this tries to avoid is when people need to figure out how to properly escape everything. If you require variable expansion, you could always do:

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

However, in cases where you want to mix PowerShell with the native command, it's probably best to use the current syntax and just be aware of what needs to be escaped properly as I would consider that advanced usage.

@essentialexch
Copy link

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

I can't believe you just suggested iex. :-)

@SteveL-MSFT
Copy link
Member Author

SteveL-MSFT commented Jun 30, 2020

@essentialexch there are a few times where it's appropriate :)

Note that in the example, expectation is that the user properly validated the contents of $mycommand before executing!

@bergmeister
Copy link
Contributor

Great writeup and a solution for problems like the one you described are definitely needed.
I myself didn't know about --% and don't remember ever seeing it unfortunately. Therefore the proposed &n operator might suffer from similar discoverability problems and it doesn't look natural to me to combine a symbol with a character (and it reminds me of %f in C printf statements). Since it's a breaking change anyway Is there a reason why it couldn't be for example &&?
Maybe it would me more intuitive to use something like brackets (or double braces?) to mark the area that you want to execute. Example: & [ wsl ls ] | less

@JustinGrote
Copy link
Contributor

I agree &! is the more intuitive way to go for those coming from other shells, however in super-duper-linter I am usually running native commands by building a parameter array and then splatting it to the command.

$command = 'linter'
$myargs = @(
    '-config'
    'linterconfig.path'
)
if ($Test) {$myargs += 'test'}
& $command @myargs

So this is the best way in case of conditional and whatnot for me

@SteveL-MSFT
Copy link
Member Author

@bergmeister I originally thought of && as visually it's similar to & which many people know today. However, && being a pipeline chain operator may cause confusion about what it is supposed to do. I like the &! suggestion currently.

@SteveL-MSFT
Copy link
Member Author

@JustinGrote, there's no reason to not continue to use that. the &! syntax is really for cases where you just want to cut and paste or have some args that conflict with PowerShell syntax and don't want to or don't know how to escape properly

@oising
Copy link
Contributor

oising commented Jun 30, 2020

Oh man, I remember having discussions with Bruce and Jason about this a decade ago. My gut feeling is that inventing another operator seems unnecessary here. I know that some people on this thread have not heard about --% but I'll wager than there are more than you think. What's wrong with:

# run ls in wsl, return results to powershell, then pipe to wsl again, to grep.
& wsl ls | wsl grep -i "foo"  

#  the entire pipeline right of wsl will run in wsl. 
& --% wsl ls | grep -i "foo"

Why hasn't anyone suggested this? It's logically consistent with usage of --% elsewhere to say "everything after this is to be passed without special powershell treatment."

@JustinGrote
Copy link
Contributor

@oising well for one, that command doesn't work because you have to preface it with the command you want to run, are you suggesting adding the functionality?
image

This kinda does what is expected:
function Invoke-LiteralCommand ($command) {Invoke-Expression "& $command --% $args"}

Invoke-LiteralCommand ping -W 200 www.google.com | grep icmp

@oising
Copy link
Contributor

oising commented Jul 1, 2020

@JustinGrote Yes, I know it doesn't work now :) I am suggesting that instead of &n or &! or whatever else is being talked about. It just makes more sense to me: & is call and --% is suppress powershell parsing; together they are coherent and more discoverable than adding something new. I don't like the idea of having a "call" and a "call native" operator.

I've expanded my example to show the differences.

@JustinGrote
Copy link
Contributor

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

@essentialexch
Copy link

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

@oising
Copy link
Contributor

oising commented Jul 1, 2020

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

True, or for those of us who'd rather not have to think about them at all.

@vexx32
Copy link
Collaborator

vexx32 commented Jul 1, 2020

Yeah I'm with @oising. This concept exists, it's just woefully insufficient. If we're going to implement something completely new we're better off deprecating/removing the old syntax.

I feel oftentimes these ideas are voiced, not enough weight is given to their actual target demographic. If users are already struggling to find the existing methods, and finding the existing methods lacking when they are found, it's a call to a) improve documentation, and b) improve the actual functionality of the existing operators.

Instead we get a weird option c) which is supposed to somehow address the past issues whilst introducing yet another operator which is relying on documentation which is already not accessible enough for users to find it naturally.

I agree that error messages and/or the suggestions system should be used to help introduce these concepts. I don't agree that we need a third? fourth? fifth? (I've literally lost count, someone help me out here) way to invoke commands. The existing methods are insufficient -- that means we should improve them, not leave them behind like cobwebs to confuse users further when they start digging into them.

The first question I tend to get when people realise there's 5 ways of doing something is "why are there 2-3 ways that literally do the same thing but don't work as well", and my answer is as it always has been -- the PS team is way too focused on making sure everything from the past decade++ still works, when trying to add/improve functionality. Here we see it again, we have existing but insufficient implementations that need revision and improvement, and the solution proposed is to further muddy the pool by adding another potentially incomplete implementation.

We should be finishing what we start, not starting another implementation all over again, IMO.

@romero126
Copy link
Contributor

I believe if we are going to be using & for more than a few Unique use cases we should start thinking about making this considered a verb or look at standardization across the language.

The last thing I would want is confusion about what & is meant to do. In its entirety.

@iSazonov
Copy link
Collaborator

iSazonov commented Jul 1, 2020

I'd consider 3 scenarios:

  1. Full expansion - current PowerShell behavior.
  2. Full literal - that is proposed in the OP.
  3. Partial expansion - wsl ls $path still works

I wonder where @mklement0 comments? :-)

@peppekerstens
Copy link

peppekerstens commented Jul 1, 2020

At start of reading this thread I thought; great idea! But after reading comments it started me thinking. I have always found the & command 'not PS standards worthy" and something reminding me of 'PERL days' (ugh). Introducing another non-standard PS command (not being noun-verb) will not help. Certainly not those less savvy at PS.

I did not know about the --% parameter either. I use the same principle as @JustinGrote as a solution.

Cutting/pasting commands into PS shell has never been my main 'thing'.
Would we not be better of replacing those native commands with PowerShell one's?
Make path for replacing cmd.exe entirely...

I vote for exploring improving existing commands. And - indeed - improving some documentation on --% and & usage

@iSazonov the use case @SteveL-MSFT is proposing is the 'cut paste' scenario. Partial expansion would make things much more difficult I think (on the AST side of things)

@SP3269
Copy link

SP3269 commented Jul 1, 2020

Immediately though of “#!” and then “!#”. Liking the “&!”.

@romero126
Copy link
Contributor

We can probably look at adding token accelerators to function calls in the tokenizer. This would allow for the use of the following.

Iex -l
Or Invoke-Expression -literal that stops parsing before passing up.

I feel adding more unique tokens at the parsing level makes the barrier to entry go up and discoverability for this feature goes down.

Get-Help & for example doesn't show up with anything. And we have to look for it in about_Operators.

However there are unique cases for the call_operator that doesn't get documented.
& modulename { command }
allows you to scope into a module. And I am certain there are more. But the discoverability on it is so low that it becomes difficult to find within the native documentation. And the community knows about it only through JSnover and a talk he gave showing us cool things.

I believe whatever is decided on it needs to fully take into account how discoverable this "feature" is, from a new user perspective and keep in mind that newer users will try to use this on Powershell 5 not without knowing its new to pwsh core if the discoverability is too low.

@rjmholt
Copy link
Collaborator

rjmholt commented Jul 1, 2020

I'd love us to change the existing behaviour, but changing such a fundamental API in PowerShell would just break so many things. We could possibly consider migrating over with a configuration.

Breaking things more directly might once have been possible before PS 6 went GA, but honestly, even then it would have been a serious threat to backward compatibility (not unlike Python's print() function).

In terms of passing arguments to subprocesses, I think both synchronous invocation and Start-Process invocation already have storied issues discussing an overhaul, and my feeling is that both need investing in at once for consistency. Start-Process in particular needs to update its support of passing an array of arguments.

For synchronous invocation with cmdline arg passing, I see four kinds of argument passing possible:

  • Current behaviour, which would become legacy, which is subject to CommandLineToArgvW on Windows and has unexpected results
  • Default behaviour, that should pass arguments by their expression value but apply the usual bareword token rules, so that things like > and | separate commands. In this mode, the value a subprocess takes from the expression should be what is displayed by Write-Host $val
  • Verbatim behaviour, where all tokens are interpreted as bareword strings until the escape token is seen. This is what --% is supposed to do today, but ideally would have only one end token that can be embedded on the same line like --% ... %--
  • Bash readline behaviour, where an alternate, sh-oriented escaping logic is applied, allowing simpler compatibility

I think the first behaviour should be slowly phased out by introducing the second as an experimental feature and then swapping them.

The second and third behaviours could have their own sigils like &! and &# or perhaps +% ... -% or something. But for the verbatim mode, I think an important facet would be simplifying the escape token so that more tokens are taken verbatim. Similar to a heredoc.

@iSazonov
Copy link
Collaborator

iSazonov commented Jul 2, 2020

If the request is only to address copy-paste scenario like "Any text content after this operator will call into the "default shell" of the OS to execute." why do not say this explicitly by shell?

PS> shell wsl ls | less
PS> (shell wsl ls *.txt) | Select-String Hello

It is more discoverable and readable than cryptic operators in Forth/APL style.

@TSlivede
Copy link

TSlivede commented Jul 2, 2020

I absolutely don't see how this would resolve #1995: See #1995 (comment)

Also, the problem with WSL is a problem of the behavior of wsl.exe (see #12975 (comment)) not of powershell.
(Ok, there is a problem with powershell, but that's #1995)

Oh, and isn't this almost a duplicate of #12975 ? Because I can essentially mostly repeat, what I mentioned there:

@bitcrazed
The problem is that the lack of an ability to delimit a portion of a command-line to be passed varbatim to the receiving command/script is something that trips users up all the time.

What do you mean by "portion of a command-line to be passed varbatim"?

Do you mean

  1. pass some sequence of characters verbatim to the called executable, such that the called executable has this sequence of characters in one element of its argument array (e.g. those characters are then available in argv[1] in main)

Or do you mean

  1. insert some sequence of characters verbatim into the lpCommandLine parameter of CreateProcess

If you mean (1.), then it essentially already works:

PS /home/User> /bin/echo 'cd / && ls . | cowsay'
cd / && ls . | cowsay
PS /home/User>

(Except for the problem of embedded quotes as discussed in #1995)

One could argue, that adding a one-line-here-string would improve some usecases, but I think, that's not really the point of this issue.

As this does already work more or less, I assume, you meant (2.)


If you mean (2.), then let me state my opinion on that in a somewhat dramatic way:

Please please please don't add special syntax for this. This is basically what --% tried to do, which also should have never ever been implemented.

Why am I so strongly against this?

  1. It is a Windows only problem, so adding syntax would mean that powershell on Windows has different syntax than on Linux. (Or the syntax would be supported but is totally meaningless, as it is currently the case for --%)

  2. If the main commandline shell on Windows published by Microsoft adds a first-class feature (via special syntax opposed to via a cmdlet) to call executables that don't follow the typical commandline parsing rules (if the tool follows the typical rules, you don't need (2.), you can usually better use (1.)), then that encourages authors of command line tool, to not follow these rules, which only worsens the "Windows command line anarchy". The less people follow the typical rules, the harder it is to programmaticly call external executables or generally write cross platform code, so I definitely think, that program authors should be encouraged to follow those typical rules.

  3. I strongly doubt, that this is a common use-case

    And this isn't just an issue that affects WSL: It also affects Windows Terminal's wt command-line invocations, and many other tools.

    Could you add some examples, where such problems occur? Because in case of WSL, I'd say that WSL's parsing of the commandline is simply broken (issue was about bash.exe but situation is by default not better with wsl.exe) in the default case - I'd consider every tool, that doesn't follow the typical commandline parsing rules broken, but WSL's default behavior is IMHO not even properly documented...

    I said the "default" behavior of wsl.exe is broken - while writing this response, I noticed, that wsl.exe actually seems to behave as expected, when using -e:

PS C:\> wsl -e bash -c 'cd / && ls . | cowsay'
 _______________________________________
/ acct bin boot cache cygdrive data dev \
| etc home init lib lib64 lost+found    |
| media mnt opt proc root run sbin snap |
\ srv sys tmp usr var                   /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
PS C:\>

So the only thing missing for this explicit usecase is IMO a parameter to wsl.exe to call the default shell with the commandline arguments parsed as in any other normal exe.

The only thing regarding "portion of a command-line to be passed varbatim to the receiving command/script", that would be improved with this operator would be meaning (2.) above, and as mentioned, I think developing in that direction is bad.

If all you want to do is a shortcut do copy&paste existing command lines, why don't you add a cmdlet (e.g. Invoke-Shell, that calls bash -c on linux and cmd /c on windows?
Then you could just do

Invoke-Shell @'
whatever existing comandline containg '"quotes"' or whatnot
'@

If it must be one line, then add one-line-here-string syntax, but please don't implement a special operator, that can't be properly found and just complicates everything.

And please don't abandon #1995 for this!

@romero126
Copy link
Contributor

romero126 commented Jan 9, 2021

After reading a ton of the community suggestions, I truly wonder if this is the right path. Based off of the syntax that people are using I honestly think it should be treated how we use a new here-string or scriptblock.

So my suggestion would be a uniquely escaped scriptblock

Example:
& {sh sh}

Syntax may change a little bit,
However I do like the following options
!{ }!
&!{ }!&
{sh sh}

Of course there are implications on parsing and adding a new ast tree visitor, I feel like this would give us the better forward facing design elements The PS community would be able to leverage for a variety of instances vs the current suggestions that all hint at the same thing but are limited in scope.

Thoughts?

@iSazonov
Copy link
Collaborator

iSazonov commented Jan 9, 2021

@romero126 You could re-read my comments above why this way is the navel - we don't need "one more way" to launch native apps.

@romero126 @peppekerstens You could look #13428. If we injected PSNativeCmdlet in Engine we would be able to solve all the problems mentioned and get tremendous flexibility in development for many years.

@mklement0
Copy link
Contributor

mklement0 commented Jan 9, 2021

I truly wonder if this is the right path.

Indeed.

We're talking about a cross-platform (in syntax only) mechanism for passing a command-line string to the platform-native shell.

In a nutshell: cmd /c ... on Windows and sh -c '...' on Unix.

Such a narrow use case does not call for an operator, but for a command - the previously discussed ins (Invoke-NativeShell).

how we use a new here-string or scriptblock.

In fact using the existing string literals, such as - if needed - a here-string is the appropriate thing to do - both conceptually and syntactically - no need to invent new syntax.

After all, you're passing a string to an external program, which then interprets it according to its rules.

Using strings is also the right choice for being able to use string interpolation, so as to be able to embed PowerShell variables and expressions in the command line being passed.

The above solves all conceptual problems that would arise from shoehorning a different shell's syntax into PowerShell's syntax without proper delimiters, via an operator, as proposed here, which would invite perennial confusion and headaches.


I do understand the desire to make this use case syntactically easier; the choices are:

  • In many cases, using regular string literals will be sufficient; e.g.,
    ins '/bin/echo "Nat \"King\" Cole"' (Unix) / ins 'echo Nat "King" Cole | findstr "g\" C"' (Windows);
    you'll just need to know that ' must be escaped as '' in '...' strings, and pass-through $ as `$ in "..." strings.

  • Here-strings obviate the need for escaping (in the verbatim variant) for quick paste-it-and-submit-it scenarios:

@aminya
Copy link

aminya commented Jan 21, 2021

Whatever decision is made about the operator, I think there should be a mode to enable globally (in my $profile for example), to make all the commands have this new operator that fixes the broken behavior implicitly included before them. This allows using PowerShell like Bash without the quoting issue.

With the new mode enabled, I should be able to do this as I can in Bash:

❯ node -e 'console.log("good")'
good

❯ node -e "console.log('good')"
good

❯ $str="good"
❯ node -e "console.log('$str')"
good

@mklement0
Copy link
Contributor

mklement0 commented Jan 21, 2021

@aminya, what you're describing is an unrelated, general problem with how PowerShell passes arguments to external programs, discussed at length in #1995 (comment); the plan is to initially provide a fix in the form of an experimental feature (not available as of this writing; an initial attempt to implement it was abandoned (#13482 (comment)), we're waiting for a new one update: see #14747 for the discussion and PR #14692 for the current PR).

@aminya
Copy link

aminya commented Jan 22, 2021

@aminya, what you're describing is an unrelated, general problem with how PowerShell passes arguments to external programs, discussed at length in #1995 (comment); the plan is to initially provide a fix in the form of an experimental feature (not available as of this writing; an initial attempt to implement it was abandoned (#13482 (comment)), we're waiting for a new one).

Thanks.

Also, it would be good to remove the need for esc operator when we want to interpolate the flags.

Notice the need for esc

$someFlag="--some-flag"

$esc = '--%'
script/build --first-flag $esc $someFlag

In bash this becomes

script/build --first-flag $someFlag

@mklement0
Copy link
Contributor

mklement0 commented Jan 25, 2021

@aminya, there's no reason to use --% in your example and, in fact, it actively prevents interpolation (if the symbol is used literally and/or via @-prefixed splatting; the behavior you're showing with --% contained in a variable actually does result in interpolation while still stripping the --% and is arguably a bug - see #11608); this thread already contains a detailed discussion of --%, the stop-parsing symbol and its appropriate uses and limitations, starting here; independently, there are various problems with passing --prefixed tokens as unnamed arguments to PowerShell commands and/or to external programs.

However, I suggest keeping this issue focused on the original intent: offering a convenient way to execute commands written for the platform-native shell.

@Luiz-Monad
Copy link

Luiz-Monad commented Jun 30, 2021

This situation is so unfortunate for me, I need to fastly process the output for CMake when tracing, besides passing compiler arguments, which are kind of annoying (bad CMakeLists, but that's another problem). I also couldn't wait for 7.2 going live.
What I needed was redirecting the error output to the normal output by the most directly means possible, and the only way to do that is at the dotnet IO.Stream source.
(I could recompile CMake, but I don't want to do that for reproducibility reasons, an external Powershell module is better anyway, as this problem could happen again in other processes).
So I made a better Invoke-NativeProcess CmdLet that solve the streaming problems, besides correctly using ArgumentLists because it uses a copy of the Start-Process the same way the internal NativeCommandProcessor does.

Please provide us with better ways to hook new CommandProcessors, so we can directly create custom replacements instead of making CmdLets.

I think if there was a simpler way to hook CommandsProcessors, or change which CommandProcessor is used for each executable, it would be much better than using & and relying on it doing the wrong thing (trying his best thou, but it doesn't know what its command does).
I bet if CommandProcessors were customizable, this problem could be alleviated, because it mostly happens when integrating with external tools.
So it would be much simpler to create customized CommandProcessors for each tool, perhaps driven by metadata, a simple XML and powershell generates all the CmdLets/CommandProcessors to build the command line for the external application, the final user doesn't even need to know anything, is just does Get-Help and get the correct parameters. It would get much more integrated. An idea would be doing what syntax highlighters do in VsCode, sometimes they have a parse-tree just for highlighting, which is kind of the same problem we have with detecting which thing is a literal char and which thing is an argument that should be expanded, this is a way to drive for any kind of syntax, and work for most languages (besides C++, because, in C++, even the syntax is Turing complete, you can't know it before you make the entire compiler).
A good side effect would also having standardized -parameter witch is better than /s or --bullshit, besides Get-Help being useful.

Meanwhile, I have a solution, just use my Invoke-FastProcess
The library I made for this purpose. https://github.com/Luiz-Monad/PowershellProcess

GitHub
Contribute to Luiz-Monad/PowershellProcess development by creating an account on GitHub.

@iSazonov
Copy link
Collaborator

iSazonov commented Jul 1, 2021

So it would be much simpler to create customized CommandProcessors for each tool, perhaps driven by metadata, a simple XML and powershell generates all the CmdLets/CommandProcessors to build the command line for the external application,

You could look https://github.com/PowerShell/Crescendo.
(I'm personally skeptical about it. It's extremely surprising that PowerShell doesn't do what it should and we need to learn yet another convoluted module just to run an application.)

My alternative proposal partially is in #13428.
(This allows us to do magical things with minimal changes in Engine (based on some meta-information) as well as very specific things.)

GitHub
a module for wrapping native applications in a PowerShell function and module - PowerShell/Crescendo

@Luiz-Monad
Copy link

Luiz-Monad commented Jul 2, 2021

So it would be much simpler to create customized CommandProcessors for each tool, perhaps driven by metadata, a simple XML and powershell generates all the CmdLets/CommandProcessors to build the command line for the external application,

You could look https://github.com/PowerShell/Crescendo.
(I'm personally skeptical about it. It's extremely surprising that PowerShell doesn't do what it should and we need to learn yet another convoluted module just to run an application.)

My alternative proposal partially is in #13428.
(This allows us to do magical things with minimal changes in Engine (based on some meta-information) as well as very specific things.)

Wow, that's exactly what I needed, thank you.
I can even replace & with my InvokeProcess-Fast which buffer the stream directly from the source, the System.IO.Stream, to an array, so it doesn't send thousands of objects onto the pipeline (sending arrays kind of partially defeats its purpose), but now I can stream 4MB of text into my console, and better, I can Tee-Objects as fast as the filesystem would write them, which gets me amazingly fast throughput, the file finishes writing in parallel faster than dumping text onto the screen, which is what I wanted.
But the most important of all, when I redirect the error stream to the output, it doesn't get stupidly slow because lots of ErrorRecords with Exceptions inside with bloody stack traces inside them... (and putting an enormous strain on the GC, exceptions are meant to be exceptional, not a container)

GitHub
a module for wrapping native applications in a PowerShell function and module - PowerShell/Crescendo

@Luiz-Monad
Copy link

Luiz-Monad commented Jul 2, 2021

(I'm personally skeptical about it. It's extremely surprising that PowerShell doesn't do what it should and we need to learn yet another convoluted module just to run an application.)

I'm not so surprised by that, I kind of hate command line integrations between programs, seriously, calling a program by the command line and passing arguments as an array of strings that need to be parsed...
Perhaps some kind of Intent for doing IPC, though the Binder, like Android does, but that's on operating systems, not shells. We can't blame Powershell too much for it.
Any "new" kind of shell will have to deal with this bullshit one way or another.

But I agree with you that a tool like Crescendo for parsing metadata and making all command line tools have a standardized interface is absolutely needed, it should be a core module and very solid with the most used software, like git, or cmake in my case.

Perhaps this Unix idea of everything being a text was not a good idea at all, powershell was supposed to solve that, it kind of does if you only use cmdlets...
So the next very best thing is actually automatically generating those cmdlets.

@yecril71pl
Copy link
Contributor

Perhaps this Unix idea of everything being a text was not a good idea at all, powershell was supposed to solve that, it kind of does if you only use cmdlets...

That idea has been superseded by the desktop bus.

Copy link
Contributor

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

2 similar comments
Copy link
Contributor

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

Copy link
Contributor

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

@microsoft-github-policy-service microsoft-github-policy-service bot added Resolution-No Activity Issue has had no activity for 6 months or more labels Nov 16, 2023
Copy link
Contributor

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Shell automation moved this from To do to Done Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Committee-Reviewed PS-Committee has reviewed this and made a decision Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif Issue-Enhancement the issue is more of a feature request than a bug Resolution-No Activity Issue has had no activity for 6 months or more
Projects
Shell
  
Done
Development

No branches or pull requests