Can we use | on the next line as a continuation character?

[Jaykul]

I would like it if PowerShell could detect the pipe character as the first non-whitespace on a line as a continuation of the previous line, meaning we could write something like this:

$result = Get-Process
    | Where WS -gt 500mb
    | Format-Table VM, WS, CPU, ID, ProcessName

without having to add backticks!

[lzybkr]

In a script - yes, but that would break the ability to use the old school right-click or Alt+e,p paste in the console because after enter, you'd have a complete command line.

[iSazonov]

Related discussion #2819

I see bad UX for interactive session: after first NewLine parser will wait next NewLine or skip spaces up to | or NewLine - this will confuse the user.

For script this is a request to delete a continuation character ` (backtick) in time as we can always type:

$result = Get-Process |
     Where WS -gt 500mb |
     Format-Table VM, WS, CPU, ID, ProcessName

[Jaykul]

The parser already does this in other places.

if( 1 -eq (get-random 2)) {
    "True"
}
else {
    "False"
}

@lzybkr obviously having the parser be able to do understand it doesn't mean people must write multiple lines that way -- many people have adopted stroustrup style as above, or even "brace on their own line" and other style things that break pasting (especially since PSReadLine messed with right-click and made Ctrl+V work). That shouldn't make any difference.

@iSazonov interactively, if you hit enter, it's not going to wait, it's just going to run it. Just like it does now with the else above. You can hit shift+enter to type it at the console (when you have PSReadLine).

[lzybkr]

PSReadline doesn't mess with right-click, although there is a corner case where a TAB character expands to something it wouldn't have w/o PSReadline - but that has more to do with how multi-line input isn't working as expected if you don't use PSReadline.

But my point was really simpler - the interactive experience was intended to be as close as possible, ideally identical, to putting the same text in a script. This feature would be a departure from that intent in a commonly used scenario - right click.

At this point, I'm saying making a judgement one way or the other on this suggestion, I'm just pointing out something that might not be obvious to everybody.

[KirkMunro]

Right-click-paste could use some modernization, pasting the entire clipboard buffer into the console as if you had shift+entered each line and then running it that way. Even without considering this issue I think that would be useful by itself -- how many times have you right-click-pasted something that wasn't quite right (parse errors) and saw a bunch of error messages, perhaps interspersed with a few commands that executed? That could be quite harmful, depending on what you were pasting (oops, that cd command didn't parse properly, so I stayed in my current C:\ directory when the del *.* -r -fo command executed...). It would be better if right-click-paste actually pasted the entire block, and PowerShell handles that well on the execution side already.

Technically the interactive experience today is already different from putting the same text in a script because right-click-paste executes commands as they are pasted in, rather than as a collection of commands. The result is that parsing errors don't prevent partial execution in right-click-paste while they do in scripts.

Also, for interactive, ad-hoc PowerShell, I think it's pretty uncommon for users to enter multi-line commands by typing them in at the console, and for users who do this today, they can continue to do it just as they have always done. For the rest of us normal folk, our pipelines will be on a single line unless we right-click-paste, which as was pointed out above, could, and probably should, be improved so that the interactive experience really is as close as possible, ideally identical, to putting the same text in a script.

All to say: +1 for having PowerShell support pipelines at the beginning of a line without line continuance characters, and a request to fix right-click-paste so that there is better consistency between interactive use and the scripting experience.

[lzybkr]

Right click is not a PowerShell feature, it's a conhost feature. Conhost can't always assume simulating typing is equivalent to paste without some coordination with the current console process.

[Jaykul]

Yes. It might make right-click paste awkward. In exactly the same way else does.

[michaeltlombardi]

This would make commands (both in the console and in files) with multiple pipes much more readable and, as pointed out before, has exactly the same issue that if/else does when executed in the prompt.

[RichardSiddaway]

Up to now a | as the last character on a line has worked as a continuation character. What does this change bring especially if it stops previous behaviour working

…

Sent from my iPad On 8 Mar 2017, at 17:04, Michael T Lombardi <notifications@github.com<mailto:notifications@github.com>> wrote: This would make commands (both in the console and in files) with multiple pipes much more readable and, as pointed out before, has exactly the same issue that if/else does when executed in the prompt. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#3020 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHgYUW-mDh_8nDUr671LdathDzLZPhS7ks5rjt9-gaJpZM4LohkD>.

[michaeltlombardi]

Would this change necessarily require that a | couldn't still be a continuation character at EOL as well as at the beginning of a new line? I wouldn't expect it to.

[Jaykul]

@RichardSiddaway it's about readability -- this has come up several times in discussions about line-wrapping in the practices & style repo. Some people are writing this way already, just using backticks at the line ending.

Having a pipe character as the first thing on the line makes it really obvious that it's a continuation --much more so than just indenting it-- particularly if the lines are long and you can't easily see the pipe on the end.

Get-Process | Where CPU | Where Path |
    Get-Item | Where FullName -match "AppData" | 
    Sort FullName -Unique

vs.

Get-Process | Where CPU | Where Path
    | Get-Item | Where FullName -match "AppData"
    | Sort FullName -Unique

[JamesWTruher]

@Jaykul in your example above the indentation is just as useful from a visual acuity perspective. Pipe symbol seems to be just an additional (and IMO unneeded) indicator.
given your example, does this appeal as much to you?

Get-Process | Where CPU | Where Path
|Get-Item | Where FullName -match "AppData"
|Sort FullName -Unique

To my eye, this seems quite understandable:

Get-Process | Where CPU | Where Path |
    Get-Item | Where FullName -match "AppData" |
    Sort FullName -Unique

it's the indentation that helps (and it's how I usually write this)

FWIW, I never understood the use of the backtick in long pipelines and tried to squash it whenever I saw it in code reviews, as it was unneeded.

[michaeltlombardi]

I do explicitly prefer the pipe as the first character, as in your first example, @JamesWTruher (albeit with a single space between the pipe and the first character of the command) - it gives a very clear and explicit visual indicator of a continuing pipeline. The indentation gives a similar, but less explicit, inference when scrolling over code.

[BrucePay]

@Jaykul This will completely break the interactive command line experience. For example, say you type a command and hit return

PS> dir
>

Do you get output? No you do not because the parser is waiting for the next token which might be a continuation. Now bang on the return key. Nothing happens except you see more prompts because the parser is still waiting for a token to see if it is a continuation.

PS> dir
>
>

OK - add | sort Length and hit return.

PS> dir
>
>
> | sort Length
>

And still nothing happens because there still might be a continuation token so it's still gotta listen for a new line. Now you say 'i'm done with this - it doesn't work, let's just get the date'. You type Get-Date and hit return. Yay - you get output. But it's not from Get-Date. Nope - you've finally completed the dir | sort Length command so this is the output you see. In the meantime, the command reader is now waiting for you to complete the command you started with Get-Date. Or the whole thing might just be an error.

This would be a horrible interactive experience for any user.

Could we make it work only in files? Yeah - probably but as @lzybkr mentioned, one of our base design principles was that what worked on the command line could simply be pasted into a file and vise versa. We even support pasting from Word documents with emdash and endash characters.

So while I do actually like how '|' looks at the start of the line in things like Haskell pattern matching:

fib n | n == 0 = 1
  | n == 1 = 1
  | n >= 2 = fib (n-1) + fib (n-2)

it won't work for PowerShell.

[michaeltlombardi]

With the caveat that I am as a newborn babe when it comes to the internals, here are some observations:

# This works everywhere
Get-Process |
Select-Object -First 1

# This errors at the prompt unless using soft returns, note the two blank lines;
# hitting enter after the first one causes it to error out.
# Note that with soft returns you can go just about forever before adding the next line.
Get-Process |


Select-Object -First 1

# This will run fine in either
If ($false) {
  ' False Output'
} Else {
  'True Output'
}
# This will error in the prompt (unless using soft returns) but pass in a file
If ($false) { 'False Output' }
Else { 'True Output' }

As an aside, I would expect / be okay with the following failing in the prompt without soft returns, as a non-cuddled if/else does:

Get-Process
| Select-Object -First 1

I do not particularly care for readability, especially readability at-a-glance, when interactive at the prompt (because, generally, I wrote it moments ago) but it's one of my principal concerns when I'm writing/maintaining scripts and modules or debugging other people's work, etc.

[Jaykul]

To address @JamesWTruher -- I would expect people would not indent continuation lines if the pipe was there at the front as an indication of what was happening:

Get-Process | Where CPU | Where Path
| Get-Item | Where FullName -match "AppData"
| Sort-Object FullName -Unique # I'm so proud of this highlighting "bug" by the way

@BrucePay So, right. The interactive experience wouldn't explicitly support it.

If you guys really think there are no differences between interactive and scripts, you need to try using PowerShell without PSReadLine once in a while.

Try using the console in ISE or VS Code

Unless you have PSReadline to handle the "missing statement block" or "empty pipe element" type exceptions, you can't put a newline after a pipe, nor write Allman-style braces. You certainly can't write either of these:

Get-Process |
Where CPU

if($True)
{

But even with PSReadLine

The interactive experience doesn't wait to see if I'm going to write else on the next line after

if($True){ "yay!"}

Or whether I will write the (necessary) generic catch { "You threw $_" } handler after

try { throw 5 }
catch [ArgumentException] { "whew!" }

And it won't let me put a comment where it thinks there should be code, so neither of these is possible:

if($True)
#This always runs:
{ "Hello World" }

Get-Process |
# I only want actual apps, with windows
Where MainWindowHandle

So yes, this would be yet another place where you can have a line break in script but not at the console. But we already have a lot of those (especially if you're not using PSReadLine).

[pauby]

To reference the code that @Jaykul 1 and @JamesWTruher 2 had, I use this:

Get-Process | Where CPU | Where Path |
    Get-Item | Where FullName -match "AppData" | 
    Sort FullName -Unique

As was pointed out the indentation makes this a lot easier to read when quickly scanning the code - I know those three lines of code are 'tied' together without having to glance at the end to see the pipe. Readability for me is one of the most important things when I'm writing code.

I sometimes add the backtick character to the end after the pipe, but when I remember it's not needed, I flagellate myself for it. It's a habit I'm trying to break (the backtick not the briching).

Copying and pasting into the console with PSReadline hasn't given me any issues (I can recall) so it's not something I'm going to get in the middle of and add to the discussion on.

[vexx32]

Having the pipe be on the next line instead of at a line end is a very F#-like idea, where you can do things like this:

let f1 str server =
    str
    |> parseUserName
    |> getUserByName server
    |> validateLogin <| DateTime.Now

And I'm all for anything that makes PS a little more functional (pun intended) ;)

[michaeltlombardi]

Not just F# but also Elm, Haskell, Elixir, and even Google's shell style guide (and note here they have to escape the newlines to do it...).

[BrucePay]

@michaeltlombardi As with the Unix shells, you can escape newlines in PowerShell if you want to

Get-Process | Where CPU | Where Path `
| Get-Item | Where FullName -match "AppData" `
| Sort-Object FullName -Unique

F# is whitespace-sensitive (offside rule) in ways that PowerShell is not. Elm, Haskell and Elixir are, of course, not shells.

complaints about bad hosts...

The interpreter provides information to the host application by throwing an IncompleteParse exception. It's up to the host implementation to catch this and prompt for more input. If the host is doing this wrong, complain to the host owner.

If you guys really think there are no differences between interactive and scripts, you need to try using PowerShell without PSReadLine once in a while.

I used PowerShell for well over a decade without PSReadLine. Is that enough?

This errors at the prompt unless using soft returns, note the two blank lines;

How that's handled is up to the host implementation. Open an issue on the offending hosts if you want this changed.

exactly the same issue that if/else does when executed in the prompt.

Yup. It's ambiguous. Sorry. I couldn't think of a way to make it work reliably. If we'd gone with OTBS instead of matching C#'s brace style, this wouldn't have been significant. In fact there would have been a lot of advantages to using OTBS (e.g. DSLs). Also note: in the original console host, this did "work" because after receiving the first IncompleteParse exception, the host just kept reading lines until it hit a blank line, then executed the result. But this means you always had to hit an extra newline.

@Jaykul So do you want the opposite of F#'s #light directive? Something like #heavy where spaces are irrelevant and semicolons are mandatory?

[vexx32]

Escaping newlines in PS is, in my opinion, more problematic than with most languages. PS's escape character is the goddamn backtick. Syntax highlighting doesn't really make it stand out for the most part -- most highlight schemes render it a pretty dull colour.

Add to that the fact that when you go back and edit something littered with backticks, the probability you'll just accidentally miss a backtick or two that got misplaced amidst the edits is pretty high.

I would use that form if tracing backtick mistakes wasn't like finding a needle in a haystack. No, I think it'd just be better to permit the pipe-at-the-beginning format as a style in its own right, without mandating escaping the line. :/

[BrucePay]

@vexx32 While I agree that '`' is hard to see, '|' is not.

[Jaykul]

@Jaykul So do you want the opposite of F#'s #light directive? Something like #heavy where spaces are irrelevant and semicolons are mandatory?

No @BrucePay, I just want another syntax trick like this:

try { throw 5 }
catch [ArgumentException] {  <# ... #> }
catch { Write-Warning Whatever }

which let's me write things neatly in scripts, without worrying about whether it works when you're typing in the console or not.

[michaeltlombardi]

@BrucePay correct, and I have some code using those backticks, but what we're seeking here is the ability to write the code without them for all the reasons @vexx32 listed.

It's totally fair to point out that several existing behaviors in PowerShell that people rely on are actually the result of host implementation, so thank you for clarifying those things! On the other hand, this also implies that there are things in the language that already break those expectations unless the host implementation does extra work to handle them, right?

[BrucePay]

@michaeltlombardi Bad host implementations will result in a poor interactive experience but don't impact the language itself. That said, the interactive experience is super important since the majority of PowerShell 'scripts' are typed interactively.

we're seeking here is the ability to write the code without them

My point is that if you put the | at the end of the line, continuation is implicit and no continuation character is needed as you request. And the | symbol is easy to see. And use indentation to make the structure of the code clearly visible. Unlike F# or Python, we don't enforce indentation because PowerShell is a shell and sometimes you need to write very dense code.

@Jaykul

without worrying about whether it works when you're typing in the console or not.

The console experience remains a primary scenario for us and the majority of our customers. However we have a meta-feature coming up for supporting experimental features. We could try to do what you want, exposed as an experimental feature, and then see what kind of feedback we get about the overall experience.

[Jaykul]

I totally understand that the console experience is primary. My point is simply that this should not break the console, as you suggested in your first comment.

It should behave in the console the same way that the second catch does above: you can only type it in a console where you have a multi-line ReadLine like PSReadLine and can Shift+Enter to inject an extra line (or you can paste it in) -- it doesn't need to change the existing behavior at all.

After all, this is a feature aimed at improving code-formatting, not reducing typing. It needs to work in the scripting experience, not the interactive console.

[BrucePay]

But one of the promises we made to the user right from the start was that you could paste from a script (example, etc.) into the console and have it work. This even included pasting from word docs with emdashes, etc. Copy and paste is by far the most common form of reuse (we did a study to validate this)

[Jaykul]

Right. But the copy-paste thing is currently only fully true only as long as you're pasting into PSReadLine's Ctrl+V and not via right-clicking -- and this change would work there as well. That is, it would require a multi-line paste capable ReadLine, but it would work by default in PowerShell 5+

As I mentioned above, there are quite a few cases (if/else, try/catch, comments, Allman braces, etc) where that is already not the case if you don't have PSReadLine.

My opinion is that one more edge case to the list of problems with non-PSReadLine hosts isn't a big enough reason to exclude a feature -- after all, this syntax wouldn't work in PowerShell 4 or 5 anyway, so it'll never work in ISE or the non-PSReadline hosts. It's driving me a little crazy that we're spending this much effort debating one more problem with pasting into {{not-our-problem}} but we're fine with the fact that it would be syntax that doesn't work in older versions of PowerShell...

P.S. I've used the pasting argument as one of the justifications for the BestPractices choosing OTBS over Stroustrup or Allman, but I'm definitely going to reference this thread in the book now 😁

[BrucePay]

I really wish we'd gone with OTBS. DSLs would have been a lot simpler.

[michaeltlombardi]

OTBS is, of course, the One True way....

[lzybkr]

@BrucePay - I'm sure there are still some folks that paste multi-line command lines and do not use Ctrl+v, but I'm guessing not so many folks, and they should adapt, pasting without right click or Alt+Spacebar,e,p is superior in multiple ways.

I think it's very reasonable to consider this change.

[vexx32]

Can anyone point me in the right direction for what would need to happen to make this work? I'd like to at least get it working and see what it's like... and then maybe put it behind an experimental flag so we can test it out a bit if need be -- but in honesty I'm not even sure if that is really necessary; after all, this won't overtly change anything about the current experience, just add a new one.

Given that as-is a pipe element at the start of a line is an instant error at the present, it's not a coding pattern that is being currently followed without also the use of escaped newlines, which will also be unaffected by making this change. So really... nothing to lose here, everything to gain?

[lzybkr]

This probably won't work, but should get you started:

diff --git a/src/System.Management.Automation/engine/parser/Parser.cs b/src/System.Management.Automation/engine/parser/Parser.cs
index a31f64fa0..37b897f7c 100644
--- a/src/System.Management.Automation/engine/parser/Parser.cs
+++ b/src/System.Management.Automation/engine/parser/Parser.cs
@@ -5653,6 +5653,11 @@ namespace System.Management.Automation.Language
                 }
 
                 pipeToken = PeekToken();
+                if (pipeToken.Kind == TokenKind.Newline)
+                {
+                    SkipNewlines();
+                    pipeToken = PeekToken();
+                }
 
                 switch (pipeToken.Kind)
                 {

[vexx32]

You're right, that doesn't quite get there. At least in part because currently the parser doesn't care about the number of newlines; so many statements never get executed. I have some potential thoughts on how to proceed, so you've given me an intriguing puzzle, thank you! 💖

[kilasuit]

Thread's like this really does show the immense benefits that open sourcing PowerShell has brought us and for us all to be able to a good and open discussion with the Product Team, MVP's, trusted community members and those that are new to the language like this is fantastic. Just to quote @Jaykul's response earlier on in this thread this would be the ideal end goal.

The key part of what @Jaykul said above is that that we tend to find it easier to comprehend and read that continuation of a command is at the beginning of a line as to opposed to at the end of a line

@RichardSiddaway it's about readability -- this has come up several times in discussions about line-wrapping in the practices & style repo. Some people are writing this way already, just using backticks at the line ending.

Having a pipe character as the first thing on the line makes it really obvious that it's a continuation --much more so than just indenting it-- particularly if the lines are long and you can't easily see the pipe on the end.

Get-Process | Where CPU | Where Path |
    Get-Item | Where FullName -match "AppData" | 
    Sort FullName -Unique

vs.

Get-Process | Where CPU | Where Path
    | Get-Item | Where FullName -match "AppData"
    | Sort FullName -Unique

So yes, I'd like to be able to do this in future

Get-Process | Where CPU | Where Path
    | Get-Item | Where FullName -match "AppData"
    | Sort FullName -Unique

[vexx32]

I need to give this another attempt at some point soon. It's an intriguing puzzle, but not one I really know how to solve as yet. 😕

[pauby]

I agree with both @kilasuit in that this is awesome we now have an open source version of PowerShell. As an aside I installed it on Manjaro Linux yesterday and wasa little in awe that it built it all from scratch. And so quickly.

I also agree with @RichardSiddaway that it would be good to be able to have the option of using the line continuation character at the start of the line. I'm a big advocate in coding for readability so the indentation (ie. first option) is what I use and I'm happy with. I actually don't like the look of the second option (I think it looks ugly :)) but it takes any ambiguity out of it. And I'm all for people having choices if it improves readability!

[KirkMunro]

Last night I felt like tinkering with the parser, and this always bugged me (I've always preferred pipes at the beginning of the line). Here is the result:

The only question I have is whether or not this needs to be hidden behind an experimental flag. If the tests are sufficient, why make it experimental at all, since it doesn't impact any existing functionality? Thoughts?

Now to wrap it up in some Pester tests...

[vexx32]

I don't think it really needs an experimental flag, provided the behaviour is predictable, consistent, and testable. It's not breaking any pre-existing behaviour as far as I can tell.

It doesn't really seem like there's any contention over the design, either.

[KirkMunro]

Also as a side note, it's too bad that PowerShell allows command names to start with "-". Otherwise we could probably wrap parameters on newlines without backticks as well. Sure we have splatting, but this could work with Intellisense and Syntax Highlighting if command names couldn't match parameter names:

Get-AzureRmAppServicePlanMetrics
    -ResourceGroupName Default-Web-WestUS
    -Name ContosoWebApp
    -StartTime 2016-11-30T22:00:00Z
    -EndTime 2016-11-30T22:30:00Z
    -Granularity PT1M
    -Metrics Requests

[dustindortch]

I am awaiting the splat-like sequel, but definitely think the option to have it as a first character is great. Sure it's different between interactive and script, but that is consistent with how PowerShell already is.

[iSazonov]

The only question I have is whether or not this needs to be hidden behind an experimental flag. If the tests are sufficient, why make it experimental at all, since it doesn't impact any existing functionality? Thoughts?

I don't know why MSFT team is so passive at last time but if you code will be under experimental tag I could fast review. Also we could enable the feature by default in powershell.config.json.

it's too bad that PowerShell allows command names to start with "-".

I'm full agree. I think it's worth it to discuss thoroughly in the new issue. (There are other characters we could put away.)

[HumanEquivalentUnit]

"An empty pipe element is not allowed", so there should be no breaking-compatibility problem if a pipe was used for line continuation and the start of the next line, at the same time:

Get-Process | Where CPU | Where Path |
    | Get-Item | Where FullName -match "AppData" |
    | Sort FullName -Unique

Which addresses many of the complaints - no backticks, pipe at the start of a line for visual indication of continuity; no change to the interactive prompt behaviour after typing the first line because it can tell whether to expect a continuing line; doesn't have much confusion with reserved || operator because it's split over multiple lines; is no more typing than using a backtick and a pipe; needn't have problems with a comment line in between as per Jaykul's if/else, try/catch examples.

[kilasuit]

@HumanEquivalentUnit - I can't say that I am a fan of that suggestion because I as a script writer may decide that I want to remove the newlines within a script for cmdline use and would be left with an unworkable and non-intuitive single line that IMO
looks atrocious

Get-Process | Where CPU | Where Path |    | Get-Item | Where FullName -match "AppData" |    | Sort FullName -Unique

Which

[HumanEquivalentUnit]

@kilasuit that edited line looks weird (doesn't look atrocious to me, just odd), but is that really any different from code with backticks which is valid now, if you removed lines without removing the line continuation? Do you currently suffer from that problem with backtick code?

Get-Process | Where CPU | Where Path `    | Get-Item | Where FullName -match "AppData" `    | Sort FullName -Unique

[vexx32]

If you ask me, that's just a clear reason why Kirk's PR is welcome, to be honest. Not requiring either backticks or an additional pipe is very neat when condensing lines.

[iSazonov]

Initial request was implemented.
Now we discuss PowerShell/PowerShell-RFC#179