ConPTY Passthrough mode

[be5invis]

Modern apps won’t read the hidden character grid and do everything in VT. So why not an API/console mode to tell Console Host to completely throw away that?

[DHowett-MSFT]

@zadjii-msft was looking into this. The main issue, if I recall correctly, is that we need to trash the entire buffer when something enters or exits “passthrough” mode. It’s also only truly applicable when there is a connected pseudoconsole session.

[DHowett-MSFT]

And the reason one console might enter and exit passthrough mode multiple times is that you may run coolNewThing.exe from CMD, and perhaps it might launch a further process that needs legacy support.

Additional concerns: if you have a tree of four processes, each of which wants passthrough to be different, should the ones that are doing ReadConsoleOutput be able to read the buffers from the other legacy/non-passthrough ones?

It’s complicated when you get into compatibility discussions. 😄

[zadjii-msft]

Yea, I tried getting this working for like a day last year, but it's something I've wanted to work on for a while.

As Dustin mentioned, there'd be real weirdness moving between passthrough mode and non-passthrough mode. However, I think it might still be something good to investigate.

[zadjii-msft]

From #1985:

I've discussed this a couple times on different threads before, but I think I never made a real issue for it.

The idea of conpty passthrough mode is that a commandline client application
that knows it's only going to use VT sequences (and not the API) to modify
the console could set a special mode, ENABLE_PASSTHROUGH_MODE. If conpty is
active, the console would then stop rendering itself over conpty, and anything
that was written to the console would go straight to the terminal.

• This would work especially well for something like wsl, where it's only ever going to be talking VT.
• It would not work for cmd.exe, because of cmd.exe's heavy reliance on the API.
• Windows Powershell again needs the API pretty heavily, but maybe Powershell
  Core, which is cross-platform, could make use of it.

This has some rough edges that need to be sorted out.

• What happens when the app exits, and passthrough mode is turned off? The
  terminal and conpty's buffers would not be equivalent anymore!
  • I've been toying with the idea of having conpty both pass the sequences
    through, and also process them itself, so that the terminal and conpty stay
    in sync.
• What happens when someone tries to call Console API's in passthrough mode?
  • I'm thinking we just cause them to fail. All save for Get/SetConsoleMode and reading input / writing output.
  • We'll also probably need to be able to read resize events.

This needs a real spec written, but it probably needs prototyping done before that.

[oising]

We need a SIGWINCH asynchronous signal for resizing somehow.

[mintty]

This is an essential feature for opening MS text-mode software (esp. WSL) to 3rd-party environments.
The current approach seems to tighly connect the Windows terminal implementation with ConPTY, however that limits applications unnecessarily and makes them highly dependent on Windows terminal progress (which may still take years, honestly).

It would not work for cmd.exe, because of cmd.exe's heavy reliance on the API.

It should be a combined mode: Whenever a console-API-based application is run, e.g. as started from a pure terminal-based application, the console API calls should be transformed into terminal escape sequences. Note as there are not so many features in the console API, this is a much easier approach than the reverse mapping, trying to squeeze terminal features through the conhost bottleneck.

[therealkenc]

Just realized #2035 is this ask framed differently.

however that limits applications unnecessarily and makes them highly dependent on Windows terminal progress

Not just progress. Behavior. This guy is writing a Tek4010 emulator. He is going to need a PTY if say his code were ever ported to native win32. And conhost sure as heck doesn't know what to do with the bytestream coming down that PTY. Never will. And need not care.

We need a SIGWINCH asynchronous signal for resizing somehow.

That too. Doesn't have to be a signal, mind, if there is some religious/philosophical reasons against. But if not it needs to be a separate (third rail) HANDLE on which I can WaitForMultipleObjects(), because no one said there is a ReadFile() byte coming, ever.

[be5invis]

@mintty
There are many absurd console API usages, like reading the screen. That's really a legacy that ... IBM PC has a video card while PDPs aren't.
One idea is that ConPTY can have a "sync" stuff to read back the screen from a console application if some console API applications want to something strange. Otherwise the console host could simply convert console API calls into VT sequence, and forward that to the terminal app.

[therealkenc]

There are many absurd console API usages, like reading the screen.

Great. Invent a new CSI sequence for that. Send the PCHAR_INFO lpBuffer back as I dunno a base64 gzip. The feature is not supported by Mintty. Yet. No biggie. Not even unusual. Any 6502/Z80 assembler programmer working 1981 was free to add their own crazy vendor-specific sequence to their company's terminal if they were bored enough on a weekend too.

[mintty]

There are many absurd console API usages, like reading the screen.

Great. Invent a new CSI sequence for that.

I assume this concern is more about the other direction of such adaptation, i.e. how would you serve a Windows console program that wants to use that "absurd" feature? You could run a second, hidden console in parallel, to maintain backwards compatibility.

[DHowett-MSFT]

run a second, hidden console in parallel

The people who hate how ConPTY is implemented today will absolutely hate how it's implemented if we do that. 😁

[mintty]

They wouldn't even notice in pure pass-through applications. It might be necessary to solve an otherwise unresolvable dilemma.

[therealkenc]

run a second, hidden console in parallel

This solution makes no sense because there is no window in sshd.exe nor someoldprogram.exe. Both those programs are text only applications that wouldn't know a Consolas Font from a hole in the ground.

To the point of #2035, neither sshd.exe nor someoldprogram.exe are terminal emulators. Gnome Terminal is a terminal emulator. And gnome-terminal is the only thing that can give the correct answer as to the contents its screen buffer; ie what ReadConsoleOutput() should return. Similarly, the only thing that knows the contents of (VS Code) xterm.js's screen buffer is xterm.js. ConPTY doesn't have a clue what xterm.js internal screen buffer contains. At best it can only guess by scraping the data passing by.

I assume this concern is more about the other direction of such adaptation,

Yes. And as a practical matter I wouldn't expect the ESC[?GIVEMEBUFFERASIBM feature being added to vteterminal anytime soon. Unless someone can point out the killer app that calls ReadConsoleOutput().

[a hidden window] might be necessary to solve an otherwise unresolvable dilemma.

I think we are closer than that. If conhost wants to read-only scrape the data going by for the sole purpose of keeping a wild-ass-guess as what the actual terminal emulator's buffer looks like, sure, I can live with that. ReadConsoleOutput() is a red herring. If conhost (call it a urxvtd analogy) wants to maintain a shadow buffer, it can knock itself out.

...So long as it is quiet about it. There is no reason for a pass-through "mode", which is how this issue was framed. There is never a reason for ConPTY to inject a VT sequence into a WriteFile() on a PTY handle -- ever. Daniel over on the VS Code team can't possibly care; because he has his own terminal emulator. PTYs don't care about VT100 sequences. Never heard of them.

[Then we need SIGWINCH (or equivalent thereof). Which isn't related to VT100 sequences or screen buffers either.]

[DHowett-MSFT]

@therealkenc,

Like it or not, conhost is the API server that, regardless of whether it presents a window, makes all existing Win32 console applications work. It must continue to make those applications work, because organizations really hate love when Microsoft rolls through with a new standard and tells everybody to drop what they're doing and throw out thousands of lines of code.

I think you're looking to turn this request, and this project, into something it's not. You may be attempting to turn CreatePseudoConsole into something it's not, too. Things work the way they work because we need to maintain compatibility with the thirty years of applications written since the VGA text mode buffer became the "official" design inspiration for how consoles should work in DOS and Windows.

ConPTY exists to--narrowly-stated--allow an application that understands a number of sequences as specified by an xterm-256color terminfo to host a windows console application; to wit: an application that would otherwise run in conhost should be able to run in a "terminal emulator" of sufficient compatibility. It's not intended to support a TEK4010 application (those are not windows console subsystem applications), and it's not likely to want to support a TEK4010 terminal emulator that is expecting to receive a bytestream from a TEK4010 application. That guy will probably end up doing what everybody ELSE who doesn't want to write a windows console subsystem application does: use pipes, because they don't have the same compatibility requirements (neé guarantees) as the windows pseudoconsole infrastructure.

Through that lens, a "passthrough" mode is required. A console application by default, and this cannot be changed for compatibility reasons, starts up in a mode where it just has full access to all of the stupid Win32 console APIs that no terminal emulator developer wants to countenance. Nobody should be able to read back the contents of a terminal buffer, local or remote, that they wrote to. Nobody should be able to write into the offscreen section of the buffer, because there's no guarantee anywhere else that it actually exists. But, they do. Developers use this. Applications expect this, because they were written as windows console subsystem applications. A passthrough mode--mode!--is the only way we can offer an application a way to say "I promise I won't use the old ways" while still being a windows console subsystem application. That's the first step we can make towards ConPTY being the dumb pipe you want it to be.

If you'd like to debate whether the Windows Console was the right choice, or was well-designed, I'm happy to have you do it--but not here.

---

Two asides.

1. If you're looking for a better SIGWINCH, follow WINDOW_BUFFER_SIZE_EVENT generated during window scrolling #281.
2. On machines where isatty(3) is a libc-provided fixture, you also have openpty(3). When somebody spawns a child application with a stdin/stdout hooked up to file descriptors they get back from openpty(3), isatty(3) suddenly returns 1. That application will, more often than not, decide that what that means is that it can send back VT100 escape sequences. That's what it means to most applications to be "on a tty." Sure, it's a terrible abstraction and a poor design and applications should be smarter than this, but they're not. The Windows Pseudoconsole fits right in, here, with the understanding that "I've allocated a PTY, which means I want VT".

[therealkenc]

I'm not intending to criticize the design or implementation -- at all. That was not the intent. I am trying (more slowly than intended) to find a solution to open-for-a-year issue WSL#3279. Of which Biswa96 has a very good start.

[zadjii-msft]

woah this thread got pretty out of hand over the weekend.

The stars aligned Friday, and I actually got a chance to play around with implementing a passthrough mode for conpty. I'm pretty happy with how it works so far, so I think it needs a spec and some polish, and maybe we can ship it one day. Here's the approach I've been taking:

• I introduced a new ENABLE_PASSTHROUGH_MODE to the SetConsoleMode flags.
• When a commandline application enables passthrough mode, and the console is currently attached to a terminal (it's in conpty mode), any text they write to the console is written straight to the terminal as well, with no munging by conhost.
• In passthrough mode, conpty stops "rendering" any changes to its buffer.
• If a commandline application knows it's not going to be doing anything with the console API, it can safely set passthrough mode to be able to talk directly to the terminal.
• [NOT DONE YET] If a commandline app in passthrough mode tries to call any API's, we'll either:
  • Convert the effects of that API call to VT, and pass that through to the terminal. We'll do this for API's where this is reasonable - SetConsoleTextAttribute is a good example.
  • For some APIs where the terminal doesn't really matter, we'll just keep doing what we're currently doing (case in point GetConsoleProcessList).
  • For APIs that don't make sense for conhost to be able to respond in passthrough mode, or we can't create a VT sequence to perform the requested operation, we'll return E_UNSUPPORTED_API (or a real error) indicating that API isn't supported in passthrough mode. Case in point: ReadConsoleOutput*, ScrollConsoleScreenBuffer, SetConsoleDisplayMode, etc.
    • Since the client app was updated to add support for passthrough mode, it should be able to be updated to handle this error case as well.
• conhost also processes the strings that a commandline app emitted. This is to try and keep conpty's state in sync. This is important for the following.
• When a client app exits passthrough mode, there's a chance that the terminal is in a torn state from the conpty. Case in point, if cmd.exe launched wsl.exe, and wsl enabled passthrough, did some stuff, then wsl exited, and cmd restored the console back to non-passthrough mode (as it needs to use the API). When passthrough mode is exited, conpty will redraw its screen, to re-sync the terminal to what conpty believes the buffer looks like.
  • For things like just launching wsl.exe or ssh.exe directly, or for mintty, this won't be as important, as the client will always be in passthrough mode, and the conpty will never "exit" passthrough.

This provides a way where we can be sure that apps that weren't updated for running in a pty will still have access to the entire console API, but apps that want to live in the new world can say "I promise I know what I'm doing", and run even smoother in conpty. I believe there's going to be an incredibly small intersection of apps that want to use VT and also call things like ReadConsoleOutput. Most apps are wither going to be console-like, using the API heavily, and might not be as likely to be updated for such a mode. However, for *nix-like apps that aren't going to be using the API so much and are primarily speak VT, this is an excellent option. This flag creates a clear distinction between the two when running in a pty.

[palves]

sshd on Windows should not be allocating a pty to host things for which sshd on Linux/BSD would not allocate a pty.
That would get ConPTY and its rendering logic entirely out of the loop, and give you stdin/stdout from the spawned
process directly and unmodified.

That is essentially what you get with "ssh -T", but it's not what I need -- I really need for GDB to be connected to a pty, not just a plain pipe, so that input works as if on a terminal, like cursor keys, and sending ctrl-c.

[DHowett]

Interesting. That makes sense; thanks for explaining it :)

[palves]

Windows Terminal Preview has a hidden setting for this. If you open the settings.json file via the "Open JSON file"
button in the settings tab, you can add this:

"experimental.connection.passthroughMode": true,

Ah, thanks. I read https://github.com/microsoft/terminal/pull/11264/files and saw that that is plumbed down to a PSEUDOCONSOLE_PASSTHROUGH_MODE flag passed to the Virtual Console creation. Since I need it with ssh, I guess sshd would have to have its own way to enable this.

So it is not possible to activate the passthrough mode from the command line application itself, like how you pass ENABLE_VIRTUAL_TERMINAL_PROCESSING to SetConsoleMode ? I.e., something like a ENABLE_VIRTUAL_TERMINAL_PASSTHROUGH flag GDB itself could enable in SetConsoleMode? Is that something that you envision would be possible if/when this work is eventually completed?

[BatmanAoD]

Other terminals, such as Wezterm, also need a way to enable this option.

[mintty]

Other terminals also need a way to enable this option.

Yes, and that makes me repeat that this is actually not a Microsoft Terminal issue but a feature needed for the WSL launcher, wsl.exe, which should have a decoupled mode to not capture any terminal interaction; see microsoft/WSL#9117

[j4james]

@mintty As has already been mentioned above, this is also the repository for the console subsystem, which underlies all console applications - that includes both Windows applications and WSL applications. And the whole point of conpty is to translate the Windows console APIs into VT sequences. WSL has got nothing to do with that. Windows Terminal has got nothing to do with that either, but it happens to share the same repository.

@BatmanAoD It's worth noting that any terminals that want to use this option would likely need to work on their VT support first. To translate Windows console APIs into equivalent VT sequences without a local buffer (which is a requirement for pass-through mode), we'll likely need VT features like rectangular area operations, which aren't that widely supported.

[mintty]

the whole point of conpty is to translate the Windows console APIs into VT sequences.
WSL has got nothing to do with that.

Yes, but wsl.exe does, unfortunately, as it intercepts and filters terminal interaction. It shouldn't, in a passthrough mode, that's my point, not really caring about the underlying architecture.

[ferdinandyb]

Put differently, while I'm personally interested in solving this, the reality is that this issue and any of those that would be fixed by it don't even show up in the first 2 pages sorted by 👍. To summarize: I'm sure we'll get to it, but just not right now.

I've been browsing issues and wanted to point out, that sorted by 👍, the third issue is sixels where @j4james said, that

We either need the full passthrough mode working, or for the conpty renderer to be capable of regenerating the sixel on the fly so it can repaint areas of the screen that have been invalidated. Of the two, passthrough mode seems more feasible.

so it seems to me that there's at least one high profile issue that would be solved by this, and I wouldn't be surprised if there were more highly voted issues where this would be the solution (I'm guessing a lot of people are like me: not knowing much about terminal internals, just wanting a WSL terminal to feel like they are on a linux box).

[jerch]

We either need the full passthrough mode working, or for the conpty renderer to be capable of regenerating the sixel on the fly so it can repaint areas of the screen that have been invalidated. Of the two, passthrough mode seems more feasible.

I also think that the sixel-regenerating is a dead-end in the long run creating lots of maintenance burden (you'd basically need an "image recorder" with decoding and encodings clipping viewport areas, geez) while only solving this particular sixel issue. What about other image sequences later on? What about other sequences in general? They will still fail at the "not implemented here in ConPty" trap.
On a sidenote - recently had a similar image forth and back encoding/decoding need for state serialization within xtermjs, and did not vote for sixel, but QOI (reasonably fast in wasm with good enough compression - much better than any PNG handling offered by the browser engines).

This prolly got stressed several times already - to me it seems that ConPty "puts the cart before the horse" with its interim terminal emulation to stay compatible to the far less capable WIN-console API. Wouldn't it be better to do it the other way around and to abstract the console caps into rectangular VT sequences? Yes, support is quite lousy for those across TEs, but even with scroll margins only one can simulate many aspects. I also think that TE devs would happily implement better rect support, if there is a serious need for those.

[j4james]

it seems that ConPty "puts the cart before the horse" with its interim terminal emulation to stay compatible to the far less capable WIN-console API. Wouldn't it be better to do it the other way around and to abstract the console caps into rectangular VT sequences?

@jerch I've been pushing for this for a while, and I think it's probably the right long term solution, but the current conpty approach was most definitely the correct decision at the time, and it may well remain the most practical solution for a long time.

The Windows console API has a lot more functionality than your average VT terminal, and even with the most advanced VT capabilities there will likely be things we just can't reproduce. To be honest, the more time I spend looking at this, the less convinced I am that it's worth the effort.

[mintty]

Windows console API has a lot more functionality

You mean like reading back the screen contents? An exotic feature rarely useful and certainly not relevant in the WSL domain.

less convinced I am that it's worth the effort

That's why I had tried to drag this discussion over to a WSL issue. For the Windows Terminal application, it may be dispensible to provide terminal transparency as it's a terminal itself. For WSL however, there is serious need for transparent terminal access, both local and remote, so passthrough mode without any Windows console legacy burden is essential for the WSL launcher.

[christianparpart]

The Windows console API has a lot more functionality than your average VT terminal, and even with the most advanced VT capabilities there will likely be things we just can't reproduce

@j4james hey, I am curious to know what Windows console API offers that doesn't exist as VT sequence or extension just yet. OTOH, I think if one can count the number of features that are available on the conhost side but not on the VT side, it might make sense to introduce VT sequence extensions for those few and still put conhost on top of VT. I do not want to mandate here anything, I am just curious to know what conhost is more advanced in. :)

On the "your average VT terminal" argument, I'd love to bring some of my main features of contour to Windows, which I can't until either ConPTY supports them, or a passthrough mode is available. currently my windows version of Contour will always be inferior than on other platforms, sadly. Not sure how to solve this without ConPTY passthrough in the future :)

[jerch]

@christianparpart I think the buffer read caps are on the annoying side of things - as far as I remember console API allows complete buffer access (in the sense, that the console app "owns" the console and thus the buffer state). Thats almost impossible with non rect-based VT mechanics or at least will be limited to active cursor area only prolly on all TEs. I still think that most buffer access primitives could be mimicked in VT.
The hard unsolvable problem might be the exclusiveness, that console API kinda provides to apps with a quite strict process IO coupling/isolation - a thing that many ppl also wish for terminals in linux/macos, but it is just not possible there with the current TTY/PTY abstraction (though its not a matter of VT mechanics but POSIX terminal API).

[j4james]

@j4james hey, I am curious to know what Windows console API offers that doesn't exist as VT sequence or extension just yet.

You can find a list of the console APIs here:
https://learn.microsoft.com/en-us/windows/console/console-functions

If you click through each of them, you should see a "Tip" section which explains which of them do or don't have a VT equivalent. If you think you can convert all of them into VT sequences, please do feel free to contribute PRs. A lot of the framework is already in place to do this.

it might make sense to introduce VT sequence extensions for those few

We aren't even at the point where most conpty terminals are supporting the standard VT sequences we would need. The chances of everyone agreeing to a bunch of Windows-specific extensions on top of that doesn't seem very likely.

I think what some of you really want is just a pass through mode for WSL. If that's the case, you can assumedly build your terminal as a Linux GUI app running on WSL. But ConPTY is specially for Windows console application support. If you aren't interested in that, this isn't the issue for you.

[palves]

There is a set of Windows console applications that want the feature (in my case, GDB (GNU Debugger) testing), which don't need anything from the console API that the supported subset of standard VT sequences doesn't already provide. IMO, there could be a console API that such applications could call to enable some restrictive/subset of the API, only. Call it "passthrough mode", or "no intermediate buffer mode" or some such, and when that mode is active, the problematic console APIs (like complete buffer access) would just fail with an error.

[zadjii-msft]

FWIW that's my preferred approach to solving this. A console mode that a client app can enable to say "I solemnly swear I am up to good". Something like WSL would opt-in, because it knows it's only ever going to do VT.

That doesn't work as well for mixed applications that want to do both. But that would let legacy console apps rely on conpty's regeneration for their own needs, and modern apps just use VT and all the new features that come with it.

[j4james]

If that's the way we want to go, that's fine, but it means the Windows shells don't gain any benefit from it. For example, you wouldn't be able to TYPE a sixel file, or ECHO a Contour-specific escape sequence and expect it to work. Personally I'd consider this a waste of time, but I can accept that's all some people might care about.

[jerch]

To avoid rewriting console API into wonky VT pendants, wouldn't the following work:

• make stream-IO/passthrough VT-mode default
• additionally make legacy console API fully abstract on both sides (process & TE side)
• a TE can decide whether to implement the legacy console primitives, if it does - processes can call into console API as before, if not - processes may not call into legacy console (may need special process flags?)
• TE operates as single source of truth for console buffer primitves - its the TEs resp. to get data from its internal terminal buffer feeding the TE-side of the console API
• provide a standard TE on OS side fully implementing legacy console primitives, so things like AttachConsole still works
• last step - retire ConPty 😺

Ofc there are details to overcome, like the question whether the console primitives can be made blocking or need true memory sharing & possible security implications.

[csdvrx]

On a sidenote - recently had a similar image forth and back encoding/decoding need for state serialization within xtermjs, and did not vote for sixel, but QOI (reasonably fast in wasm with good enough compression - much better than any PNG handling offered by the browser engines).

@jerch , am I right to say that we had a few standards already existing (sixel, kitty, ...) and now we've got one more?

I think it's just sad that the desire for technical perfections goes before practical concerns for something that'd work about everywhere, if leveraging the existing ecosystem.

So like @j4james, I think not being able to reach some baseline level of functionality is waste of time:

If that's the way we want to go, that's fine, but it means the Windows shells don't gain any benefit from it. For example, you wouldn't be able to TYPE a sixel file, or ECHO a Contour-specific escape sequence and expect it to work. Personally I'd consider this a waste of time

Actually, I'd consider that not just a waste of time, but a sad waste of time.

[jerch]

am I right to say that we had a few standards already existing (sixel, kitty, ...) and now we've got one more?

Nope, that serialization format is for TE internal usage only, not intended for outside with an explicit sequence. QOI yields the best compression/performance ratio covering RGBA in 32bit. Since xtermjs is browser based, we cannot just store raw bytes somewhere on the disk for session restore, as other desktop TEs would do here.

[mominshaikhdevs]

what is the latest update on this?

[zadjii-msft]

Nothing to share at this time. We'll make sure to update this thread when there is. In the meantime, might I recommend the Subscribe button?

That way you'll be notified of any updates to this thread, without needlessly pinging everyone on this thread ☺️