Support 1337 escape sequence to show inline images #614

cristynkells · 2017-03-17T22:26:24Z

Details

Browser and browser version: all
OS version: all
xterm.js version: 2.4.0

Feature request

Is there a way to view images from the command line?

by either clicking on the image (much like the linkify addon) or via a command:

$ cat vacation-beach-pic.png

Tyriar · 2017-03-18T20:47:46Z

Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax:

// Add a link for a particular regex and handle when it's clicked.
xterm.registerLinkMatcher(/\w\.png/, function (mouseEvent, imageFile) => {
  // imageFile is the text of the link
});

The API should hopefully be finalized in the next couple of versions.

cristynkells · 2017-03-30T14:35:15Z

Hi Daniel, thank you for your speedy reply! Was wondering if there a way to display html on the fly through a set of control characters? Similar to how DomTerm uses “\e]72;<html>\a” sequences ? https://lwn.net/Articles/670078/ From: Daniel Imms <notifications@github.com> Reply-To: "sourcelair/xterm.js" <reply@reply.github.com> Date: Saturday, March 18, 2017 at 4:47 PM To: "sourcelair/xterm.js" <xterm.js@noreply.github.com> Cc: cristynkells <cristynkells@gmail.com>, Author <author@noreply.github.com> Subject: Re: [sourcelair/xterm.js] how to view images in line (#614) Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax: // Add a link for a particular regex and handle when it's clicked. xterm.registerLinkMatcher(/\w\.png/, function (mouseEvent, imageFile) => { // imageFile is the text of the link }); — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Tyriar · 2017-03-30T17:55:37Z

@cristynkells there isn't right now, standard xterm doesn't support anything like that so if we did want to support it it would certainly need to be in an addon.

DonJayamanne · 2018-02-01T07:26:16Z

@Tyriar

it would certainly need to be in an addon

Is this supported today? Right now I don't see how this can be done even with an add on. Basically I'm trying to investigate what it would take to build something with inline plots (something we already have in other IDE, PTVS, RTVS, Spyder, and the like e.g. https://ipython.org/ipython-doc/3/interactive/qtconsole.html).

Tyriar · 2018-02-01T18:45:02Z

@DonJayamanne since this issue I've become aware of a standard some terminals have implemented for displaying images. Because of this we could add it to the core library, it's probably a pretty hefty feature though in terms of making sure everything is correct, particularly relating to getting the scroll bar right as right now there's no such thing as variable height rows in the viewport.

DonJayamanne · 2018-02-01T19:46:00Z

implemented for displaying terminals

Yes, I found iTerm does this.

Thanks for getting back.

ebraminio · 2018-02-26T21:03:03Z

Hi there. I opened a related issue on VSCode and redirected here. I wonder if supporting iTerm2 image feature is a possibility as it can be used as easy as this,

printf "\n\033]1337;File=;inline=1:`cat a.png | base64`\a\n"`

And I already used it on a project.

ztmr · 2018-02-26T22:07:15Z

@ebraminio If we are going to draw graphics and possibly some more hypertext, I would really like to be able to specify coordinates where to render it, not just blindly dump it under the current cursor position... check #1176.

rs · 2018-03-21T03:09:57Z

@ztmr: You can use other termcap commands to move the cursor and display the image wherever you want. That is what I do in jplot to redraw the image in place.

ztmr · 2018-03-21T10:08:56Z

@rs: Oh, of course, how could I forget about CUP! :o)

Tyriar · 2018-03-22T17:24:03Z

Opps this was labelled incorrectly. This would be a great feature to get a contribution for, here are some of my thoughts:

Do we need some pluggable way to support escape sequences?
Maybe this should live inside an addon?
Currently every single row is a fixed length, are images to be shown fixed to this grid as well or should the size be different? For example with row height=10px and image height=15px, should the image consume 15px or 20px on the terminal). If the latter we may want to tackle Smooth scrolling #1140 first.

jerch · 2018-04-15T19:35:07Z

+1 for custom escape sequences. Best place to start with those is to implement OSC and DCS subparser interfaces, where custom subparser can hook into the main parser. Also the main parser still has some escape code flaws and the test library still needs to be completed to cover all the basic sequences.

Btw DEC already had some terminal extensions for graphics and image loading (REGIS and SIXEL). No clue if it would make much sense to implement those, since the browser env of xterm.js has much better capabilities to draw such things.

jerch · 2018-05-10T12:58:53Z

SIXEL is supported by some programs (gnuplot) and some terminals (mlterm, xterm), there is even libsixel (https://github.com/saitoha/libsixel) for easy conversion. Seems not to be as dead as I thought...

Unsure about the OSC 1337 command, this seems to be supported by iterm2 only.

Edit: Also see jupyter console and https://github.com/liftoff/GateOne, they are both capable to show graphics, not sure though if they hack it around/on top of the terminal on a higher level. This might be a nice addon feature for xterm.js, still in doubt if this should be part of the terminal itself, as it raises many questions from data and state to output representation.

soccermitchy · 2018-05-22T21:16:36Z

From my point of view, Sixel support would be nice for a monitoring dashboard-style thing. Could easily show gauges and stuff in it - there are even browsers, iirc, that support rendering with it.

Tyriar · 2018-07-24T13:55:11Z

Here are some screenshot of how image placement/selection works in iterm2, not totally sure how the margins work yet.

dvag-sebastian-goeb · 2022-10-27T07:25:09Z

Any progress on this? #2503 seems to have been parked. I would really love to see inline images in VSCode especially (for Jest + Puppeteer + Jest-Image-Snapshot screenshot testing in my case).

markusbkk · 2022-11-20T07:27:01Z

Bump

I'd be interested in this, as well.

jerch · 2022-11-20T10:49:16Z

@dvag-sebastian-goeb @markusbkk
There is inline graphics support with xterm-addon-image addon (sixel only atm).

markusbkk · 2022-11-20T11:29:16Z

@dvag-sebastian-goeb @markusbkk There is inline graphics support with xterm-addon-image addon (sixel only atm).

I saw.

Unfortunately, that's not quite the same as iTerm2. I'm working on something that requires iTerm2 image protocol support.

blacktop · 2022-12-14T01:44:01Z

bump 🙏

jerch · 2023-02-04T22:13:55Z

@markusbkk PR for iterm IIP support: jerch/xterm-addon-image#43.

jerch · 2023-02-22T20:02:53Z

Just merged the IIP support. It passes all my basic tests, but since I have no mac to work with I cannot really compare it with iTerm and therefore labelled it as alpha. So if anyone wants to do some digging - a side by side comparison with iTerm's output from different images and imgcat settings would be really helpful.

wsshin · 2023-03-14T18:09:54Z

I landed here from the VS Code issue microsoft/vscode#65283. Could you describe how to test xterm-addon-image in VS Code? I am willing to test it on my Mac.

jerch · 2023-03-24T18:11:12Z

@wsshin Idk if vscode allows loading custom xterm.js addons - if not, you prolly have to pull source and add it there yourself.

Tyriar · 2023-03-24T20:24:29Z

@wsshin if you want to try test it out in vscode you can search the vscode codebase for xterm-addon-search which is an example usage of an addon from npm

trbot86 · 2023-03-29T06:45:35Z

thanks for the tip @Tyriar and the addon @jerch.
it was my first time even compiling vscode, but i managed to hamfist xterm-addon-image into vscode: microsoft/vscode@main...trbot86:vscode:xterm-addon-image

it seemed like i had to start with an old vscode commit (59faab44cd8287e247cfab8c389b9f0f44174bcf) to satisfy the xterm version dependency in xterm-addon-image. there's probably a better way to do this. i'm just a newbie to the code base, node.js, and typescript.

but man it's awesome to see it work!

jerch · 2023-03-29T10:46:30Z

it seemed like i had to start with an old vscode commit

This is prolly due to my rigid versioning policy in the addon:

  "peerDependencies": {
    "xterm": "~5.1.0"
  }

which basically tags addon releases to a certain xterm.js minor version (normally most recent one). This close addon entanglement to xterm.js versions is needed due to its heavy usage of private APIs (which also cannot be exposed by public API easily). This is somewhat unfortunate, but I dont know of a better way atm.
If you are sure it will work with current master (most of the time the last addon release can be carried forward until a crucial change in xterm.js happens), then you can raise the version to the next minor number of xterm.js (imho beta releases point to the next minor already, and vscode uses those).

There is a major catch though with vscode - in the past vscode used the writeSync method of xterm.js for certain tasks. If thats still the case, then you will have to disable IIP support. Background: The IIP protocol path uses an async sequence handler, which will create havoc on the terminal state, when used along with the sync write method. Since this is the case for all async handlers, writeSync got deprecated with a big warning several versions ago.

trbot86 · 2023-03-29T20:30:08Z

thanks for that remark. i haven't previously considered disregarding or updating the version requirement of xterm-addon-image. it turns out vscode:HEAD (specifically 10b2a8e42eddf7504f7b6e6b19d73eb5f888d075) compiles and works with xterm-addon-image, even though it's running xterm 5.2.x. i've updated my repo accordingly.

trbot86 · 2023-03-30T06:50:54Z

is there any chance of this making it into insiders? i hadn't previously used vscode OSS, and it's starting to set in how limiting it is... not having access to extensions renders it useless...

edit: i realize you can sideload extensions, but you can't evidently can't make remote-ssh work, and i do 100% of my work through the remote-ssh extension. archlinux guys have patches for this, but i'm trying to make windows OSS work, and there doesn't seem to be any workaround that i can see for the missing remote-ssh functionality.

edit2: if anyone wants to try to reproduce or improve what i've done, the head commit is whatever junk i've been messing around with, and might be broken. the "good" commit is trbot86/vscode@5799f44

jerch · 2023-03-30T10:57:07Z

@trbot86 Plz note that there are still some loose ends with the addon - most prominent the ability to serialize image data forth and back. So I am not sure, if it is ready for production rollout yet.
I am currently dealing with the serialization issue in jerch/xterm-addon-image#48. This is still in conceptual phase, as there is no prior art to learn from, so it will take a few more weeks to get things sorted out.

jerch · 2023-08-06T17:07:57Z

@Tyriar Should we close this issue, as the addon is in the base repo now?

trbot86 · 2023-08-06T18:26:08Z

I can't find a better issue in vscode (that isn't locked) to ask this in, but now that this is implemented in vscode OSS:main, how do we get this into insiders edition?

jerch · 2023-08-07T08:25:10Z

@trbot86 Isnt the insider edition built regularily from vscode's base repo? Because vscode's repo contains all the needed bits, so it should just work imho. Or do you mean something else with "insiders edition"?

Tyriar · 2023-08-07T13:14:46Z

@trbot86 images are now enabled by default in main, stable and insiders. See https://code.visualstudio.com/docs/terminal/advanced#_image-support

trbot86 · 2023-08-07T19:27:41Z

Thanks for the link. I had tested the feature without success, but the image I was testing was extremely large (~8k resolution), and it produced no output. It works when using a smaller image.

Do you happen to know offhand how vscode's sixel support is intended to handle images that are too large to fit in the terminal window? Trying to isolate whether the failure is in the program converting the image to sixels, or in vscode or xterm.js.

Edit: I also want to say thank you!! This is amazing! Should have said that from the beginning. :)

jerch · 2023-08-07T22:09:52Z

... sixel support is intended to handle images that are too large to fit in the terminal window?

@trbot86
Not sure if you mean in terms of memory usage and/or screen coverage:

memory
Since images grow rather big with higher resolutions and JS can only address up to 2GB - there a few limits in place to avoid OOM from exhaustive image usage by the addon:

pixelLimit - default is 16M (e.g. 4096x4096 squared), a sequence trying to allocate more pixels gets cancelled
sixelSizeLimit - default is 25MB, limits the max. size of a single sixel sequence, bigger sequences are again cancelled
iipSizeLimit - default is 20MB, same as sixelSizeLimit but for IIP sequences

screen coverage
Sixels are currently capped at 16K pixels in x-direction (truncated), but not actively capped in y-direction, only indirectly by pixelLimit. Thus it is possible to create an image with 16K x 1024 pixels up to an image with 1 x 16M (better dont try that, the browser engine prolly wont like such a canvas).
Sixel pixels are mapped 1:1 to CSS pixels, image data beyond the viewport to the right is preserved (can be "un-covered" by resizing), excess to the bottom will normally scroll the terminal window. Sixels dont support a resizing policy as IIP does.
Images from IIP have no additional hard caps, only pixelLimit. Furthermore IIP supports a resizing policy during image loading (e.g. width=50% to resize an image to 50% of the current viewport width).

Also see https://github.com/xtermjs/xterm.js/blob/master/addons/xterm-addon-image/README.md.

trbot86 · 2023-08-07T23:08:09Z

Thanks for the detailed answer. It sounds like I'm likely butting up against pixelLimit, as the image in question was something like 8000x5000 pixels. I'm not sure how much the sixel format inflates the PNG size, but the image was something like 2-3MB. I think sixels are perhaps quite wasteful, so I could conceivably also hit the sixelSizeLimit. Well, I can probably just avoid trying to show such large images. It would be nice, from a graceful degradation standpoint, to perhaps render a small image explaining the failure when a limit is hit, rather than the actual image. No big deal though. :)

jerch · 2023-08-08T00:41:46Z

8000x5000 pixels? Thats 160 MB in RGBA for just one image - no, you cannot expect this to work in an env with only 2GB address space. If you have hi-res output needs a terminal is prolly not the right tool for it.

If you want reasonable output & performance, then it is better to grab the terminals viewport size (e.g. by CSI 14 t) and zoom & tile into your big image on app side just sending the area of interest. Also sixel is not a good choice for that, as it is very costly to encode (needs quantization & dithering) and has poor output quality - IIP is def. the better choice here (better quality with bigger color space, better compression, much faster to generate).

It would be nice, from a graceful degradation standpoint, to perhaps render a small image explaining the failure when a limit is hit...

Thats not possible while maintaining the text grid - it would screw up output / screen state. Also other terminals have similar limits in place, so it is not surprising behavior.

trbot86 · 2023-08-08T01:03:44Z

Yes that's totally fair. It makes sense to slice or shrink large images before viewing them. Easy enough to add an imagemagick convert -resize step in a script.

Tyriar closed this as completed Mar 18, 2017

Tyriar added the type/question A question on how to use the library label Mar 18, 2017

DonJayamanne mentioned this issue Feb 7, 2018

Current support for data scientists is not enough microsoft/vscode#39492

Closed

DonJayamanne mentioned this issue Feb 26, 2018

iTerm2 like images displaying on the integrated terminal microsoft/vscode#44436

Closed

Tyriar reopened this Feb 26, 2018

Tyriar changed the title ~~how to view images in line~~ Support 1337 escape sequence to show inline images Feb 26, 2018

Tyriar added type/feature help wanted and removed type/question A question on how to use the library labels Mar 22, 2018

Tyriar added type/enhancement Features or improvements to existing features and removed type/feature labels Apr 4, 2018

jerch mentioned this issue May 8, 2018

new parser #1399

Merged

soccermitchy mentioned this issue May 22, 2018

Sixel support? vercel/hyper#2529

Closed

2 tasks

mme mentioned this issue Jul 23, 2018

Prototype of iTerm2 like image display #1577

Closed

Tyriar mentioned this issue Dec 18, 2018

Support sixels in integrated terminal microsoft/vscode#65283

Closed

davidanthoff mentioned this issue Jul 30, 2022

Feature request: inline HTML results (I'm happy to work on this) julia-vscode/julia-vscode#2982

Open

butlerx mentioned this issue Aug 17, 2022

Is inline image display a future possibility? butlerx/wetty#422

Closed

Tyriar mentioned this issue Jan 19, 2023

Add support for Sixels in integrated terminal microsoft/vscode#171666

Closed

This was referenced May 14, 2023

Terminal image support microsoft/vscode#182443

Closed

Remove experimental from terminal image support microsoft/vscode#182445

Closed

Tyriar closed this as completed Aug 7, 2023

Support 1337 escape sequence to show inline images #614

Support 1337 escape sequence to show inline images #614

Comments

cristynkells commented Mar 17, 2017 • edited

Details

Feature request

Tyriar commented Mar 18, 2017 • edited

cristynkells commented Mar 30, 2017 via email

Tyriar commented Mar 30, 2017

DonJayamanne commented Feb 1, 2018

Tyriar commented Feb 1, 2018 • edited

DonJayamanne commented Feb 1, 2018

ebraminio commented Feb 26, 2018 • edited

ztmr commented Feb 26, 2018

rs commented Mar 21, 2018

ztmr commented Mar 21, 2018

Tyriar commented Mar 22, 2018

jerch commented Apr 15, 2018

jerch commented May 10, 2018 • edited

soccermitchy commented May 22, 2018

Tyriar commented Jul 24, 2018

dvag-sebastian-goeb commented Oct 27, 2022 • edited

markusbkk commented Nov 20, 2022

jerch commented Nov 20, 2022

markusbkk commented Nov 20, 2022 • edited

blacktop commented Dec 14, 2022

jerch commented Feb 4, 2023

jerch commented Feb 22, 2023 • edited

wsshin commented Mar 14, 2023

jerch commented Mar 24, 2023

Tyriar commented Mar 24, 2023

trbot86 commented Mar 29, 2023

jerch commented Mar 29, 2023 • edited

trbot86 commented Mar 29, 2023

trbot86 commented Mar 30, 2023 • edited

jerch commented Mar 30, 2023 • edited

jerch commented Aug 6, 2023

trbot86 commented Aug 6, 2023

jerch commented Aug 7, 2023

Tyriar commented Aug 7, 2023

trbot86 commented Aug 7, 2023 • edited

jerch commented Aug 7, 2023 • edited

trbot86 commented Aug 7, 2023

jerch commented Aug 8, 2023 • edited

trbot86 commented Aug 8, 2023

cristynkells commented Mar 17, 2017 •

edited

Tyriar commented Mar 18, 2017 •

edited

Tyriar commented Feb 1, 2018 •

edited

ebraminio commented Feb 26, 2018 •

edited

jerch commented May 10, 2018 •

edited

dvag-sebastian-goeb commented Oct 27, 2022 •

edited

markusbkk commented Nov 20, 2022 •

edited

jerch commented Feb 22, 2023 •

edited

jerch commented Mar 29, 2023 •

edited

trbot86 commented Mar 30, 2023 •

edited

jerch commented Mar 30, 2023 •

edited

trbot86 commented Aug 7, 2023 •

edited

jerch commented Aug 7, 2023 •

edited

jerch commented Aug 8, 2023 •

edited