magic width modification considered harmful

[mintty]

Several sections assume that width of a grapheme cluster can be modified in magic ways:

6.1: variation selector 16 (VS16) that may have caused the width of the grapheme cluster to change to wide (2 grid cells)

6.2: Emoji symbols are always rendered in ... 2 grid cells.

6.3: VS16 ... will force the grapheme cluster’s width to be 2

This is a serious problem in terminals. The traditional way for applications to know the width of its own output is based on the locale mechanism, addressed via function wcwidth/wcswidth. While this method has limitations, esp. in a remote login scenario, no other reliable method has been established.
So I suggest the reference for actual width assumption should be the width given by these functions, even taking some unpleasant visual consequences into account. How else could an application know how wide its output actually appears? Look at https://unicode.org/reports/tr51/#Display :
The width of the pirate flag 🏴‍☠️shown there depends on the actual availability of its emoji in the current environment (font or platform).
This information is inaccessible to a terminal application.
(The Unicode specification does not really address terminal display.)
So applications like editors would be confused and frequently display garbage.
The mintty terminal goes another way: It takes the locale width authoritative and adjusts emoji display to the width thus determined.
This can mean that single-width emojis appear squeezed into a single cell and that emoji sequences can take up to 8 character cells. For some compensation in style preference, mintty offers placement options to align, center, or expand the emojis to their consumed space.
While, as said above, this may not be pleasant, it is the only way I see to achieve consistent screen handling and avoid garbage caused by unforeseeable positioning. The only alternative would be that an application requests cursor position reports after every questionable grapheme output which would slow down screen output considerably.

Also in particular

6.2: ... cursor will always move by 2 grid cells.

contradicts traditional cursor behaviour and would result in unreliable position assumptions. Cursor position inside a multi-column glyhp needs to be maintained somehow. Output of text there would then break the previous glyph (also for non-emoji glyphs like East Asian).

[j4james]

This can mean that single-width emojis appear squeezed into a single cell and that emoji sequences can take up to 8 character cells.

This was the one of the main reasons for creating this protocol in the first place: terminals want a way to display things like the pirate flag nicely without breaking backwards compatibility, and that's why there is a mode for it. Apps that expect widths to be calculated with the old wcwidth/wcswidth algorithm would simply not set mode ?2027, and everything should work exactly as it's always done.

But if a terminal claims to be able to support mode ?2027, and an application requests that support, then they should be able to rely on the terminal honoring that request in a predictable manner. If a particular font doesn't support the pirate flag emoji then worst case you just display a question mark or whatever fallback you'd use for any other missing glyph. The key thing is that you use the correct width calculation so that the layout of the page doesn't break.

If you don't think this is a good idea, then just don't support this mode, and carry on doing your own thing.

[mintty]

OK, 2 points:

1. The description should make a clear statement about the intention and that it describes a mode that breaks compatibility with traditional terminal width handling.
2. A complete and consistent definition of string widths should be aspired.
   Example:
   U+1F468 (man) has width 2
   U+1F680 (rocket) has width 2
   Their cluster U+1F468 U+200D U+1F680 (astronaut) shall have width 2 in your model - understandable.
   But what about, let's say U+1F468 U+200D U+1F680 (two men). There's no emoji specified for it right now, so if I understand correctly your model would leave it two separate emojis, width 4.
   Now what if the next Unicode version introduces that missing emoji, so the width changes from 4 to 2?

[j4james]

I didn't write the spec, and I'm not overly familiar with all the details, so I'm not the best person to answer these questions. However, regarding your second point, I would have thought a combination like U+1F468 U+200D U+1F680 should be interpreted as a single emoji, even if it isn't yet defined as such, or your font doesn't have an appropriate glyph to render it.

If an application has requested mode ?2027, and then deliberately used a zero-width joiner between two characters, I think that's a pretty good indication that they're expecting those characters to be joined together, and would expect them to occupy the same space as a single emoji. And that assumedly solves your problem of what happens when Unicode introduces a new combination emoji.

But again I'm not an expert on this subject - hopefully somebody else will chime in here with a more authoritative answer.

[mintty]

Having checked all emoji sequences, it seems that the issue could be resolved by two simple rules:
Text output while "Unicode width mode" is enabled will have the following width-rendering modifications:

1. Appending U+FE0F as a combining character changes any character to double-width.
2. Zero-width joiner U+200D forces the subsequent character to also be treated as a combining character (thus not add any width).

Both rules would need to be applied to any character sequence, regardless of whether it has an emoji definition, and regardless of whether it has a glyph in the current font. So the following resulting examples should be checked:

• "a U+FE0F" would render as an expanded double-width a.
• something like "a U+200D b" which now renders as ab would render single-width with undefined appearance.
• The "enclosing keycap" sequences would only render double-width if also combined with U+FE0F (both variations are listed as emojis by Unicode).

It's unclear whether additional variation selectors might affect width as recently discussed in https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/9.
Note that for wider attention to this specification proposal, I'd suggest to raise it as an issue in that Terminal Working Group Specifications.