toLocaleUpperCase() not working for Georgian locale ('ka')

[sowmyav24]

• Version: v10.7.0
• Platform: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
• Subsystem:

toLocaleUpperCase() does not work properly for Georgian locale.

v10.7.0 and above

> 'იანვარი'.toLocaleUpperCase();
'ᲘᲐᲜᲕᲐᲠᲘ'

Expected behaviour

Till node v10.6.0

> 'იანვარი'.toLocaleUpperCase();
'იანვარი'

[vsemozhetbyt]

FWIW:

> [...'იანვარი'.toLocaleUpperCase()].map(ch => ch.codePointAt().toString(16))

[
  '1c98', // GEORGIAN MTAVRULI CAPITAL LETTER IN  (U+1C98)
  '1c90', // GEORGIAN MTAVRULI CAPITAL LETTER AN  (U+1C90)
  '1c9c', // GEORGIAN MTAVRULI CAPITAL LETTER NAR (U+1C9C)
  '1c95', // GEORGIAN MTAVRULI CAPITAL LETTER VIN (U+1C95)
  '1c90', // GEORGIAN MTAVRULI CAPITAL LETTER AN  (U+1C90)
  '1ca0', // GEORGIAN MTAVRULI CAPITAL LETTER RAE (U+1CA0)
  '1c98'  // GEORGIAN MTAVRULI CAPITAL LETTER IN  (U+1C98)
]

cc @nodejs/v8 @nodejs/intl Is it regression or fix?

[targos]

Was there an ICU update between 10.6 and 10.7?

Edit: yes: 122ae24

[targos]

The ICU 62 changelog says:

The Unicode 11.0 changes may also require some code/tests to be fixed. Notably:

1. Word break now groups white space together.
2. Segmentation in general simplifies tests for emoji sequences.
3. Casing behaves differently for Georgian, and differently for that than for any other script.

[targos]

And the Unicode 11.0.0 changelog says:

Casing Issues
Casing behavior for the Georgian script has changed significantly. There is a new set of Mtavruli capital letters (U+1C90..U+1CBA, U+1CBD..U+1CBF) in Unicode 11.0, with case mappings to the existing Mkhedruli letters (U+10D0..U+10FA, U+10FD..U+10FF). In prior versions of the Unicode Standard, Mkhedruli Georgian was considered a monocameral (non-casing) script, and the Mkhedruli Georgian letters were gc=Lo. Starting with Version 11.0, those Mkhedruli Georgian letters are now gc=Ll, and have uppercase mappings to Mtavruli Georgian capital letters. This change will have major implications for Georgian implementations, including changes for input methods, fonts, casing, and string matching. Existing implementations have treated Mtavruli headlines and other uses for textual emphasis as a text style, so there will also be significant issues for document conversion and upgrade.

Another complication for Georgian is that the primary orthography does not use titlecasing, and the Mkhedruli Georgian letters do not have titlecase mappings to Mtavruli letters. This is unique among bicameral systems in the Unicode Standard, so casing implementations should be prepared for this exception.

[vsemozhetbyt]

Should we treat such changes as semver-major?

[targos]

I don't know. Is it something that should be fixed upstream in V8?

/cc @nodejs/intl

[marwahaha]

Hi, I maintain https://github.com/moment/moment , and our builds are breaking because of this issue. Any idea if/when it will be fixed?

[srl295]

@marwahaha what is the breakage?

@targos @marwahaha it's not clear what you mean by 'fixed'. What do you see is the bug here?

[srl295]

But, this is after downloading a font that supports the new characters: https://app.box.com/s/psnogufec39aq486uny1o300j7c6hk76/file/317278048647

[srl295]

https://www.unicode.org/mail-arch/unicode-ml/y2018-m07/0063.html discusses this some.

Here's the problem. The vast majority of Georgian fonts do not yet have the new uppercase characters. So when any system uses case mapping to uppercase text (e.g. browsers interpreting CSS’s text-transform: capitalize), then the users of Georgian will see boxes (“tofu”) if the font they are using does not have the glyphs.

See a site for example https://bpgfonts.wordpress.com/download/

[srl295]

@vsemozhetbyt

Should we treat such changes as semver-major?

Traditionally we haven't. Functionality isn't removed.

as an example, /\p{Emoji}/u.test('🛹') ( is Skateboard an emoji?) will also return true for 10.7.0 and false for 10.6.0. (I tried to find a similar case for node 8.9.0 / 8.10.0 which added Unicode 10, such as t-rex 🦕 but \p{Emoji} was not turned on then. There probably is some similar example i could do with regex.

(Edit: I don't have a font with the skateboard yet, either.)