Encodings in JavaScript

[bpj]

Sorry if this is a noob question. I'm not at all up to date with JavaScript...

I'm trying to port a filter which I wrote for the Lua implementation to JavaScript. The idea is to take symbols which look like a decimal (331) or hexadecimal (0x14b) number and replace them with a string containing the Unicode character whose codepoint the number is. What I don't know is whether I need to convert the string to UTF-8 myself, and if so how, or will it be taken care of automatically? In Lua >= 5.3 there is the utf8.char function, and writing a function myself in Lua is not beyond me but it seems JavaScript is so UTF-16 oriented that it doesn't even have a function to convert a number to a byte.

[matklad]

You might be look for String.fromCodePoint?

> String.fromCodePoint(0x2764)
'❤'

[bpj]

Yeah, but that gives me a UTF-16 encoded string doesn't it? Do I need to do anything to get UTF-8 in the output, and if so what?

[matklad]

You can than subsequently encode the resulting string into whatever encoding you need using the TextEncoder API:

const encoder = new TextEncoder();
const view = encoder.encode("€");
console.log(view); // Uint8Array(3) [226, 130, 172]

https://developer.mozilla.org/en-US/docs/Web/API/Encoding_API

Though it feels like an XY problem perhaps? In JS, you are expected to just use strings and mostly not care about their encoding.

Perhaps this will help: in Lua, like in C or Zig or Python2, strings are just sequence of bytes. Lua as a language is encoding-agnostic, but you, as a programmer, had to care about not confusing encodings fairly often.

In JavaScript, strings are fairly opaque objects, and are distinct from raw byte arrays: String vs UInt8Array. This is a design similar to Java and Python3. In JavaScript, the underlying string encoding is mostly opaque. There’s usually no encoding confusion, because bytes and strings are just different types — there’s no way to forget to decode a string.

So, when using djot, just use JavaScript strings throughout and ignore encoding. I think the only place where you might need to care about encoding is for converting internal offsets into user-visible columns. Although even there, you care about grapheme clusters and not encoding per se.