scanf doesn't support UTF-8 on Windows

[vtereshkov]

The C runtime on Windows supports UTF-8 in printf(), but not in scanf():

scanf doesn't currently support input from a UNICODE stream.

(https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/scanf-scanf-l-wscanf-wscanf-l?view=msvc-170)

printf() example:

fn main() {
   s := "Привет, мир!"
   printf("%s\n", s)
}

C:\Users\vtere\Desktop\umka-lang\umka_windows_mingw>umka.exe ..\test.um
╨Я╤А╨╕╨▓╨╡╤В, ╨╝╨╕╤А!

Split from #108.

[luauser32167]

C:\Users\vtere\Desktop\umka-lang\umka_windows_mingw>umka.exe ..\test.um
╨Я╤А╨╕╨▓╨╡╤В, ╨╝╨╕╤А!

I think you need to change the codepage to utf8 (65001)

> chcp 65001
> C:\Users\vtere\Desktop\umka-lang\umka_windows_mingw>umka.exe ..\test.um

You can also do this with the windows function SetConsoleOutputCP(65001).

Here is some info on windows utf8 from null program (Chris Wellons): 1, 2

[vtereshkov]

Fixed printf() with SetConsoleOutputCP(CP_UTF8). It still fails on Chinese characters, but at least prints Cyrillic correctly.

[vtereshkov]

I fixed Chinese by changing the console font to SimSun-ExtB. Cyrillic characters look weird after that, but it's a decades-old font rendering problem:

[skejeton]

@vtereshkov we need to be careful with Windows state, Umka may very well be embedded into an application that uses a different encoding, SetConsoleOutputCP(CP_UTF8) works globally.

I think the best solution is to just bite the bullet, and use the widechar functions and convert them to UTF-8 back and forth.

[vtereshkov]

@skejeton Perhaps you're right, but the benefit-to-cost ratio is low, so I'm reluctant to do anything with it now.