Support gettext-style language priority list for Minetest's built-in translation system

[y5nw]

Add compact, short information about your PR for easier understanding:

(GNU?) Gettext allows users to specify a priority list of languages (using the LANGUAGE variable) that is applied for translations. This can be seen in the screenshot below (using env LANGUAGE=zh_CN:de minetest).

The goal of this PR is to make Minetest's own translation system also support this feature.

I am partly also waiting for #14379 to avoid conflicts.

Note: This PR changes the language code for written Cantonese from yue_Hant to yue. However, considering the (from what I have seen) low translation coverage for this language, I do not think this change will be significant.

• Goal of the PR
  See title.
• How does the PR work?
  • The Translations class is extended to also hold language information in the lookup table.
  • The translation lookup tables are shared on the server-side to avoid allocating separate translation tables for (combinations of) different languages.
  • Translations parses the list of languages and looks up the translation for the languages in the list, returning the first one it finds.
  • The language setting uses GNU gettext's format of having a list of languages separated by a colon. This is parsed by the client as its language setting and by the server to send translations of multiple languages based on the preferences of the client.
  • Backward compatibility is kept:
    • When connecting to an older server from a newer client, the client only sends the first entry in the language list to the server. This allows the existing server-side language detection code to continue to work, but the client only receives translation files for the primary language.
    • When connecting to a newer server from an older client, the language string from the client is parsed as a list containing a single language.
• Does it resolve any reported issue?
  I can not find a relevant issue.
• Does this relate to a goal in the roadmap?
  Partially 2.2 Internal code refactoring.
  (The description of 2.3 UI Improvements does not seem to suggests that this PR relates to the stated goal.)
• If not a bug fix, why is this PR needed? What usecases does it solve?
  It allows players to specify a list of languages that can be used for translations. In particular, it allows users to specify zh_CN:zh_TW or zh_TW:zh_CN to use the other language variant.

To do

This PR is a Work in Progress.

• Extend Translations class to also hold language information.
• Implement priority list feature in the Translations
• Implement support for the priority list:
  • On the server side (particularly for minetest.get_translated_string)
  • On the client side
• Server:
  • Send translation files for multiple languages if this is needed by the client.
• Client:
  • Send priority list of languages to the server.
  • Send only the primary language to older servers.
  • Send priority list of languages to ContentDB (i.e. follow up Add support for ContentDB package translation #14410)
  • Handle different languages correctly when loading translation files.
  • Extend settings GUI to specify a priority list of languages (see implementation notes).
• Add testcases where appropriate
• Add documentation

How to test

• Server-side translations: covered by unittests
• Client-side translations:
  1. Use at least two languages where the primary language has a lower translation coverage than the secondary ones.
  2. Open up a (mod) formspec in a game.
  3. Note that the UI is translated in multiple languages.
     An example with the mail mod can be seen in the following screenshot:
     
• Compatibility with older servers: similar to above; note that
  1. Mod formspecs should be translated if a translation is available.
  2. Only the primary language is used. This is a limitation as older servers do not handle language priority lists.
• ContentDB: similar to testing with client-side translations, although this is quite subtle to notice as translation coverage for package meta is currently very low.
• Settings menu: manual testing. Note that the list should appear slightly different depending on whether the system language is used and the number of languages in the list.
• Android: in addition to the above:
  1. Add another language to the preferred language order.
  2. Start Minetest with the default language settings.
  3. Uncheck "Use system language".
  4. Note that the languages from the system settings are carried over to MT's language settings if the languages are supported by Minetest.

Implementation notes

• The language list setting is implemented using its own setting type, language_list. While this does make sense in that the language selector has certain cases that need to be handled in a special way:
  • The values are colon-separated entries from a specific list.
  • I have designed the formspec code to also show "invalid" entries.
  • The "empty list" should IMO be displayed differently to indicate that the system language is used. Currently, in addition to the list, there is a "User system language" checkmark that can be used to disable language selection.
  • However, this setting type is also very specific and unlikely to be used elsewhere. Thoughts?
• The LANG_CODE string has been replaced by a language list hardcoded (partly with CMake) into C++ as a relatively ugly macro. I would appreciate it if someone could suggest an alternative approach.
• I noticed that LANG_CODE is translated to (more or less) the same language string, with yue -> yue_Hant (written Cantonese) being the only exception.
• I have chosen to expose core.language_names only to the mainmenu API as I do not think it makes sense to export it to the regular/async environment when the value can vary between the server and clients.

[rubenwardy]

This is a feature I support, although in practice I should probably be reviewing other PRs first (like #13687). Any idea how much of a benefit this PR is? I assume a fair amount, as it allows priority lists like pt-br, pt, en

[y5nw]

This is a feature I support, although in practice I should probably be reviewing other PRs first (like #13687).

IMO the ideal solution would be using gettext directly for translation (e.g. by loading translation files using bindtextdomain) and deprecating the use of .tr files (or: converting tr files to a format readable by gettext). Ultimately the situation here is that Minetest has its own translation system that is not on the same level (in terms of features) as gettext.

Any idea how much of a benefit this PR is? I assume a fair amount, as it allows priority lists like pt-br, pt, en

I chose zh_CN:de for the screenshot since it makes the effect of the PR easily recognizable. From my own experience I would assume that zh_CN:zh_TW or zh_TW:zh_CN would be quite useful for some people as there do not appear to be many mods out there that cover both variants very well.