merge some qwik-i18n ideas

[wmertens]

There's 2 things that qwik-i18n does better IMHO:

1. template strings. E.g. t`Hi there ${name}!` instead of t('app.greet$$Hi there {{name}}!', {name})
2. no context needed, so translations work anywhere, also inside server logging

For 1, would it for example be possible to simply convert t`Hi there ${name}!` internally to the key "tmpl.Hi there {{1}}!" with the default value "Hi there {{1}}!" and params {1: name}?
Template strings are always static so they will always be inline-able.

For 2, qwik-i18n uses the qwik-core-builtin getLocale() to know what the current locale is, and it uses the global scope to store translations.
The global scope is fine for translations because they are the same for everyone. In browsers you only need a subset, never a different set for the same keys+locale, so it's safe.

It would be great if qwik-speak could adopt these features.

[robisim74]

Hi @wmertens, I've looked at the plans you have for qwik-i18n.

When I started this library, Miško had not yet made public his idea of using $localize for Qwik i18n. Check out this thread: #8

Indeed, I think I was the first to try $localize on a Qwik app.

At that time, I had already implemented the key syntax t('key') as in the major internationalization libraries. @@[default value] after the key is just an option, useful for extracting without adding a new parameter.

This syntax is more verbose, but has some advantages:

• Developers don't always have the basic texts ready, but often have to use keys or placeholders
• It is possible to use dynamic keys, which are known only at runtime such as response codes from a server
• Support multilingual pages (you can specifiy a different language for a single key)
• You can create scoped files and keys, making it much clearer to both developers and translators

Especially the lack of dynamic keys ($localize ids), have always been a major issue when using $localize in Angular (for direct experience and for the open questions on SO and Angular repo).

In any case, about your questions:

1. It would be easier inline template syntax compared to key syntax (for which I had to build a special parser: https://github.com/robisim74/qwik-speak/blob/main/packages/qwik-speak/tools/core/parser.ts): converting would not be needed, since the template syntax can be used directly. More complicated would be to integrate it into scoped translations: https://github.com/robisim74/qwik-speak/blob/main/docs/extract.md
2. no context: i tried to do without the context, but i couldn't! getLocale is not enough, because:

• the locale in qwik-speak is a complex object (with lang, currency etc.)
• in production, chunks sent to the browser are inlined (as in $localize), but here there are dynamic keys. Dynamic keys in runtimeAssets are serialized and sent to the browser, so no translation files are loaded on client (https://github.com/robisim74/qwik-speak/blob/main/docs/inline.md). Global scope would not be suitable, because it would have to be populated both on the server and on the client (as $localized in dev mode, if I remember well)

The library is now relatively stable (a lot depends on Qwik changes), and I know that several devs are using it for their projects.

I'm open to new ideas and contributions, but without creating other breaking changes for those using it.

[wmertens]

0. First of all, I do think your library is very nice 🌟, sorry for not being clear! The extra features can indeed be very useful.

1. Do you mean that t`...${...}` already works?
   As for the scoping, how about making all translation strings in templates that are missing $$ be keys in the scope tpl? Personally I probably won't need scopes so that would be fine for me.

2. But you can register the complex locale as a string in a global, and the getLocale then refers to that (or encodes the whole thing in the locale string with extensions)?
   I don't understand why global scope (or module scope, in any case a singleton) wouldn't work.
   On the client, when you receive dynamic translations, you add them to the singleton.
   On the server, all the translations are already loaded for all locales, in the singleton.
   The only difference is that the client needs to load the needed translations.

I think these features would be possible to add without changing the existing API, no?

[wmertens]

BTW, FYI, here's what I was considering building before I found out that qwik-speak already has a vite plugin. As you can see the ideas were very similar to what you built.

---

vite-plugin-static-i18n

This statically generates translated copies of code bundles, so that you can serve them to clients as-is.

Pro:

• 0-runtime in client
• All keys are known

Con:

• changing language means reload
• must tell client to load js from locale dir

How it works

In the server and in dev mode, all translations are loaded into memory eagerly, but for a production client build, all the _`x` calls are replaced with their translation.
You can also use the helper functions to implement dynamic translations.

types

type I18nKey = string
type I18nOrdinal = string
/** a string matching /([^$]|\$[0-9$])*/ */
type I18nTranslation = string
/** The locale JSON file format */
type I18nPlural = {
  locale: string // the locale key, e.g. en_us or en
  name?: string // the name of the locale in the locale, e.g. "Nederlands"
  translations: {
    /** Note that the object translation only works when used with `plural` */
    [key: I18nKey]: I18nTranslation | {
      [tag: number | I18nOrdinal | "*"]: I18nTranslation
    }
  }
}

lib

init(options?: {getLocale?: () => string})

Must be called before the first translation happens. Ensures that the locales are loaded into memory.
In production client builds, this is removed.

getLocale will be used to retrieve the locale on every translation. It defaults to defaultLocale.
For example, use this to grab the locale from context during SSR.

setLocale(locale: string)

sets the default locale at runtime.

localize`str` or _`str`

translate template string using in-memory maps

_`Hi ${name}!` converts into a lookup of the I18nKey "Hi $1". A literal $ will be converted to $$. Missing translations fall back to the key.

Nesting is achieved by passing result strings into translations again.

_`There are ${plural`${boys} boys`} and ${plural`${girls} girls`}.`

plural

translate template string using in-memory maps but vary based on the first interpolation. Translation is of the form

{
	"0": "There are none",
	"1": "There are some",
	"2": 1,
	"3": 1,
	"*": "There are many"
}

but if a string is given it falls back to the _ replacement

localize(key: I18nKey, ...params: any[]) or _(key: I18nKey, ...params: any[])

Translates the key, but this form does not get statically replaced with the translation.
It is your duty to call loadTranslations so the requested keys are present.

makeKey(...tpl: string[]): string

Returns the calculated key for a given template string array. For example, it returns "Hi $1" for ["Hi ", ""]

translate(translation: I18nTranslation | I18nPlural, ...params: unknown[])

Perform parameter interpolation given a translation string or plural object. Normally you won't use this.

guessLocale(acceptsLanguage: string)

Given an accepts-language header value, return the first matching locale.
If the given string is invalid, returns undefined.
Falls back to defaultLocale

defaultLocale: readonly string

Default locale, defaults to the first specified locale.

locales: readonly string[]

e.g. ['en_US', 'fr'].

names: readonly const {[key: string]: string}

e.g. {en_US: "English (US)", fr: "Français"}

vite-static-i18n/locales

The vite plugin will populate this import with an array of the locale data (I18nPlural[])

vite plugin

• pass locales to vite plugin
• during build:
  • transform server source code:
    • create missing json locale files
    • replace init() with init(locales, [await import '~/../i18n/locale1.json', ...])
    • output missing keys into all json files
    • complain if init() call was missing
  • transform client source code:
    • remove init call if any
    • replace __$I18N_LOCALES$__ and __$I18N_LOCALES$__ with literals in the library code ``
    • replace calls of localize and _ with the "global" __$LOCALIZE$__(key, ...values).
    • replace calls of plural with plural(__$LOCALIZE$__(key), ...values)
• after build for client:
  • copy bundle to each locale dir, replacing __$LOCALIZE$__ calls with the resulting translation or just the original
  • Unchanged files are symlinked to the parent dir copy (if not on windows?)

JSON

Translation maps are objects with keys based on the interpolation string. Each variable converts to $#, and $$ stands for a literal $.

They are stored under ~/../i18n/$locale.json. When new keys are encountered, the JSON is amended.

Format: I18nPlural

Questions

• how to copy bundle in vite

• how to detect the imports

• how to replace the original calls

• how to replace the replaced calls after build

• helper for qwik routing, what API?

  • I18n links use _ for the href

  • entry.ssr.tsx:

    export default function (opts: RenderToStreamOptions) {
    	return renderToStream(<Root />, {
    		manifest,
    		...opts,
    		base: extractBase, // determine the base URL for the client code
    		// Use container attributes to set attributes on the html tag.
    		containerAttributes: {
    			lang: opts.serverData!.locale,
    			...opts.containerAttributes,
    		},
    	})
    }

  • calling locale() inside layout.tsx

[robisim74]

Hey you are doing a great job! ⭐ I'll need some time to evaluate it and evaluate possible integrations.

In the meantime, I'll answer your questions above:

Do you mean that t...${...} already works?

Already no, I meant it wouldn't be hard to add that syntax to the Inline plugin I made.
The Extract command, instead, would be very different because you need to generate the keys, I think. The missing scopes, however, are already provided in the current command, they would not be a problem.

you can register the complex locale as a string in a global, and the getLocale then refers to that (or encodes the whole thing in the locale string with extensions)?

I really don't like this solution: the Qwik locale must contain the current language (then also used by this library) and nothing else in my opinion: it could create other problems.

The only difference is that the client needs to load the needed translations.

Yes, I think it's a big difference! even when dynamic keys and runtimeAssets are used, no files are loaded on the client: the library thus has zero impact on the first load (inlining + serialized dynamic keys)

In any case, even if we were able to remove the context, it would create a big breaking change: since version v0.14.0 I have already required a great effort from those used the library: pass to use* functions and comply with Qwik requirements (before there was a simple $translate function, like $localize)

[robisim74]

Just an addition: I am very happy with your proposal (and you are a great person for having done it): I also think that having a single i18n library for Qwik is a great thing.

However, if it won't be possible to merge, feel free to use what I have done (it is open source, and under the MIT license), or to ask for advice. I think diversity is a wealth, so even different libraries using different ways to solve the same problem would be a wealth!

[robisim74]

Hi @wmertens,

I did some tests: I think that implementing and maintaining also template syntax would be very expensive (especially for handling the messageId).
However, suggestions on how to do this without breaking changes would still have my attention.

As for no context, I have already explained above that I don't have an alternative solution: the current context contains the Speak locale, the loadTranslation$ custom function and the runtime assets, so serialization is necessary if you don't want to load things twice (on the server and on the client).

By the way, I see that $localize has been added to the Qwik commands: https://qwik.builder.io/docs/integrations/i18n/#internationalization-libraries.
So I think that if a user wants to work with template syntax he can use $localize, if instead he wants to work with key syntax (even dynamic ones) he can use Qwik Speak.

[robisim74]

@wmertens I finally found a way to make the new inlineTranslate function work without context (safe translations in the global scope) and using Qwik's getLocale. Unfortunately due to this issue, I have yet to ask developers to use a return function:

const t = inlineTranslate();

t('title@@Qwik Speak')

At least, this ensures that getLocale is available, even if the function is then invoked by props.

[wmertens]

I wonder if you had a look at https://github.com/wmertens/compiled-i18n?tab=readme-ov-file#compiled-i18n - maybe we can still combine to have a single library after all.