i18n at compile time

[robisim74]

At the state of the art, this library implements i18n only at runtime: this means that the translations, even if lazy loaded for each page, are downloaded at runtime during the SSR or on the client, and the lookup also happens at runtime.

Translations can be inlined at build time only using a localized router and the SSG offered by Qwik City.

After having an exchange of views with Miško Hevery, the main requirements of Qwik's i18n should be the following:

Qwik requirements

• extract/download translations at compile-time
• inline translations at compile-time
  • avoid lookup at runtime for static components
  • avoid 3 times strings: original string, key in translation map, the translated string

Example: angular/localize

Developers requirements

• translations also available at runtime (for texts coming from a server or processing, or for non-static components)
• translations also available outside the templates
• ability to change language at runtime without creating different apps
• different translations available during development (for layouts)
• fallbacks, handle missing translations

Example: qwik-speak

Customers requirements (not always, but recurrent)

• do not consume resources by publishing an app for each language
• support different routes for the same app or assign different domains based on location
• do not waste development time due to localization (build time or procedures too complicated for simple updates)
• translations must be in a simple format that can be easily shared with translators

Example: my customers...

Perhaps the list is not complete, but the main requirements should be clear: Qwik requires a compile-time solution that would inline the translated strings into the lazy loaded chunks that Qwik produces to avoid impact on the first loading time. At the same time, developers need in most cases runtime features to be compliant with the specifications or to satisfy customer requests.

This discussion aims to find the solution that satisfies the main requirements, which could be of any kind: a Vite plugin, or through middleware, or both, and create an integrated or alternative project to this with those interested.

[mhevery]

Thanks for such a great write-up! I would like to add a bit of context to better understand my reasoning.

(Disclosure: I was part of the design team on @angular.io/localize, so my opinions may be skewed.)

GLOSSARY:

• runtime i18n: No special compilation step is needed. Translations are loaded from JSON, and translation is done at runtime.
• compiletime i18n: A compilation step inlines translation into the code so that no loading is needed and no runtime lookup cost exists.

REASONING:

• Compile step is not free (conceptual, setup, integration etc...) For this reason, I think both options should be available to the developer.
  • compiletime i18n in production where runtime speed and size is important!
  • runtime i18n during development and testing.
• Runtime-only has these negative implications:
  • runtime translation bundle must be loaded before any code can execute. Such a bundle does not undergo chunking, and there is no easy way to do lazy loading. (Ideally, you would want to mirror the code chunks and translation chunks, but that would mean that you would have to build up a bundling system for translations that mimics the bundling of JS) For Qwik this is problematic as Qwik goes out of its way to lazy load a minimal amount of JS. Forcing a load of translation bundles for the whole application goes against the explicit goal of Qwik.
  • Secondary problems are runtime performance.
• Compile-only has issues negative implications:
  • In dev mode, you want to have a light build step. If we could skip i18n build step it would be ideal
  • Test runners often have custom build steps, we don't want to create a complex build system which needs to be integrated
• Ideal solution is runtime i18n in dev/tests but compile time in production.

How $localize solves this:

• The syntax is

  alert($localize`Hello World`)

  • In runtime mode $localize can get its values from JSON. This allows tests/dev to be fast and do not require any build step
  • In compile time mode the $localize acts like a marker telling the compiler how to replace the string. After replacement, nothing is left behind from the original string.
    • the compiler can run as a last step in the build pipeline. After treeshaker, optimizer and chunker. In this way the $localize gets lazy loading for free. Only the actually needed translations are loaded. It is fast because it has to parse the prod files only once, and generates all language outputs at once, adding only minimal cost to the build pipeline even if many translations.

NOTE: I am not saying that $localize is the solution. Only using $localize as an example of what kinds of properties I would like the final system to have to satisfy all of the requirements.

[robisim74]

I just tried @angular/localize in a Qwik app, and it works.

Steps:

• Add @angular/localize package
• Create a simple file like this:

import { component$ } from '@builder.io/qwik';

// Provide $localize without importing it directly:
// avoid breaking the $localize extraction
import {} from '@angular/localize/init';

export default component$(() => {
  return (
    <div>
      <h1>{$localize`Hello, Qwik!`}</h1>
    </div>
  );
});

• Add the following commands in package.json (just a simple example):

    "i18n-extract": "node_modules/.bin/localize-extract -s server/entry.express.js -f json -o public/locale/message.en.json",
    "i18n-translate": "node_modules/.bin/localize-translate -s entry.express.js -t public/locale/message.en.json -o server/{{LOCALE}} -r server",
    "serve.en": "node server/en/entry.express"

• Try the following commands:
  • dev mode: npm run start
  • prod mode:
    • npm run build
    • npm run i18n-extract (extract json files)
    • npm run i18n-translate (inline translations)
    • npm run serve.en (serve English app)

Still to try the other $localize APIs. But it seems like a good starting point...

[mhevery]

OMG! That is so exciting!

[robisim74]

I have to admit that a package like $localize might not have been suitable for an Angular SPA (where everything happens at runtime), but it has the right approach for a Qwik app.

The main problem in my opinion is that unlike Angular, a Qwik app also has a server which usually has only one entry point. Creating a localized file for each server is not possible without ad hoc development: server-side routing would be needed, and localized files should be controllers.

I got an idea.

It would be possible to inline all the translations into the server file in the following way:

children: $lang === "en-US" && "Hello Qwik!" || $lang === "it-IT" && "Ciao Qwik!"

($lang could be set as a global variable)

PROS

• One server file (reducing build and deploy time)
• No need to manually re-implement server side routing and controller logic
• (Optional) Ability to change language without redirecting to another link and reloading the page (more elegant)

CONS

• Larger server file (more languages, larger size)
• Execution of && and || operators at runtime (server side)

But there is also the problem of js chunks: the same technique could be applied or - if increasing the size of these files is not acceptable - they could be generated for each language:

- [lang]
    - build
      - chunk1.js 

and required by setting the q:base dynamically, for example:

const el = document.documentElement;
el.setAttribute('q:base', `${$lang}/build/`);

What do you think?

[mhevery]

I am a bit puzzled as to why you want to put all of the translations into the source file. Usually, the translation is extracted and set to a third-party for translation. The translator then returns the file, and requires merging. So while you could do what you described, in every organization I have been in the translation process, would not benefit from the above.

Also, the expression is error-prone to write and hard to read/test.

I think I would need to better understand the motivation behind this.

[robisim74]

Let me explain. I was describing the build process in production: last step in the build pipeline.

I'm assuming:

• development and testing: a runtime library like qwik-speak or tomorrow an official Qwik library for i18n
• production:
  • a new extract package to extract translation (like localize-extract)
  • a new translate package to inline translation (like localize-translate)

At the end of the current build process, we have the following situation (as an example I will use Express):

- server/
  - entry.express.js
- dist/
  - build/
    - q-chunk[1..n].js

If I use the localize-translate inlining logic, I will have to generate different files for each language.

• Client files (js chunks): this is not a problem (only those of the current language will be required)
• Server file (entry.express.js): this can be a big problem: usually there is only one entry point server side

So what I am proposing (at least for the entry.express.js server file) is to do the simulataneous inlining of all the translations (in the example I use $localize, but the function name could be passed as a parameter):

Extract of the file generated by the build of Qwik in production

return /* @__PURE__ */ jsxs("div", {
    children: [
      /* @__PURE__ */ jsx("h1", {
        children: $localize`Hello, Qwik!`
      }),
  });

Current behavior of localize-translate (one file for each language)

return /* @__PURE__ */ jsxs("div", {
    children: [
      /* @__PURE__ */ jsx("h1", {
        children: "Hello, Qwik!"
      }),
  });

New translate package (only one entry.express.js)

return /* @__PURE__ */ jsxs("div", {
    children: [
      /* @__PURE__ */ jsx("h1", {
        children: $lang === "en-US" && "Hello Qwik!" || $lang === "it-IT" && "Ciao Qwik!"
      }),
  });

This would allow nothing to change in the current build process (just add an extra inlining step, via a Vite plugin for example) and in the publishing process: the app would work exactly as it was developed without any extra development costs.

[robisim74]

Is coming: #11

[robisim74]

Qwik Speak v0.1.0 contains the Vite plugin for inlining translations at compile time. More info here

The goal is to make it stable and make it grow even more.

The extraction tool will arrive soon.

[robisim74]

Inline and Extract tools are available!

Inline features:

• Vite plugin: translations are inlined during the production build (no extra steps)
• Inlining happens for both server file and client files (chunks) sent to the browser
• Change locale re-rendering only components that uses translations without redirecting or reloading the page
• Ability to generate chunks once for each language (requires redirecting or reloading the page)
• Mixed solutions: inlining and runtime translations if needed (examples: meta titles and descriptions, server responses etc.)

Extract features:

• Cli command, also available programmatically
• Use of default values if provided
• Merge with existing languages and translations
• Scoped and self-explained properties

The library is about 8kB before tree shaking and minification, and the library and tools (that are apart) have no external dependencies.

[robisim74]

To improve performance, the changeLocale function has been removed and it is no longer possible to change locale at runtime without navigating/reloading the page (SPA or MPA is indifferent):

• SpeakLocale is no longer reactive
• QwikSpeakProvider no longer watches for locale change and no longer reloads files

PROS

• No subscriptions are created from the library
• Translations are loaded only during SSR and inlined in chunks sent to the browser (one for each language, making them much lighter)
• Only translation data needed at runtime is serialized

See Qwik Speak Inline Vite plugin for more details.

[sitaramyadav]

Is there any more steps required for parser for the syntax like

const text = someVarable?.?child.available ? "" : "auto_add_charging_description";

  <MyComponent text={t(text)}

instead this syntax.

`const text = someVarable?.?child.available ? "" : t("auto_add_charging_description");

  <MyComponent text={text}`

[robisim74]

From v0.17.0, the inlining of translations in js chunks occurs not only in prod mode but also in dev mode (in memory). So you can have the same app behavior in both dev and prod, and detect any problems immediately.