Web Translation API

[domenic]

こんにちは TAG-さん!

I'm requesting a TAG review of Web Translation API.

Browsers are increasingly offering language translation to their users. Such translation capabilities can also be useful to web developers. This is especially the case when browser's built-in translation abilities cannot help, such as:

• translating user input or other interactive features;
• pages with complicated DOMs which trip up browser translation;
• providing in-page UI to start the translation; or
• translating content that is not in the DOM, e.g. spoken content.

To perform translation in such cases, web sites currently have to either call out to cloud APIs, or bring their own translation models and run them using technologies like WebAssembly and WebGPU. This proposal introduces a new JavaScript API for exposing a browser's existing language translation abilities to web pages, so that if present, they can serve as a simpler and less resource-intensive alternative.

• Explainer (minimally containing user needs and example code): https://github.com/WICG/translation-api
• User research: none
• Security and Privacy self-review: https://github.com/WICG/translation-api/blob/main/security-privacy-questionnaire.md
• GitHub repo (if you prefer feedback filed there): https://github.com/WICG/translation-api
• Primary contacts (and their relationship to the specification):
  • Domenic Denicola (@domenic), Google, spec-writer
• Organization/project driving the design: Google Chrome
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://chromestatus.com/feature/5172811302961152

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): unknown. The Web Machine Learning W3C Working Group seems to explicitly rule out these sorts of high-level APIs in its charter. Maybe that can be amended though.
• Existing major pieces of multi-stakeholder review or discussion of this design:
  • Web Translation API WebKit/standards-positions#339
  • Web Translation API mozilla/standards-positions#1015
  • Web Translation API explainer 2024-05-07 > 2024-06-06 w3c/i18n-request#231
• Major unresolved issues with or opposition to this design:
  • It's unclear whether including translation from an unknown source language is worth including, or whether we should require web developers to do a two-step detect-language + translate language process. Explainer discussion, Should we include translation with an unknown source language? WICG/translation-api#1
  • The exact way in which language tags should be handled, especially to give good interoperability, is still unclear. Explainer discussion, Language tag handling WICG/translation-api#2
  • Naive implementations that give the site direct access to info about which language packs are downloaded can provide information that fingerprints the user. Various mitigations are possible but it's unclear which strike the best balance or are most feasible. Explainer discussion, Preventing fingerprinting via detecting the presence of downloaded language packs WICG/translation-api#3
  • We might want to expose some information about whether the translation is on-device or not, and about the model version. But this could also be harmful for interop. See explainer discussion at the bottom of the "Goals" section.
• This work is being funded by: Google

[shivaylamba]

I like this idea. And especially giving the capability to the user to choose between either using on-device or cloud based models. Given how performant and efficient on-device AI models have become using WASM/WebGPU, we can definitely vouch for the translations to take place while also minimizing risk of any data being sent to cloud.

[jasonmayes]

I think having an auto detect option could be really cool. Scenario: Imagine you are in a Google Meet video call and everyone is from a different country. This way, you could make a Chrome Extension that auto detects the language and is able to translate for each person speaking even if language changes between detection sessions. That would be super useful.

It would be good if you are thinking of hyrbid approach for the programmer to explicitly require offline on device model inference here in case there are privacy aspects they need to adhere to for their application. I think 3 options are useful:

1. client side (forces on device only - if cant run for some reason just fails with error to capture)
2. hybrid (browser chooses based on network conditions / device capabilities) and uses the right one in the moment
3. server side (forces server side only)

[shivaylamba]

I think having an auto detect option could be really cool. Scenario: Imagine you are in a Google Meet video call and everyone is from a different country. This way, you could make a Chrome Extension that auto detects the language and is able to translate for each person speaking even if language changes between detection sessions. That would be super useful.

It would be good if you are thinking of hyrbid approach for the programmer to explicitly require offline on device model inference here in case there are privacy aspects they need to adhere to for their application. I think 3 options are useful:

1. client side (forces on device only - if cant run for some reason just fails with error to capture)
2. hybrid (browser chooses based on network conditions / device capabilities) and uses the right one in the moment
3. server side (forces server side only)

I think we can also recommend the user based on their hardware, the type of model inference to be used (cloud vs on device). For instance, if the user's hardware doesn't have a GPU and has limited RAM, it might be better suited for cloud inference. Let me know what do you think @jasonmayes