Localization of scikit-learn project website

[steppi]

Hi everyone,

I work for Quansight Labs and am helping with a CZI supported project to support translation and localization of the brochure websites of core scientific Python projects. By "brochure website" I mean the general information from the core project website, as distinct from things like technical documentation and tutorials. For scikit-learn, I mean https://scikit-learn.org/stable/. You may have already seen language selection option at the top right of https://numpy.org.

The goal is to translate the websites of at least 8 core Scientific Python projects into at least 3 widely used languages, making this content available to a wider audience. I've been asked why machine translation services such as google translate are not sufficient, so will preemptively answer that:

1. They can struggle with the nuances of the jargon and technical language used, and have been observed to produce inaccurate and misleading translations
2. Aside from translation quality, at the moment, getting a seamless experience forces users to depend on closed, proprietary solutions, counter to the open-source ethos.

Machine translation is being used as a productivity-multiplier for translators and reviewers though.

I'd like to gauge whether scikit-learn maintainers are interested in participating, and if so get the ball rolling towards making this happen. I understand well how busy open source maintainers can be, and the difficulty of securing time to work on a seemingly endless stream of tasks. The hope is that a cross-functional team including employees from Quansight together with volunteer translators and reviewers could take on much of the burden, minimizing the effort needed from core project maintainers themselves. So far I've reached out to Pandas, where a user had already made a relevant issue on their GitHub. I'll continue reaching out to other core scientific python projects over the next week.

Some more information:

Translation Management Platform

For the numpy.org, Crowdin as a translation management platform. It was chosen for the quality of its UI and ease of set up. Crowdin has generously granted us a free open source enterprise account with support. Support has been excellent so far. Over the past 6 months I've familiarized myself with setting up translation workflows with Crowdin and can help you set up a workflow for scikit-learn.

What to translate

A decision needs to be made to determine what should be translated and what should not. You may want to click through https:://numpy.org to see the extent of what was translated there. I think some good candidates for scikit-learn are the installation instructions, the getting started section, the FAQ, and some of the other sections under the "More" drop-down

I think technical documentation like the User Guide, Examples, and the API reference are out-of-scope though, at least for now. I think the reward to effort ratio just isn't there for such a large set of things which can experience frequent changes.

What to expect

I can help set up a proof of concept Crowdin workflow on my own fork of scikit-learn to show you what that would look like. Since you are using .rst, you could use the typical sphinx internationalization workflow generating .pot files, which could be uploaded to Crowdin for translation, which would then allow download of .po files with translations. There are varying levels of automation that could be set up with Crowdin that we discuss. Once the .pot files are in-place, Sphinx makes it straightforward to build the documentation for different languages. I haven't looked into a solution for adding a language selector drop-down to Sphinx documentation like what we for numpy.org, but I'll take care of that. If this translation plans go forward, myself and/or colleagues from Quansight would help take care of finding qualified and interested translators.

Please let me know if you have any questions or concerns. I'm just trying to get things started off. I've probably missed something and would be happy to share more details.

[glemaitre]

Let me relay this discussion on discord. We have more developers looking at the traffic there.

[steppi]

Thanks @glemaitre

[reshamas]

Sharing some resources on Crowdin from Turing Way:
https://the-turing-way.netlify.app/community-handbook/translation/translation-getting-started.html?highlight=translation

[steppi]

Thanks @reshamas. These look very helpful.

[reshamas]

To share, we set up some translations for the Data Umbrella / PyMC sprint website (ES/PT):
https://pymc-data-umbrella.xyz/

We used Transifex, which worked fine for our site.
https://github.com/pymc-devs/pymc-data-umbrella

From discussions on local forums, it does seem that the community is moving towards using Crowdin.

cc: @OriolAbril

[reshamas]

Including here the list of current translations:
https://scikit-learn.org/dev/related_projects.html#translations-of-scikit-learn-documentation

Translation’s purpose is to ease reading and understanding in languages other than English. Its aim is to help people who do not understand English or have doubts about its interpretation. Additionally, some people prefer to read documentation in their native language, but please bear in mind that the only official documentation is the English one [1].

Those translation efforts are community initiatives and we have no control on them. If you want to contribute or report an issue with the translation, please contact the authors of the translation. Some available translations are linked here to improve their dissemination and promote community efforts.

Chinese translation (source)
Persian translation (source)
Spanish translation (source)
Korean translation (source)

[steppi]

Thanks! I wasn't aware of these. It's very cool so much exists already. It looks like these have all been inactive for two or more years, but still very useful.

[steppi]

Hi everyone, how does the following idea sound? I'd like to get started setting up repositories that mirror the content from each core project website, and sync these to Crowdin. The maintainers of each project can later decide whether they'd like to publish the translations directly on their webpage. Crowdin has an open source request form that will need to be filled out for each project in order to add it to Crowdin Scientific Python enterprise organization without paying. If you give me permission, I can fill out the form for you .

[glemaitre]

Could you open an issue to bring more attention to the topic. I will be easier to collect feedback from all maintainers.

[steppi]

Sure, will do. It didn't fit any of the issue templates and I saw this

but I guess it's one of the exceptions.

[glemaitre]

Yep here, we are in a specific case :) Sorry for the misdirection.

[steppi]

No worries