Stop losing translations when modifying

[Kebap]

Brief summary of issue / Description of requested feature:

We recently did some minor updates to strings, or even comments on strings. After uploading them to Crowdin, all recently done translations on these strings are gone. They should stay intact or at least give option to decide.

Steps to reproduce the issue / Reasons for adding feature:

1. Upload files to Crowdin, start translating
2. Modify translated strings in github source
3. Review Crowdin. The translations are gone!

Error output / Expected result of feature

Crowdin Knowledge Base explains this:

Updating Source Files
If some of the source strings were modified, the system shows a dialog with a list of those strings. You’ll be able to choose which existing translations you want to keep without changes or delete, and whether you want to keep or remove approvals.

That section specifically talks about manually updating a file via Crowdin website.
How can this dialogue start, when the updated files arrive via github integration?

Extra information, such as Mudlet version, operating system and ideas for how to solve / implement:

Example spanish changes from collection

The Spanish translation "so" is gone. In this case, the source string did not even change, but only the string's comment. In other cases, a string was very long (~100 words) and only 1 word changed, the rest stayed unchanged. Of course, much of the translation is still valid, so it should not be deleted like this.

The translators can still see "so" as a suggestion, if they take the effort to click through every single string again (including those who were not yet translated), but the suggestion is between other suggestions and not even marked as "this has been the translation of this very string before"

[Kebap]

Any idea how to remedy? @crowdin

[vadi2]

Where do you draw the line between keeping or discarding a translation?

To me putting the translation back in TM seems the best, it is very easy to pull it out again and the translator can decide if they want to re-use the text or not.

[Kebap]

Exactly, that line needs to be drawn individually for every string, like the screenshot from Crowdin suggests.

I would welcome an option for translators to start that popup dialogue after pulling strings from github.

Right now, they can't even distinguish, which strings had been translated previously, to quickly fix a pull.

I don't feel it's reasonable to spend hours at every modification, researching which translations got broken.

[SlySven]

When using Qt's own Linguist (or rather lupdate) to update a bunch of language specific .ts files from the source code directly, lupdate has an option:

  -no-obsolete
          Drop all obsolete and vanished strings.

Only if that option is supplied does that cause unused, or obsolete - which is how I think a change in comment is being treated by CrowdIn - strings to be purged from the .ts files. Somehow we need to get CrowdIn to behave like that for us. 🙏

Importantly I do not think that a change in a comment (though possibly a change in a disambiguation is different) should be clearing the translation - it might be acceptable for it to lose "approved" status but not for it to be forgotten.

[Andrulko]

Hi everyone,

Please be aware you can save translations for modified strings when updating the file over CLI / API / GitHub as well, by specifying update_option parameter in the configuration file (crowdin.yml) stored in your repo.

More info:
https://support.crowdin.com/configuration-file/#changed-strings-update

After you do the change, please Pause & Resume the integration in Crowdin to apply changes

Hope that's exactly what you need!

[vadi2]

Thanks for the response!

@Kebap which option would you like?

[Kebap]

Thanks guys. This is awesome. Will review options in detail.

[Kebap]

I'd like to try the option "update_as_unapproved", this seems to be a good compromise: Translations stay intact, but lose their approved status.

Now, when I tested this, it did not seem to work. Strings still lost their translations. I even went so far as to enhance the automatically created brief layout of the configuration yaml file, so as to reflect exactly the example given in the Knowledge Base section linked above.

Still, the Crowdin has seemingly unfortunately merely replaced the strings, instead of keeping their translations and just removing the approval. Am I doing it wrong?

String previously had translation and approval (approval seems invisible in .ts file?)

Update option "update_as_unapproved" was tested with short and long yaml layout, both to the same result

After modifying string (or comment), Crowdin now shows modified string without translation and approval

Crowdin Diff reports strings as "deleted and added" not "kept but lost their approval"

[crowdin]

-- Please reply above this line -- Hi everyone, It seems the problem is that in your project you files are .html and .ts. The files of such type don't have a clear KEY:VALUE structure, therefore when you update the files, every changed string are considered as new strings. It is the expected behavior of the system, but I will ask devs if there is something that can be done in this case once they are in the office! How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

…

-- Sincerely, Olga Kuhta Customer Success Manager Links: ------ [1] https://secure.helpscout.net/satisfaction/204306672/record/1818790773/1/ [2] https://secure.helpscout.net/satisfaction/204306672/record/1818790773/2/ [3] https://secure.helpscout.net/satisfaction/204306672/record/1818790773/3/ On Sat, Sep 8, 2018 at 1:34:23 EEST, Mudlet/mudlet <reply@reply.github.com> wrote:

I'd like to try the option "update_as_unapproved", this seems to be a good compromise: Translations stay intact, but lose their approved status. Now, when I tested this, it did not seem to work. Strings still lost their translations. I even went so far as to enhance the automatically created brief layout of the configuration yaml [1] file, so as to reflect exactly the example given in the Knowledge Base [2] section linked above. Still, the Crowdin has seemingly unfortunately merely replaced the strings, instead of keeping their translations and just removing the approval. Am I doing it wrong? String previously had translation and approval (approval seems invisible in .ts file?) [3] Update option "update_as_unapproved" was tested with short and long yaml layout, both to the same result [4] After modifying string (or comment), Crowdin now shows modified string without translation and approval [5] Crowdin Diff reports strings as "deleted and added" not "kept but lost their approval" [6] - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [7], or mute the thread [8]. Links: ------ [1] https://github.com/Kebap/Mudlet/blob/crowdin-test/.crowdin.yml [2] https://support.crowdin.com/configuration-file/#changed-strings-update [3] https://user-images.githubusercontent.com/117238/45245753-12b3a300-b2fe-11e8-819a-fbc1cab389cd.png [4] https://user-images.githubusercontent.com/117238/45245828-632b0080-b2fe-11e8-8457-d16f53e62976.png [5] https://user-images.githubusercontent.com/117238/45245723-e7c94f00-b2fd-11e8-831d-14f3c0151aa8.png [6] https://user-images.githubusercontent.com/117238/45245651-843f2180-b2fd-11e8-8744-b431244e39e8.png [7] #1961 (comment) [8] https://github.com/notifications/unsubscribe-auth/AA0k1tqzDWWwb2qGudFk9ENRg7Hm3D-Hks5uYvRcgaJpZM4WeaRq

[Kebap]

Yes, Qt IDE uses .ts files. Surely this is no niche. Any recommendation from Crowdin developers?

Feedback from Qt was inconclusive:

(Kebap) Hi all! Has anyone ever used a web-based service website for handling translations and translators? We use Crowdin and they seem to can't work well with Qt .ts file format. They seem to think every modified string is a new string and will delete old translations. See following thread for details. How can we solve this? #1961
(frkleint) Kebap: I would recommend posting to the http://lists.qt-project.org/mailman/listinfo/localization mailing list
(Kebap) Seems like that mailing list is about translatin Qt project itself. However maybe there are other people building their own project with Qt and translating to other languages?
(frkleint) Kebap: Yes, but the right maintainers/people will read it

[Andrulko]

Let me double check all details, @Kebap. I have one "plan B" in my head, need to test it out though.

Would it be ok for you to store the source text / translations in the element of .ts file? In the you can have a unique string ID for example

[SlySven]

At present we import/upload one file mudlet.ts which contains the source text to CrowdIn and they processes that to produce a mudlet_xx_YY.ts where xx is the two letter language code and YY is a two letter country code. The originating mudlet.ts is, I believe, generated/updated with the Qt lupdate utility which I thing is run from our source ./translations directory as lupdate -locations absolute ../src/mudlet.pro -ts ./mudlet.ts {at least from a *nux OS}.

There is an alternative but I do not know if CrowdIn can work like that in which we supply the individual .ts files (currently:

• mudlet_de_DE.ts
• mudlet_el_GR.ts`
• mudlet_en_GB.ts
• mudlet_es_ES.ts
• mudlet_fr_FR.ts
• mudlet_it_IT.ts
• mudlet_nl_NL.ts
• mudlet_pl_PL.ts
• mudlet_ru_RU.ts
• mudlet_zh_CN.ts
• mudlet_zn_TW.ts

) files to CrowdIn and have it/the translation team working on those. This would then mean that the existing translation efforts are kept within each .ts file - this is actually how Qt envisages translation to be done - because the lupdate will then update each individual translation file with the changes from the code sources when it is run but will not discard (unless explicitly told to with the -no-obsolete argument) old texts that no longer appear in the sources. The downside to this is there there is not a single source file for all translation which might confuse/not work with the CrowdIn system. The upside is that #1963 immediate becomes a non-issue as we can generate the plurals-only mudlet_en_US.ts file and just include it in the upload to CrowdIn on release/version/whatever change...

[vadi2]

I think Crowdin requires a single file as input. I remember that during setup it didn't like multiple files - namely, the problem was being the input and output file being the same.

Though we could just name the input and output files differently?

[Kebap]

Crowdin can certainly handle multiple input files, as in different strings to translate. However in SlySven's plan, they would all have identical contents. That would mean, even Polish translation team would see all the files including mudlet_it_IT.ts and mudlet_ru_RU.ts, etc. Hence we did not continue down that road much further and instead opted for one central translation file.

edit: The explanation for creating .ts files is correct, but the real command used is: lupdate -recursive .\src\ -ts .\translations\mudlet.ts

[SlySven]

Not sure what you mean there Vadim, but I think Kebap is on the same track - we could input all the files but how do we then say/tell CrowdId that only the mudlet_ru_RU.ts of the set should be shown to the Russian(Russia) translators?

[Andrulko]

@vadi2 You're right, it's recommended to upload a single source file, Crowdin will generate translated files itself.

Please don't upload files like mudlet_ru_RU.ts into the project.

Probably we can setup a call to discuss that furthermore? Please reach me at andriy(at)crowdin.com

I have one idea in mind that will work for sure, but need to hear if you will be happy with it. The thing is that .ts don't have unique identifiers for each string - like in .po, files where msgid is source & identifier at the same same, <source> is text & identifier as well (along with <context> and <name> elements, each string is considered as unique and if you modify <source>, the string is considered as new and you can't keep translations for modified strings in result).

Anyway, there is a pretty good solution/workaround I'd like to discuss/demonstrate to your team ;)

[SlySven]

... and only some content would be the same - the texts from the sources - but the translations already done for each locale are also stored in their respective file, and are present in each following update-translation cycle.

[SlySven]

If I understand it the unique message identifier scheme is also allowable in the Qt system but it is harder to work with - and a change mid-project is a non-trivial task (the two systems are mutually exclusive and you need a very good methodology to come up with meaningful identifiers) - and part of the benefit of the existing scheme is that duplicate strings do get merged into a single common translation, it is just that changing one needs the others to change as well if they are intended to be the same...

[vadi2]

When we translate all strings in a language to 100% and then edit a few in a release, wouldn't it still be very easy to go through and re-add them back thanks to TM?

This problem only seems like a real problem because we've yet to reach 100%.

[v3ctorman]

@vadi2 in case you want to have ability to automatically translate newly added strings with help of TM, you can set up such workflow using Advanced Workflows feature (I've just enabled it for your account)
https://support.crowdin.com/advanced-workflows/

[vadi2]

Thank you, appreciated :) we will study this.

[SlySven]

@Kebap wrote:

edit: The explanation for creating .ts files is correct, but the real command used is: lupdate -recursive .\src\ -ts .\translations\mudlet.ts

-recursive is the default argument case and isn't necessary - and so is the -locations absolute for new files apparently... 🙂 - I was just finding this out whilst checking to see what effect the problems lupdate still has with C++11/14 raw string literals {QTBUG, #1310} and whether strings are being lost from the mudlet.ts file as a result of them confounding it... 😦

[vadi2]

@Kebap Is this still an issue?

[Kebap]

Yes, even though not as bad as before the change. But it is still confusing, even to the initiated, as you can see by the linked issue from a week ago.

Also there is a bit of a race currently while most languages are not yet near 100%, will they translate strings faster than they loose translations again?

What is more, I found an issue which seems to bring back deleted translations. Currently I find no way to delete a translation from the Crowdin. After next update, it will have been added automatically again.

[Kebap]

@Andrulko Do you have any updates here? You were mentioning a plan B above..

Also, why does Crowdin do this weird behavior in TM now!? See screenshot below for comparison.

All that was changed by us here is h3 into a. No other tag was touched. Now what happened?

Would you expect translators to replace every {[=-lt;-=]}h2{[=-gt;-=]}{[=-lt;-=]}u{[=-gt;-=]} into <h2><u> or rather <0> manually?

Link to example: https://crowdin.com/translate/mudlet/137/en-de
Sceenshot of example:

[vadi2]

I think that's a bug in Crowdin's TM. Probably best to report as a separate issue to them.

[Kebap]

I have informed them here as well: https://crowdin.com/contacts

[Olena1234]

Hello, we have already checked your request and replied to your email. Please let us know if you have any other questions!