Hashtag translation Russian to English: # becomes ♪

[olets]

When translating hashtags from Russian to English, the # becomes a ♪

Russian:

#Docker

English translation:

♪Docker

Example: https://lor.sh/@skobkin/111977814820972935 as viewed from Hachyderm

[PJ-Finlay]

This is an eccentricity of the Transformer language model we use for translation not an issue with the LibreTranslate code.

This isn't the first time ♪ symbols have been generated incorrectly. I'm guessing the root cause is the OpenSubtitles dataset which is used for traning. OpenSubtitles has a lot of translated subtitles for movies and TV shows and I'm guessing a lot of ♪ characters too.

I used to filter certain special characters from the dataset before training the models, which might help with this issue, but I removed the filtering a while ago because it was slow.

[LynxPDA]

@olets Can you tell me which version of the model has a bad translation? With the latest version RU-EN v1.9 from the index I could not reproduce it.

The update was from this thread:
https://community.libretranslate.com/t/new-argos-model-en-ru-and-ru-en/872/20

[olets]

It's whatever version Hachyderm is running. Will see if I can find out