Design of Japanese Dictionary/日语字典的设计问题

[ghost]

In this project, we plan to develop a two-part Japanese dictionary to support Japanese singing.
Due to the lack of linguistic expertise, we hope you can participate in this project and provide us with help and support.
The following link is a tentative spelling dictionary. Please reply your valuable comments. Thank you.

---

在本项目中，我们计划开发一个两段式日语字典，以支持日语演唱。
由于语言学专业知识的缺失，我们希望您能够参与到这个项目中来，为我们提供帮助和支持。
以下链接为暂定的拼写字典，请回复您的宝贵意见，谢谢。

---

japanese_dict

[yqzhishen]

Linguistic principles of a proposed dictionary to be formally supported in the main branch

• Its syllables should be standard spelling or phonetic transcriptions (like pinyin in Chinese and romaji in Japanese) for easy integration with G2P modules.
• Its phonemes should cover all (or almost all) possible pronunciations in that language.
• Every syllable (in the leftmost column) and every phoneme (in the right column) should have one, and only one certain pronunciation, in all or almost all situations in that language. Some slightly context-based pronunciation differences are allowed as the networks can learn.
• Most native speakers/singers of that language should be able to easily cover all phonemes in the dictionary. This means the dictionary should not contain extremely rare or highly customized phonemes of some dialects or accents.
• It should not bring too much difficulty and complexity to the data labeling workflow, and it should be easy to use for voicebank users.

How to submit your own proposal or suggestions

• Read the above principle and the documentation for customized phoneme dictionaries.
• Point out which syllable or phoneme names, lines or rules in the dictionary should be modified, and how.
• Explain your reason to do so, and the corresponding use case or examples if there are any.
• (Recommended) Attach the modified version of the dictionary according to your suggestion.

[funnymdzz]

is it compatible with NNSVS?

[yqzhishen]

Can you first read the dictionary before asking questions, please?

[haru0l]

I noticed "yi" became "i" in line 40, is there any particular reason for that or was it an error?

Minor comment regarding the phonemes ty, dy, my, py, fy that is used in the language purely for borrowed words from other languages, so some users may not have those included if they chose to only record folk songs for their datasets.. (for example, the dataset of Namine Ritsu only have it included by recording "add-ons")

Apart from that, it should be good! (I added glottal stops and vocal fries as a separate phonemes due to porting datasets and wanting more control over the datasets, but I think these can be controlled through pitch context)

[haru0l]

If the user provides songs that are standard modern pop songs the borrowed words will likely be available.

For cl, if you don't mind breaking the usual rules of labeling Japanese, I think it's fine to replace it with SP as it will likely learn the context (I believe it came from TTS labeling anyway.) Merging the phonemes would be possible as well but I presume the AI will learn the context.

For yi, I believe the usual way to spell it would be ゆぃ, but that usually translates itself to "yui". It might be better to just remove it as neither the dataset of Namine Ritsu has it nor NNSVS uses it.

As for "hu" and "fu", it might not be best to combine it as there would be borrowed words that uses it (ex: "who are you" could be transliterated as "hu a ryu" in Japanese without explicit English support). Also, from what I understand during training, it should be fine to include it anyway since preprocessing seems to look into the phonemes itself rather than the words. (ex: if you have tye in your dictionary but not in your dataset, but you have tya in the dataset, it would not become a problem.)

[ghost]

Thank you for your answer. I intend to delete "yi" from the dictionary and replace "cl" with "SP"
"Hu" will change to "h u". The spelling of the left column of the dictionary does not need to be completely covered, such as "tye", so "hu" and "fu" can be marked with the actual pronunciation. If the pronunciation of "hu" does not appear in the dataset, I guess the model can use phonemes to combine "hu" by itself.
I also want to confirm that "zi → j i" and "ti → ch i" are correct?

[haru0l]

"ti → ch i" and "zi → j i" might not work sometimes due to it being associated loan words again 🤔, a lot of the time this is completely dependant on the user... qwq

[ghost]

The spelling on the left side of the dictionary does not need to be completely covered, just ensure that the dataset contains all the phonemes on the right side.
The test on the oniku database was initially successful. Now we are trying to train the MFA model to automatically align annotations.

[haru0l]

I see! Hopefully it all goes well 👍

[oxygen-dioxide]

Why do we need a different dictionary even if there are already NNSVS ones?

• The NNSVS Japanese dictionary uses kana as syllable names, which may cause encoding issues in our toolchain, so we want a fully-ascii dictionary that use romaji as syllable names.
  • We will keep phoneme symbols unchanged as possible, since there are many existing NNSVS voicebanks.
  • Kana will still be supported in GUI editors such as OpenUtau.
• We are planning to train an MFA model and use it in voicebank development, which need a unified phoneme set. For the function of MFA and how we use it, see section "About MFA" below.
  • Users can remove or add phonemes when developing their own voicebank, but custom phonemes won't be supported by MFA.

so here are the main goals of this proposal:

• Provide a reasonable and unified romaji for each syllable in Japanese, and how to express them in phoneme symbols
• Finalize a unified phoneme set which contains all (or almost all) phoneme in Japanese, but without rare or highly customized phonemes (such as those specified for cross-language tuning).

About MFA (and our current voicebank development workflow)

MFA (Montreal Forced Aligner) is a tool for performing forced alignment of speech datasets. With MFA, we only need to input lyrics for each sentence of the song, without having to label the exact timing of each phoneme. In other words, we don't need to make NNSVS .lab and ust files for our dataset manually. Here is our voicebank development workflow:

• Record songs
• Input lyrics for each sentence
• Use MFA to label phoneme timings automatically
• Train diffsinger model

Though there are official pretrained MFA models for Japanese, it isn't trained with singing voice datasets, and gives bad results on singing voice. So we're going to train a new MFA model with singing data.

For rare or highly customized phonemes: Since different user define these phonemes differently and we may not collect enough data for these phonemes, we will not support them in our MFA model. However, you are still free to add them into your voicebank, but you'll have to manually label phoneme timings for the sentences containing these phonemes.

[blueyred]

One benefit of auto labelling is it sort of "self heals".
I labeled an English dataset using whisper to transcribe and MFA for labelling.
Whisper occasionally transcribed some text "incorrectly" where the pronouciation for a word wasn't standard ("ball" rather than "boy" in the specific instance) this was actually beneficial as MFA labelled up the the sound "correctly" and the model got trained on the right sound rather than from the ground truth of the song lyric.