Simplified Chinese rather than traditional?

[lucasjinreal]

whisper ../../data/test_data/16k16bit.wav --language Chinese --model tiny
100%|█████████████████████████████████████| 72.1M/72.1M [00:09<00:00, 8.15MiB/s]
[00:00.000 --> 00:04.800] 黃維至立於把數字世界帶入每個人每個家庭每個組織
[00:04.800 --> 00:19.800] 購劍萬物互聯帶智能世界

this is not very accurate.

[jongwook]

The tiny model doesn't perform well on many non-English languages. I'd recommend using the medium or large model for better performance in Chinese.

Regarding the simplified/traditional distinction, we used a single language code zh for all varieties of Chinese, although there can be different dialects/languages/styles of the transcripts. I'd expect the training data had both simplified and traditional scripts, and the model should be capable of transcribing in both.

For example, by prompting in a certain style (i.e. --initial_prompt with a sentence that may have come before what's spoken in the audio), you can steer the model to transcribe the same audio file in either traditional or simplified Chinese, as well as to write with punctuations:

$ whisper --language Chinese --model large audio.wav
[00:00.000 --> 00:08.000] 如果他们使用航空的方式运输货物在某些航线上可能要花几天的时间才能卸货和通关
$ whisper --language Chinese --model large audio.wav --initial_prompt "以下是普通話的句子。"  # traditional
[00:00.000 --> 00:08.000] 如果他們使用航空的方式運輸貨物,在某些航線上可能要花幾天的時間才能卸貨和通關。
$ whisper --language Chinese --model large audio.wav --initial_prompt "以下是普通话的句子。"  # simplified
[00:00.000 --> 00:08.000] 如果他们使用航空的方式运输货物,在某些航线上可能要花几天的时间才能卸货和通关。

[ErzhongHoo]

works for me！

[wqw547243068]

Does that mean the following code with Python?

prompt='以下是普通话的句子'
result = model.transcribe(audioFile, task='translate',language='zh',verbose=True,initial_prompt=prompt)

[vopbs]

微型模型在许多非英语语言上表现不佳。我建议使用中型或大型型号以获得更好的中文性能。

关于简体/繁体的区分，我们对所有中文变体使用单一的语言代码zh，尽管可以有不同的方言/语言/抄本风格。我希望训练数据既有简化脚本也有繁体脚本，并且模型应该能够在这两种脚本中进行转录。

例如，通过以某种风格提示（即--initial_prompt在音频中所说的内容之前可能出现的句子），您可以引导模型以繁体或简体中文转录相同的音频文件，以及用标点符号：

$ whisper --language Chinese --model large audio.wav
[00:00.000 --> 00:08.000] 如果他们使用航空的方式运输货物在某些航线上可能要花几天的时间才能卸货和通关
$ whisper --language Chinese --model large audio.wav --initial_prompt "以下是普通話的句子。"  # traditional
[00:00.000 --> 00:08.000] 如果他們使用航空的方式運輸貨物,在某些航線上可能要花幾天的時間才能卸貨和通關。
$ whisper --language Chinese --model large audio.wav --initial_prompt "以下是普通话的句子。"  # simplified
[00:00.000 --> 00:08.000] 如果他们使用航空的方式运输货物,在某些航线上可能要花几天的时间才能卸货和通关。

I used your whole method, and I found a problem that he knew the initial beginning, which was different from the original traditional Chinese meaning.

[WorkAlbertX]

It works!Thank you very much!

[pto2k]

Thank you!
That works and I also noticed that, the punctuation in the initial_prompt changed formating of the transcription.

I wonder if there are other things initial_prompt could affect?

--initial_prompt "以下是普通话的句子"
[00:00.000 --> 00:05.000] 我有一首歌词 题目字母 歌词要求原创
[00:05.000 --> 00:10.000] 这个歌词根据你的能力可长可短
[00:10.000 --> 00:13.000] 参与对象是所有的同学
[00:13.000 --> 00:16.000] 那么我申请了一个班级的邮箱
[00:16.000 --> 00:20.000] 这个邮箱的七个字母分别是
[00:20.000 --> 00:22.000] 风华初手字母
[00:22.000 --> 00:26.000] 然后是起航中队手字母
[00:26.000 --> 00:28.000] 便于大家去记
[00:28.000 --> 00:33.000] 风华初起航中队这七个字的手字母大家可以记一下
[00:33.000 --> 00:37.000] 那你的电子稿一定要写好名字和学号
[00:37.000 --> 00:41.000] 便于我们收集的同学进行统计
[00:41.000 --> 00:46.000] 歌词上交的时间是1月31日之前

--initial_prompt "以下是普通话的句子。"
[00:00.000 --> 00:13.000] 一首歌词,题目字母,歌词要求原创,这个歌词根据你的能力可长可短,参与对象是所有的同学。
[00:13.000 --> 00:28.000] 那么我申请了一个班级的邮箱,这个邮箱的七个字母分别是风华初手字母,然后是启航中队手字母,便于大家去记,
[00:28.000 --> 00:43.000] 那你的电子狗一定要写好名字和学号,便于我们收集的同学进行统计。歌词上交的时间是1月31日之前。

[Francoyy]

I think it is very strange that the initial prompt is the solution to fix the Simplified VS Traditional Chinese.
It does not make too much sense to me for a 2 reasons:

• Only the character 話/话 is different. The rest of the sentence is identical. Why do we have to type the whole sentence then? If the goal is to place a context, then only 1 character should be enough? like --initial_prompt "話". Are we free to write whatever we want in this prompt? Also I want to note that there are many characters that are identical in both Simplified and Traditional. So some sentences would look exactly the same. Example 今天很冷 is the same in Traditional and Simplified. So it forces the user to choose carefully characters that are not identical between Simplified and Traditional Chinese.
• In Taiwan where Traditional Chinese is used, nobody says 普通話 to talk about Mandarin. People say 中文 or 國語. Therefore it's very strange to me that you have to say --initial_prompt "以下是普通話的句子。" to mean "I want traditional Chinese", in a sentence that nobody says when in a traditional chinese context.

I think that the correct way to solve Traditional/Simplified should be to use --language and choose one of the many [zh-hk, zh-cn, zh-sg, zh-tw] just like defined in the ISO 639-1 standard language codes.

It's not just a small variation.
Simplified VS Traditional is not a 1:1 mapping, it's a 1:n mapping.

Many traditional characters are all written the same in Simplified. Therefore if Simplified is outputted, it's not possible to programmatically change to traditional character. It depends on the context of the sentence.

[Francoyy]

More here: https://en.wikipedia.org/wiki/Ambiguities_in_Chinese_character_simplification

[Francoyy]

Example, if you see the simplified character 斗, in equivalent Traditional Chinese it could be either 斗 or it could be 鬥.
How do we know which one? A mapping algorithm wouldn't know. It needs to understand completely the context of the sentence in order to do that. And Whisper happens to be the tool that actually understands contexts of sentences.

[phineas-pta]

i understand your concerns but whisper was fed with badly formatted train data from the beginning, to fix it the way you want requires as i said finetune or re-train

AI is only as good as the data we put into them

implementing any language processing with whisper will just break it, we'll never manage to cover all edge cases for any language

[jongwook]

On the first question -- the model is not instruction fine-tuned like ChatGPT was, so we want to "engineer" the prompt to resemble the sentence that would've come before the input audio to steer the text outputs. Yet we don't want to constrain it too much on certain topic or vocabulary, or in a strange mode where there was a single letter "話" in the line preceding the audio. That was the motivation for using the generic sentence above. I don't speak Chinese (although I recognize some of the letters) and I used Google Translate to come up with the prompt, so that's why the phrase sounds unnatural. It's just one example prompt that worked for the example audio files that I had, but please feel free to try different prompts, which may work better!

[no1xsyzy]

Simplified vs Traditional is not even 1:n, it is sometimes n:m, e.g.
"to offend" is 干/干 , voicing gan53
"dry" is 干/乾 , voicing gan55
"sky" (occult usage) is 乾/乾 qian35

[BackMountainDevil]

In language option, there are zh and Chinese. As vopbs said "关于简体/繁体的区分，我们对所有中文变体使用单一的语言代码zh".

So what is the diff between zh and Chinese?

And it is quite smart. I use whisper --model large-v2 -output_dir audio/srt --output_format srt --verbose False --language zh audio/audio.m4a to extract word. And it knows my spoken is zh-cn while it knows Huang Zhizhong uses zh-tw. In my example, I use zh as language, so should I use zh or Chinese when I want to output zh-cn? or just add --initial_prompt "以下是普通话的句子。"

[LeeHaha314]

@Francoyy You got a good point, but actually the function of initial prompt is not just to switch between simplified/tradtional mode. Since both simplified and traditional texts are included in the whisper training dataset, the model should be capable for learning the n:m mapping in traditional Chinese (if there are enough patterns in the dataset).

[Francoyy]

Maybe i’m not explaining well enough, but you have to trust me on this.
The same exact sentence spoken by someone can be written in Simplified and Traditional and there is no right or wrong. Both would be absolutely correct.
My videos are for the Taiwan market therefore I want to have subtitle displayed in Traditional Chinese. If my videos were for China, I would want the same subtitles written in Simplified. Maybe it is more clear that way?
So it’s not about whether Whisper can detect if the subtitles should be in Simplified or Traditional, it is a personal output choice. And i want to have it in Traditional 100% of the time. hence this request.

[LeeHaha314]

@Francoyy I get you. Considering their strategy of data processing and dataset source (mainly unsupervised), I think it's hard to identify two mode clearly with the model itself , thus the usage of initial prompt could be a middle ground.

[BackMountainDevil]

guys. --initial_prompt "以下是普通话的句子" tested not work on large-v3 sometimes. I want zh-cn(Simplified Chinese) but got Traditional Chinese. Do you have the same wired thing?

[chleeab]

Is it possible to separate "simplified Chinese" and "traditional Chinese" into two set of language?

[BackMountainDevil]

Got it. I think add the prompt is a good solution. 

…

------------------ Original ------------------ From: Francoyy ***@***.***> Date: 周六,8月 5,2023 12:52 To: openai/whisper ***@***.***> Cc: Kearney ***@***.***>, Mention ***@***.***> Subject: Re: [openai/whisper] Simplified Chinese rather than traditional? (Discussion #277) @BackMountainDevil I was not talking about the difference between "zh" and "Chinese", but the fact that there is "Simplified Chinese" and "Traditional Chinese". Their full notation is usually "zh-TW" (or zh-HK) for traditional chinese, whereas simplified chinese is "zh-Cn" or sometimes more commonly "zh". The spoken part is the same, but the way it is written is using different characters. You could say that generally, traditional chinese contains more strokes, for example Turtle is 龟 in simplified Chinese, and 龜 in traditional. But as @no1xsyzy mentioned, it's not a 1:1 mapping, but n:m mapping. So once Whisper outputs Chinese text, there's no way to use a script to automatically translate from simplified to traditional, or vice versa. Hence the question if it is possible in some way to tell whisper that we would like Simplified or Traditional as output. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[cuichungman]

It is advised to label Cantonese as "yue" and Mandarin as "cmn," in accordance with their respective language codes. Continually using "zh" may result in messing up with different languages of Chinese.

[Francoyy]

Yeah, too bad… because it’s just a hack but not a guarantee. Most people don’t understand what Simplified Chinese and Traditional Chinese is… i guess the ultimate solution for us would be a script using google translate and converting Traditional and Simplified, after Whisper is done generating the subtitle… Kearney ***@***.***>於 2023年12月11日 週一，11:03寫道：

…

guys. --initial_prompt "以下是普通话的句子。" tested not work on large-v3. I want zh-cn(Simplified Chinese) but got Traditional Chinese. Do you have the same wired thing? — Reply to this email directly, view it on GitHub <#277 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVJRXP63N4QXXQDAJ52J3YI3K67AVCNFSM6AAAAAARARLKOSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMJXHE3TC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jbang2004]

mark

[BackMountainDevil]

try subscribe

[Francoyy]

Unfortunately, no. In terms of audio, it's the same. People have trouble to understand that. How could I explain it easily. It's a little bit like deciding to write in small case, or to decide to write in UPPER CASE. In China, people use simplified chinese characters to write characters. But for example in Hong Kong, Taiwan they use traditional chinese characters. So for the same video, if my audience was in Taiwan, I would want to output traditional characters. But if my audience is from China, then I want to get simplified characters.

…

On Mon, Feb 5, 2024 at 3:11 PM FarmerL ***@***.***> wrote: Is it possible to separate "simplified Chinese" and "traditional Chinese" into two set of language? — Reply to this email directly, view it on GitHub <#277 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVJRTWBSPC7MWR6EEJKSTYSCAZ5AVCNFSM6AAAAAARARLKOSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNRWGIYDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jchenny7]

How about if we specify in language whether to output zh-CN (Simplified Chinese) or zh-TW (Traditional Chinese) instead of consolidate as zh?

[Francoyy]

Yes, that makes sense to me. That was the original request. But according to the devs, that's not how Whisper work. Therefore there is a workaround of providing a sample input to influence the output...

…

On Mon, Feb 5, 2024 at 3:32 PM jchenny7 ***@***.***> wrote: How about if we specify in language whether to output zh-CN (Simplified Chinese) or zh-TW (Traditional Chinese) instead of consolidate as zh? — Reply to this email directly, view it on GitHub <#277 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABVJRTJHVDD6RLXFW5M6Z3YSCDIBAVCNFSM6AAAAAARARLKOSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNRWGM2TS> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[changtimwu]

I encountered a similar issue where Whisper kept outputting simplified Chinese despite setting the initial_prompt to "以下是普通話的句子". I found Whipser tend to work in that way when the voices are with certain accents. I finally found a solution:

initial_prompt = "以下是普通話的句子,請以繁體輸出"

I suspect that the Whisper model is just too effective at detecting Chinese accents. ;)

[Peter5Chiou]

thank you!