Whispering (in FRENCH) keeps missing a sentence, despite separating music from vocals. What can I do?

[NormalFall0]

The video has lot of large silences. The whisper keep hallucinating, and even worse it even skipped 10 seconds of talk where it was hallucinating from the previous silence and continuing throughest the second silence.
The video is here (https://www.youtube.com/watch?v=i1ZbRFvfjI8)

I tried the model "Medium", because I have always had better resultrs with medium than large.
Langage: FR.

The biggest hallucinations happen between : 4:14 (starting from 4:31) to 5:02, then from 5:57 to 8:14, where its "silence - talk - silence" but the silence is MISSED and replaced with hallucination.

Can someone try or tell me how I can improve the results?
If you want to compare, try to see if you get this part:
The sentence missed is at 6:36!

This sentence was skipped both with medium model and Large model.

Can someone do better or show me how to? THX (You must find a way to make whisper generate the sentence at 6:36 at least)
I also would love to skeep the SRT timing as they are AS MUCH as possible.

[glangford]

I used the following with the large-v2 model to get an improvement (including the sentence at 6:36, and no hallucination at 4:31) but the resulting .srt still needs manual editing. The .srt is included below.

model.transcribe(vfile, language="fr", fp16=False, patience=2, beam_size=5, task='transcribe',
                              best_of=5, word_timestamps=True, verbose=False, suppress_tokens="") ########

Viande.srt.txt

[glangford]

large-v3 seems to hallucinate more than large-v2, and that may be why you were seeing better success with the medium model (specifying just large implies large-v3). Note the use of the suppress_tokens option as well.

[NormalFall0]

Hi Glan (This is a new reply (I gave you another reply below if you check)

• What is this supress tokens thing, shoudl I use it?

• I just tried large-v2 (without pyanote code etc) and it indeed did not hallucinate and included the sentence at 6:36 thank you very much! I was knda worrief that I had to cut the video into multiple vidoes then whister the ones without silence, then reassemble them and try to keep the srt timestamps similar to the first vide uncut etc, you saved me:)! Thnak you!

• I wonder why is large v2 slower?

• Finally do you think I should use PYannote? Or is it only when you have mutliple speakers?

[glangford]

What is this supress tokens thing, should I use it?

See this discussion; in addition to quotes it identifies background sounds [music] and seems to reduce hallucinations
#844

I wonder why is large v2 slower?

I haven't tested the performance of v2 vs v3, I just use large-v2 on CUDA (Colab) and quantized medium on CPU (locally)

Finally do you think I should use PYannote? Or is it only when you have mutliple speakers?

The pyannote example I posted was for transcribing multiple languages within the same audio. If you have a single language, you can just use whisper.

[glangford]

@NormalFall0 Here is one more attempt using a quantized medium model as described in #2009 and speaker diarization. It is more resistant to hallucination, just FYI.

#2009 (comment)

Viande-diarized.srt.txt

[NormalFall0]

Is there something more powerful than reacting and saying thank you? Because I am soo grateful and I appreciate so much that you came here to write yet another answer! Love it!
I will try the code you provided and hopefully I get it right. I have my own code right now which is a simple command, that is run when I press a simple tkinter button. Not used to pyanote (only tried it once or twice) but I hope I get is easy. Otherwise Iwill leave a comment.
Thanks again Cant wait to try it later

[EtienneAb3d]

• I just tried large-v2 (without pyanote code etc) and it indeed did not hallucinate and included the sentence at 6:36 thank you very much! I was knda worrief that I had to cut the video into multiple vidoes then whister the ones without silence, then reassemble them and try to keep the srt timestamps similar to the first vide uncut etc, you saved me:)! Thnak you!

This kind of process can be obtained in an easier way like this:

1. Use WhisperHallu to get both a clean text (combining voice extraction, silence removal, voice compression, etc) and clean timestamps (possibly with hallucinations).
   https://github.com/EtienneAb3d/WhisperHallu
2. use WhisperTimeSync to sync the good text with the good timestamps.
   https://github.com/EtienneAb3d/WhisperTimeSync

[NormalFall0]

Is it possible to make it have no hallucinations?

[EtienneAb3d]

Hallucinations are mostly produced on silence or noise parts. To remove almost all hallucinations, you need to use voice extraction (Demux), non-voice parts removal (Silero VAD), silence parts removal (ffmpeg).
You may still get hallucinations on short audio. The Marker processing of WhisperHallu will do the job on most cases.
Finally, loudness normalization and speech compression (ffmpeg) will bring you we a higher recognition quality.