How not to remove disfluencies, hesitations, filler words, etc?

[tophee]

From the Readme:

The lack of robustness around speech disfluencies (fillers, hesitations, repeated words...) that are usually removed by Whisper.

Although the removal of disfluencies by whisper is often a feature, for research purposes, it may well be a disadvantage (even a dealbreaker). What is the best strategy (if any) to stop whisper from cleaning up the transcript?

I saw the discussion about supressing tokens where I learned that

suppress_tokens is -1 by default, which means those tokens that do not correspond to text.

If this also includes at least some disfluencies, like "uhs" and "mmhs", that (i.e. not suppressing -1) would be a step in the right direction, but does it?

Are there any other tricks to make whisper more truthful to the original?

[Jeronymous]

The best approach I've found so far is to give a prompt with disfluencies to Whisper.

As described for French in the paper "TRANSCRIBING AND ALIGNING CONVERSATIONAL SPEECH: A HYBRID PIPELINE APPLIED TO FRENCH CONVERSATIONS"
https://hal.science/hal-04404777/document

suppress_tokens option is not related to disfluencies (it's more designed for annotations of non-speech event like "[music]", or to forbid some sensitive words).

The fact that Whisper suppresses disfluencies is that it was mostly trained on transcriptions without disfluencies (typically, subtitles of Youtube).

[tophee]

Do I understand correctly that while the prompt improves the result in combination with the other methods, it actually introduces lots of insertions if used on its own?

[Jeronymous]

Not really.
Prompt/No prompt comparison corresponds to the comparison of "Reference" and "No prompt".
The insertion rate is not increased.

The other comparisons are about VAD methods, which are critical here because there are background voices that must be filtered out
(that's why without VAD, or with a "standard" VAD, insertion rates can be high)

[tophee]

Prompt/No prompt comparison corresponds to the comparison of "Reference" and "No prompt".

Sorry, I'm not sure I understand. Your're saying the no prompt WERs can only be compare to the "reference" WERs, right? But why? The Y-axis are absolute WER values, no?

Another question about the injected prompt. I understand that this is the "initial_prompt" option in whisper-timestamped, right? But that is only injected at the beginning ("Optional text to provide as a prompt for the first window."), while the article suggests that it has been re-injected again and again. What am I missunderstanding?

ALso, could you say something about how you constructed the prompt? What there the alternatives versions that performed less well? I'm asking because I'm not sure how to construct an equivalent version in Swedish...

[Jeronymous]

Yes every WER can be compared. But you were interested in the effect of the prompt.
"Reference" and "No Prompt" have the exact same setting, except from whether there is the prompt.

About "initial_prompt", you are raising an excellent point. Yes! (I should have mentioned)
I modified openai-whisper sources to have this...

I constructed the prompt by trying, seeing the effect on a bunch of audio, and tuning... Prompt engineering and black magic.
And unfortunately I can only recommend that prompt for French, not other languages

[tophee]

Wow, thank you for that answer. Will check out the paper right away. Looks like it's exactly what I'm looking for. Thank you so much!