Performance metrics for evaluating time segmentation of Whisper models

[mafaisalpg]

Hello everyone,

I like to compare different Whisper models on my own datasets changing some parameters like detect_disfluencies, etc. My output looks like

start, end, text

instead of word level timestamp.

Questions

• What are metrics (MSE, MAE, etc.) best fit for this purpose?

• Is there any Python implementation which I can reuse?

• Is there any paper or article have done such comparison?

Thanks

[Jeronymous]

Yes, we studied how to evaluate timestamps quality.
It's described in this paper: "Transcribing And Aligning Conversational Speech: A Hybrid Pipeline Applied To French Conversations"
https://hal.science/hal-04404777 (part "V. EVALUATION METRICS")

Unfortunately there is no open source code to compute those metrics.
But if you have any question about that, I can answer

[mafaisalpg]

thanks. do whisper-timestamped any implementation for these metrics which were used for comparing with whisper (original)?