Benchmarks scoring scripts availability

[MetaHG]

Hi!

Do you have any plan of releasing the scripts used to compute the detailed benchmarks of V2 available on HuggingFace?
I would find that very useful for quick evaluation of modified/finetuned pipelines on academical datasets.

P.S.: Congrats on the V2 release!

[hbredin]

This is something that I have thought about, and is deep in my long TODO list.

The only issue I see is that most of those academic datasets are not freely available (DIHARD, CALLHOME, REPERE, ...) so such a script would be useless for most people.

[MetaHG]

Mmm I see. I did not look into these datasets too precisely and I did not know that they were not freely available. I thought that they would be freely available with academic use licenses only. Consequently, I think you are right, for these datasets, such a script would be useless. However, for the other datasets that are freely available, I still think that this would be useful (for quick check of loss of generalization after finetuning, or reproducible academic evaluations).

But I understand that you must have a long todo list and that this point is certainly not a priority right now.

Thank you for your reply!

[jvel07]

@hbredin, congrats on the amazing job you are doing!
I certainly agree with @MetaHG in the sense that if you could share the evaluation script it would be quite useful. The availability of the academic datasets wouldn't be an issue as the script could be adapted to custom corpora :) Looking forward!