Are all HuggingFace dataset supported in the ScandEval benchmark?

[zhimin-z]

Check https://huggingface.co/ScandEval

[saattrupdan]

Hi @zhimin-z!

Almost all of those dataset are available in the scandeval framework, yes. Note however that the datasets on Hugging Face include datasets that are still undergoing review (see this PR) or which have been replaced with newer datasets.

To see a list of all the available benchmark datasets, run scandeval --help and look at the --dataset values. We'll make this more clear in the documentation in the near future 🙂

[zhimin-z]

Are these SQUAD datasets v1 or v2?

[saattrupdan]

Are these SQUAD datasets v1 or v2?

They're v2. You can see how all the datasets are built in the scripts directory:

https://github.com/ScandEval/ScandEval/tree/main/src/scripts