This repository contains the data and resources for the SemEval 2024 Task 1: Semantic Textual Relatedness (STR). For more information, please visit the shared task and competition websites.
Dataset | Languages | Shared Task Starter Kit | Citing This Work
If you use our data, please cite our papers:
@inproceedings{ousidhoum2024semrel2024,
title={SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages},
author={Ousidhoum, Nedjma and Muhammad, Shamsuddeen Hassan and Abdalla, Mohamed and Abdulmumin, Idris and Ahmad, Ibrahim Said and
Ahuja, Sanchit and Aji, Alham Fikri and Araujo, Vladimir and Ayele, Abinew Ali and Baswani, Pavan and Beloucif, Meriem and Biemann, Chris and Bourhim, Sofia and De Kock, Christine and Dekebo, Genet Shanko and
Oumaima Hourrane and Gopichand Kanumolu and Lokesh Madasu and Samuel Rutunda and Manish Shrivastava and Solorio, Thamar and Surange, Nirmal and Tilaye, Hailegnaw Getaneh and Vishnubhotla, Krishnapriya and Winata, Genta and Yimam, Seid Muhie and Mohammad, Saif M.},
booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
year = "2024",
publisher = "Association for Computational Linguistics",
}
@inproceedings{ousidhoum-etal-2024-semeval,
title = "{S}em{E}val-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages",
author = "Ousidhoum, Nedjma and Muhammad, Shamsuddeen Hassan and Abdalla, Mohamed and Abdulmumin, Idris and
Ahmad,Ibrahim Said and Ahuja, Sanchit and Aji, Alham Fikri and Araujo, Vladimir and Beloucif, Meriem and
De Kock, Christine and Hourrane, Oumaima and Shrivastava, Manish and Solorio, Thamar and Surange, Nirmal and
Vishnubhotla, Krishnapriya and Yimam, Seid Muhie and Mohammad, Saif M.",
booktitle = "Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)",
year = "2024",
publisher = "Association for Computational Linguistics"
}The annotation guidelines are available here. (The PDF named SemRel Annotation guidelines.)
Check the SemRel Baseline Folder for details about the baseline experiment.
The STR dataset is available in the data folder or can be downloaded from Hugging Face.
- For Track A: TrackA folder
- For Track B: TrackB folder
- For Track C: TrackC folder
**Note that the full BWS tuple annotations will be available soon. **
The STR task focuses on the following 14 languages:
- Afrikaans (afr released)
- Algerian Arabic (arq released)
- Amharic (amh released)
- English (eng released)
- Hausa (hau released)
- Indonesian (ind released)
- Hindi (hin released)
- Kinyarwanda (kin released)
- Marathi (mar released)
- Modern Standard Arabic (arb released)
- Moroccan Arabic (ary released)
- Punjabi (pan released)
- Spanish (esp released)
- Telugu (tel released)
A starter kit is available to help you create a baseline result. You can open the starter kit in a Colab Notebook and run the baseline system. The resultant experiment can be submitted to Codalab to ensure the submission format is clear.
To run the Colab Notebook, click the badge "Open in Colab".
- Simple Co-occurrence Baseline for Semantic Relatedness:
