Please go to https://microsoft.github.io/msmarco/Datasets for all dataset descriptions and pointers.
For both tasks, please prepare the test results file in the following TAB-separated (TSV) format: qid<TAB>pid<TAB>rank.
1124703 8766037 1
1124703 8021997 2
1124703 7816201 3
1124703 8296123 4
1124703 8790898 5
1124703 5451590 6
1124703 8021999 7
1124703 8388210 8
1124703 8702520 9
1124703 8790903 10
We report MRR@10 for both tasks. Therefore, to minimize the size of your test results file, please free to only inclde the top 10 results per query.
The official evaluation script for the two tasks are available at the below locations:
- Document ranking: https://github.com/microsoft/MSMARCO-Document-Ranking/blob/main/ms_marco_eval.py
- Passage ranking: https://github.com/microsoft/MSMARCO-Passage-Ranking/blob/master/ms_marco_eval.py
Once you have built a model that meets your expectations on evaluation with the dev set, you can submit your test results to get official evaluation on the test set. To ensure the integrity of the official test results, we do not release the correct answers for test set to the public.
To submit your model for official evaluation on the test set, follow the steps corresponding to the appropriate task:
For the document ranking task, we follow a GitHub pull request based submission process. Please find the submission guidelines for the document ranking task here: https://microsoft.github.io/MSMARCO-Document-Ranking-Submissions/.
For the passage ranking task, we follow a GitHub pull request based submission process. Please find the submission guidelines for the passage ranking task here: https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/.
{% include_relative Notice.md %}