- Date: 2024-02-23
- Change: Added MATH, GSM8K, and aqua-rat.
- Total Number of Tasks: 90
- Total Number of Instances: 45566
| Index | Task Name | Version | Construction | Arxiv Link | License | Dataset Size |
|---|---|---|---|---|---|---|
| 1 | aqua-rat | v1 | {'class': 'mcq', 'n_choices': 5} | https://arxiv.org/abs/1705.04146 | apache-2.0 | 254 |
| 2 | arc-challenge | v1 | {'class': 'mcq', 'n_choices': 'mixed'} | https://arxiv.org/abs/1803.05457 | cc-by-sa-4.0 | 1172 |
| 3 | arc-easy | v1 | {'class': 'mcq', 'n_choices': 'mixed'} | https://arxiv.org/abs/1803.05457 | cc-by-sa-4.0 | 2376 |
| 4 | commonsenseqa | v1 | {'class': 'mcq', 'n_choices': 5} | https://arxiv.org/abs/1811.00937 | mit | 1221 |
| 5 | gsm8k | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2110.14168 | mit | 1319 |
| 6 | hellaswag | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/1905.07830 | mit | 10042 |
| 7 | hhh-alignment-harmless | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2112.00861 | apache-2.0 | 58 |
| 8 | hhh-alignment-helpful | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2112.00861 | apache-2.0 | 59 |
| 9 | hhh-alignment-honest | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2112.00861 | apache-2.0 | 61 |
| 10 | hhh-alignment-other | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2112.00861 | apache-2.0 | 43 |
| 11 | htest-end-ly | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 12 | htest-end-punctuation | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 13 | htest-hyphenated-word | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 14 | htest-palindrome | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 15 | htest-repeated-word | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 16 | htest-rhyme | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 17 | htest-spelled-math | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 18 | htest-spelled-number | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 19 | htest-start-vowel | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 20 | htest-uppercase | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/2110.14168 | mit | 200 |
| 21 | math-algebra | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 1187 |
| 22 | math-counting-and-probability | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 474 |
| 23 | math-geometry | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 479 |
| 24 | math-intermediate-algebra | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 903 |
| 25 | math-number-theory | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 540 |
| 26 | math-prealgebra | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 871 |
| 27 | math-precalculus | v1 | {'class': 'frq', 'type': 'simple'} | https://arxiv.org/abs/2103.03874 | mit | 546 |
| 28 | medqa-usmle | v1 | {'class': 'mcq', 'n_choices': 5} | https://arxiv.org/abs/2009.13081 | mit | 1273 |
| 29 | mmlu-abstract-algebra | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 30 | mmlu-anatomy | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 135 |
| 31 | mmlu-astronomy | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 152 |
| 32 | mmlu-business-ethics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 33 | mmlu-clinical-knowledge | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 265 |
| 34 | mmlu-college-biology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 144 |
| 35 | mmlu-college-chemistry | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 36 | mmlu-college-computer-science | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 37 | mmlu-college-mathematics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 38 | mmlu-college-medicine | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 173 |
| 39 | mmlu-college-physics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 102 |
| 40 | mmlu-computer-security | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 41 | mmlu-conceptual-physics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 235 |
| 42 | mmlu-econometrics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 114 |
| 43 | mmlu-electrical-engineering | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 145 |
| 44 | mmlu-elementary-mathematics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 378 |
| 45 | mmlu-formal-logic | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 126 |
| 46 | mmlu-global-facts | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 47 | mmlu-high-school-biology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 310 |
| 48 | mmlu-high-school-chemistry | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 203 |
| 49 | mmlu-high-school-computer-science | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 50 | mmlu-high-school-european-history | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 165 |
| 51 | mmlu-high-school-geography | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 198 |
| 52 | mmlu-high-school-government-and-politics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 193 |
| 53 | mmlu-high-school-macroeconomics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 390 |
| 54 | mmlu-high-school-mathematics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 270 |
| 55 | mmlu-high-school-microeconomics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 238 |
| 56 | mmlu-high-school-physics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 151 |
| 57 | mmlu-high-school-psychology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 545 |
| 58 | mmlu-high-school-statistics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 216 |
| 59 | mmlu-high-school-us-history | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 204 |
| 60 | mmlu-high-school-world-history | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 237 |
| 61 | mmlu-human-aging | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 223 |
| 62 | mmlu-human-sexuality | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 131 |
| 63 | mmlu-international-law | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 121 |
| 64 | mmlu-jurisprudence | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 108 |
| 65 | mmlu-logical-fallacies | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 163 |
| 66 | mmlu-machine-learning | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 112 |
| 67 | mmlu-management | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 103 |
| 68 | mmlu-marketing | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 234 |
| 69 | mmlu-medical-genetics | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 70 | mmlu-miscellaneous | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 783 |
| 71 | mmlu-moral-disputes | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 346 |
| 72 | mmlu-moral-scenarios | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 895 |
| 73 | mmlu-nutrition | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 306 |
| 74 | mmlu-philosophy | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 311 |
| 75 | mmlu-prehistory | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 324 |
| 76 | mmlu-professional-accounting | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 282 |
| 77 | mmlu-professional-law | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 1534 |
| 78 | mmlu-professional-medicine | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 272 |
| 79 | mmlu-professional-psychology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 612 |
| 80 | mmlu-public-relations | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 110 |
| 81 | mmlu-security-studies | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 245 |
| 82 | mmlu-sociology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 201 |
| 83 | mmlu-us-foreign-policy | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 100 |
| 84 | mmlu-virology | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 166 |
| 85 | mmlu-world-religions | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/2009.03300 | mit | 171 |
| 86 | openbookqa | v1 | {'class': 'mcq', 'n_choices': 4} | https://arxiv.org/abs/1809.02789 | apache-2.0 | 500 |
| 87 | piqa | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/1911.11641 | afl-3.0 | 1838 |
| 88 | socialiqa | v1 | {'class': 'mcq', 'n_choices': 3} | https://arxiv.org/abs/1904.09728 | cc-by-4.0 | 2224 |
| 89 | truthfulqa-mc1 | v1 | {'class': 'mcq', 'n_choices': 'mixed'} | https://arxiv.org/abs/2109.07958 | apache-2.0 | 817 |
| 90 | winogrande | v1 | {'class': 'mcq', 'n_choices': 2} | https://arxiv.org/abs/1907.10641 | apache-2.0 | 1267 |