Probability of Equal Expected Rank

This package is the Python implementation of the MLIR fairness measure "Probability of Equal Expected Rank" using ir_measures.

How to use it

You can either directly install it from PyPi through

pip install peer_measure

Or install the GitHub version

pip install pip@git+https://github.com/hltcoe/peer_measure

When importing, please import both peer_measure and ir_measures.

from peer_measure import PEER
import ir_measures

Please refer to the documentation of ir_measures for the general usage.

Parameters

PEER takes two required parameters: weights and lang_mapping.

• weights: a int-to-float dictionary specifying the weight for each relevance level. The weight have be sum up to 1.0.
• lang_mapping: a str-to-str dictionary with keys being the doc_id and values being the language id of the correspoding document.

You can specify these parameters and the rank cutoff when declaring the measure instance. For example,

measure = PEER(weights={0: 0, 1: 0.5, 2:0, 3: 0.5}, lang_mapping=...)@20

Please refer to our paper for detail definition and implication of the parameters.

Citation

Please consider citing our paper if you use this measure.

@inproceedings{peer,
	author = {Eugene Yang and Thomas Jänich and James Mayfield and Dawn Lawrie},
	title = {Language Fairness in Multilingual Information Retrieval},
	booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (Short Paper) (Accepted)},
	year = {2024}, 
	doi = {10.1145/3626772.3657943}
}