FEAT: Fairness of exposure #959
Comments
|
Thank you for opening this issue @bram49! IMO the deterministic version of exposure makes sense, but I'd love to hear other people's thoughts on this as well @fairlearn/fairlearn-maintainers. Could you give an example of how utility would be defined in scenario (2)? As a side note, I think we need to be careful with overloading terminology. Demographic parity/disparate treatment* are typically use I n the context of classification and regression problems. To avoid confusion, I think it would make sense to instead focus on the types of harm that are being measured. E.g., differences in exposure can be seen as a measure of allocation harm in a ranking scenario, whereas differences in exposure/utility is an indicator of quality-of-service harm. [*In the Fairlearn community we generally try to avoid the term "disparate treatment" which originates from US laws on employment discrimination. Using that term may suggest compliance with US law even when it's not the case. First of all, most applications will fall outside of the employment domain. Moreover, considering only the output of the model (i.e., disregarding how it's used, by whom, etc.) is a very narrow frame.] |
|
Thank you for the reply @hildeweerts |
Fairness of exposure
Would like to extend Fairlearn with tools to deal with fairness in rankings (#945). I think that an intuitive and good metric for fairness in rankings is exposure. From the paper: Fairness of exposure in rankings by Ashudeep Singh and Thorsten Joachims.
Where exposure is defined as:
 = \sum_{j=1}^N P_{i,j} v_j)
})
,with document d_i, probabilistic ranking P and logarithmic discount v_j to deal with position bias (higher rankings get exponential more attention)
With exposure, all kinds of fairness metrics can be constructed. Such as:
1. Allocation harm
Where you equalize the average exposure of documents.
Denoting average exposure in group k with:
 = \frac{1}{|G_k|} \sum_{d_i \in G_k} Exposure(d_i | P))
And denoting the demographic parity constraint with:2. Quality-of-service harm
Where you try to keep the relevance of the items proportional to the exposure. Like in the example on the right, small differences in relevance between candidates can lead to huge differences in exposure.}{U(G_0 | q)} = \frac{Exposure(G_1 | P)}{U(G_1 | q)} )
,where U(G|q), is the average utility of a group. And utility is the relevance score, on which the documents are ranked.Problem
The problem with this metric is that you need a given probabilistic ranking P, which you can create when you have multiple rankings with the same documents. To work around this, and try to make the metric work for a single ranking, I thought about defining exposure as the sum of logarithmic discounts for ranking tau. Which would define demographic parity as:
Conclusion
Think that with the adjustment, this is an effective way to deal quantify fairness in rankings. Please let me know what you think
The text was updated successfully, but these errors were encountered: