Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics limited to a single Expected Calibration Error (ECE) formula #277

Open
thibaultcordier opened this issue Mar 2, 2023 · 0 comments
Labels
developers Proposed by developers. documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers

Comments

@thibaultcordier
Copy link
Collaborator

Describe the bug
MAPIE offers only one way to calculate the ECE, but there are different estimators of the ECE in the literature (reference: https://arxiv.org/abs/2109.03480).

To Reproduce
When using the expected_calibration_error(y_true: ArrayLike, y_scores: ArrayLike) function, the computed estimator of ECE is the class-specific ECE associated to class c (see https://arxiv.org/abs/2109.03480 in page 4).

To use this function correctly, the array y_true must be the boolean array indicating whether the example at index i is associated with the class considered (named c). So for any example at the index i, y_true[i] = 1 if this example is related to the considered class c.

Expected behavior
It will be interesting to implement other ECE estimators, such as the confidence ECE presented in https://arxiv.org/abs/1706.04599.

Additional context
Links for references:

  • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR. (https://arxiv.org/abs/1706.04599)
  • Posocco, N., & Bonnefoy, A. (2021). Estimating expected calibration errors. In Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part IV 30 (pp. 139-150). Springer International Publishing. (https://arxiv.org/abs/2109.03480)
@thibaultcordier thibaultcordier added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Mar 2, 2023
@thibaultcordier thibaultcordier added this to Need triage in Developments via automation Mar 2, 2023
@thibaultcordier thibaultcordier removed the bug Something isn't working label Mar 3, 2023
@thibaultcordier thibaultcordier added the good first issue Good for newcomers label Mar 16, 2023
@thibaultcordier thibaultcordier moved this from Need triage to Bugs to be fixed (Low Priority) in Developments Mar 16, 2023
@thibaultcordier thibaultcordier moved this from Bugs to be fixed (Low Priority) to Methods to be implemented (Long Term) in Developments Mar 16, 2023
@thibaultcordier thibaultcordier added the developers Proposed by developers. label Jul 4, 2023
@thibaultcordier thibaultcordier added this to the Backlog H2 milestone Aug 3, 2023
@LacombeLouis LacombeLouis added this to H2 2023 in Roadmap Aug 3, 2023
@LacombeLouis LacombeLouis removed this from the Backlog H2 2023 milestone Aug 3, 2023
@LacombeLouis LacombeLouis moved this from H2 2023 to H1 2024 in Roadmap Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
developers Proposed by developers. documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers
Projects
Developments
Methods to be implemented (Long Term)
Roadmap
H1 2024
Development

No branches or pull requests

2 participants