[WIP] Blog post on predict_proba
#152
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Open
| These probability estimates are typically accessible from the `predict_proba` method of scikit-learn's classifiers. | ||
|
|
||
| However, the quality of the estimated probabilities must be validated to provide trustworthiness, ensure fairness and robustness to operating conditions. | ||
| To be reliable, the estimated probabilities must be close to the true underlying posterior probabilities of the classes `P(Y=1|X)`. |
There was a problem hiding this comment.
As you know, there are several points of view on what such a probability may mean (controlled average error rate versus controlled individual probabilities). Maybe it would be good by first explaining what these two mean.
| Similarly to validating a discriminant classifier through accuracy or ROC curves, tools have been developed to evaluate a probabilistic classifier. | ||
| Calibration is one of them [1-4]. Calibration is used as a proxy to evaluate the closeness of the estimated probabilities to the true ones. Many recalibration techniques have been developed to improve the estimated probabilities (see [scikit-learn's user guide on calibration](https://scikit-learn.org/stable/modules/calibration.html)). Estimated probabilities of a calibrated classifier can be interpreted as probability of correctness on population of same estimated probability, but not as the true posterior class probability. | ||
|
|
||
| Indeed, it is important to highlight that calibration only captures part of the error on the estimated probabilities. The remaining term is the grouping loss [5]. Together, the calibration and grouping losses fully characterize the error on the estimated probabilities, the epistemic loss. |
There was a problem hiding this comment.
You are too formal. Give us the intuitions of why calibration is not the full story, rather than the maths.
| Indeed, it is important to highlight that calibration only captures part of the error on the estimated probabilities. The remaining term is the grouping loss [5]. Together, the calibration and grouping losses fully characterize the error on the estimated probabilities, the epistemic loss. | ||
|
|
||
| $$\text{Epistemic loss} = \text{Calibration loss} + \text{Grouping loss}$$ | ||
|
|
There was a problem hiding this comment.
First mention Brier score for model selection. Later mention grouping loss.
|
|
||
| However, estimating the grouping loss is a harder problem than calibration as its estimation involves directly the true probabilities. Recent work have focused on approximating the grouping loss through local estimations of the true probabilities [6]. | ||
|
|
||
| When working with scikit-learn's classifiers, users must be equally as cautious on results obtained from `predict_proba` as on results from `predict`. Both output estimated quantities (probabilities and labels respectively) with no prior guarantees on their quality. In both cases, model's quality must be assessed with appropriate metrics: expected calibration error, brier score, accuracy, AUC. |
There was a problem hiding this comment.
Put links to relevant pages in the scikit-learn documentation
| However, estimating the grouping loss is a harder problem than calibration as its estimation involves directly the true probabilities. Recent work have focused on approximating the grouping loss through local estimations of the true probabilities [6]. | ||
|
|
||
| When working with scikit-learn's classifiers, users must be equally as cautious on results obtained from `predict_proba` as on results from `predict`. Both output estimated quantities (probabilities and labels respectively) with no prior guarantees on their quality. In both cases, model's quality must be assessed with appropriate metrics: expected calibration error, brier score, accuracy, AUC. | ||
|
|
There was a problem hiding this comment.
Maybe mention quickly (with a separate section title) recalibration (and link to corresponding docs)
|
@aperezlebel While I value the effort for such a blog post, I do not agree with some parts of its current content, e.g. the grouping loss. Unfortunately, I can‘t promise a fast review atm. |
|
@lorentzenchr I appreciate your feedback, thanks. Could you elaborate on the parts you disagree with? |
|
I have 3 main points of critique:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #147.
Work in progress.
TODO: