Clarification on the utility of p-values

[Geethen]

Concerning the wrappedclassifier with mondrian conformal prediction would it be safe to use the p-value of the predicted class as a calibrated probability or a good estimate of uncertainty. If not, any advice on could work as an estimate of uncertainty seeing that the predict_proba outputs uncalibrated probabilities based on the wrapped classifier. Note, I am working on a multiclass problem so Venn-ABERS does not seem to be applicable. I have already considered set size per instance but I was hoping for something more informative. Thanks in advance for any response.

[henrikbostrom]

Hi,

The p-values are not to be confused with class probabilities. The conformal prediction framework guarantees that the p-values for the true targets are distributed uniformly; this is what allows us to reject some labels while still achieving the desired coverage rate. The framework hence provides no guarantees for the predicted label. As you are interested in getting a well-calibrated probability for the predicted label (in a multiclass setting), I suggest that you take a look at the following paper, which proposes to use Venn-ABERS (which you mentioned) in a specific way, providing probabilities for that the predicted label is correct or not (rather than probabilities for all class labels):

Johansson, U., Löfström, T. & Boström, H.. (2021). Calibrating multi-class models. Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 152:111-130 [Available from https://proceedings.mlr.press/v152/johansson21a.html]

Best regards,
Henrik