FEA add TunedThresholdClassifier meta-estimator to post-tune the cut-off threshold

doc/modules/classification_threshold.rst

@@ -117,23 +117,6 @@ a meaningful metric for their use case.
         >>> model.best_score_
         0.86...
 
-A second strategy aims to maximize one metric while imposing constraints on another
-metric. There are four pre-defined options that can be provided to `objective_metric`
-parameter, two use the Receiver Operating Characteristic (ROC) statistics and two use
-the Precision-Recall statistics.
-
-- `"max_tpr_at_tnr_constraint"`: maximizes the True Positive Rate (TPR) such that the
-  True Negative Rate (TNR) is the closest to a given value.
-- `"max_tnr_at_tpr_constraint"`: maximizes the TNR such that the TPR is the closest to
-  a given value.
-- `"max_precision_at_recall_constraint"`: maximizes the precision such that the recall
-  is the closest to a given value.
-- `"max_recall_at_precision_constraint"`: maximizes the recall such that the precision
-  is the closest to a given value.
-
-For these options, the `constraint_value` parameter needs to be defined. In addition,
-you can use the `pos_label` parameter to indicate the label of the class of interest.
-
 Important notes regarding the internal cross-validation
 -------------------------------------------------------
 

examples/model_selection/plot_tuned_decision_threshold.py

@@ -11,7 +11,7 @@
 
 This example shows how to use the
 :class:`~sklearn.model_selection.TunedThresholdClassifierCV` to tune the decision
-threshold, depending on a metric of interest as well as under a specific constraints.
+threshold, depending on a metric of interest.
 """
 
 # %%
@@ -184,201 +184,3 @@
 # example entitled,
 # :ref:`sphx_glr_auto_examples_model_selection_plot_cost_sensitive_learning.py`,
 # for more details.
-#
-# Tuning the decision threshold under constraint
-# ----------------------------------------------
-#
-# In some cases, we do not want to only maximize a given metric but instead to maximize
-# a metric while satisfying a constraint on another metric. In the current example, we
-# could imagine that the decision of our predictive model will be reviewed by a medical
-# doctor. In this case, this doctor will only accept a ratio of false positive lower
-# than a given value. Therefore, we are interested in maximizing the true positive rate
-# while having a false positive rate lower than this value.
-#
-# The :class:`~sklearn.model_selection.TunedThresholdClassifierCV` allows to tune the
-# decision threshold with such specification. We illustrate this strategy together with
-# a single train-test split split to display the Receiver Operating Characteristic (ROC)
-# curves to get better intuitions.
-#
-# First, we split the data into a training and testing set.
-
-# %%
-from sklearn.model_selection import train_test_split
-
-data_train, data_test, target_train, target_test = train_test_split(
-    data, target, random_state=42
-)
-
-# %%
-# Now, we will train both the vanilla and tuned model on the training set. We recall
-# that the tuned model is internally maximizing the balanced accuracy for the moment.
-model.fit(data_train, target_train)
-tuned_model.fit(data_train, target_train)
-
-# %%
-# To show the benefit on optimizing a metric under constraint, we will evaluate the
-# models using the ROC curve statistics: the true positive rate (TPR) and the false
-# positive rate (FPR).
-#
-# The FPR is not defined in scikit-learn and we define it below:
-from sklearn.metrics import confusion_matrix, make_scorer, recall_score
-
-
-def fpr_score(y, y_pred, neg_label, pos_label):
-    cm = confusion_matrix(y, y_pred, labels=[neg_label, pos_label])
-    tn, fp, _, _ = cm.ravel()
-    tnr = tn / (tn + fp)
-    return 1 - tnr
-
-
-tpr_score = recall_score  # TPR and recall are the same metric
-scoring = {
-    "fpr": make_scorer(fpr_score, neg_label=neg_label, pos_label=pos_label),
-    "tpr": make_scorer(tpr_score, pos_label=pos_label),
-}
-
-# %%
-# Now, we plot the ROC curve of both models and the FPR and TPR statistics for the
-# decision thresholds of both models.
-from sklearn.metrics import RocCurveDisplay
-
-disp = RocCurveDisplay.from_estimator(
-    model, data_test, target_test, name="Vanilla model", linestyle="--", alpha=0.5
-)
-RocCurveDisplay.from_estimator(
-    tuned_model,
-    data_test,
-    target_test,
-    name="Tuned model",
-    linestyle="-.",
-    alpha=0.5,
-    ax=disp.ax_,
-)
-disp.ax_.plot(
-    scoring["fpr"](model, data_test, target_test),
-    scoring["tpr"](model, data_test, target_test),
-    marker="o",
-    markersize=10,
-    color="tab:blue",
-    label="Default cut-off point at a probability of 0.5",
-)
-disp.ax_.plot(
-    scoring["fpr"](tuned_model, data_test, target_test),
-    scoring["tpr"](tuned_model, data_test, target_test),
-    marker=">",
-    markersize=10,
-    color="tab:orange",
-    label=f"Cut-off point at probability of {tuned_model.best_threshold_:.2f}",
-)
-disp.ax_.legend()
-_ = disp.ax_.set_title("ROC curves")
-
-# %%
-# As expected, both models have the same ROC curves since the tuned
-# model is only a post-processing step of the vanilla model. The tuning step is only
-# changing the decision threshold, as displayed by the blue and orange markers.
-# To optimize the balanced accuracy, the tuned model moved the decision threshold
-# from 0.5 to 0.22. By shifting this point, we increase the FPR while increasing
-# the TPR: in short we make more false positive but also more true positive. This is
-# exactly what we concluded in the previous section when looking at the balanced
-# accuracy score.
-#
-# However, this decision threshold might not be acceptable for our medical doctor. He
-# might be interested to have a low FPR instead, let say lower than 5%. For this level
-# of FPR, he would like our predictive model to maximize the TPR.
-#
-# The :class:`~sklearn.model_selection.TunedThresholdClassifierCV` allows to specify
-# such constraint by providing the name of the metric and the constraint value. Here, we
-# use `max_tpr_at_tnr_constraint` which is exactly what we want. Since the true negative
-# rate (TNR) is equal to 1 - FPR, we can rewrite the constraint value as `1 - 0.05 =
-# 0.95`.
-
-# %%
-constraint_value = 0.95
-tuned_model.set_params(
-    objective_metric="max_tpr_at_tnr_constraint",
-    constraint_value=constraint_value,
-    pos_label=pos_label,
-    store_cv_results=True,
-)
-tuned_model.fit(data_train, target_train)
-
-# %%
-# Now, we can plot the ROC curves and analyse the results.
-import matplotlib.pyplot as plt
-
-_, axs = plt.subplots(ncols=2, figsize=(12, 5))
-
-disp = RocCurveDisplay(
-    fpr=1 - tuned_model.cv_results_["constrained_scores"],
-    tpr=tuned_model.cv_results_["maximized_scores"],
-    estimator_name="ROC of the tuned model",
-    pos_label=pos_label,
-)
-axs[0].plot(
-    1 - tuned_model.constrained_score_,
-    tuned_model.best_score_,
-    marker="o",
-    markersize=10,
-    color="tab:blue",
-    label=f"Cut-off point at probability of {tuned_model.best_threshold_:.2f}",
-)
-axs[0].axvline(
-    1 - constraint_value, 0, 1, color="tab:blue", linestyle="--", label="FPR constraint"
-)
-axs[0].set_title("Average ROC curve for the tuned model\nacross CV folds")
-RocCurveDisplay.from_estimator(
-    model,
-    data_test,
-    target_test,
-    name="Vanilla model",
-    linestyle="--",
-    alpha=0.5,
-    ax=axs[1],
-)
-RocCurveDisplay.from_estimator(
-    tuned_model,
-    data_test,
-    target_test,
-    name="Tuned model",
-    linestyle="-.",
-    alpha=0.5,
-    ax=axs[1],
-)
-axs[1].plot(
-    scoring["fpr"](model, data_test, target_test),
-    scoring["tpr"](model, data_test, target_test),
-    marker="o",
-    markersize=10,
-    color="tab:blue",
-    label="Default cut-off point at a probability of 0.5",
-)
-axs[1].plot(
-    1 - tuned_model.constrained_score_,
-    tuned_model.best_score_,
-    marker="^",
-    markersize=10,
-    color="tab:orange",
-    label=f"Cut-off point at probability of {tuned_model.best_threshold_:.2f}",
-)
-axs[1].legend()
-axs[1].set_title("ROC curves")
-_ = disp.plot(ax=axs[0])
-
-# %%
-# We start with the right-hand side plot. It depicts the ROC curves as in the previous
-# section. We observe that the control point of the tuned model moved to a low FPR
-# that was defined by our constraint. To achieve this low FPR, the decision threshold
-# was moved to a probability of 0.72.
-#
-# The left-hand side plot shows the averaged ROC curve on the internal validation set
-# across the different cross-validation folds. This curve is used to define the decision
-# threshold. The vertical dashed line represents the FPR constraint that we defined.
-# The decision threshold corresponds to the maximum TPR on the left of this dashed line
-# and is represented by a blue marker.
-#
-# An important point to note is that the decision threshold is defined on averaged
-# statistics on an internal validation set. It means that the constraint is respected
-# on the train/validation dataset but not necessarily on the test set, in case the
-# statistical performance of the model differ from the train/validation set to the test
-# set (i.e. overfitting).