CalibratedClassifierCV with Pipeline base_estimator

[odedbd]

Hi All,

I'm following up on pull request ENH Support pipelines in CalibratedClassifierCV
, which addresses issues from 8710.

The pull request solves the issue of fitting CalibratedClassifierCV with a Pipeline base_estimator, using cv='prefit'. In the case where the input is not an array, the estimator cannot be used, due to data validation and subsequent use of X.shape in predict_proba.

Non array inputs are quite likely where pipelines are involved, as a pipeline is often used when there is need to extract features from non-table data such as text or list of records.

Am I missing something about how this should be used?

Here is a snippet of code reproducing the issue I see, slightly modifying the code from the quoted issue above-

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction import DictVectorizer
from sklearn.calibration import CalibratedClassifierCV, _CalibratedClassifier
from sklearn.pipeline import Pipeline


fake_features = [
    {'state': 'NY', 'age': 'adult'},
    {'state': 'TX', 'age': 'adult'},
    {'state': 'VT', 'age': 'child'},
    {'state': 'NY', 'age': 'adult'},
    {'state': 'TX', 'age': 'adult'},
    {'state': 'VT', 'age': 'child'}
]

labels = [1, 0, 1, 1, 0, 1]

pipeline = Pipeline([
    ('vectorizer', DictVectorizer()),
    ('clf', RandomForestClassifier())
])

pipeline.fit(fake_features, labels)

clf_isotonic = CalibratedClassifierCV(base_estimator=pipeline, cv='prefit', method='isotonic')
clf_isotonic.fit(fake_features, labels)

clf_isotonic.predict_proba(fake_features)

[NicolasHug]

Thanks for the report @odedbd . Just a note: since you're calibrating a pre-fitted model, you should be careful to calibrate the pipeline on different data from the one that was used to train it - i.e. pipeline.fit and clf_isotonic.fit should not both use fake_features (I understand this is a minimal bug report that does not necessarily reflect actual usage so feel free to ignore)

@glemaitre @lucyleeow @ogrisel This looks like a legitimate issue. From what I can tell in predict_proba we can remove the call to check_array (we don't need it, the data should properly be validated in the base estimator's predict method) and replace X.shape[0] with _num_samples(X). Thoughts?

@lesteve here's an instance where converting to an issue would be helpful ;)

[odedbd]

Thanks for picking this up @NicolasHug . Indeed, I am using separate data for the calibration in my actual script.

[glemaitre]

From what I can tell in predict_proba we can remove the call to check_array (we don't need it, the data should properly be validated in the base estimator's predict method) and replace X.shape[0] with _num_samples(X). Thoughts?

Yes, it seems fair to delegate the validation to the steps of the pipeline.

(NB: Thank you @odedbd to have taken the time to open a discussion)

[NicolasHug]

Opened #19637