minimax-risk-classifier

Supervised classification techniques use training samples to find classification rules with small expected 0-1 loss. Conventional methods achieve efficient learn- ing and out-of-sample generalization by minimizing surrogate losses over specific families of rules. This paper presents minimax risk classifiers (MRCs) that do not rely on a choice of surrogate loss and family of rules. MRCs achieve efficient learning and out-of-sample generalization by minimizing worst-case expected 0-1 loss w.r.t. uncertainty sets that are defined by linear constraints and include the true underlying distribution. In addition, MRCs’ learning stage provides perfor- mance guarantees as lower and upper tight bounds for expected 0-1 loss. We also present MRCs’ finite-sample generalization bounds in terms of training size and smallest minimax risk, and show their competitive classification performance w.r.t. state-of-the-art techniques using benchmark datasets.

Requirements

We will need have installed the following libraries:

• numpy
• sklearn
• cvxpy

We can install the requirements directly from the file "requirements.txt"

pip install -r requirements.txt

Training

To create an instance of the MRC classifier we must first define the following parameters:

• r: number of different classes in the prediction
• phi: Features of the MRC
• lambda_mrc: parameter describing the size of interval estimates (default value, 0.25)

For the MRC instance we also need to know:

• mumVars = number of product thresholds (recomended value, 400)
• k = maximum number of univariate thresholds for each dimension
• d = number of predictor variables / number of columns of dataset X

MRC_model = MRC(r=r, phi=PhiThreshold(r=r, m=int(mumVars/r), k=int(mumVars/(r*d))), lambda_mrc=lambda_mrc)

To train our MRC classifier, we have to pass as inputs, the training data set (X_train) with the predictor variables and the label of each case (y_train)

MRC_model.fit(X_train, y_train)

Evaluation

To collect the results predicted by our classifier, we run the following line:

y_pred= MRC_model.predict(X_test)

Pre-trained models

We have left in the folder "tests/pretrained-models" a pre-trained model to quickly test, the file is "mrc001.sav". To use it, just give the test set X as an input value, in the following example, we use the joblib library to import the pre-trained model:

import joblib

filename = 'pretrained-models/mrc001.sav'
loaded_model = joblib.load(filename)
y_pred= loaded_model.predict(X_test)

Results

Data set	LB	MRC	UB	QDA	DT	KNN	SVM	RF	AMC	MEM
Mammog.	.16	.18 ± .04	.21	.20 ± .04	.24 ± .04	.22 ± .04	.18 ± .03	.21 ± .06	.18 ± .03	.22 ± .04
Haberman	.24	.27 ± .03	.27	.24 ± .03	.39 ± .14	.30 ± .07	.26 ± .04	.35 ± .12	.25 ± .04	.27 ± .02
Indian liv.	.28	.29 ± .01	.30	.44 ± .08	.35 ± .09	.34 ± .05	.29 ± .02	.30 ± .05	.29 ± .01	.29 ± .01
Diabetes	.22	.26 ± .03	.28	.26 ± .03	.29 ± .07	.26 ± .05	.24 ± .04	.26 ± .05	.24 ± .04	.34 ± .04
Credit	.12	.15 ± .18	.17	.22 ± .07	.22 ± .14	.14 ± .09	.16 ± .17	.17 ± .15	.15 ± .18	.14 ± .04
Glass	.22	.36 ± .08	.47	.64 ± .04	.39 ± .18	.34 ± .08	.34 ± .11	.40 ± .14	.42 ± .14	.35 ± .08
Avg. rank	2.7			5.1	7.0	3.8	2.0	5.3	2.5	3.8

Support

Aritz Pérez: aperez@bcamath.org
Andrea Zanoni: andrea.zanoni@epfl.ch
Adrian Diaz Tajuelo: adiaz@bcamath.org
Santiago Mazuelas: smazuelas@bcamath.org

Authors

Aritz Pérez | BCAM-Basque Center of Applied Mathematics
Andrea Zanoni | École Polytechnique Fédérale de Lausanne
Adrian Diaz | BCAM-Basque Center of Applied Mathematics
Santiago Mazuelas | BCAM-Basque Center of Applied Mathematics

License

minimax risk classifiers (MRC) is released under the MIT license.

Citing

@inproceedings{MazZanPer:20,
 author = {Santiago Mazuelas and Andrea Zanoni and Aritz P\'{e}rez},
 booktitle = {Advances in Neural Information Processing Systems},
 pages = {302--312},
 title = {Minimax Classification with 0-1 Loss and Performance Guarantees},
 volume = {33},
 year = {2020}
}