Hands On Classifier Calibration

Table of contents

1. Hands On
2. Installation (optional)
3. Available packages for calibration
4. Additional material
5. Collaborate

Hands On

The hands on is divided into the following sections

1. Visualisation and evaluation tools
2. How to train a calibrator
3. Examples of classifiers and calibrators
   1. Binary
   2. Ternary
4. How to compare calibrator performance

There are three options to follow the Hands On:

1. Interactive by runing the code in your computer (following the installation steps below).
2. Interactive in your webbrowser with Blinder by clicking the following button (no installation is required).
3. Static (pre-rendered) notebooks by clicking in each of the prevous sections (if it fails, try reloading).

Installation (optional)

The Hands On can be run directly in your webbrowser with no installation (go to section Hands On). However, if you want to run the Hands On code in your own machine follow the next steps.

The following instructions has been tested in a Linux machine with Ubuntu 18.04 and Python 3.6. If you know how to install in other Operating systems feel free to add the information in a pull request. It is possible that the code does not work on Windows (the Dirichlet calibration library uses Jax which does not support natively Windows).

Clone this repository

git clone --recurse-submodules https://github.com/classifier-calibration/hands_on.git
cd hands_on
git submodule update

This code is working with Python3.6.

# Use the module venv to create a virtual environment in a folder called venv
python3.6 -m venv venv
# Activate the virtual environment
source venv/bin/activate
# Upgrade package installer for Python
pip install --upgrade pip
# Install all the required dependencies indicated in the requirements.txt file
pip install -r binder/requirements.txt
pip install -e lib/PyCalib
pip install -e lib/dirichlet_python
# Create a kernel with the current virtual environment
python -m ipykernel install --user --name ClaCal --display-name "ClaCal handson"

Then you can start a Jupyter Notebook, and load the created kernel to run the Hands On.

jupyter notebook

Dockerfile (optional)

Running Notebooks on a Docker container.

git clone --recurse-submodules https://github.com/classifier-calibration/hands_on.git
cd hands_on
# build docker image from Dockerfile
export APPUSER=appuser
export IMGNAME_VER=clacal_hands_on:latest
docker build . -t $IMGNAME_VER
# run a container from new image
docker run --rm --name clacal -d -v "$PWD":/home/$APPUSER/data -p 8889:8889 $IMGNAME_VER
# Open browser to http://localhost:8889 (pass: clacal2020)

Available packages for calibration

• Scikit-learn:
  • Calibrated classifiers with cross-validation
  • Isotonic Regression
  • Platt's scaling
• Beta Calibration
• Dirichlet Calibration
• PyCaLib: Python calibration library used in this Hands On
• NetCal
  • includes metrics, reliability diagram and calibration for Neural Nets
• Pakdaman R package: Binary Classifier Calibration Models including BBQ, ENIR, and ELiTE

Additional material

• Beta Calibration
  • Slides Aistats 2017
• Dirichlet Calibration
  • Slides NeurIPS
  • Poster NeurIPS
  • Slides PyData Bristol talk
  • 3min video slides

Collaborate

The previous code, examples, list of libraries, and additional material does not cover everything available online. If you know anything missing in any section feel free to open an issue including your suggestion, or make a pull request with the corresponding changes.