12. Evaluation Metrics for Classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Metrics\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "There are ***MANY*** metrics available for classification problems. It may be a bit ***confusing*** at first, so let's look at the ***confusion matrix*** to understand it better (pun intended!).\n",
    "\n",
    "### 1.1 Confusion Matrix\n",
    "\n",
    "The ***confusion matrix*** is the contingency table of ***actual*** (rows) vs ***predicted*** (columns) ***classes***.\n",
    "\n",
    "Some representations start with positive samples on both first row and columns. But ***Scikit-Learn*** results are returned with ***negative samples first***. So, we're sticking with its convention to avoid confusion!\n",
    "\n",
    "Therefore, a matrix has 4 values, as shown in the picture:\n",
    "\n",
    "![](confusion_matrix.png)\n",
    "\n",
    "The confusion matrix provides the necessary information to build a lot of different metrics.\n",
    "\n",
    "&nbsp; | &nbsp;\n",
    ":---:|:---:\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/264px-Precisionrecall.svg.png) | ![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e7/Sensitivity_and_specificity.svg/264px-Sensitivity_and_specificity.svg.png)\n",
    "<center>Source: Wikipedia</center> | <center>Source: Wikipedia</center>\n",
    "\n",
    "Notice that the matrix is built on top of ***predicted classes***, not ***probabilities***. It means you should first decide on a ***threshold*** to convert probabilities into classes and only then compute the matrix.\n",
    "\n",
    "Changing the ***threshold*** will change the matrix and, consequently, the metrics that depend on its values.\n",
    "\n",
    "So, it is possible to ***tweak the threshold*** to achieve a better performance on a given metric.\n",
    "\n",
    "### 1.2 Accuracy\n",
    "\n",
    "***How often my classifier is right?***\n",
    "\n",
    "This is the most straightforward metric of all - how often a classifier is right, generally speaking.\n",
    "\n",
    "It may be a ***misleading*** metric, though, if the dataset is ***imbalanced***.\n",
    "\n",
    "$$\n",
    "Accuracy = \\frac{TP + TN}{Total}\n",
    "$$\n",
    "\n",
    "### 1.3 Precision\n",
    "\n",
    "***My classifier says it's positive - how often is it right?***\n",
    "\n",
    "If ***False Positives*** are a ***problem***, this is the metric you should pay attention to.\n",
    "\n",
    "Example: if you want to classify videos as ***appropriate for kids*** (positive) or not (negative), you ***really*** don't want a ***false positive***, that is, an ***inappropriate video*** showing up. You will end up ***rejecting good videos***, but that's a lesser problem.\n",
    "\n",
    "$$\n",
    "Precision = \\frac{TP}{TP + FP}\n",
    "$$\n",
    "\n",
    "### 1.4 True Positive Rate (TPR) / Recall / Sensitivity\n",
    "\n",
    "***It IS a positive sample - how often my classifier gets it right?***\n",
    "\n",
    "If ***False Negatives*** are a ***problem***, this is the metric you should pay attention to.\n",
    "\n",
    "Example: if you want to detect if someone has a ***rare and fatal disease*** (positive) or not (negative), you ***really*** don't want a ***false negative***, that is, ***dismissing a sick person***. You will end up ***investigating further healthy people***, but that's a lesser problem.\n",
    "\n",
    "$$\n",
    "Recall = \\frac{TP}{TP + FN}\n",
    "$$\n",
    "\n",
    "### 1.5 False Positive Rate (FPR) / Specificity\n",
    "\n",
    "***It IS a negative sample - how often my classifier gets it wrong?***\n",
    "\n",
    "If ***False Positives*** are a ***problem***, this is the metric you should pay attention to.\n",
    "\n",
    "$$\n",
    "FPR = 1 - Specificity = 1 - \\frac{TN}{TN + FP} = \\frac{FP}{TN + FP}\n",
    "$$\n",
    "\n",
    "### 1.6 F1-Score\n",
    "\n",
    "It is the ***harmonic mean*** of precision and recall, so it combines both metrics into a single value.\n",
    "\n",
    "It favors classifiers that deliver similar levels of precision and recall.\n",
    "\n",
    "$$\n",
    "F_1 = \\frac{2}{\\frac{1}{precision} + \\frac{1}{recall}}\n",
    "$$\n",
    "\n",
    "### Tweaking the Threshold\n",
    "\n",
    "The metrics so far were computed for a given threshold. If we want to compare how they fare whenever we ***change the threshold*** to all its possible values, we need to construct one of these ***curves*** below.\n",
    "\n",
    "They are especially useful to evaluate classifiers on ***imbalanced datasets***.\n",
    "\n",
    "### 1.7 Precision-Recall Curve (Recall x Precision)\n",
    "\n",
    "The ***PR Curve*** depicts the trade-off between ***Recall*** on the horizontal axis and ***Precision*** on the vertical axis.\n",
    "\n",
    "![](https://scikit-learn.org/stable/_images/sphx_glr_plot_precision_recall_001.png)\n",
    "<center>Source: Scikit-Learn</center>\n",
    "\n",
    "You may have noticed the curve is somewhat ***bumpy***.\n",
    "\n",
    "If you ***raise the threshold***, you will move to the ***left*** on the curve. \n",
    "\n",
    "It means you're trying to ***avoid False Positives*** at the expense of ***trading True Positives for False Negatives***.\n",
    "1. More FN reduces Recall (TPR) (less TP has little impact as its on both numerator and denominator)\n",
    "2. Less FP increases precision, but less TP reduces precision\n",
    "\n",
    "But, as you shift the threshold, you may ***lose more TPs than FPs***, and then it will reduce your precision momentarily.\n",
    "\n",
    "### 1.8 ROC Curve (FPR x TPR)\n",
    "\n",
    "The ***ROC Curve*** depicts the trade-off between ***False Positive Rate*** on the horizontal axis and ***True Positive Rate*** on the vertical axis.\n",
    "\n",
    "The shape of the curve will depend on ***how separable*** the classes are:\n",
    "- perfectly separable: the \"curve\" would actually be a square, going straight up to 1 and staying there\n",
    "- completely overlapped: the \"curve\" would actually be a diagonal line, from the origin to the upper right corner\n",
    "- somewhat separable: a curve like the one in the figure below\n",
    "\n",
    "![](https://scikit-learn.org/stable/_images/sphx_glr_plot_roc_001.png)\n",
    "<center>Source: Scikit-Learn</center>\n",
    "\n",
    "If you ***raise the threshold***, you will move to the ***left*** on the curve. \n",
    "\n",
    "It means you're trying to ***avoid False Positives*** at the expense of ***trading True Positives for False Negatives***.\n",
    "1. More FN reduces TPR (Recall) (less TP has little impact as its on both numerator and denominator)\n",
    "2. Less FP reduces FPR\n",
    "\n",
    "Since TP is not present on the calculation of FPR, we ***do not*** observe the bumpiness as in the PR Curve.\n",
    "\n",
    "### 1.9 Area Under ROC\n",
    "\n",
    "The ROC Curve is a very popular method of evaluating a binary classifier. But how does one compare two curves? Unless one of them is strictly better than the other, this would be a difficult task.\n",
    "\n",
    "To make it easier to compare classifiers, one can use the ***area*** under the ROC Curve. The closer it is to ***one***, the better the classifier, as it achieves a high ***TPR*** with a little ***FPR***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "### 2.1 Balanced Dataset\n",
    "\n",
    "There are 1,000 data points in a ***balanced dataset*** of ***green*** (positive) and ***red*** (negative) labels.\n",
    "\n",
    "There is only ***one*** feature represented at the horizontal axis.\n",
    "\n",
    "***Positive*** samples are initially ***centered at 1.0***, while ***negative*** samples are ***centered at -1.0***.\n",
    "\n",
    "A ***Logistic Regression*** was trained on the data and it is shown on top of the distribution of the data (left plot).\n",
    "\n",
    "The ***probability threshold*** is the ***dotted horizontal line*** and it has a linked ***vertical dashed line*** indicating the corresponding ***feature value*** (horizontal axis) for that threshold.\n",
    "\n",
    "The plots on the right show the corresponding ***ROC*** and ***PR*** curves for the trained Logistic Regression, and the ***stars*** correspond to the ***chosen threshold***. At the bottom, there is also a ***confusion matrix***.\n",
    "\n",
    "The controls allow you to:\n",
    "- change the threshold\n",
    "- move the center of the distribution for negative and positive samples\n",
    "\n",
    "Use the controls to play with different configurations and answer the ***questions*** below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.classification.LogisticRegression import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylr = plotLogistic(x=(-1, 1), n_samples=1000, positive=.5)\n",
    "vb = VBox(build_figure_thresh(mylr))\n",
    "vb.layout.align_items = 'center'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e0dd4d3d7df4c30862d154881d7a12e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'marker': {'color': 'green'},\n",
       "              'name': 'Positive Samp…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. Using the ***threshold*** slider:\n",
    "    - raise the threshold - what happens to the star in PR and ROC curves?\n",
    "        - go all the way to 1.0 and check the confusion matrix, as well as the position in both curves\n",
    "    - lower the threshold - what happens to the star in PR and ROC curves?\n",
    "        - go all the way to 0.0 and check the confusion matrix, as well as the position in both curves\n",
    "\n",
    "\n",
    "2. Make the threshold 0.5 again and, using the ***negative center*** slider:\n",
    "    - change the center of the negative samples to -2.00 - what happens to the ***shape*** of PR and ROC curves? Why?\n",
    "        - could you achieve similar result using other algorithm than a logistic regression? Why?\n",
    "    - now change this center to 0.00 - how did the ***shape*** of PR and ROC curves changed? Why?\n",
    "    - which one of the settings is better? Why?\n",
    "    \n",
    "    \n",
    "3. Keep the negative center at 0.0 and, using the ***positive center*** slider:\n",
    "    - change the center of positive samples to 0.2 - what happens to the ***shape*** of PR and ROC curves? Why?\n",
    "        - could you achieve a better result using a more advanced algorithm than a logistic regression? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Imbalanced Dataset\n",
    "\n",
    "There are still 1,000 data points, but now the ***green*** points are about 5% of the total, making it a ***quite imbalanced*** dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylr2 = plotLogistic(x=(-1, 1), n_samples=1000, positive=.05)\n",
    "vb2 = VBox(build_figure_thresh(mylr2))\n",
    "vb2.layout.align_items = 'center'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "667dc7f2cfec41259c9d1bdfdf76bf37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'marker': {'color': 'green'},\n",
       "              'name': 'Positive Samp…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More Questions\n",
    "\n",
    "1. Using the ***threshold*** slider:\n",
    "    - at 0.5, check the ***positive*** samples, what is the ***recall***? What about the ***precision***?\n",
    "    - how to improve the ***recall***? What is the side-effect of doing it?\n",
    "    - what if you need to avoid ***false positives***, how to achieve it?\n",
    "    \n",
    "    \n",
    "2. Make the threshold 0.5 again and, using the ***negative center*** slider:\n",
    "    - change the center of the negative samples to -2.00 - what happens to the ***shape*** of PR and ROC curves? Why? How is this different from what you observed when the dataset was balanced?\n",
    "    - now change this center to 0.00 - how did the ***shape*** of PR and ROC curves changed? Why?\n",
    "    \n",
    "    \n",
    "3. Keep the negative center at 0.0 and, using the ***positive center*** slider:\n",
    "    - change the center of positive samples to 0.2 - what happens to the ***shape*** of PR and ROC curves? Why?\n",
    "    - can you improve the ***recal***? What's the consequence of achieving this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scikit-Learn\n",
    "\n",
    "[Classification Metrics](https://scikit-learn.org/stable/modules/classes.html#classification-metrics)\n",
    "\n",
    "Please check Aurelién Geron's \"Hand-On Machine Learning with Scikit-Learn and Tensorflow\" notebook on Classification [here](https://github.com/ageron/handson-ml/blob/master/03_classification.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[Confused by the Confusion Matrix](https://towardsdatascience.com/confused-by-the-confusion-matrix-e26d5e1d74eb)\n",
    "\n",
    "[ROC and precision-recall with imbalanced datasets](https://classeval.wordpress.com/simulation-analysis/roc-and-precision-recall-with-imbalanced-datasets/)\n",
    "\n",
    "[Introduction to the precision-recall plot](https://classeval.wordpress.com/introduction/introduction-to-the-precision-recall-plot/)\n",
    "\n",
    "[PyCM](https://github.com/sepandhaghighi/pycm/blob/master/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}