cs236756_tutorial_05_evaluation_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/dusk/64/000000/mind-map.png\" style=\"height:50px;display:inline\"> CS 236756 - Technion - Intro to Machine Learning\n",
    "---\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 05 - Evaluation & Cross-Validation\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [Motivation](#-Motivation---Why-Evaluate-Classifier's-Generalization-Ability?)\n",
    "* [Metrics for Classifier's Evaluation](#-Metrics-for-Classifier's-Evaluation)\n",
    "* [Methods for Classifier's Evaluation](#-Methods-for-Classifier's-Evaluation)\n",
    "    * [Holdout](#-Hold-Out-Method)\n",
    "    * [Stratification](#-Stratification)\n",
    "    * [K-Fold Cross-Validation](#-K-Fold-Cross-Validation)\n",
    "    * [Leave-One-Out](#-Leave-One-Out-Cross-Validation)\n",
    "* [Classification Example - Breast Cancer - From Zero to Hero](#-Classification-Example---Breast-Cancer---From-Zero-to-Hero)\n",
    "* [The ROC Curve](#-The-ROC-Curve)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/confetti.png\" style=\"height:50px;display:inline\"> Motivation - Why Evaluate Classifier's Generalization Ability?\n",
    "---\n",
    "It is important to evaluate the classifier generalization performance in order to:\n",
    "* Determine whether to employ/distribute the classifier\n",
    "* Compare classifiers (even compare the same type of classifiers, but with different parameters)\n",
    "* Optimize the classifier to perform better on unseen data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/info.png\" style=\"height:50px;display:inline\"> What Do We Need To Do That?\n",
    "---\n",
    "The first question that needs to be asked is: what preparations are needed in order to evaluate a trained model? \n",
    "\n",
    "* **Train-Test Separation** - The *naive* approach is seprating the data into train set and test set, that is, taking a portion of the data for training the model (usually about 80% of the dataset) and save another portion, that the model **has not seen** in order to test the model's performance. This is called the test set (usually about 20% of the dataset).\n",
    "\n",
    "* Note: Scikit-learn has a function we can use called `train_test_split` that makes it easy for us to split our dataset into training and testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How do we make sure the separation is fair and that each set (train and test) is a good representation of the data distribution?\n",
    "\n",
    "* **Shuffling** - Shuffling the data serves the purpose of reducing variance and making sure that models remain general and overfit less. The popular case where shuffling is very important is when the data is sorted by their class/target. By shuffling we add randomness that assures that the training/test/validation sets are representative of the overall distribution of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total samples: 569\n",
      "total positive sampels (M): 212, total negative samples (B): 357\n"
     ]
    }
   ],
   "source": [
    "# let's load the cancer dataset, shuffle it and  speratre into train and test set\n",
    "dataset = pd.read_csv('./datasets/cancer_dataset.csv')\n",
    "# print the number of rows in the data set\n",
    "number_of_rows = len(dataset)\n",
    "print(\"total samples: {}\".format(number_of_rows))\n",
    "total_positive_samples = np.sum(dataset['diagnosis'].values == 'M')\n",
    "print(\"total positive sampels (M): {}, total negative samples (B): {}\".format(\n",
    "    total_positive_samples, number_of_rows - total_positive_samples))\n",
    "num_train = int(0.8 * number_of_rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>Unnamed: 32</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>552</td>\n",
       "      <td>924084</td>\n",
       "      <td>B</td>\n",
       "      <td>12.770</td>\n",
       "      <td>29.43</td>\n",
       "      <td>81.35</td>\n",
       "      <td>507.9</td>\n",
       "      <td>0.08276</td>\n",
       "      <td>0.04234</td>\n",
       "      <td>0.01997</td>\n",
       "      <td>0.014990</td>\n",
       "      <td>...</td>\n",
       "      <td>36.00</td>\n",
       "      <td>88.10</td>\n",
       "      <td>594.7</td>\n",
       "      <td>0.12340</td>\n",
       "      <td>0.10640</td>\n",
       "      <td>0.08653</td>\n",
       "      <td>0.06498</td>\n",
       "      <td>0.2407</td>\n",
       "      <td>0.06484</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>215</td>\n",
       "      <td>8810987</td>\n",
       "      <td>M</td>\n",
       "      <td>13.860</td>\n",
       "      <td>16.93</td>\n",
       "      <td>90.96</td>\n",
       "      <td>578.9</td>\n",
       "      <td>0.10260</td>\n",
       "      <td>0.15170</td>\n",
       "      <td>0.09901</td>\n",
       "      <td>0.056020</td>\n",
       "      <td>...</td>\n",
       "      <td>26.93</td>\n",
       "      <td>104.40</td>\n",
       "      <td>750.1</td>\n",
       "      <td>0.14600</td>\n",
       "      <td>0.43700</td>\n",
       "      <td>0.46360</td>\n",
       "      <td>0.16540</td>\n",
       "      <td>0.3630</td>\n",
       "      <td>0.10590</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>855133</td>\n",
       "      <td>M</td>\n",
       "      <td>14.990</td>\n",
       "      <td>25.20</td>\n",
       "      <td>95.54</td>\n",
       "      <td>698.8</td>\n",
       "      <td>0.09387</td>\n",
       "      <td>0.05131</td>\n",
       "      <td>0.02398</td>\n",
       "      <td>0.028990</td>\n",
       "      <td>...</td>\n",
       "      <td>25.20</td>\n",
       "      <td>95.54</td>\n",
       "      <td>698.8</td>\n",
       "      <td>0.09387</td>\n",
       "      <td>0.05131</td>\n",
       "      <td>0.02398</td>\n",
       "      <td>0.02899</td>\n",
       "      <td>0.1565</td>\n",
       "      <td>0.05504</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>235</td>\n",
       "      <td>88249602</td>\n",
       "      <td>B</td>\n",
       "      <td>14.030</td>\n",
       "      <td>21.25</td>\n",
       "      <td>89.79</td>\n",
       "      <td>603.4</td>\n",
       "      <td>0.09070</td>\n",
       "      <td>0.06945</td>\n",
       "      <td>0.01462</td>\n",
       "      <td>0.018960</td>\n",
       "      <td>...</td>\n",
       "      <td>30.28</td>\n",
       "      <td>98.27</td>\n",
       "      <td>715.5</td>\n",
       "      <td>0.12870</td>\n",
       "      <td>0.15130</td>\n",
       "      <td>0.06231</td>\n",
       "      <td>0.07963</td>\n",
       "      <td>0.2226</td>\n",
       "      <td>0.07617</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>501</td>\n",
       "      <td>91504</td>\n",
       "      <td>M</td>\n",
       "      <td>13.820</td>\n",
       "      <td>24.49</td>\n",
       "      <td>92.33</td>\n",
       "      <td>595.9</td>\n",
       "      <td>0.11620</td>\n",
       "      <td>0.16810</td>\n",
       "      <td>0.13570</td>\n",
       "      <td>0.067590</td>\n",
       "      <td>...</td>\n",
       "      <td>32.94</td>\n",
       "      <td>106.00</td>\n",
       "      <td>788.0</td>\n",
       "      <td>0.17940</td>\n",
       "      <td>0.39660</td>\n",
       "      <td>0.33810</td>\n",
       "      <td>0.15210</td>\n",
       "      <td>0.3651</td>\n",
       "      <td>0.11830</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>858970</td>\n",
       "      <td>B</td>\n",
       "      <td>10.170</td>\n",
       "      <td>14.88</td>\n",
       "      <td>64.55</td>\n",
       "      <td>311.9</td>\n",
       "      <td>0.11340</td>\n",
       "      <td>0.08061</td>\n",
       "      <td>0.01084</td>\n",
       "      <td>0.012900</td>\n",
       "      <td>...</td>\n",
       "      <td>17.45</td>\n",
       "      <td>69.86</td>\n",
       "      <td>368.6</td>\n",
       "      <td>0.12750</td>\n",
       "      <td>0.09866</td>\n",
       "      <td>0.02168</td>\n",
       "      <td>0.02579</td>\n",
       "      <td>0.3557</td>\n",
       "      <td>0.08020</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>314</td>\n",
       "      <td>894047</td>\n",
       "      <td>B</td>\n",
       "      <td>8.597</td>\n",
       "      <td>18.60</td>\n",
       "      <td>54.09</td>\n",
       "      <td>221.2</td>\n",
       "      <td>0.10740</td>\n",
       "      <td>0.05847</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>22.44</td>\n",
       "      <td>56.65</td>\n",
       "      <td>240.1</td>\n",
       "      <td>0.13470</td>\n",
       "      <td>0.07767</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.3142</td>\n",
       "      <td>0.08116</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>366</td>\n",
       "      <td>9011494</td>\n",
       "      <td>M</td>\n",
       "      <td>20.200</td>\n",
       "      <td>26.83</td>\n",
       "      <td>133.70</td>\n",
       "      <td>1234.0</td>\n",
       "      <td>0.09905</td>\n",
       "      <td>0.16690</td>\n",
       "      <td>0.16410</td>\n",
       "      <td>0.126500</td>\n",
       "      <td>...</td>\n",
       "      <td>33.81</td>\n",
       "      <td>160.00</td>\n",
       "      <td>1671.0</td>\n",
       "      <td>0.12780</td>\n",
       "      <td>0.34160</td>\n",
       "      <td>0.37030</td>\n",
       "      <td>0.21520</td>\n",
       "      <td>0.3271</td>\n",
       "      <td>0.07632</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>73</td>\n",
       "      <td>859983</td>\n",
       "      <td>M</td>\n",
       "      <td>13.800</td>\n",
       "      <td>15.79</td>\n",
       "      <td>90.43</td>\n",
       "      <td>584.1</td>\n",
       "      <td>0.10070</td>\n",
       "      <td>0.12800</td>\n",
       "      <td>0.07789</td>\n",
       "      <td>0.050690</td>\n",
       "      <td>...</td>\n",
       "      <td>20.86</td>\n",
       "      <td>110.30</td>\n",
       "      <td>812.4</td>\n",
       "      <td>0.14110</td>\n",
       "      <td>0.35420</td>\n",
       "      <td>0.27790</td>\n",
       "      <td>0.13830</td>\n",
       "      <td>0.2589</td>\n",
       "      <td>0.10300</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>398</td>\n",
       "      <td>904302</td>\n",
       "      <td>B</td>\n",
       "      <td>11.060</td>\n",
       "      <td>14.83</td>\n",
       "      <td>70.31</td>\n",
       "      <td>378.2</td>\n",
       "      <td>0.07741</td>\n",
       "      <td>0.04768</td>\n",
       "      <td>0.02712</td>\n",
       "      <td>0.007246</td>\n",
       "      <td>...</td>\n",
       "      <td>20.35</td>\n",
       "      <td>80.79</td>\n",
       "      <td>496.7</td>\n",
       "      <td>0.11200</td>\n",
       "      <td>0.18790</td>\n",
       "      <td>0.20790</td>\n",
       "      <td>0.05556</td>\n",
       "      <td>0.2590</td>\n",
       "      <td>0.09158</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "552    924084         B       12.770         29.43           81.35      507.9   \n",
       "215   8810987         M       13.860         16.93           90.96      578.9   \n",
       "38     855133         M       14.990         25.20           95.54      698.8   \n",
       "235  88249602         B       14.030         21.25           89.79      603.4   \n",
       "501     91504         M       13.820         24.49           92.33      595.9   \n",
       "60     858970         B       10.170         14.88           64.55      311.9   \n",
       "314    894047         B        8.597         18.60           54.09      221.2   \n",
       "366   9011494         M       20.200         26.83          133.70     1234.0   \n",
       "73     859983         M       13.800         15.79           90.43      584.1   \n",
       "398    904302         B       11.060         14.83           70.31      378.2   \n",
       "\n",
       "     smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "552          0.08276           0.04234         0.01997             0.014990   \n",
       "215          0.10260           0.15170         0.09901             0.056020   \n",
       "38           0.09387           0.05131         0.02398             0.028990   \n",
       "235          0.09070           0.06945         0.01462             0.018960   \n",
       "501          0.11620           0.16810         0.13570             0.067590   \n",
       "60           0.11340           0.08061         0.01084             0.012900   \n",
       "314          0.10740           0.05847         0.00000             0.000000   \n",
       "366          0.09905           0.16690         0.16410             0.126500   \n",
       "73           0.10070           0.12800         0.07789             0.050690   \n",
       "398          0.07741           0.04768         0.02712             0.007246   \n",
       "\n",
       "     ...  texture_worst  perimeter_worst  area_worst  smoothness_worst  \\\n",
       "552  ...          36.00            88.10       594.7           0.12340   \n",
       "215  ...          26.93           104.40       750.1           0.14600   \n",
       "38   ...          25.20            95.54       698.8           0.09387   \n",
       "235  ...          30.28            98.27       715.5           0.12870   \n",
       "501  ...          32.94           106.00       788.0           0.17940   \n",
       "60   ...          17.45            69.86       368.6           0.12750   \n",
       "314  ...          22.44            56.65       240.1           0.13470   \n",
       "366  ...          33.81           160.00      1671.0           0.12780   \n",
       "73   ...          20.86           110.30       812.4           0.14110   \n",
       "398  ...          20.35            80.79       496.7           0.11200   \n",
       "\n",
       "     compactness_worst  concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "552            0.10640          0.08653               0.06498          0.2407   \n",
       "215            0.43700          0.46360               0.16540          0.3630   \n",
       "38             0.05131          0.02398               0.02899          0.1565   \n",
       "235            0.15130          0.06231               0.07963          0.2226   \n",
       "501            0.39660          0.33810               0.15210          0.3651   \n",
       "60             0.09866          0.02168               0.02579          0.3557   \n",
       "314            0.07767          0.00000               0.00000          0.3142   \n",
       "366            0.34160          0.37030               0.21520          0.3271   \n",
       "73             0.35420          0.27790               0.13830          0.2589   \n",
       "398            0.18790          0.20790               0.05556          0.2590   \n",
       "\n",
       "     fractal_dimension_worst  Unnamed: 32  \n",
       "552                  0.06484          NaN  \n",
       "215                  0.10590          NaN  \n",
       "38                   0.05504          NaN  \n",
       "235                  0.07617          NaN  \n",
       "501                  0.11830          NaN  \n",
       "60                   0.08020          NaN  \n",
       "314                  0.08116          NaN  \n",
       "366                  0.07632          NaN  \n",
       "73                   0.10300          NaN  \n",
       "398                  0.09158          NaN  \n",
       "\n",
       "[10 rows x 33 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reminder, the data looks like this\n",
    "# dataset.head(10) # the dataset is ordered by the diagnosis\n",
    "dataset.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total training samples: 455, total test samples: 114\n"
     ]
    }
   ],
   "source": [
    "# we will take the first 2 features as our data (X) and the diagnosis as labels (y)\n",
    "x = dataset[['radius_mean', 'texture_mean']].values\n",
    "y = dataset['diagnosis'].values == 'M'  # 1 for Malignant, 0 for Benign\n",
    "# shuffle\n",
    "rand_gen = np.random.RandomState(0)\n",
    "shuffled_indices = rand_gen.permutation(np.arange(len(x)))\n",
    "\n",
    "x_train = x[shuffled_indices[:num_train]]\n",
    "y_train = y[shuffled_indices[:num_train]]\n",
    "x_test = x[shuffled_indices[num_train:]]\n",
    "y_test = y[shuffled_indices[num_train:]]\n",
    "\n",
    "print(\"total training samples: {}, total test samples: {}\".format(num_train, number_of_rows - num_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/rating.png\" style=\"height:50px;display:inline\"> Metrics for Classifier's Evaluation\n",
    "---\n",
    "Different ML tasks use different metrics to measure the models' performance, we will introduce several of them, but there are more, and sometimes there are tasks that require hand-crafted metrics (for example, an NLP tasks of translating sentences from Chinese to English is measured by the BLEU score, which is a linguistic measure of language coherence)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Throught this part, we will use the terms: True Positive (TP), False Positive (FP), True Negative (TN), False Negative (FN) which are demonstrated below:\n",
    "<img src=\"./assets/tut_05_tp_tf.jpg\" style=\"height:400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Accuracy** - $\\frac{TP + TN}{P + N}$\n",
    "    * In simple words: how many did we get *right* out of all of the dataset?\n",
    "    * When not to use *accuracy*?\n",
    "        * When dealing with *skewed* datasets (i.e., when some classes/labels are more frequent than others). \n",
    "* **Error** - $\\frac{FP + FN}{P + N}$\n",
    "    * In simple words: how many did we get *wrong* out of all of the dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Precision** - $\\frac{TP}{TP + FP}$\n",
    "    * In simple words: out of all the samples we classified as *positive*, how many of them we got *right*. The accuracy of positive predictions.\n",
    "    * **Always** calculated along with **Recall**\n",
    "* **Recall (TP Rate, Sensitivity)** - $\\frac{TP}{P} = \\frac{TP}{TP + FN}$\n",
    "    * In simple words: out of all the *positive* samples, how many of them we got *right*. \n",
    "    * **Always** calculated along with **Precision**\n",
    "* **FP Rate** - $\\frac{FP}{N} = \\frac{FP}{FP + TN}$\n",
    "    * In simple words: out of all the *negative* samples, how many of them we got *wrong*, or, the ratio of negative instances that are incorrectly classified as positive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Quick Examples\n",
    "---\n",
    "   * A classifier that is trained to detect videos that are safe for kids. You would probably prefer a classifier that rejects many good videos (low recall) but keeps only safe ones (high precision), rather than a classifier that has a much higher recall but lets a few really bad videos.\n",
    "   * A classifier trained to detect shoplifters on surveillance images: it is probably fine if your classifier has only 30% precision as long as it has 99% recall (almost all shoplifters will get caught. but a lot of false alarms)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/classroom.png\" style=\"height:50px;display:inline\"> Example - Naive Classifier Evaluation on the Breast Cancer Dataset\n",
    "---\n",
    "Let's create a classifier that classifies new samples by the label probability it has seen in the train set. That is, if it has seen 30% Malignant labels, then with probability 0.3 a new sample is classified as malignant.\n",
    "\n",
    "We will evaluate using the above metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "M prob: 0.369, B prob: 0.631\n"
     ]
    }
   ],
   "source": [
    "# probability to tag as malignat\n",
    "m_prob = np.sum(y_train) / len(y_train)\n",
    "# probability to tag as benign\n",
    "b_prob = 1 - m_prob\n",
    "print(\"M prob: {:.3f}, B prob: {:.3f}\".format(m_prob, b_prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# now let's classify the test set\n",
    "# since we don't look at the data at all we can just randomly sample Ms and Bs in the size of the test set\n",
    "y_test_pred = np.random.choice([True, False], size=(len(y_test)), p=[m_prob, b_prob])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.544 or 54.386 %\n",
      "error: 0.456 or 45.614 %\n"
     ]
    }
   ],
   "source": [
    "# let's evaluate\n",
    "accuracy = np.sum(y_test == y_test_pred) / len(y_test)\n",
    "print(\"accuracy: {:.3f} or {:.3f} %\".format(accuracy, accuracy * 100))\n",
    "error = np.sum(y_test != y_test_pred) / len(y_test)\n",
    "print(\"error: {:.3f} or {:.3f} %\".format(error, error * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: 0.395 or 39.474 %\n",
      "recall: 0.341 or 34.091 %\n",
      "FP Rate: 0.329 or 32.857 %\n"
     ]
    }
   ],
   "source": [
    "precision = np.sum(y_test_pred[y_test]) / np.sum(y_test_pred)\n",
    "print(\"precision: {:.3f} or {:.3f} %\".format(precision, precision * 100))\n",
    "recall = np.sum(y_test_pred[y_test]) / np.sum(y_test)\n",
    "print(\"recall: {:.3f} or {:.3f} %\".format(recall, recall * 100))\n",
    "fp_rate =  np.sum(y_test_pred[~y_test]) / np.sum(~y_test)\n",
    "print(\"FP Rate: {:.3f} or {:.3f} %\".format(fp_rate, fp_rate * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: 0.395 or 39.474 %\n",
      "recall: 0.341 or 34.091 %\n"
     ]
    }
   ],
   "source": [
    "# using scikit-learn\n",
    "from sklearn.metrics import precision_score, recall_score\n",
    "precision = precision_score(y_test, y_test_pred)\n",
    "print(\"precision: {:.3f} or {:.3f} %\".format(precision, precision * 100))\n",
    "recall = recall_score(y_test, y_test_pred)\n",
    "print(\"recall: {:.3f} or {:.3f} %\".format(recall, recall * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/planner.png\"  style=\"height:50px;display:inline\"> Confusion Matrix Revisited\n",
    "---\n",
    "A better way to evaluate the performance of a classifier is the look at the *confusion matrix*. Each row in a confusion matrix represents an actual class. A perfect classifier would have only *true positives* and *true negatives*, so its confusion matrix would have non-zeros values only on its main diagonal.\n",
    "<img src=\"./assets/tut_05_conf_mat.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted Pos</th>\n",
       "      <th>Predicted Neg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Actual Pos</td>\n",
       "      <td>47</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Actual Neg</td>\n",
       "      <td>29</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Predicted Pos  Predicted Neg\n",
       "Actual Pos             47             23\n",
       "Actual Neg             29             15"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using scikit-learn\n",
    "from sklearn.metrics import confusion_matrix\n",
    "conf_mat = confusion_matrix(y_test, y_test_pred)\n",
    "conf_mat_df = pd.DataFrame(conf_mat, columns=['Predicted Pos', 'Predicted Neg'], index=['Actual Pos', 'Actual Neg'])\n",
    "conf_mat_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* It is sometimes nicer to present confusion matrix as a heat map (the same way you would do for correlation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/ios/100/000000/battle.png\" style=\"height:50px;display:inline\"> Precision/Recall Tradeoff\n",
    "---\n",
    "**Increasing precision reduces recall, and vice versa.** It is often convenient to combine them into a single metric called the $F_1$ score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/f.png\" style=\"height:50px;display:inline\"> The $F_1$ Score\n",
    "---\n",
    "It is the *harmonic mean* of precision and recall.\n",
    "$$F_1 = \\frac{2}{\\frac{1}{precision} + \\frac{1}{recall}} = 2 \\times \\frac{precision \\times recall}{precision + recall} = \\frac{TP}{TP + \\frac{FN + FP}{2}} $$\n",
    "Whereas the regular mean treats all values equally , the harmonic mean gives much more weight to **low values**. As a result, the classifier will only get a high $F_1$ score if both recall and precision are high.\n",
    "\n",
    "The $F_1$ score favors classifiers that have similar precision and recall. This is not always what we want, as seen in the \"Quick Examples\" above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1 score: 0.366\n",
      "f1 score (scikit): 0.366\n"
     ]
    }
   ],
   "source": [
    "# calculating the f1 score\n",
    "f_1_score = 2 * precision * recall / (precision + recall)\n",
    "print(\"f1 score: {:.3f}\".format(f_1_score))\n",
    "# using scikit-learn\n",
    "from sklearn.metrics import f1_score\n",
    "f_1_score = f1_score(y_test, y_test_pred)\n",
    "print(\"f1 score (scikit): {:.3f}\".format(f_1_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/clouds/100/000000/combo-chart.png\" style=\"height:50px;display:inline\"> Methods for Classifier's Evaluation\n",
    "---\n",
    "So far we have separated our data to a train set and *test set* to evaluate our classifier. This is also called **hold-out** method.\n",
    "\n",
    "Let's list the ways we can estimate the metrics for a certain classifier:\n",
    "* **Training Data**\n",
    "* **Independent Test Data** (different from hold-out)\n",
    "* **Hold-Out Method**\n",
    "* **$K$-fold Cross-Validation**\n",
    "* **Leave-One-Out Method**\n",
    "* **Bootstrap Method**\n",
    "* And more...\n",
    "\n",
    "We will now present some of them that you will use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/dumbbell.png\" style=\"height:50px;display:inline\"> Estimation with Training Data\n",
    "---\n",
    "<img src=\"./assets/tut_05_train.jpg\" style=\"height:100px\">\n",
    "\n",
    "* The accuracy/error estimates on the *training* data **are not** good indicators of performance on future data!\n",
    "* The reason: new data will probably not be exactly the same as the training data.\n",
    "* The accuracy/error estimates on the training data measure the degree of classifier's **underfitting** or **overfitting**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* A typical learning curve usually looks like this: <img src=\"./assets/tut_05_overfitting.png\" style=\"height:200px\">\n",
    "<a href=\"http://mlwiki.org/index.php/Overfitting\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/100/000000/test-passed.png\" style=\"height:50px;display:inline\"> Estimation with Independent Test Data\n",
    "---\n",
    "<img src=\"./assets/tut_05_test.jpg\" style=\"height:100px\">\n",
    "\n",
    "* Estimation with independent test data is used when we have plenty of data and there is **a natural way to generating this data**, that is, there is no need to separate the the train data.\n",
    "* For example, we wish to learn a certain function, like $f(x) = x^2$, we can generate how many samples we want and we are not bounded to a certain dataset.\n",
    "* In most ML tasks, we have no natural way of forming data, and we are limited to the data given to us, which is why it is more common to use hold-out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/000000/separate-document.png\" style=\"height:50px;display:inline\"> Hold-Out Method\n",
    "---\n",
    "<img src=\"./assets/tut_05_holdout.jpg\" style=\"height:150px\">\n",
    "\n",
    "* The hold-out method **splits** the data into training data and test data. Then, we build a classifier using the train set and test it using the test set.\n",
    "* The hold-out method is usually used when we have thousands of instances, including several hundred instances from each class.\n",
    "* Scikit-learn has a built in function that you can use to split the data: `from sklearn.model_selection import train_test_split`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/100/000000/more.png\" style=\"height:50px;display:inline\"> Train-Validation-Test Split\n",
    "---\n",
    "* Let's say you have a dataset which you separated into *train* set and *test* set. \n",
    "* You want to train a classifier that has hyper-parameters (parameters that are not trained, but are user-selected before the training) that you have to tune. \n",
    "    * For example, in the classifier \"K-Nearest Neighbours\" you need to choose $K$, the number of nearest neighbours needed to perform classification, or in \"Stochastic Gradient Descent\", you need to choose the learning rate (which is a continuous value, like 0.0001). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Would it be fair to train the classifier on the train set for each hyper-parameter and then test it on the test set and finally selecting the best hyper-parameters based on the performance on the test set? **NO!**\n",
    "    * It is like taking an open-material exam, but instead of bringing all of the material, you bring only the material relevant to the questions asked in the exam. It is sort of cheating."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "That is why we seperate into 3 sets:\n",
    "\n",
    "* **Train Set** - from which the model learns\n",
    "* **Validation Set** - on which the hyper-parameters are tuned\n",
    "* **Test Set** - untouched samples on which you test the generalization ability of the model. This set has **never** been seen by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_05_validation.jpg\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/cotton/64/000000/full-battery.png\" style=\"height:50px;display:inline\"> Making the Most of the Data\n",
    "---\n",
    "* Once evaluation is complete and you are satisfied with the model, *all* the data can be used to build the final classifier\n",
    "* Generally, the **larger the training data** the **better the classifier**.\n",
    "* The **larger the test set** the **more accurate** the error estimate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/clouds/100/000000/color-dropper.png\" style=\"height:50px;display:inline\"> Stratification\n",
    "---\n",
    "* The *holdout* method reserves a certain amount for testing and uses the remainder for training.\n",
    "* For **\"unbalanced\"/skewed** datasets, samples might not be representative.\n",
    "    * Few or None instances of some classes\n",
    "* **Stratified Sampling**: balancing the data. If the dataset is not large enough, there is a risk of introducing a significant sampling *bias*. In *stratified sampling*, the population is divided into homogenous subgroups called *strata*, and the right number of instances is sampled from each stratum to guarantee that the test set is representative of the overall population.\n",
    "    * Makes sure that each class is represented with approximately equal proportions in both subsets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* For example, when a survey company decides to call 1,000 people to ask them a few questions, they don't just pick up randomly 1,000 people from a phone book. They try to ensure that these 1,000 people are representative of the whole population. In the US, the population is composed of 51.3% female and 48.7% men, so a well-conducted survey in the US would try to maintain this ratio in the samples - 513 females and 487 males.\n",
    "* In Scikit-learn - `from sklearn.model_selection import StratifiedShuffleSplit` - <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html\">Read More Here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/000000/delete-sign.png\" style=\"height:50px;display:inline\"> K-Fold Cross-Validation\n",
    "---\n",
    "* Separating to validation and test sets is not very data efficient as we now allocate less data for training.\n",
    "* In order to make better use of the data we use a technique called **\"Cross Validation\"**. \n",
    "    * However, in this course we will always do this separation regardless! (we'll use the train set for CV).\n",
    "* Cross-validation is randomly splitting the data into $k$ groups. One of the groups is used as the test set and the rest are used as the training set. \n",
    "* The model is trained on the training set and scored on the test set. Then the process is repeated until each unique group as been used as the test set. This method is also called **\"K-Fold Cross Validation\"**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* For example, for 5-fold cross validation, the dataset is split into 5 groups, and the model is trained and tested 5 separate times so each group gets a chance to be the test set. This can be seen in the image below.\n",
    "<img src=\"./assets/tut_05_kcv.jpeg\" style=\"height:250px\" /> \n",
    "\n",
    "    * (Image from Datacamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Cross-validation is *better* than using the holdout method because the holdout method score is dependent on how the data is split into train, validation and test sets. \n",
    "* Cross-validation gives the model an opportunity to test on multiple splits so we can get a better idea on how the model will perform on unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**The steps:**\n",
    "1. Data is split into $k$ subsets of equal size.\n",
    "2. Each subset in turn is used for testing and the remainder for training.\n",
    "3. The estimates are averaged to yield an overall estimate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Implementation using Scikit-learn:**\n",
    "* `cross_val_score` - takes in a classifier, training data, $k$ (number of folds) and the scoring technique (\"accuracy\" for example). <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html\">Read the Doc Here</a>\n",
    "* `StratifiedKFold` - if you need more control over the cross-validation process than provided in `cross_val_score`, it is possible to implement cross-validation yourself. The `StratifiedKFold` class performs stratified sampling to produce folds that contain a representative ratio of each class. <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html\">Read the Doc Here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**More on Cross-Validation:**\n",
    "* Standard method for evaluation: *stratified 10-fold cross validation*\n",
    "* Stratification reduces the estimate's *variance* (more confidence in the output)\n",
    "* An even better method: **repeated** stratified cross-validation\n",
    "    * For example, 10-fold cross-validation is repeated 10 times and the results are averaged (reduces the variance).\n",
    "* Which dataset should we use for the $K$-Fold Cross Validation (Train, Test, Validation)?\n",
    "    * The **test set** - must be untouched, put aside. Don't use it for ANY purpose other than testing.\n",
    "    * The **validation set** - used to test performances of the models at various stages of the model development (like tuning the hyper-parameters of the model)\n",
    "    * The **train set** - used for training, including cross-validation. **USE ONLY THIS FOR CV!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/doodle/100/000000/big-puzzle--v1.png\" style=\"height:50px;display:inline\"> Leave-One-Out Cross-Validation\n",
    "---\n",
    "* A particular form of cross-validation where the number of folds = number of training instances\n",
    "    * That is, for training set of size $n$, the classifier is built $n$ times\n",
    "* Makes best use of the data\n",
    "* Involves **no random subsampling**\n",
    "* Very computationally **expensive**\n",
    "* **Stratification is not possible** - it is a big disadvantage\n",
    "    * It *guarantees* a non-stratified sample because there is only one instance in the test set.\n",
    "* Extreme example - random datatset split equally into two classes:\n",
    "    * Best inducer predicts majority class\n",
    "    * 50% accuracy on fresh data\n",
    "    * Leave-One-Out CV estimate is 100% error!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Pre-proceesing steps on validation steps\n",
    "---\n",
    "* Whatever data preparations steps done on the training data, should be able to be applied at the final test on unseen data.\n",
    "    * For example, if normalizing a feature with a mean, the mean is fixed at the data prep and **not recalculated**. Otherwise, a model that was trained and tested on data that used the former mean may not be adequate.\n",
    "* Data preparation steps are mostly based on descriptive statistics. So a **larger sample** that provides a **stable statistics** is an advantage.\n",
    "* By not using the validation set, we have a **better estimate of the \"true error\"**, which we measure by applying pre-processing steps to test data.\n",
    "* By using the validation set, we **separate the effect of the pre-processing** from that of the classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/color/96/000000/super-hero-female.png\" style=\"height:50px;display:inline\"> Classification Example - Breast Cancer - From Zero to Hero\n",
    "---\n",
    "We will now demonstrate all that we have learned on the Breast Cancer dataset. For the purpose of the exercise, we will use a classifier called \"SGDClassifier\" (SVM using Stochastic Gradient Descent). You are not supposed to be familiar with it at this point of the course, you can read more about it <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html\">here</a>. We will use it as a black-box algorithm that gives us classification for a certain dataset.\n",
    "\n",
    "* We will only use *train* and *test* sets since the dataset has limited number of samples and since we are not going to tune the hyper-parameters of the model.\n",
    "* Each classifier has a **decision function** - for each instance, it computes a score and if that score is greater than some threshold, it assigns the instance to the positive class, or else it assigns it to the negative class (lowering the threshold usually leads to increase in *recall* and reduce in *precision*, and vice versa).\n",
    "* Finally, will briefly introduce the **ROC Curve**, as a tool to asses to performance of the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# scikit-learn imports\n",
    "from sklearn.model_selection import cross_val_score, cross_val_predict\n",
    "from sklearn.metrics import confusion_matrix, f1_score, precision_recall_curve, roc_curve, roc_auc_score\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# prepare the dataset\n",
    "# we will take the first 2 features as our data (X) and the diagnosis as labels (y)\n",
    "x = dataset[['radius_mean', 'texture_mean']].values\n",
    "y = dataset['diagnosis'].values == 'M'  # 1 for Malignat, 0 for Benign\n",
    "# x = scaler.fit_transform(x)\n",
    "# shuffle\n",
    "rand_gen = np.random.RandomState(0)\n",
    "shuffled_indices = rand_gen.permutation(np.arange(len(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x_train = x[shuffled_indices[:num_train]]\n",
    "y_train = y[shuffled_indices[:num_train]]\n",
    "x_test = x[shuffled_indices[num_train:]]\n",
    "y_test = y[shuffled_indices[num_train:]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total training samples: 455, total test samples: 114\n"
     ]
    }
   ],
   "source": [
    "# pre-process - standartization\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(x_train)\n",
    "x_train = scaler.transform(x_train)\n",
    "x_test = scaler.transform(x_test)\n",
    "\n",
    "print(\"total training samples: {}, total test samples: {}\".format(num_train, number_of_rows - num_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy in each fold:\n",
      "[0.84782609 0.91304348 0.86956522 0.97826087 0.93478261 0.84782609\n",
      " 0.84782609 0.73333333 0.93181818 0.88636364]\n",
      "mean training accuracy:\n",
      "0.8790645586297761\n"
     ]
    }
   ],
   "source": [
    "# create the classifier\n",
    "sgd_clf = SGDClassifier(random_state=92, max_iter=1000, tol=1e-3)  \n",
    "# we make the random state constant for reproducible results\n",
    "# train (fit) using cross validation\n",
    "k_folds = 10\n",
    "cross_val_scores = cross_val_score(sgd_clf, x_train, y_train, cv=k_folds, scoring='accuracy')\n",
    "print(\"accuracy in each fold:\")\n",
    "print(cross_val_scores)\n",
    "print(\"mean training accuracy:\")\n",
    "print(cross_val_scores.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
       "              early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "              l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
       "              max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
       "              power_t=0.5, random_state=92, shuffle=True, tol=0.001,\n",
       "              validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now, fit the classifier to the train data\n",
    "sgd_clf.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted Pos</th>\n",
       "      <th>Predicted Neg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Actual Pos</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Actual Neg</td>\n",
       "      <td>14</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Predicted Pos  Predicted Neg\n",
       "Actual Pos             70              0\n",
       "Actual Neg             14             30"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluation\n",
    "y_test_pred = sgd_clf.predict(x_test)\n",
    "# confusion matrix\n",
    "conf_mat = confusion_matrix(y_test, y_test_pred)\n",
    "conf_mat_df = pd.DataFrame(conf_mat, columns=['Predicted Pos', 'Predicted Neg'], index=['Actual Pos', 'Actual Neg'])\n",
    "conf_mat_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 87.719 % , f1 score: 0.811\n"
     ]
    }
   ],
   "source": [
    "# accuracy\n",
    "accuracy = np.sum(y_test == y_test_pred) / len(y_test)\n",
    "# f1 score\n",
    "f1 = f1_score(y_test, y_test_pred)\n",
    "print(\"accuracy: {:.3f} % , f1 score: {:.3f}\".format(accuracy * 100, f1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-2.09830569]\n"
     ]
    }
   ],
   "source": [
    "# let's get the scores for some instance\n",
    "y_scores = sgd_clf.decision_function([x_train[0]])\n",
    "# for SGDClssifier, the default threshold is 0, anything below is calssified as negative and else positive\n",
    "print(y_scores)\n",
    "# let's see the effect of the threshold on the training data\n",
    "y_scores = cross_val_predict(sgd_clf, x_train, y_train, cv=k_folds, method=\"decision_function\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def plot_pr_recall_curve(labels, scores):\n",
    "    precisions, recalls, thresholds = precision_recall_curve(labels, scores)\n",
    "    fig = plt.figure(figsize=(10, 5))\n",
    "    ax = fig.add_subplot(1,1,1)\n",
    "    ax.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\")\n",
    "    ax.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\")\n",
    "    ax.set_xlabel(\"Threshold\")\n",
    "    ax.set_title(\"Precision/Recall Tradeoff\")\n",
    "    ax.legend()\n",
    "    ax.grid()\n",
    "    ax.set_ylim([0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFNCAYAAAAkdeqeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZxN9ePH8ddn9jFjrGMsYyvrWIexZ08hEikkvxYlhEol9a1o/6Z98y31jbSJivJNoRhLKGTIvmQbO9nGNmPm8/vjXBoMBjNz7tx5Px+Peczce869933niHef87mfY6y1iIiIiMjl8XM7gIiIiEhupjIlIiIicgVUpkRERESugMqUiIiIyBVQmRIRERG5AipTIiIiIldAZUpERETkCqhMicg5jDErjDEtLrJPGWNMkjHGP4diZStjTAtjTGK625uMMde6lOV5Y8yYLHgeY4wZa4w5YIyZ57lvgDFmt+fYFbjisCKiMiWSm3j+gT/m+YdwlzFmtDEmPKtfx1pbzVobf5F9tlhrw621qVnxmsaYkqfKzFnvc6cxZkx2vM/LYYz50ZMryRiTYoxJTnf7fbfznaUF0Bwoaa1tbIwJAV4FWnqO3UFX04n4CJUpkdyno7U2HKgD1AOePHsHz4hEbvvvuz3wU7rbp95nbSAWeNyVVGex1rbzFJFw4HNgxKnb1tq+Z+9vjAnI+ZSnlQU2WmuPem4XB4KttStczCTic3LbX7Yi4mGt3Qb8CFQHMMbEG2NeMMb8ChwFrjLGFDDG/NcYs8MYs81z+uj0aTljzL3GmFXGmMPGmJXGmDqe+0+f4jLG1DfGLDLGHPKMhr3uub+cMcaeKguekaXvjTF/G2PWG2PuTfc6w40x4z2nnA57TiPGnfWW2gNTMnifO4GpOKXq1PMFG2NeNcZs8WR63xgTmm57J2NMgifzBmNMW8/9d6V7v38ZY+67kmOQEWPMtZ7f3xPGmJ3Ah8aYIsaYKcaYPcaY/caYycaYUukec5UxZo4n11SgyFnP2cQYs8Bzui7BGNMs3bZoY8z/PL/3dcaYuz339wHeB5p6Rs2eAlZ4tiUZY6Zl9XsXyatUpkRyKWNMaZwCsiTd3b2APkB+YDPwCXASqIAzunMdcI/n8bcAw4H/AyKAG4F9GbzUW8Bb1toI4Gpg/HkifQkkAiWBrsCLxpjW6bbfCIwDCgLfA++mey+BQDNgegbvMxpoB6xPd/fLQCWcglUBKAU87dm/PjAWeNTzWs2ATZ7H7QY6eN7vXcAbpwpkFosGwoEyQH+cv2s/9NwuC6Tg/F5PGQcsAIoC/8Y5jnjeT2mc39cwoDAwFPjWGHOqcH0FbMT5vXcDRhhjmltrRwEDgDmeUbPngFoAntvXZcP7FsmT3Bx+FpHLM8kYcxI4CPwAvJhu25hTp3CMMVE4JaSgtfYYcMQY8wZO2foAp1SNsNYu9Dw2fVlJLwWoYIwpaq3di/OP/hk8/+BfA3Sw1h4HEowxH+GUgl88u8211k7x7P8p8GC6p2gGLLXWHj7rfVqcUjIDp0xgjDHAvUBNa+3fnvteBL7AORXYG/jYWnuqmG079YTW2h/SPf8sz+hMU+CP87z3y3USGG6tTfbcPgZMPPWzJ++PnuxX4ZTCFtbaE8BMY0z6Ebr/A7631k713P7JGLMUaGucSeX1ges8v/c/jDGjcX7vs7L4PYnIeWhkSiT3uclaW9BaW9Za299TlE7Zmu7nskAgsMNzeugATokq5tleGtiQidfrjTMKtNoYs9AY0yGDfUoCf59VhjbjjBidsjPdz0eBkHTziTI6xXeTtTY/ziTqKjijNgCRQD5gcbr39ZPn/gu+L2NMO8/psr89j2uf7nmz0q50RQpjTJgx5iPPaclDOOXw1OuWBPalm9cEzu/ulLJAj1Pv1ZO7oedxJYG91tojZz02/e9dRLKZRqZEfItN9/NW4ARQ1Fp7MoN9t+KctrvwE1q7Ducfcz+gC/B1ulNMp2wHChtj8qcrVGVINyp0Ee2Bzud5/VnGWSbgVeAmYC/OSE81z7yxs2X4vowxwcA3OCM931lrU4wxkwCTyYyXwp51ewhQHqhvrd3pmS92akRwB1DEGBOarhiXwXmP4Lyf0dbafme/iDGmPFDUGBOWrlBdyu9dRLKARqZEfJS1dgcwDXjNGBNhjPEzxlxtjGnu2eUj4BFjTF3jqGCMKXv28xhjbjfGRFpr04ADnrvPWA7BWrsVmAe8ZIwJMcbUxBnR+vxiOT2FINhau/oCu70JtDHG1Pbk+BBnvlMxz3OUMsZc79n3v8BdxpjWnvdcyhhTBQgCgoE9wEljTDucOWQ5IT/OaNx+TxF9+tQGa+0GYBkw3BgT5JlcfkO6x34KdDbGtDHG+Ht+vy2NMSWttRuBRTjz04KNMbVx5oJd9PcuIllHZUrEt/0fTolYCewHvgZKAFhrJwAv4Mw1OgxMwpngfLa2wApjTBLOpOnunvk5Z+sBlMMZpZoIDEs3b+lCbiCDT/GlZ63dgzOp/CnPXY/hzPFa4Dlt9jNQ2bPv73gml+PMK5sFlPWMmA3CmUC/H7gNZ2J3TngdKIAzwX8envlS6XQHmgB/A//CKVAAWGs34YzaPYVTBLcAD/PP39/dgIo4p1G/Bp6w1s7MpvchIhkw1p49Gi0iknM8k63fPTU5XUQkt7noyJQx5mPjXHpg+Xm2G2PM28ZZV2ZZNn3MWER8VzygkRQRybUyc5pvDM4w//m0wxlirojzkev/XHksEckrrLUjzvpEoohIrnLRMmWtnY1zHv98OgFjrWMBUNAYUyKrAoqIiIh4s6yYgF6KM9e2SURrnIiIiEgekRXrTGW0RkuGs9o914rqAxAaGlq3dOnSWfDyGUuxKWxM2kiR4CIUCTp7SRy5XGlpafj56UOgOenvv4PYuzeY0NBUoqOPYgzs3x9EUtK5//mWLn309GOOHDlzu5+fpVQp52zavn3BHD3qf8Z2f/80SpZ0PqS3Z08wx4+fuT0wMI3ixZ3tu3eHcOLEmX8OgoLSiIpytu/cGUJKypnbQ0JSiYw8AcCOHaGcPHnmXx2hoakULeps37YtlLS0M7eHhZ2kcGFnHczExFCsPXN7ePhJChVytm/dmo+zRUSkUKBACmlphm3bQs/ZXqBAChERKaSmGrZvP3d7wYLJ5M9/kpQUP3buDDlne+HCyYSFnSQ52Y9du87dXqTICfLlS+X4cX/27Ak+Z3tk5AlCQlI5dsyfvXvP3V6s2HGCg9M4ciSAv/8OOmd78eLHCQxMIykpgP37/9lurcUYQ4kSxwgIsBw6FMjBg4HnPL5UqWP4+VkOHgzk0KFzt2fvnz1LyZLOdv3Zy/yfvVPH1lv/7J2SU3/2stvatWv3WmsjM9qWFWUqEWfF4VOicT4afQ7PtaJGAcTFxdlFixZlwctnLM2mEfZiGPtO7mN2/9nERMZk22vlJfHx8bRo0cLtGHnKwIHw2Wewf3/2vo6OrW/ScfVdOrY5yxiz+XzbsqJMfQ8MMMaMAxoABz2LBbrKz/jx5c1f0vmrzjw6/VFii8ee3nbtVdfSolwL98JJntK3L/z2G+TL53yFhjpfzz4LlSvD77/DuHEQHPzPV1AQ3HEHFC0KS5dCVJTb70JERM7nomXKGPMlzrWxihpjEnEuNhoIYK19H2exvfY4C+gdxVkszyvcVOUm2lVox9T1U5m63rlGaKpNZdqGafx+7+8up5O84tVXYcAASEyEI0dg7144dgyOe5a9XLMGPvwQTpyAlJR/HteunVOmWreGJUvcyS4iIhd30TJlre1xke0WuD/LEmWxKT3PXAfwtm9uY/yK8ZR6PeM58gF+ATx+zeP0jeubE/HEh1kLaWkQHg5jxpx/v169nC9w9k9Odr7Cwpz7Bg1yRrJERMQ75bkLHQ+oP4CwwLDzbl+1dxX9fuiHtZZ+9c65rqhIpt17L/z3v/Dkk/Dcc5l7jJ8fhIQ4X6cUKpQ9+UTEN6WkpJCYmMjx4xld9UkuJiQkhOjoaAIDz50Mfz55rkw1Lt2YxqUbn3d7cmoyN427iYE/DuS2GrdRIKRADqYTX3H8uDMPqkkTqFnT7TQikpckJiaSP39+ypUrhzEZfeBezsday759+0hMTKR8+fKZflyeK1MXE+QfxN2xd/Pj+h/p8U0P8gfnz/RjQwJCeLHVi5SK0DJbeV18vDM/6oknoH17t9OISF5y/PhxFanLZIyhSJEi7Nmz55IepzKVgUbRjYgrGcfGAxsz/Zjk1GT+2v8XTcs0pUf1HoQEhODv53/xB4rPOXEChgyB/PmhZUu304hIXqQidfku53enMpWBUhGlWHjvwkt6zI7DOyj5eknunXwv906+l5jIGFb0X5FNCcWbBQc786ViYjRxXETyJn9/f2rUqMHJkyepWrUqn3zyCfnynbug6aVYtGgRY8eO5e23385w+/bt2xk0aBBff/31Fb3O5dBS1lmkRP4SjL1pLCOuHUGnyp1YuWclB44fcDuW5KD582HmTOfngQOdJQ1ERPKi0NBQEhISWL58OUFBQbz//vtnbLfWkpaWdknPGRcXd94iBVCyZElXihSoTGWpXrV68WiTR/m/Wv8HwIa/N7icSHLKsWPQsyc8/LCzJIKIiDiaNm3K+vXr2bRpE1WrVqV///7UqVOHrVu3Mm3aNBo1akSdOnW45ZZbSEpKAmDhwoU0btyYWrVqUb9+fQ4fPkx8fDwdOnQAYNasWdSuXZvatWsTGxvL4cOH2bRpE9WrVweceWN33XUXNWrUIDY2lpme/9MdM2YMXbp0oW3btlSsWJEhQ4ZkyXvUab5scHWhqwG49etbiQiOuOC+wf7BjO40mqqRVXMimmSxI0ecT+69+y5s3AgzZoCmKoiIOE6ePMmPP/5I27ZtAVizZg2jR49m5MiR7N27l+eff56ff/6ZsLAwXn75ZV5//XWGDh1Kt27d+Oqrr6hXrx6HDh0i9Kw5E6+++irvvfceTZo0ISkpiZCQM69J+N577wHw559/snr1aq677jrWrl0LQEJCAkuWLCE4OJjKlSszcOBArvRawSpT2aBasWrcW+dedh3ZdcH9Tpw8wdQNU5mwcgJPN386h9LJ+cycCQkJ0KkTlCx55lpP5zNlCtx6q/Nz9+6acC4i3iejy/fdeiv07w9Hj2b8ieM773S+9u6Frl3P3BYff/HXPHbsGLVr1wackanevXuzfft2ypYtS8OGDQFYsGABK1eupEmTJgAkJyfTqFEj1qxZQ4kSJahXrx4AERHnDko0adKEwYMH07NnT7p06UJ0dPQZ2+fOncvAgQMBqFKlCmXLlj1dplq3bk2BAs6yRzExMWzevFllyhsF+AUwquOoi+534uQJQl4IYVj8MIZeM5Qg/3Ovti1XbvRoaNsWSpRwRpGCgzMePZo3z1lgc/Bg53bBglC1qnM/OBcbPnzYmVieluYUqU6d4LXXYMsWePzxnHtPIiLe7NScqbOFhf2zaLa1ljZt2vDll1+esc+yZcsu+om6oUOHcsMNNzBlyhQaNmzIzz//fMbolL3AfIvg4ODTP/v7+3Py5MmLvp+LUZlyUXBAMMOaD+OZWc9w93d3Uzi08Dn7hASE8Pg1j1MoVMtgX45t2+Duu2HWLKdMDRwIv/7q/J9aeDgsWwZ33QXdujnLGTRrBuvWwY4dsHOnsyL5KWPGwC+/nPn899zzT/kSEfFGFxpJypfvwtuLFs3cSNTlaNiwIffffz/r16+nQoUKHD16lMTERKpUqcL27dtZuHAh9erV4/Dhw+ec5tuwYQM1atSgRo0azJ8/n9WrV58eCQNo1qwZn3/+Oa1atWLt2rVs2bKFypUr88cff2TLe1GZctmDDR/ky+VfMmXdlAy37z++n8h8kTza5NEcTuYbZsxwvp/6RG7t2rBgAXz9NRw44FxYuGRJp0wFBkLTps5XRqZPdy5WvHKl83XkCFSsmDPvQ0TE10RGRjJmzBh69OjBiRMnAHj++eepVKkSX331FQMHDuTYsWOEhoby888/n/HYN998k5kzZ+Lv709MTAzt2rVjx44dp7f379+fvn37UqNGDQICAhgzZswZI1JZzVxoKCw7xcXF2UWLFrny2rlJ/Q/r42f8WHDPArejABAfH0+LjE7Aexlr4b334NFHISoK1q+HgAz+12H/fud0niaN555jK5dGx9V3ne/Yrlq1iqpV9aGmK5HR79AYs9haG5fR/hqZ8nKVi1Zm2oZpjF8x/pxtdUrUoULhCi6k8n5vvOEsU9CunTNnKqMiBbqIsIiIXDmVKS9XLF8xdh/ZTbevu52zrUnpJsy9e64Lqbxf794QEeF816iTiIhkJ5UpL/dym5fpXaf3Ofc/+NODzE+cz60TbuXDjh9SIKSAC+m8i7Xw1FNQvrwz6fyee9xOJCIieYHKlJcL8AsgJjLmnPvvrH0niYcSmbByAv3i+tGyfN5Y4GjFCmfieKFCEBnpfJ1y8CC88AKUKwelSjnLIYiIiGQ3XU4ml7qtxm182vlTAF6a+5LLabLft99CkyZQvTpccw1Uqwa33PLP9hYtoEED5+eXXlKREhGRnKORqVwstkQsAJsPbubDxR9yV+xdBPj5ziG19p/5Tj/+CLt2ORPLY2KcT+EVSHdms04d2LQJKld21ooSERHJKb7zL28e5Gf8uL3m7Xy27DP6/K8Pe4/u5fGmvrEM9549zuUOhg+H5s2dVcbDw89cRDO911/P0XgiInIB/v7+1KhRg5MnT1K+fHk+/fRTChYsmGXPP2bMGBYtWsS7777L8OHDCQ8P55FHHsmy579UOs2Xy429aSzJTybTNaYrT8c/TZV3qzB3S+7+hN/EiVCsGMyd64xAgfPJvPMVKRER8S6nLiezfPlyChcufPrCw75K/zzlcsYYAv0D+c8N/6Fv3b5sOrCJb1Z+43asy2KtMwLVpYtze/ZsuOkmdzOJiMiVadSoEdu2bTt9+5VXXqFevXrUrFmTYcOGnb5/7Nix1KxZk1q1atGrVy8AJk+eTIMGDYiNjeXaa69l165dOZ4/M1SmfETRfEV5p/07NIxuyISVE9h9ZLfbkS7ZlCkwYoRzhfKjR6FRI7cTiYjIlUhNTeWXX37hxhtvBGDatGmsW7eO33//nYSEBBYvXszs2bNZsWIFL7zwAjNmzGDp0qW89dZbAFxzzTUsWLCAJUuW0L17d0aMGOHm2zkvzZnyMa9f/zrXfHwNrce2pk6JOgCEBoTyfKvnKZqvqMvpLuyGG5xJ5MHBOqUnIpIVHvzpQRJ2JmTpc9YuXps32755wX2OHTtG7dq12bRpE3Xr1qVNmzaAU6amTZtGbKzzAaqkpCTWrVvH0qVL6dq1K0WLOv9OFS5cGIDExES6devGjh07SE5Opnz58ln6XrKK/snyMXVK1OHzLp+TnJrM7M2zmbFxBh8s/oCp66e6He2Cdu2C556DLVtUpEREcrtTc6Y2b95McnLy6TlT1loef/xxEhISSEhIYP369fTu3RtrLSaDy1UMHDiQAQMG8Oeff/LBBx9w/PjxnH4rmaKRKR/UuWpnOlftDMCR5COEvxTOxgMbXU51fkePOgttHj8OZcs6yxuIiMiVu9gIUnYrUKAAb7/9Np06daJfv35cf/31PPXUU/Ts2ZPw8HC2bdtGYGAgrVu3pnPnzjz00EMUKVKEv//+m8KFC3Pw4EFKlSoFwCeffOLqe7kQlSkfFxYURo1iNfh65ddEhUWdd7+aUTVpEN0g2/Ps2gUPPQS33w4tW0JIiPPz8ePOquWtWmV7BBERyUGxsbHUqlWLcePG0atXL1atWkUjz6TY8PBwPvvsM6pVq8a//vUvmjdvjr+/P7GxsYwZM4bhw4dzyy23UKpUKRo2bMjGjd45MGCsta68cFxcnF20aJErr53XPDvrWYbFD7vgPv7Gn8k9JtOuYrsL7hcfH0+LFi0uK0diIjz8MIwf79z284Np0+Daa51Vy4cOvaynlSxyJcdWvJeOq+8637FdtWoVVatWzflAPiSj36ExZrG1Ni6j/TUylQc81ewp7qlzD+crzsmpyXSd0JWbx99M7eK1AWhfsT1PNnsyS3P88INTpHr1cpY/WL8eqlSBV16BO+/M0pcSERHJMSpTeYAxhpL5S15wnym3TeGhqQ+x9+heVuxZwcdLPs6SMpWUBB99BP36OdfSq1YN6taF0NB/9nFx0VoREZErpjIlAESFR/HFzV8AMHjqYN5f9D4pqSkE+gde8nPNmAFDhkBQEKxdC/v2Qfny0KmTc5FiERERX6IPocs5mpVtxrGTx5ixcUam9j9xAp59Fj7+2FnFfPhwZ37Ub785o1Dz5ztFSkREcoZb86F9weX87lSm5BzXXX0dxcOLc8ekO1i5Z+UF9126FOrUgWHDoEgRMAb+9z9YtAgOH4apU6FhwxwKLiIihISEsG/fPhWqy2CtZd++fYSEhFzS43SaT86RLzAfM/5vBq3GtqLVJ63YMGgDYUFhGe47dCjs3OlMLm/f3rkvIsL5EhGRnBcdHU1iYiJ79uxxO0quFBISQnR09CU9RmVKMlQ1sioj24+ky/gu/LT+J26OufmcfZYtg4QEiI39p0iJiIi7AgMDvfayK75Kp/nkvOqVqgfA7RNvZ9/Rfedsj4iA/PnhyaxdQUFERCRXUZmS84qOiObHnj9yMu0kg34adPr8e3w8rF7tXAJm9WrQeoAiIpKX6TSfXFDbCm15qtlTDIsfRuPoxkQdKcctXffD8YJYa3RRYhERyfNUpuSi/tX0X8zbOo8BPw5w7hgKzfL1AT5wNZeIiIg3UJmSi/L382fCLRN4b/aXDHsuicCG/2VLwWm8Nu+1M/YrV7BchhPVRUREfJnKlFzQv/8NtWvDddfl5+vH+xC4+iS9Bu3l/RUv8cj0c68D80efP4gtEetCUhEREXeoTMkFzZ0L06dD27bw2muwaNEKBt/8AiM6Pn7GfoeTDxPzXgxPzXyKyT0mY4xxKbGIiEjO0vRhuaC//4bUVOfn5s2hbt39GGPIH5z/jK+S+Uvyr6b/4od1P/DWb2+xcf9Gd4OLiIjkEJUpOa9jx5zr6sXEZG7/wY0G07h0Yx6a+hAV3qmQ6Wv7iYiI5GY6zSfneO45OHgQTq2mX6ZM5h7n7+fPtNunMWfLHPr/0J9BPw5iad+l+Pv5Z19YERERl2lkSs6wYgU8/TQcP+5coLhSJbj33sw/PiwojLYV2vL4NY+zYs8K1u5bm31hRUREvECmypQxpq0xZo0xZr0xZmgG2wsYYyYbY5YaY1YYY+7K+qiSXdatg44doVcvqFULChaEu+92ytSaNVCkyKU/Z42oGgBU/091Ap8LJPC5QEq/UZo/d/2ZxelFRETcddHTfMYYf+A9oA2QCCw0xnxvrV2Zbrf7gZXW2o7GmEhgjTHmc2ttcraklixVoQL4+cE33zgl6oUXIDLyyp6zXsl6vNLmFfYf23/6vtEJo2n3eTtevvZlulXvRoCfzjKLiEjul5l/zeoD6621fwEYY8YBnYD0ZcoC+Y3zefhw4G/gZBZnlSz2+eewcqVTnr77DqyFrFrRwN/Pn0can7kOVeeqnan3YT1un3g7BUIK0KFSh6x5MRERERdlpkyVAramu50INDhrn3eB74HtQH6gm7U27ewnMsb0AfoAREVFER8ffxmRJSusXx/GkCG1KFr0BG3aLM7045KSkq7ouH1R/wtu+/02pvw2hfDt4Zf9PJL1rvTYinfScfVdOrbeIzNlKqOxCnvW7euBBKAVcDUw3Rgzx1p76IwHWTsKGAUQFxdnW7RoccmB5fKlpsKnn8LYsTBzJpQsCZMnB1G5cotMP0d8fDxXetz6L+uPXxG/K34eyVpZcWzF++i4+i4dW++RmQnoiUDpdLejcUag0rsL+NY61gMbgSpZE1Gyip8fvPIKbN0KzzwDixZB5co5n6NC4Qos3pH50TARERFvlpkytRCoaIwpb4wJArrjnNJLbwvQGsAYEwVUBv7KyqByZdLSnPlQv/wCa9c6yx+UKOFOlp41erIgcQGzN892J4CIiEgWumiZstaeBAYAU4FVwHhr7QpjTF9jTF/Pbs8BjY0xfwK/AI9Za/dmV2i5NGPHQv36sGULFC+edZPML1efun2Ijojmjkl3sPvIbnfDiIiIXKFMfTbdWjsFmHLWfe+n+3k7cF3WRpOssHw53HGH83NQkLtZTskXmI+J3SbSdHRTek3sxdTbp7odSURE5LJpBXQft3mz871GDWdUylvElYzj8WseZ9qGaWw/fPYUPBERkdxDZcrHLV0K/v4wd67bSc51ap2pOZvnuJxERETk8mkJah/3yCNQrRpERLid5FxFQp3r1BxJOeJyEhERkcunkSkfduSIM0+qUye3k2QsMiwSf+PPhr83uB1FRETksqlM+agdO6BQIRgzxu0k55cvMB+xJWKZnzjf7SgiIiKXTWXKx3z6qTPp/LXXICUFGjd2O9GFlY4ozd6jWkVDRERyL82Z8iGHDsF998GxY87te+6BSpXczXQx4UHhJCUnuR1DRETksqlM+ZCICEhIgK++gj17YMQItxNdXHhQOFsPbaXuqLp0rNSR4S2Gux1JRETkkug0nw9ZuBAqVoSnnoK334aQELcTXVz36t1pV6Edh04c4o0Fb5Calup2JBERkUuiMuVD2rSB4cPdTnFpmpVtxvc9vufpZk9z6MQhVu9d7XYkERGRS6Iy5SN27YKDByE83O0kl6dy0coAbNivZRJERCR3UZnyAXPm/HOpGG//9N75VC5SGT/jx8JtC92OIiIicklUpnzAxInO5PNKlZzVznOjAiEFaBTdiJfmvkTESxFUG1mN5NRkt2OJiIhclD7N5wNeew2efBIKF3Y7yZV57brX+GrFVyzbtYxfNv7Cp0s/pUBIgXP2a1y6MSXzl3QhoYiIyLlUpnyAMbm/SAE0iG5Ag+gGTFw1kV82/sI9k+/JcL+QgBAebPAgj13zGAVDCuZwShERkTOpTOVyAwfC6tUwfbrbSbLOTVVuYu2AtZxIPXHOtqMpR3n7t7f596//ZtQfo+hbty89a/YkJjLGhaQiIiIqU7ne3r2waZPbKbKWMYaKRSqed/tnXct0ORkAACAASURBVD7j4UYP83T80/z713/z4twXqRVVi9tq3EaP6j0oXaB0DqYVEZG8ThPQc7lx42D9erdT5LzYErFM7jGZ7YO383bbtwkNDOWxnx/j6revZtuhbW7HExGRPERlKhf7+mu3E7gvKjyKgQ0GMr/3fCbcMoGUtBTmbZ3ndiwREclDVKZyseuvh3btYIPWuQSgfqn6ADw7+1ktqyAiIjlGc6ZyqcOHIX9+mDLF7STeo0yBMnx767d0Gd+Fq966itDAUAD8jB8jrh1BpyqdXE4oIiK+SCNTudC0ac4inca4ncT7dK7amf/c8B+al2tO/VL1qV+qPsdSjvHEjCdIs2luxxMRER+kkalcZuxYuOMO5+eHH3Y3i7fqG9eXvnF9T98evWQ0d39/N0t2LKFuybouJhMREV+kMpWLHDvmrCvVrBlMnQohIW4nyh1OLZWw9+hel5OIiIgv0mm+XOTAAef6e88+qyJ1KWIiY4gIjqDntz2ZvsGHVjcVERGvoDKVi5QoAQsWQPPmbifJXUrmL8miexdRIn8Jrv/seuZvne92JBER8SEqU7lASgo8+ihs3Qr+/m6nyZ0qFqnIr3f/SoBfABNXT3Q7joiI+BCVqVxg4kR49VX47ju3k+RuEcER1C5emz92/OF2FBER8SEqU7nAqlXO99693c3hC/IH5+fYyWNuxxARER+iMuXl5s+H4cOhfn0IDXU7Te5XIrwEWw5ucTuGiIj4EJUpL/evfznf33vP3Ry+olZULRIPJbLv6D63o4iIiI9QmfJSH38Mq1fDl1/CsmUQF+d2It9Qu3htAOZsmeNyEhER8RUqU17IWnj+eRgyBKKioEYNtxP5jnql6hEVFkXX8V3p/0N/9hzZ43YkERHJ5VSmvNCSJbBxI7Rv73YS31MwpCDL+y+nX1w/Ri0eReV3K7MzaafbsUREJBfT5WS80BNPQL58cOutbifxTUXzFeWd9u/Q5uo2dBrXiY/++Oj06b9yBctRvVh1lxOKiEhuojLlZf76y7nu3tNPQ+HCbqfxbadK01Mznzp9X1hgGPsf20+gf6BbsUREJJdRmfIyM2dC06Zw991uJ/F9VxW6inUD13Hg+AEApq6fypMzn2Td3+uIiYxxOZ2IiOQWKlNepndvLc6ZkyoUrnD656MpRwHYfni7ypSIiGSaJqB7kYQEWLzY7RR5V77AfAAcST7ichIREclNNDLlJRYuhFatoEoV+P13MMbtRHlPmQJlANh4YKPLSUREJDfRyJQXSEmB/v0hf36YNElFyi2R+SIpX7A8I34dwdp9a92OIyIiuYTKlBfo2xcWLYK33oJSpdxOk3cZY5jcYzIn007S6pNWWn9KREQyRWXKZdY6o1G33gq33OJ2GqlWrBrTe01n79G99PuhH9ZatyOJiIiXU5ly2ZYtcM890K2b20nklFrFa/Fcy+eYtHoSdUfVZcKKCaSmpbodS0REvJTKlEsOHIAOHaBcOWddqS5d3E4k6T3c+GH+e+N/SUpO4tavbyVmZAyzN892O5aIiHihTJUpY0xbY8waY8x6Y8zQ8+zTwhiTYIxZYYyZlbUxfc/998NPP0GBApon5Y38jB93x97NqvtXMb7reI6lHOOBnx5wO5aIiHihi5YpY4w/8B7QDogBehhjYs7apyAwErjRWlsN0OyfC9i0Cb74Ah55xBmhio11O5Gcj7+fP7dUu4XBjQaTsDOBdfvWuR1JRES8TGZGpuoD6621f1lrk4FxQKez9rkN+NZauwXAWrs7a2P6lv/9z/mueVK5R9sKbQGYs2WOy0lERMTbZKZMlQK2prud6LkvvUpAIWNMvDFmsTHm/7IqoC9q2xbuuksjUrlJpSKViMwXyfgV492OIiIiXiYzK6BntITk2Z8XDwDqAq2BUGC+MWaBtfaMlQ+NMX2APgBRUVHEx8dfcuDc6uRJw4QJ0dx6ayL+/pZevSA3vv2kpKQ8ddzS6xzVmVEbRvHOd+9Qo0ANt+Nkubx8bH2Zjqvv0rH1HpkpU4lA6XS3o4HtGeyz11p7BDhijJkN1ALOKFPW2lHAKIC4uDjbokWLy4yd+0yZAqNGwY03Xs0NN7id5vLFx8eTl45bevVT6vPVa1/xe9rvDGwx0O04WS4vH1tfpuPqu3RsvUdmTvMtBCoaY8obY4KA7sD3Z+3zHdDUGBNgjMkHNABWZW3U3Mta+O9/ncvFXHut22nkcuULzEe3at2YsGICMzbOcDuOiIh4iYuWKWvtSWAAMBWnII231q4wxvQ1xvT17LMK+AlYBvwOfGStXZ59sXOPTZvgvvvg229h8GAIDnY7kVyJZ1o+Q4XCFWj/eXt+Wv+T23FERMQLZOY0H9baKcCUs+57/6zbrwCvZF203C852SlSP/8MDzwAw4a5nUiuVPHw4sy6cxZNRzdlyPQhpz/lJyIieZdWQM8mqakQFOQszLltG7z5JpiMpvJLrlMkXxHuqHUHf+7+k8RDiW7HERERl6lMZbHDh6FKFQgIgG++cQpU8eJup5KsdnPMzQCMXTrW5SQiIuI2laks9uijsGaN83PJku5mkexToXAFWpZryX8W/YdjKcfcjiMiIi5SmcpCv/4KH3wAd9wBf/0FjRq5nUiy01PNniLxUCLv/P6O21FERMRFKlNZ5PBhaNwYJk+GDz+E8uXdTiTZrWX5lnSs1JHh8cNZu2/txR8gIiI+SWXqClkLn37qnNKbNw86dIDAQLdTSU55v8P7hASEMGDKALejiIiISzK1NIJk7OhReOQR+M9/4KqroGJFtxNJTiuZvyQD6w/kudnP8fZvbxPod2aTblq2KdWLVXcpnYiI5ASVqct04gTcdBNMnw4PPQT//rezFILkPd2rd+fFuS/ywE8PnLMtIjiCZX2XUbZgWReSiYhITlCZukyzZsGcOTB0KLz0kttpxE1VI6uyb8i+cz7VtzNpJ9eMvoZqI6sxutNobql2i0sJRUQkO2nO1CXYtg3eftuZJ3XddbB1q4qUOCKCI4gKjzrjq1bxWozvOp4jKUd4ZtYzWGvdjikiItlAZSqTvvsOKld2Tumt9Xxwq2hRdzOJ92tXsR0j249kxZ4VLN211O04IiKSDVSmLsBaZxXz5s2d+VExMbBunVOqRDLr1mq3EugXyKdLP3U7ioiIZAPNmbqAv/6CIUMgLQ1eeQXuvx9CQ91OJblNkXxFaFq2KXO2zHE7ioiIZAOVqbNs3gwREVCoEFx9NSxbBiEh4O/vdjLJzcoVKMfktZNJs2n4GQ0Ii4j4Ev2t7nHiBDzwAJQrB4ULOxcoXrQIwsJUpOTKXXf1dew5uocf1v7gdhQREcliKlMeH37ofFKvXz8YNAj8/ODQIbdTia/oUrULZQuUpee3PRm1eJQ+2Sci4kPyfJlKS3O+d+kCzzwDI0fCW285q5u3auVuNvEdgf6BxN8ZT71S9bjvf/fR9vO2HE056nYsERHJAnm2TP34IzRrBrGxkJjoXFvv6af/2R4c7F428U3lCpbj514/8267d5m+YTq9v++tESoRER+Q58rUr79CgwbQvj1s3w633goFC7qdSvIKYwz317+f51s9z7jl42g9tjW/bvnV7VgiInIF8tyn+a65xvn+xBMwbJiupyfuePyaxwkPCueFOS9wzehruP7q6/mw44eULlDa7WgiInKJ8sTI1P798Msvzs+//QZffAEvvKAiJe4xxjCowSD+GvQXr7R5hV82/sKbC950O5aIiFwGny9TJ09Chw7OCuaHDkH9+tCjh9upRBxhQWE80vgRioQWISk5ye04IiJyGXy6TA0bBoGBMG8e/OtfzmKcIt4oJCCEE6kn3I4hIiKXwWfLVGoqPPus8/MLL8Bjj7mbR+RCIsMi+WHdD0xaPcntKCIicol8tkz5+cFLL8GSJc5kc2PcTiRyfmM6jSE6IprOX3Xm7u/uJjUt1e1IIiKSST5bpoyBoUOhdm23k4hcXLVi1fjtnt94rMljjE4YzciFI92OJCIimZTnlkYQ8VZB/kG81Pollu5aypCfh7AzaScPNnyQyLBIt6OJiMgF+OzIlEhuZIxhdKfRdKjUgZfmvkS5t8rx8NSHdekZEREvpjIl4mWKhxdnwi0TWNF/BTdXvZnXF7yu034iIl5Mp/lEvFTVyKqM7TyWX7f+yqPTH2X/sf00Lt2YhtENKZKviNvxRETEQ2VKxMs92vhRRi0excu/vkyqdT7lV7lIZRqXbkyj6EY0LduUKkWruJxSRCTvUpkS8XJ94/rSN64vR5KPsGj7IuYnzmfe1nl8v+Z7RieMBuCBBg/w6nWvEuCn/6RFRHKa/uYVySXCgsJoXq45zcs1B8Bay7q/1/HOb+/w1m9vsXLPSr7t9i3hQeEuJxURyVs0AV0klzLGUKlIJd5p/w4fdfyIGRtn0P3r7pxMO+l2NBGRPEUjUyI+oHed3qSkpdDvh37UHVWXluVa0rh0YxqXbkx0RLTb8UREfJrKlIiP6BvXF4Phy+VfMmrxKN767S0ASkeUPl2sbq56M6UiSrmcVETEt6hMifiQ++Lu4764+0hJTWHprqXM2zrv9NdXK77izQVvsur+VQQHBLsdVUTEZ2jOlIgPCvQPJK5kHIMaDGJc13FseWgL33X/jo0HNvLMrGd0IWURkSykMiWSR3Ss1JHu1bvz0tyXaPBRAxZtX+R2JBERn6AyJZJHGGP4ossXjLt5HNsOb6PBRw1Yvnu527FERHI9lSmRPMQYQ7fq3Zhz1xzSbBrzts5zO5KISK6nMiWSB11V6CpK5i/Js7OeZe2+tW7HERHJ1VSmRPIgP+PHTz1/Ijk1mWajm7Fw20K3I4mI5FoqUyJ5VI2oGsy+azahgaE0Hd2U6bumux1JRCRXUpkSycOqFK3CwnsX0jC6IS+ufpH5W+e7HUlEJNfJVJkyxrQ1xqwxxqw3xgy9wH71jDGpxpiuWRdRRLJT0XxF+eG2H4gIiGDoL0NZtWeV25FERHKVi5YpY4w/8B7QDogBehhjYs6z38vA1KwOKSLZKywojDvL3cncLXOJGRlDnQ/q8Nq819h2aJvb0UREvF5mRqbqA+uttX9Za5OBcUCnDPYbCHwD7M7CfCKSQzqX6kziQ4m8cf0bBPgF8Mj0Ryj9Rmne/f1dt6OJiHi1zJSpUsDWdLcTPfedZowpBXQG3s+6aCKS00rkL8GDDR/k93t/Z82ANbSt0JbBUwezePtit6OJiHitzFzo2GRwnz3r9pvAY9baVGMy2t3zRMb0AfoAREVFER8fn8mY4i2SkpJ03HxURse2b7G+LN66mHZj2/Fu7XcpFlLMnXBy2fTfrO/SsfUextqze9FZOxjTCBhurb3ec/txAGvtS+n22cg/pasocBToY62ddL7njYuLs4sW6dpguU18fDwtWrRwO4Zkg/Md22W7ltF0dFPKFCjD4j6LCfIPyvlwctn036zv0rHNWcaYxdbauIy2ZeY030KgojGmvDEmCOgOfJ9+B2tteWttOWttOeBroP+FipSI5B41o2oysv1Ilu9ezm+Jv7kdR0TE61y0TFlrTwIDcD6ltwoYb61dYYzpa4zpm90BRcR97Sq2w2CYtXmW21FERLxOZuZMYa2dAkw5674MJ5tba++88lgi4k0KhxZ2VkzfPNvtKCIiXkcroItIpjQr04x5W+eRkpridhQREa+iMiUimdK8XHOOpBxhyc4lbkcREfEqKlMikinlC5YHYFfSLpeTiIh4F5UpEcmU0MBQAI6kHHE5iYiId1GZEpFMKR1RGoC1+9a6nERExLuoTIlIpuQPzk/j0o15ae5LzNw40+04IiJeQ2VKRDJtUrdJXFXoKm4cdyOHThxyO46IiFdQmRKRTIsMi+S5ls+RlJzEhr83uB1HRMQrqEyJyCWpULgCgJZIEBHxUJkSkUtSo1gNrip0FV+t+MrtKCIiXkFlSkQuiTGGpmWasmrPKrejiIh4BZUpEblkxcKKsevILvYe3et2FBER16lMicglu7P2naSmpfLsrGfdjiIi4jqVKRG5ZDGRMdxW4zbGLh1Lalqq23FERFylMiUil6XNVW04eOIgd353JzM3ziTNprkdSUTEFQFuBxCR3KlrTFfmbJnDl8u/5LNln1E6ojQ9a/SkY+WO1IqqRVhQmNsRRURyhMqUiFyW0MBQRnUcxZtt3+S71d/x6bJPeWXeK/z7139jMFQuWpnY4rHOVwnne5F8RdyOLSKS5VSmROSK5AvMR48aPehRowe7j+xmQeICluxYwpKdS/h16698ufzL0/uWjihN66taM7L9SEIDQ11MLSKSdVSmRCTLFAsrxo2Vb+TGyjeevm/f0X0k7Exgyc4lLEhcwJiEMbQo24I7at/hYlIRkayjMiUi2apIviK0vqo1ra9qjbWWqu9VZdBPg5i7ZS49a/akWdlm+Bl9FkZEci/9DSYiOcYYwze3fkOnyp0Yt2IcLT9pSZk3yvDY9Mc4fvK42/FERC6LRqZEJEdVK1aNsZ3HcjTlKJPXTObzPz9nxLwRlC5QmgH1B7gdT0TkkmlkSkRckS8wH92qd+P7Ht9TpkAZJq2epLWqRCRXUpkSEdcNqj+IXzb+wpDpQ9yOIiJyyVSmRMR1gxsN5v569/Pa/NdYkLjA7TgiIpdEZUpEXGeM4ZkWz2AwTN8w3e04IiKXRGVKRLxCkXxFqFK0Cot2LHI7iojIJVGZEhGvUaVoFdbsXeN2DBGRS6IyJSJeo3bx2qzdt5aDxw+6HUVEJNNUpkTEa7Qu3xqLpeuEruw/tt/tOCIimaIyJSJeo0mZJnx848fM2jSLhv9tyJHkI25HEhG5KJUpEfEqd8XexYcdP2TtvrUs373c7TgiIhely8mIiNepXqw6AB2+7ECLci1oWa4lLcq1oGrRqhhjXE4nInImlSkR8TqxJWL5vMvnTN0wlZkbZ/L1yq8BiAqLokW5FjzQ4AEalW7kckoREYfKlIh4HT/jx201buO2GrdhreWv/X8RvymemZtm8t2a7zhw/AA/3f6T2zFFRADNmRIRL2eM4erCV9O7Tm8+6/IZ1119HYt3LGbGxhluRxMRAVSmRCSXebjRw4QGhNJ6bGvaf96eZbuWuR1JRPI4lSkRyVUal27M2oFreaXNK8xPnE/t92tz56Q72XJwi9vRRCSPUpkSkVwnJCCERxo/wl+D/uKRxo8wbvk4Kr1TiYd+eoitB7e6HU9E8hiVKRHJtQqFFmJEmxGsHbiW7tW7887v73DV21fRa2IvEnYmuB1PRPIIlSkRyfXKFCjDmJvGsGHQBgbUG8DEVROJ/SCWL/78wu1oIpIHqEyJiM8oW7Asb7R9g60PbaV+qfo8PO1hDp847HYsEfFxKlMi4nMKhRbixVYvsjNpJz//9bPbcUTEx6lMiYhPiomMAeDzPz/nr/1/uZxGRHyZypSI+KTi4cX5v1r/x6TVk6jwdgXaf96e/639H9Zat6OJiI9RmRIRn2SM4ZObPmHzg5sZ1nwYCTsT6PhlR+787k5SUlPcjiciPiRTZcoY09YYs8YYs94YMzSD7T2NMcs8X/OMMbWyPqqIyKUrFVGKYS2GsfnBzQxvPpyxS8dy01c3cfD4QbejiYiPuGiZMsb4A+8B7YAYoIcxJuas3TYCza21NYHngFFZHVRE5EoE+gcyrMUwPujwAVPXTyX2g1h+3/a727FExAdkZmSqPrDeWvuXtTYZGAd0Sr+DtXaetXa/5+YCIDprY4qIZI0+dfsw5645pNk0mnzchI5fduTjJR+z58get6OJSC5lLjYZ0xjTFWhrrb3Hc7sX0MBaO+A8+z8CVDm1/1nb+gB9AKKiouqOGzfuCuNLTktKSiI8PNztGJIN8tqxPZxymM+2fMasPbPYdWIXfvhRvUB1mhZtSocSHQjxD3E7YpbIa8c1L9GxzVktW7ZcbK2Ny2hbQCYebzK4L8MGZoxpCfQGrslou7V2FJ5TgHFxcbZFixaZeHnxJvHx8ei4+aa8eGw70hFrLQk7E5i4eiITV0/kvQ3vsTt4N191/QpjMvrrL3fJi8c1r9Cx9R6ZOc2XCJROdzsa2H72TsaYmsBHQCdr7b6siScikr2MMcSWiOXZls/yZ78/ebHVi0xYOYGHpj7E+r/Xux1PRHKBzJSphUBFY0x5Y0wQ0B34Pv0OxpgywLdAL2vt2qyPKSKSM4Y0GcLtNW/nnd/foeI7FWkxpgVjl47laMpRt6OJiJe6aJmy1p4EBgBTgVXAeGvtCmNMX2NMX89uTwNFgJHGmARjzKJsSywiko38/fz5tPOnbHlwCy+2epFth7dxx6Q7aPVJK9JsmtvxRMQLZWqdKWvtFGttJWvt1dbaFzz3vW+tfd/z8z3W2kLW2tqerwwnaImI5BalIkrxeNPHWTtgLW+3fZvftv3G6CWjtYK6iJxDK6CLiFyAMYZ+9foRWzyWeybfQ5OPmzBl3RSVKhE5TWVKROQiAvwCmNd7HiPbj2Tb4W3c8MUNdBnfxe1YIuIlVKZERDIhJCCEfvX6sW7gOvrU6cOk1ZN0SRoRAVSmREQuSZB/EF1jugLQ+/veKlQiojIlInKprr3qWl6+9mUmrZ5E7AexLNy20O1IIuIilSkRkUtkjGFIkyHMvms2J9NO0uTjJry14C1NShfJo1SmREQuU+PSjUnom0C7iu14cOqDNB/TnLlb5rodS0RymMqUiMgVKBxamEndJvH+De+z7u91NB3dlHaft2P13tVuRxORHKIyJSJyhYwx3Bd3HxsGbeDla1/mt8TfuP6z6zlw/IDb0UQkB6hMiYhkkXyB+RjSZAg/9vyRbYe2MXjqYLcjiUgOUJkSEcliDaIb0DeuL1/8+QWHThxyO46IZDOVKRGRbNC9endOpJ7gh7U/uB1FRLJZgNsBRER8UePSjSkeXpx3F75L4dDCxJWMo0i+Im7HEpFsoDIlIpIN/IwfgxsOZsjPQ2j7eVsAyhcsT1zJOOqVrEe9UvWoU6IOEcERLicVkSulMiUikk0ebfIofer2YfGOxSzavoiF2xeycPtCJqycAIDBcEOlG/jkpk8oHFrY5bQicrlUpkREslGBkAK0Kt+KVuVbnb5vz5E9LN6xmDmb5/Dq/Fep92E9Puv8GQ2jG2KMcTGtiFwOTUAXEclhkWGRtK3Qlhdav8CsO2dxLOUYjT9uTJX3qvD0zKdZsXuF2xFF5BKoTImIuKhhdENW9F/B+ze8T6n8pXhhzgtU/091qo+szstzX2ZX0i63I4rIRahMiYi4rFBoIe6Lu48Zd8xg2+BtvNvuXQqGFGToL0OJfiOaruO7MnX9VFJSU9yOKiIZ0JwpEREvUjy8OPfXv5/769/Pqj2r+OiPj/hk6Sd8s+ob/Iwf0RHRlC1QlnIFy53+Xq5gOcoWLEvpiNIEBwS7/RZE8hyVKRERL1U1siqvXf8aL7Z+kf+t/R8JOxPYfHAzmw5sYtbmWSQeSiTNpp3e32Aokb/EGUWr7NGytKCFe29CJA9QmRIR8XLBAcHcHHMzN8fcfMb9KakpJB5KPF2wNh/YzKaDm9h0YBMLEhcwfsV4rg67mvu4z6XkInmDypSISC4V6B9I+ULlKV+ofIbbn/jlCUb8OoIlO5YQWyI2h9OJ5B2agC4i4qPuqHUHhQIL0fC/DXlrwVucOHnC7UgiPkllSkTER1UuWpmP4j7i+quv58GpD1L0laJ0/qozHy7+kG2HtrkdT8Rn6DSfiIgPKxBYgO+6f8eP639k8prJ/LDuByatngRArahatK3QltblW9OkTBPyBeZzOa1I7qQyJSLi44wxtK/YnvYV22OtZcWeFUxZN4Uf1v3Aa/Nf4+VfXybQL5CG0Q1pVb4VLcu1pGF0Qy2zIJJJKlMiInmIMYbqxapTvVh1hjQZQlJyEnO3zGXmxpnM2DSD52Y/xzOzniEkIIQGpRrQtExTmpZtSqPoRuQPzu92fBGvpDIlIpKHhQeF07ZCW9pWaAvAgeMHmL15NjM3zmTOljm8OPdF0uak4Wf8qF28Nk3LNKVeyXqUzF+SqPAoosKiKBxaWBdoljxNZUpERE4rGFKQGyvfyI2VbwTg8InDLEhcwJwtc5izZQ4fLP6At35764zHBPgFEBUWdbpcnfpePLz4ObcLhRbCz+izT+JbVKZEROS88gfnp83VbWhzdRsAklOTWf/3enYl7WLXkV3sStrFzqSdzs+e23/u/pNdSbtISTv3WoIBfgEUCyt2umSdLlxn3w53RrxUvCQ3UJkSEZFMC/IPIiYyhpjImAvuZ61l//H9p0vXzqSdZxawI87t5buXX7B4ReaLdArWqVGvUyNe6UbBKhWpRJB/UHa9ZZGLUpkSEZEsZ4yhcGhhCocWpmpk1Qvua63lwPED/4xwZVTAjuxixe4V7EzaeU7xKlOgDM+0eIZeNXvh7+efnW9LJEMqUyIi4ipjDIVCC1EotFCmi9ep0rXl4Bbe+f0d7vruLl6Z9wp96/alTok61IyqqU8fSo5RmRIRkVwjffGqUrQKALfXvJ1vV33LkzOfZNBPg07vW6FwBWKLx1K7eO3TXyXCS+iTh5LlVKZERCRXM8Zwc8zNdKnahW2Ht5GwM+H01x87/mDCygmn940Ki6JJmSY0K9OM5uWaU6NYDZ0alCumMiUiIj7BGEN0RDTREdF0qNTh9P0Hjx9k2a5lJOxMYNGORczZPIdvV30LOEtBXFPmGpqXbU5s8VgKhxamUGghCoYUJCI44v/bu//YuOs6juPPV3+sLQx6XVE2J2k32EoA5xQ2NAsMIjNASJgxTkgmv0xghgb5wz/2n5gQgyRoIBAWIGSY+AOVEBcExS0wEhTHDyvjV+sGhW1MJmVXHe5Kur394749rtd2u/Vuvev19Ugu31+f7/fe3Te3vPK5730+/jWhFcVhyszMalprcysXdGRHch+xa3AXW9/dytb+rTz33nM80ffEmPPqVEdrNT3nwQAAB/tJREFUUyup5lQuYLU1f7ocs6+lbdS6f2E4czhMmZnZjHNa62msXbKWtUvWArD3v3vpHeglnUmz/+B+9mf259bTQ5/ue/PDN7P7MmkODh884nu0NLSMCVhzT5zL4vbFLG5fTNcpXSxsW+jQVQMcpszMbMabd9I85p0075jOGRoeGh26MulR27n1ZLnnP3vYtmcb+z7el7tGvepZ0LaArvYuutq7ciGrq73rmOuxynGYMjMzm4Smhibmzp7L3Nlzj+m8dCZN30AfvR/20juQffUN9LHlnS1khjO5dlefczUbV290z9U04DBlZmY2hVLNKZbPX87y+ctH7T8ch9k1uIu+gT42v72ZO/9yJ+lMmpvOvYnOVCedqU5am1srVLUdicOUmZlZFahTHR2pDjpSHaw6fRUL2hbQ/WQ3T+14Ktcm1ZyiM9VJR2sHDQca6GnuyQWtzlQnqeZUBf+CmcthyszMrAqtO28da85ew86PdtKf7ufdwXfpT/fTn+5nx0c72Dmwk8f2PDbqnJObTuaMOWfQvayba754jcfQmiIOU2ZmZlVqTssc5syfw7L5y8Yce+aZZ1hy/pJcwBoJW8/vep4bNt3APdvu4faLb2dh20KaG5ppamiiqb4pt2yoa/Bo8GVSVJiSdClwN1APPBQRdxQcV3L8cuB/wHUR8UqZazUzM7OEJNpPaKf9hHbO/dy5uf0RwaOvP8r6zeu54ldXTHh+nepGhaumhqZs6MrbN14IK2wz3nnHdK2GJmbVz5rWA6QeNUxJqgfuA1YBu4EXJW2KiDfyml0GLEpe5wP3J0szMzObQpK46pyrWH3maja/vZkDnxxgaHiIoUNDDA0PkRnO5NaHDiXbI8cL2mSGMwwODY5uU3BeEGWpu7GucXLBrL6JlZ0rWXP2mrLUMRnF9EwtB3ZExNsAkn4NXAnkh6krgZ9HRAAvSEpJmhcRe8tesZmZmR1Vc0PzqGl1joeIYPjw8NhQNl5QGyfIHWu4G8wMjnvdlsaWqg9T84Fdedu7GdvrNF6b+YDDlJmZWY2SRGN9I431jcyeNbvS5VRMMWFqvKfTCvv0immDpBuBG5PNA5J6i3h/qy6nAB9Wugg7Lnxva5Pva+3yvZ1aHRMdKCZM7QZOy9v+PPD+JNoQEQ8ADxTxnlalJL0UEedVug4rP9/b2uT7Wrt8b6tHMY/OvwgskrRA0izgKmBTQZtNwDXK+gow6OelzMzMbCY4as9URAxL6gb+RHZohIcj4nVJ65LjG4AnyQ6LsIPs0AjXH7+SzczMzKpHUeNMRcSTZANT/r4NeesB3Fze0qxK+Wva2uV7W5t8X2uX722VUDYHmZmZmdlkTN/hRs3MzMyqgMOUFUXSpZJ6Je2QtL7S9Vj5SOqXtF1Sj6SXKl2PTZ6khyXtk/Ra3r45kv4s6Z/Jsq2SNdrkTHBvb5O0J/ns9ki6vJI1zmQOU3ZUeVMKXQacBVwt6azKVmVldnFELPXPrKe9jcClBfvWA1siYhGwJdm26WcjY+8twM+Sz+7S5PlmqwCHKStGbkqhiPgEGJlSyMyqSEQ8B3xUsPtK4JFk/RFg9ZQWZWUxwb21KuEwZcWYaLogqw0BPC3p5WSWAqstp46M+5csP1vheqy8uiW9mnwN6K9wK8RhyopR1HRBNm2tiIgvk/0a92ZJF1a6IDMryv3A6cBSsnPh3lXZcmYuhykrRlHTBdn0FBHvJ8t9wONkv9a12vGBpHkAyXJfheuxMomIDyLiUEQcBh7En92KcZiyYhQzpZBNQ5JOlHTSyDrwdeC1I59l08wm4Npk/Vrg9xWsxcpoJCQnvoE/uxVT1AjoNrNNNKVQhcuy8jgVeFwSZP8/+GVE/LGyJdlkSfoVcBFwiqTdwA+BO4DfSPou8B7wrcpVaJM1wb29SNJSso9d9AM3VazAGc4joJuZmZmVwF/zmZmZmZXAYcrMzMysBA5TZmZmZiVwmDIzMzMrgcOUmZmZWQkcpsxsykhqz5vh/l95M96nJb1xHN7vIklPHOM5z0oaM+GzpOsk3Vu+6sysVjhMmdmUiYiBkRnugQ0kM96TnQ7j8NHOl+Sx8cys6jhMmVm1qJf0oKTXJT0tqQVyPUU/lrQV+L6kz0h6TNKLyWtF0m5lXq/X30dGdgdmS/qdpLck/ULJCKWSvpa0255MEttUWJCk6yX1Je+9Yor+HcxsmnGYMrNqsQi4LyLOBtLAN/OOpSJiZUTcBdxNtkdrWdLmoaTND4Cbk56uC4CDyf4vAbcCZwELgRWSmoGNwLcj4gtkR3//Xn4xyVQdPyIbolYl55uZjeEwZWbV4p2I6EnWXwY68449mrd+CXCvpB6y886dnPRCPQ/8VNItZMPXcNJ+W0TsTiaD7Umu25W8X1/S5hHgwoJ6zgeejYh/R8QnBTWYmeX4+QMzqxZDeeuHgJa87Y/z1uuAr0bEQUa7Q9IfgMuBFyRdMsF1GwAVWZPn2zKzo3LPlJlNN08D3SMbyUSvSDo9IrZHxE+Al4Azj3CNt4BOSWck298Btha0+RvZiWTbJTXiCYLNbAIOU2Y23dwCnCfp1WQ4hXXJ/lslvSbpH2Sfl3pqogtERAa4HvitpO1kf0m4oaDNXuA24K/AZuCVcv8hZlYbFOFebDMzM7PJcs+UmZmZWQkcpszMzMxK4DBlZmZmVgKHKTMzM7MSOEyZmZmZlcBhyszMzKwEDlNmZmZmJXCYMjMzMyvB/wGpSMnf2M3sOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's plot and see the effect\n",
    "plot_pr_recall_curve(y_train, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/rock-music.png\" style=\"height:50px;display:inline\"> The ROC Curve\n",
    "---\n",
    "The *receiver operating characteristic* (ROC) curve is another common tool used with binary classifiers. It is very similar to the precision/recall curve, but uses the True Positive Rate (TPR, another name for Recall) vs. the False Positive Rate (FPR, the ratio of negtaive instances that are incorrectly classified as positive) we saw in the beginning of the tutorial.\n",
    "* Note: The FPR is equal to $1 - TNR$ (true negative rate, or, *specifity*)\n",
    "* The ROC curve plots *sensitivity* (recall) vs. 1 - *specifity*.\n",
    "* **TPR/FPR Tradeoff**: The higher the TPR (recall) the more *false positive* (FPR) the classifier produces.\n",
    "* To compare classifiers - measure the *area under the curve* (AUC). A perfect classifier will have a ROC AUC equal to 1, whereas a purely random one will have a 0.5 AUC.\n",
    "* **Precision/Recall curve vs ROC curve**: if the positive class is rare, or you care more about the false positive than the false negative use **Precision/Recall**. Otherwise, use the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def plot_roc_curve(labels, scores):\n",
    "    fpr, tpr, thresholds = roc_curve(labels, scores)\n",
    "    precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)\n",
    "    fig = plt.figure(figsize=(10,5))\n",
    "    ax = fig.add_subplot(1,1,1)\n",
    "    ax.plot(fpr, tpr, linewidth=2)\n",
    "    ax.plot([0,1], [0,1], 'k--') # random classifier\n",
    "    ax.set_xlabel(\"False Positive Rate (FPR)\")\n",
    "    ax.set_ylabel(\"True Positive Rate(TPR=Recall)\")\n",
    "    ax.set_title(\"ROC Curve\")\n",
    "    ax.grid()\n",
    "    ax.set_xlim([0, 1])\n",
    "    ax.set_ylim([0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC AUC Score: 0.943\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFNCAYAAABbpPhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZyO9f7H8ddnxlaWIqKIkWWyVCJajqwpqpMOJeqocDi0iGSL6pQlKaUIWSKlSEKcjk7b0GklEVnHTguRZcg6n98fM/WbM4eZG3PPdd8z7+fjMQ/3dd/X3Pebq5nzPt/vdV1fc3dEREREJLLEBB1ARERERP6XSpqIiIhIBFJJExEREYlAKmkiIiIiEUglTURERCQCqaSJiIiIRCCVNBEREZEIpJImIhHDzDaa2W9mlmRmP5nZJDMrlG6fq83sYzPbZ2Z7zGyOmVVNt08RMxtuZptT3ysxdbv4CT7XzKyrmS03s/1mttXMppvZxeH8+4qIZEQlTUQizZ/dvRBQA7gM6Pv7C2Z2FfBvYDZwPlAeWAp8ZmYXpu6TD/gIqAY0BYoAVwM7gTon+MwXgAeBrkAxoDIwC7jxZMObWZ6T/R4RkeMxrTggIpHCzDYCf3P3D1O3hwLV3P3G1O1PgWXufm+67/sXsMPd7zKzvwGDgArunhTCZ1YCVgFXufvXJ9gnAXjd3cenbt+TmrNu6rYD9wPdgDzA+0CSuz+c5j1mA/Pd/TkzOx8YAdQDkoDn3f3FEP6JRCQX0UiaiEQkMysDNAMSU7fPJGVEbPpxdn8LaJL6+FpgXigFLVVjYOuJCtpJuAW4AqgKvAHcbmYGYGZFgeuAqWYWA8whZQSwdOrndzOz60/z80Ukh1FJE5FIM8vM9gFbgO3A46nPFyPld9aPx/meH4Hfzzc75wT7nMjJ7n8iT7n7Lnf/DfgUcOCa1NduBb5w9x+A2kAJd3/S3Q+7+3pgHNA6CzKISA6ikiYikeYWdy8MNAAu4v/L169AMnDecb7nPOCX1Mc7T7DPiZzs/iey5fcHnnIeyVSgTepTdwBTUh+XA843s92/fwGPACWzIIOI5CAqaSISkdx9PjAJeDZ1ez/wBXDbcXZvRcrFAgAfAtebWcEQP+ojoIyZXZ7BPvuBM9Nslzpe5HTbbwK3mlk5UqZBZ6Q+vwXY4O5np/kq7O43hJhXRHIJlTQRiWTDgSZmViN1uw9wd+rtMgqbWVEzGwhcBTyRus9rpBShGWZ2kZnFmNk5ZvaImf1PEXL3tcAo4E0za2Bm+cysgJm1NrM+qbstAVqY2ZlmVhHokFlwd/8W2AGMB953992pL30N7DWz3mZ2hpnFmll1M6t9Kv9AIpJzqaSJSMRy9x3AZODR1O3/ANcDLUg5j2wTKbfpqJtatnD3Q6RcPLAK+ADYS0oxKg58dYKP6gqMBF4CdgPrgL+QcoI/wPPAYeBn4FX+f+oyM2+mZnkjzd/pGPBnUm4xsoGUadrxwFkhvqeI5BK6BYeIiIhIBNJImoiIiEgECltJM7NXzGy7mS0/wetmZi+mLtfynZnVDFcWERERkWgTzpG0SaQsyXIizYBKqV+dgNFhzCIiIiISVcJW0tx9AbArg12aA5M9xZfA2WaWFfcqEhEREYl6QZ6TVpo0N38EtqY+JyIiIpLr5Qnws+04zx33UlMz60TKlCgFChSoVbZs2XDmkjBKTk4mJkbXq0QjHbvotXFvctARRHINTz4G7lhsHpKPHOTozq2/uHuJU3mvIEvaVuCCNNtlgB+Ot6O7jwXGAsTHx/vq1avDn07CIiEhgQYNGgQdQ06Bjl3kazfxaz5ZveN/nv/9PJKNQ27M3kCSZfTzF/l+/PFHnn32WcaMGUOrVq2YOHEiAGa26VTfM8iS9i5wv5lNJWXJlD3unhWLHIuIRJUTlausdEmJ2LC+v0hutWXLFoYOHcq4ceM4evQod9xxB717986S9w5bSTOzN0lZILm4mW0FHgfyArj7GOA94AYgETgAtAtXFhGRSJaVBa1hfAkmtqvzP88nJCRk2WeIyP97+umnefnll7n77rvp27cvFSpUyLL3DltJc/c2mbzuwH3h+nwRkayUHaNdmo4UiXyrV69m8ODBdOzYkbp169K/f3969epFOM6X11nAIiIhCHdBaxh/SucVi0g2Wb58Oa1bt6ZKlSpMnz6dlStXAlCqVKmwFDQI9pw0EZGoo9EukdynU6dOjBs3jkKFCtGzZ08eeughSpYsGfbPVUkTkcBkxxSiiMipWLRoEZdddhmxsbFUr16dRx99lAcffJBzzjkn2zJoulNEAhNtBU1TkiI534IFC7juuuuoXbs2b7/9NgBdu3blySefzNaCBhpJE5EsdiqjY5pCFJEguTsfffQRAwYMYMGCBZx77rkMHTqUG28M9neTSpqIZKmTLWganRKRoCUnJ3Pfffexf/9+XnjhBTp27MgZZ5wRdCyVNBEJD42OiUikSk5OZtasWYwePZpZs2ZRsGBB5syZQ7ly5cifP3/Q8f6gkiYShQI74X7eP7P/M0VEssixY8eYPn06gwYNYvny5VSsWJENGzZQvXp1KleuHHS8/6ELB0SiUKSfcK8pTBGJNDt37qRq1aq0adOG5ORkpkyZwsqVK6levXrQ0U5II2kiESyzEbPsnFLUAs8iEm0OHTrEokWL+NOf/sQ555xDo0aNGDx4MH/5y1+IiYn8cSqVNJFU0XbPLo1WiYgc32+//cb48eMZOnQoO3bsYNOmTZQsWZLRo0cHHe2kqKSJpIrUgnaiBbNFROS/7d+/nzFjxvDss8/y008/UbduXSZMmMC5554bdLRTopImko6uShQRiU7btm2jV69eNGzYkKlTp1K/fv2gI50WlTSJOtE2LSkiIuGxa9cuXnzxRTZt2sTEiROpXLkyq1evpmLFikFHyxKRf9acSDrhLGg6z0tEJPJt376dvn37EhcXxxNPPMGePXs4cuQIQI4paKCRNIlQoYyWaVpSRCT3+de//kXLli05ePAgt912G/369eOSSy4JOlZYaCRNIlJmBU0jXiIiucfmzZtZsmQJALVr16Z169Z8//33TJs2LccWNNBImkSY9CNoGi0TEcm91q9fz1NPPcWrr75K7dq1+eyzzyhevDivvPJK0NGyhUqahEWG05UhLi2k0TIRkdxpzZo1DBo0iClTppAnTx46duxI7969g46V7VTSJCxO5+R+3RdMRCR3++ijj5g+fTpdu3bl4Ycf5vzzzw86UiBU0iQkp3rbi/TTlVpaSERE0lu8eDEDBw7k+uuv5+9//zvt2rWjZcuWUXsT2qyiCwckJKdS0DRdKSIiGfnyyy+58cYbqVWrFh9//PEft9EoUKBAri9ooJE0OUk6kV9ERLJC9+7dGT58OOeccw6DBg3ivvvu46yzzgo6VkRRScuFdMd+ERHJbu7Ohx9+yGWXXUbx4sVp1qwZpUuXpnPnzhQqVCjoeBFJ05250KkWNE1fiojIyXJ35s6dy1VXXcV1113Hyy+/DMB1113Hww8/rIKWAY2k5WKauhQRkXB65513GDBgAEuWLCEuLo6XX36Zu+++O+hYUUMlLYpp2lJERCKNu2NmAEyePJkDBw4wadIk7rjjDvLmzRtwuuii6c4odrr3IhMREckqR44cYdKkSVSrVo21a9cCMGHCBFasWMHdd9+tgnYKNJIWRU40cqZpSxERCcqhQ4eYNGkSQ4YMYePGjdSoUYPdu3cDcM455wScLrqppEWR4xU0jYiJiEhQjhw5QvXq1UlMTKROnTqMGDGCG2+88Y/pTjk9KmlRSCNnIiISlKSkJGbPns2dd95J3rx56dq1K/Hx8TRp0kTlLIuppImIiEim9uzZw8iRI3n++efZuXMn1atX59JLL+WBBx4IOlqOpQsHRERE5IT27dvH448/TlxcHP379+eKK67g888/59JLLw06Wo6nkbQooFttiIhIdktOTiYmJgYzY/To0TRs2JB+/fpRq1atoKPlGippESazQqYLBUREJJx+/PFHnnnmGRISEli4cCGFChVi9erVFC1aNOhouY5KWoQ5UUFrGF+Cie3qZHMaERHJLTZv3szTTz/NhAkTOHr0KHfeeSf79u3j7LPPVkELiEpahNIVnCIikl2+/vpr6tatC8A999xDnz59uPDCCwNOJSppIiIiudCqVatITEzkpptuolatWvTt25cOHTpQtmzZoKNJKpW0gOhiABERCcKyZcsYOHAg06dP54ILLqBZs2bExsbyxBNPBB1N0lFJC5PTKWG6OEBERLLaihUr6NevH7NmzaJw4cL06dOH7t27ExsbG3Q0OQGVtDAJpaDpYgAREQm3I0eOkDdvXnbu3ElCQgKPP/44Xbt2pVixYkFHk0yopIWZLgAQEZHs5u7Mnz+fAQMGUKVKFUaOHMk111zD1q1bKViwYNDxJERacUBERCSHcHfef/996tWrR8OGDVmxYgUXXXTRH6+roEUXjaRlMV0QICIiQRkwYACPP/44ZcqUYcSIEXTo0IEzzjgj6FhyilTSsljagqYLAEREJJySk5OZOXMmlSpV4pJLLqFNmzacd9553H333eTLly/oeHKaVNLCROeiiYhIuBw7doxp06YxaNAgVqxYQZcuXRg1ahSVKlWiUqVKQceTLBLWkmZmTYEXgFhgvLsPSff6WcDrQNnULM+6+8RwZspqmt4UEZHsNHXqVB577DHWrl1LtWrVeOONN2jVqlXQsSQMwlbSzCwWeAloAmwFFprZu+6+Is1u9wEr3P3PZlYCWG1mU9z9cLhyZZWMypmmOUVEJCsdPnwYd8fMWLZsGYUKFWLGjBnccsstxMToGsCcKpwjaXWARHdfD2BmU4HmQNqS5kBhMzOgELALOBrGTFkm/blnut+ZiIhktd9++41x48YxYMAAJkyYwM0338xjjz3GwIEDSfmfTsnJwlnSSgNb0mxvBa5It89I4F3gB6AwcLu7J6d/IzPrBHQCKFGiBAkJCeHIe0omNS0IHIioTJEsKSlJ/1ZRSscuuun4RZfffvuNd999l2nTpvHrr79SrVo1tmzZomOYy4SzpB2v4nu67euBJUAjoALwgZl96u57/+ub3McCYwHi4+O9QYMGWZ/2ZM37JwARkSWKJCQk6N8sSunYRTcdv+hy+eWX880333Dttdfy6KOPkpycrOOXC4VzInsrcEGa7TKkjJil1Q54x1MkAhuAixAREclFdu3axVNPPcXBgweBlPudffHFF3zwwQfUq1cv4HQSlHCOpC0EKplZeWAb0Bq4I90+m4HGwKdmVhKIB9aHMZOIiEjE2L59O8899xwvvfQSSUlJ1KhRg2bNmtGsWbOgo0kECNtImrsfBe4H3gdWAm+5+/dm1tnMOqfuNgC42syWAR8Bvd39l3BlEhERiQQHDx6ke/fuxMXFMXToUG666SaWLVumcib/JaSRtNSrL6sD5wO/Ad+7+87Mvs/d3wPeS/fcmDSPfwCuO5nAIiIi0SopKYlChQqRP39+vvrqK1q1akXfvn2Jj48POppEoAxLmpnFAb2ApqScL7YDKEDKNOZuYAzwurunvyBAREREUiUmJvLUU08xc+ZM1qxZQ/HixVmwYAF58mjhHzmxzP7rGEpKEbs//a0xzOw84E7gbmBSWNKJiIhEsZUrVzJ48GDeeOMN8ubNS8eOHfl9XEMFTTKT4X8h7n7CdSbc/Ufg2SxPJCIikgNs2bKF6tWrU6BAAbp3706PHj0477zzgo4lUSSz6c6bM3rd3d/N2jgiIiLRa9GiRfznP/+hW7duXHDBBUyaNImmTZtSooSWC5STl9lY620ZvOakrBYgIiKSq33++ecMGDCAefPmUbx4cdq3b0+RIkVo27Zt0NEkimU23an/ukRERE5g1apV3HvvvXzyyScUL16cp556invvvZciRYoEHU1ygMymO7tm9Lq7v5i1cURERCKbu7N7926KFi1KkSJF2LBhA8OGDePvf/87BQsWDDqe5CCZTXdqEj1Vu4lf88nqHUHHEBGRgLg7c+bMYeDAgZx55pkkJCRw/vnns27dOmJiwrnKouRWmU13PppdQSLd8Qpaw3h1WBGRnC45OZl33nmHgQMHsnTpUsqXL0/fvn1xd8xMBU3CJtQVB/ID9wDVSLmZLQDu3ik8sSLXxiE3Bh1BRESy0YQJE+jUqROVK1fm1VdfpU2bNuTNmzfoWJILhHonvcmkLHx+EzCIlIXSvw9XqEiiaU4RkdzlyJEjvP766xQrVozmzZvTpk0bihQpwq233kpsbGzQ8SQXCXWMtrK79wWS3H0CKctEVQ9frMiRtqBpelNEJOc6dOgQY8aMoVKlSrRv35433ngDgEKFCnH77beroEm2C3Uk7Ujqn7vNrArwM1AuPJGCdaKRM01ziojkXFOnTqVHjx788MMPXHnllYwaNYpmzZoFHUtyuVBL2gQzKwo8DrwPnJn6OEfIbEpTI2giIjlPUlISZkbBggVxdypVqsTkyZNp1KgRZhZ0PJHQSpq7v5z68BOgbPjiBCN9QWsYX4KJ7eoElEZERMJpz549jBgxgueff54ePXrwyCOP0Lp1a9q0aRN0NJH/EurVnQOAYe6+O3W7KNDN3XPMaBpoSlNEJCfbuXMnw4cPZ8SIEezZs4ebbrqJJk2aAGjkTCJSqBcO3PR7QQNw91+BP4cnkoiISNZr3749AwcOpHHjxixevJg5c+ZQu3btoGOJnFCoJS3WzPL9vmFmBYB8GewvIiISqG3bttG9e3c2b94MwKBBg1i+fDkzZszgsssuCzidSOZCvXBgKvCBmb0CONABmBK2VNlE90ATEcl5Nm3axJAhQ3jllVc4duwYNWvWpG3btlSvnivuHCU5SKgXDgw2s++AawEDhrr7P8OaLBvoHmgiIjmHu9OlSxcmTJiAmdGuXTv69OlD+fLlg44mckpCHUkDWALsd/dPzKyAmRV09/3hCpaVMhsx0wUDIiLRa/PmzZQtW/aPdTQ7d+5Mr169uOCCC4KOJnJaQjonzczaA+8C41OfKgvMDleorKZ7oImI5DxLly7ltttuIy4ujkWLFgEwatQoRowYoYImOUKoI2ldgTrAVwDuvsbMzg1bqjDRiJmISPRbtGgRAwYM4N1336Vw4cL07duXuLi4oGOJZLlQS9pBdz/8+31kzCyWlHPTREREss3+/ftp3LgxsbGxPPHEEzzwwAMULVo06FgiYRFqSfvMzHoBBcysIXAfMDd8sURERFIuBkhISGDq1KmMGTOGggULMmfOHGrUqEGRIkWCjicSVqHeJ60XsA9YBTwIfAT0C1coERHJ3dydefPmcc0119CoUSPmzJnDxo0bAahXr54KmuQKIZU0dz/m7qPd/S/ufou7jwZ0m2YREclymzdvpk6dOjRr1ozNmzczcuRI1q9fr1tpSK6T4XSnmcUALYHSwPvuvtLMmgKPAEWBi8MfUUREcrrk5GQ2bNhAhQoVKFWqFIULF2bcuHHcdddd5MunBW4kd8rsnLTxwIXAQmC0ma0FGgB93f3tMGcTEZEc7ujRo0ydOpXBgwezZ88e1q1bR4ECBfj444+DjiYSuMxK2hXAJe5+zMzOAH4BKrr7j+GPJiIiOdXhw4d5/fXXGTx4MOvWrePiiy/m+eefJ2/evEFHE4kYmZW0Q+5+DMDdfzOz1SpoIiJyuhYsWECHDh2oVasWM2fO5OabbyYmJtRr2URyh8xK2kVmtjj1sQHxqdsGuLvXDGs6ERHJEQ4cOMDYsWM5fPgwvXr1onHjxiQkJFCvXj1+vweniPy3zEqaLgwQEZFTtm/fPkaNGsWwYcPYsWMHN910Ez179sTMqF+/ftDxRCJahiXN3ddlVxAREclZZs6cyd/+9jd27drFddddx6OPPkrdunWDjiUSNUI+AcDM3kz7ZzRoN/Fr4vr8M+gYIiK5xs6dO9m2bRsAFSpU4Oqrr+arr77i/fffV0ETOUknc5ZmfOqfF4UjSDh8snrHH48bxpcIMImISM72888/06tXL8qVK0fPnj0BuOSSS5gzZw516tQJOJ1IdAp17c6otnHIjUFHEBHJkbZt28bQoUP/uCigdevWPPLII0HHEskRckVJExGR8Bg2bBijRo2ibdu29OnTh8qVKwcdSSTH0E1pREQkZImJiXTo0IGPPvoIgD59+rB27VpeeeUVFTSRLHYyJU03shERyaVWrlzJX//6V+Lj43njjTdYvXo1AOeeey5xcXHBhhPJoU5muvO5dH+KiEgucP/99zNq1CjOPPNMHnroIXr06EGpUqWCjiWS44U8kubur6X9M3UtTxERyYG++eYbjhw5AkD16tV55JFH2LhxI88884wKmkg2ybSkmVlJM6thZnlSt4ub2ZNAYtjTiYhItvrPf/5D06ZNufzyy3nzzZTbYnbu3JmBAwdSvHjxgNOJ5C4ZljQzewBYAYwDvjKzO4HVQFHgivDHExGRcHN3Pv74Yxo2bMg111zD4sWLGTJkCH/5y1+CjiaSq2V2TloXIN7dfzGzOGAN0NDdPwt3MBERyR7uzoMPPsjOnTt5/vnn6dSpE2eeeWbQsURyvcymOw+6+y8A7r4RWHMyBc3MmprZajNLNLM+J9ingZktMbPvzWx+yMlFROSUJCcnM3v2bK699lr27NlDTEwM77zzDuvXr6dbt24qaCIRIrORtDJmlvZqznPTbrv7Qyf6RjOLBV4CmgBbgYVm9q67r0izz9nAKKCpu282s3NP5S8hIiKZO3bsGDNmzGDgwIEsW7aMChUqsGHDBmrUqEGlSpWCjici6WRW0vpmsp2ROkCiu68HMLOpQHNSznH73R3AO+6+GcDdt5/E+4uISIiSkpKoXr06q1at4qKLLuK1116jdevW5MmjhWdEIlWGP53uPsHMigFlgfXuvvck3rs0sCXN9lb+92KDykBeM0sACgMvuPvkk/gMERE5gcOHD/PVV19xzTXXUKhQIZo0acKTTz5JixYtiI2NDTqeiGQiw5JmZu2AocBGoKyZdXD3uSG+9/FWKPDjfH4toDFwBvCFmX3p7mvS5egEdAIoUaIECQkJIUZIcbL7S/gkJSXpeEQpHbvocfjwYd577z2mTp3Kzp07eeONNzjjjDNo0aIFAJ9++mnACeVk6ecvd8psnPthoLq7/2xmFYHXgFBL2lbggjTbZYAfjrPPL+6+H9hvZguAS0m5ivQP7j4WGAsQHx/vDRo0CC3BvH8CEPL+EnYJCQk6HlFKxy7yHThwgJdffplnnnmGH3/8kauuuoqJEyfStGlT5s+fr+MXxfTzlztldnXnIXf/GcDdE4F8J/HeC4FKZlbezPIBrYF30+0zG7jGzPKY2ZmkTIeuPInPEBGRVNu3b6dXr17Ex8fz0Ucf8dlnn9GsWTPMtPSySDQ62as7Lwj16k53P2pm9wPvA7HAK+7+vZl1Tn19jLuvNLN5wHdAMjDe3Zef6l9GRCQ32b17NyNGjGDNmjW89tprxMXFsXLlSipWrBh0NBHJAuG8uhN3fw94L91zY9JtPwM8czLvKyKSm/3yyy8MHz6cESNGsHfvXm6++WYOHTpE/vz5VdBEcpDMSlqcuz+aLUmyULuJX/PJ6h1BxxARyXIffvght9xyCwcOHKBly5b069ePGjVqBB1LRMIgs3PSbsyWFFksbUFrGF8iwCQiIqdv27ZtLF68GIDLL7+c22+/neXLlzN9+nQVNJEcLLORtFgzK8zxb6fBSd43LdttHBKVHVNEBICNGzcyZMgQJk6cyMUXX8yiRYs4++yzmTBhQtDRRCQbZFbSLgK+579LmqduOyk3uRURkSyUmJjI4MGDee2114iJiaF9+/b07t076Fgiks0yK2kr3P2ybEkiIpLLuTtmxvz583nzzTe57777ePjhhylTpkzQ0UQkAJmdkyYiImG2ZMkSbr31VkaMGAFA27Zt2bhxI8OHD1dBE8nFMhtJG5ktKU6RruIUkWj29ddfM3DgQObMmUORIkWoW7cuAPny5aNkyZIBpxORoGU2klbbzKoc7wUzO8PM7jKzNmHIFZKMCpqu6hSRSNa7d2+uuOIKPvvsMwYMGMCmTZvo1q1b0LFEJIJkNpI2DhhkZvGkrAqwAygAVAKKA5OAl8IZ8HjSj6DpKk4RiXTuzscff0z16tUpWbIkTZs25ZxzzqFLly4ULlw46HgiEoEyHElz92/cvQVwFTCBlPU4/w3c7+4Xu/swdz+YDTn/i+6DJiLRwt157733uPrqq7n22msZNWoUAA0bNqRXr14qaCJyQpmNpAEp90MzswVA2dSF1iOCRtBEJJK9++67PPnkk3zzzTeULVuWUaNG0a5du6BjiUiUCOnqTjO7CVgGfJC6XcPMZoYzmIhINHL3Px5PmTKF3bt3M2HCBNauXUuXLl0oUKBAgOlEJJqEeguOJ4ArgN0A7r4E0Cq+IiKpjh49yuTJk6lWrRrff/89AKNHj2bVqlW0b9+efPnyBZxQRKJNqCXtiLvvTvecH3dPEZFc5PDhw4wfP574+Hjuvvtu8uTJw969KSvmFStWjDx5QjqrRETkf4T622OlmbUCYsysPPAg8GX4YomIRL5jx45x6aWXsmrVKi6//HKee+45/vznPxMTo/uEi8jpC/U3yf1ALSAZeAc4SEpRExHJVQ4cOMDrr7+OuxMbG0vXrl3517/+xddff03z5s1V0EQky4Q6kna9u/cG/ljh18xakFLYRERyvH379vHSSy/x3HPPsWPHDipXrkydOnXo0qVL0NFEJIcK9f/y9T/Oc/2yMoiISCTav38/Tz75JOXKlaNv377UqlWL//znP9SpUyfoaCKSw2U4kmZm1wNNgdJm9lyal4qQMvUpIpIjJScnExMTQ0xMDGPGjKFevXr069eP2rVrBx1NRHKJzKY7twPLSTkH7fs0z+8D+oQrlIhIUH766SeeffZZ/v3vf/PNN99wxhlnsGLFCs4+++ygo4lILpNhSXP3b4FvzWxKEMs/iYhkly1btvDMM88wbtw4Dh8+zB133MG+ffsoVqyYCpqIBCLUCwdKm9kgoCopC6wD4O6Vw5JKRCQbLVmyhDp16uDu3HXXXfTt25eKFXW/bhEJVvKlPNgAACAASURBVKgXDkwCJgIGNAPeAqaGKZOISNitWbOG2bNnA3DJJZfQr18/EhMTmTBhggqaiESEUEvame7+PoC7r3P3/kDD8MUSEQmP5cuX06ZNG6pUqcK9997LkSNHiImJ4fHHH6dcuXJBxxMR+UOoJe2QmRmwzsw6m9mfgXPDmEtEJEutWrWKli1bcvHFFzNnzhwefvhhFi9eTN68eYOOJiJyXKGek9YdKAR0BQYBZwHtwxVKRCSrHDlyhLx587Jnzx4++ugj+vfvT7du3TjnnHOCjiYikqGQSpq7f5X6cB/QFsDMyoQrlIjI6fr0008ZMGAA5cqVY9y4cVxxxRVs27aNggULBh1NRCQkmU53mlltM7vFzIqnblczs8logXURiTDuzocffkj9+vWpV68eS5cupWrVqn+8roImItEkw5JmZk8BU4A7gXlm1g/4BFgK6PYbIhJRnn76aZo0aUJiYiLDhw9nw4YNdO/ePehYIiKnJLPpzubApe7+m5kVA35I3V4d/mjHt3FvMucF9eEiElGSk5OZPXs25cqVo2bNmrRu3ZqzzjqLdu3aUaBAgczfQEQkgmU23XnQ3X8DcPddwKogC1paDeNLBB1BRAJy7Ngxpk2bxqWXXkqLFi0YNWoUAHFxcXTp0kUFTURyhMxG0i40s3dSHxsQl2Ybd28RtmQZ2DjkxiA+VkQiwNtvv03//v1ZvXo1VapU4fXXX+f2228POpaISJbLrKS1TLc9MlxBRERO5PDhw+TJk4eYmBhWrlxJgQIFmD59Oi1atCAmJtTbPYqIRJcMf7u5+0cZfWVXSBHJnQ4ePMjIkSOpUKECM2bMAKB37958++233HrrrSpoIpKj6TeciESc/fv389xzz1G+fHkeeOABypUrx3nnpVwylC9fPlIWQBERydlCXXFARCTbNGnShC+++IJGjRrx5ptvUr9+fRUzEcl1TqqkmVl+dz8UrjAikjv9+uuvjBkzhgcffJAzzzyTf/zjHxQqVIirr7466GgiIoEJabrTzOqY2TJgber2pWY2IqzJRCTH27FjB4888gjlypXjkUce4YMPPgDguuuuU0ETkVwv1HPSXgRuAnYCuPtSoGG4QolIznb48GF69OhBXFwcQ4YMoVmzZixdupTmzZsHHU1EJGKEOt0Z4+6b0p0TciwMeUQkB0tKSqJQoULkzZuXb775hpYtW9K3b1+qVKkSdDQRkYgTaknbYmZ1ADezWOABYE34YolITrJ+/XqGDBnCW2+9xapVqyhVqhQffvghefLo2iURkRMJ9TdkF1KmPMsCPwMfpj4nInJCq1evZvDgwUyZMoXY2Fg6dOjwx1WaKmgiIhkL9bfkUXdvHdYkIpKj/PTTT1SvXp28efPywAMP0LNnT84///ygY4mIRI1QS9pCM1sNTAPecfd9YcwkIlHq22+/5eOPP6ZHjx6UKlWKSZMmce2111KyZMmgo4mIRJ2Qru509wrAQKAWsMzMZpmZRtZEBICvvvqKm266iZo1azJo0CB+/fVXAO68804VNBGRUxTyslDu/rm7dwVqAnuBKZl9j5k1NbPVZpZoZn0y2K+2mR0zs1tDzSMiwVu7di1NmjThyiuv5Msvv2TgwIFs2LCBokWLBh1NRCTqhTTdaWaFgOZAa6AKMBvI8E6TqVeBvgQ0AbaSMmX6rruvOM5+TwPvn3R6Ecl27s7u3bspWrQoZ511FuvXr+eZZ56hc+fOFCpUKOh4IiI5RqjnpC0H5gBD3f3TEL+nDpDo7usBzGwqKUVvRbr9HgBmALVDfF8RCYC7889//pMBAwYA8MUXX3Duueeydu1aYmJCHpQXEZEQhVrSLnT35JN879LAljTbW4Er0u5gZqWBvwCNUEkTiUjJycnMmjWLPn36sHbtWuLi4ujTpw/ujpmpoImIhEmGJc3Mhrl7D2CGmXn61929RUbffpzn0r/HcKC3ux9Lt5pB+hydgE4A+UpVJCEhIaPYEsGSkpJ0/KLMvHnzePrppzn//PPp3bs31157LXny5GHBggVBR5OToJ+96KbjlztlNpI2LfXPkafw3luBC9JslwF+SLfP5cDU1IJWHLjBzI66+6y0O7n7WGAsQP7zKnmDBg1OIY5EgoSEBHT8ItuRI0d44403KFSoEC1btuSKK66gRo0alChRgsaNGwcdT06Rfvaim45f7pThPIW7f536sIq7f5T2i5QLCDKyEKhkZuXNLB8pFx28m+79y7t7nLvHAW8D96YvaCKSPQ4dOsTYsWOJj4/nnnvu4fXXXwfgjDPOoHXr1sTGxgacUEQkdwn1ZJL2x3muQ0bf4O5HgftJuWpzJfCWu39vZp3NrPPJxRSRcHr77bepWLEif//73ylRogRz5szhnXfeCTqWiEiultk5abeTMgJW3szS/sYuDOzO7M3d/T3gvXTPjTnBvvdk9n4iknX2799PcnIyhQsXJjY2lri4OCZMmECTJk3I6BxRERHJHpmdk/Y1sJOU88leSvP8PuDbcIUSkfDZu3cvI0eO5LnnnuP+++/nH//4B7fccgu33HKLypmISATJsKS5+wZgA/Bh9sQRkXDZtWsXL7zwAi+++CK7d+/mhhtuoFmzZgAqZyIiESiz6c757l7fzH7lv2+fYYC7e7GwphORLNO5c2emT5/OLbfcQv/+/alVq1bQkUREJAOZXTjQMPXP4kCJNF+/b4tIhPrxxx/p0aMHiYmJADzxxBMsXbqUmTNnqqCJiESBzKY7f19l4ALgB3c/bGZ1gUuA10lZaF1EIsiWLVt4+umnGT9+PEePHqVq1apUrFiRKlUyu2uOiIhEklBvwTELcDOrAEwm5R5pb4QtlYicNHfngQceoEKFCowdO5a2bduyevVqOnTI8G45IiISoUItacnufgRoAQx39wdIWZtTRAK2efNmgD/W0ezYsSOJiYmMGzeOChUqBJxOREROVagl7aiZ3Qa0BeamPpc3PJFEJBTLly+ndevWxMXF8fnnnwPwwgsv8NJLL1G2bNmA04mIyOk6mRUHGgJD3X29mZUH3gxfLBE5kcWLF9OiRQsuvvhi/vnPf9KrVy8qVaoUdCwREclimd3MFgB3X25mXYGKZnYRkOjug8IbTUTSO3jwIE2aNOHYsWM89thjPPjggxQrpjvhiIjkRCGVNDO7BngN2EbKPdJKmVlbd/8snOFEBObPn89rr73G2LFjKVCgALNnz+biiy/mrLPOCjqaiIiEUajTnc8DN7j7n9z9auBG4IXwxRLJ3dydf//739SrV48GDRowd+5c1q1bB0DdunVV0EREcoFQS1o+d1/x+4a7rwTyhSeSSO62bds2rrrqKq6//no2bNjAiBEj2LBhg847ExHJZUKa7gQWm9nLpEx5AtyJFlgXyTLJycmsW7eOSpUqUbJkSYoUKcLLL7/M3XffTf78+YOOJyIiAQi1pHUGugK9SDknbQEwIlyhRHKLY8eO8dZbbzFo0CC2b9/Ohg0bKFiwIP/+97+DjiYiIgHLtKSZ2cVABWCmuw8NfySRnO/IkSNMmTKFwYMHs3btWqpVq8YLL7xAgQIFgo4mIiIRIsNz0szsEVKWhLoT+MDM2mdLKpEc7osvvqBdu3YULFiQGTNm8N1339GmTRtiY2ODjiYiIhEis5G0O4FL3H2/mZUA3gNeCX8skZzlt99+Y/z48ezdu5d+/fpxzTXXkJCQQL169TCzoOOJiEgEyuzqzkPuvh/A3XeEsL+IpJGUlMSzzz5L+fLl6dq1K/Pnz8fdMTPq16+vgiYiIieU2UjahWb2TupjAyqk2cbdW4QtmUiUmzt3Lvfccw87d+6kcePGTJs2jfr16wcdS0REokRmJa1luu2R4QoikhPs2rWL/fv3c8EFF1CxYkWuvPJK+vXrx1VXXRV0NBERiTIZljR3/yi7gohEs+3bt/P8888zcuRIrr/+et5++20uuugi5s6dG3Q0ERGJUqHeJ01EjuPHH3/kmWeeYcyYMRw8eJBWrVrRr1+/oGOJiEgOoJImchpGjBjBiy++yJ133skjjzxCfHx80JFERCSHOKmrNc1M69NIrrZu3To6duzIvHnzAOjRowdr1qzh1VdfVUETEZEsFVJJM7M6ZrYMWJu6famZaVkoyTVWrVrFXXfdRXx8PK+99hqrV68G4JxzzuHCCy8MOJ2IiOREoY6kvQjcBOwEcPelQMNwhRKJJN27d6dq1arMmDGDBx98kA0bNvDggw8GHUtERHK4UEtajLtvSvfcsawOIxIpFi9ezKFDhwCoXr06ffr0YePGjQwbNozzzjsv4HQiIpIbhFrStphZHcDNLNbMugFrwphLJBBffPEFN9xwA7Vq1WLy5MkAdOjQgcGDB1OiRImA04mISG4SaknrAjwElAV+Bq5MfU4k6rk7CQkJNG7cmKuvvpqFCxcyePBgbr/99qCjiYhILhbSLTjcfTvQOsxZRALTo0cPtm3bxrPPPkvnzp0pWLBg0JFERCSXC6mkmdk4wNM/7+6dsjyRSJi5O3PnzmX48OFMnz6dYsWK8dZbb3H++edzxhlnBB1PREQECH2680Pgo9Svz4BzgUPhCiUSDsnJycyYMYOaNWty8803s379etavXw9AhQoVVNBERCSihDrdOS3ttpm9BnwQlkQiYZCUlMRVV13F8uXLqVy5MpMmTeKOO+4gb968QUcTERE5rpNacSCN8kC5rAwiktWOHDnCggULAChUqBBNmjThjTfeYMWKFdx9990qaCIiEtFCPSftV/7/nLQYYBfQJ1yhRE7HoUOHmDRpEkOGDGHLli0kJiYSFxfHc889F3Q0ERGRkGU6kmZmBlwKlEj9KuruF7r7W+EOJ3IyfvvtN1588UUqVKhA586dKVmyJLNnz6ZcOQ36iohI9Ml0JM3d3cxmunut7Agkcqp27txJz549ufLKK5k0aRKNGzcm5f9jiIiIRJ+QpjuBr82sprsvDmsakZOwZ88eRo4cyXfffce0adMoU6YMK1asoEKFCkFHExEROW0ZTnea2e8lri4pRW21mS02s2/NTIVNArFr1y4ee+wxypUrR//+/dm/fz+//fYbgAqaiIjkGJmNpH0N1ARuyYYsIpmaP38+N910E0lJSbRo0YL+/ftz2WWXBR1LREQky2VW0gzA3ddlQxaR4/rhhx/44YcfuPzyy6lZsyatWrWie/fuVK9ePehoIiIiYZNZSSthZg+d6EV31z0NJGw2bdrE008/zYQJE4iPj2fp0qUULlyYCRMmBB1NREQk7DIrabFAIVJH1ESyw/r16xk8eDCvvvoqZka7du3o3bu3rtQUEZFcJbOS9qO7P5ktSSTXc3fMjC+++IIpU6bQpUsXevbsyQUXXBB0NBERkWyX2c1sNXQhYffdd9/RqlUrhg0bBsDtt9/Ohg0bePHFF1XQREQk18qspDU+nTc3s6apt+1INLP/WUbKzO40s+9Svz43s0tP5/MkuixatIhbbrmFSy+9lHnz5uGesvJYnjx5KFWqVMDpREREgpVhSXP3Xaf6xmYWC7wENAOqAm3MrGq63TYA9d39EmAAMPZUP0+iS//+/alduzbz58/nH//4Bxs3bqRnz55BxxIREYkYoa44cCrqAInuvh7AzKYCzYEVv+/g7p+n2f9LoEwY80iA3J358+ezY8cOAJo2bUrBggW57777KFKkSMDpREREIk84S1ppYEua7a3AFRns3wH41/FeMLNOQCeAfKUqkpCQkEURJdzcnYULF/L666+zbNkyWrZsSYkSJQC46qqrWLxYC1dEi6SkJP3sRTEdv+im45c7hbOkHe+iAz/ujmYNSSlpdY/3uruPJXUqNP95lbxBgwZZFFHCae7cuTz55JMsXLiQMmXKMGLECCpVqoSOX3RKSEjQsYtiOn7RTccvd8rswoHTsRVIe2leGeCH9DuZ2SXAeKC5u+8MYx7JBr+f/A/w1ltv8csvvzB27FjWrVvH/fffT/78+QNMJyIiEj3CWdIWApXMrLyZ5QNaA++m3cHMygLvAG3dfU0Ys0iYHT16lClTplC9enW+/fZbAF544QVWr15Nx44dyZcvX8AJRUREokvYSpq7HwXuB94HVgJvufv3ZtbZzDqn7vYYcA4wysyWmNmicOWR8Dhy5AgTJ06kSpUq/PWvfyUmJob9+/cDULRoUfLmzRtwQhERkegUznPScPf3gPfSPTcmzeO/AX8LZwYJn+TkZGrVqsWyZcuoWbMm77zzDs2bNycmJpwDtCIiIrlDWEua5DwHDhzg7bffpm3btsTExPDAAw9QunRpmjVrprU1RUREspBKmoRk3759jB49mmHDhrF9+3YuvPBC6tatS8eOHYOOJiIikiNpXkoydODAAQYOHEhcXBy9e/emRo0aLFiwgLp1j3u3FBEREckiGkmT4zp27BixsbHExsYyduxY/vSnP9GvXz+uuCKj+xGLiIhIVlFJk//y888/M2zYMObMmcOSJUvInz8/y5Yt46yzzgo6moiISK6i6U4BYNu2bXTr1o3y5cszbNgwLrvsMvbt2weggiYiIhIAjaQJ33//PTVr1uTYsWO0bduWvn37Urly5aBjiYiI5GoaSculEhMTmTFjBgBVq1alf//+rF27lokTJ6qgiYiIRACVtFxm5cqV/PWvfyU+Pp4uXbpw6NAhzIxHH32U8uXLBx1PREREUqmk5RJr166lVatWVKtWjZkzZ9K9e3e+++47LXguIiISoXROWg535MgR8ubNS1JSEu+//z59+/alW7dulChRIuhoIiIikgGVtBzqs88+Y+DAgZQoUYLJkydz2WWX8cMPP1CwYMGgo4mIiEgINN2Zg7g7n3zyCY0aNaJu3bosWrSISy655I/XVdBERESih0bScpDnn3+eHj16cN555/Hcc8/RqVMnFTMREZEopZIWxdydOXPmUKpUKerUqUOrVq0oUKAA7du3p0CBAkHHExERkdOg6c4olJyczPTp06lRowbNmzdnxIgRAJQpU4Z7771XBU1ERCQHUEmLMjNnzqR69eq0atWKQ4cOMXnyZCZOnBh0LBEREclimu6MAocPHyY2NpbY2FjWrFlDnjx5mDZtGi1btiQ2NjboeCIiIhIGGkmLYAcPHmT06NFUqlSJN998E4Du3buzZMkSWrVqpYImIiKSg6mkRaADBw4wfPhwKlSowL333kvp0qUpW7YsAPny5SMmRodNREQkp9N0ZwS68cYbSUhIoH79+kyePJlGjRphZkHHEhERkWykIZkIsHv3bp566in27dsHwKOPPsqCBQtISEigcePGKmgiIiK5kEbSArRz506ef/55RowYwd69e6lYsSK33XYbjRo1CjqaiIiIBEwjaQE4evQovXr1oly5cgwaNIgmTZrw7bffcttttwUdTURERCKESlo2SkpKAiBPnjwsWbKE5s2bs3z5ct5++21q1KgRcDoRERGJJJruzAabNm1iyJAhTJkyhRUrVlCmTBnee+898uTRP7+IiIgcn1pCGCUmJjJ48GBee+01zIz27dv/cW8zFTQRERHJiJpCmPzyyy9Uq1aNmJgY7r33Xnr27EmZMmWCjiUiIiJRQiUtCy1dupT333+fXr16Ubx4cSZNmkTDhg0pVapU0NFEREQkyujCgSywcOFCmjdvTo0aNRg8eDA7duwAoE2bNipoIiIickpU0k7Dhg0baNq0KXXq1OHTTz/lySefZOPGjZQoUSLoaCIiIhLlNN15ktydX3/9lWLFinHWWWeRmJjIkCFDuPfeeylcuHDQ8URERCSHUEkLkbszb948BgwYwMGDB/nmm28oVqwYa9as0YLnIiIikuXULjKRnJzMrFmzqF27NjfccAPbtm2jY8eOJCcnA6igiYiISFhoJC0T06ZN44477qBChQqMHz+etm3bki9fvqBjiYiISA6nkpbO0aNHefPNN8mbNy+tW7emRYsWvPnmm9x66626Aa2IiIhkG83VpTp8+DDjx48nPj6eu+66i8mTJwOQP39+WrdurYImIiIi2UolDZg1axaVKlWiY8eOFC1alJkzZzJ37tygY4mIiEgulmuHhw4cOMDRo0cpUqQIefPmpXTp0owZM4amTZtiZkHHExERkVwu142k7du3j6effpq4uDieeuopAG644QY+++wzmjVrpoImIiIiESHXjKTt3r2bESNGMHz4cHbt2sX111/Pn//8ZwAVMxEREYk4uaak3X///UyZMoWbb76Zfv36UadOnaAjiYiIiJxQjp3u/Omnn+jZsyerVq0C4LHHHuPbb79l9uzZKmgiIiIS8XLcSNrWrVt55plnGDt2LIcPH6ZixYpcdNFFVK5cOehoIiIiIiHLUSXtoYce4qWXXiI5OZm77rqLPn36UKlSpaBjiYiIiJy0sE53mllTM1ttZolm1uc4r5uZvZj6+ndmVvNkP2PTpk24OwB58uShffv2rF27lgkTJqigiYiISNQKW0kzs1jgJaAZUBVoY2ZV0+3WDKiU+tUJGB3q+3///ffccccdXHjhhSQkJAAwdOhQRo8eTVxc3On/BUREREQCFM7pzjpAoruvBzCzqUBzYEWafZoDkz1lKOxLMzvbzM5z9x9P9KZ+5BC33norM2bMoGDBgvTo0YOqVdN3PxEREZHoFs6SVhrYkmZ7K3BFCPuUBk5Y0o78+gMffPAB/fr1o1u3bhQvXjyr8oqIiIhEjHCWtOPdIdZPYR/MrBMp06EAh/bu3bt80KBBDBo06DQjSgCKA78EHUJOiY5ddNPxi246ftEr/lS/MZwlbStwQZrtMsAPp7AP7j4WGAtgZovc/fKsjSrZRccveunYRTcdv+im4xe9zGzRqX5vOK/uXAhUMrPyZpYPaA28m26fd4G7Uq/yvBLYk9H5aCIiIiK5RdhG0tz9qJndD7wPxAKvuPv3ZtY59fUxwHvADUAicABoF648IiIiItEkrDezdff3SCliaZ8bk+axA/ed5NuOzYJoEhwdv+ilYxfddPyim45f9DrlY2e/3whWRERERCJHjl1gXURERCSaRWxJy44lpSQ8Qjh2d6Yes+/M7HMzuzSInHJ8mR2/NPvVNrNjZnZrduaTjIVy/MysgZktMbPvzWx+dmeU4wvhd+dZZjbHzJamHjudxx0hzOwVM9tuZstP8PopdZaILGnhXlJKwifEY7cBqO/ulwAD0LkWESPE4/f7fk+TcmGQRIhQjp+ZnQ2MAm5292rAbdkeVP5HiD979wEr3P1SoAEwLPXuCRK8SUDTDF4/pc4SkSWNNEtKufth4PclpdL6Y0kpd/8SONvMzsvuoPI/Mj127v65u/+auvklKffHk8gQys8ewAPADGB7doaTTIVy/O4A3nH3zQDurmMYGUI5dg4UNjMDCgG7gKPZG1OOx90XkHI8TuSUOkuklrQTLRd1svtI9jvZ49IB+FdYE8nJyPT4mVlp4C/AGCTShPLzVxkoamYJZvaNmd2VbekkI6Ecu5FAFVJu+r4MeNDdk7MnnpymU+osYb0Fx2nIsiWlJNuFfFzMrCEpJa1uWBPJyQjl+A0Herv7sZT/Qy8RJJTjlweoBTQGzgC+MLMv3X1NuMNJhkI5dtcDS4BGQAXgAzP71N33hjucnLZT6iyRWtKybEkpyXYhHRczuwQYDzRz953ZlE0yF8rxuxyYmlrQigM3mNlRd5+VPRElA6H+7vzF3ffzf+3deYhWVRjH8e+PsrTNaMWKmqLMNpVWSVpso6ioRLIwySBC26gQIooW6g9pgVZbsNDAIhTbbCckTTSbMp1SW7CIIMpoIWSk0qc/znnz9npz7jtjMzf6feAF57n3LO89OD6ec+89sFbSfGAY4CStb1UZu8uAKfkdo19I+hIYAizpnS5aD3QrZ6nrcqe3lPrv6nLsJO0LzAHG+3/vtdPl+EXE/hHRFhFtwGzgSidotVHld+eLwAmStpa0HXAcsLKX+2mbqjJ2X5NmQJG0J2nj7tW92kvrrm7lLLWcSfOWUv9dFcfuVmBXYGqejfnDGwfXQ8Xxs5qqMn4RsVLS68ByYAMwLSJKXxtgvafi3707gemSOkjLZzdGxA991mn7i6RnSU/c7ibpG+A2oB/0LGfxjgNmZmZmNVTX5U4zMzOz/zUnaWZmZmY15CTNzMzMrIacpJmZmZnVkJM0MzMzsxpykmZmZmZWQ07SzGwTktZL+qjwadvMuW2SevyerbyX5KeSlklaKOngbtQxsbEXpaQJkvYqHJsm6dAt3M/3JQ2vUOa6/OLYVtu6X9KJTe02xmRMjjfG6mNJsxrtNMVflrRzju+e35NmZjXnJM3MynRGxPDC56teandcRAwDZgD3tFo4v6z16fzjBGCvwrHLI2LFFunlxn5OpVo/rwNaStIk7QKMiIj5Te02xmR2jjXG6nDgN2BiSfxH4CqAiFgDfCtpZCv9MbPe5yTNzCrJM2YLJH2YP8eXnHOYpCV5Bme5pINy/JJC/HFJW3XR3HzgwFz2VElLJXVIekrStjk+RdKK3M69OXa7pMl5luloYGZuc0CeiTpa0iRJdxf6PEHSQ93s5yJg70Jdj0pql/SJpDty7FpSsjhP0rwcO0PSonwdZ0naoaTuMUCrM14LGtdtc/0EXgDGtVi3mfUyJ2lmVmZAYVnt+Rz7Hjg9Io4ExgIPlpSbCDwQEcNJSdI3kg7J54/M8fV0nSCcC3RI6g9MB8ZGxBGkrewm5VmmC4DDImIocFexcJ5lamfjzFNn4fBsYHTh57HAc93s55mkhKfh5rzF2VDgJElDI+JB0kbKoyJilKTdgFuA0/K1bAduKKl7JPBBU2xmYVx2LR6QtDVwFtDRFN+KtN9jcR/IduCELr6bmfWxWu7daWZ9rjMnKkX9gIfzPVjrgcEl5RYBN0vaB5gTEZ9LOhU4Cng/79U6gJTwlZkpqRP4CriGtIH0lxHxWT4+g7Rs9zCwDpgm6RVgbtUvFhFrkOcdHAAAAlpJREFUJK1W2uT489zGwlxvK/3cnrTH4pGF+IWSriD9bh0EHEraI7NoRI4vzO1sQ7puzQYBa5pi4yKivSk2QNJH+c8LgCeb4m2kZO+tQpnvKSwFm1k9OUkzs6quB74DhpFm4dc1nxARz0h6DzgbeEPS5aSNoGdExE0V2vhbEtI8W1Ro5w9Jx5JmiC4CrgZOaeG7PAdcCKwCno+IUMqYKvcTWAZMAR4BRkvaH5gMHBMRP0maDvQvKSvgrYi4uIs2Ov+h/CbnlSTUf8UlDSQlsVexcfazf67fzGrMy51mVtVA4NuI2ACMJ80i/Y2kA4DVeYnvJdKy39vAGEl75HN2kbRfxTZXAW2SGvdZjQfeyfdwDYyIV0k35ZclKb8CO/5DvXOA84GLSQkbrfYzIn4nLVuOyEulOwFrgV8k7Ulaeizry2JgZOM7SdpOUtms5ErK7y9rSUT8AlwLTJbUL4cHAz1+ItfM/l1O0sysqqnApZIWk/6RX1tyzljg47zMNgR4Oj9ReQvwpqTlpGW3QVUajIh1wGXALEkdwAbgMVLCMzfX9w5plq/ZdOCxxoMDTfX+BKwA9ouIJTnWcj/zvW73AZMjYhmwFPgEeIq0hNrwBPCapHn56coJwLO5ncWka9XsFeDkzbVfVUQsJc38XZRDo3L9ZlZjioi+7oOZmZWQ9C5wTkT8vIXrnQ+cl5NVM6spJ2lmZjUl6TjSvWXNDx/0pM7dSU+wvtDlyWbWp5ykmZmZmdWQ70kzMzMzqyEnaWZmZmY15CTNzMzMrIacpJmZmZnVkJM0MzMzsxr6E4CU/E/1bzCVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the ROC curve\n",
    "plot_roc_curve(y_train, y_scores)\n",
    "roc_auc = roc_auc_score(y_train, y_scores)\n",
    "print(\"ROC AUC Score: {:.3f}\".format(roc_auc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "\n",
    "* Precision, Recall and F1 - <a href=\"https://www.youtube.com/watch?v=jJ7ff7Gcq34\">Introduction to Precision, Recall and F1 | Classification Models</a>\n",
    "    * <a href=\"https://www.youtube.com/watch?v=j-EB6RqqjGI\">Precision, Recall & F-Measure</a>\n",
    "* Stratification - <a href=\"https://www.youtube.com/watch?v=rsNCCQhkKN8\">Sampling 03: Stratified Random Sampling</a>\n",
    "* Cross-Validation - <a href=\"https://www.youtube.com/watch?v=7062skdX05Y\">What is Cross Validation and its types?</a>\n",
    "    * K-Fold Cross Validation - <a href=\"https://www.youtube.com/watch?v=TIgfjmp-4BA\">K-Fold Cross Validation - Intro to Machine Learning</a>\n",
    "* ROC Curve - <a href=\"https://www.youtube.com/watch?v=OAl6eAyP-yo\">ROC Curves and Area Under the Curve (AUC) Explained</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com\n",
    "* Datasets from <a href=\"https://www.kaggle.com/\">Kaggle</a> - https://www.kaggle.com/\n",
    "* Examples and code snippets were taken from <a href=\"http://shop.oreilly.com/product/0636920052289.do\">\"Hands-On Machine Learning with Scikit-Learn and TensorFlow\"</a>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}