notebooks/13_advanced_model_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation and Advanced Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "**Data preparation**\n",
    "\n",
    "- Handling missing values\n",
    "- Handling categorical features (review)\n",
    "\n",
    "**Advanced model evaluation**\n",
    "\n",
    "- ROC curves and AUC\n",
    "- Bonus: ROC curve is only sensitive to rank order of predicted probabilities\n",
    "- Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Handling missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn models expect that all values are **numeric** and **hold meaning**. Thus, missing values are not allowed by scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891, 11)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read the Titanic data\n",
    "import pandas as pd\n",
    "url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/titanic.csv'\n",
    "titanic = pd.read_csv(url, index_col='PassengerId')\n",
    "titanic.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived      0\n",
       "Pclass        0\n",
       "Name          0\n",
       "Sex           0\n",
       "Age         177\n",
       "SibSp         0\n",
       "Parch         0\n",
       "Ticket        0\n",
       "Fare          0\n",
       "Cabin       687\n",
       "Embarked      2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for missing values\n",
    "titanic.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One possible strategy is to **drop missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(183, 11)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows with any missing values\n",
    "titanic.dropna().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(714, 11)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows where Age is missing\n",
    "titanic[titanic.Age.notnull()].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes a better strategy is to **impute missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.69911764705882"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean Age\n",
    "titanic.Age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# median Age\n",
    "titanic.Age.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    24\n",
       "dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# most frequent Age\n",
    "titanic.Age.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# fill missing values for Age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another strategy would be to build a **KNN model** just to impute missing values. How would we do that?\n",
    "\n",
    "If values are missing from a categorical feature, we could treat the missing values as **another category**. Why might that make sense?\n",
    "\n",
    "How do we **choose** between all of these strategies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Handling categorical features (Review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we include a categorical feature in our model?\n",
    "\n",
    "- **Ordered categories:** transform them to sensible numeric values (example: small=1, medium=2, large=3)\n",
    "- **Unordered categories:** use dummy encoding (0/1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>54</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "      <td>male</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>349909</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "      <td>female</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347742</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "      <td>female</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>237736</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "6                   0       3   \n",
       "7                   0       1   \n",
       "8                   0       3   \n",
       "9                   1       3   \n",
       "10                  1       2   \n",
       "\n",
       "                                                          Name     Sex  Age  \\\n",
       "PassengerId                                                                   \n",
       "1                                      Braund, Mr. Owen Harris    male   22   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female   38   \n",
       "3                                       Heikkinen, Miss. Laina  female   26   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   35   \n",
       "5                                     Allen, Mr. William Henry    male   35   \n",
       "6                                             Moran, Mr. James    male   28   \n",
       "7                                      McCarthy, Mr. Timothy J    male   54   \n",
       "8                               Palsson, Master. Gosta Leonard    male    2   \n",
       "9            Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female   27   \n",
       "10                         Nasser, Mrs. Nicholas (Adele Achem)  female   14   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "PassengerId                                                          \n",
       "1                1      0         A/5 21171   7.2500   NaN        S  \n",
       "2                1      0          PC 17599  71.2833   C85        C  \n",
       "3                0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "4                1      0            113803  53.1000  C123        S  \n",
       "5                0      0            373450   8.0500   NaN        S  \n",
       "6                0      0            330877   8.4583   NaN        Q  \n",
       "7                0      0             17463  51.8625   E46        S  \n",
       "8                3      1            349909  21.0750   NaN        S  \n",
       "9                0      2            347742  11.1333   NaN        S  \n",
       "10               1      0            237736  30.0708   NaN        C  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# encode Sex_Female feature\n",
    "titanic['Sex_Female'] = titanic.Sex.map({'male':0, 'female':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a DataFrame of dummy variables for Embarked\n",
    "embarked_dummies = pd.get_dummies(titanic.Embarked, prefix='Embarked')\n",
    "embarked_dummies.drop(embarked_dummies.columns[0], axis=1, inplace=True)\n",
    "\n",
    "# concatenate the original DataFrame and the dummy DataFrame\n",
    "titanic = pd.concat([titanic, embarked_dummies], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Sex_Female</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                     Name   Sex  Age  SibSp  \\\n",
       "PassengerId                                                                \n",
       "1                   0       3  Braund, Mr. Owen Harris  male   22      1   \n",
       "\n",
       "             Parch     Ticket  Fare Cabin Embarked  Sex_Female  Embarked_Q  \\\n",
       "PassengerId                                                                  \n",
       "1                0  A/5 21171  7.25   NaN        S           0           0   \n",
       "\n",
       "             Embarked_S  \n",
       "PassengerId              \n",
       "1                     1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we **interpret** the encoding for Embarked?\n",
    "- Why didn't we just encode Embarked using a **single feature** (C=0, Q=1, S=2)?\n",
    "- Does it matter which category we choose to define as the **baseline**?\n",
    "- Why do we only need **two dummy variables** for Embarked?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.793721973094\n"
     ]
    }
   ],
   "source": [
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived\n",
    "\n",
    "# train/test split\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "from sklearn import metrics\n",
    "print metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: ROC curves and AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# predict probability of survival\n",
    "y_pred_prob = logreg.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (8, 6)\n",
    "plt.rcParams['font.size'] = 14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x18bea128>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAGJCAYAAAD42ltKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe4XFW5x/HvL6GHIhhI6BCkF0MJClIOAipIu+gVKSpV\nREUQLzZQYs0FqSpFIgEFIlVRej8oCT20S4cQQwmEnk5J3vvH2sdMhjkz+5wz9czv8zz7OTN7r733\nOzsnZ7+z1tprKSIwMzOz9jSg0QGYmZlZ4zgRMDMza2NOBMzMzNqYEwEzM7M25kTAzMysjTkRMDMz\na2NOBMzMzNpYXRMBSdtJ+oekFyXNk/S1HPtsLOkOSbOy/X5Sj1jNzMzaQb1rBAYBjwBHAbOBsqMZ\nSVoauBmYAmyR7XespGNqHKeZmVlbUKNGFpQ0HfhWRPy5TJkjgFHAkIh4N1t3HHBERKxSn0jNzMz6\nr2bvI7AV8K+uJCBzE7CSpNUbFJOZmVm/0eyJwFDg1aJ1rxZsMzMzsz5o9kTAMyKZmZnV0EKNDqCC\nV/jwN/8hBdsWIMmJg5mZtZWIUF/2b/YagbuAbSUtWrBuZ+CliPh3qR0iwksNlxNOOKHhMfT3xdfY\n17m/LL7GvVs22CD4v//LV7Ya6j2OwCBJwyUNz869evZ+1Wz7KEm3FOwyFpgFXCBpQ0l7Az8ATq1n\n3GZmZv1VvZsGRgC3Za8D+Fm2XAAcTGoGGNZVOCKmSdoZOBO4H3gTODkiTqtjzGZm1s889xxsvDF8\n8EGjI/mwCBg0qH7nq2siEBGdlKmFiIiDSqz7P2D7GoZlPdDR0dHoEPo9X+P68HWuvWa+xu+8A+us\nA/fe2+hIPkyChReu4/mq1cbQDCRFf/o8ZmZWGxMmwKGHpp+tTBLRx86Czf7UgJmZWS7vvAPXXANT\nplQu++KLtY+nVTgRMDOzljVtGlx9NVx2GXR2QkcHrL125f0WWgiOOqrW0bUGNw2YmVlLmTFj/s3/\ntttgu+3gS1+CPfaAZZZpdHT1VY2mAScCZmbW9GbOhGuvTTf/m2+GT30q3fz32gs+8pFGR9c4TgSK\nOBEwM+sfIuCpp+Cmm+DGG+HOO+GTn4R99kk3/+WWa3SEzcGJQBEnAmZmreutt+DWW9ON/6abYN48\n+Oxn07Ljjr75l+JEoIgTATOz1vHBB+k5/q4b/2OPwTbbpBv/Zz4D662Xnqm37jkRKOJEwMysuU2a\nNL+6/7bbYPXV003/s59N7f6LLdboCFuLE4EiTgTMzJrLjBnpsb6ub/1vvTX/xr/zzjC0eH5Z6xEn\nAkWcCJiZNda8efDQQ/O/9d9/P4wYMf/m//GPw4Bmn/e2hTgRKOJEwMys/qZMSY/03Xhj+rnccvNv\n/NtvD0su2egI+y8nAkWcCJiZ1d6cOelxvq5v/ZMnp179XdX9a6zR6AjbhxOBIk4EzMyqLwKeeGL+\njX/cONhoo/nf+keMSEP2Wv05ESjiRMDMrDrefBNuuWX+zX/AgPmP9e24Iyy7bKMjNHAi8CFOBMzM\neu+FF2D06HTjf+KJNIZ/17f+ddbxM/3NyNMQm5lZ1fz5z6ntf9So9Ez/oos2OiKrBycCZmb2H1tt\nBZ/+dKOjsHry05xmZmZtzImAmZkRAc88Awsv3OhIrN6cCJiZGSecABMmwLe/3ehIrN7cR8DMrM39\n7Gfw17/C7bfD4MGNjsbqzYmAmVkb+8Uv4NJLUxKw/PKNjsYawYmAmVmb+vWvYezYlAQMGdLoaKxR\nnAiYmbWh//3fNG7A7bd7KuB250TAzKzN/OY3MGYMdHbCiis2OhprNCcCZmZ99M47cMYZcP316TG8\nZvbBBynezk5YaaVGR2PNwHMNmJn10rRp8NvfpiRg113h4INbY1jeddf1pEH9hecaMDNrgOnT4fe/\nh9NOSxPyjBuXJuUxa0VOBMzMcpoxA848E049NU3F+89/wnrrNToqs75xImBmVsGsWXDWWXDyybD9\n9qmn/QYbNDoqs+pwImBm1o3Zs+Gcc+Ckk2CbbeCWW2CjjRodlVl1OREwMysyezacey6ceCJ88pNw\n442wySaNjsqsNpwImFlLOfNM+NvfanuOJ56AESPguutg+PDansus0ZwImFlL6eyErbZKbfW1stJK\n7gNg7cOJgJm1nE02gZ12anQUZv2DEwEz67HHHoOZMxtz7jffbMx5zforJwJm1iOjRqWR9FZbrTHn\nl2CttRpzbrP+yImAmeV20klw/vkwYYLHqTfrL5wImFkup5wCo0d7shqz/saJgFkbueWW9PhdT82e\nDc8+m5KAlVeuelhm1kBOBMzayPjxsPDCsO++Pd93661hyJDqx2RmjeVEwKzNrLce/Nd/NToKM2sW\nTgTM+oGXXoJp0yqXe+01z0NvZgtyImDWD2y0EaywAgwYULns8cfXPh4zax2KiEbHUDWSoj99HrO8\nBg2CqVPTTzNrH5KICPXlGDm+P5iZmVl/5aYBswa6/XYYM6bvx3n33b4fw8zakxMBswa5+WbYf38Y\nORKWWqpvx/rCF9wsYGa940TArAFuvRX22w/++lfYdttGR2Nm7cyJgLWFGTPgrbcaHUXyyCNw4IFw\n5ZVOAsys8ZwIWFvYe2948EFYbLFGRwKLLgqXXw7bbdfoSMzMnAhYm5gzJ30D983XzGxBfnzQzMys\njblGwGrmootg3LhGR5E880yjIzAza05OBKwmfvc7OPVUOPZYUJ/GvKqOTTeFzTdvdBRmZs2n7omA\npG8CxwJDgceAoyPizjLldwV+CmwAvAuMA46NCH/Ha1JnnQWnnJLmrl9jjUZHY2Zm5VSca0DSmsCX\nge2BdYBlgLeBp4F/ApdExPO5TibtA1wIHAHcCXwLOAjYICJeKFH+Y8DjwKnAucBSwInAWhGxdony\nnmugBubOhenT85UdOxZOPDElAWuuWdOwzMzaXjXmGug2EZA0HPgVsDNwH3Av8AIwDVgaWB3YEtgc\nuBk4LiIeqhDwPcBDEXF4wbqngSsi4sclyn8RuARYuOsOL2kH4FZgcES8WVTeiUANfP/7qap/0UUr\nl11xRbjmGlhrrdrHZWbW7qqRCJRrGrgaOA04JCJeKRPEEOCArPyqZcotAmwGnFS06SZg6252GwfM\nAA6TdB6wBHAgcG9xEmC1M3MmnHwyfOtbjY7EzMyqrVwi8LGIqDiVSUS8Cpwi6fcVig4GBgKvFq2f\nSuovUOrYU7I+AlcBZ5Ied3wQ2KVSXGZmZlZZt4lAYRIgaXilav88SUNPSRpGSgLOB8aSmiR+Dlwm\n6dNuB6iecePg738vvW38eNhgg/rGY2Zm9ZH3qYEJkh4EzgMujoh3enGu14G5wJCi9UOAKd3sczjw\nQkT8oGuFpANIfRW2AsYX7zBy5Mj/vO7o6KCjo6MXobafK6+Ep58uPfLevvvCbrvVPyYzM1tQZ2cn\nnZ2dVT1mxacGACStDRwMfBVYFvgrMCYibuvRyaS7gYdLdBa8PCKOK1H+JGCHiBhRsG5F4CVgu+LH\nDt1ZsPeOOQZWWSX9NDOz1lCNzoK5hhiOiGci4kfAasB/A4sD10t6TtLxklbJeb5TgQMlHSJpfUln\nkPoHnAMgaZSkWwrK/wPYTNJPJK0taTNSM8Fk4IGc5zQzM7Nu9GiugYiYGxHXkp4S+BGwCqnN/nlJ\nl0paucL+lwFHA8eTOv1tDexaMIbAUGBYQfk7gX2APYEJwPXAHOBzETG7J7GbmZnZh+VqGvhPYekT\npCaCfUjjCYzJlhWBXwDLRcQWNYgzb3xuGuglNw2YmbWeWo8jUHii75FGAFwHuAbYD7gxIuZmRSZL\nOhiY1JdgzMzMrL7yPjVwBPBH4IIygwtNBQ6tSlRWFRFw9tnw+uuVy951F/z3f9c+JjMzay55nxpY\nA5gcEfOK1gtYNSIm1yS6HnLTwIJuvx0OOQQOOCBf+f32g/XWq21MZmZWPTWda6DoRHOBFSNiatH6\nwcCrETGwL0FUixOBBe27L2y9NRx5ZKMjMTOzWqhnIjAPGFoiEVgdeDwiBvUliGpxIjDf66/Dxz4G\nzz8Pyy7b6GjMzKwWat5ZUNLvCt7+WtKson23BB7uSwBWGxdeCLvv7iTAzMzKq9RZcOOC1+sD7xW8\nf480qM/J1Q7K+iYCzj0X/vCHRkdiZmbNrmwiEBEdAJIuAL4TEdPqEJP10bhxKRnYdttGR2JmZs0u\n1+ODEXFgjeOwKho9Gg47DNSnViMzM2sH3XYWlHQ1sH9ETMteB1Dq1hIRsUcNY8zNnQXhrbdgzTXh\nmWdg+eUbHY2ZmdVSrTsLvkG6+Re+LpkI9CUAq66LL4bPfc5JgJmZ5dOjuQaaXbvXCETA8OFw6qmw\n446NjsbMzGqtbtMQSzpa0pC+nMhq7777YMYM2GGHRkdiZmatIu80xN8DXpR0g6QDJC1Ry6Csd849\nFw49FAb0aHJpMzNrZ3lHFhwAbA/sD+wNLAJcBVwE3FQ8B0GjtHPTwPTpsNpq8MQTMHRoo6MxM7N6\nqFvTQETMi4jbI+JQYEXgq8DiwN+Al/oSgFXHX/6SmgScBJiZWU/0uBI5It4F7gbGA/8G3HegCXSN\nHWBmZtYTuRMBSUtLOljSrcBk4DDgYmCtWgVn+Tz4IEydCp/5TKMjMTOzVpNrZEFJVwCfB94BLgV+\nGBH31TIwy2/0aDjkEBjYFJNBm5lZK8mVCJAmGNobuDkiPqhhPNYLY8fCo482OgozM2tFHlCoHxgw\nAD74wI8Nmpm1m5oOMSzpGODsiJgt6XuUGUo4Ik7tSxBmZmbWGOUmHXoe2CIi3pA0ifKJwJq1Ca9n\nXCPQ6EjMzKyeqlEj4KaBfsCJgJlZe6rnXANflbRYifWLSPpqXwIwMzOzxsk7xPA8YGhETC1aPxiY\nGhFN8V3UNQKNjsTMzOqppp0Fc1oVeLuPx7AeuvJK+NnP5r/3+AFmZtZbZRMBSYVPp98hqXAMgYHA\n6sB1tQjMuvfkk7DllvCd76T3Sy3l2gAzM+udSjUCV2Y/NwSuAWYWbHsPeL6gjNXRkCGwySaNjsLM\nzFpd2UQgIkYCZI8PXhIRc+oQk5mZmdVJrj4CEXFBjeMwMzOzBig3suB0YM2IeD173Z2IiKWrH5qZ\nmZnVWrkagSOBGQWvzczMrJ/pNhEobA5w00D93XAD/Pznpbe9+CIcdFB94zEzs/4pVx8BSSsAdA0o\nJGkT4EvA4xExtnbhta9HH4U11oBvf7v09g02qGs4ZmbWT+UdUOgy4M/AmGw0wTuAl4HvSFopIk6u\nVYDtbOWVYeutGx2FmZn1Z3mHodkYuCd7/UXg2YjYEPgK8PVaBGZmZma1lzcRWBzoenJgJ+Dq7PWD\nwGrVDsrMzMzqI28i8CzwBUmrAZ8BbsrWr4DnGjAzM2tZeROBkcCJwCTg7oi4O1v/OWBC9cMyMzOz\nesg7suBfs9qAlYCHCjbdAlxRi8DMzMys9nJPQxwRrwCvFK27u5viZmZm1gLyjiMgYB9gR1K/gAFA\nACINMbxHzSI0MzOzmslbI3AScDRwOzCFlAR0iZJ7mJmZWdPLmwh8FdgvIi6vZTBmZmZWX3mfGhhA\nGjPAzMzM+pG8icBo4IBaBmJmZmb1l7dpYBlgf0k7A48A72fruzoLfqcWwbWbo45KMwsCPP007LJL\nY+MxM7P+TxGV+/pJ6ix4W7hDVyKwQ5Xj6hVJkefzNKull4Yzz4QllkjvP/EJWGWVxsZkZmbNSxIR\noT4do5VvnMX6QyLw4ovpp5mZWSXVSATy9hHoOuFgSZ+QtFhfTmpmZmbNIVciIGkpSZcDU4HxpKGG\nkXSOpJG1C8/MzMxqKW+NwInAysBmwOyC9dcAe1c7KDMzM6uPvE8N7AHsHREPSSpshH8SGFb9sMzM\nzKwe8tYILAu8UWL9UsDc6oXTvmbOhPffr1zOzMysmvImAveTagWKfZ3UZ8D6YNYs2H13+PKX/cSA\nmZnVV95E4EfAzyX9EVgY+K6k20lzEBzfkxNK+qak5yXNlnS/pG1y7HO0pCclzZH0sqRRPTlnM5s9\nG/bcE1ZeGf74x0ZHY2Zm7SZXIhAR44GtgUWA50jTEb8EfDIiHsh7Mkn7AKcDvwSGk2oTrpe0apl9\nTgWOAI4F1gN2Ae7Ie85m88478NJLaZk8GfbaC1ZYAS64AAYObHR0ZmbWbuo6oJCke4CHIuLwgnVP\nA1dExI9LlF8XeBTYOCKeynH8ph9QaNVVU1+Arpv+Zz8L554LC+XttmlmZpapxoBCPb79SFoB+Aaw\nJPCPiLgz536LkB4/PKlo002k2oZS9gQmArtKup40pPEdwLER8VpPY28Gc+bA44/D8ss3OhIzM7MK\niYCk0QARcVj2fhBwL2lAodmkvgJ7RMT1Oc41GBgIvFq0fiowtJt9hgGrA18i9UcAOBm4WtJWTf/1\n38zMrMlVqhHYhtQ23+UAYGlgHeDfwBjgf4A8iUBvDAAWBb4SEc8CSPoK8BSwBXBfjc7bI089Bb/7\nXb6y06fXNhYzM7OeqJQIrAI8UfB+J+DKiJgEIOm3wA05z/U6acyBIUXrhwBTutlnCvBBVxKQeTY7\nzmqUSARGjhz5n9cdHR10dHTkDK/3xo2D+++HAw6oXPass2Dw4JqHZGZm/VBnZyednZ1VPWbZzoKS\n3gI+ERFPZ+8nAz+LiPOy92sCj0fE4rlOJt0NPFyis+DlEXFcifI7AzcCH4uIidm6tYBngC0j4v6i\n8g1pLRgzBu68M/00MzOrl3rMPvgk2VwCkjYh1RDcXrB9NT7c5l/OqcCBkg6RtL6kM0j9A87JzjFK\n0i0F5W8BJgBjJA2XtCmpOeLu4iTAzMzMeq5S08CJwGWSdgXWB67r+mae2ZXUeTCXiLhM0kdJgxCt\nSHo0cNeIeCErMpSCuQsiIiTtBvwW+Cepg+JNwDF5z2lmZmbdK5sIRMRVknYBdgeuBYq7xM0GzurJ\nCSPibODsbrYdVGLdK6SnBszMzKzKKo4jEBG3Ard2s21ktQMyMzOz+um2j4CkHk0v3NPy/YlHMzAz\ns1ZVrrPgeEkXSdpRUskeiUp2knQxbTwL4TPPpKGDzczMWk25poF1SYMJjQUWlzQBmAzMAJYiPTGw\nKTAHODcr35Y6O2FUv5kP0czM2knFSYeyOQJ2ArYD1iYlAe+QBvb5F3BLRLxX4zhzacQ4AjNmwNCh\n8NprsHiu0RTMzMyqoy6TDmU3+euyxYqMHw+bbuokwMzMWlOlAYWsgs5OqMMoxmZmZjXhRKCP7rgD\ntt++0VGYmZn1TsU+Aq2k3n0EZs6EFVaAqVNh0KC6ndbMzAyoz1wDVsZdd8Hw4U4CzMysdTkR6AM3\nC5iZWavLnQhI2kTSmZKul7Ritu6/shkB25ITATMza3W5EgFJnwHuA1YGdgS6HpZbCzihNqE1t9mz\n4YEH4FOfanQkZmZmvZe3RuCXwDERsRfwbsH6TuAT1Q6qFdx9N2y8MSy5ZKMjMTMz6728icCGpGmI\ni70JLFe9cFrHHXd4/AAzM2t9FUcWzLwJrAJMKlq/KfBiNQNqVq++CpdeOn+mwSuugN/8prExmZmZ\n9VXeGoGxwEmSuubYW1hSB3AK8OdaBNZsbrkFzjwTJk5Myy67uKOgmZm1vrw1Aj8BzifVCAh4PPt5\nMfCrmkTWhLbYAs44o9FRmJmZVU+uRCCbeGh/ST8FNiPVJDwYEU/XMjgzMzOrrbyPD/5U0hIR8VxE\nXB4Rl0bE05IWz5IDMzMza0F5+wiMBEo9KDco22ZmZmYtqK9DDA8H3qhGIGZmZlZ/ZfsISJpe8Hai\npMKp/QYCiwHn1CIwMzMzq71KnQWPzH6OAX4MTCvY9h4wKSLG1yIwMzMzq72yiUBEXAAgaRIwLiLe\nr0NMZmZmVid5Hx/s7HotaSiwSNH2ydUNy8zMzOohVyIgaRngd8CXgIVJgwl1CVJ/ATMzM2sxeZ8a\nOBn4OLAXMAfYF/gf4AXgy7UJzczMzGot7xDDuwD7RcQ/Jc0FHoiISyVNAb4OXF6zCM3MzKxm8iYC\nH2H+zIPvAB8FngXuBs6rfljN4a67YPLk+a/NzMz6m7yJwHPAMGAy8CSwr6R7gf8iTVHcLx18MKyx\nBiy9dHq/xx4NDcfMzKzq8iYCfyL1EegERgHXAt8m9TE4qiaRNYEIOO00WG+9RkdiZmZWG3kfHzy1\n4PVtktYDtgCeiYhHahWcmZmZ1Vav5hqIiH9HxJXAE5K+VuWYzMzMrE4qJgKSBknaTNLSBeuWkvQ/\nwERgdC0DNDMzs9opmwhI+gRprID7geclbSzpK8DzwPHApcBaNY/SzMzMaqJSH4FfAA8CpwP7AVcA\nqwO/Bk6LiOll9m05s2bBHXekToIAM2Y0Nh4zM7Naq5QIDAd2iYgHJP2L9Kjg4RHRL5sDrr8evvUt\n2Gyz9H6LLWDIkMbGZGZmVkuVEoHBwIsAEfG2pFnAHTWPqkHmzYNtt4XLPU6imZm1iTxPDUTR6w9q\nFIuZmZnVWZ5xBCZK6koGBgGPFLwHiIhYusR+TW/mTDj8cJg0Kb1//XUYPryhIZmZmdVVpUTg4LpE\n0QCzZsFuu8Fqq8H//u/89cOGNS4mMzOzelNEVC7VIiRFns8zaxbsvjussgqMGQMDB9YhODMzsyqT\nRESoL8fo1ciCrWzOHNhzT1hpJScBZmZmeScd6jduvRXefhtuuMFJgJmZWdvVCMydCyuu6CTAzMwM\n2jARMDMzs/mcCJiZmbWxXImAkm9JekzSbEnDsvU/lPSl2oZoZmZmtZK3RuAo0myDxXMMvAx8u6oR\nmZmZWd3kTQSOAA6LiNNZcIjhCcBGVY/KzMzM6iJvIrAa8GiJ9e8Di1cvHDMzM6unvInA88DmJdbv\nAjxevXDMzMysnvIOKPQb4PeSFiclD1tL+irwffrxfARmZmb9Xa5EICLOl7QQMIrUFPBnUkfBIyPi\nkhrGZ2ZmZjWUe4jhiBgNjJa0PDAgIl6tXVhmZmZWD3nHEThD0hYAEfFaX5IASd+U9Hw2HsH9krbJ\nud/akqZLmt7bc5uZmdmC8nYWHAHcK+kJScdLWqM3J5O0D3A68EtgODAeuF7SqhX2WwS4BLgD6D/z\nJpuZmTVYrkQgIrYGPgZcDOwPTJR0p6RvSFq2B+c7Bjg/Is6LiKci4jvAFNI4BeWcCDwEXA70ad5l\nMzMzmy/3XAMRMTEifhkR6wNbAPeQRhuckmf/7Fv9ZsBNRZtuArYus9/ngc8DR+IkwMzMrKpydxYs\nski2LMqCIw2WMxgYCBT3L5gKDC21g6SVgHOBvSJiluQ8wMzMrJpyJwKS1iU1C+wHrA7cRqrq/2tt\nQgPgQuDsiLivLwf5/e/hgQfS68mTYdCgKkRmZmbWD+RKBCTdT6rWfxA4E7gkInI1CRR4HZgLDCla\nP4Tumxd2ALaTdEJXKMAASe8DR0TEH4t3GDly5H9ed3R00NHRwZ/+BLvsAmuumdZ//OM9jNzMzKwJ\ndHZ20tnZWdVjKqJyJ3xJvwYujIgn+nQy6W7g4Yg4vGDd08DlEXFcifIbFK3aCziO9BTDyxHxdlH5\nKPV5RoyAs85KP83MzPoLSUREn9rN844s+OO+nKTAqcCFku4lPTr4DVL/gHMAJI0CRkTETtl5F5jH\nQNKWwLzi9WZmZtY73SYCkn4L/CgiZkr6HaWf3xcQ2WOAFUXEZZI+SnraYEXSjIa7RsQLWZGhwLBK\nh8lzLjMzM6usXI3AJsDC2euNKZMI9OSEEXE2cHY32w6qsO8FwAU9OZ+ZmZl1r9tEICI6Sr02MzOz\n/iPvXAM/lbREifWLS/pp9cMyMzOzesg7suBIYMkS6wdl28zMzKwF5R5iuBvDgTeqEYiZmZnVX9nH\nB4um/J0oqbBj4EBgMbJH/5rNzJnQNaTA3LmNjcXMzKxZVRpH4Mjs5xjgx8C0gm3vAZMiYnwtAuuL\nO++E7beHxRdP7xdaCD7ykcbGZGZm1ozKJgLZ43pImgSMi4j36xBTn82YATvvDDfc0OhIzMzMmlu5\nAYWWi4g3s7ePAEt1N/tfQTkzMzNrIeVqBF6XNDQippImDOpOkPoLmJmZWYsplwh8Gnir4LWZmZn1\nM+VGFuws9drMzMz6j7wjC24oab2C95+RdLGkH0tys4CZmVmLyjug0BjS4EFIWhW4ClgW+Cbwq9qE\nZmZmZrWWNxFYF5iQvf4icE9E7Ap8Bdi3FoGZmZlZ7eVNBAYCXWMI7Ahcn72eCAypdlBmZmZWH3kT\ngceAIyRtR0oEuobqWYnyjxaamZlZE8ubCHwfOAzoBP4SEY9k6/cE7qlBXGZmZlYHleYaACAi/ilp\neWDpolEE/wDMrElkZmZmVnO5EgGAiPhA0ixJG5FGE3wuIp6vXWhmZmZWa3nHEVhY0snA26R5Bx4F\n3pb0G0kL1zJAMzMzq528NQInkh4TPBwYl63bBhhFSia+V/3QzMzMrNbyJgL7AYdExLUF656V9Bpw\nHk4EzMzMWlLepwaWAZ4tsX4i8JHqhWNmZmb1lDcReAQ4qnCFJAHfAR6qdlBmZmZWH3mbBo4Frpe0\nI3A3IOCTpAGFdqlRbGZmZlZjuWoEIuKfwDrAFcBSwCDgMmCdiPhX7cIzMzOzWqpYIyBpDWAnYFFg\nbEQ8VuOYzMzMrE7KJgLZ3ALXAUtkqz6QdGBEjK15ZGZmZlZzlZoGfgHcBqwMDAbGkMYUMDMzs36g\nUiKwMXBcREzJ5hj4HrCSpGVrH5qZmZnVWqVE4CPAq11vImImMAuPHWBmZtYv5Hl88OOS3shei5Q8\nbFxYKxARE2oRnJmZmdVWnkTgxhLrrip4HcDA6oRjZmZm9VQpERhWlyjMzMysIcomAhExqU5xmJmZ\nWQPknWvAzMzM+iEnAmZmZm3MiYCZmVkbcyJgZmbWxnqUCEgaLOkTkharVUBmZmZWP7kSAUlLSboc\nmAqMB1bK1p8jaWTtwjMzM7NaylsjcCJp4qHNgNkF668B9q52UGZmZlYfeUYWBNgD2DsiHpIUBeuf\nxIMOmZmZtay8NQLLAm+UWL8UMLd64ZiZmVk95U0E7ifVChT7OqnPgJmZmbWgvE0DPwJulLQhsDDw\nXUkbAVtI8/RQAAAUm0lEQVQC29UqODMzM6utXDUCETEe2BpYBHgO2BF4CfhkRDxQu/DMzMyslvLW\nCBARjwJfrWEsZmZmVme5EgFJy5XbHhFvViccMzMzq6e8NQKvl9kWwMAqxGJmZmZ1ljcR+HTR+4WB\n4cA3geOrGpGZmZnVTa5EICI6S6y+WdJE4FDg4moGZWZmZvXR19kHHwa2r0YgZmZmVn+9TgQkLQUc\nDbxQvXDMzMysnvI+NTC9eBWwBDAT2L/aQZmZmVl95O0seGTR+3nAa8DdEfFWdUMyMzOzeqmYCEha\nCBgEXBURL1XjpJK+CRwLDAUeA46OiDu7KdsBfBcYASwDPAucHhHnVyMWMzOzdlaxj0BEfAD8hh6M\nQliOpH2A04Ffkh5BHA9cL2nVbnbZitQp8QvAhsDZwLmS9q1GPGZmZu0sb2fBu4HNq3TOY4DzI+K8\niHgqIr4DTAGOKFU4IkZFxE8j4q6ImBQR5wB/JSUGZmZm1gd5v+WfC5wiaXXSlMQzCzdGxIQ8B5G0\nCLAZcFLRpptIkxrltQwwuQflzczMrISyiYCkMaRHBMdmq04pUawnQwwPzsq+WrR+Kqm/QEWSdiON\ndNiTxMHMzMxKqFQjcCDwQ2BY7UOpTNKnSKMYHhkR9zc6HjMzs1aXd4jhSVU63+vAXGBI0fohpH4C\n3ZK0DXAt8JOI+EN35UaOHMmzz8Kzz0JnZwcdHR19DNnMzKw5dHZ20tnZWdVjKiK63yjNA4ZGxNSq\nnVC6G3g4Ig4vWPc0cHlEHNfNPtsB1wA/jYjTyxw7IoIbboDTT4cbbqhW1GZmZs1HEhGhvhwjT43A\nK1LZc0RE9GQa4lOBCyXdS3p08Buk/gHnAEgaBYyIiJ2y9x2kmoDfA3+R1NWXYG5EvNaD85qZmVmR\nPInAYcA71TphRFwm6aOk6YtXBB4Fdo2IrjkLhrJgn4SvAYuRBiA6tmD9JJqk74KZmVmrqnvTQC25\nacDMzNpJNZoG+joNsZmZmbUwJwJmZmZtrGwfgYhwomBmZtaP+UZvZmbWxvpdIjB1Krz9dqOjMDMz\naw1VmVq4mWy0Ufr5xS82Ng4zM7NWUPbxwVbT9figmZlZO/Djg2ZmZtYnTgTMzMzamBMBMzOzNuZE\nwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzM\nrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05\nETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAz\nM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tj\nTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTMzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjTgTM\nzMzamBMBMzOzNuZEwMzMrI05ETAzM2tjdU8EJH1T0vOSZku6X9I2FcpvLOkOSbMkvSjpJ/WK1czM\nrL+rayIgaR/gdOCXwHBgPHC9pFW7Kb80cDMwBdgCOAo4VtIx9YnYzMysf6t3jcAxwPkRcV5EPBUR\n3yHd5I/opvz+wGLA1yLi8Yi4EjgxO441QGdnZ6ND6Pd8jevD17n2fI1bQ90SAUmLAJsBNxVtugnY\nupvdtgL+FRHvFpVfSdLq1Y/SKvF/7NrzNa4PX+fa8zVuDfWsERgMDAReLVo/FRjazT5DS5R/tWCb\nmZmZ9UGzPzUQjQ7AzMysP1NEfe61WdPATODLWVt/1/ozgQ0iYocS+/wJ+GhE7FawbgRwD7BmRPy7\nqLwTBzMzaysRob7sv1C1AqkkIt6T9ADwGeDKgk07A5d3s9tdwImSFi3oJ7Az8FJxEpCdo08Xw8zM\nrN3Uu2ngVOBASYdIWl/SGaS2/nMAJI2SdEtB+bHALOACSRtK2hv4QXYcMzMz66O61QgARMRlkj4K\nHA+sCDwK7BoRL2RFhgLDCspPk7QzcCZwP/AmcHJEnFbPuM3MzPqruvURMDMzs+bT7E8NLMDDE9de\nT66xpA5Jf5f0sqSZkh6WdFA9421FPf09LthvbUnTJU2vdYytrjfXWNLRkp6UNCf7nR5Vj1hbWS/+\nJu8q6W5J0yS9JukqSWvXK95WImk7Sf/I7l3zJH0txz69uue1TCLg4Ylrr6fXmDTg08PAF4ANgbOB\ncyXtW4dwW1IvrnHXfosAlwB34Mdqy+rNNZZ0KmmE02OB9YBdSNfautGLv8kfA64COrPyO5FGjr2u\nHvG2oEHAI6R712wq/L/v0z0vIlpiIT0y+IeidU8Dv+6m/BHA28CiBeuOA15s9Gdp1qWn17ibY1wK\nXNHoz9KsS2+vMXAacB7wNWB6oz9HMy+9+FuxLvAesG6jY2+lpRfX+YvAB2RN0tm6HYB5wHKN/jzN\nvADTga9WKNPre15L1Ah4eOLa6+U1LmUZUqdOK9Lbayzp88DngSMBPyJbRi+v8Z7ARGBXSROzqu4L\nJC1fw1BbWi+v8zhgBnCYpIGSlgIOBO6NCP/N6Lte3/NaIhHAwxPXQ2+u8QIk7QZ8Gji3uqH1Gz2+\nxpJWIl3P/SNiVm3D6xd683s8DFgd+BLwVeArpOaBqyU58Sqtx9c5IqYAu5KaEuaQvr1uCOxeuzDb\nSq/vea2SCPSG21HrSNKngIuBIyPi/kbH049cCJwdEfc1OpB+bACwKPCViLgzIu4kJQNbktparQok\nDSP1ETifdF07SFXelznhqope3/NaJRF4HZgLDClaP4TUMaKUV/hwFjSkYJstqDfXGICsp/B1wE8i\n4g+1Ca9f6M013gE4QdL7kt4H/ggMyt4fWrtQW1ZvrvEU4IOIeLZg3bPZcVareoT9Q2+u8+HACxHx\ng4h4OCL+BRwAbE+q1ra+6fU9ryUSgYh4D+ganrjQzqSeqqXcBWwradGi8iWHJ253vbzGSNqOlASc\nEBG/rV2Era+X13gj4OMFy09JPYg/DlxRm0hbVy+v8Z3AQtk31i7DSFXf/ltRQi+vs0gdAwt1vW+J\ne1GT6/09r9G9IXvQa/JLwLvAIcD6wBnANGDVbPso4JaC8kuTMtO/kNqh9gbeAb7b6M/SrEsvrnEH\naSKpE0mZ59BsWb7Rn6VZl55e4xL7H4ifGqjqNSbdoO5n/mNtm5IeHRzf6M/SzEsvrvM2pFqEnwBr\nkzob3gBMAhZv9OdptoX0+ODwbJmZXbfhtbjnNfzD9vDCHAE8T+poch+wTcG284GJReU3yv5DzwZe\nIlVdN/xzNPPSk2ucvZ9LyuoLl4n1jruVlp7+HhfteyAwrdGfodmXXvytGApclt3IXiX1zXBCW/3r\n/MUs6ZqeXeergPUa/TmacSF90er6m1r4d3ZMmevbq3uehxg2MzNrY26XMTMza2NOBMzMzNqYEwEz\nM7M25kTAzMysjTkRMDMza2NOBMzMzNqYEwEzM7M25kTAWoKkDknzJC3X6Fh6K5ve9pgKZQ6UNL1e\nMTUbSedJGtnoOOop+73eu+D9epLukjRb0sRSZSocryq/Q5KulHRUX49jzc+JgNVNNsf7vBLLJo2O\nDUBSZ0FMcyQ9JelHkqr1/2QL4OyC85X6434JsGaVztetous/Q9Jjkr7by+PkukHlONa6wBeA0wrW\n7S3pRklTs3NtX41zZcfeXtKtkl6TNFPSs5IukrRUtc6R01DgmoL3vwRmAOsCI7opU84Cv0OSRkp6\ntBdx/Qr4saQlerGvtRAnAlZPAdzM/DkJupbHGhlUgQDGkGJaB/gt6Y/y96py8Ig3ImJ20WoVlZkT\nEa9X43w5HEr6rBsDFwGnSNq/F8ep1hSy3wSujoh3CtYtQZoUqKsmpSpDoUragDTO/UOkoVw3BL4B\nvE2akrhuImJqpEl8unwMGBcRkyPijW7KlDteVX6HImIC8BppTgHrzxo9nrKX9lmAC4B/dLPtGOBh\n0jehF4HRwDIF2ztI42wvl71fhjQe/KukcbWfA44qKL8McG62fRppQpnNK8R3O/DbonU3kv4oAywL\n/Al4E5hFSmo2KDpnuZgmAccUvP7Q/AwUTCpESkbmARsVxfR10h/ogdn7DYBrmT9O/lhgSIXPOg/Y\nu2jd64Wfn/Rt9KbsXO8A/wI+WfR5Ss4xAexOmp1uNjCRlFAtXCGmV4D/7mbb4Owc21Xpd/Fo0pS4\n5cp0/c59npQwzCaNk79ZUbmtSeO7z8x+d88Clioq8z3gGdKY/C8Avy71b8GH5+34aal/L2Al4OLs\n32wm8CDQUeJ36MASx/waKeG9uijGAcBk4OiCdb8CbqzH3wcvjVtcI2D11t23x7nAUaSb2n7AlsDv\nyhznl6QJNj5PumEeTJpkA0ki3RhXzLYPB/4J3CapeL7uSt4FFsleX0C6Oe6RxTcLuEHSYpViyhR+\nm90i+9n1rXwERSLiadJELsXf0vcHLo2IuZJWzD7bI9kxdgSWBP6eXYdyBCBpoKQvAcuRbnRdliQl\nPttkx34IuK6gn0bJzyDps6Qaht+S/j0PJk028+tuA5HWA1bIPm89TAGWl9SRo+zJwLGkzzsRuEbS\n4gCSNiYli1cBm5BmfBtOutGSlRkFHE+6qa6fleluWtgVgaeycw7Nfi5A0iBS4rEasCepNuOEbo53\nCXBKdsyuGrhLSUny54r+P+xMmkX0woJ19wGfqmLzmDWjRmciXtpnId1I3yfNPNa1XNtN2c8Bcwre\nd7BgjcDfgfO62ffT2bEXK1r/IHBsmfhuB36XvR7QFQNpus+1s/MXzq62NKkq+eBKMWXbnyerEcje\nl/pWfiAF0wwDRwKTCt6vRkqaPpm9/zlF0xaTai7mASPKxDKPlMhMz/5N3ge+VeHfT8DLwP4VPsM/\ngeOK1u1FmemTSTUI84AB3Wyvdo3AANLNeh6pJuIfwHeBwSV+5/YtWDcIeAs4JHv/Z+CPRccenu03\nmJRMzQa+XuHfovDb/qNkNQGlygCHkWp/luvmeMW/QyOBR0uUexT4QcH7S4HLispslp171Wpcdy/N\nuTjLs3q7A/h4wXIogKRPS7pZ0guSpgFXAguX+QZ/NrCPpIck/UbSdgXbNie1Lb8maXrXQvrmNKxM\nbAK+npWdTbqxXwj8jPRNbh5wV1fhiJhG+mO6QY6YeutSYCVJ22bv9yVVwd9d8Fm3K/qck0m1D+U+\nK8D/kP4Ndib109ircKOkFST9Ies0+Tbp5rMCsGqF424OHF8U08XAEpKGdLPP0sC7ETGvwrHLkrRa\n1vmx69w/LFUuIuZFxMHAKqTrMJn0rf/JrP9AocJ/85mkf/P1s1WbAwcUfdY7Sdd/LdLvxqLArX35\nXEU2BR6OiDf7eJzRwEEAWS3PHsB5RWWmZT+X6eO5rIkt1OgArO3MjoiJhSskrU6qyv8DqQr1DdIf\n2L8wv1p+ARFxQ7bfLqTq8GslXZ79cR9AaivfpsSu00qs+89hSVWpPyM1Cbwckb4WlallV7ZfpZh6\nJSKmSrqZ1Bzwr+znxUXnv4Z0Mys2tcLhX8n+LSZK+gLwhKT9ImJstv1PwPKk9vRJwHukG1rJf5Oi\nmEYCl5fY1l0ntneARSUN6GMy8BKpir7LW+UKR8TLpGaMiyQdDzxNSggOKrObil6PpuBJhwIvF8VS\nTdXooHkRcKKkT5G++U+NiBuLyiyd/Xy7CuezJuVEwJrBFsDCwHcLbrx7VNopUo/qrj/iNwBjJR0O\nTCC1dUZEPN/DWN4pTlQyT5ASjK1JN2QkLU3qE/Cfb1HdxRQR75c45vvAwBwxXQT8XtLo7HyFj+tN\nIPXqnhwRH+Q4VkkR8Zyki4EfkTobAnwKODIirgfIvs2vmOMzTADW7+Y6dufZ7OdqpKSjVyJiLqkd\nvzf7vi3pFVL1f6GtumLK2uc3JDVzQfqsG3X3WSU9QUoqdyJ1Hq2GCaRaiI9mv2+VvEeJ37OIeFPS\nX4FDSM0Zfyqx7+qkJqSX+xCvNTk3DVgzeJr0u/hdSWtK2pfUcbBbkn4uaU9Ja0vq6oD1XES8HxE3\nA+NIHeY+lx1zK0k/k1SqluA/h6Wbb1oR8QypqeAPkrbJOoldRPomO7ZSTN2cbxKwk6ShkpYtE9dV\npETpPODeiHi2YNuZpGrbSyVtKWmYpJ2yKv0lyxyzlFOBDSXtmr1/GviKpPUljSDVlhQ/wlbqM/wc\n2C+73hspDZDzRUknljn3U6SnE7YsXClpWUnDSQkQwNqShpdpYshF0uGSzpK0s6S1JG2YxbcR8Lei\n4sdl13RDUr+Cd5mfLJ0IbCnpbEmbSvqYpN0knQMQEdOBM4BRSgP9rJX9O32jXHgVwh9Lqu35e/a7\nOEzSHuq+4+PzwOpZfIMlFdbojCbVMm1CQQfHAluSnprpU5ONNTcnAlZPQYnnwCPiUdKN/xhSW/XB\npKru4rKF7+eQemE/RGqTHUTqcNZlV+A20h+6J0lt7WuzYC/+XPEVOAi4l9Sx7B5gMeBzEfFuzpiK\nfQ/YgdQ+/UBRHPPfpLEH/sb85/0Lt00hfXOfR3ou/v+A32exvEsPZP8ON5OqxiH9OyyZxTYW+CMf\n/rb+oc8QETeRnpzYgXSd7gG+T/c95clqgi4htVMX2pP0Dfg20nUZnb0/vCefrYR7SP1Izia1+d8B\nbAt8JSL+UlT2h6Se9w+Q2v13y/5Nuq7ZdsAapEdUHyI9HfFKwf4/IiUMPwEeB64AVi4TW9mxEiJi\nFrA96VHFq7P4TyD9DpQ6xpXAdaRmnanAlwuO1Ul6nLEzIiaVON3upCY668eU1cSamTWU0siC9wBr\nRETD26Szb9i3kZ4k6GvHvKak9Bjki8C3ixMgSZsD15P+PWY1Ij6rD9cImFlTiIinSN+WPb59jSlZ\ngdQ5dxZwWYliPwZ+6SSg/3NnQTNrGhFxaKNjKNJfq0xXJ3WqfAE4KOtkuYCI+ELdo7KGcNOAmZlZ\nG3PTgJmZWRtzImBmZtbGnAiYmZm1MScCZmZmbcyJgJmZWRtzImBmZtbG/h9xF6DRzpLZmwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18be45f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot ROC curve\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.xlabel('False Positive Rate (1 - Specificity)')\n",
    "plt.ylabel('True Positive Rate (Sensitivity)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.838692434211\n"
     ]
    }
   ],
   "source": [
    "# calculate AUC\n",
    "print metrics.roc_auc_score(y_test, y_pred_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides allowing you to calculate AUC, seeing the ROC curve can help you to choose a threshold that **balances sensitivity and specificity** in a way that makes sense for the particular context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x0000000018BE8390>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x0000000018EA0198>], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAF9CAYAAAD2nEqnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGqJJREFUeJzt3X+Q5HV95/HnGyhYRJLg4c1KKjAQPA0xJ3qHp9wirXXZ\nUy5V510umnBJVhTXGMiGUKmjxAVZzwsWpCIm4bTwvNMih/lZUFSMBrzQcmH5KcQkcCgUOxzLsrsm\n8RZkQXH3fX98vwPNOLvdM/3t+X7n089HVdd0f7/94zU98/28ur/97e83MhNJklSGQ9oOIEmSmmOx\nS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqiMU+ZSLiiIj4WEQ8HhF7I+LOiFjfdi5JKy8i\njoqILRHxZxHxzYjYHxEXtZ1L47HYp89ngQuB/wlsAp4DvhARb24zlKRWvBy4BPhx4N56mnstW+UO\nazuAVk5EvAF4F3BRZl5ZT7sW+FvgSuBftBhP0srbARyXmTsj4gRgW9uBND7fsU+X/wDsB66Zn5CZ\n3wE+A5xWL9iSpkRmfjczd9YXo9UwaozFPl1eBzycmXsWTL+7/nnqCueRJDXMYp8urwCeWGT6/LTj\nVjCLJGkCLPbpciTwnUWmPzswX5K0ilns0+UZ4IhFpq8ZmC9JWsUs9unyBIuvbn9F/XPHCmaRJE2A\nxT5d7gN+NCJ+aMH0+a+5/dUK55EkNcxiny5/TPU33zg/ISKOAM4B7snMR9sKJklqhjuomSKZeVdE\n/BHw0Yg4FngY+EXgBODcVsNJakVEnA/8UH0CeGtEHF6f/+3MfLKdZFquyBy+98CIeAXwMeDtwNHA\nI8AHMvPWgetcBrwPOAa4EzgvMx+YQGaNoX6H/hHg54GXAX8DXJKZf95qMEmtiIhtVC/u4YXdyUZ9\n/sTM/L+tBNOyDS32+vPYe4Fbgd8FvgmcBDyRmQ/W17kI+BCwAfgGcCmwDnhVZn57YuklSdKLjFLs\nvwGckZlnHGB+UG1N/duZeXk9bQ2wG/j1zLxmsdtJkqTmjbLx3DuAuyLiDyJiV0TcFxHnDcw/EZgB\nbpqfkJnPUr3DP73RtJIk6aBGKfaTgF+m2tBqPfAJ4GMD5b62/rlrwe12D8yTJEkrYJSt4g8B7srM\nD9WXvxYRrwTOA64ectsXreePCI/zKw2RmVNzlC3HBOngljMejPKOfQewcOv2B4Hj6/Pzh/ybWXCd\nmYF5z8vMzp4+/OEPt55hNedbDRm7nm8atf2cr+b/F/OVnW+5Rin224BXL5j2T4C5+vw2qgJfPz+z\n3nhuHbB12ckkSdKSjVLsHwfeGBEXR8TJEfEzwK9Qr4bP6mXFVcBFEfHvIuI1wGeBp4DrJhNbkiQt\nZuhn7Jl5T0S8A/gN4BLgUWBzZn5y4DpXRMSRVGV/DHAHsD4zn55M7Mno9XptRziorueD7mfsej51\nS9f/X8w3nq7nW66R9jzX2INF5Eo+nrTaRAQ5ZRvPOSZIi1vueOBBYCRJKojFLklSQSx2SZIKYrFL\nklQQi12SpIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSCWOySJBXE\nYpckqSAWuyRJBbHYJUkqiMUuSVJBLHZJkgpisUuSVBCLXZKkgljskiQVxGKXJKkgFrskSQWx2CVJ\nKojFLklSQSx2SZIKYrFLklQQi12SpIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKx\nS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqiMUuSVJBDmvzwT/3uWu54Yabx76fQw6BTZve\ny5lnntlAKkmSVq+hxR4RlwGXLpi8MzOPW3Cd9wHHAHcC52XmA8Pu+7bb7uaGG/YD65cQ+fsdfvh/\n46yzHrbYJUlTb9R37A8CvYHL++bPRMRFwIXABuAbVC8Cbo6IV2Xmt4ff9RuAXxwxxuIOO+wrY91e\nkqRSjFrs+zJz98KJERHABcDlmXl9PW0DsBs4G7imqaCSJGm4UTeeOykiHo+IRyLi8xFxYj39RGAG\nuGn+ipn5LHArcHqzUSVJ0jCjFPsdVKvZ/zXV5+hrga0R8bL6PMCuBbfZPTBPkiStkKGr4jPzSwMX\n/zYibge2UZX9nQe76WITe70es7OzzM7OsmPHduDkpeSVitLv9+n3+8zNzTE3N9d2nFYMjgm9Xo9e\nr9d2JKkVTY0HS/66W2bujYj7qRr5hnryDLB94GozwM7Fbt/v958/v3HjpqU+vFSUhUVWbbYyXQbH\nBGmaNTUeLHkHNRGxBvgx4InM3EZV4OsXzF8HbF1WIkmStGyjfI/9N4EbgceAfwxcAhwJfK6+ylXA\nxRHxIPAQsBl4CrhuEoElSdKBjbIq/oeBzwPHAt8EbgfemJmPAWTmFRFxJHA11Q5q7gDWZ+bTk4ks\nSZIOZJSN535uhOtsAbY0kkiSJC2bB4GRJKkgFrskSQWx2CVJKojFLklSQSx2SZIKYrFLklQQi12S\npIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSCWOySJBXEYpckqSAW\nuyRJBbHYJUkqiMUuSVJBLHZJkgpisUuSVBCLXZKkgljskiQVxGKXJKkgFrskSQWx2CVJKojFLklS\nQSx2SZIKYrFLklQQi12SpIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItd\nkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqyJKKPSI+GBH7I+J3Fky/LCIej4i9EXFLRJzSbExJkjSK\nkYs9It4IvA/4ayAHpl8EXAicD5wG7AZujoiXNhtVkiQNM1KxR8QPAr8HnAN8a2B6ABcAl2fm9Zl5\nP7ABOBo4u/m4kiTpYEZ9x34N8EeZ+RUgBqafCMwAN81PyMxngVuB05sKKUmSRnPYsCtExPuAk3jh\nHXgOzF5b/9y14Ga7gePGTidJkpbkoMUeEa8C/guwLjP3zU/mxe/aDyQXm9jr9ZidnWV2dpYdO7YD\nJy8lr1SUfr9Pv99nbm6Oubm5tuO0YnBM6PV69Hq9tiNJrWhqPBj2jv1NwLHA/dXH6QAcCpwREe8H\nXlNPmwG2D9xuBti52B32+/3nz2/cuGnJgaWSLCyygeVsagyOCdI0a2o8GPYZ+/VU5f3a+nQqcA/w\n+fr8Q1QFvn4gyBpgHbB1WYkkSdKyHfQde2buAfYMTouIvcC3MvOB+vJVwMUR8SBV0W8GngKum0hi\nSZJ0QEM3nltEMvD5eWZeERFHAlcDxwB3AOsz8+lmIkqSpFEtudgz8y2LTNsCbGkkkSRJWjb3FS9J\nUkEsdkmSCmKxS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqiMUuSVJBLHZJkgpisUuSVBCL\nXZKkgljskiQVxGKXJKkgFrskSQWx2CVJKojFLklSQSx2SZIKYrFLklQQi12SpIJY7JIkFcRilySp\nIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqiMUu\nSVJBLHZJkgpisUuSVBCLXZKkgljskiQVxGKXJKkgFrskSQWx2CVJKojFLklSQSx2SZIKYrFLklSQ\nocUeEedFxNciYk992hoRZy24zmUR8XhE7I2IWyLilMlFliRJBzLKO/bHgP8EvA74Z8BfADdExGsB\nIuIi4ELgfOA0YDdwc0S8dCKJJUnSAQ0t9sy8MTP/PDMfycyHM3Mz8BTwhogI4ALg8sy8PjPvBzYA\nRwNnTzS5JEn6Pkv6jD0iDo2InwXWALcCJwIzwE3z18nMZ+t5pzeYU5IkjeCwUa4UET8B3A4cATwD\nvDMzvx4R8+W9a8FNdgPHNZZSkiSNZKRiBx4E/inwg8DPAL8fEW8ZcptcbGKv12N2dpbZ2Vl27NgO\nnDxyWKk0/X6ffr/P3Nwcc3NzbcdpxeCY0Ov16PV6bUeSWtHUeDBSsWfmc8Aj9cX7IuI04DzgI/W0\nGWD7wE1mgJ2L3Ve/33/+/MaNm5aWVirMwiKrNluZLoNjgjTNmhoPlvs99kOBQzJzG1WBrx8IsgZY\nB2xd5n1LkqRlGvqOPSI+Bvwp1Tvy+a3dzwTeVl/lKuDiiHgQeAiY32r+ukkEliRJBzbKqvgZ4PeA\ntcAe4GvA2zLzZoDMvCIijgSuBo4B7gDWZ+bTk4ksSZIOZGixZ+Y5I1xnC7ClkUSSJGnZ3Fe8JEkF\nsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqSDHFfu655xIR\njZwkSVqtRjoe++qRDdyHxS5JWr2KeccuSZIsdkmSimKxS5JUEItdkqSCWOySJBXEYpckqSAWuyRJ\nBbHYJUkqiMUuSVJBLHZJkgpisUuSVBCLXZKkgljskiQVxGKXJKkgFrskSQWx2CVJKojFLklSQSx2\nSZIKYrFLklQQi12SpIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSC\nWOySJBXEYpckqSAWuyRJBRla7BHxwYi4OyL2RMTuiLgxIn58ketdFhGPR8TeiLglIk6ZTGRJktoX\nEY2dmjTKO/Yzgd8F3gS8Ffge8OWIOGbgl7sIuBA4HzgN2A3cHBEvbTStJEmdkg2cmnXY0MiZbxu8\nHBG/AOwBTge+ENVLjQuAyzPz+vo6G6jK/WzgmqZDS5KkxS3nM/YfqG/3rfryicAMcNP8FTLzWeBW\nqvKXJEkrZDnF/gngPuD2+vLa+ueuBdfbPTBPkiStgKGr4gdFxG9RvQtfl5mjfDDwfdfp9XrMzs4y\nOzvLjh3bgZOXEkEqSr/fp9/vMzc3x9zcXNtxWjE4JvR6PXq9XtuRpFY0NR7EaP0MEfFx4J3AWzLz\nGwPTTwIeBk7LzK8OTP8CsDszzxmY9qLXAxs3buLTnz4Z2LTsXwDgJS95L3v3/nea2QghGPU5kZoW\nEWRms5vIdtjCMUFaTapNzCbXO8sdD0ZaFR8RnwDeBbx1sNRr24CdwPqB668B1gFblxpIkiQt39BV\n8RFxNfDzwDuAPREx/7n5U5n5dGZmRFwFXBwRDwIPAZuBp4DrJpRbkiQtYpTP2D9Ata7hfy2Yfhnw\nEYDMvCIijgSuBo4B7gDWZ+bTzUWVJEnDjPI99pFW12fmFmDL2IkkSdKyLWmr+GnR1O793ChIkrTS\nLPZFNbOVoyRJK82ju0mSVBCLXZKkgljskiQVxGKXJKkgFrskSQWx2CVJKojFLklSQSx2SZIK4g5q\nJElTo6k9i3aZxS5JmjJN7e67my8SXBUvSVJBLHZJkgpisUuSVBCLXZKkgljskiQVxGKXJKkgFrsk\nSQWx2CVJKojFLklSQSx2SZIKYrFLklQQi12SpIJY7JIkFcRilySpIBa7JEkFsdglSSqIxS5JUkEs\ndkmSCmKxS5JUEItdkqSCWOySJBXEYpckqSAWuyRJBbHYJUkqyGFtB5CkxTz33HPs27evsfs74ogj\niIjG7k/qKotdUidt3nwpV155JYccMv4wtW/fd3j00Uc5/vjjG0gmdZvFLqmT9u+HzP/Mvn0fHPu+\njjrKQtf08DN2SZIKYrFLklQQi12SpIJY7JIkFWRosUfEmyPixojYHhH7I2LDIte5LCIej4i9EXFL\nRJwymbiSJOlgRnnHfhTw18CvAs8AOTgzIi4CLgTOB04DdgM3R8RLm40qSZKGGVrsmfnFzNycmX8C\n7B+cF9XeHi4ALs/M6zPzfmADcDRw9iQCS5KkAxv3M/YTgRngpvkJmfkscCtw+pj3LUmSlmjcYl9b\n/9y1YPrugXmSJGmFTHLPc7nYxF6vx+zsLLOzs+zYsR04eYIRpG7r9/v0+33m5uaYm5trO04rBseE\nXq9Hr9drO5Ia0uS++TMXrZSiNDUejFvsO+ufM8D2gekzA/NepN/vP39+48ZNYz68tLotLLJpPEjJ\n4JigEjVRyNOxXDQ1Hoy7Kn4bVYGvHwiyBlgHbB3zviVJ0hINfcceEUcBr6wvHgKcEBGnAn+fmY9F\nxFXAxRHxIPAQsBl4CrhuQpklSdIBjLIq/jTgL+rzCWypT58F3pOZV0TEkcDVwDHAHcD6zHy6+biS\nJOlghhZ7ZvYZsso+M+fLXpIktcjjsUtSAUrfAn0aNyxdLotdkopR8hboTb3Y6Orv1xyP7iZJUkEs\ndkmSCmKxS5JUED9jn6CmNvbo4oYsksrlhmqrm8U+USVvyCKpXG6otpq5Kl6SpIJY7JIkFcRilySp\nIH7GriUpfe9WkrTaWexaBjcKlKSuclW8JEkFsdglSSqIxS5JUkEsdkmSCmKxS5JUEItdkqSCWOyS\nJBXEYpckqSAWuyRJBXHPc6uAx3WXJI3KYl8V3IWrJGk0roqXJKkgFrskSQWx2CVJKojFLklSQdx4\nboo0tXW9tBqdcMIJjd2X3zBRl1nsU8Wt6zXtXAZUPlfFS5JUEItdkqSCWOySJBXEz9jVmq5tzOcG\nUVppXVsGVAaLXS1qakMmN4jSatbUC0r/h1VxVbwkSQWx2CVJKojFLklSQfyMXZKWyI3e1GUWuyQt\nmRu8qbtcFS9JUkEsdkmSCmKxS5JUEItdkqSCNFbsEfHLEbEtIp6JiHsiYl1T9y2thIho5CRJbWqk\n2CPiXcBVwEeBU4GtwBcj4keauP+V0287wBD9tgOMoN92gCH6B5mXDZxUln7bAYbotx1gleu3HWAi\nmnrHfiHwPzLzM5n59czcBDwBfKCh+18h/bYDDNFvO8AI+m0HGKLfdgCtKv22AwzRbzvAKtdvO8BE\njF3sEXE48HrgpgWzbgJOH/f+JUnS6JrYQc2xwKHArgXTdwNrh9/8EeC2sQLs37/woSWVYY5xxweA\nffu+M/Z9SKtFjHsM6og4DtgOvDkz/3Jg+qXA2Zn56oFpfggpDZGZU7MFnmOCdHDLGQ+aeMf+d8A+\nYGbB9Bmqz9mfN00DlqThHBOk5o39GXtmfhf4KrB+wayfpNo6XpIkrZCmDgLzW8C1EXEXVZn/EtXn\n659q6P4lSdIIGin2zPzDiPhHwGbgFcDfAGdl5mNN3L8kSRrN2BvPSZKk7pjo8dgj4gRe+Mrbzsx8\ndJKPN6qI+Dngp4F/AK7JzHsG5r0cuDMzT2or32Ii4lzgjzPz/7WdZaGo9qP6z4HjgUcHn882RcTL\nMvMf6vMBvIlqu5KvZuYzrYardXUZmZSu/r6OCc1xPFi+xpaPzGz8RLUnuu3A/gWnx4Bfm8RjLiHb\nOcBzwB8AXwK+C7x/YP5aYH+bGQ+Q+znglLZz1FkuB95Wn385cEf9932u/rkVeHmL+V4JPFRnuaPO\neMvA/+Ec8OqWn8POLiPT9vs6Joydw/Fg/IyNLh+TCHgJ8CTwIeA0qldtx9fnLwb2AJe0+AR+Dfil\ngcs/VWfaVF9udSEGnqqfv6cWnPYDT8/Pb/mfcAfw2vr8Z+uF5ZVAAD8KfAW4tsV8N9QD9FuAzwH3\n1AvyD1NtA/Il4IYW83V6GZm239cxYex8jgfj5Wt8+ZhEyO3ATx9k/r8HHm/xSXwamF0w7V/WT94F\nVN+/b3sh/lNgA/DugdP36j/yu4F3t5Wvzvjs/HMIbAPWLZj/OmBXi/m+OTDQHE21n4UzBua/vuV8\nnV5Gpu33dUwYO5/jwXj5Gl8+JvEZ+8uA/3OQ+V+vr9OWPVSv0ubmJ2TmbRHxb4A/q+e16fXA7wNv\nBn4lM/cCRMSnqV5VPtBmuNoc8BP1z8W2vkzgyBXMs9ARVIMhwF6qdzZPDsx/CnjJSoca0PVlpGld\n/30dE8Yzh+PBOBpfPho7HvuAu4FLI+KIhTPqaZuBuybwuKO6G3j7wolZ7Q73p4DzafH4m5n5ENVG\nHU8CfxURp7WV5SA+BfxmRLwK+B3gyog4GSAiTqI6hO+XWsz3ALCx3kDmvcC3gLMH5v8s1cLSlq4v\nI03r+u/rmDAex4PxNL58TOId+/nAzcCuiPjfvHBwmLXAGVSvmH5yAo87qo9TLSTfJzNvrV+lb1jZ\nSN+X47vAr0XEl4EbIuKTbeZZKDOviojjqfZX8AhwAvCNiPge1f/UvVQLS1s+QvW52q8D3wbOonoe\n/xXVq/XXA+9qL17nl5Gmdf33dUwYg+PB2BpfPibyPfaI+AHgP1ItLGupXu3uBG4HrsvMJw9ycw2o\nD7JzLdWGH6/pwGq359Wv0P8tcBLV2p8ngL8EvpyT+MdaWraTgVOB2zLzifqdw/upNui5MQcOWNRS\nvqlaRqbt952kro4Jjgdj5Wt0+XAHNZIkFWQSn7GvahHx5Yh4pO0ckrrBMUGrzYoX+ypYSO6m+t5l\nJ62C56/zGc3XLavg93VMGIP5xrOcfBPdpewB3E21N51OyswPtp1hiE4/f7WuZzRft3T693VMGJv5\nxrPkfFP5GXtE/AjwAeB0Xtgv7xNUuz78VHpUOmmqOCaoJJPaKr6zC0lErAO+WOe5Cdhdz5qh+krB\nWqpDzra2lWSXn795Xc/Y9XwHU2ffkpnvaTtLU7r893BMGJ/5Jmc540Hjxd71hSQi7gG2ZuamA8z/\nBHB6ZrayE4iuP3/Q/YxdzzdMRJwK3JuZRWzc2vW/h2OC+drMN8xyxoNJFHvXF5JngFMzc9E9DUXE\njwH3ZeaalU32/ON3+vmrM3Q64yrIt4GD78nsBKpX6KUUe9f/Ho4JYzDfeCYxHkyi2Lu+kGwDPpqZ\nnznA/PcCmzPzxJVN9vzjd/r5qzN0OuMqyLcfeIZqr1eLOQRYk5mHrlyqyVkFfw/HhDGYbzyTGA8m\nsVX8TmAdB9737ulUq0TaciXwyYh4A9Vqmfnd980A66mOlHRBO9GA7j9/0P2MXc+3A/jVzPyTxWbO\nr3pb2UgT1fW/h2PCeMw3nsbHg0kUe6cXksz8rxHx91QHtn8PMP8qaB/wVeAXMvMP28pHx5+/Wtcz\ndj3fvVSHslx0QS5Qp/8ejgljM994mh8PlnKM11FPVDvUvxN4jmr1wv76/B3AOyfxmMvMeThwXH06\nvO08q+n563rGLuejOvzm2w8y/yig1/ZzOC1/jwU5HRPMt9LZGh8PJvo99og4HDi2vvh3WR2hSCNa\nDc9f1zN2Pd+08e8xnq4/f+brhqncQY0kSaUq4us0kiSpYrFLklQQi12SpIJY7JIkFeT/Ax7uzfbb\ngKmoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18beaf28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities grouped by actual response value\n",
    "df = pd.DataFrame({'probability':y_pred_prob, 'actual':y_test})\n",
    "df.hist(column='probability', by='actual', sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would have happened if you had used **y_pred_class** instead of **y_pred_prob** when drawing the ROC curve or calculating AUC?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1928dcc0>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAF2CAYAAABH4BBGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuQXOV55/HvgySEQFysC7qChQAjIYQFSNx0GyGIE+wt\npxyXvanEsb2xy8EuO1nXst4EE5Qtlym7UsROrWPHFcdkXc4FO1XOZeNdIjQjadAFxEUWAQGKEJFk\nIXSxhW4IafTuH2faPTPMfbr79On+fqq6pOk53fPMQcxv3vd5z3kjpYQkSSqOc/IuQJIkDY3hLUlS\nwRjekiQVjOEtSVLBGN6SJBWM4S1JUsEY3pIkFcyA4R0RyyLiHyNiT0ScjYiPDuI18yNibUSc6Hzd\n/ZUpV5IkDWbkfQHwE+B3gZNAv3d1iYiLgH8F9gELO193b0R8fmSlSpIkgBjKHdYi4ijwmZTS/+7n\nmHuAB4EpKaVTnc/dB9yTUpo5wnolSWp61eh53wasLwV3p0eB6RHxzip8PUmSmko1wnsqsL/Hc/u7\nfE6SJI1ANcLbnU4kSaqi0VV4z9d4+wh7SpfPdRMRhr0kqamklGIkr6/GyHsjsDQixnZ57i5gb0rp\n1d5ekFLyUcXHAw88kHsNjf7wHHueG+XhOR7+49SpxNq1ifvvT9x+e2L8+MQddyS+/OXE5s2JM2ey\n4yphwJF3RFwAXN354TnAOyNiAXAopbQ7Ih4EFqWU7uw85q+BB4CHI+JLwDXAF4BVFalYkqQ6cPYs\nbNsGq1dnj8cfh2uugTvvhD/6I1i8GMaNq87XHsy0+SJgTeffE/BHnY+Hgf9CNkU+u3RwSumNiLgL\n+AawBTgM/HFK6U8qV7YkSbX3yitZUD/2WPaYMAFWroRPfhK+//3s41oYMLxTSm30M72eUvp4L889\nBywfUWWqmJaWlrxLaHie49rwPFef57i7AwegtbU8uj5xIhtZv+c98NWvwuWX51PXkG7SUpUCIlLe\nNUiSBHD8OKxfXx5d79wJy5ZlgX3nnXDttRAjWmoGEUEa4YI1w1uS1LROn4Ynn8yCevVqePppuPHG\nLKhXroRFi2DMmMp+TcNbkqQhSAmef748Db5+PVxxRRbUd94JS5fCBRdUtwbDW5KkAeze3X2R2bhx\n5WnwFStg8uTa1mN4S5LUw89+Vl5k9thjcPhwNrIuPWbPHvg9qsnwliQ1vZMns2usS2H94ovZNdal\n0fX8+XBONW5JNkyGtySp6XR0wFNPlReZPfEEXH99eZHZrbfCuefmXWXfDG9JUsNLCV56qTyybmuD\nGTPKi8yWLYOLLsq7ysEzvCVJDemnPy0vMFu9Opv2Lo2s77gDpk3Lu8LhM7wlSQ3hyBFYu7Y8ut63\nLwvp0uj6qqtGfnOUemF4S5IK6dQp2LixPLJ+7rmsV10aXd9wA4walXeV1WF4S5IK4exZePbZclhv\n3Ahz55ZH1rffDuedl3eVtWF4S5LqUkrZfcFL0+Br1sCkSeXLt1pa4JJL8q4yH4a3JKluvP56FtKl\nwD51qjwNvnIlzJyZd4X1wfCWJOXm2DFYt64c1q++CsuXl0fXc+Y0ziKzSjK8JUk1c/o0bN5c7ls/\n80y261ZpdL1wIYwenXeV9c/wliRVTUrZKvDSyHr9+uySrdIisyVL4Pzz866yeAxvSVJFvfpq9x24\nLryw+w5cEyfmXWHxGd6SpBE5dKj7DlxvvNF9B65Zs/KusPEY3pKkITlxAtrby33rHTuy6e9S3/q6\n6+prB65GZHhLkvp15gxs2VIO6y1bYMGCcljffHN978DViAxvSVI3KcH27eVp8LVr4fLLy4vMli7N\n+tjKj+EtSWLv3vLI+rHHYMyY7jtwTZmSd4XqyvCWpCb0859ne1qXwvrAge47cM2e7c1R6pnhLUlN\n4M03YcOG8uj6+eezjTxKo+sFC1xkViSGtyQ1oI6ObAeu1auzx6ZN2Srw0sj6tttg7Ni8q9RwGd6S\n1ABSyi7ZKk2Dt7bC1KnlsF6+HC6+OO8qVSmGtyQV1Guvdd+Bq6Oj+w5c06fnXaGqxfCWpII4ejS7\nbKvUt96zJ9vTuhTY11zjIrNmYXhLUp16661sB65S3/onP8luiFIK6xtvdAeuZmV4S1KdOHsWtm0r\nT4O3t2ej6VLfevFiGDcu7ypVDwxvScrRK6+Up8HXrIFLLimPrFesgAkT8q5Q9cjwlqQaOniw+yKz\n48fLI+uVK7PbkEoDMbwlqYqOH4f168uj6507YdmycljPm+ciMw2d4S1JFXTmDDz5ZHmR2VNPwU03\nlUfXixZl9w2XRsLwlqQRSCm71WhpGnzdOpg1Kwvq0g5cF1yQd5VqNIa3JA3R7t3dd+AaN677DlyT\nJ+ddoRqd4S1JA/jZz7LbjZbC+vDhLKRLgT17dt4VqtkY3pLUw8mT8Pjj5dH1iy9m11iXwvr6692B\nS/kyvCU1vY4OePrp8sh68+YsoEuLzG69Fc49N+8qpTLDW1LTSQleeqkc1m1t2SYepZH18uVw0UV5\nVyn1zfCW1BT27eu+yCyi+yKzadPyrlAaPMNbUkM6cqT7Dlz79mW3Gy0F9tVXe3MUFZfhLakhnDoF\nGzeWw/q557JedalvfcMNMGpU3lVKlWF4Syqks2dh69byNPjjj8PcueWbo9x+O5x3Xt5VStVheEsq\nhJSy+4KXRtatrTBxYnkavKUF3vGOvKuUasPwllS3Xn+9+w5cp05134Fr5sy8K5TyYXhLqhvHjmX3\nBi+Nrl99NbtsqxTWc+e6yEwCw1tSjk6fhieeKI+sn34623WrNLpeuBBGj867Sqn+GN6SaialbBV4\naWS9fj1ceWV5kdmSJXD++XlXKdU/w1tSVb36ajms16yB8ePL0+ArVsCkSXlXKBWP4S2pog4dylaC\nlwL7yJHui8xmzcq7Qqn4DG9JI3LiRHaN9erV2ePll2Hp0nJYX3edO3BJlWZ4SxqSM2fgqafKi8ye\nfBIWLCiPrm++2R24pGozvCX1KyXYvr08Db52LVx2WXlkvWwZXHhh3lVKzcXwlvQ2e/d234Fr9Gi4\n667yDlxTpuRdodTcahbeEfFp4F5gKvBvwO+llNr7Of5u4A+Ba4FTwOPAvSmll3s51vCWRuDnP8/2\ntC4F9uuvZyFdGl1feaU3R5HqSU3COyI+DHwPuAdoBz4DfBy4NqW0u5fjrwKeBx4Cvg1cCHwFuDKl\ndHUvxxve0hC8+Wa2A1dpkdnzz2cbeZT61gsWuMhMqme1Cu/NwLMppU91ee4l4IcppT/o5fgPAn8L\njCmlckSsAB4DJqWUDvc43vCW+tHRAc8+W54G37gR5s0r3xzltttg7Ni8q5Q0WJUI735vXhgR5wI3\nAl/t8alHgdv7eNnjwDHgkxHxHeB84GPAEz2DW9LbpQQ7dnTfgWvKlCyoP/MZeOQRuOSSvKuUlKeB\n7jw8CRgF7O/x/Otk/e+3SSnt6+x5/wj4BnAO8AzwKyMrVWpc+/dnYV0K7I6ObBr8/e+Hr38dZszI\nu0JJ9aTi2wZExGyy4P4u8NfARcD/BB6JiDucI5fg6NFsB65S33rPnmxP6zvvhHvvhWuucZGZpL4N\nFN4HgQ6g58UlU4B9fbzmU8DulNIXSk9ExG8Cu4HbgA09X7Bq1apf/L2lpYWWlpYBypKK5a23YPPm\nct9669bshigrV8J3vgM33ugOXFKjamtro62traLvOZgFa5uArb0sWPtBSum+Xo7/KrAipbSoy3PT\ngL3Asp6XmLlgTY3o7FnYtq08Dd7eDu96V/nyrcWL3YFLala1Wm3+IbJLxT5NNmr+HbJLxeallHZH\nxIPAopTSnZ3HLwHWAqvIVp1fCHwZmAPMTSmd7PH+hrcawq5d5WnwNWuyRWVdd+CaMCHvCiXVg6qv\nNgdIKT0SEROBLwLTgG3A3V2u8Z4KzO5yfHvnteH/A/jvwAlgI/DLPYNbKrKDB7OQLo2ujx/Pgvo9\n74GvfhUuvzzvCiU1Km+PKg3S8ePZ9HdpdL1zZ3Zv8NLNUebNc5GZpIF5b3Opis6cyXbdKi0y27IF\nbrqpHNaLFsGYMXlXKaloDG+pglLKbjVamgZftw5mzSr3rZcuhfHj865SUtEZ3tII7d7dfQeu884r\n33Z0xQq49NK8K5TUaAxvaYh+9rPsdqOlwD50qDwNvnIlzJ498HtI0kgY3tIA3nwTHn+8PLLevj27\nxroU2Ndf7w5ckmrL8JZ66OiAp58uh/XmzTB/fnlkfeut7sAlKV+Gt5peSvDyy+XLt9raYPr0clgv\nXw4XXZR3lZJUZnirKe3b130HLigvMrvjDpg2Ld/6JKk/hreawhtvwNq15dH1vn3ZSvBS3/rqq705\niqTiMLzVkE6dgk2byn3rbdvgllvKo+sbboBRo/KuUpKGx/BWQzh7NtsiszQNvmEDzJlT7lvffjuM\nG5d3lZJUGYa3CmvnzvI0eGsrTJxYngZvaYF3vCPvCiWpOgxvFcbrr3ffgevNN8vT4CtXwsyZeVco\nSbVheKtuHTsG69eX+9a7dmWXbZVG13PnushMUnMyvFU3Tp+GJ54oj6yffhoWLiyPrhcuhNED7h4v\nSY3P8FZuUoLnniuH9fr1cOWV5WnwJUvgggvyrlKS6o/hrZr6j/8oT4M/9li2PWZpGnzFCpg0Ke8K\nJan+Gd6qqsOHs5XgpVXhR45034Fr1qy8K5Sk4jG8VVEnT0J7e3l0/dJLsHRpObCvu84duCRppAxv\njciZM/DUU+W+9ZNPwrvfXR5Z33ILnHtu3lVKUmMxvDUkKcGLL5anwdeuhcsuK4f1smVw4YV5VylJ\njc3w1oD27u2+A9fo0d134JoyJe8KJam5GN56myNHsj2tS33r/fuzkC71ra+80pujSFKeDG9x6lS2\nkUcprP/t3+C228qj6wULXGQmSfXE8G5CHR3w7LPlafCNG2HevHLf+rbb4Lzz8q5SktQXw7sJpAT/\n/u/lkfWaNVmfujQNvnw5XHJJ3lVKkgbL8G5Q+/dnIV1aFX7mTPdFZjNm5F2hJGm4DO8GcfQorFtX\nHl3v3p3taV0aXV9zjYvMJKlRGN4F9dZbsHlzuW+9dSssWlTuW990kztwSVKjMrwL5NAh+Ku/ysK6\nvR3e9a5yWC9eDOefn3eFkqRaMLwL5LOfhR074BOfyHbgmjAh74okSXkwvAtk3rxs5L1wYd6VSJLy\nZHgXxP79MGcOHDwIo0blXY0kKU+VCG/vvVUDbW3Zph8GtySpEgzvGmhtzfrckiRVguFdA2vWGN6S\npMoxvKts7144fBjmz8+7EklSozC8q6y1Nbtbmjt7SZIqxUipMvvdkqRKM7yrzH63JKnSDO8q2rUL\nTp6EuXPzrkSS1EgM7yoqTZm7I5gkqZIM7yqy3y1JqgbDu0pSst8tSaoOw7tKduzIpsuvuirvSiRJ\njcbwrhL73ZKkajG8q8R+tySpWtwStApSgmnTYNMmmDUr72okSfXELUHr1AsvwPnnG9ySpOowvKvA\nKXNJUjUZ3lVgeEuSqsmed4WdPQuXXgpbt8KMGXlXI0mqN/a869C2bTBxosEtSaoew7vCnDKXJFWb\n4V1hhrckqdrseVdQRwdMmgTbt8OUKXlXI0mqRzXreUfEpyPilYg4GRFbImLJIF7zexGxPSLejIif\nRsSDIym0CJ55Jut1G9ySpGoaPdABEfFh4GvAPUA78BngxxFxbUppdx+veQh4L/DfgG3AxcC0ShVd\nr5wylyTVwoDT5hGxGXg2pfSpLs+9BPwwpfQHvRx/DVlgz08pvThgAQ00bX733fCJT8AHPpB3JZKk\nelX1afOIOBe4EXi0x6ceBW7v42XvB3YCd0fEzs7p9ocjYvJICq13p09DezssX553JZKkRjdQz3sS\nMArY3+P514GpfbxmNvBO4EPAbwEfAeYA/xTRuBtkbtkCV16ZXeMtSVI1DdjzHoZzgLHAR1JKOwAi\n4iPAi8BC4MkqfM3c2e+WJNXKQOF9EOgAeq6fngLs6+M1+4AzpeDutKPzfS6nl/BetWrVL/7e0tJC\nS0vLAGXVn9ZW+Nzn8q5CklRv2traaGtrq+h7DmbB2iZgay8L1n6QUrqvl+PvAv4fcFVKaWfnc1cC\nLwM3p5S29Di+8AvWTp3Kru/eswcuvjjvaiRJ9axW13k/BHwsIn47IuZGxNfJ+t3f6iziwYhY3eX4\n1cDTwF9GxIKIuAH4S2BTz+BuFJs3w9y5BrckqTYG7HmnlB6JiInAF8mu1d4G3N3lGu+pZIvUSsen\niHgf8KfAOuAk2er0z1e49rphv1uSVEveHrUCli+H3/99+OVfzrsSSVK9q8S0ueE9QidPwuTJ8Npr\nMH583tVIkuqd+3nXgQ0b4N3vNrglSbVjeI+Q/W5JUq0Z3iO0Zo3hLUmqLXveI3DsGEydCgcOwLhx\neVcjSSoCe945a2+HhQsNbklSbRneI2C/W5KUB8N7BOx3S5LyYM97mI4cgZkz4eBBGDs272okSUVh\nzztH69bBrbca3JKk2jO8h8l+tyQpL4b3MNnvliTlxZ73MBw6BFdckf05Zkze1UiSisSed07WroUl\nSwxuSVI+DO9hsN8tScqT4T0M9rslSXmy5z1E+/fDnDnZ9d2jRuVdjSSpaOx556CtDZYtM7glSfkx\nvIfIfrckKW+G9xDZ75Yk5c3wHoK9e+HwYZg/P+9KJEnNzPAegtZWaGmBczxrkqQcGUNDYL9bklQP\nDO8hsN8tSaoHhvcg7doFJ0/C3Ll5VyJJanaG9yCVpsxjRJfVS5I0cob3INnvliTVC8N7EFKy3y1J\nqh+G9yDs2JH9edVV+dYhSRIY3oPS2gp33GG/W5JUHwzvQbDfLUmqJ24JOoCUYNo02LQJZs3KuxpJ\nUtG5JWgNvPACjBtncEuS6ofhPYBSv1uSpHpheA/Afrckqd7Y8+7H2bNw6aWwdSvMmJF3NZKkRmDP\nu8q2bYMJEwxuSVJ9Mbz7Yb9bklSPDO9+2O+WJNUje9596OiASZNg+3aYMiXvaiRJjcKedxU98wxM\nn25wS5Lqj+HdB/vdkqR6ZXj3wX63JKle2fPuxenTMHEivPJK9qckSZViz7tKtmyB2bMNbklSfTK8\ne2G/W5JUzwzvXtjvliTVM3vePZw6lV3fvWcPXHxx3tVIkhqNPe8q2LwZ5swxuCVJ9cvw7sF+tySp\n3hnePdjvliTVO3veXZw8CZMnw2uvwfjxeVcjSWpE9rwrbMMGuP56g1uSVN8M7y7sd0uSisDw7sJ+\ntySpCOx5dzp2DKZOhQMHYNy4vKuRJDWqmvW8I+LTEfFKRJyMiC0RsWSQr7s6Io5GxNGRFFkL7e1w\n000GtySp/g0Y3hHxYeBrwJeABcAG4McRcdkArzsX+FtgLZD/0HoA9rslSUUxmJH354HvppS+k1J6\nMaX0OWAfcM8Ar/sK8CzwA2BE0wO1YL9bklQU/YZ35+j5RuDRHp96FLi9n9e9F3gv8FkKENxHjsAL\nL8Att+RdiSRJAxs9wOcnAaOA/T2efx2Y2tsLImI68G3gV1NKJyLqPrtZty4L7rFj865EkqSBVeNS\nse8B30wpPVmF964K+92SpCIZaOR9EOgApvR4fgpZ37s3K4BlEfFA58cBnBMRp4F7Ukp/0fMFq1at\n+sXfW1paaGlpGbDwSmpthT/7s5p+SUlSk2hra6Otra2i7zngdd4RsQnYmlL6VJfnXgJ+kFK6r5fj\nr+3x1K8C9wGLgJ+mlH7e4/hcr/M+dAiuuCL7c8yY3MqQJDWJSlznPdDIG+Ah4HsR8QTZZWK/Q9bv\n/lZnEQ8Ci1JKdwKklJ7vUeTNwNmez9eLtWth8WKDW5JUHAOGd0rpkYiYCHwRmAZsA+5OKe3uPGQq\nMHugtxlRlVVkv1uSVDRNf3vU666Dhx+GhQtzK0GS1EQqMW3e1OG9fz/MmQMHD8KoUbmUIElqMu7n\nPUJtbbB0qcEtSSqWpg5v+92SpCJq+vD2fuaSpKJp2vDeuze7tnv+/LwrkSRpaJo2vFtbYflyOKdp\nz4AkqaiaNrrsd0uSiqqpw9t+tySpiJoyvHftghMnYO7cvCuRJGnomjK8W1uhpQUKsNW4JElv07Th\nbb9bklRUTRfeKdnvliQVW9OF944dWYBfdVXelUiSNDxNF96lUbf9bklSUTVleNvvliQVWVNtCZoS\nTJsGmzbBrFk1+ZKSJHXjlqBD9MILMG6cwS1JKramCm+nzCVJjaDpwttLxCRJRdc0Pe+zZ+HSS2Hr\nVpgxo+pfTpKkXtnzHoJt22DCBINbklR8TRPe9rslSY2iqcLbfrckqRE0Rc+7owMmTYLt22HKlKp+\nKUmS+mXPe5CeeQamTze4JUmNoSnC2363JKmRNE142++WJDWKhu95nz4NEyfCK69kf0qSlCd73oOw\nZQvMnm1wS5IaR8OHt/1uSVKjaYrwtt8tSWokDd3zPnUqu757zx64+OKqfAlJkobEnvcANm+GOXMM\nbklSY2no8LbfLUlqRA0f3va7JUmNpmF73idPwuTJ8NprMH58xd9ekqRhsefdjw0b4PrrDW5JUuNp\n2PC23y1JalQNHd72uyVJjaghe97HjsHUqXDgAIwbV9G3liRpROx596G9HW66yeCWJDWmhgxv+92S\npEbWsOFtv1uS1Kgarud95AjMnAkHD8LYsRV7W0mSKsKedy/WrYNbbjG4JUmNq+HC2363JKnRNWR4\n2++WJDWyhup5HzoEV1yR/TlmTEXeUpKkirLn3cPatbB4scEtSWpsDRXe9rslSc2g4cLbfrckqdE1\nTM97/36YMye7vnvUqAoUJklSFdjz7qKtDZYuNbglSY2vYcLbfrckqVk0VHjb75YkNYOGCO+9e7Nr\nu+fPz7sSSZKqb9DhHRGfjohXIuJkRGyJiCX9HNsSEf8QET+NiOMRsTUiPl6Zkt+utRWWL4dzGuJX\nEUmS+jeouIuIDwNfA74ELAA2AD+OiMv6eMltwFbg14B5wDeBb0fEr4+44l7Y75YkNZNBXSoWEZuB\nZ1NKn+ry3EvAD1NKfzCoLxTxd8ColNIHezw/4kvFZs+Gf/5nuPbaEb2NJElVV5NLxSLiXOBG4NEe\nn3oUuH0IX+ti4PAQjh+UXbvgxAmYO7fS7yxJUn0aPYhjJgGjgP09nn8dmDqYLxIR7wPuYGhhPyit\nrdDSAjGi32EkSSqOqi/xiojFwPeBz6aUtlT6/e13S5KazWBG3geBDmBKj+enAPv6e2HnivT/A9yf\nUvrzvo5btWrVL/7e0tJCS0vLIMqClLLwvv/+QR0uSVLNtbW10dbWVtH3HOyCtU3A1l4WrP0gpXRf\nH69ZBvwz8Icppa/1897DXrD28svZjVl273baXJJUDJVYsDaYkTfAQ8D3IuIJssvEfoes3/2tzkIe\nBBallO7s/LiFbMT9v4C/iYhSb7wjpXRgJAV3VbqrmsEtSWomgwrvlNIjETER+CIwDdgG3J1S2t15\nyFRgdpeXfBQ4D7i381Gyq8dxI9LaCr/0S5V6N0mSiqGwW4KmBNOmwaZNMGtW5euSJKkamnpL0Bde\ngHHjDG5JUvMpbHi7i5gkqVkVOry9vluS1IwK2fM+exYuvRS2boUZM6pUmCRJVdC0Pe9t22DCBINb\nktScChne9rslSc2ssOFtv1uS1KwK1/Pu6IBJk2D7dpjS827rkiTVuabseT/zDEyfbnBLkppX4cLb\nfrckqdkVMrztd0uSmlmhet6nT2f97p07YeLEKhcmSVIVNF3Pe8sWuOIKg1uS1NwKFd72uyVJKmB4\n2++WJDW7wvS8T53K+t179sDFF9egMEmSqqCpet6bN8OcOQa3JEmFCW/73ZIkZQoV3va7JUkqSM/7\n5EmYPBleew3Gj69RYZIkVUHT9Lw3bIDrrze4JUmCgoS3/W5JksoKE972uyVJytR9z/vYMZg6FQ4c\ngHHjaliYJElV0BQ97/Z2uOkmg1uSpJK6D2/73ZIkdVeI8LbfLUlSWV33vI8cgZkz4eBBGDu2xoVJ\nklQFDd/zXrcObrnF4JYkqau6Dm/73ZIkvV3dh7f9bkmSuqvbnvehQ3DFFdmfY8bkUJgkSVXQ0D3v\ntWth8WKDW5Kknuo2vO13S5LUu7oOb/vdkiS9XV32vPfvhzlzsuu7R43KqTBJkqqgYXvebW2wdKnB\nLUlSb+oyvO13S5LUt7oNb/vdkiT1ru7Ce+/e7Nru+fPzrkSSpPpUd+Hd2grLl8M5dVeZJEn1oe4i\n0n63JEn9q8vwtt8tSVLf6iq8d+2CEydg7ty8K5EkqX7VVXi3tkJLC8SILl2XJKmx1V142++WJKl/\ndRPeKdnvliRpMOomvHfsyAL8qqvyrkSSpPpWN+FdmjK33y1JUv/qLrwlSVL/6mJL0LNnE9OmwaZN\nMGtWruVIklRVDbMl6AsvwLhxBrckSYNRF+HtlLkkSYNneEuSVDB10fOeODHx7LMwc2aupUiSVHUN\n0/OeMMHgliRpsAYV3hHx6Yh4JSJORsSWiFgywPHzI2JtRJyIiD0RcX9/xztlLknS4A0Y3hHxYeBr\nwJeABcAG4McRcVkfx18E/CuwD1gI/C5wb0R8vq+vYXhLkjR4gxl5fx74bkrpOymlF1NKnyML5nv6\nOP43gPOAj6aUnk8p/T3wlc736ZXhXV1tbW15l9DwPMe14XmuPs9xMfQb3hFxLnAj8GiPTz0K3N7H\ny24D1qeUTvU4fnpEvLO3F0yZMrhiNTz+z1h9nuPa8DxXn+e4GAYaeU8CRgH7ezz/OjC1j9dM7eX4\n/V0+J0mSRqAaq83zvfZMkqQG1+913p3T5seB/9zZuy49/w3g2pTS27rVEfFXwMSU0vu6PLcI2Axc\nkVJ6tcfxhr0kqamM9Drv0QO8+VsR8RTwS8Dfd/nUXcAP+njZRuArETG2S9/7LmBvz+Du/BpuAipJ\n0hAMZtr8IeBjEfHbETE3Ir5O1rv+FkBEPBgRq7sc/9fACeDhiJgXER8AvtD5PpIkaYT6HXkDpJQe\niYiJwBeBacA24O6U0u7OQ6YCs7sc/0ZE3AV8A9gCHAb+OKX0J5UuXpKkZpT7vc0lSdLQVP3e5tW+\ntaqGdo7+u2ENAAAEQklEQVQjoiUi/iEifhoRxyNia0R8vJb1FtFQ/x13ed3VEXE0Io5Wu8aiG845\njojfi4jtEfFm57/pB2tRa5EN42fy3RGxKSLeiIgDEfGjiLi6VvUWSUQsi4h/7MyusxHx0UG8ZliZ\nV9XwrsWtVZvdUM8x2U10tgK/BswDvgl8OyJ+vQblFtIwznHpdecCfwusxUso+zWccxwRD5Hd6fFe\nYA7wK2TnWn0Yxs/kq4AfAW2dx99JdgfNf6lFvQV0AfATsuw6yQD/348o81JKVXuQXR725z2eewn4\nch/H3wP8HBjb5bn7gD3VrLPIj6Ge4z7e4++AH+b9vdTrY7jnGPgT4DvAR4GjeX8f9fwYxs+Ka4C3\ngGvyrr1Ij2Gc5w8CZ+hssXY+twI4C0zI+/up5wdwFPitAY4ZduZVbeRdq1urNrNhnuPeXEy2sFA9\nDPccR8R7gfcCnwW8HLIfwzzH7wd2AndHxM7OaeCHI2JyFUsttGGe58eBY8AnI2JURFwIfAx4IqXk\nz4yRG3bmVXPa3FurVt9wznE3EfE+4A7g25UtrWEM+RxHxHSy8/kbKaUT1S2vIQzn3/Fs4J3Ah4Df\nAj5CNnX+TxHhL0u9G/J5TintA+4mm2Z/k2yUOA/4T9Urs6kMO/OqvmBtiOwL1lBELAa+D3w2pbQl\n73oayPeAb6aUnsy7kAZ2DjAW+EhKqT2l1E4W4DeT9Q5VARExm6zn/V2y89pCNh38iL8kVcSwM6+a\n4X0Q6AB67hk2haw535vXePtvG1O6fE7dDeccA9C5wvRfgPtTSn9enfIawnDO8QrggYg4HRGngb8A\nLuj8+BPVK7WwhnOO9wFnUko7ujy3o/N9Lq94hY1hOOf5U8DulNIXUkpbU0rrgd8ElpNN+Wpkhp15\nVQvvlNJbQOnWql3dRbbCsTcbgaURMbbH8b3eWrXZDfMcExHLyIL7gZTSn1avwuIb5jm+Dnh3l8cf\nkq08fTfww+pUWlzDPMftwOjOkWHJbLJpYX9W9GKY5znIFqd1Vfq43mZui2j4mVfl1XYfAk4Bvw3M\nBb4OvAFc1vn5B4HVXY6/iOw3wL8h66t8ADgC/Ne8Vw7W62MY57iFbLOZr5D9hje18zE57++lXh9D\nPce9vP5juNq8oueYLFS2UL6E6Qayy8Q25P291PNjGOd5Cdlo/X7garIFb/8X2AWMy/v7qbcH2aVi\nCzofxzvP24JqZF4tvpl7gFfIFjs8CSzp8rnvAjt7HH9d5/+EJ4G9ZNO6uf9HqefHUM5x58cdZL89\nd33srHXdRXoM9d9xj9d+DHgj7++h3h/D+FkxFXikM3z2k6018JfQyp/nD3b+onS08zz/CJiT9/dR\njw+ywVHpZ2rXn7N/2c/5HVbmeXtUSZIKxp6FJEkFY3hLklQwhrckSQVjeEuSVDCGtyRJBWN4S5JU\nMIa3JEkFY3hLklQwhrckSQXz/wG/vscK1b4htAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18cda358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC curve using y_pred_class - WRONG!\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_class)\n",
    "plt.plot(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.780962171053\n"
     ]
    }
   ],
   "source": [
    "# AUC using y_pred_class - WRONG!\n",
    "print metrics.roc_auc_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use **y_pred_class**, it will interpret the zeros and ones as predicted probabilities of 0% and 100%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: ROC curve is only sensitive to rank order of predicted probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.84499501,  0.07452707,  0.85089451,  0.62517562,  0.87679784,\n",
       "        0.09611294,  0.13614004,  0.90132983,  0.61102779,  0.62517562])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted probabilities\n",
    "y_pred_prob[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.9192361 ,  0.27299647,  0.92243943,  0.79068048,  0.93637484,\n",
       "        0.31002087,  0.3689716 ,  0.94938392,  0.78168267,  0.79068048])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take the square root of predicted probabilities (to make them all bigger)\n",
    "import numpy as np\n",
    "y_pred_prob_new = np.sqrt(y_pred_prob)\n",
    "\n",
    "# print the modified predicted probabilities\n",
    "y_pred_prob_new[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x0000000019279EB8>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x00000000193EB630>], dtype=object)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAF9CAYAAAAdq/GkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHlhJREFUeJzt3XnUZHV95/H3F5BubRcQGRAkPLiM+4hLa2R6sHROGMTM\nZM6o4BLlMToxitEMxxOjA4oecvAExy16NGhGoo6Jidsw4gIOVBwFIhpC4gKBgcewdrtg28PWTfd3\n/ri3TVnUs1Xd+1Td3/N+nVOnn6p761u/ruX7qfrdW7ciM5EkSd23z7QHIEmSmmGoS5JUCENdkqRC\nGOqSJBXCUJckqRCGuiRJhTDUJUkqhKG+zkTEhoh4Z0TcFBF3RMTfRMRx0x6XpLUXEZsi4u0R8aWI\n+FFE7ImIN017XBqfob7+nAucCvwP4PXALuD8iDh2moOSNBUHA6cDjwf+tr7MI5J12H7THoDWTkQ8\nHTgJeFNmnl1f9gngu8DZwDOmODxJa+9m4LDMvDUijgSun/aANBk/qa8vLwD2AOfsvSAz7wb+FNhc\nv6glrROZuTMzb63PxlQHo0YY6uvLk4FrM3P70OWX1/8evcbjkSQ1yFBfXx4K3DLi8r2XHbaGY5Ek\nNcxQX1/uC9w94vK7BpZLkjrKUF9f7gQ2jLh848BySVJHGerryy2MnmJ/aP3vzWs4FklSwwz19eUK\n4BERccDQ5Xu/yvZ3azweSVKDDPX15TNUj/lv770gIjYArwC+nZk/nNbAJEmT8+Az60hmfisi/go4\nMyIeAlwLvBw4EnjVVAcnaSoi4nXAAfUJ4DkRsX/99/sz8+fTGZnGEZmLHxEwIk6h+lQ3V1/0PeDM\nzPxSvfxcqlAYdFlmHtP4SNWI+pP5O4DfBB4M/ANwemZ+daoDkzQVEXE91Rt7+OdDxEb991GZ+U9T\nGZjGslyo/weqr0BdQzVtOw/8PrA5M6+MiI9R7Xj1soGr7czMn7U2YkmSNNKS0++Zed7QRadFxGuA\npwNXUr2b25mZ21oanyRJWqEV7ygXEftGxIuovtP89friBLZExNaIuDoizomIg9sYqCRJWtqS0+8A\nEfFE4FKqg5bcCbw4M8+vl50E3E71yz5HAWcC+wJPzcydLY5bkiQNWUmo3wc4AngQ8ELgd4FnZ+a3\nR6z7UOCHwEmZ+fkRy/2dXmkJmblufinLfiAtb7U9Ydnp98zclZnXZeYVmfkW4DLglEXWvQW4EXjk\nEvUaP73tbW/rVN0ujtn7ov2665HPqW7W7eKYu1Y3c7yeMM7BZ/Zd7Hr19vTDGf1LYJIkqUVL7v0e\nEe8Evkj16fsBwEuAZwHHR8Qm4O1URym7leq77GcBW4F7Tb1LkqR2LXdEuUOATwKHAtupvsZ2fGZe\nGBEbgSdQfUf9AKpP5xcBL8jM29sb8r31er1O1W2zdtfqtlm7a3XVnK499l2r22Zt605m2R3lGr2x\niFzL25O6JCLIdbajnP1AWtw4PcEfdJEkqRCGuiRJhTDUJUkqhKEuSVIhiv899Yjm9jtypx5J0iwr\nPtQrTYTxutkpWZLUUU6/S5JUCENdkqRCGOqSJBXCUJckqRCGuiRJhTDUJUkqhKEuSVIh1sn31CVJ\nas+sHOjMUJckqRHTP9CZ0++SJBXCUJckqRCGuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKk\nQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12SpEIY6pIkFcJQlySpEIa6\nJEmFMNQlSSrEkqEeEadExJURsb0+XRIRJwytc0ZE3BQRd0TExRHxuHaHLEmSRlnuk/oNwO8DTwae\nClwEfCEingQQEW8CTgVeB2wGtgEXRsT9WxuxJEkaKTJzdVeI+AnwB8BHgZuB92fmWfWyjVTB/sbM\nPGfEdXO1tzepiACauM1grceu9SUiyMyY9jjWyjT6gdSWNrJmnJ6w4m3qEbFvRLwI2Ah8HTgKOAS4\nYO86mXlXveyY1QxCkiRNbr/lVoiIJwKXAhuAO4ETM/PqiNgb3FuHrrINOKzRUUqSpGUtG+rAVcC/\nAh4EvBD4i4h49jLXcU5NkqQ1tmyoZ+Yu4Lr67BURsRk4BXhHfdkhwI0DVzkEuHWxer1ej7m5Oebm\n5uj1evR6vbEGLnVdv9+n3++zsLDAwsLCtIczFfYDaVAfgPn5+bF7wjg7yl0E3JCZJ0fEzcAfD+0o\nt5VqR7mPjLiuO8pJi3BHOam7ZmVHuSU/qUfEO4EvUn0SfwDwEuBZwPH1Ku8F3hIRVwHXAKcBO4BP\nrWYQkiRpcstNvx8CfBI4FNgOXAkcn5kXAmTmH0XEfYEPAgcClwHHZebt7Q1ZkiSNsurp94luzOl3\naVFOv0vdNSvT7x77XZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12S\npEIY6pIkFcJQlySpEIa6JEmFMNQlSSrEcj+9qgHVr/BMzl+mkiS1wVBflWZ+Vk+SpDY4/S5JUiEM\ndUmSCmGoS5JUCENdkqRCGOqSJBXCUJckqRCGuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKk\nQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12SpEIY6pIkFWLJUI+IN0fE\n5RGxPSK2RcR5EfH4oXXOjYg9Q6dL2h22JEkattwn9WcBHwCeCTwHuAf4WkQcOLBOAhcChw6cTmh+\nqJIkaSn7LbUwM48fPB8RLwO2A8cA5++9GNiZmdtaGaEkSVqR1W5Tf2B9ndsGLktgS0RsjYirI+Kc\niDi4sRFKkqQVicxc+coRfwk8Anha1leMiJOA24HrgaOAM4F9gadm5s6h6+dqbq8JEUH1vmPiSo3V\nWev7QN0QEWRmTHsca2Ua/UBqS5NZs/d1MU5PWHL6/ZduJuLdVNPuWwZfiZn56YHVvhcR3wF+CDwP\n+PxwnV6vx9zcHHNzc/R6PXq93mrGKxWj3+/T7/dZWFhgYWFh2sOZCvuBNKgPwPz8/Ng9YUWf1CPi\nPcCJwLMz8x9XsP51wIcy8+yhy/2k7id1LcJP6lJ3deaTekS8D3ghKw/0g4HDgVtWMxBJkjSZ5b6n\n/kFgHngpsD0iDq1Pm+rlmyLiXRHxqxExFxE94DxgKyOm3iVJUnuWnH6PiD1U8wnDH//PyMx3RMRG\n4AvAk4EDqD6dXwScnpk3jajn9LvT71qE0+9Sd83K9Puq9n6flKFe1bGRaRRDXequWQl1j/0uSVIh\nDHVJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12S\npEIY6pIkFcJQlySpEIa6JEmFMNQlSSqEoS5JUiEMdUmSCmGoS5JUCENdkqRCGOqSJBXCUJckqRCG\nuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklS\nIQx1SZIKYahLklSIJUM9It4cEZdHxPaI2BYR50XE40esd0ZE3BQRd0TExRHxuPaGLEmSRlnuk/qz\ngA8AzwSeA9wDfC0iDty7QkS8CTgVeB2wGdgGXBgR929lxJIkaaTIzJWvHLEJ2A78RmaeHxEB3Ay8\nPzPPqtfZSBXsb8zMc4aun6u5vSZUQ2ziNpurs9b3gbohIsjMmPY41so0+oHUliazZu/rYpyesNpt\n6g+sr3Nbff4o4BDggr0rZOZdwNeBY1ZZW5IkTWC1of4+4Arg0vr8ofW/W4fW2zawTJIkrYH9Vrpi\nRLyb6tP3lhXOmY1cp9frMTc3x9zcHL1ej16vt9IhSEXp9/v0+30WFhZYWFiY9nCmwn4gDeoDMD8/\nP3ZPWNE29Yh4D3Ai8OzM/MeByx8OXAtszszvDFx+PrAtM18xVMdt6m5T1yLcpi51V2e2qUfE+4CT\ngOcMBnrteuBW4LiB9TcCW4BLVjMQSZI0mSWn3yPig8BvAv8R2B4Re7eT78jM2zMzI+K9wFsi4irg\nGuA0YAfwqRbHLUmShiw5/R4Re6jmE4Y//p+Rme8YWO9twKuBA4HLgFMy8/sj6jn97vS7FuH0u9Rd\nszL9vqrvqU/KUK/q2Mg0iqEuddeshLrHfpckqRCGuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhD\nXZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12SpEIY6pIkFcJQlySp\nEIa6JEmFMNQlSSqEoS5JUiEMdUmSCmGoS5JUiP2mPQCNJyIaq5WZjdWSJE2Pod5pTYRxc28OJEnT\n5fS7JEmFMNQlSSqEoS5JUiEMdUmSCmGoS5JUCENdkqRCGOqSJBXCUJckqRCGuiRJhTDUJUkqhKEu\nSVIhlg31iDg2Is6LiBsjYk9EnDy0/Nz68sHTJe0NWZIkjbKST+qbgL8H3gDcyb1/RSSBC4FDB04n\nNDhGSZK0Asv+Sltmfhn4MlSfykesEsDOzNzW7NAkSdJqNLFNPYEtEbE1Iq6OiHMi4uAG6kqSpFVo\n4vfUvwJ8FrgeOAo4E7goIp6amTsbqC9JklZg4lDPzE8PnP1eRHwH+CHwPODzw+v3ej3m5uaYm5uj\n1+vR6/UmHYLUSf1+n36/z8LCAgsLC9MezlTYD6RBfQDm5+fH7gmRObzf2xIrR+wATsnMjy+z3nXA\nhzLz7KHLczW314SI4N779o1VqbE6TdwHTf6/1vox0WgRQWbGtMexVqbRD6S2tNGTx+kJjX9Pvd6e\nfjhwS9O1JUnS4padfo+ITcCj6rP7AEdGxNHAT4CfAm8HPgPcCswBZwFbGTH1LkmS2rPs9HtE9ICL\n6rNJNQ8NcC7wWuALwJOBA6g+nV8EnJ6ZN42o5fS70+9ahNPvUnfNyvT7qrapT8pQr+oY6hrFUJe6\na1ZC3WO/S5JUCENdkqRCGOqSJBXCUJckqRBNHCZWq1TtUCFJmoS99N4M9aloam98SVrvZuXbTbPR\nk51+lySpEIa6JEmFMNQlSSqEoS5JUiEMdUmSCmGoS5JUCENdkqRC+D11SdKa88Ax7TDUJUlTUsYB\nX2aJ0++SJBXCUJckqRCGuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQ\nhrokSYUw1CVJKoShLklSIQx1SZIKYahLklQIQ12SpEIY6pIkFcJQlySpEIa6JEmFMNQlSSqEoS5J\nUiGWDfWIODYizouIGyNiT0ScPGKdMyLipoi4IyIujojHtTNcSZK0mJV8Ut8E/D3wBuBOIAcXRsSb\ngFOB1wGbgW3AhRFx/2aHKkmSlhKZufxae1eO2AGckpkfr88HcDPw/sw8q75sI1WwvzEzzxm6fq7m\n9ppQDbGJ2yy3zlo/JhotIsjMmPY41so0+oFmRzO9eZb6afM9eZyeMOk29aOAQ4AL9l6QmXcBXweO\nmbC2JElahUlD/dD6361Dl28bWCZJktbAfi3WHjkP0ev1mJubY25ujl6vR6/Xa3EI6ppqSq4Zsz61\n2+/36ff7LCwssLCwMO3hTIX9QBrUB2B+fn7snjDpNvWHA9cCmzPzOwPrnQ9sy8xXDF3fbeozWGeW\nwq/Jx2uW/l8r4TZ1rSduU1+8zjS3qV8P3Aoc94vhVDvKbQEumbC2JElahWWn3yNiE/Co+uw+wJER\ncTTwk8y8ISLeC7wlIq4CrgFOA3YAn2ppzJIkaYRlp98jogdcVJ9NqjkGgHMz87fqdd4GvBo4ELiM\naor++yNqOf0+g3VmaQrU6Xen37U+OP2+eJ1Jpt9XtU19Uob6bNaZpcZqqBvqWh8M9cXrTHObuiRJ\nmhGGuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoSh\nLklSIQx1SZIKYahLklQIQ12SpEIY6pIkFcJQlySpEIa6JEmFMNQlSSqEoS5JUiH2m/YARslM7r77\n7mkPQ1IH7d69m127djVSa8OGDUREI7WktTCToX7bbbdx0EEHse++GyaslI2MR1J3fO5zn+Okk05i\nn332n6jO7t13c+ONN3L44Yc3NDKpfTMZ6gAbNhzI3Xf/dMIqtwCHNTEcSR1y//s/nx07/mqiGve7\nn2Gu7nGbuiRJhTDUJUkqhKEuSVIhDHVJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQM3vwGXWLh9KU\npOkz1NWgJg7L65sDSRqX0++SJBXCUJckqRATh3pEnBERe4ZONzcxOEmStHJNbVO/CugNnN/dUF1J\nkrRCTYX67szc1lAtSZI0hqa2qT88Im6KiOsi4s8j4qiG6kqSpBVqItQvA04G/h3wn4FDgUsi4sEN\n1JYkSSs08fR7Zn5l4Ox3I+JS4HqqoH/P8Pq9Xo+5uTnm5ubo9Xr0er1Jh6AJeeCY6ej3+/T7fRYW\nFlhYWJj2cKZi1vvBwx72sEbqZDZxDAeVrw/A/Pz82D2h8YPPZOYdEfE94JGjlvf7/aZvUhPzoDHT\nMBxi6/HNVTf6waSvj/X3uGpcPQDOPfdcYLye0Pj31CNiI/BY4Jama0uSpMU18T31d0XEsRFxVEQ8\nA/gMcF/gzyYenSRJWrEmpt8PB/4ceAjwI+BS4Fcz84YGakuSpBVqYke5FzcxEEmSNBmP/S5JUiEM\ndUmSCmGoS5JUiMa/py7Niqa+9+2BQyR1haGugnlQHUnri9PvkiQVwlCXJKkQhrokSYUw1CVJKoSh\nLklSIQx1SZIKYahLklQIv6cuLcOD2GhSTTyHZuX509TrQe0w1KVleRAbTWrS59CsPX98Tcwqp98l\nSSqEoS5JUiEMdUmSCmGoS5JUCENdkqRCGOqSJBXCUJckqRCGuiRJhfDgM5LUAR7ZUCthqEtSJ3gU\nNy3P6XdJkgphqEuSVAhDXZKkQhjqkiQVwlCXJKkQhrokSYUw1CVJKoTfU5ekdaSpg9hoNhnqkrSu\nTHoQG98UzDKn3yVJKoShLklSIRoL9Yh4bURcHxF3RsS3I2JLU7UlSdLyGgn1iDgJeC9wJnA0cAnw\n5Yg4oon6y+t3rG6btbtWt83aXaurptxzz7aWKvet23pt606iqU/qpwIfy8w/zcyrM/P1wC3Aaxqq\nv4x+x+q2Wbtrddus3bW6asru3T9qqXLfuq3Xtu4kJg71iNgfeApwwdCiC4BjJq0vSZJWpomvtD0E\n2BfYOnT5NuDQcYvu2bML+OYK1/6nRdb98bg3L6nD9uy5g5X3j9F27767mcFIaygyJ/vOYkQcBtwI\nHJuZ3xi4/K3ASzLzMQOXTfoFSalombluvgRsP5CWt9qe0MQn9R8Du4FDhi4/hGq7+i+sp4YlaWn2\nA6l5E29Tz8ydwHeA44YW/RrVXvCSJGkNNHWY2HcDn4iIb1EF+e9QbU//cEP1JUnSMhoJ9cz8y4g4\nCDgNeCjwD8AJmXlDE/UlSdLyJt5RrlQRsS+wT2bumvZYShARD8lMv46gzrInNMd+0J41P/Z7RBwd\nEbvHvO6LI+IzEXFORDxtaNnBEXHdGDXvExFnR8QlEXFGfdkfAXcAd0TEJyNiwzjjXeI2XxURB0xw\n/eMi4j4D518aEVdGxB0RcW1EvKGhcUZEbI6I5w/f32PYFhH/OyJeNDj2pkTEgwf+jog4JiK2RMR9\nG6j9KxHxjPq+OGjSegN1j6zrPiMijmyqbpdM0g/q63e+J3SlH9S1m+oJ9oPRtSfvCZm5pifgScCe\nMa73CmAX8GngK8BO4NUDyw8ds+4fArdS7RfwPeC/AzcALwVeTvV1vT9o+D7YBTxuguvvAf5F/ffz\ngXuADwAnA+8B7qb6OuFq654FHF//fTBwWX1bu+p/LwEOnmDMF9dj3Qb8N+AxDdyXjwKuqetfVo/7\n4vr8HmBh3NsBTqE6CMKeodM3gKdNMOZT6+fVcN0bgP/S5HNt1k/j9oP6ukX0hFntB3W9VnqC/eBe\ntRvrCY08KYcGdzFwUf3vqNPlwO4x6l4J/M7A+V8HtgOvr8+P+wL+v8C/r/9+ZH1Hvmhg+YnAd8e8\nL3YAP6//HTztAW7fu3yMuoMv4m8Abx9a/kbgW2PUvRl4Uv33ufWL4lFUP6D8COCvgU+MeV/sofqa\n4xHAGQMvjv9D1Sg3jln3C1QN/dnAnwHfrp9nh1Pt3/EV4Atj1H0jcBPwOuBVwA+AtwLPBT5eP36b\nx6h7ev2c+K/AZuBX6tNm4C31c/r0ce6LWTy11Q/q2p3qCV3rB/V1W+kJ9oNfqt1oT1j1AFYwwJ3A\n56neKY46/cWYL7Tbgbmhy/51/R/+vfoJMk7dO4Ejhs4/ZuD8w4EdY94XO4AvUr1jnh843VM/WPPA\n/JgviL0v4q3AU4aWPxrYPkbdu/bex8D1wJah5U8Gto55X/xizPX5fYHnAf+T6l3/bcAHxqj7o4Gm\n8wCqYyb8m4HlTxlnzFTv6E8YOP8vqY7JsF99/n3ABWPUvRF4/hLL/xNw0zj38Sye2uoHde1O9YSu\n9YP6uq30BPvBL9VutCc09ZW2QT8Azs/Mj45aGBFHU73TXa3tVO+0FvZekJnfjIjnAV+ql41jO3AA\n1TQHwBXA/xtYvj+QY9Z+ClXTOhb43cy8AyAiPkL1bvH7Y9YFeFJE/JTqRTf8OO5D9SJZrQXgifW/\no/7PCUy8TQogM3cD5wPn10cl/K36tFobqJolVNs891C9691rB3C/MeoeDAw+PtcCD6ovvwX4GNWn\notV6MNVrZDFX1+uUoq1+AN3rCV3rB7BGPWEd9wNouCe0saPcFcBTl1i+E/jhGHUvp5rq+CVZHZr2\n16mmRcZ5of2A6t3m3nrHZOaNA8ufQLWNZtUy8xrgmVRPqr+LiM3j1FnEV6nukyOoPp0MOprx7uMP\nA++KiEcDfwycHRGPBIiIh1P9vO5Xxh7xIjLz5sw8k2o6b7W+D/x2RATwSqp3+C8ZWP4iqhfFal0D\nHD9w/t9SfYK4tT5/N+M93y4H3jpqR6v6stOAb41Rd1a11Q+gYz2hg/0AptAT1lk/gKZ7wjjTBctM\nJWwA7tdC3R7w5mWWf2yMuo8DHrHE8pcxsD1tgvE/j2qbzGlMvmPM3NDpoKHlJwMvH7P2u6ka7VVU\n04576vN7qLZPHTpm3TOATS08L06ox3cP8DOqXwbcRnWUw8uppt9eMEbdE+vH6bPAp6imet85sPw1\nwCVj1H0iVSP4GfC/gI/Wpy9SfUK8BXhC0/fTtE5t9YO6dmd7Qlf6QX39xnuC/eCXajfaE/ye+hqq\np5U+QbUTxxNysum21tTvyn+DatvhPlRPqm8AX8sZfMLUnxyOBr6ZmbfUnyBeTbUzz3k58ENDq6z7\nXKoGvj/w1cz8yMCygwAy8ydj1H0g1Z7Uz6TamSupXtSXAp/KzJ8vcXUVoiv9ALrVE7rWD+rrN9YT\nDHVJkgoxjYPPfG2cA0KUVrdNjlld0ebjbk+odG280M0xz4o1D3Wq7Rp/bd1Wn7itjLnlF1qnxty1\nujOstddXi7U79VzFHvYLXXzdrrZ2G19pW1Jmvtm6v3A5//y1mca0OOZWxgudHHPX6s6kNl9fHewJ\nXesH0L0xd/F1u6rarWxTj4gjqPYGPIZqoz9UO1ZcAnw4x/z1tq7VbZNjVle0+bjbEypdGy90c8xd\n0HioR8QW4MtUD84FVF8ngOroTr9G9eCdsNo9ELtWd6B+40/cNsfcYpPs4pjXvOnUt/n2zBznwBsz\np+XHvXM9oWv9oItjLqkfDNzuintCG6H+barv671+keXvA47JzFUdeKFrdevrttV02rov2nyhdWrM\nbTfKJW73aOBvM3Ma+7s0ruXXV6d6Qtf6QRfHXFo/qG97VT2hjVC/Ezg6M0cetSciHgtckZkbS65b\nX7etJ25b90WbzaFTY26x7sksfeSpI6nelZcS6m2+vjrVE7rWD7o45q71g/q6zfaEbP6IPtcDr1xi\n+SuB60uvW1/3TuDRSyx/LHDXrIy5rfF2ccwt1h38Ra5Rp9sZ81fLZvHU8uurUz2ha/2gi2PuWj+o\nr9toT2hj7/ezgQ9FxNOppim21pcfAhxH9UtEv7cO6kJ1RKAtLH6s4WOopnNWq60xtzVe6N6Y26p7\nM/CGzPzsqIV7p9rGqDur2nx9da0ndK0fQPfG3LV+AE33hHHeWazgncdJwN9QHSt374+976L6Ld4T\n11Hd11Idh/hPgOdTPSm21H//CdWPALxmVsbc5ni7NuYW654HnLnE8qMZ86dIZ/XU1uurzdpdeq62\nfD90asxd6wd17UZ7wtgvphUOdn/gsPq0/3qs22ZDa2PMbY+3S2NuqekcCzx3ieWbgF4T9/Osndp6\n3bZZuyvP1Tbvh66NuUv9oK7baE/w2O9rJCL2Bx5Sn/1xZu6c5niW07XxQntj7uJ9odnWxedU18a8\nXvuBoS5JUiGK+NqMJEky1CVJKoahLklSIQx1SZIK8f8BjgQNUVaf8ocAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18ccf7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities has changed\n",
    "df = pd.DataFrame({'probability':y_pred_prob_new, 'actual':y_test})\n",
    "df.hist(column='probability', by='actual', sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x19950320>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAF2CAYAAABH4BBGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHQ1JREFUeJzt3XuUHGWd//H3NwGCRhAlkATkFkRjuAWWuBK5DGpEIoir\nrMh6A5HDIiIuZ9HfEZDoeowgBlBDMGhAWVFurndCCDLRJCBEBFGRywLLxZiQwy1XNOH5/VE9ZBhn\nMj093V39TL9f59SZ6eqnur/9EObTVfXUU5FSQpIk5WNY2QVIkqSBMbwlScqM4S1JUmYMb0mSMmN4\nS5KUGcNbkqTMGN6SJGWm3/COiEMi4icR8XhEvBARH6lim70jYkFErKlsd059ypUkSdXseY8Efg+c\nDqwFNjmrS0RsDdwELAUOqGx3ZkScMbhSJUkSQAxkhrWIWAmcmlL67ibanAJMB0anlJ6vrDsLOCWl\n9JpB1itJUttrxDnvA4FfdwV3xTxgh4jYpQHvJ0lSW2lEeI8BlvVYt6zbc5IkaRAaEd7e6USSpAba\nrAGv+Vf+cQ97dLfnXiIiDHtJUltJKcVgtm/EnvetwMERMaLbuinAEyml/+ttg5SSSwOXc889t/Qa\nhvpiH9vPQ2Wxj2tbJkxI/OEP1bWth2qu8x4ZERMjYmKl/S6VxztVnp8eEfO7bXIVsAa4IiL2jIj3\nAJ8BZtSlYkmS2lw1h80nAb+s/J6Az1eWK4CPUhwiH9fVOKX0XERMAWYCS4CngAtSShfWr2xJUrv5\n3/+FvfeG9evLruQfpQQjRzbv/foN75RSJ5vYQ08pndDLuj8Ahw6qMtVNR0dH2SUMefZxc9jPjdfK\nffzss/C618Htt5ddyT+KgM03b+L71ev4e80FRKSya5Aktb4774SPfaz4mbOIIA1ywFojRptLklSV\nZ5+Fn/0Mli7tv+3jjze+nlwY3pKkpnruOfjpT+Gaa6CzEzo6YI89+t9us83g9NMbXV0ePGwuSWq4\nVas2BvYvfwmHHALvex+8613wyleWXV1z1eOwueEtSWqI1avh5z8vAvumm+DNby4C+93vhm22Kbu6\n8hjekqSWkRLcdx/Mmwc33ggLF8Kb3gTHHlsE9qtfXXaFrcHwliSV6umn4eabi7CeNw9eeAEOP7xY\n3vpWA7s3hrckqanWry+us+4K6z/+EQ46qAjrt78dxo8vrnlW3wxvSVLDPfLIxkPhv/wl7LJLEdSH\nH16cx95yy7IrzIvhLUmqu1Wriku4uvaun356Y1hPmQJjet43UgNieEuSBu2FF+CuuzbuXS9ZApMm\nbQzsffeFYY24B2WbMrwlSTVZurS4fOvGG4ufr371xrA+9FB4xSvKrnDoMrwlSVVZt664dKtr7/rR\nR4vR4F2HwnfdtewK24fhLUnqVUpw770bw3rRIthrr41715MmFdONqvkMb0nSi556CubP3xjYw4Zt\nvITrrW+FV72q7AoFhrcktb3HHoPLLivC+t57iznDu/auX/c6r7luRd4SVJLa3He/W5zLnj69uOZ6\nxIiyK1IzGN6SlLkDD4S3vKXsKtRMXrknSVJmDG9JylRK8MADsPnmZVeiZjO8JSlT554Ld94Jn/hE\n2ZWo2TznLUkZ+vzn4Yc/hFtugVGjyq5GzWZ4S1Jm/uu/4Oqri+Debruyq1EZDG9JysiXvgRXXVUE\n9+jRZVejshjekpSJL3+5uK77llu8LWe7M7wlKQNf+QrMmVPcZ3vs2LKrUdkMb0lt6dln4eKL4YYb\nikuuWtn69UW9nZ2www5lV6NW4NzmktrKc8/B175WBPfUqfDRj+YxpejrX++NRYYK5zaXpCqtXAnf\n+AZceGFx045Fi4obd0g5MrwlDWmrVsHMmTBjRnFbzF/9CsaPL7sqaXAMb0lD0po1cMklcMEFcOih\nxQjtCRPKrkqqD8Nb0pCydi1ceimcfz4cdBDMnw977VV2VVJ9Gd6ShoS1a2H2bDjvPHjTm+DGG2Gf\nfcquSmoMw1tSw82cCf/zP419j3vvhUmT4Be/gIkTG/teUtkMb0kN19kJBx5YnHtulB128Jy22ofh\nLakp9tkH3va2squQhgbDW2oTf/wjrF5dzns/9VQ57ysNVYa31AamTy9mFNt553LePwJ2372c95aG\nIsNbGuLOPx8uvxzuvNN5saWhwvCWhrCvfhUuu8wbWkhDjeEttbj584tLrQZq7Vp48MEiuHfcse5l\nSSqR4S21uMWLYfPN4bjjBr7t5MkwenT9a5JULsNbysD48fAv/1J2FZJaheEtleSJJ4p7S/fnySe9\nj7OklzK8pZLstRdsvz0MG9Z/27PPbnw9kvIRKaVyC4hIZdcglWHkSFi+vPgpqX1EBCmlGMxrVPGd\nX5IktRIPm0sDdMstMGfO4F/n+ecH/xqS2pPhLQ3ATTfBBz4A06bBVlsN7rXe+14PmUuqjeEtVenm\nm+Hf/g1++EM4+OCyq5HUzgxvtaxVq+Dpp8uuovD738Pxx8P11xvckspneKtlvec98LvfwZZbll0J\njBgB114LhxxSdiWSZHirha1bV+zpGpiS9FJeKiZJUmbc89ZL/Pd/w6JFZVdReOCBsiuQpNZkeOtF\nX/86zJgBZ54JMai5f+pjv/3gn/6p7CokqfVUFd4R8XHgTGAM8EfgUymlhZtoPxX4HDABeB5YBJyZ\nUnJfqkVdcgl89avFvZ933bXsaiRJm9Lv3OYRcSxwJXAKsBA4FTgBmJBSeqyX9q8F/gTMAGYDWwHn\nAbunlPbopb1zmzfAhg2wcmV1ba+6Cs47rwju3XZraFmS1PbqMbd5NeH9G+CulNLJ3dbdD1yXUvps\nL+2PAX4AbN6VyhFxGHAzMCql9FSP9oZ3A3z608Vh8BEj+m87diz87Gew++6Nr0uS2l09wnuTh80j\nYgtgf+D8Hk/NAyb3sdkiYBVwUkR8G3g5cDxwe8/gVuOsXg0XXACnnlp2JZKkeuvvUrFRwHBgWY/1\nyynOf/+DlNJSYCrwRWAd8AywJ3DUoCqVJElAA0abR8Q44EfA5cBVwNbAF4BrIuItHiOvn0WL4Mc/\n7v25xYthwoTm1iNJao7+wnsFsAEY3WP9aGBpH9ucDDyWUvpM14qI+CDwGHAgsLjnBtOmTXvx946O\nDjo6OvopS1DMPnb//b3PQHbccXDkkc2vSZL0Up2dnXR2dtb1NasZsHYbcHcvA9auTSmd1Uv784HD\nUkqTuq0bCzwBHNLzEjMHrNXujDPgNa8pfkqS8lCPAWvVTI86Azg+Ik6MiDdExMUU57svrRQxPSLm\nd2v/E2D/iDgnIvaIiP0pDqE/Cvx2MMVKkqQqznmnlK6JiG2Bs4GxwD3A1G7XeI8BxnVrv7Bybfj/\nAz4NrAFuBd6RUlpb5/olSWo7VQ1YSynNAmb18dwJvay7DrhucKVJkqTeeFcxSZIy441JWkxKMGsW\nrFjRf9tbb4V//dfG1yRJai3uebeYzs5iZrT16/tfpkyBqVPLrliS1Gz9XirW8AK8VOwljjsOJk+G\n004ruxJJUiM05cYkjWZ4b7RiBbz2tfDww/CqV5VdjSSpEZp1nbea5Mor4aijDG5J0qYZ3i0iJZg9\nG046qexKJEmtzvBuEYsWFQF+8MFlVyJJanWGd4u47LJirzsGdRZEktQOHLDWAp5+GnbbDR54ALbb\nruxqJEmN5IC1IeJ734N3vMPgliRVx/AuWUobD5lLklQNw7tkd9wBq1bBYYeVXYkkKReGd8lmz4aP\nfQyG+V9CklQlB6yVaOVK2HlnuPdeGDOm7GokSc3ggLXMff/7xeFyg1uSNBCGd4kcqCZJqoXhXZLf\n/Q6WL4e3v73sSiRJuTG8S3LZZXDiiTB8eNmVSJJy44C1kmyzDdxzD+y0U9mVSJKayft5Z2zYMFi/\n3kvEJKndONpckqQ2ZHhLkpQZw1uSpMwY3pIkZcbwliQpM5uVXUC7uP56+PznNz72+m5JUq0M7yb5\n85/hjW+ET36yeLzVVl4mJkmqjeHdRKNHwz77lF2FJCl37vtJkpQZw1uSpMwY3pIkZcbwliQpMw5Y\nq6O5c+ELX+j9uccfhxNOaG49kqShyfCuo3vugV13hU98ovfnJ0xoajmSpCHK8K6zHXeEyZPLrkKS\nNJR5zluSpMwY3pIkZcbwliQpM4a3JEmZMbwlScqM4S1JUmYMb0mSMmN4S5KUGcNbkqTMGN6SJGXG\n8JYkKTPObT5Ip59e3DEM4P774Ygjyq1HkjT0RUqp3AIiUtk1DMbWW8PMmfDylxeP//mf4TWvKbcm\nSVLrighSSjGo1yg7OIdCeD/+ePFTkqT+1CO8PectSVJmDG9JkjJjeEuSlBnDW5KkzBjeg7B6Nfz9\n72VXIUlqN4Z3jdasgaOOgve/35HmkqTmqiq8I+LjEfFwRKyNiCURcVAV23wqIv4cEesi4i8RMX3w\n5baGtWvh6KNhxx3hW98quxpJUrvpd4a1iDgWuAg4BVgInArcEBETUkqP9bHNDOCdwH8C9wCvBMbW\nq+hme/ZZWLWq+H3DBjjpJNh+e7jiChg+vNTSJEltqN9JWiLiN8BdKaWTu627H7gupfTZXtq/niKw\n904p3ddvARlM0rLTTsW57a6gPvxwmD0bNnNyWUnSANVjkpZNxk9EbAHsD5zf46l5wOQ+NjsaeAiY\nGhE3AAEsAM5MKT05mGLLsm4d/OlPsN12ZVciSVL/57xHAcOBZT3WLwfG9LHNOGAX4H3Ah4EPAeOB\nn0bEoL5pSJKkxtxVbBgwAvhQSulBgIj4EHAfcABwRwPec8Duuw++/vXq2q5c2dhaJEkaiP7CewWw\nARjdY/1oYGkf2ywF1ncFd8WDldfZmV7Ce9q0aS/+3tHRQUdHRz9lDd6iRbBkCXzwg/23veQSGDWq\n4SVJkoagzs5OOjs76/qa1QxYuw24u5cBa9emlM7qpf0U4EbgtSmlhyrrdgceAN6YUlrSo30pA9bm\nzIGFC4ufkiQ1S7PuKjYDOD4iToyIN0TExRTnuy+tFDE9IuZ3az8fuBOYExETI2I/YA5wW8/gliRJ\nA9fvOe+U0jURsS1wNsW12vcAU7td4z2GYpBaV/sUEUcCXwN+BaylGJ1+Rp1rlySpLVU1YC2lNAuY\n1cdzJ/Sy7q8Uo80lSVKdObe5JEmZadvwbvFJ3SRJ6lPbhvcDDxTTnkqSlJu2De/OTmjC5eSSJNVd\nv9d5N7yAEq7zXrUKxoyBJ5+El72sqW8tSWpzzbrOe8hZvBj228/gliTlqS3D20PmkqSctWV4L1gA\nhx5adhWSJNWm7c55r14N228Py5fDyJFNe1tJkgDPedfk1lth4kSDW5KUr7YLbw+ZS5JyZ3hLkpSZ\ntjrnvXYtjBoFy5bBK17RlLeUJOklPOc9QLfdBnvvbXBLkvLWVuG9YIHXd0uS8lfV/bxztWwZXH31\nxjuIXXcdfOUr5dYkSdJgDek97/nzYeZMeOihYjniCAerSZLyN6T3vAEOOAAuvrjsKiRJqp8hvect\nSdJQZHhLkpQZw1uSpMwY3pIkZcbwliQpM4a3JEmZMbwlScqM4S1JUmYMb0mSMmN4S5KUGcNbkqTM\nDLm5zW+9FR59dOPvkiQNNUMuvD/6Udh1V9h66+Lxu95VajmSJNXdkAvvlODCC2H8+LIrkSSpMTzn\nLUlSZgxvSZIyY3hLkpQZw1uSpMxkP2BtzRpYsKAYqAawalW59UiS1GjZh/cNN8Cpp8L++xePDzgA\nRo8utyZJkhop+/B+4QU4+GC49tqyK5EkqTk85y1JUmay2/NevRpOPhkeeaR4vGIFTJxYakmSJDVV\nVuG9Zg0ceSTsvDN8+csb148bV15NkiQ1WzbhvWYNHHVUEdxz5sDw4WVXJElSObI4571uHRx9NOyw\ng8EtSVIWe9433wzPPANz5xrckiRlsee9YQOMHWtwS5IEmYS3JEnayPCWJCkzhrckSZkxvCVJyozh\nLUlSZgxvSZIyY3hLkpQZw1uSpMwY3pIkZcbwliQpM1WFd0R8PCIejoi1EbEkIg6qcrs9ImJlRKwc\nXJmSJKlLv+EdEccCFwFfBCYCi4EbImKnfrbbAvgBsABIgy9VkiRBdXveZwCXp5S+nVK6L6X0SWAp\ncEo/250H3AVcC8TgypQkSV02Gd6Vvef9gXk9npoHTN7Edu8E3gmchsEtSVJd9Xc/71HAcGBZj/XL\ngTG9bRAROwCzgXenlNZEmN2SJNVTf+FdiyuBWSmlOwbzIt/4Bvz2t8Xvjz4KI0fWoTJJkoaA/sJ7\nBbABGN1j/WiK8969OQw4JCLOrTwOYFhE/B04JaX0rZ4bTJs27cXfOzo66Ojo4DvfgSOOgN12K9bv\nu28/lUqS1II6Ozvp7Oys62tGSpseCB4RtwF3p5RO7rbufuDalNJZvbSf0GPVu4GzgEnAX1JKz/Ro\nn3qrYdIkuOSS4qckSUNFRJBSGtQ55WoOm88AroyI2ykuE/t3ivPdl1aKmA5MSim9DSCl9KceRb4R\neKHnekmSVJt+wzuldE1EbAucDYwF7gGmppQeqzQZA4zr72UGVaUkSXpRv4fNG16Ah80lSW2kHofN\nndtckqTMGN6SJGXG8JYkKTOGtyRJmWnEDGs1W70ausaubdhQbi2SJLWqlgnvhQvh0EPhZS8rHm+2\nGWyzTbk1SZLUilomvFetgilTYO7csiuRJKm1ec5bkqTMGN6SJGXG8JYkKTOGtyRJmTG8JUnKjOEt\nSVJmDG9JkjJjeEuSlBnDW5KkzBjekiRlxvCWJCkzhrckSZkxvCVJyozhLUlSZgxvSZIyY3hLkpQZ\nw1uSpMwY3pIkZcbwliQpM4a3JEmZMbwlScqM4S1JUmYMb0mSMmN4S5KUGcNbkqTMGN6SJGXG8JYk\nKTOGtyRJmTG8JUnKjOEtSVJmDG9JkjJjeEuSlBnDW5KkzBjekiRlxvCWJCkzhrckSZkxvCVJyozh\nLUlSZgxvSZIyY3hLkpQZw1uSpMwY3pIkZcbwliQpM4a3JEmZMbwlScqM4S1JUmYMb0mSMmN4S5KU\nGcNbkqTMVB3eEfHxiHg4ItZGxJKIOGgTbTsi4scR8ZeIWB0Rd0fECfUpWZKk9lZVeEfEscBFwBeB\nicBi4IaI2KmPTQ4E7gbeC+wJzAJmR8Rxg65YkqQ2Fyml/htF/Aa4K6V0crd19wPXpZQ+W9UbRVwN\nDE8pHdNjfUopMXcuXHQRzJ07sA8gSVJOIoKUUgzmNfrd846ILYD9gXk9npoHTB7Ae70SeGoA7SVJ\nUi82q6LNKGA4sKzH+uXAmGreJCKOBN7CwMJekiT1ouGjzSPizcD3gNNSSksa/X6SJA111ex5rwA2\nAKN7rB8NLN3UhpUR6T8HzkkpfbOvdtOmTePBB+HBB6Gzs4OOjo4qypIkqfV1dnbS2dlZ19esdsDa\nbcDdvQxYuzaldFYf2xwC/Az4XErpok28tgPWJEltox4D1qrZ8waYAVwZEbdTXCb27xTnuy+tFDId\nmJRSelvlcQfFHvc3gO9HRNe58Q0ppScHU7AkSe2uqvBOKV0TEdsCZwNjgXuAqSmlxypNxgDjum3y\nEWBL4MzK0uWRHu0kSdIAVbvnTUppFsVkK709d0Ivj51RTZKkBnBuc0mSMmN4S5KUGcNbkqTMGN6S\nJGWmJcJ7+XJ45pmyq5AkKQ9VjzZvpL32Kn4ec8ym20mSpCpnWGtoAZUZ1iRJagdNuSWoJElqLYa3\nJEmZMbwlScqM4S1JUmYMb0mSMmN4S5KUGcNbkqTMGN6SJGXG8JYkKTOGtyRJmTG8JUnKjOEtSVJm\nDG9JkjJjeEuSlBnDW5KkzBjekiRlxvCWJCkzhrckSZkxvCVJyozhLUlSZgxvSZIyY3hLkpQZw1uS\npMwY3pIkZcbwliQpM4a3JEmZMbwlScqM4S1JUmYMb0mSMmN4S5KUGcNbkqTMGN6SJGXG8JYkKTOG\ntyRJmTG8JUnKjOEtSVJmDG9JkjJjeEuSlBnDW5KkzBjekiRlxvCWJCkzhrckSZkxvCVJyozhLUlS\nZgxvSZIyY3hLkpQZw1uSpMwY3pIkZcbwliQpM1WFd0R8PCIejoi1EbEkIg7qp/3eEbEgItZExOMR\ncU59ypUkSf2Gd0QcC1wEfBGYCCwGboiInfpovzVwE7AUOAA4HTgzIs6oV9GSJLWzava8zwAuTyl9\nO6V0X0rpkxTBfEof7T8AbAl8JKX0p5TS9cB5lddRCTo7O8suYcizj5vDfm48+zgPmwzviNgC2B+Y\n1+OpecDkPjY7EPh1Sun5Hu13iIhdai1UtfN/xsazj5vDfm48+zgP/e15jwKGA8t6rF8OjOljmzG9\ntF/W7TlJkjQIjRhtnhrwmpIkqSJS6jtrK4fNVwPvr5y77lo/E5iQUjqsl22+A2ybUjqy27pJwG+A\n3VJK/9ejvWEvSWorKaUYzPab9fPif4uI3wJvB67v9tQU4No+NrsVOC8iRnQ77z0FeKJncFfeY1Af\nQJKkdlPNYfMZwPERcWJEvCEiLqY4d30pQERMj4j53dpfBawBroiIPSPiPcBnKq8jSZIGaZN73gAp\npWsiYlvgbGAscA8wNaX0WKXJGGBct/bPRcQUYCawBHgKuCCldGG9i5ckqR1t8py3JElqPQ2f29yp\nVRtvIH0cER0R8eOI+EtErI6IuyPihGbWm6OB/jvutt0eEbEyIlY2usbc1dLHEfGpiPhzRKyr/Jue\n3oxac1bD3+SpEXFbRDwXEU9GxI8iYo9m1ZuTiDgkIn5Sya4XIuIjVWxTU+Y1NLydWrXxBtrHFJPo\n3A28F9gTmAXMjojjmlBulmro467ttgB+ACzASyg3qZY+jogZFDM9ngmMB46g6Gv1oYa/ya8FfgR0\nVtq/jWIGzV80o94MjQR+T5Fda+nn//tBZV5KqWELxeVh3+yx7n7gS320PwV4BhjRbd1ZwOONrDPn\nZaB93MdrXA1cV/ZnadWl1j4GLgS+DXwEWFn252jlpYa/Fa8H/ga8vuzac1pq6OdjgPVUTrFW1h0G\nvAC8uuzP08oLsBL4cD9tas68hu15O7Vq49XYx715JcXAQvVQax9HxDuBdwKnAV4OuQk19vHRwEPA\n1Ih4qHIY+IqI2K6BpWatxn5eBKwCToqI4RGxFXA8cHtKyb8Zg1dz5jXysLlTqzZeLX38EhFxJPAW\nYHZ9SxsyBtzHEbEDRX9+IKW0prHlDQm1/DseB+wCvA/4MPAhikPnP40Ivyz1bsD9nFJaCkylOMy+\njmIvcU/gqMaV2VZqzryGD1gbIM8LNlFEvBn4HnBaSmlJ2fUMIVcCs1JKd5RdyBA2DBgBfCiltDCl\ntJAiwN9Ice5QdRAR4yjOeV9O0a8dFIeDr/FLUl3UnHmNDO8VwAZgdI/1oylOzvfmr/zjt43R3Z7T\nS9XSxwBURpj+AjgnpfTNxpQ3JNTSx4cB50bE3yPi78C3gJGVxx9rXKnZqqWPlwLrU0oPdlv3YOV1\ndq57hUNDLf18MvBYSukzKaW7U0q/Bj4IHEpxyFeDU3PmNSy8U0p/A7qmVu1uCsUIx97cChwcESN6\ntO91atV2V2MfExGHUAT3uSmlrzWuwvzV2Md7Aft2Wz5HMfJ0X+C6xlSarxr7eCGwWWXPsMs4isPC\n/q3oRY39HBSD07rretxqR25zVHvmNXi03fuA54ETgTcAFwPPATtVnp8OzO/WfmuKb4Dfpziv8h7g\nWeA/yh452KpLDX3cQXGzmfMovuGNqSzblf1ZWnUZaB/3sv3xONq8rn1MESpL2HgJ034Ul4ktLvuz\ntPJSQz8fRLG3fg6wB8WAt7nAI8DLyv48rbZQXCo2sbKsrvTbxEZkXjM+zCnAwxSDHe4ADur23OXA\nQz3a71X5n3At8ATFYd3S/6O08jKQPq483kDx7bn78lCz685pGei/4x7bHg88V/ZnaPWlhr8VY4Br\nKuGzjGKsgV9C69/Px1S+KK2s9POPgPFlf45WXCh2jrr+pnb/OztnE/1bU+Y5PaokSZnxnIUkSZkx\nvCVJyozhLUlSZgxvSZIyY3hLkpQZw1uSpMwY3pIkZcbwliQpM4a3JEmZ+f9GOky3IBqCYAAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1969a748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC curve did not change\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob_new)\n",
    "plt.plot(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.838692434211\n"
     ]
    }
   ],
   "source": [
    "# AUC did not change\n",
    "print metrics.roc_auc_score(y_test, y_pred_prob_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84782357185298385"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate cross-validated AUC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84604951475539703"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add Fare to the model\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S', 'Fare']\n",
    "X = titanic[feature_cols]\n",
    "\n",
    "# recalculate AUC\n",
    "cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}