BinaryClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Classification\n",
    "\n",
    "In this tutorial we will explore how to perform binary classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General Imports\n",
    "\n",
    "Import the Pandas library to load and manipulate tabular datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Classification\n",
    "Load the dataset from the locally available file `SAHeart.csv`, which contains a retrospective sample of males in a heart-disease high-risk region\n",
    "of the Western Cape, South Africa. A description of the data is available here: https://web.stanford.edu/~hastie/ElemStatLearn/datasets/SAheart.info.txt\n",
    "\n",
    "The result of loading the data using the Pandas `read_csv` function is a Pandas `DataFrame` named `heart`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>famhist</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>1</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>1</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sbp  tobacco   ldl  adiposity  famhist  typea  obesity  alcohol  age  chd\n",
       "0  160    12.00  5.73      23.11        1     49    25.30    97.20   52    1\n",
       "1  144     0.01  4.41      28.61        0     55    28.87     2.06   63    1\n",
       "2  118     0.08  3.48      32.28        1     52    29.14     3.81   46    0\n",
       "3  170     7.50  6.41      38.03        1     51    31.99    24.26   58    1\n",
       "4  134    13.60  3.50      27.78        1     60    25.99    57.34   49    1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heart = pd.read_csv('SAheart.csv', sep=',', header=0)\n",
    "heart.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Separate the columns of the `heart` Pandas `DataFrame` into:\n",
    "- `X`: the feature variables, columns 0 to 9 (`:9`) (sbp, tobacco, ldl, adiposity, famhist, typea, obesity, alcohol, age)\n",
    "- `y`: the response variable, column `9` (chd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = heart.iloc[:,:9]\n",
    "y = heart.iloc[:,9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the rows that we will use to *quickly* check the prediction of the classification methods (`460:`, i.e., rows from 460 to the end of the DataFrame)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>famhist</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>460</th>\n",
       "      <td>118</td>\n",
       "      <td>5.4</td>\n",
       "      <td>11.61</td>\n",
       "      <td>30.79</td>\n",
       "      <td>0</td>\n",
       "      <td>64</td>\n",
       "      <td>27.35</td>\n",
       "      <td>23.97</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.82</td>\n",
       "      <td>33.41</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>14.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sbp  tobacco    ldl  adiposity  famhist  typea  obesity  alcohol  age  \\\n",
       "460  118      5.4  11.61      30.79        0     64    27.35    23.97   40   \n",
       "461  132      0.0   4.82      33.41        1     62    14.70     0.00   46   \n",
       "\n",
       "     chd  \n",
       "460    0  \n",
       "461    1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heart.iloc[460:,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import `LogisticRegression` from the SciKit-Learn library to perform logistic regression.\n",
    "More details about this can be found online in the documentation for SciKit-Learn: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `LogisticRegression` object to configure how to perform the logistic regression.\n",
    "\n",
    "The `random_state` parameter makes sure we get the same exact results every time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "LR = LogisticRegression(random_state=0, solver='lbfgs', multi_class='ovr', max_iter=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the dataset in the `X` and `y` sub-dataframes to train and create the logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(max_iter=200, multi_class='ovr', random_state=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LR.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quickly test the model predictions for the last two rows in the dataset (rows 460 and 461). It seems that the first prediction is incorrect while the second is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LR.predict(X.iloc[460:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the model performance by asking for the 0 to 1 `score` executed on the whole of the *training* dataset. \n",
    "Normally, two different sets of data are used for the purpose of training the model and the evaluation of the model, but here we keep it simple and use the same dataset for both training and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7338"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(LR.score(X,y), 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the general structure of the previous section, we now evaluate an SVM-based classifier.\n",
    "More details here: https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC\n",
    "\n",
    "We import the classifier class, create the SVM classifier `SVM` object to configure the SVM method (here with no parameters), fit the data to create the model, and show the quick prediction.\n",
    "\n",
    "*NOTE: Disregard the warning produced.*\n",
    "\n",
    "It seems that the first prediction is incorrect while the second is correct here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/emilie/anaconda3new/lib/python3.8/site-packages/sklearn/svm/_base.py:976: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\"Liblinear failed to converge, increase \"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([1, 1])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "SVM = svm.LinearSVC(random_state=0)\n",
    "SVM.fit(X, y)\n",
    "SVM.predict(X.iloc[460:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the model performance by asking for the 0 to 1 `score` executed on the whole of the *training* dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5411"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(SVM.score(X,y), 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forrest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the general structure of the previous section, we now evaluate a Random Forrest-based classifier.\n",
    "More details here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier\n",
    "\n",
    "\n",
    "We import the classifier class, create the Random Forrest classifier `RF` object to configure the Random Forrest method, fit the data to create the model, and show the quick prediction.\n",
    "\n",
    "The prediction is correct for the first row but incorrect for the second here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "RF = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)\n",
    "RF.fit(X, y)\n",
    "RF.predict(X.iloc[460:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the model performance by asking for the 0 to 1 `score` executed on the whole of the *training* dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7338"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(RF.score(X,y), 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*What would happen if a larger `max_depth` value were used?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network / Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the general structure of the previous section, we now evaluate a Neural Network-based classifier, specifically a Multi-layer Perceptron.\n",
    "More details here: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html\n",
    "\n",
    "We import the classifier class, create the Neural Network classifier `NN` object to configure the MLP method, fit the data to create the model, and show the quick prediction.\n",
    "\n",
    "The prediction is correct for the first row but incorrect for the second here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "NN = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=0)\n",
    "NN.fit(X, y)\n",
    "NN.predict(X.iloc[460:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the model performance by asking for the 0 to 1 `score` executed on the whole of the *training* dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.658"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(NN.score(X,y), 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "The methods evaluated here all have different performance and different advantages and disadvantages, but using Pandas and SciKit-Learn their use and evaluation is very similar and follows the same pattern of code:\n",
    "1. Import classifier\n",
    "2. Create a classifer object with optional configuration parameters\n",
    "3. Train the model on some data\n",
    "4. Evaluate the model on some data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}