3 - Classification Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Models\n",
    "Previous notebooks have focused on regression models, but another *very* import family of machine learning models are *classification models*. In this notebook we will introduce classification models and fit a range of classification models to some toy data.\n",
    "\n",
    "First, let's import the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toy classification data\n",
    "Let's load a simple toy dataset and print it's head."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_classed = pd.read_csv('Data/classed.csv', sep=',', header=0)\n",
    "df_classed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, there are two attribute variables denoted `x1` and `x2` and a target variable `y`, similar to before, but for this data the target variable `y` takes on discrete variables $[1,2,3,..]$. \n",
    "\n",
    "Let's look at the unique values of `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_classed['y'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These unique values correspond to the *classes* of the target variable `y`. \n",
    "\n",
    "Here they are just numbers, but in a real data set those numbers should correspond to meaningful groups of data.\n",
    "\n",
    "Given class values, we can plot the attributes and groups as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_classed.plot.scatter(x='x1',y='x2',c=df_classed.y.map({1:'b', 2:'r', 3:'g'}));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you see the different groups?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear classification model\n",
    "Let's now explore how we might fit a classification model to the data.\n",
    "\n",
    "As with a regression model, we need to prepare the data so that it is formatted correctly. \n",
    "\n",
    "As with a regression model, we do this by extracting the target variable and attribute matrix from the dataframe, and we will also normalise the attribute matrix to have columns of zero mean and unit variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_classed['y']\n",
    "X = df_classed.drop('y', axis=1)\n",
    "X=(X-X.mean())/X.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first classification model we will consider is the `LogisticRegression` model. \n",
    "\n",
    "Here, we will create an instance of a `LogisticRegression` model, fit it to the data, and visualise the results. \n",
    "\n",
    "A function is provided to make plotting simpler. The function `plot_results` creates a meshgrid (similar to how we created a linear space for 1D regression) and then computes the class of each point guven the classifier passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit a logistic Regreesion model to the data\n",
    "clf = LogisticRegression().fit(X, y)\n",
    "\n",
    "# Visualise the results\n",
    "def plot_results(X, y, clf):\n",
    "    xx, yy = np.meshgrid(np.arange(X['x1'].min() - .5, X['x1'].max() + .5, 0.02),\n",
    "                         np.arange(X['x2'].min() - 1.5, X['x2'].max() + 1.5, 0.02))\n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "    _, ax = plt.subplots()\n",
    "    ax.contourf(xx, yy, Z.reshape(xx.shape), cmap=plt.cm.brg, alpha=.3)\n",
    "    ax.scatter(X['x1'],X['x2'],c=df_classed.y.map({1:'b', 2:'r', 3:'g'}));\n",
    "\n",
    "plot_results(X, y, clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above plot shows the original data points as well as the *decision boundaries* of the model for each of the classes.\n",
    "\n",
    "`LogisticRegression` is a *linear* classification model and ,as a result it splits the attribute/feature space into sections that look like the above plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other classification models\n",
    "Let's now explore some other classification models and use the plotting function to visualise the *decision boundaries* of each classfification model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-Nearest Neighbour classification model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_results(X, y, KNeighborsClassifier().fit(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest Classifier model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_results(X, y, RandomForestClassifier().fit(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural Network model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_results(X, y, MLPClassifier(max_iter=100).fit(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a warning. Let's try with more interations..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_results(X, y, MLPClassifier(max_iter=100000).fit(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which model look like the best model?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}