Week12_Supervised_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 12 - Supervised Learning \n",
    "\n",
    "This week will be a bit odd as you will be doing the tutorials alongside the lectures. Between this tutorial and the exam review there will not be a lot of time during the tutorial for me to teach the basics and then have you practice coding. For this reason I have provided links to online tutorials to learn how to code some basic classifiers by yourself and I will go through the confusion matrix in the tutorials (on the board). The confusion matrix is something you can do without a computer that is relatively easy to grasp without having gone to the lectures first. I encourage you to revisit links to the online classifier tutorials after the week 12 lectures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn import neighbors\n",
    "from matplotlib.colors import ListedColormap\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Algorithms "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Intro to Machine Learning](https://scikit-learn.org/stable/tutorial/basic/tutorial.html)\n",
    "\n",
    "In general, a learning problem considers a set of n samples of data and then tries to predict properties of unknown data. If each sample is more than a single number and, for instance, a multi-dimensional entry (aka multivariate data), it is said to have several attributes or features.\n",
    "\n",
    "Learning problems fall into a few categories:\n",
    "\n",
    "- **Supervised learning**, in which the data comes with additional attributes that we want to predict [Click here to go to the scikit-learn supervised learning page](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning).This problem can be either:\n",
    "\n",
    " - **Classification**: samples belong to two or more classes and we want to learn from already labeled data how to predict the class of unlabeled data. An example of a classification problem would be handwritten digit recognition, in which the aim is to assign each input vector to one of a finite number of discrete categories. Another way to think of classification is as a discrete (as opposed to continuous) form of supervised learning where one has a limited number of categories and for each of the n samples provided, one is to try to label them with the correct category or class.\n",
    " - **Regression**: if the desired output consists of one or more continuous variables, then the task is called regression. An example of a regression problem would be the prediction of the length of a salmon as a function of its age and weight.\n",
    "\n",
    "- **Unsupervised learning**, in which the training data consists of a set of input vectors x without any corresponding target values. The goal in such problems may be to discover groups of similar examples within the data, where it is called clustering, or to determine the distribution of data within the input space, known as density estimation, or to project the data from a high-dimensional space down to two or three dimensions for the purpose of visualization [Click here to go to the Scikit-Learn unsupervised learning page](https://scikit-learn.org/stable/unsupervised_learning.html#unsupervised-learning).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training set and testing set\n",
    "\n",
    "Machine learning is about learning some properties of a data set and then testing those properties against another data set. A common practice in machine learning is to evaluate an algorithm by splitting a data set into two. We call one of those sets the **training set**, on which we learn some properties; we call the other set the **testing set**, on which we test the learned properties.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn about classifiers will will be making use of the **sklearn library**. This library has some awesome tutorials and example datasets. One such dataset is the **Iris dataset**. The Iris flower data set or Fisher's Iris data set is a multivariate data set introduced by the British statistician and biologist Ronald Fisher in his 1936 paper *The use of multiple measurements in taxonomic problems as an example of linear discriminant analysis*. The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in the dataset\n",
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "# uncomment the line below to see the data\n",
    "#iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can perform a PCA to get a look at the data \n",
    "X = iris.data\n",
    "y = iris.target\n",
    "target_names = iris.target_names\n",
    "\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X)\n",
    "\n",
    "print('explained variance ratio (first two components): %s'\n",
    "      % str(pca.explained_variance_ratio_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting the PCA\n",
    "plt.figure()\n",
    "colors = ['navy', 'turquoise', 'darkorange']\n",
    "lw = 2\n",
    "\n",
    "for color, i, target_name in zip(colors, [0, 1, 2], target_names):\n",
    "    plt.scatter(X_pca[y == i, 0], X_pca[y == i, 1], color=color, alpha=.8, lw=lw,\n",
    "                label=target_name)\n",
    "plt.legend(loc='best', shadow=False, scatterpoints=1)\n",
    "plt.title('PCA of IRIS dataset')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes\n",
    "[Gaussian Naive Bayes](https://scikit-learn.org/stable/modules/naive_bayes.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy and paste the code from this tutorial and see if you can perform Gaussian Naive Bayes classification on the Iris dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K Nearest Neighbours\n",
    "\n",
    "Sample usage of [Nearest Neighbors classification](https://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html#sphx-glr-auto-examples-neighbors-plot-classification-py). It will plot the decision boundaries for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy and paste the code from this tutorial and see if you can perform Nearest Neighbours classification on the Iris dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machine\n",
    "Comparison of different [linear SVM classifiers](https://scikit-learn.org/stable/auto_examples/svm/plot_iris_svc.html) on a 2D projection of the iris dataset. We only consider the first 2 features of this dataset:\n",
    "\n",
    "- Sepal length\n",
    "- Sepal width\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy and paste the code from this tutorial and see if you can perform SVM classification on the Iris dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix Generation \n",
    "\n",
    "By definition a confusion matrix $C$ is such that $C_ij$ is equal to the number of observations known to be in group $i$ but predicted to be in group $j$.\n",
    "\n",
    "Which is fancy jargon for this guy: \n",
    "\n",
    "<img src = \"./img/CM.png\"  width= \"500\"/>\n",
    "\n",
    "Which we can use to derive information about how well our classifiers performed\n",
    "\n",
    "<img src = \"./img/errorTypes.png\" width= \"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, lets say we have used one of the classifiers listed above on our training data and now we will asses how well it did by testing it on a separate dataset with 10 samples which results in the following confusion matrix: \n",
    "\n",
    "<img src = \"./img/CM2.png\" width= \"400\"/>\n",
    "\n",
    "In python, you can code this with the following: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "#y_true = [\"T\", \"T\", \"T\", \"T\", \"F\", \"F\", \"F\", \"F\", \"F\", \"F\"]\n",
    "#y_pred = [\"T\", \"T\", \"T\", \"F\", \"T\", \"T\", \"F\", \"F\", \"F\", \"F\"]\n",
    "tn, fp, fn, tp = confusion_matrix([\"T\", \"T\", \"T\", \"T\", \"F\", \"F\", \"F\", \"F\", \"F\", \"F\"],[\"T\", \"T\", \"T\", \"F\", \"T\", \"T\", \"F\", \"F\", \"F\", \"F\"] ).ravel()\n",
    "(tn, fp, fn, tp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy\n",
    "\n",
    ">How often is the classifier correct?\n",
    "\n",
    "$ACC = \\frac{TP + TN}{TP + TN + FP + FN}$\n",
    "\n",
    "Given the above equation, what is the accuracy of your classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision \n",
    "\n",
    ">When it predicts the positive result, how often is it correct?\n",
    "\n",
    "$PPV = \\frac{TP}{TP + FP}$\n",
    "\n",
    "Given the above equation, what is the precision of your classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall (True Positive Rate)\n",
    "\n",
    "> When it is actually the positive result, how often does it predict correctly?\n",
    "\n",
    "$TPR = \\frac{TP}{TP + FN}$\n",
    "\n",
    "Given the above equation, what is the recall of your classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### False Positive Rate\n",
    "\n",
    "> When it predicts a positive result, how often is it incorrect?\n",
    "\n",
    "$FPR = \\frac{FP}{FP + TN}$\n",
    "\n",
    "Given the above equation, what is the FPR of your classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F1 score\n",
    "\n",
    "> This is just the harmonic mean of precision and recall\n",
    "\n",
    "$F_1 =  \\frac{2*TP}{2*TP + FP + FN}$\n",
    "\n",
    "Given the above equation, what is the F1 score of your classifier? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hint at some helpful functions \n",
    "from sklearn.metrics import classification_report"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}