03-LDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quadratic Discriminant Analysis\n",
    "\n",
    "In this exercise, you will implement a QDA on your own. The QDA computes $p(\\omega|x)=\\frac{p(x|\\omega)*p(\\omega)}{p(x)}$, and assumes that the likelihood $p(x|\\omega)$ follows a normal distribution (conditional on each class $\\omega$). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1\n",
    "A fisher asks you for help in classifying fish. Recently, he catched the following fish:\n",
    "\n",
    "| Length (m)    | Type          | \n",
    "| ------------- |-------------  |\n",
    "| 1.3           | sea bass       |\n",
    "| 0.7           | salmon       |\n",
    "| 0.62           | salmon      |\n",
    "| 0.9           | salmon       |\n",
    "| 0.91          | sea bass       |\n",
    "| 0.31          | herring       |\n",
    "| 0.26           | herring       |\n",
    "\n",
    "* Calculate the class priors $p(\\omega)$ for each class of fish\n",
    "* Calculate the parameters $\\mu$ and $\\sigma^2$ for the likelihoods $p(x|\\omega)$. \n",
    "* The fisher catches a new fish with length $x = 0.82 m$. Calculate the posteriori probability $p(\\omega|x)$ for each class. Which class is most likely?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Priors: \n",
    "* $p(sea bass) = 2/7$\n",
    "* $p(salmon) = 3/7$\n",
    "* $p(herring) = 2/7$\n",
    "\n",
    "Likelihoods:\n",
    "* $\\mu_{sb} = 1.105 \\sigma^2_{sb} = 0.076$\n",
    "* $\\mu_{sa} = 0.74 \\sigma^2_{sa} = 0.021$\n",
    "* $\\mu_{h} = 0.285 \\sigma^2_{h} = 0.00125$\n",
    "\n",
    "A posteriori (without normalizing constant):\n",
    "\n",
    "* $p(sea bass | x) = 0.0046 * 2/7 =  0.0013$\n",
    "* $p(salmon | x) = 0.134 * 3/7 =  0.0038$\n",
    "* $p(herring | x) = 0 * 2/7 =  0$\n",
    "\n",
    "Thus, the fish is most likely a salmon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise \n",
    "\n",
    "Implement a function `priors(classes)` that calculates a prior $p(x)$ for each class, given a vector of class labels.\n",
    "The input should be an array of classes, e.g. `np.array([\"stand\",\"sit\",\"sit\",\"stand\"])`. The output should be a pandas data frame with columns `class` and `prior`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>prior</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b'bike'</td>\n",
       "      <td>0.257062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b'downstairs'</td>\n",
       "      <td>0.0819209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b'lie'</td>\n",
       "      <td>0.0960452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b'run'</td>\n",
       "      <td>0.163842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b'sit'</td>\n",
       "      <td>0.0621469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b'stand'</td>\n",
       "      <td>0.0451977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>b'upstairs'</td>\n",
       "      <td>0.0762712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b'walk'</td>\n",
       "      <td>0.217514</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           class      prior\n",
       "0        b'bike'   0.257062\n",
       "1  b'downstairs'  0.0819209\n",
       "2         b'lie'  0.0960452\n",
       "3         b'run'   0.163842\n",
       "4         b'sit'  0.0621469\n",
       "5       b'stand'  0.0451977\n",
       "6    b'upstairs'  0.0762712\n",
       "7        b'walk'   0.217514"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.io import arff\n",
    "import scipy.stats\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "def priors(classes):\n",
    "    unique, counts = np.unique(classes, return_counts=True)\n",
    "    counts = counts/counts.sum()\n",
    "    df = pd.DataFrame(np.array([unique,counts]).transpose(),columns=[\"class\",\"prior\"])\n",
    "    return(df)\n",
    "    \n",
    "pp = priors(np.array([\"stand\",\"sit\",\"sit\",\"sit\",\"stand\"]))\n",
    "\n",
    "\n",
    "data = arff.loadarff('features1.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "dat = df.loc[:, [\"AccX_mean\",\"class\"]]\n",
    "dat.columns = [\"x\",\"class\"]\n",
    "\n",
    "prior = priors(dat[\"class\"])\n",
    "prior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Implement a function `likelihood(data)`, that takes a data frame consisting of a column $omega$ and a column $x$, and estimates the parameters of the likelihood $p(x|\\omega)$ for each class $\\omega$, i.e. that returns a mean and a variance for each class. Thus, the output should be a data frame with columns `omega`, `mean` and `variance`. \n",
    "\n",
    "Plot the likelihoods for each class. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>variance</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>523.545587</td>\n",
       "      <td>15.843081</td>\n",
       "      <td>b'bike'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>517.316002</td>\n",
       "      <td>18.523919</td>\n",
       "      <td>b'downstairs'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>467.870175</td>\n",
       "      <td>56.697558</td>\n",
       "      <td>b'lie'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>527.009294</td>\n",
       "      <td>96.483175</td>\n",
       "      <td>b'run'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>486.721946</td>\n",
       "      <td>71.790221</td>\n",
       "      <td>b'sit'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>528.594727</td>\n",
       "      <td>1.057665</td>\n",
       "      <td>b'stand'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>545.413484</td>\n",
       "      <td>62.520043</td>\n",
       "      <td>b'upstairs'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>525.170962</td>\n",
       "      <td>1.221023</td>\n",
       "      <td>b'walk'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         mean   variance          class\n",
       "0  523.545587  15.843081        b'bike'\n",
       "1  517.316002  18.523919  b'downstairs'\n",
       "2  467.870175  56.697558         b'lie'\n",
       "3  527.009294  96.483175         b'run'\n",
       "4  486.721946  71.790221         b'sit'\n",
       "5  528.594727   1.057665       b'stand'\n",
       "6  545.413484  62.520043    b'upstairs'\n",
       "7  525.170962   1.221023        b'walk'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.io import arff\n",
    "import scipy.stats\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "def likelihood(data):\n",
    "    uc = np.unique(data[\"class\"])\n",
    "    def getMV(c):\n",
    "        dsub = data.loc[data[\"class\"]==c,\"x\"]\n",
    "        return([dsub.mean(),dsub.var()])\n",
    "    mvs = list(map(getMV, uc))\n",
    "    r = pd.DataFrame(mvs,columns = [\"mean\",\"variance\"])\n",
    "    r[\"class\"] = uc\n",
    "    return r\n",
    "    \n",
    "data = arff.loadarff('features1.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "dat = df.loc[:, [\"AccX_mean\",\"class\"]]\n",
    "dat.columns = [\"x\",\"class\"]\n",
    "lik = likelihood(dat)\n",
    "lik"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "Implement a function `myqda(newdat,lik,priors)`, that returns the most likely class for a new observation `newdat`. \n",
    "Test your implementation on the dataset `features1.arff`. Train your QDA (i.e. compute priors and likelihood parameters), and then classify the data. How good is your classification?  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47\n"
     ]
    }
   ],
   "source": [
    "from scipy.io import arff\n",
    "import scipy.stats\n",
    "\n",
    "def mylda(newdat,lik,prior):\n",
    "    def getClass(d):\n",
    "        def getProb(c,d):\n",
    "            pr = prior.loc[prior[\"class\"]==c,\"prior\"].values[0]\n",
    "            m = lik.loc[lik[\"class\"] == c,\"mean\"].values[0]\n",
    "            v = lik.loc[lik[\"class\"] == c,\"variance\"].values[0]\n",
    "            li = scipy.stats.norm(m, v).pdf(d)\n",
    "            return(li*pr)\n",
    "        probs = np.array([getProb(c,d) for c in prior[\"class\"].values])\n",
    "        return prior.loc[np.argmax(probs),\"class\"]\n",
    "    newclasses = np.array([getClass(dat) for dat in newdat])\n",
    "    return newclasses\n",
    "\n",
    "\n",
    "data = arff.loadarff('features1.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "dat = df.loc[:, [\"AccX_mean\",\"class\"]]\n",
    "dat.columns = [\"x\",\"class\"]\n",
    "\n",
    "lik = likelihood(dat)\n",
    "prior = priors(dat[\"class\"])\n",
    "\n",
    "nc = mylda(dat[\"x\"][1:100],lik,prior)\n",
    "print(sum(nc == dat[\"class\"][1:100])/100)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:ki2]",
   "language": "python",
   "name": "conda-env-ki2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}