W3-2 Testing Paradigms.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The question here is: Given that the performance of classifiers depends on how we randomly divide up our corpus, how can we decide which classifier is doing a better job?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### About overfitting and underfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We could often train a classifer that does perfectly on the training set. In that case, however, the classifier would be attending overly to the idiosyncracies of the training set. \n",
    "\n",
    "One way this could happen is if we include too many features in our featuresets.\n",
    "\n",
    "So we want to train in such a way that we pay just the right amount of attention to features in the training set - not too much and not too little."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Cross-validation\n",
    "\n",
    "In cross-validation, we shuffle the corpus, then divide it up into even-sized chunks. Then we use each one of the chunks as the test set and we average the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://scikit-learn.org/stable/_images/grid_search_cross_validation.png\" width=\"442px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Create all of the functions we'll need\n",
    "\n",
    "Mostly, I'm bottling up everything we did above. \n",
    "\n",
    "But I'm also adding a wrapper that does the cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import random\n",
    "import sklearn\n",
    "from sklearn.metrics import cohen_kappa_score\n",
    "\n",
    "# This reads in the raw data, shuffles it, and crates the feature sets.\n",
    "def prepare_feature_sets(pass_feature_func):\n",
    "    csvfile = open(\"corpora/titanic/train.csv\")\n",
    "    dlist = list(csv.DictReader(csvfile))\n",
    "    random.shuffle(dlist)\n",
    "    labeled_feature_sets = [(pass_feature_func(r), r[\"Survived\"]) for r in dlist]\n",
    "    return labeled_feature_sets\n",
    "\n",
    "# This trains and tests a classifier given the training set and the test set.\n",
    "def test_classifier(train_set, test_set):\n",
    "    classif = nltk.NaiveBayesClassifier.train(train_set)\n",
    "    accuracy = nltk.classify.accuracy(classif, test_set)\n",
    "    gold_list = [t[1] for t in test_set]\n",
    "    test_list = [classif.classify(t[0]) for t in test_set]\n",
    "    kappa = cohen_kappa_score(gold_list, test_list)\n",
    "    return {\"accuracy\": accuracy, \"kappa\": kappa}\n",
    "\n",
    "# This does the who cross validation thing.\n",
    "def cross_validate(nchunks, labeled_featuresets):\n",
    "    test_fraction = 1 / nchunks\n",
    "    test_size = round(test_fraction * len(labeled_featuresets))\n",
    "    chunk_starts = [n * test_size for n in range(nchunks)]\n",
    "    random.shuffle(labeled_featuresets)\n",
    "    data_chunks = [labeled_featuresets[start:start+test_size] for start in chunk_starts]\n",
    "    test_set = None\n",
    "    train_set = []\n",
    "    results = []\n",
    "    for n in range(nchunks):\n",
    "        for m, c in enumerate(data_chunks):\n",
    "            if m == n:\n",
    "                test_set = c\n",
    "            else:\n",
    "                train_set += c\n",
    "        res = test_classifier(train_set, test_set)\n",
    "        # print(res)\n",
    "        results.append(res)\n",
    "    average_accuracy = sum([res[\"accuracy\"] for res in results]) / len(results)\n",
    "    average_kappa = sum([res[\"kappa\"] for res in results]) / len(results)\n",
    "    print(\"average accuracy: {}, average kappa: {}\".format(round(average_accuracy, 3), round(average_kappa, 3)))\n",
    "    \n",
    "    return results, average_accuracy, average_kappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can systematically compare the different feature sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average accuracy: 0.778, average kappa: 0.526\n"
     ]
    }
   ],
   "source": [
    "def passenger_features(r):\n",
    "    return {\"sex\": r[\"Sex\"], \"pclass\": r[\"Pclass\"], \"embarked\": r[\"Embarked\"]}\n",
    "\n",
    "results, average_accuracy, average_kappa = cross_validate(10, prepare_feature_sets(passenger_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average accuracy: 0.788, average kappa: 0.545\n"
     ]
    }
   ],
   "source": [
    "def sex_features(r):\n",
    "    return {\"sex\": r[\"Sex\"]}\n",
    "results, average_accuracy, average_kappa = cross_validate(10, prepare_feature_sets(sex_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average accuracy: 0.787, average kappa: 0.536\n"
     ]
    }
   ],
   "source": [
    "def passenger_features(r):\n",
    "    return {\"sex\": r[\"Sex\"], \"pclass\": r[\"Pclass\"]}\n",
    "\n",
    "results, average_accuracy, average_kappa = cross_validate(10, prepare_feature_sets(passenger_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}