pandas_sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using pandas and scikit-learn for classification tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "```bash\n",
    "git clone git@github.com:jseabold/depy.git\n",
    "cd depy\n",
    "pip install -r requirements.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About Me\n",
    "\n",
    "Skipper Seabold <br />\n",
    "Data Scientist, [Civis Analytics](https://civisanalytics.com/)\n",
    "\n",
    "<link rel=\"stylesheet\" href=\"//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css\">\n",
    "<img src=https://g.twimg.com/twitter-bird-16x16.png style=\"float:left\"></img> &nbsp; <a href=https://twitter.com/jseabold>@jseabold</a> <br />\n",
    "<i class=\"fa fa-github fa-lg\"></i> &nbsp; <a href=\"https://github.com/jseabold\">jseabold</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "* Hitting the highlights\n",
    "* Pandas for Data Wrangling\n",
    "  * Overview\n",
    "  * Reading data\n",
    "  * Exploration\n",
    "  * GroupBy\n",
    "  * Plotting\n",
    "  * Advanced Indexing\n",
    "  * Categorical Data\n",
    "* Scikit-Learn for Classification\n",
    "  * API Overview\n",
    "  * Preprocessing\n",
    "  * Decision Trees\n",
    "  * Gradient Boosting Trees\n",
    "  * Random Forests\n",
    "  * Cross-Validation\n",
    "  * Early Stopping\n",
    "  * Custom Transformers\n",
    "  * Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Specifics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "pd.set_option(\"max_rows\", 10)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretty Graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from seaborn import set_style\n",
    "set_style(\"darkgrid\")\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(\"data/adult.names\") as fin:\n",
    "    notes = fin.read()\n",
    "    \n",
    "print(notes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas for Data Wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta = pd.read_csv(\"data/adult.data.cleaned.csv.gz\", compression=\"gzip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test = pd.read_csv(\"data/adult.test.cleaned.csv.gz\", compression=\"gzip\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Orientation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.columns.difference(test.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity checks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.columns.equals(test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.columns.difference(test.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[[5, 10, 15]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta[[\"workclass\", \"education\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(dta[[\"workclass\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(dta[\"workclass\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rows and Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[[5, 10, 15], [\"workclass\", \"education\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GroupBy Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.groupby(\"income\").education.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grouper = dta.groupby(\"education\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "grouper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "education_map = grouper.education_num.unique()\n",
    "education_map.sort_values(inplace=True)\n",
    "\n",
    "with pd.option_context(\"max_rows\", 20):\n",
    "    print(education_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "grouper.education_num.apply(lambda x : x.unique()[0])\n",
    "education_map.sort_values(inplace=True)\n",
    "\n",
    "with pd.option_context(\"max_rows\", 20):\n",
    "    print(education_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax = dta.groupby(\"education\").size().plot(kind=\"barh\", figsize=(8, 8))\n",
    "\n",
    "# ax.set_xticklabels([])  # turn off x tick labels\n",
    "\n",
    "# resize y label\n",
    "ylabel = ax.yaxis.get_label()\n",
    "ylabel.set_fontsize(24)\n",
    "\n",
    "# resize x tick labels\n",
    "labels = ax.yaxis.get_ticklabels()\n",
    "[label.set_fontsize(20) for label in labels];\n",
    "\n",
    "# resize y tick labels\n",
    "labels = ax.xaxis.get_ticklabels()\n",
    "[label.set_fontsize(20) for label in labels]\n",
    "[label.set_rotation(-45) for label in labels];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "g = sns.factorplot(\"education_num\", \"hours_per_week\", hue=\"sex\", col=\"income\", data=dta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "del dta[\"education\"]\n",
    "del dta[\"fnlwgt\"]\n",
    "del test[\"education\"]\n",
    "del test[\"fnlwgt\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexing with Booleans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.education_num <= 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[dta.education_num <= 8, \"education_num\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### .iloc vs .loc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[dta.education_num <= 8, \"education_num\"].iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[dta.education_num <= 8, \"education_num\"].loc[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing with labels (!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.groupby(\"workclass\").age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.groupby(\"workclass\").age.mean().ix[\"Federal-gov\":\"Private\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filtering Columns with Regex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.filter(regex=\"capital\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Categorical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cat = pd.Categorical(dta.workclass)\n",
    "cat.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vectorized string operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.workclass.str.contains(\"\\?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Putting it together: Strings and Boolean Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.ix[dta.workclass.str.contains(\"\\?\"), \"workclass\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Putting it together: Column Assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.workclass.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for col in dta:  # iterate through column names\n",
    "    # only look at object types\n",
    "    if not dta[col].dtype.kind == \"O\":\n",
    "        continue\n",
    "    \n",
    "    # Replace \"?\" with \"Other\"\n",
    "    if dta[col].str.contains(\"\\?\").any():\n",
    "        dta.ix[dta[col].str.contains(\"\\?\"), col] = \"Other\"\n",
    "        test.ix[test[col].str.contains(\"\\?\"), col] = \"Other\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.workclass.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replacing values using dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.income"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.income.replace({\"<=50K\": 0, \">50K\": 1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-place changes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dta.income.replace({\"<=50K\": 0, \">50K\": 1}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test.income.replace({\"<=50K.\": 0, \">50K.\": 1}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.income.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test.income.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification with Scikit-Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scikit-Learn API\n",
    "\n",
    "* Base object is the estimator\n",
    "* Any object that learns from data\n",
    "  * Classification, regression, clustering, or transformer \n",
    "  \n",
    "* parameters passed to estimator\n",
    "\n",
    "```python\n",
    "    estimator = Estimator(*args, **kwargs)\n",
    "```\n",
    "\n",
    "* `fit` method provided\n",
    "\n",
    "```python\n",
    "    estimator.fit(X, y)\n",
    "```\n",
    "    \n",
    "* Computed parameters have an underscore appended\n",
    "\n",
    "```python\n",
    "    estimator.coef_\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* scikit-learn works with numerical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = dta.pop(\"income\")\n",
    "y_test = test.pop(\"income\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Preprocessing for Text, Categorical variables, Standardization etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelBinarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dta.native_country.head(15).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "binarizer = LabelBinarizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fit_transform` is short hand for calling `fit` then `transform`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "binarizer.fit_transform(dta.native_country.head(15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "binarizer.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pre-processing with pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = pd.get_dummies(dta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test = pd.get_dummies(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deal with real life"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train.columns.equals(X_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train.columns.difference(X_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_test[X_train.columns.difference(X_test.columns)[0]] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preserve order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test = X_test[X_train.columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reported Benchmarks\n",
    "\n",
    "```\n",
    "|    Algorithm               Error\n",
    "| -- ----------------        -----\n",
    "| 1  C4.5                    15.54\n",
    "| 2  C4.5-auto               14.46\n",
    "| 3  C4.5 rules              14.94\n",
    "| 4  Voted ID3 (0.6)         15.64\n",
    "| 5  Voted ID3 (0.8)         16.47\n",
    "| 6  T2                      16.84\n",
    "| 7  1R                      19.54\n",
    "| 8  NBTree                  14.10\n",
    "| 9  CN2                     16.00\n",
    "| 10 HOODG                   14.82\n",
    "| 11 FSS Naive Bayes         14.05\n",
    "| 12 IDTM (Decision table)   14.46\n",
    "| 13 Naive-Bayes             16.12\n",
    "| 14 Nearest-neighbor (1)    21.42\n",
    "| 15 Nearest-neighbor (3)    20.35\n",
    "| 16 OC1                     15.04\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification and Regression Trees (CART)\n",
    "\n",
    "* Partition feature space into a set of rectangles via splits that lead to largest information gain\n",
    "* Fit simple model in each region (e.g., a constant)\n",
    "* Captures non-linearities and feature interactions\n",
    "* Note: not strictly necessary to dummy encode variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree = DecisionTreeClassifier(random_state=0, max_depth=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree.fit(X_train, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "export_graphviz(dtree, feature_names=X_train.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this if you have graphviz installed\n",
    "    \n",
    "    !dot -Tpng tree.dot -o tree.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(\"tree.png\", unconfined=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the full tree and look at the error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree = DecisionTreeClassifier(criterion='entropy', random_state=0)\n",
    "dtree.fit(X_train, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performs slightly worse than C4.5 with no pruning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, dtree.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beware overfitting!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree = DecisionTreeClassifier(criterion='entropy', random_state=0, max_depth=10)\n",
    "dtree.fit(X_train, y)\n",
    "metrics.mean_absolute_error(y_test, dtree.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aside: Saving Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* All of the scikit-learn models are picklable\n",
    "* Using joblib directly is often preferable to using pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import joblib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boosting\n",
    "\n",
    "* Combine many weak classifiers in to one strong one\n",
    "  * Weak classifier is slightly better than random\n",
    "* Sequentially apply a classifier to repeatedly modified versions of data\n",
    "* Each subsequent classifier improves on the mistakes of its predecessor\n",
    "* For Boosting Trees, the classifier is a decision tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a random dataset\n",
    "\n",
    "import numpy as np\n",
    "rng = np.random.RandomState(1)\n",
    "groundX = np.sort(rng.uniform(0, 10, size=250), axis=0)\n",
    "groundy = np.linspace(1, -1, 250) + np.sin(2*groundX).ravel()\n",
    "idx = np.random.randint(0, 250, size=30)\n",
    "idx.sort()\n",
    "XX = groundX[idx]\n",
    "yy = groundy[idx]\n",
    "XX = XX[:, np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "tree1 = DecisionTreeRegressor(max_depth=2)\n",
    "\n",
    "tree1.fit(XX, yy)\n",
    "y1 = tree1.predict(XX)\n",
    "\n",
    "resid1 = yy - y1\n",
    "tree1.fit(XX, resid1)\n",
    "\n",
    "y2 = tree1.predict(XX)\n",
    "resid2 = y2 - resid1\n",
    "tree1.fit(XX, resid2)\n",
    "\n",
    "y3 = tree1.predict(XX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(4, 1, figsize=(6, 18), sharey=True, sharex=True)\n",
    "ax[0].plot(XX, yy, marker='o', ls='', label='observed')\n",
    "ax[0].plot(groundX, groundy, label='truth')\n",
    "ax[0].legend(fontsize=16, loc='lower left');\n",
    "ax[1].plot(XX, yy, marker='o', ls='')\n",
    "ax[1].plot(XX, y1)\n",
    "ax[2].plot(XX, resid1, marker='o', ls='')\n",
    "ax[2].plot(XX, y2)\n",
    "ax[3].plot(XX, resid2, marker='o', ls='')\n",
    "ax[3].plot(XX, y3)\n",
    "fig.suptitle(\"Residual Fitting\", fontsize=24);\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosting\n",
    "\n",
    "* Generalizes boosting to any differentiable loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(\"models/gbt1.pkl\"):\n",
    "    gbt = GradientBoostingClassifier(max_depth=5, n_estimators=1000)\n",
    "    gbt.fit(X_train, y)\n",
    "    joblib.dump(gbt, \"models/gbt1.pkl\")\n",
    "else:\n",
    "    gbt = joblib.load(\"models/gbt1.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, gbt.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(\"models/gbt2.pkl\"):\n",
    "    gbt = GradientBoostingClassifier(max_depth=8, n_estimators=1000, subsample=.5, random_state=0,\n",
    "                                    learning_rate=.001)\n",
    "    gbt.fit(X_train, y)\n",
    "    joblib.dump(gbt, \"models/gbt2.pkl\")\n",
    "else:\n",
    "    gbt = joblib.load(\"models/gbt2.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, gbt.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bagging\n",
    "\n",
    "* Bootstrap aggregating (bagging)\n",
    "  * Simple bootstrapping is sampling with replacement\n",
    "* Fit the same learner to many bootstrap samples and average the results\n",
    "* Random forests builds on the idea of bagging and uses trees\n",
    "* Performance similar to boosting but can be easier to train and tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(\"models/rf.pkl\"):\n",
    "    rf = RandomForestClassifier(n_estimators=1000, criterion='entropy', n_jobs=4, max_depth=10)\n",
    "    rf.fit(X_train, y)\n",
    "    joblib.dump(rf, \"models/rf.pkl\")\n",
    "else:\n",
    "    rf = joblib.load(\"models/rf.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, rf.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Rule of thumb is that you can't really overfit with random forests\n",
    "  * This is true in general but only to an extent\n",
    "* Usually ok to grow large forests with full depth trees (problem dependent)\n",
    "  * Limiting the depth of the trees and the number of trees can be "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(\"models/rf_full.pkl\"):\n",
    "    rf_full = RandomForestClassifier(n_estimators=1000, criterion='entropy', \n",
    "                                     n_jobs=4, max_depth=None)\n",
    "    rf_full.fit(X_train, y)\n",
    "    joblib.dump(rf, \"models/rf_full.pkl\")\n",
    "else:\n",
    "    rf_full = joblib.load(\"models/rf_full.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, rf_full.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Validation\n",
    "\n",
    "* Sampling techniques to ensure low generalization error and avoid overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import StratifiedKFold\n",
    "\n",
    "cv = StratifiedKFold([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
    "                      0, 0, 0, 0, 0, 0,], n_folds=3)\n",
    "for idx in cv:\n",
    "    print(\"train\", idx[0], \"test\", idx[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "cv = StratifiedKFold(y, n_folds=4)\n",
    "\n",
    "params = {\"max_depth\": [3, 5, 7]}\n",
    "gbt = GradientBoostingClassifier(n_estimators=500, learning_rate=.01)\n",
    "\n",
    "if not os.path.exists(\"models/grid_search.pkl\"):\n",
    "    estimator = GridSearchCV(gbt, param_grid=params, verbose=2)\n",
    "    estimator.fit(X_train, y)\n",
    "    joblib.dump(estimator, \"models/grid_search.pkl\")\n",
    "else:\n",
    "    estimator = joblib.load(\"models/grid_search.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Out-of-bag estimates and Early-stopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gbt = GradientBoostingClassifier(learning_rate=.01, n_estimators=1000, subsample=.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gbt.fit(X_train, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metrics.mean_absolute_error(y_test, gbt.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "ax.plot(gbt.oob_improvement_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ad-hoc way to do early-stopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def monitor(i, self, local_variables):\n",
    "    start = max(0, i - 4)\n",
    "    stop = i + 1\n",
    "\n",
    "    if i > 5 and np.mean(self.oob_improvement_[start:stop]) < 1e-4:\n",
    "        print(\"Stopped at {}\".format(i))\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gbt.fit(X_train, y, monitor=monitor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(len(gbt.oob_improvement_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_obj_cols(dta, index=False):\n",
    "    \"\"\"\n",
    "    dta : pd.DataFrame\n",
    "    index : bool\n",
    "        Whether to return column names or the numeric index.\n",
    "        Default False, returns column names.\n",
    "    \"\"\"\n",
    "    columns = dta.columns.tolist()\n",
    "    obj_col_names = list(filter(lambda x : dta[x].dtype.kind == \"O\", \n",
    "                                columns))\n",
    "    if not index:\n",
    "        return obj_col_names\n",
    "    else:\n",
    "        return list(columns.index(col) for col in obj_col_names) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj_cols = get_obj_cols(dta)\n",
    "\n",
    "for col in obj_cols:\n",
    "    print(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a transformer that reliably transforms DataFrames and Arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.base import TransformerMixin, BaseEstimator\n",
    "\n",
    "\n",
    "class PandasTransformer(TransformerMixin, BaseEstimator):\n",
    "    def __init__(self, dataframe):\n",
    "        self.columns = dataframe.columns\n",
    "        self.obj_columns = get_obj_cols(dataframe, index=True)\n",
    "        obj_index = np.zeros(dataframe.shape[1], dtype=bool)\n",
    "        obj_index[self.obj_columns] = True\n",
    "        self.obj_index = obj_index\n",
    "        \n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        X = np.asarray(X)\n",
    "        # create the binarizer transforms\n",
    "        _transformers = {}\n",
    "        for col in self.obj_columns:\n",
    "            _transformers.update({col: LabelBinarizer().fit(X[:, col])})\n",
    "        \n",
    "        self._transformers = _transformers\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        X = np.asarray(X)\n",
    "        \n",
    "        dummies = None\n",
    "        for col in self.obj_columns:\n",
    "            if dummies is None:\n",
    "                dummies = self._transformers[col].transform(X[:, col])\n",
    "            else:\n",
    "                new_dummy = self._transformers[col].transform(X[:, col])\n",
    "                dummies = np.column_stack((dummies, new_dummy))\n",
    "            \n",
    "        # remove original columns\n",
    "        X = X[:, ~self.obj_index]\n",
    "        \n",
    "        X = np.column_stack((X, dummies))\n",
    "        \n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Often it makes sense to do the data transformation, feature extraction, etc. as part of a Pipeline\n",
    "* Pipelines are flexible and provide the same sklearn API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree_estimator = Pipeline([('transformer', PandasTransformer(dta)), \n",
    "                            ('dtree', dtree)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree_estimator.fit(dta, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtree_estimator.named_steps['dtree']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}