9. RandomForests.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forests\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.classification.RandomForest import *\n",
    "from intuitiveml.utils import gen_button"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "From the Scikit-Learn [website](https://scikit-learn.org/stable/modules/ensemble.html):\n",
    "\n",
    "    The goal of ensemble methods is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator.\n",
    "    \n",
    "    Two families of ensemble methods are usually distinguished:\n",
    "    \n",
    "    In averaging methods, the driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced.\n",
    "    Examples: Bagging methods, Forests of randomized trees, …\n",
    "\n",
    "    By contrast, in boosting methods, base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble.\n",
    "    Examples: AdaBoost, Gradient Tree Boosting, …\n",
    "    \n",
    "Let's stick with the ***first*** family for now, which ***Random Forests*** are a member of.\n",
    "\n",
    "The whole idea behind ***ensemble methods*** is to profit from the ***wisdom of the crowds***. In simple terms, it tries to average out the error by using multiple estimators (models). \n",
    "\n",
    "So, it is easy as training a whole ***bunch of Decision Trees*** and ***averaging their predictions***! There is a ***catch***, though...\n",
    "\n",
    "If you train multiple trees on the same dataset, they will be, er... ***all the same***! How to solve that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Bagging\n",
    "\n",
    "#### ***BAGGing*** stands for ***B***ootstrapping and ***AGG***regat***ing***.\n",
    "\n",
    "First, we ***bootstrap***, which is just fancy for ***sampling with replacement***. \n",
    "\n",
    "Resampling a dataset, keeping its original size, will yield a dataset similar to the original one, but not quite. \n",
    "\n",
    "***NOTE***: Some of items in the original dataset will ***not make it*** to the resampled dataset - these are ***out-of-bag (OOB)*** samples - they can be handy for evaluating the model later!\n",
    "\n",
    "So, if we train a Decision Tree on this dataset, it will yield a ***slightly different model***. It is ***injecting randomness*** in the process!\n",
    "\n",
    "If we repeat this several times, we'll have as many slightly different resampled datasets and, therefore, models.\n",
    "\n",
    "Next, we ***aggregate*** them, which is just ***averaging their predictions***.\n",
    "\n",
    "This is ***BAGGING*** in a nutshell! \n",
    "\n",
    "P.S.: There is actually more to it, as it is also possible to ***randomize the features used at each split*** (remember the Decision Tree notebook), for instance. Check [Scikit-Learn](https://scikit-learn.org/stable/modules/ensemble.html#random-forests) for more information on Random Forests.\n",
    "\n",
    "![](https://imgs.xkcd.com/comics/ensemble_model.png)\n",
    "<center>Source: <a href=\"https://xkcd.com/1885/\">XKCD</a></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "You have 10 data points of two colors: red and green! These are your ***labels***.\n",
    "\n",
    "Each point is has a single numerical coordinate. This is your ***feature***.\n",
    "\n",
    "The sliders below allow you to train one (shown as zero in the slider) or ***multiple Decision Trees*** and choose the ***maximum depth*** they are allowed to have.\n",
    "\n",
    "Each tree outputs a probability (refer to ***Understanding its Nodes*** section of the ***Decision Tree*** notebook for more details) associated with its predicted class.\n",
    "\n",
    "The average of these probabilities is then shown in the plot.\n",
    "\n",
    "Use the sliders to play with different configurations and answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = data()\n",
    "mydt = plotDecision(x=x, y=y, idx_mid=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aad3dbf0e65e424ba363f4ad06891ccf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'marker': {'color': 'black'},\n",
       "              'mode': 'lines',\n",
       "     …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb = VBox(build_figure_ensemble(mydt), layout={'align_items': 'center'})\n",
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. What happens if you train 5 trees (keeping depth 3)?\n",
    "2. How many trees (keeping depth 3) do you need to train to get ***all points correctly classified***? Why?\n",
    "3. How many trees (with depth 1) do you need to train to get ***all points correctly classified***? Why?\n",
    "4. Considering an individual tree, what's the main ***difference*** between a shallow (depth = 1) and a deep (depth = 3) tree?\n",
    "5. Which one is best to use in Random Forests, ***shallow*** or ***deep*** trees? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scikit-Learn\n",
    "\n",
    "[Ensemble methods](https://scikit-learn.org/stable/modules/ensemble.html)\n",
    "\n",
    "[Probability calibration](https://scikit-learn.org/stable/modules/calibration.html)\n",
    "\n",
    "Please check Aurelién Geron's \"Hand-On Machine Learning with Scikit-Learn and Tensorflow\" notebook on Ensemble Methods [here](http://nbviewer.jupyter.org/github/ageron/handson-ml/blob/master/07_ensemble_learning_and_random_forests.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[InfoGraphic](https://github.com/Avik-Jain/100-Days-Of-ML-Code/blob/master/Info-graphs/Day%2033.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}