10. GradientBoostedTrees.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Boosted Trees\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.regression.GradientBoostedTrees import *\n",
    "from intuitiveml.utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "From the Scikit-Learn [website](https://scikit-learn.org/stable/modules/ensemble.html):\n",
    "\n",
    "    The goal of ensemble methods is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator.\n",
    "    \n",
    "    Two families of ensemble methods are usually distinguished:\n",
    "    \n",
    "    In averaging methods, the driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced.\n",
    "    Examples: Bagging methods, Forests of randomized trees, …\n",
    "\n",
    "    By contrast, in boosting methods, base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble.\n",
    "    Examples: AdaBoost, Gradient Tree Boosting, …\n",
    "    \n",
    "Let's stick with the ***second*** family this time, which ***Gradient Boosted Trees*** are a member of.\n",
    "\n",
    "Once again, the main idea is to train a whole ***bunch of Decision Trees***. But, this time, it is ***not*** going to ***simply average them***. This is not ***bagging***, it is..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 (Gradient) Boosting\n",
    "\n",
    "This time, it will start by training ***ONE tree*** on the ***original dataset*** and evaluate its predictions, computing the corresponding ***residuals*** (errors).\n",
    "\n",
    "Then, for the ***NEXT*** tree, it will ***not*** use the ***original dataset*** anymore, but the ***residuals from the previous tree***!\n",
    "\n",
    "So, instead of averaging the predictions of individual trees, it ***adds them up*** to get the final predictions.\n",
    "\n",
    "Unlike bagging, which is highly parallelizable, ***boosting*** is a ***sequential*** process.\n",
    "\n",
    "This is ***Gradient Boosting*** in a nutshell!\n",
    "\n",
    "P.S.: There is actually more to it... [XGBoost](https://github.com/dmlc/xgboost) is one of the most popular and succesful algorithms and it uses many more improvements, but the underlying idea is still the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself! \n",
    "\n",
    "This time, it is a regression problem!\n",
    "\n",
    "You have 5 data points with values between 1160 and 2000. This is your ***response**.\n",
    "\n",
    "Each point is associated with a single numerical value between 750 and 950. This is your ***feature***.\n",
    "\n",
    "For a regression, the initial step is to compute the ***average*** of all points and use it to compute the ***residuals*** to train the first tree. We'll see the reasoning behind this in the ***Linear Regression*** lesson.\n",
    "\n",
    "The sliders below allow you to train one (shown as zero in the slider) or multiple Decision Trees and choose the maximum depth they are allowed to have.\n",
    "\n",
    "For each new trained tree, it will show both ***residuals from the previous step*** and corresponding ***fitted tree*** on the left. On the right, it will ***add up*** the predictions of all trees up to that one.\n",
    "\n",
    "Use the sliders to play with different configurations and answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "xreg = np.array([750., 800., 850., 900., 950.])\n",
    "yreg = np.array([1160., 1200., 1280., 1450., 2000.])\n",
    "mydtr = plotDecision(x=xreg, y=yreg)\n",
    "\n",
    "vb = VBox(build_figure_boost(mydtr), layout={'align_items': 'center'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec4f9dc8aade45989a4556ab88f9aca7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'marker': {'color': 'green', 'line': {'color': 'black', 'width': 2…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. What happens to the ***level of residuals*** as you increase the number of trees (keeping depth = 1)?\n",
    "2. What happens if you ***increase the depth*** of the trees? Why?\n",
    "3. Which one is best to use in GBTs, ***shallow*** or ***deep*** trees? Why"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scikit-Learn\n",
    "\n",
    "[Gradient Tree Boosting](https://scikit-learn.org/stable/modules/ensemble.html#gradient-tree-boosting)\n",
    "\n",
    "Please check Aurelién Geron's \"Hand-On Machine Learning with Scikit-Learn and Tensorflow\" notebook on Ensemble Methods [here](http://nbviewer.jupyter.org/github/ageron/handson-ml/blob/master/07_ensemble_learning_and_random_forests.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[Difference between Bagging and Boosting](https://quantdare.com/what-is-the-difference-between-bagging-and-boosting/)\n",
    "\n",
    "[Complete Guide to Parameter Tuning in XGBoost](https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)\n",
    "\n",
    "[Interpretable Machine Learning with XGBoost](https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27)\n",
    "\n",
    "[How to explain gradient boosting](https://explained.ai/gradient-boosting/index.html)\n",
    "\n",
    "[Mastering The New Generation of Gradient Boosting](https://towardsdatascience.com/https-medium-com-talperetz24-mastering-the-new-generation-of-gradient-boosting-db04062a7ea2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}