12_RegressionTrees.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 428,
     "status": "ok",
     "timestamp": 1603818364234,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "zBMNhWc-5rxR"
   },
   "outputs": [],
   "source": [
    "#Import required libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.tree import plot_tree\n",
    "from sklearn.model_selection import cross_val_predict, cross_validate, cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G3J3UMwkFk4Y"
   },
   "source": [
    "# Tree-Based Models: Regression Trees\n",
    "\n",
    "In this notebook we continue our discussion of regression models, but we deviate away from the usual *linear regression model*. There are many, many classes of model and we won't cover nearly all of them in our course. A second class of model which can be used for regression is called the *regression tree*. We'll discuss it here.\n",
    "\n",
    "Tree-based models are quite popular because they mimic a human's decision-making process (*If this, then that...*). They are very easily understood, even by non-experts, because they can be made quite visual -- in fact, you already know exactly what a tree-based model looks like, even though we haven't discussed them -- they are just *decision trees*. \n",
    "\n",
    "In this notebook, we'll continue to work through our insurance rate-setting application. The data will be the same data you've been using over the last three weeks, but rather than build a linear regression model, we'll build a decision tree regressor.\n",
    "\n",
    "## About tree-based models\n",
    "\n",
    "As mentioned, tree-based models are quite popular because they are intuitive. Of all the classes of models, these are the ones that non-experts will like most because they can be understood without any significant background or explanation. That being said, tree-based models can be tricky to use correctly.\n",
    "\n",
    "Let's start with the assumptions we are making when we choose to utilize a decision tree.\n",
    "\n",
    "+ Trees assume that observations can be segmented into relatively homogeneous regions, in which oservations in a similar location all exhibit a similar response.\n",
    "\n",
    "    + The idea here is that trees basically bin similar observations together. Training a tree determines what it means for observations to be similar.\n",
    "\n",
    "+ Trees assume that we can determine which observations are similar to one another by asking a series of simple yes/no questions (ie. *Is `age` greater than 55?*).\n",
    "\n",
    "    + This assumption results in the feature space being cut into [high dimentional]-\"rectangular\" regions, where each cut made is perpendicular to one of the feature axes. In 2-dimensions (one feature, one response), this is quite easy to understand -- see the image that follows.\n",
    "\n",
    "Below you can see two regression trees fit on a randomly generated set of data (don't worry about the code, just take a look at the pictures that result from it). One of the trees is unconstrained -- you can see that the unconstrained model overfits the data -- the other tree is constrained to a maximum depth of three. This means that along each branch, we ask at most three questions about the features before making a prediction. You'll notice that there are $2^3=8$ predicted levels of the response (denoted by horizontal red line segments) in the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 291
    },
    "executionInfo": {
     "elapsed": 1023,
     "status": "ok",
     "timestamp": 1603818564163,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "lJ895KcA5rxf",
    "outputId": "cd9987a7-b4e6-486b-9343-2e1ec6c20e9c"
   },
   "outputs": [],
   "source": [
    "X = 10*np.random.rand(100, 1) - 5\n",
    "y = X**2 + 1 + 4*np.random.randn(100, 1)\n",
    "\n",
    "tree_reg_NoLeash = DecisionTreeRegressor()\n",
    "tree_reg_Constrained = DecisionTreeRegressor(max_depth = 3)\n",
    "\n",
    "tree_reg_NoLeash.fit(X, y)\n",
    "tree_reg_Constrained.fit(X, y)\n",
    "\n",
    "def plot_regression_predictions(tree_reg, X, y, axes=[-5, 5, -5, 30], ylabel=\"$y$\"):\n",
    "    x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)\n",
    "    y_pred = tree_reg.predict(x1)\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "    if ylabel:\n",
    "        plt.ylabel(ylabel, fontsize=18, rotation=0)\n",
    "    plt.plot(X, y, \"b.\")\n",
    "    plt.plot(x1, y_pred, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "\n",
    "fig, axes = plt.subplots(ncols=3, figsize=(15, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plt.plot(X, y, \"b.\")\n",
    "\n",
    "plt.sca(axes[1])\n",
    "plot_regression_predictions(tree_reg_NoLeash, X, y)\n",
    "for split, style in ((0.1973, \"k-\"), (0.0917, \"k--\"), (0.7718, \"k--\")):\n",
    "    plt.plot([split, split], [-0.2, 1], style, linewidth=2)\n",
    "#plt.text(0.21, 0.65, \"Depth=0\", fontsize=15)\n",
    "#plt.text(0.01, 0.2, \"Depth=1\", fontsize=13)\n",
    "#plt.text(0.65, 0.8, \"Depth=1\", fontsize=13)\n",
    "#plt.legend(loc=\"upper center\", fontsize=18)\n",
    "#plt.title(\"max_depth=2\", fontsize=14)\n",
    "\n",
    "plt.sca(axes[2])\n",
    "plot_regression_predictions(tree_reg_Constrained, X, y, ylabel=None)\n",
    "for split, style in ((0.1973, \"k-\"), (0.0917, \"k--\"), (0.7718, \"k--\")):\n",
    "    plt.plot([split, split], [-0.2, 1], style, linewidth=2)\n",
    "for split in (0.0458, 0.1298, 0.2873, 0.9040):\n",
    "    plt.plot([split, split], [-0.2, 1], \"k:\", linewidth=1)\n",
    "#plt.text(0.3, 0.5, \"Depth=2\", fontsize=13)\n",
    "#plt.title(\"max_depth=3\", fontsize=14)\n",
    "\n",
    "#save_fig(\"tree_regression_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W8QosV3J5rxk"
   },
   "source": [
    "Can you tell what the splitting boundaries and corresponding questions are? In the plots above each decision boundary and question asked corresponds to one of the red vertical line segments in the plots. The regression tree corresponding to the rightmost plot appears below -- does it match what you expected?. \n",
    "\n",
    "When reading the plot, non-leaf nodes (the nodes that have arrows coming *out* of them) include four rows of text. The first row lists the question used to split the node into its two children. The third line gives the number of training observations contained in that node. The fourth line gives the predicted response for observations falling in this node (it is the mean resonse over all observations in the bucket). The second line lists the *mean square error* (MSE) corresponding to the model predictions if this node's mean were used as the predicted value. The leaf nodes (nodes with no children extending from them) are the nodes corresponding to model predictions. These nodes don't split any further, so they have no row corresponding to a splitting question. All other rows of text have the same interpretations though. Regarding nodes that split, an answer of \"yes\" to the splitting question moves down to the child on the left while an answer of no moves down to the child on the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "executionInfo": {
     "elapsed": 1017,
     "status": "ok",
     "timestamp": 1603819051054,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "eBogRPau5rxl",
    "outputId": "b1478ba8-50c7-45b9-f602-f127faf7b7cf"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (12, 6))\n",
    "out = plot_tree(tree_reg_Constrained, feature_names = [\"x\"], filled = True, rounded = True, fontsize = 11)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wn7YHafs5rxr"
   },
   "source": [
    "Before we jump to constructing our first decision tree regressor, it is worth noting explicitly that decision trees are enticed to overfit the training data. We need to use constraints (sometimes called *regularization*) to prevent this overfitting -- for example, in the code above, the second model was constrained by mandating a maximum depth of 3 rather than allowing the tree to grow without bound (as in the `NoLeash` tree).\n",
    "\n",
    "## Constructing and using a decision tree regressor to predict insurance claims\n",
    "\n",
    "Now that we know a bit more about decision tree models, let's actually build and use one. We'll start with a quick review of our dataset and the goal for our predictive model.\n",
    "\n",
    "### About our data\n",
    "\n",
    "- Our simple dataset contains a few attributes for each person such as \n",
    "- Age, Sex, BMI, Children, Smoker, Region and their charges\n",
    "\n",
    "### Goal\n",
    "- To use this info to predict charges for new customers\n",
    "  - An insurance company can use this information to inform *rate-setting*\n",
    "  \n",
    "## Preliminaries\n",
    "\n",
    "Remember that we'll always start by building a *training*, *test*, and *safe* set. We'll do that and just print out the head of the *training* data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "executionInfo": {
     "elapsed": 590,
     "status": "ok",
     "timestamp": 1603819254515,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "m3xySfs8Fjj7",
    "outputId": "f748750c-cd66-43aa-8e1c-990f24b95a0d"
   },
   "outputs": [],
   "source": [
    "file_name = \"https://raw.githubusercontent.com/rajeevratan84/datascienceforbusiness/master/insurance.csv\"\n",
    "insurance = pd.read_csv(file_name)\n",
    "\n",
    "#Get dummy variables\n",
    "insurance = pd.get_dummies(insurance, columns = [\"sex\", \"smoker\", \"region\"], drop_first = True)\n",
    "\n",
    "#Split to training, test, and safe data\n",
    "X = insurance.drop([\"charges\"], axis = 1)\n",
    "y = insurance[\"charges\"]\n",
    "\n",
    "X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size = 0.25, random_state = 42)\n",
    "X_safe, X_test, y_safe, y_test = train_test_split(X_temp, y_temp, test_size = 0.6, random_state = 42)\n",
    "\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vkd4N9Xz5ryW"
   },
   "source": [
    "Okay, now that we have our data processed and split, we are ready to fit our decision tree regressor. We'll fit an unconstrained regression tree and then plot its first three levels. The text printed out before the plot renders is the decision tree in text form -- it shows us information about each node, including the question used to split the node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 575
    },
    "executionInfo": {
     "elapsed": 1564,
     "status": "ok",
     "timestamp": 1603819294193,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "0YUw0yzt5ryY",
    "outputId": "db20109c-c6eb-4828-fbc7-c912f7b18774"
   },
   "outputs": [],
   "source": [
    "tree_reg = DecisionTreeRegressor()\n",
    "\n",
    "tree_reg.fit(X_train, y_train)\n",
    "\n",
    "plt.figure(figsize = (15, 10))\n",
    "out = plot_tree(tree_reg, feature_names = X_train.columns, filled = True, rounded = True, fontsize = 10, max_depth = 3)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ssedrOxN5ryf"
   },
   "source": [
    "Now that we have a tree, let's see how it performs on the training and test data. We'll just work with the $\\tt{RMSE}$ metric here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "executionInfo": {
     "elapsed": 488,
     "status": "ok",
     "timestamp": 1603819524696,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "0dviUVea5ryg",
    "outputId": "552f1dfa-3b46-4edd-f4a5-6c5253628b36"
   },
   "outputs": [],
   "source": [
    "tree1_trainRMSE = (((y_train - tree_reg.predict(X_train))**2).mean())**0.5\n",
    "tree1_testRMSE = (((y_test - tree_reg.predict(X_test))**2).mean())**0.5\n",
    "\n",
    "print(\"The training RMSE for our unconstrained tree is: $\", tree1_trainRMSE, sep = \"\")\n",
    "print(\"The test RMSE for our unconstrained tree is: $\", tree1_testRMSE, sep = \"\")\n",
    "\n",
    "#Uncomment below to see Cross-Validation Results\n",
    "X_trainCV = pd.concat([X_train, X_test])\n",
    "y_trainCV = pd.concat([y_train, y_test])\n",
    "tree_reg_cv = DecisionTreeRegressor()\n",
    "cv_results = cross_val_score(tree_reg_cv, X_trainCV, y_trainCV, cv = 10, scoring = \"neg_root_mean_squared_error\")\n",
    "print(\"Using an unconstrained tree:\")\n",
    "print(\"\\t RMSE on folds: \", -1*cv_results)\n",
    "print(\"\\tCross validation RMSE estimate: \", (-1*cv_results).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4_E0TUnI5rym"
   },
   "source": [
    "What a difference between our training and test errors! This tree is way overfit -- did you expect that? I hope so. Additionally, if you remember the end of last week, our best model had an estimated prediction error of about \\\\$6,137 and here we are significantly higher than that. It looks like our tree is not doing as good a job. At this point, there are two possible explanations -- (i) our linear regressor is just better-suited to predicting the `charges` given our features, or (ii) we just haven't used an optimal tree. Let's see if we can build a better tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PX_nQPNO5ryn"
   },
   "source": [
    "### An Aside: Hyperparameters and Model Tuning\n",
    "\n",
    "In statistical modeling, a parameter for a model is a value which is optimized by the model fitting procedure. A *hyperparameter* is a setting which is determined prior to model fitting. For example, setting `max_depth = 3` determines prior to fitting the decision tree regressor that the maximum number of splits along any branch is 3. This means that `max_depth` is a hyperparameter.\n",
    "\n",
    "It isn't very efficient to just randomly guess to find the \"best\" maximum depth or the best value for any of the other hyperparameters available when fitting decision trees. You can see the hyperparameters available to decision tree regressors in `sklearn` from the [documentation of `DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html). Since the model fitting procedure doesn't optimize hyperparameters, we have two choices. We can either choose the hyperparameters manually (which we've already said is sub-optimal) or we can try to adapt the model fitting procedure so that multiple hyperparameter combinations are tried, and the best combination is selected. How do we do this? Keep calm and cross-validate. Remember how the cross-validation process helped us decide whether to use the straight-line linear regression model, the model that included curvature associated with the `age` variable, or the model that allowed for lots of flexibility through the third-degree polynomial terms? In a sense, those curvature and interaction terms were hyperparameters too -- cross validation gave us error estimates associated with each model and we chose the one with the best cross-validation error estimate. We'll use the same strategy here.\n",
    "\n",
    "Before we do this, it is worth mentioning the complexity we are about to run up against. Remember that 10-fold cross-validation trains and validates a model 10 separate times. If we tune the `max_depth` hyperparameter considering that `max_depth` could be any of the values in the list `[2, 3, 4, 5, 6, 8, 10]`, we will be training 70 models (ten models for each choice of max depth). Now what if we also want to tune the `max_leaf_nodes` and `min_samples_split` parameters -- each with just three possibilities. All of a sudden we'll be fitting $10\\times 7\\times 3 \\times 3~=~630$ models! The run-time here is going to grow exponentially, so keep this in mind when deciding what hyperparameters to tune and which values to consider. We'll just move forward with tuning the `max_depth` hyperparameter over the list of possibilities `[2, 3, 4, 5, 6, 8, 10]`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 493
    },
    "executionInfo": {
     "elapsed": 776,
     "status": "ok",
     "timestamp": 1603819693714,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "p0lunxND5ryo",
    "outputId": "60820fbd-0e7e-49ea-f955-3b9da4b49858"
   },
   "outputs": [],
   "source": [
    "max_depth = [2, 3, 4, 5, 6, 8, 10]\n",
    "\n",
    "X_trainCV = pd.concat([X_train, X_test])\n",
    "y_trainCV = pd.concat([y_train, y_test])\n",
    "\n",
    "for depth in max_depth:\n",
    "    tree_reg = DecisionTreeRegressor(max_depth = depth)\n",
    "    \n",
    "    cv_results = cross_val_score(tree_reg, X_trainCV, y_trainCV, cv = 10, scoring = \"neg_root_mean_squared_error\")\n",
    "    print(\"Using max_depth \", depth, \":\")\n",
    "    print(\"\\t RMSE on folds: \", -1*cv_results)\n",
    "    print(\"\\tCross validation RMSE estimate: \", (-1*cv_results).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OlDzTO305ryr"
   },
   "source": [
    "It looks like using a `max_depth` of 4 yielded the best cross-validation error. It looks like we should expect a prediction error of around \\\\$4,745.47 if we use a decision tree and a max depth of 4. This is already over a \\\\$1,500 improvement over the linear regression model, and we only tuned one hyperparameter!\n",
    "\n",
    "Let's crack open the safe and see whether our decision tree with `max_depth` 4 performs as expected on this data which still has yet to be seen by our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "executionInfo": {
     "elapsed": 528,
     "status": "ok",
     "timestamp": 1603819825800,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "c0ZeIwEL7U8w",
    "outputId": "a6335f13-cd3d-4bdb-e98e-9fe3ac095671"
   },
   "outputs": [],
   "source": [
    "X = pd.concat([X_train, X_test])\n",
    "y = pd.concat([y_train, y_test])\n",
    "\n",
    "bestChargesTreeModel = DecisionTreeRegressor(max_depth = 4)\n",
    "\n",
    "bestChargesTreeModel.fit(X, y)\n",
    "print(\"RMSE on Safe Data: \", (((y_safe - bestChargesTreeModel.predict(X_safe))**2).mean())**0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XRHy_vIf7VLv"
   },
   "source": [
    "Our model performs right in the ball-park of where we expected it would. If our Safe RMSE was not comparable to any of the RMSE metrics we observed during cross-validation for the depth-4 tree, then we should be concerned about the possibility of an overfit model or training and safe data which are note representative of one another.\n",
    "\n",
    "Let's re-build our decision tree regressor with a max depth of 4, using all of the available data and see what it predicts as the cost to insure our potential new subscriber from last time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 592
    },
    "executionInfo": {
     "elapsed": 1566,
     "status": "ok",
     "timestamp": 1603819885665,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 240
    },
    "id": "YX0NsZxM5rys",
    "outputId": "a741e452-7bdd-44ac-ee9b-1ca11796eebe"
   },
   "outputs": [],
   "source": [
    "X = pd.concat([X_train, X_test, X_safe])\n",
    "y = pd.concat([y_train, y_test, y_safe])\n",
    "\n",
    "productionChargesTreeModel = DecisionTreeRegressor(max_depth = 4)\n",
    "\n",
    "productionChargesTreeModel.fit(X, y)\n",
    "\n",
    "plt.figure(figsize = (20, 10))\n",
    "out = plot_tree(productionChargesTreeModel, feature_names = X_train.columns, filled = True, rounded = True, fontsize = 10)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(3)\n",
    "plt.show()\n",
    "\n",
    "new = {\"age\" : [34], \"bmi\" : [28], \"children\" : [2], \"sex_male\" : [1], \"smoker_yes\" : [0], \"region_northwest\" : [0], \"region_southeast\" : [0], \"region_southwest\" : [0]}\n",
    "new = pd.DataFrame(new)\n",
    "\n",
    "productionChargesTreeModel.predict(new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ry5gC-XE5ryx"
   },
   "source": [
    "It looks like our new tree-based model predicts a significantly higher cost to insure (about \\\\$7,422.34) than our linear regressor from last week did. We may have just saved the company from some significant losses!\n",
    "\n",
    "## Summary\n",
    "\n",
    "Okay, that's it for now -- you know a second class of model which can be used to predict a numerical response. It won't always be the case that a regression tree outperforms a linear model or vice-versa, so it is important to consider both as options. You've got a second notebook this week that introduces the idea of a *classification tree*. Tree-based models can be employed in both the regression and classification settings. Check out the *12_ClassificationTrees* notebook next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ist-V6Gk5ryx"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "11_RegressionTrees.ipynb",
   "provenance": [
    {
     "file_id": "1b82w7y7JW2qq3zMQ0X9xs1xWf1GXstyR",
     "timestamp": 1578321565295
    },
    {
     "file_id": "1xedbseYti1b2aaeMUb7YsyOtny8I6oPF",
     "timestamp": 1573156909581
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}