13_ClassificationTrees.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 1604,
     "status": "ok",
     "timestamp": 1604419493784,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "H7R1xmMA9A6q"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.tree import plot_tree\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "churn = pd.read_csv(\"https://raw.githubusercontent.com/rajeevratan84/datascienceforbusiness/master/WA_Fn-UseC_-Telco-Customer-Churn.csv\")\n",
    "mnist = datasets.load_digits()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YtrVKOOVNzii"
   },
   "source": [
    " # Tree-Based Models: Classification Trees\n",
    "\n",
    "We've discussed linear regression models and regression trees as tools for predicting a numerical response, but what if your response variable isn't numerical? What if we wish to predict whether a customer will default on a loan or not as a way to determine whether they should be granted a line of credit?\n",
    "\n",
    "##### An Aside\n",
    "\n",
    "You may have identified that, for the *default or not* example I provided, you could easily define `default` to the value 1 and `not` to the value 0. In fact, this is how we described *one-hot encodings* in the previous notebooks. If you are being perceptive, you might be excited -- could it be that building a regression model we could interpret its output as the *probability of default*? Probabilities are numeric values between 0 and 1, afterall. If you are even more observant, you may have already noticed that using this trick can't work. All of the regression models we've discussed are *multivariate polynomials* -- try to construct a polynomial in a single variable ($x$) that is bounded between 0 and 1...there's only one of them: $y = \\beta_0$, the null model. \n",
    "\n",
    "Okay, so now that we know that approach has no hope of working, we can aknowledge a possibly more obvious reason for failure -- what if instead of `default` or `not`, we were working for Netflix trying to [classify subscribers into one of 2,000 micro-profiles](https://www.dativa.com/blogs/segmentation-relevant/) for their recommendation engine? There's certainly no natural way to define 0 or 1 here.\n",
    "\n",
    "Okay, so linear regression won't work and we need new tools. There have been multiple tools developed for tackling inference and prediction tasks within *classification*. We look at two of these methods in our class: *tree-based classifiers* and *logistic regressors*. This notebook introduces tree-based classifiers and next week we will consider logistic regression. We'll consider two separate datasets and objectives this week. \n",
    "\n",
    "+ **Application I (Customer Churn):** Our first dataset deals with the challenge of customer *churn* -- this typically involves customers with a subscriprion-based service choosing to terminate or not renew their contracts. Companies with a subscription-based model seek to identify customers at risk of *churning* and to pre-emptively outreach with enticing offers, to convince them to remain a subscriber. Predicting *churn* is known as a binary classification problem -- since customers either churn (1) or remain (0).\n",
    "+ **Application II (Digit Recognization):** Our second dataset deals with classifying handwritten digits. In this application there are ten classes in the target variable -- we have a more difficult job ahead of us.\n",
    "\n",
    "## Introducing our *churn* data\n",
    "\n",
    "Our dataset deals with customer *churn*.\n",
    "\n",
    "***Customer Churn*** - churn is when an existing customer, user, player, subscriber or any kind of return client stops doing business or ends the relationship with a company.\n",
    "\n",
    "**Aim -** is to figure our which customers may likely churn in future\n",
    "\n",
    "Source - https://www.kaggle.com/blastchar/telco-customer-churn\n",
    "1. We have customer information for a Telecommunications company\n",
    "2. We've got customer IDs, general customer info, the servies they've subscribed too, type of contract and monthly charges.\n",
    "3. This is a historic customer information so we have a field stating whether that customer has **churned**.\n",
    "\n",
    "You can see the *head* of the dataset below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9lyUd8QF9A6z"
   },
   "outputs": [],
   "source": [
    "churn.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_wxedPge9A65"
   },
   "source": [
    "**Variable Descriptions**\n",
    "- customerID - Customer ID\n",
    "- gender - Whether the customer is a male or a female\n",
    "- SeniorCitizen - Whether the customer is a senior citizen or not (1, 0)\n",
    "- Partner - Whether the customer has a partner or not (Yes, No)\n",
    "- Dependents - Whether the customer has dependents or not (Yes, No)\n",
    "- tenure - Number of months the customer has stayed with the company\n",
    "- PhoneService - Whether the customer has a phone service or not (Yes, No)\n",
    "- MultipleLines - Whether the customer has multiple lines or not (Yes, No, No phone service)\n",
    "- InternetService - Customer’s internet service provider (DSL, Fiber optic, No)\n",
    "- OnlineSecurity - Whether the customer has online security or not (Yes, No, No internet service)\n",
    "- OnlineBackup - Whether the customer has online backup or not (Yes, No, No internet service)\n",
    "- DeviceProtection - Whether the customer has device protection or not (Yes, No, No internet service)\n",
    "- TechSupport - Whether the customer has tech support or not (Yes, No, No internet service)\n",
    "- StreamingTV - Whether the customer has streaming TV or not (Yes, No, No internet service)\n",
    "- StreamingMovies - Whether the customer has streaming movies or not (Yes, No, No internet service)\n",
    "- Contract - The contract term of the customer (Month-to-month, One year, Two year)\n",
    "- PaperlessBilling - Whether the customer has paperless billing or not (Yes, No)\n",
    "- PaymentMethod - The customer’s payment method (Electronic check, Mailed check Bank transfer (automatic), Credit card (automatic))\n",
    "- MonthlyCharges - The amount charged to the customer monthly\n",
    "- TotalCharges - The total amount charged to the customer\n",
    "- Churn - Whether the customer churned or not (Yes or No)\n",
    "\n",
    "## Using trees for classification\n",
    "\n",
    "Add a description of trees for classification, in particular the tree-fitting algorithm.\n",
    "\n",
    "## Preparing our data\n",
    "\n",
    "Below we start the initial data cleaning process and then split our data into *training* and *safe* sets -- we won't use a separate *test* set now, since we know about *cross-validation*. Remember, that $k$-fold cross validation splits the data into $k$ *folds*, where each fold takes one turn being left out of the model fitting process, and is used as a validation set. The benefit to $k$-fold cross validation is that we estimate $k$ separate validation errors and can aggregate them into a single cross-validation error rate, which better approximates the true prediction error.\n",
    "\n",
    "We'll start by checking for missing data and then get dummy variables for any categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 526,
     "status": "ok",
     "timestamp": 1604419572508,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "tJhI-Le_NmlU",
    "outputId": "bd5bb4df-dff8-40b2-d354-2c75a6ed1349"
   },
   "outputs": [],
   "source": [
    "churn.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FSLyB_LRtUCQ"
   },
   "source": [
    "Our dataset does require some pre-processing. For example, there are lots of categorical columns that we need transformed into \"dummy\" columns just like we did with our regression models. I'll walk you through some automated pre-processing steps below -- be sure to work with me through the code so that you understand what each line is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 506,
     "status": "ok",
     "timestamp": 1604419612861,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "D_exhd4Z9A6_",
    "outputId": "2e95abba-9a1b-483f-f6cc-f169362eb5d3"
   },
   "outputs": [],
   "source": [
    "churn.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6g8xWVH-9A7E"
   },
   "source": [
    "It seems as though there are a lot of columns which are listed as the *object* class -- this means they are non-numerics. From the descriptions of the variables we saw earlier, this seems to be appropriate in all cases except for the `TotalCharges` column. We can see what is going wrong if we print out the `TotalCharges` column -- if you uncomment that line in the code cell below and scroll through the column entries you will find a few entries which contain a *space*, so they are not technically null. Since there are not many of these entries we will just drop them here for convenience and then convert the column to numeric values. What is dangerous about this approach?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 431,
     "status": "ok",
     "timestamp": 1604419700453,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "3Rc4iPd39A7F",
    "outputId": "5aa04c29-d912-4938-e12c-3e8b9f19f4d5"
   },
   "outputs": [],
   "source": [
    "#print(list(churn[\"TotalCharges\"]))\n",
    "print(churn.shape)\n",
    "churn = churn[churn[\"TotalCharges\"] != \" \"]\n",
    "churn[\"TotalCharges\"] = pd.to_numeric(churn[\"TotalCharges\"])\n",
    "print(churn.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bueha5qO9A7I"
   },
   "source": [
    " Now let's see what we can do to get dummy variables for these categorical variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "executionInfo": {
     "elapsed": 516,
     "status": "ok",
     "timestamp": 1604419839354,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "gbDqT4iz9A7J",
    "outputId": "c804da5e-8737-43d2-84db-514e60459519"
   },
   "outputs": [],
   "source": [
    "num_cols = [\"tenure\", \"MonthlyCharges\", \"TotalCharges\"]\n",
    "unique_id = [\"customerID\"]\n",
    "\n",
    "#We could just type out a list of the categorical columns, but this is a bit shorter\n",
    "cat_cols = [name for name in list(churn.columns) if ((name not in num_cols) & (name not in unique_id))]\n",
    "\n",
    "churn = pd.get_dummies(churn, columns = cat_cols, drop_first = True)\n",
    "churn.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cYAPuSUk9A7P"
   },
   "source": [
    "You may have noticed that, in the previous cell, we labeled `customerID` as something called a *unique identifier*. This is a non-numeric value which is unique to each observation. If we included it in our model, we could construct a perfect classifier. This sounds like a really good thing -- but, unfortunately if we used `customerID` as a predictor our classifier would be useless when applied to new data. The classifier including this variable would essentially become a *look-up table* -- find the customer with the corresponding `customerID` look to the last column of the table in that row and report back whether or not the customer *churned*. Such a model would fail on any data it was not trained on -- for this reason, we never want to include a unique identifier as a predictor in a model. Let's drop it now and then use `train_test_split` to create our *training* and *safe* sets. Since we'll use cross-validation now, there's no need to create a separate *test* set any longer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 466,
     "status": "ok",
     "timestamp": 1604419917959,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "a_-NFtZD9A7Q",
    "outputId": "38190398-c4d5-43fa-ac90-114175106941"
   },
   "outputs": [],
   "source": [
    "churn.drop([\"customerID\"], axis = 1, inplace = True)\n",
    "\n",
    "X = churn.drop([\"Churn_Yes\"], axis = 1)\n",
    "y = churn[\"Churn_Yes\"]\n",
    "\n",
    "X_train, X_safe, y_train, y_safe = train_test_split(X, y, test_size = 0.1, random_state = 42)\n",
    "\n",
    "X_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IW7ORYZ29A7W"
   },
   "source": [
    "Now that we have the training and safe data, let's see if we can build a decision tree. Decision tree classifiers are quite similar to decision tree regressors -- the model is still a decision tree which asks a series of yes/no questions. Observations land in leaf nodes and all observations falling in the same leaf node are assigned to the same class -- the most prevalent class within that leaf. Remember that unconstrained trees are enticed to overfit, so we will use cross-validation and some regularization constraints to fit various trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4877,
     "status": "ok",
     "timestamp": 1604420284507,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "ICeVEVDM9A7X",
    "outputId": "5ed1166c-4ca4-48a1-ab1c-5ebefbea0b47"
   },
   "outputs": [],
   "source": [
    "max_depth = [3, 4, 5, 8, 10]\n",
    "min_samples = [0.005, 0.01, 0.02, 0.05]\n",
    "\n",
    "for depth in max_depth:\n",
    "    for minSamp in min_samples:\n",
    "        tree_clf = DecisionTreeClassifier(max_depth = depth, min_samples_split = minSamp)\n",
    "        \n",
    "        cv_results = cross_val_score(tree_clf, X_train, y_train, cv = 10, scoring = \"accuracy\")\n",
    "        print(\"Max depth \", depth, \", minimum bucket size for splitting (as a proportion) \", minSamp, \":\")\n",
    "        print(\"\\t Accuracy Rates: \", cv_results)\n",
    "        print(\"\\t Mean accuracy: \", cv_results.mean())\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nLCGMeL39A7a"
   },
   "source": [
    "The results of cross-validation seem to suggest we get about 79% accuracy using `max_depth = 3`, and with this parameter it looks like changing the `min_samples_split` parameter did not impact training. This makes sense, since buckets are unlikely to contain a very small number of observations with just three levels of splitting. Let's see how our model actually performs on the training data by re-fitting and building a confusion matrix. You can see a visual of our model and the confusion matrix output below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 626
    },
    "executionInfo": {
     "elapsed": 1192,
     "status": "ok",
     "timestamp": 1604420456867,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "qzUptBGH9A7b",
    "outputId": "cb10a5f7-5f3b-4ce0-ae3f-b2e115fb92db"
   },
   "outputs": [],
   "source": [
    "tree_clf = DecisionTreeClassifier(max_depth = 3)\n",
    "\n",
    "tree_clf.fit(X_train, y_train)\n",
    "\n",
    "plt.figure(figsize = (15, 10))\n",
    "out = plot_tree(tree_clf, feature_names = X_train.columns, filled = True, rounded = True, fontsize = 10)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(3)\n",
    "plt.show()\n",
    "\n",
    "print(\"Confusion matrix:\\n\",confusion_matrix(y_train, tree_clf.predict(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TxPnFc2t9A7h"
   },
   "source": [
    "In the confusion matrix above, the rows correspond to the true class that the customer belongs to. The first row contains customers who did not churn (`Churn_Yes` = 0) while the second row contains customers who did churn. The columns correspond to our predictions -- again, the first column corresponds to customers who we did not expect to churn while the second column corresponds to customers who we did expect to churn. We correctly flagged 669 customers who churned but missed 1007 additional customers who churned. Refer back to our discussion on model validation metrics for the definitions of *accuracy*, *precision*, and *recall* if you need to. Our accurracy rate for this model was 79.14%, our recall rate was 39.92% (which is not so great), and our precision was 63.13%. I think we would be willing to sacrifice some of our accuracy rate and precision in order to gain some recall. Remember, the goal here was to build a model which would inform outreach efforts to customers who are at risk for *churn*. See what you can do to build a model with better recall! Once you have one, see how well it does on the *safe* data.\n",
    "\n",
    "Now that we've dealt with binary classification, let's move to predicting a response with more than just two classes.\n",
    "\n",
    "## The multiclass setting for classification\n",
    "\n",
    "In this section, we'll explore an application of classification where the target is not binary -- instead, our target will include ten different classes. We'll work with a famous dataset called `mnist` which includes 1797 hand-drawn digits. Our goal will be to build a classification tree to identify the drawn digits. This is an interesting problem because it involves something called *computer vision* -- the computer can't actually \"see\" the image of the digit, so how do we train the computer to identify the digits? We'll unpack each 8 pixel by 8 pixel image into a list of 64 grayscale values. This will give us a dataset with 64 features from which, we will try to build a decision tree to classify the digits.\n",
    "\n",
    "Below, we split the `mnist` data into a *training*, *test*, and *safe* set. Additionally, we print out the grayscale values corresponding to the first five digits in the training data, and plot the image corresponding to the first drawn digit as well as print its label. Do you think we'll have an easy or difficult time distinguishing these digits from one another?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 690
    },
    "executionInfo": {
     "elapsed": 717,
     "status": "ok",
     "timestamp": 1604420609994,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "InmtI4pn9A7p",
    "outputId": "0bad74af-f42d-4e06-b589-03ea93285986"
   },
   "outputs": [],
   "source": [
    "mnist\n",
    "digits_train, digits_temp, labels_train, labels_temp = train_test_split(mnist[\"data\"], mnist[\"target\"], test_size = 0.25, random_state = 42)\n",
    "digits_safe, digits_test, labels_safe, labels_test = train_test_split(digits_temp, labels_temp, test_size = 0.6, random_state = 42)\n",
    "print(digits_train[:6, :])\n",
    "plt.imshow(digits_train[4].reshape(8,8), cmap = \"gray\")\n",
    "print(\"The true label is: \", labels_train[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 556,
     "status": "ok",
     "timestamp": 1604420775316,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "h9zrOIHi9A7u",
    "outputId": "fb38aa9d-0ede-41fe-e8ef-36a5c4503357"
   },
   "outputs": [],
   "source": [
    "tree_clf = DecisionTreeClassifier()\n",
    "\n",
    "cv_scores = cross_val_score(tree_clf, digits_train, labels_train, cv = 10, scoring = \"accuracy\")\n",
    "\n",
    "print(cv_scores)\n",
    "print(\"Cross-Validation Error Estimate: \", 1 - (cv_scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ugvKbG6W9A70"
   },
   "source": [
    "We got around a 16% to 18% cross-validation error rate (about 82% to 84% accuracy) -- that seems pretty good. With random guessing we would have expected about a 90% error rate, so this is much better than that. Let's see how our model does on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 445,
     "status": "ok",
     "timestamp": 1604420819707,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "_VbBlgbh9A71",
    "outputId": "01a309c6-8e57-4a8e-e931-ceb73276112f"
   },
   "outputs": [],
   "source": [
    "tree_clf.fit(digits_train, labels_train)\n",
    "\n",
    "cm = confusion_matrix(labels_test, tree_clf.predict(digits_test))\n",
    "print(\"Confusion Matrix: \\n\", cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 526,
     "status": "ok",
     "timestamp": 1604421039693,
     "user": {
      "displayName": "Adam Gilbert",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1KWqLKG2SAQxx8KvqzqhM94EvqxkXs2iOw7gQWQ=s64",
      "userId": "02074648007699947871"
     },
     "user_tz": 300
    },
    "id": "_rg_1QXU9A7-",
    "outputId": "d4ecab91-6962-4dde-e924-90901bea97a8"
   },
   "outputs": [],
   "source": [
    "correct = 0\n",
    "for i in list(range(10)):\n",
    "    correct = correct + cm[i,i]\n",
    "\n",
    "accuracy = correct/len(digits_test)\n",
    "print(\"Accuracy Rate: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2AGqsFJy9A8C"
   },
   "source": [
    "An accuracy rate of about 85.6% agrees with our cross-validation error rate. Can you identify the numbers that our model has difficulty differentiating between? Did you expect this (or should we have expected it)?\n",
    "\n",
    "Thinking back to our classification tree, while we may not be overfitting is it possible that we are leaving predictive power \"on the table\"? I bet you can do better. Remember that you should be building lots of competing models and using cross-validation to tune hyperparameters and to compare those models. Take some time here to try it, and then see how your best model performs on the *safe* set. Can you get above 90% accuracy?\n",
    "\n",
    "## Summary\n",
    "\n",
    "That's it from me for now. In this notebook you learned how decision tree classifiers work and you applied them in two different scenarios. Decision trees are nice because they are so easily understood -- a non-expert can look at one and understand what is happening. Additionally, decision trees can handle the multiclass case without any extra work. What may not be so great about decision trees is that they don't typically provide us with a measure of how confident we are in our prediction (the *propensity*) -- although we can make them do it. Next week we will look at one more type of model which can be used for classification -- the *logistic regression* model. See you then!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6lNr_l_D9A8E"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "12_ClassificationTrees.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}