Project 2 - LendingClub Optimization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Homework_2_Yunxin.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/karenyxwang/Machine_Learning/blob/master/Project%202%20-%20LendingClub%20Optimization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QzFBHtFHRBkH",
        "colab_type": "text"
      },
      "source": [
        "# Download the Data \n",
        "Keep this if you are working in Google Colab. Delete this if you are working on your own computer and have the data downloaded already. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gD8xgRYrRAV3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        },
        "outputId": "234ad5ab-748d-4207-ea6d-7d75dc1ab259"
      },
      "source": [
        "!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=0B5qTk6DHjanhOV9LRE5DY3l1T2pGemVBNTVQVzVsMlFCcHF3' -O lendingclub.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-02-10 03:34:04--  https://docs.google.com/uc?export=download&id=0B5qTk6DHjanhOV9LRE5DY3l1T2pGemVBNTVQVzVsMlFCcHF3\n",
            "Resolving docs.google.com (docs.google.com)... 172.217.214.113, 172.217.214.102, 172.217.214.138, ...\n",
            "Connecting to docs.google.com (docs.google.com)|172.217.214.113|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-10-5c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/f572fqpvfs1980r71ajirnfln3c45bgf/1581305400000/09819396713149841370/*/0B5qTk6DHjanhOV9LRE5DY3l1T2pGemVBNTVQVzVsMlFCcHF3?e=download [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2020-02-10 03:34:04--  https://doc-10-5c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/f572fqpvfs1980r71ajirnfln3c45bgf/1581305400000/09819396713149841370/*/0B5qTk6DHjanhOV9LRE5DY3l1T2pGemVBNTVQVzVsMlFCcHF3?e=download\n",
            "Resolving doc-10-5c-docs.googleusercontent.com (doc-10-5c-docs.googleusercontent.com)... 172.217.214.132, 2607:f8b0:4001:c05::84\n",
            "Connecting to doc-10-5c-docs.googleusercontent.com (doc-10-5c-docs.googleusercontent.com)|172.217.214.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [text/csv]\n",
            "Saving to: ‘lendingclub.csv’\n",
            "\n",
            "lendingclub.csv         [ <=>                ]   6.03M  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-02-10 03:34:05 (267 MB/s) - ‘lendingclub.csv’ saved [6325329]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-M5G2ryQRSe",
        "colab_type": "text"
      },
      "source": [
        "# Homework 2 (Due February 9, 2020)\n",
        "Last week you trained a machine learning classifier to predict borrower success rates on LendingClub.com, using a set of features that were already present in the dataset. This week you will spend time optimizing that model.\n",
        "\n",
        "Feel free to reuse code from your first homework submission; this does not need to be done from scratch.\n",
        "\n",
        "# Question 1\n",
        "\n",
        "Choose **ONE** of the following three options:\n",
        "\n",
        "   - The `state` column is currently broken into 51 distinct values. Create a new feature, `region`, that groups similar states together.\n",
        "   - The `emp_length` column is currently a nominal value. Create a new feature, `emp_length_numeric`, that treats the variable as a numeric value instead.\n",
        "   - The `title` column is a free-form string and is currently hard or impossible to use as a feature. Define one or more features that capture information about the type of loan being applied for, based on the contents of the title.\n",
        "\n",
        "For the option you have chosen, compare how switching to your new feature in a decision tree model improves or worsens performance compared to using the original representation. Report your quantitative results along with an explanation of your new feature's possible values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sN6W49dCdBUw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "outputId": "44be15be-0c10-4693-d529-1fc4f06f8c78"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn import tree\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, cohen_kappa_score, confusion_matrix, ConfusionMatrixDisplay\n",
        "from sklearn.model_selection import train_test_split, KFold, StratifiedKFold, GroupKFold, GridSearchCV\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "lendingclub = pd.read_csv(\"lendingclub.csv\")\n",
        "lendingclub"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>amount</th>\n",
              "      <th>date</th>\n",
              "      <th>title</th>\n",
              "      <th>fico</th>\n",
              "      <th>dti</th>\n",
              "      <th>zip</th>\n",
              "      <th>state</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>year</th>\n",
              "      <th>outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2500.0</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>bike</td>\n",
              "      <td>740.0</td>\n",
              "      <td>1.00</td>\n",
              "      <td>309xx</td>\n",
              "      <td>GA</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2011</td>\n",
              "      <td>accept</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>12000.0</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Consolidation</td>\n",
              "      <td>675.0</td>\n",
              "      <td>10.78</td>\n",
              "      <td>913xx</td>\n",
              "      <td>CA</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2011</td>\n",
              "      <td>accept</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>21000.0</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Debt Cleanup</td>\n",
              "      <td>705.0</td>\n",
              "      <td>13.22</td>\n",
              "      <td>335xx</td>\n",
              "      <td>FL</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2011</td>\n",
              "      <td>accept</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31825.0</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Debt Consolidation Loan</td>\n",
              "      <td>760.0</td>\n",
              "      <td>14.03</td>\n",
              "      <td>080xx</td>\n",
              "      <td>NJ</td>\n",
              "      <td>5 years</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2011</td>\n",
              "      <td>accept</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12000.0</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Debt Consolidation</td>\n",
              "      <td>725.0</td>\n",
              "      <td>16.70</td>\n",
              "      <td>088xx</td>\n",
              "      <td>NJ</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2011</td>\n",
              "      <td>accept</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81098</th>\n",
              "      <td>2000.0</td>\n",
              "      <td>2012-12-31</td>\n",
              "      <td>other</td>\n",
              "      <td>615.0</td>\n",
              "      <td>56.33</td>\n",
              "      <td>365xx</td>\n",
              "      <td>AL</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2012</td>\n",
              "      <td>reject</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81099</th>\n",
              "      <td>2000.0</td>\n",
              "      <td>2012-12-31</td>\n",
              "      <td>car</td>\n",
              "      <td>572.0</td>\n",
              "      <td>16.90</td>\n",
              "      <td>363xx</td>\n",
              "      <td>AL</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2012</td>\n",
              "      <td>reject</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81100</th>\n",
              "      <td>35000.0</td>\n",
              "      <td>2012-12-31</td>\n",
              "      <td>debt_consolidation</td>\n",
              "      <td>556.0</td>\n",
              "      <td>21.44</td>\n",
              "      <td>347xx</td>\n",
              "      <td>FL</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2012</td>\n",
              "      <td>reject</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81101</th>\n",
              "      <td>25000.0</td>\n",
              "      <td>2012-12-31</td>\n",
              "      <td>other</td>\n",
              "      <td>506.0</td>\n",
              "      <td>12.99</td>\n",
              "      <td>404xx</td>\n",
              "      <td>KY</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2012</td>\n",
              "      <td>reject</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81102</th>\n",
              "      <td>35000.0</td>\n",
              "      <td>2012-12-31</td>\n",
              "      <td>other</td>\n",
              "      <td>597.0</td>\n",
              "      <td>4.13</td>\n",
              "      <td>211xx</td>\n",
              "      <td>MD</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2012</td>\n",
              "      <td>reject</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>81103 rows × 11 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "        amount        date                    title  ...  policy_code  year outcome\n",
              "0       2500.0    Dec-2011                     bike  ...          1.0  2011  accept\n",
              "1      12000.0    Dec-2011            Consolidation  ...          1.0  2011  accept\n",
              "2      21000.0    Dec-2011             Debt Cleanup  ...          1.0  2011  accept\n",
              "3      31825.0    Dec-2011  Debt Consolidation Loan  ...          1.0  2011  accept\n",
              "4      12000.0    Dec-2011       Debt Consolidation  ...          1.0  2011  accept\n",
              "...        ...         ...                      ...  ...          ...   ...     ...\n",
              "81098   2000.0  2012-12-31                    other  ...          0.0  2012  reject\n",
              "81099   2000.0  2012-12-31                      car  ...          0.0  2012  reject\n",
              "81100  35000.0  2012-12-31       debt_consolidation  ...          0.0  2012  reject\n",
              "81101  25000.0  2012-12-31                    other  ...          0.0  2012  reject\n",
              "81102  35000.0  2012-12-31                    other  ...          0.0  2012  reject\n",
              "\n",
              "[81103 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xhe86kjn2634",
        "colab_type": "text"
      },
      "source": [
        "The original model is as follows.\n",
        "\n",
        "Its features include amount (amount requested for their first loan), fico (credit rating of the borrower), dti (ratio of the borrower’s total monthly debt payments), state (U.S. state that the borrower resides in), and emp_length (length of time that the borrower has been employed at their current job). Among the features, 'state' and 'emp_length' are nominal values, and the rest are numerical values.\n",
        "\n",
        "The accuracy of the original model on the train set is 96.1, on the test set is 95.9."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H-zaywVX1Upv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e43cace0-9b82-4c6a-a422-89aee7d41f2b"
      },
      "source": [
        "# List the features from the data\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Prepare our dataset\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "# Use scikit-learn to create train/test split and train the decision tree\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy on the train set: 96.1\n",
            "Accuracy on the test set: 95.9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07vTIBdG25Gt",
        "colab_type": "text"
      },
      "source": [
        "The new column, region, is created to divide states into 5 regions based on their GDP in 2018. The classification method is quantile.\n",
        "\n",
        "States with GDP in the top 20 percentile include CA, TX, NY, FL, IL, PA, OH, NJ, GA, MA, WA.\n",
        "\n",
        "States with GDP in the 20-40 percentile include NC, VA, MI, MD, CO, MN, IN, TN, AZ, WI.\n",
        "\n",
        "States with GDP in the 40-60 percentile include MO, CT, LA, OR, SC, AL, KY, OK, IA, UT.\n",
        "\n",
        "States with GDP in the 60-80 percentile include NV, KS, DC, AR, NE, MS, NM, HI, NH, WV.\n",
        "\n",
        "States with GDP in the last 20 percentile include ID, DE, ME, RI, ND, AK, SD, MT, WY, VT.\n",
        "\n",
        "The new model replaces the 'state' feature in the old model with 'region'. The rest remains the same.\n",
        "\n",
        "The accuracy of the new model on the train set is 96.1, on the test set is 95.9. The model performance stayed the same."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jfiatYYZTQMw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "50f9c1f1-3056-47f6-9e28-3a1d44c82e52"
      },
      "source": [
        "# Create function for creating new column, region\n",
        "def region_column(row):\n",
        "    if row['state'] in (['CA', 'TX', 'NY', 'FL', 'IL', 'PA', 'OH', 'NJ', 'GA', 'MA', 'WA']):\n",
        "        val = \"gdp_20_percentile\"\n",
        "    elif row['state'] in (['NC', 'VA', 'MI', 'MD', 'CO', 'MN', 'IN', 'TN', 'AZ', 'WI']):\n",
        "        val = \"gdp_40_percentile\"\n",
        "    elif row['state'] in (['MO', 'CT', 'LA', 'OR', 'SC', 'AL', 'KY', 'OK', 'IA', 'UT']):\n",
        "        val = \"gdp_60_percentile\"\n",
        "    elif row['state'] in (['NV', 'KS', 'DC', 'AR', 'NE', 'MS', 'NM', 'HI', 'NH', 'WV']):\n",
        "        val = \"gdp_80_percentile\"\n",
        "    elif row['state'] in (['ID', 'DE', 'ME', 'RI', 'ND', 'AK', 'SD', 'MT', 'WY', 'VT']):\n",
        "        val = \"gdp_100_percentile\"\n",
        "    return val\n",
        "\n",
        "# Create new column, region\n",
        "lendingclub['region'] = lendingclub.apply(region_column, axis=1)\n",
        "\n",
        "# List the features from the data\n",
        "features = [\"amount\", \"fico\", \"dti\", \"region\", \"emp_length\"]\n",
        "\n",
        "# Prepare our dataset\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "# Use scikit-learn to create train/test split and train the decision tree\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy on the train set: 96.1\n",
            "Accuracy on the test set: 95.9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DT4rTebQi-Zy",
        "colab_type": "text"
      },
      "source": [
        "The new column, emp_length_numeric, is created to transform the emp_length column from nominal value to numeric value.\n",
        "\n",
        "The values in the new column, emp_length_numeric, include [0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], which respectively replaced ['< 1 year', '1 year', '2 years', '3 years', '4 years', '5 years', '6 years', '7 years', '8 years', '9 years', '10+ years'] in the old column, emp_length.\n",
        "\n",
        "The new model replaces the 'emp_length' feature in the old model with 'emp_length_numeric'. The rest remains the same.\n",
        "\n",
        "The accuracy of the new model on the train set is 95.8, on the test set is 95.6. The model performance dropped slightly.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rd-ydN8eNJxJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "942aebfb-b77c-46c3-833e-f6e19130f27b"
      },
      "source": [
        "# Create function for creating new column, emp_length_numeric\n",
        "def emp_length_numeric_column(row):\n",
        "    if row['emp_length'] == '< 1 year':\n",
        "        val = 0.5\n",
        "    elif row['emp_length'] == '1 year':\n",
        "        val = 1\n",
        "    elif row['emp_length'] == '2 years':\n",
        "        val = 2\n",
        "    elif row['emp_length'] == '3 years':\n",
        "        val = 3\n",
        "    elif row['emp_length'] == '4 years':\n",
        "        val = 4\n",
        "    elif row['emp_length'] == '5 years':\n",
        "        val = 5\n",
        "    elif row['emp_length'] == '6 years':\n",
        "        val = 6\n",
        "    elif row['emp_length'] == '7 years':\n",
        "        val = 7\n",
        "    elif row['emp_length'] == '8 years':\n",
        "        val = 8\n",
        "    elif row['emp_length'] == '9 years':\n",
        "        val = 9\n",
        "    elif row['emp_length'] == '10+ years':\n",
        "        val = 10\n",
        "    return val\n",
        "\n",
        "# Creating new column emp_length_numeric\n",
        "lendingclub['emp_length_numeric'] = lendingclub.apply(emp_length_numeric_column, axis=1)\n",
        "\n",
        "# List the features from the data\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length_numeric\"]\n",
        "\n",
        "# Prepare our dataset\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "# Use scikit-learn to create train/test split and train the decision tree\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy on the train set: 95.8\n",
            "Accuracy on the test set: 95.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hcEdh46SlJp7",
        "colab_type": "text"
      },
      "source": [
        "The new column, title_length, is created to measure the length of the titles. The rationale is that the longer the title, the more it shows that the loan borrower is more serious about his/her loan and is willing to explain the loan usage in detail.\n",
        "\n",
        "The value of the new column varies from a mininum of 1 to a maximum of 908. However, the majority of the lengths of titles (75%) was below 18 words. The mean is 13.5, meaning that, on average, loan borrowers are willing to use 13-14 words to describe their purpose."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SriX-h0zWzN6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "157c2f06-d1c7-450f-d9e9-920a5fab9f02"
      },
      "source": [
        "# Create function for creating new column, title_length\n",
        "def title_length_column(row):\n",
        "    val = len(row['title'])\n",
        "    return val\n",
        "\n",
        "# Create new column, title length, and describe the column\n",
        "lendingclub['title_length'] = lendingclub.apply(title_length_column, axis=1)\n",
        "lendingclub['title_length'].describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    81103.000000\n",
              "mean        13.505123\n",
              "std          8.979191\n",
              "min          1.000000\n",
              "25%          7.000000\n",
              "50%         14.000000\n",
              "75%         18.000000\n",
              "max        908.000000\n",
              "Name: title_length, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_rN8gK7Tmf09",
        "colab_type": "text"
      },
      "source": [
        "The new model adds the feature 'title_length'. The rest remains the same.\n",
        "\n",
        "The accuracy of the new model on the train set is 96.1, on the test set is 95.9. The model performance stayed the same with the original model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1NvQQOILlxpv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "d05d6bef-5233-4552-af61-9197a811bc5c"
      },
      "source": [
        "# List the features from the data\n",
        "features = ['title_length', \"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Prepare our dataset\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "# Use scikit-learn to create train/test split and train the decision tree\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy on the train set: 96.1\n",
            "Accuracy on the test set: 95.9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2ZEWAqydB47",
        "colab_type": "text"
      },
      "source": [
        "# Question 2\n",
        "\n",
        "Choose **THREE** of the following five options:\n",
        "1. Amount of data available for training.\n",
        "2. Subset of features included in the model.\n",
        "3. Decision tree vs. logistic regression classification.\n",
        "4. Hyperparameter tuning for the best-performing model from option 3.\n",
        "5. Number of folds or stratification strategy for cross-validation.\n",
        "\n",
        "For each option you choose, write code to see how much impact that option has on model performance. For instance, you might choose to measure prediction accuracy as the amount of available data is increased from 1% of the total training set to 100%.\n",
        "\n",
        "Note that for logistic regression classification, the default decision boundary may not produce the best model; you will likely want to check different decision boundaries if you choose options 3 and 4.\n",
        "\n",
        "Rank the three options you chose from the greatest impact on model performance to the least impact. Justify your ranking, based on specific quantitative experimental evidence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZWjqthztOFT",
        "colab_type": "text"
      },
      "source": [
        "**I did question 3 and 4 for extra credit. If there are too many mistakes for these two questions. Please prioritize question 1, 2, 5 when grading.**\n",
        "\n",
        "1.Amount of data available for training:\n",
        "\n",
        "In this part I adjusted the size of the lendingclub dataset and calculated how model accuracy varies with different sized data sets. The results show that testing accuracy increases, from as low as 87.38 to as high as more than 96, as the size of the dataset increases from 0 to 80000."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0wiABad_lYJb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "outputId": "999a46ab-c5ae-4452-d52e-05f302cdbd19"
      },
      "source": [
        "size = []\n",
        "train_accuracies = []\n",
        "test_accuracies = []\n",
        "accuracies_table = pd.DataFrame()\n",
        "\n",
        "# Select features\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Calculate accuracy of the model for different sized data sets\n",
        "for i in range(100):\n",
        "\n",
        "    # Prepare the small dataset\n",
        "    lendingclub_small = lendingclub.iloc[lendingclub.index % 100 <= i]\n",
        "    X = lendingclub_small.loc[:, features]\n",
        "    X = pd.get_dummies(X)\n",
        "    y = lendingclub_small[\"outcome\"]\n",
        "\n",
        "    # Use scikit-learn to create train/test split and train the decision tree\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "    model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "    train_pred = model.predict(X_train)\n",
        "    test_pred = model.predict(X_test)\n",
        "\n",
        "    # Caluclate accuracy\n",
        "    train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "    test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "\n",
        "    size.append(str(i+1) + '%')\n",
        "    train_accuracies.append(train_accuracy)\n",
        "    test_accuracies.append(test_accuracy)\n",
        "\n",
        "# Produce the result in a table\n",
        "accuracies_table['size'] = size\n",
        "accuracies_table['train_accuracy'] = train_accuracies\n",
        "accuracies_table['test_accuracy'] = test_accuracies\n",
        "accuracies_table"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>size</th>\n",
              "      <th>train_accuracy</th>\n",
              "      <th>test_accuracy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1%</td>\n",
              "      <td>95.377504</td>\n",
              "      <td>91.411043</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2%</td>\n",
              "      <td>94.919169</td>\n",
              "      <td>87.384615</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3%</td>\n",
              "      <td>96.560575</td>\n",
              "      <td>92.008197</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4%</td>\n",
              "      <td>96.264921</td>\n",
              "      <td>91.692308</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5%</td>\n",
              "      <td>96.241528</td>\n",
              "      <td>91.502463</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>96%</td>\n",
              "      <td>96.318654</td>\n",
              "      <td>95.960699</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>97%</td>\n",
              "      <td>96.130990</td>\n",
              "      <td>95.455701</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>98%</td>\n",
              "      <td>95.947094</td>\n",
              "      <td>95.646977</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>99%</td>\n",
              "      <td>96.257375</td>\n",
              "      <td>96.120555</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>100%</td>\n",
              "      <td>96.114485</td>\n",
              "      <td>95.912706</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>100 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    size  train_accuracy  test_accuracy\n",
              "0     1%       95.377504      91.411043\n",
              "1     2%       94.919169      87.384615\n",
              "2     3%       96.560575      92.008197\n",
              "3     4%       96.264921      91.692308\n",
              "4     5%       96.241528      91.502463\n",
              "..   ...             ...            ...\n",
              "95   96%       96.318654      95.960699\n",
              "96   97%       96.130990      95.455701\n",
              "97   98%       95.947094      95.646977\n",
              "98   99%       96.257375      96.120555\n",
              "99  100%       96.114485      95.912706\n",
              "\n",
              "[100 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bfm0RR-5Htfg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "b792c939-579f-4fa1-dc7e-b51b4352c170"
      },
      "source": [
        "# Plot the results\n",
        "ax = plt.gca()\n",
        "x_index = [x * (len(lendingclub)/100) for x in range(100)]\n",
        "ax.scatter(x_index, train_accuracies)\n",
        "ax.scatter(x_index, test_accuracies)\n",
        "plt.ylabelbest = 0\n",
        "plt.ylabel(\"Accuracy on Example Data\")\n",
        "plt.xlabel(\"Dataset Size\")\n",
        "plt.legend([\"Training Accuracy\", \"Testing Accuracy\"])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEJCAYAAACT/UyFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deZgU1dW43zMwyKDIAOICiKImKrI7\nEhQ0cQMTF/hQ3ONCFI07iQuJ+RT5vijG/KLiGhfUfCqLIkSTKEY0RjSgoMOiiKCCsqjsbqPMwPn9\nUdVNT09VdfVSPd3T532efqb79q1bp2qqz7333HPOFVXFMAzDKB3KGlsAwzAMI7+Y4jcMwygxTPEb\nhmGUGKb4DcMwSgxT/IZhGCWGKX7DMIwSI1LFLyJXisgiEXlXRK5yy3qLyGwRqRaRuSLSL0oZDMMw\njPpEpvhFpDtwIdAP6AWcICL7AX8AblLV3sAN7mfDMAwjTzSPsO0DgTmq+i2AiLwKDAMU2Nmt0wZY\nnaqhXXbZRffee++IxDQMw2iazJs3b52qdkgul6gid0XkQOCvwKFADTATmAvcC8wABGfGcZiqrvA4\nfiQwEqBLly4Hr1jRoIphGIYRgIjMU9Wq5PLITD2quhi4FXgReAGoBrYCvwRGqeqewCjgYZ/jH1DV\nKlWt6tChQYdlGIZhZEiki7uq+rCqHqyqRwAbgQ+Ac4Fn3CpP4awBGIZhGHkiaq+eXd2/XXDs+0/i\n2PR/7FY5ClgapQyGYRhGfaJc3AWYKiLtgVrgUlXdJCIXAneKSHPgO1w7vmEYhpEfIlX8qnq4R9ks\n4OAoz2sYhmH4Y5G7hmEYJUbUph7DAGD6O6u4bcYSVm+qoWNlBdcM3p+hfTo1tliGUZKY4jciZ/o7\nq/jNMwupqd0KwKpNNfzmmYUApvwNoxEwxW9Ezm0zlsSVfoya2q3cNmNJ/HubCRhG/jDFb0TO6k01\nnuWxkb/NBAwjv9jirhE5HSsrPMubiQTOBAzDiAZT/EbkXDN4fyrKm9UrqyhvxlafPFF+MwTDMHKD\nmXpKlEy8bDL1zInVST72thlLWOWh5P1mCJlg3kSG0RBT/CVIJl422XrmDO3TybNeYpvgzASuGbx/\noOxhFbl5ExmGN6b4i5hMR7NBXjZ+x2dyTCr8ZgK5UuTmTWQY3pjiL1IyHbX7mVegoW09sWPx27Vh\n9aaarMwpfjMBL9LtfMybyDC8McVfpIRVgonKXsBXgUN923pyx+JHm4rySJSoV2fip8j9yjtWVnh2\nckHeROnKbGsIRjFiir+RCaM4MlWCyco7SOnHbOupZgWJCLCpprZBebbmFL/ZTGWrcjZ+2/B8fovB\n1wze33MNwa8zC/Im8vofADZzMIoSU/yNSBhzTTZK0GtW4EUnH0UWRKrZQzbmFL/ZzA7Nyxoo7qDF\n4Fx5E3n9D0ZNrva8/mzXPQwjH5jib0TCmGuyUYJh/OE7VVbw+uijABgw7uWUSr+TqxxTzQiyMaf4\nyb25ppbbT+vtOYvwmzml603k1Y7X/yCo0wsbh2BmIqOxMMXfiIQx12SjBIOUE6TfUcTqj5pcnbJe\nJuaUGH62+Y6VFZ6KPN2Fbr+ZAHibbsLMgJLlTKXUS93V1Dq9xsUUfyMSpODC1AmjBJOJmWg6pdlR\nJNYPWgPolKE5JRE/27yfSScTV1Ove+c146mp3UozEd8o42Qqyptx5AEdUir1KNxji4VS7/QKAVP8\njUCQp02ygvNSgoLzYxkw7uUGI6Ugu36nECPPZCrKm3HLsB71jvFTzLcM6xGXIejaUo320vXvT9fb\nxw+/+ltVU65pAFRWlCMCj8/+pMF3yQveYV1qmyKl3OkVCqb484yXp43XKDxGohJMVqaJI6XEOl4I\nxG35iaTbUSTLFGQq8bq25Dpe1+Bnuho1udqzEwgzcwqDXzvJ1+LVoZ18cCemzlsVaBYKYzrKZbqK\nbIjSFJOrjtrIHNGQU9jGpKqqSufOnZvzdhvDzjhg3MueyiVxkTXdYysryvm+blugQvFrv+vov3uO\nZAX4eNzxgfKElS95ATnsNfgp1ORZiNesxWumkoowsQuJpqwwnkKJpDIZBQ0AUskdk6eNO+vY9G1t\nxs90Jvcznd9SNr8BIz1EZJ6qViWXl+yIv7HsjGFHO+n47nv50icSZB/P1WgZslus9osHmDjn0wbK\nMtkskK5pyI/k2ZXftXitD2Sz4A00mMmNmlzNVZOrU3YCyc9x4n1M95kOiuFIJzgw1XnTXcMxck/J\nKv582Bm9lHcYRZuu734QqRRHLn+E2SxW+xE2dXM6qR+CiLXjNyr16xCDrivVgrfXTCDInJfK1TSR\nsM90mNlOOsGBYYL4zKun8ShZxZ/NyDvZb9xreg3etmw/00Wiok3Xd79leZlnhxBm6pzLH2GYTsSv\njt81+JlHwrhMJpNO/XQ7xKAF78RzpBNJHKOmditjnn23niksHVfTMLbzMMF+CnGHgjD1UwXxZaro\nzRU0e0pW8Wcz8p67YkM95e01vW5ZXuapvF95fy23DOsR+OCm67sP6ac3TiSXo2UI7kTCLgzHrsGv\nowxymQzTfqa+/kHuoX71kwcJLcvL6g0SwqwP+JnCwhDGbBd2YTUoajmZTIL4Ug200jUrhW2/1CjZ\nxd0wC1h+0/10/Lq98DO/pMqTEzSCbwoPdJjZVSpl6bdIHDQr8lqszaW5L9VzFjYhXiZ4zTrSuZ+Z\nksr99byd3uTa8sm0qvkM2nSGo29g+tYBoTv/ZMLMbqNetI7i+GzxW9wtWcUP/t4Qsffp2tPTId0f\nfyZeKk0ZP2+kTPAyn+XqXof1YEkni2oQlR7PcfIzHdYdNYwZKhE/d9dkTiqbxbjyh2glW7YXllcw\nRi/i0a/7NagfZqAVxgstXW+ibL3Fwh4fZefgp/hLes/doX068froo7j9tN58X7eNjd/WojjT6iCl\n30wk63MnLn5Ban96U/r1yZW/e9QbvoddS4o9i8vHHc/tp/WO50RKBwGqbxzEOzcMCnymvRZiYybI\nTpUVCNufubBydKqsiMudqtO6tvmU+kofoLaGC7Y87lk/zOw6tuYzYNzLdB39dwaMe5np76yqVyfd\n+IFUG/mkIszxsc5h1SZnz4uY6SpZ9lxTsjb+RMJmsYTwU88whHFz9Au8KnXSXST2MwFlk1MoDJm4\ny6byLAo6V4x0nmnwd1OF1BlbE0fMqdxaATrKOp/y9Z7lzUQ4Xl7j2uZT6CjrWK278Ie6U3l220Ag\n/JqPX/fh97/INtAszPGNFcVc0iP+GGH/kbFR0P8O7VFvdFRZUU7bVuVpnzfZzTFVHWM7Q/t08hyh\n3njiQVSUN6tXt6K8GWNOOiitEW2u7vs1g/f3lCfMwrvfsWf375KyzXQ7Lr/rTbzP4AxEgs7r107i\nLHm17uJZZ7W2b1AmwPHyGuPKH6Jz2TrKBDqXrWNc+UOcVDYr/n985f21ngp0zLPvxkfUXqSKcUmn\nPGy9mHfU9HdWsXpTDSeVzWJWiyv4aIczmdXiCk4qmxVPyRLVyL+kbfwxwoysch1ZG1X0qZHdhuyQ\n+/uejQ03nQXvxDbTmS2ka7dOJ+torP3EWbKXjf9bbcHo2gvio3jYvlYwq8UVdC7zmCW02RNGLQIy\nW/NJNzguUaagY8Ou11SUN2Nos9f5b73f915k+yw2yuKuiFwJXIhzvx5U1Tvc8suBS4GtwN9V9dqg\ndqJW/LlaWA1SIpDaNbCxPQBKlaZ43zPJ0prLc6fqrM51vXpa1nzGxm07IgKVfB034/xdD4/b9j/a\n4UzKPJfVBMZsAsJ3dCeVzYqbjMoq94Sjb4Cep6a8Fr+kg14LtX6dhRdvtLyCjjTs1FZu24WBW8YD\n2aWyyLviF5HuwCSgH7AFeAG4GNgTuB44XlW/F5FdVfWLoLaiVvyQu3wnTVGJGEXEgikwcyxsXsm3\nFbvzh9rTeOzrftnl8EloM+Z6Sc9T0y/34K1n/0z3eb+jImHEW6MtuC5h9B9mxD/9nVXMmnYvVzEp\nvg5wW92p/DVhBuHnTcSJ4wOVPwR3LImdZ3K9xI4meW0C/Du1bSrs8/0TQGZ5s2I0huIfDhynqr9w\nP/838D1QBTygqi+FbSsfit8wCp5UCnXBFHjuCqhNUFAhFVvgOb3a7HUmzH8yfLmfDLd3h82fNij+\njA70/+5OIFhhT986gNtmLKHqy38yrsVDVOBvPgrqQKb/ZEbggC2VKam8TNipZfN6jgVhTFp+MqnC\nKrejmLfzsUU14j8Q+CtwKFADzATmAoe75ccB3wFXq+pbHsePBEYCdOnS5eAVK1ZEImc62GjeaDTC\nKHUfJZo4Mk55juSOZeZY7zalGaiHGcmv3E+GMZV4GUIUZxH41lpnhHxS2SyuK59CR1mPeAR8+SnQ\nRJOJ3+haEbptneS9ztPsdZg5lm2bV7J6W/sGI/Ygwsjk2aklUKMtWHTw/3LISReFOmcyec/OqaqL\nReRW4EXgG6Aax6bfHGgH9AcOAaaIyD6a1AOp6gPAA+CM+KOSMyy2a5DRqMwcW1/pg/N55tjtin/z\nSu9j/coTSe5YNn8Kz4zE1zrtpdyDyv1kaNPZs2MRoJOs49YWDyNbYO7Ox/LW4MvqbzqUsGNakIto\nzNziF33zObt4egRV//0BhsqfobaGMrZ7E1FLKOUfxm312W0DodaJbegk60gOEaqQLRzy4V1AZorf\nj0jdOVX1YVU9WFWPADYCHwArgWfU4U1gG+Dt31VAZBvMYRQhC6Y4o+gxlc7fBVMaT5YwSr1NZ+86\nbTqnvhavjiVnsdFuW7d2dV6JMhx9gzNz8aGC77mzw3O8PvqoBgOsRFdIP6W+UXeMu4J6xl2WV3DL\nluGex16w5fEG96SVbOHO8nvjbpdB+LmtfiH1y5/dNpCBW8ajfoGhYTruNIlU8YvIru7fLsAw4Elg\nOnCkW/5DoAV4LGsXGLZrUIkRGwFv/hRQ5+9zVzSe8g9S6jE8lahsH70HXUsEyqUBNRucV6IM4Jir\n2uzpf5yPbOfu9GagUt+m0E6+9jWj0GZPOHE8c3c+1vPrjmXeAWWSFEvgx0Mtzm74/yiv4NO+13jG\nYXxXsbuPnD7/+yxIqfhFpIOI/FFE/iEiL8deIdufKiLvAc8Bl6rqJmACsI+ILMLx+jk32cxTiFiA\nVYkRZFppDIKUemz03PPUJCWa6EiY9BNLvpYIlEtKEk1Voxb5K/9k2dzZy411d3gqdXVfZYL3KB8A\ncc7Z81TfQDlfRezSSrZwbfMpVFaUU96s/okqypvR+/iRCf8PiXc0h5x0EX85ZAWzW17JRzucyeyW\nV/KXQ1bQ6qdjPTsKjr4hUI5MCDPifwJYDHQFbgKWAw0WY71Q1cNVtZuq9lLVmW7ZFlU9W1W7q2pf\nVQ3biTQq2URgGkVINvbyTEhliglS6rER/Zg2jiI9+ga3XorxVOK1pDC5OKdsFvx9JqSSIVnxJczE\n/HS60DDCuAEJncnQPp3CK+IkOpetp/rGQdx2Sq8GUeFD+3Ta3qmN2RTvaFgwhUMW3sjurKVMYHfW\ncsjCG50GPTqKjD2yAkjp1eOuCh8sIgtUtadb9paqHpJzaXwoFHdO8+opIbL1kPHDy3MG0nPD9JMt\n8dgG9noPkq8lLtunNAg7CnLVbF7hmnCS8PPwCSWDj8tqqmsPQ/K9DfKYAn/PpsRr+MEgWPpiqNiF\nyJ4tDzJ25xSR2araX0RmAOOB1cDTqrpvTiUMoFAUv1GEpBFMVL++j/KLwifeT3Gm6QJZj1RKN9W1\npBOcBeF9/dORwYtU1x50P8G5pyE7k5XbduG0Vg86A7xmrze8xiCCrs33GrZHIueKbNw5/1dE2gC/\nBu4Cdgauyql0hhEFXi6KsQVFrx9kA8WckF3eS2HEjgnbsfitG/gpk80rvdv3cYGsh271GPmnuJZE\nep7q/b1fOXjfhy79t5dXtHXq1WwM1wl7EXTtseuC9GZQPua7jrJ+u9v2sAEMPXF86tF/jNoaeObC\n7aa3xE7Tr+PK4zpLmBH/AFV9PVVZlGQ74jcTTYmS7pQ63fqhTAQJijDIL96LinZQV5P+SDomczwA\nK+RspxgIG52cToccMOL3zJcTZsaVSFSznxBkM+K/C+gboqwgscCrEibdBdp0y/1G8M9fV19hx2Ya\nFW39TRBeZqVYe8ntL33RURJBJqmYovOb2RRDhxAkZyr5g2YmyRx9Q4PO5FttwR/qth9fz207zIwr\nkdoamPeov+ktzAwsx/gqfhE5FDgM6CAiv0r4amcgguX9aGisjQ6MAsDvBxrkE59Ofb8OwUu519Y4\ntmffhdcEs1JFO/92YudNVGzpKPJ0zV+NRSo5cylrQmeybfPKeLbQO8rv5VqdEs+XE8ejo0iJ73qL\n5HxBNwxB7pwtgJ1wOofWCa8vgVOiFy03WOBVCRPkHujlPhnGnTCRdG2yNRtTBCvpdvOO78zA47xe\nLoN+FFp8gh/5ltO9h/P63kor2UI7+Tq+6cut5Q9xR7el9esmu11W/SI4CM3PFbYx4icIUPyq+qqq\n3gT0V9WbEl5/UtWlfscVGhZ4VcJ4/UBj9nevqFxIz4/ar6OIjdiTadN5u5L28zSv2RA8ksw2oMdn\nlrJt08pId3xKm3zHUbgc8uFd9VJEQ2K+nASSO9sT/uT8Hfag9zNx8Hl5C84KQxgb/7cichtwENAy\nVqiqRbERrN/erBZ4VSJ4mQVu7+4/mkw1Yob6ppWKtq77YIKnCngvQCb+yNO1E0NubME+512t7Qtr\n/Stds1uuyLbDCVqDSPRwauS1lTCK/wlgMnACzkYq5wJroxQql8QeYPPqMeLkMotlzQZHqQ97oOGP\nONmN8ZmR2937vOzEmfj0p0uKhcyCWf/yuz9Rj5Bz0eFk4gqbZyxy1yg9somcjNrlE3K/mYqXTDPH\nsm3TSlZrwxzz2ez4lFMaw/sois1sGpFs3DljW8qsEZHjcSJ3fYyYhlEEZDOazJXLZyqzUpQKzx15\nHu6znWDBrH81xgg5rLtokZNp5O6oSKWKEAvmMrL6cefK5TPIrJQnhVfq61++uqCATDJRkVLxq+rf\n3LebcfPoFysWzGXESffHnSqHT5DLZ2MsUoaglNe/Sl0XBCp+ETkSuAw4wC1aDNytqv+KWK5IsGAu\nIyMyyeETo7EWKUMytE+nknz2S10XBEXuHg/cDYx1X4KTpmGCiFymqv/Ij4i5w4K5jIzw25YwzGJw\nidiMi41S1wVBI/5rgKGqOj+hrFpE5uLY+otO8XesrCjsxSyjMMmFb3ehKvpiyduTY0pdFwSlbNg9\nSekDoKoLgN2iEyk6bBctIyPC7HcLhbU5exgKbV/hPFLquiBI8X+T4XcFy9A+nbhlWA/vLdKM4iZK\npZvmloBFo0SLJW9PBJS6LvAN4BKRTcC/vb4CBqpq2ygFS8QCuIxA8hF0k+mWgBFsp5cz8rgTlNE4\nZBLANSTguz9mL5JhZICXAg4aueZK8aey0zdSUrGsKGBXUyNafBW/qr6aT0EMIyV+Odp9ty781BmJ\n52PBshiVaIG7mhrREWTjN4z6NPbipd/I3i/XOeTP1p5uLv9CwC9tdQl49ZQ6YVI2GEZh7NzkZzbx\n3Fg8gVybfbwoVn/9QnY1NSIjtOIXkVaq+m2UwhgFTLZ29Fz4i/uaUxI3FvfJcZ8PW7spUaNISGnq\nEZHDROQ94H33cy8RuTdyyYzCIhc57LN1dQwyp8R2RPLb/q6Qbe2GkWfC2PhvBwYD6wHcoK4johTK\nKEDCBjF5kSt/8TA26Uxs7Y29dmEYeSaUqUdVPxWpt0eo35bxRlMlqhz26ZqAUplT0rW1F8LahdE4\nlGi6Cgin+D8VkcMAFZFy4EqcLJ1GKRFFDvuKtv5KN9NzxWQNWzcfMQBG4VHiHX6YrRd3Ae4EjsGJ\n2n0RuFJV10cvnoNF7hY5fpG1fvvLVrSDupr8bH9n0aulSTFGWmeAX+RuShu/qq5T1bNUdTdV3VVV\nzw6r9EXkShFZJCLvishVSd/9WkTU7ViMpoyfbb5mo3f9mg35yyGTzdqFUbwUY6R1DgnKx38X3kMh\nAFT1Cr/v3OO7AxcC/YAtwAsi8jdVXSYiewKDgE8yktooPrzML0Hul15E8aO06NXSpBgjrXNI0Ih/\nLjAv4JWKA4E5qvqtqtYBrwLD3O9uB64loGMxSgA/D5yKdt71o/hRWvRqaVKMkdY5JChXz2OJn0Vk\nZ6dYvwrZ9iLg9yLSHqgBfgbMFZEhwCpVnZ/kKVQPERkJjATo0qVLyFMaRYXfgjHkdxRugVelR7FG\nWueIMIu7VcAjQGucxd1NwAhVTTnqF5FfAJfg5O9/F2gG9AIGqepmEVkOVKnquqB2bHG3BClhVzvD\nyBV+i7thFP8C4FJVfc39PBC4V1V7pinAzcDnwPVALPVDZ2A10E9VP/M71hS/kROsMzFKjEzy8cfY\nGlP6AKo6S0TqQp50V1X9QkS64Nj3+6vqnQnfLyfEiN8wsqbE/bYNI5Ewiv9VEfkzMBFnMfY04F8i\n0hdAVd8OOHaqa+OvxZk1mGO0kT65GKlboJZhxAmj+Hu5f29MKu+D0xEc5Xegqh4e1LCq7h3i/EYp\nk6uReon7bRtGIikVv6oemQ9BjBzSlGzZuRqpl7jftmEkklLxi0glcA6wd2L9VAFcRiPR1GzZuRqp\nW6CWYcQJk5b5HzhKfyHpBXAZjUGuUiAXCrlKqWCBWoYRJ4yNv6Wq/ipySYzckMsUyJmQ63PkcqRu\ngVqGAYTz4x8FfA38Dfg+Vq6qHmkVo8H8+NPAL+ugV8ZLBNDtWxdmqxT9snBmO7JuSmsWhpFHsgng\nuhT4PU7Ebqyyquo+OZfSB1P8aZBuCuTEOtkq6BJJdWsYxULGaZmBXwP7qereqtrVfeVN6Rtpkm4K\n5Bi5WAcwl0nDKArC2PiXsT3FglEMZJoCOVsFbS6ThlEUhBnxfwNUi8ifRWR87BW1YEaO8UpDm0y2\nCrrEU90aRrEQZsQ/3X0ZhUY6i5710tB+SnxhN0YuFHRyqtuKts7nZ0Y6ZbYoaxgFQcrF3ULAFnc9\nyNaDJmpPmag8fAzDCE02Xj0/AG4BugEtY+Xm1dPIFLoHTaHLZxglQDZpmR/BSdB2O3AkcD7h1gaM\nKCkkDxqv2UMhyWcYRj3CKPAKVZ2JMztYoapjgOOjFctISa5SGWRLzKSz+VNAt+cGitn3kzEPH8No\ndMIo/u9FpAxYKiKXich/ATtFLJeRirAeNAumOGaXMZXO3wVTciuHX26gmDyp5DMMI++EUfxXAq2A\nK4CDgZ8D50YplBGCMEnH/EbjuVT+fqabmo2WFM0wCpQwi7stVfW7pLJd8rldoi3uZkiUC6xxu75P\nUJgt4hpGo5NNyoa3RKR/QkMnA2/kUjgjIqJaYK03k/DATDqGUdCE8eo5E5ggIv8COgLtCdhu0Sgg\nokqh4GXXj7edo0yfhmFERpitFxeKyO+B/wO+Ao5QVfPJKwai2nXKd8YgZt4xjCIgpalHRB4GrgJ6\n4vjw/81N1WwUOlHtOlUorqSGYWREGFPPQuACdVaBPxaRHwF/ilYsI2dEseuU7V9rGEWN74hfRHYG\nUNU7NMH1R1U3AzflQTajUIlqJpFuzEHUMQqG0UQJGvH/C+gLICIzVfXohO+mx74zmhhhk7fleiaR\nnNQtFnMQO1e29Q3DiBNk45eE9+0CvjOaCvkI+IqdJ3mk7hcB7LcrWLr1DcOIE6T41ee912ejKeCn\nTJ+5MHemFL/OxS8mIN1YBEsCZxgpCTL17Coiv8IZ3cfe437uELlkRv4JUpq5MqX4dS7SDHRrw/pB\nHkS2zaNhZETQiP9BoDVOQrbY+9jnh6IXzfAkygXNVEozyg3ZdWt6Sd1sm0fDyBjfEb+qmudOoRHV\ngma9vDtJWzImE9mG7G7Eb0ZbSUa0i5hhNFEi3XpRRK4ELsTRJg+q6h0ichtwIrAF+BA4X1U3BbVj\nSdpcoki65rVFYpDyzzb5mm3JaBh5I5skbZmesDuO0u8H9AJOEJH9gH8C3VW1J/AB8JuoZEhJrswm\nUfuTx9pPdwE0DJ55dxQq2kVjSokqBsAwjNCEidzNlAOBOar6LYCIvAoMU9U/JNSZDZwSoQz+5Mps\nErU/ueeIPIlsFjSD8ukPeyAaU0oU0cSGYYQmpeIXkR2Ak4G9E+uraqpVvkXA70WkPVAD/AxItteM\nACanIW/uCPIDT0cp5aqddNpPJNtReJB3jClow2iShDH1/BUYAtQB3yS8AlHVxcCtwIvAC0A1EPfX\nE5Hr3Taf8DpeREaKyFwRmbt27doQYqZJrvzAo/YnD2onF2YS844xjJIjjKmns6oel0njqvow8DCA\niNwMrHTfnwecABytPqvLqvoA8AA4i7uZnD+QXPmBR+1PHuQFk4sUyOYdYxglRxjF/4aI9FDVhek2\nLiK7quoXItIFGAb0F5HjgGuBH8fs/41CrjJMRpWpMsjFMtcjcjPpGEZJEcbUMxCYJyJLRGSBiCwU\nkQUh258qIu8BzwGXum6bd+MEgv1TRKpF5P7MRM+SZO+SinbQvAKeGZmeZ06Ql0qm3j4NtjZU4umR\nzAvGMIwsCbPZ+l5e5aq6IhKJPIjcjz8K3/Js2oxyk3TDMEqGjP34XQVfiRN0dSJQmU+lnxeiyPSY\nTZuWgMwwjAgJs/XilTieN7u6r8dF5PKoBcsrvor208wDsrJR3ra1oWEYERLGxv8L4EeqeoOq3gD0\nx4nIbToEKdRMc9JnorzrRegmbXlgLpaGYeSIMIpfSPC/d98X10YsqRZZvXzZE0nH7JOp8rYFXcMw\n8kQYd85HgDkiMs39PBTXN78oCJNSoZ4vexb5cBos6MaUt27PPumnvP1y5oRd0A27ZaJhGCVPmMXd\nPwHnAxvc1/mqekfUguWMsIusPU91FGybPb3bCWNfT6W8gxRxNmsC+doy0TCMJkGo7Jyq+raqjndf\n70QtVE5JV6Fmk8KgsRZ0bf9ZwzDSILK0zAVDugo1m7TB2SjvxupwDMMoOaJMy1wYZJJSIdMUBtmk\nb8gmZ47tP2sYRhqESct8OfC4qm7Mgzy5J59JyLI9V2N0OIZhlBxhRvy7AW+JyNvABGCGX0bNgiWf\nScj8zhWl141l2DQMIw1C7aXUTFMAABrpSURBVLkrIgIMwvHuqQKmAA+r6ofRiudQ9Hvu2j6zhmE0\nAlntueuO8D9zX3VAW+BpEflD4IGGg3ndGIZRQISx8V8JnAOsAx4CrlHVWhEpA5bi5NY3gjCvG8Mw\nCogwNv52OJuk18vIqarbROSEaMRqYpjXjWEYBUQYU8/zOBG7AIjIziLyI4jvq2ukwva1NQyjgAij\n+O8Dvk74/LVbVnpkuqNWrnb7MgzDyAGhsnMmum+q6jZKIfArmWzz4cRyAQ17AOpqoGbD9naeGQlj\n2lgnYBhGXgij+D8SkStEpNx9XQl8FLVgBUeuPHP8ErmBJVczDCMvhFH8FwOHAauAlcCPgJFRClWQ\n5GqXrlSePObmaRhGxKQ02ajqF8DpeZClsPHzzAHvHP+ZtBNvz9w8DcOIjjB77rYUkUtF5F4RmRB7\n5UO4giJXu3SlagfMzdMwjEgJY+r5P2B3YDDwKtAZ+CpKoQqSep45PoQZqTdox/bWNQwjv4RR/Pup\n6n8D36jqY8DxOHb+0iMXu3QltjNms+Plk0nuf8MwjAwJ45ZZ6/7dJCLdcfL17BqdSEVALtMg5zNz\nqGEYBuEU/wMi0hb4HfAssBPw35FKVehYGmTDMIqYQMXvJmL70t2E5d/APnmRqhiwkbphGEVKoI3f\njdK17JuGYRhNiDCLuy+JyNUisqeItIu9IpfMMAzDiIQwNv7T3L+XJpQpZvYxDMMoSsJE7nbNtHE3\nr8+FOM7qD6rqHe5sYTKwN7AcOLVoN3I3DMMoQsLswHWOV7mq/iXFcd1xlH4/YAvwgoj8DSfPz0xV\nHScio4HRwHXpCm4YhmFkRhhTzyEJ71sCRwNvA4GKHzgQmKOq3wKIyKvAMGAI8BO3zmPAvyhkxb9g\nirltGobRpAhj6rk88bOIVAKTQrS9CPi9iLQHaoCfAXOB3VR1jVvnM2A3r4NFZCRuFtAuXbqEOF0E\nxHLwxwK10knGZhiGUaCE8epJ5hsgpd3f3ZbxVuBF4AWgGtiaVEeJJ6NvcPwDqlqlqlUdOnTIQMwc\nkKsc/IZhGAVEGBv/c2xXzmVANyBU8nlVfRh42G3nZpx8/p+LyB6qukZE9gC+yETwvOCbg9/SJhuG\nUbyEsfH/MeF9HbBCVUNpPhHZVVW/EJEuOPb9/jizhXOBce7fv6Ynch7xy51vaZMNwyhiwij+T4A1\nqvodgIhUiMjeqro8xLFTXRt/LXCpqm4SkXHAFBH5BbACKFxjeS6TsRmGYRQIYRT/UzhbL8bY6pYd\n4l19O6p6uEfZehzPoMLHkrEZhtEECaP4m6vqltgHVd0iIi0ilKmwsGRshmE0McJ49awVkZNiH0Rk\nCLAuOpEMwzCMKAkz4r8YeEJE7nY/rwQ8o3kNwzCMwidMANeHQH8R2cn9/HXkUhmGYRiRkdLUIyI3\ni0ilqn6tql+LSFsR+d98CGcYhmHknjA2/p+q6qbYBzeT5s+iE8kwDMOIkjCKv5mI7BD7ICIVwA4B\n9Q3DMIwCJszi7hPATBF5xP18PqkzcxqGYRgFSpjF3VtFZD5wjFv0P6o6I1qxDMMwjKgIM+JHVV/A\nybCJiAwUkXtU9dIUhxmGYRgFSCjFLyJ9gDNw8up8DDwTpVCGYRhGdPgqfhH5IY6yPwMnUncyIKp6\nZJ5kMwzDMCIgaMT/PvAacIKqLgMQkVF5kcowDMOIjCB3zmHAGuAVEXlQRI4GJD9iGYZhGFHhq/hV\ndbqqng4cALwCXAXsKiL3icigfAloGIZh5JaUAVyq+o2qPqmqJwKdgXeA6yKXzDAMw4iEtDZbV9WN\n7iboxbGRimEYhtGAtBS/YRiGUfyE8uM3DKN4qK2tZeXKlXz33XeNLYqRJ1q2bEnnzp0pLy8PVd8U\nv2E0MVauXEnr1q3Ze++9ETFHvKaOqrJ+/XpWrlxJ165dQx1jph7DaGJ89913tG/f3pR+iSAitG/f\nPq0Znil+w2iCmNIvLdL9f5viNwzDKDFM8RuGkVPWr19P79696d27N7vvvjudOnWKf96yZUuoNs4/\n/3yWLFkSWOeee+7hiSeeyIXIAHz++ec0b96chx56KGdtFiqiqo0tQ0qqqqp07ty5jS2GYRQFixcv\n5sADDwxdf/o7q7htxhJWb6qhY2UF1wzen6F9OuVEljFjxrDTTjtx9dVX1ytXVVSVsrLCGXvedddd\nTJkyhRYtWjBz5szIzlNXV0fz5rn3q/H6v4vIPFWtSq5bOHfdMIy8M/2dVfzmmYWs2lSDAqs21fCb\nZxYy/Z1VOT/XsmXL6NatG2eddRYHHXQQa9asYeTIkVRVVXHQQQcxduzYeN2BAwdSXV1NXV0dlZWV\njB49ml69enHooYfyxRdfAPC73/2OO+64I15/9OjR9OvXj/3335833ngDgG+++YaTTz6Zbt26ccop\np1BVVUV1dbWnfBMnTuSOO+7go48+Ys2aNfHyv//97/Tt25devXoxaJCTrearr77i3HPPpWfPnvTs\n2ZPp06fHZY0xadIkLrjgAgDOPvtsfvnLX9KvXz9++9vfMnv2bA499FD69OnDgAEDWLp0KeB0CqNG\njaJ79+707NmTe++9lxdffJFTTjkl3u7zzz/P8OHDs/pfmDunYZQwt81YQk3t1nplNbVbuW3GkpyN\n+hN5//33+ctf/kJVlTMIHTduHO3ataOuro4jjzySU045hW7dutU7ZvPmzfz4xz9m3Lhx/OpXv2LC\nhAmMHj26Qduqyptvvsmzzz7L2LFjeeGFF7jrrrvYfffdmTp1KvPnz6dv376eci1fvpwNGzZw8MEH\nM3z4cKZMmcKVV17JZ599xi9/+Utee+019tprLzZs2AA4M5kOHTqwYMECVJVNmzalvPY1a9Ywe/Zs\nysrK2Lx5M6+99hrNmzfnhRde4He/+x2TJ0/mvvvuY/Xq1cyfP59mzZqxYcMGKisrueyyy1i/fj3t\n27fnkUceYcSIEene+nrYiN8wSpjVm2rSKs+WfffdN670wRll9+3bl759+7J48WLee++9BsdUVFTw\n05/+FICDDz6Y5cuXe7Y9bNiwBnVmzZrF6aefDkCvXr046KCDPI+dNGkSp512GgCnn346EydOBOA/\n//kPRx55JHvttRcA7dq1A+Cll17i0kudTQhFhLZt26a89uHDh8dNW5s2beLkk0+me/fuXH311bz7\n7rvxdi+++GKaNWsWP19ZWRlnnXUWTz75JBs2bGDevHnxmUem2IjfMEqYjpUVrPJQ8h0rKyI53447\n7hh/v3TpUu68807efPNNKisrOfvssz190Vu0aBF/36xZM+rq6jzb3mGHHVLW8WPixImsW7eOxx57\nDIDVq1fz0UcfpdVGWVkZiWumydeSeO3XX389gwcP5pJLLmHZsmUcd9xxgW2PGDGCk08+GYDTTjst\n3jFkio34DaOEuWbw/lSU11ciFeXNuGbw/pGf+8svv6R169bsvPPOrFmzhhkzZuT8HAMGDGDKlCkA\nLFy40HNG8d5771FXV8eqVatYvnw5y5cv55prrmHSpEkcdthhvPLKK6xYsQIgbuo59thjueeeewDH\nxLRx40bKyspo27YtS5cuZdu2bUybNs1Xrs2bN9Opk2NKe/TRR+Plxx57LPfffz9bt26td74999yT\nXXbZhXHjxnHeeedld1OIWPGLyCgReVdEFonIRBFpKSJHi8jbIlItIrNEZL8oZTAMw5+hfTpxy7Ae\ndKqsQIBOlRXcMqxHJPb9ZPr27Uu3bt044IADOOeccxgwYEDOz3H55ZezatUqunXrxk033US3bt1o\n06ZNvToTJ07kv/7rv+qVnXzyyUycOJHddtuN++67jyFDhtCrVy/OOussAG688UY+//xzunfvTu/e\nvXnttdcAuPXWWxk8eDCHHXYYnTt39pXruuuu45prrqFv3771ZgkXXXQRu+++Oz179qRXr17xTgvg\nzDPPpGvXrvzwhz/M+r5E5s4pIp2AWUA3Va0RkSnAP4DfAkNUdbGIXAL0U9Xzgtoyd07DCE+67pxN\nmbq6Ourq6mjZsiVLly5l0KBBLF26NBJ3yqi5+OKLOfTQQzn33HM9v0/HnTPqq28OVIhILdAKWA0o\nsLP7fRu3zDAMI+d8/fXXHH300dTV1aGq/PnPfy5Kpd+7d2/atm3L+PHjc9JeZHdAVVeJyB+BT4Aa\n4EVVfVFELgD+ISI1wJdAf6/jRWQkMBKgS5cuUYlpGEYTprKyknnz5jW2GFnjF3uQKZHZ+EWkLTAE\n6Ap0BHYUkbOBUcDPVLUz8AjwJ6/j3Z2+qlS1qkOHDlGJaRiGUXJEubh7DPCxqq5V1VrgGWAA0EtV\n57h1JgOHRSiDYRiGkUSUiv8ToL+ItBInZ+jRwHtAGxGJLUsfCyyOUAbDMAwjiSht/HNE5GngbaAO\neAd4AFgJTBWRbcBGILvYY8MwDCMtIvXjV9UbVfUAVe2uqj9X1e9VdZqq9lDVXqr6E1VNLzzOMIyC\nJhdpmQEmTJjAZ599Fv8cJlVzOjz99NOICMuWLctZm8WCRe4aRqmzYArc3h3GVDp/F0xJfUwA7du3\np7q6murqai6++GJGjRoV/5yYfiEVyYr/kUceYf/9cxdRPHHiRAYOHBjPyxMV6aaPyAem+A2jlFkw\nBZ67AjZ/Cqjz97krslb+fjz22GP069eP3r17c8kll7Bt2zbq6ur4+c9/To8ePejevTvjx49n8uTJ\nVFdXc9ppp8VnCmFSNS9dupQf/ehH9OjRg+uvv75emuREvvzyS+bMmcODDz7IpEmT6n13880306NH\nD3r16sX1118PwAcffMBRRx1Fr1696Nu3L8uXL+ell15i6NCh8eMuvvhiHn/8cQA6d+7M6NGj6dOn\nD9OmTeP+++/nkEMOoVevXgwfPpyaGic/0meffcaQIUPikbpz5szht7/9LXfffXe83euuuy6eHiJX\nmOI3jFJm5lioTUrSVlvjlOeYRYsWMW3aNN544424Ap80aRLz5s1j3bp1LFy4kEWLFnHOOefEFX6s\nA0ieKcRSNc+fP59DDz2UCRMmAE6KhquvvpqFCxeyxx57+Moybdo0jj/+eA444AB23HFH5s+fD8Bz\nzz3H888/z5tvvsn8+fP59a9/DcAZZ5zBqFGjmD9/Pm+88Qa77rpryuvdddddeeeddxg+fDjDhw/n\nrbfeYv78+ey7777x/DyXXnopxx57LAsWLGDevHkceOCBjBgxIp4sbuvWrTz11FOceeaZad/vIEpP\n8ed4WmsYRc3mlemVZ8FLL73EW2+9RVVVFb179+bVV1/lww8/ZL/99mPJkiVcccUVzJgxo0EuHS/8\nUjXPmTMnnsUySFlOnDgxnq45MQ3zSy+9xIgRI6iocLKTtmvXjo0bN7Ju3TpOPPFEAFq2bEmrVq1S\nyhhL8wywYMECDj/8cHr06MGkSZPiaZj/9a9/cdFFFwHQvHlzdt55Z/bbbz9at27NwoULef755+nX\nr1+otM/pUHyxy9kQm9bGRjixaS1Az1MbTy7DaCzadHbNPB7lOUZVGTFiBP/zP//T4LsFCxbw/PPP\nc8899zB16lQeeOCBwLbCpmr2Yu3atbz66qssXrwYEaGuro7y8nJuueWW8BeDo6i3bdsW/xyUhvmc\nc87h+eefp3v37jz00EPMnj07/p3j7V6fX/ziFzz66KMsX7483jHkktIa8edxWmsYRcHRN0B5Uu79\n8gqnPMccc8wxTJkyhXXr1gGO988nn3zC2rVrUVWGDx/O2LFjefvttwFo3bo1X331VVrn6NevXzwd\ncrLtPsZTTz3FiBEjWLFiBcuXL2flypV07NiR//znPxx77LFMmDAhboPfsGEDbdu2pUOHDjz33HOA\no+C//fZb9tprL9599122bNnCxo0befnll33l+uabb9h9992pra3lySefjJcfeeSR3H///YBj1vny\nyy8BJzvoc889R3V1Ncccc0xa9yAMpaX48zitNYyioOepcOJ4aLMnIM7fE8dHMgPu0aMHN954I8cc\ncww9e/Zk0KBBfP7553z66accccQR9O7dm/PPP5+bb74ZcNw3L7jggrTcQMePH8+tt95Kz549+fjj\njz3NRkFpmE844QSOO+64uDnq9ttvB+CJJ57g//2//0fPnj0ZOHAga9eupWvXrgwdOpSDDjqI008/\n3XdbR4CxY8dyyCGHMGDAgHpbS959993MmDGDHj16UFVVxfvvvw845qQjjjiCM844I5IN6SNLy5xL\ncpaW+fbuPtPaPWHUouzbN4wCoJTTMn/zzTe0atUKEeHxxx9n2rRpTJ06tbHFSptt27bRu3dvpk+f\nzj777BPqmHTSMpfWiD+P01rDMPLPW2+9RZ8+fejZsycPPvggt912W2OLlDYLFy5k33335bjjjgut\n9NOltBZ3Y9PXmWMd806bzo7St4Vdw2gS/OQnP8l5CuN806NHDz7++ONIz1Faih8cJW+K3mjiqKqn\nt4jRNEnXZF9aph7DKAFatmzJ+vXr01YGRnGiqqxfv56WLVuGPqb0RvyG0cTp3LkzK1euZO3atY0t\nipEnWrZsGbi5ezKm+A2jiVFeXk7Xrl0bWwyjgDFTj2EYRolhit8wDKPEMMVvGIZRYhRF5K6IrAVW\n5KCpXYB1OWinqWP3KTx2r8Jh9yk8ubxXe6lqh+TColD8uUJE5nqFLxv1sfsUHrtX4bD7FJ583Csz\n9RiGYZQYpvgNwzBKjFJT/MG7Oxgx7D6Fx+5VOOw+hSfye1VSNn7DMAyj9Eb8hmEYJY8pfsMwjBKj\nZBS/iBwnIktEZJmIjG5sefKBiOwpIq+IyHsi8q6IXOmWtxORf4rIUvdvW7dcRGS8e48WiEjfhLbO\ndesvFZFzE8oPFpGF7jHjpYhzAYtIMxF5R0T+5n7uKiJz3GubLCIt3PId3M/L3O/3TmjjN275EhEZ\nnFDeJJ4/EakUkadF5H0RWSwih9rz5I2IjHJ/d4tEZKKItCyYZ0pVm/wLaAZ8COwDtADmA90aW648\nXPceQF/3fWvgA6Ab8AdgtFs+GrjVff8z4HlAgP7AHLe8HfCR+7et+76t+92bbl1xj/1pY193Fvfr\nV8CTwN/cz1OA09339wO/dN9fAtzvvj8dmOy+7+Y+WzsAXd1nrllTev6Ax4AL3PctgEp7njzvUyfg\nY6Ai4Vk6r1CeqVIZ8fcDlqnqR6q6BZgEDGlkmSJHVdeo6tvu+6+AxTgP5BCcHzDu36Hu+yHAX9Rh\nNlApInsAg4F/quoGVd0I/BM4zv1uZ1Wdrc5T+peEtooKEekMHA885H4W4CjgabdK8n2K3b+ngaPd\n+kOASar6vap+DCzDefaaxPMnIm2AI4CHAVR1i6puwp4nP5oDFSLSHGgFrKFAnqlSUfydgMRd1le6\nZSWDO3XsA8wBdlPVNe5XnwG7ue/97lNQ+UqP8mLkDuBaYJv7uT2wSVXr3M+J1xa/H+73m9366d6/\nYqMrsBZ4xDWJPSQiO2LPUwNUdRXwR+ATHIW/GZhHgTxTpaL4SxoR2QmYClylql8mfueOrErap1dE\nTgC+UNV5jS1LgdMc6Avcp6p9gG9wTDtx7HlycNc5huB0lh2BHYHjGlWoBEpF8a8C9kz43Nkta/KI\nSDmO0n9CVZ9xiz93p9W4f79wy/3uU1B5Z4/yYmMAcJKILMeZMh8F3IljmohtVpR4bfH74X7fBlhP\n+vev2FgJrFTVOe7np3E6AnueGnIM8LGqrlXVWuAZnOesIJ6pUlH8bwE/cFfUW+AsnjzbyDJFjmsj\nfBhYrKp/SvjqWSDmSXEu8NeE8nNcb4z+wGZ3Cj8DGCQibd2RzCBghvvdlyLS3z3XOQltFQ2q+htV\n7ayqe+M8Gy+r6lnAK8ApbrXk+xS7f6e49dUtP9310OgK/ABnsbJJPH+q+hnwqYjs7xYdDbyHPU9e\nfAL0F5FW7rXE7lVhPFONvfqdrxeOh8EHOCvh1ze2PHm65oE40+4FQLX7+hmO7XAmsBR4CWjn1hfg\nHvceLQSqEtoagbOwtAw4P6G8CljkHnM3bjR4sb6An7Ddq2cf90e2DHgK2MEtb+l+XuZ+v0/C8de7\n92IJCR4pTeX5A3oDc91najqOV449T9736ibgffd6/g/HM6cgnilL2WAYhlFilIqpxzAMw3AxxW8Y\nhlFimOI3DMMoMUzxG4ZhlBim+A3DMEoMU/xGk0BEtopItZsNcb6I/FpEAp9vEdlbRM6MQJarRKSV\nz3cnuOkO5ouTNfUit/xiETkn17IYhhfmzmk0CUTka1XdyX2/K06WzddV9caAY34CXK2qJ+RYluU4\nPuvrksrLgRVAP1VdKSI7AHur6pJcnt8wUmEjfqPJoapfACOBy9yo0b1F5DURedt9HeZWHQcc7s4U\nRvnVE5E9ROTfbr1FInK4Wz5IRP7j1n1KRHYSkStwcrO8IiKvJInWGiffzXpXzu9jSl9ExojI1SLS\n0T1P7LVVRPYSkQ4iMlVE3nJfAyK/kUaTxUb8RpMgccSfULYJ2B/4Ctimqt+JyA+AiapalTzid80z\nXvV+DbRU1d+LSDOcFLs74ORf+amqfiMi1+FEYY71G/G753gIOAkn0vVv7jm2icgY4GtV/WNC3UuB\nH6vqqSLyJHCvqs4SkS44KQ4OzNkNNEqK5qmrGEbRUw7cLSK9ga3AD9Os9xYwwTXVTFfVahH5Mc4m\nGa87qVhoAfwnlSCqeoGI9MBJ4nU1cCzOBh31cEf0F+Kk3cCt3022b0i1s4jspKpfpzqnYSRjit9o\nkojIPjjK+wvgRuBzoBeOefM7n8NGedVT1X+LyBE4G7U8KiJ/AjbibCZyRrqyqepCYKGI/B/OLk3n\nJcm+B05yvZMSFHsZ0F9V/WQ3jNCYjd9ocohIB5xt7e5Wx5bZBlijqtuAn+NsWweOCah1wqGe9URk\nL+BzVX0QZ4euvsBsYICI7OfW2VFEfujTbkyunVzzUozeOIu9iXXKcZJ1XaeqHyR89SJweUK93uHu\nhmE0xGz8RpNARLbiZIAsB+pwsiH+ybWf/wBnTwIFXgAuVdWdXCU7Aye75KM4NneveucC1wC1wNfA\nOar6sYgcBdyKY+8H+J2qPisilwOXAatV9cgEGVsDk4F9gRqcjUyuVNW5MRs/jllpBk5Wxxg/A7bg\nZLo8EGem/m9VvTgnN88oOUzxG4ZhlBhm6jEMwygxTPEbhmGUGKb4DcMwSgxT/IZhGCWGKX7DMIwS\nwxS/YRhGiWGK3zAMo8T4/85enmEap/LnAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fddhyGantaK-",
        "colab_type": "text"
      },
      "source": [
        "2.Subset of features included in the model:\n",
        "\n",
        "In this part I divided all features to three categories. Finanical features include fico (credit rating of the borrower), dti (ratio of the borrower’s total monthly debt payments), and emp_length (length of time that the borrower has been employed at their current job). Location features include state (U.S. state that the borrower resides in. Loan features include amount (amount requested for their first loan). Then I categorized these features into seven feature sets, including finanical features only, location features only, loan only, finanical and location features, finanical and loan features, location and loan features, and all features.\n",
        "\n",
        "I produced the results in a table. I considered putting them into a plot - but the x axis couldn't fit that many feature names. The result shows that the best feature sets are \"financial only\" feature and \"financial and location\" feature with 96.5% accuracy. The second best feature sets are \"financial and loan\" feature and \"all features\" feature with 95.9% accuracy. The worst feature set is \"location only\" dataset with 88.5% accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fNqLfeFNwxMW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "outputId": "4f85cc95-a0b6-4b0b-eb5e-dfc92fb51612"
      },
      "source": [
        "# Create lists of possible features\n",
        "financial_features = [\"fico\", \"dti\", \"emp_length\"]\n",
        "location_features = [\"state\"]\n",
        "loan_features = [\"amount\"]\n",
        "\n",
        "feature_sets = {\n",
        "    \"financial only\": financial_features,\n",
        "    \"location only\": location_features,\n",
        "    \"loan only\": loan_features,\n",
        "    \"financial and location\": financial_features + location_features,\n",
        "    \"financial and loan\": financial_features + loan_features,\n",
        "    \"location and loan\": location_features + loan_features,\n",
        "    \"all features\": financial_features + location_features + loan_features\n",
        "}\n",
        "\n",
        "best = 0\n",
        "best_name = None\n",
        "feature_set_names = []\n",
        "feature_set_accuracies = []\n",
        "features_accuracies_table = pd.DataFrame()\n",
        "\n",
        "# Calculate accuracies of the model with different features\n",
        "for set_name, feature_set in feature_sets.items():\n",
        "\n",
        "    # Prepare the small dataset\n",
        "    X = lendingclub.loc[:, feature_set]\n",
        "    X = pd.get_dummies(X)\n",
        "    y = lendingclub[\"outcome\"]\n",
        "\n",
        "    # Use scikit-learn to create train/test split and train the decision tree\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "    model = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "    train_pred = model.predict(X_train)\n",
        "    test_pred = model.predict(X_test)\n",
        "\n",
        "    # Calcualte accuracy\n",
        "    accuracy = 100*accuracy_score(test_pred, y_test)\n",
        "    if accuracy >= best:\n",
        "        best = accuracy\n",
        "        best_name = set_name\n",
        "    \n",
        "    feature_set_names.append(set_name)\n",
        "    feature_set_accuracies.append(accuracy)\n",
        "\n",
        "# Produce results in a table\n",
        "features_accuracies_table['feature_set_names'] = feature_set_names\n",
        "features_accuracies_table['feature_set_accuracies'] = feature_set_accuracies\n",
        "features_accuracies_table = features_accuracies_table.sort_values(by=['feature_set_accuracies'], ascending = False).reset_index(drop = True)\n",
        "features_accuracies_table"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>feature_set_names</th>\n",
              "      <th>feature_set_accuracies</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>financial only</td>\n",
              "      <td>96.529191</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>financial and location</td>\n",
              "      <td>96.523026</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>financial and loan</td>\n",
              "      <td>95.931200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>all features</td>\n",
              "      <td>95.912706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>loan only</td>\n",
              "      <td>89.248505</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>location and loan</td>\n",
              "      <td>89.211516</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>location only</td>\n",
              "      <td>88.459404</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        feature_set_names  feature_set_accuracies\n",
              "0          financial only               96.529191\n",
              "1  financial and location               96.523026\n",
              "2      financial and loan               95.931200\n",
              "3            all features               95.912706\n",
              "4               loan only               89.248505\n",
              "5       location and loan               89.211516\n",
              "6           location only               88.459404"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bwLTsCmJ1LYA",
        "colab_type": "text"
      },
      "source": [
        "3.Decision tree vs. logistic regression classification.\n",
        "\n",
        "In this part, I trained logistic regression classfication and calculated its accuracies based on different threshold. Due to the imbalance of dataset (there are a lot more \"reject\" than \"accept\" in the outcome), the threshold of the logistic regression is far above 0.5. I found that the higher the threshold, the more accurate the model is. According to my calculation, the model achieves the highest accuracy of 88.2 with a 0.098 threshold. It shows that the accuracy of the logistic regression model (88.2) is less than the accuracy of the original decision tree model (95.9)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-p6BFSES1KyH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "9a2e5e8e-b8d7-4bfa-8b5b-a085cadadae3"
      },
      "source": [
        "# List the features from the data\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Prepare data\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "\n",
        "# Train logistic regression model using scikit-learn\n",
        "model = LogisticRegression(penalty=\"elasticnet\", C=1, l1_ratio=0.5, solver=\"saga\", max_iter=5000, random_state=123)\n",
        "model.fit(X_train, y_train)\n",
        "\n",
        "# Get prediction probabilities on the test set\n",
        "predicts = model.predict_proba(X_test)\n",
        "all_probabilities = [x[1] for x in predicts]\n",
        "all_actuals = list(y_test)\n",
        "\n",
        "# Compute accuracies based on several different thresholds\n",
        "for threshold in np.arange(0.9, 0.99, 0.01):\n",
        "    all_predictions = []\n",
        "    for x in all_probabilities:\n",
        "      if x > threshold:\n",
        "        all_predictions.append('accept')\n",
        "      else:\n",
        "        all_predictions.append('reject')\n",
        "    accuracy = 100*accuracy_score(all_actuals, all_predictions)\n",
        "    print(f\"Accuracy: {accuracy:.1f} at threshold {threshold}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 67.9 at threshold 0.9\n",
            "Accuracy: 71.1 at threshold 0.91\n",
            "Accuracy: 73.8 at threshold 0.92\n",
            "Accuracy: 77.4 at threshold 0.93\n",
            "Accuracy: 80.2 at threshold 0.9400000000000001\n",
            "Accuracy: 83.8 at threshold 0.9500000000000001\n",
            "Accuracy: 88.0 at threshold 0.9600000000000001\n",
            "Accuracy: 88.2 at threshold 0.9700000000000001\n",
            "Accuracy: 88.2 at threshold 0.9800000000000001\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oShxL5GrEX6y",
        "colab_type": "text"
      },
      "source": [
        "4.Hyperparameter tuning for the best-performing model from option 3.\n",
        "\n",
        "The best-performing model from option 3 is decision tree. Therefore I'm tuning the hyperparameter for the decision tree classifier, including criterion, min_impurity_decrease, and min_samples_split. The number of folds for cross validation in the hyperparameter search is 5. The result shows that the best model has a 95.9% accuracy on the test set, which is the same as the original model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnqZ2l7VEYLo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "a536a2a6-d506-4815-ea00-4415848169d7"
      },
      "source": [
        "# Select features\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Prepare the data\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20, random_state=123)\n",
        "\n",
        "# Decide on the hyperparameters we would like to compare\n",
        "hyperparameters = {\n",
        "    \"criterion\":[\"gini\", \"entropy\"],\n",
        "    \"min_impurity_decrease\":[0, 0.0001, 0.001],\n",
        "    \"min_samples_split\":range(2, 20, 2)\n",
        "}\n",
        "\n",
        "# Decide on number of folds for cross validation in the hyerparameter search\n",
        "kfolds = 5\n",
        "\n",
        "# Initialize a search using cross validation in sklearn\n",
        "search = GridSearchCV(estimator=DecisionTreeClassifier(random_state=123),\n",
        "                            param_grid=hyperparameters, cv=kfolds, scoring=\"accuracy\")\n",
        "\n",
        "# Train a classifier with each combination of hyperparameters and take the best one and print out the results\n",
        "classifier = search.fit(X_train, y_train)\n",
        "accuracy = classifier.best_score_\n",
        "best_fit = classifier.best_estimator_\n",
        "print(f\"Best fit when training was {best_fit}\\nWith {100*accuracy:.1f}% accuracy.\")\n",
        "\n",
        "# Evaluate our best model's performance on the test set and print the results\n",
        "y_pred = classifier.predict(X_test)\n",
        "accuracy = 100*accuracy_score(y_test, y_pred)\n",
        "print(f\"Accuracy on held-out test set: {accuracy:.1f}%\")\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Best fit when training was DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='entropy',\n",
            "                       max_depth=None, max_features=None, max_leaf_nodes=None,\n",
            "                       min_impurity_decrease=0.0001, min_impurity_split=None,\n",
            "                       min_samples_leaf=1, min_samples_split=8,\n",
            "                       min_weight_fraction_leaf=0.0, presort='deprecated',\n",
            "                       random_state=123, splitter='best')\n",
            "With 95.4% accuracy.\n",
            "Accuracy on held-out test set: 95.9%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6QiQ4Th3Di_G",
        "colab_type": "text"
      },
      "source": [
        "5.Number of folds or stratification strategy for cross-validation.\n",
        "\n",
        "In this part, I calculated accuracies based on different number of folds for cross-validation and displayed the results in a table and a plot. The results show that the model accuracy is the highest with 0 fold and the lowest with 5 fold."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "94eq_RJsI0gc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "outputId": "627b9f7c-9407-4556-f0c6-6b2ce417d9bc"
      },
      "source": [
        "# List the features from the data\n",
        "features = [\"amount\", \"fico\", \"dti\", \"state\", \"emp_length\"]\n",
        "\n",
        "# Set up dataset and cross validation split\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "num_folds = 10\n",
        "\n",
        "# Set up model\n",
        "classifier = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "\n",
        "accuracies = pd.DataFrame()\n",
        "fold = []\n",
        "train_accuracies = []\n",
        "test_accuracies = []\n",
        "\n",
        "# Calculate accuracies of the model based on different cross validation split\n",
        "for i in range(num_folds):\n",
        "    X_train = X.loc[X.index % num_folds != i]\n",
        "    X_test = X.loc[X.index % num_folds == i]\n",
        "    y_train = y.loc[y.index % num_folds != i]\n",
        "    y_test = y.loc[y.index % num_folds == i]\n",
        "    \n",
        "    model = classifier.fit(X_train, y_train)\n",
        "    train_pred = model.predict(X_train)\n",
        "    test_pred = model.predict(X_test)\n",
        "    train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "    test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "\n",
        "    fold.append(i)\n",
        "    train_accuracies.append(train_accuracy)\n",
        "    test_accuracies.append(test_accuracy)\n",
        "\n",
        "accuracies['fold'] = fold\n",
        "accuracies['train accuracies'] = train_accuracies\n",
        "accuracies['test accuracies'] = test_accuracies\n",
        "\n",
        "accuracies = accuracies.sort_values(by = 'test accuracies', ascending = False).reset_index(drop = True)\n",
        "accuracies"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fold</th>\n",
              "      <th>train accuracies</th>\n",
              "      <th>test accuracies</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>96.303705</td>\n",
              "      <td>96.498582</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>96.188623</td>\n",
              "      <td>96.202688</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>96.187306</td>\n",
              "      <td>96.128237</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>96.168126</td>\n",
              "      <td>96.115906</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>96.214706</td>\n",
              "      <td>96.029593</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>95.874947</td>\n",
              "      <td>95.869297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1</td>\n",
              "      <td>96.173553</td>\n",
              "      <td>95.795833</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>3</td>\n",
              "      <td>96.092776</td>\n",
              "      <td>95.733662</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>4</td>\n",
              "      <td>95.973587</td>\n",
              "      <td>95.610358</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>5</td>\n",
              "      <td>95.970847</td>\n",
              "      <td>95.548705</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   fold  train accuracies  test accuracies\n",
              "0     0         96.303705        96.498582\n",
              "1     2         96.188623        96.202688\n",
              "2     9         96.187306        96.128237\n",
              "3     8         96.168126        96.115906\n",
              "4     7         96.214706        96.029593\n",
              "5     6         95.874947        95.869297\n",
              "6     1         96.173553        95.795833\n",
              "7     3         96.092776        95.733662\n",
              "8     4         95.973587        95.610358\n",
              "9     5         95.970847        95.548705"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o-B3yyXQRPRc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "7e3556ce-5cbf-4604-a4b7-ec1646ff59f8"
      },
      "source": [
        "# Plot the relation between test accuracy and cross validation split\n",
        "plt.title('Test Set Accuracy vs. cross-validation')\n",
        "plt.plot(range(num_folds), test_accuracies)\n",
        "plt.xlabel(\"fold\")\n",
        "plt.ylabel(\"Test set accuracy\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5fX48c/JRggQkrAnhATZ1wAJ\nS0WtAi5ViNVqq9bWpVW/1r27be36a2vtplatdasb0LqgCK51wx0M+75DyAIECCQQsp/fH/cGxzhJ\nJpDJncyc9+uVF5k7c+89cyfcM/c593keUVWMMcaYxqK8DsAYY0xosgRhjDHGL0sQxhhj/LIEYYwx\nxi9LEMYYY/yyBGGMMcYvSxDGGM+IyHARqfV5/LaIfCOQ1x7Hvn4jIvcd7/qRyBJEiBORwz4/9SJy\n1OfxN09gu5+IyOUtvOZ6Ednk7mu3iCwUkc4BbPscEdkSYBx3ioiKSFagsZvwparTVPW/J7odf3+D\nqvorVb3xRLcdSSxBhDhV7drwA+QDs3yWzQ7WfkXkbOAXwNfcfY8G5rXxPqKAy4EDwLfbctuB7Nvd\nf1gTkRivYzAdV9j/Bwl3IhItIneIyDYR2Scis0UkyX2ui4j8R0QOiMhBEVksIski8ldgIvCIe3Xw\nVz+bngi8r6qrAVR1n6o+pqpH3W13FpG7RWSXe3XxDxHpJCI9gBeAk3yudHo0Ef4MoDvwfeCbjU9m\nIvI9EdkgIuUislpExrjLM0Vkvvt+9zXE716NPOKzfuPmi09E5LcishioAFJF5DqffWwRkasbxXCx\niKxyn98sItNF5Fsi8mGj1/1MRL7wzVdErhCRDxotu11EnnF/P99n/7tE5OYmjlXj7XYRkXvddQ6J\nyCIRiWl4zyJyjYjsAl5xX/81EVnn/h28KSJDfLZ1h4gUi0iZiKwXkVPd5VNFZLm7fLeI/LGJWFp6\njxeIyEp3O/ki8rNm3texK1v3/dwjIvvdq4EzG73W72fX1N+gn7+P5o7JbhG5TUTWuMd3tojEBfLZ\nhBVVtZ8O8gPsAGY0WvYT4H0gFYgHHgf+7T53C/Ac0BmIwTnpd3Gf+wS4vJl9zcA5if4S+BIQ1+j5\nf7rbTsI5yb8O/Mp97hxgSwDvZzbwpBtfGXCez3PfAnYC4wEBhgH9gVhgPXAnkOCue7K7zp3AIz7b\nGA7U+jz+BNjmbivWPSa5wEB3HzOAo8Ao9/WnAaXAGThfpgYAQ4EubrwDfba93jd+n+WJ7nEc4LNs\nNfBV9/f9wCT39x7A+AD/Fh4F3gD6AtHAqe6/wwEFHvE5PmOAcuB0IA64A1jnvv8s95j0cY/BSQ3v\nC1gOXOz+3g2Y3EQsLb3H6cAo9xhOwLliPKeZz+hy9/db3e2kAr2ADxq9trnP7gt/g75/H80dE/f5\n3cCH7nHpBWwBrvT6HNDeP54HYD+t+LD8J4jtwFSfxwPd/6wCfA9YBIz2s61mE4T7mlzgZfdkWAb8\nyf1PHgNUA2k+rz0DWO/+3mKC8DmpNJwongD+6/P8IuA6P+udARQCUX6eCyRB/KyFuF5r2K8b0x+b\neN2/gTvc33OAvQ0nFz+vfQ74sfv7GJykE+c+3gtcBXRrxd9BLFADDPPzXEOCSPVZ9nvgSZ/H0UAJ\nMAXnxF3sHteYRttaAvwc6BFATE2+Rz+vfbDhuDbxGTUkiI98T8ru32NtMzH4fnYtJYgmj4n7eDdw\nkc/z9wJ3B/oZhcuPNTF1YCIiQDrwinuZfBDnW18UzrfRR3FOtM+JSIGI/EFEogPdvqq+pKrn4Vwl\nXAxcj/PNPhXnJLXWZ78vAr1bEf7FON/g3nQfzwZyxW0ec9/XVj/rpQPbVbW+Ffvytcv3gYjkisgS\ncZvhgGlAzxZiACd5NNwkcDkwV1WbusNmDnCp+/tlwHOqWu0+Ph/4GpAvzh08EwN4D/1wknRTsdWr\napHP41ScqzEAVLUOJ8mmqepa4Kc4J8y9blNKH/elVwBjgU3iNE+eDSAij/s03Xy/pffoNlUtEpES\nETkEXMlnx7g5qXz+89rp+2QLn10g2/Z7THxes9vn9wqga4DbDhuWIDowdb7aFALTVDXJ5ydenZpB\nlar+UlWH4zSXXAxc0rB6K/ZTr6qvA+/hFKuLgVpgkM8+u6tqQ60hkG1fgZN4CkRkN/AUThNZwy2O\nu4BBftbbBWSK/wLzEZxmlQZ9/b2dhl9EpAvwLPA7oLeqJgFv41x9NRcDOIk3XkSm4JwYn2rideDU\nAQaKyAic4z/nWDCqH6vqTJymjDd8n2vGsePfxPONj38RkNHwwP2SkIbzt4OqPqGqJ+M0L8UD/89d\nvl5Vv4GT+O8F5olInKpeqZ/dKPG3lt4j8AzwXyBdVbvjNIMKLSvGSdINBvi8h5Y+u5b+Bps9JsZh\nCaLjexC4U0TSAUSkt4jMcn+fISIj3ZNpGc5JpeGb9x6cE4JfInKROAXaJHGcDEwFPlHVGuAx4B4R\n6ek+ny4iZ/psu7eI+P3GJSInAacAZwHj3J8s4G4+u5vpEeCnIpLlbn+oiPTHaYcuB34nIgniFMtP\ndtdZAZwhImkikoxTn2lOZ5wrob1AvYjk4rRJN3gEuE5EThPnrqd0ERkKx5LzU8BDwH5VzWtqJ6pa\niVM0vdfd3yL3OHQRkUtEJBGnyaiczz6fJrnH/0mc499HnBsVTmnm6vC/wAXu+4jFuWLYD+S5fx9f\nFpFOOG34RxtiEJFvi0gP99v1IZyTrt8TbzPvUXC+ee9X1Ur3s7q4pffoega4TUT6iUhP4Mc+z7X0\n2TX7N9jcMQkwtsjgdRuX/QT+g/8aRDTOiXAzzglmC58Vi69wlx/BuVz+K27bPfBl97WlwF1+9jUd\neAfnP005sAG41ef5zsBdbkxlwFrgevc5AZ521z0IpDTa9q+BD/3sMxMniQ1xH9/kxn8YWIVbS8Gp\nsyzEKXaWAH92l0cBD+OczDYC19FE+7bPsu/jnGRKcZLePOAXPs9/HVjjHoNNOFdrDc8NwTlh3h7A\nZ3em+9q/+izrgnPVUOoew8W4hWCcYvhhnG/H/rbXBbgf55vwQfeziqFRm36j97Hefe3buPULIBvn\npFjuHs8XG/aJc4Le5z63Gji3te/RXX4pztVYubv9B/msFtBcDSIWuM+Na6v79+D72iY/O39/g3yx\nRuX3mLjP7QZO8Xn8uXUj5UfcN2+MaSUR6YbzTXW4quZ7HY8xbc2amIw5fjcB71pyMOHKelkacxzc\nwnoFzq2XxoQla2IyxhjjlzUxGWOM8Susmph69uypmZmZXodhjDEdxtKlS/epai9/z4VVgsjMzCQv\nz25jNsaYQInIzqaesyYmY4wxflmCMMYY45clCGOMMX5ZgjDGGOOXJQhjjDF+WYIwxhjjlyUIY4wx\nfkV8gqiqrePBRVt5f3OJ16EYY0xIifgEERcdxUPvbWP+iqKWX2yMMREk4hOEiJCdkUzejgNeh2KM\nMSEl4hMEQE5GMjv2V1BSXuV1KMYYEzIsQQA5mckALN1Z6nEkxhgTOixBAKPTuhMXE8XSndbMZIwx\nDSxBAJ1iohmb1p08u4IwxphjLEG4cjJTWFN4iMqaOq9DMcaYkGAJwpWTkUxNnbJy10GvQzHGmJBg\nCcKVneEUqq2ZyRhjHJYgXMld4hjUq4vdyWSMMS5LED5yMlJYurOU+nr1OhRjjPGcJQgfOZnJHDpa\nw5aSw16HYowxnrME4SMnMwWAvB3WzGSMMZYgfGT2SKBHlzjyrMOcMcZYgvDVMHCfFaqNMcYSxBfk\nZCazc38Fe8srvQ7FGGM8ZQmikYY6xFKrQxhjIpwliEZGp3anU0yUdZgzxkS8oCYIEblFRNaIyFoR\nudVn+U0issFdflcz60eLyHIRWRjMOH3FxUSR1T/JEoQxJuLFBGvDIjIauAaYBFQDr7kn+nTgfCBL\nVatEpHczm7kFWA8kBitOf7Izk3n4vW0cra6jc1x0e+7aGGNCRjCvIEYAi1W1QlVrgUXAhcD1wJ2q\nWgWgqnv9rSwi/YHzgEeCGKNfORnJ1NYrKwts4D5jTOQKZoJYA5wqIj1EJAE4F+fqYai7fLGILBKR\niU2sfzfwY6C+uZ2IyLUikicieSUlJW0S+LGB+2yeamNMBAtaglDV9cCfgDeA14AVQB1Os1YKMAX4\nEfCMiIjvuiIyE9irqksD2M9Dqpqjqjm9evVqk9iTEuIY0rur1SGMMREtqEVqVX1UVbNV9TSgFNgE\nFADz1LEE5wqhZ6NVpwK5IrID+A8wTUSeDmasjeVkJrPMBu4zxkSwYN/F1Nv9dwBO/WEO8CJwhrt8\nKBAH7PNdT1VvV9X+qpoJXAK8raqXBzPWxrIzUiirrGXzXhu4zxgTmYLdD+J5EVkHLABuUNWDwGPA\nSSKyBufq4ApVVRFJFZFXghxPwHKOTSBkdQhjTGQK2m2uAKp6qp9l1cAXrgZUtQinkN14+bvAu0EI\nr1kZPRLo2TWOvB2lfHNyRnvv3hhjPGc9qZsgIuRkpNgVhDEmYlmCaEZOZjK7Dhxlb5kN3GeMiTyW\nIJpxrD+E3e5qjIlAliCaMaph4D4b2dUYE4EsQTQjLiaKrPQkq0MYYyKSJYgWTMxMZm1RGRXVtV6H\nYowx7coSRAtyMlKoq1dW7LKB+4wxkcUSRAsmDHAK1TbDnDEm0liCaEH3hFiG9rGB+4wxkccSRACy\nM1JYtrOUOhu4zxgTQSxBBGBiZjLlVbVs2lPudSjGGNNuLEEEICcjBbAOc8aYyGIJIgDpKZ3p1a0T\nS22GOWNMBLEEEQBn4L5ku4IwxkQUSxABys5IpqD0KLsP2cB9xpjIYAkiQBMzG+oQ1sxkjIkMliAC\nNDI1kc6x0TZwnzEmYliCCFBsdBRZ6d1ZanUIY0yEsATRCjkZKawrLuNIlQ3cZ4wJf5YgWiE7M9kG\n7jPGRAxLEK0wYUAyIlgdwhgTESxBtEL3zrEM69PN7mQyxkQESxCtlJ2RzPL8gzZwnzEm7FmCaKWc\nzGQOV9WycbcN3GeMCW+WIFrps4H7rJnJGBPeLEG0Uv/kzvRJ7GSFamNM2LME0UrOwH0p1mHOGBP2\nLEEch+yMZAoPHqX40FGvQzHGmKCxBHEccjKTAesPYYwJb5YgjsOIfg0D91mh2hgTvlpMECJyvYh0\nb49gOorY6CjGD0iyCYSMMWEtkCuIDGCZiMwRkRnBDqijyMlIZn1xGYdt4D5jTJhqMUGo6k+BIcBs\n4P9EZLOI/FZEMltaV0RuEZE1IrJWRG71WX6TiGxwl9/lZ710EXlHRNa5r7mlVe+qHWRnplCvsCLf\nBu4zxoSngGoQqloP7HB/6oF+wHwR+WNT64jIaOAaYBKQBcwUkcEicgZwPpClqqOAv/hZvRb4gaqO\nBKYAN4jIyEDfVHsYPyDJGbjPOswZY8JUIDWIG0RkCXAPsBQYq6rXAOOBbzSz6ghgsapWqGotsAi4\nELgeuFNVqwBUdW/jFVW1WFWXub+XA+uBtFa9syBLjHcH7guzO5lWFxzi7L+/x54ym3vbmEgXyBVE\nKnCpqs5Q1bk+J/Z6ILeZ9dYAp4pIDxFJAM4F0oGh7vLFIrJIRCY2t3O3KWs8sLiJ568VkTwRySsp\nKQng7bSdnMxklueXUltX3677DaaH39/Gxj3lvLi80OtQjDEeCyRBvAjsaXggIt1EJAdAVdc0tZKq\nrgf+BLwBvAasAOqAGCAFp+noR8AzIiL+tiEiXYHngVtVtayJ/TykqjmqmtOrV68A3k7bmZiZwpHq\nOjaEycB9B45U89qa3QC8tLLI42iMMV4LJEE8BFT4PD4C/CuQjavqo6qaraqnAaXAJqAAmKeOJTg1\njZ6N1xWRWJzkMFtV5wWyv/aWneF0mAuXYTeeX1pAdV09l0xMZ21RGVtLDnsdkjHGQ4EkiCi3OQk4\n1rQUG8jGRaS3++8AnPrDHJwrkjPc5UOBOGBfo/UEeBRYr6p/C2RfXkhL6kzfxPiw6A+hqsxdkk9O\nRjK3nTkUEVhgVxHGRLRAEsR2t7NctIhEicgNOHczBeJ5EVkHLABuUNWDwGPASSKyBvgPcIWqqoik\nisgr7npTgW8B00RkhftzbqveWTsQEbIzk8OiR/Xi7QfYtu8Il04aQJ/EeCYPTOGllUWo2sRIxkSq\nmABecx1wP/A7QIF3cG5fbZGqnupnWTVwuZ/lRTiFbFT1A8BvXSLU5GQk8/KqYgoPHiUtqbPX4Ry3\nOYvzSYyP4byx/QCYlZXKz19Yw7riMkalWkd6YyJRIB3l9qjqRaraU1V7qerXVXVPS+tFiomZ7gRC\nHfgqoqE4feGE/sTHRgNw7uh+xESJFauNiWCB9IPoJCLXici9IvJQw097BNcRDO/bjYS46A5dqJ63\nzClOXzppwLFlyV3iOHVITxauLKbe5t82JiIFUoN4EsgEZuL0RRgEWC8qV0zDwH0dtMOcqjJnST7Z\nGckM69vtc8/Nykql8OBRluV3zPdmjDkxgSSIoap6O3BYVR8FzsEZPsO4sjNS2LC7jPLKGq9DabXF\n2w+wreTI564eGpw1qi+dYqLsbiZjIlQgCaLhrHdQREYA3YDewQup48nJSKZeYXkHHLhv7hKnOD3T\nLU776tophukjevPy6uKw6i1ujAlMIAniURFJBn4FvI7T2c3fAHsRa/yAJKKEDtcfovRINa+u/nxx\nurHcrFT2Ha7m42372zk6Y4zXmr3NVUSigX2qWopze+sX2yEM3eJjGd43kaUdbGTX593i9CWT0pt8\nzenDetO1UwwLVhZx6pD2HcrEGOOtZq8gVLUO+Fk7xdKhOQP3HewwTTENxekJA5IY3jexydfFx0Zz\n1qg+vLpmN1W1de0YoTHGa4E0Mb0hIreKSD8RSWz4CXpkHUx2RjIV1XWsL+4YA/ctcYvTl03OaPG1\nuVmplFfWsmhj+46Wa4z5vJq6ekrKq9i8p5zF2/bz2prdzF2Sz+zFO4Oyv0B6Ujf0ev6BzzLFmps+\nJ6ehw9zOA4zpH/o9j+cuyadbfAznjflicbqxqYN7kpwQy0srizhrVN92iM6Y8FdVW8fBihoOHKmm\ntKL62O8HK6o5cKTG+beimtKKGkrd15RX+p/iODkhlm8G8GWvtVpMEKradAO1OSYtqTOp3Z2B+66a\nOtDrcJpVeqSaV9bs5tKJ6XSO81+c9hUbHcW5Y/oxb1khFdW1JMQF8r3CmMhxtLqO0opq9wRfw4GK\nhhO98/hzz7lJ4Eh10022XeKiSUqII6VLHEkJsWT2SCA5Ic756RLrPJfgPJfSxVkeDC3+TxeRy/wt\nV9U5bR9Ox5admcKn2w+gqjQxxUVIeH5ZAdW19Vw6OfCLwNysVGYvzud/6/Zw/riQmtzPmHZ36GgN\n/1q0lfkrith/pIrKmqZrj93iY9wTfRw9u8YxpHdX9+Qf+7kk0HCiT0qIpVNMy1/c2kMgXwV9B9yL\nB6bhTD1qCaKRnIxkFqwsovDgUfonJ3gdjl8Nw3q3VJxubGJmCn0T41mwssgShIlYR6vreOLjHfzz\n3a2UVdYwfXgfTurVxTnBJzhJINk92Se5J/vY6EBKvaEpkCam630fu30iLDn40TCBUN6O0pBNEJ/u\nKGVryRH+fNHYVq0XFSXMHNuPJz7ewaGKGronBDQliDFhoaaunmfzCrjnrU3sKati2vDe/PCsYYxM\nDe/7dY4ntZUDJ7V1IOFgeN9udImLJi+E+0M0FKdnjk1t9bq541KpqVNeW1schMiMCT319cqClUWc\n+bdF/OyF1aQnJ/DMdV/isSsnhn1ygMBqEC/g3LUETkIZBcwPZlAdVUx0FBMykkN24L7SI9W8vLqY\nSwIsTjc2Jq07mT0SeGllEd+YaDexmfClqry3eR93vbaBtUVlDOvTjUevyGHa8N4hXV9sa4HUIO7z\n+b0W2KmqO4ITTseXnZHMPW9tpqyyhsT40GqGmbe8kOraei5rRXHal4gwKyuV+9/Zwt7ySnp3i2/j\nCI3x3rL8Uu56bQOfbDtA/+TO/P0bWeRmpREdFTmJoUEgCWIzsFdVKwFEpLOIpKvqruCG1jHlZKSg\n7sB9Xx4aOkNTNBSnx7eyON1YblYq/3h7C6+sKubKEL+d15jW2LynnD+/vpE31u2hZ9c4fpM7iksn\nDSAupuMWmU9UIO98HuB7D1c98Hxwwun4xjUM3BdiM8zl7Sxly97Dfof1bo0hfboxvG83m2nOhI2C\n0gp++OxKzr77PT7eup8fnDmURT86gytOzozo5ACBXUHEuPNIA6CqVSLSKYgxdWhdO8Uwol9iyNUh\n5izOp1sn/8N6t9asrFT+/PpGdh2oID0lNO/WMqYl+w9Xcf87W3n6k50g8J1TBnL96YNJ6RKcTmcd\nUSDpcb+InNvwQERmAqH19TjETMxMYcWug9SEyMB9Byuc4vQFE9LapBd0bpZzB9TCVXY3k+l4DlfV\ncvebmzjtrnd4/KPtXDA+jXd/eDo/P2+kJYdGAjlbXA/MEZH73cclfDY+k/EjOyOZxz/awfriMsb2\nT/I6HOYtc4rTl7TRnUfpKQmMH5DESyuLuP70QW2yTWOCraq2jqc/yef+d7Zw4Eg1Xxndlx+cNYzB\nvbt6HVrICqSj3CYgR0SS3Mcdb9q0dpaT+VmHOa8TRENxelx6Upvet52blcpvFqxjy95yBvfu1vIK\nxnikrl6Zt6yAu9/cTOHBo0wd3IMfnz2crHTvv7yFuhabmETkdyKSpKoHVfWgiCSLyG/aI7iOql/3\nzqQldQ6JDnN5O0vZvPcwl51gcbqx88b0I0rgpZXWzGRCk6ry+trdnHP3e/zouVX06BrH09+ZzOzv\nTrHkEKBAahAzfa8a3NnlZgUvpPCQ7XaYU9WWXxxEcxuK01knXpz21Tsxnikn9WDByiLP36MxjX28\ndT8XPPAR1z21lDpV/vnNCcy/YSqnDOnpdWgdSiAJIlpEjlVuRCQesEpOCyZmJrO3vIqC0qOexXCw\nopqFq4v56vi2KU43lpuVyvZ9R1hTWNbm2zbmeKwpPMS3H1vCpQ9/wu5Dlfzpa2N449bT+MqYfhHV\nA7qtBHLW+A/wPxF5zH18NTA7eCGFh+yMzyYQ8upW0BfcntMn2vehKeeM7ssd89fw0srCDjFJkglf\n2/cd4a9vbGThqmKSEmL5+bkj+NaXMoiPDY1hszuqQIrUfxCR1cB0d9FdqvpycMPq+Ib17Ua3TjHk\n7SjlgvH9233/qsqcxflktXFx2ldSQhynDenFwlXF3P6VEURF4FAExlt7yiq5563N/PfTXcRFR3HT\ntMFcc9pJITfMTUcVULuDqi4AFgQ5lrASHSWMG5DkWYe5pW5x+q6vtW5Y79bKHZfKWxv2krezlEkD\nU4K6L2MaHKqo4Z+LtvL4R9upq1cunzyAG6YNtvHB2lggo7lOBP4BjAA6AQJUqWr4j3V7gnIyUrj7\nrU0cOlpD987t+41mzpJ8ugahON3YjBF9iI+N4qWVhZYgTLv4eOt+/u/ppZRV1vDVcWncNmMoA3pY\nj/5gCKRI/QBwBbAN6AbcCNwbzKDCxcTMZFSd0SHb06GKGl5eVcxXx6cGff7oLp1imD6iD6+s3k1t\niPQcN+Hrwy37uOrxJfTu1olXbj6Vv39jnCWHIAokQUSp6kacMZlqVPVh4LwgxxUWxg1IIjpKWNrO\nzUzzlhdQFcTidGO5WakcOFLNh1v3t8v+TGR6b1MJVz/+KRkpXZh77RRG9LNGjGALJEEccW9zXSki\nfxCRmwC7NSAACXExjOyX2K4d5hp6TmelJzEqtX3uLDp9WC+6xcfw0gob4dUEx7sb9/LdJ/MY2NNJ\nDj272nih7SGQBHGl+7obgTpgCHBRIBsXkVtEZI2IrBWRW32W3yQiG9zldzWx7jkislFEtojITwPZ\nXyjKzkhu14H7luWXsmnPYS6blN4u+wPoFBPN2aP68sba3VTW1LXbfk1keGfDXq59cimDe3Vl7jVT\nbEC9dtRiglDVbapa6Q61cYeq3uyOz9QsERkNXANMArKAmSIyWETOAM4HslR1FPAXP+tGA/cDXwFG\nApeKyMhWvbMQkZOZTGVNPWuL2qcz2ZzFu5zi9HHMOX0icrNSKa+q5d2NJe26XxPe3ly3h2ufymNY\n327MuWYyyZYc2lUwZ8MYASxW1QpVrQUWARfijA57p6pWAajqXj/rTgK2uMmpGqez3vlBjDVocho6\nzLXDBEKHKmpYuKqI88el0qVTcIvTjZ08qAc9usSxwCYSMm3k9bW7uX72Ukb2S+Tp704mKcGSQ3sL\nZoJYA5wqIj1EJAE4F0gHhrrLF4vIIvc22sbSAN8pTQvcZV8gIteKSJ6I5JWUhN63177d4+mf3Jml\nO4NfqH7BLU4f75zTJyImOopzx/TjzfV7OFxV2+77N+Hl1dXF3DB7GaPTuvPUdye3+23ixhHIaK4X\nBrKsMVVdD/wJeAN4DViBU8OIAVKAKcCPgGfkBAZJUdWHVDVHVXN69QqdOaB95WQkk7czuAP3OcXp\nXWT1795uxenGcselUlVbz5vr9niyfxMeXl5VzI1zl5OVnsSTV0+yXtEeCuQK4hd+lv08kI2r6qOq\nmq2qpwGlwCacq4F56liCM8d14yEWC3GuNhr0d5d1SNmZKZSUV5F/oCJo+1iWf5CNe8rb7dZWf7IH\nJJPaPd7mqzbH7aWVRdz8n+VMGJDEE1dPopslB0812VAtImcD5wBpIvI3n6cScU7qLRKR3qq6V0QG\n4NQfprjrngG8IyJDcUaG3ddo1U+BISIyECcxXAJcFthbCj05GZ9NIJTRo0tQ9jFnsdNzelZW+xan\nfUVFCTOzUnnsg+2UHqm2gqJplReXF/L9Z1aQk5nCv6+c2O51NPNFzV1B7MWpI1QCa31+3sC5uygQ\nz4vIOpxxnG5w55V4DDhJRNbgFJ+vUFUVkVQReQXALWrfCLwOrAeeUdW1rX53IWJoH3fgviDVIbws\nTjeWm5VKbb3y2trdnsZhOpbnlhZw2zMrmDywB49fZckhVDT5KajqcmC5iMzG+dY/QFW3tGbjqnqq\nn2XV+JnTWlWLcArZDY9fAV5pzf5CVXSUMCEjmaVB6jD34orCdu053ZxRqYmc1LMLL60oCol4TOh7\n5tNd/GTeKqYO6snD386hcyGe2Z8AABpgSURBVJz1ww0VgdQgpgOrgf8BiMg4EXkhqFGFoZyMZDbt\nOcyhipo23W5Dz+mx/bszOs37ORlEhFlZqXyyfT97yiq9DseEuLlL8vnx86s4ZXBPHrnCkkOoCSRB\n/BaYDBwEUNUVwOBgBhWOsjOdOsTS/La9iliWf5ANu70tTjc2KysVVVi4yuarNk17+pOd3D5vNacP\n68XD386xyX1CUCAJosZ3TmqXTULcSuPSnYH72np+iLlL8ukSF02uh8Xpxgb37srIfonWac406cmP\nd/CLF9cwbXhv/vWtbEsOISqQBLFeRL4ORInIQBH5O/BJkOMKOwlxMYxKTWzTQvWho25xenxayBX1\ncselsmLXQfL3B+/WXtMx/fvD7fxy/lpmjOjDPy+fQKcYSw6hKpAEcSOQjVOofgGoBm5tdg3jV05G\nCit3HaS6tm0G7pu/opDKmnouC6HmpQYzxzoTFS1YZVcR5jOPvL+N3yxYx9mj+vDANy05hLpABus7\noqo/UdXxwATgt6pqXwuPQ05mMlW19awtOnTC22qYc3pMWmgUpxvrn5xAdkayNTOZY/61aCv/7+X1\nnDumL/ddNoG4mGCO9GPaQiBDbTwpIonueEqrgC0i8v3ghxZ+fDvMnajlu5zitBfjLgUqNyuVDbvL\n2bSn3OtQjMceeHcLf3x1AzPH9uOeS8YTG23JoSMI5FMaq6plwFdxbnXNwJkjwrRS78R40lM6t8kE\nQnMXO8VpL3tOt+TcMf2IEmwioQj3j7c2c9drGzl/XCp3f2OcJYcOJJBPKlZEYnCG257vdnSzyYeP\nU05GCktPcOC+ssoaFqwqIndcGl1DrDjtq1e3Tpw8qCcLVhUFdaBCE7rufnMTf/3fJi4cn8bfvj6O\nGEsOHUogn9YjQD6QDCxyx1U6HNSowlhOZjL7Dlez8wTu7nlxeegWpxvLzUpl5/4KVhWceN3FdByq\nyt/e2Mjdb27mouz+/PniLKKjjnvQZuORQIrUf1fVVFU9S52vgQXAtOCHFp6OTSB0nLe7+hanx/QP\nveJ0Y2eP7ktstNgIrxFEVfnLGxu59+0tfCMnnbu+NtaSQwfV6us9Va13m5nMcRjSuyuJ8THHPcPc\nil2h13O6Od07x/Llob1ZuKqIunprZgp3qsqdr23g/ne2cumkAfzxwjFEWXLosKxBsJ1FuQP3He8V\nxNwl+STERZM7LnSL043ljktlT1kVn7bDtKvGO6rKH15Zz78WbePyKQP4/VdHW3Lo4AK5zfULVVB/\ny0zgcjKS2bL3MAcrWnchVlZZw4KVxZw/LjWki9ONzRjRm86x0dbMFMZUld8uXMfD72/nypMz+d35\nlhzCQSBXEEsCXGYClJPp1CFaO0/1/OWFHK2p47JJGcEIK2gS4mKYMbIPr64upqbOboALN6rKr19a\ny78/3MHVUwfyq1kjOYFZhE0IaTJBiEhvEckCOovIGBEZ6/6cAiS0X4jhJ6t/EjFR0qpmJlVl9uJ8\nRqcldojidGO5WamUVtTwwZbGkweajqy+Xrlj/hqe+Hgn15w6kDtmjrDkEEaaa6c4D7gaZz7o+4GG\nT70cuCPIcYW1znHRjErr3qpC9cqCQ2zYXc7vLxgdxMiC57ShPUmMj2HBiiLOGNbb63BMG6ivV37+\n4hrmLsnn/748iJ+cM8ySQ5hpbka5fwP/FpGvq+oz7RhTRMjJSOapT3ZSVVsX0IBlcxbvdIrTIdxz\nujmdYqI5Z3RfXl5VTGVNnQ3v3MHV1yu3z1vNf/N2ccMZg/jhWZYcwlEgNYjeIpIIICIPisgSEZke\n5LjCXk5GMtW19awpLGvxtb7F6W7xse0QXXDkZqVxpLqOdzbs9ToUcwLq6pUfP7+K/+bt4ubpQyw5\nhLFAEsS1qlomImcB/YBrgLuCG1b4OzbDXADjMs1fUcTRmroO0/ehKV8a1IOeXTvZ3UwdWF298qNn\nV/Lc0gJumzGU75851JJDGAskQTT0bjoXeFJVVwa4nmlG727xZPRIaHFk14ae06NSExkTgsN6t0Z0\nlDBzbD/e2rCX8sq2nZvbBJ+q8sNnVzJveSE/PGsot8wY4nVIJsgCOdGvFJFXgJnAqyLSFZtytE1k\nZyS3OHDfyoJDrC8u49JJA8Lim9qsrH5U19bzxto9XodiWumdjXt5YXkht0wfwo3TLDlEgkASxFXA\nr4FJ7kRB8cB3ghlUpMjJSGH/kWq27zvS5GvmLnZ6Tp/fgXpON2fCgGTSkjrbTHMdjKry9/9tJj2l\nMzdOG+x1OKadBDJYXx1wEnC9u6hzIOuZluW4dYim+kOUV9bw0soicrM6dnHal4gwKyuVDzbv48AR\nG9Kro3hz/V5WFx7i5mlDbD6HCBLIUBv3AWcAl7uLjgAPBjOoSDG4V1e6d45laRN1iHApTjc2K6sf\ntfXKK6uLvQ7FBKC+Xvnb/zaR2SOBC8aneR2OaUeBfBU4WVWvAyoBVPUAEBfUqCJEVJSQnZHsd4a5\nhuL0yH6JjO2APaebM7JfIoN6dbH5qjuIN9btZn1xGTdPH2IT/kSYQD7tGhGJwi1Mi0gPbEa5NpOd\nkczWkiNfaG5ZVXCIdcVlXDY5PIrTvkSE3Kw0luw4wO5DlV6HY5pRX+/UHk7q1aXDdtI0x6+5sZga\nelnfDzwP9BKR3wAfAH9qh9giQk5GQ3+IzzczzV2ST+fY8ClONzYrqx+qsNCK1SHt1TW72binnFvs\n6iEiNfeJLwFQ1SeBXwB/AUqBi1X1P+0QW0TISk8iNlo+18wUjsXpxk7q1ZXRaYnWaS6E1dUrd7+5\nicG9uzJzbHh+UTHNay5BHGvXUNW1qnqPqt6tqmvaIa6IER8bzajU7p8rVM9fUURFdR2XTg6v4nRj\nuVmprCo4xI5mbvM13lm4qojNew9z64whNmVohGpuNNdeIvL9pp5U1b8FIZ6INDEzmSc+2kllTR2d\nYqKOFaezwqw43djMsan84ZUNLFhZxE3TreNVKKmrV+55azPD+nTj3NH9vA7HeKS5K4hooCvQrYkf\n00ayM1KorqtnTeEhVhc6xelLw7A43VhqUmcmZibz0sqiZnuTm/b30spCtpUc4bYzh9jMcBGsuSuI\nYlX97YlsXERuwRncT4CHVfVuEfm1u6zEfdnPVPUVP+veBnwX5+6p1cBVqhqWt7xkZ3zWYW7n/iNh\nXZxuLDcrlTvmr2XjnnKG9030OhwD1NbVc8+bmxnZL5GzRvb1OhzjoYBqEMdDREbjJIJJQBYwU0Qa\n+uj/XVXHuT/+kkMacDOQo6qjca5mLjmReEJZr26dyOyRwKKNJcxfUcSsrH4khmlxurGvjOlHdJTw\n0gorVoeKF5YXsmN/BbfOsKuHSNdcgjjROR9GAItVtUJVa4FFwIWtWD8GZ7rTGJwpTsP6DJKdkcLH\n2/ZTUV3HZZM71pzTJ6Jn106cPKgHC1ZZM1MoqKmr5963NzM6LZEzR/bxOhzjsSYThNtj+kSsAU4V\nkR4ikoAzXHi6+9yNIrJKRB4TkWQ/+y7Eua02HygGDqnqGycYT0ib6I7LNCICitON5WalsuvAUVbs\nOuh1KBHv+aUF7Dpw1OZ5MEAQB91T1fU4HereAF4DVgB1wD+BQcA4nJP/Xxuv6yaN84GBQCrQRUQu\nb/w697XXikieiOSVlJT4e0mHMOWkHkQJfGtKRsT9xzx7dF/ioqOsT4THqmvr+cfbW8hKT7J5ww0Q\n5FFZVfVRVc1W1dNwOtltUtU9qlqnqvXAwzg1isZmANtVtURVa4B5wMlN7OMhVc1R1ZxevXoF660E\nXWbPLrz7wzO4dFJ6yy8OM4nxsZw+rBcLVxVTV2/NTF55dukuCg8e5bYZQyLuS4rxL6gJQkR6u/8O\nwKk/zBER35uqL8BpimosH5giIgni/KVOB9YHM9ZQMKBHQsT+x8wdl0pJeRWLt+33OpSIVFVbx31v\nb2HCgCS+PLTjftEybSvYg6s8LyLrgAXADap6ELhLRFaLyCqcYcRvAxCRVHfmOlR1MfAcsAznFtco\n4KEgx2o8NH14H7rERdtEQh7576e7KD5UyffPHBaxX1LMFzXXD+KEqeqpfpZ9q4nXFuEUshse/wr4\nVfCiM6Gkc1w0Z47swyurd/Ob3NHExdjAcO2lsqaO+9/ZwqTMFKYO7uF1OCaE2P9CEzJmZaVy6GgN\n72/uuDcbdERzl+Szp6yKW8+02oP5PEsQJmScOqQX3TvH2kRC7ehodR0PvLuVKSelcPKgnl6HY0KM\nJQgTMuJiojh3TF/eWLeHo9V1XocTEWYv3klJeRW3zRjqdSgmBFmCMCFl1thUKqrreGvDHq9DCXsV\n1bU8uGgrpwzuyeSTrPZgvsgShAkpk0/qQe9unWxspnbw1Mc72Xe4mtvOtKHWjX+WIExIiY4Szhvb\nj3c3llBWWeN1OGHrcJVz9XDa0F5kZ6R4HY4JUZYgTMjJzUqluq6e19fs9jqUsPXERzsorajhthl2\n9WCaZgnChJxx6Umkp3S2sZmCpLyyhofe28a04b0ZP+ALY2Uac4wlCBNyRIQLxvfn/c37+PVLa6mu\nrfc6pLDy+Ic7OHS0xu5cMi0Kak9qY47XjWcM5nBlLY99uJ3luw5y/2Xj6Z+c4HVYHd6hozU8/P42\nZozow5gIG1betJ5dQZiQFBcTxS9njeSf35zAtr2HOe/eD3hrvd36eqIe+2A7ZZW13Gq1BxMASxAm\npH1lTD8W3HQKaUmd+c4Tefzx1fXU1lmT0/E4VFHDYx9s55xRfRmdZlcPpmWWIEzIy+zZhXnfO5nL\nJg/gX4u2cdnDi9l9qNLrsDqcRz7YRnlVLbdavwcTIEsQpkOIj43mDxeM4e5vjGNN0SHOu/d9G9Sv\nFUqPVPPYB9s5b0w/hvdN9Doc00FYgjAdylfHp/HSjVNJ6RLHtx9bwt//t8lmoQvAQ+9vo6Kmjlus\n9mBawRKE6XAG9+7G/BuncsG4NO55azNXPLaEfYervA4rZO0/XMUTH+1g1thUhvbp5nU4pgOxBGE6\npIS4GP769SzuvHAMn+44wHn3vs+S7Qe8DiskPfTeNipr6rh5ul09mNaxBGE6LBHhkkkDeOF7U+kc\nG82lD3/Cg4u2Um9NTseUlFfxxMc7OH9cGoN7d/U6HNPBWIIwHd7I1EQW3HQK54zqy52vbuCaJ/M4\nWFHtdVgh4cFFW6mpU7t6MMfFEoQJC93iY7nvsvH8JncU720u4bx7P2DFroNeh+WpPWWVPP3JTi4Y\nn8bAnl28Dsd0QJYgTNgQEa44OZNn/+9kAC5+8CMe/3A7qpHZ5PTPd7dSW6/cPM2uHszxsQRhws64\n9CRevvkUThvSi18vWMcNc5ZF3NwSxYeOMmdJPhdN6M+AHjaGlTk+liBMWEpKiOPhb+dw+1eG8/ra\nPeT+4wPWFh3yOqx288A7TrH+xmmDvQ7FdGCWIEzYiooSrvvyIP5z7RSO1tRxwQMfMXdJftg3ORUe\nPMp/Ps3n6xPTSU+xqwdz/CxBmLA3MTOFl28+lUmZKdw+bzU/eGYlFdW1XocVNPe9vQVBuOEMu3ow\nJ8YShIkIPbt24omrJ3HbjKG8sKKQ8+/7kM17yr0Oq83tOlDBs3m7uGRSOmlJnb0Ox3RwliBMxIiO\nEm6ZMYSnvzOZ0opqcu/7kBeWF3gdVpu67+0tREUJ3zvdrh7MibMEYSLO1ME9efnmUxmT1p3b/ruS\n2+etprKmzuuwTtjO/Ud4blkBl00aQN/u8V6HY8KAJQgTkfokxjPnmslcf/og5i7J58IHPmLHviNe\nh3VC7n1rCzFRwvdOH+R1KCZMWIIwESsmOoqfnDOcx67MofDgUWb+4wNeXV3sdVjHZfu+I7ywvIBv\nTcmgd6JdPZi2YQnCRLxpw/vw8s2nMKh3V66fvYzfLFhLdW3Hmtb03rc2ExcTxXVftqsH03YsQRgD\n9E9O4NnrvsRVUzP594c7uPhfH1NQWuF1WAHZsrec+SsKueJLmfTq1snrcEwYsQRhjCsuJopfzRrF\nA9+cwNa9hznv3g94e8Mer8Nq0T1vbSE+NpprTzvJ61BMmAlqghCRW0RkjYisFZFb3WW/FpFCEVnh\n/pzbxLpJIvKciGwQkfUi8qVgxmpMg3PH9GPhTaeQltSZqx/P4+cvrGbD7jKvw/Jr055yFq4q4sqT\nM+nR1a4eTNuKCdaGRWQ0cA0wCagGXhORhe7Tf1fVv7SwiXuA11T1IhGJA2zMANNuMnt2Yd73Tub3\nL6/nP5/mM3txPmPSunNxTn9ys1JJSojzOkQA7nlzM13iYrjmVLt6MG0vmFcQI4DFqlqhqrXAIuDC\nQFYUke7AacCjAKparaqRPbi/aXfxsdH87qujWfyzGfxy5khq65Vfzl/LpN+/xQ1zlrFoUwl1Hs5e\nt764jJdXF3PV1EySu4RGwjLhRYI1cJmIjADmA18CjgJvAXnAfuBKoMx9/ANVLW207jjgIWAdkAUs\nBW5R1S/cqC4i1wLXAgwYMCB7586dQXk/xgCsKTzEc0sLeHFFIQcraujXPZ4LJ6RxUXZ6u0/Kc91T\neXy0ZT8f/GQa3RNi23XfJnyIyFJVzfH7XDBHthSR7wDfA44Aa4Eq4I/APkCB3wH9VPXqRuvlAJ8A\nU1V1sYjcA5Sp6h3N7S8nJ0fz8vLa/o0Y00hVbR1vrtvLs0t38d6mEuoVJmYmc3F2OueO7UfXTkFr\nvQWcRDXzHx9w64wh3DpjaFD3ZcKbZwmiURB/AApU9QGfZZnAQlUd3ei1fYFPVDXTfXwq8FNVPa+5\nfViCMF7YfaiSecsLeC6vgG37jpAQF825Y/pxcXZ/Jg1MQUTafJ/ffSKPJdv388FPp5EYb1cP5vg1\nlyCC+jVHRHqr6l4RGYBTf5giIv1UtaG76gXAmsbrqepuEdklIsNUdSMwHae5yZiQ07d7PN87fTDX\nf3kQy/JLeTavgAUri3huaQEZPRK4aEJ/vpbdn9Q2Gl11VcFB3ly/hx+cOdSSgwmqYDcxvQ/0AGqA\n76vqWyLyFDAOp4lpB3CdqhaLSCrwiKqe6647DngEiAO2AVc1rlU0ZlcQJlRUVNfy6urdPLt0F59s\nO4AInDK4Jxdl9+fsUX2Jj40+7m1f9e8lLN91kPd/fAbdLEGYExQSTUztwRKECUX5+yt4blkBzy8t\noPDgURLjY8gdl8rF2emM7d+9VU1Qy/JLufCBj/jxOcNsSG/TJixBGBMC6uuVj7ft59m8Xby6ZjdV\ntfUM7dOVi7PT+er4tICGyfj2Y0tYU3iI9398Bl2CXAg3kcGzGoQx5jNRUcLUwT2ZOrgnv62sYeHK\nYp7J28XvX1nPn17bwOnDevP1nP6cMbw3sdFf7KKUt+MA720q4favDLfkYNqF/ZUZ44HE+FgumzyA\nyyYPYPOecp5bWsC85YW8uX4PPbvG8dVxaVyck86wvt2OrfP3NzfRs2sc3/pShoeRm0hiTUzGhIja\nunoWbSrh2bwC3tqwh5o6ZWz/7lyc3Z8+ifFc+9RSfnHeCL5rw2qYNmRNTMZ0ADHRUUwf0YfpI/qw\n/3AV81cU8UzeLu6YvxaAXt06cfkUu3ow7ccShDEhqEfXTlx9ykCumprJ2qIy5q8o5ORBPU/o9lhj\nWssShDEhTEQYndad0WndvQ7FRCCbMMgYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjjF+WIIwx\nxvhlCcIYY4xfliCMMcb4FVZjMYlICbDzOFfviTNXtrFj0Zgdj8+z4/GZcDgWGaray98TYZUgToSI\n5DU1YFWksWPxeXY8Ps+Ox2fC/VhYE5Mxxhi/LEEYY4zxyxLEZx7yOoAQYsfi8+x4fJ4dj8+E9bGw\nGoQxxhi/7ArCGGOMX5YgjDHG+BXxCUJEzhGRjSKyRUR+6nU8XhKRdBF5R0TWichaEbnF65i8JiLR\nIrJcRBZ6HYvXRCRJRJ4TkQ0isl5EvuR1TF4Skdvc/ydrRGSuiMR7HVNbi+gEISLRwP3AV4CRwKUi\nMtLbqDxVC/xAVUcCU4AbIvx4ANwCrPc6iBBxD/Caqg4Hsojg4yIiacDNQI6qjgaigUu8jartRXSC\nACYBW1R1m6pWA/8Bzvc4Js+oarGqLnN/L8c5AaR5G5V3RKQ/cB7wiNexeE1EugOnAY8CqGq1qh70\nNirPxQCdRSQGSACKPI6nzUV6gkgDdvk8LiCCT4i+RCQTGA8s9jYST90N/Bio9zqQEDAQKAH+7Ta5\nPSIiXbwOyiuqWgj8BcgHioFDqvqGt1G1vUhPEMYPEekKPA/cqqplXsfjBRGZCexV1aVexxIiYoAJ\nwD9VdTxwBIjYmp2IJOO0NgwEUoEuInK5t1G1vUhPEIVAus/j/u6yiCUisTjJYbaqzvM6Hg9NBXJF\nZAdO0+M0EXna25A8VQAUqGrDFeVzOAkjUs0AtqtqiarWAPOAkz2Oqc1FeoL4FBgiIgNFJA6nyPSS\nxzF5RkQEp415var+zet4vKSqt6tqf1XNxPm7eFtVw+4bYqBUdTewS0SGuYumA+s8DMlr+cAUEUlw\n/99MJwyL9jFeB+AlVa0VkRuB13HuQnhMVdd6HJaXpgLfAlaLyAp32c9U9RUPYzKh4yZgtvtlahtw\nlcfxeEZVF4vIc8AynLv/lhOGw27YUBvGGGP8ivQmJmOMMU2wBGGMMcYvSxDGGGP8sgRhjDHGL0sQ\nxhhj/LIEYUwbEJGb3RFOZzfx/JUicl8Tzx0ObnTGHJ+I7gdhTBv6HjBDVQu8DsSYtmIJwpgTJCIP\nAicBr4rI48Cp7uMK4FpVXdXo9QOBOUBXYH77RmtM4KyJyZgTpKr/hzPU8xlAJrBcVccCPwOe9LPK\nPTiD3o3BGQnUmJBkCcKYtnUK8BSAqr4N9BCRxEavmQrMdX9/qh1jM6ZVLEEY4w0b48aEPEsQxrSt\n94FvAojI6cA+P3NqfMhn01N+s/1CM6Z1LEEY07Z+DWSLyCrgTuAKP6+5BWe+79XYDIYmhNlorsYY\nY/yyKwhjjDF+WYIwxhjjlyUIY4wxflmCMMYY45clCGOMMX5ZgjDGGOOXJQhjjDF+/X9tJgj61qUj\n5gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Q4pI3iKwxC6",
        "colab_type": "text"
      },
      "source": [
        "My ranking from the greatest impact to the least impact on model performace is:\n",
        "\n",
        "1. Amount of data available for training.\n",
        "2. Decision tree vs. logistic regression classification.\n",
        "3. Subset of features included in the model.\n",
        "4. Number of folds or stratification strategy for cross-validation.\n",
        "5. Hyperparameter tuning for the best-performing model from option 3.\n",
        "\n",
        "I made this ranking based on how much differences in accuracy score the change has created. The change in the amount of data created the largest drop in accuracy score from 95.9 to 87.38 when the size of the data becomes really small.\n",
        "\n",
        "Logistic regression ranks the second because its best model only has a 88.2 accuracy score. Its non-best models are even more inaccurate compared to the decision tree model. Therefore logistic regression has a huge impact on the accuracy score second only to the size of data. \n",
        "\n",
        "Features ranks the third because the feature set with the worst performance only achieved an accuracy score of 88.46, which is higher than the accuracy scores after the first two changes but is also very significant. Number of folds for cross-validation ranks the fourth because its model with the worst performance still has a 95.55 accuracy score, which is not much different with the original model (95.9). Finally, hyperparameter tuning for the decision tree model ranks the last because it has the same accuracy score as the original model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOOkWEaIdEWu",
        "colab_type": "text"
      },
      "source": [
        "# Question 3\n",
        "\n",
        "Report the best-performing model and the total set of choices that you made in the options above to get to that level of performance. Compare that model performance to the result you reported in Homework 1. \n",
        "\n",
        "   - Describe one hypothetical business context for LendingClub where the amount of improvement reported in Question 3 would be worth the extra time and effort needed for optimization.\n",
        "   - Describe one hypothetical business context where the improvement would **not** have been worth the extra time and effort."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g91kwQihBon2",
        "colab_type": "text"
      },
      "source": [
        "The best model is achieved using only the financial features (\"fico\", \"dti\", \"emp_length\") while the number of fold is 0.\n",
        "\n",
        "The accuracy score of the best model is 96.77."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BiUR22p6A4Bq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "outputId": "17cb362f-4f47-47d8-bee3-6cb4e89b6def"
      },
      "source": [
        "# List the features from the data\n",
        "features = [\"fico\", \"dti\", \"emp_length\"]\n",
        "\n",
        "# Set up dataset and cross validation split\n",
        "X = lendingclub.loc[:, features]\n",
        "X = pd.get_dummies(X)\n",
        "y = lendingclub[\"outcome\"]\n",
        "\n",
        "num_folds = 10\n",
        "\n",
        "# Set up model\n",
        "classifier = DecisionTreeClassifier(criterion=\"entropy\", min_impurity_decrease=0.0001, min_samples_split=20, random_state=123).fit(X_train, y_train)\n",
        "\n",
        "accuracies = pd.DataFrame()\n",
        "fold = []\n",
        "train_accuracies = []\n",
        "test_accuracies = []\n",
        "\n",
        "# Calculate accuracies of the model based on different cross validation split\n",
        "for i in range(num_folds):\n",
        "    X_train = X.loc[X.index % num_folds != i]\n",
        "    X_test = X.loc[X.index % num_folds == i]\n",
        "    y_train = y.loc[y.index % num_folds != i]\n",
        "    y_test = y.loc[y.index % num_folds == i]\n",
        "    \n",
        "    model = classifier.fit(X_train, y_train)\n",
        "    train_pred = model.predict(X_train)\n",
        "    test_pred = model.predict(X_test)\n",
        "    train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "    test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "\n",
        "    fold.append(i)\n",
        "    train_accuracies.append(train_accuracy)\n",
        "    test_accuracies.append(test_accuracy)\n",
        "\n",
        "accuracies['fold'] = fold\n",
        "accuracies['train accuracies'] = train_accuracies\n",
        "accuracies['test accuracies'] = test_accuracies\n",
        "\n",
        "accuracies = accuracies.sort_values(by = 'test accuracies', ascending = False).reset_index(drop = True)\n",
        "accuracies"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fold</th>\n",
              "      <th>train accuracies</th>\n",
              "      <th>test accuracies</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>96.524277</td>\n",
              "      <td>96.769819</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>96.507884</td>\n",
              "      <td>96.609125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>96.499664</td>\n",
              "      <td>96.510481</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>96.574967</td>\n",
              "      <td>96.473924</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>96.672284</td>\n",
              "      <td>96.473490</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>7</td>\n",
              "      <td>96.533914</td>\n",
              "      <td>96.411837</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>3</td>\n",
              "      <td>96.521584</td>\n",
              "      <td>96.362515</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>2</td>\n",
              "      <td>96.509206</td>\n",
              "      <td>96.350635</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>6</td>\n",
              "      <td>96.325675</td>\n",
              "      <td>96.325524</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>4</td>\n",
              "      <td>96.555834</td>\n",
              "      <td>96.202219</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   fold  train accuracies  test accuracies\n",
              "0     0         96.524277        96.769819\n",
              "1     8         96.507884        96.609125\n",
              "2     9         96.499664        96.510481\n",
              "3     1         96.574967        96.473924\n",
              "4     5         96.672284        96.473490\n",
              "5     7         96.533914        96.411837\n",
              "6     3         96.521584        96.362515\n",
              "7     2         96.509206        96.350635\n",
              "8     6         96.325675        96.325524\n",
              "9     4         96.555834        96.202219"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_mII26edISl",
        "colab_type": "text"
      },
      "source": [
        "The accuracy score of the best model (96.77%) improved slightly compared to the original model (95.9%).\n",
        "\n",
        "The optimization is worth it when the costs of wrong prediction is very high. For example, the company could use the model to decide whether to provide service to a loan borrower. If the company is still at the startup stage, where every customer is very valuable and losing one customer could bring a huge loss to the company, then it is necessary for the company to improve the accuracy of the model, even just slightly. \n",
        "\n",
        "The optimization is not worth the time and effort when the costs of wrong prediction is relatively low. For example, if the company is already a large-scale developed company and it has a lot of customers, losing one customer due to wrong prediction will not cause much damage to the company. In this case, a lot of efforts made for a slight improvement in the accuracy of the model is not that necessary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-nzeiXvm05T",
        "colab_type": "text"
      },
      "source": [
        "# Extra Credit\n",
        "\n",
        "For up to 2 points of extra credit, complete all three of the analyses in Question 1.\n",
        "\n",
        "For up to 4 points of extra credit, complete all five of the analyses in Question 2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_2upx-ygRKE-",
        "colab_type": "text"
      },
      "source": [
        "# Scoring Rubric\n",
        "![](https://drive.google.com/uc?export=view&id=1dLF4RcciHT_giZEXWio1D0hwK73tKYjK)\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1nYN7X847FeHtE-0LVpqhQk8iq2YEWx2d)"
      ]
    }
  ]
}