ReproductionOfAvailableSolutions.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "V100",
      "authorship_tag": "ABX9TyMCWf5iqPWvqzZ7oX4oXlAM",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/anastasiaarsky/ML_Capstone/blob/main/ReproductionOfAvailableSolutions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Potential Public Solutions**\n",
        "The following is a summary of research papers/articles (with links) that focus on classifying spam emails. Since my dataset contains both the [SpamAssassin](https://spamassassin.apache.org/old/publiccorpus/) and [Enron Spam](https://www2.aueb.gr/users/ion/data/enron-spam/) datasets, this section will be split into two parts, each corresponding with one of the datasets."
      ],
      "metadata": {
        "id": "JxaU8Buh5NWS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Solution with the SpamAssassin dataset:**"
      ],
      "metadata": {
        "id": "lm60FF-w5XK6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **\"[Email Spam Classification](https://medium.com/@yesprabhakaran98/email-spam-classification-92b661d3b700)\"** by prabhakran98  (via medium.com):\n",
        "\n",
        "***Dataset***:\n",
        "- [SpamAssassin dataset](https://spamassassin.apache.org/old/publiccorpus/)\n",
        "\n",
        "***Data Preprocessing***:\n",
        "- Perform **TF-IDF** on the text corpus  \n",
        "  >TF-IDF is an algorithm that uses the frequency of words to determine how relevant those words are to a given document.\n",
        "  \n",
        "***Model Training Techniques & Evaluation***:\n",
        "\n",
        "| Model                     | Accuracy    | F 0.5 Score\n",
        "| :-----------------------: | :---------: | :----------:\n",
        "| **Random Forest**         | 0.98        | 0.98\n",
        "| **XGBoost**               | 0.97        | 0.96\n",
        "| **Decision Tree**         | 0.93        | 0.93\n",
        "| **Logistic Regression**   | 0.93        | 0.93\n",
        "| **Naive Bayes**           | 0.89        | 0.90\n",
        "| **KNN**                   | 0.67        | 0.54\n",
        "\n",
        "  >Notes:\n",
        "  - All models included hyperparameter tuning using a grid search\n",
        "  - F 0.5 Score is used as a metric (rather than F1 score) since the SpamAssassin dataset is imbalanced\n",
        "\n",
        "  ***Conclusion***:  \n",
        "  Ensemble Models perform better, especially the **Random Forest model**, as it had the highest accuracy and F 0.5 score.   \n",
        "  However, all models acheive a 90+ score except for K Nearest Neighbor."
      ],
      "metadata": {
        "id": "y_2SycNB8Xbj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Solution with the Enron Spam dataset:**"
      ],
      "metadata": {
        "id": "zXWAGElF5kUF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **\"[Empirical Analysis on Email Classification Using the Enron Dataset](https://towardsdatascience.com/empirical-analysis-on-email-classification-using-the-enron-dataset-19054d558697)\"** by Suleka Helmini  (via towardsdatascience.com):\n",
        "- [Github Code Link](https://github.com/suleka96/Email-Classification)\n",
        "\n",
        "***Dataset***:\n",
        "- [Enron Spam dataset](https://www2.aueb.gr/users/ion/data/enron-spam/)\n",
        "\n",
        "***Data Preprocessing***:  \n",
        "For the ML models:  \n",
        "- Utilize a **bag-of-words** approach:  \n",
        "> First, make a dictionary of the 3000 most common words in the corpus. Then iterate through the emails and record the occurence frequency of each token in the email corresponding to the token in the dictionary.\n",
        "\n",
        "For the DL model (i.e. the Recurrent Neural Network):\n",
        "- Utilize a slightly different **bag-of-words** approach:  \n",
        "> First, create a dictionary of all the words in the corpus and index them starting from 1. Then iterate through the emails, and map each word to the corresponding index in the dictionary (this ensures the sequence of words is retained). Finally, to ensure all the emails have the same length, pad all the emails to the length of the longest email by adding 0's to the beginning of each sequence.\n",
        "\n",
        "***Model Training Techniques & Evaluation***:\n",
        "\n",
        "| ML Model                          | Accuracy | F1 Score| Recall|Precision     \n",
        "| :-------------------------------: | :-------:| :-----: | :----:| :------:\n",
        "| **Logistic Regression**           | 0.973    | 0.969   | 0.972 | 0.966\n",
        "| **Random Forest Classifier**      | 0.971    | 0.966   | 0.969 | 0.964\n",
        "| **Support Vector Classification** | 0.958    | 0.952   | 0.962 | 0.943\n",
        "\n",
        "\n",
        "> Notes:\n",
        "  - All ML models included manual hyperparameter tuning\n",
        "\n",
        "| DL Model                             | Accuracy  | F1 Score| Recall|Precision\n",
        "| :----------------------------------: | :-------: | :-----: | :----:| :------:\n",
        "| **Recurrent Neural Network (LSTM)**  | 0.981     | 0.980   | 0.980 | 0.981\n",
        "> Notes:\n",
        "  - Cost function: softmax cross entropy with logits\n",
        "  - Optimization function: Adam optimizer\n",
        "  - Activation function: sigmoid\n",
        "\n",
        "  ***Conclusion***:  \n",
        "  Logistic Regression outperformed all of the ML Models, but the **RNN model** outperformed the logistic regression algorithm because of its ability to model long term dependencies."
      ],
      "metadata": {
        "id": "t-cU9lUh6I-p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Reproduction of Available Solutions**"
      ],
      "metadata": {
        "id": "C1pH9g7y50EI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "leggU7UfSo33",
        "outputId": "62a573ee-e46d-4a8b-c3a4-150d96842c73"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import os\n",
        "import datetime\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.model_selection import RandomizedSearchCV, train_test_split\n",
        "from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay\n",
        "from sklearn.metrics import f1_score, recall_score, precision_score, accuracy_score"
      ],
      "metadata": {
        "id": "JkS-f2RkSBwC"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load clean data into a pandas dataframe\n",
        "DATA_PATH = \"/content/drive/My Drive/UCSD Machine Learning Engineering Bootcamp/Capstone Project/\"\n",
        "df = pd.read_csv(DATA_PATH + 'CleanData.csv')"
      ],
      "metadata": {
        "id": "6TCBHK4OStJs"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'"
      ],
      "metadata": {
        "id": "2XKrYNKMNBef"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Reproduction of SpamAssassin dataset Solution:**\n",
        "(TF-IDF for the preprocessing, and a Random Forest model for the training)\n",
        "\n",
        "****NOTE**: the article did not have a link to the code, so I had to code it myself"
      ],
      "metadata": {
        "id": "SERlwOXUOlgS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df1 = df"
      ],
      "metadata": {
        "id": "-oohnoj_aCdE"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Step 1: TF-IDF"
      ],
      "metadata": {
        "id": "t_7L2TfZibLi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TF-IDF\n",
        "\n",
        "# initialize the tfidf vectorizer\n",
        "tfidf_vec = TfidfVectorizer(max_features=10000)\n",
        "# convert Clean_Text to numbers with TfidfVectorizer\n",
        "tfidf = tfidf_vec.fit_transform(df1['Clean_Text'])"
      ],
      "metadata": {
        "id": "5bqWw0FuPv4-"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# convert the tfidf data features to array\n",
        "tfidf = tfidf.toarray()\n",
        "# show shape of the feature vector\n",
        "tfidf.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7yRL27NDUZU_",
        "outputId": "8e9cb36b-1751-459a-bcd6-c54f5ecb5cf8"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(39763, 10000)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Step 2: Preliminary Random Forest model"
      ],
      "metadata": {
        "id": "aDYJ_nh6if9y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Split data into training and testing set\n",
        "X_idf = tfidf\n",
        "y = df1.Label\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_idf, y, test_size=0.3, shuffle=True)"
      ],
      "metadata": {
        "id": "Ut2M7nNMcTNm"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Random Forest: initial fitting\n",
        "\n",
        "# Create a random forest classifier\n",
        "rf = RandomForestClassifier()\n",
        "\n",
        "# Fit the random forest classifier to the data\n",
        "rf.fit(X_train, y_train)\n",
        "\n",
        "# Generate Predictions\n",
        "y_pred = rf.predict(X_test)\n",
        "\n",
        "# Print accuracy metrics\n",
        "accuracy = accuracy_score(y_test, y_pred)\n",
        "f1 = f1_score(y_test, y_pred, average='macro')\n",
        "recall = recall_score(y_test, y_pred, average='macro')\n",
        "precision = precision_score(y_test, y_pred, average='macro')\n",
        "\n",
        "print(\"Preliminary Random Forest Results:\")\n",
        "print(\"Test accuracy: {:.3f}\".format(accuracy))\n",
        "print(\"F1 Score: {:.3f}\".format(f1))\n",
        "print(\"Recall: {:.3f}\".format(recall))\n",
        "print(\"Precision: {:.3f}\".format(precision))\n",
        "\n",
        "# Create the confusion matrix\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "\n",
        "ConfusionMatrixDisplay(confusion_matrix=cm).plot();"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 543
        },
        "id": "mqJAjL5Xt8V1",
        "outputId": "85ea5b0f-284a-4e1c-b1a3-fe4e358e53b1"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Preliminary Random Forest Results:\n",
            "Test accuracy: 0.982\n",
            "F1 Score: 0.982\n",
            "Recall: 0.982\n",
            "Precision: 0.981\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAG1CAYAAABkoPeiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABA9UlEQVR4nO3dfVxUdfr/8feADKAy4y0gioq53mCmia2xlWmRVFaa9u1bUZKp/SyslEyzG1O7sa+uma53W1borm7anZtamqupmWRK2pqp5V1gClgKCMrdzPn9QUxOOsU4IMh5PR+Pz2Obcz7nzDUuD+biuj7nHIthGIYAAIBp+VV3AAAAoHqRDAAAYHIkAwAAmBzJAAAAJkcyAACAyZEMAABgciQDAACYHMkAAAAmRzIAAIDJkQwAAGByJAMAAFSRH3/8Uffee68aN26s4OBgde7cWdu2bXPtNwxD48ePV7NmzRQcHKy4uDh9//33buc4fvy4EhISZLPZ1KBBAw0ZMkT5+fluc/773//qmmuuUVBQkCIjIzVlyhSv4iQZAACgCpw4cUJXXXWVAgIC9PHHH+vbb7/VtGnT1LBhQ9ecKVOmaObMmZo3b562bNmievXqKT4+XoWFha45CQkJ2rVrl9asWaMVK1Zo48aNevDBB1378/Ly1KdPH7Vq1UppaWmaOnWqJkyYoNdee63CsVou5gcVOZ1OHTlyRCEhIbJYLNUdDgDAS4Zh6OTJk4qIiJCfX9X9fVpYWKji4mKfz2O1WhUUFFShuU8++aQ+//xzffbZZ+fcbxiGIiIi9Pjjj2v06NGSpNzcXIWFhSklJUV33XWXdu/erejoaG3dulXdu3eXJK1atUo333yzDh8+rIiICM2dO1dPP/20MjMzZbVaXe+9bNky7dmzp2IfzLiIZWRkGJIYDAaDcZGPjIyMKvuuOH36tBEe6l8pcYaHhxtZWVlGbm6uaxQWFp7zfTt27GiMHDnSuOOOO4ymTZsaXbt2NV577TXX/v379xuSjO3bt7sd17NnT+PRRx81DMMw3njjDaNBgwZu+0tKSgx/f3/j/fffNwzDMO677z6jX79+bnPWrVtnSDKOHz9eoX+jOrqIhYSESJJ++Kq1bPXpeKB2ur1D1+oOAagypUaJNhnLXb/Pq0JxcbEysx36Ia21bCHn/12Rd9KpVjGHFBYW5rb9ueee04QJE86af+DAAc2dO1fJycl66qmntHXrVj366KOyWq1KTExUZmamJJ11vrCwMNe+zMxMhYaGuu2vU6eOGjVq5DYnKirqrHOU7zuzLeHJRZ0MlLcGbPX9fPo/GKjJ6lgCqjsEoGoZuiCt3vohFtUPOf/3cars2IyMDNlsNtf2wMDAc893OtW9e3e99NJLkqTLL79c33zzjebNm6fExMTzjqMq8A0KADAFh+H0eUiSzWZzG56SgWbNmik6OtptW8eOHZWeni5JCg8PlyRlZWW5zcnKynLtCw8PV3Z2ttv+0tJSHT9+3G3Ouc5x5nv8EZIBAIApOGX4PLxx1VVXae/evW7bvvvuO7Vq1UqSFBUVpfDwcK1du9a1Py8vT1u2bFFsbKwkKTY2Vjk5OUpLS3PNWbdunZxOp3r06OGas3HjRpWUlLjmrFmzRu3bt69Qi0AiGQAAoEqMGjVKX3zxhV566SXt27dPixcv1muvvaakpCRJZa2RkSNH6oUXXtCHH36onTt3atCgQYqIiFD//v0llVUSbrzxRg0bNkxffvmlPv/8c40YMUJ33XWXIiIiJEn33HOPrFarhgwZol27dmnJkiWaMWOGkpOTKxzrRb1mAACAinLKKaePx3vjiiuu0AcffKBx48Zp0qRJioqK0quvvqqEhATXnDFjxqigoEAPPvigcnJydPXVV2vVqlVuly8uWrRII0aM0PXXXy8/Pz8NHDhQM2fOdO232+365JNPlJSUpJiYGDVp0kTjx493uxfBH7mo7zOQl5cnu92uE9+1YQEhaq34FjHVHQJQZUqNEq13vq/c3Fy3RXmVqfy7ImNPc5+vJojs8GOVxlpd+AYFAMDkaBMAAEzhfBYB/vb42opkAABgCk4ZcpAMnBNtAgAATI7KAADAFGgTeEYyAAAwBYdhyOHDBXS+HFvT0SYAAMDkqAwAAEzB+cvw5fjaimQAAGAKDh+vJvDl2JqOZAAAYAoOo2z4cnxtxZoBAABMjsoAAMAUWDPgGckAAMAUnLLIIYtPx9dWtAkAADA5KgMAAFNwGmXDl+NrK5IBAIApOHxsE/hybE1HmwAAAJOjMgAAMAUqA56RDAAATMFpWOQ0fLiawIdjazraBAAAmByVAQCAKdAm8IxkAABgCg75yeFDQdxRibHUNCQDAABTMHxcM2CwZgAAANRWVAYAAKbAmgHPSAYAAKbgMPzkMHxYM1CLb0dMmwAAAJOjMgAAMAWnLHL68DewU7W3NEAyAAAwBdYMeEabAAAAk6MyAAAwBd8XENImAADgola2ZsCHBxXRJgAAALUVlQEAgCk4fXw2AVcTAABwkWPNgGckAwAAU3DKj/sMeMCaAQAATI7KAADAFByGRQ4fHkPsy7E1HckAAMAUHD4uIHTQJgAAALUVlQEAgCk4DT85fbiawMnVBAAAXNxoE3hGmwAAAJOjMgAAMAWnfLsiwFl5odQ4JAMAAFPw/aZDtbeYXns/GQAAqBAqAwAAU/D92QS19+9nkgEAgCk4ZZFTvqwZ4A6EAABc1KgMeFZ7PxkAAKgQKgMAAFPw/aZDtffv59r7yQAAOIPTsPg8vDFhwgRZLBa30aFDB9f+wsJCJSUlqXHjxqpfv74GDhyorKwst3Okp6erb9++qlu3rkJDQ/XEE0+otLTUbc769evVrVs3BQYGqm3btkpJSfH634ZkAACAKtKpUycdPXrUNTZt2uTaN2rUKC1fvlzvvPOONmzYoCNHjmjAgAGu/Q6HQ3379lVxcbE2b96sBQsWKCUlRePHj3fNOXjwoPr27avevXtrx44dGjlypIYOHarVq1d7FSdtAgCAKTh9bBOcz02H6tSpo/Dw8LO25+bm6o033tDixYt13XXXSZLeeustdezYUV988YWuvPJKffLJJ/r222/1n//8R2FhYeratauef/55jR07VhMmTJDVatW8efMUFRWladOmSZI6duyoTZs2afr06YqPj69wnFQGAACmUP7UQl+Gt77//ntFRESoTZs2SkhIUHp6uiQpLS1NJSUliouLc83t0KGDWrZsqdTUVElSamqqOnfurLCwMNec+Ph45eXladeuXa45Z56jfE75OSqKygAAAF7Iy8tzex0YGKjAwMCz5vXo0UMpKSlq3769jh49qokTJ+qaa67RN998o8zMTFmtVjVo0MDtmLCwMGVmZkqSMjMz3RKB8v3l+35vTl5enk6fPq3g4OAKfSaSAQCAKThkkcOHGweVHxsZGem2/bnnntOECRPOmn/TTTe5/vuyyy5Tjx491KpVKy1durTCX9IXCskAAMAUzrfUf+bxkpSRkSGbzebafq6qwLk0aNBA7dq10759+3TDDTeouLhYOTk5btWBrKws1xqD8PBwffnll27nKL/a4Mw5v70CISsrSzabzauEgzUDAAB4wWazuY2KJgP5+fnav3+/mjVrppiYGAUEBGjt2rWu/Xv37lV6erpiY2MlSbGxsdq5c6eys7Ndc9asWSObzabo6GjXnDPPUT6n/BwVRTIAADAFh35tFZzf8M7o0aO1YcMGHTp0SJs3b9btt98uf39/3X333bLb7RoyZIiSk5P16aefKi0tTYMHD1ZsbKyuvPJKSVKfPn0UHR2t++67T19//bVWr16tZ555RklJSa4EZPjw4Tpw4IDGjBmjPXv2aM6cOVq6dKlGjRrlVay0CQAAplBZbYKKOnz4sO6++279/PPPatq0qa6++mp98cUXatq0qSRp+vTp8vPz08CBA1VUVKT4+HjNmTPHdby/v79WrFihhx56SLGxsapXr54SExM1adIk15yoqCitXLlSo0aN0owZM9SiRQvNnz/fq8sKJcliGIbh1RE1SF5enux2u05810a2EIocqJ3iW8RUdwhAlSk1SrTe+b5yc3Pd+vCVqfy7YlzqjQqqH3De5ynML9Hk2FVVGmt14RsUAACTo00AADAFQxY5fbi00PDh2JqOZAAAYAoOw08OH9YM+HJsTVd7PxkAAKgQKgMAAFM4n8cQ//b42opkAABgCg4fn1roy7E1Xe39ZAAAoEKoDAAATIE2gWckAwAAU3DKT04fCuK+HFvT1d5PBgAAKoTKAADAFByGRQ4fSv2+HFvTkQwAAEyBNQOekQwAAEzB8PGphQZ3IAQAALUVlQEAgCk4ZJHDh4cN+XJsTUcyAAAwBafhW9/faVRiMDUMbQIAAEyOZMCEfjoaoP8b0VJ3dLpUt7a5TP/vuvb67utg137DkBZMCdfdXTvp1jaXaeydl+jHA1bX/swMq15JjtSgHh11a5vLdH9sRy2cGq6SYveM+8C3QUru31a3RF2mhJhoLZ0desE+I3CmS3uc1MS39mnxtp1affgrxcbneJz76OR0rT78lW4fkn3O/QFWp+as3q3Vh79Sm+hTVRQxqoLzlwWEvozaqkZ8stmzZ6t169YKCgpSjx499OWXX1Z3SLXWyRx/Jff7k/zrGHrhnwf0+vo9enD8EdW3O1xzls4O1b/fbKpHXs7QjBXfKaiuU0/dc4mKC8u+7DP2BcrplB77v8N67dM9+n8TftTKfzTWW5Obuc5RcNJPT919icJaFGvWqu807Nkj+ue0cH30z8YX/DMDQXWdOvBtXc16JvJ35/3lxhx16FagnzIDPM4Z8vSP+jnL837UXE5ZfB61VbUnA0uWLFFycrKee+45ffXVV+rSpYvi4+OVnX3urBy+WTo7VE0iijX61Qx1uPyUwlsWK6bXSUW0LpZUVhVYNr+p7n4sU3+5MU9togs1ZuYP+jkrQJtX2SVJV/Q+qdGvZiim10k1a1Ws2Pg83TE8W59/bHe9z7r3G6qkxKLkVzLUun2hevXPUb8hx/Te35tWy+eGuW371K4FUyO0eVUDj3Mahxfr4ecz9H+PtFZpybl/6XfvnauYnnl6/fnmVRQpUD2qPRl45ZVXNGzYMA0ePFjR0dGaN2+e6tatqzfffLO6Q6uVvvjErnZdTumFB1vrzs6d9PAN7fTRokau/ZnpVh3PDlC3a/Jd2+rZnOpw+SntTqvn8bwFJ/0V0uDX6sLutHrq3KNAAdZfV9zE9Dqpw/uDdDLHv5I/FeAbi8XQmBmH9O68MP3wXfA55zRoUqKRU9I15bHWKjpd7b86cR7K70Doy6itqvUnuri4WGlpaYqLi3Nt8/PzU1xcnFJTU6sxstrraLpVKxY2UURUkV5afEC3JP6suc+20JqlDSVJx7PLLjBp0LTE7bgGTUtc+37rx4NW/fvNprr5vp9c205k11HD35yj/PWJY1zEgprlzoez5Ci1aNkbnipXhkZP/0Er/9FE3//Xc1KMmo01A55V62/ln376SQ6HQ2FhYW7bw8LCtGfPnrPmFxUVqaioyPU6Ly+vymOsbQyn9KfLTuuBcUclSW07n9ahPUFa+Y8muuHOE16f76ejAXo64RL1vCVHNyccr+xwgSrXtvMp9R+SraSbOkgeesL9Hjim4HoOLZkVfmGDAy6Qi+pPtMmTJ2vixInVHcZFrVFoqVq1K3TbFvmnQm36yO7aL0k5xwLUOKzUNSfnWIAu6XTa7bifM+tozP9coujuBXpsaobbvoahpTpxzH2RVfnrhk1LBdQUnf+crwZNSvXPLd+4tvnXkYaNP6z+Q7OVGHupuv7lpDrGFGjFge1ux876aI/WfdBIfx3V+gJHjfPhlI/PJqjFCwirNRlo0qSJ/P39lZWV5bY9KytL4eFnZ+Djxo1TcnKy63VeXp4iI39/dTDcRV9RoIz9gW7bfjwQqNDmZSX88JbFahRaou2b6uuSS8u+/AtO+mnP9rq6ZdCvbYCfjgZozP9coj91Pq3Hp6fL7zfVs44xBUr5v2YqLZHq/JITfLUxRC0uKXRbWwBUt/+810hfbQpx2/bSon1a+14jfbKk7OqXOeMjlTI1wrW/cViJJi/ep5cejtKe7bQNLhaGj1cEGCQDVcNqtSomJkZr165V//79JUlOp1Nr167ViBEjzpofGBiowMDAs7aj4gY8mK1Rt7XTv2aGquetOdq7va4++mdjjZx6WJJksUj9hx7Tv2aEqXlUkcJbFmvBlGZqHFaiv9yYK6ksEXjijrYKbV6sYeOPKPfnX3+MyisL191+QoteCdcrj7fUnUnZOrQnSMvmN9HwiUcu/IeG6QXVdSii9a8txvDIIrWJPqWTOXV07IhVJ3PcfxWWllh0IjtAhw8ESZKOHbG67S8sKMt+jxwK1E9H3feh5uKphZ5Ve5sgOTlZiYmJ6t69u/785z/r1VdfVUFBgQYPHlzdodVK7bue1vg3Duqtyc20aHq4wiOLNXzSj7puwK/rBe5MylbhKT/NGBOp/Dx/dbqiQC8uOiBrUNmVAV9tDNGRg4E6cjBQCTGd3M6/+sgOSWVXILz0r/2a9VQLjbixneyNSpUwKks33/vzBfusQLl2XU5p6jvfu14Pn/CjJOmTpY00Lbl1NUUF1BwWwzCq/W7Ls2bN0tSpU5WZmamuXbtq5syZ6tGjxx8el5eXJ7vdrhPftZEtpPau8oS5xbeIqe4QgCpTapRovfN95ebmymazVcl7lH9X3L5msALqnX8lp6SgWB/c8FaVxlpdqr0yIEkjRow4Z1sAAIDKQpvAM/6cBgDA5GpEZQAAgKrm6/MFuLQQAICLHG0Cz2gTAABgclQGAACmQGXAM5IBAIApkAx4RpsAAACTozIAADAFKgOekQwAAEzBkG+XB1b77XqrEMkAAMAUqAx4xpoBAABMjsoAAMAUqAx4RjIAADAFkgHPaBMAAGByVAYAAKZAZcAzkgEAgCkYhkWGD1/ovhxb09EmAADA5KgMAABMwSmLTzcd8uXYmo5kAABgCqwZ8Iw2AQAAJkdlAABgCiwg9IxkAABgCrQJPKNNAAAwhfLKgC/jfL388suyWCwaOXKka1thYaGSkpLUuHFj1a9fXwMHDlRWVpbbcenp6erbt6/q1q2r0NBQPfHEEyotLXWbs379enXr1k2BgYFq27atUlJSvI6PZAAAgCq0detW/f3vf9dll13mtn3UqFFavny53nnnHW3YsEFHjhzRgAEDXPsdDof69u2r4uJibd68WQsWLFBKSorGjx/vmnPw4EH17dtXvXv31o4dOzRy5EgNHTpUq1ev9ipGkgEAgCkYv7QJznecT2UgPz9fCQkJev3119WwYUPX9tzcXL3xxht65ZVXdN111ykmJkZvvfWWNm/erC+++EKS9Mknn+jbb7/VP//5T3Xt2lU33XSTnn/+ec2ePVvFxcWSpHnz5ikqKkrTpk1Tx44dNWLECN1xxx2aPn26V3GSDAAATMGQZBg+jPN4z6SkJPXt21dxcXFu29PS0lRSUuK2vUOHDmrZsqVSU1MlSampqercubPCwsJcc+Lj45WXl6ddu3a55vz23PHx8a5zVBQLCAEA8EJeXp7b68DAQAUGBp417+2339ZXX32lrVu3nrUvMzNTVqtVDRo0cNseFhamzMxM15wzE4Hy/eX7fm9OXl6eTp8+reDg4Ap9JioDAABTKL8DoS9DkiIjI2W3211j8uTJZ71XRkaGHnvsMS1atEhBQUEX+qN6jcoAAMAUKus+AxkZGbLZbK7t56oKpKWlKTs7W926dXNtczgc2rhxo2bNmqXVq1eruLhYOTk5btWBrKwshYeHS5LCw8P15Zdfup23/GqDM+f89gqErKws2Wy2ClcFJCoDAAB4xWazuY1zJQPXX3+9du7cqR07drhG9+7dlZCQ4PrvgIAArV271nXM3r17lZ6ertjYWElSbGysdu7cqezsbNecNWvWyGazKTo62jXnzHOUzyk/R0VRGQAAmILTsMhygW46FBISoksvvdRtW7169dS4cWPX9iFDhig5OVmNGjWSzWbTI488otjYWF155ZWSpD59+ig6Olr33XefpkyZoszMTD3zzDNKSkpyJSDDhw/XrFmzNGbMGD3wwANat26dli5dqpUrV3r12UgGAACmUH5VgC/HV6bp06fLz89PAwcOVFFRkeLj4zVnzhzXfn9/f61YsUIPPfSQYmNjVa9ePSUmJmrSpEmuOVFRUVq5cqVGjRqlGTNmqEWLFpo/f77i4+O9isViGJX98S6cvLw82e12nfiujWwhdDxQO8W3iKnuEIAqU2qUaL3zfeXm5rr14StT+XdFpyVPyL/u2SX9inKcKtKu/51apbFWFyoDAABT4EFFnpEMAABMgWTAM5IBAIApXMgFhBcbGu0AAJgclQEAgCnUtKsJahKSAQCAKZQlA76sGajEYGoY2gQAAJgclQEAgClwNYFnJAMAAFMwfhm+HF9b0SYAAMDkqAwAAEyBNoFnJAMAAHOgT+ARyQAAwBx8rAyoFlcGWDMAAIDJURkAAJgCdyD0jGQAAGAKLCD0jDYBAAAmR2UAAGAOhsW3RYC1uDJAMgAAMAXWDHhGmwAAAJOjMgAAMAduOuRRhZKBDz/8sMInvO222847GAAAqgpXE3hWoWSgf//+FTqZxWKRw+HwJR4AAHCBVSgZcDqdVR0HAABVrxaX+n3h05qBwsJCBQUFVVYsAABUGdoEnnl9NYHD4dDzzz+v5s2bq379+jpw4IAk6dlnn9Ubb7xR6QECAFApjEoYtZTXycCLL76olJQUTZkyRVar1bX90ksv1fz58ys1OAAAUPW8TgYWLlyo1157TQkJCfL393dt79Kli/bs2VOpwQEAUHkslTBqJ6/XDPz4449q27btWdudTqdKSkoqJSgAACod9xnwyOvKQHR0tD777LOztr/77ru6/PLLKyUoAABw4XhdGRg/frwSExP1448/yul06v3339fevXu1cOFCrVixoipiBADAd1QGPPK6MtCvXz8tX75c//nPf1SvXj2NHz9eu3fv1vLly3XDDTdURYwAAPiu/KmFvoxa6rzuM3DNNddozZo1lR0LAACoBud906Ft27Zp9+7dksrWEcTExFRaUAAAVDYeYeyZ18nA4cOHdffdd+vzzz9XgwYNJEk5OTn6y1/+orffflstWrSo7BgBAPAdawY88nrNwNChQ1VSUqLdu3fr+PHjOn78uHbv3i2n06mhQ4dWRYwAAKAKeV0Z2LBhgzZv3qz27du7trVv315/+9vfdM0111RqcAAAVBpfFwGygPBXkZGR57y5kMPhUERERKUEBQBAZbMYZcOX42srr9sEU6dO1SOPPKJt27a5tm3btk2PPfaY/vrXv1ZqcAAAVBoeVORRhSoDDRs2lMXya3mkoKBAPXr0UJ06ZYeXlpaqTp06euCBB9S/f/8qCRQAAFSNCiUDr776ahWHAQBAFWPNgEcVSgYSExOrOg4AAKoWlxZ6dN43HZKkwsJCFRcXu22z2Ww+BQQAAC4srxcQFhQUaMSIEQoNDVW9evXUsGFDtwEAQI3EAkKPvE4GxowZo3Xr1mnu3LkKDAzU/PnzNXHiREVERGjhwoVVESMAAL4jGfDI6zbB8uXLtXDhQvXq1UuDBw/WNddco7Zt26pVq1ZatGiREhISqiJOAABQRbyuDBw/flxt2rSRVLY+4Pjx45Kkq6++Whs3bqzc6AAAqCw8wtgjr5OBNm3a6ODBg5KkDh06aOnSpZLKKgblDy4CAKCmKb8DoS+jtvI6GRg8eLC+/vprSdKTTz6p2bNnKygoSKNGjdITTzxR6QECAICq5fWagVGjRrn+Oy4uTnv27FFaWpratm2ryy67rFKDAwCg0nCfAY98us+AJLVq1UqtWrWqjFgAAEA1qFAyMHPmzAqf8NFHHz3vYAAAqCoW+fjUwkqLpOapUDIwffr0Cp3MYrGQDAAAIGnu3LmaO3euDh06JEnq1KmTxo8fr5tuuklS2V18H3/8cb399tsqKipSfHy85syZo7CwMNc50tPT9dBDD+nTTz9V/fr1lZiYqMmTJ7seFChJ69evV3Jysnbt2qXIyEg988wzuv/++72KtULJQPnVAzXV7e06q44loLrDAKrEnB82VHcIQJXJP+lUt04X6M0u8IOKWrRooZdffll/+tOfZBiGFixYoH79+mn79u3q1KmTRo0apZUrV+qdd96R3W7XiBEjNGDAAH3++eeSJIfDob59+yo8PFybN2/W0aNHNWjQIAUEBOill16SVPb93LdvXw0fPlyLFi3S2rVrNXToUDVr1kzx8fEVjtViGMZFuyQiLy9PdrtdvdSPZAC11pwfNlV3CECVKUsGspWbm1tlz7Yp/65oNflF+QUFnfd5nIWF+mHc0z7F2qhRI02dOlV33HGHmjZtqsWLF+uOO+6QJO3Zs0cdO3ZUamqqrrzySn388ce65ZZbdOTIEVe1YN68eRo7dqyOHTsmq9WqsWPHauXKlfrmm29c73HXXXcpJydHq1atqnBcXl9aCACAmeXl5bmNoqKiPzzG4XDo7bffVkFBgWJjY5WWlqaSkhLFxcW55nTo0EEtW7ZUamqqJCk1NVWdO3d2axvEx8crLy9Pu3btcs058xzlc8rPUVEkAwAAc6ikZxNERkbKbre7xuTJkz2+5c6dO1W/fn0FBgZq+PDh+uCDDxQdHa3MzExZrdazbtYXFhamzMxMSVJmZqZbIlC+v3zf783Jy8vT6dOnK/xP4/OlhQAAXAx8vYtg+bEZGRlubYLAwECPx7Rv3147duxQbm6u3n33XSUmJmrDhpq3DohkAAAAL9hstgqvGbBarWrbtq0kKSYmRlu3btWMGTP0v//7vyouLlZOTo5bdSArK0vh4eGSpPDwcH355Zdu58vKynLtK//f8m1nzrHZbAoODq7wZzqvNsFnn32me++9V7Gxsfrxxx8lSf/4xz+0aRMLnQAANVQNeISx0+lUUVGRYmJiFBAQoLVr17r27d27V+np6YqNjZUkxcbGaufOncrOznbNWbNmjWw2m6Kjo11zzjxH+Zzyc1SU18nAe++9p/j4eAUHB2v79u2uhRO5ubmuSx0AAKhxLnAyMG7cOG3cuFGHDh3Szp07NW7cOK1fv14JCQmy2+0aMmSIkpOT9emnnyotLU2DBw9WbGysrrzySklSnz59FB0drfvuu09ff/21Vq9erWeeeUZJSUmu1sTw4cN14MABjRkzRnv27NGcOXO0dOlSt0cHVITXycALL7ygefPm6fXXX1dAwK+X81111VX66quvvD0dAAC1UnZ2tgYNGqT27dvr+uuv19atW7V69WrdcMMNkspu6HfLLbdo4MCB6tmzp8LDw/X++++7jvf399eKFSvk7++v2NhY3XvvvRo0aJAmTZrkmhMVFaWVK1dqzZo16tKli6ZNm6b58+d7dY8B6TzWDOzdu1c9e/Y8a7vdbldOTo63pwMA4IKorAWEFfXGG2/87v6goCDNnj1bs2fP9jinVatW+uijj373PL169dL27du9C+43vK4MhIeHa9++fWdt37Rpk9q0aeNTMAAAVJnyOxD6Mmopr5OBYcOG6bHHHtOWLVtksVh05MgRLVq0SKNHj9ZDDz1UFTECAOC7GrCAsKbyuk3w5JNPyul06vrrr9epU6fUs2dPBQYGavTo0XrkkUeqIkYAAFCFvE4GLBaLnn76aT3xxBPat2+f8vPzFR0drfr161dFfAAAVIoLvWbgYnLeNx2yWq2u6xwBAKjxfC31kwz8qnfv3rJYPC+iWLdunU8BAQCAC8vrZKBr165ur0tKSrRjxw598803SkxMrKy4AACoXD62CagMnGH69Onn3D5hwgTl5+f7HBAAAFWCNoFHlfYI43vvvVdvvvlmZZ0OAABcIJX21MLU1FQFBQVV1ukAAKhcVAY88joZGDBggNtrwzB09OhRbdu2Tc8++2ylBQYAQGXi0kLPvE4G7Ha722s/Pz+1b99ekyZNUp8+fSotMAAAcGF4lQw4HA4NHjxYnTt3VsOGDasqJgAAcAF5tYDQ399fffr04emEAICLD88m8MjrqwkuvfRSHThwoCpiAQCgypSvGfBl1FZeJwMvvPCCRo8erRUrVujo0aPKy8tzGwAA4OJS4TUDkyZN0uOPP66bb75ZknTbbbe53ZbYMAxZLBY5HI7KjxIAgMpQi/+690WFk4GJEydq+PDh+vTTT6syHgAAqgb3GfCowsmAYZT9K1x77bVVFgwAALjwvLq08PeeVggAQE3GTYc88yoZaNeu3R8mBMePH/cpIAAAqgRtAo+8SgYmTpx41h0IAQDAxc2rZOCuu+5SaGhoVcUCAECVoU3gWYWTAdYLAAAuarQJPKrwTYfKryYAAAC1S4UrA06nsyrjAACgalEZ8MjrRxgDAHAxYs2AZyQDAABzoDLgkdcPKgIAALULlQEAgDlQGfCIZAAAYAqsGfCMNgEAACZHZQAAYA60CTwiGQAAmAJtAs9oEwAAYHJUBgAA5kCbwCOSAQCAOZAMeESbAAAAk6MyAAAwBcsvw5fjayuSAQCAOdAm8IhkAABgClxa6BlrBgAAMDkqAwAAc6BN4BHJAADAPGrxF7ovaBMAAGByVAYAAKbAAkLPSAYAAObAmgGPaBMAAGByVAYAAKZAm8AzkgEAgDnQJvCINgEAACZHMgAAMIXyNoEvwxuTJ0/WFVdcoZCQEIWGhqp///7au3ev25zCwkIlJSWpcePGql+/vgYOHKisrCy3Oenp6erbt6/q1q2r0NBQPfHEEyotLXWbs379enXr1k2BgYFq27atUlJSvIqVZAAAYA5GJQwvbNiwQUlJSfriiy+0Zs0alZSUqE+fPiooKHDNGTVqlJYvX6533nlHGzZs0JEjRzRgwADXfofDob59+6q4uFibN2/WggULlJKSovHjx7vmHDx4UH379lXv3r21Y8cOjRw5UkOHDtXq1asrHKvFMIyLtguSl5cnu92uXuqnOpaA6g4HqBJzfthU3SEAVSb/pFPdOmUrNzdXNputSt6j/Lvisvtfkr816LzP4ygu1H9TnjrvWI8dO6bQ0FBt2LBBPXv2VG5urpo2barFixfrjjvukCTt2bNHHTt2VGpqqq688kp9/PHHuuWWW3TkyBGFhYVJkubNm6exY8fq2LFjslqtGjt2rFauXKlvvvnG9V533XWXcnJytGrVqgrFRmUAAAAv5OXluY2ioqIKHZebmytJatSokSQpLS1NJSUliouLc83p0KGDWrZsqdTUVElSamqqOnfu7EoEJCk+Pl55eXnatWuXa86Z5yifU36OiiAZAACYQmWtGYiMjJTdbneNyZMn/+F7O51OjRw5UldddZUuvfRSSVJmZqasVqsaNGjgNjcsLEyZmZmuOWcmAuX7y/f93py8vDydPn26Qv82XFoIADCHSrq0MCMjw61NEBgY+IeHJiUl6ZtvvtGmTTWz7UdlAAAAL9hsNrfxR8nAiBEjtGLFCn366adq0aKFa3t4eLiKi4uVk5PjNj8rK0vh4eGuOb+9uqD89R/NsdlsCg4OrtBnIhkAAJiCxTB8Ht4wDEMjRozQBx98oHXr1ikqKsptf0xMjAICArR27VrXtr179yo9PV2xsbGSpNjYWO3cuVPZ2dmuOWvWrJHNZlN0dLRrzpnnKJ9Tfo6KoE0AADCHC3wHwqSkJC1evFj//ve/FRIS4urx2+12BQcHy263a8iQIUpOTlajRo1ks9n0yCOPKDY2VldeeaUkqU+fPoqOjtZ9992nKVOmKDMzU88884ySkpJcFYnhw4dr1qxZGjNmjB544AGtW7dOS5cu1cqVKyscK5UBAACqwNy5c5Wbm6tevXqpWbNmrrFkyRLXnOnTp+uWW27RwIED1bNnT4WHh+v999937ff399eKFSvk7++v2NhY3XvvvRo0aJAmTZrkmhMVFaWVK1dqzZo16tKli6ZNm6b58+crPj6+wrFSGQAAmMKFflBRRW7jExQUpNmzZ2v27Nke57Rq1UofffTR756nV69e2r59u3cBnoFkAABgDjyoyCPaBAAAmByVAQCAKVzoNsHFhGQAAGAOtAk8IhkAAJgClQHPWDMAAIDJURkAAJgDbQKPSAYAAKZRm0v9vqBNAACAyVEZAACYg2GUDV+Or6VIBgAApsDVBJ7RJgAAwOSoDAAAzIGrCTwiGQAAmILFWTZ8Ob62IhnAWRZs+VbhkSVnbf8wpbFmP9VCktQxpkD3j81Uh26n5HBIB3YF66l72qi4kM4TqteK6S310ast3baFXXJKz637yvX6QFqIPpzaSod2hMjP31CL6AKN+McuWYPKfttnHQjSBy9Faf82mxwlFjXvUKBbHk9X+7/kus6xZ5Ndy6e10pG9dRVY16keA7N12xOH5M9vVVyE+LHFWR69qZ38/H+th7XuUKiXlxzQZ8sbSCpLBF5cdEBvzwrVnGeay+GQ2kQXyqjFWTMuLs3aFejRRd+4XvvX+fXn+UBaiGYldlL8w4d156QD8vc3dHh3PVnOWB0294FOatr6tB77105Zg5xa90aE5j4QrYkbt8keWqLD39bTnMGddOOIDCVO/045mVb966m2cjqkgc8cupAfFd6gTeBRtf4Zt3HjRt16662KiIiQxWLRsmXLqjMc/CL3eB2dOBbgGj3i8nTkoFX/Ta0nSfp/E45o2RtNtHRWmH74LkiH9wdp4/IGKimmKoCawb+OIXtoiWvUb1Tq2vfu81Hqff8RxT98WBHtTinsktOKueUnBQSW/abPP15H2QeDFf/wYbXoeEqhUYXq/+QPKj7tr6Pf1ZUkpa1ooogOBbr5sQyFti5UuyvzdPtTh7RxYTMV5vtXy2fGHyu/msCXUVtV62/vgoICdenSRbNnz67OMPA76gQ4dd3AE1r9diNJFtkbl6hjzCnl/FxH0z/8Xm9/vUtT39unTn/Or+5QAZfsg8Ead8UVevbq7nrr0XY6/mOgJOnkTwE6tN2m+o1LNPX2yzQ25s965c7O2rfV5jq2XsNShV1ySlveC1XRKT85SqXPFoUrpEmxWnYu+zkvLfJTQKB7Kcwa5FBJkb/Sd9a/cB8U3im/z4Avo5aq1mTgpptu0gsvvKDbb7+9OsPA7/jLjXmqb3Pok6WNJEnNWhVLku5LztLHixrr6YQo7dsZrJeXHFBEVFF1hgpIkqK6ntSgad8paeEu3f3iPv2UEaRX/qezCvP99VN6kCTpo1db6uq7MzViwS61vDRfM++5VNkHy/ZZLNKji75Rxq56So6O1WPtrtK6+REasWCX6todkqSO157QgTSbtv67iZwOKSfTqo9mlK1TyM0OqJ4PDvjgolozUFRUpKKiX79w8vLyqjEac4i/+2dt/dSm41llv+D8fkkfP/pnY32ypCxB2P9NXXW9Ol/xdx3XW5ObVVeogCSpU+8Tv77oeEqtu57UM1ddobQVTRTe9pQk6eqETMXemS1Jirz0oPZ83kCbl4ap/9gfZBjSkmcvUUjjEiW/818FBDn1+dvhmjskWmM/3CF7WImie+ZowFMH9a+n22rBqPaqY3XqpkcztO9Luyx0y2osbjrk2UX1Yzt58mTZ7XbXiIyMrO6QarXQ5sW6/Jp8rVrcyLXt56yy/PGH74Lc5mbsC1Ro8+ILGh9QEXXtDoVGndaxH4JkDy37GS1PCsqFtz2lE7+0EvZ+btfOtY30wKy9uuSKk2rZuUB3v7hfAUFOffFemOuY64cd0bSdX+iF1K2asmOLLrvhZ0lSk5aFF+iTwWtGJYxa6qJKBsaNG6fc3FzXyMjIqO6QarU+dx1Xzk91tOU/v/ZTszKs+uloHbW4xP0XXvM2Rco+bL3QIQJ/qLDATz/9kgg0jiySPaxI2QeC3eZkHwhWoxZlVcfiwrIFgBY/99/8Fj/jrCtmLBapQVixrEFObfuwqRpGFKrlpayfwcXnomoTBAYGKjAwsLrDMAWLxVCf/z2u/7zTUE6H5cw9enduqO4bnakD3wbrwK5gxf3PcUVeUqQXhjXyeD7gQnnvhdbqHHdcjZsXKSfLqpXTW8rPX+p+2zFZLNIN/+9HrZjeUs07FqhFpwJteTdUWfuDNWzeHklSm255qmsv1cLkdrr5sQwFBDn0+b/C9XNGkC697tcWxJp5zRXd64QsftKOjxvrk7ktNGT2HvlxMUGNRZvAs4sqGcCFc3nPfIW1KNHqtxufte+D+U0VEOTU8IlHFNLAoQPfBmnc3W109AcSNVS/nMxAvfVIexXkBKh+oxJdckWenlj2tUIal11eeN2QIyop8tO7z7fRqZw6at6xQI8s2qWmrcqqXfUblWrEwl36cGorzbj7UjlKLWr2p1Ma/vputYgucL3PrvUNtWp2pEqLLGoeXaDhr+92X6+AmoenFnpkMYzq+3T5+fnat2+fJOnyyy/XK6+8ot69e6tRo0Zq2bLlHxxdtoDQbrerl/qpjoUVvKid5vywqbpDAKpM/kmnunXKVm5urmw22x8fcB7KvyuuvHmS6gQE/fEBHpSWFOqLj8ZXaazVpVorA9u2bVPv3r1dr5OTkyVJiYmJSklJqaaoAAC1EW0Cz6o1GejVq5eqsTABADATbkfs0UV1NQEAAKh8LCAEAJgCbQLPSAYAAObgNMqGL8fXUiQDAABzYM2AR6wZAADA5KgMAABMwSIf1wxUWiQ1D8kAAMAcuAOhR7QJAAAwOSoDAABT4NJCz0gGAADmwNUEHtEmAADA5KgMAABMwWIYsviwCNCXY2s6kgEAgDk4fxm+HF9L0SYAAMDkqAwAAEyBNoFnJAMAAHPgagKPSAYAAObAHQg9Ys0AAAAmR2UAAGAK3IHQM5IBAIA50CbwiDYBAAAmR2UAAGAKFmfZ8OX42opkAABgDrQJPKJNAACAyZEMAADMwaiE4YWNGzfq1ltvVUREhCwWi5YtW+YejmFo/PjxatasmYKDgxUXF6fvv//ebc7x48eVkJAgm82mBg0aaMiQIcrPz3eb89///lfXXHONgoKCFBkZqSlTpngXqEgGAAAmUX47Yl+GNwoKCtSlSxfNnj37nPunTJmimTNnat68edqyZYvq1aun+Ph4FRYWuuYkJCRo165dWrNmjVasWKGNGzfqwQcfdO3Py8tTnz591KpVK6WlpWnq1KmaMGGCXnvtNa9iZc0AAABV4KabbtJNN910zn2GYejVV1/VM888o379+kmSFi5cqLCwMC1btkx33XWXdu/erVWrVmnr1q3q3r27JOlvf/ubbr75Zv31r39VRESEFi1apOLiYr355puyWq3q1KmTduzYoVdeecUtafgjVAYAAOZQvoDQl6Gyv8bPHEVFRV6HcvDgQWVmZiouLs61zW63q0ePHkpNTZUkpaamqkGDBq5EQJLi4uLk5+enLVu2uOb07NlTVqvVNSc+Pl579+7ViRMnKhwPyQAAwBwMSU4fxi9dgsjISNntdteYPHmy16FkZmZKksLCwty2h4WFufZlZmYqNDTUbX+dOnXUqFEjtznnOseZ71ERtAkAAKZQWY8wzsjIkM1mc20PDAz0ObbqRmUAAAAv2Gw2t3E+yUB4eLgkKSsry217VlaWa194eLiys7Pd9peWlur48eNuc851jjPfoyJIBgAA5mDIxzUDlRdKVFSUwsPDtXbtWte2vLw8bdmyRbGxsZKk2NhY5eTkKC0tzTVn3bp1cjqd6tGjh2vOxo0bVVJS4pqzZs0atW/fXg0bNqxwPCQDAABzqKQFhBWVn5+vHTt2aMeOHZLKFg3u2LFD6enpslgsGjlypF544QV9+OGH2rlzpwYNGqSIiAj1799fktSxY0fdeOONGjZsmL788kt9/vnnGjFihO666y5FRERIku655x5ZrVYNGTJEu3bt0pIlSzRjxgwlJyd7FStrBgAAqALbtm1T7969Xa/Lv6ATExOVkpKiMWPGqKCgQA8++KBycnJ09dVXa9WqVQoKCnIds2jRIo0YMULXX3+9/Pz8NHDgQM2cOdO1326365NPPlFSUpJiYmLUpEkTjR8/3qvLCiXJYhgX782W8/LyZLfb1Uv9VMcSUN3hAFVizg+bqjsEoMrkn3SqW6ds5ebmui3Kq0zl3xXXdR6rOv7nv9iv1FGkdTv/r0pjrS5UBgAAplBZVxPURqwZAADA5KgMAADMgUcYe0QyAAAwB5IBj2gTAABgclQGAADmQGXAI5IBAIA5OCVZfDy+liIZAACYApcWesaaAQAATI7KAADAHFgz4BHJAADAHJyGZPHhC91Ze5MB2gQAAJgclQEAgDnQJvCIZAAAYBI+JgOqvckAbQIAAEyOygAAwBxoE3hEMgAAMAenIZ9K/VxNAAAAaisqAwAAczCcZcOX42spkgEAgDmwZsAjkgEAgDmwZsAj1gwAAGByVAYAAOZAm8AjkgEAgDkY8jEZqLRIahzaBAAAmByVAQCAOdAm8IhkAABgDk6nJB/uFeCsvfcZoE0AAIDJURkAAJgDbQKPSAYAAOZAMuARbQIAAEyOygAAwBy4HbFHJAMAAFMwDKcMH5486MuxNR3JAADAHAzDt7/uWTMAAABqKyoDAABzMHxcM1CLKwMkAwAAc3A6JYsPff9avGaANgEAACZHZQAAYA60CTwiGQAAmILhdMrwoU1Qmy8tpE0AAIDJURkAAJgDbQKPSAYAAObgNCQLycC50CYAAMDkqAwAAMzBMCT5cp+B2lsZIBkAAJiC4TRk+NAmMEgGAAC4yBlO+VYZ4NJCAABQS1EZAACYAm0Cz0gGAADmQJvAo4s6GSjP0kpV4tN9JICaLP9k7f0FBOTnl/18X4i/un39rihVSeUFU8Nc1MnAyZMnJUmb9FE1RwJUnW6dqjsCoOqdPHlSdru9Ss5ttVoVHh6uTZm+f1eEh4fLarVWQlQ1i8W4iJsgTqdTR44cUUhIiCwWS3WHYwp5eXmKjIxURkaGbDZbdYcDVCp+vi88wzB08uRJRUREyM+v6ta0FxYWqri42OfzWK1WBQUFVUJENctFXRnw8/NTixYtqjsMU7LZbPyyRK3Fz/eFVVUVgTMFBQXVyi/xysKlhQAAmBzJAAAAJkcyAK8EBgbqueeeU2BgYHWHAlQ6fr5hVhf1AkIAAOA7KgMAAJgcyQAAACZHMgAAgMmRDAAAYHIkA6iw2bNnq3Xr1goKClKPHj305ZdfVndIQKXYuHGjbr31VkVERMhisWjZsmXVHRJwQZEMoEKWLFmi5ORkPffcc/rqq6/UpUsXxcfHKzs7u7pDA3xWUFCgLl26aPbs2dUdClAtuLQQFdKjRw9dccUVmjVrlqSy50JERkbqkUce0ZNPPlnN0QGVx2Kx6IMPPlD//v2rOxTggqEygD9UXFystLQ0xcXFubb5+fkpLi5Oqamp1RgZAKAykAzgD/30009yOBwKCwtz2x4WFqbMzMxqigoAUFlIBgAAMDmSAfyhJk2ayN/fX1lZWW7bs7KyFB4eXk1RAQAqC8kA/pDValVMTIzWrl3r2uZ0OrV27VrFxsZWY2QAgMpQp7oDwMUhOTlZiYmJ6t69u/785z/r1VdfVUFBgQYPHlzdoQE+y8/P1759+1yvDx48qB07dqhRo0Zq2bJlNUYGXBhcWogKmzVrlqZOnarMzEx17dpVM2fOVI8ePao7LMBn69evV+/evc/anpiYqJSUlAsfEHCBkQwAAGByrBkAAMDkSAYAADA5kgEAAEyOZAAAAJMjGQAAwORIBgAAMDmSAQAATI5kAPDR/fffr/79+7te9+rVSyNHjrzgcaxfv14Wi0U5OTke51gsFi1btqzC55wwYYK6du3qU1yHDh2SxWLRjh07fDoPgKpDMoBa6f7775fFYpHFYpHValXbtm01adIklZaWVvl7v//++3r++ecrNLciX+AAUNV4NgFqrRtvvFFvvfWWioqK9NFHHykpKUkBAQEaN27cWXOLi4tltVor5X0bNWpUKecBgAuFygBqrcDAQIWHh6tVq1Z66KGHFBcXpw8//FDSr6X9F198UREREWrfvr0kKSMjQ3feeacaNGigRo0aqV+/fjp06JDrnA6HQ8nJyWrQoIEaN26sMWPG6Ld39P5tm6CoqEhjx45VZGSkAgMD1bZtW73xxhs6dOiQ6374DRs2lMVi0f333y+p7KmQkydPVlRUlIKDg9WlSxe9++67bu/z0UcfqV27dgoODlbv3r3d4qyosWPHql27dqpbt67atGmjZ599ViUlJWfN+/vf/67IyEjVrVtXd955p3Jzc932z58/Xx07dlRQUJA6dOigOXPmeB0LgOpDMgDTCA4OVnFxsev12rVrtXfvXq1Zs0YrVqxQSUmJ4uPjFRISos8++0yff/656tevrxtvvNF13LRp05SSkqI333xTmzZt0vHjx/XBBx/87vsOGjRI//rXvzRz5kzt3r1bf//731W/fn1FRkbqvffekyTt3btXR48e1YwZMyRJkydP1sKFCzVv3jzt2rVLo0aN0r333qsNGzZIKktaBgwYoFtvvVU7duzQ0KFD9eSTT3r9bxISEqKUlBR9++23mjFjhl5//XVNnz7dbc6+ffu0dOlSLV++XKtWrdL27dv18MMPu/YvWrRI48eP14svvqjdu3frpZde0rPPPqsFCxZ4HQ+AamIAtVBiYqLRr18/wzAMw+l0GmvWrDECAwON0aNHu/aHhYUZRUVFrmP+8Y9/GO3btzecTqdrW1FRkREcHGysXr3aMAzDaNasmTFlyhTX/pKSEqNFixau9zIMw7j22muNxx57zDAMw9i7d68hyVizZs054/z0008NScaJEydc2woLC426desamzdvdps7ZMgQ4+677zYMwzDGjRtnREdHu+0fO3bsWef6LUnGBx984HH/1KlTjZiYGNfr5557zvD39zcOHz7s2vbxxx8bfn5+xtGjRw3DMIxLLrnEWLx4sdt5nn/+eSM2NtYwDMM4ePCgIcnYvn27x/cFUL1YM4Baa8WKFapfv75KSkrkdDp1zz33aMKECa79nTt3dlsn8PXXX2vfvn0KCQlxO09hYaH279+v3NxcHT161O2xzXXq1FH37t3PahWU27Fjh/z9/XXttddWOO59+/bp1KlTuuGGG9y2FxcX6/LLL5ck7d69+6zHR8fGxlb4PcotWbJEM2fO1P79+5Wfn6/S0lLZbDa3OS1btlTz5s3d3sfpdGrv3r0KCQnR/v37NWTIEA0bNsw1p7S0VHa73et4AFQPkgHUWr1799bcuXNltVoVERGhOnXcf9zr1avn9jo/P18xMTFatGjRWedq2rTpecUQHBzs9TH5+fmSpJUrV7p9CUtl6yAqS2pqqhISEjRx4kTFx8fLbrfr7bff1rRp07yO9fXXXz8rOfH396+0WAFULZIB1Fr16tVT27ZtKzy/W7duWrJkiUJDQ8/667hcs2bNtGXLFvXs2VNS2V/AaWlp6tat2znnd+7cWU6nUxs2bFBcXNxZ+8srEw6Hw7UtOjpagYGBSk9P91hR6Nixo2sxZLkvvvjijz/kGTZv3qxWrVrp6aefdm374YcfzpqXnp6uI0eOKCIiwvU+fn5+at++vcLCwhQREaEDBw4oISHBq/cHUHOwgBD4RUJCgpo0aaJ+/frps88+08GDB7V+/Xo9+uijOnz4sCTpscce08svv6xly5Zpz549evjhh3/3HgGtW7dWYmKiHnjgAS1btsx1zqVLl0qSWrVqJYvFohUrVujYsWPKz89XSEiIRo8erVGjRmnBggXav3+/vvrqK/3tb39zLcobPny4vv/+ez3xxBPau3evFi9erJSUFK8+75/+9Celp6fr7bff1v79+zVz5sxzLoYMCgpSYmKivv76a3322Wd69NFHdeeddyo8PFySNHHiRE2ePFkzZ87Ud999p507d+qtt97SK6+84lU8AKoPyQDwi7p162rjxo1q2bKlBgwYoI4dO2rIkCEqLCx0VQoef/xx3XfffUpMTFRsbKxCQkJ0++23/+55586dqzvuuEMPP/ywOnTooGHDhqmgoECS1Lx5c02cOFFPPvmkwsLCNGLECEnS888/r2effVaTJ09Wx44ddeONN2rlypWKioqSVNbHf++997Rs2TJ16dJF8+bN00svveTV573ttts0atQojRgxQl27dtXmzZv17LPPnjWvbdu2GjBggG6++Wb16dNHl112mdulg0OHDtX8+fP11ltvqXPnzrr22muVkpLiihVAzWcxPK18AgAApkBlAAAAkyMZAADA5EgGAAAwOZIBAABMjmQAAACTIxkAAMDkSAYAADA5kgEAAEyOZAAAAJMjGQAAwORIBgAAMDmSAQAATO7/A0dT7GAnbP9JAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Look at parameters used by our current random forest\n",
        "rf.get_params()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YRUxb3KICF9h",
        "outputId": "ed32d229-05de-4941-96cc-54b8b9d40c03"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'bootstrap': True,\n",
              " 'ccp_alpha': 0.0,\n",
              " 'class_weight': None,\n",
              " 'criterion': 'gini',\n",
              " 'max_depth': None,\n",
              " 'max_features': 'sqrt',\n",
              " 'max_leaf_nodes': None,\n",
              " 'max_samples': None,\n",
              " 'min_impurity_decrease': 0.0,\n",
              " 'min_samples_leaf': 1,\n",
              " 'min_samples_split': 2,\n",
              " 'min_weight_fraction_leaf': 0.0,\n",
              " 'n_estimators': 100,\n",
              " 'n_jobs': None,\n",
              " 'oob_score': False,\n",
              " 'random_state': None,\n",
              " 'verbose': 0,\n",
              " 'warm_start': False}"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Step 3: Random Forest Hyperparameter Tuning"
      ],
      "metadata": {
        "id": "6LBfkyPNCQIP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Random Forest: hyperparameter tuning\n",
        "\n",
        "# Number of trees in random forest\n",
        "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
        "# Number of features to consider at every split\n",
        "max_features = ['auto', 'sqrt']\n",
        "# Maximum number of levels in tree\n",
        "max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]\n",
        "max_depth.append(None)\n",
        "# Minimum number of samples required to split a node\n",
        "min_samples_split = [2, 5, 10]\n",
        "# Minimum number of samples required at each leaf node\n",
        "min_samples_leaf = [1, 2, 4]\n",
        "# Method of selecting samples for training each tree\n",
        "bootstrap = [True, False]\n",
        "\n",
        "# Create the random grid\n",
        "random_grid = {'n_estimators': n_estimators,\n",
        "               'max_features': max_features,\n",
        "               'max_depth': max_depth,\n",
        "               'min_samples_split': min_samples_split,\n",
        "               'min_samples_leaf': min_samples_leaf,\n",
        "               'bootstrap': bootstrap}\n",
        "print(random_grid)"
      ],
      "metadata": {
        "id": "o3Nh5aPVe40T",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a65f3985-2c10-4592-da95-cfb45666de40"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt'], 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None], 'min_samples_split': [2, 5, 10], 'min_samples_leaf': [1, 2, 4], 'bootstrap': [True, False]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Use the random grid to search for best hyperparameters\n",
        "\n",
        "# Keep track of time elapsed\n",
        "start = time.time()\n",
        "\n",
        "# Create a random forest classifier\n",
        "rf = RandomForestClassifier()\n",
        "\n",
        "# Use random search to find the best hyperparameters, using 3 fold cross validation\n",
        "# across 100 different combinations\n",
        "grid_search = RandomizedSearchCV(rf,\n",
        "                                 param_distributions = random_grid,\n",
        "                                 n_iter=20,\n",
        "                                 cv=3,\n",
        "                                 verbose=3)\n",
        "\n",
        "# Fit the random search object to the data\n",
        "grid_search.fit(X_train, y_train)\n",
        "\n",
        "total_time = time.time() - start\n",
        "print(\"Elapsed time: {:.3f}\".format(total_time))\n",
        "print(\"In hours & minutes: \\t\", str(datetime.timedelta(seconds = total_time)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iY4mWFV9Cr_m",
        "outputId": "f3eca0d4-ac37-4936-9ab9-3716842f65b9"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 20 candidates, totalling 60 fits\n",
            "[CV 1/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=200;, score=0.961 total time= 1.1min\n",
            "[CV 2/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=200;, score=0.956 total time= 1.1min\n",
            "[CV 3/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=200;, score=0.956 total time= 1.1min\n",
            "[CV 1/3] END bootstrap=False, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.976 total time= 3.0min\n",
            "[CV 2/3] END bootstrap=False, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.974 total time= 3.0min\n",
            "[CV 3/3] END bootstrap=False, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.973 total time= 3.0min\n",
            "[CV 1/3] END bootstrap=False, max_depth=90, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=200;, score=0.978 total time= 2.9min\n",
            "[CV 2/3] END bootstrap=False, max_depth=90, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=200;, score=0.977 total time= 2.9min\n",
            "[CV 3/3] END bootstrap=False, max_depth=90, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=200;, score=0.974 total time= 2.9min\n",
            "[CV 1/3] END bootstrap=True, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=1000;, score=0.978 total time= 9.4min\n",
            "[CV 2/3] END bootstrap=True, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=1000;, score=0.975 total time= 9.3min\n",
            "[CV 3/3] END bootstrap=True, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=1000;, score=0.975 total time= 9.4min\n",
            "[CV 1/3] END bootstrap=False, max_depth=110, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=2000;, score=0.975 total time=27.8min\n",
            "[CV 2/3] END bootstrap=False, max_depth=110, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=2000;, score=0.974 total time=27.7min\n",
            "[CV 3/3] END bootstrap=False, max_depth=110, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=2000;, score=0.973 total time=27.4min\n",
            "[CV 1/3] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1800;, score=0.975 total time=14.9min\n",
            "[CV 2/3] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1800;, score=0.974 total time=14.8min\n",
            "[CV 3/3] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1800;, score=0.973 total time=14.9min\n",
            "[CV 1/3] END bootstrap=False, max_depth=80, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=800;, score=0.976 total time=10.8min\n",
            "[CV 2/3] END bootstrap=False, max_depth=80, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=800;, score=0.974 total time=10.8min\n",
            "[CV 3/3] END bootstrap=False, max_depth=80, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=800;, score=0.974 total time=10.9min\n",
            "[CV 1/3] END bootstrap=True, max_depth=80, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=1400;, score=0.972 total time=10.7min\n",
            "[CV 3/3] END bootstrap=True, max_depth=80, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=1400;, score=0.968 total time=10.6min\n",
            "[CV 1/3] END bootstrap=True, max_depth=50, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=800;, score=0.966 total time= 5.3min\n",
            "[CV 2/3] END bootstrap=True, max_depth=50, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=800;, score=0.965 total time= 5.4min\n",
            "[CV 3/3] END bootstrap=True, max_depth=50, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=800;, score=0.964 total time= 5.4min\n",
            "[CV 1/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1200;, score=0.952 total time= 4.4min\n",
            "[CV 2/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1200;, score=0.946 total time= 4.4min\n",
            "[CV 3/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=2, min_samples_split=5, n_estimators=1200;, score=0.949 total time= 4.4min\n",
            "[CV 1/3] END bootstrap=False, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=400;, score=0.961 total time= 2.7min\n",
            "[CV 2/3] END bootstrap=False, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=400;, score=0.951 total time= 2.7min\n",
            "[CV 3/3] END bootstrap=False, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=400;, score=0.953 total time= 2.7min\n",
            "[CV 1/3] END bootstrap=True, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=1600;, score=0.960 total time= 6.6min\n",
            "[CV 2/3] END bootstrap=True, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=1600;, score=0.953 total time= 6.6min\n",
            "[CV 3/3] END bootstrap=True, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=1600;, score=0.952 total time= 6.6min\n",
            "[CV 1/3] END bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=200;, score=0.950 total time=  29.1s\n",
            "[CV 2/3] END bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=200;, score=0.946 total time=  29.1s\n",
            "[CV 3/3] END bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=200;, score=0.943 total time=  29.2s\n",
            "[CV 1/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.949 total time=  45.0s\n",
            "[CV 2/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.942 total time=  44.9s\n",
            "[CV 3/3] END bootstrap=False, max_depth=10, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=200;, score=0.939 total time=  44.8s\n",
            "[CV 1/3] END bootstrap=True, max_depth=20, max_features=auto, min_samples_leaf=4, min_samples_split=2, n_estimators=800;, score=0.957 total time= 3.3min\n",
            "[CV 2/3] END bootstrap=True, max_depth=20, max_features=auto, min_samples_leaf=4, min_samples_split=2, n_estimators=800;, score=0.950 total time= 3.2min\n",
            "[CV 3/3] END bootstrap=True, max_depth=20, max_features=auto, min_samples_leaf=4, min_samples_split=2, n_estimators=800;, score=0.950 total time= 3.3min\n",
            "[CV 1/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200;, score=0.966 total time= 6.6min\n",
            "[CV 2/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200;, score=0.961 total time= 6.5min\n",
            "[CV 3/3] END bootstrap=True, max_depth=30, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200;, score=0.961 total time= 6.6min\n",
            "[CV 1/3] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=10, n_estimators=2000;, score=0.961 total time=13.3min\n",
            "[CV 2/3] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=10, n_estimators=2000;, score=0.954 total time=13.2min\n",
            "[CV 3/3] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=10, n_estimators=2000;, score=0.953 total time=13.2min\n",
            "[CV 1/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=600;, score=0.973 total time= 4.3min\n",
            "[CV 2/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=600;, score=0.970 total time= 4.3min\n",
            "[CV 3/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=600;, score=0.968 total time= 4.3min\n",
            "[CV 1/3] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=1200;, score=0.971 total time= 8.9min\n",
            "[CV 2/3] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=1200;, score=0.968 total time= 8.9min\n",
            "[CV 3/3] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=4, min_samples_split=5, n_estimators=1200;, score=0.967 total time= 8.9min\n",
            "[CV 1/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=1400;, score=0.970 total time= 9.8min\n",
            "[CV 2/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=1400;, score=0.967 total time= 9.7min\n",
            "[CV 3/3] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=1400;, score=0.966 total time= 9.7min\n",
            "Elapsed time: 26663.649\n",
            "In hours & minutes: \t 7:24:23.648993\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.best_params_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "okXDfwtaMldC",
        "outputId": "44f993a9-901d-418e-f6ae-e0e87d9e0092"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'n_estimators': 200,\n",
              " 'min_samples_split': 10,\n",
              " 'min_samples_leaf': 1,\n",
              " 'max_features': 'sqrt',\n",
              " 'max_depth': 90,\n",
              " 'bootstrap': False}"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Tuned model metrics\n",
        "start = time.time()\n",
        "\n",
        "# Fit the tuned random forest model and generate predictions\n",
        "rf_tuned = RandomForestClassifier(n_estimators = 200, min_samples_split = 10,\n",
        "                                     min_samples_leaf =  1, max_features = 'sqrt',\n",
        "                                     max_depth = 90, bootstrap = False)\n",
        "rf_tuned.fit(X_train, y_train)\n",
        "y_pred = rf_tuned.predict(X_test)\n",
        "\n",
        "# generate metrics\n",
        "accuracy = accuracy_score(y_test, y_pred)\n",
        "f1 = f1_score(y_test, y_pred, average='macro')\n",
        "recall = recall_score(y_test, y_pred, average='macro')\n",
        "precision = precision_score(y_test, y_pred, average='macro')\n",
        "\n",
        "print(\"Tuned Random Forest Results:\")\n",
        "print(\"Test accuracy: {:.3f}\".format(accuracy))\n",
        "print(\"F1 Score: {:.3f}\".format(f1))\n",
        "print(\"Recall: {:.3f}\".format(recall))\n",
        "print(\"Precision: {:.3f}\".format(precision))\n",
        "\n",
        "total_time = time.time() - start\n",
        "print(\"Elapsed time: {:.3f} s\".format(total_time))\n",
        "print(\"In hours & minutes: \\t\", str(datetime.timedelta(seconds = total_time)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xzNJk0neFzvb",
        "outputId": "5eaa9b81-e032-4fa8-e7b3-c979be8edf5b"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tuned Random Forest Results:\n",
            "Test accuracy: 0.980\n",
            "F1 Score: 0.980\n",
            "Recall: 0.981\n",
            "Precision: 0.980\n",
            "Elapsed time: 279.844 s\n",
            "In hours & minutes: \t 0:04:39.843724\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Unfortunately, none of the Randomized Grid Search's random models acheived a higher score than the preliminary model (that used the default hyperparameters). The best model in the Randomized Grid Search achieved an accuracy of 0.980 compared to the accuracy of 0.982 of the preliminary model."
      ],
      "metadata": {
        "id": "aci1VRWR7Qxb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Random Forest Results"
      ],
      "metadata": {
        "id": "gxBbEVRb9AvU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Random Forest model metrics\n",
        "\n",
        "start = time.time()\n",
        "\n",
        "# Fit the random forest model and generate predictions\n",
        "rf_final = RandomForestClassifier()\n",
        "rf_final.fit(X_train, y_train)\n",
        "y_pred = rf_final.predict(X_test)\n",
        "\n",
        "# generate metrics\n",
        "accuracy = accuracy_score(y_test, y_pred)\n",
        "f1 = f1_score(y_test, y_pred, average='macro')\n",
        "recall = recall_score(y_test, y_pred, average='macro')\n",
        "precision = precision_score(y_test, y_pred, average='macro')\n",
        "\n",
        "print(\"Random Forest Results:\")\n",
        "print(\"Test accuracy: {:.3f}\".format(accuracy))\n",
        "print(\"F1 Score: {:.3f}\".format(f1))\n",
        "print(\"Recall: {:.3f}\".format(recall))\n",
        "print(\"Precision: {:.3f}\".format(precision))\n",
        "\n",
        "total_time = time.time() - start\n",
        "print(\"Elapsed time: {:.3f} s\".format(total_time))\n",
        "print(\"In hours & minutes: \\t\", str(datetime.timedelta(seconds = total_time)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x87am0TCA7Is",
        "outputId": "765e5d81-4290-4936-b244-e1988422cec1"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Random Forest Results:\n",
            "Test accuracy: 0.982\n",
            "F1 Score: 0.982\n",
            "Recall: 0.982\n",
            "Precision: 0.982\n",
            "Elapsed time: 97.344 s\n",
            "In hours & minutes: \t 0:01:37.343967\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My **Random Forest model acheived an accuracy, F1 score, and precision of 0.981, and a recall of 0.982**. This model's performance is roughly identical to the performance of the Random Forest model used in the article for the SpamAssassin dataset."
      ],
      "metadata": {
        "id": "98lQzp2_A7dQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Reproduction of Enron Spam dataset Solution:\n",
        "(Bag of words for the preprocessing, and a RNN (LSTM) model for the training)\n",
        "\n",
        "****NOTE**: the article did have a [Github link](https://github.com/suleka96/Email-Classification/blob/master/Enron_LSTM.py) to the code, so I just copy and pasted it and ensured that all the tensorflow methods were up to date"
      ],
      "metadata": {
        "id": "2oucUdmHOqgU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "Y-JGcR-T48mJ"
      },
      "outputs": [],
      "source": [
        "df2 = df"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Ensure the runtime is set to GPU (V100)\n",
        "import tensorflow as tf\n",
        "print(\"Tensorflow version \" + tf.__version__)\n",
        "\n",
        "device_name = tf.test.gpu_device_name()\n",
        "if device_name != '/device:GPU:0':\n",
        "  raise SystemError('GPU device not found')\n",
        "print('Found GPU at: {}'.format(device_name))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6I3zYp8-55JJ",
        "outputId": "984c6b11-ac83-4912-81e2-4f521864acbc"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tensorflow version 2.12.0\n",
            "Found GPU at: /device:GPU:0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Bag of Words model"
      ],
      "metadata": {
        "id": "b0M9Tuj6JnO2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Histogram of word frequency per email\n",
        "\n",
        "email_length = df2['Clean_Text'].str.split().apply(len)\n",
        "plt.hist(email_length, bins=50)\n",
        "plt.gca().set(title='Histogram of Word Frequency', ylabel='Frequency');"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        },
        "id": "RBWYsERr1EZ0",
        "outputId": "68149fab-5627-47f7-b5df-693b332600e2"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAAGzCAYAAACM3HvxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLwElEQVR4nO3deVwVdf///+cBPcf1gBugiYpLGq6FiZSaJpeotFj2SW0RTTMLK8W9ulyq69L0qvRK0/pVUp8y0z5pXZoa4VZKmiS5JblgVoqaCkdxQeH9+8Mvc3kEN0RH5XG/3eaWZ+Z1Zl7z5sB5NmdmjsMYYwQAAICrzsfuBgAAAIorghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGGCDWrVqqVevXna3ccObOHGiateuLV9fXzVr1szudi7ZmDFj5HA47G4DwBVEEAMuU3x8vBwOh9auXVvg8rZt26pRo0aXvZ2vv/5aY8aMuez1FBfffPONhg0bpjvvvFMzZszQP//5zwLrnnnmGfn4+OjgwYNe8w8ePCgfHx+5XC4dP37ca9mOHTvkcDj0wgsvXLH+L0WvXr3kcDgKnBYtWmR3ewDOo4TdDQDFUWpqqnx8Lu3/g77++mtNnTqVMHaRlixZIh8fH73//vtyOp3nrGvVqpWmTZumlStX6t5777Xmr1q1Sj4+Pjp58qTWrl2rVq1aWctWrlxpPfda4XK59N577+Wb37RpUxu6AXCxCGKADVwul90tXLKsrCyVLVvW7jYu2r59+1S6dOnzhjDpv2Hq+++/9wpiK1euVJMmTXTs2DF9//33XqHr+++/l4+Pj+64447L6vHUqVPKzc29YI8Xo0SJEnrssccuuv56+3kCNyo+mgRscPY5YidPntTYsWNVr149lSpVSpUqVVKrVq2UkJAg6fRHT1OnTpUkr4+d8mRlZWnw4MEKDg6Wy+VS/fr19a9//UvGGK/tHjt2TM8995wqV66s8uXL67777tOff/4ph8PhdaQt79ykzZs365FHHlGFChWsILJ+/Xr16tVLtWvXVqlSpRQUFKQnnnhCBw4c8NpW3jp+/fVXPfbYY/Lz81OVKlX097//XcYY/f7777r//vvldrsVFBSk119//aLG7tSpU3rllVdUp04duVwu1apVSy+88IJOnDhh1TgcDs2YMUNZWVnWWMXHxxe4vho1aig4ONg6ypVn5cqVuvPOO3XHHXcUuKxhw4by9/eXdDr09enTR4GBgSpVqpSaNm2qDz/80Os5O3fulMPh0L/+9S9NmjTJ6n/z5s2SToe722+/XaVKlVKdOnX0zjvvXNR4XIzz/Twl6eOPP1ZYWJhKly6tihUrqnv37vr999/zrefdd99VnTp1VLp0abVo0ULfffed2rZtq7Zt21o1eR/V79y50+u5y5Ytk8Ph0LJly7zmr169Wh07dpSfn5/KlCmju+66K9945/W/bds29erVS/7+/vLz81Pv3r119OjRfH1+/PHHatGihcqUKaMKFSqoTZs2+uabbyRJMTExqly5sk6ePJnveR06dFD9+vUvNJxAkeKIGFBEMjMz9ddff+WbX9Af/LONGTNG48aNU9++fdWiRQt5PB6tXbtWP/30k/72t7/pqaee0u7du5WQkKD//d//9XquMUb33Xefli5dqj59+qhZs2ZavHixhg4dqj///FNvvvmmVdurVy/Nnj1bjz/+uFq2bKnly5crOjr6nH39z//8j+rVq6d//vOfVqhLSEjQjh071Lt3bwUFBWnTpk169913tWnTJv3www/5Ti7v1q2bbrnlFo0fP14LFizQq6++qooVK+qdd97R3Xffrddee02ffPKJhgwZottvv11t2rQ571j17dtXH374oR566CENHjxYq1ev1rhx4/TLL79o7ty5kqT//d//1bvvvqs1a9ZYH9ed7+hVq1at9MUXX+jEiRNyuVzKzs7Wjz/+qKefflpHjx7VsGHDZIyRw+HQoUOHtHnzZvXv31/S6XDbtm1bbdu2TQMGDFBISIjmzJmjXr16KSMjQ88//7zXtmbMmKHjx4+rX79+crlcqlixojZs2KAOHTqoSpUqGjNmjE6dOqXRo0crMDDwvGNxtrNffyVLlpSfn5/1uKCf5z/+8Q/9/e9/18MPP6y+fftq//79euutt9SmTRutW7fOCpvvv/++nnrqKd1xxx0aOHCgduzYofvuu08VK1ZUcHDwJfWZZ8mSJerUqZPCwsI0evRo+fj4aMaMGbr77rv13XffqUWLFl71Dz/8sEJCQjRu3Dj99NNPeu+99xQQEKDXXnvNqhk7dqzGjBmjO+64Qy+//LKcTqdWr16tJUuWqEOHDnr88cf10UcfafHixbrnnnus56Wnp2vJkiUaPXp0ofYFKDQD4LLMmDHDSDrv1LBhQ6/n1KxZ08TExFiPmzZtaqKjo8+7ndjYWFPQr+y8efOMJPPqq696zX/ooYeMw+Ew27ZtM8YYk5ycbCSZgQMHetX16tXLSDKjR4+25o0ePdpIMj169Mi3vaNHj+ab9+mnnxpJZsWKFfnW0a9fP2veqVOnTPXq1Y3D4TDjx4+35h86dMiULl3aa0wKkpKSYiSZvn37es0fMmSIkWSWLFlizYuJiTFly5Y97/ryTJ061Ugy3333nTHGmKSkJCPJ/Pbbb2bz5s1Gktm0aZMxxpj58+cbSeaTTz4xxhgzadIkI8l8/PHH1vqys7NNRESEKVeunPF4PMYYY9LS0owk43a7zb59+7y236VLF1OqVCnz22+/WfM2b95sfH19C/yZny0mJqbA191dd91ljDn3z3Pnzp3G19fX/OMf//Cav2HDBlOiRAlrfnZ2tgkICDDNmjUzJ06csOreffddr+0Y89/fh7S0NK91Ll261EgyS5cuNcYYk5uba+rVq2eioqJMbm6uVXf06FETEhJi/va3v1nz8vp/4oknvNb5wAMPmEqVKlmPt27danx8fMwDDzxgcnJyvGrztpGTk2OqV69uunXr5rX8jTfeMA6Hw+zYscMAVxMfTQJFZOrUqUpISMg3NWnS5ILP9ff316ZNm7R169ZL3u7XX38tX19fPffcc17zBw8eLGOMFi5cKEnW1XPPPPOMV92zzz57znXnHfU5U+nSpa1/Hz9+XH/99ZdatmwpSfrpp5/y1fft29f6t6+vr5o3by5jjPr06WPN9/f3V/369bVjx45z9iKd3ldJiouL85o/ePBgSdKCBQvO+/xzOfM8Men0R4833XSTatSooQYNGqhixYrWx2Vnn6j/9ddfKygoSD169LDWV7JkST333HM6cuSIli9f7rWtrl27qkqVKtbjnJwcLV68WF26dFGNGjWs+bfccouioqIueh9KlSqV77V39se9Z/88v/jiC+Xm5urhhx/WX3/9ZU1BQUGqV6+eli5dKklau3at9u3bp/79+3udz9arVy+vI26XIiUlRVu3btUjjzyiAwcOWNvOyspS+/bttWLFCuXm5p63/9atW+vAgQPyeDySpHnz5ik3N1ejRo3KdzFM3pFaHx8fPfroo/rqq690+PBha/knn3yiO+64QyEhIYXaH6Cw+GgSKCItWrRQ8+bN882vUKFCgR9Znunll1/W/fffr5tvvlmNGjVSx44d9fjjj19UiPvtt99UrVo1lS9f3mv+LbfcYi3P+6+Pj0++N5q6deuec90FvSkdPHhQY8eO1axZs7Rv3z6vZZmZmfnqzwwXkuTn56dSpUqpcuXK+eaffZ7Z2fL24eyeg4KC5O/vb+3rpWrUqJH8/f29wtadd94p6fQbeEREhFauXKknn3xSK1euVHBwsLVfv/32m+rVq5fvjf/s8c9z9pju379fx44dU7169fL1Vb9+fSt8Xoivr68iIyPPW3P2trdu3SpjTIHblk4HSum/+3B2XcmSJVW7du2L6u9sef/TERMTc86azMxMVahQwXp89mspb9mhQ4fkdru1fft2+fj4KDQ09Lzb7tmzp1577TXNnTtXPXv2VGpqqpKTkzV9+vRC7QtwOQhiwDWgTZs22r59u7788kt98803eu+99/Tmm29q+vTpXkeUrrYzj37lefjhh7Vq1SoNHTpUzZo1U7ly5ZSbm6uOHTvmO4IhnQ4IFzNPUr6LC86lqG9y6uPjo4iICK1atUrGGK1cudLrHmF33HGHPvjgA+vcsS5duhR6WwWN6dVy9rZzc3PlcDi0cOHCAn8m5cqVu+RtnOtnk5OTk2/b0umb7p7rZrtnb/9yXzd5QkNDFRYWpo8//lg9e/bUxx9/LKfTqYcffviS1gMUBYIYcI2oWLGievfurd69e+vIkSNq06aNxowZYwWxc73B1axZU99++60OHz7sdVRsy5Yt1vK8/+bm5iotLc3ryMa2bdsuusdDhw4pMTFRY8eO1ahRo6z5hflItTDy9mHr1q3WESdJ2rt3rzIyMqx9LYxWrVpp4cKF+uqrr7Rv3z7riJh0Ooi9+OKL+vrrr3Xs2DGvKw5r1qyp9evXKzc31+uo2Nnjfy5VqlRR6dKlCxzD1NTUQu/PxahTp46MMQoJCdHNN998zrq8fdi6davuvvtua/7JkyeVlpbmda+yvKNUGRkZXus4+8hgnTp1JElut/uCR/IuVp06dZSbm6vNmzdf8JsUevbsqbi4OO3Zs0czZ85UdHS019E34GrhHDHgGnD2R3LlypVT3bp1vW7JkHfPp7Pf4Dp37qycnBxNmTLFa/6bb74ph8OhTp06SZJ1vtHbb7/tVffWW29ddJ95RyTOPgIxadKki17H5ejcuXOB23vjjTck6bxXgF5IXrh67bXXVKZMGa838hYtWqhEiRKaMGGCV21eT+np6frss8+seadOndJbb72lcuXK6a677jrvdn19fRUVFaV58+Zp165d1vxffvlFixcvLvT+XIwHH3xQvr6+Gjt2bL6fqTHGel02b95cVapU0fTp05WdnW3VxMfH53s95gWsFStWWPNycnL07rvvetWFhYWpTp06+te//qUjR47k623//v2XvD9dunSRj4+PXn755XxHZ8/evx49esjhcOj555/Xjh07LukebEBR4ogYcA0IDQ1V27ZtFRYWpooVK2rt2rX6/PPPNWDAAKsmLCxMkvTcc88pKipKvr6+6t69u+699161a9dOL774onbu3KmmTZvqm2++0ZdffqmBAwdab4xhYWHq2rWrJk2apAMHDli3r/j1118lXdzHfW63W23atNGECRN08uRJ3XTTTfrmm2+UlpZ2BUYlv6ZNmyomJkbvvvuuMjIydNddd2nNmjX68MMP1aVLF7Vr167Q627RooWcTqeSkpLUtm1blSjx3z+PZcqUUdOmTZWUlCR/f3+vr6zq16+f3nnnHfXq1UvJycmqVauWPv/8c61cuVKTJk3Kd+5eQcaOHatFixapdevWeuaZZ6wg17BhQ61fv77Q+3QhderU0auvvqqRI0dq586d6tKli8qXL6+0tDTNnTtX/fr105AhQ1SyZEm9+uqreuqpp3T33XerW7duSktL04wZM/KdI9awYUO1bNlSI0eO1MGDB1WxYkXNmjVLp06d8qrz8fHRe++9p06dOqlhw4bq3bu3brrpJv35559aunSp3G63/vOf/1zS/tStW1cvvviiXnnlFbVu3VoPPvigXC6XfvzxR1WrVk3jxo2zaqtUqaKOHTtqzpw58vf3v6wQD1wWm67WBG4YeZfr//jjjwUuv+uuuy54+4pXX33VtGjRwvj7+5vSpUubBg0amH/84x8mOzvbqjl16pR59tlnTZUqVYzD4fC6rcHhw4fNoEGDTLVq1UzJkiVNvXr1zMSJE71uC2CMMVlZWSY2NtZUrFjRlCtXznTp0sWkpqYaSV63k8i7XcD+/fvz7c8ff/xhHnjgAePv72/8/PzM//zP/5jdu3ef8xYYZ6/jXLeVKGicCnLy5EkzduxYExISYkqWLGmCg4PNyJEjzfHjxy9qO+cTERFhJJkXXngh37LnnnvOSDKdOnXKt2zv3r2md+/epnLlysbpdJrGjRubGTNmeNXk3b5i4sSJBW57+fLlJiwszDidTlO7dm0zffp0awwv5EL7er6fpzHG/N///Z9p1aqVKVu2rClbtqxp0KCBiY2NNampqV51b7/9tgkJCTEul8s0b97crFixwtx1111et68wxpjt27ebyMhI43K5TGBgoHnhhRdMQkKC1+0r8qxbt848+OCDplKlSsblcpmaNWuahx9+2CQmJl6w/3PdKuODDz4wt956q3G5XKZChQrmrrvuMgkJCfn2e/bs2flusQJcbQ5jLvEsRwA3lJSUFN166636+OOP9eijj9rdDq4zeXfVP/uO+deDL7/8Ul26dNGKFSvUunVru9tBMcU5YkAxcuzYsXzzJk2aJB8fnwve0R640fx//9//p9q1a19TX96O4odzxIBiZMKECUpOTla7du1UokQJLVy4UAsXLlS/fv0K/TU1wPVm1qxZWr9+vRYsWKDJkycX+e1QgEvBR5NAMZKQkKCxY8dq8+bNOnLkiGrUqKHHH39cL774otfJ6cDFuh4/mnQ4HCpXrpy6deum6dOn89qHrQhiAAAANuEcMQAAAJsQxAAAAGzCB+NFJDc3V7t371b58uU58RMAgOuEMUaHDx9WtWrVvL6m7GohiBWR3bt3c9UZAADXqd9//13Vq1e/6tsliBWRvK8x+f333+V2u23uBgAAXAyPx6Pg4OCL+jqyK4EgVkTyPo50u90EMQAArjN2nVbEyfoAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYpITdDeDi1Bqx4II1O8dHX4VOAABAUeGIGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2sTWITZs2TU2aNJHb7Zbb7VZERIQWLlxoLW/btq0cDofX1L9/f6917Nq1S9HR0SpTpowCAgI0dOhQnTp1yqtm2bJluu222+RyuVS3bl3Fx8fn62Xq1KmqVauWSpUqpfDwcK1Zs+aK7DMAAEAeW4NY9erVNX78eCUnJ2vt2rW6++67df/992vTpk1WzZNPPqk9e/ZY04QJE6xlOTk5io6OVnZ2tlatWqUPP/xQ8fHxGjVqlFWTlpam6OhotWvXTikpKRo4cKD69u2rxYsXWzWfffaZ4uLiNHr0aP30009q2rSpoqKitG/fvqszEAAAoFhyGGOM3U2cqWLFipo4caL69Omjtm3bqlmzZpo0aVKBtQsXLtQ999yj3bt3KzAwUJI0ffp0DR8+XPv375fT6dTw4cO1YMECbdy40Xpe9+7dlZGRoUWLFkmSwsPDdfvtt2vKlCmSpNzcXAUHB+vZZ5/ViBEjLqpvj8cjPz8/ZWZmyu12X8YIFKzWiAUXrNk5PrrItwsAwI3sSr9/X8g1c45YTk6OZs2apaysLEVERFjzP/nkE1WuXFmNGjXSyJEjdfToUWtZUlKSGjdubIUwSYqKipLH47GOqiUlJSkyMtJrW1FRUUpKSpIkZWdnKzk52avGx8dHkZGRVk1BTpw4IY/H4zUBAABcihJ2N7BhwwZFRETo+PHjKleunObOnavQ0FBJ0iOPPKKaNWuqWrVqWr9+vYYPH67U1FR98cUXkqT09HSvECbJepyenn7eGo/Ho2PHjunQoUPKyckpsGbLli3n7HvcuHEaO3bs5e08AAAo1mwPYvXr11dKSooyMzP1+eefKyYmRsuXL1doaKj69etn1TVu3FhVq1ZV+/bttX37dtWpU8fGrqWRI0cqLi7OeuzxeBQcHGxjRwAA4HpjexBzOp2qW7euJCksLEw//vijJk+erHfeeSdfbXh4uCRp27ZtqlOnjoKCgvJd3bh3715JUlBQkPXfvHln1rjdbpUuXVq+vr7y9fUtsCZvHQVxuVxyuVyXuLcAAAD/dc2cI5YnNzdXJ06cKHBZSkqKJKlq1aqSpIiICG3YsMHr6saEhAS53W7r482IiAglJiZ6rSchIcE6D83pdCosLMyrJjc3V4mJiV7nqgEAABQ1W4+IjRw5Up06dVKNGjV0+PBhzZw5U8uWLdPixYu1fft2zZw5U507d1alSpW0fv16DRo0SG3atFGTJk0kSR06dFBoaKgef/xxTZgwQenp6XrppZcUGxtrHa3q37+/pkyZomHDhumJJ57QkiVLNHv2bC1Y8N+rEOPi4hQTE6PmzZurRYsWmjRpkrKystS7d29bxgUAABQPtgaxffv2qWfPntqzZ4/8/PzUpEkTLV68WH/729/0+++/69tvv7VCUXBwsLp27aqXXnrJer6vr6/mz5+vp59+WhERESpbtqxiYmL08ssvWzUhISFasGCBBg0apMmTJ6t69ep67733FBUVZdV069ZN+/fv16hRo5Senq5mzZpp0aJF+U7gBwAAKErX3H3ErlfcRwwAgOsP9xEDAAAopghiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE1uD2LRp09SkSRO53W653W5FRERo4cKF1vLjx48rNjZWlSpVUrly5dS1a1ft3bvXax27du1SdHS0ypQpo4CAAA0dOlSnTp3yqlm2bJluu+02uVwu1a1bV/Hx8fl6mTp1qmrVqqVSpUopPDxca9asuSL7DAAAkMfWIFa9enWNHz9eycnJWrt2re6++27df//92rRpkyRp0KBB+s9//qM5c+Zo+fLl2r17tx588EHr+Tk5OYqOjlZ2drZWrVqlDz/8UPHx8Ro1apRVk5aWpujoaLVr104pKSkaOHCg+vbtq8WLF1s1n332meLi4jR69Gj99NNPatq0qaKiorRv376rNxgAAKDYcRhjjN1NnKlixYqaOHGiHnroIVWpUkUzZ87UQw89JEnasmWLbrnlFiUlJally5ZauHCh7rnnHu3evVuBgYGSpOnTp2v48OHav3+/nE6nhg8frgULFmjjxo3WNrp3766MjAwtWrRIkhQeHq7bb79dU6ZMkSTl5uYqODhYzz77rEaMGHFRfXs8Hvn5+SkzM1Nut7soh0SSVGvEggvW7BwfXeTbBQDgRnal378v5Jo5RywnJ0ezZs1SVlaWIiIilJycrJMnTyoyMtKqadCggWrUqKGkpCRJUlJSkho3bmyFMEmKioqSx+OxjqolJSV5rSOvJm8d2dnZSk5O9qrx8fFRZGSkVVOQEydOyOPxeE0AAACXwvYgtmHDBpUrV04ul0v9+/fX3LlzFRoaqvT0dDmdTvn7+3vVBwYGKj09XZKUnp7uFcLyluctO1+Nx+PRsWPH9NdffyknJ6fAmrx1FGTcuHHy8/OzpuDg4ELtPwAAKL5sD2L169dXSkqKVq9eraeffloxMTHavHmz3W1d0MiRI5WZmWlNv//+u90tAQCA60wJuxtwOp2qW7euJCksLEw//vijJk+erG7duik7O1sZGRleR8X27t2roKAgSVJQUFC+qxvzrqo8s+bsKy337t0rt9ut0qVLy9fXV76+vgXW5K2jIC6XSy6Xq3A7DQAAoGvgiNjZcnNzdeLECYWFhalkyZJKTEy0lqWmpmrXrl2KiIiQJEVERGjDhg1eVzcmJCTI7XYrNDTUqjlzHXk1eetwOp0KCwvzqsnNzVViYqJVAwAAcCXYekRs5MiR6tSpk2rUqKHDhw9r5syZWrZsmRYvXiw/Pz/16dNHcXFxqlixotxut5599llFRESoZcuWkqQOHTooNDRUjz/+uCZMmKD09HS99NJLio2NtY5W9e/fX1OmTNGwYcP0xBNPaMmSJZo9e7YWLPjvVYhxcXGKiYlR8+bN1aJFC02aNElZWVnq3bu3LeMCAACKB1uD2L59+9SzZ0/t2bNHfn5+atKkiRYvXqy//e1vkqQ333xTPj4+6tq1q06cOKGoqCi9/fbb1vN9fX01f/58Pf3004qIiFDZsmUVExOjl19+2aoJCQnRggULNGjQIE2ePFnVq1fXe++9p6ioKKumW7du2r9/v0aNGqX09HQ1a9ZMixYtyncCPwAAQFG65u4jdr3iPmIAAFx/uI8YAABAMUUQAwAAsAlBDAAAwCYEMQAAAJsQxAAAAGxCEAMAALAJQQwAAMAmBDEAAACbEMQAAABsQhADAACwCUEMAADAJgQxAAAAmxDEAAAAbEIQAwAAsAlBDAAAwCYEMQAAAJsQxAAAAGxCEAMAALAJQQwAAMAmBDEAAACbEMQAAABsQhADAACwCUEMAADAJgQxAAAAmxDEAAAAbEIQAwAAsAlBDAAAwCYEMQAAAJsQxAAAAGxCEAMAALAJQQwAAMAmBDEAAACbEMQAAABsQhADAACwCUEMAADAJgQxAAAAm9gaxMaNG6fbb79d5cuXV0BAgLp06aLU1FSvmrZt28rhcHhN/fv396rZtWuXoqOjVaZMGQUEBGjo0KE6deqUV82yZct02223yeVyqW7duoqPj8/Xz9SpU1WrVi2VKlVK4eHhWrNmTZHvMwAAQB5bg9jy5csVGxurH374QQkJCTp58qQ6dOigrKwsr7onn3xSe/bssaYJEyZYy3JychQdHa3s7GytWrVKH374oeLj4zVq1CirJi0tTdHR0WrXrp1SUlI0cOBA9e3bV4sXL7ZqPvvsM8XFxWn06NH66aef1LRpU0VFRWnfvn1XfiAAAECx5DDGGLubyLN//34FBARo+fLlatOmjaTTR8SaNWumSZMmFfichQsX6p577tHu3bsVGBgoSZo+fbqGDx+u/fv3y+l0avjw4VqwYIE2btxoPa979+7KyMjQokWLJEnh4eG6/fbbNWXKFElSbm6ugoOD9eyzz2rEiBEX7N3j8cjPz0+ZmZlyu92XMwwFqjViwQVrdo6PLvLtAgBwI7vS798Xck2dI5aZmSlJqlixotf8Tz75RJUrV1ajRo00cuRIHT161FqWlJSkxo0bWyFMkqKiouTxeLRp0yarJjIy0mudUVFRSkpKkiRlZ2crOTnZq8bHx0eRkZFWzdlOnDghj8fjNQEAAFyKEnY3kCc3N1cDBw7UnXfeqUaNGlnzH3nkEdWsWVPVqlXT+vXrNXz4cKWmpuqLL76QJKWnp3uFMEnW4/T09PPWeDweHTt2TIcOHVJOTk6BNVu2bCmw33Hjxmns2LGXt9MAAKBYu2aCWGxsrDZu3Kjvv//ea36/fv2sfzdu3FhVq1ZV+/bttX37dtWpU+dqt2kZOXKk4uLirMcej0fBwcG29QMAAK4/10QQGzBggObPn68VK1aoevXq560NDw+XJG3btk116tRRUFBQvqsb9+7dK0kKCgqy/ps378wat9ut0qVLy9fXV76+vgXW5K3jbC6XSy6X6+J3EgAA4Cy2niNmjNGAAQM0d+5cLVmyRCEhIRd8TkpKiiSpatWqkqSIiAht2LDB6+rGhIQEud1uhYaGWjWJiYle60lISFBERIQkyel0KiwszKsmNzdXiYmJVg0AAEBRs/WIWGxsrGbOnKkvv/xS5cuXt87p8vPzU+nSpbV9+3bNnDlTnTt3VqVKlbR+/XoNGjRIbdq0UZMmTSRJHTp0UGhoqB5//HFNmDBB6enpeumllxQbG2sdserfv7+mTJmiYcOG6YknntCSJUs0e/ZsLVjw3ysR4+LiFBMTo+bNm6tFixaaNGmSsrKy1Lt376s/MAAAoFiwNYhNmzZN0ulbVJxpxowZ6tWrl5xOp7799lsrFAUHB6tr16566aWXrFpfX1/Nnz9fTz/9tCIiIlS2bFnFxMTo5ZdftmpCQkK0YMECDRo0SJMnT1b16tX13nvvKSoqyqrp1q2b9u/fr1GjRik9PV3NmjXTokWL8p3ADwAAUFSuqfuIXc+4jxgAANcf7iMGAABQTBHEAAAAbEIQAwAAsAlBDAAAwCYEMQAAAJsQxAAAAGxCEAMAALAJQQwAAMAmBDEAAACbEMQAAABsQhADAACwSaGC2I4dO4q6DwAAgGKnUEGsbt26ateunT7++GMdP368qHsCAAAoFgoVxH766Sc1adJEcXFxCgoK0lNPPaU1a9YUdW8AAAA3tEIFsWbNmmny5MnavXu3PvjgA+3Zs0etWrVSo0aN9MYbb2j//v1F3ScAAMAN57JO1i9RooQefPBBzZkzR6+99pq2bdumIUOGKDg4WD179tSePXuKqk8AAIAbzmUFsbVr1+qZZ55R1apV9cYbb2jIkCHavn27EhIStHv3bt1///1F1ScAAMANp0RhnvTGG29oxowZSk1NVefOnfXRRx+pc+fO8vE5netCQkIUHx+vWrVqFWWvAAAAN5RCBbFp06bpiSeeUK9evVS1atUCawICAvT+++9fVnMAAAA3skIFsa1bt16wxul0KiYmpjCrBwAAKBYKdY7YjBkzNGfOnHzz58yZow8//PCymwIAACgOChXExo0bp8qVK+ebHxAQoH/+85+X3RQAAEBxUKggtmvXLoWEhOSbX7NmTe3ateuymwIAACgOChXEAgICtH79+nzzf/75Z1WqVOmymwIAACgOChXEevTooeeee05Lly5VTk6OcnJytGTJEj3//PPq3r17UfcIAABwQyrUVZOvvPKKdu7cqfbt26tEidOryM3NVc+ePTlHDAAA4CIVKog5nU599tlneuWVV/Tzzz+rdOnSaty4sWrWrFnU/QEAANywChXE8tx88826+eabi6oXAACAYqVQQSwnJ0fx8fFKTEzUvn37lJub67V8yZIlRdIcAADAjaxQQez5559XfHy8oqOj1ahRIzkcjqLuCwAA4IZXqCA2a9YszZ49W507dy7qfgAAAIqNQt2+wul0qm7dukXdCwAAQLFSqCA2ePBgTZ48WcaYou4HAACg2CjUR5Pff/+9li5dqoULF6phw4YqWbKk1/IvvviiSJoDAAC4kRUqiPn7++uBBx4o6l4AAACKlUIFsRkzZhR1HwAAAMVOoc4Rk6RTp07p22+/1TvvvKPDhw9Lknbv3q0jR45c9DrGjRun22+/XeXLl1dAQIC6dOmi1NRUr5rjx48rNjZWlSpVUrly5dS1a1ft3bvXq2bXrl2Kjo5WmTJlFBAQoKFDh+rUqVNeNcuWLdNtt90ml8ulunXrKj4+Pl8/U6dOVa1atVSqVCmFh4drzZo1F70vAAAAl6pQQey3335T48aNdf/99ys2Nlb79++XJL322msaMmTIRa9n+fLlio2N1Q8//KCEhASdPHlSHTp0UFZWllUzaNAg/ec//9GcOXO0fPly7d69Ww8++KC1PCcnR9HR0crOztaqVav04YcfKj4+XqNGjbJq0tLSFB0drXbt2iklJUUDBw5U3759tXjxYqvms88+U1xcnEaPHq2ffvpJTZs2VVRUlPbt21eYIQIAALgghynEpY9dunRR+fLl9f7776tSpUr6+eefVbt2bS1btkxPPvmktm7dWqhm9u/fr4CAAC1fvlxt2rRRZmamqlSpopkzZ+qhhx6SJG3ZskW33HKLkpKS1LJlSy1cuFD33HOPdu/ercDAQEnS9OnTNXz4cO3fv19Op1PDhw/XggULtHHjRmtb3bt3V0ZGhhYtWiRJCg8P1+23364pU6ZIOv0l5sHBwXr22Wc1YsSIC/bu8Xjk5+enzMxMud3uQu3/+dQaseCCNTvHRxf5dgEAuJFd6ffvCynUEbHvvvtOL730kpxOp9f8WrVq6c8//yx0M5mZmZKkihUrSpKSk5N18uRJRUZGWjUNGjRQjRo1lJSUJElKSkpS48aNrRAmSVFRUfJ4PNq0aZNVc+Y68mry1pGdna3k5GSvGh8fH0VGRlo1Zztx4oQ8Ho/XBAAAcCkKFcRyc3OVk5OTb/4ff/yh8uXLF6qR3NxcDRw4UHfeeacaNWokSUpPT5fT6ZS/v79XbWBgoNLT062aM0NY3vK8Zeer8Xg8OnbsmP766y/l5OQUWJO3jrONGzdOfn5+1hQcHFyo/QYAAMVXoYJYhw4dNGnSJOuxw+HQkSNHNHr06EJ/7VFsbKw2btyoWbNmFer5V9vIkSOVmZlpTb///rvdLQEAgOtMoW5f8frrrysqKkqhoaE6fvy4HnnkEW3dulWVK1fWp59+esnrGzBggObPn68VK1aoevXq1vygoCBlZ2crIyPD66jY3r17FRQUZNWcfXVj3lWVZ9acfaXl3r175Xa7Vbp0afn6+srX17fAmrx1nM3lcsnlcl3yvgIAAOQp1BGx6tWr6+eff9YLL7ygQYMG6dZbb9X48eO1bt06BQQEXPR6jDEaMGCA5s6dqyVLligkJMRreVhYmEqWLKnExERrXmpqqnbt2qWIiAhJUkREhDZs2OB1dWNCQoLcbrdCQ0OtmjPXkVeTtw6n06mwsDCvmtzcXCUmJlo1AAAARa1QR8QkqUSJEnrssccua+OxsbGaOXOmvvzyS5UvX946H8vPz0+lS5eWn5+f+vTpo7i4OFWsWFFut1vPPvusIiIi1LJlS0mnPyYNDQ3V448/rgkTJig9PV0vvfSSYmNjrSNW/fv315QpUzRs2DA98cQTWrJkiWbPnq0FC/57JWJcXJxiYmLUvHlztWjRQpMmTVJWVpZ69+59WfsIAABwLoUKYh999NF5l/fs2fOi1jNt2jRJUtu2bb3mz5gxQ7169ZIkvfnmm/Lx8VHXrl114sQJRUVF6e2337ZqfX19NX/+fD399NOKiIhQ2bJlFRMTo5dfftmqCQkJ0YIFCzRo0CBNnjxZ1atX13vvvaeoqCirplu3btq/f79GjRql9PR0NWvWTIsWLcp3Aj8AAEBRKdR9xCpUqOD1+OTJkzp69KicTqfKlCmjgwcPFlmD1wvuIwYAwPXnuryP2KFDh7ymI0eOKDU1Va1atSrUyfoAAADFUaG/a/Js9erV0/jx4/X8888X1SoBAABuaEUWxKTTJ/Dv3r27KFcJAABwwyrUyfpfffWV12NjjPbs2aMpU6bozjvvLJLGAAAAbnSFCmJdunTxeuxwOFSlShXdfffdev3114uiLwAAgBteoYJYbm5uUfcBAABQ7BTpOWIAAAC4eIU6IhYXF3fRtW+88UZhNgEAAHDDK1QQW7dundatW6eTJ0+qfv36kqRff/1Vvr6+uu2226w6h8NRNF0CAADcgAoVxO69916VL19eH374oXWX/UOHDql3795q3bq1Bg8eXKRNAgAA3IgKdY7Y66+/rnHjxnl91VGFChX06quvctUkAADARSpUEPN4PNq/f3+++fv379fhw4cvuykAAIDioFBB7IEHHlDv3r31xRdf6I8//tAff/yh//u//1OfPn304IMPFnWPAAAAN6RCnSM2ffp0DRkyRI888ohOnjx5ekUlSqhPnz6aOHFikTYIAABwoypUECtTpozefvttTZw4Udu3b5ck1alTR2XLli3S5gAAAG5kl3VD1z179mjPnj2qV6+eypYtK2NMUfUFAABwwytUEDtw4IDat2+vm2++WZ07d9aePXskSX369OHWFQAAABepUEFs0KBBKlmypHbt2qUyZcpY87t166ZFixYVWXMAAAA3skKdI/bNN99o8eLFql69utf8evXq6bfffiuSxgAAAG50hToilpWV5XUkLM/BgwflcrkuuykAAIDioFBBrHXr1vroo4+sxw6HQ7m5uZowYYLatWtXZM0BAADcyAr10eSECRPUvn17rV27VtnZ2Ro2bJg2bdqkgwcPauXKlUXdIwAAwA2pUEfEGjVqpF9//VWtWrXS/fffr6ysLD344INat26d6tSpU9Q9AgAA3JAu+YjYyZMn1bFjR02fPl0vvvjilegJAACgWLjkI2IlS5bU+vXrr0QvAAAAxUqhPpp87LHH9P777xd1LwAAAMVKoU7WP3XqlD744AN9++23CgsLy/cdk2+88UaRNAcAAHAju6QgtmPHDtWqVUsbN27UbbfdJkn69ddfvWocDkfRdQcAAHADu6QgVq9ePe3Zs0dLly6VdPorjf79738rMDDwijQHAABwI7ukc8SMMV6PFy5cqKysrCJtCAAAoLgo1Mn6ec4OZgAAALh4lxTEHA5HvnPAOCcMAACgcC7pHDFjjHr16mV9sffx48fVv3//fFdNfvHFF0XXIQAAwA3qkoJYTEyM1+PHHnusSJsBAAAoTi4piM2YMeNK9QEAAFDsXNbJ+gAAACg8W4PYihUrdO+996patWpyOByaN2+e1/JevXpZFwjkTR07dvSqOXjwoB599FG53W75+/urT58+OnLkiFfN+vXr1bp1a5UqVUrBwcGaMGFCvl7mzJmjBg0aqFSpUmrcuLG+/vrrIt9fAACAM9kaxLKystS0aVNNnTr1nDUdO3bUnj17rOnTTz/1Wv7oo49q06ZNSkhI0Pz587VixQr169fPWu7xeNShQwfVrFlTycnJmjhxosaMGaN3333Xqlm1apV69OihPn36aN26derSpYu6dOmijRs3Fv1OAwAA/D8Oc43cDMzhcGju3Lnq0qWLNa9Xr17KyMjId6Qszy+//KLQ0FD9+OOPat68uSRp0aJF6ty5s/744w9Vq1ZN06ZN04svvqj09HQ5nU5J0ogRIzRv3jxt2bJF0ulvCMjKytL8+fOtdbds2VLNmjXT9OnTL6p/j8cjPz8/ZWZmyu12F2IEzq/WiAUXrNk5PrrItwsAwI3sSr9/X8g1f47YsmXLFBAQoPr16+vpp5/WgQMHrGVJSUny9/e3QpgkRUZGysfHR6tXr7Zq2rRpY4UwSYqKilJqaqoOHTpk1URGRnptNyoqSklJSefs68SJE/J4PF4TAADApbimg1jHjh310UcfKTExUa+99pqWL1+uTp06KScnR5KUnp6ugIAAr+eUKFFCFStWVHp6ulVz9ndh5j2+UE3e8oKMGzdOfn5+1hQcHHx5OwsAAIqdS7p9xdXWvXt369+NGzdWkyZNVKdOHS1btkzt27e3sTNp5MiRiouLsx57PB7CGAAAuCTX9BGxs9WuXVuVK1fWtm3bJElBQUHat2+fV82pU6d08OBBBQUFWTV79+71qsl7fKGavOUFcblccrvdXhMAAMCluK6C2B9//KEDBw6oatWqkqSIiAhlZGQoOTnZqlmyZIlyc3MVHh5u1axYsUInT560ahISElS/fn1VqFDBqklMTPTaVkJCgiIiIq70LgEAgGLM1iB25MgRpaSkKCUlRZKUlpamlJQU7dq1S0eOHNHQoUP1ww8/aOfOnUpMTNT999+vunXrKioqSpJ0yy23qGPHjnryySe1Zs0arVy5UgMGDFD37t1VrVo1SdIjjzwip9OpPn36aNOmTfrss880efJkr48Vn3/+eS1atEivv/66tmzZojFjxmjt2rUaMGDAVR8TAABQfNgaxNauXatbb71Vt956qyQpLi5Ot956q0aNGiVfX1+tX79e9913n26++Wb16dNHYWFh+u6776wvHZekTz75RA0aNFD79u3VuXNntWrVyuseYX5+fvrmm2+UlpamsLAwDR48WKNGjfK619gdd9yhmTNn6t1331XTpk31+eefa968eWrUqNHVGwwAAFDsXDP3EbvecR8xAACuP9xHDAAAoJgiiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE1sDWIrVqzQvffeq2rVqsnhcGjevHley40xGjVqlKpWrarSpUsrMjJSW7du9ao5ePCgHn30Ubndbvn7+6tPnz46cuSIV8369evVunVrlSpVSsHBwZowYUK+XubMmaMGDRqoVKlSaty4sb7++usi318AAIAz2RrEsrKy1LRpU02dOrXA5RMmTNC///1vTZ8+XatXr1bZsmUVFRWl48ePWzWPPvqoNm3apISEBM2fP18rVqxQv379rOUej0cdOnRQzZo1lZycrIkTJ2rMmDF69913rZpVq1apR48e6tOnj9atW6cuXbqoS5cu2rhx45XbeQAAUOw5jDHG7iYkyeFwaO7cuerSpYuk00fDqlWrpsGDB2vIkCGSpMzMTAUGBio+Pl7du3fXL7/8otDQUP34449q3ry5JGnRokXq3Lmz/vjjD1WrVk3Tpk3Tiy++qPT0dDmdTknSiBEjNG/ePG3ZskWS1K1bN2VlZWn+/PlWPy1btlSzZs00ffr0i+rf4/HIz89PmZmZcrvdRTUsllojFlywZuf46CLfLgAAN7Ir/f59IdfsOWJpaWlKT09XZGSkNc/Pz0/h4eFKSkqSJCUlJcnf398KYZIUGRkpHx8frV692qpp06aNFcIkKSoqSqmpqTp06JBVc+Z28mrytlOQEydOyOPxeE0AAACX4poNYunp6ZKkwMBAr/mBgYHWsvT0dAUEBHgtL1GihCpWrOhVU9A6ztzGuWrylhdk3Lhx8vPzs6bg4OBL3UUAAFDMXbNB7Fo3cuRIZWZmWtPvv/9ud0sAAOA6c80GsaCgIEnS3r17vebv3bvXWhYUFKR9+/Z5LT916pQOHjzoVVPQOs7cxrlq8pYXxOVyye12e00AAACX4poNYiEhIQoKClJiYqI1z+PxaPXq1YqIiJAkRUREKCMjQ8nJyVbNkiVLlJubq/DwcKtmxYoVOnnypFWTkJCg+vXrq0KFClbNmdvJq8nbDgAAwJVgaxA7cuSIUlJSlJKSIun0CfopKSnatWuXHA6HBg4cqFdffVVfffWVNmzYoJ49e6patWrWlZW33HKLOnbsqCeffFJr1qzRypUrNWDAAHXv3l3VqlWTJD3yyCNyOp3q06ePNm3apM8++0yTJ09WXFyc1cfzzz+vRYsW6fXXX9eWLVs0ZswYrV27VgMGDLjaQwIAAIqREnZufO3atWrXrp31OC8cxcTEKD4+XsOGDVNWVpb69eunjIwMtWrVSosWLVKpUqWs53zyyScaMGCA2rdvLx8fH3Xt2lX//ve/reV+fn765ptvFBsbq7CwMFWuXFmjRo3yutfYHXfcoZkzZ+qll17SCy+8oHr16mnevHlq1KjRVRgFAABQXF0z9xG73nEfMQAArj/cRwwAAKCYIogBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNrukgNmbMGDkcDq+pQYMG1vLjx48rNjZWlSpVUrly5dS1a1ft3bvXax27du1SdHS0ypQpo4CAAA0dOlSnTp3yqlm2bJluu+02uVwu1a1bV/Hx8Vdj9wAAQDF3TQcxSWrYsKH27NljTd9//721bNCgQfrPf/6jOXPmaPny5dq9e7cefPBBa3lOTo6io6OVnZ2tVatW6cMPP1R8fLxGjRpl1aSlpSk6Olrt2rVTSkqKBg4cqL59+2rx4sVXdT8BAEDxU8LuBi6kRIkSCgoKyjc/MzNT77//vmbOnKm7775bkjRjxgzdcsst+uGHH9SyZUt988032rx5s7799lsFBgaqWbNmeuWVVzR8+HCNGTNGTqdT06dPV0hIiF5//XVJ0i233KLvv/9eb775pqKioq7qvgIAgOLlmj8itnXrVlWrVk21a9fWo48+ql27dkmSkpOTdfLkSUVGRlq1DRo0UI0aNZSUlCRJSkpKUuPGjRUYGGjVREVFyePxaNOmTVbNmevIq8lbx7mcOHFCHo/HawIAALgU13QQCw8PV3x8vBYtWqRp06YpLS1NrVu31uHDh5Weni6n0yl/f3+v5wQGBio9PV2SlJ6e7hXC8pbnLTtfjcfj0bFjx87Z27hx4+Tn52dNwcHBl7u7AACgmLmmP5rs1KmT9e8mTZooPDxcNWvW1OzZs1W6dGkbO5NGjhypuLg467HH4yGMAQCAS3JNHxE7m7+/v26++WZt27ZNQUFBys7OVkZGhlfN3r17rXPKgoKC8l1Fmff4QjVut/u8Yc/lcsntdntNAAAAl+K6CmJHjhzR9u3bVbVqVYWFhalkyZJKTEy0lqempmrXrl2KiIiQJEVERGjDhg3at2+fVZOQkCC3263Q0FCr5sx15NXkrQMAAOBKuaaD2JAhQ7R8+XLt3LlTq1at0gMPPCBfX1/16NFDfn5+6tOnj+Li4rR06VIlJyerd+/eioiIUMuWLSVJHTp0UGhoqB5//HH9/PPPWrx4sV566SXFxsbK5XJJkvr3768dO3Zo2LBh2rJli95++23Nnj1bgwYNsnPXAQBAMXBNnyP2xx9/qEePHjpw4ICqVKmiVq1a6YcfflCVKlUkSW+++aZ8fHzUtWtXnThxQlFRUXr77bet5/v6+mr+/Pl6+umnFRERobJlyyomJkYvv/yyVRMSEqIFCxZo0KBBmjx5sqpXr6733nuPW1cAAIArzmGMMXY3cSPweDzy8/NTZmbmFTlfrNaIBRes2Tk+usi3CwDAjexKv39fyDX90SQAAMCNjCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2KWF3Ayg6tUYsuGDNzvHRV6ETAABwMTgiBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGInWXq1KmqVauWSpUqpfDwcK1Zs8bulgAAwA2KIHaGzz77THFxcRo9erR++uknNW3aVFFRUdq3b5/drQEAgBuQwxhj7G7iWhEeHq7bb79dU6ZMkSTl5uYqODhYzz77rEaMGHHe53o8Hvn5+SkzM1Nut7vIe6s1YkGRr/Ncdo6PvmrbAgDATlf6/ftCSlz1LV6jsrOzlZycrJEjR1rzfHx8FBkZqaSkpHz1J06c0IkTJ6zHmZmZkk7/QK+E3BNHr8h6C1Jj0JwL1mwcG3UVOgEA4MrKe9+267gUQez/+euvv5STk6PAwECv+YGBgdqyZUu++nHjxmns2LH55gcHB1+xHq8lfpPs7gAAgKJz+PBh+fn5XfXtEsQKaeTIkYqLi7Me5+bm6uDBg6pUqZIcDkeRbsvj8Sg4OFi///67LYdNr1eMW+ExdoXH2BUeY1d4jF3h5I3b5s2bVa1aNVt6IIj9P5UrV5avr6/27t3rNX/v3r0KCgrKV+9yueRyubzm+fv7X8kW5Xa7+QUrBMat8Bi7wmPsCo+xKzzGrnBuuukm+fjYc/0iV03+P06nU2FhYUpMTLTm5ebmKjExURERETZ2BgAAblQcETtDXFycYmJi1Lx5c7Vo0UKTJk1SVlaWevfubXdrAADgBkQQO0O3bt20f/9+jRo1Sunp6WrWrJkWLVqU7wT+q83lcmn06NH5PgrF+TFuhcfYFR5jV3iMXeExdoVzLYwb9xEDAACwCeeIAQAA2IQgBgAAYBOCGAAAgE0IYgAAADYhiAEAANiEIHaNmzp1qmrVqqVSpUopPDxca9assbulq2rMmDFyOBxeU4MGDazlx48fV2xsrCpVqqRy5cqpa9eu+b4dYdeuXYqOjlaZMmUUEBCgoUOH6tSpU141y5Yt02233SaXy6W6desqPj7+auxekVqxYoXuvfdeVatWTQ6HQ/PmzfNabozRqFGjVLVqVZUuXVqRkZHaunWrV83Bgwf16KOPyu12y9/fX3369NGRI0e8atavX6/WrVurVKlSCg4O1oQJE/L1MmfOHDVo0EClSpVS48aN9fXXXxf5/halC41dr1698r0OO3bs6FVTHMdu3Lhxuv3221W+fHkFBASoS5cuSk1N9aq5mr+j19Pfy4sZu7Zt2+Z73fXv39+rpriN3bRp09SkSRPrGwQiIiK0cOFCa/l1+XozuGbNmjXLOJ1O88EHH5hNmzaZJ5980vj7+5u9e/fa3dpVM3r0aNOwYUOzZ88ea9q/f7+1vH///iY4ONgkJiaatWvXmpYtW5o77rjDWn7q1CnTqFEjExkZadatW2e+/vprU7lyZTNy5EirZseOHaZMmTImLi7ObN682bz11lvG19fXLFq06Kru6+X6+uuvzYsvvmi++OILI8nMnTvXa/n48eONn5+fmTdvnvn555/NfffdZ0JCQsyxY8esmo4dO5qmTZuaH374wXz33Xembt26pkePHtbyzMxMExgYaB599FGzceNG8+mnn5rSpUubd955x6pZuXKl8fX1NRMmTDCbN282L730kilZsqTZsGHDFR+DwrrQ2MXExJiOHTt6vQ4PHjzoVVMcxy4qKsrMmDHDbNy40aSkpJjOnTubGjVqmCNHjlg1V+t39Hr7e3kxY3fXXXeZJ5980ut1l5mZaS0vjmP31VdfmQULFphff/3VpKammhdeeMGULFnSbNy40Rhzfb7eCGLXsBYtWpjY2FjrcU5OjqlWrZoZN26cjV1dXaNHjzZNmzYtcFlGRoYpWbKkmTNnjjXvl19+MZJMUlKSMeb0G6yPj49JT0+3aqZNm2bcbrc5ceKEMcaYYcOGmYYNG3qtu1u3biYqKqqI9+bqOTtM5ObmmqCgIDNx4kRrXkZGhnG5XObTTz81xhizefNmI8n8+OOPVs3ChQuNw+Ewf/75pzHGmLfffttUqFDBGjtjjBk+fLipX7++9fjhhx820dHRXv2Eh4ebp556qkj38Uo5VxC7//77z/kcxu60ffv2GUlm+fLlxpir+zt6vf+9PHvsjDkdxJ5//vlzPoexO61ChQrmvffeu25fb3w0eY3Kzs5WcnKyIiMjrXk+Pj6KjIxUUlKSjZ1dfVu3blW1atVUu3ZtPfroo9q1a5ckKTk5WSdPnvQaowYNGqhGjRrWGCUlJalx48Ze344QFRUlj8ejTZs2WTVnriOv5kYa57S0NKWnp3vtp5+fn8LDw73Gyt/fX82bN7dqIiMj5ePjo9WrV1s1bdq0kdPptGqioqKUmpqqQ4cOWTU34nguW7ZMAQEBql+/vp5++mkdOHDAWsbYnZaZmSlJqlixoqSr9zt6I/y9PHvs8nzyySeqXLmyGjVqpJEjR+ro0aPWsuI+djk5OZo1a5aysrIUERFx3b7e+Iqja9Rff/2lnJycfF+vFBgYqC1bttjU1dUXHh6u+Ph41a9fX3v27NHYsWPVunVrbdy4Uenp6XI6nfL39/d6TmBgoNLT0yVJ6enpBY5h3rLz1Xg8Hh07dkylS5e+Qnt39eTta0H7eeY4BAQEeC0vUaKEKlas6FUTEhKSbx15yypUqHDO8cxbx/WoY8eOevDBBxUSEqLt27frhRdeUKdOnZSUlCRfX1/GTlJubq4GDhyoO++8U40aNZKkq/Y7eujQoev672VBYydJjzzyiGrWrKlq1app/fr1Gj58uFJTU/XFF19IKr5jt2HDBkVEROj48eMqV66c5s6dq9DQUKWkpFyXrzeCGK5pnTp1sv7dpEkThYeHq2bNmpo9e/YNEZBwfejevbv178aNG6tJkyaqU6eOli1bpvbt29vY2bUjNjZWGzdu1Pfff293K9edc41dv379rH83btxYVatWVfv27bV9+3bVqVPnard5zahfv75SUlKUmZmpzz//XDExMVq+fLndbRUaH01eoypXrixfX998V3vs3btXQUFBNnVlP39/f918883atm2bgoKClJ2drYyMDK+aM8coKCiowDHMW3a+GrfbfcOEvbx9Pd/rKSgoSPv27fNafurUKR08eLBIxvNGet3Wrl1blStX1rZt2yQxdgMGDND8+fO1dOlSVa9e3Zp/tX5Hr+e/l+cau4KEh4dLktfrrjiOndPpVN26dRUWFqZx48apadOmmjx58nX7eiOIXaOcTqfCwsKUmJhozcvNzVViYqIiIiJs7MxeR44c0fbt21W1alWFhYWpZMmSXmOUmpqqXbt2WWMUERGhDRs2eL1JJiQkyO12KzQ01Ko5cx15NTfSOIeEhCgoKMhrPz0ej1avXu01VhkZGUpOTrZqlixZotzcXOsNICIiQitWrNDJkyetmoSEBNWvX18VKlSwam708fzjjz904MABVa1aVVLxHTtjjAYMGKC5c+dqyZIl+T56vVq/o9fj38sLjV1BUlJSJMnrdVccx+5subm5OnHixPX7ervk0/tx1cyaNcu4XC4THx9vNm/ebPr162f8/f29rva40Q0ePNgsW7bMpKWlmZUrV5rIyEhTuXJls2/fPmPM6UuVa9SoYZYsWWLWrl1rIiIiTEREhPX8vEuVO3ToYFJSUsyiRYtMlSpVCrxUeejQoeaXX34xU6dOvS5vX3H48GGzbt06s27dOiPJvPHGG2bdunXmt99+M8acvn2Fv7+/+fLLL8369evN/fffX+DtK2699VazevVq8/3335t69ep53YIhIyPDBAYGmscff9xs3LjRzJo1y5QpUybfLRhKlChh/vWvf5lffvnFjB49+pq+BYMx5x+7w4cPmyFDhpikpCSTlpZmvv32W3PbbbeZevXqmePHj1vrKI5j9/TTTxs/Pz+zbNkyr1ssHD161Kq5Wr+j19vfywuN3bZt28zLL79s1q5da9LS0syXX35pateubdq0aWOtoziO3YgRI8zy5ctNWlqaWb9+vRkxYoRxOBzmm2++McZcn683gtg17q233jI1atQwTqfTtGjRwvzwww92t3RVdevWzVStWtU4nU5z0003mW7duplt27ZZy48dO2aeeeYZU6FCBVOmTBnzwAMPmD179nitY+fOnaZTp06mdOnSpnLlymbw4MHm5MmTXjVLly41zZo1M06n09SuXdvMmDHjauxekVq6dKmRlG+KiYkxxpy+hcXf//53ExgYaFwul2nfvr1JTU31WseBAwdMjx49TLly5Yzb7Ta9e/c2hw8f9qr5+eefTatWrYzL5TI33XSTGT9+fL5eZs+ebW6++WbjdDpNw4YNzYIFC67YfheF843d0aNHTYcOHUyVKlVMyZIlTc2aNc2TTz6Z7w9ucRy7gsZMktfvz9X8Hb2e/l5eaOx27dpl2rRpYypWrGhcLpepW7euGTp0qNd9xIwpfmP3xBNPmJo1axqn02mqVKli2rdvb4UwY67P15vDGGMu/TgaAAAALhfniAEAANiEIAYAAGATghgAAIBNCGIAAAA2IYgBAADYhCAGAABgE4IYAACATQhiAAAANiGIAQAA2IQgBgAAYBOCGAAAgE3+fyNsX6KYjwpwAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Longest email length: \", df2['Clean_Text'].str.split().apply(len).max())\n",
        "print(\"Average email length: \", df2['Clean_Text'].str.split().apply(len).mean())\n",
        "print(\"99th quantile email length: \", df2['Clean_Text'].str.split().apply(len).quantile(q=0.99))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dhUke-51KLcD",
        "outputId": "d46e1d0f-6491-4d70-a40e-8c365fd43217"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Longest email length:  28624\n",
            "Average email length:  201.44086210798983\n",
            "99th quantile email length:  1568.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The original code pads all the emails to the length of the longest email by adding 0's to the beginning of each sequence. This ensures all the emails have the same length.\n",
        "\n",
        "However, the longest email is 28,624 words long while the average email is only 201.4 words long, so I decided to pad all the email lengths to the 99th quantile (1,568 words) in my Bag of Words model."
      ],
      "metadata": {
        "id": "80zmZYtBJtCs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Method to preprocess data\n",
        "# (heavily modified from the original code)\n",
        "\n",
        "def pre_process(df):\n",
        "    # shuffle data\n",
        "    df = df.sample(frac = 1).reset_index(drop=True)\n",
        "\n",
        "    # word_freq --> dictionary of all the words and their frequencies\n",
        "    word_freq = Counter()\n",
        "    df['Clean_Text'].apply(lambda x: word_freq.update(x.split()))\n",
        "\n",
        "    # sorted_words --> a list of all unique words, sorted by frequency\n",
        "    sorted_words = sorted(word_freq, key=word_freq.get, reverse=True)\n",
        "\n",
        "    # vocab_to_int --> dictionary that maps each word to a unique integer\n",
        "    vocab_to_int = {c: i for i, c in enumerate(sorted_words, 1)}\n",
        "\n",
        "    # use the vocab_to_int dictionary to create a new column called BoW (bag of words)\n",
        "    # maps each word to the corresponding index\n",
        "    df['BoW'] = df['Clean_Text'].str.split(\" \")\n",
        "    df['BoW'] = df.BoW.apply(lambda row: [vocab_to_int[i] for i in row if vocab_to_int.get(i)])\n",
        "\n",
        "    # pad all the email lengths to the 99th quantile (1,568 words) by adding 0's\n",
        "    # to the beginning of each sequence\n",
        "    seq_length = int(df['BoW'].apply(len).quantile(q=0.99))\n",
        "\n",
        "    features = np.zeros([len(df), seq_length], dtype=int)\n",
        "    for i, row in enumerate(df['BoW']):\n",
        "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
        "\n",
        "    return features, np.array(df.Label), sorted_words"
      ],
      "metadata": {
        "id": "V_Td59z7jNFa"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### RNN (LSTM) model"
      ],
      "metadata": {
        "id": "obT8H5CaLHsA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Method to get data with the same batch size\n",
        "\n",
        "def get_batches(x, y, batch_size=100):\n",
        "    n_batches = len(x) // batch_size\n",
        "    x, y = x[:n_batches * batch_size], y[:n_batches * batch_size]\n",
        "    for ii in range(0, len(x), batch_size):\n",
        "        yield x[ii:ii + batch_size], y[ii:ii + batch_size]"
      ],
      "metadata": {
        "id": "x30Ardwc7p0S"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train and test the RNN model\n",
        "\n",
        "def train_test(df):\n",
        "    start = time.time()\n",
        "\n",
        "    # -----------------Train/Validation/Test Split------------------------------\n",
        "    start_split = time.time()\n",
        "\n",
        "    # set features and labels\n",
        "    features, labels, sorted_words = pre_process(df)\n",
        "\n",
        "    split_frac1 = 0.8\n",
        "\n",
        "    idx1 = int(len(features) * split_frac1)\n",
        "    train_x, val_x = features[:idx1], features[idx1:]\n",
        "    train_y, val_y = labels[:idx1], labels[idx1:]\n",
        "\n",
        "    split_frac2 = 0.5\n",
        "\n",
        "    idx2 = int(len(val_x) * split_frac2)\n",
        "    val_x, test_x = val_x[:idx2], val_x[idx2:]\n",
        "    val_y, test_y = val_y[:idx2], val_y[idx2:]\n",
        "\n",
        "    print(\"-----------------Train/Validation/Test Split-----------------\\n\")\n",
        "    print(\"\\nFeature Shapes:\")\n",
        "    print(\"Train set: \\t\\t{}\".format(train_x.shape),\n",
        "          \"\\nValidation set: \\t{}\".format(val_x.shape),\n",
        "          \"\\nTest set: \\t\\t{}\".format(test_x.shape))\n",
        "\n",
        "    print(\"\\nLabel Shapes:\")\n",
        "    print(\"Train set: \\t\\t{}\".format(train_y.shape),\n",
        "          \"\\nValidation set: \\t{}\".format(val_y.shape),\n",
        "          \"\\nTest set: \\t\\t{}\".format(test_y.shape))\n",
        "\n",
        "    split_time = time.time() - start_split\n",
        "    print(\"\\nTime Elapsed: \\t\\t{:.3f} s\\n\".format(split_time))\n",
        "\n",
        "    # -----------------Initialize Values----------------------------------------\n",
        "\n",
        "    start_initialize = time.time()\n",
        "\n",
        "    # set LSTM variables\n",
        "    epochs = 15\n",
        "    lstm_layers = 1\n",
        "    batch_size = 179\n",
        "    lstm_size = 30\n",
        "    learning_rate = 0.003\n",
        "\n",
        "    # set number of unique words\n",
        "    n_words = len(sorted_words) + 1\n",
        "\n",
        "    print(\"\\n---------------Initialized Values--------------------------\\n\")\n",
        "    print(\"Number of unique words: \\t{}\".format(n_words),\n",
        "          \"\\nLSTM size: \\t\\t\\t{}\".format(lstm_size),\n",
        "          \"\\nLSTM layers: \\t\\t\\t{}\".format(lstm_layers),\n",
        "          \"\\nBatch size: \\t\\t\\t{}\".format(batch_size),\n",
        "          \"\\nEpochs: \\t\\t\\t{}\".format(epochs),\n",
        "          \"\\nLearning Rate \\t\\t\\t{}\".format(learning_rate))\n",
        "\n",
        "    initialize_time = time.time() - start_initialize\n",
        "    print(\"\\nTime Elapsed: \\t\\t{:.3f} s\\n\".format(initialize_time))\n",
        "\n",
        "    # -----------------Placeholders---------------------------------------------\n",
        "\n",
        "    graph = tf.Graph()\n",
        "\n",
        "    with graph.as_default():\n",
        "\n",
        "        tf.random.set_seed(1)\n",
        "\n",
        "        inputs_ = tf.compat.v1.placeholder(tf.int32, [None, None], name=\"inputs\")\n",
        "        labels_ = tf.compat.v1.placeholder(tf.float32, [None, None], name=\"labels\")\n",
        "\n",
        "        keep_prob = tf.compat.v1.placeholder(tf.float32, name=\"keep_prob\")\n",
        "\n",
        "        embed_size = 300\n",
        "\n",
        "        embedding = tf.Variable(tf.random.uniform((n_words, embed_size), -1, 1))\n",
        "        embed = tf.compat.v1.nn.embedding_lookup(embedding, inputs_)\n",
        "\n",
        "        #print(embedding.shape)\n",
        "        #print(embed.shape)\n",
        "\n",
        "        lstm = tf.compat.v1.nn.rnn_cell.BasicLSTMCell(lstm_size)\n",
        "\n",
        "        drop = tf.compat.v1.nn.rnn_cell.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
        "\n",
        "        cell = tf.compat.v1.nn.rnn_cell.MultiRNNCell([drop] * lstm_layers)\n",
        "\n",
        "        initial_state = cell.zero_state(batch_size, tf.float32)\n",
        "\n",
        "        outputs, final_state = tf.compat.v1.nn.dynamic_rnn(cell, embed,\n",
        "                                                 initial_state=initial_state)\n",
        "\n",
        "        hidden1 = tf.compat.v1.layers.dense(outputs[:, -1], units=25, activation=tf.nn.relu)\n",
        "\n",
        "        hidden2 = tf.compat.v1.layers.dense(hidden1, units=15, activation=tf.nn.relu)\n",
        "\n",
        "        logit = tf.compat.v1.layers.dense(hidden2, units=1, activation=None)\n",
        "\n",
        "        cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(\n",
        "            logits=logit, labels=labels_))\n",
        "\n",
        "        optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate).minimize(cost)\n",
        "\n",
        "        predictions = tf.round(tf.nn.sigmoid(logit))\n",
        "\n",
        "        saver = tf.compat.v1.train.Saver()\n",
        "\n",
        "    # -----------------Training-------------------------------------------------\n",
        "\n",
        "    print(\"\\n-----------------Training----------------------------------\\n\")\n",
        "\n",
        "    start_train = time.time()\n",
        "\n",
        "    with tf.compat.v1.Session(graph=graph) as sess:\n",
        "        print(\"Starting Training...\")\n",
        "        tf.random.set_seed(1)\n",
        "        sess.run(tf.compat.v1.global_variables_initializer())\n",
        "        iteration = 1\n",
        "        for e in range(epochs):\n",
        "            state = sess.run(initial_state)\n",
        "            for ii, (x, y) in enumerate(get_batches(np.array(train_x),\n",
        "                                            np.array(train_y), batch_size), 1):\n",
        "\n",
        "                feed = {inputs_: x,\n",
        "                        labels_: y[:, None],\n",
        "                        keep_prob: 0.5,\n",
        "                        initial_state: state}\n",
        "                loss, states, _ = sess.run([cost, final_state, optimizer],\n",
        "                                           feed_dict=feed)\n",
        "\n",
        "                if iteration % 5 == 0:\n",
        "                    print(\"\\nEpoch: {}/{}\".format(e, epochs),\n",
        "                          \"Iteration: {}\".format(iteration),\n",
        "                          \"Train loss: {:.3f}\".format(loss))\n",
        "                iteration += 1\n",
        "\n",
        "        train_time = time.time() - start_train\n",
        "        print(\"\\nTime Elapsed: \\t\\t{:.3f} s\".format(train_time))\n",
        "        print(\"In hours & minutes: \\t\", str(datetime.timedelta(seconds = train_time)))\n",
        "\n",
        "        saver.save(sess, \"checkpoints/sentiment.ckpt\")\n",
        "\n",
        "    # -----------------Testing Validation Set-----------------------------------\n",
        "    print(\"Starting Validation Set...\")\n",
        "\n",
        "    start_valid = time.time()\n",
        "\n",
        "    prediction_vals = []\n",
        "    y_vals = []\n",
        "    with tf.compat.v1.Session(graph=graph) as sess:\n",
        "        tf.random.set_seed(1)\n",
        "        saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
        "\n",
        "        test_state = sess.run(cell.zero_state(batch_size, tf.float32))\n",
        "\n",
        "        for ii, (x, y) in enumerate(get_batches(np.array(val_x), np.array(val_y),\n",
        "                                                batch_size), 1):\n",
        "\n",
        "            feed = {inputs_: x,\n",
        "                    labels_: y[:, None],\n",
        "                    keep_prob: 1,\n",
        "                    initial_state: test_state}\n",
        "\n",
        "            prediction = sess.run(predictions, feed_dict=feed)\n",
        "            prediction = prediction.astype(int)\n",
        "\n",
        "            for i in range(len(prediction)):\n",
        "                prediction_vals.append(prediction[i][0])\n",
        "                y_vals.append(y[i])\n",
        "\n",
        "        accuracy = accuracy_score(y_vals, prediction_vals)\n",
        "        f1 = f1_score(y_vals, prediction_vals, average='macro')\n",
        "        recall = recall_score(y_true=y_vals, y_pred=prediction_vals, average='macro')\n",
        "        precision = precision_score(y_vals, prediction_vals, average='macro')\n",
        "\n",
        "        print(\"\\n-----------------Testing Validation Set----------------\\n\")\n",
        "        print(\"Test accuracy: {:.3f}\".format(accuracy))\n",
        "        print(\"F1 Score: {:.3f}\".format(f1))\n",
        "        print(\"Recall: {:.3f}\".format(recall))\n",
        "        print(\"Precision: {:.3f}\".format(precision))\n",
        "\n",
        "        valid_time = time.time() - start_valid\n",
        "        print(\"\\nTime Elapsed: \\t\\t{:.3f} s\\n\".format(valid_time))\n",
        "\n",
        "\n",
        "    # -----------------Testing Test Set-----------------------------------------\n",
        "    print(\"Starting Testing Set...\")\n",
        "\n",
        "    start_test = time.time()\n",
        "\n",
        "    prediction_val = []\n",
        "    y_val = []\n",
        "    with tf.compat.v1.Session(graph=graph) as sess:\n",
        "        tf.random.set_seed(1)\n",
        "        saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
        "        test_state = sess.run(cell.zero_state(batch_size, tf.float32))\n",
        "\n",
        "        for ii, (x, y) in enumerate(get_batches(np.array(test_x),\n",
        "                                                np.array(test_y), batch_size), 1):\n",
        "\n",
        "            feed = {inputs_: x,\n",
        "                    labels_: y[:, None],\n",
        "                    keep_prob: 1,\n",
        "                    initial_state: test_state}\n",
        "\n",
        "            prediction = sess.run(predictions, feed_dict=feed)\n",
        "            prediction = prediction.astype(int)\n",
        "\n",
        "            for i in range(len(prediction)):\n",
        "                prediction_val.append(prediction[i][0])\n",
        "                y_val.append(y[i])\n",
        "\n",
        "        accuracy = accuracy_score(y_val, prediction_val)\n",
        "        f1 = f1_score(y_val, prediction_val, average='macro')\n",
        "        recall = recall_score(y_true=y_val, y_pred=prediction_val, average='macro')\n",
        "        precision = precision_score(y_val, prediction_val, average='macro')\n",
        "\n",
        "        print(\"\\n-----------------Testing Test Set----------------------\\n\")\n",
        "        print(\"Test accuracy: {:.3f}\".format(accuracy))\n",
        "        print(\"F1 Score: {:.3f}\".format(f1))\n",
        "        print(\"Recall: {:.3f}\".format(recall))\n",
        "        print(\"Precision: {:.3f}\".format(precision))\n",
        "\n",
        "        test_time = time.time() - start_test\n",
        "        print(\"\\nTime Elapsed: \\t\\t{:.3f} s\\n\".format(test_time))\n",
        "\n",
        "        total_time = time.time() - start\n",
        "        print(\"\\n=======================================================\\n\")\n",
        "        print(\"\\nTotal Time Elapsed: \\t\\t{:.3f} s\\n\".format(total_time))\n",
        "        print(\"In hours & minutes: \\t\", str(datetime.timedelta(seconds = total_time)))"
      ],
      "metadata": {
        "id": "g5w3zvGl7t3J"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_test(df2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4KAvBxCjfH0o",
        "outputId": "355bc060-740c-4f94-9a08-4fc7fbf3f7e3"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:`tf.nn.rnn_cell.MultiRNNCell` is deprecated. This class is equivalent as `tf.keras.layers.StackedRNNCells`, and will be replaced by that in Tensorflow 2.0.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-----------------Train/Validation/Test Split-----------------\n",
            "\n",
            "\n",
            "Feature Shapes:\n",
            "Train set: \t\t(31810, 1568) \n",
            "Validation set: \t(3976, 1568) \n",
            "Test set: \t\t(3977, 1568)\n",
            "\n",
            "Label Shapes:\n",
            "Train set: \t\t(31810,) \n",
            "Validation set: \t(3976,) \n",
            "Test set: \t\t(3977,)\n",
            "\n",
            "Time Elapsed: \t\t4.764 s\n",
            "\n",
            "\n",
            "---------------Initialized Values--------------------------\n",
            "\n",
            "Number of unique words: \t256564 \n",
            "LSTM size: \t\t\t30 \n",
            "LSTM layers: \t\t\t1 \n",
            "Batch size: \t\t\t179 \n",
            "Epochs: \t\t\t15 \n",
            "Learning Rate \t\t\t0.003\n",
            "\n",
            "Time Elapsed: \t\t0.000 s\n",
            "\n",
            "\n",
            "-----------------Training----------------------------------\n",
            "\n",
            "Starting Training...\n",
            "\n",
            "Epoch: 0/15 Iteration: 5 Train loss: 0.688\n",
            "\n",
            "Epoch: 0/15 Iteration: 10 Train loss: 0.629\n",
            "\n",
            "Epoch: 0/15 Iteration: 15 Train loss: 0.586\n",
            "\n",
            "Epoch: 0/15 Iteration: 20 Train loss: 0.563\n",
            "\n",
            "Epoch: 0/15 Iteration: 25 Train loss: 0.591\n",
            "\n",
            "Epoch: 0/15 Iteration: 30 Train loss: 0.459\n",
            "\n",
            "Epoch: 0/15 Iteration: 35 Train loss: 0.362\n",
            "\n",
            "Epoch: 0/15 Iteration: 40 Train loss: 0.324\n",
            "\n",
            "Epoch: 0/15 Iteration: 45 Train loss: 0.196\n",
            "\n",
            "Epoch: 0/15 Iteration: 50 Train loss: 0.160\n",
            "\n",
            "Epoch: 0/15 Iteration: 55 Train loss: 0.214\n",
            "\n",
            "Epoch: 0/15 Iteration: 60 Train loss: 0.174\n",
            "\n",
            "Epoch: 0/15 Iteration: 65 Train loss: 0.213\n",
            "\n",
            "Epoch: 0/15 Iteration: 70 Train loss: 0.187\n",
            "\n",
            "Epoch: 0/15 Iteration: 75 Train loss: 0.202\n",
            "\n",
            "Epoch: 0/15 Iteration: 80 Train loss: 0.180\n",
            "\n",
            "Epoch: 0/15 Iteration: 85 Train loss: 0.177\n",
            "\n",
            "Epoch: 0/15 Iteration: 90 Train loss: 0.243\n",
            "\n",
            "Epoch: 0/15 Iteration: 95 Train loss: 0.166\n",
            "\n",
            "Epoch: 0/15 Iteration: 100 Train loss: 0.131\n",
            "\n",
            "Epoch: 0/15 Iteration: 105 Train loss: 0.145\n",
            "\n",
            "Epoch: 0/15 Iteration: 110 Train loss: 0.233\n",
            "\n",
            "Epoch: 0/15 Iteration: 115 Train loss: 0.136\n",
            "\n",
            "Epoch: 0/15 Iteration: 120 Train loss: 0.101\n",
            "\n",
            "Epoch: 0/15 Iteration: 125 Train loss: 0.161\n",
            "\n",
            "Epoch: 0/15 Iteration: 130 Train loss: 0.102\n",
            "\n",
            "Epoch: 0/15 Iteration: 135 Train loss: 0.157\n",
            "\n",
            "Epoch: 0/15 Iteration: 140 Train loss: 0.164\n",
            "\n",
            "Epoch: 0/15 Iteration: 145 Train loss: 0.040\n",
            "\n",
            "Epoch: 0/15 Iteration: 150 Train loss: 0.108\n",
            "\n",
            "Epoch: 0/15 Iteration: 155 Train loss: 0.047\n",
            "\n",
            "Epoch: 0/15 Iteration: 160 Train loss: 0.131\n",
            "\n",
            "Epoch: 0/15 Iteration: 165 Train loss: 0.097\n",
            "\n",
            "Epoch: 0/15 Iteration: 170 Train loss: 0.031\n",
            "\n",
            "Epoch: 0/15 Iteration: 175 Train loss: 0.079\n",
            "\n",
            "Epoch: 1/15 Iteration: 180 Train loss: 0.127\n",
            "\n",
            "Epoch: 1/15 Iteration: 185 Train loss: 0.122\n",
            "\n",
            "Epoch: 1/15 Iteration: 190 Train loss: 0.139\n",
            "\n",
            "Epoch: 1/15 Iteration: 195 Train loss: 0.129\n",
            "\n",
            "Epoch: 1/15 Iteration: 200 Train loss: 0.103\n",
            "\n",
            "Epoch: 1/15 Iteration: 205 Train loss: 0.059\n",
            "\n",
            "Epoch: 1/15 Iteration: 210 Train loss: 0.067\n",
            "\n",
            "Epoch: 1/15 Iteration: 215 Train loss: 0.064\n",
            "\n",
            "Epoch: 1/15 Iteration: 220 Train loss: 0.017\n",
            "\n",
            "Epoch: 1/15 Iteration: 225 Train loss: 0.045\n",
            "\n",
            "Epoch: 1/15 Iteration: 230 Train loss: 0.064\n",
            "\n",
            "Epoch: 1/15 Iteration: 235 Train loss: 0.026\n",
            "\n",
            "Epoch: 1/15 Iteration: 240 Train loss: 0.034\n",
            "\n",
            "Epoch: 1/15 Iteration: 245 Train loss: 0.044\n",
            "\n",
            "Epoch: 1/15 Iteration: 250 Train loss: 0.052\n",
            "\n",
            "Epoch: 1/15 Iteration: 255 Train loss: 0.032\n",
            "\n",
            "Epoch: 1/15 Iteration: 260 Train loss: 0.030\n",
            "\n",
            "Epoch: 1/15 Iteration: 265 Train loss: 0.044\n",
            "\n",
            "Epoch: 1/15 Iteration: 270 Train loss: 0.005\n",
            "\n",
            "Epoch: 1/15 Iteration: 275 Train loss: 0.007\n",
            "\n",
            "Epoch: 1/15 Iteration: 280 Train loss: 0.002\n",
            "\n",
            "Epoch: 1/15 Iteration: 285 Train loss: 0.015\n",
            "\n",
            "Epoch: 1/15 Iteration: 290 Train loss: 0.004\n",
            "\n",
            "Epoch: 1/15 Iteration: 295 Train loss: 0.045\n",
            "\n",
            "Epoch: 1/15 Iteration: 300 Train loss: 0.010\n",
            "\n",
            "Epoch: 1/15 Iteration: 305 Train loss: 0.058\n",
            "\n",
            "Epoch: 1/15 Iteration: 310 Train loss: 0.010\n",
            "\n",
            "Epoch: 1/15 Iteration: 315 Train loss: 0.009\n",
            "\n",
            "Epoch: 1/15 Iteration: 320 Train loss: 0.022\n",
            "\n",
            "Epoch: 1/15 Iteration: 325 Train loss: 0.005\n",
            "\n",
            "Epoch: 1/15 Iteration: 330 Train loss: 0.077\n",
            "\n",
            "Epoch: 1/15 Iteration: 335 Train loss: 0.031\n",
            "\n",
            "Epoch: 1/15 Iteration: 340 Train loss: 0.021\n",
            "\n",
            "Epoch: 1/15 Iteration: 345 Train loss: 0.058\n",
            "\n",
            "Epoch: 1/15 Iteration: 350 Train loss: 0.037\n",
            "\n",
            "Epoch: 2/15 Iteration: 355 Train loss: 0.005\n",
            "\n",
            "Epoch: 2/15 Iteration: 360 Train loss: 0.004\n",
            "\n",
            "Epoch: 2/15 Iteration: 365 Train loss: 0.011\n",
            "\n",
            "Epoch: 2/15 Iteration: 370 Train loss: 0.005\n",
            "\n",
            "Epoch: 2/15 Iteration: 375 Train loss: 0.040\n",
            "\n",
            "Epoch: 2/15 Iteration: 380 Train loss: 0.004\n",
            "\n",
            "Epoch: 2/15 Iteration: 385 Train loss: 0.030\n",
            "\n",
            "Epoch: 2/15 Iteration: 390 Train loss: 0.005\n",
            "\n",
            "Epoch: 2/15 Iteration: 395 Train loss: 0.030\n",
            "\n",
            "Epoch: 2/15 Iteration: 400 Train loss: 0.027\n",
            "\n",
            "Epoch: 2/15 Iteration: 405 Train loss: 0.022\n",
            "\n",
            "Epoch: 2/15 Iteration: 410 Train loss: 0.038\n",
            "\n",
            "Epoch: 2/15 Iteration: 415 Train loss: 0.008\n",
            "\n",
            "Epoch: 2/15 Iteration: 420 Train loss: 0.011\n",
            "\n",
            "Epoch: 2/15 Iteration: 425 Train loss: 0.009\n",
            "\n",
            "Epoch: 2/15 Iteration: 430 Train loss: 0.015\n",
            "\n",
            "Epoch: 2/15 Iteration: 435 Train loss: 0.024\n",
            "\n",
            "Epoch: 2/15 Iteration: 440 Train loss: 0.002\n",
            "\n",
            "Epoch: 2/15 Iteration: 445 Train loss: 0.044\n",
            "\n",
            "Epoch: 2/15 Iteration: 450 Train loss: 0.042\n",
            "\n",
            "Epoch: 2/15 Iteration: 455 Train loss: 0.002\n",
            "\n",
            "Epoch: 2/15 Iteration: 460 Train loss: 0.029\n",
            "\n",
            "Epoch: 2/15 Iteration: 465 Train loss: 0.012\n",
            "\n",
            "Epoch: 2/15 Iteration: 470 Train loss: 0.026\n",
            "\n",
            "Epoch: 2/15 Iteration: 475 Train loss: 0.031\n",
            "\n",
            "Epoch: 2/15 Iteration: 480 Train loss: 0.011\n",
            "\n",
            "Epoch: 2/15 Iteration: 485 Train loss: 0.044\n",
            "\n",
            "Epoch: 2/15 Iteration: 490 Train loss: 0.006\n",
            "\n",
            "Epoch: 2/15 Iteration: 495 Train loss: 0.002\n",
            "\n",
            "Epoch: 2/15 Iteration: 500 Train loss: 0.027\n",
            "\n",
            "Epoch: 2/15 Iteration: 505 Train loss: 0.032\n",
            "\n",
            "Epoch: 2/15 Iteration: 510 Train loss: 0.009\n",
            "\n",
            "Epoch: 2/15 Iteration: 515 Train loss: 0.002\n",
            "\n",
            "Epoch: 2/15 Iteration: 520 Train loss: 0.038\n",
            "\n",
            "Epoch: 2/15 Iteration: 525 Train loss: 0.003\n",
            "\n",
            "Epoch: 2/15 Iteration: 530 Train loss: 0.014\n",
            "\n",
            "Epoch: 3/15 Iteration: 535 Train loss: 0.005\n",
            "\n",
            "Epoch: 3/15 Iteration: 540 Train loss: 0.004\n",
            "\n",
            "Epoch: 3/15 Iteration: 545 Train loss: 0.003\n",
            "\n",
            "Epoch: 3/15 Iteration: 550 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 555 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 560 Train loss: 0.014\n",
            "\n",
            "Epoch: 3/15 Iteration: 565 Train loss: 0.004\n",
            "\n",
            "Epoch: 3/15 Iteration: 570 Train loss: 0.014\n",
            "\n",
            "Epoch: 3/15 Iteration: 575 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 580 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 585 Train loss: 0.006\n",
            "\n",
            "Epoch: 3/15 Iteration: 590 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 595 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 600 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 605 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 610 Train loss: 0.009\n",
            "\n",
            "Epoch: 3/15 Iteration: 615 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 620 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 625 Train loss: 0.009\n",
            "\n",
            "Epoch: 3/15 Iteration: 630 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 635 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 640 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 645 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 650 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 655 Train loss: 0.008\n",
            "\n",
            "Epoch: 3/15 Iteration: 660 Train loss: 0.004\n",
            "\n",
            "Epoch: 3/15 Iteration: 665 Train loss: 0.013\n",
            "\n",
            "Epoch: 3/15 Iteration: 670 Train loss: 0.014\n",
            "\n",
            "Epoch: 3/15 Iteration: 675 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 680 Train loss: 0.002\n",
            "\n",
            "Epoch: 3/15 Iteration: 685 Train loss: 0.006\n",
            "\n",
            "Epoch: 3/15 Iteration: 690 Train loss: 0.016\n",
            "\n",
            "Epoch: 3/15 Iteration: 695 Train loss: 0.048\n",
            "\n",
            "Epoch: 3/15 Iteration: 700 Train loss: 0.001\n",
            "\n",
            "Epoch: 3/15 Iteration: 705 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 710 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 715 Train loss: 0.038\n",
            "\n",
            "Epoch: 4/15 Iteration: 720 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 725 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 730 Train loss: 0.041\n",
            "\n",
            "Epoch: 4/15 Iteration: 735 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 740 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 745 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 750 Train loss: 0.005\n",
            "\n",
            "Epoch: 4/15 Iteration: 755 Train loss: 0.011\n",
            "\n",
            "Epoch: 4/15 Iteration: 760 Train loss: 0.002\n",
            "\n",
            "Epoch: 4/15 Iteration: 765 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 770 Train loss: 0.003\n",
            "\n",
            "Epoch: 4/15 Iteration: 775 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 780 Train loss: 0.005\n",
            "\n",
            "Epoch: 4/15 Iteration: 785 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 790 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 795 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 800 Train loss: 0.002\n",
            "\n",
            "Epoch: 4/15 Iteration: 805 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 810 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 815 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 820 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 825 Train loss: 0.064\n",
            "\n",
            "Epoch: 4/15 Iteration: 830 Train loss: 0.015\n",
            "\n",
            "Epoch: 4/15 Iteration: 835 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 840 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 845 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 850 Train loss: 0.003\n",
            "\n",
            "Epoch: 4/15 Iteration: 855 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 860 Train loss: 0.005\n",
            "\n",
            "Epoch: 4/15 Iteration: 865 Train loss: 0.007\n",
            "\n",
            "Epoch: 4/15 Iteration: 870 Train loss: 0.000\n",
            "\n",
            "Epoch: 4/15 Iteration: 875 Train loss: 0.001\n",
            "\n",
            "Epoch: 4/15 Iteration: 880 Train loss: 0.002\n",
            "\n",
            "Epoch: 4/15 Iteration: 885 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 890 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 895 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 900 Train loss: 0.004\n",
            "\n",
            "Epoch: 5/15 Iteration: 905 Train loss: 0.011\n",
            "\n",
            "Epoch: 5/15 Iteration: 910 Train loss: 0.002\n",
            "\n",
            "Epoch: 5/15 Iteration: 915 Train loss: 0.002\n",
            "\n",
            "Epoch: 5/15 Iteration: 920 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 925 Train loss: 0.005\n",
            "\n",
            "Epoch: 5/15 Iteration: 930 Train loss: 0.006\n",
            "\n",
            "Epoch: 5/15 Iteration: 935 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 940 Train loss: 0.002\n",
            "\n",
            "Epoch: 5/15 Iteration: 945 Train loss: 0.005\n",
            "\n",
            "Epoch: 5/15 Iteration: 950 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 955 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 960 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 965 Train loss: 0.002\n",
            "\n",
            "Epoch: 5/15 Iteration: 970 Train loss: 0.002\n",
            "\n",
            "Epoch: 5/15 Iteration: 975 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 980 Train loss: 0.013\n",
            "\n",
            "Epoch: 5/15 Iteration: 985 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 990 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 995 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1000 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1005 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1010 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1015 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1020 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1025 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1030 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1035 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1040 Train loss: 0.004\n",
            "\n",
            "Epoch: 5/15 Iteration: 1045 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1050 Train loss: 0.000\n",
            "\n",
            "Epoch: 5/15 Iteration: 1055 Train loss: 0.001\n",
            "\n",
            "Epoch: 5/15 Iteration: 1060 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1065 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1070 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1075 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1080 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1085 Train loss: 0.002\n",
            "\n",
            "Epoch: 6/15 Iteration: 1090 Train loss: 0.006\n",
            "\n",
            "Epoch: 6/15 Iteration: 1095 Train loss: 0.031\n",
            "\n",
            "Epoch: 6/15 Iteration: 1100 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1105 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1110 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1115 Train loss: 0.002\n",
            "\n",
            "Epoch: 6/15 Iteration: 1120 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1125 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1130 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1135 Train loss: 0.003\n",
            "\n",
            "Epoch: 6/15 Iteration: 1140 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1145 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1150 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1155 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1160 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1165 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1170 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1175 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1180 Train loss: 0.005\n",
            "\n",
            "Epoch: 6/15 Iteration: 1185 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1190 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1195 Train loss: 0.002\n",
            "\n",
            "Epoch: 6/15 Iteration: 1200 Train loss: 0.005\n",
            "\n",
            "Epoch: 6/15 Iteration: 1205 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1210 Train loss: 0.001\n",
            "\n",
            "Epoch: 6/15 Iteration: 1215 Train loss: 0.000\n",
            "\n",
            "Epoch: 6/15 Iteration: 1220 Train loss: 0.067\n",
            "\n",
            "Epoch: 6/15 Iteration: 1225 Train loss: 0.016\n",
            "\n",
            "Epoch: 6/15 Iteration: 1230 Train loss: 0.004\n",
            "\n",
            "Epoch: 6/15 Iteration: 1235 Train loss: 0.022\n",
            "\n",
            "Epoch: 7/15 Iteration: 1240 Train loss: 0.028\n",
            "\n",
            "Epoch: 7/15 Iteration: 1245 Train loss: 0.129\n",
            "\n",
            "Epoch: 7/15 Iteration: 1250 Train loss: 0.062\n",
            "\n",
            "Epoch: 7/15 Iteration: 1255 Train loss: 0.074\n",
            "\n",
            "Epoch: 7/15 Iteration: 1260 Train loss: 0.031\n",
            "\n",
            "Epoch: 7/15 Iteration: 1265 Train loss: 0.005\n",
            "\n",
            "Epoch: 7/15 Iteration: 1270 Train loss: 0.023\n",
            "\n",
            "Epoch: 7/15 Iteration: 1275 Train loss: 0.008\n",
            "\n",
            "Epoch: 7/15 Iteration: 1280 Train loss: 0.057\n",
            "\n",
            "Epoch: 7/15 Iteration: 1285 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1290 Train loss: 0.051\n",
            "\n",
            "Epoch: 7/15 Iteration: 1295 Train loss: 0.077\n",
            "\n",
            "Epoch: 7/15 Iteration: 1300 Train loss: 0.003\n",
            "\n",
            "Epoch: 7/15 Iteration: 1305 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1310 Train loss: 0.004\n",
            "\n",
            "Epoch: 7/15 Iteration: 1315 Train loss: 0.028\n",
            "\n",
            "Epoch: 7/15 Iteration: 1320 Train loss: 0.021\n",
            "\n",
            "Epoch: 7/15 Iteration: 1325 Train loss: 0.050\n",
            "\n",
            "Epoch: 7/15 Iteration: 1330 Train loss: 0.003\n",
            "\n",
            "Epoch: 7/15 Iteration: 1335 Train loss: 0.010\n",
            "\n",
            "Epoch: 7/15 Iteration: 1340 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1345 Train loss: 0.031\n",
            "\n",
            "Epoch: 7/15 Iteration: 1350 Train loss: 0.023\n",
            "\n",
            "Epoch: 7/15 Iteration: 1355 Train loss: 0.005\n",
            "\n",
            "Epoch: 7/15 Iteration: 1360 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1365 Train loss: 0.003\n",
            "\n",
            "Epoch: 7/15 Iteration: 1370 Train loss: 0.003\n",
            "\n",
            "Epoch: 7/15 Iteration: 1375 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1380 Train loss: 0.018\n",
            "\n",
            "Epoch: 7/15 Iteration: 1385 Train loss: 0.025\n",
            "\n",
            "Epoch: 7/15 Iteration: 1390 Train loss: 0.024\n",
            "\n",
            "Epoch: 7/15 Iteration: 1395 Train loss: 0.003\n",
            "\n",
            "Epoch: 7/15 Iteration: 1400 Train loss: 0.002\n",
            "\n",
            "Epoch: 7/15 Iteration: 1405 Train loss: 0.009\n",
            "\n",
            "Epoch: 7/15 Iteration: 1410 Train loss: 0.014\n",
            "\n",
            "Epoch: 7/15 Iteration: 1415 Train loss: 0.004\n",
            "\n",
            "Epoch: 8/15 Iteration: 1420 Train loss: 0.002\n",
            "\n",
            "Epoch: 8/15 Iteration: 1425 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1430 Train loss: 0.051\n",
            "\n",
            "Epoch: 8/15 Iteration: 1435 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1440 Train loss: 0.002\n",
            "\n",
            "Epoch: 8/15 Iteration: 1445 Train loss: 0.050\n",
            "\n",
            "Epoch: 8/15 Iteration: 1450 Train loss: 0.003\n",
            "\n",
            "Epoch: 8/15 Iteration: 1455 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1460 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1465 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1470 Train loss: 0.003\n",
            "\n",
            "Epoch: 8/15 Iteration: 1475 Train loss: 0.002\n",
            "\n",
            "Epoch: 8/15 Iteration: 1480 Train loss: 0.003\n",
            "\n",
            "Epoch: 8/15 Iteration: 1485 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1490 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1495 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1500 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1505 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1510 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1515 Train loss: 0.007\n",
            "\n",
            "Epoch: 8/15 Iteration: 1520 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1525 Train loss: 0.003\n",
            "\n",
            "Epoch: 8/15 Iteration: 1530 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1535 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1540 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1545 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1550 Train loss: 0.003\n",
            "\n",
            "Epoch: 8/15 Iteration: 1555 Train loss: 0.002\n",
            "\n",
            "Epoch: 8/15 Iteration: 1560 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1565 Train loss: 0.004\n",
            "\n",
            "Epoch: 8/15 Iteration: 1570 Train loss: 0.001\n",
            "\n",
            "Epoch: 8/15 Iteration: 1575 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1580 Train loss: 0.000\n",
            "\n",
            "Epoch: 8/15 Iteration: 1585 Train loss: 0.015\n",
            "\n",
            "Epoch: 8/15 Iteration: 1590 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1595 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1600 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1605 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1610 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1615 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1620 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1625 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1630 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1635 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1640 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1645 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1650 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1655 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1660 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1665 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1670 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1675 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1680 Train loss: 0.003\n",
            "\n",
            "Epoch: 9/15 Iteration: 1685 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1690 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1695 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1700 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1705 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1710 Train loss: 0.004\n",
            "\n",
            "Epoch: 9/15 Iteration: 1715 Train loss: 0.002\n",
            "\n",
            "Epoch: 9/15 Iteration: 1720 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1725 Train loss: 0.001\n",
            "\n",
            "Epoch: 9/15 Iteration: 1730 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1735 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1740 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1745 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1750 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1755 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1760 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1765 Train loss: 0.000\n",
            "\n",
            "Epoch: 9/15 Iteration: 1770 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1775 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1780 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1785 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1790 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1795 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1800 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1805 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1810 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1815 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1820 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1825 Train loss: 0.002\n",
            "\n",
            "Epoch: 10/15 Iteration: 1830 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1835 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1840 Train loss: 0.005\n",
            "\n",
            "Epoch: 10/15 Iteration: 1845 Train loss: 0.066\n",
            "\n",
            "Epoch: 10/15 Iteration: 1850 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1855 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1860 Train loss: 0.002\n",
            "\n",
            "Epoch: 10/15 Iteration: 1865 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1870 Train loss: 0.006\n",
            "\n",
            "Epoch: 10/15 Iteration: 1875 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1880 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1885 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1890 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1895 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1900 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1905 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1910 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1915 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1920 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1925 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1930 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1935 Train loss: 0.001\n",
            "\n",
            "Epoch: 10/15 Iteration: 1940 Train loss: 0.000\n",
            "\n",
            "Epoch: 10/15 Iteration: 1945 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1950 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1955 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1960 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1965 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1970 Train loss: 0.016\n",
            "\n",
            "Epoch: 11/15 Iteration: 1975 Train loss: 0.002\n",
            "\n",
            "Epoch: 11/15 Iteration: 1980 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1985 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1990 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 1995 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2000 Train loss: 0.001\n",
            "\n",
            "Epoch: 11/15 Iteration: 2005 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2010 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2015 Train loss: 0.001\n",
            "\n",
            "Epoch: 11/15 Iteration: 2020 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2025 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2030 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2035 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2040 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2045 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2050 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2055 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2060 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2065 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2070 Train loss: 0.001\n",
            "\n",
            "Epoch: 11/15 Iteration: 2075 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2080 Train loss: 0.006\n",
            "\n",
            "Epoch: 11/15 Iteration: 2085 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2090 Train loss: 0.001\n",
            "\n",
            "Epoch: 11/15 Iteration: 2095 Train loss: 0.001\n",
            "\n",
            "Epoch: 11/15 Iteration: 2100 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2105 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2110 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2115 Train loss: 0.000\n",
            "\n",
            "Epoch: 11/15 Iteration: 2120 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2125 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2130 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2135 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2140 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2145 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2150 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2155 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2160 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2165 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2170 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2175 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2180 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2185 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2190 Train loss: 0.004\n",
            "\n",
            "Epoch: 12/15 Iteration: 2195 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2200 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2205 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2210 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2215 Train loss: 0.018\n",
            "\n",
            "Epoch: 12/15 Iteration: 2220 Train loss: 0.016\n",
            "\n",
            "Epoch: 12/15 Iteration: 2225 Train loss: 0.012\n",
            "\n",
            "Epoch: 12/15 Iteration: 2230 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2235 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2240 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2245 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2250 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2255 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2260 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2265 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2270 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2275 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2280 Train loss: 0.010\n",
            "\n",
            "Epoch: 12/15 Iteration: 2285 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2290 Train loss: 0.036\n",
            "\n",
            "Epoch: 12/15 Iteration: 2295 Train loss: 0.000\n",
            "\n",
            "Epoch: 12/15 Iteration: 2300 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2305 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2310 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2315 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2320 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2325 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2330 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2335 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2340 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2345 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2350 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2355 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2360 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2365 Train loss: 0.001\n",
            "\n",
            "Epoch: 13/15 Iteration: 2370 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2375 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2380 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2385 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2390 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2395 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2400 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2405 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2410 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2415 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2420 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2425 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2430 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2435 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2440 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2445 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2450 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2455 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2460 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2465 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2470 Train loss: 0.000\n",
            "\n",
            "Epoch: 13/15 Iteration: 2475 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2480 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2485 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2490 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2495 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2500 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2505 Train loss: 0.002\n",
            "\n",
            "Epoch: 14/15 Iteration: 2510 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2515 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2520 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2525 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2530 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2535 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2540 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2545 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2550 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2555 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2560 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2565 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2570 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2575 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2580 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2585 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2590 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2595 Train loss: 0.006\n",
            "\n",
            "Epoch: 14/15 Iteration: 2600 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2605 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2610 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2615 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2620 Train loss: 0.002\n",
            "\n",
            "Epoch: 14/15 Iteration: 2625 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2630 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2635 Train loss: 0.015\n",
            "\n",
            "Epoch: 14/15 Iteration: 2640 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2645 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2650 Train loss: 0.000\n",
            "\n",
            "Epoch: 14/15 Iteration: 2655 Train loss: 0.000\n",
            "\n",
            "Time Elapsed: \t\t5804.137 s\n",
            "In hours & minutes: \t 1:36:44.136981\n",
            "Starting Validation Set...\n",
            "\n",
            "-----------------Testing Validation Set----------------\n",
            "\n",
            "Test accuracy: 0.981\n",
            "F1 Score: 0.981\n",
            "Recall: 0.981\n",
            "Precision: 0.981\n",
            "\n",
            "Time Elapsed: \t\t14.857 s\n",
            "\n",
            "Starting Testing Set...\n",
            "\n",
            "-----------------Testing Test Set----------------------\n",
            "\n",
            "Test accuracy: 0.981\n",
            "F1 Score: 0.981\n",
            "Recall: 0.981\n",
            "Precision: 0.982\n",
            "\n",
            "Time Elapsed: \t\t14.955 s\n",
            "\n",
            "\n",
            "=======================================================\n",
            "\n",
            "\n",
            "Total Time Elapsed: \t\t5841.156 s\n",
            "\n",
            "In hours & minutes: \t 1:37:21.156361\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My **RNN model acheived an accuracy, F1 score, and recall of 0.981, and a recall of 0.982**. This model's performance is roughly identical to the performance of the RNN model used in the article for the Enron Spam dataset."
      ],
      "metadata": {
        "id": "zR7eoGoOrl0D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Analysis and Conclusion**"
      ],
      "metadata": {
        "id": "ZliV2QFI57dl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Results**:\n",
        "| Model                             | Accuracy | F1 Score| Recall|Precision     \n",
        "| :-------------------------------:  | :-------:| :-----: | :----:| :------:\n",
        "| **Random Forest**                  | 0.981    | 0.981   | 0.981 | 0.982\n",
        "| **Recurrent Neural Network (LSTM)**| 0.981    | 0.981   | 0.981 | 0.982\n",
        "\n",
        "**Both models obtained identical accuracy, F1 score, recall, and precision metrics.** The accuracy of both models was 98.1%, which is very high."
      ],
      "metadata": {
        "id": "_pL__3B66Ek6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Analysis**\n",
        "**When considering the time taken train, the Random Forest model outperformed the RNN (LSTM) model**. The Random Forest model took 1 minute and 37 seconds to run, while the RNN model took around 1 hour and 37 minutes. So the RNN model took rougly 100 times the amount of time to obtain the exact same results."
      ],
      "metadata": {
        "id": "q70YKp8U_iio"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Conclusion**\n",
        "\n",
        "Moving forward, I first want to **explore other ensemble methods**, specifically boosting methods like XGBoost/AdaBoost or even CatBoost. Ensemble methods are less complex and faster to train compared to deep learning methods, so they are preferred if they are able to achieve comparable results.\n",
        "\n",
        "Next, I want to **experiment further with deep learning models**. I first want to tune the hyperparameters for the RNN model I implemented, as well as look into adding additional hidden layers to improve its results. I am also considering implementing a Convolutional Neural Network (CNN) as they are faster than RNNs and highly proficient in extracting meaningful patterns, which is ideal for spam detection.\n",
        "\n",
        "Finally, I want to **experiment with other forms of feature extraction**. So far I have tried TF-IDF Vectorization and a modified Bag of Words model, but I also want to consider Count Vectorization, Word2Vec, and possibly Word Embeddings."
      ],
      "metadata": {
        "id": "2abWFfxOhb8y"
      }
    }
  ]
}