Prototyping&Scaling.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "V100",
      "toc_visible": true,
      "machine_shape": "hm",
      "mount_file_id": "1DoAhS_GXVQned7Zk_Q-WMPaGHEWkHhP0",
      "authorship_tag": "ABX9TyNSu+2G3CHOgkHRlG00+N2I",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/anastasiaarsky/ML_Capstone/blob/main/Prototyping%26Scaling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Goal\n",
        "\n",
        "**Part 1**: Create two working implementations of a Spam Classification prototype.\n",
        "\n",
        "1. One prototype will use a traditional *Machine Learning* technique, specifically a **Random Forest**, which I tuned previously.\n",
        "  - It will be using **word level TF-IDF** as this previously resulted in the highest accuracy.\n",
        "\n",
        "2. The other model will use a *Deep Learning* technique called a **Bidirectional Long-Term Short Memory** (BiLSTM), which is a type of RNN (Recurrent Neural Network).\n",
        "  - Whereas simple RNNs only learn from the immediately preceding data, LSTMs keep track not just the immediately preceding data, but the earlier data too. This allows them to learn from data that is far away from its current position. Furthermore, the typical state in an LSTM relies on the past and the present events. However, there can be situations where a prediction depends on the past, present, and future events. So in the context of email spam detection, whether or not an email is a spam can depend on future words in the email.\n",
        "  - I will also analyze two feature extraction methods: a **Continuous Bag of Words model**, and the **pretrained GloVe Word Embeddings**. Leveraging pre-trained word embeddings is a form *deep transfer learning*, and is commonly used in NLP.\n",
        "\n",
        "**Part 2**: Scale my prototype to handle a larger volume of data.\n",
        "1. To do this, I decided to implement a **Convolutional Neural Network** (CNN) in place of a BiLSTM.\n",
        "  - CNNs are built for processing images, allowing them the capability of handling large amounts of data. This allows them to be considerably faster than BiLSTMs. They are also highly proficient in extracting meaningful patterns, which is ideal for spam detection.\n",
        "  - Once again, I will be analyzing a **Continuous Bag of Words model** and the **pretrained GloVe Word Embeddings** for my feature extraction. However, I will also be using an additional *deep transfer learning* feature extraction approach by leveraging **pretrained FastText word embeddings**.\n",
        "\n"
      ],
      "metadata": {
        "id": "MVk7qPmBkEuv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Table of Contents\n",
        "1.  [Imports and Initialization](#scrollTo=H0QcryyIz6aW)   \n",
        "2. [Part 1: Prototyping](#scrollTo=y5DBfDmMwWhr)   \n",
        "    1. [ML Prototype](#scrollTo=MjfkBLPEj206)  \n",
        "      1. [Feature Extraction](#scrollTo=vB_gYDafmthj)  \n",
        "          1. [TF-IDF](#scrollTo=971icSjk0pFS)\n",
        "      1. [Random Forest](#scrollTo=ChQAajXLnHPm)  \n",
        "          1. [Training and Evaluation](#scrollTo=Vo9YW_HWzWwm)\n",
        "          2. [Final Model](#scrollTo=QGWt5rZBnQR9)\n",
        "    2. [DL Prototype](#scrollTo=64WF95zpoa8C)\n",
        "      1. [Data Preparation & Feature Extraction](#scrollTo=4OCJvrlasOoB)  \n",
        "          1. [GloVe word embeddings](#scrollTo=yTUwR7U7glGB)\n",
        "          2. [Continuous Bag of Words model](#scrollTo=xL-4H9U_gqFL)   \n",
        "      2. [BiLSTM with GloVe word embeddings](#scrollTo=mMPBJNKAjaZc)  \n",
        "          1. [Preliminary Model](#scrollTo=8pCwZBcNnOpT)\n",
        "          2. [Larger Learning Rate](#scrollTo=o1Tf8X1Dp-lu)\n",
        "          3. [More Epochs](#scrollTo=nYkG7sWxqGpk)\n",
        "          4. [Add an Additional BiLSTM Layer](#scrollTo=o3kqehagotpf)\n",
        "          5. [Final Model](#scrollTo=VTZoh5y3qO5e)\n",
        "      3. [BiLSTM with a Continuous Bag of Words model](#scrollTo=3hJQfUNI8tcJ)  \n",
        "          1. [Preliminary Model](#scrollTo=u-H5hwWq8uQe)\n",
        "          2. [Hyperparameter Tuning](#scrollTo=AKjgqcKm8udk)\n",
        "          3. [Final Model](#scrollTo=OZqEOxVC8urr)\n",
        "3. [Part 2: Scaling](#scrollTo=Dq8YHLOKq4Xj)   \n",
        "    1. [DL Prototype](#scrollTo=HZxmi_wyxy3u)  \n",
        "      1. [Feature Extraction](#scrollTo=GRF6lHBQVvFu)  \n",
        "          1. [FastText word embeddings](#scrollTo=F3FGPgliVxN0)\n",
        "      2. [CNN with GloVe word embeddings](#scrollTo=fXEWmjc71FJw)  \n",
        "          1. [Preliminary Model](#scrollTo=XQWoQ2xYx4pj)\n",
        "          2. [Exploring Dropout Layers](#scrollTo=ZVPtsYqoAVsh)\n",
        "          3. [Hyperparameter Tuning](#scrollTo=vdObpYr3Avks)\n",
        "          4. [Final Model](#scrollTo=44i7hGzfMzcq)\n",
        "      3. [CNN with a Continuous Bag of Words model](#scrollTo=gzIaqt-HyVZR)  \n",
        "          1. [Preliminary Model](#scrollTo=0FiGzUsckEeR)\n",
        "          2. [Hyperparameter Tuning](#scrollTo=Bcib1Jrtsg5O)\n",
        "          3. [Final Model](#scrollTo=_MjezVEDmCgE)\n",
        "      4. [CNN with FastText word embeddings](#scrollTo=GN9dr4nlaX4O)\n",
        "          1. [Preliminary Model](#scrollTo=H75Y8UfpaYt2)\n",
        "          2. [Adding a Dropout Layer](#scrollTo=qaq8JYWtmuAa)\n",
        "          2. [Adding an Additional Convolutional Layer](#scrollTo=RBFnb2kPnsBd)\n",
        "          3. [Final Model](#scrollTo=c-Pmp1zsaZGB)  \n",
        "4. [Summary](#scrollTo=YpZZCfSdFsOw)\n",
        "      1. [Prototyping Analysis](#scrollTo=BGyX7z2gGDSj)\n",
        "      2. [Scaling Analysis](#scrollTo=6JNd4OtBGDhE)\n"
      ],
      "metadata": {
        "id": "2qWHZNzGyqmn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Imports and Initialization"
      ],
      "metadata": {
        "id": "H0QcryyIz6aW"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xoEKigX_ir6g",
        "outputId": "8ad9b16f-4f2f-4915-c3a0-b29b5a57f609"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import re\n",
        "import time\n",
        "import zipfile\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "\n",
        "from keras.callbacks import EarlyStopping, ReduceLROnPlateau\n",
        "from keras.layers import BatchNormalization, Bidirectional, Conv1D, Dense\n",
        "from keras.layers import Dropout, Embedding, Flatten, GlobalMaxPooling1D, LSTM\n",
        "from keras.layers import MaxPooling1D\n",
        "from keras.models import Sequential\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.utils import pad_sequences\n",
        "from keras.wrappers.scikit_learn import KerasClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.model_selection import GridSearchCV, train_test_split\n",
        "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
        "from sklearn.metrics import f1_score, precision_score, recall_score"
      ],
      "metadata": {
        "id": "MLT_UvDNjlqO"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load clean data into a pandas dataframe\n",
        "DATA_PATH = \"/content/drive/My Drive/UCSD Machine Learning Engineering Bootcamp/Capstone Project/\"\n",
        "df = pd.read_csv(DATA_PATH + 'CleanData.csv')"
      ],
      "metadata": {
        "id": "rJ2U8bhnkfQp"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Split the subset into training and testing datasets\n",
        "X_train_all, X_test, y_train_all, y_test = train_test_split(df.Clean_Text,\n",
        "                                                            df.Label, test_size=0.3,\n",
        "                                                            shuffle=True, random_state=1)\n",
        "print(\"Length of training set:\", len(X_train_all))\n",
        "print(\"Length of testing set:\", len(X_test))"
      ],
      "metadata": {
        "id": "JAOl6XFyklTF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "36bb6456-9eee-4e11-dde1-635ceafff42d"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Length of training set: 27834\n",
            "Length of testing set: 11929\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Part 1: Prototyping"
      ],
      "metadata": {
        "id": "y5DBfDmMwWhr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ML Prototype\n",
        "\n",
        "Random Forest Classifier using TF-IDF for feature extraction."
      ],
      "metadata": {
        "id": "MjfkBLPEj206"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Feature Extraction"
      ],
      "metadata": {
        "id": "vB_gYDafmthj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### TF-IDF"
      ],
      "metadata": {
        "id": "971icSjk0pFS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Split the training dataset into training and validating datasets\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_train_all, y_train_all,\n",
        "                                                  shuffle=True, random_state=1)\n",
        "print(\"Length of training set:\", len(X_train))\n",
        "print(\"Length of validation set:\", len(X_val))"
      ],
      "metadata": {
        "id": "q_gJ4tlRlFXt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a10347a4-e354-4ff8-d0ea-9e318c963a0c"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Length of training set: 20875\n",
            "Length of validation set: 6959\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Word_level TF-IDF\n",
        "vec = TfidfVectorizer()\n",
        "vec.fit(X_train_all)\n",
        "X_train_vec = vec.transform(X_train)\n",
        "X_val_vec = vec.transform(X_val)\n",
        "print(\"Shape of Vectorized Training Set: {}, \\nShape of Vectorized Validation Set: {}\"\n",
        "              .format(X_train_vec.shape, X_val_vec.shape))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "whIlnpKpkfX7",
        "outputId": "6420311d-38cb-4982-cc0b-1654397610e2"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of Vectorized Training Set: (20875, 142617), \n",
            "Shape of Vectorized Validation Set: (6959, 142617)\n",
            "CPU times: user 7.49 s, sys: 75.6 ms, total: 7.56 s\n",
            "Wall time: 7.61 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Random Forest"
      ],
      "metadata": {
        "id": "ChQAajXLnHPm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Training and Evaluation"
      ],
      "metadata": {
        "id": "Vo9YW_HWzWwm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to print metrics results\n",
        "def calculate_metrics(y_true, y_pred):\n",
        "  accuracy = accuracy_score(y_true, y_pred)\n",
        "  f1 = f1_score(y_true, y_pred, average='macro')\n",
        "  recall = recall_score(y_true, y_pred, average='macro')\n",
        "  precision = precision_score(y_true, y_pred, average='macro')\n",
        "\n",
        "  print(\"Random Forest, Word Level TF-IDF Results:\")\n",
        "  print(\"Test accuracy: {:.3f}\".format(accuracy*100))\n",
        "  print(\"F1 Score: {:.3f}\".format(f1*100))\n",
        "  print(\"Recall: {:.3f}\".format(recall*100))\n",
        "  print(\"Precision: {:.3f}\".format(precision*100))"
      ],
      "metadata": {
        "id": "FhNZn6QkoUC6"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Training Results of RF on Word Level TF-IDF Vectors\n",
        "# (using the training and validation sets)\n",
        "\n",
        "# use the default Random Forest Classifier since RandomizedSearchCV\n",
        "# produced worse results compared to the default\n",
        "classifier = RandomForestClassifier()\n",
        "\n",
        "# fit the training dataset on the classifier\n",
        "classifier.fit(X_train_vec, y_train)\n",
        "\n",
        "# predict the labels on validation dataset\n",
        "y_pred = classifier.predict(X_val_vec)\n",
        "\n",
        "# print metrics results\n",
        "calculate_metrics(y_val, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JABoGOGVkfer",
        "outputId": "b74cf77e-ea9b-4cc3-850c-f149b70293eb"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Random Forest, Word Level TF-IDF Results:\n",
            "Test accuracy: 98.276\n",
            "F1 Score: 98.274\n",
            "Recall: 98.282\n",
            "Precision: 98.267\n",
            "CPU times: user 1min 48s, sys: 200 ms, total: 1min 48s\n",
            "Wall time: 2min\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "QGWt5rZBnQR9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Perform TF-IDF Vectorization on the whole training set (training + validation),\n",
        "# as well as the testing set\n",
        "# Note that the TF-IDF vector was only fit on the training set and not the testing\n",
        "# set to avoid data leakage\n",
        "X_train_all_vec = vec.transform(X_train_all)\n",
        "X_test_vec = vec.transform(X_test)\n",
        "print(\"Shape of Vectorized Training Set: {}, \\nShape of Vectorized Testing Set: {}\"\n",
        "              .format(X_train_all_vec.shape, X_test_vec.shape))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LWYqcniKmxM7",
        "outputId": "01d4ba19-2ca4-4319-bd98-1f33e5908239"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of Vectorized Training Set: (27834, 142617), \n",
            "Shape of Vectorized Testing Set: (11929, 142617)\n",
            "CPU times: user 7.27 s, sys: 26.3 ms, total: 7.3 s\n",
            "Wall time: 8.26 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Testing Results of RF on Word Level TF-IDF Vectors\n",
        "# (using the entire training set and the testing set)\n",
        "\n",
        "classifier = RandomForestClassifier()\n",
        "\n",
        "# fit the whole training dataset on the classifier\n",
        "classifier.fit(X_train_all_vec, y_train_all)\n",
        "\n",
        "# predict the labels on testing dataset\n",
        "y_pred = classifier.predict(X_test_vec)\n",
        "\n",
        "# print metrics results\n",
        "calculate_metrics(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "id_xox81mgS4",
        "outputId": "98b81af1-a017-448a-d993-1cf21e6845c2"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Random Forest, Word Level TF-IDF Results:\n",
            "Test accuracy: 98.265\n",
            "F1 Score: 98.262\n",
            "Recall: 98.270\n",
            "Precision: 98.255\n",
            "CPU times: user 1min 16s, sys: 147 ms, total: 1min 16s\n",
            "Wall time: 1min 20s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Print classificaiton report & confusion matrix\n",
        "print(classification_report(y_test, y_pred))\n",
        "pd.DataFrame(confusion_matrix(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "id": "jCnOKTRLzCsb",
        "outputId": "3684ae92-6176-4039-d9c3-362004c26ea0"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         0.0       0.99      0.98      0.98      6221\n",
            "         1.0       0.98      0.98      0.98      5708\n",
            "\n",
            "    accuracy                           0.98     11929\n",
            "   macro avg       0.98      0.98      0.98     11929\n",
            "weighted avg       0.98      0.98      0.98     11929\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      0     1\n",
              "0  6106   115\n",
              "1    92  5616"
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-9ae6f3e1-b428-4ab2-86d3-2cd7f3d4c88e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6106</td>\n",
              "      <td>115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>92</td>\n",
              "      <td>5616</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9ae6f3e1-b428-4ab2-86d3-2cd7f3d4c88e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-11762dc0-51ca-499f-88dd-b6571346380a\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-11762dc0-51ca-499f-88dd-b6571346380a')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-11762dc0-51ca-499f-88dd-b6571346380a button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9ae6f3e1-b428-4ab2-86d3-2cd7f3d4c88e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9ae6f3e1-b428-4ab2-86d3-2cd7f3d4c88e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My Random Forest model using TF-IDF achieved a testing accuracy of 98.27%, and took 1 min and 16 s of CPU time to train."
      ],
      "metadata": {
        "id": "QLwmgGo7n7xk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## DL Prototype\n",
        "\n",
        "I will first implement two different types of feature extraction (pretrained GloVe word embeddings & a continous Bag of Words model), and then compare their results with a Bidirectional LSTM model."
      ],
      "metadata": {
        "id": "64WF95zpoa8C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Preparation & Feature Extraction"
      ],
      "metadata": {
        "id": "4OCJvrlasOoB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Ensure that punctuation is recognized as its own token\n",
        "\n",
        "to_tokenize = '.,:;!?$'\n",
        "regex_pat = re.compile(r'(['+to_tokenize+'])', flags=re.IGNORECASE)\n",
        "\n",
        "X_train_all = X_train_all.str.replace(regex_pat, r' \\1 ', regex=True)\n",
        "X_test = X_test.str.replace(regex_pat, r' \\1 ', regex=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Db9IKufsr-aT",
        "outputId": "fe0171b2-7970-401f-a495-ba3c0e8ab928"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1.86 s, sys: 146 µs, total: 1.86 s\n",
            "Wall time: 1.86 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Tokenizer parameters\n",
        "\n",
        "to_exclude = '\"#%&()*+-/:;<=>@[\\\\]^_`{|}~\\t\\n'\n",
        "oov_token = \"<OOV>\"\n",
        "\n",
        "# vocab_size parameter -> number of unique works in the training set\n",
        "unique_words = set()\n",
        "X_train_all.str.lower().str.split(\" \").apply(unique_words.update)\n",
        "vocab_size = len(unique_words)\n",
        "print(\"Vocab Size: {}\".format(vocab_size))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L3jqe4Skurm7",
        "outputId": "57d70dc3-a975-4eab-f790-784c4ef8798c"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vocab Size: 174294\n",
            "CPU times: user 729 ms, sys: 147 ms, total: 876 ms\n",
            "Wall time: 873 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Fit the Tokenizer onto the Training Data\n",
        "\n",
        "tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_token, filters=to_exclude)\n",
        "tokenizer.fit_on_texts(X_train_all)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2KtxbNvszMKm",
        "outputId": "400f4448-2e17-49e8-cdc0-909993b88902"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 3.37 s, sys: 0 ns, total: 3.37 s\n",
            "Wall time: 3.37 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Convert the Training and Testing sets to sequences\n",
        "\n",
        "X_train_all_sequences = tokenizer.texts_to_sequences(X_train_all)\n",
        "X_test_sequences = tokenizer.texts_to_sequences(X_test)\n",
        "\n",
        "print(\"Length of training set:\", len(X_train_all_sequences))\n",
        "print(\"Length of testing set:\", len(X_test_sequences))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hkKxVmDdwii9",
        "outputId": "a6257a6c-fa85-49d7-b62d-245b487754ef"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Length of training set: 27834\n",
            "Length of testing set: 11929\n",
            "CPU times: user 3.53 s, sys: 0 ns, total: 3.53 s\n",
            "Wall time: 3.52 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### GloVe words embeddings"
      ],
      "metadata": {
        "id": "yTUwR7U7glGB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Pad the sequences so they are all the same length (max length must be 100 to\n",
        "# use GloVe)\n",
        "\n",
        "max_length = 100\n",
        "padding_type = \"post\"\n",
        "trunction_type = \"post\"\n",
        "\n",
        "X_train_all_glove = pad_sequences(X_train_all_sequences, maxlen=max_length,\n",
        "                                   padding=padding_type, truncating=trunction_type)\n",
        "X_test_glove = pad_sequences(X_test_sequences, maxlen=max_length,\n",
        "                               padding=padding_type, truncating=trunction_type)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vFMICs6ywuhm",
        "outputId": "26309290-61b5-4761-faf9-2d0961373390"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 319 ms, sys: 35 ms, total: 354 ms\n",
            "Wall time: 364 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Split the training dataset into training and validation datasets\n",
        "X_train_glove, X_val_glove, y_train_glove, y_val_glove = train_test_split(X_train_all_glove,\n",
        "                                                                y_train_all,\n",
        "                                                                random_state=1)\n",
        "\n",
        "print(\"Shape of training set:\", X_train_glove.shape)\n",
        "print(\"Shape of validation set:\", X_val_glove.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "flbmlYHF0oZ0",
        "outputId": "39e8d4c1-cff2-4ff9-b2c8-bd8b1dc340cd"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of training set: (20875, 100)\n",
            "Shape of validation set: (6959, 100)\n",
            "CPU times: user 11.1 ms, sys: 1.03 ms, total: 12.1 ms\n",
            "Wall time: 28.7 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Download GloVe word embeddings\n",
        "\n",
        "!wget --no-check-certificate \\\n",
        "http://nlp.stanford.edu/data/glove.6B.zip \\\n",
        "-O /tmp/glove.6B.zip\n",
        "\n",
        "with zipfile.ZipFile('/tmp/glove.6B.zip', 'r') as zip_ref:\n",
        "    zip_ref.extractall('/tmp/glove')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WMfDhNBLwuk-",
        "outputId": "66c659f9-a4cc-42df-881f-e83ecc9ca9f7"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-08-08 21:33:23--  http://nlp.stanford.edu/data/glove.6B.zip\n",
            "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n",
            "--2023-08-08 21:33:23--  https://nlp.stanford.edu/data/glove.6B.zip\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
            "--2023-08-08 21:33:23--  https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘/tmp/glove.6B.zip’\n",
            "\n",
            "/tmp/glove.6B.zip   100%[===================>] 822.24M  5.02MB/s    in 2m 39s  \n",
            "\n",
            "2023-08-08 21:36:02 (5.18 MB/s) - ‘/tmp/glove.6B.zip’ saved [862182613/862182613]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Copy GloVe word embeddings into a dictionary\n",
        "\n",
        "glove_embeddings_index = {}\n",
        "f = open('/tmp/glove/glove.6B.100d.txt')\n",
        "for line in f:\n",
        "    values = line.split()\n",
        "    word = values[0]\n",
        "    coefs = np.asarray(values[1:], dtype='float32')\n",
        "    glove_embeddings_index[word] = coefs\n",
        "f.close()\n",
        "\n",
        "print(\"Number of word vectors: {}\".format(len(glove_embeddings_index)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xu0QQfJk1q8g",
        "outputId": "01a1cabb-72c6-46c5-ff65-39306906c838"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of word vectors: 400000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Get the embedding for every word in the training/validation sets\n",
        "glove_embeddings = np.zeros((vocab_size+1, max_length))\n",
        "\n",
        "for i, word in enumerate(unique_words):\n",
        "    embedding_vector = glove_embeddings_index.get(word)\n",
        "    # words not found in embedding index will be all-zeros\n",
        "    if embedding_vector is not None:\n",
        "        glove_embeddings[i] = embedding_vector"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "izcUP08618e3",
        "outputId": "aa3a806d-4c77-4f01-8ce9-3472f3d50bd0"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 200 ms, sys: 58.3 ms, total: 258 ms\n",
            "Wall time: 262 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the embedding layer\n",
        "embedding_layer_glove = Embedding(input_dim=vocab_size + 1,\n",
        "                            output_dim=max_length,\n",
        "                            weights=[glove_embeddings],\n",
        "                            input_length=max_length,\n",
        "                            trainable=False)"
      ],
      "metadata": {
        "id": "FATX9xSD6-bq"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Continuous Bag of Words model"
      ],
      "metadata": {
        "id": "xL-4H9U_gqFL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Pad the sequences so they are all the same length\n",
        "\n",
        "max_length = 300\n",
        "padding_type = \"post\"\n",
        "trunction_type = \"post\"\n",
        "\n",
        "X_train_all_bow = pad_sequences(X_train_all_sequences, maxlen=max_length,\n",
        "                                   padding=padding_type, truncating=trunction_type)\n",
        "X_test_bow = pad_sequences(X_test_sequences, maxlen=max_length,\n",
        "                               padding=padding_type, truncating=trunction_type)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a2AdEBxWh2F5",
        "outputId": "cd0a2474-c358-49fd-f521-e8842c49533f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 320 ms, sys: 1.15 ms, total: 321 ms\n",
            "Wall time: 319 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Split the training dataset into training and validation datasets\n",
        "X_train_bow, X_val_bow, y_train_bow, y_val_bow = train_test_split(X_train_all_bow,\n",
        "                                                                y_train_all,\n",
        "                                                                random_state=1)\n",
        "\n",
        "print(\"Shape of training set:\", X_train_bow.shape)\n",
        "print(\"Shape of validation set:\", X_val_bow.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R-WYRI9Jh2Q0",
        "outputId": "7500bb29-cd9e-48bc-bda7-e4223e773cea"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of training set: (20875, 300)\n",
            "Shape of validation set: (6959, 300)\n",
            "CPU times: user 12.1 ms, sys: 8.01 ms, total: 20.1 ms\n",
            "Wall time: 19.5 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embedding_layer_bow = Embedding(input_dim=vocab_size + 1,\n",
        "                            output_dim=max_length,\n",
        "                            input_length=max_length,\n",
        "                            trainable=False)"
      ],
      "metadata": {
        "id": "q2yikMEbgv5H"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### BiLSTM with GloVe word embeddings"
      ],
      "metadata": {
        "id": "mMPBJNKAjaZc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Preliminary Model"
      ],
      "metadata": {
        "id": "8pCwZBcNnOpT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Bidirectional(LSTM(128, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(1e-4),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "earlystop3 = EarlyStopping(monitor='val_loss', patience=3)\n",
        "\n",
        "history = rnn_model.fit(X_train_glove, y_train_glove, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_glove, y_val_glove))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RZO2ORlt91T1",
        "outputId": "ebbdad9b-cfdb-4263-94ab-f7cf781e0d45"
      },
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 29s 38ms/step - loss: 0.5400 - accuracy: 0.7263 - val_loss: 0.4020 - val_accuracy: 0.8197\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 21s 32ms/step - loss: 0.4002 - accuracy: 0.8240 - val_loss: 0.3573 - val_accuracy: 0.8444\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.3521 - accuracy: 0.8499 - val_loss: 0.3194 - val_accuracy: 0.8595\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.3182 - accuracy: 0.8662 - val_loss: 0.2934 - val_accuracy: 0.8731\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 15s 24ms/step - loss: 0.2847 - accuracy: 0.8846 - val_loss: 0.2555 - val_accuracy: 0.8906\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.2560 - accuracy: 0.8965 - val_loss: 0.2531 - val_accuracy: 0.8945\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 14s 21ms/step - loss: 0.2344 - accuracy: 0.9079 - val_loss: 0.2156 - val_accuracy: 0.9123\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 11s 18ms/step - loss: 0.2074 - accuracy: 0.9196 - val_loss: 0.2663 - val_accuracy: 0.8858\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1964 - accuracy: 0.9235 - val_loss: 0.1994 - val_accuracy: 0.9192\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1846 - accuracy: 0.9318 - val_loss: 0.2096 - val_accuracy: 0.9156\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1699 - accuracy: 0.9366 - val_loss: 0.1875 - val_accuracy: 0.9277\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1643 - accuracy: 0.9386 - val_loss: 0.1842 - val_accuracy: 0.9256\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1473 - accuracy: 0.9457 - val_loss: 0.1800 - val_accuracy: 0.9329\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1413 - accuracy: 0.9485 - val_loss: 0.1719 - val_accuracy: 0.9317\n",
            "Epoch 15/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1371 - accuracy: 0.9496 - val_loss: 0.1667 - val_accuracy: 0.9352\n",
            "Epoch 16/20\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.1314 - accuracy: 0.9518 - val_loss: 0.1844 - val_accuracy: 0.9299\n",
            "Epoch 17/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.1230 - accuracy: 0.9577 - val_loss: 0.1657 - val_accuracy: 0.9348\n",
            "Epoch 18/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1198 - accuracy: 0.9566 - val_loss: 0.1577 - val_accuracy: 0.9385\n",
            "Epoch 19/20\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.1112 - accuracy: 0.9606 - val_loss: 0.1511 - val_accuracy: 0.9405\n",
            "Epoch 20/20\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.1059 - accuracy: 0.9632 - val_loss: 0.1666 - val_accuracy: 0.9352\n",
            "Number of epochs run: 20\n",
            "CPU times: user 4min 8s, sys: 11.8 s, total: 4min 20s\n",
            "Wall time: 4min 25s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tJxS8m5H91cp",
        "outputId": "72493d76-a760-4fc1-990c-719c1a42492d"
      },
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 3s 9ms/step - loss: 0.1725 - accuracy: 0.9363\n",
            "Testing Accuracy is 93.62897276878357 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Larger Learning Rate"
      ],
      "metadata": {
        "id": "o1Tf8X1Dp-lu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Bidirectional(LSTM(64, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-4),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = rnn_model.fit(X_train_glove, y_train_glove, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_glove, y_val_glove),\n",
        "                        validation_steps=30)\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bfw094p5CcvB",
        "outputId": "a4d28bbd-1154-480b-ef23-217c86b2db70"
      },
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 23s 26ms/step - loss: 0.5075 - accuracy: 0.7407 - val_loss: 0.3579 - val_accuracy: 0.8435\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.3513 - accuracy: 0.8496 - val_loss: 0.3021 - val_accuracy: 0.8717\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.2900 - accuracy: 0.8797 - val_loss: 0.2434 - val_accuracy: 0.8978\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.2466 - accuracy: 0.9003 - val_loss: 0.2130 - val_accuracy: 0.9138\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.2105 - accuracy: 0.9194 - val_loss: 0.2042 - val_accuracy: 0.9164\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1823 - accuracy: 0.9325 - val_loss: 0.1808 - val_accuracy: 0.9251\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1629 - accuracy: 0.9403 - val_loss: 0.1751 - val_accuracy: 0.9315\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1509 - accuracy: 0.9435 - val_loss: 0.1739 - val_accuracy: 0.9325\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1362 - accuracy: 0.9501 - val_loss: 0.1854 - val_accuracy: 0.9264\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1266 - accuracy: 0.9530 - val_loss: 0.1682 - val_accuracy: 0.9362\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.1145 - accuracy: 0.9588 - val_loss: 0.1553 - val_accuracy: 0.9412\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 13s 19ms/step - loss: 0.1065 - accuracy: 0.9611 - val_loss: 0.1603 - val_accuracy: 0.9398\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0972 - accuracy: 0.9659 - val_loss: 0.1603 - val_accuracy: 0.9419\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0922 - accuracy: 0.9683 - val_loss: 0.1473 - val_accuracy: 0.9465\n",
            "Epoch 15/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0863 - accuracy: 0.9691 - val_loss: 0.1619 - val_accuracy: 0.9468\n",
            "Epoch 16/20\n",
            "653/653 [==============================] - 8s 13ms/step - loss: 0.0855 - accuracy: 0.9697 - val_loss: 0.1492 - val_accuracy: 0.9463\n",
            "Epoch 17/20\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0756 - accuracy: 0.9745 - val_loss: 0.1686 - val_accuracy: 0.9445\n",
            "Epoch 18/20\n",
            "653/653 [==============================] - 17s 27ms/step - loss: 0.0720 - accuracy: 0.9755 - val_loss: 0.1572 - val_accuracy: 0.9470\n",
            "Epoch 19/20\n",
            "653/653 [==============================] - 19s 29ms/step - loss: 0.0678 - accuracy: 0.9770 - val_loss: 0.1613 - val_accuracy: 0.9441\n",
            "Epoch 20/20\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0640 - accuracy: 0.9782 - val_loss: 0.1594 - val_accuracy: 0.9460\n",
            "Number of epochs run: 20\n",
            "CPU times: user 3min 40s, sys: 10.2 s, total: 3min 50s\n",
            "Wall time: 3min 47s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cf1TBa0gCdVt",
        "outputId": "6204def7-9649-4813-a6a8-9da7b7c395e8"
      },
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 2s 6ms/step - loss: 0.1666 - accuracy: 0.9448\n",
            "Testing Accuracy is 94.47564482688904 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### More Epochs"
      ],
      "metadata": {
        "id": "nYkG7sWxqGpk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Bidirectional(LSTM(64, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 60 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = rnn_model.fit(X_train_glove, y_train_glove, epochs=60,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_glove, y_val_glove),\n",
        "                        validation_steps=30)\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WZuCXdEIE80X",
        "outputId": "401efa9c-bf6c-482c-8892-6b87d02288d5"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/60\n",
            "653/653 [==============================] - 26s 27ms/step - loss: 0.3380 - accuracy: 0.8501 - val_loss: 0.1998 - val_accuracy: 0.9195\n",
            "Epoch 2/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.1910 - accuracy: 0.9267 - val_loss: 0.1672 - val_accuracy: 0.9292\n",
            "Epoch 3/60\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.1389 - accuracy: 0.9493 - val_loss: 0.1336 - val_accuracy: 0.9471\n",
            "Epoch 4/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1153 - accuracy: 0.9556 - val_loss: 0.1317 - val_accuracy: 0.9483\n",
            "Epoch 5/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0937 - accuracy: 0.9647 - val_loss: 0.1759 - val_accuracy: 0.9248\n",
            "Epoch 6/60\n",
            "653/653 [==============================] - 17s 26ms/step - loss: 0.0835 - accuracy: 0.9681 - val_loss: 0.1138 - val_accuracy: 0.9570\n",
            "Epoch 7/60\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0678 - accuracy: 0.9757 - val_loss: 0.1326 - val_accuracy: 0.9510\n",
            "Epoch 8/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0585 - accuracy: 0.9780 - val_loss: 0.1318 - val_accuracy: 0.9542\n",
            "Epoch 9/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0599 - accuracy: 0.9780 - val_loss: 0.1242 - val_accuracy: 0.9544\n",
            "Epoch 10/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0507 - accuracy: 0.9809 - val_loss: 0.1190 - val_accuracy: 0.9606\n",
            "Epoch 11/60\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0459 - accuracy: 0.9838 - val_loss: 0.1244 - val_accuracy: 0.9588\n",
            "Epoch 12/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0399 - accuracy: 0.9855 - val_loss: 0.1209 - val_accuracy: 0.9601\n",
            "Epoch 13/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0396 - accuracy: 0.9851 - val_loss: 0.1326 - val_accuracy: 0.9585\n",
            "Epoch 14/60\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.0436 - accuracy: 0.9849 - val_loss: 0.1412 - val_accuracy: 0.9619\n",
            "Epoch 15/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0416 - accuracy: 0.9856 - val_loss: 0.1236 - val_accuracy: 0.9622\n",
            "Epoch 16/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0347 - accuracy: 0.9866 - val_loss: 0.1336 - val_accuracy: 0.9612\n",
            "Epoch 17/60\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0321 - accuracy: 0.9890 - val_loss: 0.1461 - val_accuracy: 0.9616\n",
            "Epoch 18/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0304 - accuracy: 0.9898 - val_loss: 0.1186 - val_accuracy: 0.9618\n",
            "Epoch 19/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0287 - accuracy: 0.9897 - val_loss: 0.1474 - val_accuracy: 0.9611\n",
            "Epoch 20/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0272 - accuracy: 0.9905 - val_loss: 0.1487 - val_accuracy: 0.9606\n",
            "Epoch 21/60\n",
            "653/653 [==============================] - 17s 25ms/step - loss: 0.0261 - accuracy: 0.9902 - val_loss: 0.1523 - val_accuracy: 0.9593\n",
            "Epoch 22/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0260 - accuracy: 0.9908 - val_loss: 0.1490 - val_accuracy: 0.9608\n",
            "Epoch 23/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0269 - accuracy: 0.9904 - val_loss: 0.1300 - val_accuracy: 0.9624\n",
            "Epoch 24/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0248 - accuracy: 0.9911 - val_loss: 0.1541 - val_accuracy: 0.9602\n",
            "Epoch 25/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0247 - accuracy: 0.9908 - val_loss: 0.1557 - val_accuracy: 0.9625\n",
            "Epoch 26/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0193 - accuracy: 0.9933 - val_loss: 0.1603 - val_accuracy: 0.9609\n",
            "Epoch 27/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0217 - accuracy: 0.9929 - val_loss: 0.1689 - val_accuracy: 0.9629\n",
            "Epoch 28/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0222 - accuracy: 0.9917 - val_loss: 0.1583 - val_accuracy: 0.9603\n",
            "Epoch 29/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0173 - accuracy: 0.9934 - val_loss: 0.1870 - val_accuracy: 0.9612\n",
            "Epoch 30/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0194 - accuracy: 0.9933 - val_loss: 0.1710 - val_accuracy: 0.9619\n",
            "Epoch 31/60\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0175 - accuracy: 0.9934 - val_loss: 0.1778 - val_accuracy: 0.9590\n",
            "Epoch 32/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0209 - accuracy: 0.9930 - val_loss: 0.1599 - val_accuracy: 0.9578\n",
            "Number of epochs run: 32\n",
            "CPU times: user 5min 55s, sys: 15.9 s, total: 6min 11s\n",
            "Wall time: 6min 15s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q1VntH76E83s",
        "outputId": "a4d762ee-c16f-4c54-fe42-8e10b85b1e6c"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 3s 9ms/step - loss: 0.1902 - accuracy: 0.9543\n",
            "Testing Accuracy is 95.43130397796631 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Add an Additional BiLSTM Layer"
      ],
      "metadata": {
        "id": "o3kqehagotpf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with two stacked BiLSTM layers, as well as a dropout layer\n",
        "stacked_rnn_model = tf.keras.Sequential([\n",
        "          embedding_layer_glove,\n",
        "          Bidirectional(LSTM(64,  return_sequences=True)),\n",
        "          Bidirectional(LSTM(32)),\n",
        "          Dense(25, activation='relu'),\n",
        "          Dropout(0.5),\n",
        "          Dense(15, activation='relu'),\n",
        "          Dropout(0.5),\n",
        "          Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "stacked_rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 20 epochs with early stopping (patience of 7)\n",
        "\n",
        "earlystop7 = EarlyStopping(monitor='loss', patience=7)\n",
        "\n",
        "history = stacked_rnn_model.fit(X_train_glove, y_train_glove,\n",
        "                    epochs=60, callbacks=[earlystop7],\n",
        "                    validation_data=(X_val_glove, y_val_glove),\n",
        "                    validation_steps=len(X_val_glove)//128,\n",
        "                    steps_per_epoch=len(X_train_glove)//128)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p5C75nUw5MxW",
        "outputId": "6b0805e5-5aac-41c2-fc3c-609a2a1c48c3"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/60\n",
            "163/163 [==============================] - 15s 34ms/step - loss: 0.5143 - accuracy: 0.7399 - val_loss: 0.3080 - val_accuracy: 0.8697\n",
            "Epoch 2/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.3127 - accuracy: 0.8817 - val_loss: 0.2164 - val_accuracy: 0.9108\n",
            "Epoch 3/60\n",
            "163/163 [==============================] - 5s 30ms/step - loss: 0.2361 - accuracy: 0.9147 - val_loss: 0.1973 - val_accuracy: 0.9187\n",
            "Epoch 4/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.1781 - accuracy: 0.9396 - val_loss: 0.1802 - val_accuracy: 0.9277\n",
            "Epoch 5/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.1512 - accuracy: 0.9513 - val_loss: 0.1736 - val_accuracy: 0.9342\n",
            "Epoch 6/60\n",
            "163/163 [==============================] - 5s 32ms/step - loss: 0.1326 - accuracy: 0.9564 - val_loss: 0.1795 - val_accuracy: 0.9332\n",
            "Epoch 7/60\n",
            "163/163 [==============================] - 4s 23ms/step - loss: 0.1051 - accuracy: 0.9657 - val_loss: 0.1708 - val_accuracy: 0.9378\n",
            "Epoch 8/60\n",
            "163/163 [==============================] - 4s 23ms/step - loss: 0.0936 - accuracy: 0.9715 - val_loss: 0.1843 - val_accuracy: 0.9395\n",
            "Epoch 9/60\n",
            "163/163 [==============================] - 5s 28ms/step - loss: 0.0723 - accuracy: 0.9773 - val_loss: 0.2262 - val_accuracy: 0.9363\n",
            "Epoch 10/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0695 - accuracy: 0.9781 - val_loss: 0.2748 - val_accuracy: 0.9411\n",
            "Epoch 11/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.0622 - accuracy: 0.9818 - val_loss: 0.2323 - val_accuracy: 0.9461\n",
            "Epoch 12/60\n",
            "163/163 [==============================] - 4s 26ms/step - loss: 0.0506 - accuracy: 0.9849 - val_loss: 0.2732 - val_accuracy: 0.9437\n",
            "Epoch 13/60\n",
            "163/163 [==============================] - 5s 28ms/step - loss: 0.0557 - accuracy: 0.9826 - val_loss: 0.2805 - val_accuracy: 0.9458\n",
            "Epoch 14/60\n",
            "163/163 [==============================] - 5s 29ms/step - loss: 0.0371 - accuracy: 0.9886 - val_loss: 0.3315 - val_accuracy: 0.9458\n",
            "Epoch 15/60\n",
            "163/163 [==============================] - 5s 28ms/step - loss: 0.0367 - accuracy: 0.9902 - val_loss: 0.4310 - val_accuracy: 0.9072\n",
            "Epoch 16/60\n",
            "163/163 [==============================] - 5s 31ms/step - loss: 0.0493 - accuracy: 0.9846 - val_loss: 0.2628 - val_accuracy: 0.9509\n",
            "Epoch 17/60\n",
            "163/163 [==============================] - 5s 30ms/step - loss: 0.0388 - accuracy: 0.9877 - val_loss: 0.2560 - val_accuracy: 0.9458\n",
            "Epoch 18/60\n",
            "163/163 [==============================] - 4s 27ms/step - loss: 0.0320 - accuracy: 0.9907 - val_loss: 0.3027 - val_accuracy: 0.9507\n",
            "Epoch 19/60\n",
            "163/163 [==============================] - 5s 28ms/step - loss: 0.0236 - accuracy: 0.9924 - val_loss: 0.3530 - val_accuracy: 0.9519\n",
            "Epoch 20/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.0358 - accuracy: 0.9904 - val_loss: 0.2824 - val_accuracy: 0.9491\n",
            "Epoch 21/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.0214 - accuracy: 0.9937 - val_loss: 0.3593 - val_accuracy: 0.9514\n",
            "Epoch 22/60\n",
            "163/163 [==============================] - 7s 43ms/step - loss: 0.0243 - accuracy: 0.9934 - val_loss: 0.3254 - val_accuracy: 0.9348\n",
            "Epoch 23/60\n",
            "163/163 [==============================] - 7s 44ms/step - loss: 0.0334 - accuracy: 0.9892 - val_loss: 0.3400 - val_accuracy: 0.9470\n",
            "Epoch 24/60\n",
            "163/163 [==============================] - 10s 59ms/step - loss: 0.0285 - accuracy: 0.9910 - val_loss: 0.3106 - val_accuracy: 0.9430\n",
            "Epoch 25/60\n",
            "163/163 [==============================] - 7s 42ms/step - loss: 0.0219 - accuracy: 0.9930 - val_loss: 0.3821 - val_accuracy: 0.9478\n",
            "Epoch 26/60\n",
            "163/163 [==============================] - 5s 31ms/step - loss: 0.0295 - accuracy: 0.9905 - val_loss: 0.3017 - val_accuracy: 0.9527\n",
            "Epoch 27/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0222 - accuracy: 0.9931 - val_loss: 0.3494 - val_accuracy: 0.9481\n",
            "Epoch 28/60\n",
            "163/163 [==============================] - 8s 48ms/step - loss: 0.0184 - accuracy: 0.9944 - val_loss: 0.4093 - val_accuracy: 0.9517\n",
            "Epoch 29/60\n",
            "163/163 [==============================] - 9s 53ms/step - loss: 0.0195 - accuracy: 0.9941 - val_loss: 0.3744 - val_accuracy: 0.9523\n",
            "Epoch 30/60\n",
            "163/163 [==============================] - 7s 40ms/step - loss: 0.0424 - accuracy: 0.9880 - val_loss: 0.3561 - val_accuracy: 0.9500\n",
            "Epoch 31/60\n",
            "163/163 [==============================] - 7s 44ms/step - loss: 0.0189 - accuracy: 0.9939 - val_loss: 0.3896 - val_accuracy: 0.9483\n",
            "Epoch 32/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.0182 - accuracy: 0.9943 - val_loss: 0.3639 - val_accuracy: 0.9419\n",
            "Epoch 33/60\n",
            "163/163 [==============================] - 5s 31ms/step - loss: 0.0334 - accuracy: 0.9899 - val_loss: 0.2857 - val_accuracy: 0.9457\n",
            "Epoch 34/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0305 - accuracy: 0.9898 - val_loss: 0.3453 - val_accuracy: 0.9487\n",
            "Epoch 35/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0147 - accuracy: 0.9955 - val_loss: 0.3849 - val_accuracy: 0.9447\n",
            "Epoch 36/60\n",
            "163/163 [==============================] - 5s 30ms/step - loss: 0.0148 - accuracy: 0.9956 - val_loss: 0.3921 - val_accuracy: 0.9480\n",
            "Epoch 37/60\n",
            "163/163 [==============================] - 4s 26ms/step - loss: 0.0151 - accuracy: 0.9951 - val_loss: 0.4500 - val_accuracy: 0.9529\n",
            "Epoch 38/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0109 - accuracy: 0.9967 - val_loss: 0.5291 - val_accuracy: 0.9388\n",
            "Epoch 39/60\n",
            "163/163 [==============================] - 5s 28ms/step - loss: 0.0280 - accuracy: 0.9915 - val_loss: 0.3627 - val_accuracy: 0.9441\n",
            "Epoch 40/60\n",
            "163/163 [==============================] - 4s 26ms/step - loss: 0.0205 - accuracy: 0.9932 - val_loss: 0.4336 - val_accuracy: 0.9503\n",
            "Epoch 41/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0227 - accuracy: 0.9933 - val_loss: 0.3894 - val_accuracy: 0.9424\n",
            "Epoch 42/60\n",
            "163/163 [==============================] - 4s 24ms/step - loss: 0.0396 - accuracy: 0.9872 - val_loss: 0.3007 - val_accuracy: 0.9486\n",
            "Epoch 43/60\n",
            "163/163 [==============================] - 5s 30ms/step - loss: 0.0200 - accuracy: 0.9934 - val_loss: 0.3272 - val_accuracy: 0.9510\n",
            "Epoch 44/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0224 - accuracy: 0.9931 - val_loss: 0.3587 - val_accuracy: 0.9498\n",
            "Epoch 45/60\n",
            "163/163 [==============================] - 4s 25ms/step - loss: 0.0129 - accuracy: 0.9955 - val_loss: 0.4143 - val_accuracy: 0.9514\n",
            "CPU times: user 3min 42s, sys: 9.47 s, total: 3min 52s\n",
            "Wall time: 3min 53s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = stacked_rnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mFzxxkZY7y4g",
        "outputId": "6b8a68e3-6571-46b2-ed0d-5bfb2cbe1871"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 3s 9ms/step - loss: 0.4736 - accuracy: 0.9447\n",
            "Testing Accuracy is 94.46726441383362 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "VTZoh5y3qO5e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Bidirectional(LSTM(64, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 150 epochs with learning rate reduction (by a factor\n",
        "# of 0.5, patience of 7) once learning stagnates\n",
        "\n",
        "reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5,\n",
        "                              patience=7, min_lr=0.000001)\n",
        "\n",
        "history = rnn_model.fit(X_train_glove, y_train_glove, epochs=150,\n",
        "                        callbacks=[reduce_lr],\n",
        "                        validation_data=(X_val_glove, y_val_glove),\n",
        "                        validation_steps=32)\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sZ6ua2suXoTx",
        "outputId": "af9290e4-375f-47ba-e8e0-92db88c6ff71"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/150\n",
            "653/653 [==============================] - 18s 19ms/step - loss: 0.3337 - accuracy: 0.8529 - val_loss: 0.2062 - val_accuracy: 0.9165 - lr: 0.0030\n",
            "Epoch 2/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1883 - accuracy: 0.9281 - val_loss: 0.1711 - val_accuracy: 0.9302 - lr: 0.0030\n",
            "Epoch 3/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.1430 - accuracy: 0.9467 - val_loss: 0.1237 - val_accuracy: 0.9539 - lr: 0.0030\n",
            "Epoch 4/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.1142 - accuracy: 0.9582 - val_loss: 0.1288 - val_accuracy: 0.9509 - lr: 0.0030\n",
            "Epoch 5/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0916 - accuracy: 0.9660 - val_loss: 0.1174 - val_accuracy: 0.9585 - lr: 0.0030\n",
            "Epoch 6/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0747 - accuracy: 0.9738 - val_loss: 0.1225 - val_accuracy: 0.9575 - lr: 0.0030\n",
            "Epoch 7/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0656 - accuracy: 0.9753 - val_loss: 0.1277 - val_accuracy: 0.9560 - lr: 0.0030\n",
            "Epoch 8/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0576 - accuracy: 0.9792 - val_loss: 0.1239 - val_accuracy: 0.9595 - lr: 0.0030\n",
            "Epoch 9/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0514 - accuracy: 0.9817 - val_loss: 0.1333 - val_accuracy: 0.9556 - lr: 0.0030\n",
            "Epoch 10/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0447 - accuracy: 0.9837 - val_loss: 0.1376 - val_accuracy: 0.9589 - lr: 0.0030\n",
            "Epoch 11/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0462 - accuracy: 0.9829 - val_loss: 0.1225 - val_accuracy: 0.9601 - lr: 0.0030\n",
            "Epoch 12/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0388 - accuracy: 0.9858 - val_loss: 0.1417 - val_accuracy: 0.9573 - lr: 0.0030\n",
            "Epoch 13/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0312 - accuracy: 0.9891 - val_loss: 0.1440 - val_accuracy: 0.9601 - lr: 0.0015\n",
            "Epoch 14/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0234 - accuracy: 0.9921 - val_loss: 0.1601 - val_accuracy: 0.9606 - lr: 0.0015\n",
            "Epoch 15/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0198 - accuracy: 0.9925 - val_loss: 0.1653 - val_accuracy: 0.9619 - lr: 0.0015\n",
            "Epoch 16/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0198 - accuracy: 0.9929 - val_loss: 0.1600 - val_accuracy: 0.9616 - lr: 0.0015\n",
            "Epoch 17/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0179 - accuracy: 0.9935 - val_loss: 0.1597 - val_accuracy: 0.9621 - lr: 0.0015\n",
            "Epoch 18/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0177 - accuracy: 0.9936 - val_loss: 0.2201 - val_accuracy: 0.9546 - lr: 0.0015\n",
            "Epoch 19/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0189 - accuracy: 0.9928 - val_loss: 0.1572 - val_accuracy: 0.9609 - lr: 0.0015\n",
            "Epoch 20/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0150 - accuracy: 0.9948 - val_loss: 0.1612 - val_accuracy: 0.9626 - lr: 7.5000e-04\n",
            "Epoch 21/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0116 - accuracy: 0.9954 - val_loss: 0.2106 - val_accuracy: 0.9593 - lr: 7.5000e-04\n",
            "Epoch 22/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0117 - accuracy: 0.9956 - val_loss: 0.1968 - val_accuracy: 0.9609 - lr: 7.5000e-04\n",
            "Epoch 23/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0109 - accuracy: 0.9964 - val_loss: 0.2132 - val_accuracy: 0.9585 - lr: 7.5000e-04\n",
            "Epoch 24/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0111 - accuracy: 0.9959 - val_loss: 0.1853 - val_accuracy: 0.9618 - lr: 7.5000e-04\n",
            "Epoch 25/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0107 - accuracy: 0.9967 - val_loss: 0.2007 - val_accuracy: 0.9598 - lr: 7.5000e-04\n",
            "Epoch 26/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0097 - accuracy: 0.9965 - val_loss: 0.1855 - val_accuracy: 0.9626 - lr: 7.5000e-04\n",
            "Epoch 27/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0087 - accuracy: 0.9969 - val_loss: 0.1920 - val_accuracy: 0.9636 - lr: 3.7500e-04\n",
            "Epoch 28/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0104 - accuracy: 0.9963 - val_loss: 0.1917 - val_accuracy: 0.9628 - lr: 3.7500e-04\n",
            "Epoch 29/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0080 - accuracy: 0.9968 - val_loss: 0.2009 - val_accuracy: 0.9626 - lr: 3.7500e-04\n",
            "Epoch 30/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0084 - accuracy: 0.9968 - val_loss: 0.2039 - val_accuracy: 0.9625 - lr: 3.7500e-04\n",
            "Epoch 31/150\n",
            "653/653 [==============================] - 11s 18ms/step - loss: 0.0075 - accuracy: 0.9972 - val_loss: 0.2253 - val_accuracy: 0.9626 - lr: 3.7500e-04\n",
            "Epoch 32/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0079 - accuracy: 0.9967 - val_loss: 0.2367 - val_accuracy: 0.9622 - lr: 3.7500e-04\n",
            "Epoch 33/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0074 - accuracy: 0.9971 - val_loss: 0.2203 - val_accuracy: 0.9621 - lr: 3.7500e-04\n",
            "Epoch 34/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0067 - accuracy: 0.9977 - val_loss: 0.2200 - val_accuracy: 0.9631 - lr: 1.8750e-04\n",
            "Epoch 35/150\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0056 - accuracy: 0.9978 - val_loss: 0.2205 - val_accuracy: 0.9647 - lr: 1.8750e-04\n",
            "Epoch 36/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0060 - accuracy: 0.9977 - val_loss: 0.2276 - val_accuracy: 0.9638 - lr: 1.8750e-04\n",
            "Epoch 37/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0071 - accuracy: 0.9974 - val_loss: 0.2296 - val_accuracy: 0.9638 - lr: 1.8750e-04\n",
            "Epoch 38/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0063 - accuracy: 0.9978 - val_loss: 0.2263 - val_accuracy: 0.9648 - lr: 1.8750e-04\n",
            "Epoch 39/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0059 - accuracy: 0.9977 - val_loss: 0.2376 - val_accuracy: 0.9634 - lr: 1.8750e-04\n",
            "Epoch 40/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0057 - accuracy: 0.9977 - val_loss: 0.2355 - val_accuracy: 0.9629 - lr: 1.8750e-04\n",
            "Epoch 41/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0055 - accuracy: 0.9981 - val_loss: 0.2326 - val_accuracy: 0.9638 - lr: 9.3750e-05\n",
            "Epoch 42/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0058 - accuracy: 0.9975 - val_loss: 0.2398 - val_accuracy: 0.9634 - lr: 9.3750e-05\n",
            "Epoch 43/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0055 - accuracy: 0.9981 - val_loss: 0.2381 - val_accuracy: 0.9634 - lr: 9.3750e-05\n",
            "Epoch 44/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0049 - accuracy: 0.9979 - val_loss: 0.2428 - val_accuracy: 0.9628 - lr: 9.3750e-05\n",
            "Epoch 45/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0059 - accuracy: 0.9978 - val_loss: 0.2375 - val_accuracy: 0.9641 - lr: 9.3750e-05\n",
            "Epoch 46/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0054 - accuracy: 0.9978 - val_loss: 0.2479 - val_accuracy: 0.9629 - lr: 9.3750e-05\n",
            "Epoch 47/150\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0071 - accuracy: 0.9971 - val_loss: 0.2391 - val_accuracy: 0.9629 - lr: 9.3750e-05\n",
            "Epoch 48/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0056 - accuracy: 0.9979 - val_loss: 0.2383 - val_accuracy: 0.9634 - lr: 4.6875e-05\n",
            "Epoch 49/150\n",
            "653/653 [==============================] - 9s 13ms/step - loss: 0.0052 - accuracy: 0.9980 - val_loss: 0.2421 - val_accuracy: 0.9628 - lr: 4.6875e-05\n",
            "Epoch 50/150\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0060 - accuracy: 0.9977 - val_loss: 0.2412 - val_accuracy: 0.9631 - lr: 4.6875e-05\n",
            "Epoch 51/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0061 - accuracy: 0.9977 - val_loss: 0.2407 - val_accuracy: 0.9629 - lr: 4.6875e-05\n",
            "Epoch 52/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0051 - accuracy: 0.9979 - val_loss: 0.2436 - val_accuracy: 0.9626 - lr: 4.6875e-05\n",
            "Epoch 53/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0050 - accuracy: 0.9977 - val_loss: 0.2490 - val_accuracy: 0.9628 - lr: 4.6875e-05\n",
            "Epoch 54/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0053 - accuracy: 0.9982 - val_loss: 0.2456 - val_accuracy: 0.9632 - lr: 4.6875e-05\n",
            "Epoch 55/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0042 - accuracy: 0.9982 - val_loss: 0.2437 - val_accuracy: 0.9632 - lr: 2.3438e-05\n",
            "Epoch 56/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0052 - accuracy: 0.9979 - val_loss: 0.2476 - val_accuracy: 0.9631 - lr: 2.3438e-05\n",
            "Epoch 57/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0054 - accuracy: 0.9979 - val_loss: 0.2445 - val_accuracy: 0.9631 - lr: 2.3438e-05\n",
            "Epoch 58/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0053 - accuracy: 0.9980 - val_loss: 0.2426 - val_accuracy: 0.9632 - lr: 2.3438e-05\n",
            "Epoch 59/150\n",
            "653/653 [==============================] - 9s 15ms/step - loss: 0.0053 - accuracy: 0.9979 - val_loss: 0.2441 - val_accuracy: 0.9634 - lr: 2.3438e-05\n",
            "Epoch 60/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9981 - val_loss: 0.2429 - val_accuracy: 0.9632 - lr: 2.3438e-05\n",
            "Epoch 61/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0046 - accuracy: 0.9982 - val_loss: 0.2461 - val_accuracy: 0.9636 - lr: 2.3438e-05\n",
            "Epoch 62/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0048 - accuracy: 0.9980 - val_loss: 0.2451 - val_accuracy: 0.9631 - lr: 1.1719e-05\n",
            "Epoch 63/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0053 - accuracy: 0.9981 - val_loss: 0.2453 - val_accuracy: 0.9635 - lr: 1.1719e-05\n",
            "Epoch 64/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0045 - accuracy: 0.9986 - val_loss: 0.2482 - val_accuracy: 0.9638 - lr: 1.1719e-05\n",
            "Epoch 65/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0054 - accuracy: 0.9976 - val_loss: 0.2465 - val_accuracy: 0.9632 - lr: 1.1719e-05\n",
            "Epoch 66/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0055 - accuracy: 0.9978 - val_loss: 0.2456 - val_accuracy: 0.9635 - lr: 1.1719e-05\n",
            "Epoch 67/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0048 - accuracy: 0.9981 - val_loss: 0.2450 - val_accuracy: 0.9635 - lr: 1.1719e-05\n",
            "Epoch 68/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0053 - accuracy: 0.9982 - val_loss: 0.2455 - val_accuracy: 0.9636 - lr: 1.1719e-05\n",
            "Epoch 69/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0051 - accuracy: 0.9977 - val_loss: 0.2463 - val_accuracy: 0.9635 - lr: 5.8594e-06\n",
            "Epoch 70/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0048 - accuracy: 0.9982 - val_loss: 0.2474 - val_accuracy: 0.9632 - lr: 5.8594e-06\n",
            "Epoch 71/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0047 - accuracy: 0.9978 - val_loss: 0.2473 - val_accuracy: 0.9632 - lr: 5.8594e-06\n",
            "Epoch 72/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0056 - accuracy: 0.9978 - val_loss: 0.2469 - val_accuracy: 0.9635 - lr: 5.8594e-06\n",
            "Epoch 73/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0047 - accuracy: 0.9982 - val_loss: 0.2472 - val_accuracy: 0.9634 - lr: 5.8594e-06\n",
            "Epoch 74/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0057 - accuracy: 0.9973 - val_loss: 0.2473 - val_accuracy: 0.9635 - lr: 5.8594e-06\n",
            "Epoch 75/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0054 - accuracy: 0.9981 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 5.8594e-06\n",
            "Epoch 76/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0055 - accuracy: 0.9978 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 77/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0050 - accuracy: 0.9982 - val_loss: 0.2477 - val_accuracy: 0.9634 - lr: 2.9297e-06\n",
            "Epoch 78/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0047 - accuracy: 0.9981 - val_loss: 0.2473 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 79/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9982 - val_loss: 0.2475 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 80/150\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0050 - accuracy: 0.9982 - val_loss: 0.2470 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 81/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0047 - accuracy: 0.9983 - val_loss: 0.2467 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 82/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9983 - val_loss: 0.2470 - val_accuracy: 0.9635 - lr: 2.9297e-06\n",
            "Epoch 83/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0050 - accuracy: 0.9980 - val_loss: 0.2471 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 84/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0048 - accuracy: 0.9977 - val_loss: 0.2470 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 85/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0043 - accuracy: 0.9984 - val_loss: 0.2471 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 86/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0049 - accuracy: 0.9980 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 87/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0050 - accuracy: 0.9977 - val_loss: 0.2473 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 88/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0045 - accuracy: 0.9984 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 89/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0053 - accuracy: 0.9978 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 1.4648e-06\n",
            "Epoch 90/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0052 - accuracy: 0.9980 - val_loss: 0.2471 - val_accuracy: 0.9636 - lr: 1.0000e-06\n",
            "Epoch 91/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9981 - val_loss: 0.2472 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 92/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9982 - val_loss: 0.2473 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 93/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9982 - val_loss: 0.2476 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 94/150\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0041 - accuracy: 0.9982 - val_loss: 0.2476 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 95/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0053 - accuracy: 0.9982 - val_loss: 0.2475 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 96/150\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0053 - accuracy: 0.9979 - val_loss: 0.2476 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 97/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0043 - accuracy: 0.9982 - val_loss: 0.2475 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 98/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0043 - accuracy: 0.9984 - val_loss: 0.2474 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 99/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0050 - accuracy: 0.9981 - val_loss: 0.2475 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 100/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0051 - accuracy: 0.9981 - val_loss: 0.2475 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 101/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0047 - accuracy: 0.9978 - val_loss: 0.2476 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 102/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9983 - val_loss: 0.2477 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 103/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0049 - accuracy: 0.9978 - val_loss: 0.2478 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 104/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9979 - val_loss: 0.2480 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 105/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0053 - accuracy: 0.9977 - val_loss: 0.2484 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 106/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0048 - accuracy: 0.9980 - val_loss: 0.2485 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 107/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0052 - accuracy: 0.9982 - val_loss: 0.2484 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 108/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0048 - accuracy: 0.9981 - val_loss: 0.2486 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 109/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0050 - accuracy: 0.9986 - val_loss: 0.2485 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 110/150\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0046 - accuracy: 0.9982 - val_loss: 0.2486 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 111/150\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0049 - accuracy: 0.9981 - val_loss: 0.2486 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 112/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0052 - accuracy: 0.9977 - val_loss: 0.2484 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 113/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0047 - accuracy: 0.9983 - val_loss: 0.2484 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 114/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0047 - accuracy: 0.9982 - val_loss: 0.2483 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 115/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0049 - accuracy: 0.9979 - val_loss: 0.2483 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 116/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0050 - accuracy: 0.9979 - val_loss: 0.2482 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 117/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0047 - accuracy: 0.9981 - val_loss: 0.2481 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 118/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9980 - val_loss: 0.2482 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 119/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0045 - accuracy: 0.9982 - val_loss: 0.2482 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 120/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0048 - accuracy: 0.9982 - val_loss: 0.2483 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 121/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0057 - accuracy: 0.9982 - val_loss: 0.2483 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 122/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0046 - accuracy: 0.9984 - val_loss: 0.2483 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 123/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0040 - accuracy: 0.9986 - val_loss: 0.2484 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 124/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0043 - accuracy: 0.9982 - val_loss: 0.2485 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 125/150\n",
            "653/653 [==============================] - 9s 15ms/step - loss: 0.0050 - accuracy: 0.9980 - val_loss: 0.2486 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 126/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0046 - accuracy: 0.9980 - val_loss: 0.2486 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 127/150\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.0044 - accuracy: 0.9986 - val_loss: 0.2489 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 128/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0048 - accuracy: 0.9980 - val_loss: 0.2491 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Epoch 129/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0046 - accuracy: 0.9981 - val_loss: 0.2490 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Epoch 130/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9982 - val_loss: 0.2490 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Epoch 131/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0048 - accuracy: 0.9981 - val_loss: 0.2489 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 132/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0047 - accuracy: 0.9982 - val_loss: 0.2487 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 133/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0044 - accuracy: 0.9984 - val_loss: 0.2489 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 134/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0046 - accuracy: 0.9983 - val_loss: 0.2488 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 135/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0044 - accuracy: 0.9984 - val_loss: 0.2489 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 136/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0052 - accuracy: 0.9980 - val_loss: 0.2489 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 137/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9978 - val_loss: 0.2492 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 138/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0049 - accuracy: 0.9981 - val_loss: 0.2492 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 139/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0051 - accuracy: 0.9977 - val_loss: 0.2493 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 140/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0049 - accuracy: 0.9982 - val_loss: 0.2493 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 141/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0054 - accuracy: 0.9979 - val_loss: 0.2493 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 142/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0044 - accuracy: 0.9982 - val_loss: 0.2492 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 143/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0047 - accuracy: 0.9981 - val_loss: 0.2493 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 144/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0051 - accuracy: 0.9982 - val_loss: 0.2491 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 145/150\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.0049 - accuracy: 0.9979 - val_loss: 0.2492 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 146/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0052 - accuracy: 0.9979 - val_loss: 0.2494 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 147/150\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0058 - accuracy: 0.9982 - val_loss: 0.2495 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Epoch 148/150\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0047 - accuracy: 0.9983 - val_loss: 0.2495 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Epoch 149/150\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0054 - accuracy: 0.9977 - val_loss: 0.2494 - val_accuracy: 0.9635 - lr: 1.0000e-06\n",
            "Epoch 150/150\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0054 - accuracy: 0.9980 - val_loss: 0.2496 - val_accuracy: 0.9634 - lr: 1.0000e-06\n",
            "Number of epochs run: 150\n",
            "CPU times: user 26min 16s, sys: 1min 17s, total: 27min 34s\n",
            "Wall time: 25min 33s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "irTZuufsXoec",
        "outputId": "1170dc23-d43a-47b0-8032-972585a02c31"
      },
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 2s 6ms/step - loss: 0.2786 - accuracy: 0.9603\n",
            "Testing Accuracy is 96.02649211883545 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My final BiLSTM model that used the GloVe word embeddings achieved an accuracy of 96.03% and took 27 min and 34 s of CPU time to train."
      ],
      "metadata": {
        "id": "Zm3Iz760xEzT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### BiLSTM with a Continuous Bag of Words model"
      ],
      "metadata": {
        "id": "3hJQfUNI8tcJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Preliminary Model"
      ],
      "metadata": {
        "id": "u-H5hwWq8uQe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Bidirectional(LSTM(128, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "earlystop3 = EarlyStopping(monitor='val_loss', patience=3)\n",
        "\n",
        "history = rnn_model.fit(X_train_bow, y_train_bow, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_bow, y_val_bow))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TIp9x9Rg-ne0",
        "outputId": "fd47c726-413b-4b42-f6bc-753b021cebdd"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 27s 34ms/step - loss: 0.3390 - accuracy: 0.8388 - val_loss: 0.1870 - val_accuracy: 0.9260\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 23s 35ms/step - loss: 0.1532 - accuracy: 0.9401 - val_loss: 0.1782 - val_accuracy: 0.9270\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 0.0961 - accuracy: 0.9643 - val_loss: 0.1546 - val_accuracy: 0.9415\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 0.0585 - accuracy: 0.9791 - val_loss: 0.0842 - val_accuracy: 0.9707\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 24s 36ms/step - loss: 0.0323 - accuracy: 0.9887 - val_loss: 0.0858 - val_accuracy: 0.9711\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 27s 41ms/step - loss: 0.0187 - accuracy: 0.9940 - val_loss: 0.0894 - val_accuracy: 0.9677\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 26s 40ms/step - loss: 0.0097 - accuracy: 0.9974 - val_loss: 0.0937 - val_accuracy: 0.9736\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 26s 40ms/step - loss: 0.0153 - accuracy: 0.9950 - val_loss: 0.0819 - val_accuracy: 0.9760\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 27s 41ms/step - loss: 0.0072 - accuracy: 0.9979 - val_loss: 0.0870 - val_accuracy: 0.9746\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 24s 37ms/step - loss: 0.0039 - accuracy: 0.9991 - val_loss: 0.1015 - val_accuracy: 0.9744\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 24s 37ms/step - loss: 8.3646e-04 - accuracy: 1.0000 - val_loss: 0.0971 - val_accuracy: 0.9756\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 3.2735e-04 - accuracy: 1.0000 - val_loss: 0.1026 - val_accuracy: 0.9753\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 27s 41ms/step - loss: 1.4540e-04 - accuracy: 1.0000 - val_loss: 0.1089 - val_accuracy: 0.9756\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 26s 39ms/step - loss: 8.5354e-05 - accuracy: 1.0000 - val_loss: 0.1144 - val_accuracy: 0.9754\n",
            "Epoch 15/20\n",
            "653/653 [==============================] - 19s 29ms/step - loss: 5.6039e-05 - accuracy: 1.0000 - val_loss: 0.1190 - val_accuracy: 0.9754\n",
            "Epoch 16/20\n",
            "653/653 [==============================] - 23s 35ms/step - loss: 3.7425e-05 - accuracy: 1.0000 - val_loss: 0.1234 - val_accuracy: 0.9753\n",
            "Epoch 17/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 2.4912e-05 - accuracy: 1.0000 - val_loss: 0.1282 - val_accuracy: 0.9756\n",
            "Epoch 18/20\n",
            "653/653 [==============================] - 27s 41ms/step - loss: 1.6812e-05 - accuracy: 1.0000 - val_loss: 0.1348 - val_accuracy: 0.9747\n",
            "Epoch 19/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 1.1432e-05 - accuracy: 1.0000 - val_loss: 0.1401 - val_accuracy: 0.9749\n",
            "Epoch 20/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 7.7808e-06 - accuracy: 1.0000 - val_loss: 0.1435 - val_accuracy: 0.9749\n",
            "Number of epochs run: 20\n",
            "CPU times: user 6min 54s, sys: 13.2 s, total: 7min 7s\n",
            "Wall time: 8min 27s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_bow, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WYs16pPO-xP6",
        "outputId": "f1303b14-4176-49e0-e947-158d85536cc9"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 5s 13ms/step - loss: 0.1963 - accuracy: 0.9655\n",
            "Testing Accuracy is 96.55461311340332 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Adding Dropout Layers"
      ],
      "metadata": {
        "id": "AKjgqcKm8udk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Bidirectional(LSTM(128, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = rnn_model.fit(X_train_bow, y_train_bow, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_bow, y_val_bow))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tgYsUOO6-npf",
        "outputId": "58ade776-191e-4dd5-ea3d-40ba4cf12c1d"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 30s 31ms/step - loss: 0.3393 - accuracy: 0.8449 - val_loss: 0.1755 - val_accuracy: 0.9323\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 0.1593 - accuracy: 0.9433 - val_loss: 0.1131 - val_accuracy: 0.9566\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 19s 29ms/step - loss: 0.1021 - accuracy: 0.9644 - val_loss: 0.1159 - val_accuracy: 0.9580\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 0.0751 - accuracy: 0.9759 - val_loss: 0.0719 - val_accuracy: 0.9733\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 0.0499 - accuracy: 0.9845 - val_loss: 0.0632 - val_accuracy: 0.9777\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 0.0356 - accuracy: 0.9884 - val_loss: 0.0608 - val_accuracy: 0.9793\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 21s 32ms/step - loss: 0.0307 - accuracy: 0.9904 - val_loss: 0.0834 - val_accuracy: 0.9757\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 19s 29ms/step - loss: 0.0271 - accuracy: 0.9918 - val_loss: 0.0575 - val_accuracy: 0.9805\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 0.0199 - accuracy: 0.9947 - val_loss: 0.0701 - val_accuracy: 0.9795\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 0.0199 - accuracy: 0.9941 - val_loss: 0.0622 - val_accuracy: 0.9793\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 20s 30ms/step - loss: 0.0174 - accuracy: 0.9943 - val_loss: 0.0750 - val_accuracy: 0.9795\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 0.0139 - accuracy: 0.9964 - val_loss: 0.0721 - val_accuracy: 0.9812\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 19s 30ms/step - loss: 0.0102 - accuracy: 0.9972 - val_loss: 0.0814 - val_accuracy: 0.9807\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 0.0116 - accuracy: 0.9968 - val_loss: 0.0762 - val_accuracy: 0.9809\n",
            "Epoch 15/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 0.0115 - accuracy: 0.9967 - val_loss: 0.0868 - val_accuracy: 0.9805\n",
            "Epoch 16/20\n",
            "653/653 [==============================] - 22s 34ms/step - loss: 0.0082 - accuracy: 0.9976 - val_loss: 0.0815 - val_accuracy: 0.9832\n",
            "Epoch 17/20\n",
            "653/653 [==============================] - 23s 36ms/step - loss: 0.0111 - accuracy: 0.9967 - val_loss: 0.0749 - val_accuracy: 0.9806\n",
            "Epoch 18/20\n",
            "653/653 [==============================] - 20s 31ms/step - loss: 0.0069 - accuracy: 0.9978 - val_loss: 0.0938 - val_accuracy: 0.9807\n",
            "Epoch 19/20\n",
            "653/653 [==============================] - 24s 37ms/step - loss: 0.0110 - accuracy: 0.9962 - val_loss: 0.0826 - val_accuracy: 0.9802\n",
            "Epoch 20/20\n",
            "653/653 [==============================] - 24s 36ms/step - loss: 0.0107 - accuracy: 0.9966 - val_loss: 0.1265 - val_accuracy: 0.9769\n",
            "Number of epochs run: 20\n",
            "CPU times: user 6min 37s, sys: 13.6 s, total: 6min 50s\n",
            "Wall time: 7min 32s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_bow, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P3DkIQDC-wj0",
        "outputId": "236d9d35-d219-434d-8ba9-79f7a69ccf57"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 6s 16ms/step - loss: 0.1249 - accuracy: 0.9756\n",
            "Testing Accuracy is 97.56056666374207 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "OZqEOxVC8urr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the RNN with one BiLSTM layer, as well as dropout layers\n",
        "rnn_model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Bidirectional(LSTM(128, return_sequences=True)),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "rnn_model.compile(loss='binary_crossentropy',\n",
        "              optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Train the RNN Model for 150 epochs with early stopping (patience of 7) and\n",
        "# learning rate reduction (by a factor  of 0.5, patience of 7)\n",
        "\n",
        "reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5,\n",
        "                              patience=7, min_lr=0.000001)\n",
        "earlystop7 = EarlyStopping(monitor='loss', patience=7)\n",
        "\n",
        "history = rnn_model.fit(X_train_bow, y_train_bow, epochs=150,\n",
        "                        callbacks=[reduce_lr, earlystop7],\n",
        "                        validation_data=(X_val_bow, y_val_bow))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Vg0J_FAPwUj",
        "outputId": "3779a8eb-6454-4f47-9f18-c861c49670bb"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/150\n",
            "653/653 [==============================] - 21s 26ms/step - loss: 0.3037 - accuracy: 0.8636 - val_loss: 0.1404 - val_accuracy: 0.9457 - lr: 0.0030\n",
            "Epoch 2/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.1346 - accuracy: 0.9523 - val_loss: 0.0941 - val_accuracy: 0.9634 - lr: 0.0030\n",
            "Epoch 3/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.1112 - accuracy: 0.9616 - val_loss: 0.1007 - val_accuracy: 0.9648 - lr: 0.0030\n",
            "Epoch 4/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0883 - accuracy: 0.9697 - val_loss: 0.0781 - val_accuracy: 0.9711 - lr: 0.0030\n",
            "Epoch 5/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0601 - accuracy: 0.9814 - val_loss: 0.0910 - val_accuracy: 0.9671 - lr: 0.0030\n",
            "Epoch 6/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0427 - accuracy: 0.9861 - val_loss: 0.0722 - val_accuracy: 0.9753 - lr: 0.0030\n",
            "Epoch 7/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0324 - accuracy: 0.9904 - val_loss: 0.0671 - val_accuracy: 0.9776 - lr: 0.0030\n",
            "Epoch 8/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0291 - accuracy: 0.9910 - val_loss: 0.0806 - val_accuracy: 0.9763 - lr: 0.0030\n",
            "Epoch 9/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0251 - accuracy: 0.9918 - val_loss: 0.0835 - val_accuracy: 0.9731 - lr: 0.0030\n",
            "Epoch 10/150\n",
            "653/653 [==============================] - 17s 25ms/step - loss: 0.0347 - accuracy: 0.9884 - val_loss: 0.0737 - val_accuracy: 0.9761 - lr: 0.0030\n",
            "Epoch 11/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0227 - accuracy: 0.9931 - val_loss: 0.0846 - val_accuracy: 0.9760 - lr: 0.0030\n",
            "Epoch 12/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0145 - accuracy: 0.9960 - val_loss: 0.0742 - val_accuracy: 0.9805 - lr: 0.0030\n",
            "Epoch 13/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0138 - accuracy: 0.9957 - val_loss: 0.0609 - val_accuracy: 0.9805 - lr: 0.0030\n",
            "Epoch 14/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0194 - accuracy: 0.9939 - val_loss: 0.0703 - val_accuracy: 0.9786 - lr: 0.0030\n",
            "Epoch 15/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0195 - accuracy: 0.9941 - val_loss: 0.0722 - val_accuracy: 0.9809 - lr: 0.0030\n",
            "Epoch 16/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0142 - accuracy: 0.9954 - val_loss: 0.0897 - val_accuracy: 0.9795 - lr: 0.0030\n",
            "Epoch 17/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0119 - accuracy: 0.9961 - val_loss: 0.0793 - val_accuracy: 0.9813 - lr: 0.0030\n",
            "Epoch 18/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0121 - accuracy: 0.9961 - val_loss: 0.0900 - val_accuracy: 0.9809 - lr: 0.0030\n",
            "Epoch 19/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0118 - accuracy: 0.9960 - val_loss: 0.0777 - val_accuracy: 0.9796 - lr: 0.0030\n",
            "Epoch 20/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0102 - accuracy: 0.9967 - val_loss: 0.0897 - val_accuracy: 0.9810 - lr: 0.0030\n",
            "Epoch 21/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0071 - accuracy: 0.9978 - val_loss: 0.0931 - val_accuracy: 0.9795 - lr: 0.0015\n",
            "Epoch 22/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0067 - accuracy: 0.9979 - val_loss: 0.0982 - val_accuracy: 0.9795 - lr: 0.0015\n",
            "Epoch 23/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0067 - accuracy: 0.9977 - val_loss: 0.0868 - val_accuracy: 0.9797 - lr: 0.0015\n",
            "Epoch 24/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0060 - accuracy: 0.9984 - val_loss: 0.0821 - val_accuracy: 0.9816 - lr: 0.0015\n",
            "Epoch 25/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0044 - accuracy: 0.9986 - val_loss: 0.1022 - val_accuracy: 0.9803 - lr: 0.0015\n",
            "Epoch 26/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0053 - accuracy: 0.9985 - val_loss: 0.0931 - val_accuracy: 0.9807 - lr: 0.0015\n",
            "Epoch 27/150\n",
            "653/653 [==============================] - 17s 25ms/step - loss: 0.0031 - accuracy: 0.9993 - val_loss: 0.1178 - val_accuracy: 0.9797 - lr: 0.0015\n",
            "Epoch 28/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0041 - accuracy: 0.9988 - val_loss: 0.0961 - val_accuracy: 0.9813 - lr: 7.5000e-04\n",
            "Epoch 29/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0036 - accuracy: 0.9991 - val_loss: 0.0922 - val_accuracy: 0.9810 - lr: 7.5000e-04\n",
            "Epoch 30/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0036 - accuracy: 0.9990 - val_loss: 0.0877 - val_accuracy: 0.9810 - lr: 7.5000e-04\n",
            "Epoch 31/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0028 - accuracy: 0.9993 - val_loss: 0.0991 - val_accuracy: 0.9802 - lr: 7.5000e-04\n",
            "Epoch 32/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0027 - accuracy: 0.9994 - val_loss: 0.0935 - val_accuracy: 0.9825 - lr: 7.5000e-04\n",
            "Epoch 33/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0029 - accuracy: 0.9992 - val_loss: 0.0960 - val_accuracy: 0.9826 - lr: 7.5000e-04\n",
            "Epoch 34/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0028 - accuracy: 0.9992 - val_loss: 0.1042 - val_accuracy: 0.9803 - lr: 7.5000e-04\n",
            "Epoch 35/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0021 - accuracy: 0.9994 - val_loss: 0.0984 - val_accuracy: 0.9823 - lr: 3.7500e-04\n",
            "Epoch 36/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0016 - accuracy: 0.9997 - val_loss: 0.1033 - val_accuracy: 0.9823 - lr: 3.7500e-04\n",
            "Epoch 37/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0027 - accuracy: 0.9993 - val_loss: 0.0967 - val_accuracy: 0.9826 - lr: 3.7500e-04\n",
            "Epoch 38/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0020 - accuracy: 0.9995 - val_loss: 0.0986 - val_accuracy: 0.9818 - lr: 3.7500e-04\n",
            "Epoch 39/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0014 - accuracy: 0.9997 - val_loss: 0.1067 - val_accuracy: 0.9812 - lr: 3.7500e-04\n",
            "Epoch 40/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0020 - accuracy: 0.9993 - val_loss: 0.1041 - val_accuracy: 0.9820 - lr: 3.7500e-04\n",
            "Epoch 41/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0021 - accuracy: 0.9995 - val_loss: 0.1046 - val_accuracy: 0.9818 - lr: 3.7500e-04\n",
            "Epoch 42/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0021 - accuracy: 0.9995 - val_loss: 0.1063 - val_accuracy: 0.9818 - lr: 1.8750e-04\n",
            "Epoch 43/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0017 - accuracy: 0.9996 - val_loss: 0.1035 - val_accuracy: 0.9815 - lr: 1.8750e-04\n",
            "Epoch 44/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0014 - accuracy: 0.9996 - val_loss: 0.1074 - val_accuracy: 0.9822 - lr: 1.8750e-04\n",
            "Epoch 45/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0016 - accuracy: 0.9995 - val_loss: 0.1068 - val_accuracy: 0.9813 - lr: 1.8750e-04\n",
            "Epoch 46/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0017 - accuracy: 0.9996 - val_loss: 0.1120 - val_accuracy: 0.9825 - lr: 1.8750e-04\n",
            "Epoch 47/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0010 - accuracy: 0.9998 - val_loss: 0.1083 - val_accuracy: 0.9826 - lr: 1.8750e-04\n",
            "Epoch 48/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0017 - accuracy: 0.9995 - val_loss: 0.1090 - val_accuracy: 0.9825 - lr: 1.8750e-04\n",
            "Epoch 49/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0014 - accuracy: 0.9996 - val_loss: 0.1046 - val_accuracy: 0.9823 - lr: 9.3750e-05\n",
            "Epoch 50/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0012 - accuracy: 0.9998 - val_loss: 0.1100 - val_accuracy: 0.9823 - lr: 9.3750e-05\n",
            "Epoch 51/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0011 - accuracy: 0.9998 - val_loss: 0.1069 - val_accuracy: 0.9825 - lr: 9.3750e-05\n",
            "Epoch 52/150\n",
            "653/653 [==============================] - 16s 24ms/step - loss: 0.0015 - accuracy: 0.9996 - val_loss: 0.1106 - val_accuracy: 0.9818 - lr: 9.3750e-05\n",
            "Epoch 53/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0016 - accuracy: 0.9996 - val_loss: 0.1078 - val_accuracy: 0.9833 - lr: 9.3750e-05\n",
            "Epoch 54/150\n",
            "653/653 [==============================] - 16s 25ms/step - loss: 0.0011 - accuracy: 0.9997 - val_loss: 0.1085 - val_accuracy: 0.9813 - lr: 9.3750e-05\n",
            "Number of epochs run: 54\n",
            "CPU times: user 15min 12s, sys: 57.8 s, total: 16min 10s\n",
            "Wall time: 14min 42s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = rnn_model.evaluate(X_test_bow, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5YyaCadBPwgi",
        "outputId": "e041ae78-7b54-4ba3-b686-77df3d606872"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 4s 11ms/step - loss: 0.1335 - accuracy: 0.9782\n",
            "Testing Accuracy is 97.82043695449829 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My final BiLSTM model that used the continuous Bag of Words model achieved an accuracy of 97.82% and took 16 min and 10 s of CPU time to train."
      ],
      "metadata": {
        "id": "J1SNKFtBw8Me"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Part 2: Scaling"
      ],
      "metadata": {
        "id": "Dq8YHLOKq4Xj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## DL Prototype\n",
        "\n",
        "I will first implement a new type of feature extraction (pretrained FastText word embeddings), and then compare the results of this new form of feature extraction with the previously used ones (GloVe, continuous Bag of Words) via a CNN (Convolutional Neural Network) model."
      ],
      "metadata": {
        "id": "HZxmi_wyxy3u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Feature Extraction"
      ],
      "metadata": {
        "id": "GRF6lHBQVvFu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### FastText word embeddings"
      ],
      "metadata": {
        "id": "F3FGPgliVxN0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Pad the sequences so they are all the same length\n",
        "\n",
        "max_length = 500\n",
        "\n",
        "X_train_all_ft = pad_sequences(X_train_all_sequences, maxlen=max_length)\n",
        "X_test_ft = pad_sequences(X_test_sequences, maxlen=max_length)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V-npiqmrWLnT",
        "outputId": "2bca4b60-8e0e-4b86-b4c7-3fcce2eab59f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 386 ms, sys: 27.4 ms, total: 413 ms\n",
            "Wall time: 406 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Split the training dataset into training and validation datasets\n",
        "X_train_ft, X_val_ft, y_train_ft, y_val_ft = train_test_split(X_train_all_ft,\n",
        "                                                                y_train_all,\n",
        "                                                                random_state=1)\n",
        "\n",
        "print(\"Shape of training set:\", X_train_ft.shape)\n",
        "print(\"Shape of validation set:\", X_val_ft.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad14c500-a14a-4bf9-9e52-6328d396c860",
        "id": "sR1CEn_5WbOW"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of training set: (20875, 500)\n",
            "Shape of validation set: (6959, 500)\n",
            "CPU times: user 13.8 ms, sys: 9.98 ms, total: 23.8 ms\n",
            "Wall time: 23.1 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Method to load the pretrained FastText embeddings\n",
        "\n",
        "word2idx = tokenizer.word_index\n",
        "vocab_size = len(tokenizer.word_index)\n",
        "embed_size = 300\n",
        "FASTTEXT_INIT_EMBEDDINGS_FILE = DATA_PATH + 'wiki-news-300d-1M-subword.vec'\n",
        "\n",
        "\n",
        "def load_pretrained_embeddings(word_to_index, max_features, embedding_size, embedding_file_path):\n",
        "\n",
        "    def get_coefs(word,*arr):\n",
        "        return word, np.asarray(arr, dtype='float32')\n",
        "\n",
        "    embeddings_index = dict(get_coefs(*row.split(\" \"))\n",
        "                                for row in open(embedding_file_path, encoding=\"utf8\", errors='ignore')\n",
        "                                    if len(row)>100)\n",
        "\n",
        "    all_embs = np.stack(embeddings_index.values())\n",
        "    emb_mean, emb_std = all_embs.mean(), all_embs.std()\n",
        "    embed_size = all_embs.shape[1]\n",
        "\n",
        "    nb_words = min(max_features, len(word_to_index))\n",
        "    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embedding_size))\n",
        "\n",
        "    for word, idx in word_to_index.items():\n",
        "        if idx >= max_features:\n",
        "            continue\n",
        "        embedding_vector = embeddings_index.get(word)\n",
        "        if embedding_vector is not None:\n",
        "            embedding_matrix[idx] = embedding_vector\n",
        "\n",
        "    return embedding_matrix"
      ],
      "metadata": {
        "id": "Nei4m9S4XDNT"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Call the above method to get the FastText embeddings\n",
        "\n",
        "ft_embeddings = load_pretrained_embeddings(word_to_index=word2idx,\n",
        "                                           max_features=vocab_size,\n",
        "                                           embedding_size=embed_size,\n",
        "                                           embedding_file_path=FASTTEXT_INIT_EMBEDDINGS_FILE)\n",
        "ft_embeddings.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SqPCaamnXDfJ",
        "outputId": "fb0c0d5e-1d58-4fbc-81ab-9d4d257c6fc9"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/IPython/core/magics/execution.py:1335: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
            "  exec(code, glob, local_ns)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1min 7s, sys: 3.11 s, total: 1min 10s\n",
            "Wall time: 1min 9s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(144330, 300)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embedding_layer_ft = Embedding(input_dim=vocab_size,\n",
        "                            output_dim=embed_size,\n",
        "                            input_length=max_length,\n",
        "                            weights=[ft_embeddings],\n",
        "                            trainable=True)"
      ],
      "metadata": {
        "id": "XNXpa_MrWbOY"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### CNN with GloVe word embeddings"
      ],
      "metadata": {
        "id": "fXEWmjc71FJw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Preliminary Model"
      ],
      "metadata": {
        "id": "XQWoQ2xYx4pj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses GloVe word embeddings\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Conv1D(128, 5, activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = cnn_model.fit(X_train_glove, y_train_glove, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_glove, y_val_glove))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "id": "BYDvhvKF8n9G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ea65a7f4-8f57-49fa-e8db-22853b7934fe"
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 7s 7ms/step - loss: 0.3020 - accuracy: 0.8646 - val_loss: 0.2344 - val_accuracy: 0.9006\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 7s 10ms/step - loss: 0.1247 - accuracy: 0.9556 - val_loss: 0.2032 - val_accuracy: 0.9111\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0588 - accuracy: 0.9814 - val_loss: 0.1529 - val_accuracy: 0.9386\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 3s 5ms/step - loss: 0.0291 - accuracy: 0.9916 - val_loss: 0.1570 - val_accuracy: 0.9404\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0195 - accuracy: 0.9943 - val_loss: 0.1777 - val_accuracy: 0.9414\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0192 - accuracy: 0.9935 - val_loss: 0.2008 - val_accuracy: 0.9349\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0188 - accuracy: 0.9936 - val_loss: 0.2182 - val_accuracy: 0.9353\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 4s 7ms/step - loss: 0.0215 - accuracy: 0.9924 - val_loss: 0.2231 - val_accuracy: 0.9381\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 5s 7ms/step - loss: 0.0196 - accuracy: 0.9932 - val_loss: 0.2448 - val_accuracy: 0.9343\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 3s 5ms/step - loss: 0.0150 - accuracy: 0.9946 - val_loss: 0.2547 - val_accuracy: 0.9343\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 3s 5ms/step - loss: 0.0123 - accuracy: 0.9955 - val_loss: 0.2627 - val_accuracy: 0.9353\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 4s 5ms/step - loss: 0.0144 - accuracy: 0.9951 - val_loss: 0.2851 - val_accuracy: 0.9352\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 5s 7ms/step - loss: 0.0165 - accuracy: 0.9941 - val_loss: 0.2855 - val_accuracy: 0.9336\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 4s 5ms/step - loss: 0.0153 - accuracy: 0.9947 - val_loss: 0.2902 - val_accuracy: 0.9348\n",
            "Number of epochs run: 14\n",
            "CPU times: user 59.6 s, sys: 4.02 s, total: 1min 3s\n",
            "Wall time: 1min\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "88FMwAaP--7Y",
        "outputId": "6d7b86d2-041a-4603-9530-2584da5531f9"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.3280 - accuracy: 0.9326\n",
            "Testing Accuracy is 93.26012134552002 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My CNN model that uses GloVe word embeddings is clearly overfitting given the vast discreptancy between its training and testing accuracy (99.48% vs 93.76%)."
      ],
      "metadata": {
        "id": "B9BokPZ_na4g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Exploring Dropout Layers\n",
        "\n",
        "Adding dropout layers will prevent overfitting, so I will first see what dropout rate is ideal."
      ],
      "metadata": {
        "id": "ZVPtsYqoAVsh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Experiment with different values for dropout rate\n",
        "\n",
        "losses = []\n",
        "accs = []\n",
        "dropouts = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\n",
        "\n",
        "for dropout in dropouts:\n",
        "    print(\"Dropout: {}\".format(dropout))\n",
        "\n",
        "    # Create and compile model\n",
        "    model = Sequential([\n",
        "      embedding_layer_glove,\n",
        "      Dropout(dropout),\n",
        "      Conv1D(128, 5, activation='relu'),\n",
        "      GlobalMaxPooling1D(),\n",
        "      Dropout(dropout),\n",
        "      Dense(10, activation='relu'),\n",
        "      Dropout(dropout),\n",
        "      Dense(1, activation='sigmoid')\n",
        "    ])\n",
        "\n",
        "    model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "    # Fit model\n",
        "    callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)\n",
        "    history = model.fit(X_train_glove, y_train_glove, epochs=20,\n",
        "                            callbacks=[callback],\n",
        "                            validation_data=(X_val_glove, y_val_glove))\n",
        "    print(\"Number of epochs run: {}\".format(len(history.history['loss'])))\n",
        "\n",
        "    # Evaluate model\n",
        "    loss, accuracy = model.evaluate(X_test_glove, y_test, verbose=0)\n",
        "    print('Test loss:', loss)\n",
        "    print('Test accuracy:', accuracy)\n",
        "    losses.append(loss)\n",
        "    accs.append(accuracy)\n",
        "\n",
        "    print('\\n----------------------------------------------\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvZIYiEIHo54",
        "outputId": "d6ae48d1-efa9-431d-a1bb-a12bb63098b8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dropout: 0.0\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 26s 50ms/step - loss: 0.3445 - accuracy: 0.8427 - val_loss: 0.2973 - val_accuracy: 0.8681\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 28s 56ms/step - loss: 0.1468 - accuracy: 0.9471 - val_loss: 0.2194 - val_accuracy: 0.9100\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.0744 - accuracy: 0.9783 - val_loss: 0.2050 - val_accuracy: 0.9180\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.0386 - accuracy: 0.9910 - val_loss: 0.2018 - val_accuracy: 0.9246\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 22s 45ms/step - loss: 0.0296 - accuracy: 0.9924 - val_loss: 0.2694 - val_accuracy: 0.9086\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.0217 - accuracy: 0.9951 - val_loss: 0.2710 - val_accuracy: 0.9129\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 19s 38ms/step - loss: 0.0151 - accuracy: 0.9958 - val_loss: 0.2481 - val_accuracy: 0.9190\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.0134 - accuracy: 0.9962 - val_loss: 0.2407 - val_accuracy: 0.9242\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.0153 - accuracy: 0.9950 - val_loss: 0.2919 - val_accuracy: 0.9187\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 19s 40ms/step - loss: 0.0193 - accuracy: 0.9934 - val_loss: 0.4102 - val_accuracy: 0.8950\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.0207 - accuracy: 0.9930 - val_loss: 0.3264 - val_accuracy: 0.9164\n",
            "Number of epochs run: 11\n",
            "Test loss: 0.32636451721191406\n",
            "Test accuracy: 0.9163965582847595\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.1\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 24s 47ms/step - loss: 0.4193 - accuracy: 0.7909 - val_loss: 0.3106 - val_accuracy: 0.8566\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 21s 42ms/step - loss: 0.2572 - accuracy: 0.8921 - val_loss: 0.2284 - val_accuracy: 0.9035\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 29s 60ms/step - loss: 0.1990 - accuracy: 0.9191 - val_loss: 0.2051 - val_accuracy: 0.9128\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 26s 52ms/step - loss: 0.1598 - accuracy: 0.9386 - val_loss: 0.2050 - val_accuracy: 0.9144\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 23s 46ms/step - loss: 0.1391 - accuracy: 0.9457 - val_loss: 0.2262 - val_accuracy: 0.9115\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 20s 42ms/step - loss: 0.1245 - accuracy: 0.9536 - val_loss: 0.2145 - val_accuracy: 0.9114\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1065 - accuracy: 0.9593 - val_loss: 0.1899 - val_accuracy: 0.9280\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1012 - accuracy: 0.9627 - val_loss: 0.2208 - val_accuracy: 0.9214\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 22s 46ms/step - loss: 0.0948 - accuracy: 0.9656 - val_loss: 0.2590 - val_accuracy: 0.9130\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.0805 - accuracy: 0.9696 - val_loss: 0.1941 - val_accuracy: 0.9313\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 25s 52ms/step - loss: 0.0793 - accuracy: 0.9696 - val_loss: 0.2114 - val_accuracy: 0.9237\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.0751 - accuracy: 0.9725 - val_loss: 0.2132 - val_accuracy: 0.9252\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 23s 46ms/step - loss: 0.0718 - accuracy: 0.9721 - val_loss: 0.2270 - val_accuracy: 0.9263\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.0664 - accuracy: 0.9759 - val_loss: 0.2580 - val_accuracy: 0.9197\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.0674 - accuracy: 0.9755 - val_loss: 0.2126 - val_accuracy: 0.9304\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 21s 42ms/step - loss: 0.0595 - accuracy: 0.9785 - val_loss: 0.2222 - val_accuracy: 0.9308\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.0593 - accuracy: 0.9785 - val_loss: 0.1985 - val_accuracy: 0.9342\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 21s 42ms/step - loss: 0.0535 - accuracy: 0.9803 - val_loss: 0.2499 - val_accuracy: 0.9241\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.0499 - accuracy: 0.9820 - val_loss: 0.2218 - val_accuracy: 0.9312\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.0498 - accuracy: 0.9819 - val_loss: 0.2420 - val_accuracy: 0.9300\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.24204352498054504\n",
            "Test accuracy: 0.9300324320793152\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.2\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 25s 48ms/step - loss: 0.4907 - accuracy: 0.7459 - val_loss: 0.3216 - val_accuracy: 0.8559\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.3263 - accuracy: 0.8553 - val_loss: 0.2696 - val_accuracy: 0.8796\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.2790 - accuracy: 0.8817 - val_loss: 0.2440 - val_accuracy: 0.8949\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.2458 - accuracy: 0.8952 - val_loss: 0.2459 - val_accuracy: 0.8908\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.2197 - accuracy: 0.9078 - val_loss: 0.2100 - val_accuracy: 0.9137\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 21s 44ms/step - loss: 0.2013 - accuracy: 0.9191 - val_loss: 0.1948 - val_accuracy: 0.9209\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.1876 - accuracy: 0.9222 - val_loss: 0.1956 - val_accuracy: 0.9199\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.1763 - accuracy: 0.9281 - val_loss: 0.1984 - val_accuracy: 0.9211\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 29s 60ms/step - loss: 0.1607 - accuracy: 0.9369 - val_loss: 0.2061 - val_accuracy: 0.9164\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.1508 - accuracy: 0.9402 - val_loss: 0.2130 - val_accuracy: 0.9185\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.1484 - accuracy: 0.9423 - val_loss: 0.1760 - val_accuracy: 0.9314\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.1389 - accuracy: 0.9456 - val_loss: 0.1837 - val_accuracy: 0.9286\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.1347 - accuracy: 0.9481 - val_loss: 0.1704 - val_accuracy: 0.9346\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1270 - accuracy: 0.9509 - val_loss: 0.1789 - val_accuracy: 0.9296\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1261 - accuracy: 0.9508 - val_loss: 0.1748 - val_accuracy: 0.9341\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1201 - accuracy: 0.9545 - val_loss: 0.1707 - val_accuracy: 0.9348\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1181 - accuracy: 0.9556 - val_loss: 0.1755 - val_accuracy: 0.9337\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1101 - accuracy: 0.9560 - val_loss: 0.1786 - val_accuracy: 0.9339\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.1097 - accuracy: 0.9596 - val_loss: 0.1894 - val_accuracy: 0.9312\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.1023 - accuracy: 0.9624 - val_loss: 0.1867 - val_accuracy: 0.9346\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.18673953413963318\n",
            "Test accuracy: 0.9346149563789368\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.3\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6092 - accuracy: 0.6554 - val_loss: 0.4264 - val_accuracy: 0.7956\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 22s 45ms/step - loss: 0.4545 - accuracy: 0.7713 - val_loss: 0.3203 - val_accuracy: 0.8569\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.3883 - accuracy: 0.8168 - val_loss: 0.2825 - val_accuracy: 0.8757\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 24s 50ms/step - loss: 0.3458 - accuracy: 0.8413 - val_loss: 0.2632 - val_accuracy: 0.8854\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.3180 - accuracy: 0.8579 - val_loss: 0.2444 - val_accuracy: 0.8949\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.3004 - accuracy: 0.8670 - val_loss: 0.2295 - val_accuracy: 0.9041\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.2797 - accuracy: 0.8802 - val_loss: 0.2203 - val_accuracy: 0.9088\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 33s 67ms/step - loss: 0.2642 - accuracy: 0.8854 - val_loss: 0.2151 - val_accuracy: 0.9108\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 39s 80ms/step - loss: 0.2483 - accuracy: 0.8961 - val_loss: 0.2161 - val_accuracy: 0.9096\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.2399 - accuracy: 0.9000 - val_loss: 0.2079 - val_accuracy: 0.9126\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 29s 60ms/step - loss: 0.2340 - accuracy: 0.9040 - val_loss: 0.2019 - val_accuracy: 0.9168\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.2319 - accuracy: 0.9043 - val_loss: 0.1959 - val_accuracy: 0.9182\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 22s 46ms/step - loss: 0.2164 - accuracy: 0.9085 - val_loss: 0.1841 - val_accuracy: 0.9243\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 21s 44ms/step - loss: 0.2087 - accuracy: 0.9154 - val_loss: 0.1860 - val_accuracy: 0.9246\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.2052 - accuracy: 0.9161 - val_loss: 0.1914 - val_accuracy: 0.9185\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.2055 - accuracy: 0.9149 - val_loss: 0.1865 - val_accuracy: 0.9211\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.1969 - accuracy: 0.9192 - val_loss: 0.1765 - val_accuracy: 0.9270\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.1885 - accuracy: 0.9225 - val_loss: 0.1817 - val_accuracy: 0.9266\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.1808 - accuracy: 0.9264 - val_loss: 0.1749 - val_accuracy: 0.9270\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.1763 - accuracy: 0.9315 - val_loss: 0.1815 - val_accuracy: 0.9244\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.18149618804454803\n",
            "Test accuracy: 0.9244439601898193\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.4\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6453 - accuracy: 0.5981 - val_loss: 0.5000 - val_accuracy: 0.7780\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.5264 - accuracy: 0.7165 - val_loss: 0.4092 - val_accuracy: 0.8136\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.4704 - accuracy: 0.7569 - val_loss: 0.3444 - val_accuracy: 0.8598\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.4278 - accuracy: 0.7840 - val_loss: 0.3129 - val_accuracy: 0.8689\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 26s 52ms/step - loss: 0.4047 - accuracy: 0.8018 - val_loss: 0.2979 - val_accuracy: 0.8744\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.3882 - accuracy: 0.8114 - val_loss: 0.2840 - val_accuracy: 0.8819\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.3719 - accuracy: 0.8200 - val_loss: 0.2688 - val_accuracy: 0.8854\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.3543 - accuracy: 0.8438 - val_loss: 0.2581 - val_accuracy: 0.8870\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 26s 54ms/step - loss: 0.3474 - accuracy: 0.8467 - val_loss: 0.2506 - val_accuracy: 0.8963\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.3358 - accuracy: 0.8526 - val_loss: 0.2432 - val_accuracy: 0.8995\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.3314 - accuracy: 0.8581 - val_loss: 0.2435 - val_accuracy: 0.8986\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 31s 63ms/step - loss: 0.3197 - accuracy: 0.8617 - val_loss: 0.2289 - val_accuracy: 0.9032\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.3071 - accuracy: 0.8678 - val_loss: 0.2260 - val_accuracy: 0.9099\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.3036 - accuracy: 0.8705 - val_loss: 0.2178 - val_accuracy: 0.9102\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.2973 - accuracy: 0.8726 - val_loss: 0.2215 - val_accuracy: 0.9049\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.2938 - accuracy: 0.8743 - val_loss: 0.2187 - val_accuracy: 0.9110\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 32s 65ms/step - loss: 0.2877 - accuracy: 0.8767 - val_loss: 0.2086 - val_accuracy: 0.9164\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 21s 42ms/step - loss: 0.2816 - accuracy: 0.8763 - val_loss: 0.2156 - val_accuracy: 0.9120\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 23s 46ms/step - loss: 0.2732 - accuracy: 0.8853 - val_loss: 0.2038 - val_accuracy: 0.9161\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.2659 - accuracy: 0.8884 - val_loss: 0.1976 - val_accuracy: 0.9190\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.19761542975902557\n",
            "Test accuracy: 0.9189672470092773\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.5\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 24s 47ms/step - loss: 0.7016 - accuracy: 0.5176 - val_loss: 0.6773 - val_accuracy: 0.5157\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.6440 - accuracy: 0.5934 - val_loss: 0.5598 - val_accuracy: 0.7443\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 22s 45ms/step - loss: 0.5628 - accuracy: 0.6863 - val_loss: 0.4505 - val_accuracy: 0.8137\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.5225 - accuracy: 0.7119 - val_loss: 0.4142 - val_accuracy: 0.8253\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 26s 54ms/step - loss: 0.4965 - accuracy: 0.7320 - val_loss: 0.3851 - val_accuracy: 0.8408\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 26s 52ms/step - loss: 0.4772 - accuracy: 0.7511 - val_loss: 0.3490 - val_accuracy: 0.8539\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.4575 - accuracy: 0.7621 - val_loss: 0.3429 - val_accuracy: 0.8524\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.4478 - accuracy: 0.7758 - val_loss: 0.3284 - val_accuracy: 0.8669\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 21s 44ms/step - loss: 0.4340 - accuracy: 0.7848 - val_loss: 0.3081 - val_accuracy: 0.8763\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 28s 56ms/step - loss: 0.4204 - accuracy: 0.7927 - val_loss: 0.3028 - val_accuracy: 0.8721\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 30s 61ms/step - loss: 0.4134 - accuracy: 0.7997 - val_loss: 0.2981 - val_accuracy: 0.8730\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.4063 - accuracy: 0.8011 - val_loss: 0.2898 - val_accuracy: 0.8764\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.3981 - accuracy: 0.8073 - val_loss: 0.2815 - val_accuracy: 0.8788\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 27s 55ms/step - loss: 0.3890 - accuracy: 0.8160 - val_loss: 0.2718 - val_accuracy: 0.8887\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 27s 55ms/step - loss: 0.3893 - accuracy: 0.8142 - val_loss: 0.2725 - val_accuracy: 0.8821\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 24s 50ms/step - loss: 0.3855 - accuracy: 0.8173 - val_loss: 0.2638 - val_accuracy: 0.8887\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.3760 - accuracy: 0.8216 - val_loss: 0.2626 - val_accuracy: 0.8952\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 26s 54ms/step - loss: 0.3678 - accuracy: 0.8231 - val_loss: 0.2623 - val_accuracy: 0.8892\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.3666 - accuracy: 0.8280 - val_loss: 0.2568 - val_accuracy: 0.8907\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.3585 - accuracy: 0.8281 - val_loss: 0.2469 - val_accuracy: 0.8985\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.246877521276474\n",
            "Test accuracy: 0.8985134959220886\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.6\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.7232 - accuracy: 0.5230 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.6925 - accuracy: 0.5229 - val_loss: 0.6915 - val_accuracy: 0.5157\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6890 - accuracy: 0.5241 - val_loss: 0.6794 - val_accuracy: 0.5157\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.6774 - accuracy: 0.5431 - val_loss: 0.6407 - val_accuracy: 0.6317\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.6515 - accuracy: 0.5802 - val_loss: 0.5954 - val_accuracy: 0.7512\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.6355 - accuracy: 0.5943 - val_loss: 0.5473 - val_accuracy: 0.7731\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 25s 52ms/step - loss: 0.6230 - accuracy: 0.6035 - val_loss: 0.5452 - val_accuracy: 0.7709\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.6082 - accuracy: 0.6120 - val_loss: 0.5052 - val_accuracy: 0.7866\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.5946 - accuracy: 0.6284 - val_loss: 0.4709 - val_accuracy: 0.8062\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.5757 - accuracy: 0.6526 - val_loss: 0.4568 - val_accuracy: 0.8167\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.5685 - accuracy: 0.6635 - val_loss: 0.4536 - val_accuracy: 0.8144\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.5667 - accuracy: 0.6745 - val_loss: 0.4485 - val_accuracy: 0.8193\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.5556 - accuracy: 0.6859 - val_loss: 0.4256 - val_accuracy: 0.8199\n",
            "Epoch 14/20\n",
            "490/490 [==============================] - 25s 50ms/step - loss: 0.5475 - accuracy: 0.6895 - val_loss: 0.4139 - val_accuracy: 0.8237\n",
            "Epoch 15/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.5367 - accuracy: 0.7084 - val_loss: 0.4039 - val_accuracy: 0.8255\n",
            "Epoch 16/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.5331 - accuracy: 0.7245 - val_loss: 0.3941 - val_accuracy: 0.8315\n",
            "Epoch 17/20\n",
            "490/490 [==============================] - 21s 44ms/step - loss: 0.5252 - accuracy: 0.7331 - val_loss: 0.4004 - val_accuracy: 0.8280\n",
            "Epoch 18/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.5178 - accuracy: 0.7426 - val_loss: 0.3788 - val_accuracy: 0.8353\n",
            "Epoch 19/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.5074 - accuracy: 0.7550 - val_loss: 0.3837 - val_accuracy: 0.8321\n",
            "Epoch 20/20\n",
            "490/490 [==============================] - 25s 52ms/step - loss: 0.5061 - accuracy: 0.7549 - val_loss: 0.3698 - val_accuracy: 0.8368\n",
            "Number of epochs run: 20\n",
            "Test loss: 0.3697502911090851\n",
            "Test accuracy: 0.8368167877197266\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.7\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 25s 49ms/step - loss: 0.8339 - accuracy: 0.4957 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 25s 52ms/step - loss: 0.6968 - accuracy: 0.5219 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6933 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6929 - accuracy: 0.5221 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6923 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.6926 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.6922 - accuracy: 0.5222 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 22s 45ms/step - loss: 0.6925 - accuracy: 0.5222 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.6922 - accuracy: 0.5220 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.6922 - accuracy: 0.5220 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Number of epochs run: 10\n",
            "Test loss: 0.6927679181098938\n",
            "Test accuracy: 0.515703558921814\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.8\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 25s 48ms/step - loss: 1.0345 - accuracy: 0.5194 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 25s 51ms/step - loss: 0.7017 - accuracy: 0.5213 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 25s 50ms/step - loss: 0.6970 - accuracy: 0.5213 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6936 - accuracy: 0.5215 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 21s 44ms/step - loss: 0.6944 - accuracy: 0.5220 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 26s 52ms/step - loss: 0.6929 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6934 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 22s 46ms/step - loss: 0.6926 - accuracy: 0.5225 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 26s 53ms/step - loss: 0.6931 - accuracy: 0.5224 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 21s 43ms/step - loss: 0.6928 - accuracy: 0.5220 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 22s 45ms/step - loss: 0.6927 - accuracy: 0.5220 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Number of epochs run: 11\n",
            "Test loss: 0.6927429437637329\n",
            "Test accuracy: 0.515703558921814\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "Dropout: 0.9\n",
            "Epoch 1/20\n",
            "490/490 [==============================] - 25s 48ms/step - loss: 2.3970 - accuracy: 0.5209 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 2/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.8465 - accuracy: 0.5222 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 3/20\n",
            "490/490 [==============================] - 24s 50ms/step - loss: 0.7452 - accuracy: 0.5213 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 4/20\n",
            "490/490 [==============================] - 23s 47ms/step - loss: 0.7238 - accuracy: 0.5210 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 5/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.7052 - accuracy: 0.5221 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 6/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6999 - accuracy: 0.5216 - val_loss: 0.6928 - val_accuracy: 0.5157\n",
            "Epoch 7/20\n",
            "490/490 [==============================] - 25s 52ms/step - loss: 0.7011 - accuracy: 0.5223 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 8/20\n",
            "490/490 [==============================] - 24s 48ms/step - loss: 0.6974 - accuracy: 0.5217 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 9/20\n",
            "490/490 [==============================] - 25s 50ms/step - loss: 0.6958 - accuracy: 0.5215 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 10/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6924 - accuracy: 0.5218 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 11/20\n",
            "490/490 [==============================] - 24s 49ms/step - loss: 0.6943 - accuracy: 0.5232 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 12/20\n",
            "490/490 [==============================] - 22s 44ms/step - loss: 0.6930 - accuracy: 0.5226 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Epoch 13/20\n",
            "490/490 [==============================] - 23s 48ms/step - loss: 0.6930 - accuracy: 0.5219 - val_loss: 0.6927 - val_accuracy: 0.5157\n",
            "Number of epochs run: 13\n",
            "Test loss: 0.6927050352096558\n",
            "Test accuracy: 0.515703558921814\n",
            "\n",
            "----------------------------------------------\n",
            "\n",
            "CPU times: user 1h 21min 4s, sys: 1min 56s, total: 1h 23min\n",
            "Wall time: 1h 8min 32s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot Accuracy Results\n",
        "\n",
        "width = 0.08\n",
        "\n",
        "plt.bar(dropouts, accs, width, align='center')\n",
        "\n",
        "plt.tick_params(axis='both', which='major', labelsize=8)\n",
        "plt.tick_params(axis='both', which='minor', labelsize=8)\n",
        "\n",
        "plt.ylabel('Accuracy',size = 15)\n",
        "plt.xlabel('Dropout', size = 15)\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        },
        "id": "YF1JkmWtKyRu",
        "outputId": "39ce5bb1-2185-4c25-928a-19a08d62233e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGyCAYAAAAGdNXrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAnTUlEQVR4nO3de3TU9Z3/8deQgbggidgSMuRCgBjkooRAkIsI0bqNLYGuKC2HCFECwcJ6NF1FrF1EKHrsuq4VqQHaCIJ30Noq2lUIgsVCFImAksASMxIu9qCZBBCTnc/vD5f8TJPA5JtJZvjwfJwz5zAzn3y/73yP4tPvXL4uY4wRAACAZTqEegAAAIC2QOQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEruUA8QKn6/X5WVleratatcLleoxwEAAAEwxqi6ulo9e/ZUhw5nP1dzwUZOZWWlEhISQj0GAABwwOv1Kj4+/qxrLtjI6dq1q6RvD1JUVFSIpwEAAIHw+XxKSEio/+/42VywkXPmJaqoqCgiBwCA80wgbzXhjccAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKzkDvUAQDAk3ft6qEdooPzhH4d6BAC44BE5aIBYAADYgperAACAlYgcAABgJV6uAkKIlwcBoO1wJgcAAFiJMzkAWowzUADOB0ROG+E/AgAAhBYvVwEAACsROQAAwEpEDgAAsBKRAwAArETkAAAAK/HpKgAXDD71CFxYOJMDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASu5QDwAAOLuke18P9QgNlD/841CPAASEMzkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASmEZOWVlZRo1apRSUlKUnp6uPXv2NFrj9/uVn5+vAQMG6Morr1RGRob2798fgmkBAEA4CsvIycvL06xZs1RaWqp58+YpJyen0ZrXXntN7733nnbt2qWSkhJdd911uu+++9p/WAAAEJbCLnKOHTum4uJiZWdnS5ImTZokr9fb6CyNy+XS6dOn9fXXX8sYI5/Pp/j4+Ga3e/r0afl8vgY3AABgr7C7QKfX65XH45Hb/e1oLpdLiYmJqqioUHJycv26rKwsbdq0SbGxseratavi4uK0efPmZrf70EMPaeHChW0+PwAACA9hdyYnUMXFxdq9e7cOHTqkyspKXXfddZo9e3az6+fPn6+qqqr6m9frbcdpAQBAewu7MzkJCQk6fPiw6urq5Ha7ZYxRRUWFEhMTG6xbvXq1rr32Wl1yySWSpOnTp+uf//mfm91uZGSkIiMj23J0AAAQRsLuTE5MTIzS0tK0Zs0aSdK6desUHx/f4KUqSerTp482btyob775RpL05z//WYMGDWr3eQEAQHgKuzM5klRQUKCcnBwtWbJEUVFRKiwslCTl5uZqwoQJmjBhgubMmaNPPvlEgwcPVseOHRUbG6unnnoqxJMDAIBwEZaR069fP23btq3R4ytXrqz/c2RkpFasWNGeYwEAgPNI2L1cBQAAEAxEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwkqPIWbVqlb7++utgzwIAABA0jiLn1ltvVc+ePfWv//qv2rVrV7BnUllZmUaNGqWUlBSlp6drz549Ta77+OOPNW7cOPXv31/9+/fX+vXrgz4LAAA4PzmKnNzcXNXV1enJJ59UWlqaRowYod///vc6ceJEUIbKy8vTrFmzVFpaqnnz5iknJ6fRmpMnT2rixIlavHixPvnkE+3evVtjxowJyv4BAMD5z1HkLF++XIcPH9by5cuVnp6u7du3a9asWerZs6dmz56t4uJixwMdO3ZMxcXFys7OliRNmjRJXq9X+/fvb7Du2Wef1YgRI3T11VdLkiIiItS9e/dmt3v69Gn5fL4GNwAAYC/Hbzzu0qWLcnNz9f7776ukpERz5syR2+3W8uXLddVVV2nIkCF66qmnWhwTXq9XHo9HbrdbkuRyuZSYmKiKiooG6/bu3avIyEiNHz9eqampmjZtmr744otmt/vQQw8pOjq6/paQkNDyXxoAAJw3gvLpqkGDBum3v/2tKisrtWbNGl1zzTXatWuX5syZo549e2rGjBn64IMPgrGrenV1dXr77bdVUFCgnTt3Ki4uTrfffnuz6+fPn6+qqqr6m9frDeo8AAAgvAT1I+S1tbWqrq5WdXW1JMkYo9raWhUWFmr48OG66aab9NVXX511GwkJCTp8+LDq6urqt1FRUaHExMQG6xITE5WRkaG4uDi5XC5lZ2fr/fffb3a7kZGRioqKanADAAD2CkrkvP/++5oxY4Y8Ho9+/vOfq6SkRDfeeKP+8pe/yOfzae3atbriiiv0yiuv6I477jjrtmJiYpSWlqY1a9ZIktatW6f4+HglJyc3WDd58mTt2LGj/uWwN954Q4MHDw7GrwMAACzgdvqDX375pZ555hmtWLFCe/fulTFGCQkJmjdvnnJzcxUbG1u/dsqUKbr55ps1ZMgQvfHGG+fcdkFBgXJycrRkyRJFRUWpsLBQ0ref6powYYImTJigxMRE3XfffRo1apQ6dOiguLg4LV++3OmvAwAALOMocrKzs7V+/XqdPn1aLpdLN9xwg2bPnq0f/ehH6tCh6ZNDbrdb6enpWrVq1Tm3369fP23btq3R4ytXrmxw/5ZbbtEtt9zi5FcAAACWcxQ5zz77rGJjY3Xbbbdp1qxZjd4v05x/+Zd/Ua9evZzsEgAAoEUcRc5LL72kiRMn1n/MO1BZWVnKyspysksAAIAWcRQ5kyZNCvYcAAAAQeXo01Uffvih8vPztWPHjmbXbN++Xfn5+froo4+czgYAAOCYo8hZunSpli1bpqSkpGbX9O7dW8uWLdOTTz7pdDYAAADHHEXOli1blJaWdtZrRXXv3l1paWnavHmz4+EAAACcchQ5hw4dOutZnDN69eqlyspKJ7sAAABoFUeRExkZec7LM0iSz+dTRESEk10AAAC0iqPIGThwoLZu3arjx483u+b48eN69913NWDAAMfDAQAAOOUocrKzs1VTU6ObbrpJn3/+eaPnDx06pMmTJ+vkyZOaOnVqq4cEAABoKUffk5Obm6vnnntORUVFSklJUWZmpvr27StJOnDggN566y2dOnVKo0eP1uzZs4M6MAAAQCAcRY7b7daGDRt0xx13aNWqVXr11VcbPB8REaFbb71Vjz/+eIu/FRkAACAYHBdI586dtXLlSi1atEhFRUXyer2SpISEBI0bN04ejydoQwIAALRUq0+zeDweTZkyJRizAAAABI2jNx4DAACEu1adyTl58qQ2bdqksrIyVVdXyxjTaI3L5dKvfvWr1uwGAACgxRxHztNPP6277rpLPp+v/jFjjFwuV6P7RA4AAGhvjl6uevvttzVjxgy5XC7dd999GjlypCSpoKBAd999t5KTk2WM0dy5c/WHP/whqAMDAAAEwlHkPProo3K5XNq0aZMWLVqkyy67TJI0c+ZMPfzww9qzZ4/uvPNO/eEPf9DQoUODOjAAAEAgHEXOjh07NGLECA0ePLjJ591ut/7jP/5DMTExWrBgQasGBAAAcMJR5NTU1CgxMbH+fmRkpCSpurr6/2+4QwddddVV2rJlSytHBAAAaDlHkRMbG9vg4pxnvvivtLS0wbrjx4/r1KlTrRgPAADAGUeRc/nll6usrKz+/qhRo2SM0SOPPFL/MfK//vWv2rhxo/r16xecSQEAAFrAUeT8+Mc/1sGDB7V9+3ZJ0nXXXacrr7xSL7/8suLi4jR06FBlZGTI7/frzjvvDOa8AAAAAXEUOdOmTdOGDRvUo0ePbzfSoYNef/11XX/99Tp27Jh27typzp07a/HixcrOzg7qwAAAAIFw9GWA0dHR+uEPf9jgsbi4OL355ps6efKkqqqqFBMTo4iIiKAMCQA4/yTd+3qoR2ig/OEfB7SOuYMj0LnbkqPIyc/PV7du3Zr8JuPOnTurc+fOrR4MAACgNRy9XLV06VKVlJQEexYAAICgcRQ58fHx8vv9wZ4FAAAgaBxFzk9+8hNt3ry5wZf/AQAAhBNHkbNw4UIlJibqRz/6kXbu3BnsmQAAAFrN0RuPJ06cqMjISL333nsaNmyYPB6PEhMTddFFFzVa63K59M4777R6UAAAgJZwFDlFRUX1fzbGqLKyUpWVlU2udblcjgYDAABoDUeRc/DgwWDPAQAAEFSOIqdXr17BngMAACCoHL3xGAAAINw5OpNTUVHRovWJiYlOdgMAAOCYo8hJSkoK+A3FLpdLdXV1TnYDAADgmKPIueaaa5qMHL/fL6/Xq4qKCvn9fo0cOVKdOnVq9ZAAAAAt1eqPkDeltLRUubm5MsZow4YNTnYBAADQKm3yxuOUlBStX79ee/fu1YIFC9piFwAAAGfVZp+u+v73v6+rrrpKzz//fFvtAgAAoFlt+hFyY4yOHj3alrsAAABoUptFzs6dO7V582a+OBAAAISEozceP/jgg80+V1NTo9LSUm3YsEF1dXXKy8tzPBwAAIBTjiLngQcekMvlkjGm2TWdO3fW/PnzlZ+f73g4AAAApxxFTmFhYbPPderUSR6PR+np6erSpYvjwQAAAFrDUeRMnz492HMAAAAEFRfoBAAAVnIUOR9++KHy8/O1Y8eOZtds375d+fn5+uijj5zOBgAA4JijyFm6dKmWLVumpKSkZtf07t1by5Yt05NPPul0NgAAAMccRc6WLVuUlpam7t27N7ume/fuSktL0+bNmx0PBwAA4JSjyDl06NBZz+Kc0atXL1VWVjrZBQAAQKs4ipzIyEh99dVX51zn8/kUERHhZBcAAACt4ihyBg4cqK1bt+r48ePNrjl+/LjeffddDRgwwPFwAAAATjmKnOzsbNXU1Oimm27S559/3uj5Q4cOafLkyTp58qSmTp3a6iEBAABaytGXAebm5uq5555TUVGRUlJSlJmZqb59+0qSDhw4oLfeekunTp3S6NGjNXv27KAODAAAEAhHkeN2u7VhwwbdcccdWrVqlV599dUGz0dEROjWW2/V448/Lrfb0S4AAABaxXGBdO7cWStXrtSiRYtUVFQkr9crSUpISNC4cePk8XiCNiQAAEBLtfo0i8fj0ZQpU4IxCwAAQNA4euOx3++Xz+dTbW1ts2tqa2vl8/nk9/sdDwcAAOCUo8h57LHH1K1bt7N+m/HmzZvVrVs3PfHEE46HAwAAcMpR5LzyyitKSEjQD37wg2bX/OAHP1B8fLzWrVvneDgAAACnHEVOWVmZBg4ceM51gwYNUllZmZNdAAAAtIqjyKmqqlJ0dPQ510VHR+vLL790sgsAAIBWcRQ5Ho9HJSUl51xXUlKimJgYJ7sAAABoFUeRc+211+qTTz7RCy+80OyaF198UXv37lVGRkaLt19WVqZRo0YpJSVF6enp2rNnT7NrjTG69tprdckll7R4PwAAwF6OIufuu+9Wp06dNG3aNM2dO1clJSU6ceKETpw4oZKSEs2dO1e33HKLOnXqpLvvvrvF28/Ly9OsWbNUWlqqefPmKScnp9m1jz32WP0lJQAAAM5wFDmXX365Vq9erYiICP3ud7/TkCFDFBUVpaioKA0ZMkTLli1TRESEVq1apUGDBrVo28eOHVNxcbGys7MlSZMmTZLX69X+/fsbrd2zZ49effVV3Xvvvefc7unTp+Xz+RrcAACAvRxFjiTdfPPNKikpUV5enpKTkxUZGanIyEglJyfr9ttv165du/TTn/60xV8G6PV65fF46q955XK5lJiYqIqKigbramtrNXPmTBUUFCgiIuKc233ooYcUHR1df0tISGjRXAAA4PziOHIkKTk5WcuWLdO+fft08uRJnTx5Uvv27dOTTz6pmpoa5efnKz4+PlizNrBw4ULdeOON6t+/f0Dr58+fr6qqqvrbmWttAQAAOwX1EuFer1dr167VmjVr9Mknn8gYI5fL1aJtJCQk6PDhw6qrq5Pb7ZYxRhUVFUpMTGywbvPmzaqoqNDSpUtVV1cnn8+npKQk7dixQ927d2+03TNnmgAAwIWh1ZFTXV2tl156SWvWrNG7774rY4yMMYqLi9NPf/rTFl+8MyYmRmlpaVqzZo1ycnK0bt06xcfHKzk5ucG6LVu21P+5vLxcqampKi8vb+2vAwAALOEocv73f/9Xb775pp555hn96U9/0tdffy1jjKRv30NTVFSkMWPGtPgszhkFBQXKycnRkiVLFBUVpcLCQklSbm6uJkyYoAkTJjjaLgAAuHC0KHJ27NihZ555Ri+88IL+/ve/yxijjh07asKECcrOztYjjzyi4uJiXXPNNa0aql+/ftq2bVujx1euXNnk+qSkJH311Vet2icAALBLQJGzePFirV27VqWlpfVnbEaNGqXs7GxNnjxZl156qSTpv/7rv9psUAAAgJYIKHL+/d//XS6XS7Gxsfr5z3+uqVOnKikpqY1HAwAAcC7gj5AbY3TkyBG99dZb+u///m9eHgIAAGEtoMj529/+pjlz5uh73/uetm7dqtmzZ8vj8WjSpElav369amtr23pOAACAFgkoctLT0/XEE0+osrJSf/zjH3XTTTfJ5XLplVde0c033yyPx6O8vDwdPXq0recFAAAISIu+8djtdisrK0svvPCCjhw5ohUrVmjMmDH68ssvtWLFCh04cECSdO+99+qjjz5qi3kBAAAC4viyDlFRUZoxY4aKiopUXl6uX//617r88stljNFvfvMbDR06VP3799eiRYuCOS8AAEBAWnXtqjMSEhI0f/587dmzR8XFxbrjjjsUExOjffv26YEHHgjGLgAAAFokKJHzXWlpaXrsscd06NAhvf766/rZz34W7F0AAACcU1Av0PldHTp00A033KAbbrihrXYBAADQrKCfyQEAAAgHRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAK4Vl5JSVlWnUqFFKSUlRenq69uzZ02jNxo0bNXz4cA0YMEADBw7UPffcI7/fH4JpAQBAOArLyMnLy9OsWbNUWlqqefPmKScnp9Gabt266fnnn9fevXv1wQcf6K9//atWr17d/sMCAICwFHaRc+zYMRUXFys7O1uSNGnSJHm9Xu3fv7/BuiFDhqhPnz6SpIsuukipqakqLy9vdrunT5+Wz+drcAMAAPYKu8jxer3yeDxyu92SJJfLpcTERFVUVDT7M0eOHNHLL7+s8ePHN7vmoYceUnR0dP0tISEh6LMDAIDwEXaR01I+n09ZWVm65557NGzYsGbXzZ8/X1VVVfU3r9fbjlMCAID25g71AP8oISFBhw8fVl1dndxut4wxqqioUGJiYqO11dXVyszM1MSJE5Wfn3/W7UZGRioyMrKtxgYAAGEm7M7kxMTEKC0tTWvWrJEkrVu3TvHx8UpOTm6wrqamRpmZmcrMzNT9998filEBAEAYC7vIkaSCggIVFBQoJSVFDz/8sAoLCyVJubm5eu211yRJjz/+uLZv367169crNTVVqamp+vWvfx3KsQEAQBgJu5erJKlfv37atm1bo8dXrlxZ/+df/vKX+uUvf9meYwEAgPNIWJ7JAQAAaC0iBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWCsvIKSsr06hRo5SSkqL09HTt2bOnyXW///3vddlll6lv376aOXOmamtr23lSAAAQrsIycvLy8jRr1iyVlpZq3rx5ysnJabTm4MGD+tWvfqUtW7Zo//79Onr0qJYvX97+wwIAgLDkDvUA/+jYsWMqLi7WX/7yF0nSpEmTNHfuXO3fv1/Jycn1615++WVNmDBBsbGxkqTZs2dryZIlmjNnTpPbPX36tE6fPl1/v6qqSpLk8/na5Pfwnz7ZJtt1KtDfk7mDg7nbF3O3L+ZuX7bP7XS7xphzLzZhpri42KSkpDR4LD093bzzzjsNHps7d65ZsmRJ/f09e/aYhISEZre7YMECI4kbN27cuHHjZsHN6/WesynC7kxOW5k/f77y8/Pr7/v9fh0/flzf+9735HK5QjhZ83w+nxISEuT1ehUVFRXqcazH8W5fHO/2xfFuXxzvtmOMUXV1tXr27HnOtWEXOQkJCTp8+LDq6urkdrtljFFFRYUSExMbrEtMTNSBAwfq75eXlzda812RkZGKjIxs8Ngll1wS1NnbSlRUFP+StCOOd/vieLcvjnf74ni3jejo6IDWhd0bj2NiYpSWlqY1a9ZIktatW6f4+PgG78eRvn2vzmuvvaYjR47IGKOnnnpKP/vZz0IxMgAACENhFzmSVFBQoIKCAqWkpOjhhx9WYWGhJCk3N1evvfaaJKlPnz5auHChRo8ereTkZHXv3l15eXmhHBsAAISRsHu5SpL69eunbdu2NXp85cqVDe7PnDlTM2fObK+x2l1kZKQWLFjQ6GU2tA2Od/vieLcvjnf74niHB5cxgXwGCwAA4PwSli9XAQAAtBaRAwAArETkAAAAKxE5YYALkravQI73xo0bNXz4cA0YMEADBw7UPffcI7/fH4Jpz3+B/vMtffslX9dee+158x1W4SjQ4/3xxx9r3Lhx6t+/v/r376/169e386R2COR4+/1+5efna8CAAbryyiuVkZGh/fv3h2DaC1DA11tAm8nIyDCFhYXGGGNeeuklM2zYsEZr/ud//sd4PB5z+PBh4/f7TVZWllm6dGk7T2qHQI73hx9+aA4cOGCMMebUqVNm9OjR9T+DlgnkeJ/x6KOPmtzcXBMdHd0+w1kokON94sQJ07t3b7NlyxZjjDF1dXXm2LFj7TmmNQI53q+88ooZPny4+eabb4wxxixatMjcfPPN7TnmBYvICbGjR4+arl27mtraWmOMMX6/3/To0cOUlZU1WPfII4+YvLy8+vuvv/66GT16dLvOaoNAj/c/mjNnjlmwYEE7TGiXlhzv3bt3mzFjxpj9+/cTOQ4FerxXrFhhpkyZEooRrRLo8X711VfN4MGDjc/nM36/39x9993mrrvuCsXIFxxergoxr9crj8cjt/vbryxyuVxKTExURUVFg3UVFRXq1atX/f2kpKRGa3BugR7v7zpy5IhefvlljR8/vr3GtEagx7u2tlYzZ85UQUGBIiIiQjGqFQI93nv37lVkZKTGjx+v1NRUTZs2TV988UUoRj6vBXq8s7KyNG7cOMXGxsrj8eidd97Rgw8+GIqRLzhEDnAWPp9PWVlZuueeezRs2LBQj2OthQsX6sYbb1T//v1DPcoFoa6uTm+//bYKCgq0c+dOxcXF6fbbbw/1WNYqLi7W7t27dejQIVVWVuq6667T7NmzQz3WBYHICbHvXpBU0lkvSPrZZ5/V3z/XBUnRtECPtyRVV1crMzNTEydObHAFewQu0OO9efNmPfHEE0pKStLVV18tn8+npKQkzi60UEv+PsnIyFBcXJxcLpeys7P1/vvvh2Lk81qgx3v16tX1b6jv0KGDpk+frk2bNoVi5AsOkRNiXJC0fQV6vGtqapSZmanMzEzdf//9oRjVCoEe7y1btuizzz5TeXm5tm7dqqioKJWXl6t79+6hGPu8Fejxnjx5snbs2CGfzydJeuONNzR48OB2n/d8F+jx7tOnjzZu3KhvvvlGkvTnP/9ZgwYNavd5L0ihfEMQvvXpp5+aESNGmMsuu8wMHTrUlJSUGGOMmTFjhvnjH/9Yv2758uWmT58+pk+fPua2226rf6c+WiaQ47148WLjdrvN4MGD62+LFy8O5djnrUD/+T7j4MGDvPG4FQI93qtXrzYDBw40V1xxhcnMzDQVFRWhGvm8Fsjx/vrrr01ubq65/PLLzRVXXGGuv/76+k9vom1x7SoAAGAlXq4CAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCV3qAcAcH5yuVwN7rvdbkVHR8vj8Wjo0KHKysrSxIkT6y9eCADtjS8DBODImciZPn26JMnv96uqqkqlpaXat2+fjDFKTk7W2rVrNXz48FCOGnLl5eXq3bu3xo4dq6KiolCPA1ww+F8sAK3y9NNPN3rswIEDuu+++/Tiiy8qIyND7733nlJTU9t9NgAXNt6TAyDo+vbtqxdeeEEzZszQyZMnddttt4V6JAAXICIHQJt59NFH1aVLF+3cuVNbt26tf7y8vFwul0vjxo2Tz+dTfn6+evfurY4dO+rOO++sX7d3715NnTpVHo9HnTp1UlxcnKZNm6Z9+/Y12ldRUZFcLpdycnJ0+PBh5eTkqEePHvqnf/onpaWlafXq1c3O2ZL9PP3003K5XHrggQea3Na4cePkcrlUXl4uSXrggQfUu3dvSdLmzZvlcrnqbzk5Oec+iAAcI3IAtJno6GjdcMMNkqRNmzY1ev7UqVMaO3asnn76aaWmpmrChAnq1q2bJOmdd97RsGHD9Oyzz8rj8WjSpEmKiYnRM888o2HDhmnLli1N7vP48eMaMWKE3nzzTY0bN05jxozRxx9/rOnTpzcZJk73E6jU1FRNmjRJktSjRw9Nnz69/nb11Ve3atsAziGk10AHcN6SZAL5K2Tx4sVGkpkyZUr9YwcPHqz/+ZEjR5ovv/yywc/U1NSYHj16GElm6dKlDZ77z//8TyPJxMfHm1OnTtU/vmnTpvptXn/99aampqb+ue3bt5uLL77YdOjQwXzwwQet2k9hYaGRZBYsWNDk7zt27FgjyRw8eLDR7zt27NhzHS4AQcSZHABt6vvf/74k6csvv2zy+d/+9re65JJLGjz24osv6ujRoxo5cqTmzJnT4Lm77rpLQ4cO1eeff65169Y12l6HDh30xBNPqEuXLvWPpaena86cOfL7/Vq2bFlQ9gMg/BE5ANqU+b9vqfjH79WRJI/Ho2HDhjV6/MxLRFOnTm1ym9nZ2Q3WfVdqaqr69evX6PEpU6Y0+pnW7AdA+CNyALSpv//975KkSy+9tNFziYmJTf5MZWWlJCkpKanJ5888fujQoUbP9erV66w/c2bbrd0PgPBH5ABoUzt37pQkDRgwoNFzF110kaNtNnVWqC042Y/f72+DSQA4QeQAaDNVVVV66623JEkZGRkB/1zPnj0lSZ999lmTz5/5eHZcXFyj55r7mTOPn9m20/106tRJklRTU9Pkz3i93iYfB9D+iBwAbeYXv/iFTpw4ofT0dI0cOTLgnxszZowk6bnnnmvy+TVr1jRY910fffSRysrKGj3+/PPPS1KDj2072Y/H45EklZaWNlpfWlqqioqKRo+fCaO6urom9wOgjYT6410Azk86y0fIDxw4YCZPnmwkmS5dupiSkpIGz5/rI9Xf/Wh3QUFBg+cef/xxI8nExcU1+xHyH/7wh+bEiRP1zxUXF5uuXbsal8tlduzY0ar9VFdXm86dOxu3222Ki4vrH//iiy/MmDFj6mf47kfIT58+bTp27GhiY2NNXV1dk78zgODj2lUAWuXMt/b6/X75fD6Vlpbq008/lTFGl112mZ599lldccUVLdpmly5dtHbtWmVlZSkvL0/Lly9XSkqKPv30U+3cuVMXX3yxnnvuuSbf0zN+/Hjt2rVLffv21TXXXKOqqipt3LhRtbW1uv/++xt8msvJfi6++GL927/9mx588EFdffXVGjt2rFwul/72t7+pf//+GjlypLZt29Zgpk6dOikzM1N/+tOfNHjwYKWlpalTp04aPXq0br311hYdGwAtEOrKAnB+0v+dsThzc7vd5tJLLzWDBg0y06dPN+vXr2/2rEWgX463e/duM2XKFNOjRw/TsWNH4/F4THZ2tvn0008brT1zJmf69Onm0KFDJjs723Tv3t1ERkaawYMHm8LCwqDsxxhj/H6/+c1vfmOSk5NNx44dTXx8vPnFL35hTpw40eSXARpjzNGjR80tt9xiYmNjTURERP2sANqOy5j/+xILADiPFRUVKSMjQ9OnT2/yyugALjy88RgAAFiJyAEAAFYicgAAgJV4Tw4AALASZ3IAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAVvp/LnZCcHd4iLkAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot Loss Results\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "width = 0.08\n",
        "\n",
        "plt.bar(dropouts, losses, width, align='center',color = 'green')\n",
        "\n",
        "plt.tick_params(axis='both', which='major', labelsize=8)\n",
        "plt.tick_params(axis='both', which='minor', labelsize=8)\n",
        "\n",
        "plt.ylabel('Loss',size = 15)\n",
        "plt.xlabel('Dropout', size = 15)\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        },
        "id": "orOiCes0LECt",
        "outputId": "91e7e8f6-a88e-4ee9-bdaf-a77c2cd3e29f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGyCAYAAAAGdNXrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmF0lEQVR4nO3df3RU9Z3/8dfAwLQiGWv5FZKZIoRBQEwMxIUAJXHXY7YHcDWnHKmpRA1JW3v2tLiFxuJSqpVse2x16/HsWGxcNv7YSsCm9gc9FXFDN1uJC4phIQlLmAEi0ZXNBKSRmM/3D5b5Ok2QCZPMXD48H+fcc8ydz9x55x5rnr3zy2WMMQIAALDMsFQPAAAAMBSIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYyZ3qAVKlt7dXx44d0+jRo+VyuVI9DgAAiIMxRl1dXZo4caKGDfvkazWXbeQcO3ZMPp8v1WMAAICLEA6HlZmZ+YlrLtvIGT16tKSzJyktLS3F0wAAgHhEIhH5fL7o3/FPctlGzrmnqNLS0ogcAAAuMfG81IQXHgMAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACs5MjIaWlpUX5+vgKBgPLy8tTU1NRnTXV1tXJycqLbmDFjdPvtt6dgWgAA4ESOjJyKigqVl5erublZa9asUWlpaZ81d999t/bs2RPdJkyYoDvvvDP5wwIAAEdyGWNMqof4uI6ODmVlZen999+X2+2WMUbp6enauXOnsrKy+r3PH//4Ry1ZskRHjx7ViBEj+l3T3d2t7u7u6M/nvvuis7OTr3UAAOASEYlE5PV64/r77bgrOeFwWOnp6XK7z36tlsvlkt/vVygUOu99nn76aX35y18+b+BI0oYNG+T1eqMb30AOAIDdHBc5A3Xq1Cm98MILuvfeez9xXWVlpTo7O6NbOBxO0oQAACAVHPct5D6fT+3t7erp6Yk+XRUKheT3+/td/+KLL2rmzJmaMWPGJx7X4/HI4/EMxcgAAMCBHHclZ9y4ccrNzVVNTY0kqba2VpmZmed9Pc7TTz99was4AADg8uO4Fx5L0oEDB1RaWqr/+Z//UVpamqqrqzVr1iyVlZVp6dKlWrp0aXTdnDlzdOzYMY0ePXpAjzGQFy4BAAbOtd6V6hFimHXx/blj7sER79wDNZC/3457ukqSpk2bpoaGhj77N27c2GddV1dXssYCAACXEMc9XQUAADAYiBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCVHRk5LS4vy8/MVCASUl5enpqamftft3btXBQUFmj59uqZPn64tW7YkeVIAAOBU7lQP0J+KigqVl5ertLRUmzdvVmlpqXbt2hWz5oMPPtCtt96qTZs2acGCBfroo4/0/vvvp2hiAADgNI67ktPR0aHGxkaVlJRIkoqLixUOh9Xa2hqz7rnnntPcuXO1YMECSdLw4cM1duzY8x63u7tbkUgkZgMAAPZyXOSEw2Glp6fL7T57kcnlcsnv9ysUCsWs27dvnzwejxYvXqycnBzdddddevfdd8973A0bNsjr9UY3n883pL8HAABILcdFTrx6enr0+9//XsFgULt371ZGRoa++tWvnnd9ZWWlOjs7o1s4HE7itAAAINkc95ocn8+n9vZ29fT0yO12yxijUCgkv98fs87v96uwsFAZGRmSpJKSEt1yyy3nPa7H45HH4xnS2QEAgHM47krOuHHjlJubq5qaGklSbW2tMjMzlZWVFbNu2bJl2rVrV/S1Nb/+9a+VnZ2d9HkBAIAzOe5KjiQFg0GVlpbqkUceUVpamqqrqyVJZWVlWrp0qZYuXSq/368HHnhA+fn5GjZsmDIyMvTUU0+leHIAAOAULmOMSfUQqRCJROT1etXZ2am0tLRUjwMA1nGtd6V6hBhmXXx/7ph7cMQ790AN5O+3456uAgAAGAxEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwkiMjp6WlRfn5+QoEAsrLy1NTU1OfNTt27NCnP/1p5eTkRLfTp0+nYFoAAOBE7lQP0J+KigqVl5ertLRUmzdvVmlpqXbt2tVn3bRp07Rnz57kDwgAABzPcVdyOjo61NjYqJKSEklScXGxwuGwWltbEzpud3e3IpFIzAYAAOzluMgJh8NKT0+X2332IpPL5ZLf71coFOqz9uDBg8rNzVVeXp6efPLJTzzuhg0b5PV6o5vP5xuS+QEAgDM48umqeOTm5urIkSPyer06cuSIvvCFL2jMmDFatmxZv+srKyu1atWq6M+RSITQAQDAYo67kuPz+dTe3q6enh5JkjFGoVBIfr8/Zl1aWpq8Xq8kKTMzU8uXL1d9ff15j+vxeJSWlhazAQAAezkucsaNG6fc3FzV1NRIkmpra5WZmamsrKyYde3t7ert7ZUkdXV16eWXX9YNN9yQ9HkBAIAzOS5yJCkYDCoYDCoQCKiqqkrV1dWSpLKyMtXV1Uk6Gz+zZs1Sdna25s6dq5tvvll33313KscGAAAO4jLGmFQPkQqRSERer1ednZ08dQUAQ8C13pXqEWKYdfH9uWPuwRHv3AM1kL/fjrySAwAAkCgiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJUdGTktLi/Lz8xUIBJSXl6empqbzrjXG6KabbtJVV12VvAEBAIDjOTJyKioqVF5erubmZq1Zs0alpaXnXfvjH/9YU6ZMSd5wAADgkuC4yOno6FBjY6NKSkokScXFxQqHw2ptbe2ztqmpSS+99JK+/e1vX/C43d3dikQiMRsAALCX4yInHA4rPT1dbrdbkuRyueT3+xUKhWLWnTlzRitXrlQwGNTw4cMveNwNGzbI6/VGN5/PNyTzAwAAZ0gocj744AOFQiGdOnUqZv+JEyf07W9/W4sXL9bXvvY1HTx4MKEh+7N+/Xrdfvvtmj59elzrKysr1dnZGd3C4fCgzwQAAJzDncidH3roIf3gBz/Q66+/rtmzZ0s6+7TQ3Llz1draKmOMJGnz5s168803lZ6efsFj+nw+tbe3q6enR263W8YYhUIh+f3+mHWvvfaaQqGQnnjiCfX09CgSiWjSpEnatWuXxo4d2+e4Ho9HHo8nkV8XAABcQhK6krN9+3ZNmTIlGjiSVFNTo5aWFhUWFmrbtm3627/9W7333nv68Y9/HNcxx40bp9zcXNXU1EiSamtrlZmZqaysrJh19fX1Onz4sNra2rRz506lpaWpra2t38ABAACXn4QiJxQKaerUqTH76urq5HK5VF1drZtvvlmPPfaYAoGAfvOb38R93GAwqGAwqEAgoKqqKlVXV0uSysrKVFdXl8jIAADgMpHQ01UnTpyI+XwaY4x27typ66+/PuaFvdnZ2dq2bVvcx502bZoaGhr67N+4cWO/6ydNmqT//d//jfv4AADAfgldyZkwYYIOHToU/fmNN97QiRMntGjRoph1LpcrkYcBAAAYsIQiJycnR6+//rpeeukldXV16aGHHpLL5dLixYtj1rW0tGjixIkJDQoAADAQCUXO6tWrJZ39wL6rrrpKv/zlL5Wdna2bbropuub48eN68803Y16cDAAAMNQSipz8/Hxt3bpVCxYs0LXXXquSkhLV1dVp2LD/f9jnn39eo0ePVlFRUcLDAgAAxMtlzn2YzWUmEonI6/Wqs7NTaWlpqR4HAKzjWu+s12OadfH9uWPuwRHv3AM1kL/fjvtaBwAAgMGQUOQcP35c//Zv/6bjx4/H7D948KDuuOMOXXfddfrCF77Q79vBAQAAhlJCkVNVVaXCwkJ1dnZG90UiES1YsEAvvvii9u3bp9/+9rf6q7/6K7W0tCQ8LAAAQLwSipwdO3ZoxowZCgQC0X3PPPOMjh8/ruXLl+vAgQP60Y9+pNOnT+vRRx9NeFgAAIB4JRQ5R48e1eTJk2P2/epXv5Lb7dZjjz2mqVOn6hvf+Iays7P12muvJTQoAADAQCT0tQ5dXV264ooroj9/9NFHamho0OzZszVmzJjo/muvvVYvv/xyIg8FAJety+VdM8BgS+hKzsSJE7V///7ozzt37tTJkydVUFAQs66np0cjR45M5KEAAAAGJKHImTdvnt566y099thj2rt3r9auXSuXy6UlS5bErPuv//ovZWRkJDQoAADAQCQUOZWVlfJ4PLr//vuVk5OjP/zhDyooKFB+fn50TVtbm/bt26e/+Iu/SHhYAACAeCX0mpyZM2dq586devzxx/Xee+9p9uzZ+ta3vhWzZtu2bcrOztbf/M3fJPJQAAAAA5JQ5EhSbm6u/vmf//m8t1dUVKiioiLRhwEAABgQvtYBAABYKeErOdLZr3f42c9+pvr6eh09elSSlJGRoc9//vO6++67NX78+MF4GAAAgLglHDm1tbW65557dPLkSX38C8337t2rbdu2qaqqSk8//bSKi4sTfSgAAIC4JfR0VWNjo5YvX65Tp07ptttu09atW7V7927t2bNHL730km6//XadPHlSX/rSl9TY2DhYMwMAAFxQQldyNmzYoI8++kibN2/WbbfdFnPb9ddfr6VLl2rr1q0qLi5WVVWVNm/enNCwAAAA8UroSs7OnTuVn5/fJ3A+7rbbbtP8+fNVX1+fyEMBAAAMSEKR09nZKb/ff8F1fr9fnZ2diTwUAADAgCQUORMmTNDu3bsvuG7Pnj2aMGFCIg8FAAAwIAlFzi233KIDBw7ogQce0EcffdTndmOM1q5dq/3796uoqCiRhwIAABiQhF54/OCDD2rLli36h3/4Bz3//PNatmyZJk2aJEk6fPiwXnzxRbW1temzn/2s1q5dOxjzAgAAxCWhyMnMzNT27dt155136u2339YPf/hDuVwuSYp+Zs6sWbP07LPPKjMzM/FpAQAA4pTwhwHOmjVLb731lnbs2KH6+nodO3ZMkjRx4kQtXLhQBQUFiT4EAADAgA3K1zpIUkFBwXmD5mc/+5mOHDmiv//7vx+shwMAAPhESfmCzp/+9Kdav359Mh4KAABAEt9CDgAALEXkAAAAKw3aa3IQy7XeleoRYph15sKLAACwCFdyAACAlYgcAABgJSIHAABYaUCvyRk+fPhQzQEAADCoBhQ5576q4WKc+7oHAACAZBhQ5PT29g7VHAAAAIOK1+QAAAArETkAAMBKRA4AALCSIyOnpaVF+fn5CgQCysvLU1NTU581DQ0NysnJUU5OjmbOnKmKigp1d3enYFoAAOBEjoyciooKlZeXq7m5WWvWrFFpaWmfNdnZ2dq1a5f27NmjvXv3qqOjQ08++WTyhwUAAI7kuMjp6OhQY2OjSkpKJEnFxcUKh8NqbW2NWXfFFVdoxIgRkqQPP/xQp0+f/sS3qXd3dysSicRsAADAXo6LnHA4rPT0dLndZ9/d7nK55Pf7FQqF+qxta2tTdna2xowZI6/Xq6997WvnPe6GDRvk9Xqjm8/nG7LfAQAApJ7jImcgJk2apDfffFPvvPOOuru7tWXLlvOuraysVGdnZ3QLh8NJnBQAACSb4yLH5/Opvb1dPT09ks5+ynIoFJLf7z/vfa688krdcccdevbZZ8+7xuPxKC0tLWYDAAD2clzkjBs3Trm5uaqpqZEk1dbWKjMzU1lZWTHrWltbdebMGUlnX5OzdetWXX/99UmfFwAAOJPjIkeSgsGggsGgAoGAqqqqVF1dLUkqKytTXV2dJGn79u264YYblJ2drRtuuEHjx4/Xgw8+mMqxAQCAgwzou6uSZdq0aWpoaOizf+PGjdF/Li8vV3l5eTLHAgAAlxBHXskBAABIFJEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsBKRAwAArETkAAAAKxE5AADASkQOAACwEpEDAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEpEDgAAsJI71QMAQLK41rtSPUIMs86kegTAalzJAQAAViJyAACAlXi6CjG4nA8AsAVXcgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlRwZOS0tLcrPz1cgEFBeXp6ampr6rNm+fbtuvPFGzZgxQzNnztTq1avV29ubgmkBAIATOTJyKioqVF5erubmZq1Zs0alpaV91nzmM5/RCy+8oH379umNN97Qv//7v2vTpk3JHxYAADiS4yKno6NDjY2NKikpkSQVFxcrHA6rtbU1Zt0NN9ygyZMnS5I+9alPKScnR21tbec9bnd3tyKRSMwGAADs5bjICYfDSk9Pl9vtliS5XC75/X6FQqHz3uedd97R5s2btXjx4vOu2bBhg7xeb3Tz+XyDPjsAAHAOd6oHSFQkEtGSJUu0evVqzZkz57zrKisrtWrVqpj7ETrAxXGtd6V6hBhmnUn1CAAcyHGR4/P51N7erp6eHrndbhljFAqF5Pf7+6zt6upSUVGRbr311piA6Y/H45HH4xmqsQEAgMM47umqcePGKTc3VzU1NZKk2tpaZWZmKisrK2bdyZMnVVRUpKKiIq1duzYVowIAAAdzXORIUjAYVDAYVCAQUFVVlaqrqyVJZWVlqqurkyQ9/vjjev3117Vlyxbl5OQoJydH3//+91M5NgAAcBDHPV0lSdOmTVNDQ0Of/Rs3boz+83e+8x195zvfSeZYAADgEuLIKzkAAACJcuSVHGCgeLcPAODPcSUHAABYicgBAABW4ukqIIV4mg0Ahg5XcgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJUcGTktLS3Kz89XIBBQXl6empqa+qxpa2tTQUGBvF6vcnJykj8kAABwNEdGTkVFhcrLy9Xc3Kw1a9aotLS0z5q0tDQ9/PDDeu6555I/IAAAcDzHRU5HR4caGxtVUlIiSSouLlY4HFZra2vMuquvvloLFizQqFGj4jpud3e3IpFIzAYAAOzluMgJh8NKT0+X2+2WJLlcLvn9foVCoYSOu2HDBnm93ujm8/kGY1wAAOBQjoucoVJZWanOzs7oFg6HUz0SAAAYQu5UD/DnfD6f2tvb1dPTI7fbLWOMQqGQ/H5/Qsf1eDzyeDyDNCUAAHA6x13JGTdunHJzc1VTUyNJqq2tVWZmprKyslI8GQAAuJQ4LnIkKRgMKhgMKhAIqKqqStXV1ZKksrIy1dXVSZI++OADZWZm6otf/KL27dunzMxMVVZWpnJsAADgII57ukqSpk2bpoaGhj77N27cGP3nK664QkeOHEnmWAAA4BLiyCs5AAAAiSJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFiJyAEAAFYicgAAgJWIHAAAYCUiBwAAWInIAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICVHBk5LS0tys/PVyAQUF5enpqamvpd9/TTT2vq1KmaMmWKVq5cqTNnziR5UgAA4FSOjJyKigqVl5erublZa9asUWlpaZ81hw4d0oMPPqj6+nq1trbq+PHjeuqpp5I/LAAAcCR3qgf4cx0dHWpsbNTvfvc7SVJxcbG+/vWvq7W1VVlZWdF1mzdv1tKlSzVhwgRJ0le+8hU98sgjuu+++/o9bnd3t7q7u6M/d3Z2SpIikcjQ/CJ/GprDXqy4f0/mHhTMnVzMnVzMnVzWz32RxzXGXHixcZjGxkYTCARi9uXl5ZlXXnklZt/Xv/5188gjj0R/bmpqMj6f77zHXbdunZHExsbGxsbGZsEWDocv2BSOu5IzVCorK7Vq1aroz729vXr//ff12c9+Vi6XK4WTnV8kEpHP51M4HFZaWlqqx7Ee5zu5ON/JxflOLs730DHGqKurSxMnTrzgWsdFjs/nU3t7u3p6euR2u2WMUSgUkt/vj1nn9/t18ODB6M9tbW191nycx+ORx+OJ2XfVVVcN6uxDJS0tjf+RJBHnO7k438nF+U4uzvfQ8Hq9ca1z3AuPx40bp9zcXNXU1EiSamtrlZmZGfN6HOnsa3Xq6ur0zjvvyBijf/qnf9Idd9yRipEBAIADOS5yJCkYDCoYDCoQCKiqqkrV1dWSpLKyMtXV1UmSJk+erPXr12v+/PnKysrS2LFjVVFRkcqxAQCAgzju6SpJmjZtmhoaGvrs37hxY8zPK1eu1MqVK5M1VtJ5PB6tW7euz9NsGBqc7+TifCcX5zu5ON/O4DImnvdgAQAAXFoc+XQVAABAoogcAABgJSIHAABYichxAL6QNLniOd/bt2/XjTfeqBkzZmjmzJlavXq1ent7UzDtpS/ef7+lsx/yddNNN10yn2HlRPGe771796qgoEDTp0/X9OnTtWXLliRPaod4zndvb69WrVqlGTNm6Prrr1dhYaFaW1tTMO1lKO7vW8CQKSwsNNXV1cYYY1588UUzZ86cPmv++7//26Snp5v29nbT29trlixZYp544okkT2qHeM73f/7nf5qDBw8aY4w5ffq0mT9/fvQ+GJh4zvc5jz76qCkrKzNerzc5w1konvN96tQpc80115j6+npjjDE9PT2mo6MjmWNaI57zvXXrVnPjjTeaDz/80BhjzEMPPWS++MUvJnPMyxaRk2LHjx83o0ePNmfOnDHGGNPb22vGjx9vWlpaYtb94Ac/MBUVFdGff/WrX5n58+cndVYbxHu+/9x9991n1q1bl4QJ7TKQ8/3222+bhQsXmtbWViLnIsV7vn/605+a5cuXp2JEq8R7vl966SWTnZ1tIpGI6e3tNd/61rfMN7/5zVSMfNnh6aoUC4fDSk9Pl9t99iOLXC6X/H6/QqFQzLpQKKTPfe5z0Z8nTZrUZw0uLN7z/XHvvPOONm/erMWLFydrTGvEe77PnDmjlStXKhgMavjw4akY1Qrxnu99+/bJ4/Fo8eLFysnJ0V133aV33303FSNf0uI930uWLFFBQYEmTJig9PR0vfLKK/re976XipEvO0QO8AkikYiWLFmi1atXa86cOakex1rr16/X7bffrunTp6d6lMtCT0+Pfv/73ysYDGr37t3KyMjQV7/61VSPZa3Gxka9/fbbOnr0qI4dO6a//Mu/1Fe+8pVUj3VZIHJS7ONfSCrpE7+Q9PDhw9GfL/SFpOhfvOdbkrq6ulRUVKRbb7015hvsEb94z/drr72mn/zkJ5o0aZIWLFigSCSiSZMmcXVhgAby35PCwkJlZGTI5XKppKRE//Ef/5GKkS9p8Z7vTZs2RV9QP2zYMK1YsUKvvvpqKka+7BA5KcYXkiZXvOf75MmTKioqUlFRkdauXZuKUa0Q7/mur6/X4cOH1dbWpp07dyotLU1tbW0aO3ZsKsa+ZMV7vpctW6Zdu3YpEolIkn79618rOzs76fNe6uI935MnT9b27dv14YcfSpJefvllXXfddUmf97KUyhcE4az9+/ebuXPnmqlTp5rZs2ebt956yxhjzL333mt+8YtfRNc99dRTZvLkyWby5Mnmnnvuib5SHwMTz/l++OGHjdvtNtnZ2dHt4YcfTuXYl6x4//0+59ChQ7zwOAHxnu9NmzaZmTNnmlmzZpmioiITCoVSNfIlLZ7z/ac//cmUlZWZa6+91syaNcvcfPPN0XdvYmjx3VUAAMBKPF0FAACsROQAAAArETkAAMBKRA4AALASkQMAAKxE5AAAACsROQAAwEruVA8A4NLkcrlifna73fJ6vUpPT9fs2bO1ZMkS3XrrrdEvLwSAZOPDAAFclHORs2LFCklSb2+vOjs71dzcrAMHDsgYo6ysLD377LO68cYbUzlqyrW1temaa67RokWLtGPHjlSPA1w2+L9YABLyzDPP9Nl38OBBPfDAA/r5z3+uwsJC/eEPf1BOTk7SZwNweeM1OQAG3ZQpU/Sv//qvuvfee/XBBx/onnvuSfVIAC5DRA6AIfPoo49q1KhR2r17t3bu3Bnd39bWJpfLpYKCAkUiEa1atUrXXHONRowYoW984xvRdfv27dOdd96p9PR0jRw5UhkZGbrrrrt04MCBPo+1Y8cOuVwulZaWqr29XaWlpRo/frw+/elPKzc3V5s2bTrvnAN5nGeeeUYul0vf/e53+z1WQUGBXC6X2traJEnf/e53dc0110iSXnvtNblcruhWWlp64ZMI4KIROQCGjNfr1V//9V9Lkl599dU+t58+fVqLFi3SM888o5ycHC1dulSf+cxnJEmvvPKK5syZo+eee07p6ekqLi7WuHHj9C//8i+aM2eO6uvr+33M999/X3PnztVvf/tbFRQUaOHChdq7d69WrFjRb5hc7OPEKycnR8XFxZKk8ePHa8WKFdFtwYIFCR0bwAWk9DvQAVyyJJl4/hPy8MMPG0lm+fLl0X2HDh2K3n/evHnmxIkTMfc5efKkGT9+vJFknnjiiZjbfvSjHxlJJjMz05w+fTq6/9VXX40e8+abbzYnT56M3vb666+bK6+80gwbNsy88cYbCT1OdXW1kWTWrVvX7++7aNEiI8kcOnSoz++7aNGiC50uAIOIKzkAhtSYMWMkSSdOnOj39n/8x3/UVVddFbPv5z//uY4fP6558+bpvvvui7ntm9/8pmbPnq0jR46otra2z/GGDRumn/zkJxo1alR0X15enu677z719vbqySefHJTHAeB8RA6AIWX+71Mq/vxzdSQpPT1dc+bM6bP/3FNEd955Z7/HLCkpiVn3cTk5OZo2bVqf/cuXL+9zn0QeB4DzETkAhtR7770nSbr66qv73Ob3+/u9z7FjxyRJkyZN6vf2c/uPHj3a57bPfe5zn3ifc8dO9HEAOB+RA2BI7d69W5I0Y8aMPrd96lOfuqhj9ndVaChczOP09vYOwSQALgaRA2DIdHZ2atu2bZKkwsLCuO83ceJESdLhw4f7vf3c27MzMjL63Ha++5zbf+7YF/s4I0eOlCSdPHmy3/uEw+F+9wNIPiIHwJC5//77derUKeXl5WnevHlx32/hwoWSpOeff77f22tqamLWfdyePXvU0tLSZ/8LL7wgSTFv276Yx0lPT5ckNTc391nf3NysUCjUZ/+5MOrp6en3cQAMkVS/vQvApUmf8BbygwcPmmXLlhlJZtSoUeatt96Kuf1Cb6n++Fu7g8FgzG2PP/64kWQyMjLO+xbyW265xZw6dSp6W2Njoxk9erRxuVxm165dCT1OV1eXueKKK4zb7TaNjY3R/e+++65ZuHBhdIaPv4W8u7vbjBgxwkyYMMH09PT0+zsDGHx8dxWAhJz71N7e3l5FIhE1Nzdr//79MsZo6tSpeu655zRr1qwBHXPUqFF69tlntWTJElVUVOipp55SIBDQ/v37tXv3bl155ZV6/vnn+31Nz+LFi/Xmm29qypQp+vznP6/Ozk5t375dZ86c0dq1a2PezXUxj3PllVfq7/7u7/S9731PCxYs0KJFi+RyufTHP/5R06dP17x589TQ0BAz08iRI1VUVKRf/vKXys7OVm5urkaOHKn58+fr7rvvHtC5ATAAqa4sAJcm/d8Vi3Ob2+02V199tbnuuuvMihUrzJYtW8571SLeD8d7++23zfLly8348ePNiBEjTHp6uikpKTH79+/vs/bclZwVK1aYo0ePmpKSEjN27Fjj8XhMdna2qa6uHpTHMcaY3t5e88Mf/tBkZWWZESNGmMzMTHP//febU6dO9fthgMYYc/z4cfPlL3/ZTJgwwQwfPjw6K4Ch4zLm/z7EAgAuYTt27FBhYaFWrFjR7zejA7j88MJjAABgJSIHAABYicgBAABW4jU5AADASlzJAQAAViJyAACAlYgcAABgJSIHAABYicgBAABWInIAAICViBwAAGAlIgcAAFjp/wFsMg1cek5wuAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Accuracy is highest and loss is lowest when the dropout rate is 0.2."
      ],
      "metadata": {
        "id": "6j_XqniFi8Lt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Hyperparameter Tuning"
      ],
      "metadata": {
        "id": "vdObpYr3Avks"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the model to be optimized\n",
        "\n",
        "def model_to_optimize(num_filters, kernel_size):\n",
        "  model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Dropout(0.2),\n",
        "    Conv1D(num_filters, kernel_size, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    GlobalMaxPooling1D(),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "  ])\n",
        "  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "  return model"
      ],
      "metadata": {
        "id": "_I1pomtAAx5i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the parameter grid\n",
        "\n",
        "params = {\n",
        "    \"num_filters\":[32, 64, 128],\n",
        "    \"kernel_size\":[3, 5, 7],\n",
        "}"
      ],
      "metadata": {
        "id": "JJcWby1ZBJtB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the Keras Classifier\n",
        "\n",
        "model = KerasClassifier(build_fn=model_to_optimize, epochs=20,\n",
        "                        batch_size=10, verbose=3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VW4Do42rBWCF",
        "outputId": "988c1857-c62b-4303-ddb3-b5d5096fd561"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-46-48e61dca825b>:3: DeprecationWarning: KerasClassifier is deprecated, use Sci-Keras (https://github.com/adriangb/scikeras) instead. See https://www.adriangb.com/scikeras/stable/migration.html for help migrating.\n",
            "  model = KerasClassifier(build_fn=model_to_optimize, epochs=20,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Perform the Grid Search\n",
        "\n",
        "grid_search = GridSearchCV(estimator=model, param_grid=params,\n",
        "                              cv=3, verbose=3)\n",
        "grid_search.fit(X_train_all_glove, y_train_all)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "cYmOXtYkB57w",
        "outputId": "12053ba4-6142-4e64-c8c7-838c565e4c43"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 9 candidates, totalling 27 fits\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=3, num_filters=32;, score=0.926 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=3, num_filters=32;, score=0.928 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=3, num_filters=32;, score=0.931 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=3, num_filters=64;, score=0.940 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=3, num_filters=64;, score=0.935 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=3, num_filters=64;, score=0.941 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END ....kernel_size=3, num_filters=128;, score=0.946 total time= 3.7min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END ....kernel_size=3, num_filters=128;, score=0.944 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END ....kernel_size=3, num_filters=128;, score=0.946 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=5, num_filters=32;, score=0.905 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=5, num_filters=32;, score=0.930 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=5, num_filters=32;, score=0.923 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=5, num_filters=64;, score=0.921 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=5, num_filters=64;, score=0.925 total time= 4.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=5, num_filters=64;, score=0.937 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END ....kernel_size=5, num_filters=128;, score=0.937 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END ....kernel_size=5, num_filters=128;, score=0.947 total time= 3.7min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END ....kernel_size=5, num_filters=128;, score=0.947 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=7, num_filters=32;, score=0.890 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=7, num_filters=32;, score=0.906 total time= 3.6min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=7, num_filters=32;, score=0.924 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END .....kernel_size=7, num_filters=64;, score=0.931 total time= 3.8min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END .....kernel_size=7, num_filters=64;, score=0.898 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END .....kernel_size=7, num_filters=64;, score=0.922 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 1/3] END ....kernel_size=7, num_filters=128;, score=0.930 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 2/3] END ....kernel_size=7, num_filters=128;, score=0.942 total time= 3.7min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV 3/3] END ....kernel_size=7, num_filters=128;, score=0.944 total time= 4.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "CPU times: user 1h 57min 16s, sys: 7min 19s, total: 2h 4min 35s\n",
            "Wall time: 1h 56min 13s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=3,\n",
              "             estimator=<keras.wrappers.scikit_learn.KerasClassifier object at 0x7c0df01a8190>,\n",
              "             param_grid={'kernel_size': [3, 5, 7],\n",
              "                         'num_filters': [32, 64, 128]},\n",
              "             verbose=3)"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=3,\n",
              "             estimator=&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7c0df01a8190&gt;,\n",
              "             param_grid={&#x27;kernel_size&#x27;: [3, 5, 7],\n",
              "                         &#x27;num_filters&#x27;: [32, 64, 128]},\n",
              "             verbose=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=3,\n",
              "             estimator=&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7c0df01a8190&gt;,\n",
              "             param_grid={&#x27;kernel_size&#x27;: [3, 5, 7],\n",
              "                         &#x27;num_filters&#x27;: [32, 64, 128]},\n",
              "             verbose=3)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: KerasClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7c0df01a8190&gt;</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KerasClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7c0df01a8190&gt;</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check the cross-validated results of the Grid Search\n",
        "\n",
        "pd.DataFrame(grid_search.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 864
        },
        "id": "jiPu57A4DX9v",
        "outputId": "5e4fd312-e24f-43c3-ccda-bc0ac5340890"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0     228.886374     24.505122         2.752888        0.002102   \n",
              "1     263.826116      0.304330         2.356581        0.289024   \n",
              "2     247.997033     22.011251         2.139256        0.055938   \n",
              "3     228.272594     24.962870         4.497849        1.233063   \n",
              "4     246.401566     24.836475         2.558052        0.324129   \n",
              "5     231.150077     23.189388         2.755982        0.002000   \n",
              "6     246.803588     24.322281         2.703593        0.161299   \n",
              "7     251.703463     16.845218         2.617659        0.293497   \n",
              "8     247.021159     23.433117         3.756544        1.116092   \n",
              "\n",
              "  param_kernel_size param_num_filters                                  params  \\\n",
              "0                 3                32   {'kernel_size': 3, 'num_filters': 32}   \n",
              "1                 3                64   {'kernel_size': 3, 'num_filters': 64}   \n",
              "2                 3               128  {'kernel_size': 3, 'num_filters': 128}   \n",
              "3                 5                32   {'kernel_size': 5, 'num_filters': 32}   \n",
              "4                 5                64   {'kernel_size': 5, 'num_filters': 64}   \n",
              "5                 5               128  {'kernel_size': 5, 'num_filters': 128}   \n",
              "6                 7                32   {'kernel_size': 7, 'num_filters': 32}   \n",
              "7                 7                64   {'kernel_size': 7, 'num_filters': 64}   \n",
              "8                 7               128  {'kernel_size': 7, 'num_filters': 128}   \n",
              "\n",
              "   split0_test_score  split1_test_score  split2_test_score  mean_test_score  \\\n",
              "0           0.926169           0.927786           0.930588         0.928181   \n",
              "1           0.940397           0.934900           0.940612         0.938636   \n",
              "2           0.945570           0.943738           0.946109         0.945139   \n",
              "3           0.905368           0.929511           0.923367         0.919415   \n",
              "4           0.920888           0.925307           0.937379         0.927858   \n",
              "5           0.937163           0.946756           0.946864         0.943594   \n",
              "6           0.890063           0.905691           0.924445         0.906733   \n",
              "7           0.930588           0.898470           0.921858         0.916972   \n",
              "8           0.930050           0.942445           0.943630         0.938708   \n",
              "\n",
              "   std_test_score  rank_test_score  \n",
              "0        0.001826                5  \n",
              "1        0.002644                4  \n",
              "2        0.001015                1  \n",
              "3        0.010245                7  \n",
              "4        0.006970                6  \n",
              "5        0.004548                2  \n",
              "6        0.014056                9  \n",
              "7        0.013560                8  \n",
              "8        0.006142                3  "
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-9f76501a-e2b5-4ac9-9310-dbcc86510943\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_kernel_size</th>\n",
              "      <th>param_num_filters</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>228.886374</td>\n",
              "      <td>24.505122</td>\n",
              "      <td>2.752888</td>\n",
              "      <td>0.002102</td>\n",
              "      <td>3</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 32}</td>\n",
              "      <td>0.926169</td>\n",
              "      <td>0.927786</td>\n",
              "      <td>0.930588</td>\n",
              "      <td>0.928181</td>\n",
              "      <td>0.001826</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>263.826116</td>\n",
              "      <td>0.304330</td>\n",
              "      <td>2.356581</td>\n",
              "      <td>0.289024</td>\n",
              "      <td>3</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 64}</td>\n",
              "      <td>0.940397</td>\n",
              "      <td>0.934900</td>\n",
              "      <td>0.940612</td>\n",
              "      <td>0.938636</td>\n",
              "      <td>0.002644</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>247.997033</td>\n",
              "      <td>22.011251</td>\n",
              "      <td>2.139256</td>\n",
              "      <td>0.055938</td>\n",
              "      <td>3</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 128}</td>\n",
              "      <td>0.945570</td>\n",
              "      <td>0.943738</td>\n",
              "      <td>0.946109</td>\n",
              "      <td>0.945139</td>\n",
              "      <td>0.001015</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>228.272594</td>\n",
              "      <td>24.962870</td>\n",
              "      <td>4.497849</td>\n",
              "      <td>1.233063</td>\n",
              "      <td>5</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 32}</td>\n",
              "      <td>0.905368</td>\n",
              "      <td>0.929511</td>\n",
              "      <td>0.923367</td>\n",
              "      <td>0.919415</td>\n",
              "      <td>0.010245</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>246.401566</td>\n",
              "      <td>24.836475</td>\n",
              "      <td>2.558052</td>\n",
              "      <td>0.324129</td>\n",
              "      <td>5</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 64}</td>\n",
              "      <td>0.920888</td>\n",
              "      <td>0.925307</td>\n",
              "      <td>0.937379</td>\n",
              "      <td>0.927858</td>\n",
              "      <td>0.006970</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>231.150077</td>\n",
              "      <td>23.189388</td>\n",
              "      <td>2.755982</td>\n",
              "      <td>0.002000</td>\n",
              "      <td>5</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 128}</td>\n",
              "      <td>0.937163</td>\n",
              "      <td>0.946756</td>\n",
              "      <td>0.946864</td>\n",
              "      <td>0.943594</td>\n",
              "      <td>0.004548</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>246.803588</td>\n",
              "      <td>24.322281</td>\n",
              "      <td>2.703593</td>\n",
              "      <td>0.161299</td>\n",
              "      <td>7</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 32}</td>\n",
              "      <td>0.890063</td>\n",
              "      <td>0.905691</td>\n",
              "      <td>0.924445</td>\n",
              "      <td>0.906733</td>\n",
              "      <td>0.014056</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>251.703463</td>\n",
              "      <td>16.845218</td>\n",
              "      <td>2.617659</td>\n",
              "      <td>0.293497</td>\n",
              "      <td>7</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 64}</td>\n",
              "      <td>0.930588</td>\n",
              "      <td>0.898470</td>\n",
              "      <td>0.921858</td>\n",
              "      <td>0.916972</td>\n",
              "      <td>0.013560</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>247.021159</td>\n",
              "      <td>23.433117</td>\n",
              "      <td>3.756544</td>\n",
              "      <td>1.116092</td>\n",
              "      <td>7</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 128}</td>\n",
              "      <td>0.930050</td>\n",
              "      <td>0.942445</td>\n",
              "      <td>0.943630</td>\n",
              "      <td>0.938708</td>\n",
              "      <td>0.006142</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9f76501a-e2b5-4ac9-9310-dbcc86510943')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-1baa4b16-ca22-4c85-99d4-d46467605844\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1baa4b16-ca22-4c85-99d4-d46467605844')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-1baa4b16-ca22-4c85-99d4-d46467605844 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9f76501a-e2b5-4ac9-9310-dbcc86510943 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9f76501a-e2b5-4ac9-9310-dbcc86510943');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Grid Search best parameters and score\n",
        "\n",
        "print(\"Best parameters: {}\".format(grid_search.best_params_))\n",
        "print(\"Highest accuracy obtained: {}\".format(grid_search.best_score_))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pAAaj9RvDL6q",
        "outputId": "d96e1579-09e7-4cb9-e8c7-8e24c8b4182e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best parameters: {'kernel_size': 3, 'num_filters': 128}\n",
            "Highest accuracy obtained: 0.9451390306154887\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN with tuned hyperparameters, batch normalization, and dropout layers\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Dropout(0.2),\n",
        "    Conv1D(128, 3, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    GlobalMaxPooling1D(),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = cnn_model.fit(X_train_glove, y_train_glove, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_glove, y_val_glove))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "id": "i2Mqst3Eq--5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88eef450-280b-457a-cc53-25a020abd26c"
      },
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 15s 18ms/step - loss: 0.4327 - accuracy: 0.7953 - val_loss: 0.2569 - val_accuracy: 0.8964\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.2952 - accuracy: 0.8740 - val_loss: 0.2148 - val_accuracy: 0.9073\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.2510 - accuracy: 0.8935 - val_loss: 0.1792 - val_accuracy: 0.9286\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.2238 - accuracy: 0.9075 - val_loss: 0.1671 - val_accuracy: 0.9320\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.2058 - accuracy: 0.9167 - val_loss: 0.1604 - val_accuracy: 0.9352\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 8s 12ms/step - loss: 0.1913 - accuracy: 0.9205 - val_loss: 0.1509 - val_accuracy: 0.9396\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.1835 - accuracy: 0.9261 - val_loss: 0.1498 - val_accuracy: 0.9389\n",
            "Epoch 8/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.1736 - accuracy: 0.9284 - val_loss: 0.1498 - val_accuracy: 0.9419\n",
            "Epoch 9/20\n",
            "653/653 [==============================] - 7s 10ms/step - loss: 0.1652 - accuracy: 0.9330 - val_loss: 0.1397 - val_accuracy: 0.9432\n",
            "Epoch 10/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.1622 - accuracy: 0.9342 - val_loss: 0.1407 - val_accuracy: 0.9422\n",
            "Epoch 11/20\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.1530 - accuracy: 0.9407 - val_loss: 0.1402 - val_accuracy: 0.9451\n",
            "Epoch 12/20\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.1551 - accuracy: 0.9368 - val_loss: 0.1326 - val_accuracy: 0.9467\n",
            "Epoch 13/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.1501 - accuracy: 0.9396 - val_loss: 0.1410 - val_accuracy: 0.9431\n",
            "Epoch 14/20\n",
            "653/653 [==============================] - 6s 10ms/step - loss: 0.1517 - accuracy: 0.9408 - val_loss: 0.1342 - val_accuracy: 0.9474\n",
            "Epoch 15/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.1436 - accuracy: 0.9429 - val_loss: 0.1325 - val_accuracy: 0.9487\n",
            "Epoch 16/20\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.1376 - accuracy: 0.9466 - val_loss: 0.1327 - val_accuracy: 0.9483\n",
            "Epoch 17/20\n",
            "653/653 [==============================] - 6s 8ms/step - loss: 0.1377 - accuracy: 0.9449 - val_loss: 0.1349 - val_accuracy: 0.9440\n",
            "Epoch 18/20\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.1270 - accuracy: 0.9489 - val_loss: 0.1336 - val_accuracy: 0.9474\n",
            "Epoch 19/20\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.1318 - accuracy: 0.9474 - val_loss: 0.1346 - val_accuracy: 0.9464\n",
            "Epoch 20/20\n",
            "653/653 [==============================] - 5s 8ms/step - loss: 0.1248 - accuracy: 0.9509 - val_loss: 0.1299 - val_accuracy: 0.9468\n",
            "Number of epochs run: 20\n",
            "CPU times: user 2min 6s, sys: 7.93 s, total: 2min 14s\n",
            "Wall time: 2min 15s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y4jSQy58r0Sn",
        "outputId": "6fb9a60a-396e-47a6-d2e2-3ba09cd856ed"
      },
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.1325 - accuracy: 0.9485\n",
            "Testing Accuracy is 94.85287666320801 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "44i7hGzfMzcq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN with tuned hyperparameters, batch normalization, and dropout layers\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_glove,\n",
        "    Dropout(0.2),\n",
        "    Conv1D(256, 3, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    GlobalMaxPooling1D(),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.5),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "                  metrics=['accuracy'])\n",
        "\n",
        "# Train the CNN Model for 60 epochs with learning rate reduction (by a factor\n",
        "# of 0.2, patience of 5) once learning stagnates\n",
        "\n",
        "reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,\n",
        "                              patience=5, min_lr=0.001)\n",
        "\n",
        "history = cnn_model.fit(X_train_glove, y_train_glove, epochs=60,\n",
        "                        callbacks=[reduce_lr],\n",
        "                        validation_data=(X_val_glove, y_val_glove),\n",
        "                        validation_steps=len(X_val_glove)//128,\n",
        "                        steps_per_epoch=len(X_train_glove)//128)\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "owKXDjocVK_k",
        "outputId": "6ea15414-f05f-412f-ece7-2866b1e699bc"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/60\n",
            "163/163 [==============================] - 9s 10ms/step - loss: 0.4265 - accuracy: 0.7988 - val_loss: 0.3334 - val_accuracy: 0.9004 - lr: 0.0030\n",
            "Epoch 2/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.2544 - accuracy: 0.8993 - val_loss: 0.1971 - val_accuracy: 0.9231 - lr: 0.0030\n",
            "Epoch 3/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.2097 - accuracy: 0.9187 - val_loss: 0.1650 - val_accuracy: 0.9352 - lr: 0.0030\n",
            "Epoch 4/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.1772 - accuracy: 0.9315 - val_loss: 0.1504 - val_accuracy: 0.9388 - lr: 0.0030\n",
            "Epoch 5/60\n",
            "163/163 [==============================] - 1s 8ms/step - loss: 0.1596 - accuracy: 0.9402 - val_loss: 0.1386 - val_accuracy: 0.9440 - lr: 0.0030\n",
            "Epoch 6/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.1506 - accuracy: 0.9429 - val_loss: 0.1339 - val_accuracy: 0.9481 - lr: 0.0030\n",
            "Epoch 7/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.1335 - accuracy: 0.9513 - val_loss: 0.1414 - val_accuracy: 0.9442 - lr: 0.0030\n",
            "Epoch 8/60\n",
            "163/163 [==============================] - 2s 13ms/step - loss: 0.1285 - accuracy: 0.9512 - val_loss: 0.1369 - val_accuracy: 0.9475 - lr: 0.0030\n",
            "Epoch 9/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.1204 - accuracy: 0.9580 - val_loss: 0.1355 - val_accuracy: 0.9519 - lr: 0.0030\n",
            "Epoch 10/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.1182 - accuracy: 0.9582 - val_loss: 0.1253 - val_accuracy: 0.9536 - lr: 0.0030\n",
            "Epoch 11/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.1086 - accuracy: 0.9612 - val_loss: 0.1278 - val_accuracy: 0.9507 - lr: 0.0030\n",
            "Epoch 12/60\n",
            "163/163 [==============================] - 2s 9ms/step - loss: 0.1111 - accuracy: 0.9602 - val_loss: 0.1382 - val_accuracy: 0.9507 - lr: 0.0030\n",
            "Epoch 13/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0983 - accuracy: 0.9661 - val_loss: 0.1288 - val_accuracy: 0.9537 - lr: 0.0030\n",
            "Epoch 14/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0983 - accuracy: 0.9644 - val_loss: 0.1270 - val_accuracy: 0.9517 - lr: 0.0030\n",
            "Epoch 15/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0958 - accuracy: 0.9669 - val_loss: 0.1273 - val_accuracy: 0.9527 - lr: 0.0030\n",
            "Epoch 16/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0822 - accuracy: 0.9718 - val_loss: 0.1242 - val_accuracy: 0.9556 - lr: 0.0010\n",
            "Epoch 17/60\n",
            "163/163 [==============================] - 2s 13ms/step - loss: 0.0725 - accuracy: 0.9750 - val_loss: 0.1214 - val_accuracy: 0.9555 - lr: 0.0010\n",
            "Epoch 18/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0756 - accuracy: 0.9738 - val_loss: 0.1246 - val_accuracy: 0.9550 - lr: 0.0010\n",
            "Epoch 19/60\n",
            "163/163 [==============================] - 2s 9ms/step - loss: 0.0706 - accuracy: 0.9764 - val_loss: 0.1233 - val_accuracy: 0.9573 - lr: 0.0010\n",
            "Epoch 20/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0673 - accuracy: 0.9771 - val_loss: 0.1258 - val_accuracy: 0.9546 - lr: 0.0010\n",
            "Epoch 21/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0654 - accuracy: 0.9784 - val_loss: 0.1299 - val_accuracy: 0.9537 - lr: 0.0010\n",
            "Epoch 22/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0634 - accuracy: 0.9789 - val_loss: 0.1234 - val_accuracy: 0.9566 - lr: 0.0010\n",
            "Epoch 23/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0643 - accuracy: 0.9774 - val_loss: 0.1203 - val_accuracy: 0.9583 - lr: 0.0010\n",
            "Epoch 24/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0641 - accuracy: 0.9777 - val_loss: 0.1226 - val_accuracy: 0.9573 - lr: 0.0010\n",
            "Epoch 25/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0636 - accuracy: 0.9781 - val_loss: 0.1278 - val_accuracy: 0.9559 - lr: 0.0010\n",
            "Epoch 26/60\n",
            "163/163 [==============================] - 2s 11ms/step - loss: 0.0589 - accuracy: 0.9802 - val_loss: 0.1257 - val_accuracy: 0.9557 - lr: 0.0010\n",
            "Epoch 27/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0626 - accuracy: 0.9781 - val_loss: 0.1307 - val_accuracy: 0.9576 - lr: 0.0010\n",
            "Epoch 28/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0583 - accuracy: 0.9803 - val_loss: 0.1268 - val_accuracy: 0.9563 - lr: 0.0010\n",
            "Epoch 29/60\n",
            "163/163 [==============================] - 4s 22ms/step - loss: 0.0607 - accuracy: 0.9795 - val_loss: 0.1273 - val_accuracy: 0.9578 - lr: 0.0010\n",
            "Epoch 30/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0590 - accuracy: 0.9804 - val_loss: 0.1267 - val_accuracy: 0.9549 - lr: 0.0010\n",
            "Epoch 31/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0560 - accuracy: 0.9805 - val_loss: 0.1264 - val_accuracy: 0.9583 - lr: 0.0010\n",
            "Epoch 32/60\n",
            "163/163 [==============================] - 2s 13ms/step - loss: 0.0541 - accuracy: 0.9815 - val_loss: 0.1297 - val_accuracy: 0.9580 - lr: 0.0010\n",
            "Epoch 33/60\n",
            "163/163 [==============================] - 2s 13ms/step - loss: 0.0531 - accuracy: 0.9818 - val_loss: 0.1314 - val_accuracy: 0.9557 - lr: 0.0010\n",
            "Epoch 34/60\n",
            "163/163 [==============================] - 1s 8ms/step - loss: 0.0599 - accuracy: 0.9801 - val_loss: 0.1254 - val_accuracy: 0.9590 - lr: 0.0010\n",
            "Epoch 35/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0566 - accuracy: 0.9813 - val_loss: 0.1249 - val_accuracy: 0.9573 - lr: 0.0010\n",
            "Epoch 36/60\n",
            "163/163 [==============================] - 1s 8ms/step - loss: 0.0507 - accuracy: 0.9825 - val_loss: 0.1405 - val_accuracy: 0.9566 - lr: 0.0010\n",
            "Epoch 37/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0533 - accuracy: 0.9824 - val_loss: 0.1339 - val_accuracy: 0.9566 - lr: 0.0010\n",
            "Epoch 38/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0540 - accuracy: 0.9815 - val_loss: 0.1279 - val_accuracy: 0.9560 - lr: 0.0010\n",
            "Epoch 39/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0531 - accuracy: 0.9821 - val_loss: 0.1334 - val_accuracy: 0.9566 - lr: 0.0010\n",
            "Epoch 40/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0561 - accuracy: 0.9809 - val_loss: 0.1314 - val_accuracy: 0.9562 - lr: 0.0010\n",
            "Epoch 41/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0571 - accuracy: 0.9811 - val_loss: 0.1257 - val_accuracy: 0.9580 - lr: 0.0010\n",
            "Epoch 42/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0544 - accuracy: 0.9822 - val_loss: 0.1256 - val_accuracy: 0.9599 - lr: 0.0010\n",
            "Epoch 43/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0525 - accuracy: 0.9826 - val_loss: 0.1274 - val_accuracy: 0.9590 - lr: 0.0010\n",
            "Epoch 44/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0528 - accuracy: 0.9825 - val_loss: 0.1334 - val_accuracy: 0.9582 - lr: 0.0010\n",
            "Epoch 45/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0521 - accuracy: 0.9823 - val_loss: 0.1250 - val_accuracy: 0.9588 - lr: 0.0010\n",
            "Epoch 46/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0515 - accuracy: 0.9830 - val_loss: 0.1266 - val_accuracy: 0.9578 - lr: 0.0010\n",
            "Epoch 47/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0481 - accuracy: 0.9840 - val_loss: 0.1253 - val_accuracy: 0.9585 - lr: 0.0010\n",
            "Epoch 48/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0536 - accuracy: 0.9814 - val_loss: 0.1234 - val_accuracy: 0.9592 - lr: 0.0010\n",
            "Epoch 49/60\n",
            "163/163 [==============================] - 2s 13ms/step - loss: 0.0534 - accuracy: 0.9822 - val_loss: 0.1200 - val_accuracy: 0.9590 - lr: 0.0010\n",
            "Epoch 50/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0502 - accuracy: 0.9831 - val_loss: 0.1363 - val_accuracy: 0.9536 - lr: 0.0010\n",
            "Epoch 51/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0521 - accuracy: 0.9826 - val_loss: 0.1281 - val_accuracy: 0.9589 - lr: 0.0010\n",
            "Epoch 52/60\n",
            "163/163 [==============================] - 2s 11ms/step - loss: 0.0510 - accuracy: 0.9829 - val_loss: 0.1244 - val_accuracy: 0.9618 - lr: 0.0010\n",
            "Epoch 53/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0433 - accuracy: 0.9850 - val_loss: 0.1449 - val_accuracy: 0.9589 - lr: 0.0010\n",
            "Epoch 54/60\n",
            "163/163 [==============================] - 2s 11ms/step - loss: 0.0492 - accuracy: 0.9828 - val_loss: 0.1320 - val_accuracy: 0.9576 - lr: 0.0010\n",
            "Epoch 55/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0503 - accuracy: 0.9828 - val_loss: 0.1235 - val_accuracy: 0.9603 - lr: 0.0010\n",
            "Epoch 56/60\n",
            "163/163 [==============================] - 2s 10ms/step - loss: 0.0506 - accuracy: 0.9827 - val_loss: 0.1259 - val_accuracy: 0.9585 - lr: 0.0010\n",
            "Epoch 57/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0510 - accuracy: 0.9833 - val_loss: 0.1287 - val_accuracy: 0.9595 - lr: 0.0010\n",
            "Epoch 58/60\n",
            "163/163 [==============================] - 2s 12ms/step - loss: 0.0465 - accuracy: 0.9848 - val_loss: 0.1318 - val_accuracy: 0.9590 - lr: 0.0010\n",
            "Epoch 59/60\n",
            "163/163 [==============================] - 1s 9ms/step - loss: 0.0505 - accuracy: 0.9828 - val_loss: 0.1251 - val_accuracy: 0.9596 - lr: 0.0010\n",
            "Epoch 60/60\n",
            "103/163 [=================>............] - ETA: 0s - loss: 0.0459 - accuracy: 0.9839"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 9780 batches). You may need to use the repeat() function when building your dataset.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r163/163 [==============================] - 1s 7ms/step - loss: 0.0459 - accuracy: 0.9839 - val_loss: 0.1305 - val_accuracy: 0.9592 - lr: 0.0010\n",
            "Number of epochs run: 60\n",
            "CPU times: user 1min 43s, sys: 5.84 s, total: 1min 49s\n",
            "Wall time: 1min 48s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_glove, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aEoHSnAhVLH-",
        "outputId": "63c58be4-0849-4586-9bf6-bda290ea1f0c"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.1350 - accuracy: 0.9566\n",
            "Final Testing Accuracy for the CNN is 95.6576406955719 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My final CNN model that used the pretrained GloVe word embeddings achieved an accuracy of 95.66% and took 1 min and 49 s of CPU time to train."
      ],
      "metadata": {
        "id": "3nawrmJFlBQB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### CNN with a Continuous Bag of Words model"
      ],
      "metadata": {
        "id": "gzIaqt-HyVZR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Preliminary Model"
      ],
      "metadata": {
        "id": "0FiGzUsckEeR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses a continuous Bag of Words model\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Conv1D(128, 5, activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "history = cnn_model.fit(X_train_bow, y_train_bow, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_bow, y_val_bow))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['val_loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s5n07PhCiHPS",
        "outputId": "d4ac0e95-45a3-4910-f28a-772b81ed8e2e"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 7s 8ms/step - loss: 0.2342 - accuracy: 0.9028 - val_loss: 0.1139 - val_accuracy: 0.9582\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0617 - accuracy: 0.9817 - val_loss: 0.0835 - val_accuracy: 0.9671\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0221 - accuracy: 0.9950 - val_loss: 0.0783 - val_accuracy: 0.9707\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 5s 7ms/step - loss: 0.0081 - accuracy: 0.9986 - val_loss: 0.0710 - val_accuracy: 0.9723\n",
            "Epoch 5/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0032 - accuracy: 0.9994 - val_loss: 0.0755 - val_accuracy: 0.9734\n",
            "Epoch 6/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0037 - accuracy: 0.9994 - val_loss: 0.0807 - val_accuracy: 0.9738\n",
            "Epoch 7/20\n",
            "653/653 [==============================] - 4s 6ms/step - loss: 0.0050 - accuracy: 0.9989 - val_loss: 0.0886 - val_accuracy: 0.9694\n",
            "Number of epochs run: 7\n",
            "CPU times: user 31.7 s, sys: 2.16 s, total: 33.9 s\n",
            "Wall time: 31.2 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model2.evaluate(X_test_bow, y_test)\n",
        "print('Testing Accuracy is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "voT6OnqWiL8j",
        "outputId": "b20bcc64-3600-4ab7-8697-f24fa45ee42d"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.0995 - accuracy: 0.9697\n",
            "Testing Accuracy is 96.97375893592834 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Hyperparameter Tuning"
      ],
      "metadata": {
        "id": "Bcib1Jrtsg5O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the model to be optimized\n",
        "\n",
        "def model_to_optimize(num_filters, kernel_size):\n",
        "  model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Dropout(0.2),\n",
        "    Conv1D(num_filters, kernel_size, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    GlobalMaxPooling1D(),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "  ])\n",
        "  model.compile(loss='binary_crossentropy', optimizer=tf.keras.optimizers.Adam(3e-3), metrics=['accuracy'])\n",
        "  return model"
      ],
      "metadata": {
        "id": "yOJEwGlKshPW"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the parameter grid\n",
        "\n",
        "params = {\n",
        "    \"num_filters\":[32, 64, 128],\n",
        "    \"kernel_size\":[3, 5, 7],\n",
        "}\n",
        "\n",
        "# Define the Keras Classifier\n",
        "\n",
        "model = KerasClassifier(build_fn=model_to_optimize, epochs=20,\n",
        "                        batch_size=10, verbose=3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ftlF5ALZshbM",
        "outputId": "20d6b2a3-ef14-49df-dda1-2256c5835203"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-33-37a7a4be389f>:10: DeprecationWarning: KerasClassifier is deprecated, use Sci-Keras (https://github.com/adriangb/scikeras) instead. See https://www.adriangb.com/scikeras/stable/migration.html for help migrating.\n",
            "  model = KerasClassifier(build_fn=model_to_optimize, epochs=20,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Perform the Grid Search with learning rate reduction (by a factor\n",
        "# of 0.2, patience of 3) once learning stagnates\n",
        "\n",
        "grid_search = GridSearchCV(estimator=model, param_grid=params,\n",
        "                              cv=3, verbose=2)\n",
        "\n",
        "reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.2,\n",
        "                              patience=3, min_lr=0.001)\n",
        "\n",
        "grid_search.fit(X_train_all_bow, y_train_all, callbacks=[reduce_lr],\n",
        "                validation_steps=len(X_val_bow)//32, steps_per_epoch=len(X_train_bow)//32)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "z5t99nOmshly",
        "outputId": "28694eb1-4c86-4ab3-9aa5-4068ca81a37a"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 9 candidates, totalling 27 fits\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=32; total time= 1.8min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=64; total time= 1.9min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=64; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=3, num_filters=64; total time= 1.8min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=3, num_filters=128; total time= 2.3min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=3, num_filters=128; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=3, num_filters=128; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=32; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=32; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=64; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=64; total time= 1.8min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=5, num_filters=64; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=5, num_filters=128; total time= 1.9min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=5, num_filters=128; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=5, num_filters=128; total time= 1.7min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=32; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=64; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=64; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END ......................kernel_size=7, num_filters=64; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=7, num_filters=128; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=7, num_filters=128; total time= 2.5min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "[CV] END .....................kernel_size=7, num_filters=128; total time= 2.4min\n",
            "Epoch 1/20\n",
            "Epoch 2/20\n",
            "Epoch 3/20\n",
            "Epoch 4/20\n",
            "Epoch 5/20\n",
            "Epoch 6/20\n",
            "Epoch 7/20\n",
            "Epoch 8/20\n",
            "Epoch 9/20\n",
            "Epoch 10/20\n",
            "Epoch 11/20\n",
            "Epoch 12/20\n",
            "Epoch 13/20\n",
            "Epoch 14/20\n",
            "Epoch 15/20\n",
            "Epoch 16/20\n",
            "Epoch 17/20\n",
            "Epoch 18/20\n",
            "Epoch 19/20\n",
            "Epoch 20/20\n",
            "CPU times: user 53min 34s, sys: 2min 38s, total: 56min 13s\n",
            "Wall time: 1h 4min 56s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=3,\n",
              "             estimator=<keras.wrappers.scikit_learn.KerasClassifier object at 0x7e98434a27d0>,\n",
              "             param_grid={'kernel_size': [3, 5, 7],\n",
              "                         'num_filters': [32, 64, 128]},\n",
              "             verbose=2)"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=3,\n",
              "             estimator=&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7e98434a27d0&gt;,\n",
              "             param_grid={&#x27;kernel_size&#x27;: [3, 5, 7],\n",
              "                         &#x27;num_filters&#x27;: [32, 64, 128]},\n",
              "             verbose=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=3,\n",
              "             estimator=&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7e98434a27d0&gt;,\n",
              "             param_grid={&#x27;kernel_size&#x27;: [3, 5, 7],\n",
              "                         &#x27;num_filters&#x27;: [32, 64, 128]},\n",
              "             verbose=2)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: KerasClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7e98434a27d0&gt;</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KerasClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;keras.wrappers.scikit_learn.KerasClassifier object at 0x7e98434a27d0&gt;</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check the cross-validated results of the Grid Search\n",
        "\n",
        "pd.DataFrame(grid_search.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 864
        },
        "id": "Wt4d_NyfuZDq",
        "outputId": "1235d54b-281b-4683-90b3-95ecab38e491"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0     131.540622     17.598878         3.794272        1.402769   \n",
              "1     120.913908     16.216127         3.032613        0.500690   \n",
              "2     141.155036      6.571026         4.533743        1.461524   \n",
              "3     143.778200      0.101945         2.780806        0.334586   \n",
              "4     130.948888     17.993415         3.671083        1.201634   \n",
              "5     115.860033     21.159110         4.754145        1.007725   \n",
              "6     144.804616      1.313748         3.984537        1.392820   \n",
              "7     144.289852      0.385506         3.574270        1.722041   \n",
              "8     144.255539      0.583658         2.867557        0.065113   \n",
              "\n",
              "  param_kernel_size param_num_filters                                  params  \\\n",
              "0                 3                32   {'kernel_size': 3, 'num_filters': 32}   \n",
              "1                 3                64   {'kernel_size': 3, 'num_filters': 64}   \n",
              "2                 3               128  {'kernel_size': 3, 'num_filters': 128}   \n",
              "3                 5                32   {'kernel_size': 5, 'num_filters': 32}   \n",
              "4                 5                64   {'kernel_size': 5, 'num_filters': 64}   \n",
              "5                 5               128  {'kernel_size': 5, 'num_filters': 128}   \n",
              "6                 7                32   {'kernel_size': 7, 'num_filters': 32}   \n",
              "7                 7                64   {'kernel_size': 7, 'num_filters': 64}   \n",
              "8                 7               128  {'kernel_size': 7, 'num_filters': 128}   \n",
              "\n",
              "   split0_test_score  split1_test_score  split2_test_score  mean_test_score  \\\n",
              "0           0.962384           0.948696           0.963678         0.958252   \n",
              "1           0.965186           0.968097           0.964648         0.965977   \n",
              "2           0.964863           0.972516           0.972408         0.969929   \n",
              "3           0.958288           0.952145           0.952145         0.954193   \n",
              "4           0.964324           0.956995           0.963785         0.961702   \n",
              "5           0.966156           0.950205           0.964216         0.960193   \n",
              "6           0.956564           0.953977           0.955917         0.955486   \n",
              "7           0.958073           0.953007           0.958396         0.956492   \n",
              "8           0.962600           0.958504           0.965294         0.962133   \n",
              "\n",
              "   std_test_score  rank_test_score  \n",
              "0        0.006778                6  \n",
              "1        0.001515                2  \n",
              "2        0.003582                1  \n",
              "3        0.002896                9  \n",
              "4        0.003335                4  \n",
              "5        0.007107                5  \n",
              "6        0.001099                8  \n",
              "7        0.002468                7  \n",
              "8        0.002792                3  "
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-9fc26b00-1741-4c3e-81ba-372fdf421ad9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_kernel_size</th>\n",
              "      <th>param_num_filters</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>131.540622</td>\n",
              "      <td>17.598878</td>\n",
              "      <td>3.794272</td>\n",
              "      <td>1.402769</td>\n",
              "      <td>3</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 32}</td>\n",
              "      <td>0.962384</td>\n",
              "      <td>0.948696</td>\n",
              "      <td>0.963678</td>\n",
              "      <td>0.958252</td>\n",
              "      <td>0.006778</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>120.913908</td>\n",
              "      <td>16.216127</td>\n",
              "      <td>3.032613</td>\n",
              "      <td>0.500690</td>\n",
              "      <td>3</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 64}</td>\n",
              "      <td>0.965186</td>\n",
              "      <td>0.968097</td>\n",
              "      <td>0.964648</td>\n",
              "      <td>0.965977</td>\n",
              "      <td>0.001515</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>141.155036</td>\n",
              "      <td>6.571026</td>\n",
              "      <td>4.533743</td>\n",
              "      <td>1.461524</td>\n",
              "      <td>3</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 3, 'num_filters': 128}</td>\n",
              "      <td>0.964863</td>\n",
              "      <td>0.972516</td>\n",
              "      <td>0.972408</td>\n",
              "      <td>0.969929</td>\n",
              "      <td>0.003582</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>143.778200</td>\n",
              "      <td>0.101945</td>\n",
              "      <td>2.780806</td>\n",
              "      <td>0.334586</td>\n",
              "      <td>5</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 32}</td>\n",
              "      <td>0.958288</td>\n",
              "      <td>0.952145</td>\n",
              "      <td>0.952145</td>\n",
              "      <td>0.954193</td>\n",
              "      <td>0.002896</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>130.948888</td>\n",
              "      <td>17.993415</td>\n",
              "      <td>3.671083</td>\n",
              "      <td>1.201634</td>\n",
              "      <td>5</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 64}</td>\n",
              "      <td>0.964324</td>\n",
              "      <td>0.956995</td>\n",
              "      <td>0.963785</td>\n",
              "      <td>0.961702</td>\n",
              "      <td>0.003335</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>115.860033</td>\n",
              "      <td>21.159110</td>\n",
              "      <td>4.754145</td>\n",
              "      <td>1.007725</td>\n",
              "      <td>5</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 5, 'num_filters': 128}</td>\n",
              "      <td>0.966156</td>\n",
              "      <td>0.950205</td>\n",
              "      <td>0.964216</td>\n",
              "      <td>0.960193</td>\n",
              "      <td>0.007107</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>144.804616</td>\n",
              "      <td>1.313748</td>\n",
              "      <td>3.984537</td>\n",
              "      <td>1.392820</td>\n",
              "      <td>7</td>\n",
              "      <td>32</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 32}</td>\n",
              "      <td>0.956564</td>\n",
              "      <td>0.953977</td>\n",
              "      <td>0.955917</td>\n",
              "      <td>0.955486</td>\n",
              "      <td>0.001099</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>144.289852</td>\n",
              "      <td>0.385506</td>\n",
              "      <td>3.574270</td>\n",
              "      <td>1.722041</td>\n",
              "      <td>7</td>\n",
              "      <td>64</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 64}</td>\n",
              "      <td>0.958073</td>\n",
              "      <td>0.953007</td>\n",
              "      <td>0.958396</td>\n",
              "      <td>0.956492</td>\n",
              "      <td>0.002468</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>144.255539</td>\n",
              "      <td>0.583658</td>\n",
              "      <td>2.867557</td>\n",
              "      <td>0.065113</td>\n",
              "      <td>7</td>\n",
              "      <td>128</td>\n",
              "      <td>{'kernel_size': 7, 'num_filters': 128}</td>\n",
              "      <td>0.962600</td>\n",
              "      <td>0.958504</td>\n",
              "      <td>0.965294</td>\n",
              "      <td>0.962133</td>\n",
              "      <td>0.002792</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9fc26b00-1741-4c3e-81ba-372fdf421ad9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-414068cd-4dc3-4dc0-951b-37bd2f675280\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-414068cd-4dc3-4dc0-951b-37bd2f675280')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-414068cd-4dc3-4dc0-951b-37bd2f675280 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9fc26b00-1741-4c3e-81ba-372fdf421ad9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9fc26b00-1741-4c3e-81ba-372fdf421ad9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Grid Search best parameters and score\n",
        "\n",
        "print(\"Best parameters: {}\".format(grid_search.best_params_))\n",
        "print(\"Highest accuracy obtained: {}\".format(grid_search.best_score_))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MAnc2SmsuZPD",
        "outputId": "9c109a99-d90d-4bfa-aeae-bfcafb2ae3e4"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best parameters: {'kernel_size': 3, 'num_filters': 128}\n",
            "Highest accuracy obtained: 0.9699288606643677\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "_MjezVEDmCgE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN with tuned hyperparameters, batch normalization, and dropout layers\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_bow,\n",
        "    Dropout(0.2),\n",
        "    Conv1D(128, 3, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    GlobalMaxPooling1D(),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.2),\n",
        "    Dense(10, activation='relu'),\n",
        "    BatchNormalization(),\n",
        "    Dropout(0.5),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer=tf.keras.optimizers.Adam(3e-3),\n",
        "                  metrics=['accuracy'])\n",
        "# Train the CNN Model for 60 epochs with learning rate reduction (by a factor\n",
        "# of 0.2, patience of 3) once learning stagnates\n",
        "\n",
        "reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,\n",
        "                              patience=3, min_lr=0.001)\n",
        "\n",
        "history = cnn_model.fit(X_train_bow, y_train_bow, epochs=60,\n",
        "                        callbacks=[reduce_lr],\n",
        "                        validation_data=(X_val_bow, y_val_bow))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MASKeuzLlJLE",
        "outputId": "ad961ae0-6520-4175-9ae9-9cc3ddf63861"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/60\n",
            "653/653 [==============================] - 15s 10ms/step - loss: 0.3056 - accuracy: 0.8702 - val_loss: 0.1529 - val_accuracy: 0.9477 - lr: 0.0030\n",
            "Epoch 2/60\n",
            "653/653 [==============================] - 8s 12ms/step - loss: 0.1765 - accuracy: 0.9355 - val_loss: 0.0934 - val_accuracy: 0.9644 - lr: 0.0030\n",
            "Epoch 3/60\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.1421 - accuracy: 0.9483 - val_loss: 0.0842 - val_accuracy: 0.9664 - lr: 0.0030\n",
            "Epoch 4/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.1235 - accuracy: 0.9562 - val_loss: 0.0758 - val_accuracy: 0.9692 - lr: 0.0030\n",
            "Epoch 5/60\n",
            "653/653 [==============================] - 8s 13ms/step - loss: 0.1110 - accuracy: 0.9607 - val_loss: 0.0748 - val_accuracy: 0.9738 - lr: 0.0030\n",
            "Epoch 6/60\n",
            "653/653 [==============================] - 14s 21ms/step - loss: 0.0959 - accuracy: 0.9666 - val_loss: 0.0781 - val_accuracy: 0.9714 - lr: 0.0030\n",
            "Epoch 7/60\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0924 - accuracy: 0.9676 - val_loss: 0.0749 - val_accuracy: 0.9730 - lr: 0.0030\n",
            "Epoch 8/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0801 - accuracy: 0.9729 - val_loss: 0.0791 - val_accuracy: 0.9705 - lr: 0.0030\n",
            "Epoch 9/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0666 - accuracy: 0.9771 - val_loss: 0.0648 - val_accuracy: 0.9764 - lr: 0.0010\n",
            "Epoch 10/60\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.0603 - accuracy: 0.9797 - val_loss: 0.0628 - val_accuracy: 0.9772 - lr: 0.0010\n",
            "Epoch 11/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0504 - accuracy: 0.9844 - val_loss: 0.0676 - val_accuracy: 0.9766 - lr: 0.0010\n",
            "Epoch 12/60\n",
            "653/653 [==============================] - 15s 23ms/step - loss: 0.0497 - accuracy: 0.9830 - val_loss: 0.0648 - val_accuracy: 0.9772 - lr: 0.0010\n",
            "Epoch 13/60\n",
            "653/653 [==============================] - 9s 13ms/step - loss: 0.0466 - accuracy: 0.9850 - val_loss: 0.0659 - val_accuracy: 0.9766 - lr: 0.0010\n",
            "Epoch 14/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0435 - accuracy: 0.9859 - val_loss: 0.0771 - val_accuracy: 0.9764 - lr: 0.0010\n",
            "Epoch 15/60\n",
            "653/653 [==============================] - 13s 19ms/step - loss: 0.0408 - accuracy: 0.9851 - val_loss: 0.0681 - val_accuracy: 0.9776 - lr: 0.0010\n",
            "Epoch 16/60\n",
            "653/653 [==============================] - 8s 13ms/step - loss: 0.0389 - accuracy: 0.9865 - val_loss: 0.0684 - val_accuracy: 0.9784 - lr: 0.0010\n",
            "Epoch 17/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0365 - accuracy: 0.9884 - val_loss: 0.0776 - val_accuracy: 0.9772 - lr: 0.0010\n",
            "Epoch 18/60\n",
            "653/653 [==============================] - 14s 21ms/step - loss: 0.0381 - accuracy: 0.9879 - val_loss: 0.0701 - val_accuracy: 0.9795 - lr: 0.0010\n",
            "Epoch 19/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0369 - accuracy: 0.9882 - val_loss: 0.0683 - val_accuracy: 0.9796 - lr: 0.0010\n",
            "Epoch 20/60\n",
            "653/653 [==============================] - 13s 19ms/step - loss: 0.0333 - accuracy: 0.9892 - val_loss: 0.0684 - val_accuracy: 0.9789 - lr: 0.0010\n",
            "Epoch 21/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0347 - accuracy: 0.9895 - val_loss: 0.0671 - val_accuracy: 0.9787 - lr: 0.0010\n",
            "Epoch 22/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0365 - accuracy: 0.9874 - val_loss: 0.0663 - val_accuracy: 0.9799 - lr: 0.0010\n",
            "Epoch 23/60\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0321 - accuracy: 0.9900 - val_loss: 0.0689 - val_accuracy: 0.9790 - lr: 0.0010\n",
            "Epoch 24/60\n",
            "653/653 [==============================] - 11s 18ms/step - loss: 0.0330 - accuracy: 0.9888 - val_loss: 0.0704 - val_accuracy: 0.9799 - lr: 0.0010\n",
            "Epoch 25/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0308 - accuracy: 0.9897 - val_loss: 0.0718 - val_accuracy: 0.9800 - lr: 0.0010\n",
            "Epoch 26/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0320 - accuracy: 0.9898 - val_loss: 0.0690 - val_accuracy: 0.9806 - lr: 0.0010\n",
            "Epoch 27/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0295 - accuracy: 0.9909 - val_loss: 0.0715 - val_accuracy: 0.9810 - lr: 0.0010\n",
            "Epoch 28/60\n",
            "653/653 [==============================] - 9s 13ms/step - loss: 0.0292 - accuracy: 0.9903 - val_loss: 0.0734 - val_accuracy: 0.9796 - lr: 0.0010\n",
            "Epoch 29/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0277 - accuracy: 0.9914 - val_loss: 0.0743 - val_accuracy: 0.9796 - lr: 0.0010\n",
            "Epoch 30/60\n",
            "653/653 [==============================] - 8s 12ms/step - loss: 0.0331 - accuracy: 0.9904 - val_loss: 0.0724 - val_accuracy: 0.9799 - lr: 0.0010\n",
            "Epoch 31/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0279 - accuracy: 0.9907 - val_loss: 0.0692 - val_accuracy: 0.9793 - lr: 0.0010\n",
            "Epoch 32/60\n",
            "653/653 [==============================] - 14s 22ms/step - loss: 0.0278 - accuracy: 0.9914 - val_loss: 0.0641 - val_accuracy: 0.9812 - lr: 0.0010\n",
            "Epoch 33/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0234 - accuracy: 0.9919 - val_loss: 0.0712 - val_accuracy: 0.9812 - lr: 0.0010\n",
            "Epoch 34/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0256 - accuracy: 0.9921 - val_loss: 0.0740 - val_accuracy: 0.9800 - lr: 0.0010\n",
            "Epoch 35/60\n",
            "653/653 [==============================] - 11s 17ms/step - loss: 0.0239 - accuracy: 0.9922 - val_loss: 0.0798 - val_accuracy: 0.9793 - lr: 0.0010\n",
            "Epoch 36/60\n",
            "653/653 [==============================] - 9s 13ms/step - loss: 0.0273 - accuracy: 0.9911 - val_loss: 0.0691 - val_accuracy: 0.9803 - lr: 0.0010\n",
            "Epoch 37/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.0292 - accuracy: 0.9907 - val_loss: 0.0723 - val_accuracy: 0.9793 - lr: 0.0010\n",
            "Epoch 38/60\n",
            "653/653 [==============================] - 10s 15ms/step - loss: 0.0289 - accuracy: 0.9913 - val_loss: 0.0666 - val_accuracy: 0.9802 - lr: 0.0010\n",
            "Epoch 39/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0249 - accuracy: 0.9922 - val_loss: 0.0667 - val_accuracy: 0.9797 - lr: 0.0010\n",
            "Epoch 40/60\n",
            "653/653 [==============================] - 13s 21ms/step - loss: 0.0209 - accuracy: 0.9936 - val_loss: 0.0634 - val_accuracy: 0.9833 - lr: 0.0010\n",
            "Epoch 41/60\n",
            "653/653 [==============================] - 12s 19ms/step - loss: 0.0303 - accuracy: 0.9905 - val_loss: 0.0652 - val_accuracy: 0.9802 - lr: 0.0010\n",
            "Epoch 42/60\n",
            "653/653 [==============================] - 13s 19ms/step - loss: 0.0233 - accuracy: 0.9930 - val_loss: 0.0703 - val_accuracy: 0.9802 - lr: 0.0010\n",
            "Epoch 43/60\n",
            "653/653 [==============================] - 8s 13ms/step - loss: 0.0226 - accuracy: 0.9930 - val_loss: 0.0615 - val_accuracy: 0.9823 - lr: 0.0010\n",
            "Epoch 44/60\n",
            "653/653 [==============================] - 12s 18ms/step - loss: 0.0220 - accuracy: 0.9934 - val_loss: 0.0647 - val_accuracy: 0.9809 - lr: 0.0010\n",
            "Epoch 45/60\n",
            "653/653 [==============================] - 13s 21ms/step - loss: 0.0232 - accuracy: 0.9927 - val_loss: 0.0644 - val_accuracy: 0.9816 - lr: 0.0010\n",
            "Epoch 46/60\n",
            "653/653 [==============================] - 13s 20ms/step - loss: 0.0205 - accuracy: 0.9931 - val_loss: 0.0720 - val_accuracy: 0.9806 - lr: 0.0010\n",
            "Epoch 47/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0220 - accuracy: 0.9929 - val_loss: 0.0664 - val_accuracy: 0.9815 - lr: 0.0010\n",
            "Epoch 48/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.0263 - accuracy: 0.9911 - val_loss: 0.0688 - val_accuracy: 0.9822 - lr: 0.0010\n",
            "Epoch 49/60\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.0204 - accuracy: 0.9936 - val_loss: 0.0705 - val_accuracy: 0.9819 - lr: 0.0010\n",
            "Epoch 50/60\n",
            "653/653 [==============================] - 8s 12ms/step - loss: 0.0201 - accuracy: 0.9935 - val_loss: 0.0630 - val_accuracy: 0.9820 - lr: 0.0010\n",
            "Epoch 51/60\n",
            "653/653 [==============================] - 13s 21ms/step - loss: 0.0229 - accuracy: 0.9924 - val_loss: 0.0648 - val_accuracy: 0.9826 - lr: 0.0010\n",
            "Epoch 52/60\n",
            "653/653 [==============================] - 10s 16ms/step - loss: 0.0194 - accuracy: 0.9935 - val_loss: 0.0644 - val_accuracy: 0.9815 - lr: 0.0010\n",
            "Epoch 53/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0236 - accuracy: 0.9924 - val_loss: 0.0673 - val_accuracy: 0.9813 - lr: 0.0010\n",
            "Epoch 54/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.0218 - accuracy: 0.9929 - val_loss: 0.0646 - val_accuracy: 0.9800 - lr: 0.0010\n",
            "Epoch 55/60\n",
            "653/653 [==============================] - 8s 12ms/step - loss: 0.0213 - accuracy: 0.9934 - val_loss: 0.0640 - val_accuracy: 0.9813 - lr: 0.0010\n",
            "Epoch 56/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.0212 - accuracy: 0.9935 - val_loss: 0.0611 - val_accuracy: 0.9825 - lr: 0.0010\n",
            "Epoch 57/60\n",
            "653/653 [==============================] - 7s 11ms/step - loss: 0.0205 - accuracy: 0.9932 - val_loss: 0.0758 - val_accuracy: 0.9796 - lr: 0.0010\n",
            "Epoch 58/60\n",
            "653/653 [==============================] - 6s 9ms/step - loss: 0.0212 - accuracy: 0.9930 - val_loss: 0.0616 - val_accuracy: 0.9810 - lr: 0.0010\n",
            "Epoch 59/60\n",
            "653/653 [==============================] - 9s 14ms/step - loss: 0.0249 - accuracy: 0.9923 - val_loss: 0.0818 - val_accuracy: 0.9790 - lr: 0.0010\n",
            "Epoch 60/60\n",
            "653/653 [==============================] - 11s 16ms/step - loss: 0.0204 - accuracy: 0.9935 - val_loss: 0.0653 - val_accuracy: 0.9823 - lr: 0.0010\n",
            "Number of epochs run: 60\n",
            "CPU times: user 8min 21s, sys: 25.6 s, total: 8min 46s\n",
            "Wall time: 10min 25s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_bow, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O9vVREaFlW0s",
        "outputId": "0daddebe-3e8e-42b1-f0fb-19345c69f57f"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.0868 - accuracy: 0.9796\n",
            "Final Testing Accuracy for the CNN is 97.96294569969177 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My final CNN model that used the continuous Bag of Words model achieved an accuracy of 97.96% and took 8 min and 46 s of CPU time to train."
      ],
      "metadata": {
        "id": "UdCjqgYUwys8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### CNN with FastText word embeddings"
      ],
      "metadata": {
        "id": "GN9dr4nlaX4O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Preliminary Model"
      ],
      "metadata": {
        "id": "H75Y8UfpaYt2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses the FastText word embeddings\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_ft,\n",
        "    Conv1D(128, 5, padding='same', activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "earlystop3 = EarlyStopping(monitor='val_loss', patience=3)\n",
        "\n",
        "history = cnn_model.fit(X_train_ft, y_train_ft, epochs=20,\n",
        "                        callbacks=[earlystop3],\n",
        "                        validation_data=(X_val_ft, y_val_ft))\n",
        "\n",
        "print(\"Number of epochs run: {}\".format(len(history.history['val_loss'])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_1w5AD3AbYFU",
        "outputId": "4c7e782d-909d-47a6-e379-e0212de80ebf"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 181s 257ms/step - loss: 0.1075 - accuracy: 0.9565 - val_loss: 0.0371 - val_accuracy: 0.9872\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 103s 157ms/step - loss: 0.0093 - accuracy: 0.9975 - val_loss: 0.0399 - val_accuracy: 0.9866\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 58s 89ms/step - loss: 8.6833e-04 - accuracy: 0.9999 - val_loss: 0.0438 - val_accuracy: 0.9882\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 37s 56ms/step - loss: 1.4284e-04 - accuracy: 1.0000 - val_loss: 0.0432 - val_accuracy: 0.9885\n",
            "Number of epochs run: 4\n",
            "CPU times: user 3min 53s, sys: 5.1 s, total: 3min 58s\n",
            "Wall time: 6min 19s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_ft, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KwG231DbYPo",
        "outputId": "152f9a1d-6689-419e-ae6e-bcc96c062499"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.0407 - accuracy: 0.9869\n",
            "Final Testing Accuracy for the CNN is 98.69226217269897 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Adding a Dropout Layer"
      ],
      "metadata": {
        "id": "qaq8JYWtmuAa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses the FastText word embeddings and multiple\n",
        "# dropout layers\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_ft,\n",
        "    Conv1D(128, 5, padding='same', activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.2),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "cnn_model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_R3fM566d24D",
        "outputId": "79bc8236-ce24-41fc-9f47-886a05d4ff39"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_9\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_1 (Embedding)     (None, 500, 300)          43299000  \n",
            "                                                                 \n",
            " conv1d_17 (Conv1D)          (None, 500, 128)          192128    \n",
            "                                                                 \n",
            " global_max_pooling1d_13 (Gl  (None, 128)              0         \n",
            " obalMaxPooling1D)                                               \n",
            "                                                                 \n",
            " dense_24 (Dense)            (None, 10)                1290      \n",
            "                                                                 \n",
            " dropout_13 (Dropout)        (None, 10)                0         \n",
            "                                                                 \n",
            " dense_25 (Dense)            (None, 1)                 11        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 43,492,429\n",
            "Trainable params: 43,492,429\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "CPU times: user 78.5 ms, sys: 2 ms, total: 80.5 ms\n",
            "Wall time: 86.4 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 3)\n",
        "\n",
        "earlystop3 = EarlyStopping(monitor='val_loss', patience=3,\n",
        "                           restore_best_weights=True, verbose=1)\n",
        "\n",
        "history = cnn_model.fit(X_train_ft, y_train_ft, epochs=20,\n",
        "                        validation_data=(X_val_ft, y_val_ft),\n",
        "                        callbacks=[earlystop3])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8-apFq1Ud3C2",
        "outputId": "379f9897-2943-4a4c-b534-3dcf6bafb283"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 126s 191ms/step - loss: 0.0263 - accuracy: 0.9941 - val_loss: 0.0667 - val_accuracy: 0.9851\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 63s 96ms/step - loss: 0.0035 - accuracy: 0.9989 - val_loss: 0.0861 - val_accuracy: 0.9846\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 41s 63ms/step - loss: 0.0012 - accuracy: 0.9992 - val_loss: 0.0968 - val_accuracy: 0.9842\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - ETA: 0s - loss: 0.0011 - accuracy: 0.9992Restoring model weights from the end of the best epoch: 1.\n",
            "653/653 [==============================] - 35s 53ms/step - loss: 0.0011 - accuracy: 0.9992 - val_loss: 0.1570 - val_accuracy: 0.9749\n",
            "Epoch 4: early stopping\n",
            "CPU times: user 3min 49s, sys: 3.83 s, total: 3min 53s\n",
            "Wall time: 4min 24s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_ft, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n9EoagMimFYv",
        "outputId": "cdd07ac3-20f6-4538-bd35-a00f339f1193"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.0720 - accuracy: 0.9815\n",
            "Final Testing Accuracy for the CNN is 98.14737439155579 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Adding a Second Convolutional Layer"
      ],
      "metadata": {
        "id": "RBFnb2kPnsBd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses the FastText word embeddings, with dropouts\n",
        "# and two convolutional layers\n",
        "cnn_model = Sequential([\n",
        "    embedding_layer_ft,\n",
        "    Conv1D(128, 5, padding='same', activation='relu'),\n",
        "    MaxPooling1D(),\n",
        "    Conv1D(64, 5, padding='same', activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Flatten(),\n",
        "    Dense(25, activation='relu'),\n",
        "    Dropout(0.25),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dropout(0.25),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "cnn_model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8mMAyxhmkhih",
        "outputId": "7ac98971-80ab-4725-83db-18fddc73d839"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_7\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_1 (Embedding)     (None, 500, 300)          43299000  \n",
            "                                                                 \n",
            " conv1d_14 (Conv1D)          (None, 500, 128)          192128    \n",
            "                                                                 \n",
            " max_pooling1d (MaxPooling1D  (None, 250, 128)         0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv1d_15 (Conv1D)          (None, 250, 64)           41024     \n",
            "                                                                 \n",
            " global_max_pooling1d_11 (Gl  (None, 64)               0         \n",
            " obalMaxPooling1D)                                               \n",
            "                                                                 \n",
            " flatten_2 (Flatten)         (None, 64)                0         \n",
            "                                                                 \n",
            " dense_19 (Dense)            (None, 25)                1625      \n",
            "                                                                 \n",
            " dropout_10 (Dropout)        (None, 25)                0         \n",
            "                                                                 \n",
            " dense_20 (Dense)            (None, 10)                260       \n",
            "                                                                 \n",
            " dropout_11 (Dropout)        (None, 10)                0         \n",
            "                                                                 \n",
            " dense_21 (Dense)            (None, 1)                 11        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 43,534,048\n",
            "Trainable params: 43,534,048\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "CPU times: user 169 ms, sys: 5.04 ms, total: 174 ms\n",
            "Wall time: 196 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Train the CNN Model for 100 epochs with early stopping (patience of 3)\n",
        "\n",
        "earlystop3 = EarlyStopping(monitor='val_loss', patience=3,\n",
        "                           restore_best_weights=True, verbose=1)\n",
        "\n",
        "history = cnn_model.fit(X_train_ft, y_train_ft, epochs=100,\n",
        "                        validation_data=(X_val_ft, y_val_ft),\n",
        "                        callbacks=[earlystop3],\n",
        "                        batch_size = 128)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t3pMlHddkh_w",
        "outputId": "631e7b7c-02d6-4403-dbf6-3c8c7c62cadb"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/100\n",
            "164/164 [==============================] - 51s 295ms/step - loss: 0.0488 - accuracy: 0.9840 - val_loss: 0.0905 - val_accuracy: 0.9842\n",
            "Epoch 2/100\n",
            "164/164 [==============================] - 47s 284ms/step - loss: 0.0077 - accuracy: 0.9981 - val_loss: 0.0893 - val_accuracy: 0.9830\n",
            "Epoch 3/100\n",
            "164/164 [==============================] - 43s 260ms/step - loss: 0.0024 - accuracy: 0.9991 - val_loss: 0.1155 - val_accuracy: 0.9816\n",
            "Epoch 4/100\n",
            "164/164 [==============================] - 36s 221ms/step - loss: 0.0013 - accuracy: 0.9993 - val_loss: 0.1565 - val_accuracy: 0.9832\n",
            "Epoch 5/100\n",
            "164/164 [==============================] - ETA: 0s - loss: 0.0018 - accuracy: 0.9991Restoring model weights from the end of the best epoch: 2.\n",
            "164/164 [==============================] - 27s 164ms/step - loss: 0.0018 - accuracy: 0.9991 - val_loss: 0.1500 - val_accuracy: 0.9825\n",
            "Epoch 5: early stopping\n",
            "CPU times: user 2min 25s, sys: 2.41 s, total: 2min 27s\n",
            "Wall time: 3min 23s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model.evaluate(X_test_ft, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a889b9e9-923a-416e-e23c-8a2054106afc",
        "id": "VfnfnviAmbzA"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 2s 4ms/step - loss: 0.0930 - accuracy: 0.9827\n",
            "Final Testing Accuracy for the CNN is 98.27311635017395 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Final Model"
      ],
      "metadata": {
        "id": "c-Pmp1zsaZGB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tf.keras.backend.clear_session()"
      ],
      "metadata": {
        "id": "kAii53A0G2rk"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Create and compile the CNN that uses the FastText word embeddings\n",
        "cnn_model_final = Sequential([\n",
        "    embedding_layer_ft,\n",
        "    Conv1D(128, 5, padding='same', activation='relu'),\n",
        "    GlobalMaxPooling1D(),\n",
        "    Dense(10, activation='relu'),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "cnn_model_final.compile(loss='binary_crossentropy',\n",
        "                  optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "cnn_model_final.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "65761eb3-9022-44a1-cad7-a17518bd75e1",
        "id": "-X1hls7SGgx2"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, 500, 300)          43299000  \n",
            "                                                                 \n",
            " conv1d (Conv1D)             (None, 500, 128)          192128    \n",
            "                                                                 \n",
            " global_max_pooling1d (Globa  (None, 128)              0         \n",
            " lMaxPooling1D)                                                  \n",
            "                                                                 \n",
            " dense (Dense)               (None, 10)                1290      \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 11        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 43,492,429\n",
            "Trainable params: 43,492,429\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "CPU times: user 993 ms, sys: 123 ms, total: 1.12 s\n",
            "Wall time: 1.1 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "# Train the CNN Model for 20 epochs with early stopping (patience of 2)\n",
        "\n",
        "earlystop2 = EarlyStopping(monitor='val_loss', patience=2,\n",
        "                           verbose=1)\n",
        "\n",
        "history = cnn_model_final.fit(X_train_ft, y_train_ft, epochs=20,\n",
        "                        callbacks=[earlystop2],\n",
        "                        validation_data=(X_val_ft, y_val_ft))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b0892440-4ff9-4997-a2de-6e77607b0eeb",
        "id": "7ZBL-eI2Ggx5"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "653/653 [==============================] - 117s 174ms/step - loss: 0.1084 - accuracy: 0.9571 - val_loss: 0.0347 - val_accuracy: 0.9874\n",
            "Epoch 2/20\n",
            "653/653 [==============================] - 67s 102ms/step - loss: 0.0108 - accuracy: 0.9968 - val_loss: 0.0331 - val_accuracy: 0.9888\n",
            "Epoch 3/20\n",
            "653/653 [==============================] - 46s 71ms/step - loss: 0.0012 - accuracy: 0.9997 - val_loss: 0.0484 - val_accuracy: 0.9843\n",
            "Epoch 4/20\n",
            "653/653 [==============================] - 30s 46ms/step - loss: 1.4741e-04 - accuracy: 1.0000 - val_loss: 0.0361 - val_accuracy: 0.9882\n",
            "Epoch 4: early stopping\n",
            "CPU times: user 3min 23s, sys: 7.33 s, total: 3min 30s\n",
            "Wall time: 4min 20s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "loss, accuracy = cnn_model_final.evaluate(X_test_ft, y_test)\n",
        "print('Final Testing Accuracy for the CNN is {} '.format(accuracy*100))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "36b21f67-cac2-47e5-a109-c8a225618953",
        "id": "MUdh0fz5Ggx5"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 3ms/step - loss: 0.0435 - accuracy: 0.9870\n",
            "Final Testing Accuracy for the CNN is 98.7006425857544 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Print accuracy, classification report, and confusion matrix\n",
        "y_pred = (cnn_model_final.predict(X_test_ft) > 0.5).astype(\"int32\")\n",
        "\n",
        "print(\"Accuracy: %.2f%%\" % (accuracy_score(y_test, y_pred)*100))\n",
        "print(\"F1 score: %.2f%%\" % (f1_score(y_test, y_pred, average='macro')*100))\n",
        "print(\"Recall: %.2f%%\" % (recall_score(y_test, y_pred, average='macro')*100))\n",
        "print(\"Precision: %.2f%%\" % (precision_score(y_test, y_pred, average='macro')*100))\n",
        "\n",
        "print(classification_report(y_test, y_pred))\n",
        "pd.DataFrame(confusion_matrix(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "outputId": "bda864b4-6957-4812-c953-da434b6ae408",
        "id": "MmeH6A1QGgx5"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "373/373 [==============================] - 1s 2ms/step\n",
            "Accuracy: 98.70%\n",
            "F1 score: 98.70%\n",
            "Recall: 98.72%\n",
            "Precision: 98.69%\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         0.0       0.99      0.98      0.99      6221\n",
            "         1.0       0.98      0.99      0.99      5708\n",
            "\n",
            "    accuracy                           0.99     11929\n",
            "   macro avg       0.99      0.99      0.99     11929\n",
            "weighted avg       0.99      0.99      0.99     11929\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      0     1\n",
              "0  6120   101\n",
              "1    54  5654"
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-53349898-2c1e-49c9-868a-0ef96aafa696\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6120</td>\n",
              "      <td>101</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>54</td>\n",
              "      <td>5654</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-53349898-2c1e-49c9-868a-0ef96aafa696')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-6b377a08-1d75-4e0a-a909-6e53120ef934\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6b377a08-1d75-4e0a-a909-6e53120ef934')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-6b377a08-1d75-4e0a-a909-6e53120ef934 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-53349898-2c1e-49c9-868a-0ef96aafa696 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-53349898-2c1e-49c9-868a-0ef96aafa696');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My final CNN model that used the pretrained FastText word embeddings achieved an accuracy of 98.70% and took 3 min and 30 s of CPU time to train."
      ],
      "metadata": {
        "id": "i2OAudQ_yH6O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Summary\n",
        "\n"
      ],
      "metadata": {
        "id": "YpZZCfSdFsOw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prototyping Analysis"
      ],
      "metadata": {
        "id": "BGyX7z2gGDSj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The results of my first two prototypes were as follows:\n",
        "\n",
        "\n",
        "| Model         | Feature Extraction Method | Training Time (CPU)  | Accuracy |\n",
        "| ------------- | ------------------------- | -------------------- | -------- |\n",
        "| Random Forest (ML)| TF-IDF                | 1min 16s             | 98.27%   |\n",
        "| BiLSTM (DL)   | GloVe Word Embeddings     | 27min 34s            | 96.03%   |\n",
        "| BiLSTM (DL)   | Continuous Bag of Words   | 16min 10s            | 97.82%   |\n",
        ""
      ],
      "metadata": {
        "id": "7LTJmumbtZ2b"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "My **Random Forest model easily outperformed my two BiLSTMs**. Although my BiLSTM model that utilized a Continuous Bag of Words model for feature extraction was somewhat close in terms of model accuracy (97.82% vs 98.27%), the Random Forest was much, much faster in terms of CPU training time."
      ],
      "metadata": {
        "id": "C8GB2Ptru4Xu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Scaling Analysis"
      ],
      "metadata": {
        "id": "6JNd4OtBGDhE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "I decided to use a **CNN model rather than a BiLSTM** in order to be able to scale up my protoype for larger volumes of data. CNNs are typically used for image processing and recognition (while BiLSTMs are typically only used for text) so they are more powerful compared to BiLSTMs. Not only are they considerably faster during training, but they can also handle more data.\n",
        "\n",
        "\n",
        "\n",
        "The results of my CNN models were as follows:\n",
        "\n",
        "| Model       | Feature Extraction Method | Training Time (CPU)  | Accuracy |\n",
        "| ----------- | ------------------------- | -------------------- | -------- |\n",
        "| CNN         | GloVe Word Embeddings     | 1min 49s             | 95.66%   |\n",
        "| CNN         | Continuous Bag of Words   | 8min 46s             | 97.96%   |\n",
        "| CNN         | FastText Word Embeddings  | 3min 30s             | 98.70%   |"
      ],
      "metadata": {
        "id": "S6v8agqtpYyF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Furthermore, I found that my previously-used methods of feature extraction (GloVe word embeddings, Continuous Bag of Words) did not result in a higher accuracy than a simple Random Forest model. So, I did some research and decided to utilize **FastText word embeddings** instead, which resulted in a **testing accuracy of 98.70%** with my CNN model.\n",
        "\n",
        "Though my CNN model that leveraged FastText word embeddings took slighlty more CPU time to train (3 min 23 s) compared to the simple Random Forest model (1 min 8 s), it boasted a higher accuracy (98.70% vs 98.27%), as well as a higher recall, precision, and f1 score.\n",
        "\n",
        "\n",
        "\n",
        "Below is a comparison of the two models:\n",
        "\n",
        "| Model | Feature Extraction Method | Training Time (CPU) | Accuracy | F1 Score | Recall | Precision |  \n",
        "| --- | --- | --- | --- | --- | --- | --- |\n",
        "| Random Forest| TF-IDF  | 1min 8s  | 98.27%   |  98.26%  | 98.27% | 98.25%  |\n",
        "| CNN | FastText Word Embeddings| 3min 30s | 98.70% | 98.70% | 98.72% | 98.69% |"
      ],
      "metadata": {
        "id": "Qi7GIvd12fGr"
      }
    }
  ]
}