Project 3 - Sentiment Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Homework_5_Yunxin",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/karenyxwang/Machine_Learning/blob/master/Sentiment%20Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QzFBHtFHRBkH",
        "colab_type": "text"
      },
      "source": [
        "# Upload your data\n",
        "To upload your own files for access in Google Colab:\n",
        "\n",
        "1. Save the file in Google Drive.\n",
        "\n",
        "2. Once it's saved, find it in your drive, right-click, and click `Get Shareable Link`.\n",
        "\n",
        "3. Flip the toggle in the pop-up to `Link Sharing On`, then click `Sharing Settings`.\n",
        "\n",
        "4. In the dropdown menu, choose `Anyone with the link can view`, then click `Copy Link`.\n",
        "\n",
        "5. Paste that link in the code field below. \n",
        "\n",
        "6. Alter the link to the format below. Specifically, the link provided by Google Drive is written by default as:\n",
        "\n",
        "`https://docs.google.com/document/d/ABCDEFG`\n",
        "\n",
        "Replace this with:\n",
        "\n",
        "`https://docs.google.com/uc?export=download&id=ABCDEFG`\n",
        "\n",
        "Make sure that the unique identifier (`ABCDEFG`) above is identical to what you copied from Google Drive. Once you have saved the file to the Google Colab server, you can now open that file in Python, as in the next cell below.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gD8xgRYrRAV3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        },
        "outputId": "d7ab6100-4377-4ccc-d32d-739f5997989e"
      },
      "source": [
        "!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Z0DH7oezZ6RMqxXlMwjUNATx12IHUZkC' -O twitter_200.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-04-09 00:25:53--  https://docs.google.com/uc?export=download&id=1Z0DH7oezZ6RMqxXlMwjUNATx12IHUZkC\n",
            "Resolving docs.google.com (docs.google.com)... 74.125.31.138, 74.125.31.113, 74.125.31.101, ...\n",
            "Connecting to docs.google.com (docs.google.com)|74.125.31.138|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-0k-7s-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/1fnk06lousd1itaka3m52u15mdhdr0ef/1586391900000/15910814371148022664/*/1Z0DH7oezZ6RMqxXlMwjUNATx12IHUZkC?e=download [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2020-04-09 00:25:54--  https://doc-0k-7s-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/1fnk06lousd1itaka3m52u15mdhdr0ef/1586391900000/15910814371148022664/*/1Z0DH7oezZ6RMqxXlMwjUNATx12IHUZkC?e=download\n",
            "Resolving doc-0k-7s-docs.googleusercontent.com (doc-0k-7s-docs.googleusercontent.com)... 74.125.139.132, 2607:f8b0:400c:c05::84\n",
            "Connecting to doc-0k-7s-docs.googleusercontent.com (doc-0k-7s-docs.googleusercontent.com)|74.125.139.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 26731 (26K) [text/csv]\n",
            "Saving to: ‘twitter_200.csv’\n",
            "\n",
            "\rtwitter_200.csv       0%[                    ]       0  --.-KB/s               \rtwitter_200.csv     100%[===================>]  26.10K  --.-KB/s    in 0s      \n",
            "\n",
            "2020-04-09 00:25:54 (54.8 MB/s) - ‘twitter_200.csv’ saved [26731/26731]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9NyeAhBd3K3h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "outputId": "8febc5a9-f03d-455b-853b-6e578edeba95"
      },
      "source": [
        "# You are going to need to run this cell, restart the runtime after running this command,\n",
        "# then start over before you can run the code in this notebook.\n",
        "!pip3 install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.0/en_core_web_md-2.2.0.tar.gz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.0/en_core_web_md-2.2.0.tar.gz\n",
            "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.0/en_core_web_md-2.2.0.tar.gz (96.4MB)\n",
            "\u001b[K     |████████████████████████████████| 96.4MB 57kB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from en-core-web-md==2.2.0) (2.2.4)\n",
            "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (1.0.0)\n",
            "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (0.4.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (46.1.3)\n",
            "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (1.1.3)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (2.0.3)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (1.0.2)\n",
            "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (1.0.2)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (2.21.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (3.0.2)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (0.6.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (4.38.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (1.18.2)\n",
            "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.0->en-core-web-md==2.2.0) (7.4.0)\n",
            "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.0->en-core-web-md==2.2.0) (1.6.0)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en-core-web-md==2.2.0) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en-core-web-md==2.2.0) (2019.11.28)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en-core-web-md==2.2.0) (3.0.4)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en-core-web-md==2.2.0) (2.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.0->en-core-web-md==2.2.0) (3.1.0)\n",
            "Building wheels for collected packages: en-core-web-md\n",
            "  Building wheel for en-core-web-md (setup.py) ... \u001b[?25l\u001b[?25hcanceled\n",
            "\u001b[31mERROR: Operation cancelled by user\u001b[0m\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cUrMXdBGAmEn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.metrics import accuracy_score, cohen_kappa_score, confusion_matrix, ConfusionMatrixDisplay\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.naive_bayes import BernoulliNB, ComplementNB, GaussianNB, MultinomialNB \n",
        "from sklearn.model_selection import train_test_split, KFold, StratifiedKFold\n",
        "from sklearn.svm import LinearSVC, SVC\n",
        "from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0ttQy1Fz3aCd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "de03dc92-b4e9-46b8-9531-c0d2536014e0"
      },
      "source": [
        "twitter = pd.read_csv(\"twitter_200.csv\")\n",
        "print(twitter.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(200, 3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-M5G2ryQRSe",
        "colab_type": "text"
      },
      "source": [
        "# Homework 5 (Due February 27, 2020)\n",
        "\n",
        "Last week you annotated a set of at least 200 microblog posts (your Training Set) with a supervised learning task designed by you and your partner, with your own annotation manual. \n",
        "\n",
        "By now, you should have two columns of labels for each post, and you should know the approximate upper bound of machine learning performance on this data, equal to your inter-rater reliability.\n",
        "\n",
        "Save your work to a CSV file with the two annotator's labels (name the columns `labels_A` and `labels_B`), and a third column with the original text of the annotated post. \n",
        "\n",
        "# Task 1\n",
        "Divide your data into a training set and a test set made up of 20% of the data. If you have 200 rows, your training set should have 160 examples, and your test set should have 40 rows."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sN6W49dCdBUw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "twitter_training, twitter_test = train_test_split(twitter, test_size=0.20, random_state=123)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2ZEWAqydB47",
        "colab_type": "text"
      },
      "source": [
        "# Task 2\n",
        "\n",
        "Use 10-fold cross-validation on your training set. Each fold should contain 8% of the original training data (if you had 200 rows originally and use an 80% training set, each fold for optimization will contain 16 examples). \n",
        "\n",
        "Optimize a machine learning classifier predicting `labels_A`, using features you extract from the microblog texts. \n",
        "\n",
        "Choose and perform **THREE** of the following optimizations:\n",
        "\n",
        "   - Compare Naïve Bayes, Logistic Regression, and SVMs on a unigram feature space.\n",
        "   - Compare a unigram feature space with a feature space that also includes longer N-grams.\n",
        "   - Compare a unigram feature space with a feature space that removes stopwords.\n",
        "   - Vary the vocabulary size of your n-gram feature space and evaluate how performance changes.\n",
        "   - If you are working with English data, additional options include:\n",
        "      - Compare an n-gram feature space with a feature space that also includes part-of-speech n-grams.\n",
        "      - Compare an n-gram feature space to a word embedding feature space.\n",
        "      - Compare a standard n-gram feature space with a lemmatized feature space.\n",
        "   - For the one most accurate model so far, tune your hyperparameters.\n",
        "       - For Naive Bayes, evaluate different implementations: ComplementNB, MultinomialNB, BernoulliNB\n",
        "       - For Support Vector Machines, evaluate different kernels including a polynomial kernel and a radial basis function kernel.\n",
        "       - For Logistic Regression, try L1 and L2 regularization, as well as unregularized features.\n",
        "\n",
        "Report the performance of your best-tuned model on the cross-validated training set.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0wiABad_lYJb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Start by defining a function to evaluate a classifier's predictions\n",
        "def evaluate(y_pred, y_actual, metrics, model_name = 'model'):\n",
        "    # Compute Confusion Matrix\n",
        "    conf_matrix = confusion_matrix(y_actual, y_pred)\n",
        "\n",
        "    # Compute and store each metric\n",
        "    model_metrics = {}\n",
        "    for (metric_name, metric) in metrics.items():\n",
        "        result = metric(y_actual, y_pred)\n",
        "        model_metrics[metric_name] = result\n",
        "\n",
        "    return conf_matrix, model_metrics\n",
        "\n",
        "# Then define a function that trains a classifier and evaluates it on one fold\n",
        "def evaluate_one_fold(classifier_name, classifier, X_train, y_train, X_test, y_test, metrics, fold_num, noisy = 'loud', labels=[]):\n",
        "\n",
        "    # Train and Evaluate Model\n",
        "    model = classifier.fit(X_train, y_train)\n",
        "    y_pred = model.predict(X_test)\n",
        "    conf_matrix, model_metrics = evaluate(y_pred, y_test, metrics, model_name = classifier_name)\n",
        "\n",
        "    # Display Results appropriately when noisy is set to 'loud' or 'quiet'\n",
        "    if noisy == 'quiet' and fold_num == 0:\n",
        "        print(f\"{classifier_name}: Fold {fold_num}\", end = '')\n",
        "    elif noisy == 'quiet':\n",
        "        print(f'...{fold_num}', end ='')\n",
        "    elif noisy == 'loud':\n",
        "        print(f\"{classifier_name}: Fold {fold_num} Results\")\n",
        "        ConfusionMatrixDisplay(conf_matrix, labels).plot(values_format='.4g')\n",
        "        plt.show()\n",
        "        print(model_metrics)\n",
        "        print(\"------------------------\")\n",
        "\n",
        "    return model_metrics\n",
        "\n",
        "# Define a function to evaluate over all folds\n",
        "def evaluate_all_folds(classifier_name, classifier, X, y, kf, metrics, noisy = 'loud', labels=[]):\n",
        "\n",
        "    # Initialize tracking variables\n",
        "    all_fold_metrics = {metric_name: [] for metric_name in metrics}\n",
        "\n",
        "    # Iterate over each fold\n",
        "    for fold_num, (train_index, test_index) in enumerate(kf.split(X, y)):\n",
        "        # Get the data subset for the current fold\n",
        "        X_train = X.iloc[train_index]\n",
        "        X_test = X.iloc[test_index]\n",
        "        y_train = y.iloc[train_index]\n",
        "        y_test = y.iloc[test_index]\n",
        "\n",
        "        # Train and Evaluate the Model\n",
        "        model_metrics =  evaluate_one_fold(classifier_name, classifier, X_train, y_train, X_test, y_test, metrics, fold_num, noisy, labels=labels)\n",
        "\n",
        "        # Update our tracking variables\n",
        "        [all_fold_metrics[metric_name].append(metric_val) for metric_name, metric_val in model_metrics.items()]\n",
        "\n",
        "    return all_fold_metrics\n",
        "\n",
        "# Define a function to compare different classifiers\n",
        "def compare_classifiers(classifiers, metrics, metric_to_optimize, df, feature_set,\n",
        "                        target, folds = 10, shuffle = True, noisy='loud', labels=[]):\n",
        "    # Initialize tracking variables\n",
        "    best = 0\n",
        "    best_name = None\n",
        "    classifier_comparison = {}\n",
        "\n",
        "    # Set up dataset and cross validation\n",
        "    X = df.loc[:, feature_set]\n",
        "    X = pd.get_dummies(X)\n",
        "    y = df[target]\n",
        "    kf = StratifiedKFold(n_splits=folds, shuffle=shuffle)\n",
        "\n",
        "    # For each classifier\n",
        "    for classifier_name, classifier in classifiers.items():\n",
        "        # Evaluate on all metrics for all folds\n",
        "        all_fold_metrics = evaluate_all_folds(classifier_name, classifier, X, y, kf, metrics, noisy = noisy, labels=labels)\n",
        "\n",
        "        # Compute average performance on metric to optimize over\n",
        "        optimization_metric_avg = np.mean(all_fold_metrics[metric_to_optimize])\n",
        "\n",
        "        # Update Tracking Variables\n",
        "        if optimization_metric_avg > best:\n",
        "            best = optimization_metric_avg\n",
        "            best_name = classifier_name\n",
        "        classifier_comparison[classifier_name] = all_fold_metrics\n",
        "        if noisy == 'quiet': \n",
        "            print()\n",
        "            print(f\"Average {metric_to_optimize}: {optimization_metric_avg:.3f}\")\n",
        "            print('-------------')\n",
        "    # Return our results\n",
        "    return best, best_name, classifier_comparison"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s0y14dXb-ulK",
        "colab_type": "text"
      },
      "source": [
        "## Question 1: compare Naïve Bayes, Logistic Regression, and SVMs on a unigram feature space.\n",
        "\n",
        "This part compares the performance of Bernoulli NB, Logistic Regression, and Linear SVM classifier on a unigram feature space with a 1000 vocabulary size bag-of-words presentation using 10-fold classification. The results show that Linear SVM has the highest average Kappa 0.243, followed by Logistic Regression with a 0.126 Kappa and Bernoulli NB with a 0.038 Kappa.\n",
        "\n",
        "However, it is worth mentioning that the results change everytime I rerun the code, sometimes the Kappa of Logistic Regression is higher and it is close to the result of Linear SVM, but most times the Kappa of Linear SVM is the highest."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nJYMRhagFdrw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "vocab_size = 1000\n",
        "\n",
        "# Build a unigram model - capturing individual words only\n",
        "vectorizer = CountVectorizer(max_features=vocab_size)\n",
        "X = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "# Make a DataFrame with BOW model representations for each message\n",
        "bow_df = pd.DataFrame(X.toarray())\n",
        "column_names = [str(i) for i in range(vocab_size)]\n",
        "\n",
        "# Make the column names the words\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  column_names[v] = k\n",
        "bow_df.columns = column_names\n",
        "\n",
        "# Add Y labels to our DataFrame\n",
        "bow_df[\"labels_A\"] = twitter_training[\"labels_A\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "egjid58QR8PM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        },
        "outputId": "8369feab-0b01-465c-de87-b7a789c54b26"
      },
      "source": [
        "# Pick Classifiers to Compare\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Logistic Regression\": LogisticRegression(),\n",
        "    \"Linear SVM\": LinearSVC()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "bow_features = column_names\n",
        "feature_set = bow_features\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bow_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet', folds=10)\n",
        "\n",
        "print(f\"Best classifier is: {best_name} \\nWith K={best:.3f}.\")    \n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.021\n",
            "-------------\n",
            "Logistic Regression: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.156\n",
            "-------------\n",
            "Linear SVM: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.174\n",
            "-------------\n",
            "Best classifier is: Linear SVM \n",
            "With K=0.174.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WtoWz2dkHHkO",
        "colab_type": "text"
      },
      "source": [
        "## Question 2: Compare unigram feature space with a feature space that includes longer N-grams.\n",
        "\n",
        "This part compares the performance of Bernoulli NB, Complement NB, and Multinomial NB classifiers under the unigram feature space, the bigram feature space and the trigram feature space. The results show that the unigram feature space is the best with a 0.087 Kappa, bigram is the second best feature space with a 0.067 Kappa, and trigram is the worst with a 0.052 Kappa. Therefore, Bigram and Trigram feature spaces seem to decrease the performance of the model, which is a surprise to me.\n",
        "\n",
        "It is worth mentioning that the results change everytime I rerun the code. However, the results always show that the unigram has the best Kappa result, followed by bigram and trigram feature space. Among the Naive Bayes classifiers, Complement NB performs the best in unigram and bigram feature space while Multinomial NB performs the best in trigram feature space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P_yGRS7LQSWm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create a vocab of n-grams from the dataframe\n",
        "def ngrams(df, vocab_size = 1000, min_n=1, max_n=1):\n",
        "  vectorizer = CountVectorizer(max_features=vocab_size, ngram_range=(1,max_n))\n",
        "  X = vectorizer.fit_transform(df['text'])\n",
        "\n",
        "  bow_df = pd.DataFrame(X.toarray())\n",
        "  column_names = [str(i) for i in range(vocab_size)]\n",
        "  for k, v in vectorizer.vocabulary_.items():\n",
        "    column_names[v] = k\n",
        "  bow_df.columns = column_names\n",
        "  return column_names, bow_df\n",
        "\n",
        "# Unigrams DataFrame\n",
        "unigram_names, unigram_df = ngrams(twitter_training, max_n=1)\n",
        "unigram_df[\"labels_A\"] = twitter_training[\"labels_A\"].values\n",
        "\n",
        "# DataFrame of Bigrams\n",
        "bigram_names, bigram_df = ngrams(twitter_training, vocab_size = 3000, max_n=2)\n",
        "bigram_df[\"labels_A\"] = twitter_training[\"labels_A\"].values\n",
        "\n",
        "# DataFrame of Trigrams\n",
        "trigram_names, trigram_df = ngrams(twitter_training, vocab_size = 5000, max_n=3)\n",
        "trigram_df[\"labels_A\"] = twitter_training[\"labels_A\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YtDIYL7GQjWx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 589
        },
        "outputId": "1b106f7a-055e-48a3-81ab-0f690c1f2f8d"
      },
      "source": [
        "# Setup classifiers and metrics to be used on all n-grams\n",
        "# Pick Classifiers to Compare\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Complement NB\": ComplementNB(), \n",
        "    \"Multinomial NB\": MultinomialNB()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "### Compare classifiers on unigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(unigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, unigram_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best unigram classifier is: {best_name} \\nWith K={best:.3f}.\")    \n",
        "\n",
        "### Compare classifiers on bigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(bigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bigram_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best bigram classifier is: {best_name} \\nWith K={best:.3f}.\")    \n",
        "\n",
        "### Compare classifiers on trigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(trigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, trigram_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best trigram classifier is: {best_name} \\nWith K={best:.3f}.\")    "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.035\n",
            "-------------\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.108\n",
            "-------------\n",
            "Multinomial NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.084\n",
            "-------------\n",
            "Best unigram classifier is: Complement NB \n",
            "With K=0.108.\n",
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.000\n",
            "-------------\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.071\n",
            "-------------\n",
            "Multinomial NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.122\n",
            "-------------\n",
            "Best bigram classifier is: Multinomial NB \n",
            "With K=0.122.\n",
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.000\n",
            "-------------\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: -0.009\n",
            "-------------\n",
            "Multinomial NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.043\n",
            "-------------\n",
            "Best trigram classifier is: Multinomial NB \n",
            "With K=0.043.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZRlKrVtcXK2q",
        "colab_type": "text"
      },
      "source": [
        "## Question 3: Compare a unigram feature space with a feature space that removes stopwords.\n",
        "\n",
        "**Also: For Naive Bayes, evaluate different implementations: ComplementNB, MultinomialNB, BernoulliNB**\n",
        "\n",
        "This part compares the performance of Bernoulli NB, Complement NB, and Multinomial NB with a unigram feature space that removes stopwords. The results show that the feature space without stopwords has a higher Kappa than the feature space with stopwords. While the highest Kappa without stopwords is 0.173 with Complement NB, the highest Kappa with stopwords is 0.118 with Complement NB. Therefore, we can conclude preliminarily that removing stopwards increase the performance of the models.\n",
        "\n",
        "However, it is worth mentioning that the results change everytime I rerun the code. From my observation, in the majority of times, the performance of classifiers are better without stopwords and usually Complement NB performs the best."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r2kNUbf8XNzX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "0b317a2f-9fe2-4fcf-d815-dd81baede65f"
      },
      "source": [
        "# Extract features for unigrams with no stopwords included.\n",
        "vectorizer = CountVectorizer(max_features=1000, ngram_range=(1,1), stop_words='english')\n",
        "X = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "no_stopwords_df = pd.DataFrame(X.toarray())\n",
        "no_stopwords_columns = [str(i) for i in range(1000)]\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  no_stopwords_columns[v] = k\n",
        "no_stopwords_df.columns = no_stopwords_columns\n",
        "no_stopwords_df[\"labels_A\"] = twitter_training[\"labels_A\"].values\n",
        "\n",
        "# Pick Classifiers to Compare\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Complement NB\": ComplementNB(), \n",
        "    \"Multinomial NB\": MultinomialNB()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "# Re-run our classifier with stopwords included, as a baseline.\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, unigram_df, unigram_names, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Classifier with stopwords K={best:.3f}.\")    \n",
        "\n",
        "# Then run our classifier on the feature space with stopwords removed.\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, no_stopwords_df, no_stopwords_columns, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Classifier without stopwords K={best:.3f}.\")   "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.011\n",
            "-------------\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.143\n",
            "-------------\n",
            "Multinomial NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.114\n",
            "-------------\n",
            "Classifier with stopwords K=0.143.\n",
            "Bernoulli NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.046\n",
            "-------------\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.146\n",
            "-------------\n",
            "Multinomial NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.102\n",
            "-------------\n",
            "Classifier without stopwords K=0.146.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8IMTQzAMcywV",
        "colab_type": "text"
      },
      "source": [
        "## Question 4: Vary vocabulary size of n-gram feature space and evaluate how performance changes.\n",
        "\n",
        "This part evaluates the performance of Complement NB while varying the vocabulary size of bigram feature space and observes how performance changes. The size of vocabulary varies from 200, 400, 600, 800 to 1000. The results show that as the vocabulary size increases, the Kappa of the Complement NB classifier also increases from around 0.1 to around 0.15. Therefore, it seems that smaller vacabulary sizes could decrease the performance of the models.\n",
        "\n",
        "However, it must be stressed that the result is not consistent everytime. While sometimes the performance of the classifier increases with the vocabulary size, other times the performance decreases or shows no pattern at all. Very few times the Kappa even becomes negative as the size increases. Therefore I feel cautious to conclude that the performance of the classifier improves as the vocabulary size increases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dVQDro7_Xn-c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "e74a615c-531d-4d06-9fa8-f3f949548c83"
      },
      "source": [
        "# set vocabulary sizes\n",
        "kappa_results = []\n",
        "sizes_to_test = [200, 400, 600, 800, 1000]\n",
        "\n",
        "for size in sizes_to_test:\n",
        "  classifiers = {\"Complement NB\": ComplementNB()}\n",
        "\n",
        "  # Set a list of metrics we want to use to compare our classifiers \n",
        "  metrics = {\n",
        "      \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "      \"Kappa\"    : cohen_kappa_score\n",
        "  }\n",
        "\n",
        "  # Choose a metric to optimize over\n",
        "  metric_to_optimize = 'Kappa'\n",
        "\n",
        "  # Pick features to use\n",
        "  sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "  feature_set = list(bigram_df.columns[:-1])\n",
        "\n",
        "  # DataFrame of Bigrams\n",
        "  bigram_names, bigram_df = ngrams(twitter_training, vocab_size = size, min_n=2,max_n=2)\n",
        "  bigram_df[\"labels_A\"] = twitter_training[\"labels_A\"].values\n",
        "\n",
        "  best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bigram_df, bigram_names, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "  print(f\"At vocab size {size}, K={best:.3f}\")\n",
        "  kappa_results.append(best)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.122\n",
            "-------------\n",
            "At vocab size 200, K=0.122\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.211\n",
            "-------------\n",
            "At vocab size 400, K=0.211\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.057\n",
            "-------------\n",
            "At vocab size 600, K=0.057\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.098\n",
            "-------------\n",
            "At vocab size 800, K=0.098\n",
            "Complement NB: Fold 0...1...2...3...4...5...6...7...8...9\n",
            "Average Kappa: 0.070\n",
            "-------------\n",
            "At vocab size 1000, K=0.070\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VpH07wD2buK6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "aceaa652-ed58-4621-db56-b7bc9968ae16"
      },
      "source": [
        "plt.plot(sizes_to_test, kappa_results)\n",
        "plt.ylim(0, 0.2)\n",
        "plt.ylabel(\"Kappa\")\n",
        "plt.xlabel(\"Vocabulary Size\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXgUZbb48e/Jzr6EsIUkJMCAIGvC\nJoIgiriBjqigKDoq4zjjOD9Hr3gdR+XOeNU7jss4OuKuoIg7riiCoiJLAgiENQQCCVvYlwAhyfn9\nURVtYpA0pFKd5Hyep59Uv/VW1elOJ6drO6+oKsYYY0xFhfkdgDHGmOrFEocxxpigWOIwxhgTFEsc\nxhhjgmKJwxhjTFAscRhjjAmKp4lDRIaLyGoRyRKRCeXMv11EVojIUhH5UkSSAuaNE5G17mNcQHuq\niCxz1/mkiIiXr8EYY8yxxKv7OEQkHFgDnAvkAguBMaq6IqDPEGC+qhaIyO+Awap6pYg0BdKBNECB\nDCBVVXeLyALgj8B84BPgSVX91JMXYYwx5me83OPoA2SparaqFgJTgZGBHVR1tqoWuE/nAW3c6fOA\nL1R1l6ruBr4AhotIK6Chqs5TJ+O9Clzi4WswxhhTRoSH644HNgU8zwX6/kL/G4DSPYfylo13H7nl\ntP+MiIwHxgPUq1cvtVOnTsHEbgLsKShk0+5DNKsfTatGMUEte6SohDXb9tO8QTQtGga3rDHGXxkZ\nGTtUNa5su5eJo8JEZCzOYamzKmudqjoJmASQlpam6enplbXqWmXF5n38+pnvGNmmMVNu7EtkePA7\nqTe8vJDFm/bwzV1nUycq3IMojTFeEJGc8tq9PFSVByQEPG/jth1DRM4B7gFGqOqREyybx0+Hs467\nTlM59hYc5ebJGTSqE8m/r+p1UkkDYPygFHYdLOSdRbkn7myMCXleJo6FQAcRSRaRKGA0MD2wg4j0\nBJ7FSRrbA2bNAIaJSBMRaQIMA2ao6hZgn4j0c6+muhb4wMPXUGuVlCi3T1vClr2HePrqXsQ1iD7p\ndfVJbkr3No14/ptsikusqKYx1Z1niUNVi4A/4CSBlcA0Vc0UkYkiMsLt9n9AfeAtEVkiItPdZXcB\n/4OTfBYCE902gFuA54EsYB0/nRcxlehfs7L4ctV27r2oM6lJTU9pXSLC+EHt2LCzgC9WbKukCI0x\nfvHsctxQYuc4gjN79XZ+8/JCLu0Rz6NXdKcybpUpKi5hyKNfEVc/mndvGVAJURpjvCYiGaqaVrbd\n7hw3x9i4s4A/TV1Cp5YN+fulXSslaQBEhIdx45kpLNq4h/QNu068gDEmZFniMD86VFjMzZMzUFX+\nM7ZXpV8BdXlaGxrXjWTSnOxKXa8xpmpZ4jAAqCr3vL+MFVv28cToniTF1qv0bdSNiuCafkl8sXIb\n2fkHKn39xpiqYYnDADB5/kbeXZTHbUM7MKRTc8+2c23/tkSGh/H8t+s924YxxluWOAwZObuZ+GEm\nQzrGcdvQDp5uK65BNJf1iuftjFx2HDhy4gWMMSHHEkctl7//CLdMyaBloxgeu7IHYWHeFxu+cWAK\nhUUlvPp9uTelGmNCnCWOWqyouIRb31jEnoKj/GdsKo3rRlXJdtvF1eec01rw2vcbOFRYXCXbNMZU\nHksctdgjM1YzL3sXD17alS6tG1Xptn97Vgq7C47yVsamE3c2xoQUSxy11MdLtzBpTjbX9EvistQ2\nJ16gkqUlNaFnYmOe/2a9lSExppqxxFELrd22nzvf/oFeiY2596LOvsQgIowfmMLGXQXMyNzqSwzG\nmJNjiaOW2X/4KL+dnEHdqHCevjqVqAj/PgLDurQkKbYuz87JpjaUvjGmprDEUYuoKne+tZScnQU8\ndVUvWgY5KFNlCw8TbjwzmR827WHhht2+xmKMqThLHLXIf77O5rPMrdx9fif6pcT6HQ4Ao1ITaGJl\nSIypVixx1BLfZe3g/2as4sJurbjhzGS/w/lRnahwrunflpkrt5G13cqQGFMdWOKoBfL2HOLWNxbT\nLq4+j1zWrdIq3laWcf2TiI4I4/lvbK/DmOrAEkcNd/hoMbdMzqCwqIT/XJNKveiQGGb+GLH1oxmV\n2oZ3F+Wxff9hv8MxxpyAJY4a7oEPV/BD7l7+cXl32sXV9zuc47pxYApHS0p4da6VITEm1FniqMGm\nLdzEGws28rvB7Rh+eku/w/lFyc3qMaxzC16bl0NBYZHf4RhjfoGniUNEhovIahHJEpEJ5cwfJCKL\nRKRIREYFtA9xxyAvfRwWkUvceS+LyPqAeT28fA3V1dLcPfzlg+Wc2b4Zdwzr6Hc4FTJ+UAp7Dx1l\n2kIrQ2JMKPMscYhIOPBv4HygMzBGRMreprwRuA54PbBRVWerag9V7QGcDRQAnwd0ubN0vqou8eo1\nVFe7Dhbyu8mLaFYviidG9yC8CireVobUpKakJjXhhe/WU1Rc4nc4xpjj8HKPow+QparZqloITAVG\nBnZQ1Q2quhT4pf8So4BPVbXAu1BrjuIS5bapi8nff4RnxqYSWz/a75CCctPAFDbtOsRnVobEmJDl\nZeKIBwKPOeS6bcEaDbxRpu3vIrJURB4Tker1n9Fjj32xhm/W7mDiyC50T2jsdzhBO7dzC5Kb1WOS\nlSExJmSF9MlxEWkFdAVmBDTfDXQCegNNgbuOs+x4EUkXkfT8/HzPYw0Fn2du5anZWVyZlsDoPol+\nh3NSwsOEGwcmszR3L/PX7/I7HGNMObxMHHlAQsDzNm5bMK4A3lPVo6UNqrpFHUeAl3AOif2Mqk5S\n1TRVTYuLiwtys9VPdv4B/jztB7rGN+KBkV38DueUXNarDbH1oqwMiTEhysvEsRDoICLJIhKFc8hp\nepDrGEOZw1TuXgji3P58CbC8EmKt1goKi7h5cgYR4cIzY3sRExnud0inJCYynGv7t2XWqu2s3bbf\n73CMMWV4ljhUtQj4A85hppXANFXNFJGJIjICQER6i0gucDnwrIhkli4vIm1x9li+LrPqKSKyDFgG\nNAP+5tVrqA5UlbveWUbW9gM8OaYnbZrU9TukSnFN/yRiIsN4zsqQGBNyPK0/oaqfAJ+UaftrwPRC\nnENY5S27gXJOpqvq2ZUbZfX24ncb+PCHzdx5XkcGdqg5h+Sa1ovi8tQE3ly4iTuGdaR5Q39LwBtj\nfhLSJ8fNL1uwfhcPfrKSczu34HdntfM7nEp3w5nJHC0p4eW5G/wOxRgTwBJHNbVt32FumbKIpKZ1\nefSK7oRVk5v8gtG2WT2Gd2nJ5Hk5HDhiZUiMCRWWOKqhwqISbpmyiINHivjPNak0jIn0OyTPjB+U\nwr7DRVaGxJgQYomjGnrwk5Vk5OzmkVHd+FWLBn6H46meiU3o3bYJL3xrZUiMCRWWOKqZ9xbn8vLc\nDdxwZjIXd2/tdzhVYvygduTtOcQny60MiTGhwBJHNbJi8z7ufncZfZKbMuH8Tn6HU2WGdmpOSlw9\nJs1ZZ2VIjAkBljiqib0FR7l5cgaN6kTy1FU9iQyvPb+6sDDhpoEpLM/bx/fZO/0Ox5har/b896nG\nSkqU26ctYfOeQzx9dS+aN6h99zRc2jOeZvWtDIkxocASRzXw1Owsvly1nXsv6kxqUlO/w/FFTGQ4\n4/q35avV+azeamVIjPGTJY4QN3v1dh6buYZLe8Zzbf8kv8Px1dh+SdSJDLcyJMb4zBJHCNu4s4A/\nTV1CxxYNePDSrjh1HWuvJvWiuCKtDR8syWPr3sN+h2NMrWWJI0QdPlrMzZMzUFWevSaVOlHVu+Jt\nZbnhzBSKS9TKkBjjI0scIUhVuee95azYso/HR/cgKbae3yGFjMTYupx/eiumzLcyJMb4xRJHCJo8\nfyPvLMrltqEdOLtTC7/DCTnjB6Ww/3ARUxds9DsUY2olSxwhZtHG3Uz8MJPBHeO4bWgHv8MJSd0T\nGtMnuSkvfrueo1aGxJgqZ4kjhOTvP8ItkxfRslEMj1/Zo0ZWvK0svx2Uwua9h/l46Ra/QzGm1rHE\nESKKiku49Y1F7C4o5JmrU2lcN8rvkELakI7Nad+8PpPmZFsZEmOqmCWOEPHIjNXMy97Fg5d25fT4\nRn6HE/KcMiTJrNiyj++yrAyJMVXJEkcI+HjpFibNyeaafklcllruSLqmHJf0jKdZ/Wgm2Q2BxlQp\nTxOHiAwXkdUikiUiE8qZP0hEFolIkYiMKjOvWESWuI/pAe3JIjLfXeebIlKtj+lkbd/PnW//QM/E\nxtx7UWe/w6lWoiPCuX5AW+asyWflln1+h2NMreFZ4hCRcODfwPlAZ2CMiJT9z7gRuA54vZxVHFLV\nHu5jRED7w8Bjqtoe2A3cUOnBV5H9h48y/rUM6kaF88zVqURF2A5gsMb2TaJulJUhMaYqefmfqg+Q\nparZqloITAVGBnZQ1Q2quhSo0DWV4tTcOBt42216Bbik8kKuOqrKnW8tJWdnAf8a04uWjWpfxdvK\n0KhuJFf2TmD6ks1s2XvI73CMqRW8TBzxQOBA0bluW0XFiEi6iMwTkdLkEAvsUdXSW4aPu04RGe8u\nn56fnx9s7J57dk42n2VuZcLwTvRvF+t3ONXabwYko8BL323wOxRjaoVQPjaSpKppwFXA4yLSLpiF\nVXWSqqapalpcXJw3EZ6k77J28Mhnq7iwWytuHJjsdzjVXkLTulzQtRWvz9/IvsNH/Q7HmBrPy8SR\nByQEPG/jtlWIqua5P7OBr4CewE6gsYhEnMw6Q0HenkPc+sZiUuLq88hl3Wp9xdvKMn5gCgeOWBkS\nY6qCl4ljIdDBvQoqChgNTD/BMgCISBMRiXanmwEDgBXq3Ok1Gyi9Amsc8EGlR+6RI0XF3DI5g8Ki\nEp69JpV60REnXshUSNc2jeifEsuL326gsMjKkBjjJc8Sh3se4g/ADGAlME1VM0VkooiMABCR3iKS\nC1wOPCsime7ipwHpIvIDTqJ4SFVXuPPuAm4XkSyccx4vePUaKtv901fwQ+5e/nF5d9rF1fc7nBpn\n/KAUtu47zEdLN/sdijE1mtSGcg1paWmanp7uawzTFm7iv95Zys1ntWPC+Z18jaWmUlXOe3wOYSJ8\nettAOwxozCkSkQz3XPMxQvnkeI2xLHcvf/lgOQPax3LHsF/5HU6NJSLcNDCFVVv3883aHX6HY0yN\nZYnDY7sPFnLz5Aya1YviydE9iQi3t9xLI3q0pnmDaCbNsRsCjfGK/RfzUHGJ8sepi8nff4RnxqYS\nWz/a75BqPKcMSTLfZu0gc/Nev8MxpkayxOGhx75Ywzdrd/DAyC50T2jsdzi1xlV9E6kXFc5zttdh\njCcscXjk88ytPDU7iyvTEhjTJ9HvcGqVRnUiGd0nkQ+XbmHzHitDYkxls8ThgfU7DvLnaT/QNb4R\nD4zs4nc4tdL1A9oC8OK36/0NxJgayBJHJSsoLOLm1zKICBeeGduLmMhwv0Oqldo0qctF3VrxxoKN\n7D1kZUiMqUyWOCqRqnLXO8tYs30/T47pSZsmdf0OqVa7aWAKBwuLecPKkBhTqSxxVKKXvtvAhz9s\n5o5hHRnYIbQKK9ZGp8c3YkD7WF76br2VITGmElniqCQL1u/iwU9Wcm7nFvzurKAK+RoPjR/Ujm37\njjD9BytDYkxlscRRCbbtO8wtUxaR0LQuj17RnbAwK3URKgZ1aEanlg14bk42taG8jjFVwRLHKSos\nKuGWKYs4eKSI/4xNpWFMpN8hmQClZUhWb9vP12tCb0AvY6ojSxyn6MFPVpKRs5uHR3WjY8sGfodj\nynFx99a0bBhjZUiMqSSWOE7Be4tzeXnuBn4zIJkR3Vv7HY45jqiIMK4f0Ja563ayPM/KkBhzqixx\nnKSVW/Zx97vL6JPclLsvsDLpoW5M30TqR0fYXocxlcASx0nYe+goN0/OoGFMJE9d1ZNIq3gb8hrG\nRDKmTwIfL9tC7u4Cv8Mxplqz/3hBKilRbn9zCXm7D/HM2F40bxDjd0imgq4fkIwAL367we9QjKnW\nLHEE6anZWXy5ajv3XtSZ1KSmfodjgtC6cR0u7t6aqQs3srfAypAYc7I8TRwiMlxEVotIlohMKGf+\nIBFZJCJFIjIqoL2HiHwvIpkislRErgyY97KIrBeRJe6jh5evIdDs1dt5bOYaLunRmmv7J1XVZk0l\numlgCgWFxUxZkON3KMZUW54lDhEJB/4NnA90BsaISOcy3TYC1wGvl2kvAK5V1S7AcOBxEQkc0OJO\nVe3hPpZ48gLK2LSrgD9NXULHFg343193s/Gsq6nOrRsysEMzXvpuA0eKiv0Ox5hqycs9jj5Alqpm\nq2ohMBUYGdhBVTeo6lKgpEz7GlVd605vBrYDvhV/Ony0mN++lkGJKs9ek0qdKKt4W52NH5RC/v4j\nfLDEypAYczK8TBzxwKaA57luW1BEpA8QBawLaP67ewjrMREpdzxWERkvIukikp6ff/J3DKsq97y3\nnBVb9vH4lT1Iiq130usyoeHM9s04rVVDnpuTTUmJlSExJlghfXJcRFoBrwHXq2rpXsndQCegN9AU\nuKu8ZVV1kqqmqWpaXNzJ76xMmb+Rdxbl8sehHRh6WouTXo8JHSLC+EHJrN1+gK/WbPc7HGOqHS8T\nRx6QEPC8jdtWISLSEPgYuEdV55W2q+oWdRwBXsI5JOaJRRt388CHmQzuGMefhnbwajPGBxd1a02r\nRlaGxJiT4WXiWAh0EJFkEYkCRgPTK7Kg2/894FVVfbvMvFbuTwEuAZZXatQuVeXBj1fSslEMj1/Z\nwyre1jCR4WH8ZkAy87J3sTR3j9/hGFOteJY4VLUI+AMwA1gJTFPVTBGZKCIjAESkt4jkApcDz4pI\nprv4FcAg4LpyLrudIiLLgGVAM+BvXsQvIky6No2XrutN47pRXmzC+Gx0nwQaWBkSY4ImtWGMgrS0\nNE1PT/c7DBOC/veTlTz3TTZf3zmEhKY21K8xgUQkQ1XTyraH9MlxY7x2/YBkwsOEF75d73coxlQb\nljhMrdayUQwjusfz5sJN7Cko9DscY6oFSxym1rtpUDKHjhYzeZ6VITGmIixxmFqvU8uGnPWrOF6e\nm8Pho1aGxJgTscRhDE4Zkh0HjvD+4grfamRMrWWJwxjgjHaxdGndkOe+sTIkxpyIJQ5jKC1DksK6\n/IPMWmVlSIz5JZY4jHFd0LUV8Y3rMOkbuyHQmF9S4cQhIk1EpI87+NIgERnkZWDGVLXI8DB+c2Yy\nC9bvYskmK0NizPFUKHGIyI3AHJzyIQ+4P+/3Lixj/HFl7wQaxEQwac66E3c2ppaq6B7HbThlzHNU\ndQjQE7CvZKbGqR8dwdh+SXy2fCs5Ow/6HY4xIamiieOwqh4GEJFoVV0FdPQuLGP8c90Zba0MiTG/\noKKJI9cd8/t94AsR+QCw22xNjdSiYQyX9IhnWvomdh+0MiTGlFWhxKGql6rqHlW9H7gXeAFnLAxj\naqSbBqVw+GgJr1kZEmN+JpirqnqJyB+BbkCuqtpXMVNj/apFA4Z0jOOVuRusDIkxZVT0qqq/Aq8A\nsTiDJ70kIn/xMjBj/DZ+UDt2Hizk3UVWhsSYQBXd47ga6K2q96nqfUA/4BrvwjLGf/1SmtI1vhHP\nWxkSY45R0cSxGYgJeB4N2NcwU6OVliHJ3nGQmSu3+R2OMSGjooljL5ApIi+LyEvAcmCPiDwpIk8e\nbyERGS4iq0UkS0QmlDN/kIgsEpEiERlVZt44EVnrPsYFtKeKyDJ3nU+KiFTwNRgTtPNPb0mbJnVs\nXHJjAkRUsN977qPUVydaQETCgX8D5wK5wEIRma6qKwK6bQSuA+4os2xT4D4gDVAgw112N/AMcBMw\nH/gEGA58WsHXYUxQIsLDuOHMZB74cAUZObtJTWrid0jG+K5CiUNVXxGRKKATzj/y1RW4qqoPkKWq\n2QAiMhUYCfyYOFR1gzuvpMyy5wFfqOoud/4XwHAR+QpoqKrz3PZXcS4LtsRhPHNFWgKPz1zLc3Oy\nSb0m1e9wjPFdRa+qugBYBzwJPAVkicj5J1gsHtgU8DzXbauI4y0b706fcJ0iMl5E0kUkPT8/v4Kb\nNebn6kVHMLZfIjNWbGXDDitDYkxFz3H8ExiiqoNV9SxgCPCYd2GdOlWdpKppqpoWFxfndzimmht3\nRlsiw8J4/ls71+GH3N0F3DIlg+4PfM6Ed5ayPG+v3yHVahVNHPtVNSvgeTaw/wTL5AEJAc/bUPEr\nsY63bJ47fTLrNOakNW8Qw6U943krPZedB474HU6tcfhoMU/MXMs5//yaWau20y+lKe8vyeOif33L\nyH9/x1vpm+wGTR9UNHGki8gnInKde4XThzgnu38tIr8+zjILgQ4ikuyeHxkNTK/g9mYAw9wxQJoA\nw4AZqroF2Cci/dyrqa4FPqjgOo05JTcNSuZIkZUhqQqqymfLtzD00a95bOYahp7Wgll/Hsyz16Qx\n/7/P4b6LO3Pg8FHufHspfR/8kv/5aAXZ+Qf8DrvWENUT39jkXoJ7PKqqvznOchcAjwPhwIuq+ncR\nmQikq+p0EemNc7VWE+AwsFVVu7jL/gb4b3dVf1fVl9z2NOBloA7OSfFb9QQvIi0tTdPT00/4Oo05\nkRteXsjiTXv47q6zqRMV7nc4NdLabft54MMVfJu1g04tG3DfxV3o3y72Z/1Ule+zdzJl3kZmZG6l\nqEQZ0D6WsX2TOKdzCyLDbYDTUyUiGaqa9rP2iiSO6s4Sh6ks87N3cuWkefztktMZ2y/J73BqlH2H\nj/LEzLW8MncDdaPC+fOwjlzdN5GICiSA7fsO8+bCTbyxYCOb9x6mRcNoRvdOZEyfRFo2ijnh8qZ8\np5Q4RCQGuAHoQsAd5Mfb0wg1ljhMZVFVLnl6LnsLCvnyz4MJD7P7T09VSYnydkYuj8xYxc6DhYzu\nncid53Wkab2ooNdVVFzC7NX5TJ6Xw5y1+YSJcM5pzRnbL4kB7ZoRZr+voBwvcVT0BsDXgFU491dM\nxKldtbLywjOmehARxg9M4fevL+KLFVsZfnorv0Oq1pZs2sN9Hyznh9y9pCY14eXr+3B6fKOTXl9E\neBjndm7BuZ1bsHFnAVMW5PBWei4zMrfRNrYuV/dNYlRqG5qcRFIyP6noHsdiVe0pIktVtZuIRALf\nqGo/70M8dbbHYSpTcYky+B+ziasfzbu3DPA7nGopf/8RHvlsFW9l5NK8QTR3X9CJS3rE40UFocNH\ni/l0+RYmz9tIRs5uoiLCuKhbK8b2S6JnQmNPtllTnOoex1H35x4ROR3YCjSvrOCMqU7Cw4Qbz0zh\nvumZpG/YRVrbpn6HVG0cLS7hlbkbeGLmWg4XFfPbs1K49ewO1I+u6L+i4MVEhnNpzzZc2rMNK7fs\nY/K8HN5fnMe7i/Lo0rohY/slMbJHa+pGeRdDTVPRPY4bgXeArjhXNNUH7lXVZz2NrpLYHoepbAWF\nRZzx0Cz6tG3KpGt/9oXMlOObtfk88OEKsrYfYHDHOP56UWdS4ur7EsuBI0W8tziPKfNyWLV1Pw2i\nI/h1r3jG9kuiQ4sGvsQUik7q5LiIJKjqpuPMu0hVP6rEGD1jicN44dHPV/PU7Cy+vP0s3/4BVgeb\ndhXwt49XMCNzG0mxdfnrRZ05u1PzkDhEpKpk5OzmtXk5fLpsK4XFJfRJbsrYfkkM79KSqIjafUnv\nySaOVcDw0mKEAe3XA39R1XaVHagXLHEYL+TvP8KAh2cxKrUND17a1e9wQs6hwmKe+Xodz369jjAR\n/nB2e24cmEx0RGje/7LzwBGmpefy+oIcNu06RLP6UVyRlsBVfRNp06Su3+H54mQTR+kNfBeq6lq3\n7W7gKuB8Vc097sIhxBKH8crd7y7jnUW5zJ1wNs3qR/sdTkhQVT5dvpW/f7ySvD2HGNG9NXdf0IlW\njer4HVqFlJQoX6/NZ8q8HGat2o4CZ3d0Lukd9Ku4WnUJ9knfxyEiQ4FnccqX34hTLv1Cd2yMasES\nh/HKuvwDnPPPr7l1SHtuH9bR73B8t3rrfh74MJO563ZyWquGPDCiC32Sq+/FA3l7DvHG/I1MXbiJ\nHQeO0KZJHa7qm8gVaQm14ovCqd4AOBCnNMhc4ApVPVz5IXrHEofx0k2vprNwwy6+nzC01pYh2Vtw\nlMdmruG1eTk0iIngz8M6clWfxBrz7bywqIQZmVuZPC+H+et3ERkunH+6c0lv77ZNQuJ8jRdO6nJc\nEdmPM3CT4IwzPhTY7hYYVFVt6EWwxlQn4wel8MWKbbyVsYlr+7f1O5wqVVyivJW+iUdmrGZPQSFX\n9U3kz+d2rHE32EVFhHFx99Zc3L01a7ftZ8r8jbyTkcv0HzbTsUUDru6XyKU942kQE+l3qFXCalUZ\nc4pUlV8/M5edBwqZfUftKUOSkbOb+6dnsixvL33aNuW+EZ3p0vrk7/qubgoKi5i+ZDOT5+ewPG8f\ndaPCGdkjnrH9EmvM+2BFDi1xGA99umwLv5uyiKev7sUFXWt2GZLt+w7z0GereHdRHi0aRvPfF5zG\niO6ta+zhmhNRVX7I3cvkeTl8+MNmjhSV0CuxMWP7JXFB11bERFbfw5eWOCxxGA8VlyhDH/2KRnWj\neP+WM2rkP9HCohJenrueJ7/MorCohBsHJvP7Ie2p5+Fd39XNnoJC3s7IZcr8jazfcZAmdSO5PC2B\nq/smkhRbz+/wgmaJwxKH8dhr83K49/3lTPtt/2p9JVF5vl6TzwMfZpKdf5ChnZpz70Wdadus+v0j\nrColJcrcdTuZPC+HL1Zuo7hEGdihGWP7JTG0U/MKlYoPBZY4LHEYjx0qLGbAw7PoldiY58f19juc\nSrFxZwETP1rBzJXbSG5Wj79e1JkhnaxMXTC27j3M1IUbeWPBRrbtO0KrRjGM6ZPI6N4JNG8Y2mOF\nWOKwxGGqwGNfrOGJL9cy8/azaN+8+pYhKSgs4unZ65j0TTaRYcKtQztw/YC2IXvXd3VQVFzCzJXb\nmTI/h2/W7iAiTBjWpQVj+ybRv11sSB7etMRhicNUgZ0HjnDGQ7O4tGc8D13Wze9wgqaqfLR0Cw9+\nspItew9zac94JpzfiRYh/s24ulm/4yCvz89hWnouew8dJSWunjNWSK82NKobOpf0+pI4RGQ48ATO\nmOPPq+pDZeZHA68CqcBO4NZyJtcAABWySURBVEpV3SAiVwN3BnTtBvRS1SUi8hXQCjjkzhumqtt/\nKQ5LHKYq3fPeMt5Kz+XbCUNo3qD6/MNduWUf90/PZP76XXRp7dz1bSXjvXX4aDEfLd3C5Hk5LNm0\nh5jIMC7u1pqx/ZLontDY7/CqPnGISDiwBjgXyAUWAmNUdUVAn1uAbqp6s4iMBi5V1SvLrKcr8H5p\nQUU3cdyhqhXOBJY4TFVav+MgZz/6Fb8f3J47zgv9MiR7Cgr55xdrmDwvh0Z1IrnzvE5c2Tuh1tyP\nEiqW5+1lyvwc3l+8mUNHi+ka34ix/RIZ0T3et4oEx0scXp7a7wNkqWq2qhYCU4GRZfqMBF5xp98G\nhsrPD/SNcZc1plpIblaPYZ1b8Nq8HAoKi/wO57iKS5Qp83MY8o+vmDwvh2v6JTH7jsFc1bfmlAqp\nTk6Pb8T//rob8+8ZygMjunD4aDF3vbOMPg/O5IEPM8nafsDvEH/k5QXY8UDgWB65QN/j9VHVIhHZ\nC8QCOwL6XMnPE85LIlKMM7jU37Sc3SYRGQ+MB0hMTDyFl2FM8MYPaseMzG1MW7iJ6wYk+x3Oz6Rv\n2MV90zPJ3LyPvslNuX9EF05rZRWEQkHDmEjGndGWa/snsWD9LibP38jkeTm89N0G+qfEMrZfEsO6\ntCDSx0t6Q/rOHRHpCxSo6vKA5qtVNU9EGuAkjmtwzpMcQ1UnAZPAOVRVFfEaUyo1qQmpSU14/tv1\njO2XFDLX7W/de5iHPl3J+0s206pRDP8a05OLurUKySt6ajsRoW9KLH1TYsnf35lp6Zt4ff5Gfv/6\nIuIaRDO6dwJj+iTSunHVl6v38tOcByQEPG/jtpXbR0QigEY4J8lLjQbeCFxAVfPcn/uB13EOiRkT\ncsYPSiF39yE+y9zqdygcKSrmma/WcfajX/HJ8q3cenZ7vvzzWVxci0uFVCdxDaL5/ZD2zPmvIbww\nLo3TWzfkqdlZnPnwLG58JZ2vVm+npKTqvh97ucexEOggIsk4CWI0zgBQgaYD44DvgVHArNLDTiIS\nBlwBDCzt7CaXxqq6Q0QigYuAmR6+BmNO2jmntSC5WT0mzcnmwq7+faufvWo7Ez9awfodBzm3cwvu\nvbAzibG1c0S76i48TBh6WguGntaCTbsKeH3BRqYt3MTMlc6wvFf1SeTytASaelyd2OvLcUtHEAwH\nXlTVv4vIRCBdVaeLSAzwGtAT2AWMVtVsd9nBwEOq2i9gffWAOUCku86ZwO2qWvxLcdhVVcYvU+bn\ncM97y5k6vh/9UmKrdNsbdhxk4kcrmLVqOylxzl3fgzvaXd81zZGiYj5bvpUp8zayYMMuoiLCuLBr\nK8b2S6RX4qmNFWI3AFriMD44fLSYAQ/NontCY168rmrKkBw8UsRTs7N44Zv1REWEcdvQDow7oy1R\nEaFxnsV4Z/XW/Uyel8N7i/M4cKSITi0b8K8xPenQosFJre+kBnIyxpyamMhwru3flsdmrmHttv0n\n/QdcEarK9B828+AnK9m27wiX9WrDXcM7hnw9JFN5OrZswP9ccjp3nd+JD5bk8d6iPFp5cPLc9jiM\n8diug4Wc8dCXjOjemkdGdfdkG5mb93L/9EwWbthN1/hG3D+iC6lJTTzZlqk9bI/DGJ80rRfF5akJ\nvLlwE3cMq9w9gN0HC/nH56t5Y8FGmtSN4uHLunJ5agJhdgOf8ZAd9DSmCtw4MJmjJSW8PHdDpayv\nuER57fsNDP7HV0xd6Ix1PuuOwVzZO9GShvGc7XEYUwWSYusxvEtLJs/L4ZYh7al/CqPmzc/eyf0f\nrmDlln30T4nl/hFd6NjSu3MnxpRlexzGVJHxg1LYd7iIaQs3nbhzObbsPcStbyzmyknz2HfoKE9f\n3YvXb+prScNUOdvjMKaK9ExsQp+2TXnh2/Vc27/iZUgOHy3mhW/X89SsLIpV+ePQDvzurHa+VUw1\nxvY4jKlCNw1KIW/PIT5ZfuIyJKrKzBXbOO/xOfzfjNWc9as4vrz9LG4/91eWNIyvbI/DmCo0tFNz\nUuLqMWnOOi7+heKC2fkHeODDFXy9Jp/2zevz2g19GNghroqjNaZ8ljiMqUJhYcJNA1O4+91lfL9u\nJ2e0b3bM/ANHivjXrLW8+O16YiLC+cuFpzHujLa+ltA2pixLHMZUsUt7xvPo56uZ9E32j4mjpER5\nf0ke//vpKvL3H+Hy1Db81/BOxDWI9jlaY37OEocxVSwmMpxx/dvy6BdrWL11P4VFJdw3fTmLNu6h\ne5tGPHdtGj1CYLxpY47HEocxPhjbL4mnv1rHja8uJHf3IWLrRfHIqG6M6tXGbuAzIc8ShzE+aFIv\niqv7JvLy3A38ZkAyt53TgYYxkX6HZUyFWOIwxicTzu/ELUPaez7ojjGVzS7VMMYnEeFhljRMtWSJ\nwxhjTFAscRhjjAmKp4lDRIaLyGoRyRKRCeXMjxaRN93580WkrdveVkQOicgS9/GfgGVSRWSZu8yT\ncioD6hpjjAmaZ4lDRMKBfwPnA52BMSLSuUy3G4DdqtoeeAx4OGDeOlXt4T5uDmh/BrgJ6OA+hnv1\nGowxxvycl3scfYAsVc1W1UJgKjCyTJ+RwCvu9NvA0F/agxCRVkBDVZ2nzpi3rwKXVH7oxhhjjsfL\nxBEPBA48kOu2ldtHVYuAvUCsOy9ZRBaLyNciMjCgf+4J1gmAiIwXkXQRSc/Pzz+1V2KMMeZHoXpy\nfAuQqKo9gduB10WkYTArUNVJqpqmqmlxcVZV1BhjKouXiSMPSAh43sZtK7ePiEQAjYCdqnpEVXcC\nqGoGsA74ldu/zQnWaYwxxkNeJo6FQAcRSRaRKGA0ML1Mn+nAOHd6FDBLVVVE4tyT64hICs5J8GxV\n3QLsE5F+7rmQa4EPPHwNxhhjyvCs5IiqFonIH4AZQDjwoqpmishEIF1VpwMvAK+JSBawCye5AAwC\nJorIUaAEuFlVd7nzbgFeBuoAn7oPY4wxVUSci5NqtrS0NE1PT/c7DGOMqVZEJENV08q2h+rJcWOM\nMSHKEocxxpigWOIwxhgTFEscxhhjgmKJwxhjTFAscRhjjAmKJQ5jjDFBscRhjDEmKJY4jDHGBMUS\nhzHGmKBY4jDGGBMUSxzGGGOCYonDGGNMUCxxGGOMCYolDmOMMUGxxGGMMSYoljiMMcYExRKHMcaY\noHiaOERkuIisFpEsEZlQzvxoEXnTnT9fRNq67eeKSIaILHN/nh2wzFfuOpe4j+ZevgZjjDHHivBq\nxSISDvwbOBfIBRaKyHRVXRHQ7QZgt6q2F5HRwMPAlcAO4GJV3SwipwMzgPiA5a5WVRtE3BhjfODl\nHkcfIEtVs1W1EJgKjCzTZyTwijv9NjBURERVF6vqZrc9E6gjItEexmqMMaaCvEwc8cCmgOe5HLvX\ncEwfVS0C9gKxZfpcBixS1SMBbS+5h6nuFRGp3LCNMcb8kpA+OS4iXXAOX/02oPlqVe0KDHQf1xxn\n2fEiki4i6fn5+d4Ha4wxtYSXiSMPSAh43sZtK7ePiEQAjYCd7vM2wHvAtaq6rnQBVc1zf+4HXsc5\nJPYzqjpJVdNUNS0uLq5SXpAxxhhvE8dCoIOIJItIFDAamF6mz3RgnDs9CpilqioijYGPgQmq+l1p\nZxGJEJFm7nQkcBGw3MPXYIwxpgzPEod7zuIPOFdErQSmqWqmiEwUkRFutxeAWBHJAm4HSi/Z/QPQ\nHvhrmctuo4EZIrIUWIKzx/KcV6/BGGPMz4mq+h2D59LS0jQ93a7eNcaYYIhIhqqmlW0P6ZPjxhhj\nQo8lDmOMMUGxxGGMMSYoljiMMcYExRKHMcaYoFjiMMYYExRLHMYYY4JiicMYY0xQLHEYY4wJiiUO\nY4wxQbHEYYwxJiiWOIwxxgTFEocxxpigWOIwxhgTFEscxhhjgmKJwxhjTFAscRhjjAmKJQ5jjDFB\nscRhjDEmKJ4mDhEZLiKrRSRLRCaUMz9aRN50588XkbYB8+5221eLyHkVXacxxhhveZY4RCQc+Ddw\nPtAZGCMinct0uwHYrartgceAh91lOwOjgS7AcOBpEQmv4DqNMcZ4yMs9jj5Alqpmq2ohMBUYWabP\nSOAVd/ptYKiIiNs+VVWPqOp6IMtdX0XWaYwxxkMRHq47HtgU8DwX6Hu8PqpaJCJ7gVi3fV6ZZePd\n6ROtEwARGQ+Md58eEJHVJ/EaAJoBO05yWS9ZXMGxuIJjcQWnpsaVVF6jl4nDV6o6CZh0qusRkXRV\nTauEkCqVxRUciys4FldwaltcXh6qygMSAp63cdvK7SMiEUAjYOcvLFuRdRpjjPGQl4ljIdBBRJJF\nJArnZPf0Mn2mA+Pc6VHALFVVt320e9VVMtABWFDBdRpjjPGQZ4eq3HMWfwBmAOHAi6qaKSITgXRV\nnQ68ALwmIlnALpxEgNtvGrACKAJ+r6rFAOWt06vX4Drlw10esbiCY3EFx+IKTq2KS5wv+MYYY0zF\n2J3jxhhjgmKJwxhjTFBqdeIQkQQRmS0iK0QkU0Ruc9ubisgXIrLW/dnEbRcRedItd7JURHp5FFeM\niCwQkR/cuB5w25Pd0ixZbqmWKLf9uKVbPIovXEQWi8hHIRbXBhFZJiJLRCTdbfP1d+luq7GIvC0i\nq0RkpYj09zsuEenovk+lj30i8ie/43K39f/cz/1yEXnD/Xvw/TMmIre5MWWKyJ/ctip/v0TkRRHZ\nLiLLA9qCjkNExrn914rIuPK2dVyqWmsfQCuglzvdAFiDU8rkEWCC2z4BeNidvgD4FBCgHzDfo7gE\nqO9ORwLz3e1NA0a77f8BfudO3wL8x50eDbzp8ft2O/A68JH7PFTi2gA0K9Pm6+/S3dYrwI3udBTQ\nOBTiCogvHNiKc7OX35/9eGA9UCfgs3Wd358x4HRgOVAX56KimUB7P94vYBDQC1h+sp9zoCmQ7f5s\n4k43qXAMXn8oq9MD+AA4F1gNtHLbWgGr3elngTEB/X/s52FMdYFFOHfI7wAi3Pb+wAx3egbQ352O\ncPuJR/G0Ab4EzgY+cj+QvsflbmMDP08cvv4uce5NWl/2dfsdV5lYhgHfhUJc/FRNoqn7mfkIOM/v\nzxhwOfBCwPN7gf/y6/0C2nJs4ggqDmAM8GxA+zH9TvSo1YeqArm7uD1xvt23UNUt7qytQAt3urwy\nKvF4wD0ctATYDnwBrAP2qGpROds+pnQLUFq6xQuP4/zBlLjPY0MkLgAFPheRDHFKzoD/v8tkIB94\nyT2897yI1AuBuAKNBt5wp32NS1XzgH8AG4EtOJ+ZDPz/jC0HBopIrIjUxfkmn0Do/B6DjeOU4rPE\nAYhIfeAd4E+qui9wnjrpuMqvWVbVYlXtgfMNvw/QqapjKEtELgK2q2qG37Ecx5mq2gunevLvRWRQ\n4EyffpcROIcVnlHVnsBBnEMJfscFgHuuYATwVtl5fsTlHpsfiZNwWwP1cCpk+0pVV+JU7/4c+AxY\nAhSX6ePb77Gq46j1iUNEInGSxhRVfddt3iYirdz5rXC+9YMPJU9UdQ8wG2f3vLE4pVnKbvt4pVsq\n2wBghIhswKlMfDbwRAjEBfz4bRVV3Q68h5Nw/f5d5gK5qjrfff42TiLxO65S5wOLVHWb+9zvuM4B\n1qtqvqoeBd7F+dz5/hlT1RdUNVVVBwG7cc6J+v1+lQo2jlOKr1YnDhERnLvXV6rqPwNmBZZCGYdz\n7qO0/Vr3SoV+wN6A3cPKjCtORBq703VwzrusxEkgo44TV3mlWyqVqt6tqm1UtS3O4Y1Zqnq133EB\niEg9EWlQOo1z3H45Pv8uVXUrsElEOrpNQ3EqIvgaV4Ax/HSYqnT7fsa1EegnInXdv8/S9ysUPmPN\n3Z+JwK9xLhDx+/0qFWwcM4BhItLE3csb5rZVTGWfRKpOD+BMnF26pTi7nktwjl3G4pwAXotz9URT\nt7/gDCS1DlgGpHkUVzdgsRvXcuCvbnsKTs2uLJxDC9Fue4z7PMudn1IF791gfrqqyve43Bh+cB+Z\nwD1uu6+/S3dbPYB09/f5Ps5VLKEQVz2cb+eNAtpCIa4HgFXuZ/81IDpEPmPf4CSxH4Chfr1fOIl+\nC3AUZ4/2hpOJA/iN+75lAdcHE4OVHDHGGBOUWn2oyhhjTPAscRhjjAmKJQ5jjDFBscRhjDEmKJY4\njDHGBMUSh6nxxKmAfF6Ztj+JyDOVtP7rROSpIJd5WURGnbhn0LF0FJGvxKl4u1JEJrntaSLyZGVv\nz9ROng0da0wIeQPnhsXAG5xG49TcqhZEJEJ/qtX0S54EHlPVD9zlugKoajrOvSTGnDLb4zC1wdvA\nhfLTGA5tceogfSMiY8QZw2O5iDxcuoCIDBeRReKMifKl29ZHRL53ixXODbgbHCDB/aa/VkTuK92O\nHDtmwh0icn/Z4ETkryKy0I1hknvHNO76HhdnbJF7RGS9WyIHEWkY+DxAK5ybwgBQ1WVu/8Hy0/gp\nn8hP43DsFWdchnAR+T83jqUi8tuTe6tNbWCJw9R4qroL567i892m0TjjO7TCKVx3Ns7d3b1F5BIR\niQOeAy5T1e44JbXBuZt5oDrFCv8KPBiwmT7AZTh3/V8uImlBhPiUqvZW1dOBOsBFAfOiVDVNVR8A\nvgIuDHgN76pTzynQY8AsEflUnAGRGpfzflygTgHNG4AcnLvZb8ApR9Eb6A3cJCLJQbwGU4tY4jC1\nRenhKvipjHhv4Ct1CuoVAVNwBsnpB8xR1fXwY+IBp4DeW+5exGNAl4D1f6GqO1X1EE5hvjODiG2I\nOKPXLcNJYoHrfTNg+nngenf6euClsitS1ZeA03DKcAwG5olIdNl+ItIMp5zHVaq6F6dW0bXilPKf\nj1PCokMQr8HUIpY4TG3xATBUnKEz6+rJlYb/H2C2u2dwMU6dpFJla/coUMSxf2MxZfogIjHA08Ao\nVe2Ks6cT2O/gjytU/Q5oKyKDgXBVXU45VHWzqr6oqiPdGE4vs81wnOrGEwPWIcCtqtrDfSSr6ufl\nrd8YSxymVlDVAzgVVl/kp2qwC4CzRKSZ+890DPA1MA8YVHqoRkSauv0b8VPp6evKbOJcccZ9rgNc\nAnwHbAOaizP4TzTHHoIqVZokdogzLsyJrrR6Facq68/2NtxYhwecB2mJs+dQtlz2Q8BSVZ0a0DYD\n+F3Asr8Sp8qwMT9jV1WZ2uQNnHE6RgOo6hYRmYCTUAT4OOBqpPHAuyIShjO2wbk44zq/IiJ/AT4u\ns+4FOOO6tAEmu1cxISIT3Xl5OOdIjqGqe0TkOZxKsFuBhSd4DVOAv3FsKfRAw4AnROSw+/xOVd0q\nIoEDgd0BZLqHpcA5X/M8znCki9yT8/k4CdCYn7HquMZUI+69HyNV9Rq/YzG1l+1xGFNNiMi/cK4M\nu8DvWEztZnscxhhjgmInx40xxgTFEocxxpigWOIwxhgTFEscxhhjgmKJwxhjTFD+PykDKBTbrYYR\nAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DuVY-p-ESjvA",
        "colab_type": "text"
      },
      "source": [
        "## Question 5: Compare an n-gram feature space to a word embedding feature space.\n",
        "\n",
        "This part compares bigram and trigram feature spaces with a word embedding feature space with Linear SVM classifier. The results show that the performance of Linear SVM classifier is much better with a word embedding featuer space than the bigram and trigram feature space. While the Kappa for embedding feature space is 0.185, it is only 0.126 in bigram feature space and 0.005 in trigram feature space. Therefore, it seems that the embedding feature space increases the performance of the model with a Linear SVM classifier. However, the result is not consistent every time. Sometimes the performance with a bigram feature space is the best among all three."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iQ6rgP26S5_t",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 381
        },
        "outputId": "1f1ca492-f0e8-4dc0-d03d-b9be089fa696"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en_core_web_md', disable=[\"parser\", \"ner\"])\n",
        "\n",
        "# Tokenize Twitter data\n",
        "def tokenize(nlp, row):\n",
        "  text = row[\"text\"]\n",
        "  tokens = nlp(text)\n",
        "  return tokens\n",
        "\n",
        "all_tagged = twitter_training.apply(lambda x: tokenize(nlp, x), axis=1)\n",
        "\n",
        "# Prepare Embeddings as a DataFrame\n",
        "dimensions = 300\n",
        "X_dict = {\n",
        "    f\"D{i}\":[] for i in range(dimensions)\n",
        "}\n",
        "for row in all_tagged:\n",
        "    vector = row.vector\n",
        "    for i in range(len(vector)):\n",
        "        key = f\"D{i}\"\n",
        "        X_dict[key].append(vector[i])\n",
        "\n",
        "embedding_df = pd.DataFrame(X_dict)\n",
        "embedding_df[\"text\"] = [x.text for x in all_tagged]\n",
        "embedding_df[\"labels_A\"] = twitter_training[\"labels_A\"].values"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "OSError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-14-f087c8631e67>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mspacy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mnlp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mspacy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'en_core_web_md'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdisable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"parser\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ner\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# Tokenize Twitter data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnlp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/spacy/__init__.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(name, **overrides)\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mdepr_path\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m         \u001b[0mdeprecation_warning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mWarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW001\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdepr_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mutil\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moverrides\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     31\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/spacy/util.py\u001b[0m in \u001b[0;36mload_model\u001b[0;34m(name, **overrides)\u001b[0m\n\u001b[1;32m    167\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"exists\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Path or Path-like to model data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mload_model_from_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moverrides\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 169\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mIOError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mErrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mE050\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    170\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mOSError\u001b[0m: [E050] Can't find model 'en_core_web_md'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory."
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zcHpEOrVS-dn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick Classifiers to Compare\n",
        "classifiers = {\"Linear SVM\": LinearSVC()}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "# Compare models and display final result\n",
        "\n",
        "# Bigram feature space\n",
        "feature_set = list(bigram_df.columns[:-1])\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bigram_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "print(f\"Bigram feature space Kappa: K={best:.3f}.\")\n",
        "\n",
        "# Trigram feature space\n",
        "feature_set = list(trigram_df.columns[:-1])\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, trigram_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet')\n",
        "print(f\"Trigram feature space Kappa: K={best:.3f}.\")\n",
        "\n",
        "# Embedding feature space\n",
        "feature_set = X_dict.keys()\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, embedding_df, feature_set, \"labels_A\", labels=sentiments, noisy = 'quiet', folds=10)    \n",
        "print(f\"Embedding feature space Kappa: K={best:.3f}.\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbGH-C2P7Irb",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion \n",
        "\n",
        "The best performing model in Task 2 is Linear SVM classifier on a unigram feature space with a Kappa of 0.243. The setting of the unigram feature space has a 1000 vocabulary size bag-of-words presentation and the classifier was tested using 10-fold classification. \n",
        "\n",
        "Although the results vary from time to time, I ran all of my optimization codes several times and this model is indeed the best performer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOOkWEaIdEWu",
        "colab_type": "text"
      },
      "source": [
        "# Task 3\n",
        "\n",
        "Train two models, each **trained on the full 80% training set, and tested on the held-out 20% test set**:\n",
        "   - A Naïve Bayes classifier with unigram features.\n",
        "   - The best-tuned model from task two, retrained on the full 80% training set. \n",
        "   \n",
        "Report three sets of evaluation metrics:\n",
        "   - The estimated performance from cross-validation in Task 2.\n",
        "   - The performance of the simple Naïve Bayes unigram classifier on the held-out test set.\n",
        "   - The performance of the best-tuned model on the held-out test set.\n",
        "\n",
        "At minimum, your evaluations should include percent accuracy and Kappa values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2egNUeysnsmm",
        "colab_type": "text"
      },
      "source": [
        "## Model 1: A Naïve Bayes classifier with unigram features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGH2ZB88oWsE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "vocab_size = 400\n",
        "\n",
        "# Build a unigram model - capturing individual words only - for the training set\n",
        "vectorizer = CountVectorizer(max_features=vocab_size)\n",
        "X_train = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "# Make a DataFrame with BOW model representations for each message\n",
        "bow_train = pd.DataFrame(X_train.toarray())\n",
        "column_names_train = [str(i) for i in range(vocab_size)]\n",
        "\n",
        "# Make the column names the words\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  column_names_train[v] = k\n",
        "bow_train.columns = column_names_train\n",
        "\n",
        "# Add our Y labels to our DataFrame\n",
        "bow_train[\"labels_A\"] = twitter_training[\"labels_A\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gcQrQsWeDE2f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Build a unigram model - capturing individual words only - for the test set\n",
        "vectorizer = CountVectorizer(max_features=vocab_size)\n",
        "X_test = vectorizer.fit_transform(twitter_test[\"text\"])\n",
        "\n",
        "# Make a DataFrame with BOW model representations for each message\n",
        "bow_test = pd.DataFrame(X_test.toarray())\n",
        "column_names_test = [str(i) for i in range(vocab_size)]\n",
        "\n",
        "# Make the column names the words\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  column_names_test[v] = k\n",
        "bow_test.columns = column_names_test\n",
        "\n",
        "# Add our Y labels to our DataFrame\n",
        "bow_test[\"labels_A\"] = twitter_test[\"labels_A\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBHwBrUI7UBw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick features to use\n",
        "bow_train_features = column_names_train\n",
        "feature_train_set = bow_train_features\n",
        "bow_test_features = column_names_test\n",
        "feature_test_set = bow_test_features\n",
        "\n",
        "# Set up dataset\n",
        "X_train = bow_train.loc[:, feature_train_set]\n",
        "y_train = bow_train['labels_A']\n",
        "X_test = bow_test.loc[:, feature_test_set]\n",
        "y_test = bow_test['labels_A']\n",
        "\n",
        "# Fit the model\n",
        "model = BernoulliNB().fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")\n",
        "\n",
        "# Calculate and display Kappa on the train and test sets\n",
        "train_kappa = cohen_kappa_score(y_train, train_pred)\n",
        "test_kappa = cohen_kappa_score(y_test, test_pred)\n",
        "print(f\"Kappa on the train set: {train_kappa:.3f}\")\n",
        "print(f\"Kappa on the test set: {test_kappa:.3f}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yWw6GSYn1Y7R",
        "colab_type": "text"
      },
      "source": [
        "## Model 2: The best-tuned model from task two, retrained on the full 80% training set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hc6yXfev1XbJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick features to use\n",
        "bow_train_features = column_names_train\n",
        "feature_train_set = bow_train_features\n",
        "bow_test_features = column_names_test\n",
        "feature_test_set = bow_test_features\n",
        "\n",
        "# Set up dataset\n",
        "X_train = bow_train.loc[:, feature_train_set]\n",
        "y_train = bow_train['labels_A']\n",
        "X_test = bow_test.loc[:, feature_test_set]\n",
        "y_test = bow_test['labels_A']\n",
        "\n",
        "# Fit the model\n",
        "model = LinearSVC().fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")\n",
        "\n",
        "# Calculate and display Kappa on the train and test sets\n",
        "train_kappa = cohen_kappa_score(y_train, train_pred)\n",
        "test_kappa = cohen_kappa_score(y_test, test_pred)\n",
        "print(f\"Kappa on the train set: {train_kappa:.3f}\")\n",
        "print(f\"Kappa on the test set: {test_kappa:.3f}\")\n",
        "\n",
        "# Prepare for calculating statistical differences with labels_B\n",
        "labels_A_y = y_test\n",
        "labels_A_y_pred = test_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SL8aid8x-1ZX",
        "colab_type": "text"
      },
      "source": [
        "## Task 3 Report\n",
        "\n",
        "1. The best estimated performance from cross-validation in Task 2 I got was Linear SVM classifier on a unigram feature space with a Kappa of 0.243. Other better models I got in Task 2 had Kappa values ranging from 0.14 to 0.2.\n",
        "\n",
        "2. The performance of the simple Naïve Bayes unigram classifier on the held-out test set has a 37.5% accuracy and a -0.004 Kappa. \n",
        "\n",
        "3. The performance of the best-tuned model in task 2 on the held-out test set, which is the Linear SVM classifier on a unigram feature space, has a 37.5% accuracy and a -0.04 Kappa. \n",
        "\n",
        "I was surprised to find that the Kappa score dropped dramatically on the held-out test set and even reached negative. Compared to a former 0.243 Kappa during the cross validation, the results on the test set were way worse and showed that the model didn't have any predictative ability. The reason I could think of was that our sample is so small. Also, we extracted our twitter posts from the dataset very randomly, so our train and held out sets may have nothing in common in terms of topics and therefore the model is bad at generalization.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-nzeiXvm05T",
        "colab_type": "text"
      },
      "source": [
        "# Extra Credit\n",
        "\n",
        "For up to 3 points of extra credit:\n",
        "   - Perform five of the optimizations above instead of three.\n",
        "\n",
        "For up to 3 points of extra credit:\n",
        "   - Re-run the same set of code above but on `labels_B` instead of `labels_A`. Choose a new set of optimized hyperparameters and features based on these results. How do the chosen models differ, and are the model quality metrics different between the two columns in a statistically significant way?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Raudnlb7Bx5u",
        "colab_type": "text"
      },
      "source": [
        "## Rerun the same optimization code on labels_B"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RDwnQCtCmO8",
        "colab_type": "text"
      },
      "source": [
        "**Compare Naïve Bayes, Logistic Regression, and SVMs on a unigram feature space.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOweJRme_npu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "vocab_size = 1000\n",
        "\n",
        "# By default we can build a unigram model - capturing individual words only\n",
        "vectorizer = CountVectorizer(max_features=vocab_size)\n",
        "X = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "# Now lets make a DataFrame with our BOW model representations for each message\n",
        "bow_df = pd.DataFrame(X.toarray())\n",
        "column_names = [str(i) for i in range(vocab_size)]\n",
        "\n",
        "# Make the column names the words\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  column_names[v] = k\n",
        "bow_df.columns = column_names\n",
        "\n",
        "# Add our Y labels to our DataFrame\n",
        "bow_df[\"labels_B\"] = twitter_training[\"labels_B\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eCXEyue__v5-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick Classifiers to Compare\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Logistic Regression\": LogisticRegression(),\n",
        "    \"Linear SVM\": LinearSVC()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "bow_features = column_names\n",
        "feature_set = bow_features\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bow_df, feature_set, \"labels_B\", labels=sentiments, noisy = 'quiet', folds=10)\n",
        "\n",
        "print(f\"Best classifier is: {best_name} \\nWith K={best:.3f}.\")    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DCUmxpDmCqkb",
        "colab_type": "text"
      },
      "source": [
        "**Compare unigram feature space with a feature space that includes longer N-grams.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5vSzJ2f0_y_i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def ngrams(df, vocab_size = 1000, min_n=1, max_n=1):\n",
        "  vectorizer = CountVectorizer(max_features=vocab_size, ngram_range=(1,max_n))\n",
        "  X = vectorizer.fit_transform(df['text'])\n",
        "\n",
        "  bow_df = pd.DataFrame(X.toarray())\n",
        "  column_names = [str(i) for i in range(vocab_size)]\n",
        "  for k, v in vectorizer.vocabulary_.items():\n",
        "    column_names[v] = k\n",
        "  bow_df.columns = column_names\n",
        "  return column_names, bow_df\n",
        "\n",
        "# Unigrams DataFrame\n",
        "unigram_names, unigram_df = ngrams(twitter_training, max_n=1)\n",
        "unigram_df[\"labels_B\"] = twitter_training[\"labels_B\"].values\n",
        "\n",
        "# DataFrame of Bigrams\n",
        "bigram_names, bigram_df = ngrams(twitter_training, vocab_size = 3000, max_n=2)\n",
        "bigram_df[\"labels_B\"] = twitter_training[\"labels_B\"].values\n",
        "\n",
        "# DataFrame of Trigrams\n",
        "trigram_names, trigram_df = ngrams(twitter_training, vocab_size = 5000, max_n=3)\n",
        "trigram_df[\"labels_B\"] = twitter_training[\"labels_B\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A9A69eFS_2t0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### Setup classifiers and metrics to be used on all n-grams\n",
        "# Pick Classifiers to Compare\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Complement NB\": ComplementNB(), \n",
        "    \"Multinomial NB\": MultinomialNB()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "### Compare classifiers on unigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(unigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, unigram_df, feature_set, \"labels_B\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best unigram classifier is: {best_name} \\nWith K={best:.3f}.\")    \n",
        "\n",
        "### Compare classifiers on bigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(bigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, bigram_df, feature_set, \"labels_B\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best bigram classifier is: {best_name} \\nWith K={best:.3f}.\")    \n",
        "\n",
        "### Compare classifiers on trigrams ###\n",
        "# Pick features to use\n",
        "feature_set = list(trigram_df.columns[:-1])\n",
        "\n",
        "# Compare models and display final result\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, trigram_df, feature_set, \"labels_B\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Best trigram classifier is: {best_name} \\nWith K={best:.3f}.\")    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEPm6KF9Cvox",
        "colab_type": "text"
      },
      "source": [
        "**Compare a unigram feature space with a feature space that removes stopwords.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wU9yn5aE_6hI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract features for unigrams with no stopwords included.\n",
        "vectorizer = CountVectorizer(max_features=1000, ngram_range=(1,1), stop_words='english')\n",
        "X = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "no_stopwords_df = pd.DataFrame(X.toarray())\n",
        "no_stopwords_columns = [str(i) for i in range(1000)]\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  no_stopwords_columns[v] = k\n",
        "no_stopwords_df.columns = no_stopwords_columns\n",
        "no_stopwords_df[\"labels_B\"] = twitter_training[\"labels_B\"].values\n",
        "\n",
        "classifiers = {\n",
        "    \"Bernoulli NB\": BernoulliNB(),\n",
        "    \"Complement NB\": ComplementNB(), \n",
        "    \"Multinomial NB\": MultinomialNB()\n",
        "}\n",
        "\n",
        "# Set a list of metrics we want to use to compare our classifiers \n",
        "metrics = {\n",
        "    \"Accuracy\" : lambda y,y_pred: 100*accuracy_score(y,y_pred),\n",
        "    \"Kappa\"    : cohen_kappa_score\n",
        "}\n",
        "\n",
        "# Choose a metric to optimize over\n",
        "metric_to_optimize = 'Kappa'\n",
        "\n",
        "# Pick features to use\n",
        "sentiments = [\"Positive\", \"Negative\", \"Neutral\"]\n",
        "\n",
        "# Re-run our classifier with stopwords included, as a baseline.\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, unigram_df, unigram_names, \"labels_B\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Classifier with stopwords K={best:.3f}.\")    \n",
        "\n",
        "# Then run our classifier on the feature space with stopwords removed.\n",
        "best, best_name, classifier_comparison = compare_classifiers(classifiers, metrics, metric_to_optimize, no_stopwords_df, no_stopwords_columns, \"labels_B\", labels=sentiments, noisy = 'quiet')\n",
        "\n",
        "print(f\"Classifier without stopwords K={best:.3f}.\")   "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGNAyZm-CHia",
        "colab_type": "text"
      },
      "source": [
        "## Choose a new best set of optimized hyperparameters and features: Multinomial NB with a unigram feature space that removes stopwords.\n",
        "\n",
        "The performance of the best-tuned model for label B on the held-out test set has a 32.5% accuracy and a -0.061 Kappa. The difference between the performance on predicting label B and label A is not statistically significant with p = 0.66 > 0.1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "doRFsFUhD6R-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract features for the train set unigrams with no stopwords included.\n",
        "vectorizer = CountVectorizer(max_features=300, ngram_range=(1,1), stop_words='english')\n",
        "X_train = vectorizer.fit_transform(twitter_training[\"text\"])\n",
        "\n",
        "no_stopwords_train = pd.DataFrame(X_train.toarray())\n",
        "no_stopwords_train_columns = [str(i) for i in range(300)]\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  no_stopwords_train_columns[v] = k\n",
        "no_stopwords_train.columns = no_stopwords_train_columns\n",
        "no_stopwords_train[\"labels_B\"] = twitter_training[\"labels_B\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NAcMy-4gMQuN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract features for the test unigrams with no stopwords included.\n",
        "vectorizer = CountVectorizer(max_features=300, ngram_range=(1,1), stop_words='english')\n",
        "X_test = vectorizer.fit_transform(twitter_test[\"text\"])\n",
        "\n",
        "no_stopwords_test = pd.DataFrame(X_test.toarray())\n",
        "no_stopwords_test_columns = [str(i) for i in range(300)]\n",
        "for k, v in vectorizer.vocabulary_.items():\n",
        "  no_stopwords_test_columns[v] = k\n",
        "no_stopwords_test.columns = no_stopwords_test_columns\n",
        "no_stopwords_test[\"labels_B\"] = twitter_test[\"labels_B\"].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CGXpWVVlDt4Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick features to use\n",
        "feature_train_set = no_stopwords_train_columns\n",
        "feature_test_set = no_stopwords_test_columns\n",
        "\n",
        "# Set up dataset\n",
        "X_train = no_stopwords_train.loc[:, feature_train_set]\n",
        "y_train = no_stopwords_train['labels_B']\n",
        "X_test = no_stopwords_test.loc[:, feature_test_set]\n",
        "y_test = no_stopwords_test['labels_B']\n",
        "\n",
        "model = MultinomialNB().fit(X_train, y_train)\n",
        "train_pred = model.predict(X_train)\n",
        "test_pred = model.predict(X_test)\n",
        "\n",
        "# Calculate and display accuracy on the train and test sets\n",
        "train_accuracy = 100*accuracy_score(y_train, train_pred)\n",
        "test_accuracy = 100*accuracy_score(y_test, test_pred)\n",
        "print(f\"Accuracy on the train set: {train_accuracy:.1f}\")\n",
        "print(f\"Accuracy on the test set: {test_accuracy:.1f}\")\n",
        "\n",
        "# Calculate and display Kappa on the train and test sets\n",
        "train_kappa = cohen_kappa_score(y_train, train_pred)\n",
        "test_kappa = cohen_kappa_score(y_test, test_pred)\n",
        "print(f\"Kappa on the train set: {train_kappa:.3f}\")\n",
        "print(f\"Kappa on the test set: {test_kappa:.3f}\")\n",
        "\n",
        "labels_B_y = y_test\n",
        "labels_B_y_pred = test_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVGGhUiPe3a8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.stats import ttest_rel\n",
        "labels_A_y_matches = (labels_A_y == labels_A_y_pred)\n",
        "labels_A_y_matches = labels_A_y_matches.apply(int)\n",
        "labels_B_y_matches = (labels_B_y == labels_B_y_pred)\n",
        "labels_B_y_matches = labels_B_y_matches.apply(int)\n",
        "\n",
        "t, p = ttest_rel(labels_A_y_matches, labels_B_y_matches)\n",
        "print(f\"T-test results: t={t:.3f}, p={p:.3f}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_2upx-ygRKE-",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# Scoring Rubric\n",
        "![](https://drive.google.com/uc?export=view&id=1JqI8Tfmi3YrnjVdDxjwNu1ZOuOOgQCuI)\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1VfVuKGmNBu6oJgXBTX4YB6Lxe_0t9cWN)"
      ]
    }
  ]
}