14_working_with_text_data/05_sentiment_analysis_twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification and sentiment analysis: Twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once text data has been converted into numerical features using the natural language processing techniques discussed in the previous sections, text classification works just like any other classification task.\n",
    "\n",
    "In this notebook, we will apply these preprocessing technique to news articles, product reviews, and Twitter data and teach various classifiers to predict discrete news categories, review scores, and sentiment polarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:48.348039Z",
     "start_time": "2020-06-21T00:42:48.344849Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:49.188022Z",
     "start_time": "2020-06-21T00:42:48.362034Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# spacy, textblob and nltk for language processing\n",
    "from textblob import TextBlob\n",
    "\n",
    "# sklearn for feature extraction & modeling\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score, roc_curve, accuracy_score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:49.206503Z",
     "start_time": "2020-06-21T00:42:49.189954Z"
    }
   },
   "outputs": [],
   "source": [
    "sns.set_style('white')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Twitter Sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a dataset that contains 1.6 million training and 350 test tweets from 2009 with algorithmically assigned binary positive and negative sentiment scores that are fairly evenly split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow the [instructions](../data/twitter_sentiment.ipynb) to create the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 0 - the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive); training data has no neutral tweets\n",
    "- 1 - the id of the tweet (2087)\n",
    "- 2 - the date of the tweet (Sat May 16 23:58:44 UTC 2009)\n",
    "- 3 - the query (lyx). If there is no query, then this value is NO_QUERY. (only test data uses query)\n",
    "- 4 - the user that tweeted (robotickilldozr)\n",
    "- 5 - the text of the tweet (Lyx is cool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read and preprocess train/test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:49.216045Z",
     "start_time": "2020-06-21T00:42:49.208663Z"
    }
   },
   "outputs": [],
   "source": [
    "data_path = Path('..', 'data', 'sentiment140')\n",
    "if not data_path.exists():\n",
    "    data_path.mkdir(parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:49.224549Z",
     "start_time": "2020-06-21T00:42:49.217690Z"
    }
   },
   "outputs": [],
   "source": [
    "names = ['polarity', 'id', 'date', 'query', 'user', 'text']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a few preprocessing steps:\n",
    "- remove tweets above the legal (at the time) length of 140 characters,\n",
    "- binarize polarity, and \n",
    "- move the data to the faster parquet format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:49.232438Z",
     "start_time": "2020-06-21T00:42:49.225444Z"
    }
   },
   "outputs": [],
   "source": [
    "def load_train_data():\n",
    "    parquet_file = data_path / 'train.parquet'\n",
    "    if not parquet_file.exists():\n",
    "        df = (pd.read_csv(data_path / 'train.csv',\n",
    "                          low_memory=False,\n",
    "                          encoding='latin1',\n",
    "                          header=None,\n",
    "                          names=names,\n",
    "                          parse_dates=['date'])\n",
    "              .drop(['id', 'query'], axis=1)\n",
    "              .drop_duplicates(subset=['polarity', 'text']))\n",
    "        df = df[df.text.str.len() <= 140]\n",
    "        df.polarity = (df.polarity > 0).astype(int)\n",
    "        df.to_parquet(parquet_file)\n",
    "        return df\n",
    "    else:\n",
    "        return pd.read_parquet(parquet_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:50.926809Z",
     "start_time": "2020-06-21T00:42:49.233284Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1566668 entries, 0 to 1599999\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count    Dtype         \n",
      "---  ------    --------------    -----         \n",
      " 0   polarity  1566668 non-null  int64         \n",
      " 1   date      1566668 non-null  datetime64[ns]\n",
      " 2   user      1566668 non-null  object        \n",
      " 3   text      1566668 non-null  object        \n",
      "dtypes: datetime64[ns](1), int64(1), object(2)\n",
      "memory usage: 59.8+ MB\n"
     ]
    }
   ],
   "source": [
    "train = load_train_data()\n",
    "train.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:50.931373Z",
     "start_time": "2020-06-21T00:42:50.927606Z"
    }
   },
   "outputs": [],
   "source": [
    "def load_test_data():\n",
    "    parquet_file = data_path / 'test.parquet'\n",
    "    if not parquet_file.exists():\n",
    "        df = (pd.read_csv('data/sentiment140/test.csv',\n",
    "                          low_memory=False,\n",
    "                          encoding='latin1',\n",
    "                          header=None,\n",
    "                          names=names,\n",
    "                          parse_dates=['date'])\n",
    "              .drop(['id', 'query'], axis=1)\n",
    "              .drop_duplicates(subset=['polarity', 'text']))\n",
    "        df = df[(df.text.str.len() <= 140) &\n",
    "                (df.polarity.isin([0, 4]))]\n",
    "        df.to_parquet(parquet_file)\n",
    "        return df\n",
    "    else:\n",
    "        return pd.read_parquet(parquet_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:51.013369Z",
     "start_time": "2020-06-21T00:42:50.932868Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 354 entries, 0 to 497\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype              \n",
      "---  ------    --------------  -----              \n",
      " 0   polarity  354 non-null    int64              \n",
      " 1   date      354 non-null    datetime64[ns, UTC]\n",
      " 2   user      354 non-null    object             \n",
      " 3   text      354 non-null    object             \n",
      "dtypes: datetime64[ns, UTC](1), int64(1), object(2)\n",
      "memory usage: 13.8+ KB\n"
     ]
    }
   ],
   "source": [
    "test = load_test_data()\n",
    "test.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:51.031897Z",
     "start_time": "2020-06-21T00:42:51.015643Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>polarity</th>\n",
       "      <th>date</th>\n",
       "      <th>user</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2009-04-06 22:19:45</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2009-04-06 22:19:49</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2009-04-06 22:19:53</td>\n",
       "      <td>mattycus</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>2009-04-06 22:19:57</td>\n",
       "      <td>ElleCTF</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2009-04-06 22:19:57</td>\n",
       "      <td>Karoli</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   polarity                date             user  \\\n",
       "0         0 2009-04-06 22:19:45  _TheSpecialOne_   \n",
       "1         0 2009-04-06 22:19:49    scotthamilton   \n",
       "2         0 2009-04-06 22:19:53         mattycus   \n",
       "3         0 2009-04-06 22:19:57          ElleCTF   \n",
       "4         0 2009-04-06 22:19:57           Karoli   \n",
       "\n",
       "                                                text  \n",
       "0  @switchfoot http://twitpic.com/2y1zl - Awww, t...  \n",
       "1  is upset that he can't update his Facebook by ...  \n",
       "2  @Kenichan I dived many times for the ball. Man...  \n",
       "3    my whole body feels itchy and like its on fire   \n",
       "4  @nationwideclass no, it's not behaving at all....  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:51.097846Z",
     "start_time": "2020-06-21T00:42:51.033755Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    784335\n",
       "0    782333\n",
       "Name: polarity, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.polarity = (train.polarity>0).astype(int)\n",
    "train.polarity.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:51.121466Z",
     "start_time": "2020-06-21T00:42:51.103598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    180\n",
       "0    174\n",
       "Name: polarity, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.polarity = (test.polarity>0).astype(int)\n",
    "test.polarity.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:52.174735Z",
     "start_time": "2020-06-21T00:42:51.123083Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEECAYAAAAh5uNxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAaG0lEQVR4nO3df2zV1f3H8dft7Q/x3luROeMyvIYqN8GZ0kKHbl6KJI66RDdCoHKvuc7onDAB29iuCJRKBLHZ2m9EU3+FZVlr29Uf2S9nNiWErqOSpbOgVyuxmaCOKb8m917g3tJ+vn8Y76j9PW+59/Y8H8lNuOeetu9D2vu6n/P5nPOxWZZlCQBgnIxkFwAASA4CAAAMRQAAgKEIAAAwFAEAAIZKmwC45557kl0CAEwpaRMAJ0+eTHYJADClpE0AAAASiwAAAEMRAABgKAIAAAxFAACAoQgAADAUAQAAhiIAAMBQBAAAGCoz2QUAgGk+Ox1TKHpuSLsrJ1OXXJx9weogAADgAgtFz6n94LEh7cWeyy5oADAFBACGIgAAwFAEAAAYigAAAEMRAABgKAIAAAxFAACAoQgAADAUAQAAhiIAAMBQBAAAGIoAAABDEQAAYCgCAAAMRQAAgKEIAAAwFAEAAIYa1x3Bli5dKpfLJUmaOXOmVq1apfXr18tms2n27NmqqalRRkaG2tra1NraqszMTK1evVqLFy/W2bNnVVlZqePHj8vhcKi2tlYzZsxQd3e3tm3bJrvdLq/XqzVr1kzqQAEAg40ZANFoVJLU2NgYb1u1apXKysp0/fXXa/Pmzdq1a5cKCgrU2Niol156SdFoVH6/XzfeeKNaWlrk8Xi0du1avfLKK2poaNCmTZtUU1OjJ554QldeeaV+8pOfKBgM6lvf+tbkjRQAMMiYU0A9PT06c+aM7r77bt15553q7u5WMBjUggULJEnFxcXau3evDhw4oMLCQmVnZ8vlcsntdqunp0ddXV1auHBhvG9nZ6fC4bBisZjcbrdsNpu8Xq86Ozsnd6QAgEHGPAK46KKLdM8992jFihX64IMPdO+998qyLNlsNkmSw+FQKBRSOByOTxN90R4Ohwe1n9/X6XQO6vvhhx8memwAgFGMGQCzZs3SVVddJZvNplmzZmn69OkKBoPx1yORiHJzc+V0OhWJRAa1u1yuQe2j9c3NzU3kuHABfHY6plD03JB2V06mLrk4OwkVAZiIMaeAXnzxRT322GOSpE8++UThcFg33nij9u3bJ0lqb29XUVGR8vPz1dXVpWg0qlAopN7eXnk8Hs2bN0979uyJ950/f76cTqeysrJ0+PBhWZaljo4OFRUVTeIwMRlC0XNqP3hsyGO4UACQesY8Ali+fLkeeugh+Xw+2Ww2Pfroo7r00ktVXV2t+vp65eXlqaSkRHa7XYFAQH6/X5Zlqby8XDk5OfL5fKqqqpLP51NWVpbq6uokSVu2bFFFRYX6+/vl9Xo1d+7cSR8s/jcjfdKP9vUn5PtwxAAkh82yLCvZRYzHsmXL9PLLLye7DCN9dPK02g8eG9Je6J6uNw//Z0h7secyzbz04nF/n5H6A1NVqvwtjGsdADCZzvUP6KOTp4e0c2QATC4CAEl3pm9Ab/aeGNJe7LmMAAAmEVtBAIChCAAAMBQBAACG4hyAgdL9csx0rx9IFQSAgb5YwPVl6XLSNd3rB1IFU0AAYCgCAAAMRQAAgKE4BzBFjXSiVJr4Hj7pghXFwMQQAFPUSCdKpc/38JmKWFEMTAxTQABgKAIAAAxFAACAoQgAADAUAQAAhiIAAMBQXAYKY422VoK1AzABAQBjjbZWgrUDMAFTQABgKI4AgGGwrQRMQAAAw2BbCZiAKSAAMBQBAACGYgooTXAfXACJRgCkCe6DCyDRmAICAEMRAABgqHEFwPHjx7Vo0SL19vbq0KFD8vl88vv9qqmp0cDAgCSpra1Ny5YtU2lpqXbv3i1JOnv2rNauXSu/3697771XJ058flldd3e3VqxYoZUrV+rJJ5+cpKEBAEYzZgD09fVp8+bNuuiiiyRJ27dvV1lZmZqbm2VZlnbt2qWjR4+qsbFRra2t2rlzp+rr6xWLxdTS0iKPx6Pm5mYtXbpUDQ0NkqSamhrV1dWppaVF+/fvVzAYnNxRAgCGGDMAamtrtXLlSl1++eWSpGAwqAULFkiSiouLtXfvXh04cECFhYXKzs6Wy+WS2+1WT0+Purq6tHDhwnjfzs5OhcNhxWIxud1u2Ww2eb1edXZ2TuIQAQDDGTUAXn75Zc2YMSP+Ji5JlmXJZrNJkhwOh0KhkMLhsFwuV7yPw+FQOBwe1H5+X6fTOahvKBRK6KCAC+2z0zF9dPL0kMdnp2PJLg0Y0aiXgb700kuy2Wzq7OzUu+++q6qqqvg8viRFIhHl5ubK6XQqEokMane5XIPaR+ubm5ub6HEBFxSX6SIdjXoE8Pzzz6upqUmNjY2aM2eOamtrVVxcrH379kmS2tvbVVRUpPz8fHV1dSkajSoUCqm3t1cej0fz5s3Tnj174n3nz58vp9OprKwsHT58WJZlqaOjQ0VFRZM/UgDAIBNeCFZVVaXq6mrV19crLy9PJSUlstvtCgQC8vv9sixL5eXlysnJkc/nU1VVlXw+n7KyslRXVydJ2rJliyoqKtTf3y+v16u5c+cmfGAAgNGNOwAaGxvj/25qahryemlpqUpLSwe1TZs2TTt27BjSt6CgQG1tbROpEwCQYGwFAUwi7iuAVEYApJiRNn2L9vUnoRp8VdxXAKmMAEgxI11NUuienoRqAExl7AUEAIbiCAAAJkmqT+kSAAAwSVJ9SpcAAJKAq4OQCggAIAm4OgipgJPAAGAoAgAADEUAAIChCAAAMBQBAACG4iogIA2MtKCIy0bxVRAAQBrgjmNmuNDrQwgAAEgRF3p9COcAAMBQBAAAGIoAAABDcQ4ASGNsKoevggAA0hibyuGrIACQcCN9Kk2Vm2AA+BwBgIQb6VNpqtwEA8DnCACkLI4kgMlFACAu1d5wOZJIPLaUwPkIAMTxhjv1saUEzsc6AAAwFAEAAIZiCgjAiOd/JM4PTGVjBkB/f782bdqkf/7zn7Lb7dq+fbssy9L69etls9k0e/Zs1dTUKCMjQ21tbWptbVVmZqZWr16txYsX6+zZs6qsrNTx48flcDhUW1urGTNmqLu7W9u2bZPdbpfX69WaNWsuxHgBDGOk8z8S5wemsjGngHbv3i1Jam1t1bp167R9+3Zt375dZWVlam5ulmVZ2rVrl44eParGxka1trZq586dqq+vVywWU0tLizwej5qbm7V06VI1NDRIkmpqalRXV6eWlhbt379fwWBwckcKABhkzAC4+eab9cgjj0iS/vWvf+myyy5TMBjUggULJEnFxcXau3evDhw4oMLCQmVnZ8vlcsntdqunp0ddXV1auHBhvG9nZ6fC4bBisZjcbrdsNpu8Xq86OzsncZgAgC8b1zmAzMxMVVVV6bXXXtOOHTu0e/du2Ww2SZLD4VAoFFI4HJbL5Yp/jcPhUDgcHtR+fl+n0zmo74cffpjIcQFxqba+AUgV4z4JXFtbq4qKCpWWlioajcbbI5GIcnNz5XQ6FYlEBrW7XK5B7aP1zc3NTcR4Ug4Lb5KP9Q3A8MacAvrtb3+rZ555RpI0bdo02Ww2XXfdddq3b58kqb29XUVFRcrPz1dXV5ei0ahCoZB6e3vl8Xg0b9487dmzJ953/vz5cjqdysrK0uHDh2VZljo6OlRUVDSJw0yeLxbefPkxXCgA6eSz0zF9dPL0kMdnp2PJLg3jNOYRwJIlS/TQQw/pjjvu0Llz57RhwwZdffXVqq6uVn19vfLy8lRSUiK73a5AICC/3y/LslReXq6cnBz5fD5VVVXJ5/MpKytLdXV1kqQtW7aooqJC/f398nq9mjt37qQPdipiegPJwqri9DdmAFx88cV6/PHHh7Q3NTUNaSstLVVpaemgtmnTpmnHjh1D+hYUFKitrW0itWIYTG8A+F+xEAxAQnGXsvRBAABIqKl6l7KRLuiQ0jfcCAAAGIeRznlI6RtubAYHAIbiCACYAK66wlRCAAATwFVXicdiyeQhAAAkVaqtJxgpkKbiUR4BACAlJety0pECaSoe5REAAFLSRC8nHemTe2aGdG5g6PdniokAADBFjPbJ/c3D/xnSnq6XbiYSl4ECgKEIAAAwFAEAAIYiAADAUAQAABiKAAAAQxEAAGAo1gEAKYTN5nAhEQBACmGzOVxITAEBgKEIAAAwFFNAScJcL4BkIwCShLleAMnGFBAAGIojgAQx6S5CAKYGAiBBTLqLEICpgQAApiAuMsB4EADAFMRFBhgPAgAAvqJ0PeIaNQD6+vq0YcMGffzxx4rFYlq9erWuueYarV+/XjabTbNnz1ZNTY0yMjLU1tam1tZWZWZmavXq1Vq8eLHOnj2ryspKHT9+XA6HQ7W1tZoxY4a6u7u1bds22e12eb1erVmz5kKNFwASLl2PuEa9DPT3v/+9pk+frubmZj333HN65JFHtH37dpWVlam5uVmWZWnXrl06evSoGhsb1draqp07d6q+vl6xWEwtLS3yeDxqbm7W0qVL1dDQIEmqqalRXV2dWlpatH//fgWDwQsyWADAf40aALfccoseeOCB+HO73a5gMKgFCxZIkoqLi7V3714dOHBAhYWFys7OlsvlktvtVk9Pj7q6urRw4cJ4387OToXDYcViMbndbtlsNnm9XnV2dk7iEAEAwxk1ABwOh5xOp8LhsNatW6eysjJZliWbzRZ/PRQKKRwOy+VyDfq6cDg8qP38vk6nc1DfUCg0GWMDAIxizJXAR44c0Z133qkf/vCHuu2225SR8d8viUQiys3NldPpVCQSGdTucrkGtY/WNzc3N5FjApBAX5zg/PIj1U9wYmyjBsCxY8d09913q7KyUsuXL5ckXXvttdq3b58kqb29XUVFRcrPz1dXV5ei0ahCoZB6e3vl8Xg0b9487dmzJ953/vz5cjqdysrK0uHDh2VZljo6OlRUVDTJwwTwvzrTN6D2g8eGPGL9VrJLw1c06lVATz/9tE6dOqWGhob4CdyNGzdq69atqq+vV15enkpKSmS32xUIBOT3+2VZlsrLy5WTkyOfz6eqqir5fD5lZWWprq5OkrRlyxZVVFSov79fXq9Xc+fOnfyRAgAGGTUANm3apE2bNg1pb2pqGtJWWlqq0tLSQW3Tpk3Tjh07hvQtKChQW1vbRGsFACQQC8EAXBDpulhqKiMAAFwQ6bpYairjfgAAYCiOAACkFaaSEocAAJBWEjWVRJAQAAAMxTkJzgEAgLEIAAAwFAEAAIYiAADAUAQAABiKAAAAQxEAAGAoAgAADEUAAIChCAAAMBQBAACGIgAAwFAEAAAYigAAAEMRAABgKAIAAAxFAACAoQgAADAUAQAAhiIAAMBQBAAAGIoAAABDEQAAYKhxBcD+/fsVCAQkSYcOHZLP55Pf71dNTY0GBgYkSW1tbVq2bJlKS0u1e/duSdLZs2e1du1a+f1+3XvvvTpx4oQkqbu7WytWrNDKlSv15JNPTsa4AABjGDMAnnvuOW3atEnRaFSStH37dpWVlam5uVmWZWnXrl06evSoGhsb1draqp07d6q+vl6xWEwtLS3yeDxqbm7W0qVL1dDQIEmqqalRXV2dWlpatH//fgWDwckdZQJ9djqmj06eHvKI9vUnuzQAmJAxA8DtduuJJ56IPw8Gg1qwYIEkqbi4WHv37tWBAwdUWFio7OxsuVwuud1u9fT0qKurSwsXLoz37ezsVDgcViwWk9vtls1mk9frVWdn5yQNL/FC0XNqP3hsyCPWbyW7NACYkMyxOpSUlOijjz6KP7csSzabTZLkcDgUCoUUDoflcrnifRwOh8Lh8KD28/s6nc5BfT/88MOEDShRPjsdUyh6bkg7n/QBTBVjBsCXZWT896AhEokoNzdXTqdTkUhkULvL5RrUPlrf3NzcrzKGSfHFJ/0vK3RPT0I1AJB4E74K6Nprr9W+ffskSe3t7SoqKlJ+fr66uroUjUYVCoXU29srj8ejefPmac+ePfG+8+fPl9PpVFZWlg4fPizLstTR0aGioqLEjgoAMKYJHwFUVVWpurpa9fX1ysvLU0lJiex2uwKBgPx+vyzLUnl5uXJycuTz+VRVVSWfz6esrCzV1dVJkrZs2aKKigr19/fL6/Vq7ty5CR8YAGB04wqAmTNnqq2tTZI0a9YsNTU1DelTWlqq0tLSQW3Tpk3Tjh07hvQtKCiIfz8AQHKwEAwADEUAAIChCAAAMBQBAACGIgAAwFAEAAAYigAAAEMRAABgKAIAAAxFAACAoQgAADAUAQAAhiIAAMBQBAAAGIoAAABDEQAAYKgJ3xFsquHm7wBMZXwAcPN3AKZiCggADEUAAIChCAAAMBQBAACGIgAAwFAEAAAYigAAAEMRAABgKAIAAAxFAACAoQgAADBU0vYCGhgY0MMPP6z33ntP2dnZ2rp1q6666qpJ+3ls+gYAgyUtAF5//XXFYjH95je/UXd3tx577DE99dRTk/bz2PQNAAZL2hRQV1eXFi5cKEkqKCjQ22+/naxSAMBINsuyrGT84I0bN2rJkiVatGiRJOmmm27S66+/rszM4Q9Krr/+en3zm9+8kCUCQNq79NJLtXPnzmFfS9oUkNPpVCQSiT8fGBgY8c1fkvbt23chygIAYyRtCmjevHlqb2+XJHV3d8vj8SSrFAAwUtKmgL64CujgwYOyLEuPPvqorr766mSUAgBGSloAAACSi4VgAGAoAgAADEUAAICh0iIABgYGtHnzZt1+++0KBAI6dOhQsksaUV9fnyorK+X3+7V8+XLt2rVLhw4dks/nk9/vV01NjQYGBpJd5rCOHz+uRYsWqbe3N21qfuaZZ3T77bdr2bJleuGFF1K+7r6+Pj344INauXKl/H5/Wvxf79+/X4FAQJJGrLWtrU3Lli1TaWmpdu/encxyJQ2u+d1335Xf71cgENA999yjY8c+3xEg1WqWBtf9hT/84Q+6/fbb488TWreVBv785z9bVVVVlmVZ1ptvvmmtWrUqyRWN7MUXX7S2bt1qWZZlnThxwlq0aJF13333WW+88YZlWZZVXV1t/eUvf0lmicOKxWLWT3/6U2vJkiXW+++/nxY1v/HGG9Z9991n9ff3W+Fw2NqxY0fK1/3aa69Z69atsyzLsjo6Oqw1a9akdM3PPvusdeutt1orVqywLMsattZPP/3UuvXWW61oNGqdOnUq/u9UqfmOO+6w3nnnHcuyLKulpcV69NFHU65myxpat2VZ1jvvvGPdeeed8bZE150WRwDptG3ELbfcogceeCD+3G63KxgMasGCBZKk4uJi7d27N1nljai2tlYrV67U5ZdfLklpUXNHR4c8Ho/uv/9+rVq1SjfddFPK1z1r1iz19/drYGBA4XBYmZmZKV2z2+3WE088EX8+XK0HDhxQYWGhsrOz5XK55Ha71dPTk6ySh9RcX1+vOXPmSJL6+/uVk5OTcjVLQ+s+efKkfvGLX2jDhg3xtkTXnRYBEA6H5XQ648/tdrvOnRu6s2cqcDgccjqdCofDWrduncrKymRZlmw2W/z1UCiU5CoHe/nllzVjxox4yEpK+Zqlz/9A3n77bT3++OPasmWLKioqUr7uiy++WB9//LG+//3vq7q6WoFAIKVrLikpGbRCf7haw+GwXC5XvI/D4VA4HL7gtX7hyzV/8aHmH//4h5qamnTXXXelXM3S4Lr7+/u1ceNGbdiwQQ6HI94n0XUnbSuIiZjothHJduTIEd1///3y+/267bbb9POf/zz+WiQSUW5ubhKrG+qll16SzWZTZ2en3n33XVVVVenEiRPx11OxZkmaPn268vLylJ2drby8POXk5Ojf//53/PVUrPtXv/qVvF6vHnzwQR05ckQ/+tGP1NfXF389FWs+X0bGfz8zflHrl/8+I5HIoDepVPCnP/1JTz31lJ599lnNmDEj5WsOBoM6dOiQHn74YUWjUb3//vvatm2bbrjhhoTWnRZHAOm0bcSxY8d09913q7KyUsuXL5ckXXvttfG9jNrb21VUVJTMEod4/vnn1dTUpMbGRs2ZM0e1tbUqLi5O6Zolaf78+frrX/8qy7L0ySef6MyZM/rOd76T0nXn5ubG/2AvueQSnTt3LuV/P843XK35+fnq6upSNBpVKBRSb29vSv2N/u53v4v/fl955ZWSlPI15+fn65VXXlFjY6Pq6+t1zTXXaOPGjQmvO3U/Rp/ne9/7nv72t79p5cqV8W0jUtXTTz+tU6dOqaGhQQ0NDZI+3/l069atqq+vV15enkpKSpJc5diqqqpUXV2d0jUvXrxYf//737V8+XJZlqXNmzdr5syZKV33XXfdpQ0bNsjv96uvr0/l5eW67rrrUrrm8w33e2G32xUIBOT3+2VZlsrLy5WTk5PsUiV9PpWybds2feMb39DatWslSd/+9re1bt26lK15NF//+tcTWjdbQQCAodJiCggAkHgEAAAYigAAAEMRAABgKAIAAAxFAADjEI1G9cILL0zoa1577TV98sknk1QR8NURAMA4HD16dMIB8Otf/zrp2wsAoyEAgHF4+umn9f777+vJJ5+MLyIKBAJ67733dOTIES1ZskSffvqpOjs75ff79frrr8e31YjFYskuHxhWWqwEBpJt1apVOnjwoM6cOaMbbrhBfr9fH3zwgR566CG1tLSosrJS69ev17Fjx/Tss8/qiiuu0Jw5c/Twww8rOzs72eUDwyIAgAk4ePCg3njjDb366quSpFOnTkmSbr75Zv3f//2fvvvd7+qKK65IZonAuBEAwDhkZGRoYGBAeXl5+sEPfqDbbrtNx48fj58X+OUvf6kbb7xRb731lrq7u1VQUCCbzSZ2WkEq4xwAMA5f+9rX1NfXp0gkoldffVWBQEA//vGPNXv2bL311lv64x//qMrKSm3btk0bNmxQKBRSYWGhfvazn+k///lPsssHhsVmcABgKI4AAMBQBAAAGIoAAABDEQAAYCgCAAAMRQAAgKEIAAAw1P8D6Z+et4miunwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(train.text.str.len(), kde=False)\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:52.367559Z",
     "start_time": "2020-06-21T00:42:52.175614Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                 1566668\n",
       "unique                 765666\n",
       "top       2009-06-15 12:53:14\n",
       "freq                       20\n",
       "first     2009-04-06 22:19:45\n",
       "last      2009-06-25 10:28:31\n",
       "Name: date, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.date.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:52.817105Z",
     "start_time": "2020-06-21T00:42:52.370055Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "650606"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.user.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:42:53.591563Z",
     "start_time": "2020-06-21T00:42:52.832364Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "lost_dog           549\n",
       "webwoke            341\n",
       "SallytheShizzle    276\n",
       "VioletsCRUK        275\n",
       "mcraddictal        274\n",
       "tsarnick           247\n",
       "what_bugs_u        246\n",
       "Karen230683        237\n",
       "DarkPiano          232\n",
       "SongoftheOss       226\n",
       "Name: user, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.user.value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create text vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a document-term matrix with 934 tokens as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.651633Z",
     "start_time": "2020-06-21T00:42:53.596830Z"
    }
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(min_df=.001, max_df=.8, stop_words='english')\n",
    "train_dtm = vectorizer.fit_transform(train.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.655414Z",
     "start_time": "2020-06-21T00:43:11.652745Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1566668x934 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 6332930 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.669425Z",
     "start_time": "2020-06-21T00:43:11.656403Z"
    }
   },
   "outputs": [],
   "source": [
    "test_dtm = vectorizer.transform(test.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.897040Z",
     "start_time": "2020-06-21T00:43:11.671003Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB()"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb = MultinomialNB()\n",
    "nb.fit(train_dtm, train.polarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict Test Polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.910282Z",
     "start_time": "2020-06-21T00:43:11.898724Z"
    }
   },
   "outputs": [],
   "source": [
    "predicted_polarity = nb.predict(test_dtm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.928292Z",
     "start_time": "2020-06-21T00:43:11.912307Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7768361581920904"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(test.polarity, predicted_polarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextBlob for Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.978158Z",
     "start_time": "2020-06-21T00:43:11.929636Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ok its cake and ice cream time! Ha! See what I'm talking about! The temptation is there! \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_positive = train.text.loc[256332]\n",
    "print(sample_positive)\n",
    "parsed_positive = TextBlob(sample_positive)\n",
    "parsed_positive.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:11.987157Z",
     "start_time": "2020-06-21T00:43:11.980272Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " i hate this place\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-0.8"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_negative = train.text.loc[636079]\n",
    "print(sample_negative)\n",
    "parsed_negative = TextBlob(sample_negative)\n",
    "parsed_negative.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.000962Z",
     "start_time": "2020-06-21T00:43:11.998282Z"
    }
   },
   "outputs": [],
   "source": [
    "def estimate_polarity(text):\n",
    "    return TextBlob(text).sentiment.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.122410Z",
     "start_time": "2020-06-21T00:43:12.001988Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>392473</th>\n",
       "      <td>No one will speak to me on this  Seems useless...</td>\n",
       "      <td>-0.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>492394</th>\n",
       "      <td>Fuck pacsun for not having any smalls in anyth...</td>\n",
       "      <td>-0.4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>613144</th>\n",
       "      <td>I'm absolutely JOYFUL that Shahid Afridi made ...</td>\n",
       "      <td>-0.1500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1189887</th>\n",
       "      <td>I tell you something, I think you'll understan...</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1001446</th>\n",
       "      <td>OWWW! Hurt myself. Keno ftw! Taking her up on ...</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92768</th>\n",
       "      <td>@AlexaNDYE Yup, didn't manage to dodge a 12 ho...</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>902549</th>\n",
       "      <td>finished watching the movie 'mirrors'. I liked...</td>\n",
       "      <td>0.2875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1578586</th>\n",
       "      <td>The BBC (R4) will 'Keep in touch with Demotix ...</td>\n",
       "      <td>0.4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1441092</th>\n",
       "      <td>Well everyone, I'm going to bed, mighty night ...</td>\n",
       "      <td>0.4000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>409953</th>\n",
       "      <td>@2kutekreations Nope... no chocolate.    i rea...</td>\n",
       "      <td>0.4750</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      text  sentiment\n",
       "392473   No one will speak to me on this  Seems useless...    -0.5000\n",
       "492394   Fuck pacsun for not having any smalls in anyth...    -0.4000\n",
       "613144   I'm absolutely JOYFUL that Shahid Afridi made ...    -0.1500\n",
       "1189887  I tell you something, I think you'll understan...     0.0000\n",
       "1001446  OWWW! Hurt myself. Keno ftw! Taking her up on ...     0.0000\n",
       "92768    @AlexaNDYE Yup, didn't manage to dodge a 12 ho...     0.0000\n",
       "902549   finished watching the movie 'mirrors'. I liked...     0.2875\n",
       "1578586  The BBC (R4) will 'Keep in touch with Demotix ...     0.4000\n",
       "1441092  Well everyone, I'm going to bed, mighty night ...     0.4000\n",
       "409953   @2kutekreations Nope... no chocolate.    i rea...     0.4750"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['text']].sample(10).assign(sentiment=lambda x: x.text.apply(estimate_polarity)).sort_values('sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare with TextBlob Polarity Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also obtain TextBlob sentiment scores for the tweets and note (see left panel in below figure) that positive test tweets receive a significantly higher sentiment estimate. We then use the MultinomialNB ‘s model .predict_proba() method to compute predicted probabilities and compare both models using the respective Area Under the Curve (see right panel below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.346372Z",
     "start_time": "2020-06-21T00:43:12.123460Z"
    }
   },
   "outputs": [],
   "source": [
    "test['sentiment'] = test.text.apply(estimate_polarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.355982Z",
     "start_time": "2020-06-21T00:43:12.348343Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7429378531073446"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(test.polarity, (test.sentiment>0).astype(int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ROC AUC Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.372365Z",
     "start_time": "2020-06-21T00:43:12.357859Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8254948914431672"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_true=test.polarity, y_score=test.sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.378394Z",
     "start_time": "2020-06-21T00:43:12.373345Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.848595146871009"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_true=test.polarity, y_score=nb.predict_proba(test_dtm)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.400769Z",
     "start_time": "2020-06-21T00:43:12.388229Z"
    }
   },
   "outputs": [],
   "source": [
    "fpr_tb, tpr_tb, _ = roc_curve(y_true=test.polarity, y_score=test.sentiment)\n",
    "roc_tb = pd.Series(tpr_tb, index=fpr_tb)\n",
    "fpr_nb, tpr_nb, _ = roc_curve(y_true=test.polarity, y_score=nb.predict_proba(test_dtm)[:, 1])\n",
    "roc_nb = pd.Series(tpr_nb, index=fpr_nb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Naive Bayes model outperforms TextBlob in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T00:43:12.751843Z",
     "start_time": "2020-06-21T00:43:12.401797Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAGoCAYAAADVZM+hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeXxM9/7H8dfMZM+E2HcRia1VjaW1VKiUUkrtW4uW0uqmy68uxb1BRFq01C33tr1UW8UtsVa1lKK2ElJLK3YaVdIkyD6ZzPz+yO20qS1IZpJ4Px8Pj8ec5Xu+75mqM5/5fs85BrvdbkdEREREREREXMro6gAiIiIiIiIiogJdREREREREpEhQgS4iIiIiIiJSBKhAFxERERERESkCVKCLiIiIiIiIFAEq0EVERERERESKADdXBxApziIiIti9ezcAx48fp1q1anh5eQGwZMkSx+v82L9/P0uXLmXSpEnEx8fToUMH6tatC4DNZsPLy4sxY8bQtGlTdu3axeTJk1mzZs11j1mvXj127NhB2bJlr7vfhQsXiIyM5Pjx4wB4eXnxzDPP0L59+3znv977OXDgAB988AHvvvvuLR8vv4YOHcr06dOv+p6jo6P59NNPsVqt5OTkEBISwpgxY/Dz8yv0XCIicmerV68edevWxWg0YjAYyMjIwGw2Ex4ezj333ANAeno6s2fPZuPGjXh4eAAQFhbGyJEj83ynWL58OYsXLyYzM5Ps7GyaNm3K66+/TqlSpa7a983uLyKuowJd5DaMHz/e8TosLIzp06c7TrI369ixY5w/f96x7OXlxcqVKx3La9euZezYsXz99de3Hvgaxo8fT6tWrZg5c6Yjy4ABAwgMDCQoKOiWjvnn93PPPfc4pTgH2LZt21XX79+/n/fee49ly5bh7+9PTk4OEydOJDw8nBkzZjglm4iI3NkWLFiQ5wfk//znP0RERLBkyRKsVitPPfUUISEhrFixAm9vbzIyMpgxYwbDhg1jwYIFuLm58a9//YstW7bw3nvvUb58ebKzs4mMjOTZZ5/ls88+u6LPm91fRFxLBbpIIfn8889ZtGgRNpsNf39/JkyYQGBgIE899RR33303o0ePZvv27YwZM4ZFixbx7rvvkpKSwtixY3n++eevON7FixepUKHCFetTUlKYOHEihw8fxmAwEBoayquvvoqbW+7/3jNnzuTAgQPYbDZefvll2rVrd8UxEhISyMzMxGazYTQaCQ4OZu7cuY5f1o8fP86UKVO4ePEiOTk5DBo0iN69e7Nr1y7eeecdatSowdGjR7FarUycOJGqVavmeT/du3d3jPiPGTMGLy8vjhw5QmJiImFhYfj7+7Np0yYSEhKIiIigZcuWWCwWpk+fzu7du8nJyeGuu+5i/PjxmM1mwsLC6NGjBzt27ODcuXM89thjvPzyy4wdOxaAIUOG8P7771OlSpU879Fut5OZmQmAyWRi1KhRHD16FACr1cq0adP49ttvMZlMNG7cmH/84x8YDAaioqLYsWMHJpOJRo0aMXbsWEeORo0aERcXx6uvvkqjRo2YNGkS586dIzs7my5duvDss89itVqZPHkye/fuxd3dnerVqzN16lR8fX1v82+ZiIgUV1arlXPnzlG6dGkA1q1bh81mc5zLALy9vRk3bhzdu3dn/fr1tG3bln//+98sX76c8uXLA+Du7s7o0aNZv349FovFMfIOuSPyN9r/3//+N8nJyfz9738HYPbs2Y7lQYMGUbp0aU6cOEG/fv2YM2cOW7duxcPDg5ycHB588EE++ugjKlasyJQpUzhy5AjZ2dm0bNmS0aNH4+bmxrvvvsv69etxd3enTJkyTJ06lYoVKzrrYxYpdnQNukgh+P7771mxYgULFy5kxYoVPP3007zwwgsYjUamTZvGypUr2bBhA2PGjGHGjBlUq1aNl156iWbNmjF16lQAMjMzeeyxx3jsscdo164dkZGRjBgx4oq+IiIi8Pf3Z/Xq1Sxbtoy4uDjmzZvn2F69enWWL1/OtGnTGDNmDElJSVccY/To0Xz66ae0bNmSkSNH8uGHH1KjRg0qVKiA1WrlpZde4rXXXnNMEZ83bx6xsbFA7sj00KFDWbFiBT179uSdd96hSpUqV7yfP/vxxx9ZsGCB41g+Pj4sXryYwYMH88EHHwDw/vvvYzKZiI6OZtWqVVSsWJHp06c7jpGens5nn33G4sWLmTdvHj///LOjrwULFuQpzgHatGlD48aNHcX971PvmzdvDsBnn33GoUOHWLlyJWvWrCEtLY21a9cyd+5cLly4wMqVK1m5ciU2m4233nrLcdw6derw5Zdf0qFDB15//XV69epFdHQ0S5cuZfv27axdu5bY2Fi+//57Vq1aRXR0NDVq1CAuLi5/f5lERKTEGDJkCF27dqV169Z07NgRwHHu2rdvH82aNbuijcFgoGXLlsTExHDixAm8vLyoVatWnn28vb3p1q1bnuIcuOn9r6ZUqVKsXbuWIUOGUKdOHTZu3AjAd999R/Xq1QkKCiIyMpK7776b6OhoVqxYQXJyMvPnz+fcuXMsWLCAZcuWER0dzQMPPMD+/fvz+3GJ3JE0gi5SCL799ltOnz5N//79HesuX77MxYsXqVixIpMnT+a5557jxRdf5L777rvqMf46xX379u08//zzrFq1Ks9+W7ZsYdGiRRgMBjw8POjfvz8LFixwFPMDBgwAoG7dugQFBbFv3z4eeuihPMdo2bIl3377LbGxsezZs4dNmzbx3nvvsWDBAnx8fDhz5gxvvPGGY//MzEx+/PFHgoKCqFq1Kg0aNADgrrvuYvny5Tf8fNq1a4e7uzsVKlTAx8eH0NBQAGrWrMnFixcdn2FKSgrbt28HIDs7m3LlyjmO8ft7qFSpEuXKlePSpUvUqFHjmn26u7szY8YMRo8eza5du9i9ezd/+9vfaNmyJTNnzmT79u089thjjmv8fp/u37t3b1555RXc3d0BGDRoUJ4ZDr9/mUpPT2f37t1cunSJWbNmOdYdPnyY1q1bYzKZ6NOnj+NLWaNGjW74OYmISMny+xT3Q4cOMWLECJo3b57n3Ga1Wq/azmKxYDKZMBqN2Gy2fPd3s/tfzZ9/NOjduzfLly+nU6dOREdH07dvXyD3nH3gwAGWLl0K4JitVqlSJerXr0+PHj1o06YNbdq0oWXLlreVR6SkU4EuUghsNhuPPfYYr7/+umP5woULjmlsx44do3z58jf1K3KrVq2oWbMmBw4cyHMyt9lsGAyGPMt/PsEbjcY8236f+v67xMREZs+ezYQJE2jWrBnNmjXj2WefZdy4caxYsYJ+/frh5+eX58eC3377DT8/P2JjY/PctMZgMGC322/4Xv76i/1fM/2e9Y033qBt27YApKWlkZWV5dju6el5U/0uXbqUMmXK8NBDD9GtWze6devGyJEjCQsLIykp6YoMv/32Gzab7aqfb3Z2tmPZx8fHsd5ut7N48WK8vb0BSEpKwtPTE19fX1auXMnevXvZuXMnL7/8MsOGDePxxx+/bmYRESmZ7r77bsaOHcuYMWNo0KAB1atXp0mTJnz44YeOy81+Z7PZ2L17NyNHjiQ4OBir1cqpU6fyjIpnZWXxwgsvEBERQaVKlRzr87P/X8+hfz7HwR/nOYBHHnmEqKgojh8/zu7du4mKinJknDVrluO+NZcvX8ZgMGA0Gvn00085cOAAO3bsIDIyktDQUEaPHl0gn6NISaQp7iKFoHXr1nzxxRdcuHABgEWLFjFkyBAgd0r4xx9/zLJly0hJSWHBggVA7jXR1/rlHODkyZOcPXvWMVr9574+/fRT7HY7FouF//73v7Rq1cqx/fcR7UOHDnHmzBnuvffePO1Lly7N9u3b+fjjjx0n6IyMDM6cOcNdd91FYGBgntH8c+fO8eijj3Lw4MHrfgY3ej830rp1axYuXIjFYsFmszFhwgTefvvtG7a7Vr9Go5Hp06fz66+/OtYdPXqUqlWrUrp0aVq2bMmaNWsc/YWHh/PFF18QGhrKokWLyM7OxmazsXDhQh544IErjm82mwkJCWH+/PlA7peTAQMG8M0337Bp0yaefPJJGjduzIsvvkj37t1v+PmJiEjJ9uijj9KoUSPHFPeOHTvi7e1NZGSkYwQ6MzOTyZMn4+vrS4cOHfDw8GD48OGMGzeO3377DcgdXY+MjCQjIyNPcQ7ka/8yZcpw6NAh7HY7qampbNq06ZqZPT096dKlC2PGjOHhhx92/CDdunVrPvroI8d3kZEjR/Lpp59y+PBhHn30UYKCgnjmmWd48sknOXDgQIF/liIliUbQRQpB69atGT58OEOHDsVgMGA2m/nnP/9JWloar776KuPHj6dSpUpERUXRp08f7rvvPkJCQnjvvfd44YUXGDNmjOMa9N/ZbDYmTZpEYGCgo/CH3DuwR0RE0LVrV7KzswkNDeXZZ591bP/555/p3r07BoOBt99+G39//zxZ3dzc+M9//sO0adP45JNP8PHxwWAw0KNHD3r37g3AnDlzmDJlCh9++CFWq5VRo0Y5Hvd2LX9+P4MGDbrpz/C5557jzTffpEePHuTk5NCgQQPGjBlzw3adOnVi0KBBzJ492/GYOoCePXuSkZHB8OHDsVgsGAwGatWqxX/+8x9MJhP9+/fn7Nmz9OzZE7vdzv3338+gQYOwWq28+eabdO/eHavVSqNGjZgwYcJV+54+fTqTJ0+ma9euWCwWHn30Ubp160ZOTg5btmzh0UcfxcfHh9KlSzN58uSb/kxERKRkmTBhAt26dWPr1q2EhoYyb9485syZQ8+ePTEajeTk5BAWFsa8efMcl1o9++yzeHt7M2zYMCB3NPz+++9nzpw5V+3jRvv/3v/DDz9MpUqVuP/++687K61Pnz58+umnhIeHO9aNGzeOKVOmOL6LtGrViqeffhp3d3ceeeQRevXqhY+PD15eXnmegCMiVzLY8zMfVUREREREREQKlaa4i4iIiIiIiBQBKtBFREREREREigAV6CIiIiIiIiJFgAp0ERERERERkSKgRBbov9+lUkRERFxP52UREZH8KZEFenJysqsjiIiIyP/ovCwiIpI/JbJAFxERERERESluVKCLiIiIiIiIFAEq0EVERERERESKADdXBxAREZE7T3Z2NvHx8WRmZro6Sonn5eVF9erVcXd3d3UUERG5ARXoIiIi4nTx8fH4+flRq1YtDAaDq+OUWHa7ncTEROLj4wkMDHR1HBERuQFNcRcRERGny8zMpFy5cirOC5nBYKBcuXKaqSAiUkyoQBcRERGXUHHuHPqcRUSKDxXoIiIiIiIiIkWACnQRERG54+zatYtmzZpx7tw5x7rp06cTHR19zTbR0dF88803t9xnWFgYjz/+OE888QQ9e/Zk4cKFt3wsEREpmXSTOBEREbkjubu7M3bsWObPn5+vaeA9e/a87T7nzZuHp6cnFouFzp0706lTJ8qVK3fbxxURkZJBBbqIiIjckVq0aIHNZmPhwoU88cQTebbNmDGDgwcPkpaWRlBQEFOnTmX27NmUL1+eU6dOUb9+fXr06EFCQgLPPPMM0dHRzJgxg927d2O323nyySd55JFHrtl3ZmYmnp6e+Pn5kZqayrhx40hJSSE5OZk+ffrQtWtXevTowVdffYXJZGLatGk0bNiQ2rVrExERAYC/vz+RkZFkZ2fz8ssvY7fbyc7OZuLEidSrV69QPzsRESkcKtClyDl27BijRo1i1qxZBAcHuzqOiIg4wcPvbObI+dQCO17dSma+fqXtDfcLDw+nT58+tG7d2rEuNTWVUqVKMX/+fGw2G126dOH8+fOO7X379mXixIn06NGDlStX0rNnTzZv3kx8fDyLFy8mKyuLvn378sADD1CqVKk8/Q0dOhSDwcCJEydo37497u7uHD16lC5duvDwww9z/vx5Bg0axMCBA2natCnfffcdrVu3ZsuWLYwaNYonnniCyMhIgoOD+fzzz/nwww9p3Lgxfn5+zJgxg2PHjpGaWnCfo4iIOJfTC/QffviB6dOn88knn+RZv3HjRt577z3c3Nzo1asXffv2xWazER4eTlxcHB4eHkRERBAQEODsyOJkERERpKWlERERwUcffeTqOCIid6SbOV8XhPwU04WhTJkyvPHGG4wZM4YmTZoA4OnpSVJSEq+++io+Pj6kp6eTnZ3taBMUFEROTg5nz55l7dq1fPTRRyxZsoRDhw4xaNAgAKxWK7/88ssVBfqfp7iPGDGCVatW0aJFCxYsWMDXX3+N2WzGarUC0KdPHz755BNsNhutWrXCw8OD48ePM3HiRACys7MJDAykTZs2nDp1iueeew43NzdGjhzpjI9OREQKgVML9A8++IBVq1bh7e2dZ312djZTp05l6dKleHt7M2DAANq1a8e+ffuwWCwsWbKE2NhYoqKimDt3rjMji5MdO3aMU6dOAXDq1CmOHTumUXQRESe72fN1hQoVXJS0YISFhbF+/XqWL1/O66+/zpYtWzh37hwzZ84kKSmJ9evXY7fb87Tp3bs306ZNIzg4mFKlSlG7dm2aN2/O5MmTsdlszJkzh+rVq1+zTw8PD8qVK0d2djbz5s0jJCSEgQMHsnPnTjZv3gxAs2bNiIyMZOnSpbz88ssABAYG8uabb1K1alViYmJISEhg165dVKxYkXnz5rFv3z7efvvtK35YERGR4sGpBXrNmjWZPXs2o0ePzrP++PHj1KxZk9KlSwPQtGlT9uzZQ2xsLKGhoQCEhIRw8OBBZ8YVF/j9uro/L2sUXUqCr776irVr17o6xk1LTk4GckcZi5POnTvTsWNHV8cotm72fH29a62Li3HjxrFz504AGjVqxJw5c+jbty8eHh7UqFGDCxcu5Nm/U6dOTJkyxTFwEBYWxvfff8/AgQNJT0+nffv2mM3mK/oZOnQoRqMRm81G5cqV6datG3v37iU8PJzVq1fj7++PyWTCYrHg4eFB165dWbduHXXq1AFyp+T/7W9/IycnB4ApU6bg7+/PK6+8woIFCzAajTz//POF+VGJiJQ4R/ZuJvn4nmtuD3qgN+WrOmcmt1ML9I4dOxIfH3/F+tTUVPz8/BzLvr6+pKamkpqamufkZjKZsFqtuLnp0vmS6vfR82sti4hzJSYmAsWvQJfbc7Pn6+KoefPmNG/e3LFsNpvZtGmTY3nZsmVXtGnatKnjtbe3N3v2/PFlzmAwMHbs2Ov2uXHjxquub9GiBevWrbvqNqvVSp8+fRzLDRs2vOrouH7MFhG5dWkbovDJySTDt9pVt6dcbFMyC/RrMZvNpKWlOZbT0tLw8/O7Yr3NZlNxXsLVqlUrT1Feq1Ytl2URKUgdO3YsliO6o0aNAmDWrFkuTiJFwbXO11I4xowZQ3JyMrNnz3Z1FBG5HRfPwJ55YLM6rctTien8einTaf0Vd9UzTpDQcgL3P/zEjXcuZEWi2g0KCuL06dNcvHgRHx8f9uzZw7BhwzAYDGzatInOnTsTGxtL3bp1XR1VCtn48eN5+umn8yyLiEjRcK3ztRSOqKgoV0cQkdt1fCNEPwP39AG/Sk7rds/heDzd/Kjg5+m0Pouz+KD+1Gv8kKtjAC4u0FevXk16ejr9+vVjzJgxDBs2DLvdTq9evahUqRIdOnRg27Zt9O/fH7vdTmRkpCvjihMEBwc7RtFr1aqlG8SJiBQBNzpfi4jI/6Sch28mQUYy5Fjg1wPQex4EhubZLeZ0Ev/afKLQYsSmXCSie0Na3F250PqQwmGw//W2pCVAz549iY6OdnUMuUV6DrpI0aEp7lIQrnZe/umnn2jQoIGLEt159HmLOMGZnfD5U3BvP6jWLHdd9fuuOnI+f9tJthxJoP/9NQsligFoXac8Ph5FYsK03AT9F5MiJzg4mC+++MLVMURERETkTnVoOWyZfnPXjacnwmNzoO7D+do9oJwvHTXCLX+hAl1ERERERAQgxwob/gE/rYKu74LfTRTQ5krgU7bwsskdQQW6iIiI3HGioqI4dOgQCQkJZGZmUqNGDcqUKcO77757w7a7d+/Gz8+P+vXrExYWRpUqVTAYDKSnp9OrVy8ef/xxoqOjOXHiBP/3f/931WPMnj2b8uXLM2DAgIJ+ayJyK/55PySdALsNgsJgxOYCK7Z/Tkrn4Xe2YLXZHOtybHZeaKdLOeVKKtBFRETkjjNmzBiAGxbSV7Ns2TI6d+5M/fr1AZg3bx6enp5YLBY6d+5Mp06dCiWziBQAuz13Knp2et71KefglUPgXQZM7mAw3HZXWdYcElKyOJ6QSo2y3qx5Me+N4txNt9+HlDwq0EVEROSOl52dzT/+8Q9Onz6NzWbj5Zdf5u6776Zv37688847mEwmXnnlFSZMmMDWrVs5dOjQFTcyzczMxNPT84pnw8+bN48vvvgCNzc3mjVrxuuvvw7Ahg0b+PLLL8nMzGT8+PE0atTIae9XpETLSoWLpyH5NCSf+svrM+DmAR7mvG38qoBXqdxtBWTmhqMs3HkaPy93mgaUwcPNWGDHlpJLBbqIiIi43nstIOGngjtehQbw/M587/75559TpkwZIiMjSU5O5oknnuCLL74gKiqKCRMmYLfbeeutt2jQoAGhoaF07tyZqlWrAjB06FAMBgMnTpygffv2uLu7O44bFxfHl19+yeLFi3Fzc+PFF19k06ZNAFSrVo1JkyZx9OhRRo8ezfLlywvu/YsUVb/sg4yLN90sITWLXy9l5VlnsFvxTD+HV2o8Xqln8EqNxzP1Z0zWdLJ8q5Nprk6muQaZ5hpkVW5EZnBNMs3VyXH3u3onp9KAtFt4U1d3MiGNlx6qw9OhtQvsmFLyqUAXERER17uJYrowHDlyhJiYGPbv3w+A1WolOTmZRo0a4efnh7u7+zUfU/bnKe4jRoxg1apVjm0nTpzg3nvvdRTtzZo14+jRowDcd999ANSpU4eEhITCfHsirme1wPoJ8NNqKHfz114n/pqC1ZKDu9sf08LtGDlrKk+CW2UuuDUkwfQQCWUrc9FY9o8p6un/+3Ph91bn//fHOe6qWsppfUnJoAJdRERE7ni1a9emcuXKPPvss2RmZjJ37lxKly7NunXr8PX1xWazsW7dOjp16oTBYMBut19xDA8PD8qVK0d2djZGo9Fx3Pnz52O1WjGZTOzevZvu3btz+PBh9u/fT9euXYmLi3OMxouUSJfPwedPgrc/jNwG3mU4diGVHScS832Izy6e4aXOwTxyT5XCyylSBKhAFxERkTte//79GT9+PE888QSpqakMHDiQc+fOMWvWLBYuXIjdbmfgwIHcc8893HvvvUyfPp3q1asDuVPcjUYjNpuNypUr061bN9asWQNAvXr1eOSRRxgwYAA2m42mTZvSvn17Dh8+THx8PIMHD8ZisTBp0iRXvn2RwnNqGywbBs2GQehr8L8frxZ/f4aYM8ncVSV/I8xNA/xpWK10YSYVKRIM9qv9BFzM9ezZk+joaFfHEBEp9kaNGgXArFmzXJxEirOrnZd/+umna04Zl4Knz1sKlSUdflwBvx78y/oUiPsSuv8L6rTPsylizY9UKuXF8Da6PlvkzzSCLiIiIiIiN++XfbD3YzgYDTWaQ63WYPjTncoNBgj9PygT4LqMIsWMCnQREREREbmmv688yNnkDBqnfUfr1HUAlLNewMueznfmTmwrP5fk7Apw9CqN4y7wpzu0ORy5kMKQlrUKNbdIcaQCXURERFzCbrdjMBhuvKPclhJ4NaM42aoffiG86900/uk07r7VuFC9A7+6l+JS+Sb4GYx0usXjNqtVpkBzipQEKtBFRETE6by8vEhMTKRcuXIq0guR3W4nMTERLy8vV0eRYubg2UuMWrwPOzDTOoXQramY0hOg3TiqNh/g6ngiJZYKdBEREXG66tWrEx8fr+d/O4GXl5fjjvMi+fXrpUzKmz2Z0uMeAuedxtTjv+DlD2VquTqaSImmAl1ERESczt3dncDAQFfHECmxwmZ8y6nf0m65vR3oEVKN4IpmMADlgsGnbIHlE5GrU4EuIiIiIlJE/XIxA2vOzd9HICEli+/HtaeMj8ct923U1SciTqcCXURERESkCDqRkMrD72yhiv/N30OgSmkvzJ5umFRlixQrKtBFRERERIqAg2cvkZhmcSz/nJROUAUzX73SpnA7ttvh9HawZlx9e0524fYvIg4q0EVEREREioAh876nTiUz7iajY91DDSoWbqeZl2Hlc3DhJ/CvefV9aj8IHubCzSEigAp0ERERERGn23cmmf3xl/KsS7fkMOfxppT1vfXrxm/KL/sgegQEtIJnt4G7Hscn4moq0EVEREREnOz9LSdIs+RQq5yPY93glgGU8nLC1/OEONg0BX7+HtqHw739C79PEckXFegiIiIiIoUo7tcUFn1/Js+6H89d5m+d6tP5nirOC5J8Cr59E45+DQ+8BN3/BR4+N2wmIs6jAl1EREREpBDtPJHIoV8u8UjDP4rxJ1vVokXtcs4JcPkcbJkGh5bD/SPgpb3gVdo5fYvITVGBLiIiIiJSQJLTLLy+dD9Wm82xLj45g9bB5RnaOtC5YdISYds7sO9TaDwIXtgDvk76UUBEbokKdBERERGRAnIhJYufzl0monvDPOsbVCnlvBCZl2HHe/D9+9CwJ4zcAaWcOJVeRG6ZCnQRERERkdv02a4zvL/lOBarjTK+HrSrX8iPR7saS3puUb59NtR5GEZsgjK1nJ9DRG6ZCnQRERERkdt08rdUOjasTP/7auLv7e7czq0W2LsAts6AGvfDU2uhQj3nZhCRAqECvYT76quvWLt2ratj3JTk5GQAypQp4+IkN69z58507NjR1TFERESkEO0+lUTff+/Abs+7/q1ejQgs7+u8IDlW2L84987sFevDgMVQNcR5/YtIgVOBLkVOYmIiUDwLdBERESm5UrOsJKRkceR8Cu3qVWTek/e5JojNBj+ugE2RYK4IvT6Ami1ck0VECpQK9BKuY8eOxW5Ed9SoUQDMmjXLxUlERERE/vBG9AG2H0/E7GmiU0MX3HTNbs99hvnGyWB0g0fehKAwMBicn0VECoUKdBERERGR69h1IpF0Sw5nL2Yw6XyU47wAACAASURBVLG76XyPC4rzk1vhm0mQlQJh46F+FxXmIiWQCnQRERERkWtIt1gZ8MFO2tStQCkvN4Irmp0bID4GNk6C5NPQ7g1o2AuMJudmEBGnUYEuIiIiInINdjt4uZv46Kn7ndvx+UOwcQqci4W2oyHkcTA5+e7wIuJ0TivQbTYb4eHhxMXF4eHhQUREBAEBAQAkJCTw6quvOvb96aefeO211xgwYADdu3fHz88PgOrVqzN16lRnRRYRERERca7E47k3fzu5BVq/Ar3ngbuXq1OJiJM4rUDfsGEDFouFJUuWEBsbS1RUFHPnzgWgQoUKfPLJJwDs27ePd955h759+5KVlQXg2CYiIiIi4gyHf73MZ7vOkJ1jc06Hl+Jh85tw+AtoMRK6zgJPJ0+nFxGXc1qBHhMTQ2hoKAAhISEcPHjwin3sdjuTJ09m+vTpmEwmDh48SEZGBkOHDsVqtfLqq68SEqJnO4qIiIhI4dp1IonD51Lo0qgKrXtXKLyOUi/A1hmwfwk0fQpejAFvPWpW5E7ltAI9NTUVs/mPXwFNJhNWqxU3tz8ibNy4kTp16lC7dm0AvLy8GDZsGH369OHUqVMMHz6cdevW5WkjIiIiIvJn1hwbLy7aR2qW9ZaP8cvFDELrVGBIq1oFF+zPMpJh27sQMx8a9YPnv899prmI3NGcVumazWbS0tIcyzab7YpCe9WqVQwePNixHBgYSEBAAAaDgcDAQPz9/UlISKBKFRc82kJEREREioUsq41vDl/gg8HNbus4DSr7FVCiP8lKhV1zYefc3EelPbMV/GsUfD8iUiw5rUBv0qQJmzZtonPnzsTGxlK3bt0r9jl06BBNmjRxLC9dupQjR44QHh7O+fPnSU1NpUKFQpxiJCIiIiIuN+PrOFbEnr3l9jYb+HiYaFu3CH1vzM6EPfPgu3cgsA0MWw/lglydSkSKGKcV6B06dGDbtm30798fu91OZGQkq1evJj09nX79+pGUlISvry8Gg8HRpnfv3owdO5YBAwZgMBiIjIzU9HYRERGREu7YhVSGh9bmwbq3PuXb17OIPCs8Jxv2fQpbpkGVe2HQcqjc0NWpRKSIclq1azQamTRpUp51QUF//GpYtmxZVq5cmWe7h4cHM2bMcEo+ERERESk6yps9qVnOx9Uxbp0tBw4uy31kWpkA6PsxVL+9KfciUvJpOFpEREREpKDY7XB4DWycAp5+0O3d3CntIiL5oAJdREREROR22e1wfCNsjMid1t5hItR5GP50+aaIyI2oQBcRERERuR1ndsI3kyH1PLR7A+7qDkajq1OJSDGkAl1ERERE5Fb8Eps7Yp4QBw/+DRr1B5O+XovIrdO/ICIiIiJSpFhtdldHuL6EONg0Bc7sgjb/B/0Xgpunq1OJSAmguTciIiIiUmQs3xdP7M8XuadaaVdHuVLyKVg+EuZ3hqpN4KV9cP9wFeciUmA0gi4iIiIiRcJ/9/zMjK/j+Ozp5tQoW4QesXb5XO5zzA9Fw/0j4KW94FUEf0AQkWJPBbqIiIiIuNzCXaf558ZjfDa8BUEVzK6OkystEba9A/s+hcZPwAsx4FvO1alEpARTgS4iIiIiLvXRtpN8sPUki4a3oFZ5X1fHgczLsOM9+P59uLsHjNwOpaq6OpWI3AFUoIuIiIiIy3y49QQLdpxi8YgWrp/WbknPLcq3z4Y6HWD4Rigb6NpMInJHUYEuIiIiIi4x59tj/Hf3zywZ0ZKq/t6uC2K1wN4FsHUGVL8PnvwCKtZ3XR4RuWOpQBcRERERp5u14SgrfzjL4hEtqVzayzUhcqywfwl8GwUV6sGAxVA1xDVZRERQgS4iIiIiTmS323l7/RHWHfyVxSNaUNHPBcW5zQY/roBNkWCuCD3fh4CWzs8hIvIXKtBFRERExCnsdjtR6w6zOS6BxSNaUM7s5OeH2+1w9GvYOBkMJnjkTQgKA4PBuTlERK5BBbqIiIiIOMXJ39JYFhPP+lfaUsbXw8mdb80tzDMvQ9g4qP+oCnMRKXJUoIuIiIiIU1htdsr4eDi3OI+PgY2TIPkUPPgG3NMbjCbn9S8ichNUoIuIiIhIyXP+EGycAr/sg7ajofETYHJ3dSoRketSgS4iIiIONpuN8PBw4uLi8PDwICIigoCAAMf2VatWMX/+fIxGI7169WLgwIEuTCtF3crYs/x6KdOxnJCSVfidJh7Pvfnbyc3Q+hXo/R9wd+Ej3EREboIKdBEpdmbPns2xY8dcHeOO8PvnPGrUKBcnuTMEBwfz4osvujTDhg0bsFgsLFmyhNjYWKKiopg7d65j+1tvvcWaNWvw8fGhS5cudOnShdKlS7swsRRlE1f/SJd7quDtkTul3Gg0MLxN7cLp7FI8bH4TfloDLZ6DrjPB069w+hIRKSQq0EWk2Dl27BhHD+2jpjnH1VFKvFL23BsoZZ3e4+IkJd+Z1KJxTWxMTAyhoaEAhISEcPDgwTzb69WrR0pKCm5ubtjtdgy6yZbcwMvt6xTu3dpTL8DWt2H/Ymj6JLwYAz5lC68/EZFCpAJdRIqlmuYc3mhy2dUxRApM5N5Sro4AQGpqKmaz2bFsMpmwWq24ueV+ZahTpw69evXC29ubDh06UKpU0cgtd6CMZNj2LsTMh0b94Lld4FfJ1alERG6LCnQRERFxMJvNpKWlOZZtNpujOD98+DDffvst33zzDT4+Prz++ut8+eWXPPLII66KK0XQsQup/H3lQWx2O5czsjEZC3iWRVYq7JoLO+dCvc7wzFbwr1GwfYiIuIgKdBEREXFo0qQJmzZtonPnzsTGxlK3bl3HNj8/P7y8vPD09MRkMlG2bFkuX9ZMFsnrTFIaaZYc/tapHt7uJvx9CuiRatmZsGcefPcOBIbC0K+hfHDBHFtEpIhQgS4iIiIOHTp0YNu2bfTv3x+73U5kZCSrV68mPT2dfv360a9fPwYOHIi7uzs1a9akR48ero4sRcRb6w4TvfcsmdYcWgSWo1VQ+fw1PLwWvnjtxvtZ0qDWAzBoOVRueHthRUSKKBXoIiIi4mA0Gpk0aVKedUFBQY7XAwYMYMCAAc6OJcXAqcQ0XggL5qEGFfH3volR80vxUPtBCBt//f1MHmCucDsRRUSKPBXoIiIiIlIgyvh4UKX0LTxz3MMXSlcr+EAiIsWMCnQRERERuWWnE9NIt+RwOcPq6igiIsWeCnQRERERuSWZ2TmEzdhMnYq5j+arXuYWRs9FRMRBBbqIiIiI3JSL6RZ2nkgky2rDw2Rk3ctt8tfQlgPHN0J2xh/rft0Pbl6FE1REpJhRgS4iIiIiN2X1D7/w/tYT3FWlFH2aVc9/w4PLYONkqNwo7/pG/Qo2oIhIMaUCXURERERuih14sG5FJne/yced7fo3dIqC+l0KJZeISHFndHUAEREREbkDnI2BtAtQt5Ork4iIFFkq0EVERESk8O16H+57GowmVycRESmyVKCLiIiISOFKvQBHvoTGg1ydRESkSHPaNeg2m43w8HDi4uLw8PAgIiKCgIAAx/b58+ezdOlSypYtC8DEiROpVavWdduIiIiIiPO8ue4wMaeTOX85k3b1Kua/4Q+LoEE38ClbeOFEREoApxXoGzZswGKxsGTJEmJjY4mKimLu3LmO7YcOHeLNN9+kYcM/bjby9ddfX7eNiIiIiDjPtmO/0e++GgRVMBP8v2ef50taApSvU3jBRERKCKcV6DExMYSGhgIQEhLCwYMH82w/dOgQ77//PgkJCTz44IM888wzN2wjIiIiIoXr/OVMes7ZTnaOjaQ0C00DylC/cqn8Nbakw48rIG4dtHyucIOKiJQATivQU1NTMZv/+KXVZDJhtVpxc8uN0KVLFwYOHIjZbOaFF15g06ZNN2wjIiIiIoXDZrOTac3h10uZeLgZiX6uFSajgfJmz2s0yAFrZu7r347A3o/hYDTUaA7tw3X3dhGRfHBapWs2m0lLS3Ms22w2R6Ftt9sZMmQIfn5+ALRt25Yff/zxum1EREREpPDM3niM2RuP4m4y0qxWGSqV8rp+g1UvwYH/gtENzJUg5HEYuR1KV3NOYBGREsBp1W6TJk3YtGkTnTt3JjY2lrp16zq2paam8uijj7J27Vp8fHzYtWsXvXr1IjMz85ptRERERKRg5NjsHDmfgs1ud6w7nZTG6x3r8UzboPwdxJICPd+Hu3sUUkoRkZLPaQV6hw4d2LZtG/3798dutxMZGcnq1atJT0+nX79+vPLKKwwePBgPDw9atmxJ27ZtsdlsV7QRERERkYK17dhvPLdwLzXK+uRZ37lhFRclEhG5MzmtQDcajUyaNCnPuqCgP36R7d69O927d79hGxEREREpGMcupBL3awoHzl7ivlplmP/U/Td/kKMbckfPL50t+IAiIncYXdAtIiIicod6Z/0Rzl7MoKq/Fx3uqnzzB7Ckw2d9oEFXKF0dKt5d8CFFRO4gKtBFRERE7lB27AwPrU2XRrc6ld0Obl7Q9+MCzSUicqcyujqAiIiIiIiIiGgEXUREROSOcPDsJT7afirPuh9+vkSXe6q6JpCIiFxBBbqIiIjIHWDvmWTOJmfQo8kfzyVvHliW1nXKuzCViIj8mQp0ERERkRImKc3CcwtjsOb88Vzz8ymZPFS/En2b1Si4ji4cBqO+ToqIFBT9iyoiIiJSwvyWmsXPSRnM7B+SZ31QBXPBdbJvIayfAN3eLbhjiojc4VSgi4iIiJRAPh4m7qtVtmAPunQonNwCdjt4l4En10LF+gXbh4jIHUwFuoiIiIhcX1Yq2G3w21Ho/i+o0ii3QDe5uzqZiEiJogJdRERERPJK+w1Obs4dLT+xGVJ+zb3W3GiCsoFgrujqhCIiJZIKdBEpdpKSkvgtxUTk3lKujiJSYE6nmCiflOTqGFJM/ZyUzsX0bMfy6aS0mztA5mU4vf2PovziGQhoBYFt4f4RUKEBGI0FnFpERP5KBbqIiIhIMddz7nbK+XrgZjI41t0XeJ3rz7MzIf773NHxk1vgwo9QrQkEtoFHZ0LVxmDS10QREWfTv7wiUuyULVsW35QTvNHksqujiBSYyL2l8CxbwDf0kjtGjs3OwqebU87see2dUhNg38e5RfnZGKhQH2q3hbBxUKM5uHs7L7CIiFyVCnQRERGRO8GPK+CnNdDmdaj1AHiVdnUiERH5CxXoIiIiIsVQYmoWi74/g80O6RbrtXf8aTVc+Ani9+ROY6/f2XkhRUTkpuhuHyIiIiLF0A/xF/k8Jp7sHBsvhtXB38fj6jtunQEXT+c+Gu2evs4NKSIiN0Uj6CIiIiLFyMrYs2yOS+DcpUxql/fltYfr3bhRs6FQrWnhhxMRkduiAl1ERESkGPnq0K+UN3vSO7g69Sr7uTqOiIgUIBXoIiIiIsVM88BydGlUxdUxRESkgOkadBEREREREZEiQCPoIiIiIiXN5V/g/QchJxsyL4G7j6sTiYhIPqhAFxERESkG0i1Wsq12LFb7tXey5UDWZbj4M3j6wdCvwWgCb3/nBRURkVumAl1ERESkiMuy5hAyaT1ebkYMBgNPhwZefcfNb8G2meDmCTWag2855wYVEZHbogJdREREpIjLsdkxGQzsD+949R2ST0FaIiSdgHZvwAOjnJpPREQKhgp0ERERkeIs6ST8uy2UC8pdvqePa/OIiMgtU4EuIiIiUgRYc2ysO/Qr1pwrrzHPsuZcu+E3k6DVC9B2dCGmExERZ1CBLiIiIlIEnPwtjbHLDhDWoOJVtz/RouaVK+Nj4MwOeOyfhZxOREScQQW6iIiIiAsdiL/Ehp/Ok5iWReXSXszq3zh/De12WP93eHAsePgWbkgREXEKo6sDiIiIiNzJVu//hZjTyZTz9eTFh+rkv+GRdZD+G4Q8XnjhRETEqTSCLiIiIuJioXXK80zboPw3yLHC+n/Aw5PBpK9zIiIlhUbQRURERIqbfZ+AuSLUedjVSUREpADpJ1cRERGR4iQrFb6NggGLwGBwdRoRESlAGkEXERERKU52vAe1WkO1Jq5OIiIiBUwj6CIiIiLFRcp52DUXRnzr6iQiIlIInFag22w2wsPDiYuLw8PDg4iICAICAhzb16xZw4IFCzCZTNStW5fw8HCMRiPdu3fHz88PgOrVqzN16lRnRRYREREpWjZH5d61vUwtVycREZFC4LQCfcOGDVgsFpYsWUJsbCxRUVHMnTsXgMzMTGbOnMnq1avx9vbm1VdfZdOmTbRu3RqATz75xFkxRURERIqmhCPw40p4YY+rk4iISCFx2jXoMTExhIaGAhASEsLBgwcd2zw8PFi8eDHe3t4AWK1WPD09OXz4MBkZGQwdOpTBgwcTGxvrrLgiIiIiRcuGcHjgZfAp6+okIiJSSJw2gp6amorZbHYsm0wmrFYrbm5uGI1GypcvD+SOlqenp/PAAw9w5MgRhg0bRp8+fTh16hTDhw9n3bp1uLnp0nkRERG5g2RnwLH10Hueq5OIiEghclqlazabSUtLcyzbbLY8hbbNZmPatGmcPHmS2bNnYzAYCAwMJCAgwPHa39+fhIQEqlSp4qzYIiIiIq5nt4HRDdy9XJ1EREQKkdOmuDdp0oQtW7YAEBsbS926dfNs//vf/05WVhZz5sxxTHVfunQpUVFRAJw/f57U1FQqVKjgrMgiIiIiIiIiTuO0EfQOHTqwbds2+vfvj91uJzIyktWrV5Oenk7Dhg1ZunQpzZo1Y8iQIQAMHjyY3r17M3bsWAYMGIDBYCAyMlLT20VERERERKREclq1azQamTRpUp51QUFBjteHDx++arsZM2YUai4RERERERGRokDD0SIiIuJgs9kIDw8nLi4ODw8PIiIiCAgIcGzfv38/UVFR2O12KlSowLRp0/D09HRhYhERkZLDadegi4iISNG3YcMGLBYLS5Ys4bXXXnPcCwbAbrczYcIEpk6dyqJFiwgNDeXs2bMuTCsiIlKyaARdREREHGJiYggNDQUgJCSEgwcPOradPHkSf39/FixYwJEjR2jbti21a9d2VVQREZESRyPoIiIi4pCamorZbHYsm0wmrFYrAMnJyezbt4+BAwcyf/58du7cyY4dO1wVVUREpMRRgS4iIiIOZrOZtLQ0x7LNZnM8QcXf35+AgACCg4Nxd3cnNDQ0zwi73Jyus7+j4T++Yv62k/h4alKjiIioQBcREZE/adKkCVu2bAEgNjaWunXrOrbVqFGDtLQ0Tp8+DcCePXuoU6eOS3KWBGcvZrD2pVD2TujAE81rujqOiIgUAfq5VkRERBw6dOjAtm3b6N+/P3a7ncjISFavXk16ejr9+vVjypQpvPbaa9jtdho3bsyDDz7o6sjFmq+nCT8vd1fHEBGRIkIFuoiIiDgYjUYmTZqUZ11QUJDjdcuWLVm6dKmzY5UY2Tk29sdfxGaH6jk/4xa/E7w9btzQmlH44URExOVUoIuIiIg4ybZjv/HSon3UreTHx8ZJ+G0JBFM+CnSAOg8XbjgREXE5FegiIiIiTmKz22kaUIb5A++Ct23w+OfgW97VsUREpIjQTeJEREREnMFuxy/xB55MfBveuQuC2oGXv6tTiYhIEaIRdBEREZHCdmwDfP137k5PYa1bexixC0pVcXUqEREpYjSCLiIiIlLY9n4Mjfqys8vXfFF6gIpzERG5Ko2g59Ps2bM5duyYq2PcEX7/nEeNGuXiJHeG4OBgXnzxRVfHEBEp0X5OzmBdUg5f2U9SyluPVRMRkatTgZ5Px44dI/bgT+T4lHV1lBLPkJP71zLmxHkXJyn5TOlJro4gInJHSEjJpExFD54KCSS4otnVcUREpIhSgX4TcnzKklG/s6tjiBQY78NrXR3hlp1JNRG5t5SrY5R4lywGAEp72F2cpOQ7k2qijqtDSIGy5tgY+MEuUrKsvJJiIbihmZBGmtouIiLXpgJdRIqd4OBgV0e4Y1z+3yUnFQP0mRe2OujvdkmTnWMnNv4iy59rRY315fCtVtrVkUREpIjLV4H++eef06dPH8fyxx9/zODBgwstlIjI9eiaeef5/V4Qs2bNcnESkeLJaIC7K3pB2inw9HV1HBERKeKuW6CvWbOGjRs3smvXLnbu3AlATk4OR48eVYEuIiIickN2WPk8lA+G4A6uDiMiIkXcdQv00NBQKlSowMWLF+nXrx8ARqORGjVqOCWciIiISHH2vGEZJJ2EJ78Ao55uKyIi13fdAr106dI0b96c5s2bk5iYSFZWFpA7ii4iIiIi12Y6+Dk9Dd9C/+3g7u3qOCIiUgzk6xr0iRMnsnnzZipWrIjdbsdgMLB48eLCziYiIiJSPJ3egfv6Nxhp+xur/Cq5Oo2IiBQT+SrQf/jhBzZs2IBRU7NEREREri/5NPx3MJZucznymc3VaUREpBjJV8UdEBDgmN4uIiIiItdx9GsIbo8tqL2rk4iISDGTrxH0c+fO0a5dOwICAgA0xV1ERETkenTNuYiI3IJ8FegzZswo7BwiIiIiIiIid7R8Fehubm5MmzaN5ORkOnbsSL169ahWrVphZxMREREp0i5lZPPG8gNYc/641rztpTNUtSTwaeJeDBhcmE5ERIqbfF2DPmHCBHr16oXFYqFZs2ZMmTKlsHOJiIiIFHkXLmcScyqZHo2rOf7cW8Of2uV96d20Oh89dZ+rI4qISDGSrxH0rKwsWrZsydy5c6lduzaenp6FnUtERESkWDB7udGpYZU/VqSXApMPNf+8TkREJB/yNYLu4eHB1q1bsdlsxMbG4uHhUdi5RERERERERO4o+SrQJ0+eTHR0NMnJycybN4/w8PBCjiUiIiJSDNntcDYGPHxdnURERIqhfE1xr1y5MpMnT9az0EVERESuxW6H9RMg4TA88qar04iISDGUrwJ99OjR7N27Fz8/P+x2OwaDgeXLlxd2NhEREZHi49soOL4JhqwGr9KuTiMiIsVQvgr0kydPsmHDhtvqyGazER4eTlxcHB4eHkRERBAQEODYvnHjRt577z3c3Nzo1asXffv2vWEbEREREVdISrNw+NfLxCdn5K74biYcXAZPfQk+ZV0bTkREiq18FeiNGjXixIkT1K5d+5Y72rBhAxaLhSVLlhAbG0tUVBRz584FIDs7m6lTp7J06VK8vb0ZMGAA7dq1Y9++fddsIyIiIuIqH2w9wRf7z1HV34v/8/8WYqJzi3NzBVdHExGRYixfBbrZbKZ37974+Pg41n333Xc31VFMTAyhoaEAhISEcPDgQce248ePU7NmTUqXzp0O1rRpU/bs2UNsbOw12zhbUlISpvREvA+vdVkGkYJmSk8kKcnd1TFERIqNb346z/nLWRw8e4mBzWvyrN822Pw5PPkFlKrq6ngiIlLM5atA37VrF99//z1ubvna/apSU1Mxm82OZZPJhNVqxc3NjdTUVPz8/BzbfH19SU1NvW4bEREREWcbvXQ/bepWoHoZbzrbt8KmKBiyBsroEjwREbl9+ap0a9WqRWJiIpUqVbrljsxmM2lpaY5lm83mKLT/ui0tLQ0/P7/rtnG2smXLcvJiNhn1O7ukf5HC4H14LWXL6lpJEZGbMa5LA8r/+h0snwKDV0L5YFdHEhGREiJfz0GPiYkhLCyM1q1bO/7crCZNmrBlyxYAYmNjqVu3rmNbUFAQp0+f5uLFi1gsFvbs2UPjxo2v20ZERETEZU58Cy2ehUp3uTqJiIiUIPkajl6/fv1td9ShQwe2bdtG//79sdv/v727D46qPvQ//tndPBg2gUiiCGoEArFKREigtdcJZZQMD7ao0BBCJVxEcn9DQSuRC79eTRnGJqDiVaDRiw8xpNeEp7RXQFGiFHopiomCwjVoQFO1yEMCmmxIls2e3x/83N4USIKye87uvl8zzGTPA/vJGWa+58P3PBgqLCzUpk2b1NLSouzsbC1atEizZs2SYRiaPHmy+vTpc959AAAATHfmtORMNDsFACDEdFrQi4uLNWfOHM2fP182m63DuuXLl1/UF9ntdi1ZsqTDsuTkZN/Pt912m2677bYu9wEAADCT/eu/nn2l2n3f7xW0AAD8o04L+reFeerUqQEJAwAAYGmGodhtC6R/mislJHe9PQAAF6HTe9AHDx4st9utNWvWaPjw4Ro2bJiGDh2qVatWBSofAACAZYw3/iy766j0T/ebHQUAEII6nUHfuHGjnn32WZ04cULjxo2TYRhyOBxKT08PVD4AAABTFPzXfpXv+WuHZW9FVMg9tkQRjkiTUgEAQlmnBX3KlCmaMmWKNmzYoJ///OeBygQAAGC6hma3nsi6WeNT+/qWRT7ple2q601MBQAIZd16ivutt96q5557Tm1tbb5lc+fO9VsoAAAAK3DYbYqK6NZbaQEA+N66NeI88MADam5uVmJiou8PAAAAAAC4dLo1g+50OvXggw/6OwsAAAAAAGGrWwV98ODB2rJli2644Qbf+9AHDBjg12AAAAAAAISTbhX0jz76SB999JHvs81m05o1a/wWCgAAAACAcNOtgl5WVqampiZ9+eWXuvbaa+V0Ov2dCwAAIOC8XkNPvfmJXG0e/c+RbzT+pqvMjgQACCPdKuivv/66nnnmGbW3t2vcuHGy2WyaM2eOv7MBAAAEVJvHq+LtdVo0/gf6xY+S9OOBCWZHAgCEkW49xb2kpETr1q1TfHy85syZo6qqKn/nAgAAMEWEw6b7MgbqvoyBSoiNNjsOACCMdKug2+12RUVFyWazyWazKSYmxt+5AAAAAAAIK90q6CNGjFB+fr6OHj2qgoIC3XTTTf7OBQAAAABAWOlWQf/FL36h4cOHa+LEidq1a5cmTpzo71wAAAAAAISVbhX0RYsWadCgQTp48KDmz5+voqIif+cCAAAImE37/qaB/3eLbvzNVvWKiTQ7DgAgTHWroHs8Ho0cOVJNTU2644475PV6/Z0LAAAgYE61uDX1h0mq++0E7V50kNhYnAAAHTxJREFUu9lxAABhqluvWTtz5oyKioqUnp6ut99+W+3t7f7OBQAAEFB2m+Sw28yOAQAIY92aQV+6dKkGDBigvLw8NTY26vHHH/d3LgAAAAAAwkq3ZtD79++v/v37S5ImTJjgzzwAAAAAAISlbs2gAwAAAAAA/6KgAwAAAABgARR0AACA7jJ4kw0AwH8o6AAAAN3x5XtSVKzUo7fZSQAAIYqCDgAA0B17Vks/vE+yO8xOAgAIURR0AACArjQflw6+Kg2fbnYSAEAIo6ADAAAfr9ergoICZWdna/r06aqvrz/vdo888oieeOKJAKczUc1L0o13cnk7AMCvKOgAAMCnqqpKbrdba9euVX5+vpYuXXrONhUVFfr4449NSGeS9jNS9YvSD//F7CQAgBBHQQcAAD41NTXKyMiQJA0bNkz79+/vsP7999/Xvn37lJ2dbUY8c3y0Seo9ULoq1ewkAIAQR0EHAAA+zc3Nio2N9X12OBzyeDySpGPHjmnVqlUqKCgwK5459qyWfpRndgoAQBiIMDsAAACwjtjYWLlcLt9nr9eriIizpwtbt27VyZMnlZeXp+PHj6u1tVUDBw7UpEmTzIrrf0f2Sac+l66/w+wkAIAwQEEHAAA+aWlp2r59uyZMmKC9e/cqJSXFty43N1e5ubmSpMrKSh0+fDi0y7kkvbNaGnmv5OCUCQDgf4w2AADAJzMzU7t27dLUqVNlGIYKCwu1adMmtbS0hNd955LkapBqN0nz3jc7CQAgTFDQAQCAj91u15IlSzosS05OPme7kJ85l6T3SqUf/ExyJpidBAAQJgJW0FtbW7VgwQI1NDTI6XRq2bJl6t2747tEX3rpJW3ZskWS9JOf/ERz586VYRgaNWqU+vfvL+nsE2Xz8/MDFRsAAISjdo/07gtSzstmJwEAhJGAFfTy8nKlpKRo3rx52rJli4qLi/Xwww/71n/++ed65ZVXtH79etlsNk2bNk1jxoxRTEyMhgwZomeffTZQUQEAQLg7uEXqdY3U92azkwAAwkjAXrP2v9+rOmrUKO3evbvD+quuukrPP/+8HA6H7Ha7PB6PoqOjdeDAAR09elTTp0/X7Nmzdfjw4UBFBgAA4eodXq0GAAg8v8ygr1+/XqWlpR2WJSQkKC4uTpLkdDrV1NTUYX1kZKR69+4twzD02GOP6cYbb9SAAQN04sQJ5eXlafz48aqurtaCBQu0ceNGf8QGAACQvtovNR6SbphodhIAQJjxS0HPyspSVlZWh2Vz5871vVfV5XKpZ8+e5+zX1tamX//613I6nfrNb34jSUpNTZXD4ZAkjRgxQkePHpVhGLLZbP6IDgAAwt2e1dKIWZIj0uwkAIAwE7BL3NPS0rRjxw5J0s6dO5Went5hvWEYmjNnjq6//notWbLEV8pXrVrlm42vra1Vv379KOcAAMA/Whql//mjlP7PZicBAIShgD0kLicnRwsXLlROTo4iIyO1fPlySVJJSYmSkpLk9Xq1Z88eud1u/fnPf5YkzZ8/X3l5eVqwYIF27Nghh8OhoqKiQEUGAADh5v0yKWW8FHuF2UkAAGEoYAU9JiZGK1asOGf5zJkzfT9/+OGH59139erVfssFAAAgSTajXXr3eSnrJbOjAADCVMAKOgAAgBUYhqG/NrboTLvhW3a8qU3Xf/MXyXmldHV6J3sDAOA/FHQAABBW6htalPnvO3Rt7x4dlv9n5B+k0f9iUioAACjoAAAgzJxp9+q6BKeq5v/k7wuPfSStqZduvMu8YACAsBewp7gDAABYVv0uKWWsFBFldhIAQBijoAMAABiGZOe95wAAc1HQAQAAAACwAAo6AAAAAAAWQEEHAAAAAMACeIo7AATA66+/rldffdXsGBetrq5OkvTAAw+YnOTiTJgwQWPHjjU7BoJJu1uy2cxOAQAIc8ygAwAuKCEhQQkJCWbHAPzvwB+lgaPNTgEACHPMoANAAIwdO5YZXcCqvnxPajoipYw3OwkAIMwxgw4AAMLbntXSyPskB/MWAABzUdABAED4aj4uHXxVSss1OwkAAFzifjEcLY2KqQ2+hzwFG9uZ05IkIzLG5CShz9HSKKmP2TEAICBa3B59cfK06hta/r7wvZekGyZKPXqblgsAgG9R0Ltp0KBBZkcIG98+NXrQQIqj//Xh3zaAsPG77XVa++7nurxHlG6+Jv7swo9fl24vMDcYAAD/HwW9m+bNm2d2hLDx7eucnn76aZOTAABCiafd0H0ZA/V/fpL894XedinSaV4oAAD+F+5BBwAAAADAAijoAAAAAABYAAUdAAAAAAALoKADAAAAAGABFHQAAAAAACyAgg4AAAAAgAVQ0AEAAAAAsAAKOgAAAAAAFhBhdgAAAAB/OtPuVbvX0Jl2w+woAAB0ioIOAABC2pgnd+jI162ySXr0rlSz4wAAcEEUdAAAENKaWz36y6LblBgbbXYUAAA6xT3oAAAAAABYAAUdAAAAAAALoKADAAAAAGABFHQAAAAAACyAgg4AAAAAgAVQ0AEAAAAAsAAKOgAAAAAAFhCw96C3trZqwYIFamhokNPp1LJly9S7d+8O2zz66KN677335HQ6JUnFxcWKjIzscj8AAAAAAIJdwGbQy8vLlZKSopdffll33XWXiouLz9nmwIEDev7551VWVqaysjLFxcV1az8AAAAAAIJdwAp6TU2NMjIyJEmjRo3S7t27O6z3er2qr69XQUGBpk6dqg0bNnRrPwAAgO/EMKS2byRHwC4oBACgU34ZkdavX6/S0tIOyxISEhQXFydJcjqdampq6rC+paVF99xzj2bOnKn29nbl5uYqNTVVzc3Nne4HAADwnXzxruT1SH1SzU4CAIAkPxX0rKwsZWVldVg2d+5cuVwuSZLL5VLPnj07rI+JiVFubq5iYmIkSbfccotqa2sVGxvb6X4AAADfyTv/IY2cLdkdZicBAEBSAC9xT0tL044dOyRJO3fuVHp6eof1n332maZNm6b29nadOXNG7733noYMGdLlfgAAABet6Supbps0/B6zkwAA4BOwm65ycnK0cOFC5eTkKDIyUsuXL5cklZSUKCkpSbfffrt+9rOfacqUKYqMjNSdd96pwYMH65prrjnvfgAAAN9ZdYmUOlmKiTc7CQAAPgEr6DExMVqxYsU5y2fOnOn7efbs2Zo9e3a39gMAAPhOPG6ppkTK/S+zkwAA0EHALnEHAACwhMbDUlSsdOUNZicBAKAD3isCAABCTqPLrYp3/yrDkFrc7eduYOcUCABgPcygAwCAkLPv81Oq2PO5XG0ezRmdrMt7RJkdCQCALvHfxwAAIGS8su9v2vnxcR35+rQGXuHUv477gdmRAADoNgo6AAAIGa/v/0rxPSJ157CrdcNVPc2OAwDARaGgAwCAkPLj5AT9dGg/s2MAAHDRuAcdAAAAAAALoKADAAAAAGABFHQAAAAAACyAe9ABAICP1+vV4sWLdfDgQUVFRenRRx/Vdddd51u/efNmlZaWyuFwKCUlRYsXL5bdzv/3AwBwKTCiAgAAn6qqKrndbq1du1b5+flaunSpb11ra6ueeuoprVmzRhUVFWpubtb27dtNTAsAQGihoAMAAJ+amhplZGRIkoYNG6b9+/f71kVFRamiokIxMTGSJI/Ho+joaFNyAgAQiijoAADAp7m5WbGxsb7PDodDHo9HkmS325WYmChJKisrU0tLi2699VZTcgIAEIq4Bx0AAPjExsbK5XL5Pnu9XkVERHT4/Pjjj+vTTz/VypUrZbPZzIj5/Xz9ueS8wuwUAACcgxl0AADgk5aWpp07d0qS9u7dq5SUlA7rCwoK1NbWpuLiYt+l7kHnk23SoNvMTgEAwDmYQQcAAD6ZmZnatWuXpk6dKsMwVFhYqE2bNqmlpUWpqanasGGDRowYoRkzZkiScnNzlZmZaXLqi1S3TcoqNTsFAADnoKADAAAfu92uJUuWdFiWnJzs+7m2tjbQkS6thkOS2yVddZPZSQAAOAeXuAMAgPBR96Y0aIwUjPfOAwBCHjPoAAAgqHnavZr50rtqbvPo0xMu/ezmvhfeuG6bdHNO4MIBAHARKOgAACConWk39M7hRpXn3SKbTUrt1+sCG7ZK9bulSasDGxAAgG6ioAMAgKBnt0vp113e+Ub1/y31GSLFdLEdAAAm4R50AAAQHj6pkgaPMTsFAAAXREEHAADhoa7q7APiAACwKAo6AAAIfSc/k1pPSVfdbHYSAAAuiIIOAABC3yfbzs6e2zn1AQBYF6MUAAAIfVzeDgAIAhR0AAAQ2s60Sp/tkpJvMzsJAACdoqADAIDQ9tfd0pU/kHr0NjsJAACdoqADAIDQVlclDco0OwUAAF2ioAMAgNBW9yb3nwMAggIFHQAAhDbXcSk+yewUAAB0iYIOAAAAAIAFUNABAAAAALCACLMDAAAAXHJf1ki/nywZXsntkiKizE4EAECXAlbQW1tbtWDBAjU0NMjpdGrZsmXq3fvvrzv56KOPVFhY6Pu8d+9e/e53v1NGRoZGjRql/v37S5KGDRum/Pz8QMUGAADByNUg9b1ZynpJskdK0bFmJwIAoEsBK+jl5eVKSUnRvHnztGXLFhUXF+vhhx/2rb/hhhtUVlYmSXrttdd05ZVXatSoUaqvr9eQIUP07LPPBioqAAAIBfZIKeZys1MAANBtAbsHvaamRhkZGZKkUaNGaffu3efdrqWlRStXrtS//du/SZIOHDigo0ePavr06Zo9e7YOHz4cqMgAAAAAAASMX2bQ169fr9LS0g7LEhISFBcXJ0lyOp1qamo6774bNmzQuHHjfJe/X3HFFcrLy9P48eNVXV2tBQsWaOPGjf6IDQAAAACAafxS0LOyspSVldVh2dy5c+VyuSRJLpdLPXv2PO++mzZt0ooVK3yfU1NT5XA4JEkjRozQ0aNHZRiGbDabP6IDAICQYJgdAACAixawS9zT0tK0Y8cOSdLOnTuVnp5+zjZNTU1yu93q27evb9mqVat8s/G1tbXq168f5RwAAFyYxy39ZaV09bnnGgAAWFnAHhKXk5OjhQsXKicnR5GRkVq+fLkkqaSkRElJSbr99tv16aef6uqrr+6wX15enhYsWKAdO3bI4XCoqKgoUJEBAECwMQxpy4NSdJz0k381Ow0AABclYAU9Jiamw6Xr35o5c6bv56FDh6q4uLjD+l69emn16tV+zwcAAELAf/+7dOQDaeZrkt1hdhoAAC5KwAo6AACAXx34o/TuC9J923jvOQAgKFHQAQBA0BuqT6QtT0vT/yD17Gd2HAAAvhMKOgAACG5ej1bZn5TufEbqO9TsNAAAfGcBe4o7AACAX7S7FSeXdP14s5MAAPC9UNABAAAAALAACjoAAAhKhmHoeFObGprbzI4CAMAlwT3oAAAgKK2v+UKP/HG/EqPb9abNZnYcAAC+N2bQAQBAUGo7066sEddo18LbdFkEpzQAgODHaAYAAAAAgAVQ0AEAAAAAsAAKOgDgghoaGnT//feroaHB7CgAAAAhj4IOALig0tJSffjhh1qzZo3ZUYBOGGYHAADgkqCgAwDOq6GhQVu3bpVhGNq6dSuz6LCub/4mxV5hdgoAAL43CjoA4LxKS0vl9XolSe3t7cyiw7r+tlfqO8zsFAAAfG8UdADAeVVVVcnj8UiSPB6Ptm3bZnIi4AKO7JX6UdABAMGPgg4AOK8xY8YoIiJCkhQREaHMzEyTEwEXcGSf1Pdms1MAAPC9UdABAOc1Y8YM2e1nhwmHw6Hc3FyTEwHnshle6cgHXOIOAAgJFHQAwHklJCRo3LhxstlsGjdunBISEsyOBJzjcvcRKTpOciaaHQUAgO8twuwAAADrmjFjhj777DNmz2EpLW6Pmlo9+qbVo34ttVzeDgAIGRR0AMAFJSQkaMWKFWbHADr455J39cnRJkU67PqPvp/ygDgAQMigoAMAgKDSdqZdJTN/qGHXxktrVkp9J5gdCQCAS4J70AEAQHAyDJ7gDgAIKRR0AAAQnL7+XHJES3FXmZ0EAIBLgoIOAACC09/2MnsOAAgpFHQAABCcjuzjAXEAgJBCQQcAAMGJ+88BACGGgg4AAIKPYUhH9kp9mUEHAIQOCjoAAAg6kS1Hz5b0nv3MjgIAwCVDQQcAAEEn5sSHZy9vt9nMjgIAwCVDQQcAAEGnR8N+HhAHAAg5FHQAABB0Yhr284A4AEDIoaADAICgc/YSd2bQAQChhYIOAACCSry3UXZPqxSfZHYUAAAuqYAX9G3btik/P/+869atW6dJkyZpypQp2r59uySptbVV8+bN07Rp0zR79mw1NjYGMi4AAGHF6/WqoKBA2dnZmj59uurr6zusf+uttzR58mRlZ2dr3bp1pmRM9hzS6cRUHhAHAAg5AS3ojz76qJYvXy6v13vOuuPHj6usrEwVFRV64YUX9OSTT8rtdqu8vFwpKSl6+eWXddddd6m4uDiQkQEACCtVVVVyu91au3at8vPztXTpUt+6M2fOqKioSC+++KLKysq0du1aHT9+POAZB3oOqSXhpoB/LwAA/hYRyC9LS0vTmDFjtHbt2nPWffDBBxo+fLiioqIUFRWlpKQk1dbWqqamRvfdd58kadSoURT0i/T666/r1VdfNTvGRamrq5MkPfDAAyYnuXgTJkzQ2LFjzY4BAN9ZTU2NMjIyJEnDhg3T/v37fesOHTqkpKQk9erVS5KUnp6u6upqjR8/PqAZkz11Op34i4B+JwAAgeCXgr5+/XqVlpZ2WFZYWKgJEybonXfeOe8+zc3NiouL8312Op1qbm7usNzpdKqpqckfkWEhCQkJZkcAgLDV3Nys2NhY32eHwyGPx6OIiIgLjtWBFnHjRPW+if8MBQCEHr8U9KysLGVlZV3UPrGxsXK5XL7PLpdLcXFxHZa7XC717NnzkmYNdWPHjmVGFwDQbf84Hnu9XkVERJx33bdjdaCNvGtOwL8TAIBAsMxT3IcOHaqamhq1tbWpqalJhw4dUkpKitLS0rRjxw5J0s6dO5Wenm5yUgAAQldaWpp27twpSdq7d69SUlJ865KTk1VfX69Tp07J7Xarurpaw4cPNysqAAAhJ6D3oJ9PSUmJkpKSdPvtt2v69OmaNm2aDMPQgw8+qOjoaOXk5GjhwoXKyclRZGSkli9fbnZkAABCVmZmpnbt2qWpU6fKMAwVFhZq06ZNamlpUXZ2thYtWqRZs2bJMAxNnjxZffr0MTsyAAAhw2YYhmF2iEtt0qRJqqysNDsGAAAQ4zIAAN1lmUvcAQAAAAAIZxR0AAAAAAAsgIIOAAAAAIAFUNABAAAAALAACjoAAAAAABZAQQcAAAAAwAIo6AAAAAAAWAAFHQAAAAAAC6CgAwAAAABgARR0AAAAAAAsgIIOAAAAAIAFUNABAAAAALAAm2EYhtkhLrUf/ehHuvrqq82OAQCAJVx++eV64YUXTPt+xmUAADq60NgckgUdAAAAAIBgwyXuAAAAAABYAAUdAAAAAAALoKADAAAAAGABFHQAAAAAACyAgg4AAAAAgAVQ0AEAAAAAsAAKOizF6/WqoKBA2dnZmj59uurr682OBIS1ffv2afr06WbHQJjqakx46623NHnyZGVnZ2vdunUmpQwOXR3LzZs3KysrS1OnTlVBQYG8Xq9JSa2tu+cpjzzyiJ544okApwsuXR3LDz74QNOmTVNOTo7uv/9+tbW1mZTU+ro6lq+88oruvvtuTZ48WS+//LJJKYPHhc59AjXmUNBhKVVVVXK73Vq7dq3y8/O1dOlSsyMBYeu5557Tww8/zEkRTNPZmHDmzBkVFRXpxRdfVFlZmdauXavjx4+bmNbaOjuWra2teuqpp7RmzRpVVFSoublZ27dvNzGtdXXnPKWiokIff/yxCemCS2fH0jAMPfLIIyoqKlJ5ebkyMjL05ZdfmpjW2rr6d/nYY4+ppKRE5eXlKikp0ddff21SUuu70LlPIMccCjospaamRhkZGZKkYcOGaf/+/SYnAsJXUlKSVq5caXYMhLHOxoRDhw4pKSlJvXr1UlRUlNLT01VdXW1WVMvr7FhGRUWpoqJCMTExkiSPx6Po6GhTclpdV+cp77//vvbt26fs7Gwz4gWVzo7lp59+qvj4eJWWluqee+7RqVOnNHDgQLOiWl5X/y6vv/56NTU1ye12yzAM2Ww2M2IGhQud+wRyzKGgw1Kam5sVGxvr++xwOOTxeExMBISvsWPHKiIiwuwYCGOdjQnNzc2Ki4vzrXM6nWpubg54xmDR2bG02+1KTEyUJJWVlamlpUW33nqrKTmtrrPjeOzYMa1atUoFBQVmxQsqnR3LkydP6v3339e0adNUUlKit99+W7t37zYrquV1df48ePBgTZ48WXfccYdGjx6tnj17mhEzKFzo3CeQYw4FHZYSGxsrl8vl++z1eikIABCmOhsT/nGdy+XqcPKEjroaX71er5YtW6Zdu3Zp5cqVzLBdQGfHcevWrTp58qTy8vK0evVqbd68WZWVlWZFtbzOjmV8fLyuu+46DRo0SJGRkcrIyOCqyk50dixra2v1pz/9SW+++abeeustNTY26rXXXjMratAK5JhDQYelpKWlaefOnZKkvXv3KiUlxeREAACzdDYmJCcnq76+XqdOnZLb7VZ1dbWGDx9uVlTL62p8LSgoUFtbm4qLi32XuuNcnR3H3NxcVVZWqqysTHl5efrpT3+qSZMmmRXV8jo7ltdee61cLpfvYWfV1dUaPHiwKTmDQWfHMi4uTpdddpmio6PlcDjUu3dvffPNN2ZFDVqBHHOYmoSlZGZmateuXZo6daoMw1BhYaHZkQAAJjnfmLBp0ya1tLQoOztbixYt0qxZs2QYhiZPnqw+ffqYHdmyOjuWqamp2rBhg0aMGKEZM2ZIOls2MzMzTU5tPV39m0T3dXUsf/vb3yo/P1+GYWj48OEaPXq02ZEtq6tjmZ2drWnTpikyMlJJSUm6++67zY4cNMwYc2yGYRh++ZsBAAAAAEC3cYk7AAAAAAAWQEEHAAAAAMACKOgAAAAAAFgABR0AAAAAAAugoAMAAAAAYAEUdADf2zvvvKMHH3yw29vPnTtXknTw4EG9++67/ooFAAAABBUKOoCAW7VqlSTpjTfeUF1dnclpAAAAAGuIMDsAAOuqrKzUm2++qebmZp08eVK//OUvFRsbq6eeekrR0dGKj49XYWFhh31+//vf64033pDH41FcXJxWrlypzZs3a+PGjfJ6vbr//vv10EMPqbKyUn/4wx8UGRmpIUOGaMmSJdqwYYMk6Ve/+pXuvfdeDR061IxfGwAAADAFBR1Ap1paWlRSUqLGxkZlZWXJZrOpvLxcffr0UWlpqZ555hmNHj1akuT1enXq1Cm99NJLstvtmjVrlj788ENJUs+ePfXMM8/4/t4+ffro7rvvVmJiooYOHarLLrtMdXV1SkxM1BdffEE5BwAAQNihoAPo1MiRI2W325WYmKgePXrI4/GoT58+vnVPPvmkr6Db7XZFRkZq/vz56tGjh7766it5PB5J0oABAzr9nqysLFVWVqpfv36aOHGiX38nAAAAwIoo6AA6deDAAUnSiRMndPr0aUnSsWPHdOWVV2rPnj3q37+/b9va2lpVVVVp/fr1On36tCZNmiTDMCSdLe//yGazyev1SpLGjRunF198UfHx8Xr66af9/FsBAAAA1kNBB9CpEydOaMaMGWpqatLixYsVERGhefPmyWazqVevXioqKtInn3wiSbruuusUExOjSZMmKSoqSldccYWOHTt2wb87NTVVjz32mJKTk3XLLbdo5MiRamxsVHx8fKB+PQAAAMAybMa301sA8A8qKyt1+PBhPfTQQwH5vsWLF2vs2LH68Y9/HJDvAwAAAKyE16wBsIR7771Xra2tlHMAAACELWbQAQAAAACwAGbQAQAAAACwAAo6AAAAAAAWQEEHAAAAAMACKOgAAAAAAFgABR0AAAAAAAv4f/77HnVo8nVPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(ncols=2, figsize=(14, 6))\n",
    "sns.boxplot(x='polarity', y='sentiment', data=test, ax=axes[0])\n",
    "axes[0].set_title('TextBlob Sentiment Scores')\n",
    "roc_nb.plot(ax=axes[1], label='Naive Bayes', legend=True, lw=1, title='ROC Curves')\n",
    "roc_tb.plot(ax=axes[1], label='TextBlob', legend=True, lw=1)\n",
    "sns.despine()\n",
    "fig.tight_layout();"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "316px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}