SMS Classification Practice.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.linear_model import SGDClassifier, LogisticRegression\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "\n",
    "from scipy.stats import randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('sms_spam.csv', encoding = \"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook's focus is on using classifiers, and later on classification with train-validation-test splits + cross-validation. I've chosen to use a ready-made dataset since the focus is on the classification methods rather than on preprocessing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   labels                                               text\n",
       "0       0  Go until jurong point, crazy.. Available only ...\n",
       "1       0                      Ok lar... Joking wif u oni...\n",
       "2       1  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3       0  U dun say so early hor... U c already then say...\n",
       "4       0  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The last 3 columns are not meaningful, so we'll drop those\n",
    "df = df.drop(df.columns[2:5], axis=1)\n",
    "\n",
    "# We can also rename the columns to be a bit more informative\n",
    "df = df.rename(index=str, columns={\"v1\": \"labels\", \"v2\": \"text\"})\n",
    "\n",
    "# We'll also convert the spam/ham labels to 1 and 0\n",
    "df['labels'] = df.labels.map({'ham':0, 'spam':1})\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    4825\n",
       "1     747\n",
       "Name: labels, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.labels.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 747 spam texts and 4825 good texts, which indicates that we do not suffer from a severe class imbalance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.text\n",
    "Y = df.labels\n",
    "\n",
    "# Split the data into train and test\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y)\n",
    "\n",
    "vectorizer = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit vectorizer to the training data, transform training data\n",
    "X_train_df = vectorizer.fit_transform(X_train)\n",
    "\n",
    "# Just transform the test data \n",
    "X_test_df = vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multinomial Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb = MultinomialNB()\n",
    "nb.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, ..., 1, 0, 0])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dict for the predictions of all the models we'll try\n",
    "prediction = dict()\n",
    "\n",
    "prediction[\"Multinomial\"] = nb.predict(X_test_df)\n",
    "\n",
    "# Looks like this (an array of 0s, 1s)\n",
    "prediction['Multinomial']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9863603732950467"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Score the predictions\n",
    "\n",
    "accuracy_score(Y_test, prediction['Multinomial'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! 98% accuracy. We should make sure to take a look at the sensitivity and specificity of these predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        Ham       0.99      1.00      0.99      1197\n",
      "       Spam       0.97      0.93      0.95       196\n",
      "\n",
      "avg / total       0.99      0.99      0.99      1393\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(Y_test, prediction['Multinomial'], target_names = ['Ham', 'Spam']), sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the classification_report, we can see that the performance is excellent for real texts and very strong for spam. Since it's worse for a user to miss real messages from people they know than to get the occasional spam message, this is an ideal situation.\n",
    "\n",
    "Let's take a look at some of the messages that were misclassified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4254    Block Breaker now comes in deluxe format with ...\n",
       "68      Did you hear about the new \\Divorce Barbie\\\"? ...\n",
       "671            SMS. ac sun0819 posts HELLO:\\You seem cool\n",
       "1457    CLAIRE here am havin borin time & am now alone...\n",
       "3458    Not heard from U4 a while. Call me now am here...\n",
       "1939    More people are dogging in your area now. Call...\n",
       "2247    Back 2 work 2morro half term over! Can U C me ...\n",
       "2662    Hello darling how are you today? I would love ...\n",
       "2821    ROMCAPspam Everyone around should be respondin...\n",
       "3417    LIFE has never been this much fun and great un...\n",
       "4067    TBS/PERSOLVO. been chasing us since Sept forå£...\n",
       "4674    Hi babe its Chloe, how r u? I was smashed on s...\n",
       "4371    Ur balance is now å£600. Next question: Comple...\n",
       "3862    Oh my god! I've found your number again! I'm s...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find texts in X_test that were actually spam (Y_test = 1), but predicted Ham (0)\n",
    "X_test[Y_test > prediction[\"Multinomial\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9791816223977028"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate predictions\n",
    "prediction[\"Logistic\"] = logreg.predict(X_test_df)\n",
    "\n",
    "# Score predictions\n",
    "accuracy_score(Y_test, prediction['Logistic'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slightly worse performance than the Naive Bayes classifier, but still pretty darned good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        Ham       0.98      1.00      0.99      1197\n",
      "       Spam       0.99      0.86      0.92       196\n",
      "\n",
      "avg / total       0.98      0.98      0.98      1393\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Generate another classification report\n",
    "print(classification_report(Y_test, prediction['Logistic'], target_names = ['Ham', 'Spam']), sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall for Spam is a bit worse with the logistic regression, pulling the f1-score down."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm = SVC()\n",
    "svm.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8592964824120602"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate predictions\n",
    "prediction[\"SVM\"] = svm.predict(X_test_df)\n",
    "\n",
    "# Score predictions\n",
    "accuracy_score(Y_test, prediction['SVM'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes. The SVM performed pretty poorly on this task out of the box. Let's investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        Ham       0.86      1.00      0.92      1197\n",
      "       Spam       0.00      0.00      0.00       196\n",
      "\n",
      "avg / total       0.74      0.86      0.79      1393\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlenechambliss/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(Y_test, prediction['SVM'], target_names = ['Ham', 'Spam']), sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the warning generated above that the SVM failed to predict \"spam\" for any texts. \n",
    "\n",
    "We can verify that this matches the accuracy score by noticing that the accuracy score is simply the percentage of non-spam texts in the dataset, meaning that the SVM chose \"Ham\" and was right 86% of the time simply because 86% of our texts are indeed not spam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8659368269921034"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "4825 / (4825 + 747)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps the SVM just needs some tuning and coaxing in order to work well for this data, but we will come back to that later. First, we'll try one last classification method.\n",
    "\n",
    "# Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = DecisionTreeClassifier()\n",
    "tree.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9633883704235463"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate predictions\n",
    "prediction[\"Tree\"] = tree.predict(X_test_df)\n",
    "\n",
    "# Score predictions\n",
    "accuracy_score(Y_test, prediction['Tree'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This accuracy is the third best out of the four classifiers we've tried so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        Ham       0.98      0.98      0.98      1197\n",
      "       Spam       0.88      0.86      0.87       196\n",
      "\n",
      "avg / total       0.96      0.96      0.96      1393\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(Y_test, prediction['Tree'], target_names = ['Ham', 'Spam']), sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tree seems to be letting a lot more spam slip through than our first two classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning\n",
    "The Naive Bayes classifier was our best classifier by far. Let's see if we can tune the other classifiers a bit to get them up to, or near, the same accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'C': array([1.00000e-05, 8.48343e-05, 7.19686e-04, 6.10540e-03, 5.17947e-02,\n",
       "       4.39397e-01, 3.72759e+00, 3.16228e+01, 2.68270e+02, 2.27585e+03,\n",
       "       1.93070e+04, 1.63789e+05, 1.38950e+06, 1.17877e+07, 1.00000e+08])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Set up a \"grid\" of values we'd like to test to find out which results in the best performance\n",
    "c_space = np.logspace(-5, 8, 15)\n",
    "param_grid = {'C': c_space}\n",
    "\n",
    "# Perform a grid search for the logistic regression classifier, then re-fit the data\n",
    "logreg_cv = GridSearchCV(logreg, param_grid, cv = 5)\n",
    "logreg_cv.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned Logistic Regression Parameters: {'C': 1389495.494373136}\n",
      "Best score is 0.9856424982053122\n"
     ]
    }
   ],
   "source": [
    "print(\"Tuned Logistic Regression Parameters: {}\".format(logreg_cv.best_params_)) \n",
    "print(\"Best score is {}\".format(logreg_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome. The performance is now on par with the NB classifier. Let's try this same strategy for the other classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=5, error_score='raise',\n",
       "          estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best'),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=1,\n",
       "          param_distributions={'max_depth': [3, None], 'max_features': <scipy.stats._distn_infrastructure.rv_frozen object at 0x1a1b8a0da0>, 'min_samples_leaf': <scipy.stats._distn_infrastructure.rv_frozen object at 0x1a1b8a0a90>, 'criterion': ['gini', 'entropy']},\n",
       "          pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "          return_train_score='warn', scoring=None, verbose=0)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Decision trees take a lot of parameters, making them an ideal use case for RandomizedSearchCV\n",
    "\n",
    "param_dist = {\"max_depth\": [3, None],\n",
    "              \"max_features\": randint(1, 9),\n",
    "              \"min_samples_leaf\": randint(1, 9),\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "tree_cv = RandomizedSearchCV(tree, param_dist, cv=5)\n",
    "tree_cv.fit(X_train_df, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned Tree Parameters: {'criterion': 'gini', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 2}\n",
      "Best score is 0.8724575257238574\n"
     ]
    }
   ],
   "source": [
    "print(\"Tuned Tree Parameters: {}\".format(tree_cv.best_params_)) \n",
    "print(\"Best score is {}\".format(tree_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very interesting - we see that the tree with parameters chosen from the grid actually performs worse than an out-of-the-box tree with default parameters. Perhaps we ought to just leave this one alone.\n",
    "\n",
    "Let's try to work on the SVM now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned SVM Parameters: {'C': 2275.845926074791}\n",
      "Best score is 0.9844460397224216\n"
     ]
    }
   ],
   "source": [
    "# Reusing the same param_grid we used for the logistic regression, since both classifiers take a C value\n",
    "svm_cv = GridSearchCV(svm, param_grid, cv = 5)\n",
    "svm_cv.fit(X_train_df, Y_train)\n",
    "\n",
    "print(\"Tuned SVM Parameters: {}\".format(svm_cv.best_params_)) \n",
    "print(\"Best score is {}\".format(svm_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! The SVM is now our second-best performer, going by best score. Impressive how much it was able to improve. \n",
    "\n",
    "While running the CV, however, a significant amount of processing power and time were used, making SVM impractical for very large datasets.\n",
    "\n",
    "To wrap up, let's play with the Naive Bayes classifier a bit and see if we can get it to perform any better than it already has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned NB Parameters: {'alpha': 1.8000000000000003}\n",
      "Best score is 0.9818138310600623\n"
     ]
    }
   ],
   "source": [
    "alpha_values = np.arange(0.1, 4, 0.1)\n",
    "alpha_grid = {'alpha': alpha_values}\n",
    "\n",
    "nb_cv = GridSearchCV(nb, alpha_grid, cv = 5)\n",
    "nb_cv.fit(X_train_df, Y_train)\n",
    "\n",
    "print(\"Tuned NB Parameters: {}\".format(nb_cv.best_params_)) \n",
    "print(\"Best score is {}\".format(nb_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm. Looks like the performance is roughly equal or even a little worse. For this dataset, using out-of-the-box NB seems to be more ideal than fussing with the hyperparameters."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}