Text Classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Classifiers\n",
    "Now that we have explored and built some text features (i.e. features extracted from text data), we want something useful to feed the text data into. A common machine learning problem involving text data is *text classification*. In this notebook we will explore text classification using a real dataset.\n",
    "\n",
    "First let's import the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "pd.set_option('display.max_colwidth', 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading text data\n",
    "Often, text data will come in the format of *JavaScript Object Notation* (JSON). Let's import some JSON data.\n",
    "\n",
    "Run the following code to load the JSON file `Data/reviews.json` into a Pandas DataFrame and print the first $10$ lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('Data/reviews.json','r') as in_file:\n",
    "    df = pd.DataFrame(json.load(in_file))\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This above data consists of user reviews of Google Play applications and the corresponding rating for each review. [SOURCE](http://jmcauley.ucsd.edu/data/amazon/). All reviews with *medium* ratings ($2-4$) were removed from the raw data, and a *balanced* number of reviews with ratings $1$ and $5$ were extracted. This was done to make it easier to train a classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simpler Text Classifier\n",
    "Here we will introduce common classifier for text classification, the [*Multinomial naive Bayes*](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes) classifier. Using Sklearn, it is simple to add a classifer as the last step in the text processing pipeline. \n",
    "\n",
    "Run the following code to setup a text classification pipeline and fit it to the rating and review data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_pipe = Pipeline([\n",
    "    ('vect', CountVectorizer(stop_words='english')),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', MultinomialNB()),\n",
    "])\n",
    "\n",
    "text_pipe.fit(df.review, df.rating);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a text classification pipeline, we create our own inputs and pass them through the pipeline as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = [\n",
    "    \"Love this app!\",\n",
    "    \"Hate this app!\",\n",
    "    \"Total rubbish\",\n",
    "    \"Works perfectly\",\n",
    "]\n",
    "print(text_pipe.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the ratings output by the text classification pipeline makes sense given the example text inputs? Experiment with your own inputs and try to fool the system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance on training set\n",
    "Given a text classification piepline, we can evaluate how well the pipeline performs on the training set.\n",
    "\n",
    "We can compute the *confusion matrix* as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.confusion_matrix(df.rating, text_pipe.predict(df.review)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix simply calculates the number of correctly and incorrectly classified examples in the data.\n",
    "\n",
    "These numbers can be used to calculate the *precision*, *recall* and *f1-score* as follows. See [here](https://en.wikipedia.org/wiki/Precision_and_recall) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.classification_report(df.rating, text_pipe.predict(df.review)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance on test set\n",
    "As with other classifiers, we need to evaluate a text classification pipeline on a tes set (i.e. a data set it has not seen before). Let's first split the ratings and reviews data into training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df.review, df.rating, test_size=0.2, random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's build another text classification pipeline, but this time let's train it on the training data and evaluate it on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_pipe = Pipeline([\n",
    "    ('vect', CountVectorizer(stop_words='english')),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', MultinomialNB()),\n",
    "])\n",
    "\n",
    "text_pipe.fit(X_train, y_train);\n",
    "print(text_pipe.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Selection\n",
    "There is a chance there might exist better parameter choices for the above pipeline. For example, maybe stop-words are useful...\n",
    "\n",
    "Run the following code using the same pipeline, but without removing stop-words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_pipe_wstops = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', MultinomialNB()),\n",
    "])\n",
    "\n",
    "text_pipe_wstops.fit(X_train, y_train);\n",
    "print(text_pipe_wstops.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, we get slightly better performance on the test data without stop words!\n",
    "\n",
    "We can also compare the same pipeline with and without using a Tfidf transform. Let's do that, but this time let's do it using a grid search.\n",
    "\n",
    "Run the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "grid_params = {\n",
    "    'tfidf__use_idf': (True, False),\n",
    "}\n",
    "search = GridSearchCV(text_pipe_wstops, grid_params)\n",
    "search.fit(X_train, y_train);\n",
    "print(search.best_params_)\n",
    "print(search.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like, for this data, we don't really need a Tfidf transform either!\n",
    "\n",
    "*Note*: This is often not the case."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}