bbs-opinion-teamwork-sols.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Opinion Mining and Sentiment Analysis: Teamwork\n",
    "\n",
    "**Text Mining unit**\n",
    "\n",
    "_Prof. Gianluca Moro, Dott. Ing. Roberto Pasolini – DISI, University of Bologna_\n",
    "\n",
    "**Bologna Business School** - Alma Mater Studiorum Università di Bologna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "- The provided exercises must be executed by teams of 2 or 3 persons, different teams should not communicate with each other\n",
    "- It is allowed to consult course material and the Web for advice\n",
    "- If still in doubt about anything, ask the teacher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "The following cell contains all necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following to download the necessary files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from urllib.request import urlretrieve\n",
    "def download(file, url):\n",
    "    if not os.path.exists(file):\n",
    "        urlretrieve(url, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "download(\"100k_reviews.tsv.gz\", \"https://www.dropbox.com/s/9fkjz84dnzfyimt/estore_reviews_100k.tsv.gz?dl=1\")\n",
    "download(\"positive-words.txt\", \"https://github.com/datascienceunibo/bbs-opinion-lab-2019/raw/master/positive-words.txt\")\n",
    "download(\"negative-words.txt\", \"https://github.com/datascienceunibo/bbs-opinion-lab-2019/raw/master/negative-words.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "We provide in the `100k_reviews.tsv.gz` file a dataset of 100,000 reviews posted on Amazon.com about DVDs of movies and TV series. Each review is labeled with a score between 1 and 5 stars.\n",
    "\n",
    "Run the following to correctly load the file into a pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"100k_reviews.tsv.gz\", sep=\"\\t\", compression=\"gzip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>stars</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Romero did the right thing when he pick...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK, that makes it sound like something out of ...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>- At a tribal village, a pensive Elizabeth Cur...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Wow! This has to be one of the more unusual mo...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Kevin Costner is one of those actors that I ne...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  stars\n",
       "0  George Romero did the right thing when he pick...      5\n",
       "1  OK, that makes it sound like something out of ...      5\n",
       "2  - At a tribal village, a pensive Elizabeth Cur...      5\n",
       "3  Wow! This has to be one of the more unusual mo...      5\n",
       "4  Kevin Costner is one of those actors that I ne...      5"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within the teamwork you will also make use of the Hu and Liu sentiment lexicon: run the following to load sets of positive and negative words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scan_hu_liu(f):\n",
    "    for line in f:\n",
    "        line = line.decode(errors=\"ignore\").strip()\n",
    "        if line and not line.startswith(\";\"):\n",
    "            yield line\n",
    "\n",
    "def load_hu_liu(filename):\n",
    "    with open(filename, \"rb\") as f:\n",
    "        return set(scan_hu_liu(f))\n",
    "\n",
    "hu_liu_pos = load_hu_liu(\"positive-words.txt\")\n",
    "hu_liu_neg = load_hu_liu(\"negative-words.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1)** Verify the distribution of the number of stars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5    47458\n",
       "4    26486\n",
       "3    13898\n",
       "2     6791\n",
       "1     5367\n",
       "Name: stars, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"stars\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2)** Add a `label` column to the DataFrame whose value is `\"pos\"` for reviews with 4 or 5 stars and `\"neg\"` for reviews with 3 stars or less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"label\"] = np.where(data[\"stars\"] >= 4, \"pos\", \"neg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3)** Split the dataset randomly into a training set with 80\\% of data and a test set with the remaining 20\\%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset, testset = train_test_split(data, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4)** Create a function which accepts a text as input, counts the occurrences of positive and negative words from the Hu \\& Liu lexicon and return `\"pos\"` if there are more positive words than negative or `\"neg\"` otherwise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentiment_label(text):\n",
    "    words = nltk.word_tokenize(text)\n",
    "    pos_count = sum(1 for word in words if word in hu_liu_pos)\n",
    "    neg_count = sum(1 for word in words if word in hu_liu_neg)\n",
    "    return \"pos\" if pos_count > neg_count else \"neg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('pos', 'neg')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "(sentiment_label(\"This is awesome!\"),\n",
    " sentiment_label(\"This is horrible!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4)** Apply the function above to test reviews and get the percentage of cases where the function output matches the known label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 31.2 s, sys: 8 ms, total: 31.2 s\n",
      "Wall time: 31.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lexicon_label = testset[\"text\"].apply(sentiment_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6696"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(lexicon_label == testset[\"label\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5)** Create a tf.idf vector space model from training reviews excluding words appearing in less than 3 documents and extract the document-term matrix for them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = TfidfVectorizer(min_df=3)\n",
    "train_dtm = vect.fit_transform(trainset[\"text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6)** Train a logistic regression classifier on the training reviews, using the representation created above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.1 s, sys: 36 ms, total: 3.13 s\n",
      "Wall time: 3.14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model = LogisticRegression()\n",
    "model.fit(train_dtm, trainset[\"label\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7)** Verify the accuracy of the classifier on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8471"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dtm = vect.transform(testset[\"text\"])\n",
    "model.score(test_dtm, testset[\"label\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8)** Extract the 10 words with the highest regression coefficient and the 10 words with the lowest coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs = pd.Series(\n",
    "          model.coef_[0],\n",
    "    index=vect.get_feature_names()\n",
    ").sort_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "worst           -7.779192\n",
       "boring          -7.559625\n",
       "unfortunately   -7.267638\n",
       "waste           -6.985140\n",
       "bad             -5.771118\n",
       "nothing         -5.712796\n",
       "terrible        -5.704467\n",
       "wasted          -5.389687\n",
       "fails           -5.230198\n",
       "poorly          -5.142444\n",
       "dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coefs.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "awesome      4.599625\n",
       "amazing      4.604864\n",
       "hilarious    4.719440\n",
       "superb       4.745584\n",
       "favorite     5.077927\n",
       "highly       5.789346\n",
       "wonderful    6.359046\n",
       "perfect      6.655664\n",
       "excellent    6.878745\n",
       "great        7.736444\n",
       "dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coefs.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9)** Create a function which accepts a text as input and returns a list of the only words from the text which are also present in the Hu and Liu lexicon (each distinct word must appear in the list as many times as it appears in the text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "hu_liu_all = hu_liu_pos | hu_liu_neg\n",
    "def tokenize_hu_liu(text):\n",
    "    words = nltk.word_tokenize(text)\n",
    "    return [word for word in words if word in hu_liu_all]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['awesome'], ['horrible'])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "(tokenize_hu_liu(\"This is awesome!\"),\n",
    " tokenize_hu_liu(\"This is horrible!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**10)** Repeat points from 5 to 7 with a tf.idf vectorizer which uses the function above to extract tokens from text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = TfidfVectorizer(min_df=3, tokenizer=tokenize_hu_liu)\n",
    "train_dtm = vect.fit_transform(trainset[\"text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 680 ms, sys: 0 ns, total: 680 ms\n",
      "Wall time: 684 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model = LogisticRegression()\n",
    "model.fit(train_dtm, trainset[\"label\"]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8189"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dtm = vect.transform(testset[\"text\"])\n",
    "model.score(test_dtm, testset[\"label\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}