ch08/ch08.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Machine Learning with PyTorch and Scikit-Learn  \n",
    "# -- Code Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package version checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add folder to path in order to load from the check_packages.py script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check recommended package versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] Your Python version is 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:35:11) \n",
      "[Clang 11.1.0 ]\n",
      "[OK] numpy 1.22.0\n",
      "[OK] pandas 1.3.5\n",
      "[OK] sklearn 1.0.2\n",
      "[OK] pyprind 2.11.3\n",
      "[OK] nltk 3.6.3\n"
     ]
    }
   ],
   "source": [
    "from python_environment_check import check_packages\n",
    "\n",
    "\n",
    "d = {\n",
    "    'numpy': '1.21.2',\n",
    "    'pandas': '1.3.2',\n",
    "    'sklearn': '1.0',\n",
    "    'pyprind': '2.11.3',\n",
    "    'nltk': '3.6',\n",
    "}\n",
    "check_packages(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8 - Applying Machine Learning To Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Preparing the IMDb movie review data for text processing](#Preparing-the-IMDb-movie-review-data-for-text-processing)\n",
    "  - [Obtaining the IMDb movie review dataset](#Obtaining-the-IMDb-movie-review-dataset)\n",
    "  - [Preprocessing the movie dataset into more convenient format](#Preprocessing-the-movie-dataset-into-more-convenient-format)\n",
    "- [Introducing the bag-of-words model](#Introducing-the-bag-of-words-model)\n",
    "  - [Transforming words into feature vectors](#Transforming-words-into-feature-vectors)\n",
    "  - [Assessing word relevancy via term frequency-inverse document frequency](#Assessing-word-relevancy-via-term-frequency-inverse-document-frequency)\n",
    "  - [Cleaning text data](#Cleaning-text-data)\n",
    "  - [Processing documents into tokens](#Processing-documents-into-tokens)\n",
    "- [Training a logistic regression model for document classification](#Training-a-logistic-regression-model-for-document-classification)\n",
    "- [Working with bigger data – online algorithms and out-of-core learning](#Working-with-bigger-data-–-online-algorithms-and-out-of-core-learning)\n",
    "- [Topic modeling](#Topic-modeling)\n",
    "  - [Decomposing text documents with Latent Dirichlet Allocation](#Decomposing-text-documents-with-Latent-Dirichlet-Allocation)\n",
    "  - [Latent Dirichlet Allocation with scikit-learn](#Latent-Dirichlet-Allocation-with-scikit-learn)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing the IMDb movie review data for text processing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining the IMDb movie review dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IMDB movie review set can be downloaded from [http://ai.stanford.edu/~amaas/data/sentiment/](http://ai.stanford.edu/~amaas/data/sentiment/).\n",
    "After downloading the dataset, decompress the files.\n",
    "\n",
    "A) If you are working with Linux or MacOS X, open a new terminal windowm `cd` into the download directory and execute \n",
    "\n",
    "`tar -zxf aclImdb_v1.tar.gz`\n",
    "\n",
    "B) If you are working with Windows, download an archiver such as [7Zip](http://www.7-zip.org) to extract the files from the download archive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional code to download and unzip the dataset via Python:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36% | 29.38 MB | 4.67 MB/s | 6.29 sec elapsed"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The Jupyter server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--ServerApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "ServerApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67% | 54.32 MB | 6.44 MB/s | 8.43 sec elapsed"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The Jupyter server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--ServerApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "ServerApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "86% | 69.35 MB | 7.26 MB/s | 9.55 sec elapsed"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The Jupyter server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--ServerApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "ServerApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import tarfile\n",
    "import time\n",
    "import urllib.request\n",
    "\n",
    "source = 'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'\n",
    "target = 'aclImdb_v1.tar.gz'\n",
    "\n",
    "if os.path.exists(target):\n",
    "    os.remove(target)\n",
    "\n",
    "def reporthook(count, block_size, total_size):\n",
    "    global start_time\n",
    "    if count == 0:\n",
    "        start_time = time.time()\n",
    "        return\n",
    "    duration = time.time() - start_time\n",
    "    progress_size = int(count * block_size)\n",
    "    speed = progress_size / (1024.**2 * duration)\n",
    "    percent = count * block_size * 100. / total_size\n",
    "\n",
    "    sys.stdout.write(f'\\r{int(percent)}% | {progress_size / (1024.**2):.2f} MB '\n",
    "                     f'| {speed:.2f} MB/s | {duration:.2f} sec elapsed')\n",
    "    sys.stdout.flush()\n",
    "\n",
    "\n",
    "if not os.path.isdir('aclImdb') and not os.path.isfile('aclImdb_v1.tar.gz'):\n",
    "    urllib.request.urlretrieve(source, target, reporthook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.isdir('aclImdb'):\n",
    "\n",
    "    with tarfile.open(target, 'r:gz') as tar:\n",
    "        tar.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing the movie dataset into more convenient format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install pyprind by uncommenting the next code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pyprind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:00:21\n"
     ]
    }
   ],
   "source": [
    "import pyprind\n",
    "import pandas as pd\n",
    "import os\n",
    "import sys\n",
    "\n",
    "# change the `basepath` to the directory of the\n",
    "# unzipped movie dataset\n",
    "\n",
    "basepath = 'aclImdb'\n",
    "\n",
    "labels = {'pos': 1, 'neg': 0}\n",
    "pbar = pyprind.ProgBar(50000, stream=sys.stdout)\n",
    "df = pd.DataFrame()\n",
    "for s in ('test', 'train'):\n",
    "    for l in ('pos', 'neg'):\n",
    "        path = os.path.join(basepath, s, l)\n",
    "        for file in sorted(os.listdir(path)):\n",
    "            with open(os.path.join(path, file), \n",
    "                      'r', encoding='utf-8') as infile:\n",
    "                txt = infile.read()\n",
    "            df = df.append([[txt, labels[l]]], \n",
    "                           ignore_index=True)\n",
    "            pbar.update()\n",
    "df.columns = ['review', 'sentiment']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shuffling the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(0)\n",
    "df = df.reindex(np.random.permutation(df.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional: Saving the assembled data as CSV file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('movie_data.csv', index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "\n",
    "# the following is necessary on some computers:\n",
    "df = df.rename(columns={\"0\": \"review\", \"1\": \"sentiment\"})\n",
    "\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 2)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Note\n",
    "\n",
    "If you have problems with creating the `movie_data.csv`, you can find a download a zip archive at \n",
    "https://github.com/rasbt/python-machine-learning-book-pytorch-edition/tree/master/code/ch08/\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing the bag-of-words model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming documents into feature vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling the fit_transform method on CountVectorizer, we just constructed the vocabulary of the bag-of-words model and transformed the following three sentences into sparse feature vectors:\n",
    "1. The sun is shining\n",
    "2. The weather is sweet\n",
    "3. The sun is shining, the weather is sweet, and one and one is two\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer()\n",
    "docs = np.array([\n",
    "        'The sun is shining',\n",
    "        'The weather is sweet',\n",
    "        'The sun is shining, the weather is sweet, and one and one is two'])\n",
    "bag = count.fit_transform(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us print the contents of the vocabulary to get a better understanding of the underlying concepts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 6, 'sun': 4, 'is': 1, 'shining': 3, 'weather': 8, 'sweet': 5, 'and': 0, 'one': 2, 'two': 7}\n"
     ]
    }
   ],
   "source": [
    "print(count.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from executing the preceding command, the vocabulary is stored in a Python dictionary, which maps the unique words that are mapped to integer indices. Next let us print the feature vectors that we just created:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each index position in the feature vectors shown here corresponds to the integer values that are stored as dictionary items in the CountVectorizer vocabulary. For example, the  rst feature at index position 0 resembles the count of the word and, which only occurs in the last document, and the word is at index position 1 (the 2nd feature in the document vectors) occurs in all three sentences. Those values in the feature vectors are also called the raw term frequencies: *tf (t,d)*—the number of times a term t occurs in a document *d*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 0 1 1 0 1 0 0]\n",
      " [0 1 0 0 0 1 1 0 1]\n",
      " [2 3 2 1 1 1 2 1 1]]\n"
     ]
    }
   ],
   "source": [
    "print(bag.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessing word relevancy via term frequency-inverse document frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are analyzing text data, we often encounter words that occur across multiple documents from both classes. Those frequently occurring words typically don't contain useful or discriminatory information. In this subsection, we will learn about a useful technique called term frequency-inverse document frequency (tf-idf) that can be used to downweight those frequently occurring words in the feature vectors. The tf-idf can be de ned as the product of the term frequency and the inverse document frequency:\n",
    "\n",
    "$$\\text{tf-idf}(t,d)=\\text{tf (t,d)}\\times \\text{idf}(t,d)$$\n",
    "\n",
    "Here the tf(t, d) is the term frequency that we introduced in the previous section,\n",
    "and the inverse document frequency *idf(t, d)* can be calculated as:\n",
    "\n",
    "$$\\text{idf}(t,d) = \\text{log}\\frac{n_d}{1+\\text{df}(d, t)},$$\n",
    "\n",
    "where $n_d$ is the total number of documents, and *df(d, t)* is the number of documents *d* that contain the term *t*. Note that adding the constant 1 to the denominator is optional and serves the purpose of assigning a non-zero value to terms that occur in all training examples; the log is used to ensure that low document frequencies are not given too much weight.\n",
    "\n",
    "Scikit-learn implements yet another transformer, the `TfidfTransformer`, that takes the raw term frequencies from `CountVectorizer` as input and transforms them into tf-idfs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.   0.43 0.   0.56 0.56 0.   0.43 0.   0.  ]\n",
      " [0.   0.43 0.   0.   0.   0.56 0.43 0.   0.56]\n",
      " [0.5  0.45 0.5  0.19 0.19 0.19 0.3  0.25 0.19]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf = TfidfTransformer(use_idf=True, \n",
    "                         norm='l2', \n",
    "                         smooth_idf=True)\n",
    "print(tfidf.fit_transform(count.fit_transform(docs))\n",
    "      .toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in the previous subsection, the word is had the largest term frequency in the 3rd document, being the most frequently occurring word. However, after transforming the same feature vector into tf-idfs, we see that the word is is\n",
    "now associated with a relatively small tf-idf (0.45) in document 3 since it is\n",
    "also contained in documents 1 and 2 and thus is unlikely to contain any useful, discriminatory information.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we'd manually calculated the tf-idfs of the individual terms in our feature vectors, we'd have noticed that the `TfidfTransformer` calculates the tf-idfs slightly differently compared to the standard textbook equations that we de ned earlier. The equations for the idf and tf-idf that were implemented in scikit-learn are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{idf} (t,d) = log\\frac{1 + n_d}{1 + \\text{df}(d, t)}$$\n",
    "\n",
    "The tf-idf equation that was implemented in scikit-learn is as follows:\n",
    "\n",
    "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d)+1)$$\n",
    "\n",
    "While it is also more typical to normalize the raw term frequencies before calculating the tf-idfs, the `TfidfTransformer` normalizes the tf-idfs directly.\n",
    "\n",
    "By default (`norm='l2'`), scikit-learn's TfidfTransformer applies the L2-normalization, which returns a vector of length 1 by dividing an un-normalized feature vector *v* by its L2-norm:\n",
    "\n",
    "$$v_{\\text{norm}} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v_{1}^{2} + v_{2}^{2} + \\dots + v_{n}^{2}}} = \\frac{v}{\\big (\\sum_{i=1}^{n} v_{i}^{2}\\big)^\\frac{1}{2}}$$\n",
    "\n",
    "To make sure that we understand how TfidfTransformer works, let us walk\n",
    "through an example and calculate the tf-idf of the word is in the 3rd document.\n",
    "\n",
    "The word is has a term frequency of 3 (tf = 3) in document 3 ($d_3$), and the document frequency of this term is 3 since the term is occurs in all three documents (df = 3). Thus, we can calculate the idf as follows:\n",
    "\n",
    "$$\\text{idf}(\"is\", d_3) = log \\frac{1+3}{1+3} = 0$$\n",
    "\n",
    "Now in order to calculate the tf-idf, we simply need to add 1 to the inverse document frequency and multiply it by the term frequency:\n",
    "\n",
    "$$\\text{tf-idf}(\"is\", d_3)= 3 \\times (0+1) = 3$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf-idf of term \"is\" = 3.00\n"
     ]
    }
   ],
   "source": [
    "tf_is = 3\n",
    "n_docs = 3\n",
    "idf_is = np.log((n_docs+1) / (3+1))\n",
    "tfidf_is = tf_is * (idf_is + 1)\n",
    "print(f'tf-idf of term \"is\" = {tfidf_is:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we repeated these calculations for all terms in the 3rd document, we'd obtain the following tf-idf vectors: [3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]. However, we notice that the values in this feature vector are different from the values that we obtained from the TfidfTransformer that we used previously. The  nal step that we are missing in this tf-idf calculation is the L2-normalization, which can be applied as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{tfi-df}_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n",
    "\n",
    "$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n",
    "\n",
    "$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the results match the results returned by scikit-learn's `TfidfTransformer` (below). Since we now understand how tf-idfs are calculated, let us proceed to the next sections and apply those concepts to the movie review dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.39, 3.  , 3.39, 1.29, 1.29, 1.29, 2.  , 1.69, 1.29])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfTransformer(use_idf=True, norm=None, smooth_idf=True)\n",
    "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n",
    "raw_tfidf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l2_tfidf = raw_tfidf / np.sqrt(np.sum(raw_tfidf**2))\n",
    "l2_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven.<br /><br />Title (Brazil): Not Available'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'review'][-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def preprocessor(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n",
    "                           text)\n",
    "    text = (re.sub('[\\W]+', ' ', text.lower()) +\n",
    "            ' '.join(emoticons).replace('-', ''))\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven title brazil not available'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(df.loc[0, 'review'][-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test :) :( :)'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(\"</a>This :) is :( a test :-)!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['review'] = df['review'].apply(preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing documents into tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "\n",
    "def tokenizer(text):\n",
    "    return text.split()\n",
    "\n",
    "\n",
    "def tokenizer_porter(text):\n",
    "    return [porter.stem(word) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runners', 'like', 'running', 'and', 'thus', 'they', 'run']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'and', 'thu', 'they', 'run']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_porter('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/sebastian/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'run', 'lot']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop = stopwords.words('english')\n",
    "[w for w in tokenizer_porter('a runner likes running and runs a lot')\n",
    " if w not in stop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a logistic regression model for document classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Strip HTML and punctuation to speed up the GridSearch later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df.loc[:25000, 'review'].values\n",
    "y_train = df.loc[:25000, 'sentiment'].values\n",
    "X_test = df.loc[25000:, 'review'].values\n",
    "y_test = df.loc[25000:, 'sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "tfidf = TfidfVectorizer(strip_accents=None,\n",
    "                        lowercase=False,\n",
    "                        preprocessor=None)\n",
    "\n",
    "\"\"\"\n",
    "param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              {'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'vect__use_idf':[False],\n",
    "               'vect__norm':[None],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              ]\n",
    "\"\"\"\n",
    "\n",
    "small_param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "                     'vect__stop_words': [None],\n",
    "                     'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "                     'clf__penalty': ['l2'],\n",
    "                     'clf__C': [1.0, 10.0]},\n",
    "                    {'vect__ngram_range': [(1, 1)],\n",
    "                     'vect__stop_words': [stop, None],\n",
    "                     'vect__tokenizer': [tokenizer],\n",
    "                     'vect__use_idf':[False],\n",
    "                     'vect__norm':[None],\n",
    "                     'clf__penalty': ['l2'],\n",
    "                  'clf__C': [1.0, 10.0]},\n",
    "              ]\n",
    "\n",
    "lr_tfidf = Pipeline([('vect', tfidf),\n",
    "                     ('clf', LogisticRegression(solver='liblinear'))])\n",
    "\n",
    "gs_lr_tfidf = GridSearchCV(lr_tfidf, small_param_grid,\n",
    "                           scoring='accuracy',\n",
    "                           cv=5,\n",
    "                           verbose=1,\n",
    "                           n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important Note about `n_jobs`**\n",
    "\n",
    "Please note that it is highly recommended to use `n_jobs=-1` (instead of `n_jobs=1`) in the previous code example to utilize all available cores on your machine and speed up the grid search. However, some Windows users reported issues when running the previous code with the `n_jobs=-1` setting related to pickling the tokenizer and tokenizer_porter functions for multiprocessing on Windows. Another workaround would be to replace those two functions, `[tokenizer, tokenizer_porter]`, with `[str.split]`. However, note that the replacement by the simple `str.split` would not support stemming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 8 candidates, totalling 40 fits\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('vect',\n",
       "                                        TfidfVectorizer(lowercase=False)),\n",
       "                                       ('clf',\n",
       "                                        LogisticRegression(solver='liblinear'))]),\n",
       "             n_jobs=-1,\n",
       "             param_grid=[{'clf__C': [1.0, 10.0], 'clf__penalty': ['l2'],\n",
       "                          'vect__ngram_range': [(1, 1)],\n",
       "                          'vect__stop_words': [None],\n",
       "                          'vect__tokenizer': [<function tokenizer at 0x150effb80>,\n",
       "                                              <function tokenizer_porter at 0x150effc10>]},\n",
       "                         {'...\n",
       "                          'vect__stop_words': [['i', 'me', 'my', 'myself', 'we',\n",
       "                                                'our', 'ours', 'ourselves',\n",
       "                                                'you', \"you're\", \"you've\",\n",
       "                                                \"you'll\", \"you'd\", 'your',\n",
       "                                                'yours', 'yourself',\n",
       "                                                'yourselves', 'he', 'him',\n",
       "                                                'his', 'himself', 'she',\n",
       "                                                \"she's\", 'her', 'hers',\n",
       "                                                'herself', 'it', \"it's\", 'its',\n",
       "                                                'itself', ...],\n",
       "                                               None],\n",
       "                          'vect__tokenizer': [<function tokenizer at 0x150effb80>],\n",
       "                          'vect__use_idf': [False]}],\n",
       "             scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_lr_tfidf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameter set: {'clf__C': 10.0, 'clf__penalty': 'l2', 'vect__ngram_range': (1, 1), 'vect__stop_words': None, 'vect__tokenizer': <function tokenizer at 0x150effb80>}\n",
      "CV Accuracy: 0.897\n"
     ]
    }
   ],
   "source": [
    "print(f'Best parameter set: {gs_lr_tfidf.best_params_}')\n",
    "print(f'CV Accuracy: {gs_lr_tfidf.best_score_:.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.899\n"
     ]
    }
   ],
   "source": [
    "clf = gs_lr_tfidf.best_estimator_\n",
    "print(f'Test Accuracy: {clf.score(X_test, y_test):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Start comment:\n",
    "    \n",
    "Please note that `gs_lr_tfidf.best_score_` is the average k-fold cross-validation score. I.e., if we have a `GridSearchCV` object with 5-fold cross-validation (like the one above), the `best_score_` attribute returns the average score over the 5-folds of the best model. To illustrate this with an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.6, 0.4, 0.6, 0.2, 0.6])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "np.random.seed(0)\n",
    "np.set_printoptions(precision=6)\n",
    "y = [np.random.randint(3) for i in range(25)]\n",
    "X = (y + np.random.randn(25)).reshape(-1, 1)\n",
    "\n",
    "cv5_idx = list(StratifiedKFold(n_splits=5, shuffle=False).split(X, y))\n",
    "    \n",
    "lr = LogisticRegression()\n",
    "cross_val_score(lr, X, y, cv=cv5_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By executing the code above, we created a simple data set of random integers that shall represent our class labels. Next, we fed the indices of 5 cross-validation folds (`cv3_idx`) to the `cross_val_score` scorer, which returned 5 accuracy scores -- these are the 5 accuracy values for the 5 test folds.  \n",
    "\n",
    "Next, let us use the `GridSearchCV` object and feed it the same 5 cross-validation sets (via the pre-generated `cv3_idx` indices):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV 1/5] END ..................................., score=0.600 total time=   0.0s\n",
      "[CV 2/5] END ..................................., score=0.400 total time=   0.0s\n",
      "[CV 3/5] END ..................................., score=0.600 total time=   0.0s\n",
      "[CV 4/5] END ..................................., score=0.200 total time=   0.0s\n",
      "[CV 5/5] END ..................................., score=0.600 total time=   0.0s\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "lr = LogisticRegression()\n",
    "gs = GridSearchCV(lr, {}, cv=cv5_idx, verbose=3).fit(X, y) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the scores for the 5 folds are exactly the same as the ones from `cross_val_score` earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the best_score_ attribute of the `GridSearchCV` object, which becomes available after `fit`ting, returns the average accuracy score of the best model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the result above is consistent with the average score computed the `cross_val_score`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "cross_val_score(lr, X, y, cv=cv5_idx).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### End comment.\n",
    "\n",
    "<hr>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with bigger data - online algorithms and out-of-core learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell is not contained in the book but\n",
    "# added for convenience so that the notebook\n",
    "# can be executed starting here, without\n",
    "# executing prior code in this notebook\n",
    "\n",
    "import os\n",
    "import gzip\n",
    "\n",
    "\n",
    "if not os.path.isfile('movie_data.csv'):\n",
    "    if not os.path.isfile('movie_data.csv.gz'):\n",
    "        print('Please place a copy of the movie_data.csv.gz'\n",
    "              'in this directory. You can obtain it by'\n",
    "              'a) executing the code in the beginning of this'\n",
    "              'notebook or b) by downloading it from GitHub:'\n",
    "              'https://github.com/rasbt/python-machine-learning-'\n",
    "              'book-2nd-edition/blob/master/code/ch08/movie_data.csv.gz')\n",
    "    else:\n",
    "        with gzip.open('movie_data.csv.gz', 'rb') as in_f, \\\n",
    "                open('movie_data.csv', 'wb') as out_f:\n",
    "            out_f.write(in_f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "\n",
    "# The `stop` is defined as earlier in this chapter\n",
    "# Added it here for convenience, so that this section\n",
    "# can be run as standalone without executing prior code\n",
    "# in the directory\n",
    "stop = stopwords.words('english')\n",
    "\n",
    "\n",
    "def tokenizer(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', text.lower())\n",
    "    text = re.sub('[\\W]+', ' ', text.lower()) +\\\n",
    "        ' '.join(emoticons).replace('-', '')\n",
    "    tokenized = [w for w in text.split() if w not in stop]\n",
    "    return tokenized\n",
    "\n",
    "\n",
    "def stream_docs(path):\n",
    "    with open(path, 'r', encoding='utf-8') as csv:\n",
    "        next(csv)  # skip header\n",
    "        for line in csv:\n",
    "            text, label = line[:-3], int(line[-2])\n",
    "            yield text, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('\"In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70\\'s, they discover the criminal and a net of power and money to cover the murder.<br /><br />\"\"Murder in Greenwich\"\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.<br /><br />Title (Brazil): Not Available\"',\n",
       " 1)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(stream_docs(path='movie_data.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatch(doc_stream, size):\n",
    "    docs, y = [], []\n",
    "    try:\n",
    "        for _ in range(size):\n",
    "            text, label = next(doc_stream)\n",
    "            docs.append(text)\n",
    "            y.append(label)\n",
    "    except StopIteration:\n",
    "        return None, None\n",
    "    return docs, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "\n",
    "vect = HashingVectorizer(decode_error='ignore', \n",
    "                         n_features=2**21,\n",
    "                         preprocessor=None, \n",
    "                         tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version\n",
    "\n",
    "clf = SGDClassifier(loss='log', random_state=1)\n",
    "\n",
    "\n",
    "doc_stream = stream_docs(path='movie_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:00:16\n"
     ]
    }
   ],
   "source": [
    "import pyprind\n",
    "pbar = pyprind.ProgBar(45)\n",
    "\n",
    "classes = np.array([0, 1])\n",
    "for _ in range(45):\n",
    "    X_train, y_train = get_minibatch(doc_stream, size=1000)\n",
    "    if not X_train:\n",
    "        break\n",
    "    X_train = vect.transform(X_train)\n",
    "    clf.partial_fit(X_train, y_train, classes=classes)\n",
    "    pbar.update()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.868\n"
     ]
    }
   ],
   "source": [
    "X_test, y_test = get_minibatch(doc_stream, size=5000)\n",
    "X_test = vect.transform(X_test)\n",
    "print(f'Accuracy: {clf.score(X_test, y_test):.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = clf.partial_fit(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decomposing text documents with Latent Dirichlet Allocation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent Dirichlet Allocation with scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "\n",
    "# the following is necessary on some computers:\n",
    "df = df.rename(columns={\"0\": \"review\", \"1\": \"sentiment\"})\n",
    "\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer(stop_words='english',\n",
    "                        max_df=.1,\n",
    "                        max_features=5000)\n",
    "X = count.fit_transform(df['review'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "lda = LatentDirichletAllocation(n_components=10,\n",
    "                                random_state=123,\n",
    "                                learning_method='batch')\n",
    "X_topics = lda.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 5000)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.components_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 1:\n",
      "worst minutes awful script stupid\n",
      "Topic 2:\n",
      "family mother father children girl\n",
      "Topic 3:\n",
      "american war dvd music tv\n",
      "Topic 4:\n",
      "human audience cinema art sense\n",
      "Topic 5:\n",
      "police guy car dead murder\n",
      "Topic 6:\n",
      "horror house sex girl woman\n",
      "Topic 7:\n",
      "role performance comedy actor performances\n",
      "Topic 8:\n",
      "series episode war episodes tv\n",
      "Topic 9:\n",
      "book version original read novel\n",
      "Topic 10:\n",
      "action fight guy guys cool\n"
     ]
    }
   ],
   "source": [
    "n_top_words = 5\n",
    "feature_names = count.get_feature_names_out()\n",
    "\n",
    "for topic_idx, topic in enumerate(lda.components_):\n",
    "    print(f'Topic {(topic_idx + 1)}:')\n",
    "    print(' '.join([feature_names[i]\n",
    "                    for i in topic.argsort()\\\n",
    "                        [:-n_top_words - 1:-1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on reading the 5 most important words for each topic, we may guess that the LDA identified the following topics:\n",
    "    \n",
    "1. Generally bad movies (not really a topic category)\n",
    "2. Movies about families\n",
    "3. War movies\n",
    "4. Art movies\n",
    "5. Crime movies\n",
    "6. Horror movies\n",
    "7. Comedies\n",
    "8. Movies somehow related to TV shows\n",
    "9. Movies based on books\n",
    "10. Action movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To confirm that the categories make sense based on the reviews, let's plot 5 movies from the horror movie category (category 6 at index position 5):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Horror movie #1:\n",
      "House of Dracula works from the same basic premise as House of Frankenstein from the year before; namely that Universal's three most famous monsters; Dracula, Frankenstein's Monster and The Wolf Man are appearing in the movie together. Naturally, the film is rather messy therefore, but the fact that ...\n",
      "\n",
      "Horror movie #2:\n",
      "Okay, what the hell kind of TRASH have I been watching now? \"The Witches' Mountain\" has got to be one of the most incoherent and insane Spanish exploitation flicks ever and yet, at the same time, it's also strangely compelling. There's absolutely nothing that makes sense here and I even doubt there  ...\n",
      "\n",
      "Horror movie #3:\n",
      "<br /><br />Horror movie time, Japanese style. Uzumaki/Spiral was a total freakfest from start to finish. A fun freakfest at that, but at times it was a tad too reliant on kitsch rather than the horror. The story is difficult to summarize succinctly: a carefree, normal teenage girl starts coming fac ...\n"
     ]
    }
   ],
   "source": [
    "horror = X_topics[:, 5].argsort()[::-1]\n",
    "\n",
    "for iter_idx, movie_idx in enumerate(horror[:3]):\n",
    "    print(f'\\nHorror movie #{(iter_idx + 1)}:')\n",
    "    print(df['review'][movie_idx][:300], '...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the preceeding code example, we printed the first 300 characters from the top 3 horror movies and indeed, we can see that the reviews -- even though we don't know which exact movie they belong to -- sound like reviews of horror movies, indeed. (However, one might argue that movie #2 could also belong to topic category 1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Readers may ignore the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n",
      "[NbConvertApp] Converting notebook ch08.ipynb to script\n",
      "[NbConvertApp] Writing 24007 bytes to ch08.py\n"
     ]
    }
   ],
   "source": [
    "! python ../.convert_notebook_to_script.py --input ch08.ipynb --output ch08.py"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}