2017-10-24 - 10 Naive Bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Naive Bayes (the easy way)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We'll cheat by using sklearn.naive_bayes to train a spam classifier! Most of the code is just loading our training data into a pandas DataFrame that we can play with. And as a little remark here, we can say that if we'd like to scale that problem up into a big data problem, we could easily do what we do using hdfs datastore to hold our email data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import numpy\n",
    "from pandas import DataFrame\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "# method for reading files from path\n",
    "def readFiles(path):\n",
    "    for root, dirnames, filenames in os.walk(path):\n",
    "        for filename in filenames:\n",
    "            \n",
    "            # join whole file path\n",
    "            path = os.path.join(root, filename)\n",
    "            \n",
    "            # prepare variables\n",
    "            inBody = False\n",
    "            \n",
    "            # start with an empty list of lines\n",
    "            lines = []\n",
    "            \n",
    "            # open file in read mode\n",
    "            f = io.open(path, 'r', encoding='latin1')\n",
    "            \n",
    "            # read line by line; body starts with an empty line\n",
    "            for line in f:\n",
    "                if inBody:\n",
    "                    lines.append(line)\n",
    "                elif line == '\\n':\n",
    "                    inBody = True\n",
    "            f.close()\n",
    "            \n",
    "            # concat all list items to message with linebreaks between lines\n",
    "            message = '\\n'.join(lines)\n",
    "            yield path, message\n",
    "\n",
    "\n",
    "def dataFrameFromDirectory(path, classification):\n",
    "    rows = []\n",
    "    index = []\n",
    "    \n",
    "    # reuse readFiles method\n",
    "    for filename, message in readFiles(path):\n",
    "        rows.append({'message': message, 'class': classification})\n",
    "        index.append(filename)\n",
    "\n",
    "    return DataFrame(rows, index=index)\n",
    "\n",
    "data = DataFrame({'message': [], 'class': []})\n",
    "\n",
    "my_dir = \"C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1\"\n",
    "\n",
    "data = data.append(dataFrameFromDirectory(my_dir + '/emails/spam', 'spam'))\n",
    "data = data.append(dataFrameFromDirectory(my_dir + '/emails/ham', 'ham'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's have a look at that DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1/emails/spam\\00001.7848dde101aa985090474a91ec93fcf0</th>\n",
       "      <td>spam</td>\n",
       "      <td>&lt;!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Tr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1/emails/spam\\00002.d94f1b97e48ed3b553b3508d116e6a09</th>\n",
       "      <td>spam</td>\n",
       "      <td>1) Fight The Risk of Cancer!\\n\\nhttp://www.adc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1/emails/spam\\00003.2ee33bc6eacdb11f38d052c44819ba6c</th>\n",
       "      <td>spam</td>\n",
       "      <td>1) Fight The Risk of Cancer!\\n\\nhttp://www.adc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1/emails/spam\\00004.eac8de8d759b7e74154f142194282724</th>\n",
       "      <td>spam</td>\n",
       "      <td>##############################################...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:/Users/kohleggermichael/OneDrive/Teaching/[WCIS] CRM und Information Mining II/Block-1/emails/spam\\00005.57696a39d7d84318ce497886896bf90d</th>\n",
       "      <td>spam</td>\n",
       "      <td>I thought you might like these:\\n\\n1) Slim Dow...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   class  \\\n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  spam   \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  spam   \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  spam   \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  spam   \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  spam   \n",
       "\n",
       "                                                                                              message  \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  <!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Tr...  \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  1) Fight The Risk of Cancer!\\n\\nhttp://www.adc...  \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  1) Fight The Risk of Cancer!\\n\\nhttp://www.adc...  \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  ##############################################...  \n",
       "C:/Users/kohleggermichael/OneDrive/Teaching/[WC...  I thought you might like these:\\n\\n1) Slim Dow...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we will use a CountVectorizer to **split up each message into its list of words**. The counts object is a sparse matrix with emails on cols, words (aka terms) on rows and word count per email in its cells. As the overall dictionary is huge (429,785.00 words), the matrix gets quite big. Since we cannot access the matrix properly, we simply plot some meta information about it. *Also mind that we have not done any NLP stuff yet (e.g., stemming, stop-word reduction).*\n",
    "\n",
    "![Sparse Matrix Example](img/sparsematrix.png \"Sparse Matrix Example\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3000x62964 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 429785 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# vecotrice message into words with count\n",
    "vectorizer = CountVectorizer()\n",
    "counts = vectorizer.fit_transform(data['message'].values)\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets throw our sparse matrix into a **MultinomialNB (naive bayes)** classifier that also gets the classification (spam/ham) for each handled email. Call fit() and we've got a trained spam filter ready to go! It's just that easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train a multinomial naive bayes classifier on the counts data\n",
    "classifier = MultinomialNB()\n",
    "targets = data['class'].values\n",
    "classifier.fit(counts, targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam', 'ham'], \n",
       "      dtype='<U4')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples = ['Free Viagra now!!!', \"Hi Bob, how about a game of golf tomorrow?\"]\n",
    "example_counts = vectorizer.transform(examples)\n",
    "predictions = classifier.predict(example_counts)\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam', 'spam', 'spam', ..., 'ham', 'ham', 'ham'], \n",
       "      dtype='<U4')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# you could also hand over a larger set of emails (in this case I take the original training data)\n",
    "classifier.predict(vectorizer.transform(data[\"message\"].values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this all mean? This is a pretty good example of a fully implementable classifier, that you could **directly use in an application**. Lets assume that you would implement a guest book app in php. You would probably establish a little form that lets users add a new post to your guest book, and would very likely also attract spammers. Now you could do is go and get some example spam that you could train a classifier with. Then you couls take the trained model and implement it into a function that you can call from your php-submit method and that would return a binary marker (for spam/ham). OnSubmit, you could let the post content run through that method and would immediately get a classification. If that classification is spam, you could reject the post right away."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Our data set is small, so our spam classifier isn't actually very good. Try running some different test emails through it and see if you get the results you expect. If you really want to challenge yourself, try applying train/test to this spam classifier - see how well it can predict some subset of the ham and spam emails."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}