2_naive_bayes_for_text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><h1>Naive Bayes for Text Classification</h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "By The End Of This Session You Should Be Able To:\n",
    "----\n",
    "\n",
    "- Define Text Classification\n",
    "- Apply the Naive Bayes formula to classify text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What is classification?\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Predict a single group from a discrete set of groups.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Puppy or Lion?\n",
    "-----\n",
    "\n",
    "<center><img src=\"images/prediction.png\" width=\"55%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What is Text Classification?\n",
    "---\n",
    "\n",
    "For a given document, predict a single category from a set of pre-defined categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are real world examples of text classification?\n",
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Is an email is \"Spam\" or \"Not Spam\"?\n",
    "- Is a movie review positive 👍 or negative 👎?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Methods for Text Classification\n",
    "-----\n",
    "\n",
    "1. Rules\n",
    "1. Statistical \n",
    "1. Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Rules for Text Classification\n",
    "----\n",
    "\n",
    "Write a series of `if then` statements by hand\n",
    "\n",
    "Examples:\n",
    "\n",
    "- \"Viagra\" or \"Nigerian Prince\" in email would indicate spam\n",
    "- Words such as \"suck\" and \"terrible\" would indicate a bad movie review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are the downsides of rules?\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Time intensive - To be complete, a large set of rules would be needed.\n",
    "1. Brittle - Rules could not easily adapt to news situations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Statistical Models\n",
    "-----\n",
    "\n",
    "Learn from the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Machine Learning for Text Classification\n",
    "------\n",
    "\n",
    "<center><img src=\"images/pipeline.jpg\" width=\"65%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Learning > Static\n",
    "------\n",
    "<center><img src=\"images/google.jpg\" width=\"55%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different Models of The World\n",
    "------\n",
    "<center><img src=\"images/bayesian_evol.png\" width=\"75%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- θ is a model of the world. Just use model, ingore any data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- X is data we observe. Just the data, ingore any model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different Models of The World\n",
    "------\n",
    "<center><img src=\"images/bayesian_evol.png\" width=\"500\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [X | θ] Given a model, which is data is most likely?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [X, θ] What is the joint occurrence of the data and the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [θ | X] Given the data, which model is most likely?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bayes Rule\n",
    "-------\n",
    "<center><img src=\"images/Thomas_Bayes.png\" width=\"75%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why do we want to use Naive Bayes as our ML algoithm?\n",
    "----\n",
    "\n",
    "<center><img src=\"https://imgs.xkcd.com/comics/frequentists_vs_bayesians.png\" width=\"35%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bayes Theorem for Text Classification\n",
    "-----\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Likelihood\n",
    "------\n",
    "\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>\n",
    "\n",
    "The probability of observing the document given a specific label. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Prior\n",
    "------\n",
    "\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>\n",
    "\n",
    "The probability of observing each of the label, ignoring the words. \n",
    "\n",
    "This is the baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Predictor Prior Probability\n",
    "------\n",
    "\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>\n",
    "\n",
    "The probability of observing any given document $p(doc)$ \n",
    "\n",
    "Assumed to be constant, typically dropped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Posterior Probability\n",
    "------\n",
    "\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>\n",
    "\n",
    "The goal - Predict probability of a label given a new document\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "NB Summary\n",
    "------\n",
    "\n",
    "  $$p(\\text{class }| \\text{ doc}) = \\frac{p(\\text{doc } | \\text{ class}) \\times p(\\text{class})}{p(\\text{doc})}$$\n",
    "  \n",
    "\n",
    "- $p(\\text{class }| \\text{ doc})$ is the probability of observing a particular class given a document (**Posterior**)\n",
    "- $p(\\text{doc } | \\text{ class})$ is the probability of observing a document given a particular class (**Likelihood**)\n",
    "- $p(\\text{class})$ is the probability of observing each of the classes (**Prior**)\n",
    "- $p(\\text{doc})$ is the probability of observing a document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dropping p(doc) from the equation\n",
    "-----\n",
    "\n",
    "The probability of observing a document is assumed to be the same for each document. Since it is the same, that term can be dropped.\n",
    "\n",
    "Which reduces the calculation to…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Prediction the class for a new document\n",
    "-------\n",
    "\n",
    "$$ P(\\text{class } |\\text{ doc}) = P(\\text{class }) •  \\prod_{i=1}^n P(word_i | \\text{class })$$\n",
    "\n",
    "$$P(\\text{class }| \\text{ doc}) = P(\\text{class}) \\times (P(word_1 \\text{ | class}) \\cdot P(word_2 \\text{ | class}) \\cdot ... \\cdot P(word_n \\text{ | class}))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Being Naive: Bayes' Greatest Strength and Greatest Weakness\n",
    "------\n",
    "\n",
    "<center><img src=\"images/naive.jpeg\" width=\"65%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Naive: Each word conditionally independent for each label\n",
    "-------\n",
    "\n",
    "<center><img src=\"images/bayes_rule.png\" width=\"70%\"/></center>\n",
    "\n",
    "Each word can predict many labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Naive Bayes works very well\n",
    "----\n",
    "\n",
    "__Despite feature independence assumption__, NB is very common in real world applications\n",
    "\n",
    "- Email filtering\n",
    "- Fraud detection\n",
    "- Medical diagnosis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"images/supervised_learning.png\" width=\"60%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What features might you want to use for text classification? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Words - single or common phrases\n",
    "- Length of document\n",
    "- Author\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What about Deep Learning for Text Classification?\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Deep Learning is state-of-the-art.\n",
    "\n",
    "But sometimes, you don't have enough data or time.\n",
    "\n",
    "Naive Bayes is fast and works with very few examples. It is \"good enough\" for many uses cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Peter Norvig on Naive Bayes at Google\n",
    "-----\n",
    "\n",
    "> And it was fun looking at the comments, because you’d see things like ‘well, I’m throwing in this naive Bayes now, but I’m gonna come back and fix it it up and come up with something better later. And the comment would be from several years ago\n",
    "\n",
    "> … when you have enough data, sometimes, you don’t have to be too clever about coming up with the best algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Summary\n",
    "---\n",
    "\n",
    "- Classification is supervised maching learning to predict a discrete label for data.\n",
    "- Naive Bayes (NB) is among the simplest (and most effective) machine learning classifiers. \n",
    "- Naive Bayes (NB) makes the strong assumption that all features are conditionally independent given the class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}