intro-to-topic-modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Joy of Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matt Burton (@mcburton)\n",
    "\n",
    "- Visiting Assistant Professor \n",
    "- SIS & ULS\n",
    "- PhD from Michigan\n",
    "- Studied Digital Humanities Blogs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Plan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- What you do before you start topic modeling\n",
    "- An intro to generative topic models\n",
    "- What comes out the other end\n",
    "- Then we'll do a thing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Centric Research Workflow\n",
    "![Idealized Workflow](Data-science-workflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reality\n",
    "![Real workflow](real-workflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Work\n",
    "\n",
    "![Data work](data-work.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What **is** topic modeling?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### A method for finding *latent* patterns of co-occurance within large amounts of data (most often, but not always, text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### \"Distant Reading\" - Franco Moretti"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### \"Macroanalysis\" - Matthew Jockers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### \"non-consumptive reading\" - some lawyer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Topic modeling is *not* magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What do I mean when I say \"words?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Words are transformed into numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Documents and sentences are broken down and chopped up into little units called tokens, or unigrams(or ngrams)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  “digital humanities” could be a bigram or two individual unigrams, “digital” and “humanities.”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stopwords are removed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"and\", \"but\", or \"or\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# \"to be or not to be\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stemming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Are “model” and “models”  separate tokens or do we want to treat them as one token?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# We typically don't use stemming when topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What Do I mean when I say \"document?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A document is a bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "doc_one = \"John likes to watch movies. Mary likes movies too.\"\n",
    "doc_two = \"John also likes to watch football games.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the corpus dictionary\n",
    "{\n",
    "    \"John\": 1,\n",
    "    \"likes\": 2,\n",
    "    \"to\": 3,\n",
    "    \"watch\": 4,\n",
    "    \"movies\": 5,\n",
    "    \"also\": 6,\n",
    "    \"football\": 7,\n",
    "    \"games\": 8,\n",
    "    \"Mary\": 9,\n",
    "    \"too\": 10\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the vector representation\n",
    "doc_one_as_vec = [1, 2, 1, 1, 2, 0, 0, 0, 1, 1]\n",
    "doc_two_as_vec = [1, 1, 1, 1, 0, 1, 1, 1, 0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The vector representation is what the computer chews on when topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# In Topic Modeling, documents are composed from a *mixture* of \"topics\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# OK, so what is a \"topic?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A \"topic\" is a probability distribution over words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "![A \"topic\" is a distribution over words](files/word-distribution2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basically a topic is a bag that contains all of the words, but more of some words than others"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# OK, no we can talk about models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The technical term for topic modeling is Latent Dirichlet Allocation or LDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#  LDA models a *generative* process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "-  Select a set of K topics from a probability distribution \n",
    "-  For each document\n",
    "  -  Determine a topic mix\n",
    "  -  For each word\n",
    "    -  Select a topic from the mix\n",
    "    -  Select an observed word from the topic\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# Basically you \"author\" a document by repeatedly selected a word token from a series of different bags-of-words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What I've just described is *not* what actually happens when you *do* topic modeling (it is merely the process being modeled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Computering, or more technically \"estimating\" these numbers with MATH is what happens when you *do* topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The parameters (the various bags-of-words) must be *estimated*\n",
    "![Estimating the model](files/estimating.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The \"parameters\" of the model are the *N* topic distributions and each document's topic mixture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# There are multiple ways to estimate the parameters using *fancy math.* \n",
    "## \"Variational Inference\" & \"Gibbs Sampling\", are the most common. \n",
    "## MALLET uses Gibbs Sampling. [This is a great video of David Mimno explaining how it works](http://journalofdigitalhumanities.org/2-1/the-details-by-david-mimno/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The big pile of numbers you get after you run MALLET, those are the estimated parameters, the distributions the model would use to generate the corpus you have. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Reading these parameters are what scholars use to \"distantly read\" a corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Many scholars focus on the list of top words for each of the k topic distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "0\t2.5\tscience network networks scientific analysis research history social data study publication juggling statistics published information natural time ideas sorts \n",
    "1\t2.5\tstudents learning education student experience teaching model college class building courses ve free thinking general teach graduate approach business \n",
    "2\t2.5\twords word figure collection results literary common language capitalism good interesting corpus lot appears love case period similar women \n",
    "3\t2.5\tarchives day archival viagra archivists society cancer conference information studies heart web american archive treatment prostate october mid blood \n",
    "4\t2.5\ttr report link hcil html shneiderman cs version published umiacs acm plaisant information computer human conference car video proc \n",
    "5\t2.5\thistory project music american research virginia war center university workshop civil part america asian freedom events archive summer september \n",
    "6\t2.5\tdigital humanities work dh scholarship scholars projects scholarly review tools ways project critical criticism technology field process discourse peer \n",
    "7\t2.5\ttopic topics documents modeling models model http april crowdsourcing analysis mallet words accessed document text antiquities lda network set \n",
    "8\t2.5\thistory book human historians historical technology big century "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# There is a tension interpreting these clusters of top words. Are they meaningful clusters or artifacts of the model? of bad data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "3\t2.5\tarchives day archival viagra archivists society cancer conference information studies heart web american archive treatment prostate october mid blood "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# This is why it is really important to go back to the original documents and, if possible, check your validity against some external source "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Ryan Heuser and Long Le-Khac did a good job of this in the Stanford Literary Lab Pamphlet #4 - *\"A Quantitative Literary History of 2,958 Nineteenth-Century British Novels: The Semantic Cohort Method\"*\n",
    "## Also see Alan Liu's discussion of this article in his PMLA article *\"The Meaning of the Digital Humanities\"* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![My network of topics visualization](dh-blog-map-small.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Nice Topic Modeling Links\n",
    "- Andrew Goldstone & Ted Underwood's [dfr-browser](http://agoldst.github.io/dfr-browser/demo/#)\n",
    "\n",
    "- Michael Nelson's [Mining the Dispatch](http://dsl.richmond.edu/dispatch/)\n",
    "\n",
    "- Lisa Rhody's [Revising Ekphrasis](http://www.lisarhody.com/revising-ekphrasis/)\n",
    "\n",
    "- Cameron Blevin's [Topic Modeling Martha Ballard's Diary](http://www.cameronblevins.org/posts/topic-modeling-martha-ballards-diary/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Great Tutorials\n",
    "\n",
    "- [The Joy of Topic Modeling](http://mcburton.net/blog/joy-of-tm/) - *my post that covers this presentation & more*\n",
    "- [Getting Started with Topic Modeling and MALLET from the programing historian](http://programminghistorian.org/lessons/topic-modeling-and-mallet) - *a great set of tutorials for historians*\n",
    "- [Journal of Digital Humanities issue on Topic Modeling](http://journalofdigitalhumanities.org/2-1/) - *This issue is dedicated to discussing topic modeling*\n",
    "- [The Historian's Macroscope - a DH methods book](http://www.themacroscope.org/) - *Another great set of tutorials for historians*\n",
    "- [Topic Modeling a Guided Tour](http://www.scottbot.net/HIAL/?p=19113) - *yet another blog post about topic modeling...Scott has written a lot about topic modeling*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook intro-to-topic-modeling.ipynb to slides\n",
      "[NbConvertApp] Writing 237469 bytes to intro-to-topic-modeling.slides.html\n",
      "[NbConvertApp] Redirecting reveal.js requests to https://cdn.jsdelivr.net/reveal.js/2.6.2\n",
      "Serving your slides at http://127.0.0.1:8000/intro-to-topic-modeling.slides.html\n",
      "Use Control-C to stop this server\n",
      "WARNING:tornado.access:404 GET /custom.css (127.0.0.1) 0.68ms\n",
      "WARNING:tornado.access:404 GET /custom.css (127.0.0.1) 0.96ms\n"
     ]
    }
   ],
   "source": [
    "# ignore me, command to launch slides\n",
    "!ipython nbconvert intro-to-topic-modeling.ipynb --to slides --post serve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}