topic_modeling.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:4c023635c799d6f60d069f05798a5daab940cab88169b0ec9ff9b59a6f28f649"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Your mission:\n",
      "\n",
      "Tell me something interesting about how the The New York Times covers abortion. I've pulled every story that the Times ran that contained the word abortion between 1987 and 2008 which includes 14,269 articles totalling 14,626,680 words (90 MB). This is a lot, so I would recommend restricting your analysis to just those that were nationally focused, which incluces just 5,406 and a mere 5,906,630 words (36 MB). \n",
      "\n",
      "This dataset of articles is not far from the one the data used in [DiMaggio, Paul, Manish Nag, and David Blei. \"Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of US government arts funding.\" Poetics 41.6 (2013): 570-606.](http://www.sciencedirect.com/science/article/pii/S0304422X13000661). This should give you a starting point for thinking about the kinds of things that can be operationalized and the kinds of analysis that can be accomplished.\n",
      "\n",
      "There are two major goals for today, and keep them both in mind as you work on the mission.\n",
      "2. Understanding the benefits and limitations of topic models in the social sciences.\n",
      "1. Mastering the application of topic models in Python.\n",
      "\n",
      "As always, after inspecting the data, attempt to come up with your theoretical informed analysis plan, and then implement it!\n",
      "\n",
      "Teams:  \n",
      "* Taylor and Ali  \n",
      "* Karam and Mary Beth  \n",
      "* Lena and Akram  \n",
      "* Joseph, David And Didem  \n",
      "\n",
      "\n",
      "\n",
      "Be preprared to present your results at 2pm."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline\n",
      "\n",
      "import pandas as pd\n",
      "\n",
      "import numpy as np\n",
      "from sklearn.feature_extraction.text import TfidfVectorizer\n",
      "from sklearn.decomposition import NMF\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Note that this will only work if you have stored the data in the current directory. \n",
      "# You can get the data from  \n",
      "# https://github.com/nealcaren/quant-text-fall-2014/tree/master/times_data\n",
      "# Remember that Pandas can read a file or a URL.\n",
      "\n",
      "df = pd.read_json('times_abortion.json')\n",
      "\n",
      "print 'Article Count:', len(df)\n",
      "print 'Word Count:', df.word_count.sum()\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Article Count: 14269\n",
        "Word Count: 14626680\n"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>byline</th>\n",
        "      <th>dsk</th>\n",
        "      <th>print_page_number</th>\n",
        "      <th>print_section</th>\n",
        "      <th>publication_day_of_month</th>\n",
        "      <th>publication_month</th>\n",
        "      <th>publication_year</th>\n",
        "      <th>text</th>\n",
        "      <th>title</th>\n",
        "      <th>word_count</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1000008</th>\n",
        "      <td> Baker, Russell</td>\n",
        "      <td>    Editorial Desk</td>\n",
        "      <td> 23</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Despite Monica Lewinsky and Stonewall Clinton,...</td>\n",
        "      <td>               Observer; Comfort for the Afflicted</td>\n",
        "      <td>  749</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000090</th>\n",
        "      <td>    Gross, Jane</td>\n",
        "      <td>     National Desk</td>\n",
        "      <td> 17</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Contradicting Sgt. Maj. Gene C. McKinney, Sgt....</td>\n",
        "      <td>      Ex-Top Soldier Is Contradicted By An Accuser</td>\n",
        "      <td>  421</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000104</th>\n",
        "      <td>     Barry, Dan</td>\n",
        "      <td> Metropolitan Desk</td>\n",
        "      <td>  1</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Just two months into his second term, Mayor Ru...</td>\n",
        "      <td> Giuliani Fund-Raising Points To Running for Hi...</td>\n",
        "      <td> 1318</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000208</th>\n",
        "      <td>  Schmitt, Eric</td>\n",
        "      <td>      Foreign Desk</td>\n",
        "      <td>  4</td>\n",
        "      <td> A</td>\n",
        "      <td> 7</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> The Senate committee that controls spending bi...</td>\n",
        "      <td> Senators Back Extra Military Spending but Faul...</td>\n",
        "      <td>  820</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000225</th>\n",
        "      <td> Purdum, Todd S</td>\n",
        "      <td>     National Desk</td>\n",
        "      <td>  6</td>\n",
        "      <td> A</td>\n",
        "      <td> 7</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Californians are long used to the televised cu...</td>\n",
        "      <td>        Interest Groups Run Own Race in California</td>\n",
        "      <td> 1169</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "                 byline                dsk  print_page_number print_section  \\\n",
        "1000008  Baker, Russell     Editorial Desk                 23             A   \n",
        "1000090     Gross, Jane      National Desk                 17             A   \n",
        "1000104      Barry, Dan  Metropolitan Desk                  1             A   \n",
        "1000208   Schmitt, Eric       Foreign Desk                  4             A   \n",
        "1000225  Purdum, Todd S      National Desk                  6             A   \n",
        "\n",
        "         publication_day_of_month  publication_month  publication_year  \\\n",
        "1000008                         6                  3              1998   \n",
        "1000090                         6                  3              1998   \n",
        "1000104                         6                  3              1998   \n",
        "1000208                         7                  3              1998   \n",
        "1000225                         7                  3              1998   \n",
        "\n",
        "                                                      text  \\\n",
        "1000008  Despite Monica Lewinsky and Stonewall Clinton,...   \n",
        "1000090  Contradicting Sgt. Maj. Gene C. McKinney, Sgt....   \n",
        "1000104  Just two months into his second term, Mayor Ru...   \n",
        "1000208  The Senate committee that controls spending bi...   \n",
        "1000225  Californians are long used to the televised cu...   \n",
        "\n",
        "                                                     title  word_count  \n",
        "1000008                Observer; Comfort for the Afflicted         749  \n",
        "1000090       Ex-Top Soldier Is Contradicted By An Accuser         421  \n",
        "1000104  Giuliani Fund-Raising Points To Running for Hi...        1318  \n",
        "1000208  Senators Back Extra Military Spending but Faul...         820  \n",
        "1000225         Interest Groups Run Own Race in California        1169  "
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dfn = pd.read_json('times_national_abortion.json')\n",
      "\n",
      "print 'Article Count:', len(dfn)\n",
      "print 'Word Count:', dfn.word_count.sum()\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Article Count: 5406\n",
        "Word Count: 5906630\n"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>byline</th>\n",
        "      <th>dsk</th>\n",
        "      <th>print_page_number</th>\n",
        "      <th>print_section</th>\n",
        "      <th>publication_day_of_month</th>\n",
        "      <th>publication_month</th>\n",
        "      <th>publication_year</th>\n",
        "      <th>text</th>\n",
        "      <th>title</th>\n",
        "      <th>word_count</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1000008</th>\n",
        "      <td> Baker, Russell</td>\n",
        "      <td>    Editorial Desk</td>\n",
        "      <td> 23</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Despite Monica Lewinsky and Stonewall Clinton,...</td>\n",
        "      <td>               Observer; Comfort for the Afflicted</td>\n",
        "      <td>  749</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000090</th>\n",
        "      <td>    Gross, Jane</td>\n",
        "      <td>     National Desk</td>\n",
        "      <td> 17</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Contradicting Sgt. Maj. Gene C. McKinney, Sgt....</td>\n",
        "      <td>      Ex-Top Soldier Is Contradicted By An Accuser</td>\n",
        "      <td>  421</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000104</th>\n",
        "      <td>     Barry, Dan</td>\n",
        "      <td> Metropolitan Desk</td>\n",
        "      <td>  1</td>\n",
        "      <td> A</td>\n",
        "      <td> 6</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Just two months into his second term, Mayor Ru...</td>\n",
        "      <td> Giuliani Fund-Raising Points To Running for Hi...</td>\n",
        "      <td> 1318</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000208</th>\n",
        "      <td>  Schmitt, Eric</td>\n",
        "      <td>      Foreign Desk</td>\n",
        "      <td>  4</td>\n",
        "      <td> A</td>\n",
        "      <td> 7</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> The Senate committee that controls spending bi...</td>\n",
        "      <td> Senators Back Extra Military Spending but Faul...</td>\n",
        "      <td>  820</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1000225</th>\n",
        "      <td> Purdum, Todd S</td>\n",
        "      <td>     National Desk</td>\n",
        "      <td>  6</td>\n",
        "      <td> A</td>\n",
        "      <td> 7</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1998</td>\n",
        "      <td> Californians are long used to the televised cu...</td>\n",
        "      <td>        Interest Groups Run Own Race in California</td>\n",
        "      <td> 1169</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "                 byline                dsk  print_page_number print_section  \\\n",
        "1000008  Baker, Russell     Editorial Desk                 23             A   \n",
        "1000090     Gross, Jane      National Desk                 17             A   \n",
        "1000104      Barry, Dan  Metropolitan Desk                  1             A   \n",
        "1000208   Schmitt, Eric       Foreign Desk                  4             A   \n",
        "1000225  Purdum, Todd S      National Desk                  6             A   \n",
        "\n",
        "         publication_day_of_month  publication_month  publication_year  \\\n",
        "1000008                         6                  3              1998   \n",
        "1000090                         6                  3              1998   \n",
        "1000104                         6                  3              1998   \n",
        "1000208                         7                  3              1998   \n",
        "1000225                         7                  3              1998   \n",
        "\n",
        "                                                      text  \\\n",
        "1000008  Despite Monica Lewinsky and Stonewall Clinton,...   \n",
        "1000090  Contradicting Sgt. Maj. Gene C. McKinney, Sgt....   \n",
        "1000104  Just two months into his second term, Mayor Ru...   \n",
        "1000208  The Senate committee that controls spending bi...   \n",
        "1000225  Californians are long used to the televised cu...   \n",
        "\n",
        "                                                     title  word_count  \n",
        "1000008                Observer; Comfort for the Afflicted         749  \n",
        "1000090       Ex-Top Soldier Is Contradicted By An Accuser         421  \n",
        "1000104  Giuliani Fund-Raising Points To Running for Hi...        1318  \n",
        "1000208  Senators Back Extra Military Spending but Faul...         820  \n",
        "1000225         Interest Groups Run Own Race in California        1169  "
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# LDA uses frequencies, but I find that NMF likes Tf-Idf better.\n",
      "\n",
      "vectorizer = TfidfVectorizer(max_df=0.7, \n",
      "                             min_df=5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# This may not be quick. The whole notebook takes my relatively modern MacBook Air 5 minutes to run.\n",
      "\n",
      "vectorizer.fit(df['text'])\n",
      "tfidf = vectorizer.transform(df['text'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 52
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# This will not be quick. Note that more topics = more time.\n",
      "\n",
      "number_of_topics = 20\n",
      "\n",
      "nmf = NMF(n_components = number_of_topics, max_iter = 5000).fit(tfidf)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 53
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Print out most informative features \n",
      "# This also stores the topics in a list topic_labels\n",
      "\n",
      "feature_names = vectorizer.get_feature_names()\n",
      "topic_labels = []\n",
      "\n",
      "for topic_idx, topic in enumerate(nmf.components_):\n",
      "    print(\"Topic #%d\" % topic_idx)\n",
      "    values = [topic[item] for  item in topic.argsort()[:-10 - 1:-1]]\n",
      "    words = [feature_names[i] for i in topic.argsort()[:-10 - 1:-1]]\n",
      "    print '-','\\n- '.join(['%s (%.2f)' % c for c in zip(words,values)])\n",
      "    topic_labels.append(\" \".join([feature_names[i]\n",
      "                    for i in topic.argsort()[:-3 - 1:-1]]))\n",
      "    print ''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Topic #0:\n",
        "- gore (2.68)\n",
        "- mr (1.12)\n",
        "- bradley (0.88)\n",
        "- clinton (0.71)\n",
        "- al (0.39)\n",
        "- campaign (0.38)\n",
        "- vice (0.36)\n",
        "- president (0.27)\n",
        "- nader (0.26)\n",
        "- lieberman (0.22)\n",
        "\n",
        "Topic #1:\n",
        "- she (2.57)\n",
        "- her (2.30)\n",
        "- ms (0.98)\n",
        "- mother (0.28)\n",
        "- mrs (0.26)\n",
        "- woman (0.25)\n",
        "- husband (0.22)\n",
        "- herself (0.18)\n",
        "- baby (0.18)\n",
        "- when (0.17)\n",
        "\n",
        "Topic #2:\n",
        "- court (2.22)\n",
        "- justice (1.13)\n",
        "- law (0.75)\n",
        "- supreme (0.74)\n",
        "- justices (0.64)\n",
        "- case (0.61)\n",
        "- decision (0.51)\n",
        "- roe (0.43)\n",
        "- state (0.40)\n",
        "- ruling (0.39)\n",
        "\n",
        "Topic #3:\n",
        "- church (1.55)\n",
        "- catholic (1.27)\n",
        "- pope (1.09)\n",
        "- cardinal (0.96)\n",
        "- bishops (0.77)\n",
        "- bishop (0.76)\n",
        "- catholics (0.60)\n",
        "- vatican (0.53)\n",
        "- archbishop (0.47)\n",
        "- roman (0.39)\n",
        "\n",
        "Topic #4:\n",
        "- mrs (1.92)\n",
        "- whitman (1.79)\n",
        "- jersey (0.73)\n",
        "- governor (0.66)\n",
        "- she (0.64)\n",
        "- lazio (0.61)\n",
        "- her (0.59)\n",
        "- clinton (0.50)\n",
        "- state (0.38)\n",
        "- mcgreevey (0.34)\n",
        "\n",
        "Topic #5:\n",
        "- you (1.41)\n",
        "- we (1.12)\n",
        "- what (0.68)\n",
        "- my (0.65)\n",
        "- people (0.61)\n",
        "- there (0.57)\n",
        "- do (0.54)\n",
        "- our (0.54)\n",
        "- can (0.52)\n",
        "- so (0.50)\n",
        "\n",
        "Topic #6:\n",
        "- party (1.88)\n",
        "- republican (1.77)\n",
        "- republicans (1.19)\n",
        "- democrats (1.01)\n",
        "- democratic (0.93)\n",
        "- state (0.66)\n",
        "- voters (0.64)\n",
        "- percent (0.59)\n",
        "- election (0.55)\n",
        "- candidates (0.53)\n",
        "\n",
        "Topic #7:\n",
        "- clinic (1.78)\n",
        "- clinics (1.22)\n",
        "- protesters (0.82)\n",
        "- rescue (0.81)\n",
        "- operation (0.73)\n",
        "- anti (0.68)\n",
        "- police (0.65)\n",
        "- arrested (0.49)\n",
        "- federal (0.48)\n",
        "- were (0.42)\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Topic #8:\n",
        "- dole (3.22)\n",
        "- mr (1.31)\n",
        "- buchanan (0.98)\n",
        "- clinton (0.64)\n",
        "- kemp (0.42)\n",
        "- bob (0.39)\n",
        "- gramm (0.39)\n",
        "- convention (0.38)\n",
        "- party (0.34)\n",
        "- republican (0.32)\n",
        "\n",
        "Topic #9:\n",
        "- bill (1.75)\n",
        "- house (1.30)\n",
        "- senate (1.00)\n",
        "- clinton (0.91)\n",
        "- veto (0.68)\n",
        "- congress (0.68)\n",
        "- measure (0.63)\n",
        "- legislation (0.58)\n",
        "- spending (0.58)\n",
        "- president (0.54)\n",
        "\n",
        "Topic #10:\n",
        "- 95 (2.46)\n",
        "- 25 (0.57)\n",
        "- 24 (0.55)\n",
        "- novel (0.44)\n",
        "- book (0.43)\n",
        "- 11 (0.38)\n",
        "- books (0.36)\n",
        "- 12 (0.35)\n",
        "- schuster (0.34)\n",
        "- 14 (0.34)\n",
        "\n",
        "Topic #11:\n",
        "- percent (0.92)\n",
        "- children (0.76)\n",
        "- health (0.75)\n",
        "- teen (0.68)\n",
        "- care (0.68)\n",
        "- school (0.54)\n",
        "- state (0.53)\n",
        "- education (0.52)\n",
        "- parents (0.49)\n",
        "- welfare (0.46)\n",
        "\n",
        "Topic #12:\n",
        "- judge (2.84)\n",
        "- bork (1.52)\n",
        "- souter (0.73)\n",
        "- thomas (0.71)\n",
        "- nomination (0.64)\n",
        "- court (0.63)\n",
        "- confirmation (0.56)\n",
        "- senate (0.54)\n",
        "- senator (0.51)\n",
        "- committee (0.50)\n",
        "\n",
        "Topic #13:\n",
        "- women (2.96)\n",
        "- abortions (0.65)\n",
        "- woman (0.53)\n",
        "- men (0.47)\n",
        "- rights (0.46)\n",
        "- editor (0.41)\n",
        "- pro (0.40)\n",
        "- choice (0.40)\n",
        "- life (0.36)\n",
        "- pregnancy (0.32)\n",
        "\n",
        "Topic #14:\n",
        "- bush (3.90)\n",
        "- mr (1.64)\n",
        "- president (0.77)\n",
        "- mccain (0.69)\n",
        "- reagan (0.44)\n",
        "- george (0.42)\n",
        "- dukakis (0.41)\n",
        "- campaign (0.38)\n",
        "- quayle (0.35)\n",
        "- vice (0.28)\n",
        "\n",
        "Topic #15:\n",
        "- drug (1.81)\n",
        "- ru (1.63)\n",
        "- 486 (1.62)\n",
        "- pill (1.08)\n",
        "- roussel (0.69)\n",
        "- company (0.54)\n",
        "- mifepristone (0.51)\n",
        "- french (0.45)\n",
        "- uclaf (0.38)\n",
        "- surgical (0.38)\n",
        "\n",
        "Topic #16:\n",
        "- population (1.21)\n",
        "- china (1.12)\n",
        "- nations (1.11)\n",
        "- united (1.06)\n",
        "- planning (1.00)\n",
        "- family (0.73)\n",
        "- countries (0.67)\n",
        "- chinese (0.66)\n",
        "- international (0.66)\n",
        "- world (0.58)\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Topic #17:\n",
        "- giuliani (3.00)\n",
        "- dinkins (1.33)\n",
        "- mayor (1.26)\n",
        "- mr (1.09)\n",
        "- city (0.81)\n",
        "- york (0.53)\n",
        "- lazio (0.43)\n",
        "- koch (0.38)\n",
        "- yesterday (0.33)\n",
        "- rudolph (0.32)\n",
        "\n",
        "Topic #18:\n",
        "- mr (3.83)\n",
        "- pataki (0.65)\n",
        "- amato (0.64)\n",
        "- campaign (0.56)\n",
        "- state (0.46)\n",
        "- cuomo (0.43)\n",
        "- governor (0.43)\n",
        "- him (0.41)\n",
        "- race (0.32)\n",
        "- senator (0.32)\n",
        "\n",
        "Topic #19:\n",
        "- dr (2.81)\n",
        "- medical (0.80)\n",
        "- cells (0.64)\n",
        "- doctors (0.59)\n",
        "- research (0.58)\n",
        "- foster (0.57)\n",
        "- fetal (0.57)\n",
        "- health (0.53)\n",
        "- tissue (0.52)\n",
        "- patients (0.46)\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Get some predicted values. Takes a tfidf.\n",
      "\n",
      "y_hat = nmf.transform(tfidf)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# get the most likely topic and add it as a column\n",
      "topics = []\n",
      "for row in y_hat:\n",
      "    topics.append(topic_labels[np.argsort(row)[0]])\n",
      "    \n",
      "df['top_topic'] = topics"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# get the full suite of predictions as values\n",
      "# This is normalized so that the topic values for a given article sum to 1\n",
      "\n",
      "y_norm = [y/ y.sum() for y in y_hat] #Normalize\n",
      "\n",
      "df_y_hat = pd.DataFrame(y_norm, \n",
      "                        columns=topic_labels,  # Use our informative topic names\n",
      "                        index=df.index.values) # Makes merge easier\n",
      "\n",
      "df_topics = df.join(df_y_hat) # This is a new data set, so you can remerge"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#an example of the stats could do\n",
      "\n",
      "for topic in topic_labels:\n",
      "    print df_topics[topic].mean(), topic"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.0358335251903 gore mr bradley\n",
        "0.0678877400687 court justice law\n",
        "0.0508140434073 dr medical health\n",
        "0.0596895877918 church catholic pope\n",
        "0.0571415795769 party republican platform\n",
        "0.0281974330593 95 25 24\n",
        "0.0512134159213 bush mr president\n",
        "0.0340913871595 judge bork souter\n",
        "0.0640827560929 house clinton senate\n",
        "0.0347392110001 dole mr buchanan\n",
        "0.0546154243498 population planning nations\n",
        "0.0767100844845 mr republican state\n",
        "0.0288833196368 cells research tissue\n",
        "0.0475806548793 clinic clinics protesters\n",
        "0.0269208243834 mrs whitman jersey\n",
        "0.0681708102034 women ms abortions\n",
        "0.0211392526363 drug ru 486\n",
        "0.026087049339 giuliani mr dinkins\n",
        "0.123480218835 people ms like\n",
        "0.0427216819846 mr pataki amato\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_topics.groupby('publication_year').mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>print_page_number</th>\n",
        "      <th>publication_day_of_month</th>\n",
        "      <th>publication_month</th>\n",
        "      <th>word_count</th>\n",
        "      <th>gore mr bradley</th>\n",
        "      <th>court justice law</th>\n",
        "      <th>dr medical health</th>\n",
        "      <th>church catholic pope</th>\n",
        "      <th>party republican platform</th>\n",
        "      <th>95 25 24</th>\n",
        "      <th>...</th>\n",
        "      <th>population planning nations</th>\n",
        "      <th>mr republican state</th>\n",
        "      <th>cells research tissue</th>\n",
        "      <th>clinic clinics protesters</th>\n",
        "      <th>mrs whitman jersey</th>\n",
        "      <th>women ms abortions</th>\n",
        "      <th>drug ru 486</th>\n",
        "      <th>giuliani mr dinkins</th>\n",
        "      <th>people ms like</th>\n",
        "      <th>mr pataki amato</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>publication_year</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1987</th>\n",
        "      <td> 16.078125</td>\n",
        "      <td> 15.440625</td>\n",
        "      <td> 6.889062</td>\n",
        "      <td> 1037.596875</td>\n",
        "      <td> 0.013062</td>\n",
        "      <td> 0.097396</td>\n",
        "      <td> 0.060587</td>\n",
        "      <td> 0.092524</td>\n",
        "      <td> 0.033881</td>\n",
        "      <td> 0.024220</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.081611</td>\n",
        "      <td> 0.040472</td>\n",
        "      <td> 0.031659</td>\n",
        "      <td> 0.037868</td>\n",
        "      <td> 0.011171</td>\n",
        "      <td> 0.055880</td>\n",
        "      <td> 0.014596</td>\n",
        "      <td> 0.016393</td>\n",
        "      <td> 0.124166</td>\n",
        "      <td> 0.022838</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1988</th>\n",
        "      <td> 16.131772</td>\n",
        "      <td> 15.781845</td>\n",
        "      <td> 7.007321</td>\n",
        "      <td>  919.093704</td>\n",
        "      <td> 0.018898</td>\n",
        "      <td> 0.079396</td>\n",
        "      <td> 0.053036</td>\n",
        "      <td> 0.053688</td>\n",
        "      <td> 0.053024</td>\n",
        "      <td> 0.027054</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.072635</td>\n",
        "      <td> 0.068815</td>\n",
        "      <td> 0.038239</td>\n",
        "      <td> 0.050122</td>\n",
        "      <td> 0.013610</td>\n",
        "      <td> 0.065434</td>\n",
        "      <td> 0.038249</td>\n",
        "      <td> 0.013891</td>\n",
        "      <td> 0.119832</td>\n",
        "      <td> 0.025828</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1989</th>\n",
        "      <td> 15.372093</td>\n",
        "      <td> 15.536906</td>\n",
        "      <td> 7.145602</td>\n",
        "      <td>  922.744186</td>\n",
        "      <td> 0.010631</td>\n",
        "      <td> 0.130093</td>\n",
        "      <td> 0.067381</td>\n",
        "      <td> 0.046344</td>\n",
        "      <td> 0.031252</td>\n",
        "      <td> 0.026215</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.056746</td>\n",
        "      <td> 0.072783</td>\n",
        "      <td> 0.034361</td>\n",
        "      <td> 0.056964</td>\n",
        "      <td> 0.018407</td>\n",
        "      <td> 0.077171</td>\n",
        "      <td> 0.022146</td>\n",
        "      <td> 0.062115</td>\n",
        "      <td> 0.126975</td>\n",
        "      <td> 0.027854</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1990</th>\n",
        "      <td> 14.048096</td>\n",
        "      <td> 15.927856</td>\n",
        "      <td> 6.660321</td>\n",
        "      <td>  993.942886</td>\n",
        "      <td> 0.012016</td>\n",
        "      <td> 0.085574</td>\n",
        "      <td> 0.045054</td>\n",
        "      <td> 0.102904</td>\n",
        "      <td> 0.044839</td>\n",
        "      <td> 0.023684</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.052115</td>\n",
        "      <td> 0.114025</td>\n",
        "      <td> 0.026436</td>\n",
        "      <td> 0.044151</td>\n",
        "      <td> 0.014907</td>\n",
        "      <td> 0.069399</td>\n",
        "      <td> 0.023694</td>\n",
        "      <td> 0.014248</td>\n",
        "      <td> 0.121351</td>\n",
        "      <td> 0.043393</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1991</th>\n",
        "      <td> 13.740741</td>\n",
        "      <td> 15.361111</td>\n",
        "      <td> 7.291005</td>\n",
        "      <td>  875.859788</td>\n",
        "      <td> 0.009539</td>\n",
        "      <td> 0.102363</td>\n",
        "      <td> 0.062877</td>\n",
        "      <td> 0.063908</td>\n",
        "      <td> 0.026945</td>\n",
        "      <td> 0.020312</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.064820</td>\n",
        "      <td> 0.044731</td>\n",
        "      <td> 0.026793</td>\n",
        "      <td> 0.062423</td>\n",
        "      <td> 0.011360</td>\n",
        "      <td> 0.077972</td>\n",
        "      <td> 0.028532</td>\n",
        "      <td> 0.010961</td>\n",
        "      <td> 0.126635</td>\n",
        "      <td> 0.027325</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1992</th>\n",
        "      <td> 13.712329</td>\n",
        "      <td> 15.927478</td>\n",
        "      <td> 6.970185</td>\n",
        "      <td> 1002.409347</td>\n",
        "      <td> 0.043044</td>\n",
        "      <td> 0.071893</td>\n",
        "      <td> 0.030479</td>\n",
        "      <td> 0.033666</td>\n",
        "      <td> 0.076023</td>\n",
        "      <td> 0.019914</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.042105</td>\n",
        "      <td> 0.086067</td>\n",
        "      <td> 0.031344</td>\n",
        "      <td> 0.041717</td>\n",
        "      <td> 0.019782</td>\n",
        "      <td> 0.085820</td>\n",
        "      <td> 0.019573</td>\n",
        "      <td> 0.016465</td>\n",
        "      <td> 0.131338</td>\n",
        "      <td> 0.061834</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1993</th>\n",
        "      <td> 13.840438</td>\n",
        "      <td> 15.993910</td>\n",
        "      <td> 6.456760</td>\n",
        "      <td> 1103.561510</td>\n",
        "      <td> 0.024206</td>\n",
        "      <td> 0.070857</td>\n",
        "      <td> 0.077052</td>\n",
        "      <td> 0.056207</td>\n",
        "      <td> 0.036414</td>\n",
        "      <td> 0.034490</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.057861</td>\n",
        "      <td> 0.044883</td>\n",
        "      <td> 0.015811</td>\n",
        "      <td> 0.064460</td>\n",
        "      <td> 0.025750</td>\n",
        "      <td> 0.076564</td>\n",
        "      <td> 0.023507</td>\n",
        "      <td> 0.056577</td>\n",
        "      <td> 0.155919</td>\n",
        "      <td> 0.028816</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1994</th>\n",
        "      <td> 12.462985</td>\n",
        "      <td> 14.862515</td>\n",
        "      <td> 7.203290</td>\n",
        "      <td> 1082.405405</td>\n",
        "      <td> 0.023328</td>\n",
        "      <td> 0.064905</td>\n",
        "      <td> 0.064554</td>\n",
        "      <td> 0.056195</td>\n",
        "      <td> 0.050726</td>\n",
        "      <td> 0.023520</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.072791</td>\n",
        "      <td> 0.079209</td>\n",
        "      <td> 0.014948</td>\n",
        "      <td> 0.080529</td>\n",
        "      <td> 0.026017</td>\n",
        "      <td> 0.069725</td>\n",
        "      <td> 0.017058</td>\n",
        "      <td> 0.023994</td>\n",
        "      <td> 0.110916</td>\n",
        "      <td> 0.092517</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1995</th>\n",
        "      <td> 13.065410</td>\n",
        "      <td> 14.683278</td>\n",
        "      <td> 6.763012</td>\n",
        "      <td> 1053.384275</td>\n",
        "      <td> 0.020591</td>\n",
        "      <td> 0.044787</td>\n",
        "      <td> 0.083096</td>\n",
        "      <td> 0.076079</td>\n",
        "      <td> 0.068480</td>\n",
        "      <td> 0.024624</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.058084</td>\n",
        "      <td> 0.051223</td>\n",
        "      <td> 0.016017</td>\n",
        "      <td> 0.068693</td>\n",
        "      <td> 0.026639</td>\n",
        "      <td> 0.053164</td>\n",
        "      <td> 0.012264</td>\n",
        "      <td> 0.015796</td>\n",
        "      <td> 0.125740</td>\n",
        "      <td> 0.033118</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1996</th>\n",
        "      <td> 13.109336</td>\n",
        "      <td> 14.906644</td>\n",
        "      <td> 6.521447</td>\n",
        "      <td> 1105.935240</td>\n",
        "      <td> 0.025712</td>\n",
        "      <td> 0.036355</td>\n",
        "      <td> 0.029837</td>\n",
        "      <td> 0.038603</td>\n",
        "      <td> 0.109390</td>\n",
        "      <td> 0.019184</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.040071</td>\n",
        "      <td> 0.097379</td>\n",
        "      <td> 0.019533</td>\n",
        "      <td> 0.031189</td>\n",
        "      <td> 0.031042</td>\n",
        "      <td> 0.048318</td>\n",
        "      <td> 0.020521</td>\n",
        "      <td> 0.016263</td>\n",
        "      <td> 0.118087</td>\n",
        "      <td> 0.036658</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1997</th>\n",
        "      <td> 13.075931</td>\n",
        "      <td> 15.842407</td>\n",
        "      <td> 6.978510</td>\n",
        "      <td> 1094.520057</td>\n",
        "      <td> 0.022722</td>\n",
        "      <td> 0.059533</td>\n",
        "      <td> 0.060747</td>\n",
        "      <td> 0.065003</td>\n",
        "      <td> 0.049072</td>\n",
        "      <td> 0.025290</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.066867</td>\n",
        "      <td> 0.056209</td>\n",
        "      <td> 0.036049</td>\n",
        "      <td> 0.044989</td>\n",
        "      <td> 0.054031</td>\n",
        "      <td> 0.080167</td>\n",
        "      <td> 0.023658</td>\n",
        "      <td> 0.025400</td>\n",
        "      <td> 0.148564</td>\n",
        "      <td> 0.042747</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1998</th>\n",
        "      <td> 12.952333</td>\n",
        "      <td> 15.615619</td>\n",
        "      <td> 6.907708</td>\n",
        "      <td> 1056.229209</td>\n",
        "      <td> 0.027463</td>\n",
        "      <td> 0.034178</td>\n",
        "      <td> 0.057726</td>\n",
        "      <td> 0.045059</td>\n",
        "      <td> 0.061258</td>\n",
        "      <td> 0.023378</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.055877</td>\n",
        "      <td> 0.115560</td>\n",
        "      <td> 0.023804</td>\n",
        "      <td> 0.054153</td>\n",
        "      <td> 0.032660</td>\n",
        "      <td> 0.069413</td>\n",
        "      <td> 0.015702</td>\n",
        "      <td> 0.025967</td>\n",
        "      <td> 0.132262</td>\n",
        "      <td> 0.079051</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1999</th>\n",
        "      <td> 13.520223</td>\n",
        "      <td> 16.460251</td>\n",
        "      <td> 6.859135</td>\n",
        "      <td> 1233.340307</td>\n",
        "      <td> 0.051156</td>\n",
        "      <td> 0.055697</td>\n",
        "      <td> 0.035389</td>\n",
        "      <td> 0.066548</td>\n",
        "      <td> 0.062988</td>\n",
        "      <td> 0.022525</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.061454</td>\n",
        "      <td> 0.062307</td>\n",
        "      <td> 0.029933</td>\n",
        "      <td> 0.042405</td>\n",
        "      <td> 0.046123</td>\n",
        "      <td> 0.050060</td>\n",
        "      <td> 0.018823</td>\n",
        "      <td> 0.032756</td>\n",
        "      <td> 0.140397</td>\n",
        "      <td> 0.049157</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2000</th>\n",
        "      <td> 14.455581</td>\n",
        "      <td> 15.667426</td>\n",
        "      <td> 6.397115</td>\n",
        "      <td> 1153.927866</td>\n",
        "      <td> 0.160920</td>\n",
        "      <td> 0.047862</td>\n",
        "      <td> 0.031540</td>\n",
        "      <td> 0.054476</td>\n",
        "      <td> 0.069495</td>\n",
        "      <td> 0.016007</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.031736</td>\n",
        "      <td> 0.088508</td>\n",
        "      <td> 0.019501</td>\n",
        "      <td> 0.018913</td>\n",
        "      <td> 0.043970</td>\n",
        "      <td> 0.042960</td>\n",
        "      <td> 0.022533</td>\n",
        "      <td> 0.034758</td>\n",
        "      <td> 0.097493</td>\n",
        "      <td> 0.031331</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2001</th>\n",
        "      <td> 12.358736</td>\n",
        "      <td> 15.568773</td>\n",
        "      <td> 4.355019</td>\n",
        "      <td> 1026.721190</td>\n",
        "      <td> 0.024562</td>\n",
        "      <td> 0.051293</td>\n",
        "      <td> 0.042686</td>\n",
        "      <td> 0.050982</td>\n",
        "      <td> 0.040053</td>\n",
        "      <td> 0.040324</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.054684</td>\n",
        "      <td> 0.082735</td>\n",
        "      <td> 0.062117</td>\n",
        "      <td> 0.047477</td>\n",
        "      <td> 0.035458</td>\n",
        "      <td> 0.057140</td>\n",
        "      <td> 0.017795</td>\n",
        "      <td> 0.024889</td>\n",
        "      <td> 0.100591</td>\n",
        "      <td> 0.049060</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2002</th>\n",
        "      <td> 14.284483</td>\n",
        "      <td> 15.577586</td>\n",
        "      <td> 6.672414</td>\n",
        "      <td>  944.560345</td>\n",
        "      <td> 0.026272</td>\n",
        "      <td> 0.063409</td>\n",
        "      <td> 0.052596</td>\n",
        "      <td> 0.064712</td>\n",
        "      <td> 0.040770</td>\n",
        "      <td> 0.026255</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.071979</td>\n",
        "      <td> 0.146718</td>\n",
        "      <td> 0.042075</td>\n",
        "      <td> 0.056218</td>\n",
        "      <td> 0.029891</td>\n",
        "      <td> 0.094442</td>\n",
        "      <td> 0.010625</td>\n",
        "      <td> 0.017899</td>\n",
        "      <td> 0.097881</td>\n",
        "      <td> 0.036671</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2003</th>\n",
        "      <td> 16.865546</td>\n",
        "      <td> 14.554622</td>\n",
        "      <td> 6.739496</td>\n",
        "      <td>  555.067227</td>\n",
        "      <td> 0.009983</td>\n",
        "      <td> 0.055296</td>\n",
        "      <td> 0.054321</td>\n",
        "      <td> 0.039519</td>\n",
        "      <td> 0.053010</td>\n",
        "      <td> 0.042566</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.062391</td>\n",
        "      <td> 0.040871</td>\n",
        "      <td> 0.079818</td>\n",
        "      <td> 0.066128</td>\n",
        "      <td> 0.012550</td>\n",
        "      <td> 0.159312</td>\n",
        "      <td> 0.014042</td>\n",
        "      <td> 0.015675</td>\n",
        "      <td> 0.132637</td>\n",
        "      <td> 0.020225</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2004</th>\n",
        "      <td> 13.968750</td>\n",
        "      <td> 15.895833</td>\n",
        "      <td> 7.442708</td>\n",
        "      <td>  583.322917</td>\n",
        "      <td> 0.012243</td>\n",
        "      <td> 0.059577</td>\n",
        "      <td> 0.031000</td>\n",
        "      <td> 0.170174</td>\n",
        "      <td> 0.057674</td>\n",
        "      <td> 0.031449</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.033617</td>\n",
        "      <td> 0.068182</td>\n",
        "      <td> 0.079784</td>\n",
        "      <td> 0.032475</td>\n",
        "      <td> 0.019164</td>\n",
        "      <td> 0.114356</td>\n",
        "      <td> 0.027721</td>\n",
        "      <td> 0.010045</td>\n",
        "      <td> 0.107228</td>\n",
        "      <td> 0.017171</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2005</th>\n",
        "      <td> 17.229730</td>\n",
        "      <td> 14.310811</td>\n",
        "      <td> 7.198198</td>\n",
        "      <td>  626.756757</td>\n",
        "      <td> 0.010039</td>\n",
        "      <td> 0.090207</td>\n",
        "      <td> 0.030611</td>\n",
        "      <td> 0.073633</td>\n",
        "      <td> 0.045787</td>\n",
        "      <td> 0.179883</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.032185</td>\n",
        "      <td> 0.037664</td>\n",
        "      <td> 0.029561</td>\n",
        "      <td> 0.036765</td>\n",
        "      <td> 0.016483</td>\n",
        "      <td> 0.097684</td>\n",
        "      <td> 0.023273</td>\n",
        "      <td> 0.014569</td>\n",
        "      <td> 0.093052</td>\n",
        "      <td> 0.018048</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2006</th>\n",
        "      <td> 14.105000</td>\n",
        "      <td> 15.876847</td>\n",
        "      <td> 5.783251</td>\n",
        "      <td>  847.177340</td>\n",
        "      <td> 0.012318</td>\n",
        "      <td> 0.068902</td>\n",
        "      <td> 0.034938</td>\n",
        "      <td> 0.038151</td>\n",
        "      <td> 0.059957</td>\n",
        "      <td> 0.117163</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.042576</td>\n",
        "      <td> 0.099477</td>\n",
        "      <td> 0.062200</td>\n",
        "      <td> 0.018421</td>\n",
        "      <td> 0.018968</td>\n",
        "      <td> 0.122815</td>\n",
        "      <td> 0.029100</td>\n",
        "      <td> 0.016103</td>\n",
        "      <td> 0.114982</td>\n",
        "      <td> 0.030360</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2007</th>\n",
        "      <td> 12.122222</td>\n",
        "      <td> 15.611111</td>\n",
        "      <td> 3.300000</td>\n",
        "      <td>  781.288889</td>\n",
        "      <td> 0.021512</td>\n",
        "      <td> 0.101952</td>\n",
        "      <td> 0.031070</td>\n",
        "      <td> 0.033012</td>\n",
        "      <td> 0.053122</td>\n",
        "      <td> 0.087377</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.038568</td>\n",
        "      <td> 0.072421</td>\n",
        "      <td> 0.052941</td>\n",
        "      <td> 0.026113</td>\n",
        "      <td> 0.012229</td>\n",
        "      <td> 0.083278</td>\n",
        "      <td> 0.015896</td>\n",
        "      <td> 0.100654</td>\n",
        "      <td> 0.109495</td>\n",
        "      <td> 0.026641</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>21 rows \u00d7 24 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 34,
       "text": [
        "                  print_page_number  publication_day_of_month  \\\n",
        "publication_year                                                \n",
        "1987                      16.078125                 15.440625   \n",
        "1988                      16.131772                 15.781845   \n",
        "1989                      15.372093                 15.536906   \n",
        "1990                      14.048096                 15.927856   \n",
        "1991                      13.740741                 15.361111   \n",
        "1992                      13.712329                 15.927478   \n",
        "1993                      13.840438                 15.993910   \n",
        "1994                      12.462985                 14.862515   \n",
        "1995                      13.065410                 14.683278   \n",
        "1996                      13.109336                 14.906644   \n",
        "1997                      13.075931                 15.842407   \n",
        "1998                      12.952333                 15.615619   \n",
        "1999                      13.520223                 16.460251   \n",
        "2000                      14.455581                 15.667426   \n",
        "2001                      12.358736                 15.568773   \n",
        "2002                      14.284483                 15.577586   \n",
        "2003                      16.865546                 14.554622   \n",
        "2004                      13.968750                 15.895833   \n",
        "2005                      17.229730                 14.310811   \n",
        "2006                      14.105000                 15.876847   \n",
        "2007                      12.122222                 15.611111   \n",
        "\n",
        "                  publication_month   word_count  gore mr bradley  \\\n",
        "publication_year                                                    \n",
        "1987                       6.889062  1037.596875         0.013062   \n",
        "1988                       7.007321   919.093704         0.018898   \n",
        "1989                       7.145602   922.744186         0.010631   \n",
        "1990                       6.660321   993.942886         0.012016   \n",
        "1991                       7.291005   875.859788         0.009539   \n",
        "1992                       6.970185  1002.409347         0.043044   \n",
        "1993                       6.456760  1103.561510         0.024206   \n",
        "1994                       7.203290  1082.405405         0.023328   \n",
        "1995                       6.763012  1053.384275         0.020591   \n",
        "1996                       6.521447  1105.935240         0.025712   \n",
        "1997                       6.978510  1094.520057         0.022722   \n",
        "1998                       6.907708  1056.229209         0.027463   \n",
        "1999                       6.859135  1233.340307         0.051156   \n",
        "2000                       6.397115  1153.927866         0.160920   \n",
        "2001                       4.355019  1026.721190         0.024562   \n",
        "2002                       6.672414   944.560345         0.026272   \n",
        "2003                       6.739496   555.067227         0.009983   \n",
        "2004                       7.442708   583.322917         0.012243   \n",
        "2005                       7.198198   626.756757         0.010039   \n",
        "2006                       5.783251   847.177340         0.012318   \n",
        "2007                       3.300000   781.288889         0.021512   \n",
        "\n",
        "                  court justice law  dr medical health  church catholic pope  \\\n",
        "publication_year                                                               \n",
        "1987                       0.097396           0.060587              0.092524   \n",
        "1988                       0.079396           0.053036              0.053688   \n",
        "1989                       0.130093           0.067381              0.046344   \n",
        "1990                       0.085574           0.045054              0.102904   \n",
        "1991                       0.102363           0.062877              0.063908   \n",
        "1992                       0.071893           0.030479              0.033666   \n",
        "1993                       0.070857           0.077052              0.056207   \n",
        "1994                       0.064905           0.064554              0.056195   \n",
        "1995                       0.044787           0.083096              0.076079   \n",
        "1996                       0.036355           0.029837              0.038603   \n",
        "1997                       0.059533           0.060747              0.065003   \n",
        "1998                       0.034178           0.057726              0.045059   \n",
        "1999                       0.055697           0.035389              0.066548   \n",
        "2000                       0.047862           0.031540              0.054476   \n",
        "2001                       0.051293           0.042686              0.050982   \n",
        "2002                       0.063409           0.052596              0.064712   \n",
        "2003                       0.055296           0.054321              0.039519   \n",
        "2004                       0.059577           0.031000              0.170174   \n",
        "2005                       0.090207           0.030611              0.073633   \n",
        "2006                       0.068902           0.034938              0.038151   \n",
        "2007                       0.101952           0.031070              0.033012   \n",
        "\n",
        "                  party republican platform  95 25 24    ...     \\\n",
        "publication_year                                         ...      \n",
        "1987                               0.033881  0.024220    ...      \n",
        "1988                               0.053024  0.027054    ...      \n",
        "1989                               0.031252  0.026215    ...      \n",
        "1990                               0.044839  0.023684    ...      \n",
        "1991                               0.026945  0.020312    ...      \n",
        "1992                               0.076023  0.019914    ...      \n",
        "1993                               0.036414  0.034490    ...      \n",
        "1994                               0.050726  0.023520    ...      \n",
        "1995                               0.068480  0.024624    ...      \n",
        "1996                               0.109390  0.019184    ...      \n",
        "1997                               0.049072  0.025290    ...      \n",
        "1998                               0.061258  0.023378    ...      \n",
        "1999                               0.062988  0.022525    ...      \n",
        "2000                               0.069495  0.016007    ...      \n",
        "2001                               0.040053  0.040324    ...      \n",
        "2002                               0.040770  0.026255    ...      \n",
        "2003                               0.053010  0.042566    ...      \n",
        "2004                               0.057674  0.031449    ...      \n",
        "2005                               0.045787  0.179883    ...      \n",
        "2006                               0.059957  0.117163    ...      \n",
        "2007                               0.053122  0.087377    ...      \n",
        "\n",
        "                  population planning nations  mr republican state  \\\n",
        "publication_year                                                     \n",
        "1987                                 0.081611             0.040472   \n",
        "1988                                 0.072635             0.068815   \n",
        "1989                                 0.056746             0.072783   \n",
        "1990                                 0.052115             0.114025   \n",
        "1991                                 0.064820             0.044731   \n",
        "1992                                 0.042105             0.086067   \n",
        "1993                                 0.057861             0.044883   \n",
        "1994                                 0.072791             0.079209   \n",
        "1995                                 0.058084             0.051223   \n",
        "1996                                 0.040071             0.097379   \n",
        "1997                                 0.066867             0.056209   \n",
        "1998                                 0.055877             0.115560   \n",
        "1999                                 0.061454             0.062307   \n",
        "2000                                 0.031736             0.088508   \n",
        "2001                                 0.054684             0.082735   \n",
        "2002                                 0.071979             0.146718   \n",
        "2003                                 0.062391             0.040871   \n",
        "2004                                 0.033617             0.068182   \n",
        "2005                                 0.032185             0.037664   \n",
        "2006                                 0.042576             0.099477   \n",
        "2007                                 0.038568             0.072421   \n",
        "\n",
        "                  cells research tissue  clinic clinics protesters  \\\n",
        "publication_year                                                     \n",
        "1987                           0.031659                   0.037868   \n",
        "1988                           0.038239                   0.050122   \n",
        "1989                           0.034361                   0.056964   \n",
        "1990                           0.026436                   0.044151   \n",
        "1991                           0.026793                   0.062423   \n",
        "1992                           0.031344                   0.041717   \n",
        "1993                           0.015811                   0.064460   \n",
        "1994                           0.014948                   0.080529   \n",
        "1995                           0.016017                   0.068693   \n",
        "1996                           0.019533                   0.031189   \n",
        "1997                           0.036049                   0.044989   \n",
        "1998                           0.023804                   0.054153   \n",
        "1999                           0.029933                   0.042405   \n",
        "2000                           0.019501                   0.018913   \n",
        "2001                           0.062117                   0.047477   \n",
        "2002                           0.042075                   0.056218   \n",
        "2003                           0.079818                   0.066128   \n",
        "2004                           0.079784                   0.032475   \n",
        "2005                           0.029561                   0.036765   \n",
        "2006                           0.062200                   0.018421   \n",
        "2007                           0.052941                   0.026113   \n",
        "\n",
        "                  mrs whitman jersey  women ms abortions  drug ru 486  \\\n",
        "publication_year                                                        \n",
        "1987                        0.011171            0.055880     0.014596   \n",
        "1988                        0.013610            0.065434     0.038249   \n",
        "1989                        0.018407            0.077171     0.022146   \n",
        "1990                        0.014907            0.069399     0.023694   \n",
        "1991                        0.011360            0.077972     0.028532   \n",
        "1992                        0.019782            0.085820     0.019573   \n",
        "1993                        0.025750            0.076564     0.023507   \n",
        "1994                        0.026017            0.069725     0.017058   \n",
        "1995                        0.026639            0.053164     0.012264   \n",
        "1996                        0.031042            0.048318     0.020521   \n",
        "1997                        0.054031            0.080167     0.023658   \n",
        "1998                        0.032660            0.069413     0.015702   \n",
        "1999                        0.046123            0.050060     0.018823   \n",
        "2000                        0.043970            0.042960     0.022533   \n",
        "2001                        0.035458            0.057140     0.017795   \n",
        "2002                        0.029891            0.094442     0.010625   \n",
        "2003                        0.012550            0.159312     0.014042   \n",
        "2004                        0.019164            0.114356     0.027721   \n",
        "2005                        0.016483            0.097684     0.023273   \n",
        "2006                        0.018968            0.122815     0.029100   \n",
        "2007                        0.012229            0.083278     0.015896   \n",
        "\n",
        "                  giuliani mr dinkins  people ms like  mr pataki amato  \n",
        "publication_year                                                        \n",
        "1987                         0.016393        0.124166         0.022838  \n",
        "1988                         0.013891        0.119832         0.025828  \n",
        "1989                         0.062115        0.126975         0.027854  \n",
        "1990                         0.014248        0.121351         0.043393  \n",
        "1991                         0.010961        0.126635         0.027325  \n",
        "1992                         0.016465        0.131338         0.061834  \n",
        "1993                         0.056577        0.155919         0.028816  \n",
        "1994                         0.023994        0.110916         0.092517  \n",
        "1995                         0.015796        0.125740         0.033118  \n",
        "1996                         0.016263        0.118087         0.036658  \n",
        "1997                         0.025400        0.148564         0.042747  \n",
        "1998                         0.025967        0.132262         0.079051  \n",
        "1999                         0.032756        0.140397         0.049157  \n",
        "2000                         0.034758        0.097493         0.031331  \n",
        "2001                         0.024889        0.100591         0.049060  \n",
        "2002                         0.017899        0.097881         0.036671  \n",
        "2003                         0.015675        0.132637         0.020225  \n",
        "2004                         0.010045        0.107228         0.017171  \n",
        "2005                         0.014569        0.093052         0.018048  \n",
        "2006                         0.016103        0.114982         0.030360  \n",
        "2007                         0.100654        0.109495         0.026641  \n",
        "\n",
        "[21 rows x 24 columns]"
       ]
      }
     ],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_topics['decade'] =df_topics['publication_year'] > 1999"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_topics.groupby('decade').mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>print_page_number</th>\n",
        "      <th>publication_day_of_month</th>\n",
        "      <th>publication_month</th>\n",
        "      <th>publication_year</th>\n",
        "      <th>word_count</th>\n",
        "      <th>gore mr bradley</th>\n",
        "      <th>court justice law</th>\n",
        "      <th>dr medical health</th>\n",
        "      <th>church catholic pope</th>\n",
        "      <th>party republican platform</th>\n",
        "      <th>...</th>\n",
        "      <th>population planning nations</th>\n",
        "      <th>mr republican state</th>\n",
        "      <th>cells research tissue</th>\n",
        "      <th>clinic clinics protesters</th>\n",
        "      <th>mrs whitman jersey</th>\n",
        "      <th>women ms abortions</th>\n",
        "      <th>drug ru 486</th>\n",
        "      <th>giuliani mr dinkins</th>\n",
        "      <th>people ms like</th>\n",
        "      <th>mr pataki amato</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>decade</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>False</th>\n",
        "      <td> 13.849621</td>\n",
        "      <td> 15.54402</td>\n",
        "      <td> 6.883107</td>\n",
        "      <td> 1993.112448</td>\n",
        "      <td> 1036.046722</td>\n",
        "      <td> 0.023789</td>\n",
        "      <td> 0.070563</td>\n",
        "      <td> 0.054512</td>\n",
        "      <td> 0.059326</td>\n",
        "      <td> 0.056926</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.058107</td>\n",
        "      <td> 0.075266</td>\n",
        "      <td> 0.026097</td>\n",
        "      <td> 0.051743</td>\n",
        "      <td> 0.025270</td>\n",
        "      <td> 0.067942</td>\n",
        "      <td> 0.021065</td>\n",
        "      <td> 0.025422</td>\n",
        "      <td> 0.128822</td>\n",
        "      <td> 0.045277</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>True </th>\n",
        "      <td> 14.234073</td>\n",
        "      <td> 15.51877</td>\n",
        "      <td> 6.021452</td>\n",
        "      <td> 2001.735073</td>\n",
        "      <td>  980.032892</td>\n",
        "      <td> 0.085233</td>\n",
        "      <td> 0.056916</td>\n",
        "      <td> 0.035647</td>\n",
        "      <td> 0.061179</td>\n",
        "      <td> 0.058027</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.040295</td>\n",
        "      <td> 0.082633</td>\n",
        "      <td> 0.040312</td>\n",
        "      <td> 0.030507</td>\n",
        "      <td> 0.033692</td>\n",
        "      <td> 0.069110</td>\n",
        "      <td> 0.021444</td>\n",
        "      <td> 0.028816</td>\n",
        "      <td> 0.101571</td>\n",
        "      <td> 0.032242</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>2 rows \u00d7 25 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 36,
       "text": [
        "        print_page_number  publication_day_of_month  publication_month  \\\n",
        "decade                                                                   \n",
        "False           13.849621                  15.54402           6.883107   \n",
        "True            14.234073                  15.51877           6.021452   \n",
        "\n",
        "        publication_year   word_count  gore mr bradley  court justice law  \\\n",
        "decade                                                                      \n",
        "False        1993.112448  1036.046722         0.023789           0.070563   \n",
        "True         2001.735073   980.032892         0.085233           0.056916   \n",
        "\n",
        "        dr medical health  church catholic pope  party republican platform  \\\n",
        "decade                                                                       \n",
        "False            0.054512              0.059326                   0.056926   \n",
        "True             0.035647              0.061179                   0.058027   \n",
        "\n",
        "                  ...              population planning nations  \\\n",
        "decade            ...                                            \n",
        "False             ...                                 0.058107   \n",
        "True              ...                                 0.040295   \n",
        "\n",
        "        mr republican state  cells research tissue  clinic clinics protesters  \\\n",
        "decade                                                                          \n",
        "False              0.075266               0.026097                   0.051743   \n",
        "True               0.082633               0.040312                   0.030507   \n",
        "\n",
        "        mrs whitman jersey  women ms abortions  drug ru 486  \\\n",
        "decade                                                        \n",
        "False             0.025270            0.067942     0.021065   \n",
        "True              0.033692            0.069110     0.021444   \n",
        "\n",
        "        giuliani mr dinkins  people ms like  mr pataki amato  \n",
        "decade                                                        \n",
        "False              0.025422        0.128822         0.045277  \n",
        "True               0.028816        0.101571         0.032242  \n",
        "\n",
        "[2 rows x 25 columns]"
       ]
      }
     ],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}