day_3_answers.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:c3970259adc93bff730c67fe267a940e78ce3e385f9a596ddaaee2d0c853b3dc"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## In Class Assignment for September 3\n",
      "### From words to a networks \n",
      "\n",
      "Last week, we read Ryan Light's \"[From Words to Networks and Back: Digital Text, Computational Social Science, and the Case of Presidential Inaugural Addresses](http://scu.sagepub.com/content/1/2/111)\". In it it, he used the tools of network analysis to identify major themes in presidential addresses. Let's do that for ASA abstracts.\n",
      "\n",
      "Work with a different partner from last time.\n",
      "\n",
      "Break up the assignment into the following discrete tasks:\n",
      "\n",
      "1. Review the functions created from last time. You can either use mine (below) or your own.  \n",
      "2. Take a look `combinations` which is part of `itertools`. What does it doe and how is it related to network analysis of text?\n",
      "3. What word do you think is most central to ASA abstracts? Hypothesize.\n",
      "3. Create a network of the most commonly co-occurring word pairs in the abstracts. Don't forget to clean the strings first. You might thing about eliminating commonly used words, like \"the\" or \"in\". You might think about eliminating words that are only mentioned one or two times in the entire corpus.\n",
      "4. Get your word-pair list into a format that it can be added into `networkx`. \n",
      "5. Compute and display the most central words in the abstracts.\n",
      "3. `community` is a module that divides networks into cliques based on a comparison of [\"the density of links inside communities as compared to links between communities.\"](http://arxiv.org/pdf/0803.0476v2.pdf). Given a network, it produces a dictionary where the keys are the nodes and the values are the arbitrarily named community numbers. Partition your word network!\n",
      "4. Think about your network. Before you look at the results, how do you think you will be able to classify the partitions? That is, what do you think it will tell you about the data?\n",
      "4. Print out the most frequently occurring words in your network. Note: This is tricky.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from itertools import combinations\n",
      "\n",
      "pairs = combinations('abcde', 3)\n",
      "\n",
      "for item in pairs:\n",
      "    print item"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "('a', 'b', 'c')\n",
        "('a', 'b', 'd')\n",
        "('a', 'b', 'e')\n",
        "('a', 'c', 'd')\n",
        "('a', 'c', 'e')\n",
        "('a', 'd', 'e')\n",
        "('b', 'c', 'd')\n",
        "('b', 'c', 'e')\n",
        "('b', 'd', 'e')\n",
        "('c', 'd', 'e')\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sentence = 'This paper contributes to the growing comparative scholarship on regional trade agreements (RTAs) and the dynamics they engender in national and local life.'\n",
      "\n",
      "for word_pair in combinations(sentence.split(),2):\n",
      "    print word_pair"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "('This', 'paper')\n",
        "('This', 'contributes')\n",
        "('This', 'to')\n",
        "('This', 'the')\n",
        "('This', 'growing')\n",
        "('This', 'comparative')\n",
        "('This', 'scholarship')\n",
        "('This', 'on')\n",
        "('This', 'regional')\n",
        "('This', 'trade')\n",
        "('This', 'agreements')\n",
        "('This', '(RTAs)')\n",
        "('This', 'and')\n",
        "('This', 'the')\n",
        "('This', 'dynamics')\n",
        "('This', 'they')\n",
        "('This', 'engender')\n",
        "('This', 'in')\n",
        "('This', 'national')\n",
        "('This', 'and')\n",
        "('This', 'local')\n",
        "('This', 'life.')\n",
        "('paper', 'contributes')\n",
        "('paper', 'to')\n",
        "('paper', 'the')\n",
        "('paper', 'growing')\n",
        "('paper', 'comparative')\n",
        "('paper', 'scholarship')\n",
        "('paper', 'on')\n",
        "('paper', 'regional')\n",
        "('paper', 'trade')\n",
        "('paper', 'agreements')\n",
        "('paper', '(RTAs)')\n",
        "('paper', 'and')\n",
        "('paper', 'the')\n",
        "('paper', 'dynamics')\n",
        "('paper', 'they')\n",
        "('paper', 'engender')\n",
        "('paper', 'in')\n",
        "('paper', 'national')\n",
        "('paper', 'and')\n",
        "('paper', 'local')\n",
        "('paper', 'life.')\n",
        "('contributes', 'to')\n",
        "('contributes', 'the')\n",
        "('contributes', 'growing')\n",
        "('contributes', 'comparative')\n",
        "('contributes', 'scholarship')\n",
        "('contributes', 'on')\n",
        "('contributes', 'regional')\n",
        "('contributes', 'trade')\n",
        "('contributes', 'agreements')\n",
        "('contributes', '(RTAs)')\n",
        "('contributes', 'and')\n",
        "('contributes', 'the')\n",
        "('contributes', 'dynamics')\n",
        "('contributes', 'they')\n",
        "('contributes', 'engender')\n",
        "('contributes', 'in')\n",
        "('contributes', 'national')\n",
        "('contributes', 'and')\n",
        "('contributes', 'local')\n",
        "('contributes', 'life.')\n",
        "('to', 'the')\n",
        "('to', 'growing')\n",
        "('to', 'comparative')\n",
        "('to', 'scholarship')\n",
        "('to', 'on')\n",
        "('to', 'regional')\n",
        "('to', 'trade')\n",
        "('to', 'agreements')\n",
        "('to', '(RTAs)')\n",
        "('to', 'and')\n",
        "('to', 'the')\n",
        "('to', 'dynamics')\n",
        "('to', 'they')\n",
        "('to', 'engender')\n",
        "('to', 'in')\n",
        "('to', 'national')\n",
        "('to', 'and')\n",
        "('to', 'local')\n",
        "('to', 'life.')\n",
        "('the', 'growing')\n",
        "('the', 'comparative')\n",
        "('the', 'scholarship')\n",
        "('the', 'on')\n",
        "('the', 'regional')\n",
        "('the', 'trade')\n",
        "('the', 'agreements')\n",
        "('the', '(RTAs)')\n",
        "('the', 'and')\n",
        "('the', 'the')\n",
        "('the', 'dynamics')\n",
        "('the', 'they')\n",
        "('the', 'engender')\n",
        "('the', 'in')\n",
        "('the', 'national')\n",
        "('the', 'and')\n",
        "('the', 'local')\n",
        "('the', 'life.')\n",
        "('growing', 'comparative')\n",
        "('growing', 'scholarship')\n",
        "('growing', 'on')\n",
        "('growing', 'regional')\n",
        "('growing', 'trade')\n",
        "('growing', 'agreements')\n",
        "('growing', '(RTAs)')\n",
        "('growing', 'and')\n",
        "('growing', 'the')\n",
        "('growing', 'dynamics')\n",
        "('growing', 'they')\n",
        "('growing', 'engender')\n",
        "('growing', 'in')\n",
        "('growing', 'national')\n",
        "('growing', 'and')\n",
        "('growing', 'local')\n",
        "('growing', 'life.')\n",
        "('comparative', 'scholarship')\n",
        "('comparative', 'on')\n",
        "('comparative', 'regional')\n",
        "('comparative', 'trade')\n",
        "('comparative', 'agreements')\n",
        "('comparative', '(RTAs)')\n",
        "('comparative', 'and')\n",
        "('comparative', 'the')\n",
        "('comparative', 'dynamics')\n",
        "('comparative', 'they')\n",
        "('comparative', 'engender')\n",
        "('comparative', 'in')\n",
        "('comparative', 'national')\n",
        "('comparative', 'and')\n",
        "('comparative', 'local')\n",
        "('comparative', 'life.')\n",
        "('scholarship', 'on')\n",
        "('scholarship', 'regional')\n",
        "('scholarship', 'trade')\n",
        "('scholarship', 'agreements')\n",
        "('scholarship', '(RTAs)')\n",
        "('scholarship', 'and')\n",
        "('scholarship', 'the')\n",
        "('scholarship', 'dynamics')\n",
        "('scholarship', 'they')\n",
        "('scholarship', 'engender')\n",
        "('scholarship', 'in')\n",
        "('scholarship', 'national')\n",
        "('scholarship', 'and')\n",
        "('scholarship', 'local')\n",
        "('scholarship', 'life.')\n",
        "('on', 'regional')\n",
        "('on', 'trade')\n",
        "('on', 'agreements')\n",
        "('on', '(RTAs)')\n",
        "('on', 'and')\n",
        "('on', 'the')\n",
        "('on', 'dynamics')\n",
        "('on', 'they')\n",
        "('on', 'engender')\n",
        "('on', 'in')\n",
        "('on', 'national')\n",
        "('on', 'and')\n",
        "('on', 'local')\n",
        "('on', 'life.')\n",
        "('regional', 'trade')\n",
        "('regional', 'agreements')\n",
        "('regional', '(RTAs)')\n",
        "('regional', 'and')\n",
        "('regional', 'the')\n",
        "('regional', 'dynamics')\n",
        "('regional', 'they')\n",
        "('regional', 'engender')\n",
        "('regional', 'in')\n",
        "('regional', 'national')\n",
        "('regional', 'and')\n",
        "('regional', 'local')\n",
        "('regional', 'life.')\n",
        "('trade', 'agreements')\n",
        "('trade', '(RTAs)')\n",
        "('trade', 'and')\n",
        "('trade', 'the')\n",
        "('trade', 'dynamics')\n",
        "('trade', 'they')\n",
        "('trade', 'engender')\n",
        "('trade', 'in')\n",
        "('trade', 'national')\n",
        "('trade', 'and')\n",
        "('trade', 'local')\n",
        "('trade', 'life.')\n",
        "('agreements', '(RTAs)')\n",
        "('agreements', 'and')\n",
        "('agreements', 'the')\n",
        "('agreements', 'dynamics')\n",
        "('agreements', 'they')\n",
        "('agreements', 'engender')\n",
        "('agreements', 'in')\n",
        "('agreements', 'national')\n",
        "('agreements', 'and')\n",
        "('agreements', 'local')\n",
        "('agreements', 'life.')\n",
        "('(RTAs)', 'and')\n",
        "('(RTAs)', 'the')\n",
        "('(RTAs)', 'dynamics')\n",
        "('(RTAs)', 'they')\n",
        "('(RTAs)', 'engender')\n",
        "('(RTAs)', 'in')\n",
        "('(RTAs)', 'national')\n",
        "('(RTAs)', 'and')\n",
        "('(RTAs)', 'local')\n",
        "('(RTAs)', 'life.')\n",
        "('and', 'the')\n",
        "('and', 'dynamics')\n",
        "('and', 'they')\n",
        "('and', 'engender')\n",
        "('and', 'in')\n",
        "('and', 'national')\n",
        "('and', 'and')\n",
        "('and', 'local')\n",
        "('and', 'life.')\n",
        "('the', 'dynamics')\n",
        "('the', 'they')\n",
        "('the', 'engender')\n",
        "('the', 'in')\n",
        "('the', 'national')\n",
        "('the', 'and')\n",
        "('the', 'local')\n",
        "('the', 'life.')\n",
        "('dynamics', 'they')\n",
        "('dynamics', 'engender')\n",
        "('dynamics', 'in')\n",
        "('dynamics', 'national')\n",
        "('dynamics', 'and')\n",
        "('dynamics', 'local')\n",
        "('dynamics', 'life.')\n",
        "('they', 'engender')\n",
        "('they', 'in')\n",
        "('they', 'national')\n",
        "('they', 'and')\n",
        "('they', 'local')\n",
        "('they', 'life.')\n",
        "('engender', 'in')\n",
        "('engender', 'national')\n",
        "('engender', 'and')\n",
        "('engender', 'local')\n",
        "('engender', 'life.')\n",
        "('in', 'national')\n",
        "('in', 'and')\n",
        "('in', 'local')\n",
        "('in', 'life.')\n",
        "('national', 'and')\n",
        "('national', 'local')\n",
        "('national', 'life.')\n",
        "('and', 'local')\n",
        "('and', 'life.')\n",
        "('local', 'life.')\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "word_pair_frequencies = {}\n",
      "word_pair_frequencies[('agreements', '(RTAs)')] = 15"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#importing the community network. If you don't have it, this will install it. This is a hack\n",
      "\n",
      "try:\n",
      "    import community\n",
      "except Exception, e:\n",
      "    import urllib2\n",
      "    url = 'https://gist.githubusercontent.com/nealcaren/72804f2bf068aeae3644/raw/3b56e880a0432f87952f35611ab292e2f37c536b/community.py'\n",
      "    text = urllib2.urlopen(url).read()\n",
      "    with open('community.py', 'w') as outfile:\n",
      "        outfile.write(text)\n",
      "    import community"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import networkx as nx"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fruit_network = [('apple','banana',{'weight':5}),\n",
      "                 ('apple','orange',{'weight':3}),\n",
      "                 ('orange','banana',{'weight':2}),\n",
      "                 ('orange','lemon',{'weight':5}),\n",
      "                 ('lemon','pineapple',{'weight':3}),\n",
      "                 ('pineapple','orange',{'weight':2})]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "G = nx.Graph()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "G.add_edges_from(fruit_network)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print G.nodes()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['orange', 'pineapple', 'lemon', 'apple', 'banana']\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# What are the most central elements?\n",
      "centrality=nx.eigenvector_centrality(G)\n",
      "print centrality"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'orange': 0.5891294035599184, 'lemon': 0.4529372814977233, 'apple': 0.444572575737449, 'banana': 0.4008813581266987, 'pineapple': 0.29903800176440903}\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Now print it nicer for just the top results\n",
      "for word in sorted(centrality, key=centrality.get, reverse=True)[:3]:\n",
      "    print '%.02f %s' % (centrality[word], word)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.59 orange\n",
        "0.45 lemon\n",
        "0.44 apple\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Now partition the network\n",
      "\n",
      "partition = community.best_partition(G)\n",
      "print partition"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'orange': 0, 'pineapple': 0, 'lemon': 0, 'apple': 1, 'banana': 1}\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Convert to a more useful format:\n",
      "\n",
      "fruit_freqs = {'apple': 8,\n",
      "             'orange': 12,\n",
      "             'banana' : 7,\n",
      "             'lemon' : 8,\n",
      "             'pineapple': 5}\n",
      "\n",
      "\n",
      "communities = {}\n",
      "for fruit in partition:\n",
      "    clique = partition[fruit]\n",
      "    frequncy = fruit_freqs[fruit]\n",
      "    \n",
      "    try:\n",
      "        communities[clique][fruit] = frequncy\n",
      "    except Exception, e:\n",
      "        communities[clique] = {fruit : frequncy}\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Now it is a dictionary of dictionaries!!!!\n",
      "print communities"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{0: {'orange': 12, 'lemon': 8, 'pineapple': 5}, 1: {'apple': 8, 'banana': 7}}\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Print it out all pretty \n",
      "for clique in communities:\n",
      "    community_freqs = communities[clique]\n",
      "    print clique\n",
      "    for fruit in sorted(community_freqs, key=community_freqs.get, reverse=True)[:5]:\n",
      "        print fruit\n",
      "    print ''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0\n",
        "orange\n",
        "lemon\n",
        "pineapple\n",
        "\n",
        "1\n",
        "apple\n",
        "banana\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Old stuff\n",
      "\n",
      "# Read in the abstracts\n",
      "\n",
      "import csv\n",
      "import urllib2 \n",
      "from string import punctuation\n",
      "\n",
      "file_location = 'https://gist.githubusercontent.com/nealcaren/886c85645695d04887d0/raw/fec90e5cc1257f277f97a904a4f1cb77f60b2012/asa_2014_prelim.csv'\n",
      "response = urllib2.urlopen(file_location)\n",
      "abstracts = list(csv.reader(response))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Create two lists, one for the roundtables and one for the panel papers\n",
      "\n",
      "roundtable_abstracts = []\n",
      "panel_abstracts = []\n",
      "\n",
      "for abstract in abstracts:\n",
      "    if len(abstract[-1]) > 100:\n",
      "        if 'Roundtable' in abstract[3]:\n",
      "            roundtable_abstracts.append(abstract[-1])\n",
      "        elif 'Regular Session' in abstract[3] or 'Section on '  in abstract[3]:\n",
      "            panel_abstracts.append(abstract[-1])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Your functions go here\n",
      "def word_clean(word):\n",
      "    w = word.lower()\n",
      "    w = w.strip(punctuation)\n",
      "    return w\n",
      "\n",
      "def text_clean(sentence):\n",
      "    words = sentence.split()\n",
      "    clean_words = []\n",
      "    for word in words:\n",
      "        w = word_clean(word)\n",
      "        clean_words.append(w)\n",
      "    clean_words = set(clean_words)\n",
      "    clean_words = list(clean_words)  \n",
      "    return clean_words\n",
      "\n",
      "def text_list_to_word_freq(text_list):\n",
      "    '''\n",
      "    Generate word frequencies from a list of texts\n",
      "    '''\n",
      "    \n",
      "    word_freq = {}\n",
      "\n",
      "    for text in text_list:\n",
      "        words = text_clean(text)\n",
      "\n",
      "        for word in words:\n",
      "            if word not in word_freq:\n",
      "                word_freq[word] = 1\n",
      "            else:\n",
      "                word_freq[word] = word_freq[word] + 1\n",
      "    return word_freq"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "word_freq = {}\n",
      "\n",
      "for abstract in panel_abstracts:\n",
      "    clean_abstract = text_clean(abstract)\n",
      "    for word in clean_abstract:\n",
      "        try:\n",
      "            word_freq[word] = word_freq[word] + 1\n",
      "        except:\n",
      "            word_freq[word] = 1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#we had stopwords available all along\n",
      "\n",
      "from nltk.corpus import stopwords\n",
      "stopword_list = stopwords.words('english')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#create the pair frequencies\n",
      "\n",
      "word_pair_freq = {}\n",
      "\n",
      "for abstract in panel_abstracts:\n",
      "    clean_abstract = text_clean(abstract)\n",
      "    shorter = []\n",
      "    for word in clean_abstract:\n",
      "        if word not in stopword_list:\n",
      "            shorter.append(word)\n",
      "            \n",
      "    abstract_pairs = combinations(shorter, 2)\n",
      "    \n",
      "    for abstract_pair in abstract_pairs:\n",
      "        try:\n",
      "            word_pair_freq[abstract_pair] = word_pair_freq[abstract_pair] + 1\n",
      "        except:\n",
      "            word_pair_freq[abstract_pair] = 1\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Now print it nicer for just the top results\n",
      "for word in sorted(word_pair_freq, key=word_pair_freq.get, reverse=True)[:50]:\n",
      "    print '%s %s' % (word_pair_freq[word], word)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "262 ('paper', 'social')\n",
        "223 ('research', 'social')\n",
        "198 ('using', 'data')\n",
        "193 ('social', 'data')\n",
        "192 ('study', 'social')\n",
        "191 ('paper', 'research')\n",
        "188 ('research', 'data')\n",
        "186 ('study', 'data')\n",
        "186 ('paper', 'data')\n",
        "174 ('research', 'study')\n",
        "162 ('paper', 'analysis')\n",
        "157 ('using', 'social')\n",
        "149 ('paper', 'also')\n",
        "146 ('however', 'social')\n",
        "146 ('paper', 'two')\n",
        "146 ('findings', 'data')\n",
        "143 ('new', 'social')\n",
        "142 ('paper', 'new')\n",
        "140 ('using', 'study')\n",
        "138 ('paper', 'use')\n",
        "137 ('states', 'united')\n",
        "136 ('paper', 'study')\n",
        "135 ('findings', 'social')\n",
        "133 ('social', 'analysis')\n",
        "133 ('paper', 'using')\n",
        "131 ('paper', 'one')\n",
        "128 ('paper', 'work')\n",
        "125 ('findings', 'study')\n",
        "125 ('among', 'data')\n",
        "124 ('results', 'data')\n",
        "121 ('find', 'social')\n",
        "121 ('use', 'data')\n",
        "120 ('paper', 'within')\n",
        "119 ('two', 'data')\n",
        "119 ('also', 'data')\n",
        "119 ('data', 'analysis')\n",
        "118 ('however', 'data')\n",
        "118 ('paper', 'however')\n",
        "118 ('findings', 'research')\n",
        "117 ('find', 'data')\n",
        "116 ('paper', 'interviews')\n",
        "116 ('examine', 'data')\n",
        "116 ('using', 'research')\n",
        "115 ('research', 'however')\n",
        "114 ('survey', 'data')\n",
        "113 ('however', 'study')\n",
        "112 ('research', 'two')\n",
        "112 ('research', 'analysis')\n",
        "111 ('social', 'within')\n",
        "110 ('research', 'work')\n"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print len(word_pair_freq)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3845166\n"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#get the data ready for networkx\n",
      "# Use only the top 50,000 ties--there are 5+ million in the dataset\n",
      "\n",
      "edge_list = []\n",
      "for pair in sorted(word_pair_freq, key=word_pair_freq.get, reverse=True)[:50000]:\n",
      "    weight = word_pair_freq[pair]\n",
      "    pair_weighted = pair + ({'weight' : weight},)\n",
      "    edge_list.append(pair_weighted)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "G = nx.Graph()\n",
      "G.add_edges_from(edge_list)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "centrality=nx.eigenvector_centrality(G)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Now print it nicer for just the top results\n",
      "for word in sorted(centrality, key=centrality.get, reverse=True)[:15]:\n",
      "    print '%.02f %s' % (centrality[word], word)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.24 paper\n",
        "0.19 social\n",
        "0.18 research\n",
        "0.18 data\n",
        "0.15 study\n",
        "0.13 using\n",
        "0.13 analysis\n",
        "0.13 however\n",
        "0.12 two\n",
        "0.12 findings\n",
        "0.12 new\n",
        "0.11 find\n",
        "0.11 use\n",
        "0.11 work\n",
        "0.11 results\n"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Now partition the network\n",
      "\n",
      "partition = community.best_partition(G)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#convert it to a more useful format\n",
      "\n",
      "communities = {}\n",
      "for word in partition:\n",
      "    clique = partition[word]\n",
      "    frequncy = word_freq[word]\n",
      "    \n",
      "    try:\n",
      "        communities[clique][word] = frequncy\n",
      "    except Exception, e:\n",
      "        communities[clique] = {word : frequncy}\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 29
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Print it out all pretty \n",
      "for clique in communities:\n",
      "    community_freqs = communities[clique]\n",
      "    print clique\n",
      "    for word in sorted(community_freqs, key=community_freqs.get, reverse=True)[:20]:\n",
      "        print '%s (%s)' % (word,word_freq[word])\n",
      "    print ''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0\n",
        "findings (316)\n",
        "find (305)\n",
        "among (280)\n",
        "examine (250)\n",
        "results (243)\n",
        "may (243)\n",
        "time (231)\n",
        "women (231)\n",
        "economic (218)\n",
        "important (214)\n",
        "across (212)\n",
        "suggest (199)\n",
        "gender (196)\n",
        "relationship (195)\n",
        "studies (195)\n",
        "health (189)\n",
        "national (179)\n",
        "effects (177)\n",
        "support (176)\n",
        "survey (174)\n",
        "\n",
        "1\n",
        "within (254)\n",
        "argue (244)\n",
        "interviews (231)\n",
        "different (224)\n",
        "show (219)\n",
        "based (219)\n",
        "three (216)\n",
        "role (215)\n",
        "well (208)\n",
        "cultural (205)\n",
        "ways (200)\n",
        "political (198)\n",
        "first (196)\n",
        "many (193)\n",
        "often (188)\n",
        "groups (186)\n",
        "recent (186)\n",
        "theory (185)\n",
        "examines (185)\n",
        "states (185)\n",
        "\n",
        "2\n",
        "paper (630)\n",
        "social (606)\n",
        "data (507)\n",
        "research (488)\n",
        "study (473)\n",
        "using (376)\n",
        "also (369)\n",
        "analysis (324)\n",
        "two (314)\n",
        "however (309)\n",
        "new (307)\n",
        "one (295)\n",
        "use (292)\n",
        "work (288)\n",
        "cases (67)\n",
        "discuss (66)\n",
        "must (64)\n",
        "focused (64)\n",
        "set (64)\n",
        "focuses (63)\n",
        "\n",
        "3\n",
        "et (16)\n",
        "al (16)\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 30
    }
   ],
   "metadata": {}
  }
 ]
}