day_2_answers.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:93bdb83b19871604d4f27112874812605b100c783bc5d12970ba53b494fe5748"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## In Class Assignment for August 25th \n",
      "### What is the difference between a roundtable paper and regular/section session paper?  \n",
      "\n",
      "Work with a partner to compare the most frequently used words in #asa14 papers that are presentend at roundtables with those that are presented in regular or section sessions. \n",
      "\n",
      "To make things easier, I've created two lists of #asa14 abstracts. `roundtable_abstracts` has the abstracts of roundtable papers while `panel_abstracts` has the abstracts of papers presented on panels. I've extracted them from a csv file that I've [made available](https://gist.github.com/nealcaren/886c85645695d04887d0) on Github.\n",
      "\n",
      "Break up the assignment into the following discrete tasks:\n",
      "\n",
      "1. Create a `word_clean` function that that takes a single word and returns a lower-cased version of it with all punctation removed.  \n",
      "2. Create a  function `text_clean` that takes a string of words and returns a list of unique, cleaned words. Use your `word_clean` function as part of it.\n",
      "3. Create a dictionary where the keys are the words in the roundtable abstracts and the values are the frequencies. \n",
      "4. Print out the 50 most frequently used words in roundtable abstracts.\n",
      "5. Repeat steps 3 and 4 for panel sessions, and compare the lists.\n",
      "6. Bonus: If you have time, replace the counts in the dictionaries with percents. This requires you to know the total number of words used for each presentation type. Then create a new dictionary that is the ratio of the percents, so you can look at the words that are dispropritaly found in each type of paper. You might limit yourself to words that are used at least 10 times.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Potential useful things to know\n",
      "\n",
      "word = 'Hi!'\n",
      "print word\n",
      "print word.lower()\n",
      "print word.strip('!')\n",
      "print word.strip('.!,')\n",
      "\n",
      "from string import punctuation \n",
      "print punctuation "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Hi!\n",
        "hi!\n",
        "Hi\n",
        "Hi\n",
        "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sentence = 'We investigate the following question: how have RTAs helped separatist and autonomous movements in their ambitions?'\n",
      "print sentence.split()\n",
      "\n",
      "words = sentence.split()\n",
      "print words\n",
      "\n",
      "print set(['A','A','B','B'])\n",
      "print set(words)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['We', 'investigate', 'the', 'following', 'question:', 'how', 'have', 'RTAs', 'helped', 'separatist', 'and', 'autonomous', 'movements', 'in', 'their', 'ambitions?']\n",
        "['We', 'investigate', 'the', 'following', 'question:', 'how', 'have', 'RTAs', 'helped', 'separatist', 'and', 'autonomous', 'movements', 'in', 'their', 'ambitions?']\n",
        "set(['A', 'B'])\n",
        "set(['and', 'separatist', 'We', 'their', 'RTAs', 'helped', 'how', 'investigate', 'ambitions?', 'have', 'in', 'following', 'the', 'movements', 'autonomous', 'question:'])\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "word_freqs = {'investigate ' : 50,\n",
      "         'following ' : 20,\n",
      "         'question' : 55}\n",
      "\n",
      "print word_freqs\n",
      "\n",
      "for word in sorted(word_freqs, key = word_freqs.get, reverse=True):\n",
      "    print word_freqs[word], word "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'following ': 20, 'question': 55, 'investigate ': 50}\n",
        "55 question\n",
        "50 investigate \n",
        "20 following \n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Read in the abstracts\n",
      "\n",
      "import csv\n",
      "import urllib2 \n",
      "\n",
      "file_location = 'https://gist.githubusercontent.com/nealcaren/886c85645695d04887d0/raw/fec90e5cc1257f277f97a904a4f1cb77f60b2012/asa_2014_prelim.csv'\n",
      "response = urllib2.urlopen(file_location)\n",
      "abstracts = list(csv.reader(response))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Sample abstract\n",
      "\n",
      "abstracts[500]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "['721813',\n",
        " 'Francesco Giovanni Duina, University of British Columbia; Jared Bok, Emory University',\n",
        " 'Tue, August 19, 2:30 to 4:10pm, TBA',\n",
        " 'Regular Session. Citizenship and Identities in the Globalized World',\n",
        " 'Capitalizing on Regional Integration: Sub-national Movements and the Rhetorical Leveraging of NAFTA and the EU',\n",
        " 'Regular',\n",
        " 'This paper contributes to the growing comparative scholarship on regional trade agreements (RTAs) and the dynamics they engender in national and local life. An objective of that scholarship is to identify patterns across RTAs. We investigate the following question: how have RTAs helped separatist and autonomous movements in their ambitions? We propose that both the left- and right-leaning movements have successfully appropriated, in positive and negative language, RTAs in their rhetoric to articulate not only their goals against their nation states but also their claims against those who oppose them. We identify four factors that might explain the observable differences in rhetorical approaches. The empirical evidence concerns the Quebecois nationalists in Canada, Converg\\xc3\\xa8ncia i Uni\\xc3\\xb3 in Spain, the Zapatistas in Mexico, and the Lega Nord in Italy. We conclude by reflecting on the possible local and regional impact of the observed rhetorical leveraging across RTAs.']"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Create two lists, one for the roundtables and one for the panel papers\n",
      "\n",
      "roundtable_abstracts = []\n",
      "panel_abstracts = []\n",
      "\n",
      "for abstract in abstracts:\n",
      "    if len(abstract[-1]) > 100:\n",
      "        if 'Roundtable' in abstract[3]:\n",
      "            roundtable_abstracts.append(abstract[-1])\n",
      "        elif 'Regular Session' in abstract[3] or 'Section on '  in abstract[3]:\n",
      "            panel_abstracts.append(abstract[-1])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Your functions go here\n",
      "def word_clean(word):\n",
      "    w = word.lower()\n",
      "    w = w.strip(punctuation)\n",
      "    return w\n",
      "\n",
      "def text_clean(sentence):\n",
      "    words = sentence.split()\n",
      "    clean_words = []\n",
      "    for word in words:\n",
      "        w = word_clean(word)\n",
      "        clean_words.append(w)\n",
      "    clean_words = set(clean_words)\n",
      "    clean_words = list(clean_words)  \n",
      "    return clean_words\n",
      "\n",
      "def text_list_to_word_freq(text_list):\n",
      "    '''\n",
      "    Generate word frequencies from a list of texts\n",
      "    '''\n",
      "    \n",
      "    word_freq = {}\n",
      "\n",
      "    for text in text_list:\n",
      "        words = text_clean(text)\n",
      "\n",
      "        for word in words:\n",
      "            if word not in word_freq:\n",
      "                word_freq[word] = 1\n",
      "            else:\n",
      "                word_freq[word] = word_freq[word] + 1\n",
      "    return word_freq"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "roundtable_word_freq = text_list_to_word_freq(roundtable_abstracts)\n",
      "panel_word_freq = text_list_to_word_freq(panel_abstracts)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#print out the 50 most common words\n",
      "for word in sorted(roundtable_word_freq, key = roundtable_word_freq.get, reverse = True)[:50]:\n",
      "    print word, roundtable_word_freq[word] / float(len(roundtable_abstracts))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "the 0.997281377435\n",
        "of 0.995015858632\n",
        "and 0.991391028546\n",
        "to 0.967829632986\n",
        "in 0.966017217943\n",
        "a 0.908019936565\n",
        "this 0.871318531944\n",
        "that 0.85727231536\n",
        "on 0.751699139103\n",
        "for 0.708654281831\n",
        "is 0.681014952424\n",
        "with 0.680108744903\n",
        "as 0.66062528319\n",
        "are 0.622111463525\n",
        "from 0.588128681468\n",
        "by 0.575441776167\n",
        "their 0.481196193928\n",
        "these 0.450385138197\n",
        "paper 0.449932034436\n",
        "an 0.448572723154\n",
        "social 0.44630720435\n",
        "have 0.439057544178\n",
        "how 0.423198912551\n",
        "study 0.391934753058\n",
        "between 0.383325781604\n",
        "which 0.376076121432\n",
        "has 0.374263706389\n",
        "research 0.365201631174\n",
        "more 0.356592659719\n",
        "or 0.356139555958\n",
        "i 0.346624376982\n",
        "be 0.342546443135\n",
        "not 0.331218849116\n",
        "data 0.330765745356\n",
        "at 0.311735387404\n",
        "it 0.310376076121\n",
        "we 0.29768917082\n",
        "using 0.283189850476\n",
        "but 0.271409152696\n",
        "they 0.265065700045\n",
        "also 0.262347077481\n",
        "both 0.236067059357\n",
        "while 0.231536021749\n",
        "analysis 0.229270502945\n",
        "through 0.225645672859\n",
        "about 0.223833257816\n",
        "than 0.222927050295\n",
        "been 0.221567739012\n",
        "who 0.22020842773\n",
        "other 0.219302220208\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#print out the 50 most common words\n",
      "for word in sorted(roundtable_word_freq, key = roundtable_word_freq.get, reverse = True)[:50]:\n",
      "    print word, roundtable_word_freq[word] / float(len(roundtable_abstracts))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "the 0.997281377435\n",
        "of 0.995015858632\n",
        "and 0.991391028546\n",
        "to 0.967829632986\n",
        "in 0.966017217943\n",
        "a 0.908019936565\n",
        "this 0.871318531944\n",
        "that 0.85727231536\n",
        "on 0.751699139103\n",
        "for 0.708654281831\n",
        "is 0.681014952424\n",
        "with 0.680108744903\n",
        "as 0.66062528319\n",
        "are 0.622111463525\n",
        "from 0.588128681468\n",
        "by 0.575441776167\n",
        "their 0.481196193928\n",
        "these 0.450385138197\n",
        "paper 0.449932034436\n",
        "an 0.448572723154\n",
        "social 0.44630720435\n",
        "have 0.439057544178\n",
        "how 0.423198912551\n",
        "study 0.391934753058\n",
        "between 0.383325781604\n",
        "which 0.376076121432\n",
        "has 0.374263706389\n",
        "research 0.365201631174\n",
        "more 0.356592659719\n",
        "or 0.356139555958\n",
        "i 0.346624376982\n",
        "be 0.342546443135\n",
        "not 0.331218849116\n",
        "data 0.330765745356\n",
        "at 0.311735387404\n",
        "it 0.310376076121\n",
        "we 0.29768917082\n",
        "using 0.283189850476\n",
        "but 0.271409152696\n",
        "they 0.265065700045\n",
        "also 0.262347077481\n",
        "both 0.236067059357\n",
        "while 0.231536021749\n",
        "analysis 0.229270502945\n",
        "through 0.225645672859\n",
        "about 0.223833257816\n",
        "than 0.222927050295\n",
        "been 0.221567739012\n",
        "who 0.22020842773\n",
        "other 0.219302220208\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Some ratios:\n",
      "ratios = {}\n",
      "\n",
      "for word in  sorted(roundtable_word_freq, key = roundtable_word_freq.get, reverse = True)[:500]:\n",
      "    rountable_percent =  roundtable_word_freq[word] / float(len(roundtable_abstracts))\n",
      "    panel_percent = panel_word_freq[word] / float(len(panel_abstracts))\n",
      "    ratio = panel_percent/rountable_percent\n",
      "    ratios[word] = ratio"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'More common in Roundtables'\n",
      "for word in sorted(ratios, key=ratios.get)[:30]:\n",
      "    print '%.2f %s'% (ratios[word],word)\n",
      "\n",
      "print ''\n",
      "print 'More common in Panels'\n",
      "for word in sorted(ratios, key=ratios.get, reverse= True)[:30]:\n",
      "    print '%.2f %s'% (ratios[word],word)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "More common in Roundtables\n",
        "0.55 regression\n",
        "0.60 discussed\n",
        "0.60 2012\n",
        "0.61 student\n",
        "0.63 demographic\n",
        "0.63 respondents\n",
        "0.65 behaviors\n",
        "0.65 students\n",
        "0.65 attitudes\n",
        "0.65 2011\n",
        "0.65 educational\n",
        "0.65 school\n",
        "0.67 will\n",
        "0.67 socioeconomic\n",
        "0.70 examined\n",
        "0.70 higher\n",
        "0.71 physical\n",
        "0.71 significant\n",
        "0.72 positive\n",
        "0.72 uses\n",
        "0.73 explores\n",
        "0.73 significantly\n",
        "0.74 regarding\n",
        "0.74 compared\n",
        "0.75 china\n",
        "0.75 type\n",
        "0.75 content\n",
        "0.75 society\n",
        "0.77 several\n",
        "0.77 issues\n",
        "\n",
        "More common in Panels\n",
        "1.78 increasingly\n",
        "1.64 global\n",
        "1.63 communities\n",
        "1.58 ethnographic\n",
        "1.54 larger\n",
        "1.48 even\n",
        "1.46 information\n",
        "1.43 workers\n",
        "1.42 find\n",
        "1.38 we\n",
        "1.38 often\n",
        "1.38 shape\n",
        "1.37 multiple\n",
        "1.37 drawing\n",
        "1.36 particularly\n",
        "1.36 networks\n",
        "1.36 network\n",
        "1.32 empirical\n",
        "1.31 across\n",
        "1.31 our\n",
        "1.31 dynamics\n",
        "1.30 labor\n",
        "1.29 little\n",
        "1.29 account\n",
        "1.29 address\n",
        "1.29 despite\n",
        "1.29 market\n",
        "1.29 consequences\n",
        "1.28 practices\n",
        "1.28 conditions\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}