TextAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# [Introduction to Text Analysis with Python](https://github.com/michellejm/NLTK_DHRI)\n",
    "### Digital Humanities Research Institute <br> June 13, 2018\n",
    "#### Michelle A. McSweeney, PhD <br> Rachel Rakov\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Concordance**: Words in their contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 25 of 1226 matches:\n",
      "s , and to teach them by what name a whale - fish is to be called in our tongue\n",
      "t which is not true .\" -- HACKLUYT \" WHALE . ... Sw . and Dan . HVAL . This ani\n",
      "ulted .\" -- WEBSTER ' S DICTIONARY \" WHALE . ... It is more immediately from th\n",
      "ISH . WAL , DUTCH . HWAL , SWEDISH . WHALE , ICELANDIC . WHALE , ENGLISH . BALE\n",
      "HWAL , SWEDISH . WHALE , ICELANDIC . WHALE , ENGLISH . BALEINE , FRENCH . BALLE\n",
      "least , take the higgledy - piggledy whale statements , however authentic , in \n",
      " dreadful gulf of this monster ' s ( whale ' s ) mouth , are immediately lost a\n",
      " patient Job .\" -- RABELAIS . \" This whale ' s liver was two cartloads .\" -- ST\n",
      " Touching that monstrous bulk of the whale or ork we have received nothing cert\n",
      " of oil will be extracted out of one whale .\" -- IBID . \" HISTORY OF LIFE AND D\n",
      "ise .\" -- KING HENRY . \" Very like a whale .\" -- HAMLET . \" Which to secure , n\n",
      "restless paine , Like as the wounded whale to shore flies thro ' the maine .\" -\n",
      ". OF SPERMA CETI AND THE SPERMA CETI WHALE . VIDE HIS V . E . \" Like Spencer ' \n",
      "t had been a sprat in the mouth of a whale .\" -- PILGRIM ' S PROGRESS . \" That \n",
      "EN ' S ANNUS MIRABILIS . \" While the whale is floating at the stern of the ship\n",
      "e ship called The Jonas - in - the - Whale . ... Some say the whale can ' t ope\n",
      " in - the - Whale . ... Some say the whale can ' t open his mouth , but that is\n",
      " masts to see whether they can see a whale , for the first discoverer has a duc\n",
      " for his pains . ... I was told of a whale taken near Shetland , that had above\n",
      "oneers told me that he caught once a whale in Spitzbergen that was white all ov\n",
      "2 , one eighty feet in length of the whale - bone kind came in , which ( as I w\n",
      "n master and kill this Sperma - ceti whale , for I could never hear of any of t\n",
      " . 1729 . \"... and the breath of the whale is frequendy attended with such an i\n",
      "ed with hoops and armed with ribs of whale .\" -- RAPE OF THE LOCK . \" If we com\n",
      "contemptible in the comparison . The whale is doubtless the largest animal in c\n"
     ]
    }
   ],
   "source": [
    "text1.concordance(\"whale\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 24 of 24 matches:\n",
      " to bespeak a monument for her first love , who had been killed by a whale in \n",
      "erlasting itch for things remote . I love to sail forbidden seas , and land on\n",
      "astic our stiff prejudices grow when love once comes to bend them . For now I \n",
      "ng . Now , it was plainly a labor of love for Captain Sleet to describe , as h\n",
      "he whole , I greatly admire and even love the brave , the honest , and learned\n",
      "to - night with hearts as light , To love , as gay and fleeting As bubbles tha\n",
      "he fleece of celestial innocence and love ; and hence , by bringing together t\n",
      "s this visible world seems formed in love , the invisible spheres were formed \n",
      "tism in them , still , while for the love of it they give chase to Moby Dick ,\n",
      "own hearty good - will and brotherly love about it at all . As touching Slave \n",
      "stubborn , as malicious . He did not love Steelkilt , and Steelkilt knew it . \n",
      " of sea - usages and the instinctive love of neatness in seamen ; some of whom\n",
      "g to let that rascal beat ye ? Do ye love brandy ? A hogshead of brandy , then\n",
      "h a sog ! such a sogger ! Don ' t ye love sperm ? There goes three thousand do\n",
      "atever they may reveal of the divine love in the Son , the soft , curled , her\n",
      " come to deadly battle , and all for love . They fence with their long lower j\n",
      "de overtakes the sated Turk ; then a love of ease and virtue supplants the lov\n",
      "ove of ease and virtue supplants the love for maidens ; our Ottoman enters upo\n",
      "go , the Virgin ! that ' s our first love ; we marry and think to be happy for\n",
      "Tranquo , being gifted with a devout love for all matters of barbaric vertu , \n",
      "ght worship is defiance . To neither love nor reverence wilt thou be kind ; an\n",
      " is woe . Come in thy lowest form of love , and I will kneel and kiss thee ; b\n",
      "es , yet full of the sweet things of love and gratitude . Come ! I feel proude\n",
      "dihood of a Nantucketer ' s paternal love , had thus early sought to initiate \n"
     ]
    }
   ],
   "source": [
    "text1.concordance(\"love\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Similar**: Words that appear in a similar environment as the target word\n",
    "This is determined based on concordance, so it returns words that appear in similar contexts\n",
    "\n",
    "Might do this with two different texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sea man it ship by him hand them whale view ships land me life death\n",
      "water way head nature fear\n"
     ]
    }
   ],
   "source": [
    "#How does Melville use \"love\"\n",
    "text1.similar(\"love\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "affection sister heart mother time see town life it dear elinor\n",
      "marianne me word family her him do regard head\n"
     ]
    }
   ],
   "source": [
    "#How does Austen use \"love\"\n",
    "text2.similar(\"love\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "join part hi hey and wb well ty lmao yeah hiya ok oh hello you what\n",
      "yes haha no all\n"
     ]
    }
   ],
   "source": [
    "text5.similar('lol')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For the next module, we need a plotting package\n",
    "\n",
    "import matplotlib as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGTVJREFUeJzt3XucJWV95/HPbxiYiYAzcgmCwDSCN/CCMBok4DQroiIafUUXCK7gqogbdY0ShYU4bfblZgGFEHUDagiJKAFNzLIaF4hKUJDLgFxFbgKiIDBhkYvI9bd/1FN2zZlzerqfvs30fN6v13mdOk899dTzVFWfb1ed6tORmUiSVGPebHdAkrTuMkQkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBGt8yLi2xFx6CTbOCwifjDJNq6PiOHJtDGVpmK7VKxzJCLOmMl1anYZIppREXF7ROw7lW1m5hsy8++mss2uiBiKiIyIh8vjnoj4ZkS8tqcfu2TmBdPVj4maru0SEadHxONlW9wfEedHxAsr2pnyY0EzzxCRxm9xZm4CvAw4H/hGRBw2W52JiPmztW7g+LIttgXuBU6fxb5oFhkiWmtExAERcVVEPBARF0fES0v5juU33t3K620iYmV76SgiLoiI93TaeW9E3BARD0XEjzvLHRURt3bK31rTz8z8ZWaeDIwAx0XEvNL+b3+zjohXRsSKiHiwnLmcWMrbs5rDI+KuiLg7Ij7a6fu8Tj//PSLOjojNepZ9d0T8DPhuRCyMiDNK3Qci4vKI2Kp3u5R2j42IOyLi3oj4+4hY1NPuoRHxs7Jtjxnntvg18FXgxf3mR8Sby2W+B0p/XlTKvwxsD/yfckbzsYnuB60dDBGtFcob/WnA+4DNgVOBcyJiQWbeCnwc+EpEPAP4W+D0fpeOIuLtNG/u7wSeCbwZ+Pcy+1Zgb2AR8EngjIjYehLd/ifgd4EX9Jl3MnByZj4T2BE4u2f+PsDzgP2AozqXdT4EvAVYBmwD/D/g8z3LLgNeBLwOOLSMZzua7XYE8Gif/hxWHvsAzwU2AT7XU2evMpbXAJ9o3/DHEhGbAIcAP+oz7/nAmcCHgS2Bf6EJjY0y8z8BPwPelJmbZObxa1qX1k6GiNYW7wVOzcxLM/Opci3/MWAPgMz8InAzcCmwNTDoN+X30FxquTwbt2TmHaWNr2XmXZn5dGaeVdp75ST6fFd53qzPvCeAnSJii8x8ODMv6Zn/ycx8JDOvpQnFg0v5+4BjMvPnmfkYTSC+refS1UhZ9tGyns2Bncp2uyIzH+zTn0OAEzPzp5n5MHA0cFBPu5/MzEcz82rgaprLdoMcGREPALfQBNJhfeocCHwrM8/PzCeATwO/A+w5RrtaxxgiWlssAT5aLns8UN6gtqP5bbz1RZrLJp8tb7D9bEdzxrGaiHhn53LZA6WtLSbR5+eU5/v7zHs38HzgJ+US0wE98+/sTN/B6DiX0HzW0vbxBuApYKsBy34ZOBf4h3J57PiI2LBPf7Yp6+muc35Pu7/sTP+aJhwG+XRmLs7MZ2fmm8vZ4pjrzMynS9+f06eu1lGGiNYWdwKfKm9M7eMZmXkm/PayyV8CfwOMtJ8TDGhnx97CiFhCE0IfADbPzMXAdUBMos9vpflQ+cbeGZl5c2YeTHO56zjg6xGxcafKdp3p7Rk9q7kTeEPPdliYmb/oNt9ZzxOZ+cnM3JnmN/wDaC7l9bqLJqC663wSuGecY62xyjojImjG3Y7FrxCfAwwRzYYNywfC7WM+zRv8ERHxe9HYOCLeGBGblmVOBq7IzPcA3wJOGdD2l2gutexe2tmpBMjGNG9a9wFExLsY8GHwmkTEVhHxAWA5cHT5Dbu3zjsiYssy74FS/FSnyp9FxDMiYhfgXcBZpfwU4FOlz0TElhHxB2P0ZZ+IeElEbAA8SHN566k+Vc8E/iQidiiB/D+AszLzyYmMfYLOBt4YEa8pZ0cfpblEeXGZfw/N5zNahxkimg3/QvPhb/sYycwVNJ+LfI7mw+RbKNfZy5vo62k+NAb4CLBbRBzS23Bmfg34FM0dQw8B/wxslpk/Bj4D/JDmzeslwEUT7PcDEfEIcC2wP/D2zDxtQN3XA9dHxMM0AXhQZv6mM//fyhi/Q3Np6LxSfjJwDnBeRDwEXAL83hh9ejbwdZoAuaG02++P/U6jufR1IXAb8Bvgg2MPd3Iy80bgHcBngZXAm2g+SH+8VPkL4Nhy6e7I6eyLpk/4T6mkmRMRQzRv4htO81mANCM8E5EkVTNEJEnVvJwlSarmmYgkqdpsfoHbjNhiiy1yaGhotrshSeuMK664YmVmbjmeunM+RIaGhlixYsVsd0OS1hkRcceaazW8nCVJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqdqMhkgED0+w/mERfG66+rMmw8Mwbx5ENI9582Dx4tH5vdMjI6Ovu9P92u33eni4eQwN9V9uZGT1ZUdGmvq9y4yMrN6HhQv796193T7Pn9+Mt11f2/bw8OpjHBkZ3T7dx7x5Tf2FC1ddpm1vZGR0Xlu/ba+7nqGhpj9tW+2y/bbR8HBTd/Hipm53OmJ02/WOodun+fNX3879tnFbp7de7zHTjq3tS+82Xrx4dJ0Ro31duLB5tGNq1927ndv1L1w4eny2j6Gh5rntQzuGdlu2y3a36fDw6PLtONrt0vtz0Lt9u/1s19/W7bYxPDzaVtuntp320dbv3cbdbdg+d4+z3jGMjIy23dZrj41++7k73Xuc9R6f7THbXXasn/veef1+tro/t93+ttuxXaYdZ/vc9rUdW3sMzYTIzJlZE02IZLLJBOofBizN5AO161y6dGmuWLGiatmI/uXtJotYdXrQvH7tdue1r7vr67ds7zp6+9ivfNB6xirvN+5uee+Yx6N2mbHq926j8bTdO/7xtN9vG6+pbDx9mKjx7JupanMypqtNWPUYHOt4rW27Zv/29mUiP/fd5fu1OVm1b+8RcUVmLh1P3Sk9E4ngYxF8qEyfFMF3y/RrIjijTH8qgqsjuCSCrUrZmyK4NIIfRfCvbXlP21tG8I8RXF4evz+VfZckTdxUX866ENi7TC8FNolgQ2Av4PvAxsAlmbys1H1vqfsDYI9MXg78A/CxPm2fDJyUySuAPwS+NKgTEXF4RKyIiBX33XffFAxLktTP/Clu7wpg9wg2BR4DrqQJk72BDwGPA9/s1H1tmd4WOCuCrYGNgNv6tL0vsHPnFO+ZEWyayUO9FTPzC8AXoLmcNflhSZL6mdIQyeSJCG4H3gVcDFwD7APsCNwAPJFJ+6b+VGf9nwVOzOScCIaBkT7NzwNelcmjU9lnSVK96bg760LgyPL8feAI4KpOePSzCPhFmT50QJ3zYPQD9gh2nXxXx7Zs2aofbkXAokWjr3unly8ffd2d7tduv9fLljWPJUv6L7d8+erLLl/e1O9dZvny1fuwYEH/vrWv2+cNNlh1fW3by5atPsZB44xo6i9YsGqdtr3ly1edFzHaXnc9S5Y0/Wnbapftt42WLWvqLlrU1O1Ot/N7x98d4/LlzTK927nfNm7r9NbrPWbasbV96d3GixaNrhNG+7pgQfNoxzTomGjXv2DB6PHZPpYsGT1GI0bH0G7LdtnuNl22bHT5dhztdumOp+13d/t2+9muv63bbaO7zdo+te20j7Z+7zbubsP2uXuc9Y5h+fLRttt67bHRbz93p3uPs97jsz1mu8uO9XPfO6/fz1b3Z6rb33Y7tsu042yf2762Y4PR/TLdpvzurAheA/xfYHEmj0RwE3BKJid2786K4G3AAZkcFsEfACfRBMklwCsyGe7enRXBFsDngRfRnMFcmMkRa+rPZO7OkqT10UTuzprRW3xngyEiSRMza7f4SpLWL4aIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqmaISJKqGSKSpGqGiCSpmiEiSapmiEiSqhkikqRqhogkqZohIkmqZohIkqoZIpKkaoaIJKmaISJJqrZWhUgEH47gGbPdj67h4dHpkZFV5/W+nqvGO87p2h7Dw03b7WOiRkZG2xg0v/d17Vi6x0s73a+97ut2urvsmqypz/36P5753fJuf4aG+rc1aFtNZCwzZTL7dV0zk+OMzJy5ta1BBLcDSzNZOYFlNsjkqUHzly5dmitWrJhMn2g3UXe63+u5arzjnK7tEbHq64muo7t8v2X77dea9fS21U73a29QvfGuc0197tfWWMfymtoYdOwP2lZr48/GZPbrumay2z8irsjMpeOpu8YzkQiGIvhJBF+K4LoIvhLBvhFcFMHNEbwygs0i+OcIrongkgheWpYdieC0CC6I4KcRfKiUbxzBtyK4urR5YJm3DfC9CL5X6u0XwQ8juDKCr0WwSSm/PYJPRPAD4O2V20mSNEnzx1lvJ5o368OBy4E/AvYC3gz8N+BO4EeZvCWC/wD8PbBrWfaFwD7ApsCNEfw18HrgrkzeCBDBokx+FcFHgH0yWRnBFsCxwL6ZPBLBx4GPAH9e2v1NJnv162xEHF76yvbbbz/OIUqSJmq8n4nclsm1mTwNXA98J5MErgWGaALlywCZfBfYPIJFZdlvZfJYuUR1L7BVWW7fCI6LYO9MftVnnXsAOwMXRXAVcCiwpDP/rEGdzcwvZObSzFy65ZZbjnOIkqSJGu+ZyGOd6ac7r58ubTzZZ5n2ilx32aeA+ZncFMHuwP7AX0RwXuZvzzBaAZyfycED+vTIOPsuSZom4w2RNbkQOAT47xEMAyszebD3A9FWBNsA92dyRgQPA4eVWQ/RXPZaCVwCfD6CnTK5pdy1tW0mN01Rn8dl2bLR6eXLV53X+3quGu84p2t7LFs2ubt9li+HCy4Y3MZU7tfu8dJO92uvW9ZOd5ddkzX1ebzrHGuZbn+WLOlfd9C2mshYZsr68vMKMzvWNd6dFcEQ8M1MXlxen15ef72dB7wa+FtgB+DXwOGZXBPBCPBwJp8uy14HHAC8ADiB5kzmCeD9mayI4IPAHwN3Z7JP+XzlOGBB6c6xmZwzkbu4Jnt3liStbyZyd9ZadYvvdDBEJGlipvQWX0mSBjFEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0QkSdUMEUlSNUNEklTNEJEkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVK1yMzZ7sO0ioj7gDsqF98CWDmF3VlbrS/jhPVnrOvLOGH9GetMjnNJZm45nopzPkQmIyJWZObS2e7HdFtfxgnrz1jXl3HC+jPWtXWcXs6SJFUzRCRJ1QyRsX1htjswQ9aXccL6M9b1ZZyw/ox1rRynn4lIkqp5JiJJqmaISJKqGSJ9RMTrI+LGiLglIo6a7f6MV0TcHhHXRsRVEbGilG0WEedHxM3l+VmlPCLir8oYr4mI3TrtHFrq3xwRh3bKdy/t31KWjRkc22kRcW9EXNcpm/axDVrHLIx1JCJ+UfbtVRGxf2fe0aXfN0bE6zrlfY/jiNghIi4tYzorIjYq5QvK61vK/KFpHud2EfG9iLghIq6PiP9ayufUfh1jnHNjn2amj84D2AC4FXgusBFwNbDzbPdrnH2/Hdiip+x44KgyfRRwXJneH/g2EMAewKWlfDPgp+X5WWX6WWXeZcCryjLfBt4wg2N7NbAbcN1Mjm3QOmZhrCPAkX3q7lyO0QXADuXY3WCs4xg4GzioTJ8CvL9M/xfglDJ9EHDWNI9za2C3Mr0pcFMZz5zar2OMc07s0xl5A1iXHuWAO7fz+mjg6Nnu1zj7fjurh8iNwNZlemvgxjJ9KnBwbz3gYODUTvmppWxr4Ced8lXqzdD4hlj1jXXaxzZoHbMw1kFvOKscn8C55RjuexyXN9OVwPxS/tt67bJlen6pFzO4f/838Nq5vF97xjkn9qmXs1b3HODOzuufl7J1QQLnRcQVEXF4KdsqM+8GKM+/W8oHjXOs8p/3KZ9NMzG2QeuYDR8ol3FO61x+mehYNwceyMwne8pXaavM/1WpP+3KZZaXA5cyh/drzzhhDuxTQ2R1/a7zryv3Qf9+Zu4GvAH444h49Rh1B41zouVro7k4tr8GdgR2Be4GPlPKp3Kss7IdImIT4B+BD2fmg2NV7VO2zuzXPuOcE/vUEFndz4HtOq+3Be6apb5MSGbeVZ7vBb4BvBK4JyK2BijP95bqg8Y5Vvm2fcpn00yMbdA6ZlRm3pOZT2Xm08AXafYtTHysK4HFETG/p3yVtsr8RcD9Uz+aURGxIc0b61cy859K8Zzbr/3GOVf2qSGyusuB55W7HTai+TDqnFnu0xpFxMYRsWk7DewHXEfT9/ZulUNprsdSyt9Z7njZA/hVOa0/F9gvIp5VTq/3o7m+ejfwUETsUe5weWenrdkyE2MbtI4Z1b7hFW+l2bfQ9O+gchfODsDzaD5M7nscZ3Nx/HvA28ryvdutHevbgO+W+tM1pgD+BrghM0/szJpT+3XQOOfMPp2pD5PWpQfNXSA30dwJccxs92ecfX4uzd0aVwPXt/2muf75HeDm8rxZKQ/g82WM1wJLO239Z+CW8nhXp3wpzYF+K/A5ZvZD1zNpTvmfoPnt6t0zMbZB65iFsX65jOUamjeGrTv1jyn9vpHOHXODjuNyrFxWtsHXgAWlfGF5fUuZ/9xpHudeNJdWrgGuKo/959p+HWOcc2Kf+rUnkqRqXs6SJFUzRCRJ1QwRSVI1Q0SSVM0QkSRVM0S03ouIkyLiw53X50bElzqvPxMRH5lE+yMRceSAeYdHxE/K47KI2Kszb+/yra9XRcTvRMQJ5fUJE1z/UET8UW3/pbEYIhJcDOwJEBHzgC2AXTrz9wQuGk9DEbHBeFcaEQcA7wP2yswXAkcAX42IZ5cqhwCfzsxdM/PRUne3zPzT8a6jGAIMEU0LQ0RqAmLPMr0LzR+nPVT+AnoB8CLgR+UvpU+IiOui+R8VBwJExHA0/y/iqzR/PEZEHBPN/334V+AFA9b7ceBPM3MlQGZeCfwdzfeevQf4j8AnIuIrEXEOsDFwaUQcGBFvL/24OiIuLOvcoPTv8vKlfu8r6/mfwN7ljOZPpnLDSfPXXEWa2zLzroh4MiK2pwmTH9J8++mraL719JrMfDwi/pDmy/JeRnO2cnn7Bk7zvUcvzszbImJ3mq+keDnNz9iVwBV9Vr1Ln/IVwKGZ+Wfl0tY3M/PrABHxcGbuWqavBV6Xmb+IiMVl2XfTfBXIK0r4XRQR59H8v4wjM/OAyW0paXWGiNRoz0b2BE6kCZE9aULk4lJnL+DMzHyK5gv8/g14BfAgcFlm3lbq7Q18IzN/DVDOIsYrGN+3rF4EnB4RZwPtFxfuB7w0ItrvUFpE871Lj09g/dKEeDlLarSfi7yE5nLWJTRnIt3PQ8b6d8CP9LweTxD8GNi9p2y3Uj6mzDwCOJbmG1qviojNS/8+WD5D2TUzd8jM88bRD6maISI1LgIOAO7P5uu57wcW0wTJD0udC4EDy2cPW9L8G9vL+rR1IfDWckfVpsCbBqzzeOC4EgBExK7AYcD/WlNnI2LHzLw0Mz9B81Xg29F8m+37o/nacSLi+dF8o/NDNP+WVZpyXs6SGtfSfM7x1Z6yTdoPvmn+R8uraL4pOYGPZeYvI+KF3YYy88qIOIvm21rvAL7fb4WZeU5EPAe4OCKS5s3+HVn+494anBARz6M5+/hO6dM1NHdiXVm+fvw+4C2l/MmIuBo4PTNPGkf70rj4Lb6SpGpezpIkVTNEJEnVDBFJUjVDRJJUzRCRJFUzRCRJ1QwRSVK1/w/tl/AKdPV2bAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1e6a4f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text1.dispersion_plot([\"whale\", \"monster\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Count a specific word - how many times does this sequence of characters occur in my document?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "282"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1.count(\"Whale\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How many **tokens** are in my text?\n",
    "\n",
    "**tokens** are unique sequences, let's start with an example:\n",
    "\n",
    "\"love\", \"bowie\", \"Bowie\", \"!\" and \":)\" are all unique **tokens**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove punctuation by only capturing the items that \n",
    "#\"are alpha\" and then lowering those\n",
    "text1_tokens = []\n",
    "for t in text1:\n",
    "    if t.isalpha():\n",
    "        t=t.lower()\n",
    "        text1_tokens.append(t)\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "218361"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#First figure out how many words are in our text. \n",
    "\n",
    "len(text1_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How many **unique** words are in my text? \n",
    "\n",
    "* first make a set that groups all the \"words\" together (numbers, punctuation sequences, etc.) - this groups together **types**. \n",
    "* Token = instance\n",
    "* Type = more general (\"bowie\" and \"Bowie\" are different types - why?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16948"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#set tells us how many unique items - it makes a set\n",
    "set(text1_tokens)\n",
    "len(set(text1_tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07761459234936642"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text1_tokens))/(len(text1_tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1_slice = text1_tokens[0:10000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2816"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text1_slice))/(len(text1_slice))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To clean our corpus, we need to:\n",
    "1. Remove capitalizaton and punctuation (DONE)\n",
    "2. Remove the stop words\n",
    "3. Lemmatize the words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import the stopwords from nltk.corpus\n",
    "from nltk.corpus import stopwords\n",
    "stops = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Remove the stop words\n",
    "for t in text1_tokens:\n",
    "    if t in stops:\n",
    "        text1_tokens.remove(t)\n",
    "    else:\n",
    "        pass\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "144313"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(text1_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16946"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text1_tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the lemmatizer function from nltk.stem \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "#the lemmatizer requires that an instance be called before it is used\n",
    "wordnet_lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lemmatize the words\n",
    "text1_clean = [wordnet_lemmatizer.lemmatize(t) for t in text1_tokens]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "144313"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(text1_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14883"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text1_clean))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'aback',\n",
       " 'abaft',\n",
       " 'abandon',\n",
       " 'abandoned',\n",
       " 'abandonedly',\n",
       " 'abandonment',\n",
       " 'abased',\n",
       " 'abasement',\n",
       " 'abashed',\n",
       " 'abate',\n",
       " 'abated',\n",
       " 'abatement',\n",
       " 'abating',\n",
       " 'abbreviate',\n",
       " 'abbreviation',\n",
       " 'abeam',\n",
       " 'abed',\n",
       " 'abednego',\n",
       " 'abel',\n",
       " 'abhorred',\n",
       " 'abhorrence',\n",
       " 'abhorrent',\n",
       " 'abhorring',\n",
       " 'abide',\n",
       " 'abided',\n",
       " 'abiding',\n",
       " 'ability',\n",
       " 'abjectly',\n",
       " 'abjectus',\n",
       " 'able',\n",
       " 'ablution',\n",
       " 'aboard',\n",
       " 'abode',\n",
       " 'abominable',\n",
       " 'abominate',\n",
       " 'abominated',\n",
       " 'abomination',\n",
       " 'aboriginal',\n",
       " 'aboriginally',\n",
       " 'aboriginalness',\n",
       " 'abortion',\n",
       " 'abound',\n",
       " 'abounded',\n",
       " 'abounding',\n",
       " 'aboundingly',\n",
       " 'about',\n",
       " 'above',\n",
       " 'abraham',\n",
       " 'abreast',\n",
       " 'abridged',\n",
       " 'abroad',\n",
       " 'abruptly',\n",
       " 'absence',\n",
       " 'absent',\n",
       " 'absolute',\n",
       " 'absolutely',\n",
       " 'absorbed',\n",
       " 'absorbing',\n",
       " 'absorbingly',\n",
       " 'abstained',\n",
       " 'abstemious',\n",
       " 'abstinence',\n",
       " 'abstract',\n",
       " 'abstracted',\n",
       " 'abstraction',\n",
       " 'absurd',\n",
       " 'absurdly',\n",
       " 'abundance',\n",
       " 'abundant',\n",
       " 'abundantly',\n",
       " 'academy',\n",
       " 'accelerate',\n",
       " 'accelerated',\n",
       " 'accelerating',\n",
       " 'accept',\n",
       " 'accessible',\n",
       " 'accessory',\n",
       " 'accident',\n",
       " 'accidental',\n",
       " 'accidentally',\n",
       " 'accommodate',\n",
       " 'accommodated',\n",
       " 'accommodation',\n",
       " 'accompanied',\n",
       " 'accompanies',\n",
       " 'accompaniment',\n",
       " 'accompany',\n",
       " 'accompanying',\n",
       " 'accomplish',\n",
       " 'accomplished',\n",
       " 'accomplishing',\n",
       " 'accomplishment',\n",
       " 'accordance',\n",
       " 'according',\n",
       " 'accordingly',\n",
       " 'accosted',\n",
       " 'account',\n",
       " 'accountable',\n",
       " 'accountant',\n",
       " 'accounted',\n",
       " 'accounting',\n",
       " 'accumulate',\n",
       " 'accumulated',\n",
       " 'accumulating',\n",
       " 'accuracy',\n",
       " 'accurate',\n",
       " 'accurately',\n",
       " 'accursed',\n",
       " 'accustomed',\n",
       " 'acerbity',\n",
       " 'ache',\n",
       " 'ached',\n",
       " 'achieve',\n",
       " 'achieved',\n",
       " 'achilles',\n",
       " 'acknowledges',\n",
       " 'acknowledging',\n",
       " 'acquaintance',\n",
       " 'acquainted',\n",
       " 'acquiesce',\n",
       " 'acquiesced',\n",
       " 'acquiescence',\n",
       " 'acquired',\n",
       " 'acre',\n",
       " 'acridness',\n",
       " 'across',\n",
       " 'act',\n",
       " 'acted',\n",
       " 'actest',\n",
       " 'action',\n",
       " 'actium',\n",
       " 'active',\n",
       " 'actively',\n",
       " 'activity',\n",
       " 'actual',\n",
       " 'actually',\n",
       " 'actuated',\n",
       " 'acushnet',\n",
       " 'acute',\n",
       " 'acuteness',\n",
       " 'adam',\n",
       " 'adamite',\n",
       " 'adapted',\n",
       " 'add',\n",
       " 'added',\n",
       " 'adding',\n",
       " 'addition',\n",
       " 'additional',\n",
       " 'address',\n",
       " 'addressed',\n",
       " 'addressing',\n",
       " 'adequate',\n",
       " 'adequately',\n",
       " 'adhering',\n",
       " 'adhesiveness',\n",
       " 'adieu',\n",
       " 'adios',\n",
       " 'adjacent',\n",
       " 'adjoining',\n",
       " 'adjust',\n",
       " 'adjusting',\n",
       " 'admeasurement',\n",
       " 'admeasurements',\n",
       " 'administered',\n",
       " 'administering',\n",
       " 'admirable',\n",
       " 'admirably',\n",
       " 'admiral',\n",
       " 'admire',\n",
       " 'admirer',\n",
       " 'admit',\n",
       " 'admits',\n",
       " 'admitted',\n",
       " 'admitting',\n",
       " 'admonish',\n",
       " 'admonished',\n",
       " 'admonishing',\n",
       " 'admonition',\n",
       " 'admonitory',\n",
       " 'ado',\n",
       " 'adolescence',\n",
       " 'adopt',\n",
       " 'adopted',\n",
       " 'adopting',\n",
       " 'adoption',\n",
       " 'adoration',\n",
       " 'adoring',\n",
       " 'adorned',\n",
       " 'adorning',\n",
       " 'adornment',\n",
       " 'adown',\n",
       " 'adrift',\n",
       " 'adroit',\n",
       " 'adroitly',\n",
       " 'adroop',\n",
       " 'adult',\n",
       " 'adulterer',\n",
       " 'advance',\n",
       " 'advanced',\n",
       " 'advancement',\n",
       " 'advancing',\n",
       " 'advantage',\n",
       " 'advent',\n",
       " 'adventure',\n",
       " 'adventurous',\n",
       " 'adventurously',\n",
       " 'adverse',\n",
       " 'advert',\n",
       " 'advertised',\n",
       " 'advice',\n",
       " 'advised',\n",
       " 'advocate',\n",
       " 'aerated',\n",
       " 'aesthetic',\n",
       " 'aesthetically',\n",
       " 'afar',\n",
       " 'affair',\n",
       " 'affect',\n",
       " 'affected',\n",
       " 'affecting',\n",
       " 'affection',\n",
       " 'affectionate',\n",
       " 'affectionately',\n",
       " 'affghanistan',\n",
       " 'affidavit',\n",
       " 'affinity',\n",
       " 'affirm',\n",
       " 'affirmative',\n",
       " 'affirms',\n",
       " 'affixed',\n",
       " 'afflicted',\n",
       " 'affliction',\n",
       " 'affluent',\n",
       " 'afford',\n",
       " 'afforded',\n",
       " 'affording',\n",
       " 'affords',\n",
       " 'affright',\n",
       " 'affrighted',\n",
       " 'affronted',\n",
       " 'afire',\n",
       " 'afloat',\n",
       " 'afoam',\n",
       " 'afore',\n",
       " 'aforesaid',\n",
       " 'aforethought',\n",
       " 'afoul',\n",
       " 'afraid',\n",
       " 'afresh',\n",
       " 'afric',\n",
       " 'africa',\n",
       " 'african',\n",
       " 'aft',\n",
       " 'after',\n",
       " 'afternoon',\n",
       " 'afterwards',\n",
       " 'again',\n",
       " 'againe',\n",
       " 'against',\n",
       " 'agassiz',\n",
       " 'age',\n",
       " 'aged',\n",
       " 'agency',\n",
       " 'agent',\n",
       " 'aggravate',\n",
       " 'aggregate',\n",
       " 'aggregated',\n",
       " 'aggregation',\n",
       " 'aggrieved',\n",
       " 'aghast',\n",
       " 'agile',\n",
       " 'agitated',\n",
       " 'aglow',\n",
       " 'ago',\n",
       " 'agonized',\n",
       " 'agonizing',\n",
       " 'agonizingly',\n",
       " 'agony',\n",
       " 'agrarian',\n",
       " 'agree',\n",
       " 'agreeable',\n",
       " 'agreed',\n",
       " 'agrees',\n",
       " 'aground',\n",
       " 'ague',\n",
       " 'ah',\n",
       " 'ahab',\n",
       " 'ahasuerus',\n",
       " 'ahaz',\n",
       " 'ahead',\n",
       " 'ahoy',\n",
       " 'aid',\n",
       " 'aide',\n",
       " 'ail',\n",
       " 'aim',\n",
       " 'aimed',\n",
       " 'aimlessly',\n",
       " 'ain',\n",
       " 'aint',\n",
       " 'air',\n",
       " 'airley',\n",
       " 'airth',\n",
       " 'aisle',\n",
       " 'ajar',\n",
       " 'akin',\n",
       " 'ala',\n",
       " 'alabama',\n",
       " 'aladdin',\n",
       " 'alarm',\n",
       " 'alarmed',\n",
       " 'alb',\n",
       " 'albatross',\n",
       " 'albemarle',\n",
       " 'albert',\n",
       " 'albicore',\n",
       " 'albino',\n",
       " 'alcove',\n",
       " 'alderman',\n",
       " 'aldrovandi',\n",
       " 'aldrovandus',\n",
       " 'ale',\n",
       " 'aleak',\n",
       " 'alert',\n",
       " 'alewife',\n",
       " 'alexander',\n",
       " 'alfred',\n",
       " 'algerine',\n",
       " 'algiers',\n",
       " 'alien',\n",
       " 'alights',\n",
       " 'alike',\n",
       " 'aliment',\n",
       " 'alive',\n",
       " 'all',\n",
       " 'allay',\n",
       " 'allaying',\n",
       " 'alleged',\n",
       " 'alleghanian',\n",
       " 'alleghanies',\n",
       " 'allegiance',\n",
       " 'allegorical',\n",
       " 'allegory',\n",
       " 'alley',\n",
       " 'allotted',\n",
       " 'allow',\n",
       " 'allowance',\n",
       " 'allowed',\n",
       " 'allowing',\n",
       " 'allude',\n",
       " 'alluded',\n",
       " 'alluding',\n",
       " 'allure',\n",
       " 'allured',\n",
       " 'allurement',\n",
       " 'alluring',\n",
       " 'alluringly',\n",
       " 'allurings',\n",
       " 'allusion',\n",
       " 'ally',\n",
       " 'almanac',\n",
       " 'almanack',\n",
       " 'almighty',\n",
       " 'almost',\n",
       " 'alms',\n",
       " 'aloft',\n",
       " 'alone',\n",
       " 'along',\n",
       " 'alongside',\n",
       " 'aloof',\n",
       " 'aloud',\n",
       " 'alow',\n",
       " 'alp',\n",
       " 'alpaca',\n",
       " 'alpine',\n",
       " 'already',\n",
       " 'also',\n",
       " 'altar',\n",
       " 'alter',\n",
       " 'altered',\n",
       " 'altering',\n",
       " 'alternate',\n",
       " 'alternately',\n",
       " 'alternating',\n",
       " 'although',\n",
       " 'altitude',\n",
       " 'altogether',\n",
       " 'always',\n",
       " 'am',\n",
       " 'amain',\n",
       " 'amaze',\n",
       " 'amazement',\n",
       " 'amazing',\n",
       " 'amazingly',\n",
       " 'amber',\n",
       " 'ambergriese',\n",
       " 'ambergris',\n",
       " 'ambiguous',\n",
       " 'ambition',\n",
       " 'ambitious',\n",
       " 'amelia',\n",
       " 'amen',\n",
       " 'amend',\n",
       " 'america',\n",
       " 'american',\n",
       " 'amid',\n",
       " 'amidst',\n",
       " 'amittai',\n",
       " 'among',\n",
       " 'amongst',\n",
       " 'amorous',\n",
       " 'amount',\n",
       " 'amounted',\n",
       " 'amour',\n",
       " 'amphibious',\n",
       " 'amphitheatrical',\n",
       " 'ample',\n",
       " 'amplified',\n",
       " 'amplify',\n",
       " 'amputate',\n",
       " 'amputated',\n",
       " 'amputating',\n",
       " 'amputation',\n",
       " 'amsterdam',\n",
       " 'amuck',\n",
       " 'amusing',\n",
       " 'an',\n",
       " 'anacharsis',\n",
       " 'anaconda',\n",
       " 'anak',\n",
       " 'analogical',\n",
       " 'analogous',\n",
       " 'analogy',\n",
       " 'analyse',\n",
       " 'analysed',\n",
       " 'analysis',\n",
       " 'analytic',\n",
       " 'anathema',\n",
       " 'anatomical',\n",
       " 'anatomist',\n",
       " 'anatomy',\n",
       " 'ancestor',\n",
       " 'ancestress',\n",
       " 'ancestry',\n",
       " 'anchor',\n",
       " 'anchored',\n",
       " 'ancient',\n",
       " 'ancientest',\n",
       " 'and',\n",
       " 'andes',\n",
       " 'andiron',\n",
       " 'andrew',\n",
       " 'andromeda',\n",
       " 'anew',\n",
       " 'angel',\n",
       " 'angelo',\n",
       " 'anger',\n",
       " 'angle',\n",
       " 'anglo',\n",
       " 'angrily',\n",
       " 'angry',\n",
       " 'anguish',\n",
       " 'angular',\n",
       " 'angularly',\n",
       " 'animal',\n",
       " 'animate',\n",
       " 'animated',\n",
       " 'animating',\n",
       " 'animation',\n",
       " 'animosity',\n",
       " 'ankers',\n",
       " 'ankle',\n",
       " 'annals',\n",
       " 'annawon',\n",
       " 'anne',\n",
       " 'annihilated',\n",
       " 'annihilating',\n",
       " 'annihilation',\n",
       " 'anno',\n",
       " 'announced',\n",
       " 'announcement',\n",
       " 'announces',\n",
       " 'announcing',\n",
       " 'annual',\n",
       " 'annually',\n",
       " 'annuitant',\n",
       " 'annus',\n",
       " 'anoint',\n",
       " 'anointed',\n",
       " 'anointing',\n",
       " 'anoints',\n",
       " 'anomalous',\n",
       " 'anomalously',\n",
       " 'anomaly',\n",
       " 'anon',\n",
       " 'anonymous',\n",
       " 'another',\n",
       " 'answer',\n",
       " 'answered',\n",
       " 'ant',\n",
       " 'antagonistic',\n",
       " 'antarctic',\n",
       " 'antecedent',\n",
       " 'antediluvian',\n",
       " 'antelope',\n",
       " 'antemosaic',\n",
       " 'anti',\n",
       " 'antic',\n",
       " 'antichronical',\n",
       " 'anticipated',\n",
       " 'anticipatingly',\n",
       " 'anticipation',\n",
       " 'anticipative',\n",
       " 'antidote',\n",
       " 'antilles',\n",
       " 'antiochus',\n",
       " 'antique',\n",
       " 'antiquity',\n",
       " 'antler',\n",
       " 'antlered',\n",
       " 'antony',\n",
       " 'antwerp',\n",
       " 'anus',\n",
       " 'anvil',\n",
       " 'anxiety',\n",
       " 'anxious',\n",
       " 'any',\n",
       " 'anybody',\n",
       " 'anyhow',\n",
       " 'anyone',\n",
       " 'anything',\n",
       " 'anyway',\n",
       " 'anyways',\n",
       " 'anywhere',\n",
       " 'aorta',\n",
       " 'apart',\n",
       " 'apartment',\n",
       " 'ape',\n",
       " 'apeak',\n",
       " 'aperture',\n",
       " 'apex',\n",
       " 'apollo',\n",
       " 'apology',\n",
       " 'apoplectic',\n",
       " 'apoplexy',\n",
       " 'apostolic',\n",
       " 'apothecary',\n",
       " 'apotheosis',\n",
       " 'appal',\n",
       " 'appalled',\n",
       " 'appalling',\n",
       " 'appallingly',\n",
       " 'appals',\n",
       " 'apparatus',\n",
       " 'apparel',\n",
       " 'apparelled',\n",
       " 'apparent',\n",
       " 'apparently',\n",
       " 'apparition',\n",
       " 'appeal',\n",
       " 'appear',\n",
       " 'appearance',\n",
       " 'appeared',\n",
       " 'appearing',\n",
       " 'appears',\n",
       " 'appellation',\n",
       " 'appellative',\n",
       " 'append',\n",
       " 'appendage',\n",
       " 'appetite',\n",
       " 'apple',\n",
       " 'appliance',\n",
       " 'applicable',\n",
       " 'application',\n",
       " 'applied',\n",
       " 'applies',\n",
       " 'apply',\n",
       " 'applying',\n",
       " 'appoint',\n",
       " 'appointed',\n",
       " 'appointment',\n",
       " 'apportioned',\n",
       " 'appreciative',\n",
       " 'apprehension',\n",
       " 'apprehensiveness',\n",
       " 'apprise',\n",
       " 'apprised',\n",
       " 'approach',\n",
       " 'approached',\n",
       " 'approaching',\n",
       " 'appropriate',\n",
       " 'appropriated',\n",
       " 'approval',\n",
       " 'approve',\n",
       " 'approved',\n",
       " 'approving',\n",
       " 'approvingly',\n",
       " 'approximate',\n",
       " 'apricot',\n",
       " 'april',\n",
       " 'apron',\n",
       " 'apt',\n",
       " 'aptitude',\n",
       " 'aquarius',\n",
       " 'arbitrary',\n",
       " 'arboring',\n",
       " 'arbour',\n",
       " 'arc',\n",
       " 'arch',\n",
       " 'archaeological',\n",
       " 'archaeologist',\n",
       " 'archangel',\n",
       " 'archangelic',\n",
       " 'archangelical',\n",
       " 'archbishop',\n",
       " 'archbishopric',\n",
       " 'arched',\n",
       " 'archer',\n",
       " 'archiepiscopacy',\n",
       " 'arching',\n",
       " 'archipelago',\n",
       " 'architect',\n",
       " 'architecture',\n",
       " 'archy',\n",
       " 'arctic',\n",
       " 'ardour',\n",
       " 'are',\n",
       " 'area',\n",
       " 'arethusa',\n",
       " 'argo',\n",
       " 'argosy',\n",
       " 'argue',\n",
       " 'argued',\n",
       " 'arguing',\n",
       " 'argument',\n",
       " 'arid',\n",
       " 'aries',\n",
       " 'aright',\n",
       " 'arion',\n",
       " 'arise',\n",
       " 'arisen',\n",
       " 'arises',\n",
       " 'arising',\n",
       " 'aristotle',\n",
       " 'arithmetic',\n",
       " 'ark',\n",
       " 'arkansas',\n",
       " 'arkite',\n",
       " 'arm',\n",
       " 'armada',\n",
       " 'armed',\n",
       " 'armor',\n",
       " 'army',\n",
       " 'arnold',\n",
       " 'aroma',\n",
       " 'aromatic',\n",
       " 'aroostook',\n",
       " 'arose',\n",
       " 'around',\n",
       " 'arpens',\n",
       " 'arrah',\n",
       " 'arrange',\n",
       " 'arranged',\n",
       " 'arrangement',\n",
       " 'arranging',\n",
       " 'arrant',\n",
       " 'arrantest',\n",
       " 'array',\n",
       " 'arrayed',\n",
       " 'arrested',\n",
       " 'arrival',\n",
       " 'arrive',\n",
       " 'arrived',\n",
       " 'arrives',\n",
       " 'arriving',\n",
       " 'arrogance',\n",
       " 'arrow',\n",
       " 'arrowy',\n",
       " 'arsacidean',\n",
       " 'arsacides',\n",
       " 'art',\n",
       " 'artedi',\n",
       " 'arter',\n",
       " 'arterial',\n",
       " 'artful',\n",
       " 'article',\n",
       " 'articulated',\n",
       " 'artificial',\n",
       " 'artificially',\n",
       " 'artificialness',\n",
       " 'artisan',\n",
       " 'artist',\n",
       " 'artistic',\n",
       " 'as',\n",
       " 'asa',\n",
       " 'ascend',\n",
       " 'ascended',\n",
       " 'ascendency',\n",
       " 'ascending',\n",
       " 'ascertained',\n",
       " 'ascertaining',\n",
       " 'ascribable',\n",
       " 'ascribe',\n",
       " 'ascribed',\n",
       " 'ascription',\n",
       " 'ash',\n",
       " 'ashantee',\n",
       " 'ashore',\n",
       " 'asia',\n",
       " 'asiatic',\n",
       " 'aside',\n",
       " 'ask',\n",
       " 'askance',\n",
       " 'asked',\n",
       " 'aslant',\n",
       " 'asleep',\n",
       " 'aslope',\n",
       " 'aspect',\n",
       " 'aspersion',\n",
       " 'asphaltic',\n",
       " 'asphaltites',\n",
       " 'aspiration',\n",
       " 'aspiring',\n",
       " 'ass',\n",
       " 'assail',\n",
       " 'assailable',\n",
       " 'assailant',\n",
       " 'assailed',\n",
       " 'assailing',\n",
       " 'assails',\n",
       " 'assassin',\n",
       " 'assault',\n",
       " 'assaulted',\n",
       " 'assembled',\n",
       " 'assembly',\n",
       " 'assented',\n",
       " 'assert',\n",
       " 'asserted',\n",
       " 'assertion',\n",
       " 'assigned',\n",
       " 'assignment',\n",
       " 'assigns',\n",
       " 'assist',\n",
       " 'assistance',\n",
       " 'assistant',\n",
       " 'assisted',\n",
       " 'assisting',\n",
       " 'associate',\n",
       " 'associated',\n",
       " 'association',\n",
       " 'assuaging',\n",
       " 'assume',\n",
       " 'assumed',\n",
       " 'assumes',\n",
       " 'assuming',\n",
       " 'assumption',\n",
       " 'assurance',\n",
       " 'assure',\n",
       " 'assured',\n",
       " 'assuredly',\n",
       " 'assures',\n",
       " 'assuring',\n",
       " 'assyrian',\n",
       " 'astern',\n",
       " 'astir',\n",
       " 'astonished',\n",
       " 'astonishing',\n",
       " 'astonishment',\n",
       " 'astral',\n",
       " 'astrological',\n",
       " 'astronomer',\n",
       " 'astronomical',\n",
       " 'astronomy',\n",
       " 'asunder',\n",
       " 'asylum',\n",
       " 'at',\n",
       " 'ate',\n",
       " 'atheism',\n",
       " 'atheistical',\n",
       " 'athirst',\n",
       " 'athletic',\n",
       " 'athwartships',\n",
       " 'atlantic',\n",
       " 'atmosphere',\n",
       " 'atmospheric',\n",
       " 'atom',\n",
       " 'atrocious',\n",
       " 'attached',\n",
       " 'attaching',\n",
       " 'attack',\n",
       " 'attacked',\n",
       " 'attacking',\n",
       " 'attain',\n",
       " 'attainable',\n",
       " 'attained',\n",
       " 'attaining',\n",
       " 'attains',\n",
       " 'attar',\n",
       " 'attempt',\n",
       " 'attend',\n",
       " 'attendance',\n",
       " 'attendant',\n",
       " 'attended',\n",
       " 'attending',\n",
       " 'attends',\n",
       " 'attention',\n",
       " 'attentive',\n",
       " 'attentively',\n",
       " 'attenuated',\n",
       " 'attest',\n",
       " 'attestation',\n",
       " 'attested',\n",
       " 'attic',\n",
       " 'attitude',\n",
       " 'attract',\n",
       " 'attracted',\n",
       " 'attraction',\n",
       " 'attribute',\n",
       " 'attuned',\n",
       " 'auction',\n",
       " 'audacious',\n",
       " 'audacity',\n",
       " 'audible',\n",
       " 'auger',\n",
       " 'aught',\n",
       " 'augment',\n",
       " 'augmented',\n",
       " 'august',\n",
       " 'aunt',\n",
       " 'auspex',\n",
       " 'austere',\n",
       " 'australia',\n",
       " 'australian',\n",
       " 'austrian',\n",
       " 'authentic',\n",
       " 'authenticated',\n",
       " 'author',\n",
       " 'authoritative',\n",
       " 'authoritatively',\n",
       " 'authority',\n",
       " 'authorized',\n",
       " 'auto',\n",
       " 'automaton',\n",
       " 'autumn',\n",
       " 'autumnal',\n",
       " 'auxiliary',\n",
       " 'avail',\n",
       " 'available',\n",
       " 'availing',\n",
       " 'availle',\n",
       " 'avast',\n",
       " 'avatar',\n",
       " 'avenger',\n",
       " 'avenue',\n",
       " 'average',\n",
       " 'averred',\n",
       " 'avers',\n",
       " 'averse',\n",
       " 'averted',\n",
       " 'avocation',\n",
       " 'avoid',\n",
       " 'await',\n",
       " 'awaited',\n",
       " 'awaiting',\n",
       " 'awake',\n",
       " 'awaken',\n",
       " 'awakened',\n",
       " 'awarded',\n",
       " 'aware',\n",
       " 'away',\n",
       " 'awe',\n",
       " 'awed',\n",
       " 'awestruck',\n",
       " 'awful',\n",
       " 'awfulness',\n",
       " 'awhile',\n",
       " 'awkwardness',\n",
       " 'awl',\n",
       " 'awoke',\n",
       " 'awry',\n",
       " 'axe',\n",
       " 'axis',\n",
       " 'axle',\n",
       " 'ay',\n",
       " 'aye',\n",
       " 'azimuth',\n",
       " 'azore',\n",
       " 'azores',\n",
       " 'azure',\n",
       " 'babbling',\n",
       " 'babe',\n",
       " 'babel',\n",
       " 'baboon',\n",
       " 'baby',\n",
       " 'babyish',\n",
       " 'babylon',\n",
       " 'babylonian',\n",
       " 'bachelor',\n",
       " 'back',\n",
       " 'backbone',\n",
       " 'backed',\n",
       " 'background',\n",
       " 'backing',\n",
       " 'backstay',\n",
       " 'backward',\n",
       " 'backwardly',\n",
       " 'backwards',\n",
       " 'backwoods',\n",
       " 'backwoodsman',\n",
       " 'bacon',\n",
       " 'bad',\n",
       " 'bade',\n",
       " 'baden',\n",
       " 'badge',\n",
       " 'badger',\n",
       " 'badly',\n",
       " 'baffle',\n",
       " 'baffled',\n",
       " 'bag',\n",
       " 'baggage',\n",
       " 'bagged',\n",
       " 'bagging',\n",
       " 'bailer',\n",
       " 'bailiff',\n",
       " 'bait',\n",
       " 'bake',\n",
       " 'baked',\n",
       " 'baker',\n",
       " 'balaene',\n",
       " 'balanced',\n",
       " 'balancing',\n",
       " 'bald',\n",
       " 'bale',\n",
       " 'baleen',\n",
       " 'baleful',\n",
       " 'baleine',\n",
       " 'balena',\n",
       " 'baliene',\n",
       " 'baling',\n",
       " 'ball',\n",
       " 'ballast',\n",
       " 'ballasted',\n",
       " 'ballena',\n",
       " 'balloon',\n",
       " 'ballroom',\n",
       " 'bally',\n",
       " 'balmed',\n",
       " 'balmy',\n",
       " 'baltic',\n",
       " 'baltimore',\n",
       " 'bamboo',\n",
       " 'bamboozingly',\n",
       " 'bamboozle',\n",
       " 'band',\n",
       " 'bandage',\n",
       " 'bandaged',\n",
       " 'bandana',\n",
       " 'bandbox',\n",
       " 'banded',\n",
       " 'bandied',\n",
       " 'banding',\n",
       " 'bang',\n",
       " 'banished',\n",
       " 'banister',\n",
       " 'bank',\n",
       " 'banked',\n",
       " 'banker',\n",
       " 'bankrupt',\n",
       " 'banned',\n",
       " 'banner',\n",
       " 'bannered',\n",
       " 'bannister',\n",
       " 'banquet',\n",
       " 'banqueter',\n",
       " 'bantam',\n",
       " 'bantering',\n",
       " 'banteringly',\n",
       " 'banterings',\n",
       " 'baptised',\n",
       " 'baptism',\n",
       " 'baptismal',\n",
       " 'baptized',\n",
       " 'baptizo',\n",
       " 'bar',\n",
       " 'barb',\n",
       " 'barbacued',\n",
       " 'barbarian',\n",
       " 'barbaric',\n",
       " 'barbarous',\n",
       " 'barbary',\n",
       " 'barbecue',\n",
       " 'barbed',\n",
       " 'barber',\n",
       " 'bare',\n",
       " 'barely',\n",
       " 'barest',\n",
       " 'bargain',\n",
       " 'barge',\n",
       " 'bark',\n",
       " 'barn',\n",
       " ...]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(set(text1_clean))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'aback',\n",
       " 'abaft',\n",
       " 'abandon',\n",
       " 'abandonedli',\n",
       " 'abas',\n",
       " 'abash',\n",
       " 'abat',\n",
       " 'abbrevi',\n",
       " 'abe',\n",
       " 'abeam',\n",
       " 'abednego',\n",
       " 'abel',\n",
       " 'abhor',\n",
       " 'abhorr',\n",
       " 'abid',\n",
       " 'abil',\n",
       " 'abjectli',\n",
       " 'abjectu',\n",
       " 'abl',\n",
       " 'ablut',\n",
       " 'aboard',\n",
       " 'abod',\n",
       " 'abomin',\n",
       " 'aborigin',\n",
       " 'abort',\n",
       " 'abound',\n",
       " 'aboundingli',\n",
       " 'about',\n",
       " 'abov',\n",
       " 'abraham',\n",
       " 'abreast',\n",
       " 'abridg',\n",
       " 'abroad',\n",
       " 'abruptli',\n",
       " 'absenc',\n",
       " 'absent',\n",
       " 'absolut',\n",
       " 'absorb',\n",
       " 'absorbingli',\n",
       " 'abstain',\n",
       " 'abstemi',\n",
       " 'abstin',\n",
       " 'abstract',\n",
       " 'absurd',\n",
       " 'absurdli',\n",
       " 'abund',\n",
       " 'abundantli',\n",
       " 'academi',\n",
       " 'acceler',\n",
       " 'accept',\n",
       " 'access',\n",
       " 'accessori',\n",
       " 'accid',\n",
       " 'accident',\n",
       " 'accommod',\n",
       " 'accompani',\n",
       " 'accomplish',\n",
       " 'accord',\n",
       " 'accordingli',\n",
       " 'accost',\n",
       " 'account',\n",
       " 'accumul',\n",
       " 'accur',\n",
       " 'accuraci',\n",
       " 'accurs',\n",
       " 'accustom',\n",
       " 'acerb',\n",
       " 'ach',\n",
       " 'achiev',\n",
       " 'achil',\n",
       " 'acknowledg',\n",
       " 'acquaint',\n",
       " 'acquiesc',\n",
       " 'acquir',\n",
       " 'acr',\n",
       " 'acrid',\n",
       " 'across',\n",
       " 'act',\n",
       " 'actest',\n",
       " 'action',\n",
       " 'actium',\n",
       " 'activ',\n",
       " 'actual',\n",
       " 'actuat',\n",
       " 'acushnet',\n",
       " 'acut',\n",
       " 'ad',\n",
       " 'adam',\n",
       " 'adamit',\n",
       " 'adapt',\n",
       " 'add',\n",
       " 'addit',\n",
       " 'address',\n",
       " 'adequ',\n",
       " 'adher',\n",
       " 'adhes',\n",
       " 'adieu',\n",
       " 'adieux',\n",
       " 'adio',\n",
       " 'adjac',\n",
       " 'adjoin',\n",
       " 'adjust',\n",
       " 'admeasur',\n",
       " 'administ',\n",
       " 'admir',\n",
       " 'admit',\n",
       " 'admonish',\n",
       " 'admonit',\n",
       " 'admonitori',\n",
       " 'ado',\n",
       " 'adolesc',\n",
       " 'adopt',\n",
       " 'ador',\n",
       " 'adorn',\n",
       " 'adown',\n",
       " 'adrift',\n",
       " 'adroit',\n",
       " 'adroitli',\n",
       " 'adroop',\n",
       " 'adult',\n",
       " 'adulter',\n",
       " 'advanc',\n",
       " 'advantag',\n",
       " 'advent',\n",
       " 'adventur',\n",
       " 'advers',\n",
       " 'advert',\n",
       " 'advertis',\n",
       " 'advic',\n",
       " 'advis',\n",
       " 'advoc',\n",
       " 'aerat',\n",
       " 'aesthet',\n",
       " 'afar',\n",
       " 'affair',\n",
       " 'affect',\n",
       " 'affection',\n",
       " 'affghanistan',\n",
       " 'affidavit',\n",
       " 'affin',\n",
       " 'affirm',\n",
       " 'affix',\n",
       " 'afflict',\n",
       " 'affluent',\n",
       " 'afford',\n",
       " 'affright',\n",
       " 'affront',\n",
       " 'afir',\n",
       " 'afloat',\n",
       " 'afoam',\n",
       " 'afor',\n",
       " 'aforesaid',\n",
       " 'aforethought',\n",
       " 'afoul',\n",
       " 'afraid',\n",
       " 'afresh',\n",
       " 'afric',\n",
       " 'africa',\n",
       " 'african',\n",
       " 'aft',\n",
       " 'after',\n",
       " 'afternoon',\n",
       " 'afterward',\n",
       " 'again',\n",
       " 'against',\n",
       " 'agassiz',\n",
       " 'age',\n",
       " 'agenc',\n",
       " 'agent',\n",
       " 'aggrav',\n",
       " 'aggreg',\n",
       " 'aggriev',\n",
       " 'aghast',\n",
       " 'agil',\n",
       " 'agit',\n",
       " 'aglow',\n",
       " 'ago',\n",
       " 'agon',\n",
       " 'agoni',\n",
       " 'agonizingli',\n",
       " 'agrarian',\n",
       " 'agre',\n",
       " 'agreeabl',\n",
       " 'aground',\n",
       " 'agu',\n",
       " 'ah',\n",
       " 'ahab',\n",
       " 'ahasueru',\n",
       " 'ahaz',\n",
       " 'ahead',\n",
       " 'ahoy',\n",
       " 'aid',\n",
       " 'ail',\n",
       " 'aim',\n",
       " 'aimlessli',\n",
       " 'ain',\n",
       " 'aint',\n",
       " 'air',\n",
       " 'airley',\n",
       " 'airth',\n",
       " 'aisl',\n",
       " 'ajar',\n",
       " 'akin',\n",
       " 'ala',\n",
       " 'alabama',\n",
       " 'aladdin',\n",
       " 'alarm',\n",
       " 'alb',\n",
       " 'albatross',\n",
       " 'albemarl',\n",
       " 'albert',\n",
       " 'albicor',\n",
       " 'albino',\n",
       " 'alcov',\n",
       " 'aldermen',\n",
       " 'aldrovandi',\n",
       " 'aldrovandu',\n",
       " 'ale',\n",
       " 'aleak',\n",
       " 'alert',\n",
       " 'alew',\n",
       " 'alexand',\n",
       " 'alfr',\n",
       " 'algerin',\n",
       " 'algier',\n",
       " 'alien',\n",
       " 'alight',\n",
       " 'alik',\n",
       " 'aliment',\n",
       " 'aliv',\n",
       " 'all',\n",
       " 'allay',\n",
       " 'alleg',\n",
       " 'alleghani',\n",
       " 'alleghanian',\n",
       " 'allegi',\n",
       " 'allegor',\n",
       " 'allegori',\n",
       " 'alley',\n",
       " 'alli',\n",
       " 'allot',\n",
       " 'allow',\n",
       " 'allud',\n",
       " 'allur',\n",
       " 'alluringli',\n",
       " 'allus',\n",
       " 'alm',\n",
       " 'almanac',\n",
       " 'almanack',\n",
       " 'almighti',\n",
       " 'almost',\n",
       " 'aloft',\n",
       " 'alon',\n",
       " 'along',\n",
       " 'alongsid',\n",
       " 'aloof',\n",
       " 'aloud',\n",
       " 'alow',\n",
       " 'alp',\n",
       " 'alpaca',\n",
       " 'alpin',\n",
       " 'alreadi',\n",
       " 'also',\n",
       " 'altar',\n",
       " 'alter',\n",
       " 'altern',\n",
       " 'although',\n",
       " 'altitud',\n",
       " 'altogeth',\n",
       " 'alway',\n",
       " 'am',\n",
       " 'amain',\n",
       " 'amaz',\n",
       " 'amazingli',\n",
       " 'amber',\n",
       " 'ambergri',\n",
       " 'ambergries',\n",
       " 'ambigu',\n",
       " 'ambit',\n",
       " 'ambiti',\n",
       " 'amelia',\n",
       " 'amen',\n",
       " 'amend',\n",
       " 'america',\n",
       " 'american',\n",
       " 'amid',\n",
       " 'amidst',\n",
       " 'amittai',\n",
       " 'among',\n",
       " 'amongst',\n",
       " 'amor',\n",
       " 'amount',\n",
       " 'amour',\n",
       " 'amphibi',\n",
       " 'amphitheatr',\n",
       " 'ampl',\n",
       " 'amplifi',\n",
       " 'amput',\n",
       " 'amsterdam',\n",
       " 'amuck',\n",
       " 'amus',\n",
       " 'an',\n",
       " 'anacharsi',\n",
       " 'anaconda',\n",
       " 'anak',\n",
       " 'analog',\n",
       " 'analys',\n",
       " 'analysi',\n",
       " 'analyt',\n",
       " 'anathema',\n",
       " 'anatom',\n",
       " 'anatomi',\n",
       " 'anatomist',\n",
       " 'ancestor',\n",
       " 'ancestress',\n",
       " 'ancestri',\n",
       " 'anchor',\n",
       " 'ancient',\n",
       " 'ancientest',\n",
       " 'and',\n",
       " 'andiron',\n",
       " 'andrew',\n",
       " 'andromeda',\n",
       " 'anew',\n",
       " 'angel',\n",
       " 'angelo',\n",
       " 'anger',\n",
       " 'angl',\n",
       " 'anglo',\n",
       " 'angri',\n",
       " 'angrili',\n",
       " 'anguish',\n",
       " 'angular',\n",
       " 'angularli',\n",
       " 'ani',\n",
       " 'anim',\n",
       " 'animos',\n",
       " 'anker',\n",
       " 'ankl',\n",
       " 'ann',\n",
       " 'annal',\n",
       " 'annawon',\n",
       " 'annihil',\n",
       " 'anno',\n",
       " 'announc',\n",
       " 'annu',\n",
       " 'annual',\n",
       " 'annuit',\n",
       " 'anoint',\n",
       " 'anomal',\n",
       " 'anomali',\n",
       " 'anon',\n",
       " 'anonym',\n",
       " 'anoth',\n",
       " 'answer',\n",
       " 'ant',\n",
       " 'antagonist',\n",
       " 'antarct',\n",
       " 'anteced',\n",
       " 'antediluvian',\n",
       " 'antelop',\n",
       " 'antemosa',\n",
       " 'anti',\n",
       " 'antic',\n",
       " 'antichron',\n",
       " 'anticip',\n",
       " 'anticipatingli',\n",
       " 'antidot',\n",
       " 'antil',\n",
       " 'antiochu',\n",
       " 'antiqu',\n",
       " 'antler',\n",
       " 'antoni',\n",
       " 'antwerp',\n",
       " 'anu',\n",
       " 'anvil',\n",
       " 'anxieti',\n",
       " 'anxiou',\n",
       " 'anybodi',\n",
       " 'anyhow',\n",
       " 'anyon',\n",
       " 'anyth',\n",
       " 'anyway',\n",
       " 'anywher',\n",
       " 'aorta',\n",
       " 'apart',\n",
       " 'ape',\n",
       " 'apeak',\n",
       " 'apertur',\n",
       " 'apex',\n",
       " 'apollo',\n",
       " 'apolog',\n",
       " 'apoplect',\n",
       " 'apoplexi',\n",
       " 'apostol',\n",
       " 'apothecari',\n",
       " 'apotheosi',\n",
       " 'appal',\n",
       " 'appallingli',\n",
       " 'appar',\n",
       " 'apparatu',\n",
       " 'apparel',\n",
       " 'apparit',\n",
       " 'appeal',\n",
       " 'appear',\n",
       " 'appel',\n",
       " 'append',\n",
       " 'appendag',\n",
       " 'appetit',\n",
       " 'appl',\n",
       " 'appli',\n",
       " 'applianc',\n",
       " 'applic',\n",
       " 'appoint',\n",
       " 'apport',\n",
       " 'appreci',\n",
       " 'apprehens',\n",
       " 'appris',\n",
       " 'approach',\n",
       " 'appropri',\n",
       " 'approv',\n",
       " 'approvingli',\n",
       " 'approxim',\n",
       " 'apricot',\n",
       " 'april',\n",
       " 'apron',\n",
       " 'apt',\n",
       " 'aptitud',\n",
       " 'aquariu',\n",
       " 'arbitrari',\n",
       " 'arbor',\n",
       " 'arbour',\n",
       " 'arc',\n",
       " 'arch',\n",
       " 'archaeolog',\n",
       " 'archaeologist',\n",
       " 'archangel',\n",
       " 'archbishop',\n",
       " 'archbishopr',\n",
       " 'archer',\n",
       " 'archi',\n",
       " 'archiepiscopaci',\n",
       " 'archipelago',\n",
       " 'architect',\n",
       " 'architectur',\n",
       " 'arctic',\n",
       " 'ardour',\n",
       " 'are',\n",
       " 'area',\n",
       " 'arethusa',\n",
       " 'argo',\n",
       " 'argosi',\n",
       " 'argu',\n",
       " 'argument',\n",
       " 'ari',\n",
       " 'arid',\n",
       " 'aright',\n",
       " 'arion',\n",
       " 'aris',\n",
       " 'arisen',\n",
       " 'aristotl',\n",
       " 'arithmet',\n",
       " 'ark',\n",
       " 'arkansa',\n",
       " 'arkit',\n",
       " 'arm',\n",
       " 'armada',\n",
       " 'armi',\n",
       " 'armor',\n",
       " 'arnold',\n",
       " 'aroma',\n",
       " 'aromat',\n",
       " 'aroostook',\n",
       " 'aros',\n",
       " 'around',\n",
       " 'arpen',\n",
       " 'arrah',\n",
       " 'arrang',\n",
       " 'arrant',\n",
       " 'arrantest',\n",
       " 'array',\n",
       " 'arrest',\n",
       " 'arriv',\n",
       " 'arrog',\n",
       " 'arrow',\n",
       " 'arrowi',\n",
       " 'arsacid',\n",
       " 'arsacidean',\n",
       " 'art',\n",
       " 'artedi',\n",
       " 'arter',\n",
       " 'arteri',\n",
       " 'articl',\n",
       " 'articul',\n",
       " 'artifici',\n",
       " 'artisan',\n",
       " 'artist',\n",
       " 'as',\n",
       " 'asa',\n",
       " 'ascend',\n",
       " 'ascertain',\n",
       " 'ascrib',\n",
       " 'ascript',\n",
       " 'ash',\n",
       " 'ashante',\n",
       " 'ashor',\n",
       " 'asia',\n",
       " 'asiat',\n",
       " 'asid',\n",
       " 'ask',\n",
       " 'askanc',\n",
       " 'aslant',\n",
       " 'asleep',\n",
       " 'aslop',\n",
       " 'aspect',\n",
       " 'aspers',\n",
       " 'asphalt',\n",
       " 'asphaltit',\n",
       " 'aspir',\n",
       " 'ass',\n",
       " 'assail',\n",
       " 'assassin',\n",
       " 'assault',\n",
       " 'assembl',\n",
       " 'assent',\n",
       " 'assert',\n",
       " 'assign',\n",
       " 'assist',\n",
       " 'associ',\n",
       " 'assuag',\n",
       " 'assum',\n",
       " 'assumpt',\n",
       " 'assur',\n",
       " 'assuredli',\n",
       " 'assyrian',\n",
       " 'astern',\n",
       " 'astir',\n",
       " 'astonish',\n",
       " 'astral',\n",
       " 'astrolog',\n",
       " 'astronom',\n",
       " 'astronomi',\n",
       " 'asund',\n",
       " 'asylum',\n",
       " 'at',\n",
       " 'ate',\n",
       " 'atheism',\n",
       " 'atheist',\n",
       " 'athirst',\n",
       " 'athlet',\n",
       " 'athwartship',\n",
       " 'atlant',\n",
       " 'atmospher',\n",
       " 'atom',\n",
       " 'atroci',\n",
       " 'attach',\n",
       " 'attack',\n",
       " 'attain',\n",
       " 'attar',\n",
       " 'attempt',\n",
       " 'attend',\n",
       " 'attent',\n",
       " 'attenu',\n",
       " 'attest',\n",
       " 'attic',\n",
       " 'attitud',\n",
       " 'attract',\n",
       " 'attribut',\n",
       " 'attun',\n",
       " 'auction',\n",
       " 'audac',\n",
       " 'audaci',\n",
       " 'audibl',\n",
       " 'auger',\n",
       " 'aught',\n",
       " 'augment',\n",
       " 'august',\n",
       " 'aunt',\n",
       " 'auspic',\n",
       " 'auster',\n",
       " 'australia',\n",
       " 'australian',\n",
       " 'austrian',\n",
       " 'authent',\n",
       " 'author',\n",
       " 'authorit',\n",
       " 'auto',\n",
       " 'automaton',\n",
       " 'autumn',\n",
       " 'auxiliari',\n",
       " 'avail',\n",
       " 'avast',\n",
       " 'avatar',\n",
       " 'aveng',\n",
       " 'avenu',\n",
       " 'aver',\n",
       " 'averag',\n",
       " 'avers',\n",
       " 'avert',\n",
       " 'avoc',\n",
       " 'avoid',\n",
       " 'aw',\n",
       " 'await',\n",
       " 'awak',\n",
       " 'awaken',\n",
       " 'awar',\n",
       " 'award',\n",
       " 'away',\n",
       " 'awe',\n",
       " 'awestruck',\n",
       " 'awhil',\n",
       " 'awkward',\n",
       " 'awl',\n",
       " 'awok',\n",
       " 'awri',\n",
       " 'axe',\n",
       " 'axi',\n",
       " 'axl',\n",
       " 'ay',\n",
       " 'aye',\n",
       " 'azimuth',\n",
       " 'azor',\n",
       " 'azur',\n",
       " 'babbl',\n",
       " 'babe',\n",
       " 'babel',\n",
       " 'babi',\n",
       " 'baboon',\n",
       " 'babyish',\n",
       " 'babylon',\n",
       " 'babylonian',\n",
       " 'bachelor',\n",
       " 'back',\n",
       " 'backbon',\n",
       " 'background',\n",
       " 'backstay',\n",
       " 'backward',\n",
       " 'backwardli',\n",
       " 'backwood',\n",
       " 'backwoodsman',\n",
       " 'bacon',\n",
       " 'bad',\n",
       " 'bade',\n",
       " 'baden',\n",
       " 'badg',\n",
       " 'badger',\n",
       " 'badli',\n",
       " 'baffl',\n",
       " 'bag',\n",
       " 'baggag',\n",
       " 'bailer',\n",
       " 'bailiff',\n",
       " 'bait',\n",
       " 'bake',\n",
       " 'baker',\n",
       " 'balaen',\n",
       " 'balanc',\n",
       " 'bald',\n",
       " 'bale',\n",
       " 'baleen',\n",
       " 'balein',\n",
       " 'balena',\n",
       " 'balien',\n",
       " 'ball',\n",
       " 'ballast',\n",
       " 'ballena',\n",
       " 'balli',\n",
       " 'balloon',\n",
       " 'ballroom',\n",
       " 'balm',\n",
       " 'balmi',\n",
       " 'baltic',\n",
       " 'baltimor',\n",
       " 'bamboo',\n",
       " 'bamboozingli',\n",
       " 'bamboozl',\n",
       " 'ban',\n",
       " 'band',\n",
       " 'bandag',\n",
       " 'bandana',\n",
       " 'bandbox',\n",
       " 'bandi',\n",
       " 'bang',\n",
       " 'banish',\n",
       " 'banist',\n",
       " 'bank',\n",
       " 'banker',\n",
       " 'bankrupt',\n",
       " 'banner',\n",
       " 'bannist',\n",
       " 'banquet',\n",
       " 'bantam',\n",
       " 'banter',\n",
       " 'banteringli',\n",
       " 'baptis',\n",
       " 'baptism',\n",
       " 'baptiz',\n",
       " 'baptizo',\n",
       " 'bar',\n",
       " 'barb',\n",
       " 'barbacu',\n",
       " 'barbar',\n",
       " 'barbari',\n",
       " 'barbarian',\n",
       " 'barbecu',\n",
       " 'barber',\n",
       " 'bare',\n",
       " 'barest',\n",
       " 'barg',\n",
       " 'bargain',\n",
       " 'bark',\n",
       " 'barn',\n",
       " 'barnacl',\n",
       " 'baromet',\n",
       " 'baron',\n",
       " 'baroni',\n",
       " 'barqu',\n",
       " 'barrel',\n",
       " 'barren',\n",
       " 'barricad',\n",
       " 'barrier',\n",
       " 'barrow',\n",
       " 'barter',\n",
       " 'bartholomew',\n",
       " 'base',\n",
       " 'basement',\n",
       " 'baser',\n",
       " 'bash',\n",
       " 'bashaw',\n",
       " 'bashe',\n",
       " 'basi',\n",
       " 'basilosauru',\n",
       " 'basin',\n",
       " 'bask',\n",
       " 'basket',\n",
       " 'bass',\n",
       " 'basso',\n",
       " 'bastil',\n",
       " 'bastion',\n",
       " 'bat',\n",
       " 'bath',\n",
       " 'baton',\n",
       " 'battalion',\n",
       " 'batten',\n",
       " 'batter',\n",
       " 'batteri',\n",
       " 'battl',\n",
       " 'baulk',\n",
       " 'bawl',\n",
       " 'bay',\n",
       " 'bayonet',\n",
       " 'be',\n",
       " 'beach',\n",
       " 'beacon',\n",
       " 'bead',\n",
       " 'beadl',\n",
       " 'beak',\n",
       " 'beaker',\n",
       " 'beal',\n",
       " 'beam',\n",
       " 'bear',\n",
       " 'beard',\n",
       " 'bearer',\n",
       " 'bearskin',\n",
       " 'beast',\n",
       " 'beat',\n",
       " 'beaten',\n",
       " 'beater',\n",
       " 'beauti',\n",
       " 'beaver',\n",
       " 'becalm',\n",
       " 'becam',\n",
       " 'becaus',\n",
       " 'becharm',\n",
       " 'becket',\n",
       " 'beckon',\n",
       " 'becom',\n",
       " 'bed',\n",
       " 'bedarn',\n",
       " 'bedeaden',\n",
       " 'bedevil',\n",
       " 'bedfellow',\n",
       " 'bedford',\n",
       " 'bedraggl',\n",
       " 'bedroom',\n",
       " 'bedsid',\n",
       " 'bedstead',\n",
       " 'bedward',\n",
       " 'bee',\n",
       " 'beech',\n",
       " 'beef',\n",
       " 'beefsteak',\n",
       " 'beehiv',\n",
       " 'beelzebub',\n",
       " 'been',\n",
       " 'beer',\n",
       " 'befal',\n",
       " 'befallen',\n",
       " 'befel',\n",
       " 'befog',\n",
       " 'befool',\n",
       " 'befor',\n",
       " 'befriend',\n",
       " 'beg',\n",
       " 'began',\n",
       " 'begat',\n",
       " 'beget',\n",
       " 'beggar',\n",
       " 'begin',\n",
       " 'beginn',\n",
       " 'begon',\n",
       " 'begotten',\n",
       " 'begrim',\n",
       " 'begun',\n",
       " 'behalf',\n",
       " 'behavior',\n",
       " 'behead',\n",
       " 'beheld',\n",
       " 'behind',\n",
       " 'behold',\n",
       " 'beholdest',\n",
       " 'behoov',\n",
       " 'behr',\n",
       " 'bejuggl',\n",
       " 'belat',\n",
       " 'belay',\n",
       " 'belfast',\n",
       " 'belfri',\n",
       " 'beli',\n",
       " 'belial',\n",
       " 'belief',\n",
       " 'beliest',\n",
       " 'believ',\n",
       " 'belik',\n",
       " 'belisariu',\n",
       " 'bell',\n",
       " 'belli',\n",
       " 'bellow',\n",
       " 'belong',\n",
       " 'belongest',\n",
       " 'belov',\n",
       " 'below',\n",
       " 'belshazzar',\n",
       " 'belt',\n",
       " 'belub',\n",
       " 'bench',\n",
       " 'bend',\n",
       " 'bendigo',\n",
       " 'beneath',\n",
       " 'benedict',\n",
       " 'benefit',\n",
       " 'benevol',\n",
       " 'bengal',\n",
       " 'benign',\n",
       " 'benjamin',\n",
       " 'bennett',\n",
       " 'bent',\n",
       " 'bentham',\n",
       " 'bepatch',\n",
       " 'bequeath',\n",
       " 'berg',\n",
       " 'berkshir',\n",
       " 'berlin',\n",
       " 'bermuda',\n",
       " 'bernard',\n",
       " 'berri',\n",
       " 'berth',\n",
       " 'beseech',\n",
       " 'beset',\n",
       " 'besid',\n",
       " 'besieg',\n",
       " 'besmok',\n",
       " 'besoot',\n",
       " 'bespatt',\n",
       " 'bespeak',\n",
       " 'bess',\n",
       " 'best',\n",
       " 'bestir',\n",
       " 'bestow',\n",
       " 'bestreak',\n",
       " 'bet',\n",
       " 'betak',\n",
       " 'betaken',\n",
       " 'bethink',\n",
       " 'bethought',\n",
       " 'betoken',\n",
       " 'betray',\n",
       " 'better',\n",
       " 'betti',\n",
       " 'between',\n",
       " 'beverag',\n",
       " 'bevi',\n",
       " 'bewar',\n",
       " 'bewild',\n",
       " 'bewitch',\n",
       " 'beyond',\n",
       " 'bias',\n",
       " 'bibl',\n",
       " 'bibliograph',\n",
       " 'bid',\n",
       " 'bidden',\n",
       " 'bier',\n",
       " 'big',\n",
       " 'bigamist',\n",
       " 'bigger',\n",
       " 'biggest',\n",
       " 'bigot',\n",
       " 'bigotri',\n",
       " 'bildad',\n",
       " 'biliou',\n",
       " 'bill',\n",
       " 'billet',\n",
       " 'billiard',\n",
       " 'billion',\n",
       " 'billow',\n",
       " 'bilocular',\n",
       " 'bind',\n",
       " 'binder',\n",
       " 'binnacl',\n",
       " 'biograph',\n",
       " 'biographi',\n",
       " 'bipe',\n",
       " 'birch',\n",
       " 'bird',\n",
       " 'birmah',\n",
       " 'birth',\n",
       " 'birthmark',\n",
       " 'biscuit',\n",
       " 'bishop',\n",
       " 'bison',\n",
       " 'bit',\n",
       " 'bite',\n",
       " 'bitin',\n",
       " 'bitingli',\n",
       " 'bitt',\n",
       " 'bitter',\n",
       " 'bitterest',\n",
       " 'bitterli',\n",
       " 'bivouack',\n",
       " 'black',\n",
       " 'blackberri',\n",
       " 'blacken',\n",
       " 'blackest',\n",
       " 'blackish',\n",
       " 'blackl',\n",
       " 'blacksmith',\n",
       " 'blackston',\n",
       " 'bladder',\n",
       " 'blade',\n",
       " 'blame',\n",
       " 'blameworthi',\n",
       " 'blanc',\n",
       " 'blanch',\n",
       " 'blanco',\n",
       " 'bland',\n",
       " 'blandish',\n",
       " 'blang',\n",
       " 'blank',\n",
       " 'blanket',\n",
       " 'blasphem',\n",
       " 'blasphemi',\n",
       " 'blast',\n",
       " 'blaze',\n",
       " 'bleach',\n",
       " 'bleak',\n",
       " 'bled',\n",
       " 'bleed',\n",
       " 'blemish',\n",
       " 'blend',\n",
       " 'blent',\n",
       " 'bless',\n",
       " 'blew',\n",
       " 'blight',\n",
       " 'blind',\n",
       " 'blindest',\n",
       " 'blindfold',\n",
       " 'blindli',\n",
       " 'bliss',\n",
       " 'blister',\n",
       " 'blisteringli',\n",
       " 'blith',\n",
       " 'bloat',\n",
       " 'block',\n",
       " 'blockhead',\n",
       " 'blocksburg',\n",
       " 'blood',\n",
       " 'bloodi',\n",
       " 'bloodiest',\n",
       " 'bloodsh',\n",
       " 'bloodshot',\n",
       " 'bloodthirsti',\n",
       " 'bloom',\n",
       " 'blossom',\n",
       " 'blot',\n",
       " 'blow',\n",
       " 'blown',\n",
       " 'blubber',\n",
       " 'blue',\n",
       " 'blueness',\n",
       " ...]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "porter_stemmer = PorterStemmer()\n",
    "t1_porter = [porter_stemmer.stem(t) for t in text1_tokens]\n",
    "sorted(set(t1_porter))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_dist = FreqDist(text1_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually nothing happens here, so check the **type** to be sure it worked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nltk.probability.FreqDist"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's **plot** the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAElCAYAAADz3wVRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXmcHVWV+L+n9yX7Bg0JCUsA2Uk3q6gsyqYzIAqCChkGJ/6EwQUHkFEGRRxxgwEcERSYgAICgiQBCSEQ9iXdWQkhZiFkJSTprL0v5/fHvdVd772q16+X1510zvfzqc97derWPffVq7qn7rnn3iuqimEYhmFkSk5fF8AwDMPYvTDDYRiGYXQKMxyGYRhGpzDDYRiGYXQKMxyGYRhGpzDDYRiGYXQKMxyGYRhGpzDDYRiGYXQKMxyGYRhGp8jLZuYi8j3gG4ACC4HLgTLgUWAYMAe4VFUbRaQQeBAoBzYDX1HVlT6fG4ArgBbg26o6PZ3eESNG6Lhx47pc7rq6OoqLi3td3t9194YO0937untDh+nuvu5MqKqq2qSqIztMqKpZ2YB9gQ+AYr//GPAv/vNiL/s98C3//Urg9/77xcBf/PfDgPlAIbA/sBzITae7vLxcu0NlZWWfyPu77t7QYbr7pw7T3TPndARQqRnU79l2VeUBxSKSB5QA64HTgSf88cnA+f77eX4ff/wMEREvf1RVG1T1A2AZcHyWy20YhmHEkDXDoaprgV8Dq3AGYxtQBWxV1WafbA2uZYL/XO3Pbfbph4flEecYhmEYvYxolmbHFZGhwF+BrwBbgcf9/k2qepBPMwZ4VlWPFJFFwFmqusYfW45rWdwMvKmqf/Ly+/w5f03SNwmYBFBWVlY+derULpe9traWkpKSXpf3d929ocN0977u3tBhuruvOxMqKiqqVLWiw4SZ+LO6sgEXAveF9i8D7gY2AXledhIw3X+fDpzkv+f5dALcANwQyqctXdxmfRy7pu7e0GG6+6cO090z53QEu0AfxyrgRBEp8X0VZwDvAS8BX/ZpJgJP++9T/D7++Iv+h0wBLhaRQhHZHxgPvJPFchuGYRhpyFo4rqq+LSJP4EJum4G5wL3AM8CjInKLl93nT7kPeEhElgHVuMgqVHWRiDyGMzrNwFWq2pKtchuGYRjpyeo4DlW9CbgpSbyCiKgoVa3Hubei8vkZ8LMeL6BhGIbRaWzkeIiH3vqQk38+kycX7+zrohiGYeyymOEI0dDUwrpt9WxraO3rohiGYeyymOEIUVLgPHf1zdkJUTYMw+gPmOEIUVqYC5jhMAzDSIcZjhDF+WY4DMMwOsIMR4jSQueqamgxw2EYhhGHGY4QxQXW4jAMw+gIMxwhSsxwGIZhdIgZjhClbVFVFo5rGIYRhxmOEIGrqsFaHIZhGLGY4QhRauM4DMMwOsQMR4ii/BxEoLEVWlrNeBiGYURhhiOEiFDix3LUNjZ3kNowDGPPxAxHEsXeXVXXaDO3G4ZhRGGGI4lg2pEaMxyGYRiRmOFIothcVYZhGGkxw5FEMO2IuaoMwzCiMcORRDB63FxVhmEY0WTNcIjIISIyL7RtF5HvisgwEZkhIkv951CfXkTkThFZJiILRGRCKK+JPv1SEZmYrTJDu+GoM1eVYRhGJFkzHKq6RFWPUdVjgHKgFngK+AEwU1XHAzP9PsA5wHi/TQLuBhCRYbh1y0/ArVV+U2BsskGwmFNNg7U4DMMwougtV9UZwHJV/RA4D5js5ZOB8/3384AH1fEWMEREyoCzgBmqWq2qW4AZwNnZKmjQ4qhtMsNhGIYRhahmf4S0iNwPzFHV34rIVlUdEjq2RVWHisg04FZVfc3LZwLXA6cCRap6i5ffCNSp6q+TdEzCtVQoKysrnzp1apfKOnn+dqb8o5ZLjxrI+YeUJhyrra2lpKQk5ZyekveGjr7U3Rs6THfv6+4NHaa7+7ozoaKiokpVKzpMqKpZ3YACYBOwl9/fmnR8i/98BjglJJ+Jc3FdC/woJL8R+H46neXl5dpVbnt+iY69fpre9vySlGOVlZWR5/SUvDd09KXu3tBhuvunDtPdM+d0BFCpGdTrveGqOgfX2tjg9zd4FxT+82MvXwOMCZ03GliXRp4V2lxV1jluGIYRSW8YjkuAR0L7U4AgMmoi8HRIfpmPrjoR2Kaq64HpwJkiMtR3ip/pZVmhxI/jqLVwXMMwjEjyspm5iJQAnwO+GRLfCjwmIlcAq4ALvfxZ4FxgGS4C63IAVa0WkZ8Cs326m1W1Oltlbp/k0AyHYRhGFFk1HKpaCwxPkm3GRVklp1Xgqph87gfuz0YZkwnmqjJXlWEYRjQ2cjyJYHZca3EYhmFEY4YjidICc1UZhmGkwwxHEsG64zUN5qoyDMOIwgxHEsG643U2ctwwDCMSMxxJtM2Oa3NVGYZhRGKGI4mStvU4zFVlGIYRhRmOJNpWAGxqCaY4MQzDMEKY4UgiN0coyAVVqG9q7eviGIZh7HKY4YigKFcAqDF3lWEYRgpmOCIozHOGw9YdNwzDSMUMRwRFee6yWIvDMAwjFTMcERT5FoeNHjcMw0jFDEcEbYbDxnIYhmGkYIYjgsLcoMVhrirDMIxkzHBEYK4qwzCMeMxwRGCGwzAMIx4zHBG0Gw5zVRmGYSSTVcMhIkNE5AkReV9EFovISSIyTERmiMhS/znUpxURuVNElonIAhGZEMpnok+/VEQmxmvsGQqtxWEYhhFLtlscdwDPqeqhwNHAYuAHwExVHQ/M9PsA5wDj/TYJuBtARIYBNwEnAMcDNwXGJlsELQ4bx2EYhpFK1gyHiAwCPg3cB6Cqjaq6FTgPmOyTTQbO99/PAx5Ux1vAEBEpA84CZqhqtapuAWYAZ2er3NA+5YiNHDcMw0glmy2OA4CNwAMiMldE/igipcBeqroewH+O8un3BVaHzl/jZXHyrNHW4rBxHIZhGClItqYOF5EK4C3gk6r6tojcAWwHrlbVIaF0W1R1qIg8A/xcVV/z8pnAdcDpQKGq3uLlNwK1qvqbJH2TcC4uysrKyqdOndrlsr+0bBu/nVvHSaML+Y+T2r1itbW1lJSUpKTvKXlv6OhL3b2hw3T3vu7e0GG6u687EyoqKqpUtaLDhKqalQ3YG1gZ2v8U8AywBCjzsjJgif9+D3BJKP0Sf/wS4J6QPCFd1FZeXq7d4Z6pr+vY66fpZfe9nSCvrKyMTN9T8t7Q0Ze6e0OH6e6fOkx3z5zTEUClZlC/Z81VpaofAatF5BAvOgN4D5gCBJFRE4Gn/fcpwGU+uupEYJs6V9Z04EwRGeo7xc/0sqxRZLPjGoZhxJKX5fyvBv4sIgXACuByXL/KYyJyBbAKuNCnfRY4F1gG1Pq0qGq1iPwUmO3T3ayq1dkstEVVGYZhxJNVw6Gq84Aof9kZEWkVuComn/uB+3u2dPEUWlSVYRhGLDZyPIJia3EYhmHEYoYjAhs5bhiGEY8ZjgjChkOzFK5sGIaxu2KGI4L8HCE/V2hpVRpbWvu6OIZhGLsUZjhiKClwcQO2CqBhGEYiZjhiKCnIBaC2yQyHYRhGGDMcMbQZjgaLrDIMwwhjhiOGNleVRVYZhmEkYIYjhqDFYWM5DMMwEjHDEUNgOGz0uGEYRiJmOGIoKXSuqhozHIZhGAmY4YihJD9ocZiryjAMI4wZjhhKgxaHjeMwDMNIwAxHDMVBH4eN4zAMw0jADEcMgauqxsZxGIZhJGCGI4agc9zGcRiGYSRihiMGC8c1DMOIxgxHDDYA0DAMI5qsGg4RWSkiC0VknohUetkwEZkhIkv951AvFxG5U0SWicgCEZkQymeiT79URCZms8wBwZQj1uIwDMNIpDdaHKep6jGqGqw9/gNgpqqOB2b6fYBzgPF+mwTcDc7QADcBJwDHAzcFxiablFqLwzAMI5K+cFWdB0z23ycD54fkD6rjLWCIiJQBZwEzVLVaVbcAM4Czs13IYuvjMAzDiESyuTSqiHwAbAEUuEdV7xWRrao6JJRmi6oOFZFpwK2q+pqXzwSuB04FilT1Fi+/EahT1V8n6ZqEa6lQVlZWPnXq1C6Xu7a2lk1N+Xzv+c2MHpjLHWePbJOXlJREpu8JeW/o6EvdvaHDdPe+7t7QYbq7rzsTKioqqkLeoXhUNWsbsI//HAXMBz4NbE1Ks8V/PgOcEpLPBMqBa4EfheQ3At9Pp7e8vFy7Q2Vlpa7aXKNjr5+mJ/98ZoI8Ln1PyHtDR1/q7g0dprt/6jDdPXNORwCVmkHdnlVXlaqu858fA0/h+ig2eBcU/vNjn3wNMCZ0+mhgXRp5VmmbcsT6OAzDMBLImuEQkVIRGRh8B84E3gWmAEFk1ETgaf99CnCZj646EdimquuB6cCZIjLUd4qf6WVZpW0FQOvjMAzDSCAvi3nvBTwlIoGeh1X1ORGZDTwmIlcAq4ALffpngXOBZUAtcDmAqlaLyE+B2T7dzapancVyA1CYl0OOQGNzK80treTl2pAXwzAMyKLhUNUVwNER8s3AGRFyBa6Kyet+4P6eLmM6RISSgjx2NjRT29TCIDMchmEYgI0cT0ubu8qmVjcMw2jDDEca2vs5rIPcMAwjwAxHGoJpR6yD3DAMox0zHGmwyCrDMIxUzHCkocTGchiGYaTQacPhx1MclY3C7GoEqwDafFWGYRjtZGQ4RGSWiAzyM9XOBx4QkduyW7S+p6TQlo81DMNIJtMWx2BV3Q5cADygquXAZ7NXrF2DtlUAm6zFYRiGEZCp4cjz80pdBEzLYnl2KUp9VFWNjeMwDMNoI1PD8RPc/FDLVHW2iBwALM1esXYN2tfkMFeVYRhGQKZTjqxX1bYOcVVdsSf0cbS1OKxz3DAMo41MWxx3ZSjrVxTbOA7DMIwU0rY4ROQk4GRgpIhcEzo0CMjNZsF2BUoLbcoRwzCMZDpyVRUAA3y6gSH5duDL2SrUrkJxvk05YhiGkUxaw6GqLwMvi8j/qeqHvVSmXQZrcRiGYaSSaed4oYjcC4wLn6Oqp2ejULsKNleVYRhGKpkajseB3wN/BPaYWrTNVWXjOAzDMNrINKqqWVXvVtV3VLUq2DI5UURyRWSuiEzz+/uLyNsislRE/iIiBV5e6PeX+ePjQnnc4OVLROSsTv7GLtPmqmoyV5VhGEZApoZjqohcKSJlIjIs2DI89zvA4tD+L4DbVXU8sAW4wsuvALao6kHA7T4dInIYcDFwOHA28DsR6ZWIrmJbAdAwDCOFTA3HROBa4A2gym+VHZ0kIqOBz+NcXIiIAKcDT/gkk4Hz/ffz/D7++Bk+/XnAo6raoKofAMuA4zMsd7cotYWcDMMwUsioj0NV9+9i/v8DXEd7KO9wYKuqBr6fNcC+/vu+wGqvr1lEtvn0+wJvhfIMn5NVivPbJzlsbdXeUGkYhrHLI6odV4giclmUXFUfTHPOF4BzVfVKETkV+A/gcuBN745CRMYAz6rqkSKyCDhLVdf4Y8txLYub/Tl/8vL7/Dl/TdI3CZgEUFZWVj516tQOf1cctbW1lJSUAPDVJzfQ0KL86Yuj0Mb6Nnlc+u7IezKvXVF3b+gw3b2vuzd0mO7u686EioqKKlWt6DChqna44aYXCbY/ACuAJzo45+e41sFK4COgFvgzsAnI82lOAqb779OBk/z3PJ9OgBuAG0L5tqWL28rLy7U7VFZWtn2fcPPzOvb6abphe12CPC59d+Q9mdeuqLs3dJju/qnDdPfMOR0BVGoGNiGjPg5VvTq0/RtwLG5UebpzblDV0ao6Dte5/aKqfg14ifZR5xOBp/33KX4ff/xF/0OmABf7qKv9gfHAO5mUuycIFnOyVQANwzAcmY7jSKYWV4F3heuBR0XkFmAucJ+X3wc8JCLLgGqcsUFVF4nIY8B7QDNwlar2Wi1ekm9rchiGYYTJyHCIyFQg6AzJBT4BPJapElWdBczy31cQERWlqvXAhTHn/wz4Wab6epK2FoeN5TAMwwAyb3H8OvS9GfhQfSd2fyeYdqSmoYXSPi6LYRjGrkCmfRwvA+/jwmqHAo3ZLNSuRImN5TAMw0ggI8MhIhfhOqQvxK07/raI9Ptp1SE80aG5qgzDMCBzV9UPgeNU9WMAERkJvED7CPB+S0KLI7+PC2MYhrELkOmUIzmB0fBs7sS5uzXW4jAMw0gk0xbHcyIyHXjE738FeDY7Rdq1KA2vyWG944ZhGB2uOX4QsJeqXisiFwCn4EZzv4kbBd7vKbbOccMwjAQ6cjf9D7ADQFWfVNVrVPV7uNbG/2S7cLsCtnysYRhGIh0ZjnGquiBZqKqVuGVk+z3BDLm2JodhGIajI8NRlOZYcU8WZFeltNBcVYZhGGE6MhyzReTfkoUicgVuMad+T7AKYI25qgzDMICOo6q+CzwlIl+j3VBU4GbG/WI2C7arEKwC6GbH7eqckIZhGP2HtDWhqm4AThaR04AjvPgZVX0x6yXbRWibq8pcVYZhGEDmS8e+hFtHY48jMBx15qoyDMMA9pDR390hmHLEWhyGYRgOMxwdYCsAGoZhJGKGowNK8tsHALqVbA3DMPZszHB0QF5uDgV5ObQqNLb2dWkMwzD6nqwZDhEpEpF3RGS+iCwSkZ94+f4i8raILBWRv4hIgZcX+v1l/vi4UF43ePkSETkrW2WOI+ggb2i2FodhGEY2WxwNwOmqejRwDHC2iJwI/AK4XVXHA1uAK3z6K4AtqnoQcLtPh4gcBlwMHA6cDfxORHKzWO4UAndVfbM1OQzDMLJmONSx0+/m+02B02lfAGoycL7/fp7fxx8/Q0TEyx9V1QZV/QBYBhyfrXJHUeKnHam3FodhGAaSzQ5f3zKoAg4C/hf4FfCWb1UgImOAv6vqESLyLnC2qq7xx5YDJwA/9uf8ycvv8+c8kaRrEjAJoKysrHzq1KldLndtbS0lJSVt+9e9sInlW5q56eQSjtp3UIfpuyrvybx2Rd29ocN0977u3tBhuruvOxMqKiqqVLWiw4SqmvUNGIIbQPgpYFlIPgZY6L8vAkaHji0HhuMMztdD8vuAL6XTV15ert2hsrIyYf8r97yhY6+fpg88+3pG6bsq78m8dkXdvaHDdPdPHaa7Z87pCKBSM6jTeyWqSlW3ArOAE4EhIhKMWB8NrPPf13hDgj8+GKgOyyPO6RWCQYDmqjIMw8huVNVIERnivxcDnwUW41oeX/bJJgJP++9T/D7++IveAk4BLvZRV/sD44F3slXuKIKoKjMchmEY2Z3utQyY7Ps5coDHVHWaiLwHPCoitwBzca4n/OdDIrIM19K4GEBVF4nIY8B7QDNwlar26jBuMxyGYRjtZM1wqFs58NgI+QoioqJUtR64MCavnwE/6+kyZkrgqrJxHIZhGDZyPCOsxWEYhtGOGY4MCJaPrW8xw2EYhmGGIwOK863FYRiGEWCGIwNKC22uKsMwjAAzHBlQHKw7bobDMAzDDEcmlNrsuIZhGG2Y4ciAAb5zfHuDzY5rGIZhhiMDDi1zExuu3NZEQ7MtIWsYxp6NGY4MGFycz8F7DaC5Fd5du72vi2MYhtGnmOHIkPKxQwGY8+GWPi6JYRhG32KGI0Mm7OcNxyozHIZh7NmY4ciQoMVR+eGWYF0QwzCMPRIzHBmy/4hSBhYIG3c0sGZLXV8XxzAMo88ww5EhIsIhwwsAc1cZhrFnY4ajExwyPB+AKusgNwxjD8YMRyc4ZIRrcZjhMAxjT8YMRyc4aGg+uTnC4vXbqWlo7uviGIZh9AnZXHN8jIi8JCKLRWSRiHzHy4eJyAwRWeo/h3q5iMidIrJMRBaIyIRQXhN9+qUiMjFOZ7YpzBMO32cQrQrzV2/tq2IYhmH0KdlscTQD31fVTwAnAleJyGHAD4CZqjoemOn3Ac4BxvttEnA3OEMD3AScgFty9qbA2PQFNp7DMIw9nawZDlVdr6pz/PcdwGJgX+A8YLJPNhk4338/D3hQHW8BQ0SkDDgLmKGq1aq6BZgBnJ2tcndEMJ7D+jkMw9hT6ZU+DhEZBxwLvA3sparrwRkXYJRPti+wOnTaGi+Lk/cJbVOPrNpKa6sNBDQMY89Dsj0KWkQGAC8DP1PVJ0Vkq6oOCR3foqpDReQZ4Oeq+pqXzwSuA04HClX1Fi+/EahV1d8k6ZmEc3FRVlZWPnXq1C6Xuba2lpKSklj5pGkfs7mulTvOGsHoQXkdps9U3pVzdifdvaHDdPe+7t7QYbq7rzsTKioqqlS1osOEqpq1DcgHpgPXhGRLgDL/vQxY4r/fA1ySnA64BLgnJE9IF7WVl5drd6isrEwrv/LPVTr2+mn66DsfZpQ+U3lP5rUr6u4NHaa7f+ow3T1zTkcAlZpB3Z7NqCoB7gMWq+ptoUNTgCAyaiLwdEh+mY+uOhHYps6VNR04U0SG+k7xM72szwg6yK2fwzCMPZG8LOb9SeBSYKGIzPOy/wRuBR4TkSuAVcCF/tizwLnAMqAWuBxAVatF5KfAbJ/uZlWtzmK5O8Q6yA3D2JPJmuFQ11chMYfPiEivwFUxed0P3N9zpeseh5UNojAvh+Uba9ha29jXxTEMw+hVbOR4FyjIy+Ho0a5/f+4qGwhoGMaehRmOLjLB3FWGYeyhmOHoItbPYRjGnooZji4yYT/nqpq3eistNhDQMIw9CDMcXWT4gEL2H1FKXVMLH26zmXINw9hzMMPRDYLxHO9vtsgqwzD2HMxwdIMJY527asmmpj4uiWEYRu9hhqMbBB3k8zc0WCe5YRh7DGY4usH4UQM5eswQdjQqX7r7DW54ciHbaq31YRhG/8YMRzfIzRH+MulEvvSJUvJzhUfeWcUZt83i6XlrgwkZDcMw+h1mOLpJUX4uXz1iIM9++1McN24om3Y28p1H53HZ/e+wfqdFWxmG0f8ww9FDjN9rIH+ZdBK/+NKRDC7O59Wlm/j+85tZtG5bXxfNMAyjRzHD0YPk5AhfOW4/Zn7/M3zusL1oaFG+9ac5bKuzfg/DMPoPZjiywIgBhdx1ybHsPySPVdW1fP+x+bbMrGEY/QYzHFmiKD+X/zhpCIOK8nhh8QbueWVFXxfJMAyjRzDDkUX2HpDH7V85BoBfTX+fN5Zv6uMSGYZhdB8zHFnmjE/sxVWnHUirwrcfmcuG7fV9XSTDMIxuYYajF7jmc4fwyYOGs2lnI1f9eQ7N1t9hGMZuTNYMh4jcLyIfi8i7IdkwEZkhIkv951AvFxG5U0SWicgCEZkQOmeiT79URCZmq7zZJDdHuOPiY9l7UBGVH27hTwt39HWRDMMwukw2Wxz/B5ydJPsBMFNVxwMz/T7AOcB4v00C7gZnaICbgBOA44GbAmOzuzFiQCH/+7VjycsRpv6jlopbZnDxvW/yo78tZPIbK3l92SY+NjeWYRi7AXnZylhVXxGRcUni84BT/ffJwCzgei9/UN08HW+JyBARKfNpZ6hqNYCIzMAZo0eyVe5sUj52GD+/4Ehu+ttCNu1sZNPOat5aUZ2QZu8BuZy15l0+c8hITjxgOCUFWfuLDMMwuoRkc04lbzimqeoRfn+rqg4JHd+iqkNFZBpwq6q+5uUzcQblVKBIVW/x8huBOlX9dYSuSbjWCmVlZeVTp07tcrlra2spKSnJmnxnTQ21FLF2RzNrtrdvq7Y1U9vc/n/k58BhIws4Zu9CJgxXRg8fkLUy9WRefanDdPe+7t7QYbq7rzsTKioqqlS1osOEqpq1DRgHvBva35p0fIv/fAY4JSSfCZQD1wI/CslvBL7fkd7y8nLtDpWVlX0ib2pu0T9Pf0N/8/wS/effvqbjfjBNx17vtoNumKb//cx7urWmMSu6ezKvvtRhuvunDtPdM+d0BFCpGdTtve0H2SAiZaq63ruiPvbyNcCYULrRwDovPzVJPqsXytkn5OXmcMjwAr5afjDXfO5gqmsaeXXpRv6+8COeW/QR97yygkdnr+bq0w/i0pPGUpiX29dFNgxjD6S3w3GnAEFk1ETg6ZD8Mh9ddSKwTVXXA9OBM0VkqO8UP9PL9giGlRZw3jH78vtLy/nFGcM56YDhbKtr4pZnFnP6r1/mb3PX0mrTtxuG0ctkrcUhIo/gWgsjRGQNLjrqVuAxEbkCWAVc6JM/C5wLLANqgcsBVLVaRH4KzPbpblbfUb6ncdCwfB7+7ARm/WMjtz77Pks27OC7f5nHmEF5fKNxJecfuy+Di/P7upiGYewBZDOq6pKYQ2dEpFXgqph87gfu78Gi7baICKcdMopPjx/JX+es4bbn/8Hq7fXcNGURP//7Ys49soyvHr9f25K2hmEY2cBiPXdDcnOEiyrGcP4x+3LPtDd5e1Mery3bxJNz1vLknLWMHzWAipGwkjUMKy1I2NRcW4ZhdBMzHLsxBXk5nDymiKvPL+fDzTX8ZfZqHqtcw9KPd7L0Y3hk0fyUc/JzYMj0FxhcnM+gojz3WZzv3Fw1NWwp3sD+I0sZM7SEgjybkcYwjFTMcPQTxg4v5bqzD+V7nzuYmYs/Ztrbi8krHcLmmka21DZSvbORzTWNNDS3snFHAxt3NETm8+CCSsC1asYMLWb/EaUUt9Qwt3YFY4aVMGZoCaOHFTOoyPpTDGNPxQxHPyM/N4ezj9ibkQ1rKS8/NuGYqvLGO5UceOiRbK9vYltdE9vrmthe30R1TRNV769kZ04pH2zayZotdazcXMvKzbUAPLtscUJeg4vzGZyv7D37TQYV5zGoOJ9BRa7lsnOza7mMHlbMmKEllBbabWYY/Ql7ovcgRISivBz2HlzE3oOLUo4fU1xNeXk5APVNLayurmX5xhreXPAPtHQYq6trWbOljtVbatlW18S2Oli1PTrI7b55lW3fh5UWMHpoMfsMLmbn9q2MXDYPEcgRIVeEnBzYsnk7z330HoV5uRTk5VDot43ra9lc9JEzVCX5DCkuYHBxvvXVGEYfYobDiKQoP5fxew1k/F4DfevliLZjqsqmnY288s5c9hk3nu31ruWyra6J7fXNvP/BGhryBrB6izM01TWNVNc0smDNNpfB6rXRSpd/EC2vrEoR5QoUT51OYV4ORfm5FObnUJiXS0tDHUNmv0l+npCXk0N+bg75uUJ+bg4127cxZu0iSgpyKSnIpbggj5KCXDby1iDAAAAgAElEQVSuq6Nm4EbfanL9PgPNFWcYsZjhMDqNiDByYCHjhuRTfuDwlONVVTvaWi6trcqmnQ2s3lLL+m31LF22grHjxtGq7lirKi2qrPjgQ/baZ18amlppaG6lsaWV+qYWVq3bQF7JILbVNbG11hmnrXVNNDa3srOhmZ1RXTXVaYb6fLAyWv7WOymiAfnC2NdfZd8hxYweWsLoocWMHlrMluomBn+8k9LCXEoK8igtyCUv1wIJjD0HMxxGVsnJEUYNKmLUIOcaq2paT/mE0SnpqvI2UV5+YKq8qqHNCIV5a3Ylhx95NPVNzsA0NLdQ39TK/Hff48DxB9PU0kpzi9LY9tnCkmUfMLJsNHWNzdQ2tlDb2EJdYwurPtpIbtGApH6fZnY2KYvWbWfRuu2pP2zmywm7BXk5FOUow158icElzp02uDifIf5z88adVO5cTkFeDgV5riVUmJfDqtX1bCz8iPxcITfHtZLycoWV1Y0M2biTgUV5DCrKpyjfppcxdh3McBi7Jfk5wsCifAYmddU0fFRA+QGprSCAqtYNlJfvnyqvqkoxTqrKS2/MZth+B7N2Sx1rvNtt7dY6Vn5UDfmF1Da0UNPYTE1DM43NrTQC2zfXgg8oSOHd96Plb6W64oAE41SQm8PAojxyaaFo5kvkiGv5CYBAY309Ja+8TI4IIkKO70PKyREGUs+ndi7niH0Hc/g+gxhSUhCtzzAyxAyHYUQgIgwuyuWYMUM4ZsyQhGPJhkZVaWhu5Y13qhh78GEucCBwq9U2sq2umQ/XrGX4yFHOwLS00tjsWkMbN21m4OAhtLQqTS2ttLQqza3K5q3backpYEd9M9vrm2hsaWVzTaNTWBNjmHbsjP09r61uN1r7DinmsH0Gkduwg2fXv5fQAsrPFdavrWF+XWp/09o1NSxtWUVxQS6FebkUF+RSnO+2dTua2W9HAwOL8ijMy0FEOnG1jd0NMxyG0U1EhKL8XAYX5XLgyNQ1UwCqqrZTXn5YhDy1tZMsDwzT9vomZs+Zz+GHH4ECraq44DJl4buLOOyww2lVbZO3qtLY3MrM2YvYmT+Ud9dtY/H67azd6lpOQHxAwvz3ouXzFsZfiOdeACAvRxhQlMeAwjxamxspffllcnMkYWuorWH4vLcpys91W14Ohfk5bNm8nWfXv4cAEmpVbdiwgxkfR7fYNmzYwYub3ie3rbXlWlwfb6hhUeNKirxxCz5XbGxEV1YntMxyc4QPtjZR+tF2cqW9nDkibK5tobqmkSIfgJGbY0bRDIdh7OIEhqkoP5e9B+QxbkRpSpptg/M5ZO+B0edvLmmLimtpVT7YtJNF67azYMly9i4b7VtArTT5z3UfbWDUqFEp+Xy04WMGDx1OXVMLdU0t1De5PqLaxhaqt9fQRC47vNtua60LZgDiW0KbNkXLl8YYsyXLo+UA78ccW7AoWj7rzWj5jFej5c/MaPuanysU5eVSmJ+LtDYxaNasdgOYn0NRXi41O7ax19K55Oe6Pq0C/7nx4x28tGlJm6vRGS1Yv24ns3cu9+HpQl6O+1y7upaVrGmLGgxC1QvyclhW3UTe6q0pRV2xpYnUV5GexQyHYexB5OYIB40ayEGjBjK65SPKyw9ISVNVVUd5+eER8nrKy4+KzDfcQmpobmFnfTM7G5qZt+BdDv3EYbQEEXTeFbdo8fvst/+B1De10tDcQkNTK/XNLaxYuYrRo0e3tZoUUIU1a9ew7777Rupeu2Yte5ft4yL1/EJDLaqsXrueQcNGUN/U6gydN3Jbt2+npHSAa521att5O2tqKSwqoqXVlbNFldZWqK1vQHNyqW9yARhNLUpTSzM7GpoB2FhbE32x166Llr+/LFoe1wdWlTp1UBszX08RDSnM4cLPxp/SE5jhMAyjRynMy6VwQC7DBxSyaVBedEtoUwHlh6S2aqoKNscYs62Ulx8Uqa+qahvl5eMj5DWUlx8ZIe/YPRgnD9yGDc2tNDS1MHvufA4+9DAX3RcYwKYW3l+6jNH7jWtrzQUtug9Xr6EsZOSCbe26jxi1117tRssbrg0fb2TQkGE0NnsD29zqQtZbWtm5cyelpamtz5ymusjr1JOY4TAMw8iQsNuQ4nz2HpDH+L1SDePQujWUH5vaQnIGMM7IfSJCXkV5+TGRZUln6LKNjVoyDMMwOoUZDsMwDKNT7DaGQ0TOFpElIrJMRH7Q1+UxDMPYU9ktDIeI5AL/C5wDHAZcIiKpQfGGYRhG1tktDAdwPLBMVVeoaiPwKHBeH5fJMAxjj2R3MRz7AqtD+2u8zDAMw+hlZHdYEEdELgTOUtVv+P1LgeNV9epQmknAJICysrLyqVOndllfbW0tJSUlvS7v77p7Q4fp7n3dvaHDdHdfdyZUVFRUqWpFhwnVj7TclTfgJGB6aP8G4Ia49OXl5dodKisr+0Te33X3hg7T3T91mO6eOacjgErNoE7eXVocecA/gDOAtcBs4KuqGjkRjYhsBD7shsoRQNREOtmW93fdvaHDdPdPHaa7Z87piLGqOrLDVJlYl11hA87FGY/lwA+zrCvS6mZb3t919/fft6fq7u+/r7/o7sltt5lyRFWfBZ7t63IYhmHs6ewuUVWGYRjGLoIZjmju7SN5f9fdGzpMd//UYbp75pweYbfoHDcMwzB2HazFYRiGYXQKMxyGYRhGpzDD0QEiMlREjheRTwdbD+dfmInM6Ft2t/8p2+UVkQtFZKD//iMReVJEJvRU/saujfVxpEFEvgF8BxgNzANOBN5U1dP98VJVjVlwOCWv76jqHckyYKKqTkiSz0mWReR3MjCO0CqOqvpgJmVJymcv4Di/+46qfpx0vDO/sQA4FLdU9BJ1E1IiIp8E5qlqjYh8HZgA3KGqsYM0/YzIe5H4+1aJyBG4GZKLQvJO/26vY19gbJKOV0Rkb9zEmgrMVtWPov6TQCYipwDjVfUBERkJDFDVD5LSXpCuLKr6ZCfKPayDvKrTlbeDvAfhrsfdwLeAS5PvW59ugaoe5X/7z4FfA/+pqifE3ZtR1xv4vKpen5T3U8BDaYo51ZcteIl7Gfg9cElU4nT3h4g8pKqXJsuAqar6WJoyZIyIXKiqj3ckSzp+NPApv/uqqs738g7vtd7ADAdtled/A/uo6jl+yvaTgO/iKtW3VPUYETkU+AlwB/BH3J+2n/+Tv6mqV4rIL4FbgDrgOeBon8814YfWV06zcJXTVwHxhwbhHoJ/xj28e6nqESJyFPDPqnqLv7EPxBmzFuCbPp+G5J/m5TcCDwA7fLmPBX4ADAF+5cshuBv1WlV9wj/84d84F3gTuNTnmazjEl/u5V62v78mfxeRBf46HAXMAZpwlUdt8n+hqoNE5GrgJmAD0BocAv4KnIozHM/iptl/Dfg6cAVwOCGDAtya5hr+AvgK8J6/hoGOKcB/AS/633E6cD9wEdH/0yNABXCIqh4sIvsAjwMXkHhPPQmMArYBJ/v8AU7z1/8HEWV9Hkgw5J7AOK8D9gO2+HINwc2s8EXgTxHlvQd3/44jsVK/WUS+CdyMu28VN4noRz7vU0P5BMxU1WNF5OfAQlV92N8j75J4bwbXtS7meo+OMHDVwNP+ekVdq2ogH5js5Zf6PMOLbRfhZpqYo6pfFpGpJN634P6LU3EVcb3XnQcsADapaop3QUR2JOVTGtK9HmgMkgLqjeucpGd/Ku5ZeyU5f89M4N+A4GXii7hIqWFE3Guq+kkROZiYez1GR/fI9gjD3WED/o6rGOb7/TxgIe5tE9xDUBj6/jYwBpgbyuPd4Lj//CLuxv4G7gbdgquUgm0x7gHYAbwU2qbgKp2XcW+9UToW441+hr8v+F1n+fyPxlXg84FRoXQjQ2ljf2OMjveBg0L7BwLv++9z/Od/AVf47+uBK4GBuErtW8B1/tgyYHiEjoU492pQxr1wb5+PAz/FGa2JuAr3jg6u4ZLgP03SsSSs25exNs3/NA9XSYR1LEhzT00DykJpy3AVRFRZl+De0H/ptyP9dqu/lr8Hzg2lP8frfSmmvHOBvwDXAd8PNn/uUmBEKK9v+/usAVgR2j7wn9Nwhmg5zmAV+vsp8t5Mvt7+/14I1PjrFWwfAH/yaeKu1fy4ezxJNhiY4r/fATwM/JPf5uIq+Vb/ud1vm3EtqBuB/8A9A8NC209wL2rBffsD4DfAVbgXq7GhbSJwF+4F6M7Q9nd/ne7w/0dQpodxLxsLgNLQ7yj1ssh7zX/G3utZqTOzlfHutNFuIMIXfR7wlH8ofox7O3ga96b7dkT6oIJY5D//AJztb6Bl/qb6TGibgKtMvtSZMvnPx5MeqGEdbMHNdQfwxSBf3JtiWGdOIEv3G2PK+0rSvgQyf1PfgJsyZm8gF6iNyCPQ+RKQl+aaVPmHVoBFQRlDvzMf95aa7hr+HdeaStYxEygI7RcAL6T5n97xn4FxDB7yuHvq3aTzc3Bv6enK+nqE3teBqgh5pf9MKW+y7qRjzwElEfK7cS8aV/vtaC8vwRnO8X6/DDgz+d4M5ZNwvXGV+jhciy1c2Q6LK2/oWs0BDgzJDwiuf1L6fGBx1P0ZyHBGYjFwAu3P5qdxBix5WxHco0n5vBX1fPjrNhE3b97E0HYBMDRNmRYCRSFZkZdF3msd1RfZ2HabKUeyTI2IDMc3QUXkRGCbqn7RH/+xiLyEu9mfAx72rhz1fv3g7Qxgqoi8j2u6XokzPJtV9aQY3X8Vkc+T6mbZJCIHhsr0ZdxbOrhJzN4TkXdwb4RneHnYraG0u5FeFpHnce6jG3ynZivwdxGZjnt4wbkSgmldVqf5jVEsEpFngce8zguB2d63P9lfhyvU9RfsB6wVka/hFuVSnKsrcGGsAGaJyDMkut9mi8gQnFGuAnYC7/hrB7DV94F8hKuUlqS5hrXAPBGZ6XUc4+UrgbdF5Gl/3nm4Bzbuf3pMRO4BhojIvwH/6sv3pah7ClgYuuYKXIwzlAelKWupiJyiqq/5YyfjKo0NIvIjnFtKcS67zf6cmSJyG4n9ALNF5EhVXUgqNwBviMjbSdf8fZ//k7j76SER+YOq3kW7KwWcK0Vxb+LhezMg+XoDoKqXJPntR4jI/ur89rNirtVTwEsissJnMw64PMkdlQt8Anc/AowUkf1UdZW/hvvhnqMV/vznSOzH3D/iGiEib4rIRcATXlQD5IvIdtz/tD1I6n6eDhKRh1W1KSKvkSJygKqu8Pv741r9f8Ddg0/5pOcD9wF5MfcapK8vehzr4wDERYPcBRyBe6MZCXxZVRfEpB+Be3v/LO4GeR74jqpu9seHAttVtUVE3sC9YSwlum/gYdzb22m4PoUv4yrDn+EexpNxbq4PgK+r6koR+UxUuVT1Zd9xOp7Eyu1VXMW4QlW3+gptX+BrOJfUKb48rwAnqur1Hf3GiGvyQJS8vWj6r0npx/n8P+mvw+vAd/3vuykmn4N8GV8F6oFBqrpAXBDDX3F9KA8AA3CunOdJvYZfU9UPRWRiUt7BipLzY3SXEf0/zcMZquNx12m6qs4I3VOH41pFbfeUN6ZBx+crqvqUiBwQUdbg/y7H9bMM9udsxVUaK3F9QYFxeAX4ibrO8b/i7uXJuJeBIb78xbjKsoFEP/w7uP6ihbT3K4FzZ52kPkBCREpxFetR4YsTd0+GGJdGHum39/lGXasLgen+3PP8NfshzmgFNAMfquoan8+5pPbBXQn8Avg/3LU+xt8X5+JcSFHMw923J+Hu27eA7+H6lsoD4x5GRMbjWjYJQR1e/724/yO4Ft9U1en+/ml7LlV1rs/rc7iWXdu95uVR98/XNE0ASncww+HxnWKH4P6QJVFvCB2cf7qqvigx0TMaEzUj7dEpwecA4ElVPdMfLwVyVHVHBmVIjgI7CVchXxtzyh81tWMyKMcwVa1OOha8CWaEiLymqqdEdCi2vY11cP5An26n3z8d9zB9CueemId7qFKifnz6QlwFPw7nstvu87s5098Qyivyf8IZj4tx7pP7cQ+zikgR8O+4fqUdOFflXeo7YdPoif2/xUU8iapuy6C881T1GP99rBc/i6sUE/CG9A1VPTkin4XAcdreeVyEc4scGaP3F5oaJZUiC5cTF6wxR1WP9bIFyYYp6ZxwRNd/4/oYgoiu2ChBfz8cirv/3lfVehGZrarH+XKcAPwnzhjNjVAd9QI0SFW3S0ykmzfir+EM/O24vozLcf/jTaEy4cvUICI3416O3tCIiEZpj3wL68j1L6oZ1xfdwVxV7RxPe7TJBBFBk8L4ROQuUiMzAk7w2z/5/SBd0LKIC7cMIkFq/dvWZmB/f0N9KSiTiIBz9YyNq4hxvtQgCuw0EQnemn6TpHMfXItDxEU8BQzEGRpwLrdzVHW7/+2fwPmvjwhnJCLXqeovY67NHABVHZgkD5rp/0n7Ncen/Vdx7qaHcJU9IrIJuMwb5pf9bzwN+H/A4SLyKNFRcRfi3s7n4CKQwvrj3gJPwHUeJ7ukIv8nVT1TRG7EvQVeDvxWRB7D+bfX+3KBe0P+bxGpJ/q/2w+4jMT/+xjgbxHXLvj6UFRZ1YWL14l3b3nD8EnaO/nDBPsviVtFcyqJLqYHiHabxPE5oM1I+GtxqH/jj3p26r2hDVwspaFzL8C1CEbhrlNwrZb7JJ8Hfq+qT4vIj8W5kMJRgneJyLWqGriVymm/347y13GNOPfn34AZuLf1dap6edSPExcC+2+hfD4rIi/g7sfAPRyguBecYlWdKSLi/4tXgJ+ISHLr9kBfppU41+2d/ll/Fdea3Jv2yLfW0PU4APhARJ7DtZReJMuY4aAtbjsqhDA5/rsyTTZV/vNbhCr8UF5xTPM37q9wFZziXCFP43ziVbQ/yHdCdEXsf8ds/xaFiBSq6lfCb56hdINxnXM/x0WFBOwItTL+G2c8Po9riT2Ic20lE/R7pLs2UTyNeyBeoP2aB9yLC19+yZf3VOAPIlKH8+2/6c89TlU/FpG/4yq4H/rz/4F7gERVz47R/wDtb4Gn4d8CgT/7c7+AM0wTgY3Azoj/6Q/gXkNF5COcy6oZd21Px7UwXvL6CkRkvqoeLSLHkOh+mS/OpfkWia6i0SS6X5KJKyu4+3Cy/6/BVYij/PFw+O56EfkYF6HWiOvrCFBVPUBEZtHuNrk8cJuEEZFv4VwvByS9jAzBVXo/TD7Hc6HE++1/CfyTqib0rYnINH/OZ4Ff+JesHK/juKCV4Sv5F4An4p5xjenHlPgQ/StIvG+f8RkltESSqBeRHGCpiPw77r8fRftLZpigVXO/uJD9i3DRXZNw99fhqhq1SNMhPr+rgPtEZBrwaJTrrEfQLPW6704bnQxv7SCv53AdvuGQx2syPLcQGOy/dzqUjpgoMH/sZFxc/2XBlkF+5wNv4Cqz8T18zWMjPogJt8RV8q/g3gx/jKuci4mPYLoXODJGR5X/XBiSvRqSLwjJX07zP30bZ9yn41o4+V7+f8Ca0DknAL/z6RfiwjpvxkVgXU1EVFAG1zC2rL6MX8P19dyGM5KzgbNCac/0x04kIlqok2XpMEoq5ryr/XX7FW4Q4edCx1Kiybw8LqIrXZRgZ0PY48KpI+9b3LiWSBmuhTwA9yLwAK4/7sQ0uv/on7ungGtw3pA8YiLfIs4finvRa+nOf5pusxaH411cMzCjKAT/JnM9qSOYT8cNaIp7y43LL2GkrW+uviHxETCRaPzbU6Ytqih33CBc593V3n337ZjfcDDuzajtd/gynR5T3Gkicq66BbqSWeHdP8Ho4a8DH6jq97yuAbgWwgO4/+1NSYxgWublBbhom5TOYFLfAtfi3gK3eJ3rfWtrHTBaRPJJHK08y7/1jgAuUN8JKSILvdslH9hHRFb6co3FDX47BVdpBJ3Nv8C1oB7wb9zTSIw6qhYXeBDVag364RLK6mVP0+6mW+tle6vq9FDez4vIXap6jYiMkoj+Oc18RLuq68i/KvmAOP//wbhggU/g/pdcXETSnbh+uaCP6IXQqZXe3fo3Eq/Jk4Rcv6q63l+DdFGCnXrGcWNaHhORG7yOZhFpIem+FdfnUwKMEBcUEx5wuY8/d7ZPq5rkApPoSL3h/vpsxY312uT1R0a+Bc+kuACFr+DG88zGGb6ssEcbDmkP34sMIVTVf445NXARfJ5UF0GnKvw0lfopxFd6HaKqL4d0VACHqX8d6YBkl1NVZKpUHsdFrfyRVNdTFN8B/lNEGnAVYLjD/F9xb+RBCOgruGsRNPPLcf059+NaCVtxA9wOFJHXcW+aX8KFksbxXdwD/23c4MHTcC2xkd69831cRTcIFzVzN84Y/M6ffylwt6p+IynfL3Twu6eReH1a/G9sxL11/5B2IxH4r6eF0hfhBpeuA26JKStEvMCIyFkicj2uRQyukskTN73LIJyrI9O+uWQexv32KqJ9/dW4IILHcVFUl+EGjP5IovuI7vNlqvXHwnnFlWkNzgh/yuu/V1WDvpnkEHaXWfwzHhmiT+p9W+C3Jv/bg+u2A/itP/ck/3sGAG0zTeDu05RIveAFUFy/4lm4/qdc3H/+IqmRb4jIB7g65DHc7A8ZTRPUVfboqCpvoQXXAXdd+BDwC1U9Iea8KlUtl1D0h4jsxL2Z5+HCYTOq8EVkMRGVurhImKGEfOHAVu1CeJ2IPA5827+ZZYXgmnTynJTQ4SSDNxhoVR8hIiLX4q5Dlao2h9IF4ZljcAbjBOBGVZ2TRncFrpIeizMIXn3s/zRfVY/uSNYRInIN7kUj3Nn8fziXzQka7b9OziMHeCFNaw4RuRfXx7IwJBuBc1kF/RWv4Qz0Nlyo87Ek9c1pJyPQ/IvQK7j5ld4PyStVtSLpmWmL5PKV6eW4QbMv4dxnM1T1uhQl8bpvISLCzR+LDWGPyauzIfr/BfyPugirG3EDfH+qqnN8C+HLuFHsQeTYu7h7OypS707cc/9pXB0Q9Ol9QyMi33x+g9QHsvQGe3SLI7hpRCQ/+QYSkeI0p0a5CDYT3dnVEXFN6PNx05W0DbzCdRrelWnGXWlRichjqnqRuDDMlLeK5IpV2sMQp4rIlbgKMcHVElO2qAkk3wDOEJHjcA9+MPvqNuBfVfVXMT/1RlV93LsKPouLIrsbZ0Di+DMuTDnh7U1So2YCWkTkQFVd7tMdQGYtqwRU9TaJ6GwWF2qcMndXDONxb67p5ic6BfgX/yaayQvMbbS7toKQ4a68VT7gdd/lr9FcXKVXK24g6Txx87mtxw2Y+zbOkG7CvXVfq6pN3jguE5EPSY0ai+yITtd6UTfGKe2Enkl5zfHGJgjRR1XflfgZgL+sbs6vU3CRZQn3oKquFgk3wmih/TkJIvWqceNLLsC9CN2hquugzaUZGfnmn7FG7ybM6Fp1lz3acEh8JEg4LDWKKBfB1Z1pDXRUqeMqrihfeMaGA9fZGLSozg+r97IovuM/O3K5BCS7Jq4lscI5II2ecOhwMIEkuGb9lar6KoB/GB/ADfCLIqjAE8IzOyj3RlWdkiwUF90UFe31ZyJGK3egIxLfEkpuDbXgKtWXSPJfS3v4deAG+QjXx/YH3PW+x6ddICIP4ybZPCdC9SMikhKq2dW+uZjfFhUyfQTO5ZKDG9vyPVzr8AL8FBzJz46qtopIMEXNWbhAgq+RfvYCVCMj3J4QkVW48OZZRIfqRhEO0b9cXF9T1H+upL8H42ZhWCTRkXpfiKjwz6E9wi4h8g33jD2Ec81mfK26hWap13132OhiJEgP6f4MbmbOt0mcwyqQRc5X00VdUfP4LOhKXml0XIQbyQ1ugringAlp0kdOIOk/I+dmSpNX5IR7HZT3DNwb7iW4CizY4qJminCurRf9dkP4/+mB6zcxauvgnE7NT4TrGwq2T+Iiqn7pj8VGoHXyd8zEhRXf7q/nKC//TkTaFFnS8cg5yNKkj4twy8EZ48gJPWPyegjXAv4d7mXtLuDOrtyDuP6VP+MmO/wYN4XLcF/GgaFnZh5uhonYiR976lp1d9ujWxzqRuBuI2Ye/zji3BnaiWahduwme4DODbyKKmenW1SSOriw7RDpR3v/SF0USmRTPYKEgVcisoX2QXrviItYCuYo+gouimkCtL2xh7kI5xv/tbopVcqIHy0fcDluxG4+iVO3x0V7PYgbef5Tv38JrnK5sAM9GaGqk9MdF5F/JhTRparT6OT8RKqaHOjwuojsFJGzaX+r7lIwRogFOMN0BO7Z2ioib+IMYfII/3+JkIUJXMLJc5DFkRDhFqCu9fKhJrqmNpN+IbvYgBKJWG+EmHtQXKf2paqaMgZKRAIXa/DMBH2tU4gYXyURkX3APepmuejsteoWe3TneFcJuTOqCLkzVPWvncijrVKnfSQs+EpdVb8uMfPVdEJHJgP9egQRmasR6zOo7wzs4NzP4EOHVbXRu2sgNcInqMxiO4U7Ud6FGpo2I8kdVIqrPNuivXDhwN3uHI8oR7o+JVU3YPBWnOvnz15+CS767Q90Yn4iSZwWIwdXwf8ONx4mkri8OkLaQ6b/i/bpXl4NJRkENKvqZ9PkEcxBdiQugGAArj/rni6U51c4V2c4VHeBxk+FEhlQIvEDCSPD1P05s1T11Ah5p54ZEfkjEeuQqOo3evJaZYIZji4gEaOxu5BHr1XqvYG4kaprcZ3T5bhpEd7pSsUqqZMcKoB2YY6pNDr+ANyuqu9FHIuaKPJynO/6LZ/mBJwr6cpulqNMVdf7TtxwK0lwbqSLfGvxGFVt9efk4jqdj6MTc3H5jvLAODbjDM3N2oOjiyU1ZHo+sBr3Rp5wn+Mq7uaUTNrzCk+7E45869J9ICJfwrnoghexpyLShPsej8HNRRbuexxP5qHtQZ4/w70Y/QXnhgq4mU48M1EvKtI+G0GPXquO2KNdVd0g3eC1jOiqm2wXpivuojh2hr4X4Trqe7qj7xRgYnLUEe0D0oJor8/gHvY1wGW+kzU8oGwQNBYAAAbPSURBVK9bhN5oD0p+u/cBAwFDcFE30D5LbniQX8JcXDG69u9eaTOiGNd3UpVkFG6XxCV5l6QzGp6oaXe6jPcIdOQV6CigpLMDCcG1CKE9+CO4175A556ZdJF9PXqtOsJaHJ0gyf8/APcHBTd/Ov+/0Q3829QUVT2rB/McG3NoGonLBZ+OiwL696jEXXXlhMqRicvyYtyqf7Nwlc6ncZ3zP1LVI8iQDnzkWUVErsCNIQmW5P0MrrVzf5pz3u3M74vJo0t9dhK93OuncUY6pSWi8QMJEZHvkxh5qLjWYaWqzuvEbzkd54ZKiOxT1Zd64lp1BmtxdAL1kwt6P+eruEFO2Qt5MwJKiA/r7RJp+gHqNXGiyBdF5LbuGog0PIybFymdy/LzuHEtW4BVwPXqFsQ6XTo3LU3k6HfceKFscx1wrLavWTMcF7UUazjowrQ7yWjMhKBxpAkoGYkzFoPIPLQ9oBzX2T7Fp/88bkqQb4rI46r6ywyLNxwXdDCO9nVIgin2u32tOoO1OLqApK4LMRdnRNJFiBgZktRRnIt7aG9W1d/2gu6ncP0Z38V1Gm/BhXWmrGPRW0TcbyNw7rxtdG6Wgh4Z/d4VxK38d46qNvr9AtwEnCmd46H/v1OzMPRQOdP2PSa3RPw5Ha0fMh23lG+wrswA3AqCX8S59A7LsGxR65A8gluaulevlRmOLuI7KMODnOpU9dD0ZxmZkORGagY2ZOAPz0Y5EqK9elt/UlnC99tVuNHdZ0SlTdOamgNcmOQjfyK5IswGIvIgLuInYUle3BT4qOptobRxbkR82my1/mLJxKWY5tzFuLXaA6NZiBtv84lMIw/9eVFRWO/iWjCRZOtamauqC/i3p5R1Ifq2VP2HvqgYotCYeYx6m4j7bUIX77draR/9LrgO/i6Nfu8Cy0mscJ/2nymupF3l/08iE5diunPfEreOPbipiR4Rt2hVZwIs1krqOiQtfWJIrcXReUTkdpzfsgE3kO4V3DrMdWlPNIwu0JP3m69sgvmX3lfVrEfgGCBu3fi2ySVVtbMLnyEiJbgorIWqutRHYR2pqs/3bGkzKIsZjq4TGuT0H7i1Dgr7uEhGP6Yn7jeJGPWsSUskZwNxsy3ELXNr7GaYq6oLRAxyCtaFMIwep6fut7hRz0Qs6JUF0i1za+xmmOHoGnGDnAwjG/TU/daZBb16muGqep+IfMf3Hb0sbhZdYzfEDEcX0Ph1IQyjx+nB+60ro557inTL3Bq7GWY4DKOfI11fIrkniVrD5ru9oNfIAmY4DKP/05UFvXqaC3HRRO8Cp/mJJH+NW83O2M0ww2EY/Rzt+hLJPclRqro1VKZqEclo4Jux62GGwzD6OWnmX+poieSeJEdEhqrqFl+mYVj9s9tif5xh9H+6M+q5p/gNbiK+J3D9LRcBP+sl3UYPYwMADcPoFUTkMNzEkQLM1IhFtIzdAzMchmEYRqdIt1i7YRiGYaRghsMwDMPoFGY4DKMDROSHIrJIRBaIyDwROSGLumaJSEW28jeMnsCiqgwjDSJyEm5ivgmq2iAiI4CCPi6WYfQp1uIwjPSUAZuCdStUdZOqrhOR/xKR2SLyrojcKyICbS2G20XkFRFZLCLHiciTIrJURG7xacaJyPsiMtm3Yp7way0kICJnisibIjJHRB7306ojIreKyHv+3F/34rUwDMAMh2F0xPPAGBH5h4j8zi8nC/BbVT1OVY/AzV77hdA5jar6aeD3uJXurgKOAP5FRIb7NIcA9/o1obfjBui14Vs2PwI+65d2rQSu8QPnvggc7s+9JQu/2TDSYobDMNKgqjtx62BMwq0f8RcR+RfcfEtvi8hC3NiEw0OnTfGfC4FFqrret1hWAGP8sdWqGoza/hNudbgwJwKHAa+LyDzc+hVjcUamHvijiFwA1PbYjzWMDLE+DsPoAFVtAWYBs7yh+CZwFFChqqtF5MeEVrWjfebZ1tD3YD945pIHUCXvCzBDVS9JLo+IHA+cAVwM/P/27hiloSCKwvB/OpGArkAsjLbpsw0bKzdgY+8uLCQrsBbEFVgEsgARLey1EGyFsZgRQorAvEAg8H/V4w3MvO5w74V5V9TgkrbGikNaI8lZkvHSqwnw2p6/2tzhfMDWR23wDnABPK+sz4FpkpP2HftJTtt5B6WUJ+q15JMBZ0sbseKQ1hsBt0kOgV/gndq2+qa2oj6AxYB9X4DLJDPgDbhbXiylfLaW2H2S/3+L3wA/wEOSPWpVcj3gbGkjXjkibVmSY+CxDdalnWOrSpLUxYpDktTFikOS1MXgkCR1MTgkSV0MDklSF4NDktTF4JAkdfkDEcs+JJtlu64AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a28c22a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_dist.plot(50,cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It may be a little easier to look at as a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 7757),\n",
       " ('a', 2531),\n",
       " ('whale', 1494),\n",
       " ('his', 1348),\n",
       " ('it', 1044),\n",
       " ('that', 1006),\n",
       " ('one', 940),\n",
       " ('in', 812),\n",
       " ('all', 719),\n",
       " ('this', 700),\n",
       " ('to', 685),\n",
       " ('like', 650),\n",
       " ('i', 620),\n",
       " ('ship', 605),\n",
       " ('is', 575),\n",
       " ('and', 573),\n",
       " ('upon', 566),\n",
       " ('wa', 550),\n",
       " ('sea', 542),\n",
       " ('he', 537)]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_dist.most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's check to see if some words we are interested in appear in our text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "b_words = ['god', 'apostle', 'angel']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "my_list = []\n",
    "for word in b_words:\n",
    "    if word in text1_clean:\n",
    "        my_list.append(word)\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['god', 'angel']\n"
     ]
    }
   ],
   "source": [
    "print(my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's pull a book in from the Internet\n",
    "\n",
    "Project Gutenburg is a great source! www.gutenberg.org\n",
    "Let's say I'm interested in writings about East Africa from the early 20th century. I find the book, Zanzibar Tales, translated by George W. Bateman. In Project Gutenberg's database, this is text number 37472 \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To make a book into a Text NLTK can deal with, we have to:\n",
    "\n",
    "* open the file from a location\n",
    "* read it/decode it \n",
    "* tokenize it (go from a string to a list of word)\n",
    "* nltk.Text() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#import the urlopen command\n",
    "from urllib.request import urlopen\n",
    "#set the url to a variable \n",
    "#DO NOT NAVIGATE TO THE SITE AND JUST COPY THE LINK - THIS IS THE TXT VERSION! BETTER TO TYPE THIS LINK\n",
    "my_url = \"http://www.gutenberg.org/cache/epub/996/pg996.txt\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#open the file from the url\n",
    "file = urlopen(my_url)\n",
    "#read the opened file\n",
    "raw = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "don=raw.decode()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the type to be sure it worked. I expect a string now.\n",
    "type(don)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#split the string into words with word_tokenize (uses spaces to distinguish words)\n",
    "don_tokens = nltk.word_tokenize(don)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check to make sure it worked\n",
    "type(don_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "496482"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get an idea of how big the file is\n",
    "len(don_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\ufeffThe',\n",
       " 'Project',\n",
       " 'Gutenberg',\n",
       " 'EBook',\n",
       " 'of',\n",
       " 'Don',\n",
       " 'Quixote',\n",
       " ',',\n",
       " 'by',\n",
       " 'Miguel']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#look at the first 10 words to be sure its correct\n",
    "don_tokens[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "dq_text = don_tokens[120:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#turn the list of words into a text nltk can recognize\n",
    "dq_nltk_text = nltk.Text(don_tokens[120:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Remove stop words\n",
    "mystops = stopwords.words('english')\n",
    "dq_clean = [w for w in dq_text if w not in mystops]\n",
    "#Lowercase and remove punctuation\n",
    "dq_clean = [t.lower() for t in dq_clean if t.isalpha()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lemmatize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "wordnet_lemmatizer = WordNetLemmatizer()\n",
    "dq_clean = [wordnet_lemmatizer.lemmatize(t) for t in dq_clean]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['chapter',\n",
       " 'i',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'character',\n",
       " 'and',\n",
       " 'pursuit',\n",
       " 'of',\n",
       " 'the',\n",
       " 'famous',\n",
       " 'gentleman',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'of',\n",
       " 'la',\n",
       " 'mancha',\n",
       " 'chapter',\n",
       " 'ii',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'first',\n",
       " 'sally',\n",
       " 'the',\n",
       " 'ingenious',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'made',\n",
       " 'from',\n",
       " 'home',\n",
       " 'chapter',\n",
       " 'iii',\n",
       " 'wherein',\n",
       " 'is',\n",
       " 'related',\n",
       " 'the',\n",
       " 'droll',\n",
       " 'way',\n",
       " 'in',\n",
       " 'which',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'had',\n",
       " 'himself',\n",
       " 'dubbed',\n",
       " 'a',\n",
       " 'knight',\n",
       " 'chapter',\n",
       " 'iv',\n",
       " 'of',\n",
       " 'what',\n",
       " 'happened',\n",
       " 'to',\n",
       " 'our',\n",
       " 'knight',\n",
       " 'when',\n",
       " 'he',\n",
       " 'left',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'chapter',\n",
       " 'v',\n",
       " 'in',\n",
       " 'which',\n",
       " 'the',\n",
       " 'narrative',\n",
       " 'of',\n",
       " 'our',\n",
       " 'knight',\n",
       " 'mishap',\n",
       " 'is',\n",
       " 'continued',\n",
       " 'chapter',\n",
       " 'vi',\n",
       " 'of',\n",
       " 'the',\n",
       " 'diverting',\n",
       " 'and',\n",
       " 'important',\n",
       " 'scrutiny',\n",
       " 'which',\n",
       " 'the',\n",
       " 'curate',\n",
       " 'and',\n",
       " 'the',\n",
       " 'barber',\n",
       " 'made',\n",
       " 'in',\n",
       " 'the',\n",
       " 'library',\n",
       " 'of',\n",
       " 'our',\n",
       " 'ingenious',\n",
       " 'gentleman',\n",
       " 'chapter',\n",
       " 'vii',\n",
       " 'of',\n",
       " 'the',\n",
       " 'second',\n",
       " 'sally',\n",
       " 'of',\n",
       " 'our',\n",
       " 'worthy',\n",
       " 'knight',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'of',\n",
       " 'la',\n",
       " 'mancha',\n",
       " 'chapter',\n",
       " 'viii',\n",
       " 'of',\n",
       " 'the',\n",
       " 'good',\n",
       " 'fortune',\n",
       " 'which',\n",
       " 'the',\n",
       " 'valiant',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'had',\n",
       " 'in',\n",
       " 'the',\n",
       " 'terrible',\n",
       " 'and',\n",
       " 'adventure',\n",
       " 'of',\n",
       " 'the',\n",
       " 'windmill',\n",
       " 'with',\n",
       " 'other',\n",
       " 'occurrence',\n",
       " 'worthy',\n",
       " 'to',\n",
       " 'be',\n",
       " 'fitly',\n",
       " 'recorded',\n",
       " 'chapter',\n",
       " 'ix',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'concluded',\n",
       " 'and',\n",
       " 'finished',\n",
       " 'the',\n",
       " 'terrific',\n",
       " 'battle',\n",
       " 'between',\n",
       " 'the',\n",
       " 'gallant',\n",
       " 'biscayan',\n",
       " 'and',\n",
       " 'the',\n",
       " 'valiant',\n",
       " 'manchegan',\n",
       " 'chapter',\n",
       " 'x',\n",
       " 'of',\n",
       " 'the',\n",
       " 'pleasant',\n",
       " 'discourse',\n",
       " 'that',\n",
       " 'passed',\n",
       " 'between',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'and',\n",
       " 'his',\n",
       " 'squire',\n",
       " 'sancho',\n",
       " 'panza',\n",
       " 'chapter',\n",
       " 'xi',\n",
       " 'of',\n",
       " 'what',\n",
       " 'befell',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'with',\n",
       " 'certain',\n",
       " 'goatherd',\n",
       " 'chapter',\n",
       " 'xii',\n",
       " 'of',\n",
       " 'what',\n",
       " 'a',\n",
       " 'goatherd',\n",
       " 'related',\n",
       " 'to',\n",
       " 'those',\n",
       " 'with',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'chapter',\n",
       " 'xiii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'ended',\n",
       " 'the',\n",
       " 'story',\n",
       " 'of',\n",
       " 'the',\n",
       " 'shepherdess',\n",
       " 'marcela',\n",
       " 'with',\n",
       " 'other',\n",
       " 'incident',\n",
       " 'chapter',\n",
       " 'xiv',\n",
       " 'wherein',\n",
       " 'are',\n",
       " 'inserted',\n",
       " 'the',\n",
       " 'despairing',\n",
       " 'verse',\n",
       " 'of',\n",
       " 'the',\n",
       " 'dead',\n",
       " 'shepherd',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'incident',\n",
       " 'not',\n",
       " 'looked',\n",
       " 'for',\n",
       " 'chapter',\n",
       " 'xv',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'related',\n",
       " 'the',\n",
       " 'unfortunate',\n",
       " 'adventure',\n",
       " 'that',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'fell',\n",
       " 'in',\n",
       " 'with',\n",
       " 'when',\n",
       " 'he',\n",
       " 'fell',\n",
       " 'out',\n",
       " 'with',\n",
       " 'certain',\n",
       " 'heartless',\n",
       " 'yanguesans',\n",
       " 'chapter',\n",
       " 'xvi',\n",
       " 'of',\n",
       " 'what',\n",
       " 'happened',\n",
       " 'to',\n",
       " 'the',\n",
       " 'ingenious',\n",
       " 'gentleman',\n",
       " 'in',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'which',\n",
       " 'he',\n",
       " 'took',\n",
       " 'to',\n",
       " 'be',\n",
       " 'a',\n",
       " 'castle',\n",
       " 'chapter',\n",
       " 'xvii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'are',\n",
       " 'contained',\n",
       " 'the',\n",
       " 'innumerable',\n",
       " 'trouble',\n",
       " 'which',\n",
       " 'the',\n",
       " 'brave',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'and',\n",
       " 'his',\n",
       " 'good',\n",
       " 'squire',\n",
       " 'sancho',\n",
       " 'panza',\n",
       " 'endured',\n",
       " 'in',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'which',\n",
       " 'to',\n",
       " 'his',\n",
       " 'misfortune',\n",
       " 'he',\n",
       " 'took',\n",
       " 'to',\n",
       " 'be',\n",
       " 'a',\n",
       " 'castle',\n",
       " 'chapter',\n",
       " 'xviii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'related',\n",
       " 'the',\n",
       " 'discourse',\n",
       " 'sancho',\n",
       " 'panza',\n",
       " 'held',\n",
       " 'with',\n",
       " 'his',\n",
       " 'master',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'and',\n",
       " 'other',\n",
       " 'adventure',\n",
       " 'worth',\n",
       " 'relating',\n",
       " 'chapter',\n",
       " 'xix',\n",
       " 'of',\n",
       " 'the',\n",
       " 'shrewd',\n",
       " 'discourse',\n",
       " 'which',\n",
       " 'sancho',\n",
       " 'held',\n",
       " 'with',\n",
       " 'his',\n",
       " 'master',\n",
       " 'and',\n",
       " 'of',\n",
       " 'the',\n",
       " 'adventure',\n",
       " 'that',\n",
       " 'befell',\n",
       " 'him',\n",
       " 'with',\n",
       " 'a',\n",
       " 'dead',\n",
       " 'body',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'notable',\n",
       " 'occurrence',\n",
       " 'chapter',\n",
       " 'xx',\n",
       " 'of',\n",
       " 'the',\n",
       " 'unexampled',\n",
       " 'and',\n",
       " 'adventure',\n",
       " 'which',\n",
       " 'wa',\n",
       " 'achieved',\n",
       " 'by',\n",
       " 'the',\n",
       " 'valiant',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'of',\n",
       " 'la',\n",
       " 'mancha',\n",
       " 'with',\n",
       " 'le',\n",
       " 'peril',\n",
       " 'than',\n",
       " 'any',\n",
       " 'ever',\n",
       " 'achieved',\n",
       " 'by',\n",
       " 'any',\n",
       " 'famous',\n",
       " 'knight',\n",
       " 'in',\n",
       " 'the',\n",
       " 'world',\n",
       " 'chapter',\n",
       " 'xxi',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'exalted',\n",
       " 'adventure',\n",
       " 'and',\n",
       " 'rich',\n",
       " 'prize',\n",
       " 'of',\n",
       " 'helmet',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'thing',\n",
       " 'that',\n",
       " 'happened',\n",
       " 'to',\n",
       " 'our',\n",
       " 'invincible',\n",
       " 'knight',\n",
       " 'chapter',\n",
       " 'xxii',\n",
       " 'of',\n",
       " 'the',\n",
       " 'freedom',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'conferred',\n",
       " 'on',\n",
       " 'several',\n",
       " 'unfortunate',\n",
       " 'who',\n",
       " 'against',\n",
       " 'their',\n",
       " 'will',\n",
       " 'were',\n",
       " 'being',\n",
       " 'carried',\n",
       " 'where',\n",
       " 'they',\n",
       " 'had',\n",
       " 'no',\n",
       " 'wish',\n",
       " 'to',\n",
       " 'go',\n",
       " 'chapter',\n",
       " 'xxiii',\n",
       " 'of',\n",
       " 'what',\n",
       " 'befell',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'in',\n",
       " 'the',\n",
       " 'sierra',\n",
       " 'morena',\n",
       " 'which',\n",
       " 'wa',\n",
       " 'one',\n",
       " 'of',\n",
       " 'the',\n",
       " 'rarest',\n",
       " 'adventure',\n",
       " 'related',\n",
       " 'in',\n",
       " 'this',\n",
       " 'veracious',\n",
       " 'history',\n",
       " 'chapter',\n",
       " 'xxiv',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'continued',\n",
       " 'the',\n",
       " 'adventure',\n",
       " 'of',\n",
       " 'the',\n",
       " 'sierra',\n",
       " 'morena',\n",
       " 'chapter',\n",
       " 'xxv',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'strange',\n",
       " 'thing',\n",
       " 'that',\n",
       " 'happened',\n",
       " 'to',\n",
       " 'the',\n",
       " 'stout',\n",
       " 'knight',\n",
       " 'of',\n",
       " 'la',\n",
       " 'mancha',\n",
       " 'in',\n",
       " 'the',\n",
       " 'sierra',\n",
       " 'morena',\n",
       " 'and',\n",
       " 'of',\n",
       " 'his',\n",
       " 'imitation',\n",
       " 'of',\n",
       " 'the',\n",
       " 'penance',\n",
       " 'of',\n",
       " 'beltenebros',\n",
       " 'chapter',\n",
       " 'xxvi',\n",
       " 'in',\n",
       " 'which',\n",
       " 'are',\n",
       " 'continued',\n",
       " 'the',\n",
       " 'refinement',\n",
       " 'wherewith',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'played',\n",
       " 'the',\n",
       " 'part',\n",
       " 'of',\n",
       " 'a',\n",
       " 'lover',\n",
       " 'in',\n",
       " 'the',\n",
       " 'sierra',\n",
       " 'morena',\n",
       " 'chapter',\n",
       " 'xxvii',\n",
       " 'of',\n",
       " 'how',\n",
       " 'the',\n",
       " 'curate',\n",
       " 'and',\n",
       " 'the',\n",
       " 'barber',\n",
       " 'proceeded',\n",
       " 'with',\n",
       " 'their',\n",
       " 'scheme',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'matter',\n",
       " 'worthy',\n",
       " 'of',\n",
       " 'record',\n",
       " 'in',\n",
       " 'this',\n",
       " 'great',\n",
       " 'history',\n",
       " 'chapter',\n",
       " 'xxviii',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'strange',\n",
       " 'and',\n",
       " 'delightful',\n",
       " 'adventure',\n",
       " 'that',\n",
       " 'befell',\n",
       " 'the',\n",
       " 'curate',\n",
       " 'and',\n",
       " 'the',\n",
       " 'barber',\n",
       " 'in',\n",
       " 'the',\n",
       " 'same',\n",
       " 'sierra',\n",
       " 'chapter',\n",
       " 'xxix',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'droll',\n",
       " 'device',\n",
       " 'and',\n",
       " 'method',\n",
       " 'adopted',\n",
       " 'to',\n",
       " 'extricate',\n",
       " 'our',\n",
       " 'knight',\n",
       " 'from',\n",
       " 'the',\n",
       " 'severe',\n",
       " 'penance',\n",
       " 'he',\n",
       " 'had',\n",
       " 'imposed',\n",
       " 'upon',\n",
       " 'himself',\n",
       " 'chapter',\n",
       " 'xxx',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'address',\n",
       " 'displayed',\n",
       " 'by',\n",
       " 'the',\n",
       " 'fair',\n",
       " 'dorothea',\n",
       " 'with',\n",
       " 'other',\n",
       " 'matter',\n",
       " 'pleasant',\n",
       " 'and',\n",
       " 'amusing',\n",
       " 'chapter',\n",
       " 'xxxi',\n",
       " 'of',\n",
       " 'the',\n",
       " 'delectable',\n",
       " 'discussion',\n",
       " 'between',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'and',\n",
       " 'sancho',\n",
       " 'panza',\n",
       " 'his',\n",
       " 'squire',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'incident',\n",
       " 'chapter',\n",
       " 'xxxii',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'what',\n",
       " 'befell',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'party',\n",
       " 'at',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'chapter',\n",
       " 'xxxiii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'related',\n",
       " 'the',\n",
       " 'novel',\n",
       " 'of',\n",
       " 'the',\n",
       " 'curiosity',\n",
       " 'chapter',\n",
       " 'xxxiv',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'continued',\n",
       " 'the',\n",
       " 'novel',\n",
       " 'of',\n",
       " 'the',\n",
       " 'curiosity',\n",
       " 'chapter',\n",
       " 'xxxv',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'heroic',\n",
       " 'and',\n",
       " 'prodigious',\n",
       " 'battle',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'had',\n",
       " 'with',\n",
       " 'certain',\n",
       " 'skin',\n",
       " 'of',\n",
       " 'red',\n",
       " 'wine',\n",
       " 'and',\n",
       " 'brings',\n",
       " 'the',\n",
       " 'novel',\n",
       " 'of',\n",
       " 'the',\n",
       " 'curiosity',\n",
       " 'to',\n",
       " 'a',\n",
       " 'close',\n",
       " 'chapter',\n",
       " 'xxxvi',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'more',\n",
       " 'curious',\n",
       " 'incident',\n",
       " 'that',\n",
       " 'occurred',\n",
       " 'at',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'chapter',\n",
       " 'xxxvii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'is',\n",
       " 'continued',\n",
       " 'the',\n",
       " 'story',\n",
       " 'of',\n",
       " 'the',\n",
       " 'famous',\n",
       " 'princess',\n",
       " 'micomicona',\n",
       " 'with',\n",
       " 'other',\n",
       " 'droll',\n",
       " 'adventure',\n",
       " 'chapter',\n",
       " 'xxxviii',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'curious',\n",
       " 'discourse',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'delivered',\n",
       " 'on',\n",
       " 'arm',\n",
       " 'and',\n",
       " 'letter',\n",
       " 'chapter',\n",
       " 'xxxix',\n",
       " 'wherein',\n",
       " 'the',\n",
       " 'captive',\n",
       " 'relates',\n",
       " 'his',\n",
       " 'life',\n",
       " 'and',\n",
       " 'adventure',\n",
       " 'chapter',\n",
       " 'xl',\n",
       " 'in',\n",
       " 'which',\n",
       " 'the',\n",
       " 'story',\n",
       " 'of',\n",
       " 'the',\n",
       " 'captive',\n",
       " 'is',\n",
       " 'continued',\n",
       " 'chapter',\n",
       " 'xli',\n",
       " 'in',\n",
       " 'which',\n",
       " 'the',\n",
       " 'captive',\n",
       " 'still',\n",
       " 'continues',\n",
       " 'his',\n",
       " 'adventure',\n",
       " 'chapter',\n",
       " 'xlii',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'what',\n",
       " 'further',\n",
       " 'took',\n",
       " 'place',\n",
       " 'in',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'and',\n",
       " 'of',\n",
       " 'several',\n",
       " 'other',\n",
       " 'thing',\n",
       " 'worth',\n",
       " 'knowing',\n",
       " 'chapter',\n",
       " 'xliii',\n",
       " 'wherein',\n",
       " 'is',\n",
       " 'related',\n",
       " 'the',\n",
       " 'pleasant',\n",
       " 'story',\n",
       " 'of',\n",
       " 'the',\n",
       " 'muleteer',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'strange',\n",
       " 'thing',\n",
       " 'that',\n",
       " 'came',\n",
       " 'to',\n",
       " 'pas',\n",
       " 'in',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'chapter',\n",
       " 'xliv',\n",
       " 'in',\n",
       " 'which',\n",
       " 'are',\n",
       " 'continued',\n",
       " 'the',\n",
       " 'adventure',\n",
       " 'of',\n",
       " 'the',\n",
       " 'inn',\n",
       " 'chapter',\n",
       " 'xlv',\n",
       " 'in',\n",
       " 'which',\n",
       " 'the',\n",
       " 'doubtful',\n",
       " 'question',\n",
       " 'of',\n",
       " 'mambrino',\n",
       " 'helmet',\n",
       " 'and',\n",
       " 'the',\n",
       " 'is',\n",
       " 'finally',\n",
       " 'settled',\n",
       " 'with',\n",
       " 'other',\n",
       " 'adventure',\n",
       " 'that',\n",
       " 'occurred',\n",
       " 'in',\n",
       " 'truth',\n",
       " 'and',\n",
       " 'earnest',\n",
       " 'chapter',\n",
       " 'xlvi',\n",
       " 'of',\n",
       " 'the',\n",
       " 'end',\n",
       " 'of',\n",
       " 'the',\n",
       " 'notable',\n",
       " 'adventure',\n",
       " 'of',\n",
       " 'the',\n",
       " 'officer',\n",
       " 'of',\n",
       " 'the',\n",
       " 'holy',\n",
       " 'brotherhood',\n",
       " 'and',\n",
       " 'of',\n",
       " 'the',\n",
       " 'great',\n",
       " 'ferocity',\n",
       " 'of',\n",
       " 'our',\n",
       " 'worthy',\n",
       " 'knight',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'chapter',\n",
       " 'xlvii',\n",
       " 'of',\n",
       " 'the',\n",
       " 'strange',\n",
       " 'manner',\n",
       " 'in',\n",
       " 'which',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'of',\n",
       " 'la',\n",
       " 'mancha',\n",
       " 'wa',\n",
       " 'carried',\n",
       " 'away',\n",
       " 'enchanted',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'remarkable',\n",
       " 'incident',\n",
       " 'chapter',\n",
       " 'xlviii',\n",
       " 'in',\n",
       " 'which',\n",
       " 'the',\n",
       " 'canon',\n",
       " 'pursues',\n",
       " 'the',\n",
       " 'subject',\n",
       " 'of',\n",
       " 'the',\n",
       " 'book',\n",
       " 'of',\n",
       " 'chivalry',\n",
       " 'with',\n",
       " 'other',\n",
       " 'matter',\n",
       " 'worthy',\n",
       " 'of',\n",
       " 'his',\n",
       " 'wit',\n",
       " 'chapter',\n",
       " 'xlix',\n",
       " 'which',\n",
       " 'treat',\n",
       " 'of',\n",
       " 'the',\n",
       " 'shrewd',\n",
       " 'conversation',\n",
       " 'which',\n",
       " 'sancho',\n",
       " 'panza',\n",
       " 'held',\n",
       " 'with',\n",
       " 'his',\n",
       " 'master',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'chapter',\n",
       " 'l',\n",
       " 'of',\n",
       " 'the',\n",
       " 'shrewd',\n",
       " 'controversy',\n",
       " 'which',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'and',\n",
       " 'the',\n",
       " 'canon',\n",
       " 'held',\n",
       " 'together',\n",
       " 'with',\n",
       " 'other',\n",
       " 'incident',\n",
       " 'chapter',\n",
       " 'li',\n",
       " 'which',\n",
       " 'deal',\n",
       " 'with',\n",
       " 'what',\n",
       " 'the',\n",
       " 'goatherd',\n",
       " 'told',\n",
       " 'those',\n",
       " 'who',\n",
       " 'were',\n",
       " 'carrying',\n",
       " 'off',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'chapter',\n",
       " 'lii',\n",
       " 'of',\n",
       " 'the',\n",
       " 'quarrel',\n",
       " 'that',\n",
       " 'don',\n",
       " 'quixote',\n",
       " 'had',\n",
       " 'with',\n",
       " 'the',\n",
       " 'goatherd',\n",
       " 'together',\n",
       " 'with',\n",
       " 'the',\n",
       " 'rare',\n",
       " 'adventure',\n",
       " 'of',\n",
       " 'the',\n",
       " 'penitent',\n",
       " 'which',\n",
       " 'with',\n",
       " 'an',\n",
       " 'expenditure',\n",
       " 'of',\n",
       " 'sweat',\n",
       " 'he',\n",
       " 'brought',\n",
       " 'to',\n",
       " 'a',\n",
       " 'happy',\n",
       " 'conclusion',\n",
       " 'translator',\n",
       " 'preface',\n",
       " 'i',\n",
       " 'about',\n",
       " 'this',\n",
       " ...]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dq_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One step further! \n",
    "**Part-of-Speech Tagging**\n",
    "\n",
    "NLTK uses the Penn Tag Set. \n",
    "\n",
    "There are other options as well that you many want to look at (i.e., tree tagger and polyglot)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#make a new object that has all the words and tags in it\n",
    "dq_tagged = nltk.pos_tag(dq_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('chapter', 'NN'), ('i', 'NN'), ('which', 'WDT'), ('treats', 'NNS'), ('of', 'IN'), ('the', 'DT'), ('character', 'NN'), ('and', 'CC'), ('pursuits', 'NNS'), ('of', 'IN')]\n"
     ]
    }
   ],
   "source": [
    "print(dq_tagged[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We have a **list of tuples**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I'm going to put it in a for-loop\n",
    "\n",
    "Have to deal with this in a special way \n",
    "<br>(a, b) in my_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#create an empty dictionary\n",
    "tag_dict = {}\n",
    "#for every word/tag combo in my list, \n",
    "for (word, tag) in dq_tagged:\n",
    "    if tag in tag_dict: \n",
    "        tag_dict[tag]+=1\n",
    "    else:\n",
    "        tag_dict[tag] = 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\"''\": 1,\n",
       " 'CC': 1245,\n",
       " 'CD': 2867,\n",
       " 'DT': 2715,\n",
       " 'EX': 68,\n",
       " 'FW': 435,\n",
       " 'IN': 7055,\n",
       " 'JJ': 35160,\n",
       " 'JJR': 678,\n",
       " 'JJS': 960,\n",
       " 'MD': 4390,\n",
       " 'NN': 66182,\n",
       " 'NNP': 115,\n",
       " 'NNPS': 1,\n",
       " 'NNS': 16427,\n",
       " 'PDT': 33,\n",
       " 'POS': 1,\n",
       " 'PRP': 1923,\n",
       " 'PRP$': 305,\n",
       " 'RB': 11117,\n",
       " 'RBR': 404,\n",
       " 'RBS': 40,\n",
       " 'RP': 275,\n",
       " 'TO': 322,\n",
       " 'UH': 6,\n",
       " 'VB': 7527,\n",
       " 'VBD': 16105,\n",
       " 'VBG': 7465,\n",
       " 'VBN': 6809,\n",
       " 'VBP': 11250,\n",
       " 'VBZ': 3170,\n",
       " 'WDT': 396,\n",
       " 'WP': 391,\n",
       " 'WP$': 138,\n",
       " 'WRB': 265,\n",
       " '``': 2}"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('NN', 66182),\n",
       "             ('JJ', 35160),\n",
       "             ('NNS', 16427),\n",
       "             ('VBD', 16105),\n",
       "             ('VBP', 11250),\n",
       "             ('RB', 11117),\n",
       "             ('VB', 7527),\n",
       "             ('VBG', 7465),\n",
       "             ('IN', 7055),\n",
       "             ('VBN', 6809),\n",
       "             ('MD', 4390),\n",
       "             ('VBZ', 3170),\n",
       "             ('CD', 2867),\n",
       "             ('DT', 2715),\n",
       "             ('PRP', 1923),\n",
       "             ('CC', 1245),\n",
       "             ('JJS', 960),\n",
       "             ('JJR', 678),\n",
       "             ('FW', 435),\n",
       "             ('RBR', 404),\n",
       "             ('WDT', 396),\n",
       "             ('WP', 391),\n",
       "             ('TO', 322),\n",
       "             ('PRP$', 305),\n",
       "             ('RP', 275),\n",
       "             ('WRB', 265),\n",
       "             ('WP$', 138),\n",
       "             ('NNP', 115),\n",
       "             ('EX', 68),\n",
       "             ('RBS', 40),\n",
       "             ('PDT', 33),\n",
       "             ('UH', 6),\n",
       "             ('``', 2),\n",
       "             (\"''\", 1),\n",
       "             ('NNPS', 1),\n",
       "             ('POS', 1),\n",
       "             ('reverse', False)])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "tag_dict = OrderedDict(sorted(tag_dict.items(), key=lambda t: t[1], reverse=True))\n",
    "tag_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How do you know to put all that other stuff in (i.e., sorted, lambda, etc)?!?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Read the docs*\n",
    "\n",
    "https://docs.python.org/3.1/whatsnew/3.1.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So far, we have counted things in our texts by looking at \n",
    "\n",
    "* Concordance\n",
    "* Words in similar environments\n",
    "* Words in common contexts\n",
    "* Unique words \n",
    "* Length of words \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then we performed some operations, but still counted things:\n",
    "\n",
    "* Frequency Distributions\n",
    "* Lexical Density \n",
    "* Found words from a list in a text\n",
    "* Part-of-Speech Tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we will perform operations on the Text itself before doing those operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#Let's check out the lexical density\n",
    "len(set(zt_clean))/len(zt_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if I want to read in my OWN corpus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "f = open(\"/Users/mam/books/hungerGames/catchingFire.txt\", 'r')\n",
    "my_file = f.read()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "hg_tokens = nltk.word_tokenize(my_file)\n",
    "hg_tokens[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Going Forward**\n",
    "\n",
    "* Use a text editor to write complete programs\n",
    "    * Run these in the terminal\n",
    "* Use Spyder to write complete programs\n",
    "* Often save the program you write in the same file as the file you will be working with to shorted the path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do I know where to go?!?\n",
    "* http://www.nltk.org/book_1ed\n",
    "* http://www.nltk.org/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}