18-NaturalLanguageProcessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing\n",
    "\n",
    "Most of the data we have encountered so far has been numerical (or at least, numerically encoded).\n",
    "\n",
    "However, one of the most powerful aspects of data science is acknowledging and considering that there are vasts amounts of data available in many other modalities, with potentially valuable information, if the data can be leveraged and analyzed. \n",
    "\n",
    "Here, we will introduce natural language processing (NLP), or the computational analysis of text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "Natural Language Processing (NLP) is the approach of analyzing text data, with computers.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "Natural Language Processing on \n",
    "<a href=\"https://en.wikipedia.org/wiki/Natural-language_processing\" class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NTLK: Natural Language Tool Kit\n",
    "\n",
    "There are many tools for analyzing text data in Python. Here, we will use one of biggest and most prominent ones: NLTK.\n",
    "\n",
    "NLTK provides interfaces to over 50 corpora and lexical resources, as well as a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.\n",
    "\n",
    "In this notebook, we will walk through some basic text-analysis using the `NLTK` package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "The Natural Language Tool Kit, or NLTK, is a Python module for text-analysis. \n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "The NLTK organization website is \n",
    "<a href=\"http://www.nltk.org/\" class=\"alert-link\">here</a>\n",
    "and they have a whole book of tutorials \n",
    "<a href=\"http://www.nltk.org/book/\" class=\"alert-link\">here</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NLTK\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set an example sentence of 'data' to play with\n",
    "sentence = \"UC San Diego is a great place to study cognitive science.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading Corpora\n",
    "\n",
    "To work with text-data, you often need corpora - text datasets to compare to. \n",
    "\n",
    "NLTK has many such datasets available, but doesn't install them by default (as the full set of them would be quite large). Below we will download some of these datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you hit an error downloading things in the cell below, come back to this cell, uncomment it, and run this code.\n",
    "#   This code gives python permission to write to your disk (if it doesn't already have persmission to do so).\n",
    "# import ssl\n",
    "\n",
    "# try:\n",
    "#     _create_unverified_https_context = ssl._create_unverified_context\n",
    "# except AttributeError:\n",
    "#     pass\n",
    "# else:\n",
    "#     ssl._create_default_https_context = _create_unverified_https_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/tom/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /Users/tom/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /Users/tom/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     /Users/tom/nltk_data...\n",
      "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
      "[nltk_data] Downloading package words to /Users/tom/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n",
      "[nltk_data] Downloading package treebank to /Users/tom/nltk_data...\n",
      "[nltk_data]   Package treebank is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download some useful data files from NLTK\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('maxent_ne_chunker')\n",
    "nltk.download('words')\n",
    "nltk.download('treebank')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "Tokenization is the process of splitting text data into 'tokens', which are meaningful pieces of data.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "More information on tokenization\n",
    "<a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html\" class=\"alert-link\">here</a>.\n",
    "</div>\n",
    "\n",
    "Tokenization can be done at different levels - you can, for example tokenize text into sentences, and/or into words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize our sentence, at the word level\n",
    "tokens = nltk.word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['UC', 'San', 'Diego', 'is', 'a', 'great', 'place', 'to', 'study', 'cognitive', 'science', '.']\n"
     ]
    }
   ],
   "source": [
    "# Check out the word-tokenized data\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part-of-speech (POS) Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "Part-of-Speech tagging is the process of labelling words with respect to their 'types' and relationships to other words.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "Part-of-speech tagging on \n",
    "<a href=\"https://en.wikipedia.org/wiki/Part-of-speech_tagging\" class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply part-of-speech tagging to our sentence\n",
    "tags = nltk.pos_tag(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('UC', 'NNP'), ('San', 'NNP'), ('Diego', 'NNP'), ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('place', 'NN'), ('to', 'TO'), ('study', 'VB'), ('cognitive', 'JJ'), ('science', 'NN'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "# Check the POS tags for our data\n",
    "print(tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NNP: noun, proper, singular\n",
      "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
      "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
      "    Shannon A.K.C. Meltex Liverpool ...\n",
      "DT: determiner\n",
      "    all an another any both del each either every half la many much nary\n",
      "    neither no some such that the them these this those\n",
      "JJ: adjective or numeral, ordinal\n",
      "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
      "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
      "    multilingual multi-disciplinary ...\n"
     ]
    }
   ],
   "source": [
    "# Check out the documentation for describing the abbreviations\n",
    "nltk.help.upenn_tagset(tagpattern='NNP')\n",
    "nltk.help.upenn_tagset(tagpattern='DT')\n",
    "nltk.help.upenn_tagset(tagpattern='JJ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition (NER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "Named entity recognition seeks to label words with the kinds of entities that they relate to.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "Named entity recognition on \n",
    "<a href=\"https://en.wikipedia.org/wiki/Named-entity_recognition\" class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply named entity recognition to our POS tags\n",
    "entities = nltk.chunk.ne_chunk(tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  UC/NNP\n",
      "  (PERSON San/NNP Diego/NNP)\n",
      "  is/VBZ\n",
      "  a/DT\n",
      "  great/JJ\n",
      "  place/NN\n",
      "  to/TO\n",
      "  study/VB\n",
      "  cognitive/JJ\n",
      "  science/NN\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "# Check out the named entities\n",
    "print(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "'Stop words' are the most common words of a language, that we often want to filter out before text analysis. \n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "Stop words on \n",
    "<a href=\"https://en.wikipedia.org/wiki/Stop_words\" class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "# Check out the corpus of stop words in English\n",
    "print(nltk.corpus.stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Encoding\n",
    "\n",
    "In order to analyze text _as_ data, we often need to encode it in some way. \n",
    "\n",
    "By encoding here, we just mean choosing a representation of the data, and for text data the goal is to choose a representation that is more amenable for computational analysis. There are many possibilities, and which approach works best depends largely on the context of the data and the analyses to be performed. Choosing how to encode text data is a key topic in NLP. \n",
    "\n",
    "Here, we will explore a couple simple encoding approaches, which in this case are basically ways to count the words and measure occurrences in text data. By measuring how often certain words occur, we can characterize the text as numerical data, and open up access to numerical analysis of the data. \n",
    "\n",
    "Some common encodings for text data are:\n",
    "- Bag of Words (BoW)\n",
    "    - Text is encoded as a collection of words & frequencies\n",
    "- Term Frequency / Inverse Document Frequency (TF/IDF)\n",
    "    - TF/IDF is a weighting that stores words with relation to their commonality across a corpus.\n",
    "\n",
    "Next we will walk through an example of encoding text as BoW and TF-IDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "%matplotlib inline\n",
    "\n",
    "# Standard Python has some useful string tools\n",
    "import string\n",
    "\n",
    "# Collections is a part of standard Python, with some useful data objects\n",
    "from collections import Counter\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Scikit-learn has some useful NLP tools, such as a TFIDF vectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load some Data\n",
    "\n",
    "The data we will be looking at is a small subset of the BookCorpus dataset. The original dataset can be found here: http://yknzhu.wixsite.com/mbweb.\n",
    "\n",
    "The original dataset was collected from more than 11,000 books, and has already been tokenised at both the sentence and word level. \n",
    "\n",
    "The small subset provided and used here contains the first 10,000 sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data\n",
    "with open('files/book10k.txt', 'r') as file:\n",
    "    sents = file.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the half-ling book one in the fall of igneeria series kaylee soderburg copyright 2013 kaylee soderburg all rights reserved .\n",
      "\n",
      "alejo was sure the fact that he was nervously repeating mass along with five wrinkly , age-encrusted spanish women meant that stalin was rethinking whether he was going to pay the price .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check out the data - print out the first and last sentences, as examples\n",
    "print(sents[0])\n",
    "print(sents[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-Processing\n",
    "\n",
    "First, let's do some standard text pre-processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocessing: strip all extra whitespace from the sentences\n",
    "sents = [sent.strip() for sent in sents]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first take a look at the word frequencies in this data.\n",
    "\n",
    "To do so, we can tokenize the text, count occurences, and then we can have a look at the most frequent words in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize all the sentences into words\n",
    "#  This collects all the word tokens together into one big list\n",
    "tokens = []\n",
    "for x in sents:\n",
    "    tokens.extend(nltk.word_tokenize(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words in the data: \t 140094\n",
      "Number of unique words: \t 8221\n"
     ]
    }
   ],
   "source": [
    "# Check out how many words are in the data\n",
    "print('Number of words in the data: \\t', len(tokens))\n",
    "print('Number of unique words: \\t', len(set(tokens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag-of-Words\n",
    "\n",
    "Next, let's try a 'bag-of-words' representation of the data. \n",
    "\n",
    "After tokenization, a 'bag-of-words' can be computed by counting how often each token occurs, which we can do with the `Counter` object. \n",
    "\n",
    "<div class=\"alert alert-success\">\n",
    "A 'bag of words' model, of representation, is way to represent text data by counting occurences of tokens.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "Bag of words on \n",
    "<a href=https://en.wikipedia.org/wiki/Bag-of-words_model class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the 'counter' object to count how many times each word appears\n",
    "counts = Counter(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can explore the counter object, which is basically a 'bag-of-words' representation of the data. \n",
    "\n",
    "Note that in this encoding we have lost word order and grammar. All we have is a collection of words. \n",
    "\n",
    "This representation is quite different from how humans interact with language, but can be useful for some analyses. \n",
    "\n",
    "What we do have is a list of all the words present, and how often they appear. Basically, we have turned the text into a _distribution_ and we can try and analyze this distribution to try and programmatically analyze the text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('.', 8601),\n",
       " (',', 6675),\n",
       " ('the', 6062),\n",
       " ('and', 3382),\n",
       " ('to', 3328),\n",
       " ('``', 2852),\n",
       " ('i', 2747),\n",
       " ('a', 2480),\n",
       " ('of', 2122),\n",
       " ('was', 1752),\n",
       " ('he', 1678),\n",
       " ('in', 1616),\n",
       " ('you', 1483),\n",
       " ('her', 1353),\n",
       " ('his', 1349),\n",
       " ('?', 1153),\n",
       " ('she', 1153),\n",
       " ('that', 1134),\n",
       " ('it', 1050),\n",
       " (\"'s\", 1023),\n",
       " ('had', 898),\n",
       " ('with', 894),\n",
       " ('alejo', 890),\n",
       " ('wara', 875),\n",
       " ('at', 818)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out the counts object, printing out some of the most common tokens\n",
    "counts.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing you might notice if you scroll through the word list above is that it still contains punctuation. Let's remove those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "# The 'string' module (standard library) has a useful list of punctuation\n",
    "print(string.punctuation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop all punction markers from the counts object\n",
    "for punc in string.punctuation:\n",
    "    if punc in counts:\n",
    "        counts.pop(punc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the top 10 most frequent words\n",
    "top10 = counts.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the top words, and counts\n",
    "top10_words = [it[0] for it in top10]\n",
    "top10_counts = [it[1] for it in top10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEWCAYAAACdaNcBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAWQUlEQVR4nO3dfbxdVX3n8c/XBAElEhCkqIwRi6XQIgg+RKhDq6Xi1BFaFBCftXTqA1JHfWEZ27FOp4w61OJ0VMq0VYexCCpFahXUMjAoDwkkmKiMIKmgIAKChiDy8Osfe185XG+Sm+Sce85d+bxfr/u6e6+9z95r3Zz7vStr77NXqgpJUlseMe4KSJKGz3CXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcNVJJ1g58PZjknoH148ZQn1OS3DetXifMdT2kUVs47gqobVW1w9RykjXA66vqi5tzrCQLq+r+IVTro1X1+o2c6xEAVfXgEM4nzTl77hqrJAuSvCvJt5PcluTMJIv7bXsnuT/J7yW5EfjcQNnrknw3ye1JXptkaZJVSe5Mcupm1uWyJH+a5HJgHfD4JDsn+ViSW5LcmORPpoI/ycIkf9nX4bokb05y/8DxbklyyMD6KUnOGFj/tSSX93W+KsnB0+ryJ/33HyX5XJKdBrYf2m+7K8l3krysP96NU/Xr9zsuyWWb8/PQ/Ga4a9zeDhwGHAI8EbgP+IuB7QuAZwG/BLx4oGw/YE/gNcAHgbcB/7Yvf02SZ21mfV4OvBJYBNwCnAnc1Z/rmcARwCv6fd8E/Abwq8BS4OjZniTJEuBc4GRgZ+A/AecOBjjwMuA4YHdgMfCW/rW/CJwPvA94LHAgsLqqLgF+SvdzGGzPx2dbL7XDcNe4/T5wUlV9r6p+ArwbODpJBvb546paV1X3DJT9aVXdW1Xn9esfq6rbq+o7wFeAAzZwzlf0veWpr50Htp1RVddW1X3AE4DnAm/tz38zcBpwTL/vS4H/3tf9B8B7N6HdrwI+XVVfrKoHq+pzwNfp/tBN+euqur6q7gbOAfafqj/w2ar6VFXdX1U/qKqVUz8HukAnyW50QX/WJtRLjXDMXWPTB/gedMMtg0+wewRdjxTgwar63rSXPlBVtw+s3wN8f9r6Dqzfxzcw5n7jwPKTgO2AHwz8rXkEcF2//Php+//LBs453ZOAY5O8ZKBsm/6YU24ZWF7HQ23aA7h+Pcf9GHBVkjcCxwIXVtVtm1AvNcJw19hUVSX5LvA7VbV8+vYkuwBz/djSwfPdCKwFdqqZH596M13QTvk307bfDTxqYP0XgKkx+Rvp/pfw5s2o443AU2faUFU3JLkGeBFdD//PN+P4aoDDMhq3DwOnJNkDIMnjkrxozHUCuqAELgPem2RRkkck2WvgIukngT9Msnv/h+gd0w6xgq53vjDJs3nomgHAR4GXJHlef1F5+375F2ZRtY8Bv53kyP61uybZb9r2d9FdJ/jsprdcLTDcNW7vBb4IfDnJj+nGy58+3io9zLF0FzO/CdxBN369W7/tfwCXAKuBy+nCftAf0V1svRN4J/D3Uxuq6tvA79JdY7iNbkjnLczid7Kqrqf7Q/FHwA+BZcC+A7ucDfwi8MmqunfWLVVT4mQd0nAk2RtYVVVjHe7sb4X8DnBMVf2/cdZF42PPXWrPscCPDPatmxdUpYb0H1haQnePvLZiDstIUoMclpGkBk3MsMwuu+xSS5YsGXc1JGleWb58+W1Vtev08okJ9yVLlrBs2bJxV0OS5pUkM34y2mEZSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMm5kNMX/vuXSw56R/HXY2mrDnl3427CpLGxJ67JDXIcJekBhnuktQgw12SGjS0cE/ylWEdS5K0ZYYW7lX1nGEdS5K0ZYbZc1/bfz80yUVJzknyzSRnJsmwziNJ2rhRjbkfAJwI7APsCRw8005Jjk+yLMmyB9bdNaKqSNLWZ1ThfkVV3VRVDwIr6GZj/zlVdXpVHVRVBy141I4jqookbX1GFe73Diw/wAR9ElaStgbeCilJDTLcJalBQxsuqaod+u8XARcNlL9pWOeQJM2OPXdJapDhLkkNmpi7WH71CTuyzOePS9JQ2HOXpAYZ7pLUIMNdkho0MWPuzqE6Ws6nKm1d7LlLUoMMd0lqkOEuSQ0y3CWpQVsU7kmWJFk1rMpIkobDnrskNWgY4b4gyV8nWZ3kgiTbJ3lKks8nWZ7kkiR7D+E8kqRZGka47wX8VVXtC9wJ/C5wOvDmqjoQeBvwP2d6oXOoStJoDONDTDdU1Yp+eTndfKnPAc5OMrXPtjO9sKpOp/tDwLa771VDqIskieGE+/T5UncD7qyq/YdwbEnSZhjFBdUfATckeQlAOk8bwXkkSesxqrtljgNel2QlsBp48YjOI0mawRYNy1TVGuBXBtbfP7D5BVtybEnS5vM+d0lqkOEuSQ0y3CWpQRMzWYcTZEvS8Nhzl6QGGe6S1CDDXZIaNDFj7k6QPbecMFtqmz13SWqQ4S5JDTLcJalBhrskNchwl6QGbTTck7wjyQn98l8k+XK//Lwk/zvJh/qp8lYneffA605J8vUk1yR5//qOL0kavtncCnkx8B+B04CDgG2TbAMcAlwCnF1VdyRZAHwpyX7ATcCRwN5VVUkWj6b6kqSZzGZYZjlwYJJFdFPqfZUu5H+NLtxfmuQq4GpgX2AfutmYfgKckeR3gHUzHdgJsiVpNDYa7lV1H7AGeA3wFbpA/3XgKcA9wNuA51XVfsA/AttV1f3AM4FPAUcAn1/PsU+vqoOq6qAFj9pxy1sjSQJmf0H1YroQv5gu3P8DsAJ4DHA3cFeS3YDDAZLsAOxYVZ8DTgScLFuS5tBsHz9wCXAy8NWqujvJT4BLqmplkqvp5kn9NnBpv/8i4B+SbAcE+MMh11uStAGzCveq+hKwzcD6UweWX72elz1zi2omSdps3ucuSQ0y3CWpQYa7JDVoYp7n7hyqkjQ89twlqUGGuyQ1yHCXpAZNzJi7c6jOX87HKk0ee+6S1CDDXZIaZLhLUoMMd0lq0MjDPckJSb6R5MxRn0uS1JmLu2XeABxeVTfMwbkkSQy5557krUlW9V8nJvkwsCdwXhKf6S5Jc2RoPfckB9JNxfcsugk6LgdeDrwA+PWqum2G1xwPHA+w4DG7DqsqkrTVG2bP/RDgM1V1d1WtBT5NN4n2ejmHqiSNxjDDPUM8liRpCwwz3C8GjkjyqCSPBo6km3tVkjTHhjbmXlVXJfk74Iq+6IyqujqxQy9Jc22ot0JW1anAqdPKlgzzHJKkjfMTqpLUIMNdkho0Mc9zdw5VSRoee+6S1CDDXZIaZLhLUoMmZszdOVQFzscqDYs9d0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSgkYV7knOTLE+yup9OT5I0R0Z5n/trq+qOJNsDVyb5VFXdPriDc6hK0miMcljmhCQrgcuAPYC9pu/gHKqSNBoj6bknORR4PrC0qtYluQjYbhTnkiT9vFH13HcEftgH+97As0d0HknSDEYV7p8HFia5BngP3dCMJGmOjGRYpqruBQ4fxbElSRvnfe6S1CDDXZIaNDHPc3cOVUkaHnvuktQgw12SGmS4S1KDJmbM3TlUtSWce1V6OHvuktQgw12SGmS4S1KDDHdJatCchHuSr8zFeSRJnTkJ96p6zlycR5LUmaue+9q5OI8kqTPWMfckxydZlmTZA+vuGmdVJKkpYw1351CVpNHwbhlJapDhLkkNMtwlqUFzdSvkDnNxHklSx567JDXIcJekBk3M89ydQ1WShseeuyQ1yHCXpAYZ7pLUoIkZc3cOVY2Cc6tqa2XPXZIaZLhLUoMMd0lq0NDCPcnCJBck2XemdUnS3BlauFfV/cDLgf+aZJvp68M6jyRp44Z6t0xV3Qq8eH3rkqS54Zi7JDXIOVQlqUHOoSpJDXJYRpIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDVoYp7n7hyqkjQ89twlqUGGuyQ1yHCXpAZNzJi7c6hqvnKeVk0ie+6S1CDDXZIaZLhLUoMMd0lq0FDCPcniJG8YxrEkSVtuWD33xYDhLkkTYli3Qp4CPCXJCuDCvuxwoID/UlVnDek8kqRZGFbP/STg+qraH7gM2B94GvB84H1Jdh/SeSRJszCKC6qHAJ+oqgeq6vvA/wWeMdOOTpAtSaMxinDPbHd0gmxJGo1hhfuPgUX98sXA0UkWJNkVeC5wxZDOI0mahaFcUK2q25NcmmQV8E/ANcBKuguq76iqW4ZxHknS7AztwWFV9bJpRW8f1rElSZvGT6hKUoMMd0lqkOEuSQ2amMk6nCBbkobHnrskNchwl6QGGe6S1KCJGXN3gmy1wMmyNSnsuUtSgwx3SWqQ4S5JDTLcJalBIw/3JGtHfQ5J0sPZc5ekBs0q3JOcm2R5ktVJju/L1ib5syQrk1yWZLe+/MlJvprkyiTvGWXlJUkzm23P/bVVdSBwEHBCkscCjwYuq6qn0c2+9Hv9vn8JfKiqngFscJIO51CVpNGYbbifkGQlcBmwB7AX8FPg/H77cmBJv3ww8Il++eMbOqhzqErSaGz0E6pJDgWeDyytqnVJLgK2A+6rqup3e2DasQpJ0tjMpue+I/DDPtj3Bp69kf0vBY7pl4/bkspJkjbPbML988DCJNcA76EbmtmQtwBvTHIl3R8GSdIc2+iwTFXdCxw+w6YdBvY5BzinX74BWDqw3ylbWEdJ0ibyPndJapDhLkkNmpjnuTuHqiQNjz13SWqQ4S5JDTLcJalBEzPm7hyqkrZGo5p31567JDXIcJekBhnuktQgw12SGrRJ4Z5kcZI39MuHJjl/Y6+RJM29Te25LwbeMIqKSJKGZ1NvhTwFeEqSFcB9wN1JzgF+hW42ppdXVSU5EDiV7smRtwGvrqqbh1hvSdIGbGrP/STg+qraH3g7cABwIrAPsCdwcJJtgA8CR/Xzrv4N8GczHcw5VCVpNLb0Q0xXVNVNAH1vfglwJ11P/sIkAAuAGXvtVXU6cDrAtrvv5dR8kjQkWxru9w4sT82jGmB1VS2d+SWSpFHb1GGZHwOLNrLPtcCuSZYCJNkmyb6bUzlJ0ubZpJ57Vd2e5NIkq4B7gO/PsM9PkxwFnJZkx/4cHwBWD6PCkqSN2+Rhmap62XrK3zSwvAJ47hbUS5K0BfyEqiQ1yHCXpAZNzPPcnUNVkobHnrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDUoVZMxR0aSH9M9LrgVu9BNMdiS1trUWnvANs0Hw27Pk6pq1+mFE/P4AeDaqjpo3JUYliTLWmoPtNem1toDtmk+mKv2OCwjSQ0y3CWpQZMU7qePuwJD1lp7oL02tdYesE3zwZy0Z2IuqEqShmeSeu6SpCEx3CWpQWMP9yQvSHJtkuuSnDTu+mxIkr9JcmuSVQNlOye5MMm3+u879eVJclrfrmuSPH3gNa/q9/9WkleNoy19PfZI8s9JvpFkdZK3NNCm7ZJckWRl36Z39+VPTnJ5X7+zkjyyL9+2X7+u375k4Fjv7MuvTfJb42nRz+qyIMnVSc7v1+d7e9Yk+VqSFUmW9WXz+X23OMk5Sb7Z/z4tHXt7qmpsX8AC4HpgT+CRwEpgn3HWaSP1fS7wdGDVQNl7gZP65ZOA/9YvvxD4JyDAs4HL+/KdgW/333fql3caU3t2B57eLy8C/j+wzzxvU4Ad+uVtgMv7un4SOKYv/zDwB/3yG4AP98vHAGf1y/v078dtgSf379MFY3zvvRX4P8D5/fp8b88aYJdpZfP5ffdR4PX98iOBxeNuz1j+YQd+IEuBLwysvxN45zjrNIs6L+Hh4X4tsHu/vDvdh7EAPgIcO30/4FjgIwPlD9tvzG37B+A3W2kT8CjgKuBZdJ8IXDj9fQd8AVjaLy/s98v09+LgfmNoxxOBLwG/AZzf12/etqc//xp+Ptzn5fsOeAxwA/0NKpPSnnEPyzwBuHFg/aa+bD7ZrapuBui/P64vX1/bJrLN/X/fD6Dr6c7rNvVDGCuAW4EL6Xqpd1bV/TPU72d177ffBTyWyWrTB4B3AA/2649lfrcHoIALkixPcnxfNl/fd3sCPwD+th86OyPJoxlze8Yd7pmhrJV7M9fXtolrc5IdgE8BJ1bVjza06wxlE9emqnqgqvan6/E+E/jlmXbrv090m5L8NnBrVS0fLJ5h13nRngEHV9XTgcOBNyZ57gb2nfQ2LaQbrv1QVR0A3E03DLM+c9KecYf7TcAeA+tPBL43prpsru8n2R2g/35rX76+tk1Um5NsQxfsZ1bVp/vied2mKVV1J3AR3bjm4iRTz1IarN/P6t5v3xG4g8lp08HAv0+yBvh7uqGZDzB/2wNAVX2v/34r8Bm6P8Lz9X13E3BTVV3er59DF/Zjbc+4w/1KYK/+yv8j6S4AnTfmOm2q84Cpq9qvohu3nip/ZX9l/NnAXf1/zb4AHJZkp/7q+WF92ZxLEuB/Ad+oqlMHNs3nNu2aZHG/vD3wfOAbwD8DR/W7TW/TVFuPAr5c3YDnecAx/d0nTwb2Aq6Ym1Y8pKreWVVPrKoldL8fX66q45in7QFI8ugki6aW6d4vq5in77uqugW4Mckv9UXPA77OuNszrgsqAxcNXkh3l8b1wMnjrs9G6voJ4GbgPrq/sq+jG8/8EvCt/vvO/b4B/qpv19eAgwaO81rguv7rNWNszyF0/+27BljRf71wnrdpP+Dqvk2rgD/uy/ekC7PrgLOBbfvy7fr16/rtew4c6+S+rdcCh0/A++9QHrpbZt62p6/7yv5r9dTv/Tx/3+0PLOvfd+fS3e0y1vb4+AFJatC4h2UkSSNguEtSgwx3SWqQ4S5JDTLcJalBkzRBtrTJkjxAdzvZlCOqas2YqiNNDG+F1LyWZG1V7bCB7QvroWewSFsNh2XUnCSvTnJ2ks8CF/Rlb09yZf/87HcP7Hty/3zzLyb5RJK39eUXJTmoX96l//j/1EPJ3jdwrN/vyw/tXzP1TO8z+08Ak+QZSb6S7hnzVyRZlOSSJPsP1OPSJPvN1c9I7XNYRvPd9v0TIAFuqKoj++WlwH5VdUeSw+g+bv9Muk8Hntc/qOpuuo/0H0D3u3AVsJwNex3dx8WfkWRb4NIkF/TbDgD2pXseyKXAwUmuAM4Cjq6qK5M8BrgHOAN4NXBikqfSfcL0mi36SUgDDHfNd/dU9wTI6S6sqjv65cP6r6v79R3own4R8JmqWgeQZDbPNToM2C/J1HNdduyP9VPgiqq6qT/WCrpn/98F3FxVVwJU/9TNJGcD70rydrqPnP/dbBsszYbhrlbdPbAc4M+r6iODOyQ5kfU/UvV+Hhq23G7asd5cVQ97oFOSQ4F7B4oeoPv9ykznqKp1SS4EXgy8FDhoI+2RNolj7toafAF4bf/cepI8IcnjgIuBI5Ns3z+l8EUDr1kDHNgvHzXtWH/QPyqZJE/tn2y4Pt8EHp/kGf3+iwYe1XsGcBpw5cD/MqShsOeu5lXVBUl+Gfhqf41zLfDyqroqyVl0T8P8F+CSgZe9H/hkklcAXx4oP4NuuOWq/oLpD4AjNnDunyY5Gvhg/wjie+geQ7y2qpYn+RHwt0NqqvQz3gop9ZL8Z7rQff8cne/xdJOJ7F1VD25kd2mTOCwjjUGSV9LNV3uywa5RsOcuSQ2y5y5JDTLcJalBhrskNchwl6QGGe6S1KB/BVVCN+YAqPcGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot a barplot of the most frequent words in the text\n",
    "plt.barh(top10_words, top10_counts)\n",
    "plt.title('Term Frequency');\n",
    "plt.xlabel('Frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, 'the', 'was', 'a', etc. appear a lot in the document. \n",
    "\n",
    "However, these frequently appearing words aren't particularly useful for figuring out what these documents are about.\n",
    "\n",
    "They do not really help us to understand this text data. \n",
    "\n",
    "These words are all 'stop words', so let's drop them from the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop all stop words\n",
    "for stop in nltk.corpus.stopwords.words('english'):\n",
    "    if stop in counts:\n",
    "        counts.pop(stop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the top 20 most frequent words, of the stopword-removed data\n",
    "top20 = counts.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEWCAYAAACNJFuYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5hcVZ3u8e9LQG4JN7kIDBJBFAQhkKAyXAThoKIMoCggILch6lGR8QkOiChy5iLqyDmgIwQGEy4DKHITGa6KCfeQEJIgIApBFBQQCHdI4D1/7NWk6KrudKeruqq738/z1FO79l5777VSnf712nv/1pJtIiIiai3T7gpERETnSXCIiIg6CQ4REVEnwSEiIuokOERERJ0Eh4iIqJPgEBERdRIcoqNJer7m9bqkl2o+H9iG+nxH0sJu9TpqsOsR0WrLtrsCEb2xPbprWdJ84B9tX780x5K0rO1FTajWVNv/uIRzLQNg+/UmnC9i0KXnEEOapFGSTpD0oKQnJZ0vabWybVNJiyQdKekR4KqadUdI+rOkv0k6XNJ2kuZJekbSD5ayLrdJOknS7cCLwHqS1pB0jqS/SHpE0re6AoekZSX9v1KH30v6sqRFNcf7i6Qdaj5/R9JZNZ93lHR7qfMsSdt3q8u3yvuzkq6StHrN9p3LtgWS/ijpM+V4j3TVr5Q7UNJtS/PvEUNbgkMMdccAuwM7AH8HLAROqdk+Cng/8G5gr5p1WwIbAYcBpwGTgA+W9YdJev9S1ucg4LPAGOAvwPnAgnKu9wF7AweXsl8CPgS8F9gO2K+vJ5E0FrgMOB5YA/gGcFltAAA+AxwIrAusBnyl7PtO4Erge8BbgfHAPbanA69S/TvUtufcvtYrho8EhxjqPgcca/tR2y8D3wb2k6SaMt+0/aLtl2rWnWT7FdtXlM/n2P6b7T8CtwBb93LOg8tf612vNWq2nWX7ftsLgfWBnYCvlvM/BpwK7F/Kfhr4j1L3J4Dv9qPdhwCX2L7e9uu2rwJ+SxUou5xp+w+2XwAuBsZ11R/4he2f215k+wnbd3f9O1AFBCStQxUoLupHvWKYyD2HGLJKANiA6nJR7QiSy1D9RQzwuu1Hu+36mu2/1Xx+Cfhrt8+j6dm5vdxzeKRmeUNgBeCJmli1DPD7srxet/IP93LO7jYEDpD0qZp1y5VjdvlLzfKLLG7TBsAfejjuOcAsSV8EDgCus/1kP+oVw0SCQwxZti3pz8AnbM/svl3SmsBgDztce75HgOeB1d14+OPHqH5Rd3l7t+0vACvVfH4b0HVP4hGqXsqXl6KOjwDvarTB9kOS5gB7UvUw/n0pjh/DQC4rxVB3OvAdSRsASFpb0p5trhNQ/aIFbgO+K2mMpGUkbVJzk/mnwD9JWrcEsq91O8Rsqt7BspI+wOJ7JgBTgU9J2rXclF+xLL+tD1U7B/i4pH3KvmtJ2rLb9hOo7pP8ov8tj+EgwSGGuu8C1wO/kvQc1f2CbdpbpTc5gOpm8H3AU1TX79cp234ITAfuAW6nCha1vk51s/oZ4Djgwq4Nth8EPkl1j+VJqktSX6EP/6dt/4Eq0HwdeBq4E9i8psjPgHcCP7X9Sp9bGsOKMtlPRGeQtCkwz3ZbL/eWR1n/COxv+6Z21iXaJz2HiOjuAODZBIaRLTekI+INJeFtLFWORIxguawUERF1clkpIiLqDJvLSmuuuabHjh3b7mpERAwZM2fOfNL2Wo22DZvgMHbsWO688852VyMiYsiQ1GNWfi4rRUREnQSHiIiok+AQERF1EhwiIqJOgkNERNRJcIiIiDoJDhERUSfBISIi6gybJLi5f17A2GN/2fLzzP/Ox1p+joiIdhvUnoOkkyTtNpjnjIiI/hvUnoPtbw7m+SIiYukMuOcg6QRJ90m6TtIFkiZJGifpNklzJF0qafVSdoqkfcvyfEnfljRL0twyCxZlPtvryvozJD1c5teNiIhBMqDgIGkC1Ty2WwOfACaUTecA/2x7S2Au8K0eDvGk7W2AHwOTyrpvAb8q6y8F3t7L+SdKulPSna+9uGAgTYmIiBoD7TnsAFxu+yXbzwG/AFYGVrP9m1JmKrBTD/tfUt5nUs0+1XXMCwFsX001AXpDtifbnmB7wqiVVh1QQyIiYrGBBgcNcP9XyvtrLL7/MdBjRkTEAA00ONwE7ClpBUmjgY8BLwBPS9qxlDkY+E1PB+jhmJ8GkLQ7sPoA6xgREf00oKeVbM+QdAVwN/AwcCewADgEOF3SSsCDwGH9OOy3gQsk7UcVVB4DnlvSTu9df1XuTA5CRERTyPbADiCNtv18CQTTgIm2Zw3geMsDr9leJGk74Me2xy1pvwkTJjgzwUVE9J2kmbYnNNrWjDyHyZLeA6wATB1IYCjeDvxU0jLAq8CRfdlpsDKkayVbOiKGqwEHB9ufaUZFao73ANWjsRER0SYZeC8iIuq0feA9SWOBK21vUT5PAkYDTwGfBxYBv7W9f7vqGBEx0rQ9OPTiWOAdtl+RtFqjApImAhMBRq2y1mDWLSJiWOvky0pzgPMlHUTVe6iTDOmIiNbohOCwiDfXY4Xy/jHgR8B4YKakTu7lREQMK50QHP4KrC3prSXH4eNU9drA9q+BrwGrUd2HiIiIQdD2v8ZtL5R0EnA78BBwHzAKOE/SqlRjLZ1i+5nejpMM6YiI5ml7cACwfSpwarvrERERlY4IDs3QjgzpLsmUjojhphPuOfRK0okl9yEiIgZJU4KDpFHNOE5ERHSGPgUHSZdJminpnpJ4hqTnJZ0k6XZgO0m7SrqrzAd9dnnyqGuu6DXL8gRJN5blE0u5GyU9KOmomvMdL+l+SdcD725ymyMiYgn62nM43PZ4qjmij5L0VqrpQOfZfj/VPA5TgP1sv5fqXsYX+nDcTYEPA+8DviVpOUnjgf1ZPC/1tj3tnDmkIyJao6/B4ShJdwO3ARsAm1BN7fnzsv3dwEO2f1c+9zZvdK1f2n7F9pPA48A6wI7ApbZftP0scEVPOydDOiKiNZb4tJKknYHdgO1sv1guC60AvGz7ta5ivRyiNgN6hW7bXqlZrp1HemAzEEVExID0peewKvB0CQybAh9oUOY+YKykd5bPtfNGz6caAgPgk3043zRgH0krShoD7NmHfSIioon6kudwNfB5SXOA+6kuLb2J7ZclHQb8rIyBNAM4vWz+NvBfkr5OlQXdK9uzJF0EzKaal3p6XxqSDOmIiOYZ8BzSnSJzSEdE9E+r55DuCO3MkIZkSUfE8NL0DGlJUyTt24dy60m6eAllxkqa17zaRUREX7Sl5yBpWduPAksMIhERMfj6miF9gqT7JF0n6QJJkyQdKWmGpLsl/VzSSjW77CZpuqTfSfp4Ocahkn4m6RfAtbW9AkmjJH2vHG+OpM81v6kREdFXSwwOkiZQPYLalbHcdfPiEtvb2t4KuBc4oma3scAHqWZzO11SV37DdsAhtj/U7TRHAAtsb0uVEX2kpHf0oW7JkI6IaIG+XFbaAbjc9ksA5S9/gC0k/QuLZ2m7pmafn9p+HXhA0oNUw2QAXGf7qQbn2B3YsuZexapUWdi/a1D2DbYnA5MBll93k+Hx2FVERAfoS3DoKft5CrC37bslHQrsXLOt+y/qrs8v9HKOL9u+5k0rpbF9qF9ERDRZX+453ATsKWkFSaOpLhUBjAEek7QccGC3fT4laRlJGwMbUSXP9eYa4AvlWEh6l6SV+9yKiIhoqiX2HGzPkHQFcDdVxvKdwALgBKqM54eBuVTBosv9VMNnrAN8vmRQ93aas6juU8xSVfAJYO/+NCQZ0hERzdOnDGlJo20/X55ImgZMtD2r5bXrh2RIR0T0TzMypCdLeg/VqKpTOy0wQPszpCFZ0hExfPQpONj+TG/by43jK21v0YQ6RUREmzV9+IyIiBj6WjG20kZlLuljJF0i6WpJD0j6bk2ZA8pc0/MknVzWfVrSD8ryV0p+BJI2lnRTs+sZERE9a+rYSpLeDVwIHAaMK6+tqWZ8u1/SaVQzvp1MNQHQ01RDaexNdaP7mHKoHYG/SVqfKgmv4ZwOkiYCEwFGrbJWM5sSETGiNbPnsBZwOXCQ7dll3Q22F9h+GfgtsCHV8Bg32n7C9iLgfGAn238BRpfZ3zYA/ptqHuod6SE4ZA7piIjWaGZwWAA8Amxfs67RHNG9JTzcStXruJ8qIOxINR7TzU2sZ0RELEEzg8OrVIlrn5XU29NNtwMflLSmpFHAASyeb3oaMKm83wXsArxiO6PqRUQMoqbec7D9Qhmi+zrgvB7KPCbpOODXVL2Iq2xfXjZPp7qkNM32a5IeAe7ry7mTIR0R0TyZQzoiYoTKHNJtkGzpiBjK2poEJ2mCpFPL8qGSftjO+kRERKWtPQfbd1KN8hoRER2kKT0HSQdJukPSbElnlDmhn5d0sqSZkq6X9D5JN0p6UNI/lP12lnRlt2ONkfRQzdwOq0ia3/U5IiJab8DBQdJmwH7A9rbHUeUzHAisTJXsNh54DvgX4H8B+wAn9XQ8288BN7J4UqH9gZ/bXtjg3JlDOiKiBZpxWWlXqqEwZpQJfVYEHqfKe7i6lJlLla+wUNJcqol9enMW8DXgMqqkuCMbFcoc0hERrdGM4CCqOR6Oe9NKaZIXPyf7OiVb2vbrkno9r+2bJY2V9EFglO15TahnRET0UTPuOdwA7CtpbQBJa0jasAnHPQe4APhJE44VERH9MOCeg+3fSvoG1eiqywALgS8OuGbVgHz/QhUgligZ0hERzdOxGdKS9gX2sn1wX8onQzoion+GXIZ0mffho8Aefd0nGdIREc0zaBnSkp4v7+tJurgsN8yKtv1l2++0/bvBql9ERCw26D0H248C+w72eSMiou8GfWyl8ohq3aOpkj4m6dYyz8Nakn4uaUZ5bd/oWBER0Rodcc9B0j7AV4E9bD8t6b+BU2zfJOntwDXAZg32yxzSEREt0AnBYRdgArC77WfLut2A95SMa4BVJI0pQ2u8IRnSERGt0QnB4UFgI+BdLB6hdRlgO9svta1WEREjWFvncygeBj4BnCNp87LuWuBLXQUkjWtHxSIiRqpO6Dlg+35JBwI/k7QncBTwI0lzqOo4Dfh8b8dIhnRERPN0bIZ0fyVDOiKif4ZchvTS6LQM6VrJlo6IoaYT7jnUkTSljK0UERFt0JHBISIi2qupwaFkP98r6UxJ90i6VtKKksZJuk3SHEmXSlq9lD+yZEDfXTKiV6o53E6SbilzTqcXERExiFrRc9gE+JHtzYFngE9STdzzz7a3pJoy9Ful7CW2t7W9FXAvcETNcdYFdgA+Dnyn0Ykyh3RERGu0Ijg8ZHt2WZ4JbAysZvs3Zd1UYKeyvIWk6WVe6QOBzWuOc5nt123/Flin0YlsT7Y9wfaEUSut2vyWRESMUK0IDq/ULL8GrNZL2SnAl2y/F/g2sEIPxxERETFoBuOG9ALgaUk7ls8HA129iDHAY5KWo+o5REREBxisPIdDgNPLDecHgcPK+hOA26mG0JhLFSyWSjKkIyKaJxnSEREjVDKk2ywZ0hEx1HREEpykEyVNarC+4axxERHRWh0RHCIiorMsVXCQ9DVJR5XlUyT9qizvKuk8SQdImitpnqSTa/Z7vmZ5X0lTGhx7fMmYvhX44tLULyIiBmZpew7TgK5HUycAo8vjqDsADwAnAx8CxgHbStq7H8f+CXCU7e2WVDAZ0hERrbG0wWEmMF7SGKpktVupgsSOVENm3Gj7CduLgPNZnBHdK0mr8uZs6nN7K58M6YiI1liq4GB7ITCfKl/hFmA6sAvVUBl/7G3XmuUVGmxXtzIREdEGA7khPQ2YVN6nU03jORu4DfigpDUljQIOYHFG9F8lbSZpGWCf7ge0/QywQNIOZVWypiMi2mAgeQ7TgeOBW22/IOllYLrtxyQdB/yaqidwle3Lyz7HAlcCjwDzgNENjnsYcLakF4Fr+lqZZEhHRDRPMqQjIkaoZEh3kGRLR8RQkCS4iIiok+AQERF1WhIcJB0k6Q5JsyWdIekISafUbD9S0g96KDuqvKaUDOu5kv6pFfWMiIjGmh4cJG0G7Adsb3sc1Wxwi4B/KFnUUD2R9JMeyh5IlVm9vu0tyixxP+nhXMmQjohogVbckN4VGA/MkASwIvA48Cvg45LuBZazPVfSl3oo+wtgI0mnAb8Erm10ItuTgckAy6+7yfB47CoiogO0IjgImGr7uDetlN4PfB24j8U9gYZlS/mtgA9TDb73aeDwFtQ1IiIaaMU9hxuAfSWtDSBpDUkb2r4d2AD4DHBBb2UlrQksY/vnVFOJbtOCekZERA+a3nOw/VtJ3wCuLcNkLKT66/9h4KfAONtPL6HsS1T3JLqCV13PortkSEdENE9LkuBsXwRc1GDTDsApfSyb3kJERJsMSoa0pNWAO4C7bd/QinMMlQzpLsmUjohONihJcLafsf0u259qtL23uaIl3Sip4dgfERHRGsmQjoiIOk0JDpI+K2lOmfv53PLE0Q1l3Q2S3l7KTZG0b81+zzc41oqSLiz7XkSV+xAREYNowPccJG1ONa/D9raflLQGMBU4x/ZUSYcDpwJ9nUf6C8CLtreUtCUwq5dzTwQmAoxaZa2BNCMiImo0o+fwIeBi208C2H4K2A7477L9XKqnlPpqJ+C8cqw5wJyeCmYO6YiI1mhGcOjLvM9d2xd1nVPVeBlvWUL5iIhog2YEhxuAT0t6K1RZzsAtwP5l+4HATWV5PtVYSgB7ActRb1rZB0lbAFs2oY4REdEPA77nYPseSf8K/EbSa8BdwFFU80AfAzxBNQorwJnA5ZLuoAoqLzQ45I+psqPnALOp8iOWKBnSERHNkzmkIyJGqMwh3cGSKR0RnShJcBERUSfBISIi6gxacJD01TIn9DxJR5fxlO6VdKakeyRdK2nFUnZjSVdLmilpuqRNB6ueERExSMFB0niqJ5beD3wAOBJYHdgE+JHtzYFngE+WXSYDX7Y9HpgE/GcPx80c0hERLTBYN6R3AC61/QKApEuAHYGHbM8uZWYCYyWNBv4e+FmZVxpg+UYHzRzSERGtMVjBQT2sf6Vm+TWqQfaWAZ6xPa7ltYqIiIYG657DNGBvSStJWhnYB5jeqKDtZ4GHJH0KqmE2JG01SPWMiAgGqedge5akKSzOdj4LeLqXXQ4Eflzml14OuBC4u7dzJEM6IqJ5kiEdETFCJUO6wyVLOiI6TUckwUk6VNJ6NZ/nS1qznXWKiBjJOiI4AIcC6y2pUEREDI6WBQdJK0v6ZZlXep6k/SSNl/Sbkvl8jaR1y5zSE4DzJc3uypIGvixplqS5yZCOiBhcrew5fAR41PZWtrcArgZOA/Ytmc9nA/9q+2LgTuBA2+Nsv1T2f9L2NlTzO0xqdIJkSEdEtEYrb0jPBb4v6WTgSqpHV7cAriuZz6OAx3rZ/5LyPhP4RKMCyZCOiGiNlgUH278rYyrtAfw7cB1wj+3t+niIruzp1xhGT1VFRAwFrbznsB7wou3zgO9TDbq3lqTtyvblJG1eij8HjGlVXSIion9a+Rf5e4HvSXodWAh8AVgEnCpp1XLu/wvcA0wBTpf0EtDXnsWbT5YM6YiIpkmGdETECJUM6WEimdQRMViafs+hzPA2b4DH2FnSlc2qU0RE9E+nZEhHREQHaVVwWFbSVElzJF1c5nH4pqQZJVt6skqyg6R3Srq+ZFLPkrRx7YEkbSvpLkkbtaiuERHRTauCw7uByba3BJ4F/jfwQ9vblmzpFYGPl7LnU80jvRXV9KBvJMZJ+nvgdGAv2w92P0kypCMiWqNVweER2zeX5fOo5pDeRdLtkuYCHwI2lzQGWN/2pQC2X7b9YtlvM6rs5z1t/7HRSWxPtj3B9oRRK63aoqZERIw8rQoO3Z+PNfCfVOMqvRc4E1iBnueWhqoH8TKwdUtqGBERPWpVcHh7VyY0cABwU1l+UtJoYF94Y77oP0naG0DS8pJWKmWfAT4G/JuknVtUz4iIaKBVeQ73AodIOgN4gGpk1dWpBuObD8yoKXswcIakk6gyqT/VtcH2XyXtCfyPpMNt397TCZMhHRHRPMmQjogYoZIhPYwkSzoiBkPHJMFJOknSbg3WJ1s6ImKQdUzPwfY3212HiIiotDQ4SFoZ+Cnwd1Qzv/0fqgS5PakS4W4BPmfbkqYAV9q+WNJHqIbzfhKY1co6RkREvVZfVmo0j3RPmdIASFqBKg9iT2BH4G09HTwZ0hERrdHq4DAX2E3SyZJ2tL2ABpnS3fbZFHjI9gOuHqU6r6eDJ0M6IqI1WnpZqfs80pKuBb4ITLD9iKQTqTKl63ZtZb0iIqJ3Le05NJhHepuy6U2Z0t3cB7yjZnTWA1pZx4iIqNfqp5UazSO9N40zpYFq8D1JE4FfSnqSauiNLZZ4omRIR0Q0TTKkIyJGqGRIR58k+zoiunRMhnQXSVO6RmGVdHTNKK0RETFIOi44dHM0kOAQETHI2hYcJI2VdK+kMyXdI+laSSsCC4BXJR0FrAf8WtKv21XPiIiRqN09h02o5o/enGpyn0/a/ortW2yfCjwK7GJ7l0Y7J0M6IqI12h0cHrI9uyzPBMb2Z+dkSEdEtEa7g8MrNcuvMYyenoqIGMraHRyW5DlgTLsrEREx0nT6X+qTqeaPfqyn+w5dkiEdEdE8bQsOtudTMyyG7e83KHMacNogVisiIuj8nkOfJUO6/ZJhHTF8dPo9h4iIaIOOCg6SRrW7DhER0cTLSpK+Brxs+1RJpwBb2f6QpF2Bw6iePNqWamrQi21/q+w3Hzgb2B34oaQxwETgLcDvgYNtv9isekZExJI1s+cwjWrOZ4AJwGhJywE7ANOB48vQsFsCH5S0Zc2+L9vewfaFwCVljumtgHuBI3o6YTKkIyJao5nBYSYwvvzl/wpwK1WQ2JEqOHxa0izgLqp5o99Ts+9FNctbSJpe5pg+kPo5pt+QDOmIiNZo2mUl2wvLJaLDgFuAOcAuwMbAS8AkYFvbT0uawpvnjn6hZnkKsLftuyUdCuzcrDpGRETfNPuG9DSqIDCNqrfweWA2sApVAFggaR3go70cYwzwWLkkdWCT6xcREX3Q7DyH6cDxwK22X5D0MjC99ALuAu4BHgRu7uUYJwC3Aw9TzTXdp+EzkiEdEdE8mUM6ImKEyhzSMaiSKR0x9A1KEpyk+ZLWXEKZqyStNhj1iYiI3nVMz8H2Hu2uQ0REVJrec5B0maSZZV7oiQ22HyTpDkmzJZ3RNWRGbe9C0lclzSuvo5tdx4iI6F0rLisdbns8VQLcUZLe2rVB0mbAfsD2tsdRzf72psdVJY2nypV4P/AB4EhJWzc6UTKkIyJaoxWXlY6StE9Z3gDYpGbbrsB4YIYkqMZZerzb/jsAl9p+AUDSJVRZ1nd1P5HtyVQTArH8upsMj8euIiI6QFODg6Sdgd2A7Wy/KOlG3pwJLWCq7eN6O0wz6xQREf3X7MtKqwJPl8CwKdVloVo3APtKWhtA0hqSNuxWZhqwt6SVJK0M7EOVXBcREYOk2ZeVrgY+L2kOcD9wW+1G27+V9A3gWknLAAuBL1JlQ5cinlXGXrqjrDvLdt0lpe6SIR0R0TwdkSFdnlh6HHib7YVLc4xkSEdE9M9QyJC+h6qHsFSBAZIhPVwkuzqiM3REcLC9KVS5DrbHtrk6EREjXkfNIR0REZ2h04LDEwCS1pU0rWRRz5O045J2jIiI5umIy0pdbG9bFj8DXGP7X8vN6pUalS/Dc0wEGLXKWoNTyYiIEaCjgkONGcDZZTa4y2zPblQoGdIREa3RaZeVALA9DdgJ+DNwrqTPtrlKEREjSkcGh5I1/bjtM4H/ArZpc5UiIkaUTr2stDNwjKSFwPPAEnsOyZCOiGiejgwOtqcCU9tdj4iIkaojgoOkZYGrgH+yfU/3z305RjKkI2KkaeWIAh1xz8H2IuAg4N8kLdf9c3trFxEx8nREzwHA9uPAXj19joiIwdMRPYeIiOgsQzo4ZA7piIjWGNLBwfZk2xNsTxi10qrtrk5ExLAxpINDRES0RoJDRETU6ZinlQYqGdIREc2TnkNERNRJcIiIiDoJDhERUSfBISIi6iQ4REREnQSHiIiok+AQERF1EhwiIqJOgkNERNSR7XbXoSkkPQfc3+56tMiawJPtrkQLpX1DW9o3dG1oe61GG4bN8BnA/bYntLsSrSDpzuHaNkj7hrq0b3jKZaWIiKiT4BAREXWGU3CY3O4KtNBwbhukfUNd2jcMDZsb0hER0TzDqecQERFNkuAQERF1hnxwkPQRSfdL+r2kY9tdn6Ulab6kuZJmS7qzrFtD0nWSHijvq5f1knRqafMcSdu0t/b1JJ0t6XFJ82rW9bs9kg4p5R+QdEg72tJID+07UdKfy3c4W9IeNduOK+27X9KHa9Z33M+vpA0k/VrSvZLukfSVsn5YfH+9tG9YfH9NY3vIvoBRwB+AjYC3AHcD72l3vZayLfOBNbut+y5wbFk+Fji5LO8B/A8g4APA7e2uf4P27ARsA8xb2vYAawAPlvfVy/Lq7W5bL+07EZjUoOx7ys/m8sA7ys/sqE79+QXWBbYpy2OA35U2DIvvr5f2DYvvr1mvod5zeB/we9sP2n4VuBDYq811aqa9gKlleSqwd836c1y5DVhN0rrtqGBPbE8Dnuq2ur/t+TBwne2nbD8NXAd8pPW1X7Ie2teTvYALbb9i+yHg91Q/ux3582v7MduzyvJzwL3A+gyT76+X9vVkSH1/zTLUg8P6wCM1n/9E719yJzNwraSZkiaWdevYfgyqH2hg7bJ+qLa7v+0Ziu38Urm0cnbXZReGcPskjQW2Bm5nGH5/3doHw+z7G4ihHhzUYN1QfTZ3e9vbAB8Fvihpp17KDqd2Q8/tGWrt/DGwMTAOeAz4j7J+SLZP0mjg58DRtp/trWiDdUOxfcPq+xuooR4c/gRsUPP574BH21SXAbH9aHl/HLiUqsv6167LReX98VJ8qLa7v+0ZUu20/Vfbr9l+HTiT6juEIdg+SctR/eI83/YlZfWw+f4atW84fX/NMNSDwwxgE0nvkPQWYH/gijbXqd8krSxpTNcysDswj6otXU94HAJcXkuHJhAAAANkSURBVJavAD5bnhL5ALCgq7vf4frbnmuA3SWtXrr4u5d1HanbfZ99qL5DqNq3v6TlJb0D2AS4gw79+ZUk4L+Ae23/oGbTsPj+emrfcPn+mqbdd8QH+qJ6UuJ3VE8NHN/u+ixlGzaietLhbuCernYAbwVuAB4o72uU9QJ+VNo8F5jQ7jY0aNMFVF3zhVR/YR2xNO0BDqe6Afh74LB2t2sJ7Tu31H8O1S+JdWvKH1/adz/w0U7++QV2oLo8MgeYXV57DJfvr5f2DYvvr1mvDJ8RERF1hvplpYiIaIEEh4iIqJPgEBERdRIcIiKiToJDRETUWbbdFYhoJ0mvUT2+2GVv2/PbVJ2IjpFHWWNEk/S87dG9bF/W9qLBrFNEJ8hlpYhuJB0q6WeSfgFcW9YdI2lGGZTt2zVljy/j+V8v6QJJk8r6GyVNKMtrSppflkdJ+l7NsT5X1u9c9rlY0n2Szi+ZvEjaVtItku6WdIekMZKmSxpXU4+bJW05WP9GMfzlslKMdCtKml2WH7K9T1neDtjS9lOSdqcaMuF9VNnAV5SBEV+gGjJha6r/S7OAmUs43xFUw0tsK2l54GZJ15ZtWwObU43PczOwvaQ7gIuA/WzPkLQK8BJwFnAocLSkdwHL254zoH+JiBoJDjHSvWR7XIP119numq9h9/K6q3weTRUsxgCX2n4RQFJfxtXZHdhS0r7l86rlWK8Cd9j+UznWbGAssAB4zPYMAJfRUSX9DDhB0jFUQ1RM6WuDI/oiwSGisRdqlgX8u+0zagtIOpqeh2hexOLLtit0O9aXbb9pADpJOwOv1Kx6jer/pxqdw/aLkq6jmlzm08CEJbQnol9yzyFiya4BDi/j/yNpfUlrA9OAfSStWEbV3bNmn/nA+LK8b7djfaEMGY2kd5WReHtyH7CepG1L+TGSuv6oOws4FZhR08uJaIr0HCKWwPa1kjYDbi33iJ8HDrI9S9JFVKN6PgxMr9nt+8BPJR0M/Kpm/VlUl4tmlRvOT7B4us1G535V0n7AaZJWpLrfsBvwvO2Zkp4FftKkpka8IY+yRjSJpBOpfml/f5DOtx5wI7CpqwlqIpoml5UihiBJn6Wa9/j4BIZohfQcIiKiTnoOERFRJ8EhIiLqJDhERESdBIeIiKiT4BAREXX+P8h1mEmxzJucAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot a barplot of the most frequent words in the text\n",
    "plt.barh([it[0] for it in top20], [it[1] for it in top20])\n",
    "plt.title('Term Frequency');\n",
    "plt.xlabel('Frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks potentially more relevant / useful! \n",
    "\n",
    "As a distribution of meaningful word, bag-of-word representations can be used to analyze text data in various ways.\n",
    "\n",
    "If you are interested in further analysis of this data, look through the `nltk` tutorials for further analyses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term Frequency - Inverse Document Frequency (TF-IDF)\n",
    "\n",
    "Finally, let's look at another approach for encoding text data - term frequency / inverse document frequency.\n",
    "\n",
    "Note that TF-IDF is a similar kind of word counting to bag-of-words. \n",
    "\n",
    "First, let's consider a difficulty with bag-of-words encodings, which is it's difficult to interpret the word counts. For example, knowing a word occurs 10 times isn't itself really enough information to understand something about the document the word come from. If that document is 100 words long, that word seems like it must be quite important. But if the document is 10,000 words long, then this word seems less important. \n",
    "\n",
    "TF-IDF tries to address this, and does so by scaling the counts of words by the typical occurrence. \n",
    "\n",
    "- The term-frequency is the count of the term in the document, which is the same as in the bag-of-words. \n",
    "\n",
    "- The inverse-document-frequency is a measurement of how commonly the term occurs across a corpus. \n",
    "  - Since it's calculated as an inverse, a higher IDF score is a rarer word. \n",
    "\n",
    "The TF-IDF score is calculated by multiplying the TF by the IDF. One way to think of this is that it normalizes, or scales, term occurrences in a document by a population measure of the occurrence of the term. \n",
    "\n",
    "This allows for a representation of the text data which indicates if terms in the document of interest occur more or less frequently than expected (given a corpus). A high TF-IDF score could occur, for example, due to a relatively large number of occurrences of a typically rare term. This may be an interesting and useful feature to describe the text.\n",
    "\n",
    "Here, we will briefly examine applying TF/IDF to text data. Note that in this case, we are using an object from `sklearn` that can be used to compute the TF/IDF. \n",
    "\n",
    "<div class=\"alert alert-success\">\n",
    "Term Frequency - Inverse Document Frequency is representation of text data that scales term occurrences by corpus statistics.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "TF-IDF on \n",
    "<a https://en.wikipedia.org/wiki/Tf%E2%80%93idf class=\"alert-link\">wikipedia</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a TFIDF object, applying some settings\n",
    "tfidf = TfidfVectorizer(analyzer='word',\n",
    "                        sublinear_tf=True,\n",
    "                        max_features=5000,\n",
    "                        tokenizer=nltk.word_tokenize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TfidfVectorizer will calculate the inverse document frequency (IDF) for each word across our corpus of words. \n",
    "\n",
    "The TFIDF for each term, for a particular document, can then be calculated as TF * IDF.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learn the TFIDF representation of our data\n",
    "#   Note that this takes the raw sentences, tokenizes them, and learns the TF/IDF representation\n",
    "tdidf = tfidf.fit(sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we apply this representation to our data, let's explore what the data object and calculated scores. \n",
    "\n",
    "If you explore the tfidf object you'll see it includes attributes:\n",
    "- `vocabulary_`, which maps the terms to their indices\n",
    "- `idf_`, which has the IDF values for each term. \n",
    "\n",
    "Let's now plot out the IDF for each of the top 10 most frequently appeared words (from the first analysis). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the IDF weights for the top 10 most common words\n",
    "IDF_weights = [tfidf.idf_[tfidf.vocabulary_[token]] for token in top10_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXg0lEQVR4nO3de7RkZX3m8e9jg4BCYARGEZEGJBJBuYqgxMEFawJxRkziBVFHjJFRRhEiqHFm6TIGxxiWTNAYRUXBGMNFBwlRBo1B8ILQrc2lgybcFMQryP0Ov/lj74byeLpPnT5Vdc55+/tZq9bZVfv2vrW7n3rr3bv2m6pCktSex8x3ASRJ42HAS1KjDHhJapQBL0mNMuAlqVEGvCQ1yoCXpEYZ8A1Icn2SA+e7HGsryaeT3J/kjv5xZZL/nWTT+S7bqCTZP8mNMyyz6n24c+Dx8kmVUe0x4LVWkqw34k1+oKo2AbYEXgvsA3wzyeNHvJ+F7gNVtfHA4/SpC4zhvVejDPjGJDk8yTeSnJDkV0muS3JwP+/QJMumLH9MknP66Q369X6U5GdJPppko37e/kluTPL2JD8FPpVkiyTnJrk1yS1JLkrymH75Jyf5fJJf9GU4apjyV9W9VXUp8CJgc7qwJ8ljkvyvJD9M8vMkpw228JPsl+RbfVluSHJ4//oFSf5k6vsz8LySHJnk3/tvD+9NskOSbye5PckZSR47sPx/SbKi38+3kjxrYN71SY5NcnmS25KcnmTD/kPqy8CTB1rmTx7qgP76tt+e5HLgriTrrek9TrJR/43gV0n+Nclxg98g+no/beD5p5P8xVzqOTD/kH7d25Nck+SgJC9NsnxKnd6a5OzZvA+aHQO+Tc8BfgBsAXwA+GSSAOcAT0+y48CyhwF/30//JfDbwG7A04CtgXcNLPsk4AnAtsARwFuBG+la3U8E3glUH/L/CFzWb+MA4OgkvzdsBarqDuArwO/2Lx3eP14AbA9sDHwYIMlT6QL0Q31ZdgNWDLsv4CBgT7pvDW8DTgZeCWwD7AK8ot/PHsApwH+n+/D5GHBOkg0GtvWyfnvbAc8CDq+qu4CDgZsGWuY3zaJ8q7wCeCGwGfAwa36P3w3s0D9+D3jNsDtZ23r26+4NnAYc15fz+cD1dP/2tkvyOwPbeBXwmWHLpdkz4Nv0w6r6eFU9BJwKbAU8saruBr7Io4G1I7AT3X/eAK8HjqmqW/qAfR9w6MB2HwbeXVX3VdU9wAP9tretqgeq6qLqbm70bGDLqvrzqrq/qq4FPj5lW8O4ie4DBbrA/WBVXVtVdwJ/BhyarrvilcBXq+pzfTlurqrZBPxfVtXtVbUSuBI4v9/PbXQfHLv3y70e+FhVfaeqHqqqU4H76D4YVjmpqm6qqlvoAni3Wdb52L7VfGuSX06Zd1JV3dC/9zO9xy8Dju+P5Q3ASbMow1zq+TrglKr6SlU9XFU/rqrvV9V9wOl0oU6SnYGlwLmzKJdmyYBv009XTfShDl2LF7rW+iv66cOAs/tltgQeByxfFTDAef3rq/yiqu4deP5XwNXA+UmuTfKO/vVt6bojbh3Y1jvpWvmzsTVwSz/9ZOCHA/N+CKzXb3Mb4JpZbnvQzwam75nm+ar3blvgrVPqtU1ftlV+OjB998C6wzqhqjbrH1tMmXfDwPRM7/GTpyw/+N7NZC71XNOxOBU4rG9MvBo4ow9+jYkna9Y95wNbJNmNLuiP6V//JV2Y7VxVP17Nur9269G+lf9WujDYGfiXJJfSBct1VbXjNNsYSpKNgQOB4/uXbqILnlWeCjxIF8Y3AHuvZlN30X1wrfKktS1Tv5/jq+r4GZf8TaO4bevgNmZ6j39CF7Yr++dPnTL/bn7zfVnVRz+Xet5A1y30G6rq4iT303W7HdY/NEa24NcxVfUgcBZd6/sJdP3cVNXDdF/xT0zyHwGSbL2mfvP+RNzT+hbZ7cBD/eMS4Pb+pOBGSZYk2SXJs2cqX7oTvXsCZwO/Aj7Vz/occEyS7frwfx9wel+fzwIHJnlZf/Jx8/4DDLq++D9M8rj+pOLrZvF2TfVx4A1JnpPO45O8MMkmQ6z7M2DzjO7Sz5ne4zOAP0vyH5I8BXjzlPVX0LWmlyQ5CPhPA/PmUs9PAq9NckC6E+NbJ9lpYP5pdOdOHqyqb0y/CY2KAb9u+nu61vGZfUCu8na6LpeLk9wOfBV4+hq2s2O/zJ3At4GPVNUFfd//f6Xrl72O7tvBJ4A1hdvbktxB1yVzGrAceG5/ghK6k36fAS7st3kvfWhV1Y+A36f7NnELXXjt2q93InA/XcCeSvdhsFaqahld//SH6T58rqY/uTjEut+n+5C6tu/2mNVVNNNsb6b3+D103TLX0X1rm3oy8y39+rfSncN45GqWOdbzErorn04EbgO+zq9/8/oM3YlrT65OQBzwQ2pfkv2Bv6uqp8xzOTYCfg7sUVX/Pp9lWRfYgpc0SW8ELjXcJ8OTrJImIsn1QIAXz3NR1hl20UhSo+yikaRGLZgumi222KKWLl0638WQpEVl+fLlv6yqLaebt2ACfunSpSxbtmzmBSVJj0iy2l8p20UjSY0y4CWpUQa8JDXKgJekRhnwktQoA16SGmXAS1KjDHhJatSC+aHTFT++jaXv+Kf5LoYkTdT173/h2LZtC16SGmXAS1KjDHhJapQBL0mNGlnAJ/nWqLYlSZq7kQV8VT13VNuSJM3dKFvwd/Z/909yQZKzknw/yWeTZFT7kSQNZ1x98LsDRwPPALYHnjfdQkmOSLIsybKH7r5tTEWRpHXTuAL+kqq6saoeBlYAS6dbqKpOrqq9qmqvJY/bdExFkaR107gC/r6B6YdYQL+YlaR1hZdJSlKjDHhJatTIuk6qauP+7wXABQOvv2lU+5AkDc8WvCQ1yoCXpEYtmKtbnrn1piwb432RJWldYwtekhplwEtSowx4SWrUgumDd0xWSeuScY7FuooteElqlAEvSY0y4CWpUQa8JDVqTgGfZGmSK0dVGEnS6NiCl6RGjSLglyT5eJKVSc5PslGSHZKcl2R5kouS7DSC/UiSZmEUAb8j8DdVtTNwK/BHwMnAm6tqT+BY4CPTreiYrJI0PqP4odN1VbWin15ON/7qc4Ezk6xaZoPpVqyqk+k+DNhgqx1rBGWRJPVGEfBTx199InBrVe02gm1LktbSOE6y3g5cl+SlAOnsOob9SJLWYFxX0bwSeF2Sy4CVwCFj2o8kaTXm1EVTVdcDuww8P2Fg9kFz2bYkaW68Dl6SGmXAS1KjDHhJatSCGfDDQbclabRswUtSowx4SWqUAS9JjVowffAOui2pNZMYWHtNbMFLUqMMeElqlAEvSY0y4CWpUQa8JDVqxoBP8rYkR/XTJyb5Wj99QJK/S/K3/bB7K5O8Z2C99yf51ySXJzlhdduXJI3HMJdJXgi8FTgJ2AvYIMn6wH7ARcCZVXVLkiXAPyd5FnAj8AfATlVVSTYbT/ElSaszTBfNcmDPJJvQDc/3bbqg/126gH9Zku8C3wN2Bp5BN6rTvcAnkvwhcPd0G3bQbUkanxkDvqoeAK4HXgt8iy7UXwDsANwDHAscUFXPAv4J2LCqHgT2Bj4PvBg4bzXbPrmq9qqqvZY8btO510aS9IhhT7JeSBfkF9IF/BuAFcBvAXcBtyV5InAwQJKNgU2r6kvA0YADcEvShA17q4KLgP8JfLuq7kpyL3BRVV2W5Ht0465eC3yzX34T4ItJNgQCHDPickuSZjBUwFfVPwPrDzz/7YHpw1ez2t5zKpkkaU68Dl6SGmXAS1KjDHhJatSCuR+8Y7JK0mjZgpekRhnwktQoA16SGrVg+uAdk1XSQjPfY6rOlS14SWqUAS9JjTLgJalRBrwkNWrsAZ/kqCRXJfnsuPclSXrUJK6iORI4uKqum8C+JEm9kbbgk/xpkiv7x9FJPgpsD5yTxHvCS9IEjawFn2RPumH9nkM3yMd3gFcBBwEvqKpfTrPOEcARAEt+a8tRFUWSxGhb8PsB/7eq7qqqO4Ev0A3MvVqOySpJ4zPKgM8ItyVJmqNRBvyFwIuTPC7J44E/oBvLVZI0D0bWB19V303yaeCS/qVPVNX3Ehv2kjQfRnqZZFV9EPjglNeWjnIfkqTh+EtWSWqUAS9JjVow94N3TFZJGi1b8JLUKANekhplwEtSoxZMH7xjskoal8U+turasgUvSY0y4CWpUQa8JDXKgJekRhnwktQoA16SGjW2gE9ydpLlSVb2Q/NJkiZonNfB/3FV3ZJkI+DSJJ+vqpsHF3BMVkkan3F20RyV5DLgYmAbYMepCzgmqySNz1ha8En2Bw4E9q2qu5NcAGw4jn1JkqY3rhb8psCv+nDfCdhnTPuRJK3GuAL+PGC9JJcD76XrppEkTdBYumiq6j7g4HFsW5I0HK+Dl6RGGfCS1KgFcz94x2SVpNGyBS9JjTLgJalRBrwkNWrB9ME7Jquk2VpXx1odli14SWqUAS9JjTLgJalRBrwkNWoiAZ/kW5PYjyTpURMJ+Kp67iT2I0l61KRa8HdOYj+SpEfNax98kiOSLEuy7KG7b5vPokhSc+Y14B2TVZLGx6toJKlRBrwkNcqAl6RGTeoyyY0nsR9J0qNswUtSowx4SWrUgrkfvGOyStJo2YKXpEYZ8JLUKANekhq1YPrgHZNV0kLQ0jivtuAlqVEGvCQ1yoCXpEaNLOCTrJfk/CQ7T/dckjRZIwv4qnoQeBXwviTrT30+qv1IkoYz0qtoqurnwCGrey5Jmhz74CWpUY7JKkmNckxWSWqUXTSS1CgDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQvmfvCOySpJo2ULXpIaZcBLUqMMeElq1ILpg3dMVmnd0dK4pwuZLXhJapQBL0mNMuAlqVEGvCQ1aiQBn2SzJEeOYluSpNEYVQt+M8CAl6QFZFSXSb4f2CHJCuAr/WsHAwX8RVWdPqL9SJKGNKoW/DuAa6pqN+BiYDdgV+BA4K+SbDWi/UiShjSOk6z7AZ+rqoeq6mfA14FnT7egg25L0viMI+Az7IIOui1J4zOqgL8D2KSfvhB4eZIlSbYEng9cMqL9SJKGNJKTrFV1c5JvJrkS+DJwOXAZ3UnWt1XVT0exH0nS8EZ2s7GqOmzKS8eNatuSpNnzl6yS1CgDXpIaZcBLUqMWzIAfDrotSaNlC16SGmXAS1KjDHhJatSC6YN30G1p8XIQ7YXJFrwkNcqAl6RGGfCS1CgDXpIaNfaAT3LnuPchSfpNtuAlqVFDBXySs5MsT7IyyRH9a3cmOT7JZUkuTvLE/vXtknw7yaVJ3jvOwkuSVm/YFvwfV9WewF7AUUk2Bx4PXFxVu9KN4vT6ftm/Bv62qp4NrHGgD8dklaTxGTbgj0pyGXAxsA2wI3A/cG4/fzmwtJ9+HvC5fvoza9qoY7JK0vjM+EvWJPsDBwL7VtXdSS4ANgQeqKrqF3toyrYKSdK8GqYFvynwqz7cdwL2mWH5bwKH9tOvnEvhJElrb5iAPw9YL8nlwHvpumnW5C3A/0hyKd2HgyRpHszYRVNV9wEHTzNr44FlzgLO6qevA/YdWO79cyyjJGkteB28JDXKgJekRi2Y+8E7JqskjZYteElqlAEvSY0y4CWpUQumD94xWaXRcYxUgS14SWqWAS9JjTLgJalRBrwkNWpWAZ9ksyRH9tP7Jzl3pnUkSfNjti34zYAjx1EQSdJozfYyyfcDOyRZATwA3JXkLGAXulGdXlVVlWRP4IN0d5z8JXB4Vf1khOWWJM1gti34dwDXVNVuwHHA7sDRwDOA7YHnJVkf+BDwkn4c11OA46fbmGOyStL4zPWHTpdU1Y0Afat+KXArXYv+K0kAlgDTtt6r6mTgZIANttrRYf4kaYTmGvD3DUyvGpc1wMqq2nf6VSRJkzDbLpo7gE1mWOYHwJZJ9gVIsn6SndemcJKktTerFnxV3Zzkm0muBO4BfjbNMvcneQlwUpJN+338H2DlKAosSRrOrLtoquqw1bz+poHpFcDz51AuSdIc+UtWSWqUAS9JjVow94N3TFZJGi1b8JLUKANekhplwEtSowx4SWqUAS9JjTLgJalRBrwkNcqAl6RGGfCS1KhULYxxNpLcQXer4dZsQTdsYYtarVur9YJ269ZqvWDmum1bVVtON2PB3KoA+EFV7TXfhRi1JMtarBe0W7dW6wXt1q3VesHc6mYXjSQ1yoCXpEYtpIA/eb4LMCat1gvarVur9YJ269ZqvWAOdVswJ1klSaO1kFrwkqQRMuAlqVETDfgkByX5QZKrk7xjmvkbJDm9n/+dJEsnWb65GKJuhyf5RZIV/eNP5qOcs5XklCQ/T3LlauYnyUl9vS9Psseky7g2hqjX/kluGzhe75p0GddWkm2S/EuSq5KsTPKWaZZZdMdtyHotyuOWZMMklyS5rK/be6ZZZvb5WFUTeQBLgGuA7YHHApcBz5iyzJHAR/vpQ4HTJ1W+CdTtcODD813Wtajb84E9gCtXM//3gS8DAfYBvjPfZR5RvfYHzp3vcq5l3bYC9uinNwH+bZp/j4vuuA1Zr0V53PrjsHE/vT7wHWCfKcvMOh8n2YLfG7i6qq6tqvuBfwAOmbLMIcCp/fRZwAFJMsEyrq1h6rYoVdWFwC1rWOQQ4LTqXAxslmSryZRu7Q1Rr0Wrqn5SVd/tp+8ArgK2nrLYojtuQ9ZrUeqPw5390/X7x9QrYGadj5MM+K2BGwae38hvHpxHlqmqB4HbgM0nUrq5GaZuAH/Ufx0+K8k2kyna2A1b98Vo3/4r85eT7DzfhVkb/df43elahIMW9XFbQ71gkR63JEuSrAB+DnylqlZ7zIbNx0kG/HSfNFM/oYZZZiEaptz/CCytqmcBX+XRT+LFbrEes5l8l+4eH7sCHwLOnufyzFqSjYHPA0dX1e1TZ0+zyqI4bjPUa9Eet6p6qKp2A54C7J1klymLzPqYTTLgbwQGW61PAW5a3TJJ1gM2ZXF8jZ6xblV1c1Xd1z/9OLDnhMo2bsMc10Wnqm5f9ZW5qr4ErJ9ki3ku1tCSrE8Xgp+tqi9Ms8iiPG4z1WuxHzeAqroVuAA4aMqsWefjJAP+UmDHJNsleSzdSYJzpixzDvCafvolwNeqP6OwwM1Ytyn9my+i6z9swTnAf+uvytgHuK2qfjLfhZqrJE9a1b+ZZG+6/ys3z2+phtOX+5PAVVX1wdUstuiO2zD1WqzHLcmWSTbrpzcCDgS+P2WxWefjxO4mWVUPJnkT8P/orjo5papWJvlzYFlVnUN38D6T5Gq6T6ZDJ1W+uRiybkcleRHwIF3dDp+3As9Cks/RXZmwRZIbgXfTnQCiqj4KfInuioyrgbuB185PSWdniHq9BHhjkgeBe4BDF0ljA+B5wKuBK/o+XYB3Ak+FRX3chqnXYj1uWwGnJllC96F0RlWdO9d89FYFktQof8kqSY0y4CWpUQa8JDXKgJekRhnwktQoA15NSHJn/3dpknuSfK+/6+AlSV4zsNzUu3qeNs22np7kgn7+VUlaHi1IDZvYdfDSBF1TVbsDJNke+EKSx1TVp/r5p1fVm9aw/knAiVX1xX4bz5xrgZIsqaqH5rodaTZswatpVXUt8KfAUbNYbSu6n4Wv2sYV8MjNoE5IckV/07g3968f0H9juCLdfeY36F+/Psm7knwDeGmSHZKcl2R5kouS7DSyikrTMOC1LvguMBimLx/oopnuF5wnAl/r70Z4zKqfkANHANsBu/c3jftskg2BTwMvr6pn0n0rfuPAtu6tqv2q6h/oBk9+c1XtCRwLfGSUlZSmMuC1Lph6F77Tq2q3/vGpqQv3r/0OcCbd7Qwu7lvlB9INuPBgv9wtwNOB66rq3/rVT6UbTOSRfcEjd0B8LnBm/zP7j9F9U5DGxj54rQt2Z5Y3d6uqm4BTgFPSDeu3C90HxTC3uB50V//3McCt/e1gpYmwBa+m9QNDnEB3b/Bh1zmovy0tSZ5EN6jCj4HzgTf0t2olyRPo7vi3NMnT+tVfDXx96jb7+5Zfl+Sl/bpJsutaVksaigGvFu2w6jJJ4AzgQ9N1xazBfwauTHIZ3R1Cj6uqnwKfAH4EXN7PO6yq7qW7E+OZSa4AHgY+uprtvhJ4Xb/uShoZ1lELl3eTlKRG2YKXpEYZ8JLUKANekhplwEtSowx4SWqUAS9JjTLgJalR/x86qmnSRg2J3AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the IDF scores for the very common words\n",
    "plt.barh(top10_words, IDF_weights)\n",
    "plt.title('Inverse Document Frequency');\n",
    "plt.xlabel('IDF Score');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compare the plot with the following plot that shows the words with top 10 highest IDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the terms with the highest IDF score\n",
    "inds = np.argsort(tfidf.idf_)[::-1][:10]\n",
    "top_IDF_tokens = [list(tfidf.vocabulary_)[ind] for ind in inds]\n",
    "top_IDF_scores = tfidf.idf_[inds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAawAAAEWCAYAAAA6maO/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5xd473H8c83Q0OEBFGNFEOkVISQVF1TxUu12urFXVXUaY6eXlSrGr3QOtVW6xxO9UK0mrgcJYoiLdEigrjMRJJJXNskLqHKIROkCUl+54/1TLONPTN7MrNnzZr9fb9e+zVrnvWsZz1rYb6etdZejyICMzOz3q5f3h0wMzOrhAPLzMwKwYFlZmaF4MAyM7NCcGCZmVkhOLDMzKwQHFhmZlYIDizrlSQtlnRw3v1YV5ImS3pD0qvpM1/SjyQNyrtv3UXSAZKe7aBOy3l4reRzdE/10foWB5ZZImm9bm7yJxGxMbAFcBKwF3CvpI26eT+93U8iYmDJ55rWFapw7q0PcmBZrydpvKR7JJ0v6RVJiyR9OK07RlJDq/qnSbopLfdP2z0t6QVJF0vaMK07QNKzkr4p6e/AbyUNkXSLpKWSXpY0U1K/VH8rSb+X9GLqw1cq6X9ErIiIh4CPA5uThReS+kn6jqSnJP1D0uWlIzBJ+0m6L/XlGUnjU/ldkv6t9fkp+T0k/YekJ9Po7j8lDZc0S9IySddKekdJ/Y9KmpP2c5+kXUvWLZZ0uqR5kpolXSNpgxS6fwK2Khk5bVXRP9C3tv1NSfOA1yWt1945lrRhGrG9IukRSd8oHeGl496h5PfJkn7QleMsWX942naZpL9JOlTSkZIaWx3T1yXd2JnzYJVzYFlRvB94HBgC/AT4jSQBNwE7ShpRUvc44H/T8nnAe4DRwA7AMOCskrrvAjYDtgUmAF8HniUbFW0JfAuIFFo3A3NTGwcBX5X0oUoPICJeBW4H9k9F49Png8D2wEDg5wCStiELhItSX0YDcyrdF3AoMIZsVHcGMAk4Htga2AU4Nu1nD+Ay4N/JwvQS4CZJ/UvaOiq1tx2wKzA+Il4HPgw8VzJyeq4T/WtxLHAYMBhYQ/vn+GxgePp8CDix0p2s63GmbfcELge+kfo5DlhM9u/edpLeW9LGZ4ArKu2XdY4Dy4riqYi4NCJWA1OAocCWEbEc+ANr/wCPAHYi+2Mk4PPAaRHxcgqMHwLHlLS7Bjg7IlZGxD+BN1Pb20bEmxExM7IXbr4P2CIizomINyJiIXBpq7Yq8RxZQEIWIP8dEQsj4jXgTOAYZZfHjgf+HBFXp378X0R0JrDOi4hlEbEAmA9MT/tpJgvC3VO9zwOXRMQDEbE6IqYAK8mCrsXPIuK5iHiZLFBGd/KYT0+jmqWSXmq17mcR8Uw69x2d46OAc9M/y2eAn3WiD105zpOByyLi9ohYExFLIuKxiFgJXEMWUkgaCdQDt3SiX9YJDiwrir+3LKSQgmxEAtlo6ti0fBxwY6qzBTAAaGz5gwncmspbvBgRK0p+/ynwV2C6pIWSJqbybckufy0taetbZKOwzhgGvJyWtwKeKln3FLBeanNr4G+dbLvUCyXL/yzze8u52xb4eqvj2jr1rcXfS5aXl2xbqfMjYnD6DGm17pmS5Y7O8Vat6peeu4505Tjb+2cxBTgu/c/RCcC1KcisCnyj0/qC6cAQSaPJguu0VP4S2R/nkRGxpI1t3zJdQRqFfZ3sj9tI4E5JD5H9oVwUESPKtFERSQOBg4FzU9FzZH9IW2wDrCILl2eAPdto6nWyIG7xrnXtU9rPuRFxboc13647pnoobaOjc/w8WXgsSL9v02r9ct5+XlrucXXlOJ8huwz5NhFxv6Q3yC7zHpc+ViUeYVnhRcQq4Dqy0dFmZPeJiIg1ZJeULpD0TgBJw9q775RuzO+Q/o95GbA6fR4ElqWHBDaUVCdpF0nv66h/yh78GAPcCLwC/Datuho4TdJ2Kcx+CFyTjucq4GBJR6WHETZPgQzZvaxPSRqQHjI4uROnq7VLgVMkvV+ZjSQdJmnjCrZ9Adhc3feofkfn+FrgTEmbSno38OVW288hG+3USToU+EDJuq4c52+AkyQdpOxBmWGSdipZfznZvcdVEXFP+SasOziwrK/4X7LRy9T0B7/FN8ku8d0vaRnwZ2DHdtoZkeq8BswCfhkRd6V7Zx8ju6+xiGz09mugvT/WZ0h6lewS4OVAI7BPemABsocArgDuTm2uIP0RjoingY+QjfZeJvtjvFva7gLgDbLAmEIWbuskIhrI7u/8nCxM/0p62KCCbR8jC92F6TJbp54SLNNeR+f4+2SXAReRjapbP9xwatp+Kdk9wH89rdfF43yQ7MnOC4BmYAZvHRlfQfYgix+2qDJ5AkczKyJJBwBXRsS7c+7HhsA/gD0i4sk8+9LXeYRlZtY1XwAeclhVnx+6MDNbR5IWAwI+kXNXaoIvCZqZWSH4kqCZmRWCLwlWyZAhQ6K+vj7vbpiZFUpjY+NLEbFFuXUOrCqpr6+noaGh44pmZvYvktp8g4kvCZqZWSE4sMzMrBAcWGZmVggOLDMzKwQHlpmZFYIDy8zMCsGBZWZmheDAMjOzQvAXh6ukaUkz9ROn5d0NM7MetfjHh1WtbY+wzMysEBxYZmZWCA4sMzMrBAeWmZkVQqECS1K9pPm9oB+TJR2Rdz/MzGpJoQLLzMxqVxEDaz1JUyTNk3SdpAGSxkiaIalR0m2ShgJIGi7p1lQ+U9JOqXyypItT2ROSPprK6yT9VNJDqf1/T+WS9HNJj0iaBrwzt6M3M6tRRfwe1o7AyRFxr6TLgC8CnwQOj4gXJR0NnAt8DpgEnBIRT0p6P/BL4MDUTj3wAWA4cKekHYDPAs0R8T5J/YF7JU0Hdk/7HQVsCTwCXNa6Y5ImABMA6jYpO2GmmZmtoyIG1jMRcW9avhL4FrALcLskgDrgeUkDgX2AqakcoH9JO9dGxBrgSUkLgZ2AQ4BdS+5PDQJGAOOAqyNiNfCcpDvKdSwiJpGFJP2HjojuOFgzM8sUMbBaB8GrwIKI2Lu0UNImwNKIGF1hOwEI+HJE3NaqrY+UqW9mZj2oiPewtpHUEk7HAvcDW7SUSVpf0siIWAYsknRkKpek3UraOVJSP0nDge2Bx4HbgC9IWj9t8x5JGwF3A8eke1xDgQ/2xIGamdlaRQysR4ETJc0DNgMuAo4AzpM0F5hDdikQ4Hjg5FS+ADi8pJ3HgRnAn8juc60Afk12f2p2enz+ErJR6A3Ak0AT8Ku0nZmZ9aBCXRKMiMXAzmVWzSG7z9S6/iLg0DaauzciTmtVfw3ZPbFvlan/pU511szMulURR1hmZlaDCjXC6i4RMT7vPpiZWefUZGD1hFHDBtFQxXlhzMxqjS8JmplZITiwzMysEBxYZmZWCL6HVSVNS5qpnzgt726YmfWoxVW8d+8RlpmZFYIDy8zMCsGBZWZmheDAMjOzQlinwEoz9h7Rcc2K2nqtO9qpJkkHSLol736YmdUyj7DMzKwQKgosSZ+VNE/SXElXpOJxku6TtLB0tCXpG5IeSvW/X1L+GUkPSpoj6RJJda32MUTSLElveyZS0pGS5qf9353Kxku6UdLNkhZJ+pKkr0l6WNL9kjZL9Uan3+dJukHSph2U7yDpz2lfs9N8WQADJV0n6TFJV6lkGmMzM6u+DgNL0kjg28CBEbEbcGpaNRTYD/go8ONU9xCyKeX3BEYDYySNk/Re4Ghg3zQD8Gqyuapa9rElMA04KyLKfXnpLOBDaf8fLynfBTgu7e9cYHlE7A7MAj6b6lwOfDMidiWbz+rsDsqvAn6R9rUP8Hwq3x34Ktn0JtsD+5Y5VxMkNUhqWL28udzpNDOzdVTJF4cPBK6LiJcAIuLlNLi4Mc0f9UgKHIBD0ufh9PtAsgDbFRgDPJS23RD4R6qzPvAX4IsR0dbEiPcCkyVdC1xfUn5nRLwKvCqpGbg5lTcBu0oaBAwuaXcKMLWd8o2BYRFxQzrWFQCpzw9GxLPp9zlAPXBPaScjYhIwCaD/0BHRxrGYmdk6qCSwBJT747uyVZ2Wnz+KiEve0oD0ZWBKRJxZpp1VQCPwIdJMvpLOBQ4DiIjREXGKpPensjmSRpfpw5qS39dUeGyttXeZr3Rfq9exfTMzW0eV3MP6C3CUpM0BWu4NteE24HOSBqa6wyS9M7VxRFpG0maStk3bBPA5YCdJEwEi4tspqEan+sMj4oGIOAt4Cdi6koOLiGbgFUn7p6ITgBntlC8DnpX0ibTf/pIGVLIvMzOrrg5HCRGxII14ZkhazdrLfeXqTk/3q2aly2ivAZ+JiEckfQeYLqkf8CbwReCptN1qSccAN0taFhG/bNX0TyWNIBsB/QWYS3aPrBInAhen4FkInNRB+QnAJZLOSf08ssL9mJlZFSnCt1qqof/QETH0xAvz7oaZWY/q6stvJTVGxNhy6/w9LDMzKwQHlpmZFYIDy8zMCsGPZlfJqGGDaKjiRGZmZrXGIywzMysEB5aZmRWCA8vMzArB97CqpGlJM/UTy73H18ys7+rq97Da4xGWmZkVggPLzMwKwYFlZmaF4MAyM7NCcGBVQJIfTjEzy1lugSWpXtKjki6VtEDSdEkbShou6VZJjZJmStop1d9S0g2S5qbPPqn8M5IelDRH0iWS6tJnsqT5kpoknZbqfkXSI5LmSfpdKttT0n2SHk4/d0zl4yVNlXQz2bQoV0g6vKT/V0n6eI+fODOzGpX3yGEEcGxEfF7StcCnyealOiUinkyzDP8SOBD4Gdkki5+UVAcMTHNvHQ3sGxFvSvolcDywgGyq+10AJA1O+5sIbBcRK0vKHgPGRcQqSQcDP0z9ANgb2DUiXpb0AeA04A+SBgH7kM2pZWZmPSDvwFoUEXPSciNQTxYEU9MEkAD9088Dgc9CNuEj0CzpBGAM8FCqvyHwD+BmYHtJFwHTgOmpjXnAVZJuBG5MZYOAKWmCyADWL+nf7RHxctrnDEm/SLMmfwr4fUSsKj0YSROACQB1m2yxrufEzMzKyDuwVpYsrwa2BJZGRKWzCQuYEhFnvm2FtBvwIbKZjY8CPgccBowDPg58V9JI4D+BO9PIrR64q6SZ11s1ewXZCO6Y1N5bRMQkYBJkEzhWeAxmZlaB3vbQxTJgkaQjAZTZLa37C/CFVF4naZNUdkQa9SBpM0nbShoC9IuI3wPfBfaQ1A/YOiLuBM4ABgMDyUZYS9I+xnfQv8nAVwEiYkE3HK+ZmVWotwUWZCOYkyXNJbsX1fKgw6nAByU1kV0+HBkRjwDfIXsoYh5wOzAUGAbcJWkOWcicCdQBV6btHwYuiIilwE+AH0m6N9VpU0S8ADwK/LYbj9fMzCqgCF+5qpSkAUATsEdENLdXt//QETH0xAt7pmNmZr1EV98lKKkxIsaWW9cbR1i9UnqC8DHgoo7CyszMul/eD10URkT8Gdgm736YmdUqj7DMzKwQPMKqklHDBtFQxXlhzMxqjUdYZmZWCA4sMzMrBAeWmZkVgu9hVUnTkmbqJ07LuxtmZj2qq9/Dao9HWGZmVggOLDMzKwQHlpmZFYIDy8zMCqHPBJakuySVfWFiF9tdnKYrMTOzHPWZwDIzs74tt8CSdIakr6TlCyTdkZYPknSlpEMkzZI0W9JUSQPT+jGSZkhqlHSbpKGt2u0naYqkH6Tf22pnsaTvp/ImSTul8s0lTZf0sKRLyGY1RtJGkqZJmitpvqSje+xkmZlZriOsu4H90/JYYKCk9YH9yOac+g5wcETsATQAX0vrLwKOiIgxwGXAuSVtrgdcBTwREd9Jl/Le1k5J/ZdS+a+A01PZ2cA9EbE7cBNr39B+KPBcROwWEbsAt7Y+IEkTJDVIali93DOQmJl1pzy/ONwIjJG0MbASmE0WXPuTBcXOwL2SAN4BzAJ2BHYBbk/ldcDzJW1eAlwbES0htlcb7bS4vqQvn0rL41qWI2KapFdSeRNwvqTzgFsiYmbrA4qIScAkyCZw7NzpMDOz9uQWWBHxpqTFwEnAfcA84IPAcGARcHtEHFu6jaRRwIKI2LuNZu8DPijpvyJiBdnlvLe1U2Jl+rmat56Lt4VNRDwhaQzwEeBHkqZHxDkVHKqZmXWDvB+6uJvsUtzdwEzgFGAOcD+wr6QdIJuaXtJ7gMeBLSTtncrXlzSypL3fAH8Epkpar512OurT8an+h4FN0/JWwPKIuBI4H9ijqwdvZmaVyzuwZgJDgVkR8QKwApgZES8C44GrJc0jC56dIuIN4AjgPElzycJtn9IGI+K/yS4vXgH8X7l2OujT94FxkmYDhwBPp/JRwIOS5gDfBn7QheM2M7NOUoRvtVRD/6EjYuiJF+bdDTOzHtXVl99KaoyIst+pzXuEZWZmVhEHlpmZFYLnw6qSUcMG0VDFeWHMzGqNR1hmZlYIDiwzMysEB5aZmRWC72FVSdOSZuonTsu7G2ZmPaqrj7W3xyMsMzMrBAeWmZkVggPLzMwKwYFlZmaFUBOBJekuSWPT8h8lDa60fqvy0ZI+Uq1+mplZ22oisEpFxEciYuk6bj6abD4sMzPrYX0usCRtJGmapLmS5ks6utX6xZKGpOXvSnpM0u2SrpZ0eknVIyU9KOkJSftLegdwDnC0pDmt2zUzs+rqi9/DOhR4LiIOA5A0CPhC60rpkt+ngd3JzsNsoLGkynoRsWe6BHh2RBws6SxgbER8qdyOJU0AJgDUbbJFNx6SmZn1uREW0AQcLOk8SftHRHMb9fYD/hAR/4yIV4GbW62/Pv1sBOor2XFETIqIsRExtm7AoHXpu5mZtaHPjbAi4glJY8juNf1I0vQ2qqqDplamn6vpg+fJzKxo+twIS9JWwPKIuBI4H9ijjar3AB+TtIGkgUAl7xN5Fdi4e3pqZmad0ecCCxgFPChpDvBt4AflKkXEQ8BNwFyyy38NQFuXD1vcCezshy7MzHpen7vUFRG3Abe1Kj6gZH19Sfn5EfE9SQOAu4H/SnVK679EuocVES8D76tCt83MrAN9LrA6aZKknYENgCkRMTvvDpmZWXk1HVgRcVzefTAzs8rUdGBV06hhg2io4rwwZma1pi8+dGFmZn2QA8vMzArBgWVmZoXge1hV0rSkmfqJ0/LuhplZj1pcxXv3HmGZmVkhOLDMzKwQHFhmZlYIDiwzMyuEmgwsSfWS5pcpP0fSwXn0yczM2uenBEtExFl598HMzMqryRFWUifpUkkLJE2XtKGkyZKOAJC0WNIPJc2S1CBpD0m3SfqbpFPy7ryZWa2p5cAaAfwiIkYCS4FPl6nzTETsDcwEJgNHAHsB55RrUNKEFG4Nq5d3NLWWmZl1Ri0H1qKImJOWG0lzXrVyU/rZBDwQEa9GxIvACkmDW1eOiEkRMTYixtYNGFSVTpuZ1apaDqyVJcurKX8/r6XOmlb117RR38zMqqSWA8vMzArEgWVmZoVQk5e1ImIxsEvJ7+eXqVNfsjyZ7KGLt60zM7Oe4RGWmZkVggPLzMwKoSYvCfaEUcMG0VDFeWHMzGqNR1hmZlYIDiwzMysEB5aZmRWC72FVSdOSZuonTsu7G2ZmPWpxFe/de4RlZmaF4MAyM7NCcGCZmVkhOLDMzKwQaiqwJH1P0unrsN0Bkm6pRp/MzKwyNRVYZmZWXH06sCR9VtI8SXMlXdFq3WhJ96f1N0jaNJXvIOnPaZvZkoa32u59kh6WtH1PHouZWa3rs4ElaSTwbeDAiNgNOLVVlcuBb0bErkATcHYqvwr4RdpmH+D5kjb3AS4GDo+IhWX2OUFSg6SG1cubu/2YzMxqWZ8NLOBA4LqIeAkgIl5uWSFpEDA4ImakoinAOEkbA8Mi4oa0zYqIWJ7qvBeYBHwsIp4ut8OImBQRYyNibN2AQdU5KjOzGtWXA0tArMM2bXkeWAHsvs49MjOzddaXA+svwFGSNgeQtFnLiohoBl6RtH8qOgGYERHLgGclfSJt01/SgFRnKXAY8ENJB/TQMZiZWdJn3yUYEQsknQvMkLQaeBhYXFLlRODiFEgLgZNS+QnAJZLOAd4Ejixp8wVJHwP+JOlzEfFADxyKmZnRhwMLICKmkN2fKrduDrBXmfInye5/lVoI3JXWPw2M7NaOmplZh/ryJUEzM+tDHFhmZlYIffqSYJ5GDRtEQxXnhTEzqzUeYZmZWSE4sMzMrBAcWGZmVgi+h1UlTUuaqZ84Le9umJn1qMVVvHfvEZaZmRWCA8vMzArBgWVmZoXgwDIzs0Los4El6XuSTi9TfoCkW7rQbpe2NzOzdVPowJLkpxzNzGpErw4sSd+V9Jik2yVdLel0SXdJ+qGkGcCpkj4m6QFJD0v6s6QtS5rYTdIdkp6U9PmS8k0k3SDpEUkXS+qX9neIpFmSZkuaKmlgKj809eMe4FM9dwbMzKxFrx2hSBoLfJpsht/1gNlAY1o9OCI+kOptCuwVESHp34AzgK+neruSTSGyEfCwpJYvRu0J7Aw8BdwKfErSXcB3gIMj4nVJ3wS+JuknwKVkU478FbimekdtZmZt6bWBBewH/CEi/gkg6eaSdaWh8W7gGklDgXcAi0rWtWz/T0l3kgXVUuDBiFiY2r067WsFWYjdK4nU1ixgJ2BRmicLSVcCE8p1WNKElnV1m2yx7kduZmZv05svCaqdda+XLF8E/DwiRgH/DmxQsi5abRftlAu4PSJGp8/OEXFyG/XLiohJETE2IsbWDRhUySZmZlah3hxY9wAfk7RBupfU1vs+BgFL0vKJrdYdnrbfHDgAeCiV7ylpu3Tv6ui0r/uBfSXtACBpgKT3AI8B20kanrY9thuOzczMOqnXBlZEPATcBMwFrgcagOYyVb8HTJU0E3ip1boHgWlkYfSfEfFcKp8F/BiYT3YJ8YaIeBEYD1wtaV7aZqeIWEF2mW9aeujiqe46RjMzq5wiKrralQtJAyPiNUkDgLuBCRExO+9+VaL/0BEx9MQL8+6GmVmP6urLbyU1RsTYcut680MXAJMk7Ux2X2pKUcLKzMy6X68OrIg4Lu8+mJlZ79Br72GZmZmV6tUjrCIbNWwQDVWcyMzMrNZ4hGVmZoXgwDIzs0JwYJmZWSH4HlaVNC1ppn7itI4rmpn1IV39HlZ7PMIyM7NCcGCZmVkhOLDMzKwQHFhmZlYIDqx1IGmypCPy7oeZWS1xYJmZWSEUOrAknSHpK2n5Akl3pOWDJF0p6VhJTZLmSzqvZLvXSpaPkDQ5LU+W9DNJ90la2DKKUubnkh6RNA14Z08ep5mZFTywyObI2j8tjwUGSlof2A94EjgPOBAYDbxP0icqaHNo2v6jZJM8AnwS2BEYBXwe2KfchpImSGqQ1LB6ebm5Js3MbF0VPbAagTGSNgZWks0kPJYsxJYCd0XEixGxCrgKGFdBmzdGxJqIeATYMpWNA66OiNVp1uI7ym0YEZMiYmxEjK0bMKhrR2ZmZm9R6MCKiDeBxcBJwH3ATOCDwHDg6fY2LVneoNW6lSXLamMbMzPrYYUOrORu4PT0cyZwCjAHuB/4gKQhkuqAY4EZaZsXJL1XUj+yy32V7OMYSXWShpKFopmZ9aC+EFgzye47zYqIF4AVwMyIeB44E7gTmAvMjog/pG0mAreQXdp7voJ93EB2T6wJ+BVrg8/MzHqIInylqxr6Dx0RQ0+8MO9umJn1qK6+/FZSY0SMLbeuL4ywzMysBjiwzMysEDwfVpWMGjaIhirOC2NmVms8wjIzs0JwYJmZWSE4sMzMrBB8D6tKmpY0Uz9xWt7dMDPrUV19rL09HmGZmVkhOLDMzKwQHFhmZlYIDiwzMyuEXhVYkgZL+o+0fICkW6q8v3pJ89tYN17SVtXcv5mZVa5XBRYwGPiPvDuRjAccWGZmvURvC6wfA8MlzQF+Sjbl/XWSHpN0lSQBSDpI0sOSmiRdJql/Kl8saUhaHivprrS8haTbJc2WdImkp1rqAXWSLpW0QNJ0SRtKOoJs5uKrJM1JZT+W9IikeZLO7+HzYmZW83pbYE0E/hYRo4FvALsDXwV2BrYH9pW0ATAZODoiRpF9l+wLHbR7NnBHROxBNrfVNiXrRgC/iIiRwFLg0xFxHdAAHJ/6siHZRI8jI2JX4AfldiJpgqQGSQ2rlzd3/ujNzKxNvS2wWnswIp6NiDVkswjXAzsCiyLiiVRnCjCug3b2A34HEBG3Aq+UrFsUEXPScmPaR2vLyCaG/LWkTwHLy+0kIiZFxNiIGFs3YFBHx2ZmZp3Q2wNrZcnyarLRlNqpv4q1x7RBSXl725Tbx1tExCpgT+D3wCeAW9tpz8zMqqC3BdarwMYd1HkMqJe0Q/r9BNZOWb8YGJOWP12yzT3AUQCSDgE27UxfJA0EBkXEH8kuUY6uYHszM+tGvSqwIuL/gHvTo+Y/baPOCuAkYKqkJmANcHFa/X3gfyTNJBstUVJ+iKTZwIeB58kCqT2TgYvTAyAbA7dImkcWjqetw+GZmVkXKCLy7kPVpacIV0fEKkl7A79KD1NUTf+hI2LoiRdWcxdmZr1OV19+K6kxIsaWW1crb2vfBrhWUj/gDeDzOffHzMw6qSYCKyKeJHtE3szMCqomAisPo4YNoqGK88KYmdWaXvXQhZmZWVscWGZmVggOLDMzKwQHlpmZFYIDy8zMCsGBZWZmheDAMjOzQnBgmZlZITiwzMysEGri5bd5kPQq8Hje/eglhgAv5d2JXsLnYi2fi7V8LtbaNiK2KLfCr2aqnsfbeuNwrZHU4HOR8blYy+diLZ+LyviSoJmZFYIDy8zMCsGBVT2T8u5AL+JzsZbPxVo+F2v5XFTAD12YmVkheIRlZmaF4MAyM7NCcGBVgaRDJT0u6a+SJubdn7xI2lrSnZIelbRA0ql59ylvkuokPSzplrz7kidJgyVdJ+mx9O/H3nn3KS+STkv/fcyXdLWkDfLuU2/lwOpmkuqAXwAfBnYGjpW0c769ys0q4OsR8V5gL+CLNXwuWpwKPJp3J3qB/wFujYidgN2o0XMiaRjwFWBsROwC1AHH5Nur3suB1f32BP4aEQsj4g3gd8DhOfcpFxHxfETMTsuvkv1RGpZvr/Ij6d3AYcCv8+5LniRtArBuKK0AAAPKSURBVIwDfgMQEW9ExNJ8e5Wr9YANJa0HDACey7k/vZYDq/sNA54p+f1ZaviPdAtJ9cDuwAP59iRXFwJnAGvy7kjOtgdeBH6bLo/+WtJGeXcqDxGxBDgfeBp4HmiOiOn59qr3cmB1P5Upq+nvDkgaCPwe+GpELMu7P3mQ9FHgHxHRmHdfeoH1gD2AX0XE7sDrQE3e65W0KdkVmO2ArYCNJH0m3171Xg6s7vcssHXJ7++mhof4ktYnC6urIuL6vPuTo32Bj0taTHaZ+EBJV+bbpdw8CzwbES2j7evIAqwWHQwsiogXI+JN4Hpgn5z71Gs5sLrfQ8AISdtJegfZDdSbcu5TLiSJ7D7FoxHx33n3J08RcWZEvDsi6sn+nbgjImry/6Qj4u/AM5J2TEUHAY/k2KU8PQ3sJWlA+u/lIGr0AZRK+G3t3SwiVkn6EnAb2RM/l0XEgpy7lZd9gROAJklzUtm3IuKPOfbJeocvA1el/6lbCJyUc39yEREPSLoOmE32VO3D+DVNbfKrmczMrBB8SdDMzArBgWVmZoXgwDIzs0JwYJmZWSE4sMzMrBAcWGa9lKTX0s96Sf9MrzF6VNKDkk4sqTde0ouS5qTP5WXa2lHSXWn9o5L86LQVjr+HZVYMf0uvMULS9sD1kvpFxG/T+msi4kvtbP8z4IKI+ENqY1RXOySpLiJWd7Uds0p5hGVWMBGxEPga2bQUlRpK9kqkljaa4F/zc50vqUnSPElfTuUHpRFdk6TLJPVP5YslnSXpHuBIScMl3SqpUdJMSTt124GateLAMium2UBpOBxdckmw3FsjLgDukPSnNGHg4FQ+gezFq7tHxK5kb5/YAJgMHB0Ro8iuxHyhpK0VEbFfRPyO7K0MX46IMcDpwC+78yDNSjmwzIqp9awA10TE6PT5bevKqey9wFTgAOD+NGo6GLg4Ilalei8DO5K9kPWJtPkUsvmr/rUv+Ndb+PcBpqZXb11CNpIzqwrfwzIrpt3p5EtSI+I54DLgMknzgV3Igq/1+9nKTZFT6vX0sx+wNCJGd6YfZuvKIyyzgkmTYZ4PXNSJbQ5NU70g6V3A5sASYDpwSprtFkmbAY8B9ZJ2SJufAMxo3Waa22yRpCPTtpK02zoellmHHFhmxTC85bF24FrgonKX/tpxCDBf0lyymQS+kab5+DXZFBfz0rrjImIF2dvTp0pqIpsh+eI22j0eODltu4BsMkKzqvDb2s3MrBA8wjIzs0JwYJmZWSE4sMzMrBAcWGZmVggOLDMzKwQHlpmZFYIDy8zMCuH/AaHX4pnB9rhJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the terms with the highest IDF score\n",
    "plt.barh(top_IDF_tokens, top_IDF_scores)\n",
    "plt.title('Inverse Document Frequency');\n",
    "plt.xlabel('IDF Score');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, comparing across the two plots, the frequently appearing words have much lower values for their IDF scores, as compared to much rarer words. This is basically by definition, for the IDF score. \n",
    "\n",
    "What this means for TF-IDF is that the weighting helps account for which words in a document are specific to that document. Because the TF and IDF values are multiplied, rare terms get a higher TF-IDF score, per occurrence, than common words, which helps to compare across terms and documents. Ultimately, this allows us to represent a document by distribution of terms that are most unique to the particular document, as compared to the average across the corpus. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Applying TF-IDF\n",
    "\n",
    "Now that we have learned the frequencies, we can apply this representation to our data. \n",
    "\n",
    "In the next line, we will apply the TF-IDF representation to our data, and convert this to an array. \n",
    "\n",
    "This array is an `n_documents` x `n_terms` matrix that encodes the documents in a TFIDF representation.\n",
    "\n",
    "Note that in our TFIDF object above we set the number of features to use as 5000, which is the number of terms. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply TF/IDF representation to our data\n",
    "tfidf_books = tdidf.transform(sents).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents: \t\t 10000\n",
      "Number of terms: \t\t 5000\n",
      "TFIDF representation shape: \t (10000, 5000)\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of documents: \\t\\t\", len(sents))\n",
    "print(\"Number of terms: \\t\\t\", len(tfidf.vocabulary_))\n",
    "print(\"TFIDF representation shape: \\t\", tfidf_books.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the TFIDF array, each row stores a representation of the document based on the TF-IDF score of our 5000 terms. \n",
    "\n",
    "This is a new representation of our text data, a numerical one, that we could now use for analysis and comparison of text data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Text analysis and NLP is itself a huge field, and an area in which one can take whole classes or research programs. \n",
    "\n",
    "Here, we have introduced some of the core idea related to NLP. For more information on these topics, look into NLP focused resources and classes."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}