Twitter-Sentiment-Analysis-Jupyter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About\n",
    "\n",
    "---\n",
    "\n",
    "Twitter Sentiment Analysis project using Python, NLTK, Twitter API interface and Nick Sander's tweets corpus to evaluate the sentiment of Netflix users about it's platform services.\n",
    "\n",
    "Developed by Jaques D'Erasmo S. Fernando as part of his final project on his cerification course through **University of California - San Diego at Edx Platform** in ***Python for Data Science***.\n",
    "\n",
    "\n",
    "\n",
    "Contact --> jaques.ufba@gmail.com\n",
    "\n",
    "Portfolio --> https://github.com/Jaquesd/Portifolio\n",
    "\n",
    "Verified Certificate link --> https://courses.edx.org/certificates/2a8732fa24024af5be308fe4ee77d8a2\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Acknowledgment**\n",
    "\n",
    "Thank you ***Felipe Solares*** (https://github.com/fsolares) and ***Eduardo Passos*** (https://github.com/eduardosteps), old friends and also Data Science students (12/19/2019) for your immeasurable contribution on this project, all your feedback, code sujection and support during my path gave me the strength to overcome this challenge. Congratualation for us, that was an ammazing and real team work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis using Python,  Twitter API and NLTK Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Project purpose**\n",
    "\n",
    "Perform a sentiment analysis using NLTK Naive Bayes Classifer to identify the sentiment from tweets users related with Netflix services (or any other high tech company as Apple, Microsoft, Google, for example).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Importing libraries\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "README!\n",
    "\n",
    "It is recommended that you install the most recent version of Anaconda distribution.\n",
    "\n",
    "Additionally, few other packaged will need to be installed separately in case you do not have those installed. Bellow you can find each package and the installation command that can be executed from jupyter notebook cell except for the worldcloud package that need to be executed from the Anaconda Terminal (Windows) or the Terminal (Linux or MAC)\n",
    "\n",
    "* tweepy - ***!pip install tweepy***\n",
    "* textblob - ***!pip install textblob***\n",
    "* worldcloud - ***conda install -c https://conda.anaconda.org/conda-forge wordcloud***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install tweepy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install textblob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     /home/jaques/nltk_data...\n",
      "[nltk_data]   Package movie_reviews is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /home/jaques/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /home/jaques/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "#General Libraries\n",
    "\n",
    "import pickle\n",
    "import os\n",
    "import tweepy #tweepy is a libraries that work similar as 'twitter' used during the course. I decided \n",
    "              #to use thatsince I previuosly new about that and I feel more confortble. It is important\n",
    "              #to mention that the principles behind both libriries are the same\n",
    "              \n",
    "        \n",
    "import pandas as pd\n",
    "import nltk\n",
    "from textblob import TextBlob #it is necessary install the textblob library if you don't have it yet! \n",
    "                              #This is a veryuseful library that will be used during this work in two ways, \n",
    "                              #as a translator for tweets in different languages and in a simplefied \n",
    "                              #sentiment analysis that will be aslo presented\n",
    "            \n",
    "            \n",
    "\n",
    "# if you don't have the wordcloud library instaled, one option, if using Anaconda, you can install this \n",
    "# package with the command bellow (type exactly as it is bellow on your Anaconda Prompt):\n",
    "# conda install -c conda-forge wordcloud\n",
    "\n",
    "\n",
    "# Visualization Libraries\n",
    "\n",
    "from operator import itemgetter\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "# Preprocess Libraries\n",
    "\n",
    "import re #module used to work with regular expressions\n",
    "from nltk.tokenize import word_tokenize\n",
    "from string import punctuation\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('movie_reviews')\n",
    "nltk.download(\"punkt\")\n",
    "nltk.download(\"stopwords\")\n",
    "\n",
    "# Image visualization\n",
    "\n",
    "from IPython.display import Image\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Saving my credentials and creating the OAuth object to access tweets on Twitter\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This strucre will save your credentials at the first time you exetute it, after that, erase your\n",
    "# credentials to avoind anyone else to use that\n",
    "if not os.path.exists('secret_twitter_credentials.pkl'):\n",
    "    Twitter={}\n",
    "    Twitter['Consumer Key'] = '<type your credential>'\n",
    "    Twitter['Consumer Secret'] = '<type your credential>'\n",
    "    Twitter['Access Token'] = '<type your credential>'\n",
    "    Twitter['Access Token Secret'] = '<type your credential>'\n",
    "    with open('secret_twitter_credentials.pkl','wb') as f:\n",
    "        pickle.dump(Twitter, f)\n",
    "else:\n",
    "    Twitter=pickle.load(open('secret_twitter_credentials.pkl','rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticating and creating an API object\n",
    "\n",
    "auth = tweepy.OAuthHandler(Twitter['Consumer Key'], Twitter['Consumer Secret'])\n",
    "auth.set_access_token(Twitter['Access Token'], Twitter['Access Token Secret'])\n",
    "api = tweepy.API(auth, wait_on_rate_limit=True,wait_on_rate_limit_notify=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Getting tweets as test data\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that will access the Tweet and get tweets based on a search term, number of tweets, inicial \n",
    "# date and final date. This function will return a list of dictionaries where the key is the tweet text \n",
    "# and the values is a label, in this casa we are considering 'None' in the labor. This format is necessary\n",
    "# for futures aplication\n",
    "\n",
    "def get_tweets(word='google',number=100, since='2019-11-01', until='2019-11-28'):\n",
    "\n",
    "    search_term = str(input('Hi! What are you looking for today? - '))\n",
    "    num_of_terms = int(input('How many tweets do you want? - '))\n",
    "    since = input('From which date? ex.: 2019-11-01 - ')\n",
    "    until = input('Until which date? ex.: 2019-11-28 - ')\n",
    "\n",
    "    data = []\n",
    "\n",
    "    tweets = tweepy.Cursor(api.search,\n",
    "                                #since = f'{since},\n",
    "                                #until = f'{until},\n",
    "                                q=search_term,\n",
    "                                lang='en').items(num_of_terms)                                \n",
    "                                \n",
    "    for tweet in tweets:\n",
    "        try:\n",
    "            data.append(tweet.extended_tweet.full_text)\n",
    "        except:\n",
    "            data.append(tweet.text)\n",
    "\n",
    "    \n",
    "    new_data = set(data) #converting in a set to remove the duplicates\n",
    "\n",
    "    new_data_dic = []\n",
    "\n",
    "    for i in new_data:\n",
    "        new_data_dic.append({'text':i, 'label':None}) #this strucure as a dictionary is necessary for  \n",
    "                                                      #further purposes\n",
    "\n",
    "    print()\n",
    "\n",
    "    #Will print a message to the user \n",
    "    print('Great Job! We got '+str(len(data))+' tweets with the term '+word+'!!')\n",
    "\n",
    "\n",
    "    return [i for i in new_data_dic]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi! What are you looking for today? - Netflix\n",
      "How many tweets do you want? - 2000\n",
      "From which date? ex.: 2019-11-01 - 2019-12-18\n",
      "Until which date? ex.: 2019-11-28 - 2019-12-19\n",
      "\n",
      "Great Job! We got 2000 tweets with the term google!!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'text': '@sarahwittman bussy netflix success', 'label': None},\n",
       " {'text': 'RT @notviking: therapy:\\n\\n- expensive\\n- might not work\\n\\nwatching hot rod (2007) on netflix:\\n\\n- free\\n- will always make you feel better since…',\n",
       "  'label': None},\n",
       " {'text': 'Netflix has bs scary movies.', 'label': None}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating a tweets objecting and printing the first 4 tweets\n",
    "tweets = get_tweets()\n",
    "tweets[:3]\n",
    "\n",
    "\n",
    "# IMPORTANT!!!\n",
    "\n",
    "# The Twitter API have limitations on the total of tweets and how many requests can be made.\n",
    "# A recomendation is do not request more than 3000 tweets in an interval of 15 min to avoid long waiting\n",
    "# time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1179"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the final lengh of the request we can see the total is not the 1000 tweets requested. The \n",
    "# reason for that is that the tweet api return duplicates and the above function do the job of removing\n",
    "# those.\n",
    "\n",
    "len(tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Getting training data\n",
    "---\n",
    "\n",
    "We'll use the Niek Sander's tweets corpus woth ~5000 classified tweets labelled as positive, negative, neutral or irrelevant. Those tweets are related with tech companies like, Apple, Google, Twitter, Youtube, Microsoft and others. Said that, the model that will be created is recommended to a search term related with tech companies. The performance lower if the sentiment analysis is related with another kind of topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Getting Nick Sander's tweets corpus!**\n",
    "\n",
    "* access the link https://github.com/zfz/twitter_corpus;\n",
    "* download the file;\n",
    "* we will use the **full-corpus.csv** file. This file already contain the tweets texts and it respective labels of *positive*, *negative*, *neutral* or *irrelevant*.\n",
    "\n",
    "\n",
    "After download the file it is recommend that you explore it using Pandas for a better comprehension "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic</th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>TweetId</th>\n",
       "      <th>TweetDate</th>\n",
       "      <th>TweetText</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>apple</td>\n",
       "      <td>positive</td>\n",
       "      <td>126415614616154112</td>\n",
       "      <td>Tue Oct 18 21:53:25 +0000 2011</td>\n",
       "      <td>Now all @Apple has to do is get swype on the i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>apple</td>\n",
       "      <td>positive</td>\n",
       "      <td>126404574230740992</td>\n",
       "      <td>Tue Oct 18 21:09:33 +0000 2011</td>\n",
       "      <td>@Apple will be adding more carrier support to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>apple</td>\n",
       "      <td>positive</td>\n",
       "      <td>126402758403305474</td>\n",
       "      <td>Tue Oct 18 21:02:20 +0000 2011</td>\n",
       "      <td>Hilarious @youtube video - guy does a duet wit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>apple</td>\n",
       "      <td>positive</td>\n",
       "      <td>126397179614068736</td>\n",
       "      <td>Tue Oct 18 20:40:10 +0000 2011</td>\n",
       "      <td>@RIM you made it too easy for me to switch to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>apple</td>\n",
       "      <td>positive</td>\n",
       "      <td>126395626979196928</td>\n",
       "      <td>Tue Oct 18 20:34:00 +0000 2011</td>\n",
       "      <td>I just realized that the reason I got into twi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Topic Sentiment             TweetId                       TweetDate  \\\n",
       "0  apple  positive  126415614616154112  Tue Oct 18 21:53:25 +0000 2011   \n",
       "1  apple  positive  126404574230740992  Tue Oct 18 21:09:33 +0000 2011   \n",
       "2  apple  positive  126402758403305474  Tue Oct 18 21:02:20 +0000 2011   \n",
       "3  apple  positive  126397179614068736  Tue Oct 18 20:40:10 +0000 2011   \n",
       "4  apple  positive  126395626979196928  Tue Oct 18 20:34:00 +0000 2011   \n",
       "\n",
       "                                           TweetText  \n",
       "0  Now all @Apple has to do is get swype on the i...  \n",
       "1  @Apple will be adding more carrier support to ...  \n",
       "2  Hilarious @youtube video - guy does a duet wit...  \n",
       "3  @RIM you made it too easy for me to switch to ...  \n",
       "4  I just realized that the reason I got into twi...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Function to read the nick_sanders_corpus csv file and return a list containing the text and the label\n",
    "# for each tweet\n",
    "\n",
    "# reading the nick_sanders_corpus\n",
    "df = pd.read_csv('full_corpus.csv')\n",
    "\n",
    "#function\n",
    "def trainingData():\n",
    "    trainingData = [{'text':row[4], 'label':row[1]} for index,row in df.iterrows()]\n",
    "    return trainingData\n",
    "\n",
    "# The function return a list of dict containing the key (tweets text) and the values (the labels)\n",
    "\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'text': 'Now all @Apple has to do is get swype on the iphone and it will be crack. Iphone that is',\n",
       "  'label': 'positive'},\n",
       " {'text': '@Apple will be adding more carrier support to the iPhone 4S (just announced)',\n",
       "  'label': 'positive'},\n",
       " {'text': \"Hilarious @youtube video - guy does a duet with @apple 's Siri. Pretty much sums up the love affair! http://t.co/8ExbnQjY\",\n",
       "  'label': 'positive'}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_Data = trainingData()\n",
    "training_Data[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Preprocessing tweets from test and training data\n",
    "---\n",
    "\n",
    "The preprocess step will use few python tools to work with strings as detailed bellow:\n",
    "\n",
    "        USING .LOWER() STRING FUNCTION\n",
    "        \n",
    "            1 - Convert to lower case\n",
    "        \n",
    "        USING REGULAR EXPRESSIONS\n",
    "            2 - Replace links with the string 'url'\n",
    "            3 - Replace @ ... with 'at_user'\n",
    "            4 - Replace #word with the word itself\n",
    "            5 - Remove emoticons using ASCII encode and decode\n",
    "        \n",
    "        USING NLTK\n",
    "            7 - Tokenize the tweet into words (a list of words)\n",
    "            8 - Remove stopwords (including url and user, RT and '...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a class to preprocess the test and training tweets\n",
    "\n",
    "class Preprocess:\n",
    "    def __init__(self):\n",
    "        \n",
    "        lst = ['AT_USER','URL','rt','...', \"'s\", \"n't\", \"``\", \"''\"]\n",
    "        self._stopwords = set(stopwords.words('english')+list(punctuation)+lst)\n",
    "        \n",
    "        \n",
    "    def processTweets(self, tweets):\n",
    "        #tweets is a list of dict with Keys, 'text' and 'label'\n",
    "        processedTweets = []\n",
    "        #this list will be a list of tuple. Each tuple is a tweet which is a list of words and its label\n",
    "        \n",
    "        for tweet in tweets:\n",
    "            processedTweets.append((self.cleanTweet(tweet['text']),tweet['label']))\n",
    "                                         #it will apply the cleanTweet function only to the tweets text\n",
    "        return processedTweets\n",
    "    \n",
    "    \n",
    "    def processTweets_words(self, tweets):\n",
    "        \n",
    "        processedTweets = []\n",
    "        #this list will be a list of each word in all tweets\n",
    "        \n",
    "        for tweet in tweets:\n",
    "            processedTweets.append(self.cleanTweet(tweet['text']))\n",
    "                                         #it will apply the cleanTweet function only to the tweets text\n",
    "        return processedTweets\n",
    "            \n",
    "        \n",
    "    def cleanTweet(self,tweet):\n",
    "        #1 - Convert to lower case\n",
    "        tweet = tweet.lower()\n",
    "                \n",
    "        #2 - Replace links with word 'URL'\n",
    "        tweet = re.sub('((www\\.[^\\s]+)|(https?://[^\\s]+))','URL',tweet)\n",
    "        \n",
    "        #3 - Replace @username with 'AT_USER'\n",
    "        tweet = re.sub('@[^\\s]+','AT_USER',tweet)\n",
    "        \n",
    "        #4 - Repalce #word (hashtag) with just the word, witouth the '#' simble\n",
    "        tweet=re.sub(r'#([^\\s]+)',r'\\1',tweet)\n",
    "        \n",
    "        #5 - Remove emoticons\n",
    "        tweet = tweet.encode('ascii', 'ignore').decode('ascii')\n",
    "        \n",
    "        #6 - Tokenizing the tweets\n",
    "        tweet = word_tokenize(tweet)\n",
    "        \n",
    "        #7 - Removing stopwords \n",
    "        return [word for word in tweet if word not in self._stopwords]        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiating a Preprocess Class called 'tprocessor'\n",
    "tprocessor = Preprocess()\n",
    "\n",
    "# Creating an object that will contain the result of the TRAINING DATA after been preprocessed by the\n",
    "# 'tprocessor' class\n",
    "cleanedtrainingData = tprocessor.processTweets(training_Data)\n",
    "\n",
    "# Creating an object that will contain the result of the TEST DATA after been preprocessed by the\n",
    "# 'tprocessor' class\n",
    "cleanedtestData = tprocessor.processTweets(tweets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['get', 'swype', 'iphone', 'crack', 'iphone'], 'positive'), (['adding', 'carrier', 'support', 'iphone', '4s', 'announced'], 'positive'), (['hilarious', 'video', 'guy', 'duet', 'siri', 'pretty', 'much', 'sums', 'love', 'affair'], 'positive'), (['made', 'easy', 'switch', 'iphone', 'see', 'ya'], 'positive'), (['realized', 'reason', 'got', 'twitter', 'ios5', 'thanks'], 'positive'), ([\"'m\", 'current', 'user', 'little', 'bit', 'disappointed', 'move'], 'positive'), (['16', 'strangest', 'things', 'siri', 'said', 'far', 'sooo', 'glad', 'gave', 'siri', 'sense', 'humor', 'via'], 'positive'), (['great', 'close', 'personal', 'event', 'tonight', 'regent', 'st', 'store'], 'positive'), (['companies', 'experience', 'best', 'customer', 'service', 'aside'], 'positive'), (['apply', 'job', 'hope', 'call', 'lol'], 'positive')]\n"
     ]
    }
   ],
   "source": [
    "print(cleanedtrainingData[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['bussy', 'netflix', 'success'], None), (['therapy', 'expensive', 'might', 'work', 'watching', 'hot', 'rod', '2007', 'netflix', 'free', 'always', 'make', 'feel', 'better', 'since'], None), (['netflix', 'bs', 'scary', 'movies'], None), (['nooooo', 'daybreak', 'one', 'favorite', 'shows', 'year', 'mad', 'max', 'set', 'john', 'hughes', 'high', 'school', 'even', 'matth'], None), (['dont', 'watch', 'dont', 'fuck', 'cats', 'netflix', 'alone', '.i', 'repeat', 'dont', 'watch', 'fucking', 'bizarre', 'terrifying'], None), (['ok', 'im', 'dead', 'serious', 'us', 'planning', 'day', 'cancel', 'netflix', 'accounts', 'bc', 'wont', 'even', 'consider'], None), (['+as', 'christian', 'religion', 'nakakabastos', 'lang', 'nakakadisappoint', 'amp', 'even', 'funny', 'inopen', 'nila', 'yung', 'bible'], None), (['ryan', 'murphy', 'best', 'ever', 'thank', 'never', 'ageist'], None), (['since', 'tf', 'netflix', '16.95'], None), (['burlesque', 'netflix'], None)]\n"
     ]
    }
   ],
   "source": [
    "print(cleanedtestData[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.1 - Word Frequence Visualizations\n",
    "\n",
    "* Word Frequence using a bar plot\n",
    "* Word Frequence using a Word Cloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bussy', 'netflix', 'success', 'therapy', 'expensive']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the result of the 'processTweets_words' is a list fo lists, so it will be necessary unify all words in\n",
    "# only one list\n",
    "words = tprocessor.processTweets_words(tweets)\n",
    "\n",
    "unif_words =[]\n",
    "for i in words:\n",
    "    for j in i:\n",
    "        unif_words.append(j)\n",
    "        \n",
    "unif_words[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word frequence using 'Counter' class from the 'collection' package\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "word_counter = Counter(unif_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_common_words = word_counter.most_common()[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('netflix', 853), ('watch', 102), ('dont', 87), ('cats', 81), ('jesus', 79), ('show', 79), ('watching', 77), ('like', 63), ('trailer', 61), ('im', 56), ('season', 55), ('anne', 50), ('good', 49), ('movie', 48), ('series', 47)]\n"
     ]
    }
   ],
   "source": [
    "print(most_common_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEaCAYAAAAL7cBuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XXWd//HX52Zt1i5p032j0I2ypq1g0aqooKyyFkcUkeL8RHSYGQd0XMZR3EYdEBSLICBLYVgLVFHQUJaWFihSliKUbukC3dOmLWmTz++Pc1Ju03uzNScnN/f9fDzyaO73LPeTb27vO+d8z/0ec3dERESaS8RdgIiIdE8KCBERSUkBISIiKSkgREQkJQWEiIikpIAQEZGUFBASGzP7npndHncdIpKaAkIAMLOrzGxus7Y307Sd3wX1TDezRjPbkfT1cNTPm6nMbIWZnRh3HU0U/j1DbtwFSLcxD7jSzHLcvcHMBgJ5wDHN2saE67aZmRlg7t7YzprWuvvQNuw/1933tnPfItIKHUFIk0UEgXBU+PhDwN+AN5q1LXP3tQBmdryZLTKzbeG/xzftzMyqzeyHZvYMsBMYbWajzOxJM9tuZn8BKjpSaPjX6b1mdruZ1QJfMLOEmV1pZsvMbJOZ3WNmfZO2+ZyZrQyXfSv5L24zu8XMfpC07nQzq0l6PNjM7jOzDWa23Mwub1bLPWZ2W/hzvWpmVUnLh5nZ/eG2m8zsuqRlXzSz181si5k9ZmYjOtIfbeivS8zsLTPbbGZzzGxw0rJPmNkb4e/w1+Hv50tp9jPXzH6e9PhuM7u5A/WMD18fW8P+Oi1pWT8ze9jMasPX1A/M7On2Pod0DgWEAODu9cBzBCFA+O9TwNPN2uYBhG++jwLXAv2AXwCPmlm/pN1+DpgJlAIrgTuBFwiC4b+Bzx9EyacD9wK9gTuAy4EzgA8Dg4EtwPVhrROA34T1DA7rbfXIJNw2ATwM/B0YAnwM+LqZfTJptdOA2WEtc4Drwm1zgEcIfvaR4fazw2VnAN8EPgP0J+jru9rZB22p/6PAj4BzgUFhLU01VBD04VUEffIGcHzqPQHwReBzZvZRM/ssMBn4WjvrySPozz8DA4CvAneY2dhwleuBOmAgwevjYF4jcrDcXV/6wt0Bvgc8EH7/d+BQ4KRmbZ8Pv/8csLDZ9vOBL4TfVwPfT1o2HNgLFCe13QncnqaW6UAjsDXp69ykOuc1W/914GNJjwcBewhOo34HmJ20rBioB04MH98C/KDZc9eE308FVjV7rquA3yfV8njSsgnArvD744ANQG6Kn++PwMVJjxMER1ojOvi7W9H08zRrvwn4adLjkrBfRgIXAvOTlhmwGvhSC8/zmXCdjcC0Vl5LB/xugROA9UAiqe2ucP2csLaxSct+ADwd9/+NbP3SEYQkmwdMM7M+QH93fxN4Fjg+bDuc98cfBhP8NZpsJcFfyU1WJ30/GNji7nXN1m/JWnfvnfR1T5p9A4wAHghPW2wlCIwGoDJ87n3rhzVsauW5k/c7uGm/4b6/Ge63yfqk73cChWaWCwwDVnrq8ZERwDVJ+9xM8AY9pPmKZnZD0kD9N9tYd5P9fk/uvoPgZx/Cgf3iQE3zHTTzCMEb+Rvu3pFTP4OB1b7/eFTT66Y/QaAn/26b/56lCykgJNl8oJzgtNAzAO5eC6wN29a6+/Jw3bUEb3LJhgNrkh4nTxW8DuhjZsXN1u+o5tMQrwZObhYohe6+JnzuYU0rmlkRwSmVJnVAUdLjgc32u7zZfkvd/VNtqHE1MDwMi1TLLm22317u/uwBP6j7l929JPy6ug3Pm2y/31PY//0Ifk/rSDrVFl5M0Nqptx8ShO8gM5vRzlqa6hkWnrpr0vS62UBwlJlcwzAkNgoI2cfddwHPA1cQnBNv8nTYlnz10lzgMDO7wMxyzew8gtMrj6TZ98pw3/9lZvlmNg04tRPLvwH4YdNAr5n1N7PTw2X3AqeY2TQzywe+z/6v/ZeAT5lZXwuu1Pp60rKFQK2Z/YeZ9TKzHDM73Mwmt6GmhQRvwj82s2IzKzSzDybVe5WZTQzrLTezczr4szfJC5+j6SuX4DTeRWZ2lJkVAFcDz7n7CoIxpElmdka47lfYPxz3Y2YfAi4iODV1IfArMzvgiCdJolk9BQTjXHXAN8wsz8ymE7wOZrt7A3A/8D0zKzKzceHzSEwUENLckwSDh8mnD54K2/YFhLtvAk4B/pXglMU3gFPcfWML+76A4Jz+ZuC7wG2dWPc1BAPEfzaz7cCC8Llw91cJ3vzuJHjD3sL+p1L+QDC+soJg8PTupgXhm9apBFdyLSc49/47giOtFiVtOwZYFT7neeGyB4CfALMtuBLrFeDkjvzgSeYCu5K+vufuTwDfBu4j+NkPAc4Pa9gInAP8lOB3OIEgxN9rvmMzKyP4fV3m7mvC00s3Ab8PjzxSmdGsnmUeXAxxWvizbgR+DVzo7kvDbS4j6Nv1BL+Xu1LVI13DgtOOItnFzFYQDMY+Hnct3UV42qcG+Ky7/y3uegDM7CfAQHfX1Uwx0BGESBYzs0+aWe/w9M83CQbKF8RYzzgzO8ICU4CLgQfiqifb6ZPUItntOIJTb/nAa8AZ4VhUXEoJTisNBt4Ffg48FGM9WU2nmEREJCWdYhIRkZQUECIiklJGj0FUVFT4yJEjO7RtXV0dxcXFra8o7aa+jYb6NRrZ2K8vvPDCRnfv39p63SYgzGw8wcRfFcAT7v6b1rYZOXIkzz//fIeer7q6munTp3doW2mZ+jYa6tdoZGO/mllr09wAEZ9iMrObzexdM3ulWftJ4RTDb5nZlQDu/rq7f5lg1smqVPsTEZGuE/UYxC0Es4HuE06BfD3BJyknADPC6ZgJ54V/Gngi4rpERKQVkQaEu88jmFYh2RTgLXd/O/zY/WyCuf1x9znufjzw2SjrEhGR1sUxBjGE/afwrQGmhpN2fQYoIJhTJiUzm0kwsyiVlZVUV1d3qIgdO3Z0eFtpmfo2GurXaKhf04sjIFJN7OXuXk1wk5kWufssYBZAVVWVd3RwKRsHprqK+jYa6tdoqF/Ti+NzEDXsP8f7UII54kVEpBuJIyAWAYdacAP7fIKph+e0ZwdmdqqZzdq2bVskBYqISPSXud5FcJeysWZWY2YXh7dfvAx4jODOVPeE8/W3mbs/7O4zy8tbnZJfREQ6KNIxCHdPeUtCd59LCwPRIiISv4yci0mnmEREopeRAaFTTCIi0cvIgBARkegpIEREJKWMDAiNQYiIRC8jA0JjECIi0cvIgBARkegpIEREJCUFhIiIpJSRAaFBahGR6GVkQGiQWkQkehkZECIiEj0FhIiIpKSAEBGRlDIyIDRILSISvYwMCA1Si4hELyMDQkREoqeAEBGRlBQQIiKSkgJCRERSUkCIiEhKGRkQusxVRCR6GRkQusxVRCR6GRkQIiISPQWEiIikpIAQEZGUFBAiIpKSAkJERFJSQIiISEoZGRD6HISISPQyMiD0OQgRkehlZECIiEj0FBAiIpKSAkJERFJSQIiISEoKCBERSUkBISIiKSkgREQkJQWEiIikpIAQEZGUFBAiIpJSRgaE5mISEYleRgaE5mISEYleRgaEiIhETwEhIiIpKSBERCQlBYSIiKSkgBARkZQUECIikpICQkREUlJAiIhISgoIERFJSQEhIiIpKSBERCQlBYSIiKSkgBARkZQUECIikpICQkREUupWAWFmZ5jZjWb2kJl9Iu56RESyWW7UT2BmNwOnAO+6++FJ7ScB1wA5wO/c/cfu/iDwoJn1Af4H+HNL+97T0Mi6bbvSPzeWdlndHm/PjyEiknUiDwjgFuA64LamBjPLAa4HPg7UAIvMbI67vxau8p/h8hYtXb+d43701w4X9t+LnmDcoFLGDSxjfPjv6P7F5OV0qwMrEZFYRB4Q7j7PzEY2a54CvOXubwOY2WzgdDN7Hfgx8Ed3fzHV/sxsJjAToHflUC6amJ/6eVupa0vde2yo38uytZt46h8baAg3yDEYXJJgWGnwNbTEGFaaoLzAMEt/RCLv27FjB9XV1XGX0eOoX6Ohfk2vK44gUhkCrE56XANMBb4KnAiUm9kYd7+h+YbuPguYBVBVVeXf/dzHO1RAdXU106dPB4JTVW9vqGPp+lpeX7edpetrWbpuO8+u3b1v/b7F+YwbGBxljBtUyviBZRxaWUJhXk6Hnr8nS+5b6Tzq12ioX9OLKyBS/Snu7n4tcG1XF5OXk2DswFLGDizl9KPeb99SV8/S9e8HxtL1tdy5cCW79zQCkDAYVVHMuEFljE8KjyG9e+loQ0QyXlwBUQMMS3o8FFjb1o3N7FTg1DFjxnR2XfvpU5zPcYf047hD+u1ra2h0Vm6qC4NjO0vX1bKkZhuPvrxu3zqlBbmMHVjKuEGljK0sZXi/Ykb2K2Jw714a3xCRjBFXQCwCDjWzUcAa4HzggrZu7O4PAw9XVVVdElF9aeUkjNH9Sxjdv4RPTRq0r33He3t5o9nRxkOL17L9vb37bTukdy9G9CsKvvoWh98XM7xvEb3ydbpKRLqPrrjM9S5gOlBhZjXAd939JjO7DHiM4DLXm9391ahriVJJQS7HjujDsSP67Gtzd96pfY+Vm+pYuWknKzcH/67avJM5L62ldvfe/fZRWVbAiL7FDO9XxMh+RQzrW0TvonzKe+XRu1ce5b3yKOuVR05Cp69EJHpdcRXTjDTtc4G5UT9/nMyMgeWFDCwvZOrofgcs37qzPgyOnazcWMfKzTtZtWknT725gXtfeC/tfksLcykPA6N3UfBvUX5uC5/6gIQZ/UsLGFheyKCwpsHlvehdlKfxEhFJKa5TTAelq8Ygota7KJ/eRfkcOaz3Act21TewZusutu2qZ+vOPWzbFXw1fV+7aw9bw7Z3anew8729KZ7hfXsbnU119TQ07n8BcEFugsMqS7nm/KMY3b+kU38+EclsGRkQcY5BdJVe+TmMGdC5b9gNjc7GHe+xbttu1m/bxdqtu1lfu5v7Xqjhghuf4+5LP8CIfsWd+pwikrkyMiCkY3ISRmVZIZVlhZB01PKZY4YwY9YCZsxawN2XHsewvkUxViki3YWuuRTGDSzj9i9Npa6+gRk3LqBmy864SxKRbiAjA8LMTjWzWdu2bYu7lB5j4uBy7vjSVGp37eGCG59rcRJEEckOGRkQ7v6wu88sLy+Pu5Qe5fAh5fzh4qlsqatnxqwFvFO7u/WNRKTH0hiE7OfIYb259eIpXHjTQj728yc5fEgZEwaVM3FwGRMGlzFmQIk+DS6SJRQQcoBjhvdh9swPcOfCVby2dv/5pypKCvi3TxzGOVXD9IE9kR4uIwOip3wOojs7fEg5V585CQguj12+cQevrq3ltvkrufL+Jdw6fyXfPmU8xx9SEXOlIhKVjAyIbPgcRHeSkzDGDChlzIBSTjtyMI+8vI4f/3EpF9z4HCeOH8AJh/ZneN8ihvcrYkjvXnGXKyKdJCMDQuJjZpx65GA+PqGSm55ezm+fXMbjr7+7b3l+ToKzDs3lw+6awkMkwykgpEMK83L4ykfG8P+mH8KGHe+xenMwCeEjf1/HXUvfZftdi/nJWUdQXKCXmEim0uUoclDMjAGlhRw7oi9nHj2UGy+s4uzD8pi7ZB1nXP8ML9dsPWD+JxHJDPrzTjpVImGcMjqfM044hstnL+a0656hV14O4weVMnFwcLnsxMHlul2rSAbIyIDQVUzd37RDK/jT10/gyTc28OraWl5bW8sDi9fwhwUrAcjLMQ4fUs7kkX2pCu+j0a+kIOaqRSRZRgaErmLKDANKCzmnahjnhI8bG51Vm3fy6tpaXl6zlRdWbOGWZ1Ywa97bAIzuX8xxo/sxbUwF4weVkTCjrFcuvYvy4/shRLJYRgaEZKZEwhhZUczIimI+fURwu9bdexpYsmYbz6/YwqIVm3lw8RrueG7Vvm0K8xLcPfO4lPfMEJFoKSAkVoV5OUwe2ZfJI/vyzxzCnoZGXlq9lVWbduLAT/+0lG8+sISHvvJBcjXFh0iXUkBIt5KXk9gXGAC98nL4yp0vcuv8lVw8bVTM1YlkF/1JJt3apyYNZPrY/vziz2+wapPuUyHSlTIyIHQ/iOxhZvz36YeTkzC+cMtCttTVx12SSNZo0ykmM7uipeXu/ovOKadtdBVTdhnWt4jffX4y/3TTc1x86yLOPnYYBbkJThxfSXlRXtzlifRYbR2DqAImA3PCx6cC84DVURQl0tyUUX353/OO4muzF/Piqq1AMD7xiYmVFOXnckj/YmZMGa6pPUQ6UVv/N1UAx7j7dgAz+x7wf+7+pagKE2nuU5MGMe3QCnbVN/BO7W7+MH8l897cQEOjc9eOen5dvYyfnX0EHxtfGXepIj1CWwNiOJB88rceGNnp1Yi0oqwwj7LCPCrLCvnZOe9/NuLFVVv49oOv8M93vMjNn5/MMSN6U5SvowmRg9HW/0F/ABaa2QOAA2cCt0VWlUg7HTO8D7dfPJVzfzuff7rpOQBmTBnG1WdO0rTjIh3UpoBw9x+a2R+BE8Kmi9x9cXRlibRfn+J87pr5Aea8tJbX1tVy18LVDCrvxQmHBne9G9a3iArN9yTSZu05Bi8Cat3992bW38xGufvyqAoT6YiKkgK+OG0UjY1O7a49/OIv/+AXf/kHACUFufz4rEmccsTgmKsUyQxtvcz1uwRXMo0Ffg/kAbcDH4yutBbr0Wyu0qJEwrjugmNYuHwzexobaWx0rv/bW1x252Kee3szJx0+kGNH9NGU4yItaOsRxJnA0cCLAO6+1sxKI6uqFfochLRFfm6CaeHpJYAPHdafn/5pKTc+tZw/LFjJlJF9ufvSD2iMQiSNtn6Sut7dnWCAGjMrjq4kkWjk5ST41qcn8PgVH+KKjx/GwhWbmbtkfdxliXRbbQ2Ie8zst0BvM7sEeBy4MbqyRKIzZkApX/nIGMYNLOW7c17hzXe2x12SSLfUpoBw9/8B7gXuIxiH+I67/yrKwkSilBOOUZgZM25coJAQSaHVMQgzywEec/cTgb9EX5JI1xgzoITZMz/AjFkL+PSvnqY0nKajuCCX/znnSKaM6htzhSLxajUg3L3BzHaaWbm7a/pU6VEO6V/CPZcex63zV7CnoRGAp97cyJduXcRRw/vQpyiPb58yQZ+fkKzU1quYdgNLzOwvQF1To7tfHklVIl1oZEUx3z114r7HKzbW8Z05r1K7aw8L3t7Emi27+Mi4AQCccGgFRwzV7U8lO7Q1IB4Nv0R6vJEVxdz2xSkA3PP8ar71wBKeX7kFgN9UL+PWL06mpCCPAaUF9CnOj7NUkUi1GBBmNtzdV7n7rV1VkEh3cm7VMM48egiN7qzZsotPX/s0Z/1mPgClBbnM/doJDOtbFHOVItFo7QjiQeAYADO7z93Pir4kke4lLye42G90/xIevXwaS9dvp35vI1fe/zJn3/As/YqD8YmchPGtT4/nA6P7xVmuSKdpLSCSP2I6OspCRDLB6P4ljO5fAkBBboL7Xlyzb9niVVv4r4df46IPjgTgmOG9GTMgtgkHRA5aawHhab6PleZiku7g5EmDOHnSoH2P73l+Nd+492W+ce/LAAzvW0T1v00nkdBUHpKZWvug3JFmVmtm24Ejwu9rzWy7mdV2RYGpuPvD7j6zvLw8rhJEDnDOsUN57psf4+n/+AhXnzmJVZt38qlrn2JvePmsSKZp8QjC3TXVpUgbmRmVZYUAnFM1lD+9up55/9jAjU8tZ+zA4LRURUmBLpOVjKF7MopEIC8nwU2fr2LaT/7KT/60dF+7Gcz794/oyifJCAoIkYjk5SR4+KvTWLd1NwCbd9Zz0e8X8a//93dGhAHxsfGVnHT4wDjLFElLASESoQGlhQwoLdz3+PSjBrNo+WZqNu9k2649PLtsE5+cWKl7Uki3pIAQ6ULXnH/0vu9vX7CS/3zwFT768ycxC64p//dPjuWkwwel34FIF2rr/SBEpJOdcsQgzjl2KBMHlzFhUBmb6+q55/mauMsS2UdHECIx6V2Uz8/OOXLf4yvve5m7n1/NhO/8id698njk8hPoq7meJEYKCJFu4tIPH0J5rzw21dVz7ws1LFy+WQPYEisFhEg3MaqimKs+NZ7dexp4cPEavjZ7MQW5wVngRMI4f0yC6fGWKFlGASHSzRTm5XD1Zybx2tr3Jyt48KU1vLRhb4xVSTZSQIh0Q+dWDdvv8cpNdSxYtoFzbniWc44dxrmTh6XZUqTz6ComkQzw2akjGFWeYNmGOu5YuCruciRL6AhCJAOcOKGS3Hd78cTWCh5YvIbb5q8AoLKskE9O1EC2REMBIZJBjhnRmz8sWMl3Hnp1X9vCb36MAWWFLWwl0jEKCJEMcubRQ/no2Er2NjYy/+1NXHbnYlZt3qmAkEh0m4Aws9HAt4Bydz877npEuqvyojwAxg0sA+DLt79Ar/z3Z+YvKcjj9oun0K+kIJb6pOeIdJDazG42s3fN7JVm7SeZ2Rtm9paZXQng7m+7+8VR1iPSk4yuKObSD4/mQ4f2Z/KIvkwe0ZdD+pfw+rpaXlsX2/28pAeJ+gjiFuA64LamBjPLAa4HPg7UAIvMbI67vxZxLSI9SiJhXHXy+P3aVmysY/ob1SzfWMfEwfvfcbF3rzzd/lTaJdKAcPd5ZjayWfMU4C13fxvAzGYDpwMKCJGDVFlWSMLgOw+9ut9ANsAFU4dz9ZmTYqpMMlEcYxBDgNVJj2uAqWbWD/ghcLSZXeXuP0q1sZnNBGYCVFZWUl1d3aEiduzY0eFtpWXq22i0tV8vP7qAjbt8v7bHV+3h+X/UUF29KaLqMpder+nFERCpjnHd3TcBX25tY3efBcwCqKqq8unTp3eoiOrqajq6rbRMfRuNtvZrqjW23PEiS9fX6veSgl6v6cUREDVA8jwBQ4G1MdQhkjX6leSzbEMdo656dF9bUV4OD132QcYMKI2xMunO4giIRcChZjYKWAOcD1zQnh2Y2anAqWPGjImgPJGe56IPjqJ3UT54cOppY109dz63irferVNASFqRBoSZ3UVwxFthZjXAd939JjO7DHgMyAFudvdXW9jNAdz9YeDhqqqqSzq7ZpGeaFRFMVd8/LB9j1dv3smdz62idveeGKuS7i7qq5hmpGmfC8yN8rlFJL2yXsGH7R56aQ3L3t2Rdr0hfXpx4XEju6gq6W66zSep20OnmEQOTmlBLuMHlfHCyi28sHJLynX2Njh7G53TjhwcnJ6SrJORAaFTTCIHJ5Ew/vi1E1pcZ/bCVVx5/xJ21jfQu6iLCpNuRfeDEJGUmuZ32lnfEHMlEpeMPIIQkegV5QdvDz+a+/q+CQKbGMYFU4dx7Ii+cZQmXSQjA0JjECLRGzewlMMqS3jjne0HLFu3bTc5CRQQPVxGBoTGIESiN6xvEX/+lw+nXDb9Z3/jvb2NXVyRdDWNQYhIuxXk5vDeHgVET5eRRxAiEq+CvATbdu1h7dZdadfJSRgDSgsw0xTjmUoBISLtVlKQy7PLNnH8j//a4no/PesIzp08rMV1pPvKyIDQILVIvL5/+uG8sHJz2uXucOX9S1i3bXcXViWdLSMDQoPUIvEaM6CEMQNKWlznqgeWsLdR4xSZTIPUIhKJvJwEexq89RWl21JAiEgk8hLGngYdQWSyjDzFJCLdX15ugjl/X8viVaknA4RgVtnrLjiGkgK9FXVHGXkEYWanmtmsbdu2xV2KiKRx4XEjGTewlOKC3JRfu/c0Uv3GhhanG5d4ZWRsa5BapPtLvkFRKvP+sYELb15IvU5DdVsZeQQhIpmvIDd4+6nXlB3dlgJCRGKRr4Do9hQQIhKLpoDQpH/dV0aOQYhI5ms6xfT6ulr6Fqe/penIiiIGlBZ2VVmSRAEhIrEo6xXchOiaJ97kmifeTLvexMFlPHp5y7dHlWhkZEBoLiaRzDegtJC5l5/Alp31ade54cllLN9Y14VVSbKMDAhd5irSM0wYXNbi8odeWsNb+pxEbDRILSLdVk5C8znFSQEhIt1WbsJo0IywsVFAiEi3lZtj7G3UEURcFBAi0m0FRxAKiLgoIESk28pJJNirMYjYZORVTCKSHXITRn1DIydf81SbtzmssoRrzj86wqqyR0YGhD4HIZIdPjlxIG+9u4MGb9tRxLINO3jopbX873lHYWYRV9fzZWRA6HMQItlh0tBybvjcsW1e/5rH3+SXj/+DRocc5cNB0xiEiPQYOeE7mga2O4cCQkR6jJxE8JamgOgcCggR6TH2HUG0ccxCWqaAEJEeIxEOTOsIonMoIESkx8hJKCA6kwJCRHqMXAVEp1JAiEiPkQgDolFjEJ1CASEiPUZOOAahCf46hwJCRHqMpjGIRgVEp8jIT1KLiKSSF17netL/ztt3uqk1e/fuJbf6sbTLL/vIGC798CGdUl+myciA0FxMIpLKhw7rz5c/fAjv7W1o8zY1NTUMHTo05bL7X1zDy2u2dVZ5GScjA0JzMYlIKn2L87ny5HHt2qa6egPTp09MuezpNzdCFp+t0hiEiEgaZtl9RZQCQkQkjYQZWZwPCggRkZboCEJERA6QMMvmIQgFhIhIOmbgOoIQEZHmEmZk82fuFBAiImnoCEJERFIy0BGEiIgcyDRILSIiqegUk4iIpKQPyomISErBGET2JoQCQkQkDR1BiIhIapqsT0REUklYVs/23X3uB2FmxcCvgXqg2t3viLkkEclyhuHeGHcZsYn0CMLMbjazd83slWbtJ5nZG2b2lpldGTZ/BrjX3S8BTouyLhGRtkgkyOoxiKiPIG4BrgNua2owsxzgeuDjQA2wyMzmAEOBJeFqbb9foIhIRAzj7Y11XHH3S3GXcoALpg6namTfSJ8j0oBw93lmNrJZ8xTgLXd/G8DMZgOnE4TFUOAlWjiyMbOZwEyAyspKqqurO1Tbjh07OryttEx9Gw31azRa6tdBiXreaNzLU0vXdm1RbTCEjexYEe3f+HGMQQwBVic9rgGmAtcC15nZp4GH023s7rOAWQBVVVU+ffr0DhVRXV1NR7eVlqlvo6F+jUZL/Zrt3R1HQFiKNnf3OuCiri5GRETCcn5PAAAGWUlEQVRSi+My1xpgWNLjoUC7jt/M7FQzm7Vt27ZOLUxERN4XR0AsAg41s1Fmlg+cD8xpzw7c/WF3n1leXh5JgSIiEv1lrncB84GxZlZjZhe7+17gMuAx4HXgHnd/Nco6RESk/aK+imlGmva5wNwon1tERA5ORk61oTEIEZHoZWRAaAxCRCR6GRkQIiISPcvk2+mZ2QZgJVAOND/f1Lyt+eMKYGOkBaZ+3qi2bcu66dZpT3tb2rqibw+mX9uz/cH0a7pl3blf09USxbatrdvefk3XrveCA5f3dvf+re7J3TP+C5jVWluKx8/HVVsU27Zl3XTrtKe9jX0ded8eTL+2Z/uD6deD6cO4+vVg+7YzX7Pt7dd29KP6tY376imnmFJNzdG8Le30HRE7mOdtz7ZtWTfdOu1pb2tb1A72Odu6/cH0a7pl3blfD/Z5O/M1295+Tdeu94IO7iujTzEdDDN73t2r4q6jJ1LfRkP9Gg31a3o95QiiI2bFXUAPpr6Nhvo1GurXNLL2CEJERFqWzUcQIiLSAgWEiIikpIAQEZGUFBAhMys2s1vN7EYz+2zc9fQUZjbazG4ys3vjrqWnMbMzwtfrQ2b2ibjr6SnMbLyZ3WBm95rZP8ddT5x6dECY2c1m9q6ZvdKs/SQze8PM3jKzK8PmzwD3uvslwGldXmwGaU+/uvvb7n5xPJVmnnb27YPh6/ULwHkxlJsx2tmvr7v7l4Fzgay+/LVHBwRwC3BScoOZ5QDXAycDE4AZZjaB4M52TffKbujCGjPRLbS9X6V9bqH9ffuf4XJJ7xba0a9mdhrwNPBE15bZvfTogHD3ecDmZs1TgLfCv2zrgdnA6QS3Qh0artOj++VgtbNfpR3a07cW+AnwR3d/satrzSTtfc26+xx3Px7I6tPN2fhGOIT3jxQgCIYhwP3AWWb2G+L7KH4mS9mvZtbPzG4Ajjazq+IpLeOle81+FTgRONvMvhxHYRku3Wt2uplda2a/JctvbBbpHeW6KUvR5u5eB1zU1cX0IOn6dROgN6+Dk65vrwWu7epiepB0/VoNVHdtKd1TNh5B1ADDkh4PBdbGVEtPon6Njvo2GurXVmRjQCwCDjWzUWaWD5wPzIm5pp5A/Rod9W001K+t6NEBYWZ3AfOBsWZWY2YXu/te4DLgMeB14B53fzXOOjON+jU66ttoqF87RpP1iYhISj36CEJERDpOASEiIikpIEREJCUFhIiIpKSAEBGRlBQQIiKSkgJCsoaZ/dLMvp70+DEz+13S45+b2RUHsf/vmdm/pWlfY2YvmdlrZjbjIJ5jupk90tHtRdpDASHZ5FngeAAzSwAVwMSk5ccDz7RlR+FU0e3xS3c/imC20N+aWV47txfpcgoIySbPEAYEQTC8Amw3sz5mVgCMBxaH02j/zMxeMbMlZnYe7Pvr/W9mdiewJGz7VnjDmceBsa0V4O5vAjuBPuH2l5jZIjP7u5ndZ2ZFYfst4Yyiz5rZ22Z2dvN9mdlkM1tsZqMPtmNEUsnG2VwlS7n7WjPba2bDCYJiPsGUz8cB24CX3b3ezM4CjgKOJDjKWGRm88LdTAEOd/flZnYswfw9RxP8X3oReKGlGszsGOBNd383bLrf3W8Ml/0AuBj4VbhsEDANGEcwR9C9Sfs5PlzvdHdf1dE+EWmJAkKyTdNRxPHALwgC4niCgHg2XGcacJe7NwDvmNmTwGSgFljo7svD9U4AHnD3nQBm1tJEb/9iZpcAo9n/zmaHh8HQGyghmBeoyYPu3gi8ZmaVSe3jgVnAJ9xds49KZHSKSbJN0zjEJIJTTAsIjiCSxx9S3SegSV2zx22dzOyX7j6W4N7Rt5lZYdh+C3CZu08C/gsoTNrmvaTvk2taB+wmOHIRiYwCQrLNM8ApwGZ3b3D3zQR/vR9HcMoJYB5wnpnlmFl/4EPAwhT7mgecaWa9zKwUOLW1J3f3+4Hngc+HTaXAunDQuq23t9wKfBq42symt3EbkXZTQEi2WUIwrrCgWds2d98YPn4AeBn4O/BX4Bvuvr75jsL7QN8NvATcBzzVxhq+D1wRXkn1beA54C/A0rb+EO7+DkEgXW9mU9u6nUh7aLpvERFJSUcQIiKSkgJCRERSUkCIiEhKCggREUlJASEiIikpIEREJCUFhIiIpKSAEBGRlP4/P4PYf17WAjUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Word distribution present a large vocabulary\n",
    "sorted_word_counts = sorted(list(word_counter.values()), reverse=True)\n",
    "\n",
    "plt.loglog(sorted_word_counts)\n",
    "plt.ylabel(\"Freq\")\n",
    "plt.xlabel(\"Word Rank\")\n",
    "plt.title('Word Frequence - Log x Log')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGTlJREFUeJzt3Xm0ZWV9p/HnCwgoRcCBVKBAC1YhEWE5XYMQkhSRCIilxrYFwjJiE4hZIUq0l4KmDaY1wdXBCWm1NIZWkZKAyGjjWOKAMjgBEhqUQgqQUQsLMQT89R97Xzhc9h3qUrvOPXWfz1pn1dnvnn7nvbvu9+7h7J2qQpKkiTYZdgGSpLnJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyIDSnJDkhyaeGXYdmJsnhSb4wMFxJlgyzJq0/BoQmleT4JBdOaLtukrZDN0A9S5P8Jsnagdd5fa9Xk6uq06rqRcOuQ/3YbNgFaE67GDguyaZV9WCS3wEeBzx3QtuSdtoZSxIgVfWbdazplqracQbL36yqHljHZUsa4B6EpnIZTSA8ux3+Q+CrwLUT2n5cVbcAJNknyWVJ1rT/7jO+sCQrk7wryTeBXwG7JNk5ydeS/DLJF4GnzKbQ9tDUmUk+leQe4IgkmyQ5LsmPk9yV5IwkTxqY59VJbmzHvS3JqiT7t+NOTfLOgWmXJlk9MLxDkrOS3JHkhiSvn1DLGUk+0X6uq5OMDYzfKcln23nvSvLBgXH/Lck1SX6e5KIkT5tNf7TLekuSm9sark3ywrZ90n5Jsrg9TPTaJDe1dbwuyfOT/DDJLybUe0SSb0yy/m3aPrij7ee/S7JJO27TJCclubPtv2Pa9fpH6xxiQGhSVXU/8B2aEKD99+vANya0XQzQ/pK5APgA8GTgPcAFSZ48sNhXA0cDWwM3Ap8GrqAJhv8JvOYxlPwy4ExgW+A04PXAy4E/AnYAfg6c0ta6O/Chtp4d2nqn3TNp590EOA/4AbAIeCFwbJIDBiZ7KbCireVc4IPtvJsC59N89sXt/CvacS8H3gq8AtiOpq9PX8c+GK9xN+AY4PlVtTVwALCqHT1pvwzYC9gVOAR4H/A2YH/gmcCrkvzRDMo4GdgG2KVd158Dr23HHQUcRPOHxnPbejTXVJUvX5O+gBOAs9v3P6D5pXHghLbXtO9fDVw6Yf5LgCPa9yuBfxgY91TgAWCrgbZPA5+apJalwG+AXwy8XjVQ58UTpr8GeOHA8PbAf9IcWn07sGJg3FbA/cD+7fCpwDsnrHt1+34v4KcT1nU88K8DtXxpYNzuwH3t+72BO4DNOj7f54EjB4Y3odnTetosfm5LgNtpfqk/bh36ZTFQwKKB8XcBhwwMnwUc274/AvjGwLhq170p8B/A7gPj/hJY2b7/CvCXA+P2b+d9VL/4Gt7LPQhN52Jg3yRPBLarquuAbwH7tG178PD5hx1o/jIedCPNX8njbhp4vwPw86q6d8L0U7mlqrYdeJ0xybIBngac3R4W+QXNL8YHgYXtuh+avq3hrmnWPbjcHcaX2y77re1yx/1s4P2vgC3bwyc7ATdW9/mRpwHvH1jm3UB4ZP8BkOTDAyfq3zpxfFVdDxxLE1a3J1mRZIeB9UzWL+NuG3h/X8fwgo76Bz0F2JxH/jwHt4VH9D+P/tlpDjAgNJ1LaA4THA18E6Cq7gFuadtuqaob2mlvofnlM+ipwM0Dw4O3D74VeGKSrSZMP1sTb018E3DQhEDZsqpubte90/iESZ5Ac5hp3L3AEwaGf2fCcm+YsNytq+rFM6jxJuCpkxxrv4nmr+rB5T6+qr71qA9a9bqqWtC+/rFrRVX16aral+ZnUsC7B9YzWb+sL3fS7JUMbg+D28KtPPKQ3k5ozjEgNKWqug+4HHgjzTHxcd9o2wavXroQeHqSP0uyWZJDaA6vnD/Jsm9sl/2OJJsn2RdYth7L/zDwrvETvUm2S/KydtyZwEuS7Jtkc+AfeOT/h+8DL07ypDRXah07MO5S4J72JPDj2xOueyR5/gxqupTml+OJSbZKsmWS3x+o9/gkz2zr3SbJf53NB0+yW5I/TrIF8Guav/ofHFjPZP2yXlTVg8AZ7Xq2btf1RmD8Oy5nAG9IsijJtsBb1uf6tX4YEJqJrwG/TRMK477etj0UEFV1F/AS4E00h2veDLykqu6cYtl/RnNM/27g74FPrMe6309zgvgLSX4JfLtdF1V1NfDXNOc8bqU5Ubt6YN5P0pxfWQV8AfjM+Ij2l98ymhOsN9D8tfwxmj2tKQ3MuwT4abvOQ9pxZ9P8lb8izZVYV9GcyJ2NLYAT29p+RvOzGj8UNWm/rGd/Q7Mn9hOabefTwMfbcR+l6dcfAt+j+ePiAR4OMc0BqfKBQRJAklXAX1TVl4Zdy3yT5CDgw1U168t6tf65ByFpg2sPzb24PRS5iGbv8exh16VHMiAkDUOAd9Ac2vsezZVUbx9qRXoUDzFJkjq5ByFJ6jSS9z1JsgxYtvXWWx/19Kc/fVbLuPfee9lqq62mn3Aesm+mZv9Mzr6Z2lzpnyuuuOLOqtpuuulG+hDT2NhYXX755bOad+XKlSxdunT9FrSRsG+mZv9Mzr6Z2lzpnyRXVNXYdNN5iEmS1MmAkCR1MiAkSZ0MCElSp5EMiCTLkixfs2bNsEuRpI3WSAZEVZ1XVUdvs82090aTJM3SSAaEJKl/BoQkqdNIfpN6fbjy5jUccdwFj2pfdeLBQ6hGkuYe9yAkSZ0MCElSp5EMCC9zlaT+jWRAeJmrJPVvJANCktQ/A0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdRjIg/Ca1JPVvJAPCb1JLUv9GMiAkSf0zICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUqc5ExBJnpHkw0nOTPJXw65Hkua7XgMiyceT3J7kqgntBya5Nsn1SY4DqKprqup1wKuAsT7rkiRNr+89iFOBAwcbkmwKnAIcBOwOHJZk93bcS4FvAF/uuS5J0jRSVf2uIFkMnF9Ve7TDewMnVNUB7fDxAFX1TwPzXFBVB0+yvKOBowEWLlz4vBUrVsyqrtvvXsNt9z26fc9F3kJ87dq1LFiwYNhlzFn2z+Tsm6nNlf7Zb7/9rqiqaY/UbLYhiplgEXDTwPBqYK8kS4FXAFsAF042c1UtB5YDjI2N1dKlS2dVxMmnncNJVz764686fHbL25isXLmS2fbrfGD/TM6+mdqo9c8wAiIdbVVVK4GVG7YUSdJkhnEV02pgp4HhHYFb1mUBPnJUkvo3jIC4DNg1yc5JNgcOBc5dlwX4yFFJ6l/fl7meDlwC7JZkdZIjq+oB4BjgIuAa4IyqurrPOiRJ667XcxBVddgk7RcyxYno6SRZBixbsmTJbBchSZrGnPkm9brwEJMk9W8kA0KS1L+RDAivYpKk/o1kQHiISZL6N5IBIUnqnwEhSeo0kgHhOQhJ6t9IBoTnICSpfyMZEJKk/hkQkqROIxkQnoOQpP6NZEB4DkKS+jeSASFJ6p8BIUnqZEBIkjoZEJKkTiMZEF7FJEn9G8mA8ComSerfSAaEJKl/BoQkqZMBIUnqZEBIkjoZEJKkTiMZEF7mKkn9G8mA8DJXSerfSAaEJKl/BoQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6jWRA+E1qSerfSAaE36SWpP6NZEBIkvpnQEiSOhkQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE5zKiCSvDzJR5Ock+RFw65Hkuaz3gMiyceT3J7kqgntBya5Nsn1SY4DqKrPVdVRwBHAIX3XJkma3IbYgzgVOHCwIcmmwCnAQcDuwGFJdh+Y5O/a8ZKkIek9IKrqYuDuCc2/B1xfVT+pqvuBFcDL0ng38Pmq+m7ftUmSJrfZkNa7CLhpYHg1sBfwN8D+wDZJllTVhyfOmORo4GiAhQsXsnLlylkVsPDx8KY9H3hU+2yXtzFZu3at/TAF+2dy9s3URq1/hhUQ6WirqvoA8IGpZqyq5cBygLGxsVq6dOmsCjj5tHM46cpHf/xVh89ueRuTlStXMtt+nQ/sn8nZN1Mbtf4Z1lVMq4GdBoZ3BG6Z6cw+clSS+jesgLgM2DXJzkk2Bw4Fzp3pzD5yVJL6tyEucz0duATYLcnqJEdW1QPAMcBFwDXAGVV1dd+1SJJmbkbnIJK8carxVfWeKcYdNkn7hcCFM1l/Rz3LgGVLliyZzeySpBmY6R7EGPBXNFcfLQJeR/P9ha3b1wblISZJ6t9Mr2J6CvDcqvolQJITgH+rqr/oqzBJ0nDNdA/iqcD9A8P3A4vXezUz5FVMktS/mQbEJ4FLk5yQ5O+B7wCf6K+sqXmISZL6N6NDTFX1riSfB/6gbXptVX2vv7IkScO2Lpe5PgG4p6reD6xOsnNPNUmS5oAZBUR7WOktwPFt0+OAT/VV1Azq8RyEJPVspnsQfwq8FLgXoKpuYQiXt47zHIQk9W+mAXF/VRVQAEm26q8kSdJcMNOAOCPJR4BtkxwFfAn4aH9lSZKGbaZXMf1zkj8B7gF2A95eVV/stTJJ0lBNGxDt40Evqqr9gTkRCt6LSZL6N+0hpqp6EPhVkjlzRtiT1JLUv5nei+nXwJVJvkh7JRNAVb2+l6okSUM304C4oH1JkuaJKQMiyVOr6qdV9X82VEGSpLlhunMQnxt/k+SsnmuZMb9JLUn9m+4QUwbe79JnIeuiqs4DzhsbGztqfS978XHdR9JWnXjw+l6VJM1p0+1B1CTvJUkbuen2IJ6V5B6aPYnHt+9ph6uqfqvX6iRJQzNlQFTVphuqEEnS3LIuz4OQJM0jBoQkqdNIBoSXuUpS/0YyILwXkyT1byQDQpLUPwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnUYyIPwmtST1byQDwm9SS1L/RjIgJEn9MyAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnTYbdgGjYvFxF0w6btWJB2/ASiRpw3APQpLUac4ERJJdkvxLkjOHXYskqeeASPLxJLcnuWpC+4FJrk1yfZLjAKrqJ1V1ZJ/1SJJmru89iFOBAwcbkmwKnAIcBOwOHJZk957rkCSto1RVvytIFgPnV9Ue7fDewAlVdUA7fDxAVf1TO3xmVb1yiuUdDRwNsHDhwuetWLFiVnXdfvcabrtvVrM+yp6LNq7nUqxdu5YFCxYMu4w5y/6ZnH0ztbnSP/vtt98VVTU23XTDuIppEXDTwPBqYK8kTwbeBTwnyfHjgTFRVS0HlgOMjY3V0qVLZ1XEyaedw0lXrp+Pv+rw2dUwV61cuZLZ9ut8YP9Mzr6Z2qj1zzACIh1tVVV3Aa+b0QKSZcCyJUuWrNfCJEkPG8ZVTKuBnQaGdwRuWZcF+MhRSerfMALiMmDXJDsn2Rw4FDh3CHVIkqbQ92WupwOXALslWZ3kyKp6ADgGuAi4Bjijqq5ex+UuS7J8zZo1679oSRLQ8zmIqjpskvYLgQsfw3LPA84bGxs7arbLkCRNbc58k1qSNLeMZEB4iEmS+jeSAeFVTJLUv5EMCElS/wwISVKnkQwIz0FIUv9GMiA8ByFJ/RvJgJAk9c+AkCR1MiAkSZ1GMiA8SS1J/RvJgPAktST1byQDQpLUPwNCktTJgJAkdRrGM6kfs7n2TOrFx12wTtOvOvHgniqRpPVnJPcgPEktSf0byYCQJPXPgJAkdTIgJEmdDAhJUicDQpLUaSQDwnsxSVL/RjIgvMxVkvo3kgEhSeqfASFJ6mRASJI6GRCSpE4GhCSpkwEhSepkQEiSOhkQkqROPjBoCCZ7wJAPEpI0l4zkHoTfpJak/o1kQEiS+mdASJI6GRCSpE4GhCSpkwEhSepkQEiSOhkQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaEJKnTnLmba5KtgP8N3A+srKrThlySJM1rve5BJPl4ktuTXDWh/cAk1ya5PslxbfMrgDOr6ijgpX3WJUmaXt+HmE4FDhxsSLIpcApwELA7cFiS3YEdgZvayR7suS5J0jRSVf2uIFkMnF9Ve7TDewMnVNUB7fDx7aSrgZ9X1flJVlTVoZMs72jgaICFCxc+b8WKFbOq6/a713DbfbOatTd7Lpobz7dYu3YtCxYsGHYZc5b9Mzn7Zmqz6Z8rb17T2f5Yfl/st99+V1TV2HTTDeMcxCIe3lOAJhj2Aj4AfDDJwcB5k81cVcuB5QBjY2O1dOnSWRVx8mnncNKVc+YUDACrDl867BIAWLlyJbPt1/nA/pmcfTO12fTPEZM9gXID/L4Yxm/IdLRVVd0LvHZGCxjxR45K0igYxmWuq4GdBoZ3BG5ZlwX4yFFJ6t8wAuIyYNckOyfZHDgUOHcIdUiSptD3Za6nA5cAuyVZneTIqnoAOAa4CLgGOKOqrl7H5S5LsnzNmu6TN5Kkx67XcxBVddgk7RcCFz6G5Z4HnDc2NnbUbJchSZqat9qQJHUyICRJnUYyIDwHIUn96/2b1H1Kcgdw4yxnfwpw53osZ2Ni30zN/pmcfTO1udI/T6uq7aabaKQD4rFIcvlMvmo+H9k3U7N/JmffTG3U+mckDzFJkvpnQEiSOs3ngFg+7ALmMPtmavbP5OybqY1U/8zbcxCSpKnN5z0ISdIUDAhJUqd5FxCTPA97XkmyU5KvJrkmydVJ3tC2PynJF5Nc1/77xLY9ST7Q9tkPkzx3uJ+gf0k2TfK9JOe3wzsn+U7bN59p70RMki3a4evb8YuHWfeGkGTbJGcm+fd2G9rbbaeR5G/b/1NXJTk9yZajvO3Mq4CY4nnY880DwJuq6hnAC4C/bvvhOODLVbUr8OV2GJr+2rV9HQ18aMOXvMG9geZuw+PeDby37ZufA0e27UfSPCp3CfDedrqN3fuB/1tVvws8i6af5v22k2QR8HpgrH3E8qY0jzMY3W2nqubNC9gbuGhg+Hjg+GHXNewXcA7wJ8C1wPZt2/bAte37jwCHDUz/0HQb44vmIVZfBv4YOJ/mKYh3AptN3I5oblu/d/t+s3a6DPsz9Ng3vwXcMPEzuu0UPPw45Se128L5wAGjvO3Mqz0Iup+HvWhItcwJ7W7tc4DvAAur6laA9t/fbiebb/32PuDNwG/a4ScDv6jmWSbwyM//UN+049e002+sdgHuAP61PQT3sSRb4bZDVd0M/DPwU+BWmm3hCkZ425lvAdH5POwNXsUckWQBcBZwbFXdM9WkHW0bZb8leQlwe1VdMdjcMWnNYNzGaDPgucCHquo5wL08fDipy7zpn/a8y8uAnYEdgK1oDrFNNDLbznwLiMf8POyNRZLH0YTDaVX12bb5tiTbt+O3B25v2+dTv/0+8NIkq4AVNIeZ3gdsm2T8AVuDn/+hvmnHbwPcvSEL3sBWA6ur6jvt8Jk0geG2A/sDN1TVHVX1n8BngX0Y4W1nvgWEz8OmubIE+Bfgmqp6z8Coc4HXtO9fQ3NuYrz9z9srUl4ArBk/nLCxqarjq2rHqlpMs318paoOB74KvLKdbGLfjPfZK9vp59RfgetTVf0MuCnJbm3TC4Ef4bYDzaGlFyR5Qvt/bLxvRnfbGfZJkCGcSHox8P+AHwNvG3Y9Q+qDfWl2ZX8IfL99vZjm+OeXgevaf5/UTh+aq79+DFxJc5XG0D/HBuinpcD57ftdgEuB64F/A7Zo27dsh69vx+8y7Lo3QL88G7i83X4+BzzRbeehvnkH8O/AVcAngS1GedvxVhuSpE7z7RCTJGmGDAhJUicDQpLUyYCQJHUyICRJnQwIzRtJ3pvk2IHhi5J8bGD4pCRvfAzLPyHJf5+k/eYk30/yoySHPYZ1LB2/w6zUNwNC88m3aL7ZSpJNgKcAzxwYvw/wzZksqL0z8Lp4b1U9m+ZWDB9pv8kuzWkGhOaTb9IGBE0wXAX8MskTk2wBPAP4Xvut3//V3tP/yiSHwEN/vX81yadpvvRFkreleb7Il4DdHr3KR6qq64Bf0Xy5jCRHJbksyQ+SnJXkCW37qe1zFL6V5CdJXjlxWUme394wb5fH2jFSl82mn0TaOFTVLUkeSPJUmqC4hOaOmnvT3Enzh1V1f5L/QvNt4WfR7GVcluTidjG/B+xRVTckeR7N7TieQ/N/6bs0d++cVPvAnOuqavxeRZ+tqo+2495J84yAk9tx29N86/13aW7LcObAcvZpp3tZVf10tn0iTcWA0HwzvhexD/AemoDYhyYgvtVOsy9welU9SHMTuq8BzwfuAS6tqhva6f4AOLuqfgWQZKr7ev1tkqNobrtw4ED7Hm0wbAssoHlGwLjPVdVvgB8lWTjQ/gxgOfCiqtpYb3ynOcBDTJpvxs9D7ElziOnbNHsQg+cfum7DPO7eCcMzvVfNe6tqN+AQ4BNJtmzbTwWOqao9ae7js+XAPP8x8H6wpluBX9PsuUi9MSA033wTeAlwd1U9WFV30/z1vjfNISeAi4FD0jyXejvgD2lupjbRxcCfJnl8kq2BZdOtvJpbq1/Ow3fx3Bq4tT1pffgMP8MvgIOBf0yydIbzSOvMgNB8cyXNeYVvT2hbU1V3tsNn09yp9AfAV4A3V3Ob60eoqu8Cn6G5G+5ZwNdnWMM/AG9sr6T6HzRP8/sizV1AZ6SqbqMJpFOS7DXT+aR14d1cJUmd3IOQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSp/8PQnlY7swvb00AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Word distribution that allowed detect the count of words in a specific range\n",
    "\n",
    "plt.hist(sorted_word_counts, bins=50, log=True);\n",
    "plt.ylabel(\"Freq\")\n",
    "plt.xlabel(\"Word Rank\")\n",
    "plt.title('Word Frequence - semilog')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>netflix</td>\n",
       "      <td>853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>watch</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>dont</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cats</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>jesus</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>show</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>watching</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>like</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>trailer</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>im</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>season</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>anne</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>good</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>movie</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>series</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        word  freq\n",
       "0    netflix   853\n",
       "1      watch   102\n",
       "2       dont    87\n",
       "3       cats    81\n",
       "4      jesus    79\n",
       "5       show    79\n",
       "6   watching    77\n",
       "7       like    63\n",
       "8    trailer    61\n",
       "9         im    56\n",
       "10    season    55\n",
       "11      anne    50\n",
       "12      good    49\n",
       "13     movie    48\n",
       "14    series    47"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converting the dictionary to a pandas DataFrame\n",
    "df = pd.DataFrame(most_common_words, columns=['word','freq'])\n",
    "\n",
    "# sorting the words based on their frequence\n",
    "df = df.sort_values(by=['freq'], ascending=False)\n",
    "\n",
    "# getting the top 20 more frequent words\n",
    "top_20 = df.head(20)\n",
    "top_20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAElCAYAAACI+8edAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu4XVV59v/vzUmBcAa3EIKg4JEIhS2ioiZgXwUPUAseShUoNvqTeqi0Gm0tyqtVWtF6aNEoKuAhIkpBQCuNBIyCQBAICL4gBAihQTEEwkEavH9/zLHIyj4kOztzrr0yc3+ua197rbnmGs+YOzv7WXPMZ44h20RERLTFRhPdgYiIiDolsUVERKsksUVERKsksUVERKsksUVERKsksUVERKsksUU0TNJHJH1jovsRsaFIYosNiqQPSrpoyLZbRtn2ph70Z5qkP0pa3vX1g6bjRrTZJhPdgYgeuwyYKWlj249LeiqwKbDfkG17ln3HTJIA2f7jWvZpse1dx9D+JrZXrGXbERucnLHFhuYqqkS2b3n+MuAS4NdDtv3G9mIASS+WdJWkZeX7izuNSZor6eOSfgY8DDxd0h6SLpX0oKSLgR3H09EyhHmOpG9IegA4VtJGkmZK+o2k+ySdLWn7rve8RdId5bV/kLRQ0ivKa1+X9LGufadJWtT1fBdJ35P0W0m3S3r3kL6cLenMclw3Shrsen2KpO+X994n6Qtdr/2VpJskLZX0X5KeNp6fR8RYJbHFBsX2Y8AvqJIX5ftPgXlDtl0GUJLGhcDngB2ATwMXStqhq9m3ADOArYA7gG8B86kS2v8FjlmHLh8OnANsC3wTeDdwBPByYBdgKfDvpa/PBU4r/dml9HeNZ4LlvRsBPwCuAyYDhwDvlfTKrt1eB8wufTkf+EJ578bABVTHvnt5/+zy2hHAh4DXAztR/ay/vZY/g4i1ksQWG6JLWZnEXkr1x/anQ7ZdWh6/GrjF9lm2V9j+NnAz8Nqu9r5u+8YyTLgz8ALgw7b/YPsyqoSxOrtIur/r6w1dr11u+z9t/9H2I8DbgX+wvcj2H4CPAEdK2gQ4ErjA9mXltQ8DYx0WfQGwk+2TbT9m+zbgy0D3dcZ5ti+y/ThwFrBP2X4AVSL9e9sP2X7U9rzy2tuBT9i+qfx8/hnYN2dt0aRcY4sN0WXACZK2o/pjfoukJcAZZdverLy+tgvVmUi3O6jOSjru6nq8C7DU9kND9p+ymv6s7hrbXUOePw04V1J3wnocGCixn9jf9kOS7ltN3KHt7iLp/q5tG1Ml/I7/6Xr8MPDkklCnAHeMcv3vacBnJZ3atU1UP7+hP9eIWiSxxYbocmAbquHDnwHYfkDS4rJtse3by76Lqf44d9sN+FHX8+4lMu4BtpO0ZVdy223IPmtj6PvuAv7K9s+G7ijpHuA5Xc+3oBqO7HgI2KLr+VOHtHu77b3G0ce7gN1GKW65C/i47W+Oo92IcclQZGxwypDe1cD7WPWMZF7Z1l0NeRHwTEl/IWkTSW8Enkt1TWmktu8obX9U0maSDmLVYct19UXg452hPEk7STq8vHYO8BpJB0naDDiZVf+PXwscJmn7Uvn53q7XrgQekPQBSZtL2ljS3pJeMIY+XUmV0D8paUtJT5b0kq7+flDS80p/t5F01DiPPWJMkthiQ3Up8BSqZNbx07LticRm+z7gNcCJwH3A+4HX2P7datr+C+CFwO+Bk4Aza+z3Z6kKN34s6UHgihIL2zcCJ1AVr9xDVViyqOu9Z1EVhywEfgx8p/NCuW72WqrK0NuB3wFfoTqzXa2u9+4J3FlivrG8di5wCjC7VHbeABw6ngOPGCtlodGI9pK0EHib7f+e6L5E9ErO2CIiolWS2CIiolUyFBkREa2SM7aIiGiVJLaIiGiV9foG7R133NG77777Wr3noYceYsstt2ymQz2M0as4bYnRqzg5lv6L0as4OZbmY8yfP/93tnda446219uv/fff32vrkksuWev39GOMXsVpS4xexcmx9F+MXsXJsTQfA7jaY8gNGYqMiIhWSWKLiIhWSWKLiIhWSWKLiIhWSWKLiIhWSWKLiIhWSWKLiIhWWa9v0B7N7jMvHPW1E6eu4NgRXl/4yVc32aWIiOiRnLFFRESrJLFFRESrJLFFRESrJLFFRESrJLFFRESrJLFFRESrJLFFRESrNJrYJP2tpBsl3SDp25KeLGkPSb+QdIuk70jarOz7pPL81vL67k32LSIi2qmxxCZpMvBuYND23sDGwJuAU4DP2N4LWAocX95yPLDU9p7AZ8p+ERERa6XpochNgM0lbQJsAdwDHAycU14/AziiPD68PKe8fogkNdy/iIhomcYSm+27gU8Bd1IltGXAfOB+2yvKbouAyeXxZOCu8t4VZf8dmupfRES0k2w307C0HfA94I3A/cB3y/OTynAjkqYAF9meKulG4JW2F5XXfgMcYPu+Ie3OAGYADAwM7D979uxhsRfcvWzUfg1sDkseGb596uRt1v4gR7F8+XImTZpUW3sTGactMXoVJ8fSfzF6FSfH0nyM6dOnz7c9uKb9mpwE+RXA7bZ/CyDp+8CLgW0lbVLOynYFFpf9FwFTgEVl6HIb4PdDG7U9C5gFMDg46GnTpg0LPNIkxx0nTl3BqQuGH/bCo4e3M15z585lpH7VrRdx2hKjV3FyLP0Xo1dxciz9E6PJa2x3AgdK2qJcKzsE+BVwCXBk2ecY4Lzy+PzynPL6T9zU6WRERLRWk9fYfkFVBHINsKDEmgV8AHifpFuprqGdXt5yOrBD2f4+YGZTfYuIiPZqdD022ycBJw3ZfBtwwAj7Pgoc1WR/IiKi/TLzSEREtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEoSW0REtEpjiU3SsyRd2/X1gKT3Stpe0sWSbinftyv7S9LnJN0q6XpJ+zXVt4iIaK8m12P7te19be8L7A88DJxLtc7aHNt7AXNYue7aocBe5WsGcFpTfYuIiPbq1VDkIcBvbN8BHA6cUbafARxRHh8OnOnKFcC2knbuUf8iIqIlepXY3gR8uzwesH0PQPn+lLJ9MnBX13sWlW0RERFjJtvNBpA2AxYDz7O9RNL9trften2p7e0kXQh8wva8sn0O8H7b84e0N4NqqJKBgYH9Z8+ePSzmgruXjdqfgc1hySPDt0+dvM04jm5ky5cvZ9KkSbW1N5Fx2hKjV3FyLP0Xo1dxcizNx5g+ffp824Nr2m+Tcfdq7A4FrrG9pDxfImln2/eUocZ7y/ZFwJSu9+1KlRBXYXsWMAtgcHDQ06ZNGxbw2JkXjtqZE6eu4NQFww974dHD2xmvuXPnMlK/6taLOG2J0as4OZb+i9GrODmW/onRi6HIN7NyGBLgfOCY8vgY4Lyu7W8t1ZEHAss6Q5YRERFj1egZm6QtgD8F3t61+ZPA2ZKOB+4EjirbLwIOA26lqqA8rsm+RUREOzWa2Gw/DOwwZNt9VFWSQ/c1cEKT/YmIiPbLzCMREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqjSY2SdtKOkfSzZJukvQiSdtLuljSLeX7dmVfSfqcpFslXS9pvyb7FhER7dT0GdtngR/ZfjawD3ATMBOYY3svYE55DtVK23uVrxnAaQ33LSIiWqixxCZpa+BlwOkAth+zfT9wOHBG2e0M4Ijy+HDgTFeuALaVtHNT/YuIiHZq8ozt6cBvga9J+qWkr0jaEhiwfQ9A+f6Usv9k4K6u9y8q2yIiIsZM1cLVDTQsDQJXAC+x/QtJnwUeAN5le9uu/Zba3k7ShcAnbM8r2+cA77c9f0i7M6iGKhkYGNh/9uzZw2IvuHvZqP0a2ByWPDJ8+9TJ26z1MY5m+fLlTJo0qbb2JjJOW2L0Kk6Opf9i9CpOjqX5GNOnT59ve3BN+20y7l6t2SJgke1flOfnUF1PWyJpZ9v3lKHGe7v2n9L1/l2BxUMbtT0LmAUwODjoadOmDQt87MwLR+3UiVNXcOqC4Ye98Ojh7YzX3LlzGalfdetFnLbE6FWcHEv/xehVnBxL/8RobCjS9v8Ad0l6Vtl0CPAr4HzgmLLtGOC88vh84K2lOvJAYFlnyDIiImKsmjxjA3gX8E1JmwG3AcdRJdOzJR0P3AkcVfa9CDgMuBV4uOwbERGxVhpNbLavBUYaDz1khH0NnNBkfyIiov0y80hERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKEltERLRKo4lN0kJJCyRdK+nqsm17SRdLuqV8365sl6TPSbpV0vWS9muybxER0U69OGObbnvfruW8ZwJzbO8FzCnPAQ4F9ipfM4DTetC3iIhomYkYijwcOKM8PgM4omv7ma5cAWwraecJ6F9ERKzHmk5sBn4sab6kGWXbgO17AMr3p5Ttk4G7ut67qGyLiIgYM1ULVzfUuLSL7cWSngJcDLwLON/2tl37LLW9naQLgU/Ynle2zwHeb3v+kDZnUA1VMjAwsP/s2bOHxV1w97JR+zSwOSx5ZPj2qZO3WfsDHMXy5cuZNGlSbe1NZJy2xOhVnBxL/8XoVZwcS/Mxpk+fPr/rstaoNhl3r8bA9uLy/V5J5wIHAEsk7Wz7njLUeG/ZfREwpevtuwKLR2hzFjALYHBw0NOmTRsW99iZF47apxOnruDUBcMPe+HRw9sZr7lz5zJSv+rWizhtidGrODmW/ovRqzg5lv6J0dhQpKQtJW3VeQz8H+AG4HzgmLLbMcB55fH5wFtLdeSBwLLOkGVERMRYjemMTdL7Vve67U+PsHkAOFdSJ863bP9I0lXA2ZKOB+4Ejir7XwQcBtwKPAwcN6YjiIiI6DLWochB4AVUZ1UArwUuY9Vij1XYvg3YZ4Tt9wGHjLDdwAlj7E9ERMSIxprYdgT2s/0ggKSPAN+1/bamOhYRETEeY73GthvwWNfzx4Dda+9NRETEOhrrGdtZwJWlstHAnwFnNtariIiIcRpTYrP9cUk/BF5aNh1n+5fNdSsiImJ81qbcfwvgAdufBRZJ2qOhPkVERIzbmBKbpJOADwAfLJs2Bb7RVKciIiLGa6xnbH8GvA54CJ6YUWSrpjoVERExXmNNbI+V+8wMT8wkEhER0XfGmtjOlvQlqqVk/hr4b+DLzXUrIiJifMZaFfkpSX8KPAA8C/gn2xc32rOIiIhxWGNik7Qx8F+2X0G19ExERETfWuNQpO3HgYcl1bdgWUREREPGOvPIo8ACSRdTKiMBbL+7kV5FRESM01gT24Xla62Vocyrgbttv6bc2D0b2B64BniL7cckPYlqmq79gfuAN9peOJ6YERGx4VptYpO0m+07bZ+xDjHeA9wEbF2enwJ8xvZsSV8EjgdOK9+X2t5T0pvKfm9ch7gREbEBWtM1tv/sPJD0vbVtXNKuwKuBr5TnAg4Gzim7nAEcUR4fXp5TXj+k7B8RETFma0ps3Ynl6eNo/9+A9wN/LM93AO63vaI8XwRMLo8nUxYuLa8vK/tHRESMmaoJRUZ5UbrG9n5DH4+pYek1wGG23ylpGvB3wHHA5bb3LPtMAS6yPVXSjcArbS8qr/0GOKCsuN3d7gxgBsDAwMD+s2fPHhZ7wd3LRu3XwOaw5JHh26dOrq/oc/ny5UyaNKm29iYyTlti9CpOjqX/YvQqTo6l+RjTp0+fb3twTfutqXhkH0kPUJ25bV4eU57b9tajv5WXAK+TdBjwZKprbP9GNXvJJuWsbFdgcdl/ETCFauWATYBtgN8PbdT2LGAWwODgoKdNmzYs8LEzR69zOXHqCk5dMPywFx49vJ3xmjt3LiP1q269iNOWGL2Kk2Ppvxi9ipNj6Z8Yqx2KtL2x7a1tb2V7k/K483x1SQ3bH7S9q+3dgTcBP7F9NHAJcGTZ7RjgvPL4/PKc8vpPvLrTyYiIiBGszXpsdfkA8D5Jt1JdQzu9bD8d2KFsfx8wcwL6FhER67mx3se2TmzPBeaWx7cBB4ywz6PAUb3oT0REtNdEnLFFREQ0JoktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJaJYktIiJapbHEJunJkq6UdJ2kGyV9tGzfQ9IvJN0i6TuSNivbn1Se31pe372pvkVERHs1ecb2B+Bg2/sA+wKvknQgcArwGdt7AUuB48v+xwNLbe8JfKbsFxERsVYaS2yuLC9PNy1fBg4GzinbzwCOKI8PL88prx8iSU31LyIi2km2m2tc2hiYD+wJ/Dvwr8AV5awMSVOAH9reW9INwKtsLyqv/QZ4oe3fDWlzBjADYGBgYP/Zs2cPi7vg7mWj9mlgc1jyyPDtUydvM44jHNny5cuZNGlSbe1NZJy2xOhVnBxL/8XoVZwcS/Mxpk+fPt/24Jr2a3QFbduPA/tK2hY4F3jOSLuV7yOdnQ3LurZnAbMABgcHPW3atGFvOnbmhaP26cSpKzh1wfDDXnj08HbGa+7cuYzUr7r1Ik5bYvQqTo6l/2L0Kk6OpX9i9KQq0vb9wFzgQGBbSZ3MsiuwuDxeBEwBKK9vA/y+F/2LiIj2aLIqcqdypoakzYFXADcBlwBHlt2OAc4rj88vzymv/8RNjpNGREQrNTkUuTNwRrnOthFwtu0LJP0KmC3pY8AvgdPL/qcDZ0m6lepM7U0N9i0iIlqqscRm+3rgT0bYfhtwwAjbHwWOaqo/ERGxYcjMIxER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpJbBER0SpNrsc2RdIlkm6SdKOk95Tt20u6WNIt5ft2ZbskfU7SrZKul7RfU32LiIj2avKMbQVwou3nUK2cfYKk5wIzgTm29wLmlOcAhwJ7la8ZwGkN9i0iIlqqscRm+x7b15THD1Ktnj0ZOBw4o+x2BnBEeXw4cKYrVwDbStq5qf5FREQ7yXbzQaTdgcuAvYE7bW/b9dpS29tJugD4pO15Zfsc4AO2rx7S1gyqMzoGBgb2nz179rB4C+5eNmpfBjaHJY8M3z518jZre1ijWr58OZMmTaqtvYmM05YYvYqTY+m/GL2Kk2NpPsb06dPn2x5c036NraDdIWkS8D3gvbYfkDTqriNsG5Z1bc8CZgEMDg562rRpw9507MwLR+3PiVNXcOqC4Ye98Ojh7YzX3LlzGalfdetFnLbE6FWcHEv/xehVnBxL/8RotCpS0qZUSe2btr9fNi/pDDGW7/eW7YuAKV1v3xVY3GT/IiKifZqsihRwOnCT7U93vXQ+cEx5fAxwXtf2t5bqyAOBZbbvaap/ERHRTk0ORb4EeAuwQNK1ZduHgE8CZ0s6HrgTOKq8dhFwGHAr8DBwXIN9i4iIlmossZUikNEuqB0ywv4GTmiqPxERsWHIzCMREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqSWwREdEqTa7H9lVJ90q6oWvb9pIulnRL+b5d2S5Jn5N0q6TrJe3XVL8iIqLdmjxj+zrwqiHbZgJzbO8FzCnPAQ4F9ipfM4DTGuxXRES0WGOJzfZlwO+HbD4cOKM8PgM4omv7ma5cAWwraeem+hYREe2lan3PhhqXdgcusL13eX6/7W27Xl9qeztJFwCfLIuTImkO8AHbV4/Q5gyqszoGBgb2nz179rC4C+5eNmqfBjaHJY8M3z518jZrc2irjTNajPHGGc3y5cuZNGlSbe21OUav4uRY+i9Gr+LkWJqPMX369Pm2B9e0X2MraK+lkVbaHjHj2p4FzAIYHBz0tGnThu1z7MwLRw104tQVnLpg+GEvPHp4O2syWpzRYow3zmjmzp3LSMdfp7bE6FWcHEv/xehVnBxL/8TodVXkks4QY/l+b9m+CJjStd+uwOIe9y0iIlqg12ds5wPHAJ8s38/r2v43kmYDLwSW2b6nx33rS7uv4exzpLPGhZ98dZNdiojoa40lNknfBqYBO0paBJxEldDOlnQ8cCdwVNn9IuAw4FbgYeC4pvoVIxstgSZ5RsT6prHEZvvNo7x0yAj7Gjihqb5Ef1jb5Alrn0B7dYbbi2OJiPHJzCMREdEqSWwREdEq/VLuHxFDpHAoYnyS2CI2cCkcirZJYouIxrWpcCj6XxJbRMRaSJLuf0lsEREbqF4MQ0/ErTGpioyIiFZJYouIiFZJYouIiFZJYouIiFZJYouIiFbpq8Qm6VWSfi3pVkkzJ7o/ERGx/umbxCZpY+DfgUOB5wJvlvTcie1VRESsb/omsQEHALfavs32Y8Bs4PAJ7lNERKxn+imxTQbu6nq+qGyLiIgYM1VrfE48SUcBr7T9tvL8LcABtt81ZL8ZwIzy9FnAr9cy1I7A79axu/0Qo1dx2hKjV3FyLP0Xo1dxcizNx3ia7Z3WtFM/Tam1CJjS9XxXYPHQnWzPAmaNN4ikq20Pjvf9/RKjV3HaEqNXcXIs/RejV3FyLP0To5+GIq8C9pK0h6TNgDcB509wnyIiYj3TN2dstldI+hvgv4CNga/avnGCuxUREeuZvklsALYvAi5qOMy4hzH7LEav4rQlRq/i5Fj6L0av4uRY+iRG3xSPRERE1KGfrrFFRESssyS2iIholSS26DlJW0raqDx+pqTXSdp0ovsVEe2wQSQ2SU8eYduODcTZWNIuknbrfNXc/lFj2VZjvO0kPb+Bpi8DnixpMjAHOA74et1BJB0saYu6251okrZsoE1JmrLmPWMoSQdJOq483knSHhPdp/VBg39fNoziEUkLgL+2fUV5/ufAJ2w/s8YY7wJOApYAfyybbbu2fzhJ19jeb03b1jHGXOB1VBWz1wK/BS61/b4aY1xje7/yM9vc9r9I+qXtP6krRolzJnAgcB/w0/I1z/bSmuP8ABj6H2kZcDXwJduP1hTnxcBXgEm2d5O0D/B22++sqf35tvevo601xNkDeBewO12V2bZfV3OcycDThsS4rOYYJwGDwLNsP1PSLsB3bb+khrYXMPz36gl1/m0p8Z4JnAYM2N67JJ3X2f5YjTHm0vDfF+izcv8G/QXw1fJD3QXYATi45hjvofrlvq/mdpF0KHAYMFnS57pe2hpYUXO4bWw/IOltwNdsnyTp+ppjSNKLgKOB48u22n8Xbb+1BNsFOJJq9YhdGoh1G7AT8O3y/I1UH3CeCXwZeEtNcT4DvJIycYHt6yS9rKa2Aa6Q9ALbV9XY5kj+Ezgd+AErPwTWStIpVP8OvwIeL5tNNVpQpz8D/gS4BsD2Yklb1dT2a8r3E8r3s8r3o4GHa4rR7cvA3wNfArB9vaRvAbUlNnrz92XDSGy2F0j6ONUvxoPAy2wvqjnMXVSf0puwmOrT/+uA+V3bHwT+tuZYm0jaGXgD8A81t93xXuCDwLm2b5T0dOCSuoNI+kvgpcBUqjnpvkB11la3P7HdnWB+IOky2y+TVOskA7bvktS96fHR9h2H6cA7JC0EHgJEzaMOxaO2P7fm3dbJEVQfNP/QcJzHbFuSod5hYtt3lDZfMuQMcKaknwEn1xWr2ML2lUN+v+r+4NyLvy8bRmKTdDrwDOD5VJ+ifyDpC7b/vYa2O6fQtwFzJV0IPPGfyfan1zWG7euA6yR9y/b/rmt7a/BRqtlf5tm+qiSdW+oMYPtS4NKu57cB764zRvFvwG+ALwKX2F7YQAyAnSTtZvtOgHJttXMN97Ea49xVhiNdpp17N3BTje0fWmNbq/PZMoT3Y1b9v3JNjTFuAzbtbr8hZ0v6ErCtpL8G/orqzKdOW0o6yPY8eGJIuvbrrMDvJD2DMvwp6UjgnppjnEz19+VnTf19gQ0ksQE3AG9zdUHxdkkHAuuccIrOsMOd5Wuz8tWEAyR9hJXXDTqfqJ9eY4x7uj+h275NUl0/KwAkXcII1w5s1zo8bHtHSc8DXgZ8XNJewK9t1zU02HEiME/Sb6j+TfYA3lk+vZ9RY5x3AJ+lWs5pEVViOGG171gLtu+QdBCwl+2vSdoJmFRX+12mUg3PHkzX9WjqvTzwMHCtpDmsmjxr/QBl+1OS/hR4gGq1kX+yfXGdMaiG678qaZvy/H6qBFq3E6hmA3m2pLuB24G/rDOA7e8C3+16fhvw53XGgA2keKQtJN1MNfQ4n64hqDqv6/WoQKW7QOHJVL/YK2y/v64YJc7WwEuAl1MNSe4IXGH7mDrjlFhPAp5NldhurqtgZEiM7W3/fsi2PWzfXlP7jRVCDIlzM/D8sqBwIySN+G9su84PGj1Vfp9lu6lLHp04WwIb2X6wgbYbL1CBlp+xSTrb9htGqC6q/dqBpIuBo2zfX55vB8y2/cq6YgDLbP+wxvaeUIo5Xkw1rNZdobQ11aTUtbE9f8imn0m6dMSd1828rq8vNHBdtdv+rKzye74kbJ9Zc4wfSDrU9gMAkp5D9el375rab7IQott1wLbAvQ20DVQJrAzXdiqff93EML6k1wOnAE+h+rvS+duydY0xtqGquH5ZeX4pcHJdCU7SX9r+xpD/93SutdVxOaVLLwpU2p3YqCoVYWV1UZN26iQ1ANtLJT2l5hiXSPpX4PvUf21iM6php01YObwK1RDLkTW0/wRJ23c93YgqKTy1zhiwshy6/HFubGhC0llU13CvZdUKvLoT2z9TJbdXUw17nUlVIVeXxgohhhgAbpZ0Fav+HtdW7i9pGtUw8EKqZDNF0jF1l/sD/wK81nad1zqH+irV5ZQ3lOdvAb4GvL6m9jv/zk18iBmqFwUq7U5stu8p3+/oQbjHhxQQPI36/5i+sHzvXpyvlmsTnYIOSV/vwc9rPlW/RfVLfTsry/5rI2lvqkrY7aun+i1wjO0bag41CDwLfiR1AAANW0lEQVTXDY/r275Q1QwtP6b6I3SE7TovvPeiEAKqs4+mnQr8H9u/hieGwL5N9SGqTksaTmoAz7DdfR3qo5Kuratx218qD//D9m/rancUvShQaXdik/QgK/+AjjQUWdtwAVXp6ryuIbWXAW+vsX1sT6+zvVE8XM4Kn0d1/asTu7YL+7Z7NTPDLOB9ti+BJz7Fz6Iacq3TDVRnnLX/BwWQ9HlW/f3dmqrq711lyLOWgogeFUJ0PkQ1bdNOUisx/5+ambbtaknfobo3r/vs8/s1xnhkSFXkS4BHamy/4+eSbge+A3zfNU9kUDReoAItLx6R9PRSddOreDtSzXQh4HLbv6u5/VXG2qlK5msbay8xfkz1i/13VFV4xwC/tf2BGmMcBfzI9oOS/hHYD/hYzeXeSLrO9j5r2lZDnEuAfYEraWBobbRCiK4460VBhKR5tg/q+sD5xEvUf13qqyVG56bmvwQ2tn1cXTFKnK+NsNm2a6talLQv1bDqNlQ/q99TjTzUfmOzpAOAN1HdB/grqjqBbzQQp7ECFWh/Yptve39Jc2wf0nCsYTHqjivpe1RnB50/ZG8B9rFd11h798/s+q5rVJfafnmNMa63/fxSWv4J4FPAh2y/cA1vXds451IVQnT/cRu0fUTNcUb82fTozGSdjZBonniJ+kc2eqJUqZ4AHER1HJdRDbU1fV9bY0pVJJ3ioYZj7Uh1S9TRtte5eGy0ApWOmgtU2j0UCWxUSpifOdIPtI4fpqoJlrcAdiyVkJ2roltTTd9Up0bH2otO5dg9pUhhMbBrzTE6BRavBk6zfZ6q+/Pq9ldUN5x/n5V/3Gr9xA7NJ7DVVPd24q9Tda/tXhQN9FRJYJ8GPl2KlXZtIqmVe/3+muHzXtZ5xtZoVWRXnK2pKmPfRFUMdS5wQE3N97JApfWJrXNKPbTSr05vp5oiaheqoohOYnuAam7COvVirP1j5T/SicDnqRL0e2uOcXcpUngFcEr5dF37ShPlGkETM5oAPR1aa7S6V9LWrubv236k1z3k3rn1gUaYbLeMPNQ62S5wHtU0bf9NvdObdWu6KrLjOqprhSfbvrzOhm1/SdLGwAO2P1Nn2yNp9VBkh6R32v6PIdtqu7G1tPcu25+vq71RYuxDVeLdmYFgKTWPtUs6A3hP1/142wOfqvkT6BbAq4AFtm9RNXfcVNs/ritGifNMqmuFu7Pqp+m6J8Ber0m6wPZrSuFAp9iqw653ZpueUFktQtVku1NcJttd17PbEeJca3vfOtscS4wm4kpSud1jK6p/9+V1tl9iXNKLIri2n7F1vA34jyHbzqHG0l/bny/l5c9l1WrCdb6Xacgw6pmsPK1/iOqsp86LyM/3qvfj/V5SrcvJ2H5Y0r1U1z9uoSr5r32+OKqbl79ItdRLU5+mASifRgdYNYHeWVPbjV4Ds/2a8r1N64j1ZLJd4AJJh9m+qMEYvaqKfJ6qezKbvD3m55K+QFWg9lBnY92FY61ObJKeTVW2vo2qGQI6tqYr+dQU6yRgGlViu4hqQtl51HOTbmcY9VnAC6iGP0RVDFH3DacbSdquU+pbzthq/T1R19RNVEMqmwLfoJr+qk4rbJ9Wc5vDaJS1+Kgm3V5nvbwGVq4T78WqH87q/h3rhc5ku41N5l28B/iQpD9QXZ9uouDmHcCZWjlX5FKqauW69eL2mE5b3SsT1D1PaLuHIiUdTnWN7XWUNayKB6nKWH9eY6wFwD7AL23vI2kA+Irt19YY48fAn3dKZMuQwXdtv6rGGG+lWlLmHKpfuDcAH7d91mrfuHYxrqVM3eSyuGidw0Rd14reTTVt07msWoZf6zUjSbcCL3QDa/H1Uhm2ew9VsdC1VLeuXJ6h29Urv29DPwzUVlDUNWLTmZB6OdUSWfNt11Y81qvbY3qh1Wdsts8DzpP0orovho7gUdt/lLSiVBfdC9R9bWI3Vl0G5TGq60e1sX2mpKupPkEJeL3tX9UZg+anbuqe2QSquem6P8HV/e/S5Fp8vfQeqhGBK2xPLyMeH53gPo1LqVY+nuETDdQ6K/4oHwZ+DtR5e9Fg+Tqf6nf6L4CrqNbO+67tf6kpzm2SPsyqt8fUVocAUD7w/zOwi+1DJT0XeJHt0+uM0+rE1uU+VctXNDmj9FWStqWagmg+1aeqK2tsH6pfuCvL/VmmKs2t/ebcksjqTmbdhk7ddDzVdbBadK4VSXoD1Y3gD5T/sPsB/7euOOrBWnw99qjtRyUh6Um2b5b0rInu1DidBdxMteL4yVRzajYx9VUvPgzsAOzXKeYoQ/nnUJX/z6ear7IOndtjvsfK22OOrantjq9TXX7oXPf8f1TX22pNbLWXWPepL1MNr/0vVDNKU90KUKetgKOAucCfUl10rfWeKdsfp7oPaynVmkzH2f5EnTF6wfanqP5jfo9q9vV/dDMrKv9jSWoHUf2bfJ1qyYy6bFW+7gQupppIequur/XNovLh7D+BiyWdR3Uf4/poT9sfBh5yNTPLq6nWgavboy5LFHU+DFBdO67T0JGa/wWeZvsR6l1I9RnAFKq8sCnVWWfd11d3tH025Vq07RU0UNi1oZyx9WJG6a9RVfl9nmqo61pJl9n+bJ1BSvVQrRVEvTLCfV+df5B3SPoj1VRB/zr01ox10H0j+BfrvhHc9no5TDca239WHn5E1TRh2wA/msAurYvORAP3l2rl/6HmYfti6IeBpdT/YeBbwBXlgwbAa4FvlyH8OkdWvkl1e8wNrCyCqttDknZg5STIB9LAMH6ri0c6JP0Q+BuqQov9VM0ofbztQ2uOszHVsMR0qkqmR2w/u84YbVZ+4X9uu5ZPvJIuAO6muiVif6oS6Svrvhiu3qzF1yhJGwHX265rbbcJVa59fY+qMvVrVIUX/2T7iw3GfDnlw4BrXkRV1eK8nenB5tm+us72S4x5tg+qu90hMfaj+vC/N1UC3Qk4ss57cWHDSWxPZ2XZ6lKqC6JHu8blWco1vC2By6lmIphnu7GFFNtK0s4uyw3V0FavbgQf6QbaX3YqPtcXkr4JfLCu++9i/SLpEODNwBwaWqlA1QTo/0U15PnnVEtxfTj3sY3P3VSf2i6huvnwAar7QE5e3ZvW0vVUZwV7U51a3y/p8jIOHmNUV1IrbT1MNU9kd9tNLC3Ti7X4emFn4EZJV7LqzbO1LQDaK72qvmuZ44BnU11f674fs84leD5s+7tlVOMVVOvmncbKtSZrsaEktvOoii2uoaGL4bb/FkDSJKpfkK9RrdH1pCbiRV9pfC2+HpnEqvNRCjhlgvqyrr5OD6rvWmYf200U2HRr9Lp3x4aS2Hat8ybmkUj6G+ClVGdtd1BNXPrTJmNGf7D9o3LtoLMW39+65rX4emSToTcWS9p8ojqzjna0fbakD0JVfSep0WnVWuAKSc9t4L7Vbj2ZAH1DSWw/lzTV9oIGY2xOtUzG/FLCGhsIrVx374IRtvU9Sf8f8E7g6ZK6L+JvBfxsYnq1znpSfdcyBwHHqJoM+w+snB6szomj30B13ftTtu8v173/vsb2gQ2neORXwJ5URSNN/YPFBkYr1+K7hGqe0O61+H5o+zkT1LW1UuYg3I5q0deZXS89WPf0Y73Sq+q7NinXhoeps8iuVzaUM7Zay/ojil6uxdcYVwtWLqOqiGuLZ1D9v++uvttQ/t6Ny/qYwEazQZyxRTRJPViLL9ZOZ1LtMuvMP1NV333Idq3Vd9GfktgiaqCG1uKL8dHKhUY/QXUf47fWx3sLY3xyah6xjtTsWnwxPj2pvov+lH/oiHV3JNWEsf9TJr7eh9y/ONHeQDXDxavKVGfb00D1XfSnnLFFrLterMUXa6GHs85EH0pii1h3vViLLyLGKMUjEetI0llU61b9FHgU2Dr3S0VMnCS2iHUk6WCqWRteSlmLD6h9Lb6IGJsktogaZC2+iP6Ra2wR62iEtfhekLX4IiZOyv0j1t31wGNU8xI+H9h7PZ4VP2K9l6HIiJp0rcX3d8BTbedetogJkKHIiHWUtfgi+ksSW8S6y1p8EX0kQ5EREdEqKR6JiIhWSWKLiIhWSWKLaClJ0yRdMNH9iOi1JLaIliizn0Rs8JLYIvqApPdLend5/BlJPymPD5H0DUlvlrRA0g2STul633JJJ0v6BfAiSa+SdLOkecDrJ+ZoIiZWEltEf7iM6l44gEFgkqRNqSZXvgU4BTgY2Bd4gaQjyr5bAjfYfiFwNdXSOa8tbT21d92P6B9JbBH9YT6wv6StgD9QzTs5SJWg7gfm2v5tuU/um8DLyvseB75XHj8buN32La7u4/lGLw8gol8ksUX0Adv/CyykmpLr51Qzl0wHngHcuZq3Pmr78e6mmupjxPoiiS2if1xGNc9kZ9HSd1Ct7XYF8HJJO5YCkTcDl47w/puBPSQ9ozx/c/Ndjug/SWwR/eOnwM7A5baXUK3G/VPb9wAfBC4BrgOusX3e0DfbfhSYAVxYikfu6FnPI/pIptSKiIhWyRlbRES0ShJbRES0ShJbRES0ShJbRES0ShJbRES0ShJbRES0ShJbRES0ShJbRES0yv8PqjNlJEcpVhIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Bar ploting from the top 20 more frequent words\n",
    "\n",
    "axes = top_20.plot.bar(x='word',y='freq',legend=False)\n",
    "plt.gcf().tight_layout()\n",
    "plt.ylabel('Freq')\n",
    "plt.grid()\n",
    "plt.title('Word Frequence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "dic = {}\n",
    "for item in unif_words:\n",
    "    dic[item] = dic.get(item, 0) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bulding a World Cloud\n",
    "\n",
    "\n",
    "\n",
    "wordcloud = WordCloud(width=1600,   height=900,\n",
    "                      prefer_horizontal=0.5,\n",
    "                      min_font_size=10,\n",
    "                      colormap='prism')\n",
    "\n",
    "wordcloud = wordcloud.fit_words(dic)\n",
    "\n",
    "wordcloud = wordcloud.to_file('TrendingTwitter.png')\n",
    "\n",
    "\n",
    "#IMPORTANT!!!\n",
    "\n",
    "#The result will create a png file in the fouder where the jupyter notebook are running. In order to see the\n",
    "#result you will need to go check this file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"TrendingTwitter.png\" width=\"900\" class=\"unconfined\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url = 'TrendingTwitter.png',width=900, unconfined=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Using NLTK to extract features and train the Classifier Model\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extract features from both the test data (the 100 tweets to be classifed) and the training data, downloaded from the corpus**\n",
    "    \n",
    "The NLTK model that will be used is known as NAVIE BAYES CLASSIFICATION\n",
    "\n",
    "       1.1 - Build a vocabulary (list of all unique words in all the tweets in the training data);\n",
    "\n",
    "       2.2 - Represent each tweet with the presence or absence of theses words;\n",
    "             Ex.: Given a *vocabulary: {'the','worst','thing,'in','the','world'} and a *tweet: {'the','worst','thing'}        \n",
    "                                    \n",
    "                  This tweet will be represented as a \"Feature Vector\" (1,1,1,0,0,0) -> indicatinting that the \n",
    "                  first three words from the vocabulary are in the tweet, and there is other 3 words in the \n",
    "                  vocabulary that is not part of the mentioned tweet.\n",
    "\n",
    "       2.3 - Use NLTK's built in Navie Bayes Classifier to train a Classifier                                 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Defining the function to create a word Vocabulary or bag_of_words\n",
    "\n",
    "def wordVocab(cleanedtrainingData):\n",
    "    training_features = []\n",
    "    for (words, sentiment) in cleanedtrainingData:\n",
    "        training_features.extend(words)\n",
    "    return list(set(training_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating the word_features_vocab object\n",
    "\n",
    "word_features_vocab = wordVocab(cleanedtrainingData)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The NLTK library have a function called apply_features that takes a user-defined function to extract\n",
    "# featrues from training data. In this case the function will be called extract_features_func that will\n",
    "# take each tweet in the in the training data and repersent it with the presence or absence of a word in\n",
    "# the vocabulary, as previously explaned in the item 2.2 above.\n",
    "\n",
    "def extract_features_func(tweet):\n",
    "    tweet_words = set(tweet)\n",
    "    features = {}\n",
    "    for word in word_features_vocab:\n",
    "        features[f'contains {word}'] = (word in tweet_words)\n",
    "        # this step will creat a dictionary with keys like 'contains word1' and 'contains word2', and values\n",
    "        # as True or False. The statement that create the True or False return is the '(word in tweet_words)'\n",
    "        \n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the traning_features object\n",
    "\n",
    "# apply_features will take the extract_features_func defined above, and apply it to each element of\n",
    "# cleanedtrainingData. it automatically recognize that each of these elements are tuples where the\n",
    "# first element is the text and the second is the label. The apply_features apply the extract_features_func\n",
    "# only on the text element.\n",
    "\n",
    "trainingFeature = nltk.classify.apply_features(extract_features_func,cleanedtrainingData)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the classifier objected, trained using the training data features \n",
    "\n",
    "NBclassifier = nltk.NaiveBayesClassifier.train(trainingFeature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - Run the Classifier on the 2000 downloaded tweets\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment_classifier = [NBclassifier.classify(extract_features_func(tweet[0])) for tweet in cleanedtestData]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n",
      "                 SENTIMENT RESULTS USING NAIVE BAYES CLASSIFIER                 \n",
      "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n",
      "\n",
      "Positive Sentiment = 0.17%\n",
      "Negative Sentiment = 1.53%\n",
      "Neutral Sentiment = 96.52%\n",
      "Irrelevant Sentiment = 1.78%\n",
      "\n",
      "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n",
      "                                First 10 tweets                                 \n",
      "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'text': '@sarahwittman bussy netflix success', 'label': None},\n",
       " {'text': 'RT @notviking: therapy:\\n\\n- expensive\\n- might not work\\n\\nwatching hot rod (2007) on netflix:\\n\\n- free\\n- will always make you feel better since…',\n",
       "  'label': None},\n",
       " {'text': 'Netflix has bs scary movies.', 'label': None},\n",
       " {'text': \"RT @adamfgoldberg: NOOOOO! Daybreak is one of my favorite shows this year. It's Mad Max set in a John Hughes high school and even has Matth…\",\n",
       "  'label': None},\n",
       " {'text': 'Don’t watch don’t fuck with cats  on Netflix alone....I REPEAT DONT WATCH IT...IT IS FUCKING BIZARRE AND TERRIFYING',\n",
       "  'label': None},\n",
       " {'text': 'RT @lettersforanne: ok so im dead serious about us planning a day where we all just cancel our netflix accounts bc they wont even consider…',\n",
       "  'label': None},\n",
       " {'text': \"RT @hebiancao__: +as a Christian, or from any religion nakakabastos lang and nakakadisappoint &amp; it's not even funny. Inopen nila yung bible…\",\n",
       "  'label': None},\n",
       " {'text': '@MiraSorvino @Rubytea53 @DEADLINE @netflix Ryan Murphy is just the best ever. Thank you for never being #ageist ,… https://t.co/FcyF8vZGpm',\n",
       "  'label': None},\n",
       " {'text': 'Since when tf is Netflix $16.95 ?', 'label': None},\n",
       " {'text': 'Burlesque is on Netflix. 😁', 'label': None}]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('-='*40)\n",
    "print('{:^80}'.format('SENTIMENT RESULTS USING NAIVE BAYES CLASSIFIER'))\n",
    "print('-='*40)\n",
    "print('')\n",
    "\n",
    "\n",
    "if sentiment_classifier.count('positive'):\n",
    "    print('Positive Sentiment = {:.2f}'.format(100*sentiment_classifier.count('positive')/len(sentiment_classifier))+\"%\")\n",
    "          \n",
    "if sentiment_classifier.count('negative'):\n",
    "    print('Negative Sentiment = {:.2f}'.format(100*sentiment_classifier.count('negative')/len(sentiment_classifier))+\"%\")\n",
    "    \n",
    "if sentiment_classifier.count('neutral'):\n",
    "    print('Neutral Sentiment = {:.2f}'.format(100*sentiment_classifier.count('neutral')/len(sentiment_classifier))+\"%\")\n",
    "    \n",
    "if sentiment_classifier.count('irrelevant') or sentiment_classifier.count('irrelevant') == 0:\n",
    "    print('Irrelevant Sentiment = {:.2f}'.format(100*sentiment_classifier.count('irrelevant')/len(sentiment_classifier))+\"%\")\n",
    "    \n",
    "\n",
    "print('')\n",
    "print('-='*40)\n",
    "print('{:^80}'.format('First 10 tweets'))\n",
    "print('-='*40)\n",
    "print('')\n",
    "\n",
    "tweets[0:10]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A simplified approach using TextBlob library\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object of this section is to present a simple and very interest tool that I could discover on my journey to complete this project. I will also perform the same kind of sentiment analysis in a much more simplefied version using a powerful library called TextBlob.\n",
    "\n",
    "TextBlob is an object-oriented NLP text-processing library that is built on the NLTK and pattern NLP libraries and simplifies many of their capabilities.\n",
    "\n",
    "*During my exploration to perform this work I had the chance to learn about this library and I strongly recomend that you dedicate a time to read and understand better the main tasks that TextBlob can perform.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries\n",
    "\n",
    "import pickle\n",
    "import os\n",
    "import tweepy\n",
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Saving/loading the credations to access the tweet API\n",
    "\n",
    "# This structure will save your credentials at the first time you exetute it, after that, erase your\n",
    "# credentials to avoind anyone else to use that\n",
    "if not os.path.exists('secret_twitter_credentials.pkl'):\n",
    "    Twitter={}\n",
    "    Twitter['Consumer Key'] = 'your_cosumer_key_here'\n",
    "    Twitter['Consumer Secret'] = 'your_cosumer_secret_here'\n",
    "    Twitter['Access Token'] = 'your_access_token_here'\n",
    "    Twitter['Access Token Secret'] = 'your_access_token_secret_here'\n",
    "    with open('secret_twitter_credentials.pkl','wb') as f:\n",
    "        pickle.dump(Twitter, f)\n",
    "else:\n",
    "    Twitter=pickle.load(open('secret_twitter_credentials.pkl','rb'))\n",
    "    \n",
    "    \n",
    "# Authenticating and creating an API object\n",
    "\n",
    "auth = tweepy.OAuthHandler(Twitter['Consumer Key'], Twitter['Consumer Secret'])\n",
    "auth.set_access_token(Twitter['Access Token'], Twitter['Access Token Secret'])\n",
    "api = tweepy.API(auth, wait_on_rate_limit=True,wait_on_rate_limit_notify=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that will access the Tweet and get tweets based on a search term, number of tweets, inicial \n",
    "# date and final date. This function will return a list of dictionaries where the key is the tweet text\n",
    "# and the values is a label, in this casa we are considering 'None' in the labor. This format is necessary\n",
    "# for futures aplication\n",
    "\n",
    "\n",
    "def get_tweets2(word='google',number=5, since='2019-11-01', until='2019-11-28'):\n",
    "\n",
    "    search_term = str(input('Hi! What are you looking for today? - '))\n",
    "    num_of_terms = int(input('How many tweets do you want? - '))\n",
    "    since = input('From which date? ex.: 2019-11-01 - ')\n",
    "    until = input('Until which date? ex.: 2019-11-28 - ')\n",
    "\n",
    "    tweets = tweepy.Cursor(api.search,\n",
    "                                #since = f'{since},\n",
    "                                #until = f'{until},\n",
    "                                q=search_term,\n",
    "                                lang='en').items(num_of_terms)                                \n",
    "                                \n",
    "    \n",
    "    positive = 0\n",
    "    negative = 0\n",
    "    neutral = 0\n",
    "\n",
    "    for tweet in tweets:\n",
    "        try:\n",
    "            analysis = TextBlob(tweet.extended_tweet.full_text)\n",
    "        except:\n",
    "            analysis = TextBlob(tweet.text)\n",
    "            \n",
    "            if analysis.sentiment[0] > 0.00:\n",
    "                positive += 1\n",
    "            elif analysis.sentiment[0] < 0.00:\n",
    "                negative += 1\n",
    "            else:\n",
    "                neutral += 1\n",
    "                \n",
    "    print()\n",
    "    print('-=' * 3, 'TOTALS', '-=' * 3)\n",
    "    print('Total Positives:', positive)\n",
    "    print()\n",
    "    print('Total Negatives:', negative)\n",
    "    print()\n",
    "    print('Total Neutrals:', neutral)\n",
    "    print()\n",
    "    print()\n",
    "    print('-=' * 3, 'PERCENTAGE', '-=' * 3)\n",
    "    print(int((positive / num_of_terms) * 100), '% Positives')\n",
    "    print()\n",
    "    print(int((negative / num_of_terms) * 100), '% Negatives')\n",
    "    print()\n",
    "    print(int((neutral / num_of_terms) * 100), '% Neutrals')\n",
    "    print()\n",
    "    \n",
    "    return positive, negative, neutral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi! What are you looking for today? - Netflix\n",
      "How many tweets do you want? - 200\n",
      "From which date? ex.: 2019-11-01 - 2019-12-18\n",
      "Until which date? ex.: 2019-11-28 - 2019-12-18\n",
      "\n",
      "-=-=-= TOTALS -=-=-=\n",
      "Total Positives: 96\n",
      "\n",
      "Total Negatives: 17\n",
      "\n",
      "Total Neutrals: 87\n",
      "\n",
      "\n",
      "-=-=-= PERCENTAGE -=-=-=\n",
      "48 % Positives\n",
      "\n",
      "8 % Negatives\n",
      "\n",
      "43 % Neutrals\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tweets2 = get_tweets2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adjusting the TextBlob to perform the sentiment analysis using a Navie Bayes Analyser\n",
    "\n",
    "By default, a TextBlob and the Sentences and Words you get from it determine sentiment using a PatternAnalyzer, which uses the same sentiment analysis techniques as in the Pattern library. The TextBlob library also comes with a NaiveBayesAnalyzer9 (module text-blob.sentiments), **which was trained on a database of movie reviews**.\n",
    "\n",
    "Considering that we can not re-train the algorithm in the TextBlob, it is important to understand that the performance compromised since the train data was not related with the topic of the search term that will be considered on this analysis, a term realated to a tech company for the sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob.sentiments import NaiveBayesAnalyzer\n",
    "\n",
    "\n",
    "# Recommendation!!!\n",
    "\n",
    "# The sentiment analysis using TextBlob NaiveBayesAnalyzer presented a low performance in terms of speed,\n",
    "# basically it takes long time to process the analysis. Request a max of 20 tweets in order to be able to\n",
    "# quickly have a return and evaluate the result. If you have more time and a powerful machine you maybe want\n",
    "# to try a larger tweet request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tweets3(word='google',number=5, since='2019-11-01', until='2019-11-28'):\n",
    "\n",
    "    search_term = str(input('Hi! What are you looking for today? - '))\n",
    "    num_of_terms = int(input('How many tweets do you want? - '))\n",
    "    since = input('From which date? ex.: 2019-11-01 - ')\n",
    "    until = input('Until which date? ex.: 2019-11-28 - ')\n",
    "\n",
    "    tweets = tweepy.Cursor(api.search,\n",
    "                                #since = f'{since},\n",
    "                                #until = f'{until},\n",
    "                                q=search_term,\n",
    "                                lang='en').items(num_of_terms)                                \n",
    "                                \n",
    "    \n",
    "    positive = 0\n",
    "    negative = 0\n",
    "\n",
    "    for tweet in tweets:\n",
    "        \n",
    "        blob = TextBlob(tweet.text, analyzer=NaiveBayesAnalyzer())\n",
    "\n",
    "        if blob.sentiment.p_pos > blob.sentiment.p_neg:\n",
    "            positive += 1\n",
    "        else:\n",
    "            blob.sentiment.p_pos < blob.sentiment.p_neg\n",
    "            negative += 1\n",
    "\n",
    "                \n",
    "    print()\n",
    "    print('-=' * 3, 'TOTALS', '-=' * 3)\n",
    "    print('Total Positives:', positive)\n",
    "    print()\n",
    "    print('Total Negatives:', negative)\n",
    "    print()\n",
    "    print()\n",
    "    print('-=' * 3, 'PERCENTAGE', '-=' * 3)\n",
    "    print(int((positive / num_of_terms) * 100), '% Positives')\n",
    "    print()\n",
    "    print(int((negative / num_of_terms) * 100), '% Negatives')\n",
    "    print()\n",
    "    \n",
    "    return positive, negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi! What are you looking for today? - Netflix\n",
      "How many tweets do you want? - 20\n",
      "From which date? ex.: 2019-11-01 - 2019-12-19\n",
      "Until which date? ex.: 2019-11-28 - 2019-12-19\n",
      "\n",
      "-=-=-= TOTALS -=-=-=\n",
      "Total Positives: 14\n",
      "\n",
      "Total Negatives: 6\n",
      "\n",
      "\n",
      "-=-=-= PERCENTAGE -=-=-=\n",
      "70 % Positives\n",
      "\n",
      "30 % Negatives\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tweets3 = get_tweets3()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}