Chapter 10 - Natural Language Processing/NLP-MasterTemplate.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "83708667-4fdc-1563-7b3a-06b6575d2865"
   },
   "source": [
    "# NLP-Master Template\n",
    "\n",
    "How do you work through various NLP related tasks using different Python packages for NLP ? \n",
    "\n",
    "In this jupyter notebook you will work through the above mentioned steps of Data-preprocessing and feature extraction using different libraries for NLP.\n",
    "\n",
    "There are many python packages for NLP out there, but we can cover the important bases once we master a handful of them. In this jupyter notebook we will describe following Python NLP libraries we’ve found to be the most useful and will be using in the case studies. \n",
    "\n",
    "   * NLTK Book: https://www.nltk.org/    \n",
    "   * TextBlob : https://textblob.readthedocs.io/en/dev/index.html    \n",
    "   * Spacy : https://spacy.io/\n",
    "\n",
    "In additional to these there are few other libraries such as Gensim and Stanford’s CoreNLP that can be explored as well.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [1. Loading Libraries and Packages](#1)\n",
    "* [2. Data Preprocessing](#2)\n",
    "    * [2.1. Tokenization](#2.1)    \n",
    "    * [2.2. Removing Stop Words](#2.2)\n",
    "    * [2.3. Stemming](#2.3)\n",
    "    * [2.4. Lemmetization](#2.4)\n",
    "    * [2.5. PoS tagging](#2.5)\n",
    "    * [2.6. Name Entity Recognition](#2.6)  \n",
    "* [3. Feature Representation](#3)\n",
    "    * [3.1. Bag-of Words](#3.1)    \n",
    "    * [3.2. TF-IDF](#3.2)\n",
    "    * [3.3. Word Embedding](#3.3)\n",
    "* [4. Inference](#4)\n",
    "    * [4.1. Supervised (Example Naive Bayes)](#4.1)    \n",
    "    * [4.2. Unsupervised (Example LDA)](#4.2)\n",
    "* [5. NLP Recipies](#5)\n",
    "    * [5.1. Sentiment Analysis](#5.1)\n",
    "    * [5.2. Words and Sentences similarity](#5.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "# 1. Load libraries and Packages "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a first step we check if the additional packages needed are present, if not install them. These are checked separately as they aren't included in requirement.txt as they aren't used for all case studies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pkg_resources\n",
    "import pip\n",
    "import sys\n",
    "installedPackages = {pkg.key for pkg in pkg_resources.working_set}\n",
    "required = {'nltk', 'spacy', 'textblob','gensim' }\n",
    "missing = required - installedPackages\n",
    "if missing:\n",
    "    !pip install nltk==3.4\n",
    "    !pip install textblob==0.15.3\n",
    "    !pip install gensim==3.8.2    \n",
    "    !pip install -U SpaCy==2.2.0\n",
    "    !python -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For NLTK, import NLTK and run nltk.download().This will open the NLTK downloader from where you can choose the corpora and models to download. You can also download all packages at once. \n",
    "\n",
    "For spacy, download spacy core model by running the command \"python -m spacy download en_core_web_sm\". Once downloaded, load this int he jupyter notebook by using spacy.load()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\tatsa\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import nltk.data\n",
    "nltk.download('punkt')\n",
    "from textblob import TextBlob\n",
    "import spacy\n",
    "#Run the command python -m spacy download en_core_web_sm to download this\n",
    "import en_core_web_lg\n",
    "nlp = en_core_web_lg.load()\n",
    "\n",
    "#Other helper packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#Download nltk data lobraries. All can be downloaded by using nltk.download('all')\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Diable the warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "# 2. Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.1'></a>\n",
    "## 2.1. Tokenization\n",
    "Tokenization is just the term used to describe the process of converting the normal text strings into a list of tokens i.e words that we actually want. Sentence tokenizer can be used to find the list of sentences and Word tokenizer can be used to find the list of words in strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "_cell_guid": "5d8fee34-f454-2642-8b06-ed719f0317e1"
   },
   "outputs": [],
   "source": [
    "#Text to tokenize\n",
    "text = \"This is a tokenize test\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The NLTK data package includes a pre-trained Punkt tokenizer for English, which has alreayd been loaded before "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['This', 'is', 'a', 'tokenize', 'test']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "word_tokenize(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['This', 'is', 'a', 'tokenize', 'test'])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text).words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.2'></a>\n",
    "## 2.2. Stop Words Removal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, some extremely common words which would appear to be of little value in helping select documents matching a user need are excluded from the vocabulary entirely. These words are called stop words. The code for removing stop words using SpaCy library is shown below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first load the language model and store it in the stop_words variable. The stopwords.words('english') is a set of default stop words for English language model in NLTK. Next, we simply iterate through each word in the input text and if the word exists in the stop word set of the NLTK language model, the word is removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"S&P and NASDAQ are the two most popular indices in US\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['S', '&', 'P', 'NASDAQ', 'two', 'popular', 'indices', 'US']\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize \n",
    "stop_words = set(stopwords.words('english'))\n",
    "text_tokens = word_tokenize(text)\n",
    "tokens_without_sw= [word for word in text_tokens if not word in stop_words]\n",
    "\n",
    "print(tokens_without_sw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see some of the stop words such as \"are\", \"of\", \"most\" etc are removed from the sentence. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.3'></a>\n",
    "## 2.3. Stemming\n",
    "Stemming is the process of reducing inflected (or sometimes derived) words to their stem, base or root form — generally a written word form. Example if we were to stem the following words: “Stems”, “Stemming”, “Stemmed”, “and Stemtization”, the result would be a single word “stem”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"It's a Stemming testing\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_text = word_tokenize(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Stemming', 'stem'), ('testing', 'test')]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize stemmer.\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "# Stem each word.\n",
    "[(word, stemmer.stem(word)) for i, word in enumerate(parsed_text) \n",
    " if word.lower() != stemmer.stem(parsed_text[i])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.4'></a>\n",
    "## 2.4. Lemmetization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A slight variant of stemming is lemmatization. The major difference between these is, that, stemming can often create non-existent words, whereas lemmas are actual words. So, your root stem, meaning the word you end up with, is not something you can just look up in a dictionary, but you can look up a lemma. Examples of Lemmatization are that “run” is a base form for words like “running” or “ran” or that the word “better” and “good” are in the same lemma so they are considered the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"This world has a lot of faces \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['This', 'world', 'has', 'a', 'lot', 'of', 'faces'])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textblob import Word\n",
    "parsed_data= TextBlob(text).words\n",
    "parsed_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('has', 'ha'), ('faces', 'face')]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(word, word.lemmatize()) for i, word in enumerate(parsed_data) \n",
    " if word != parsed_data[i].lemmatize()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.5'></a>\n",
    "## 2.5. POS Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, some extremely common words which would appear to be of little value in helping select documents matching a user need are excluded from the vocabulary entirely. These words are called stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Google is looking at buying U.K. startup for $1 billion'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Google', 'NNP'),\n",
       " ('is', 'VBZ'),\n",
       " ('looking', 'VBG'),\n",
       " ('at', 'IN'),\n",
       " ('buying', 'VBG'),\n",
       " ('U.K.', 'NNP'),\n",
       " ('startup', 'NN'),\n",
       " ('for', 'IN'),\n",
       " ('1', 'CD'),\n",
       " ('billion', 'CD')]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text).tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spacy- doing all at ones "
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1oAAADKCAYAAAC8LDFAAAAgAElEQVR4Ae3dCXwc1YHn8cxmZpPJzE6ymSQk2qx2Nposk2RYZ8kEspuLyWFm1sskHzAJEDLBxAHPcA6ELIsDwYy5YhjABCeObwsbYSGMjY1tydiWbMuXbPCp+7IsWz4kG9nybb/9vNf9qkutavVR9aqqq3/9+fRHUlfX9X1PVe/fr/rV+wQPBBBAAAEEEEAAAQQQQAABTwXe5+nSWBgCCCCAAAIIIIAAAggggIAgaFEJEEAAAQQQQAABBBBAAAGPBQhaHoOyOAQQQAABBBBAAAEEEECAoEUdQAABBBBAAAEEEEAAAQQ8FiBoeQzK4hBAAAEEEEAAAQQQQAABghZ1AAEEEEAAAQQQQAABBBDwWICg5TEoi0MAAQQQQAABBBBAAAEECFrUAQQQQAABBBBAAAEEEEDAYwGClsegLA4BBBBAAAEEEEAAAQQQIGhRBxBAAAEEEEAAAQQQQAABjwUIWh6DsjgEEEAAAQQQQAABBBBAgKBFHUAAAQQQQAABBBBAAAEEPBYgaHkMyuIQQAABBBBAAAEEEEAAAYIWdQABBBBAAAEEEEAAAQQQ8FiAoOUxKItDAAEEEEAAAQQQQAABBAha1AEEEEAAAQQQQAABBBBAwGMBgpbHoCwOAQQQQAABBBBAAAEEECBoUQcQQAABBBBAAAEEEEAAAY8FCFoeg7I4BBBAAAEEEEAAAQQQQICgRR1AAAEEEEAAAQQQQAABBDwWIGh5DMriEEAAAQQQQAABBBBAAAGCFnUAAQQQQAABBBBAAAEEEPBYgKDlMSiLQwABBBBAAAEEEEAAAQQIWtQBBBBAAAEEEEAAAQQQQMBjAYKWx6AsDgEEEEAAAQQQQAABBBAgaFEHEEAAAQQQQAABBBBAAAGPBQhaHoOyOAQQQAABBBBAAAEEEECAoEUdQAABBBBAAAEEEEAAAQQ8FiBoeQzK4hBAAIF8E5i4/ZC45u1O8YmyevHHpbsj/bxySau4e9N+sbSrP9+Kie1FAAEEEMgzAYJWnhUYm4sAAgh4JbDn2Glx+eIWFawuKWsQX1rUKr6+pD3Sz0srmq0gOWbdPq8oWQ4CCCCAAAJDBAhaQ0h4AQEEECgMAd179UTdEdHUc75gnjN2vSc+U96kAtf8tmOFUdjsJQIIIICA7wIELd/JWSECCCAQvMAdG/eroPHT6v0FE7CSw+RflDcqgx19p4IvELYAAQQQQCByAgStyBUpO4QAAgikF7j09SZR8lpTwYYsGboe3nRYBa3Je46kB+MdCCCAAAIIZClA0MoSjLcjgAAC+S7Q2n9GBYwfvr2voIPWstaTyuEna/muVr7XabYfAQQQCKMAQSuMpcI2IYAAAgYFqntOqIBxX+3Bgg5asldLfk/t6sp2g9osGgEEEECgUAUIWoVa8uw3AggUrABBKzHwB0GrYP8N2HEEEEDAuABByzgxK0AAAQTCJUDQImiFq0ayNQgggEA0BQha0SxX9goBBBBIKUDQImilrBxMQAABBBDwTICg5RklC0IAAQTyQ4CgRdDKj5rKViKAAAL5LUDQyu/yY+sRQACBrAWCDVpbxMPFJaJIP8eUiY0B3iyZ72hlXX2YAQEEEEAgQwGCVoZQvA0BBBCIikCwQUv3JsUDF0ErKtWK/UAAAQQQSBIgaCWB8CcCCCAQdQGClg57DO8e9brO/iGAAAJBChC0gtRn3QgggEAAAmaDVqyn6oY5ZbZLBG8VU3Ykwo28f1VTj3OP1sY5t4qi4sfFFPUzfonh41uM3e+LSwcDqICsEgEEECgQAYJWgRQ0u4kAAghoAT+ClvwO1sMrbYFqSFgaLmiViCJ9SeHKx9X3uWLLSg5r7v8maOlawU8EEEAAAa8FCFpei7I8BBBAIOQCvgQtK1g5B6rhe7RsPWA7ysQNcuAMa3nuw1WsRy22HIJWyCsrm4cAAgjksQBBK48LrxA2fW7LUXF1ZYd6yt+TH9v7Tnk2XS4r+WFfv9vpycuWf9uX7zT951sOWPvnZrpcjtND27qd7lQ2cn16+U7Tsyk7J3v7/E7T7bZupzvZmS4bvXy3ZeM0f14FLX2JoY9BS9vL+uv00NOdbO313u10p/8b+/Kdptv/L9xOd/t/k27+4WzT2aebnso+yLIzXTb25aezdzs9l7LrOH5WyCcPBApJgKBVSKUd0n2Vjb7barvFsTPnh2zhxO2HhPzEWT7l78kP3WD0YrpcVvLDvn6305OXLf+2L99p+sjKdmv/3UyXy3F6aFu3053KRq5PL99pejZl52Rvn99put3W7XQnO9Nlo5fvtmyc5td299UeNPDdp3gPlhWMvOnRumFOj4FtdR4MQ9vL+uv00NOdbO313u10p/8b+/KdpuuyzddjorZNZ59ueip7vfxU0+Vy5dPt9CDKxl72bo956eYf7v8iXdlcuaTV8XzvtExeQyDfBQha+V6Ceb799saw06ev8jV5wpNPp+nyUzmvpqf6hE8v3+10p6Ky75/TdPnpq16/m+mpPt3Vy3Y73als5Pbq5TtNz6bsnOzt8ztNt9u6ne7GXho4PXTZprJPN13bppp/uOm6QZYPQWvh43JADNulhB7fc0s2Cq9OKiNtn2vZDWdv/79IV3ZO/zf2+Z2m2/8v3E53+3+Tbv7h/i/S2aebnspWl22q6SbLznTZ2Jefzt7t9FzKTofYS19vcpqd1xCIpABBK5LFmj87JS8j0AffVCe+/NkbthSB/BAINGjp71zpGxbrn/HBL2KjDtpuaGwwZMnvajkFrfwoRbYSgfwSWLy3X/xsfbd4sb43vzacrUXAhQBBywUes3ojIA+6TpcpeLN0loIAAskCZoOWu8EqYkHLXA+WfSAMglZyzeBvBIIRkF8dkEGMBwJREyBoRa1E2R8EEEAgjQBBKxEG6dFKU1mYjIAPAnJwE/m/6HS5qw+rZxUIGBMgaBmjZcF2AflplfwCLAdRuwq/IxCMAEGLoBVMzWOtCAwV0McjGbQuKatnoIyhRLySxwIErTwuvHzadDmqoDyIyieXB+RTybGtURTQDRszg2EkQkzyZXph/Fsek5IHw4himbNPCIRZQH4Ie8WSVuE0SEeYt5ttQyCdAEErnRDTPRHQowvKA6nTMO6erISFIIBARgIErUQYJGhlVGV4EwIIIIBADgIErRzQmCU3Adm449Oq3OyYCwEvBQhaBC0v6xPLQgABBBBwFiBoObvwKgIIIBBZAYIWQSuylZsdQwABBEIkQNAKUWGwKQgggIAfAgQtgpYf9Yx1IJCrgPyKQWnrUb7TnSsg84VGgKAVmqJgQxBAAAF/BAhaBC1/ahprQSA3gU++2qAGz7p+9d7cFsBcCIREgKAVkoKI4mbI0QXlJ1IMfhHF0mWf8llAfldSDgLx0+r9IowjAfq1Tes7TyuH62jM5XN1ZtsjKCADljxGySdtiAgWcAHtEkGrgArb712V982SB8lLX2/ye9WsDwEE0gjIT4wvrWgu6KD1q82H1TFKjorKAwEEwiMgP6iV/5ey950HAvksQNDK59IL+bbrT6Po+g95QbF5BSkwOv6JcSH3av1FeaMKWlXdxwuyDrDTCCCAAAJmBQhaZn0Leunykyj5qRSfSBV0NWDnQyrQdeKsChnyA5En6o4UVM/WjF3vic+UN6n9n9LQG9ISYrMQQAABBPJdgKCV7yXI9iOAAAI5Crzbe0p8dH69ChyXlDWILy1qFV9f0h7pp7xcUve2892sHCsOsyGAAAIIZCRA0MqIiTchgAAC0RQ4eua8uLG6S1z2RrP4yPw9VgjRYSRqP/9LeaP4xrI28fzuI9EsUPYKAQQQQCA0AgSt0BQFG4IAAggggAACCCAgBeRgGPI5t+UoIAjkrQBBK2+Ljg1HAAEEEEAAAQSiKaB700dWtkdzB9mrghAgaBVEMfu/kx3Hz1qfRjEYhv/+rBEBBBBAAIF8FiBo5XPpse1agKClJfjpqYAMV/ogyT1qPKVlYQgggAACCEReQLYj5FPeYJ0HAvkqQNDK15IL+XYTtEJeQGweAggggAACCCCAgFEBgpZR3sJd+LEz59UnUTJwycsIeSCAAAIIIIAAAgggUEgCBK1CKm32FQEEEEAAAQQQQAABBHwRIGj5wsxKEEAAAQQQQAABBBBAoJAECFqFVNrsKwIIIIAAAgggkAcC3EcrDwqJTUwrQNBKS8QbEEAAAQQQQAABBPwU0CMXcx8tP9VZl9cCBC2vRVmeEuA+WlQEBBBAAAEEEMhVgKCVqxzzhUkg66C1a9cuwTN/DIKqbAzvHpQ860UAAQQQQCD/BWQ7Qj7Dch+t+s2tYue6Rp55YnB64Ewo/gmyDlozZ84UU6dO5ZkHBnV1dYFVMoJWYPSsGAEEEEAAAQQ8FpBBa8r983nmiUFeB63y8nKxZ88eniE2kGE4yKDl8fGNxSGAAAIIIIAAAoEJ6KC1buE2sXHJdp4hNah4oVKF4bwPWgMDA4JnOA1aW1tVj2O+Bq39a2vE5l/9kqePBjtefN73k1fnkqsET/MGjXM+IvSz/Y0rMDdU7zoWfxVnQ7bJx4nmeZ+yrJOn8beZY8rR+t/7fo4I2wrtQatz9wHBM5wGBC0CmvGAmu9BSx5c199/r1g++vs8fTQ41tTo63ntxN63RP209/HEgDpAHaAOhLwONM39c1/PD2FcGUErnMEqOfAStAhaBK0MjqBHGxtUyHpnwsPi9LY6ngYNDi5aqKxlL6Lfj84l31QNrJN75gixv46nIYPOisuV8ymcjdcx+eFB6/xi4+sp9P+XnsrrVJ3uq30Aa0PHDV3Hjqy9U1kfrnvY11NE2O6jRdAiaOXyD5DTYBjyO1pcNhjOywZluUShR0tW5m1PP6ECwIGKcoKWwaAlg+y2R8bHrGvX53IcyXmeM0fr1Qm8a9HXaCwZbCydaXot5rz4mzgbdJYN04OrfqKsj9T8M9aGrRtm/JFomv1n4nznaqwNWzeXfkLV63MnD+R8vM92xrAN707QImhlW4fl+wlaEexhC0PQ8uI+Wqf7+lTjf+P99xC0DAetY6tWKuuaO8flchxxNc/+6lvUCfy9uidoLBlsLO1fdo1y7t/2a5wNOsuw1Tjzg6JhxgfE2dalWBu0PrrxIVWnD668CWeDzrJOH9vymLLev+Ynro732cxM0MqPYJN86V7Qf3PpYASDTdh698IQtLwa3r1+1gwVADpmzSBsGQ5buyc9pazb31yUzbnQ9XsvnBtQJ/C2Vz5DY8lgY+lC1zrl3L7gr3A26GxvlB5YcR3Whq2bSz+u6vWp+pexNmzdUX5ZzPrQJtfH/XxcAD1a+RH8CFoELeOXXUYpaMmDsRwUo/pnYwhahoPWqa1bROUN14vKG0aLi+fP+XoePLx1gjqBH6m5g8aSwcbS4erblXPv+vtwNugsw1ZL6SXK+uTumVgbtD7dWKacuxZfhbNBZ1mnB3b+Tll3Lv2Wr+eHsKyMoEXQyqUuculgBINfGIJWLpUx1TwdS5eosNXwwnOELcNhq2XqFGUtexL9fjTM+pBomPkBca59OQ0mgw2mhpkfFE1zPiwu7K3B2aDz6cZXYwGA7x8ar2cd5V9Q1v1bnza+Lhk4Cvm5b8m3Y9Ztr/l9igh8fQQtglYulZCgRdDKpd74Ps/qn90qKn94nXivehVhy3DYqr7tVhW2Bnr8+9KzrFDHmuaqE/iBFd8v6IaM6UbcsS2x3sODb/8IZ8ON5vYFn1d1mu8fmg0nF7vWK+e2Vz9LnTZcp880L1TWrQs+63s7IOgVErQIWrnUQYIWQSuXeuP7PAe3bFaN/+0THyVoGQ5aXfNLlfU7k572vZzbXot9Mj2waxoNJoMNprayz6jGkux1MR3sCnn5F/ZWxwJA2V/ibLA+yzrWvfTvlHXvuruxNmzds/LGmPXO53w/RwS5QoIWQSuX+kfQImjlUm8CmWfjLx9UAeDg4oWELcNha2P8htG9O3f4WtYD+9eoE3jnwitoLBlsLA3s+r1y3rfkOzgbdJYBYN+b34oHgHuwNmzdMP39onHmH4sLnWuwNmh9sXujaJj+h0JehizERWPniKsrO4R8/nyLv1dXpNohghZBK1XdGO51ghZBa7j6Eapp/e1tKmht/sV9BC3DQevwksXKesODD/heB7pWjFIN02Obf0VjyWBjSd67TN5c9/i7L+Bs0FmGrfrpfyAaZ/2pkCM/FnIPn+l9P1ITu7HugcrROBuu073r7lHHj4Mb/sXYOYLh3fMj2AQ9nHvy+hl1MILBhuHdhx5nt/edUp9EyU+j5rYcHfqGHF/ZMfk5FQD2ls4hbBkOW+88+rCy7l69KsfSym22s/1t6gTeMu8/0Vgy2Fg627JYOcshm003gAt9+YfiNzHuqboBa4N1WtazpjkfUfX6VH0p1oatW+YVKeuzxztyO9inmYugRdBKDlGZ/E3QImgxvHuag+twk88eP64a/zXjxhK0DAet/rXVynr12DHDFYmRaT21sU+mD68ZS2PJYGOpp+qHqqF0dMODOBt0lgGgcdaHlPWZpnKsDVqf2DFFOe9943/ibNBZ1un3tj6prPetHG3kPBC2hXLpYH4EP4IWQYug5fLo2Vw2XwWAppcmE7YMh609/zZJWbeUv+qy1LKc/eJ5dQKXl7ada6+kwWSqwdS9WTk3l34CY1PG8eUe2/xorFH65t9ibdi6raxEWcsgUOi9qab3v6Pii8p64EBNlgf5/Hs7QYuglUut5TtaEQx+UbuPllPFljfVrbrpB+Lk5o2ELcNhq+pHP1Rh6/ypU05FYey13h3PqBN491v/m8aSwYZp77p7lfOhVbfgbNBZNnhbXv6ksj6x/UWsDVrLwTDkhzSt84txNugs6/TJ3bOUdceiK42dC8KyYIIWQSuXukjQImjlUm8Cn6drZaVq/O988l8JWoaDVtv0qcp6129f8r3cm17+hDqJn9w9gwaTwQZT09yPKmd6D83e7+l0w3zl3L7gc9Rng/VZBoCuRV9X1kdq/hlrw9bdS0cq6/eaX/b9HOHnCglaBK1c6lt0glZrqbiluEQUpX0+JtZ5Fa46V4vJz5SLDq+W59FyCqFHS1b2tXffoQJA38oVhC3DYUt+J2756O+L4117cznO5DxPf+ur6gTe8doIGksGG0v9W59Szt1v/T3OBp1lANA3MT668SGsDVvXT/93Qg75fmFvDdYGrc+1L1fHj+Z5RTkf6/NhxrwJWtXTxE2ObeHPif91za1iwktVYvfu/AhNmQx+kfwevqPlUZgYMtJfT62YPfkpMdl63hWraDffZXtNTl8kGj3ahnXPlIii20oJWgEdIQ+/+45q/G/5fw8QtAwHre7yMmVdN3GC76UtL0mRlwEd2zyBxpLBxlJH+V8r51N75uBs0PnivtpYo3TuR3E26CxD7cG3f6ys9y/7B6xNW6/6R2V95N0nPDtHcB+tHMOQDlrXjhNPTngs8XxwnLhp5AjVIXHZmJdEdV2Oyw95SCNoeRRyhgStIcutFZNkon+m1tjgE4UatOrq6kR/f79nB1M3C5INf9nTcqCinLBlOGxteuA+ZX34nW1uiizreU8e3BBvmH6MxpLBxtLJPbHvWnRWXI6zQWcZAPa9+beqTh9afSvWhq0bZ/2Jsj7VMA9rw9YNMz+grC+e9+b7vH4N737mzBkh2zXy53CPvOvRGl8lknt75N8bJt8sLisuESMfqRLNIQ9NTtuf7jWC1pBANGAoCA0TtPo7xcrJt4tRX4tdanjFNbeLSYt2ij69bVufF6OKS8R3nqlNvDYwILZM+a4oKv6umLSpWZTdNvgyxVsWdBraj+x9TF862NjYKKZOnSpWr16dMnCZuo9W8kHwxL4u1fhfd8ftBC3DQat3xbKY9b13JReD8b+7344NQ85w72a/Q9S99LuqodS/9WkapYYbpfXT36+sz3esxNqg9Xtb/lU58wGC2WOH/AChr/YXyvrAun/y5JzgV9CSGyvbM7NmzRo2cEUlaHXubhZzxso27DjxymZbr9bmajFn/K1i5Fdi7du/GXmrmDBtw5Aw1rx8rpgw9tvib9TliSPEyLHjxZzlnY6hLl0oMjGdoKXDjPGfKYJWX62YfP3nRdHXbhSTFiwXK1ctF2XP3CiuKC4Ro6xg1Se2PBcLVZO3xoNOPHyNmrJVDAz0iY5Nq8XU+0tE0Q2PiYWrVostrX0FE7TkQWnevHkqbKUKXNU9J4Q+SE7cfsiTg26qhez63RQVANpnTs86bHW9MFqMeWFllvPNF8/Gr39+9pW6LOfN/v21D8ZD/YPzja/rdJqwtn3io8q6q2pFquIw8vq5gW51ApeXEJpqmNZNLBEvVJpvjMgGSVif59qWKWduFm2+jPRlbXwvzrx167xPq3rNBwg+WM8vVtZnjjUaOReYWqi8Ske2Z+QzVeCKTtA6IJpLH1CXED60IB606qrEhJElomjE98RDL70uKubPFc/e9vexnq/xiZ6v5orxYmRxibjsmnHi2WlLRMW058Tt13xOFI24WcyptoW2AHvKCFrGA5buAXIOWo2l14qiy8eLpT36fbGffVXjxYjia8Xs+vjrfbVi0jUloujmGaKxf6uYrH+3bX+hXjooD3S6V0sfmJIDl59B6/zp06rx//aPb8wqiOgAQ9DKPPwNbKxV1it/fKPn57v29vaUPaRyZQc3PaBO4N1Lr/Y4qKwUr4+JhdlCD1oyAOoAIId9D2sgjMp2WZe11ZdibfADiLMti2IfILx8Cc4GneX/5fF3n1PWXctHeX6OML1A2atlb9MkB64oBa3ON58SVxWXiJte2q16oqomfE0UFd8spq0aHJaqf/09UVT8PfGbKvn6NvGba0tE0bUviQ32IFX3uvj5V74sbnpuQyh6tQhatqCS/ntWg8NQdu93ClpNouxWOYDFU2LpqtVipf1Z8Zi4rrhE2C8B7Nv0lBhV/Hlx3fWyd+sWUdY0eHvCHrTkQaO7u9vYc+7cuYMOSvoANdwlhaYOlK2vLVABoP75ZzMOW/kStNL1Mvk9vXHyc8q66ZV5nhan/kRxuPojRxGTvVry3i3eNbYJWsmWerQ20b3ZQ2fzn6Yn70fY/z666WFVnzsr/gfOhgOAvHRQHjuOrL0Ta9PWC7+srE/sq3J9jjDZhklednNzs2ObRgeunRsaxZT754t1C7eFIlCkvOxOD4aR4jtaar74e2JBq1o8eVWJKLrv9SGXCXZumCtuLy4R//DsBtG5bq4a3VuHs5Trt4ewAH4naAUatOLhy3HYy9gn2iOesw+e0SfWPfMN1b16S2nTkEsDwx60dPAJ6qf8cqmfj7dv+bEKACfWr00TtlaKBTfGL8ez6sJosaBK9+wMnT74EkGHSwdfuSd+awH7cuqEDnOx2w4MniYvW1Sv3/hr0WXNL7dr8PusZcQvHbT+trY9vi9yOdalf4lttNZhTbNt14P3WJdBFmVxaaLsPZSDkJz1eFAU+yeKToGrb9dkdQJvL/+CR42l+eKFZMfie0RdvCEiLyccdMuIMb8WBwc1UhIhTb5v7OyVwppn4vzENr77azHWWs9o8fq7ifUO6kUb9L4SUWRfxn7bPBN1fTNzuWNfbaz3sKfqhsQ+DNrv7EKT3cT6XXoM8axL+Gkv+3ssn9HihYnx/59i6Sm3J+Gjy2yQrWOZ6nlj+3NwdnyZY+4RL8R7OZ220csA11J6iarT/dsmeWKd+bYlvF6oHFyPB9e7mM2gcrPMg3PLfD9j2yjfXz/tD5T1ha51PlsntiGX7c63eU43vqKc2youc90EqK6udgw/QbRrZkyfIX77q7nRCFpvPKYuC4yFpioxQYepIcEoNq1Ihrb4PBMqBvd6hS1wEbTCELQyHolwp5h9c7yhJS8hTNr2sAetxYsXiy1bthh72r+nZT/oydflpYV+P7rXrFaN/3cnPOIiaCUFFN3YKy6xfZcr8R4VwGwhKRHIhoY13fjT77GClm0d+j1FxfeI2ngwsoJVpkGr6tdiTKbLtL0vm0soO+bMVNY7Xnze02LWvVr2+pQcuFrKPqNO4vJL7u4bIInGpt1eBq3BDUtb4LKFn5Tvka7W+5zWkVieFQYqE+EpsS32MOK0nMFBwb1HokEov6clewDkZVdulzusky1IpXyftrSCVsJP/q/UOYQsbWj57k8KE7a6r99jBS3btEQ5JmzcetjnP1U/Vzm3vPxJ18725ab/3ak+JVzlhwaxZYTTLf3+DS2v/cu/p6wPLP++z9ZDtyWX7c+neeSQ+vL4caxxhqtzhMk2TPKyN27cKKZPn+4Y7GR7asOqusj0aO2eOU4UFX9OxEJThkGrYrz64JGglV2Vjs4Ni5NC0MCA06WDO8XsG+QAFkNDk9NliY2zrxVFxdeKqYumqHty3TR756BerbAHLZM9SrLL3d4Ylr8HFbDsVb42PgT5kWVL0oStRK+OPWBYocYWdE5bQUr3NNmC1guJUGNfjjWPvZdJLyf+mj1o6fBlzVdcIvRr1jY59DjZl6HXr9+v/5aXFia/pv9O7j3L5jLEdXeMU2Fr747tore317Pn8uXLh9QtWb904Dre8YY6gTfN/jOPGkuJhqRucCd6RxK9W1YjXAcDW6Pfms/2mm6gW/PZesoSr+keKadtSH7N1jDW2+Cilyldo+34O88q567F33TtnAhQtmBoBUv9mt6/zMy1r9oPyz0xr7VOHdL0+ux2Sa8NLRd/GsjtCz6nrHvX3+faOl25JqZrb3uYT9Q5qycvyUjNn/RaUG6Jfcm8nPQQ5LLXJZf5mScz6wuda1SdbpzzYfspOqvf/b6PlmwzJbdrZMCS7R35iM53tOKjDo54QFS8I3un0l86eNPkbcNcOrhbzBkzQox80OHSwyE9ZOZ7w+jRGhKIBn/vySnw5PaaU9AaEGowjOJviIk1g0cJ7Kt5THzn8ivFxKqeWJiqn6HCVWyUQT20u22wjIEBUchBSx589AEpDARmqMYAABoSSURBVAFLH717d+1Ujf+N99+TQ9CyBahBowkmeqdi4SXxPv2peZE9UNmCjTXd/gl5PMRZIWnQvIllpw1aOrjJZVshLDG/47qTe8Ws+fRlk5n/7FlYoazn/uY3Vl3QdcLkz/Xr14vOJVepk/jhNbd50FhKNC6twGQPMFYjPv5pv26oJzU2dQMsuYGv/070EMiGSqKRq9aZvI5B9SV2SeKQeezbaOj3zte/pJwHdk515awNBoUjWw/TYJs6IZI9tLn1ug5nutGX8IzV+0TgGlIuSbb291uBQa/PkKveJvtP+em/fNpfM/t7wsxe75MNrLILqVu2Rn3r71fOsm5nOy/v1/9vmf2Ux2dZpw9vfVSforP6qUcuHlnZntV8ubxZ3kNLfhdLn7vsAUsvLypBy7qP1sRq67tmww+G8W3x7NJhBsNYNS3WGREfWCPoSwkJWgEHrYHD8dEEi78obplcHhveffJd4rrLS8SIO0vjlwfGLxm85imxpc82CqEc+tJ2CeGWKfL7W9eKSYsKa3h33Zs1XMDy6z5a+gCof+oercNvvekiaOmeq0To0D1AKYOWrQfK3oPkGHbi38FyF7RsgSpFUHNcd/y9en8SAS2xr5n2aukerbeXLhU1NTWePSsqKqyTnT7pyZ8zZ85U9zjpbVmoTt6Ns/7Uo4aSU9BKvDbEMd4IT26Q6kaY9brqSUksZ3CYSLyeSdCKBRTnhrFer9c/rR6tRV937awb64MNEpdnxl5PmKQyTwSw5KBVJ4QOvkmBQK9Tb8OQZav3x5ZnlZ3PQat9wV+pOu3vSI/O9SnZIMxuudR53aPFDYwzC0u5GMt5vOjRkgFLPn++5YA+xRv7qXuznAKWXmneBa1rx4knJzyWeD44Toz+xufU5X+XjZmWNHJglsO7Xz9eTJsjh3d/Stwk77s18jFRpXrHzPdYpQtyBK2gg5Zc/+GdYuEziRsWj/j2KHHP5NWioz8WqhpfuUUUOfV6qSHgS8Qtr8QHxmhaJB6++Yuq0g4eRMNUL11myzV9w2J5CVe672D5Oby7Pgjq72ht+9X4tCHLHoYSl9glwovuTYqFjmF6tGRw0T1LtsCTSZDJPWgltmfopX+p9mFwkMpk+4YLXKa+o5X8qaI9YMlp8qG/o3Vs0yOuA0CsEZFo4Fuf7Nsa7fq15AZo8t+6QWI1TIcNWkmN3JQ9NfbGUNI8hntb9He0zjRXuHYebKL3KeGuwlAG5sMGLbuH5Sl7IWO9W87boLcl9jNVmeqyNfFTjqApP/VvLv2Ya+fsts+5PiUbhNUtu32Nla/+jtb+ZaN8th5cz3LZ9nybx6vvaOnzu8mf8tyyYsUK6xLBVOvKu6CV9KFTUfEIMfJH48ST06rFbqdL+hxuWPzknG1DRiIcdMPiEV8Wo++bJqrrgg9YOoARtHwLWpmFktwuSwz3sk0HrVQHIfvrskdLfxo1t+WofZKx3/Wog/01a7IKWvZeHSuAZPodLXWJYSL4WKFNhy/bCILJwSr571iwGRqUrG1KvuwvqRdNByPr/VbwG7p91ntyvHTQ1KiD+lNFp4AlK44edbDt1c962FBKavDLe0lZo8/pUQYT77G+u2JrzOswlggCicEwrGVl+B2txOV1yQ3h5L/NNd70qIMHVlzribPVWLePVmcFq1S9ScOZJ/VoWcuyXTJolU/8vdZ7EvNaZZOml9JkI7e59OMqaL1X97gn1plvq3N9SjZJ9BSGyy3z/Uz8n+hRB893rvLZOrENuWx3vs3j5aiDxhoMOSw4b4KWU4gqoNcIWgStQQNqmAh6YQhaORzDXM2i76O1e9JTGYUsGUqssBH/1Ef1YqUcsW+YUQfl6IBWsNKjBSbCTfKlSrq3LLeglQhiycu1erdS7oPeNtu+5xC0TN1HS/dm6XuW6B4se8XQ99Ea2PlbDxtKtga9vozs1RQjAMrptsvKEgFCj9Y2WozVw4LrQRhs38eyymzMaGu4dyukWUFALyvpO2G25Vjz2HtxPPxdO1/YW+OJ81An2z5qp1T7bzdPDk/WPieXoW35Vnmlfo/2HBIyrOWbaSj3bfi/KmTJwTD8b0xnGLRs36Wz6m/8mBmUWy5W+j5ah9eMDcA6g/qj679DfR3irP9nDNfPXJzlPJ0e3kfLfvwP+neCVnh6rXTvldNPghZBi6Dl8dHy/OnTalCGyhtGZxyyknuPZANCB6DT24aGpMQ0eQleIuzYX9fBzerVSrMcY0FLBr8hYSsRsgaFzCyD1sDGWmW98sc3elyKQuzYsUN9B8spYMmVHdwUu7dTlwffGRrSgLAa8LKBHvvk3mp0Ww193TBNfLIvkhqhskFkhQp7Y2jQ8mWvi16WHnUw3hDTja14Q9Ye6vwaDOPg2z9WjX9vBhqJ7ZfdxPpd7qPdyN6TmMrccrSXgW7EOgSppOUnl5f8v9eNWFknrDK3Grt62WZ+Ns78oLJ2O9jIkPqcUQPcuQ46Gwy1DdIt2/2VtyiQl2c2zf5wOEOWLC/9v2/VvYS5trbKZki9NlM/s3WW7z/+7nPKumv5KM/PEUEvkKBF0MqlDkZ4ePdwX95noidLL7PQerR2/W6Kavw3//Y3WQatwd9b0pfe8TO1y/aJjyrrrqoVuRxvcp7n3EC3OnnLxtKZ5oXhaCxZjX49KqBs7CQaR3oQBqtxZL9szprXKTAE12g617Ys1iCd8xFPja1wFeIGYi6NSjfz9FReH2uQmvjgIKOgFVw9c+OWy7yt8z6trI9u+qWn9TqXbYn6PK3zi5X1mWP+30sz5xNMhjMStAhaGVaVQW8jaEWwh62QgtaJfV2q4b9m7C2ErPgNjk0Fxd4Vy5T1unvvGnQQ8eOP7rd/qE7ePVU/CFFDKRGqki+p0r1iqlFlhSrb5WyOPVbBN3y7l3431iDd+JCnzgStoWWrL8/0YrCRqDfe3eyfvLG5/ICm7dX/5mmddrNNUZ23r/YXyvrAun/y5LTg93200m00QYugla6OOE0naBG0nOpF3rxWN3GCavzvLZ1D0DIctDbFbwZ9+J1tvtaPkwc3qJN3w4wPiIvdG0PWWHIKWw69VE5hy7pEaGgjPIiG2Mk9sdHv2sr+0nNjgtbgMt77xldiDVKPBhsJor7kyzobZ/2Jsj6x4yXP63W+GPi1nXro/IvnT3lyjvDzPlqZbDBBi6CVST1Jfg9Bi6CVXCc8+duP4d0Pv/uOClm1995JyDIcsrrLy5S1DLZ+PzoWXakaSr1r76ahZPCSsI7yv1bO8jsWfjXMCnE95ztXK+eG6e8XXg02UoiOmeyz/r5h12L394LLZH2F/J6Dq/5R1esj7z7h2SlCj1zsx320MtloghZBK5N6kvweghZBK7lOePK3H0Fr7d13qMZ/z8IKgpbhoFUzbqyyPt6115P6kelC+ltfVSdved1/ITdiTO97/9anlLPsaTG9rkJfftsrn1HWXg42UuimqfafyzMH96SmcnL7+rn25apON88ryvTQnpfvI2gRtHKpuAQtglYu9SbtPKbvo9W1slI1/LeOf5CQZThktU2fqqx3/faltOXu9RuaXv6EOoG/t/VJAoDB3qymuR9VzqfqS3E26Hxi+2Tl3FJ6Cc4GnWVwkKOTyu9myUFH3AYJ5h8+sHUvHRk7Tje/7PUpIFTLI2gRtHKpkAQtglYu9SbweeRQ7stHf1/0rVxB0DIctKp+9ENlff6UN9fdZ1p5enc8o07e8p4sNHSGb+i48eldd69y3r/s/+BsuPHfNOc/KuujHg824qb8ozjvhc41yrlx1ofExX211GuD9frk7th3O+Ul3lF/ELQIWrnUcYIWQSuXehPoPM1l81XDf+eTEwlZhkPWnn+bpKxbyl/1t8wvnlcNJfmJtDyRR7ExGIp96t5sOcvLf0KxTQYbhUHu35G1dynr9vIv4Gy4jNvKSpT1kbV3Ym3YuqPii8p64ECNv+eIANZG0CJo5VLtCFoErVzqTWDznD1+XDX8q276gZA3zzU1lDnLrRP9a6uV9eqxY3wv757aO9XJm14Wcz1ZMnT0VMWGzT+06ic0SA03SBtm/HtVpxlsxGydPrFjinJue+W/UqcN12l5Sbf8MGzfytG+nyOCWCFBi6CVS70jaBG0cqk3gc2zY/JzqvHf+OLzhCzDvVnvPPqwsu5evcrX8j7b36ZO3vKL7PSymGuUnm1ZrJyb536MBqnhBqn8wEA2SLsWfxNrw9ZNcz6irN+rexxrw9Yt84qU9dnjHUbOEQzvnh/BpnN3uLaz4oVKMeX++eL0wBkj9TLbhRK0CFrZ1pnA3t/f3qYa/tW33UrIMhyyDi9ZrKw3PPiA7+XdtWKUOnkfXjOWhpLBhlLXoq8p57719+Ns0Fn2HMqQJZ8MNmLugwPpfKQm1hPO6JlmnaV177p7VJ0+uOFfjJ0jCFrhCjBhC1SptoegFcFgMxCyfWptbRVTp04VdXV1xg6A6RZsYnj3jb98UDX+22dNJ2gZDlob779XWffu3JGuqD2dPrA/9iX2lpc/RePfYON/YNfvVSOpfcGlOBt0lg3SzorLlfUBbk5svK7Je5PJQHty9wzj65JlW6hPeeP4hul/KBpmflAIcdHTc4B9YdxHi6CVKkwN9zpBK2ShJGwhyYvtiWLQOrhlc6yH5b67CVmGQ1bX/FJl/c6kp+3nPF9+b3vtC6qhdHTj+IJtxPjReGsri93LiWHzzTaW9eWZDTM+wGWwhoNJ99K/iwXa5d/n2GHYumfljcq6d+dzvpwXwrISvqOVH8GPoEXQEl6EqeGWEYag5fV9tFb/7FbV+O8uLyNoGQ5a8tJMOXT+QM8BX89vx5rmqpO3HMXKj7BRqOs4tmWCcu5a9FWcDTdIW+f/Z2XNzYnNBtqLXeuVc8OMPxLn2t6iXhus12eaFyrr1gWf9fX8EIaVEbQIWrnUQ76jFcHgF4aglUtlTDVPx9IlquG/7eGHCFmGQ1bL1CnKun7WjFTFYez1hlkfUidweVPXQg1Bfuy3vNxHXl41sGsazgYbpP3xEdnkgAF+lGshr6OjPNYTfrj6dqwN1mlZx/Yt+bY6fvS3vWbsXBDWBRO0CFq51E2CFkErl3rj6zyyd0U+j7y1hKBlMGid2rpFVN5wvZA3g754/pyvZXx4a6yXpXvp1TSUDDaUZENUhqz9y7+Hs0Fn2SBtmv0flDU3Jzbbm3W6sUw5t877NHXacJ0e2Pk7Zd259Fu+nh/CsjKCFkErl7pI0CJo5VJvfJtH9qzIkLXr6ScIWQZDlrxv2O5JTynr9jcX+Va+ckUXzg2ok7cMAKcbX6WxZKixdKFrnXKWAwacaX4dZ0POMmTJETNlfe547b/jbNBZWjeXflxZH9v0CNaGrTvKL1PWpw5t8vUcEZaVEbQIWrnUxZyDVk1NjeAZXoOgRx3MpTImz3O6r081/FfefIPor1lD0DIYtI6tWqmsa+4cl1wMxv/eX32LOnlz01yzn/zvX3ZNzHn1T2mQGm6QyhHZZNA6/s6zWBu0lr2F0rnz9S/hbNBZBtpjWx5T1vvX/MT4OUGvIKzDuy+buVbwDK9BJAbDkI14nuE3CHJ4d32gdPNz29NPqMZ/00uTCVkGQ5bszdr2yHhlfaB2vZsiy3reM0fr1clb3jT3wt41NJYMNZbONL2mnFtKLxEX9tbgbMhZNkj3Lfmust735rdwNugsreXgFzJondjxEtaGrZtLP6Gsz530b5CksAYteTNcnuE3yNsbFh86dEjwzB+DrFu+Hs3g9j5aRxsbYj0s48YSsgyHrIOLFirrzb/6pUeln/liOpd8U528+2p/QUPJYENJ38upbwPOsoFu8ikb/vJ5as8co+sxuQ/5sOyeyuuU8/63RuFsuE4fWRu7EfThuoczP7h78M6w3UfrUFev4Jk/BnkbtDz432ERBSDgNmitj98wVw+Ewc/YgCAmHY41NfpaM0/sfUs1lHTDlJ+xBjoOOFAHqANhqwNNc//c1/MDK0MgKgJZf0crKjvOfpgV6Dh+Vkzcfkg9ZejK5rF/bY2QvSs8/TPY8eLz2RSRJ+/tXHKV4GneoHHOR4R+tr9xBeaG6l3H4q9aztRrs/W6ed6nlHXzy5dQnw3V5+Q6fLT+954c91kIAoUmQNAqtBJnfxFAAAEEEEAAAQQQQMC4AEHLODErQAABBBBAAAEEEEAAgUITIGgVWomzvwgggAACCCCAAAIIIGBcgKBlnJgVIIAAAggggAACCGQjELbh3bPZdt6LgBYgaGkJfiKAAAIIIIAAAgiEQoCgFYpiYCNcChC0XAIyu7OA2+HdnZfKqwgggAACCCBQCAJ65OK5LUcLYXfZx4gKELQiWrBB7xZBK+gSYP0IIIAAAggggAACQQoQtILUj/C63dxHK8Is7BoCCCCAAAIIIIBAgQgQtAqkoNlNBBBAAAEEEEAAAQQQ8E+AoOWfNWtCAAEEEEAAAQQQQACBAhEgaBVIQbObCCCAAAIIIIBAvgi8ubdf1PQM5Mvmsp0IOAoQtBxZeBEBBBBAAAEEEEAgCAEG1ApCnXWaECBomVBlmZaA/DRKDtHKAwEEEEAAAQQQyETgxfpeoe+jJUMXDwTyVYCgla8llwfb/fMtB6wD5fa+U3mwxWwiAggggAACCAQtcOzMeSHvn/Wz9d1BbwrrR8CVAEHLFR8zDycgw5X+REqGLh4IIIAAAggggAACCBSKAEGrUEo6oP2Un0Yt3tsf0NpZLQIIIIAAAggggAACwQgQtIJxZ60IIIAAAggggAACCCAQYQGCVoQLl11DAAEEEEAAAQTyRUAO6c4DgSgJELSiVJrsCwIIIIAAAgggkIcCegCtqys7hBwMgwcCURAgaEWhFPNoH+QoQlcuaeUgmkdlxqYigAACCCBgUkAGq0vK6tUAWvJnx/GzJlfHshHwTYCg5Rs1K5IhS49CKMMWDwQQQAABBBBAQArIkYplyOJ2MNSHKAkQtKJUmiHfF/mJ1RVLWlXYkqGLBwIIIIAAAgggoAW4ZFBL8DMqAgStqJRknuyHPIgSsvKksNhMBBBAAAEEPBaQlwU+wL01PVZlcWEVIGiFtWQKcLvkF2Dlwbe650QB7j27jAACCCCAQLQF5Pldf4WAD12jXdbsXUyAoEVNCIWAvCZbH3zlTY6TH/aDs1MQm7j9kDW/2+nJ65Z/25fvNH1kZbu1fjfT5XKcHtrG7XS5H8kPu63b6U729uU7Tbfbup2evG/yb9Nlo5fvtmzSze+2bNLN79Y+3fxOZWMve6fp2lbWf6dHptPT2bqd7mQrt1f/3zpNl15eTU9n73a6G/t0ZZfKXpdtqunazu30IMrGXvZuyybd/EGU3XBlI69qGW660/byGgL5LEDQyufSi9C2L97bLy59vUkdgJ0+5Qr6xBTlBqHd1m2jw+mkb1++03S7rdvpTv8SusGWrsGXbnqqBp1efqrp6RoVmU53Wzbp5ndrn25+p7Kxl73TdG2brmzSTQ+ibOT+6LJNZ+92ejp7t9ODKBtd9kGUnfTyquyc7O3Ld5pu/79wOz2IstN2qcpO7p/Ted5pW3kNgXwXIGjlewlGbPtlz5bTl2Hl6/KgLZ9OIxLJg7ZX051I7ct3mi7v/6HX72a6XI7TQy/b7XSnk5vd1u10p7KxL99put3W7XQnO9Nlo5fvtmzSze+2bNLN79Y+3fxOZWMve6fp2lbWf6dHptPT2bqd7mQrt1f/3zpNl15eTU9n73a6G/t0ZZfKXpdtqunazu30IMrGXvZuyybd/EGUXbqycdomXkMgqgIEraiWLPuFAAIIIIAAAggggAACgQkQtAKjZ8UIIIAAAggggAACCCAQVQGCVlRLlv1CAAEEEEAAAQQQQACBwAQIWoHRs2IEEEAAAQQQQAABBBCIqgBBK6oly34hgAACCCCAAAIIIIBAYAIErcDoWTECCCCAAAIIIIAAAghEVYCgFdWSZb8QQAABBBBAAAEEEEAgMAGCVmD0rBgBBBBAAAEEEEAAAQSiKkDQimrJsl8IIIAAAggggAACCCAQmABBKzB6VowAAggggAACCCCAAAJRFSBoRbVk2S8EEEAAAQQQQAABBBAITICgFRg9K0YAAQQQQAABBBBAAIGoChC0olqy7BcCCCCAAAIIIIAAAggEJkDQCoyeFSOAAAIIIIAAAggggEBUBQhaUS1Z9gsBBBBAAAEEEEAAAQQCEyBoBUbPihFAAAEEEEAAAQQQQCCqAgStqJYs+4UAAggggAACCCCAAAKBCRC0AqNnxQgggAACCCCAAAIIIBBVAYJWVEuW/UIAAQQQQAABBBBAAIHABAhagdGzYgQQQAABBBBAAAEEEIiqAEErqiXLfiGAAAIIIIAAAggggEBgAgStwOhZMQIIIIAAAggggAACCERVgKAV1ZJlvxBAAAEEEEAAAQQQQCAwAYJWYPSsGAEEEEAAAQQQQAABBKIqQNCKasmyXwgggAACCCCAAAIIIBCYAEErMHpWjAACCCCAAAIIIIAAAlEVIGhFtWTZLwQQQAABBBBAAAEEEAhMgKAVGD0rRgABBBBAAAEEEEAAgagKELSiWrLsFwIIIIAAAggggAACCAQmQNAKjJ4VI4AAAggggAACCCCAQFQFCFpRLVn2CwEEEEAAAQQQQAABBAITIGgFRs+KEUAAAQQQQAABBBBAIKoCBK2oliz7hQACCCCAAAIIIIAAAoEJELQCo2fFCCCAAAIIIIAAAgggEFUBglZUS5b9QgABBBBAAAEEEEAAgcAECFqB0bNiBBBAAAEEEEAAAQQQiKoAQSuqJct+IYAAAggggAACCCCAQGACBK3A6FkxAggggAACCCCAAAIIRFWAoBXVkmW/EEAAAQQQQAABBBBAIDABglZg9KwYAQQQQAABBBBAAAEEoipA0IpqybJfCCCAAAIIIIAAAgggEJjA/wfYWeJhLroKjQAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you call nlp on a text, spaCy first tokenizes the text to produce a Doc object. The Doc is then processed in several different steps – this is also referred to as the processing pipeline. The pipeline used by the default models consists of a tagger, a parser and an entity recognizer. Each pipeline component returns the processed Doc, which is then passed on to the next component.\n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "All the preprocessing items including tokenization, stop words removal, lemmatization, getting POS and NER etc. can be performed in one go using spaCy. An example is demonstrated below. We will go through the example of NER in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Google is looking at buying U.K. startup for $1 billion'\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Token</th>\n",
       "      <th>is_stop_word</th>\n",
       "      <th>lemma</th>\n",
       "      <th>POS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Google</td>\n",
       "      <td>False</td>\n",
       "      <td>Google</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>is</td>\n",
       "      <td>True</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>looking</td>\n",
       "      <td>False</td>\n",
       "      <td>look</td>\n",
       "      <td>VERB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>at</td>\n",
       "      <td>True</td>\n",
       "      <td>at</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>buying</td>\n",
       "      <td>False</td>\n",
       "      <td>buy</td>\n",
       "      <td>VERB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>U.K.</td>\n",
       "      <td>False</td>\n",
       "      <td>U.K.</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>startup</td>\n",
       "      <td>False</td>\n",
       "      <td>startup</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>for</td>\n",
       "      <td>True</td>\n",
       "      <td>for</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>$</td>\n",
       "      <td>False</td>\n",
       "      <td>$</td>\n",
       "      <td>SYM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>NUM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>billion</td>\n",
       "      <td>False</td>\n",
       "      <td>billion</td>\n",
       "      <td>NUM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Token  is_stop_word    lemma    POS\n",
       "0    Google         False   Google  PROPN\n",
       "1        is          True       be   VERB\n",
       "2   looking         False     look   VERB\n",
       "3        at          True       at    ADP\n",
       "4    buying         False      buy   VERB\n",
       "5      U.K.         False     U.K.  PROPN\n",
       "6   startup         False  startup   NOUN\n",
       "7       for          True      for    ADP\n",
       "8         $         False        $    SYM\n",
       "9         1         False        1    NUM\n",
       "10  billion         False  billion    NUM"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[t.text, t.is_stop, t.lemma_, t.pos_]\n",
    "              for t in doc],\n",
    "             columns=['Token', 'is_stop_word','lemma', 'POS'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy also performs NER that we will discuss in the next section, along with the word embedding which we will also cover in the next section. Given NER performs a wide range of NLP related tasks in one go, it is highly recommended. We will be using spaCy extensively in our case studies. The list of all the task that can be performed using spaCy is mentioned in the list below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cats', 'char_span', 'count_by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', 'from_disk', 'get_extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_nered', 'is_parsed', 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'merge', 'noun_chunks', 'noun_chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', 'sentiment', 'sents', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws', 'to_array', 'to_bytes', 'to_disk', 'to_json', 'user_data', 'user_hooks', 'user_span_hooks', 'user_token_hooks', 'vector', 'vector_norm', 'vocab']\n"
     ]
    }
   ],
   "source": [
    "attributes = [a for a in dir(doc) if not a.startswith('_')]\n",
    "print(attributes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2.6'></a>\n",
    "## 2.6. Name Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named Entity Recognition, popularly referred to as N.E.R is a process that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. NER is used in many fields in Natural Language Processing (NLP), and it can help answering many real-world questions. The NER performed using spaCy is shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Google is looking at buying U.K. startup for $1 billion'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SpaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity:  Google\n",
      "Entity Type: ORG | Companies, agencies, institutions, etc.\n",
      "--\n",
      "Entity:  U.K.\n",
      "Entity Type: GPE | Countries, cities, states\n",
      "--\n",
      "Entity:  $1 billion\n",
      "Entity Type: MONEY | Monetary values, including unit\n",
      "--\n"
     ]
    }
   ],
   "source": [
    "for entity in nlp(text).ents:\n",
    "    print(\"Entity: \", entity.text)\n",
    "    print(\"Entity Type: %s | %s\" % (entity.label_, spacy.explain(entity.label_)))\n",
    "    print(\"--\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Google\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is looking at buying \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    U.K.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " startup for \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    $1 billion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
       "</mark>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "displacy.render(nlp(text), style=\"ent\", jupyter = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "df6a4523-b385-69ee-c933-592826d81431"
   },
   "source": [
    "<a id='3'></a>\n",
    "# 3. Feature Representation\n",
    "\n",
    "The vast majority of NLP related data is created for human consumption and as such is stored\n",
    "in an unstructured format, such as news feed articles, PDF reports, social media posts\n",
    "and audio files, which cannot be readily processed by computers. Following the preprocessing steps discussed in the previous section, in order for the information content to be conveyed to the statistical inference algorithm, the preprocessed tokens need to be translated into predictive features. A model is used to embed raw text into a vector space where we can use the data science tool.\n",
    "\n",
    "Feature representation involves two things:\n",
    "* A vocabulary of known words.\n",
    "* A measure of the presence of known words.\n",
    "\n",
    "The intuition behind the Feature Representation is that documents are similar if they have similar content. Also, we can learn something about the meaning of the document from its content alone.\n",
    "For example, if our dictionary contains the words {Learning, is, the, not, great}, and we want to vectorize the text “Learning is great”, we would have the following vector: (1, 1, 0, 0, 1).\n",
    "\n",
    "Some of the feature representation methods are as follows: \n",
    "* Bag of Words- word count\n",
    "* Tf-Idf\n",
    "* Word Embedding \n",
    "    * Pretrained word embedding models ( Word2vec, GloVe)\n",
    "    * Customized deep Learning based\n",
    "\n",
    "There are Feature representation(or vector representation) such as one-hot encoding of text, n-grams etc which are similar to the types mentioned above. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3.1'></a>\n",
    "## 3.1. Bag of Words - Word Count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In natural language processing, a common technique for extracting features from text is to place all of the words that occur in the text in a bucket. This aproach is called a bag of words model or BoW for short. It’s referred to as a “bag” of words because any information about the structure of the sentence is lost.The CountVectorizer from sklearn provides a simple way to both tokenize a collection of text documents and encode new documents using that vocabulary.The fit_transform\n",
    "function learns the vocabulary from one or more documents and encodes each document in the word as a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [\n",
    "'The stock price of google jumps on the earning data today',\n",
    "'Google plunge on China Data!'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 1 1 1 1 1 0 1 1 2 1]\n",
      " [1 1 0 1 0 0 1 1 0 0 0 0]]\n",
      "{'the': 10, 'stock': 9, 'price': 8, 'of': 5, 'google': 3, 'jumps': 4, 'on': 6, 'earning': 2, 'data': 1, 'today': 11, 'plunge': 7, 'china': 0}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "print( vectorizer.fit_transform(sentences).todense() )\n",
    "print( vectorizer.vocabulary_ )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see an array version of the encoded vector showing a count of 1 occurrence for each word except the (index and id 10) that has an occurrence of 2. Word counts are a good starting point, but are very basic.One issue with simple counts is that some words like “the” will appear many times and their large counts will not be very meaningful in the encoded vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3.2'></a>\n",
    "## 3.2. TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative is to calculate word frequencies, and by far the most popular method is called TF-IDF. This is an acronym than stands for “Term Frequency – Inverse Document” Frequency which are the components of the resulting scores assigned to each word.\n",
    "\n",
    "* Term Frequency: This summarizes how often a given word appears within a document.\n",
    "* Inverse Document Frequency: This downscales words that appear a lot across documents.\n",
    "Without going into the math, TF-IDF are word frequency scores that try to highlight words that are more interesting, e.g. frequent in a document but not across documents.\n",
    "\n",
    "The TfidfVectorizer will tokenize documents, learn the vocabulary and inverse document frequency weightings, and allow you to encode new documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['china', 'data', 'earning', 'google', 'jumps', 'plunge', 'price', 'stock', 'today']\n",
      "(2, 9)\n",
      "[[0.         0.29017021 0.4078241  0.29017021 0.4078241  0.\n",
      "  0.4078241  0.4078241  0.4078241 ]\n",
      " [0.57615236 0.40993715 0.         0.40993715 0.         0.57615236\n",
      "  0.         0.         0.        ]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(max_features=1000, stop_words='english')\n",
    "TFIDF = vectorizer.fit_transform(sentences)\n",
    "print(vectorizer.get_feature_names()[-10:])\n",
    "print(TFIDF.shape)\n",
    "print(TFIDF.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A vocabulary of 9 words is learned from the documents and each word is assigned a unique integer index in the output vector. The sentences are encoded as an 9-element sparse array and we can review the final scorings of each word with different values from the other words in the vocabulary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3.3'></a>\n",
    "## 3.3. Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A word embedding is a class of approaches for representing words and documents using a dense vector representation.\n",
    "\n",
    "It is an improvement over more the traditional bag-of-word model encoding schemes where large sparse vectors were used to represent each word or to score each word within a vector to represent an entire vocabulary. These representations were sparse because the vocabularies were vast and a given word or document would be represented by a large vector comprised mostly of zero values.\n",
    "\n",
    "Instead, in an embedding, words are represented by dense vectors where a vector represents the projection of the word into a continuous vector space.The position of a word within the vector space is learned from text and is based on the words that surround the word when it is used. The position of a word in the learned vector space is referred to as its embedding.\n",
    "\n",
    "Two popular examples of methods of learning word embeddings from text include:\n",
    "* Pretained models( i.e. Word2Vec, glove etc.)\n",
    "* Developing custom models\n",
    "\n",
    "In addition to these carefully designed methods, a word embedding can be learned as part of a deep learning model. This can be a slower approach, but tailors the model to a specific training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.1 Pretrained word embedding models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.1.1  Pretrained model- SpaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SpaCy comes with inbuilt representation of text as vectors at different levels of word, sentence and document. The underlying vector representations come from a word embedding model which generally produces a dense multi-dimensional semantic representation of words (as shown in the example below). The word embedding model includes 20k unique vectors with 300 dimensions. Using this vector representation, we can calculate similarities and dissimilarities between tokens, named entities, noun phrases, sentences and documents. \n",
    "\n",
    "The word embedding in Spacy is performed first by first loading the model, and then processing text. The vectors can be accessed directly using the .vector attribute of each processed token (word). The mean vector for the entire sentence is also calculated simply using .vector, providing a very convenient input for machine learning models based on sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"Apple orange cats dogs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector representation of the sentence for first 10 features: \n",
      " [-0.30732775  0.22351399 -0.110111   -0.367025   -0.13430001  0.13790375\n",
      " -0.24379876 -0.10736975  0.2715925   1.3117325 ]\n"
     ]
    }
   ],
   "source": [
    "print(\"Vector representation of the sentence for first 10 features: \\n\", doc.vector[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.1.2. Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word2Vec(vocab=10, size=100, alpha=0.025)\n",
      "['The', 'stock', 'price', 'of', 'Google', 'increases', 'plunge', ' on', 'China', ' Data!']\n",
      "[ 0.00217071 -0.00090912 -0.00315378  0.00301918]\n"
     ]
    }
   ],
   "source": [
    "sentences = [\n",
    "['The','stock','price', 'of', 'Google', 'increases'],\n",
    "['Google','plunge',' on','China',' Data!']]\n",
    "# train model\n",
    "model = Word2Vec(sentences, min_count=1)\n",
    "# summarize the loaded model\n",
    "print(model)\n",
    "words = list(model.wv.vocab)\n",
    "print(words)\n",
    "print(model['Google'][1:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4'></a>\n",
    "# 4. Interpretation\n",
    "Like all other artificial intelligence tasks, the inference generated by an NLP application\n",
    "usually needs to be translated into a decision in order to be actionable.Inference in ML falls under three broad categories, namely supervised, unsupervised and reinforcement learning. While the type of inference required depends on the business problem and the type of training data, in NLP the most commonly used algorithms are\n",
    "supervised or unsupervised. \n",
    "\n",
    "In the past years, neural network architectures, such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), have dominated NLP-based inference. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4.1'></a>\n",
    "## 4.1. Supervised Learning Example-Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most commonly used supervised methodologies in NLP is the Naïve\n",
    "Bayes model, which assumes that all word features are independent of each other given\n",
    "the class labels. Due to this simplifying assumptions, Naïve Bayes is very compatible with a bag-of-words word representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "senteces = [\n",
    "'The stock price of google jumps on the earning data today',\n",
    "'Google plunge on China Data!']\n",
    "sentiment = (1, 0)\n",
    "data = pd.DataFrame({'Sentence':senteces,\n",
    "        'sentiment':sentiment})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vect = CountVectorizer().fit(data['Sentence'])\n",
    "X_train_vectorized = vect.transform(data['Sentence'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1], dtype=int64)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clfrNB = MultinomialNB(alpha = 0.1)\n",
    "clfrNB.fit(X_train_vectorized, data['sentiment'])\n",
    "\n",
    "preds = clfrNB.predict(vect.transform(['Apple price plunge', 'Amazon Price jumps']))\n",
    "preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see that the Naive Bayes trains the model fairly well from the two sentences. The model gives a sentiment of 0 for the sentence \"Apple price plunge\" and 1 for the sentence \"Amazon Price jumps\", given the sentence used for training also had keywords \"plunge\" and \"jumps\" as were assigned to sentiments of 0 and 1 respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4.2'></a>\n",
    "## 4.2. Unsupervised Learning Example-LDA \n",
    "LDA model is the most popular topic model because it tends to produce meaningful topics that\n",
    "humans can relate to, can assign topics to new documents, and is extensible. Variants of\n",
    "LDA models can include metadata such as authors, or image data, or learn hierarchical\n",
    "topics\n",
    "Given a set of documents, assume that there are some latent topics of documents that are not observed. Each document has a distribution over these topics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [\n",
    "'The stock price of google jumps on the earning data today',\n",
    "'Google plunge on China Data!'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.04283242, 0.91209857, 0.04506902],\n",
       "       [0.0679334 , 0.07059544, 0.86147116]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Getting the bag-of words\n",
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "vect=CountVectorizer(ngram_range=(1,1),stop_words='english')\n",
    "sentences_vec=vect.fit_transform(sentences)\n",
    "\n",
    "#Running LDA on the bag of words. \n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "lda=LatentDirichletAllocation(n_components=3)\n",
    "lda.fit_transform(sentences_vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model produces two smaller matrices. We will be discussing the interpretation further in the third case study. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='5'></a>\n",
    "# 5 NLP Recipies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='5.1'></a>\n",
    "## 5.1. Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment analysis is contextual mining of text which identifies and extracts subjective information in source material, and helping us understand the sentiments behind a text. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the help of Sentiment Analysis using Textblob the sentiment analysis can be performed in few lines of code. TextBlob provides polarity and subjectivity estimates for parsed documents using dictionaries provided by the Pattern library. The polarity defines the phase of emotions expressed in the analyzed sentence. Polarity alone is not enough to deal with complex text sentences. Subjectivity helps in determining personal states of the speaker including Emotions, Beliefs and opinions. It has values from 0 to 1 and a value closer to 0 shows the sentence is objective and vice versa.\n",
    "\n",
    "The texblob sentiment function is pretrained and map adjectives frequently found in movie reviews(source code: https://textblob.readthedocs.io/en/dev/_modules/textblob/en/sentiments.html) to sentiment polarity scores, ranging from -1 to +1 (negative ↔ positive) and a similar subjectivity score (objective ↔ subjective).\n",
    "\n",
    "The .sentiment attribute provides the average for each over the relevant tokens, whereas the .sentiment_assessments attribute lists the underlying values for each token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"Bayer (OTCPK:BAYRY) started the week up 3.5% to €74/share in Frankfurt, touching their highest level in 14 months, after the U.S. government said a $25M glyphosate decision against the company should be reversed.\"\n",
    "text2 = \"Apple declares poor in revenues\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text1).sentiment.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentiment(polarity=0.5, subjectivity=0.5, assessments=[(['touching'], 0.5, 0.5, None)])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text1).sentiment_assessments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.4"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text2).sentiment.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentiment(polarity=-0.4, subjectivity=0.6, assessments=[(['poor'], -0.4, 0.6, None)])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(text2).sentiment_assessments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the first sentiment has positive sentiment and the second text has negative sentiments. Looking at the subjectivity, the second sentence has more subjectivity as compared to the first one. However, looking at the words that give rise to the sentiments, the word \"touching\" and not \"high\" causes positive sentiment in sentence one. So, probably a sentiment analysis algorithm pretrained on movie/product reviews might not perform well with news sentiment analysis. Hence probably, additional training for the stock sentiments might be needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='5.2'></a>\n",
    "## 5.2. Text Similarity\n",
    "Finding similarity between text is at the heart of almost all text mining methods, for example, text classification, clustering, recommendation, and many more. In order to calculate similarity between two text snippets, the usual way is to convert the text into its corresponding vector representation, for which there are many methods like word embedding of text, and then calculate similarity or difference using different distance metrics such as cosine-similarity and euclidean distance applicable to vectors. The underlying vector representations come from a word embedding model which generally produces a dense multi-dimensional semantic representation of words (as shown in the example). Using this vector representation, we can calculate similarities and dissimilarities between tokens, named entities, noun phrases, sentences and documents. The example below shows how to calculate similarities between two documents and tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"Barack Obama was the 44th president of the United States of America.\"\n",
    "text2 = \"Donald Trump is the 45th president of the United States of America.\"\n",
    "text3 = \"SpaCy and NLTK are two popular NLP libraries in Python community.\"\n",
    "doc1 = nlp(text1); doc2 = nlp(text2); doc3 = nlp(text3); "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_similarity(inp_obj1, inp_obj2):\n",
    "    return inp_obj1.similarity(inp_obj2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between doc1 and doc2:  0.9525886414220489\n",
      "Similarity between doc1 and doc3:  0.5184867892507579\n"
     ]
    }
   ],
   "source": [
    "print(\"Similarity between doc1 and doc2: \", text_similarity(doc1, doc2))\n",
    "print(\"Similarity between doc1 and doc3: \", text_similarity(doc1, doc3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token 1: Apple, Token 2: Apple - Similarity: 1.000000\n",
      "Token 1: Apple, Token 2: orange - Similarity: 0.561892\n",
      "Token 1: Apple, Token 2: cats - Similarity: 0.218511\n",
      "Token 1: orange, Token 2: Apple - Similarity: 0.561892\n",
      "Token 1: orange, Token 2: orange - Similarity: 1.000000\n",
      "Token 1: orange, Token 2: cats - Similarity: 0.267099\n",
      "Token 1: cats, Token 2: Apple - Similarity: 0.218511\n",
      "Token 1: cats, Token 2: orange - Similarity: 0.267099\n",
      "Token 1: cats, Token 2: cats - Similarity: 1.000000\n"
     ]
    }
   ],
   "source": [
    "def token_similarity(doc):\n",
    "    for token1 in doc:\n",
    "        for token2 in doc:\n",
    "            print(\"Token 1: %s, Token 2: %s - Similarity: %f\" % (token1.text, token2.text, token1.similarity(token2)))\n",
    "\n",
    "doc4 = nlp(\"Apple orange cats\")\n",
    "token_similarity(doc4)"
   ]
  }
 ],
 "metadata": {
  "_change_revision": 206,
  "_is_fork": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}