notebooks/target_model.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "import math\n",
    "from textwrap import fill\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "from sklearn.model_selection import GridSearchCV, StratifiedKFold\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  \n",
    "\n",
    "# Models\n",
    "from bella.models.target import TargetInd, TargetDepMinus, TargetDep, TargetDepPlus\n",
    "# Word Vector methods\n",
    "from bella.word_vectors import VoVectors, SSWE\n",
    "from bella.helper import read_config\n",
    "# Sentiment lexicons\n",
    "from bella import lexicons\n",
    "# Get the data\n",
    "from bella.parsers import dong\n",
    "# Tokenisers\n",
    "from bella.tokenisers import ark_twokenize, stanford"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Target dependent models\n",
    "This notebook shows how to use the target dependent models and comparing the results of our implementation to the one in the original [paper](https://www.ijcai.org/Proceedings/15/Papers/194.pdf). We also show the problems we encountered when attempting to reproduce the methods from the description in the paper and the affects of not stating certain processes.\n",
    "\n",
    "The paper had four different models:\n",
    "1. **Target-Ind** -- Uses only the full Tweet as context.\n",
    "2. **Target-Dep-** -- Uses the left and right context of the target word as well as the target word as context.\n",
    "3. **Target-Dep** -- Uses all of the above contexts.\n",
    "4. **Target-Dep+** -- Uses all of the above as well as including two additional left and right contexts which ignores all words in the contexts unless they are part of the given sentiment lexicon (or any lexicon).\n",
    "\n",
    "The above models correspond to the following classes in our implementation:\n",
    "1. [TargetInd](../tdparse/models/target.py), 2. [TargetDepC](../tdparse/models/target.py), 3. [TargetDep](../tdparse/models/target.py), 4. [TargetDepSent](../tdparse/models/target.py)\n",
    "\n",
    "All of the results shown below are 5 fold cross validation over the training data of [Dong et al.](https://aclanthology.coli.uni-saarland.de/papers/P14-2009/p14-2009) or where appropriate on the test data as reported in the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "##\n",
    "#  ADD YOUR CONFIG FILE PATH HERE \n",
    "##\n",
    "CONFIG_FP = Path('..', 'config.yaml')\n",
    "\n",
    "# Load the training data\n",
    "dong_train = dong(read_config('dong_twit_train_data', CONFIG_FP))\n",
    "dong_test = dong(read_config('dong_twit_test_data', CONFIG_FP))\n",
    "train_data = dong_train.data()\n",
    "test_data = dong_test.data()\n",
    "train_y = dong_train.sentiment_data()\n",
    "test_y = dong_test.sentiment_data()\n",
    "\n",
    "# Get word vectors\n",
    "# To reduce the number of vectors we load we only load\n",
    "# the words that are within the train and test set\n",
    "# which is still as fair in practice as loading the whole\n",
    "# word vectors\n",
    "train_words = dong_train.word_list(ark_twokenize)\n",
    "test_words = dong_test.word_list(ark_twokenize)\n",
    "all_words = list(set(train_words + test_words))\n",
    "w2v = VoVectors(filter_words=all_words)\n",
    "sswe = SSWE(filter_words=all_words)\n",
    "\n",
    "###\n",
    "#    Set here the number of cpus to use for all of the\n",
    "#    grid searching\n",
    "###\n",
    "\n",
    "n_cpus = 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing the three base models\n",
    "\n",
    "In the paper the base models (target-ind, target-dep- and target-dep) using the the word2vec word vectors were compared after they found the best C-values therefore we are going to use the C-Values stated in the paper to compare our results to theres. **random_state** is used here to ensure that the results are reproducible, it stops the data from randomly shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple, List, Union\n",
    "\n",
    "def add_row(data: dict, row_name: str, df: pd.DataFrame = pd.DataFrame()) -> pd.DataFrame:\n",
    "    data_df = pd.DataFrame(data, index=[row_name])\n",
    "    return pd.concat((df, data_df))\n",
    "\n",
    "def get_score(grid_scores: pd.DataFrame, \n",
    "              associated_param: Union[None, str] = None) -> Union[List[float], List[Tuple[float, str]]]:\n",
    "    '''\n",
    "    :param grid_scors: Return of the :py:func:`grid_search_model`\n",
    "    :param associated_param: Optional. The name of the parameter you want to \n",
    "                             associate to the score. E.g. lexicon as you have\n",
    "                             grid searched over different lexicons and you \n",
    "                             want the return to be associated with the \n",
    "                             lexicon name e.g. [(0.68, 'MPQA), (0.70, 'NRC')]\n",
    "    :return: A list of test scores from the grid search and if associated_param\n",
    "             is not None a list of scores and parameter name.\n",
    "    '''\n",
    "    extracted_scores = grid_scores['mean_test_score'].astype(float).round(4) * 100\n",
    "    extracted_scores = extracted_scores.tolist()\n",
    "    if associated_param is not None:\n",
    "        for column in grid_scores.columns:\n",
    "            if associated_param in column:\n",
    "                associated_param = column\n",
    "                break\n",
    "        else:\n",
    "            raise Exception(f'Could not find the associated column using {associated_param}')\n",
    "        associated_param = grid_scores[associated_param].apply(lambda x: str(x)).tolist()\n",
    "        extracted_scores = list(zip(extracted_scores, associated_param))\n",
    "    return extracted_scores\n",
    "    \n",
    "        \n",
    "\n",
    "def grid_search_model(model, X, y, **kwargs) -> pd.DataFrame:\n",
    "    stratified_folds = StratifiedKFold(5)\n",
    "    # If no kwargs then we grid search over the model\n",
    "    # parameters already set.\n",
    "    grid_params = model.get_cv_parameters(**kwargs)\n",
    "    # model attribute is the sklearn pipeline model that represents\n",
    "    # the machine learning model\n",
    "    grid_model = GridSearchCV(model.pipeline(), grid_params, cv=stratified_folds, n_jobs=n_cpus)\n",
    "    grid_model.fit(X, y)\n",
    "    grid_results = pd.DataFrame(grid_model.cv_results_)\n",
    "    return grid_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instances of the models\n",
    "target_ind = TargetInd([w2v])\n",
    "target_dep_minus = TargetDepMinus([w2v])\n",
    "target_dep = TargetDep([w2v])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grid search each of the models and compare to paper results\n",
    "all_results = {}\n",
    "for model in [target_ind, target_dep_minus, target_dep]:\n",
    "    all_results[str(model)] = get_score(grid_search_model(model, train_data, train_y, word_vectors=[[w2v]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target Dependent</th>\n",
       "      <th>Target Dependent Minus</th>\n",
       "      <th>Target Indepdent</th>\n",
       "      <th>Target Independent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Our Results</th>\n",
       "      <td>66.81</td>\n",
       "      <td>65.67</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Paper Results</th>\n",
       "      <td>65.72</td>\n",
       "      <td>65.38</td>\n",
       "      <td>59.22</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Target Dependent  Target Dependent Minus  Target Indepdent  \\\n",
       "Our Results               66.81                   65.67               NaN   \n",
       "Paper Results             65.72                   65.38             59.22   \n",
       "\n",
       "               Target Independent  \n",
       "Our Results                 60.98  \n",
       "Paper Results                 NaN  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_results = add_row(all_results, 'Our Results')\n",
    "base_results = add_row({'Target Dependent': 65.72, 'Target Dependent Minus': 65.38, \n",
    "                        'Target Indepdent': 59.22}, 'Paper Results', base_results)\n",
    "base_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the results above that we get similar results and the order of the models stays the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Target-Dep+ and sentiment lexicons\n",
    "The **Target-Dep+** model uses sentiment lexicons to remove words therefore in this section we compare:\n",
    "1. The statistics on the sentiment lexicons\n",
    "2. The results of the model using different lexicons\n",
    "\n",
    "All the experiments below again use the Word2Vec word embeddings.\n",
    "## Sentiment lexicon statistics\n",
    "\n",
    "Below we present the size of the sentiment lexicon once it has been processed and the size of that lexicon stated in the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the sentiment lexicons and remove all words that are not associated\n",
    "# to the Positive or Negative class.\n",
    "\n",
    "hu_liu_fp = Path(read_config('hu_liu_lexicon', CONFIG_FP))\n",
    "mpqa_fp = Path(read_config('mpqa_lexicon', CONFIG_FP))\n",
    "nrc_fp = Path(read_config('nrc_emotion_lexicon', CONFIG_FP))\n",
    "\n",
    "subset_cats = {'positive', 'negative'}\n",
    "mpqa = lexicons.Mpqa(mpqa_fp, subset_cats=subset_cats)\n",
    "nrc = lexicons.NRC(nrc_fp, subset_cats=subset_cats)\n",
    "hu_liu = lexicons.HuLiu(hu_liu_fp, subset_cats=subset_cats)\n",
    "# Combine sentiment lexicons - Removes words that contradict each other.\n",
    "mpqa_huliu = lexicons.Lexicon.combine_lexicons(mpqa, hu_liu)\n",
    "all_three = lexicons.Lexicon.combine_lexicons(mpqa_huliu, nrc)\n",
    "\n",
    "# Load the sentiment lexicons but lower case all the words\n",
    "mpqa_low = lexicons.Mpqa(mpqa_fp, subset_cats=subset_cats, lower=True)\n",
    "nrc_low = lexicons.NRC(nrc_fp, subset_cats=subset_cats, lower=True)\n",
    "hu_liu_low = lexicons.HuLiu(hu_liu_fp, subset_cats=subset_cats, lower=True)\n",
    "mpqa_huliu_low = lexicons.Lexicon.combine_lexicons(mpqa_low, hu_liu_low)\n",
    "all_three_low = lexicons.Lexicon.combine_lexicons(mpqa_huliu_low, nrc_low)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Paper No. Positive</th>\n",
       "      <th>Ours No. Positive</th>\n",
       "      <th>Ours low No. Positive</th>\n",
       "      <th>Paper No. Negative</th>\n",
       "      <th>Ours No. Negative</th>\n",
       "      <th>Ours low No. Negative</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>MPQA</th>\n",
       "      <td>2289</td>\n",
       "      <td>2298</td>\n",
       "      <td>2298</td>\n",
       "      <td>4114</td>\n",
       "      <td>4148</td>\n",
       "      <td>4148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hu Liu</th>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>4780</td>\n",
       "      <td>4780</td>\n",
       "      <td>4780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NRC</th>\n",
       "      <td>2231</td>\n",
       "      <td>2231</td>\n",
       "      <td>2231</td>\n",
       "      <td>3243</td>\n",
       "      <td>3243</td>\n",
       "      <td>3243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MPQA &amp; Hu Liu</th>\n",
       "      <td>2706</td>\n",
       "      <td>2725</td>\n",
       "      <td>2725</td>\n",
       "      <td>5069</td>\n",
       "      <td>5080</td>\n",
       "      <td>5076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All Three</th>\n",
       "      <td>3940</td>\n",
       "      <td>4016</td>\n",
       "      <td>4016</td>\n",
       "      <td>6490</td>\n",
       "      <td>6530</td>\n",
       "      <td>6526</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Paper No. Positive  Ours No. Positive  Ours low No. Positive  \\\n",
       "MPQA                         2289               2298                   2298   \n",
       "Hu Liu                       2003               2003                   2003   \n",
       "NRC                          2231               2231                   2231   \n",
       "MPQA & Hu Liu                2706               2725                   2725   \n",
       "All Three                    3940               4016                   4016   \n",
       "\n",
       "               Paper No. Negative  Ours No. Negative  Ours low No. Negative  \n",
       "MPQA                         4114               4148                   4148  \n",
       "Hu Liu                       4780               4780                   4780  \n",
       "NRC                          3243               3243                   3243  \n",
       "MPQA & Hu Liu                5069               5080                   5076  \n",
       "All Three                    6490               6530                   6526  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def filter_cat(lexicon, filter_cat):\n",
    "    return [word for word, cat in lexicon.lexicon if cat == filter_cat]\n",
    "\n",
    "all_lexicons = [mpqa, hu_liu, nrc, mpqa_huliu, all_three]\n",
    "num_positive = [len(filter_cat(lexicon, 'positive')) for lexicon in all_lexicons]\n",
    "num_negative = [len(filter_cat(lexicon, 'negative')) for lexicon in all_lexicons]\n",
    "\n",
    "all_lexicons_low = [mpqa_low, hu_liu_low, nrc_low, mpqa_huliu_low, all_three_low]\n",
    "num_positive_low = [len(filter_cat(lexicon, 'positive')) for lexicon in all_lexicons_low]\n",
    "num_negative_low = [len(filter_cat(lexicon, 'negative')) for lexicon in all_lexicons_low]\n",
    "\n",
    "columns = ['Paper No. Positive', 'Ours No. Positive', 'Ours low No. Positive', \n",
    "           'Paper No. Negative', 'Ours No. Negative', 'Ours low No. Negative']\n",
    "index = ['MPQA', 'Hu Liu', 'NRC', 'MPQA & Hu Liu', 'All Three']\n",
    "data = [[2289, 2003, 2231, 2706, 3940], num_positive, num_positive_low, \n",
    "        [4114, 4780, 3243, 5069, 6490], num_negative, num_negative_low]\n",
    "senti_info = dict(list(zip(columns, data)))\n",
    "pd.DataFrame(senti_info, columns=columns, index=index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['anti-Semites', 'anti-Israeli', 'anti-US', 'anti-American']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Words that are shared between the MPQA and Hu Liu sentiment lexicons\n",
    "[word for word, cat in list(set(mpqa_huliu.lexicon).difference(set(mpqa_huliu_low.lexicon))) if cat == 'negative']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see we never agree on the number of words within the lexicons. We get the lexicons from the sources described in the paper. Intrestingly if we do not lower case the words in the lexicons we won't see the same similarities between the MPQA and Hu Liu sentiment lexicon as they both share the words above just the Hu Liu lexicon has the words lower cased already where as MPQA has not.\n",
    "\n",
    "## Showing the affect of using different sentiment lexicons in the Target-Dep+ model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instances of the model\n",
    "target_dep_plus = TargetDepPlus([w2v], hu_liu_low)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "senti_scores = grid_search_model(target_dep_plus, train_data, train_y, \n",
    "                                 word_vectors=[[w2v]], senti_lexicon=all_lexicons_low)\n",
    "senti_scores = get_score(senti_scores, 'lexicon')\n",
    "senti_scores = {f'Target Dependent Plus {name}': score for score, name in senti_scores}\n",
    "senti_scores['Target Dependent'] = all_results['Target Dependent']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target Dependent</th>\n",
       "      <th>Target Dependent Plus HuLiu</th>\n",
       "      <th>Target Dependent Plus Mpqa</th>\n",
       "      <th>Target Dependent Plus Mpqa HuLiu</th>\n",
       "      <th>Target Dependent Plus Mpqa HuLiu NRC</th>\n",
       "      <th>Target Dependent Plus NRC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Our Results</th>\n",
       "      <td>66.81</td>\n",
       "      <td>68.61</td>\n",
       "      <td>66.81</td>\n",
       "      <td>68.37</td>\n",
       "      <td>68.23</td>\n",
       "      <td>67.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Paper Results</th>\n",
       "      <td>65.72</td>\n",
       "      <td>67.24</td>\n",
       "      <td>65.56</td>\n",
       "      <td>67.40</td>\n",
       "      <td>67.30</td>\n",
       "      <td>66.05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Target Dependent  Target Dependent Plus HuLiu  \\\n",
       "Our Results               66.81                        68.61   \n",
       "Paper Results             65.72                        67.24   \n",
       "\n",
       "               Target Dependent Plus Mpqa  Target Dependent Plus Mpqa HuLiu  \\\n",
       "Our Results                         66.81                             68.37   \n",
       "Paper Results                       65.56                             67.40   \n",
       "\n",
       "               Target Dependent Plus Mpqa HuLiu NRC  Target Dependent Plus NRC  \n",
       "Our Results                                   68.23                      67.13  \n",
       "Paper Results                                 67.30                      66.05  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "senti_results = add_row(senti_scores, 'Our Results')\n",
    "paper_results = [('NRC', 66.05), ('HuLiu', 67.24), ('Mpqa', 65.56), \n",
    "                 ('Mpqa HuLiu', 67.40), ('Mpqa HuLiu NRC', 67.30)]\n",
    "paper_results = {f'Target Dependent Plus {name}': score for name, score in paper_results}\n",
    "paper_results['Target Dependent'] = 65.72\n",
    "senti_results = add_row(paper_results, 'Paper Results', senti_results)\n",
    "senti_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the results shown above we get different results but the results also have a different rank between the lexicons as in the best lexicon was **Hu and Liu** where as the papers original results show the combination of **MPQA and Hu & Liu** was the best. However in general we can see that it is better to use a sentiment lexicon than not. Also that both our implmentation and the original paper show that the best single sentiment lexicon is **Hu & Liu** and that using **all three** sentiment lexicons is worse than using **MPQA and Hu & Liu**.\n",
    "\n",
    "# Showing the affect of the different word vectors\n",
    "As presented in the paper they show the affect of using different word vectors accross the four models using the best sentiment lexicon for the sentiment dependent model. As we had different result for the sentiment lexicons compared to the original paper we will show the results of using **Hu & Liu** lexicon and using the combination of **Hu & Liu and MPQA**. The word vectors used are the following:\n",
    "1. Word2Vec - Which has been used throughout the previous experiments (100 dimensions)\n",
    "2. SSWE - Sentiment Specific Word Embeddings (50 dimensions)\n",
    "3. Concatenation of Word2vec and SSWE (150 dimensions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors = [[w2v], [sswe], [w2v, sswe]]\n",
    "vector_results = {}\n",
    "# Non Sentiment lexicon models\n",
    "for model in [target_ind, target_dep_minus, target_dep]:\n",
    "    vector_result = grid_search_model(model, train_data, train_y, word_vectors=vectors)\n",
    "    vector_result = get_score(vector_result, 'vector')\n",
    "    vector_results[str(model)] = vector_result\n",
    "# Sentiment lexicon model, requires a lexicon when grid searching\n",
    "for model, lexicon in [(target_dep_plus, hu_liu_low), (target_dep_plus, mpqa_huliu_low)]:\n",
    "    vector_result = grid_search_model(model, train_data, train_y, \n",
    "                                      word_vectors=vectors, senti_lexicon=[lexicon])\n",
    "    vector_result = get_score(vector_result, 'vector')\n",
    "    vector_results[f'{str(model)} {str(lexicon)}'] = vector_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fbae46b5cc0>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAFjCAYAAADGq4k2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XucHVWd7/3PLzdCEiZAjIxjEAIi\nILdOk6DAdDQEjBcgyISDXAY8gDkOGm6DAyLj6KPwcAB9uHkERhA5g4DgBQhyG7mkZTTQIQgCAUS5\nNKJEIIGEiQnwe/6o6nTTJLv37q61115V3/fr1a/0rl27evU3q3rtVbvqV+buiIiISJqGxW6AiIiI\nDJ4GchERkYRpIBcREUmYBnIREZGEaSAXERFJmAZyERGRhAUbyM1sWzN7sM/Xq2Z2gpltamZ3mNmT\n+b+bhGqDiIhI2VkzriM3s+HA88CHgC8AL7v7WWZ2KrCJu58SvBEiIiIl1KxD6zOBp9z9GWA28IN8\n+Q+AA5rUBhERkdIZ0aSf8xng6vz7zdz9hfz7PwGbresFZjYXmAswduzYXbfbbrvgjRQREWkFixYt\n+ou7T6xn3eCH1s1sFPBHYAd3/7OZLXP3jfs8/4q71/ycfOrUqd7V1RW0nSIiIq3CzBa5+9R61m3G\nofVPAA+4+5/zx382s/cA5P++2IQ2iIiIlFIzBvJD6D2sDnAjcGT+/ZHADU1og4iISCkFHcjNbCyw\nD/CTPovPAvYxsyeBvfPHIiIiMghBT3Zz95XAhH7LXiI7i31I1qxZQ3d3N6tWrRrqpqSP0aNHM2nS\nJEaOHBm7KSIiUodmnbVeuO7ubjbaaCO23HJLzCx2c0rB3XnppZfo7u5m8uTJsZsjIiJ1SLZE66pV\nq5gwYYIG8QKZGRMmTNBRDhGRhCQ7kAMaxANQpiIiaUl6IBcREam6ZD8j72+zeZet/X7FwuyKtnEf\nml3Xa9e1/p8vPHrQbXn99dc56KCDeOqppxg+fDj77bcfZ52lk/NFRKR4mpEHcvLJJ7NkyRIWL17M\nvffeyy233BK7SSIiUkIayAfpnHPO4YILLgDgxBNPZK+99gLgzjvv5HOf+xwzZswAYNSoUbS3t9Pd\n3c3y5cvZYosteOuttwBYuXIlm2++OWvWrInzS4iISPI0kA9SR0cHnZ2dAHR1dbFixQrWrFlDZ2cn\n06dPX7vesmXLuOmmm5g5cybjx4+nra2Ne+65B4D58+cza9YsXbMtIiKDVprPyJtt1113ZdGiRbz6\n6qtssMEGtLe309XVRWdn59qZ+htvvMEhhxzCcccdx1ZbbQXAwQcfzLXXXsuMGTO45pprOPbYY2P+\nGiKSmDuOmhR0+/tc3h10+1I8zcgHaeTIkUyePJkrrriCPfbYg46ODu666y5+97vfsf322wMwd+5c\nttlmG0444YS1r9t///259dZbefnll1m0aNHaQ/IiIiKDoYF8CDo6Ojj33HOZPn06HR0dXHzxxUyZ\nMgUz4/TTT2f58uWcd955b3vNuHHjmDZtGscffzz77rsvw4cPj9R6EREpg9IcWn/75WKNXjo2uEvN\nOjo6OOOMM9h9990ZO3Yso0ePpqOjg+7ubs444wy222472tvbAfjiF7/IMcccA2SH1w866CDuvvvu\nQf1cERGRHqUZyGOYOXPm2844f+KJJ9Z+7+7rfd2cOXNqPi8iIlIvDeQiIgm5cvGrQbe/T9CtSwj6\njFxERCRhmpGLiCTkiCl/E7sJ0mI0kIuIiPQT8nr9oq/V16F1ERGRhGkgFxERSVhpDq1PPOcLNZ9f\n/dTzvPYft7HR4bMYtfV7B9ze0i99p6imiYiIBFOZGfmord/LRofP4rX/uI3VTz0fuzkiIiKFqMxA\nDsUP5itXruRTn/oUu+yyCzvuuCPXXnstp556Kh/84AfZeeedOfnkk3nzzTeZPHky7s6yZcsYPnw4\nCxYsAGD69Ok8+eSTrFy5kqOOOorddtuNKVOmcMMNNwy5bSIiUg2lObRer76Deb2H2dfn1ltv5e/+\n7u+4+eabAXjmmWf46le/ypIlSzCztQP3tttuy6OPPsof/vAH2tvb6ezs5EMf+hDPPfcc22yzDaed\ndhp77bUXl19+OcuWLWO33XZj7733ZuzYsUX92iIiUlKVmpH3KGpmvtNOO3HHHXdwyimn0NnZyXvf\n+15Gjx7N0UcfzU9+8hPGjBkDZDXZFyxYwIIFC/jyl7/ML3/5S+6//36mTZsGwO23385ZZ51FW1sb\nH/3oR1m1ahXPPvtsIb+riIiUWyUHcihmMP/ABz7AAw88wE477cTpp5/OmWeeyX333cecOXOYP38+\nH//4x4HsEHpnZyf33Xcfn/zkJ1m2bBl33303HR0dQFaX/cc//jEPPvggDz74IM8+++zaW6GKiIjU\nUrlD630N9TD7H//4RzbddFMOP/xwNt54Y8477zw+//nP88lPfpI999yTrbbaCoDddtuNf/zHf2Sr\nrbZi9OjRtLW1cckllzB//nwAZs2axYUXXsiFF16ImbF48WKmTJlS+O8rIulTrfXmCJlz0RmXZiAf\n0uViFw/uZQ8//DBf+tKXGDZsGCNHjuTb3/42++67L6tWrcLd+fa3vw3ABhtswOabb86HP/xhIDvU\nfvXVV7PTTjsB8K//+q+ccMIJ7Lzzzrz11ltMnjx57SAvIsUzM8ZM26+udV+//yaAhtbX3Q2lmUoz\nkMcwa9YsZs2a9bZl99133zrX7ezsXPv9oYceyqGHHrr28YYbbsgll1wSppEi8g5jpu3HuA/Nbug1\nja4fimqtN0dKOVf2M3IREZEy0EAuIiKSMA3kIiIiCdNn5CIiUrjN5l22zuUrFmaVK+s952B96//5\nwqOH0Lpy0YxcREQkYRrIRUREElaaQ+t3HDWp0O3tc3n3kF7/+uuvc9BBB/HUU08xfPhw9ttvP846\n66yCWiciIpLRjDygk08+mSVLlrB48WLuvfdebrnllthNEhGRktFAPkjnnHMOF1xwAQAnnngie+21\nFwB33nknhx12GGPGjGHGjBkAjBo1ivb2drq7u1m+fDlbbLEFb731FpDdCnXzzTdnzZo16/1Z99xz\nD21tbbS1tTFlyhRee+01XnjhBaZPn05bWxs77rgjnZ2dXHfddZx00kkAnH/++WtLxP7+979nzz33\nBGDRokV85CMfYdddd2XWrFm88MILYQISEZGmCHpo3cw2Br4H7Ag4cBTw32RFUUcDbwDHuvu6y6E1\noOi6uAPVwu3o6OBb3/oWxx13HF1dXfz1r39lzZo1dHZ2Mn369Letu2zZMm666SaOP/54xo8fT1tb\nG/fccw8zZsxg/vz5zJo1i5EjR673Z5177rl85zvfYc8992TFihWMHj2aSy+9lFmzZvGVr3yFN998\nk9dff52VK1dy9tlnA1kluQkTJvD888+vbdOaNWuYN28eN9xwAxMnTuTaa6/lK1/5CpdffvlQ4xJJ\nSk/Z1aLXHcz6jUql1nrP2eb9FZdn2LPWVWu91/nAre4+x8xGAWOAHwFfd/dbzOyTwNnARwO3o3C7\n7rorixYt4tVXX2WDDTagvb2drq4uOjs7187UAd544w0OOeQQjjvuuLUz5IMPPphrr72WGTNmcM01\n13DsscfW/Fl77rknJ510EocddhgHHnggkyZNYtq0aRx11FGsWbOGAw44gLa2NjbaaCNWrFjBa6+9\nxnPPPcehhx7KggUL6Ozs5MADD+Txxx/nt7/9Lfvsk3WjN998k/e85z3hQhIRkeCCDeRmNh6YDnwW\nwN1XA6vNzIGeIrbjgT8W8fOaXRd35MiRTJ48mSuuuII99tiDnXfembvuuovf/e53b7sF6dy5c9lm\nm2044YQT1i7bf//9Oe2003j55ZdZtGjR2sPy63PqqafyqU99ip///Ofsueee3HbbbUyfPp0FCxZw\n880389nPfpaTTjqJI444gj322IPvf//7bLvttnR0dHD55Zfzq1/9im9961s8++yz7LDDDvzqV78K\nlotUW9EnnfY31JNQe4SutT7+8wcM+q6KA/lhIjXAB8qr1Wvdq9Z6ZjKwFPi+mS02s++Z2VjgBOAc\nM3sOOBf48rpebGZzzazLzLqWLl0asJmD19HRwbnnnsv06dPp6Ojg4osvZsqUKZgZAKeffjrLly/n\nvPPOe9vrxo0bx7Rp0zj++OPZd999GT58OAAXXXQRF1100Tt+zlNPPcVOO+3EKaecwrRp01iyZAnP\nPPMMm222GZ/73Oc45phjeOCBB97RpilTpnDXXXexwQYbMH78eLbddluWLl26diBfs2YNjzzySMiI\nRCqp7y2SVz/1fOzmSMmFPLQ+AmgH5rn7QjM7HziVbBZ+orv/2Mz+B3AZsHf/F7v7pcClAFOnTh3w\nnoBFvVNvREdHB2eccQa77747Y8eOZfTo0XR0dADQ3d3NGWecwXbbbUd7ezsAX/ziFznmmGOA7PD6\nQQcdxN133712e0uWLFl7Ulpf5513HnfddRfDhg1jhx124BOf+ATXXHMN55xzDiNHjmTcuHFceeWV\na9v03HPPMX36dIYPH87mm2/OdtttB2Qn3V1//fUcd9xxLF++nDfeeIMTTjiBHXbYIWRMIpXUdzAP\nMTMX6RFyIO8Gut19Yf74erKB/O+B4/Nl15GdDJekmTNnvu1s8yeeeGLt95MmTap5T+I5c+a84/mn\nn3567T3M+7rwwgvfsezII4/kyCOPfMfyrbfe+m3bvf3229/2fFtbGwsWLFhvu0SkOBrMpRmCHVp3\n9z8Bz5nZtvmimcCjZJ+JfyRfthfwZKg2pGb+/PmMGjUqdjNEpEA6zC6hhT5rfR5wVX7G+u+B/wnc\nAJxvZiOAVcDcwG0QEYlKM3MJKehA7u4PAlP7Lf4lsGtB2197YpkUo9bHASIyeBrMJZRka62PHj2a\nl156iQkTJmgwL4i789JLLzF69OjYTREpJQ3mxZl4zhcG9brVTz1fV/4/HGzDIkh2IJ80aRLd3d20\n6qVpqRo9ejSTJoW9FlikyjSYx1XG/JMdyHsKsoiIpKaMg0lKypZ/sgO5iLSeVOqAh661Puag2mWX\nYfCDSSoZh661PuZ9mzfcpr4Gyj+lWuu6+5mISCS6NC2usuSvGbmIFCaV+tSha603otGZeSoZh661\n/hYPNNymdVlf/qnkDBrIpUJSuaGHVE/ZPrNNTer569C6iEgLKMth3lSlnL8GchGRFpHyYNJsIfLp\nm39KNJCLiLQQDeb1CZVPT/4p0UAuItJiNJgPLGQ++oxcRESGTIN5bcqnlwZyEZEWpcGqNuWT0UAu\nItLCNFjVpnw0kIuItLxUz6ZulqoP5ioII5WRSo3qlKWSceha6yPaZxd+wlTPYHXlwjsL3W5/qdZa\nL7qoi2qti4hUmM6mjqOqM3PNyKUyUqqdnKpUMg5da31E+7Bg5T5TyThWrfWiZuap5AyakYuIFK6q\nM8NWUbX8NZCLiARQtcGk1VQpfx1aF2khm8277B3Lek4aqvdQZK31/3zh0UNonTQq9btqpa4q+WtG\nLiISUJVmhq2oCvlrRi4tY+I5XxjS61c/9XzNd94/HNLWRQavKjPDVlX2/DUjl9KowjtvSZf6Z1xl\nzl8DuZRKmXdWSZ/6Z1xlzV8DuZROWXdWKQf1z7jKmL8GcimlMu6sUh7qn3GVLX+d7Cal1f8ElxTq\ngK+rPnVRtakzYS8/SyFjCF9rfcxBxw643mBPwEol42bXWm/UQPmr1rrIIISqTa27RkmrKtvMMDVl\nyV8zcmkZoS4N6dlZj1jdVeh2Q6hV9CVkbfCipFKfOnSt9UY0OjNPJeNYtdYbtb78U8kZNJBLCwl5\nneeord8Lj7X+QB5arWv1B7oOvx66Vn9wyn6dc6tLPX8dWpeWUZbDXKlS/nEp/7hSzl8DubSUlHem\nMlD+cSn/+umcml4ayKXl6I9ZXMo/LuVfn1D59OSfEg3k0pL0xywu5R+X8h9YyHz0GblIQfTHLC7l\nH5fyr0359Ao6kJvZxmZ2vZktMbPHzGz3fPm8fNkjZnZ2yDZI2rSzxqX841L+tSmfTOgZ+fnAre6+\nHbAL8JiZzQBmA7u4+w7AuYHbIInTzhqX8o9L+demfAIO5GY2HpgOXAbg7qvdfRnwT8BZ7v7XfPmL\nodog5aGdNS7lH1eqZ1M3S9X7Z8iCMJOBpcD3zWwXYBFwPPABoMPMzgBWASe7+/39X2xmc4G5AO97\n3/sCNlNSMdSiDSnUqA5da30o9anryT+FjCF8rfUR7bODVSi8cuGdhW63v1RrrRdd1EW11jMjgHbg\nu+4+BVgJnJov3xT4MPAl4EdmZv1f7O6XuvtUd586ceLEgM2UlFT9nXdsyr8+Ops6jqr2z5Az8m6g\n290X5o+vJxvIu4GfuLsD95nZW8C7yGbvIgMa7DvvFGonh661XkR96lr5p5AxhK+1PqJ9WLByn6lk\nHKvWelEz81RyhoAzcnf/E/CcmW2bL5oJPAr8DJgBYGYfAEYBfwnVDimnqr7zbhXKvzblE1fV8g99\n1vo84CozewhoA84ELge2MrPfAtcAR+azc5GGVG1nbTXKvzblE1eV8g86kLv7g/nn3Du7+wHu/kp+\n9vrh7r6ju7e7e9gzN6TUqrSztiLlX5vyiasq+auymySvKjtrEXSjieZT/4yrCvnrfuRSt83mXVbX\nej2XndR7MkvP+mMOCntplBD0fu8bHT4LVuue7+ui/hlX2fPXjFxKowrvvIdKN5qIR/0zrjLnr4Fc\nSqXMO2sRlE9cyj+usuavgVxKp6w7a1GUT1zKP64y5q+BXEqpjDtrkZRPXMo/rrLlr5PdpG7rq53c\n32BrKY856NiG21RL/xNcUqgD3sxa6yFOAEohYwhfa72evjzY/FPJuNm11hs1UP6qtS4yCLo0qvnK\nNjNJjfKPqyz5a0YudQtZmxrCXxp1RAKXRsWotV7kzDyV+tSha603otH8U8k4Vq31Rq0v/1RyBs3I\npYXo0qh4yjIzSZXyjyv1/DWQS8tIfWdKnfKPS/nHlXL+GsilpaS8M5WB8o9L+ddP59T0qnsgN7MP\nm9mtZna3mR0QslFSbfpjFpfyj0v51yfkx3AbHT6r8O2GtN6B3Mz+tt+ik4BPA58EvhGyUSL6YxaX\n8o9L+Q9M59T0qjUjv9jMvmpmo/PHy4A5ZIN52AsZRdAfs9iUf1zKvzbl02u9A7m7HwAsBuab2RHA\nCcAGwARAh9alKbSzxqX841L+tSmfTM3PyN39JmAWMB74KfCEu1/g7kub0TgR0M4am/KPS/nXpnxq\nf0a+v5ndBdwK/BY4GJhtZteY2dbNaqAIaGeNTfnHlerZ1M1S9f5Zq7LbN4HdgA2B29x9N+CfzWwb\n4AzgM01on7SQ2LXWh1qBLIUa1c2std6oevJPIWMIX2t9RPvsYBUKr1x4Z6Hb7S/VWutF3zugLLXW\nlwMHAv8AvNiz0N2fdHcN4hJF1d95x6b866OzqeOoav+sNSP/NHAIsAY4tDnNkVaWam3qHinUTo5R\na71RtfJPIWMIX2t9RPuwYPcOSCXjWLXWi5qZp5Iz1D5r/S/ufqG7X+zuutxMWkpV33m3CuVfm/KJ\nq2r5q0SrJKtqO2urUf61KZ+4qpS/BnJJWpV21lak/GtTPnFVJf8BB3Izm2dmmzSjMSKDUZWdtQi6\n0UTzqX/GVYX865mRbwbcb2Y/MrOPm5mFbpRIo6qwsxZBN5qIQ/0zrrLnP+BA7u6nA9sAlwGfBZ40\nszNVFEZaTdl31iLoRhPxqH/GVeb86/qM3N0d+FP+9QawCXC9mZ0dsG0iDSvzzloE5ROX8o+rrPnX\n8xn58Wa2CDgbuBfYyd3/CdiVrFiMSEsp685aFOUTl/KPq4z51zMj3xQ40N1nuft17r4GwN3fAvYN\n2jqRQSrjzlok5ROX8o+rbPnXquzW4xbg5Z4HZvY3wPbuvtDdHwvWMmk5sWutN6p/hacU6oA3s9Z6\n0bWpQbXWe9avpy8PNv9UMm52rfVGDZR/WWqt9/gusKLP4xX5MpFC6dKo5ivbzCQ1yj+usuRfz4zc\n8pPdgOyQupnV8zopmdC11kPVpu7ZWY9Y3VXodkOIUWu9yJl5KvWpQ9dab0Sj+aeScaxa641aX/6p\n5Az1zch/b2bHmdnI/Ot44PehGybVo0uj4inLzCRVyj+u1POvZyD/PLAH8DzQDXwImBuyUVJNqe9M\nqVP+cSn/uFLOv56CMC+6+2fc/d3uvpm7H+ruLw70OpHBSHlnKgPlH5fyr5/OqelVz3Xko83sC2b2\nf8zs8p6vejZuZhub2fVmtsTMHjOz3fs8989m5mb2rqH8AlI++mMWl/KPS/nXR+WGe9VzaP3/An8L\nzALuASYBr9W5/fOBW919O2AX4DEAM9sc+BjwbKMNlmrQH7O4lH9cyn9gOqemVz0D+fvd/V+Ble7+\nA+BTZJ+T12Rm44HpZDXacffV7r4sf/r/A/4F8PW8XER/zCJT/nEp/9qUT696BvI1+b/LzGxHYDzw\n7jpeNxlYCnzfzBab2ffMbKyZzQaed/ff1Hqxmc01sy4z61q6dGkdP07KSDtrXMo/LuVfm/LJ1DOQ\nX5rfj/x04EbgUeB/1/G6EUA78F13nwKsBL4GnAZ8daAXu/ul7j7V3adOnDixjh8nZaWdNS7lH5fy\nr035DDCQm9kw4FV3f8XdF7j7VvnZ65fUse1uoNvdF+aPrycb2CcDvzGzp8k+b3/AzP528L+CVIF2\n1riUf1ypnk3dLFXvnzUrtOVV3P4F+FGjG3b3P5nZc2a2rbs/DswEHnD3mT3r5IP5VHf/S6Pbl+aL\nXWt9qBXIUqhR3cxa642qJ/8UMobwtdZHtM8OVqHwyoV3Frrd/lKttV70vQPKVmv9P83sZDPb3Mw2\n7fmqc/vzgKvM7CGgDThz0C0VQe+8Y1P+9dHZ1HFUtX/WUzP94PzfL/RZ5sBWA73Q3R8EptZ4fss6\nfr60iFRrU/dIoXZyjFrrjaqVfwoZQ/ha6yPahwW7d0AqGceqtV7UzDyVnKG+ym6T1/E14CAuElJV\n33m3CuVfm/KJq2r511PZ7Yh1fTWjcSK1VG1nbTXKvzblE1eV8q/nM/Jpfb46yC4h2z9gm0TqVqWd\ntRUp/9qUT1xVyb+eQ+vz+nx9juwSsnHhmyZSn6rsrEXQjSaaT/0zrirkX8+MvL+VZNeCi7SMKuys\nRdCNJuJQ/4yr7PnX8xn5TWZ2Y/41H3gc+Gn4pok0puw7axF0o4l41D/jKnP+9czIzwW+lX/9v8B0\ndz81aKtEBqnMO2sRlE9cyj+usuZfz0D+LLDQ3e9x93uBl8xsy6CtEhmCsu6sRVE+cSn/uMqYfz0D\n+XXAW30ev5kvE2lZZdxZi6R84lL+cZUt/3oqu41w99U9D9x9tZmNCtgmaVGxa603qn+FpxTqgDez\n1nrRtalBtdZ71q+nLw82/1Qybnat9UYNlH/Zaq0vNbO1143n9xPXTU6kcLo0qvnKNjNJjfKPqyz5\n1zMj/zzZjU8uyh93A6rsVkGha62Hqk3ds7Mesbqr0O2GEKPWepEz81TqU4eutd6IRvNPJeNYtdYb\ntb78U8kZ6hjI3f0p4MNmNi5/vCJ4qyrmjqMmBd3+Ppd3B91+UYo+zNvXqK3fC4+1/kAeS4jD7FI/\n5R9X6vnXcx35mWa2sbuvcPcVZraJmX2zGY2TainLYa5UKf+4lH9cKedfz2fkn3D3ZT0P3P0V4JPh\nmiRVlvLOVAbKPy7lXz+dU9OrnoF8uJlt0PPAzDYENqixvsiQ6I9ZXMo/LuVfH5Ub7lXPQH4V8Asz\nO9rMjgbuAK4M2yypOv0xi0v5x6X8B6Zyw73qufvZ/wa+CWyff30jXyYSlP6YxaX841L+tSmfXnXd\n/czdb3X3k939ZGClmX0ncLtEAO2ssSn/uJR/bconU9dAbmZTzOxsM3sa+AawJGirRPrQzhqX8o9L\n+demfGoM5Gb2ATP7NzNbAlwIPAeYu89w9wub1kIRtLPGpvzjSvVs6mapev+sVRBmCdAJ7OvuvwMw\nsxOb0qqKSb12cn+haq0PtWhDCjk3s9Z6o+rJP4WMIXyt9RHts4NVKLxy4Z2Fbre/VGutF13UpSy1\n1g8EXgDuMrN/N7OZgBX880UaUvV33rEp//robOo4qto/1zsjd/efAT8zs7HAbOAE4N1m9l3gp+5+\ne5PaWHqp1PRNtTZ1jxRyjlFrvVG18k8hYwhfa31E+7Bg5T5TyThWrfWiZuap5Az1XX620t1/6O77\nAZOAxcApwVsmUkNV33m3CuVfm/KJq2r513XWeg93f8XdL3X3maEaJFKvqu2srUb516Z84qpS/g0N\n5CKtpko7aytS/rUpn7iqkr8GckleVXbWIuhGE82n/hlXFfLXQC6lUIWdtQi60UQc6p9xlT1/DeRS\nGmXfWYugG03Eo/4ZV5nz10AupVLmnbUIyicu5R9XWfPXQC6lU9adtSjKJy7lH1cZ89dALqVUxp21\nSMonLuUfV9nyr1VrXZoklfrUsWutN6p/hacUcm5mrfWia1NDOn05dK31evryYPNPJeNm11pv1ED5\nl6XWukhT6dKo5ivbzCQ1yj+usuSvGXkLSKWmb+ha66FqU/fsrEes7ip0uyHEqLVe5Mw8lb4cutZ6\nIxrNP5WMY9Vab9T68k8lZwg8Izezjc3sejNbYmaPmdnuZnZO/vghM/upmW0csg2SDl0aFU9ZZiap\nUv5xpZ5/6EPr5wO3uvt2wC7AY8AdwI7uvjPwBPDlwG2QRKS+M6VO+cel/ONKOf9gA7mZjQemA5cB\nuPtqd1/m7re7+xv5ar8mu6OaCJD2zlQGyj8u5V8/nVPTK+SMfDKwFPi+mS02s+/l9zbv6yjglnW9\n2MzmmlmXmXUtXbo0YDOl1eiPWVzKPy7lXx+VG+4VciAfAbQD33X3KcBK4NSeJ83sK8AbwFXrenF+\nu9Sp7j514sSJAZsprUh/zOJS/nEp/4HpnJpeIQfybqDb3Rfmj68nG9gxs88C+wKHubsHbIMkTH/M\n4lL+cSn/2pRPr2ADubv/CXjOzLbNF80EHjWzjwP/Auzv7q+H+vlSDtpZ41L+cSn/2pRPJvRZ6/OA\nq8zsIaANOBO4CNgIuMPMHjQjuZbvAAAXuUlEQVSziwO3QRKnnTUu5R+X8q9N+QQeyN39wfxz7p3d\n/QB3f8Xd3+/um7t7W/71+ZBtkHLQzhqX8o8r1bOpm6Xq/VOV3VpA6rWT+wtVa32oFchSyLmZtdYb\nVU/+KWQM4Wutj2ifHaxC4ZUL7yx0u/2lWmu96HsHqNa6SCBVf+cdm/Kvj86mjqOq/VMz8haQSk3f\nVGtT90gh5xi11htVK/8UMobwtdZHtA8Ldu+AVDKOVWu9qJl5KjmDZuSSqKq+824Vyr825RNX1fLX\nQC7JqtrO2mqUf23KJ64q5a+BXJJWpZ21FSn/2pRPXFXJXwO5JK8qO2sRdKOJ5lP/jKsK+Wsgl1Ko\nws5aBN1oIg71z7jKnr8GcimNsu+sRdCNJuJR/4yrzPlrIJdSKfPOWgTlE5fyj6us+Wsgl9Ip685a\nFOUTl/KPq4z5ayCXUirjzlok5ROX8o+rbPmrslsLSKU+dexa643qX+EphZybWWu96NrUkE5fDl1r\nvZ6+PNj8U8m42bXWGzVQ/qq1LjIIujSq+co2M0mN8o+rLPlrRt4CUqnpG7rWeqja1D076xGruwrd\nbggxaq0XOTNPpS+HrrXeiEbzTyXjWLXWG7W+/FPJGTQjlxaiS6PiKcvMJFXKP67U89dALi0j9Z0p\ndco/LuUfV8r5ayCXlpLyzlQGyj8u5V8/nVPTqzSfkW8277K13/ecLVnvZzD1rP/nC48eQuukESHO\nppb6Kf+4lH99Qp9TQwLn1PTQjFxakmYmcSn/uJT/wHROTS8N5NKy9McsLuUfl/KvTfn00kAuLU07\na1zKPy7lX5vyyWggl5annTUu5R+X8q9N+Wggl0RoZ41L+ceV6tnUzVL1/lmas9b71vUtsjZ1r3Bn\nradeO7m/ULXWh3o2bwo5N7PWeqPqyT+FjCF8rfUR7bODnU195cI7C91uf6nWWi/6bH/VWhcJpOrv\nvGNT/vXR2dRxVLV/lmZGvq5rwFullvJAUqnp2yp5Dvaddwo5x6i13qha+aeQMYSvtT6ifViw65xT\nyThWrfWiZuap5AyakUuiqvrOu1Uo/9qUT1xVy18DuSSrajtrq1H+tSmfuKqUvwZySVqVdtZWpPxr\nUz5xVSV/DeSSvKrsrEXQjSaaT/0zrirkr4FcSqEKO2sRQp5NvdHhswrfblmof8ZV9vw1kEtplH1n\nLYJuNBGP+mdcZc5fA7mUSpl31iIon7iUf1xlzV8DuZROWXfWoiifuJR/XGXMXwO5lFIZd9YiKZ+4\nlH9cZcu/NJXdQtdan3jO4CpirX7q+QErDH08kfrUsWutN6p/hacU6oA3s9Z60bWpQbXWe9avpy8P\nNv9UMm52rfVGDZS/aq3nzGxjM7vezJaY2WNmtruZbWpmd5jZk/m/m4RsQ2xle+cXki6Naj71z7iU\nf1xlyT/0jPx84FZ3n2Nmo4AxwGnAL9z9LDM7FTgVOGWoPyh0rfWh1Kge6J1fKjV9Q9daD1Wbuif/\nI1Z3FbrdEGLUWi9yZp5KXw5da70RjeafSsaxaq03an35p5IzBJyRm9l4YDpwGYC7r3b3ZcBs4Af5\naj8ADgjVhlZSlnd+IenSqHjUP+NS/nGlnn/IQ+uTgaXA981ssZl9z8zGApu5+wv5On8CNlvXi81s\nrpl1mVnX0qVLAzazeVLvLKEpn7iUf1zKP66U8w85kI8A2oHvuvsUYCXZYfS13N0BX9eL3f1Sd5/q\n7lMnTpwYsJnNlXJnaQblE5fyj0v510/n1PQKOZB3A93uvjB/fD3ZwP5nM3sPQP7viwHb0JK0s9am\nfOJS/nEp//qo3HCvYAO5u/8JeM7Mts0XzQQeBW4EjsyXHQnUd01TyaT6zq9Z9McsLuUfl/IfmM6p\n6RW6IMw84CozewhoA84EzgL2MbMngb3zxy1P7/yaT3/M4lL+cSn/2pRPr6ADubs/mH/OvbO7H+Du\nr7j7S+4+0923cfe93f3lkG0oit75xaGdNS7lH5fyr035ZFSitU7qLPFoZ41L+cel/GtTPhrI66bO\nEpfyj0v5x6Vzamqrev9UrfU61x/zvs2D1KaG9Gsn9xeq1vpQ808h52bWWm9UPfmnkDGEr7U+on12\nsAqFVy68s9Dt9pdqrfWi/z6r1nqJVf2dX2zKPy7lXx+dUxNHVftnaWbkzay1XvQ7v1Rq+qZam7pH\nCjnHqLXeqFr5p5AxhK+1PqJ9WLB7B6SScaxa60X9fU4lZ9CMfNCq+s6vVSj/uJR/bconrqrlr4F8\nCKrWWVqN8o9L+demfOKqUv4ayIeoSp2lFSn/uJR/bconrqrkr4G8AFXpLK1K+ddPN5poPvXPuKqQ\nvwbyglShs7Qy5V8f3WgiDvXPuMqevwbyApW9s7Q65T8w3WgiHvXPuMqcvwbygpW5s6RA+demfOJS\n/nGVNX8N5AGUtbOkQvnXpnziUv5xlTF/DeSBlLGzpET516Z84lL+cZUt/9JUdmtGrfVG1VthKJX6\n1LFrrTeqf/4p5NzMWush7h2QQsYQvtZ6PX15sPmnknGza603aqD8VWtd1irbO7+QdGlU86l/xqX8\n4ypL/qWZkTez1nqjBnrnl0pN39C11kPVpu7J/4jVXYVuN4QYtdaLnJmn0pdD11pvRKP5p5JxrFrr\njVpf/qnkDJqRN01Z3vmFpEuj4lH/jEv5x5V6/hrImyj1zhKa8olL+cel/ONKOX8N5E2WcmdpBuUT\nl/KPS/nXT+fU9NJAHoF21tqUT1zKPy7lXx+VG+6lgTySVN/5NYv+mMWl/ONS/gPTOTW9NJDXSe/8\nmk9/zOJS/nEp/9qUTy8N5HXSO784tLPGpfzjUv61KZ+MBvI6qbPEo501LuUfl/KvTfloIK+bOktc\nyj8u5R+Xzqmprer9szSV3ZpRaz1EbWpIv3Zyf6FqrQ81/xRybmat9UbVk38KGUP4Wusj2mcHq1B4\n5cI7C91uf6nWWi/677NqrZdY1d/5xab841L+9dE5NXFUtX+WZkbezFrrRb/zS6Wmb6q1qXukkHOM\nWuuNqpV/ChlD+FrrI9qHBbt3QCoZx6q1XtTf51RyBs3IB62q7/xahfKPS/nXpnziqlr+GsiHoGqd\npdUo/7iUf23KJ64q5a+BfIiq1FlakfKPS/nXpnziqkr+GsgLUJXO0qqUf/10o4nmU/+Mqwr5ayAv\nSBU6SytT/vXRjSbiUP+Mq+z5ayAvUNk7S6tT/gPTjSbiUf+Mq8z5ayAvWJk7SwqUf23KJy7lH1dZ\n8w86kJvZ02b2sJk9aGZd+bI2M/t1zzIz2y1kG2Ioa2dJhfKvTfnEpfzjKmP+zZiRz3D3Nnefmj8+\nG/i6u7cBX80fl04ZO0tKlH9tyicu5R9X2fKPUdnNgZ6SOeOBPxax0WbUWm9UvRWGUqlPHbvWeqP6\n559Czs2stR7i3gEpZAzha63X05cHm38qGTe71nqjBso/pVrr5u4Fb7LPxs3+ALxCNnhf4u6Xmtn2\nwG2AkR0R2MPdn1nHa+cCc/OH2wKPB2toGO8C/hK7ESWnjMNTxuEp4+ZILect3H1iPSuGHsjf6+7P\nm9m7gTuAecAc4B53/7GZ/Q9grrvvHawRkZhZV5+PEyQAZRyeMg5PGTdHmXMO+hm5uz+f//si8FNg\nN+BI4Cf5Ktfly0RERGQQgg3kZjbWzDbq+R74GPBbss/EP5KvthfwZKg2iIiIlF3Ik902A35qZj0/\n54fufquZrQDON7MRwCp6Pwcvm0tjN6AClHF4yjg8Zdwcpc056GfkIiIiEpYqu4mIiCRMA7mIiEjC\nYhSEaXlmNgH4Rf7wb4E3gaX5493cfXWAn9kOvNvdby16261IGTeHcg5PGYenjGvTQL4O7v4S0AZg\nZl8DVrj7ufW+3syGu/ubDf7YdmBHoOU7TRGUcXMo5/CUcXjKuDYdWm+Qmd1kZovM7BEzOyZfNsLM\nlpnZeWb2ELCbme1vZo/n615oZj/L1x1nZleY2X1mttjM9jOzDcnqzh+W30xmTsRfMTpl3BzKOTxl\nHJ4yBtxdXzW+gK8BJ/d5vGn+7xjgUWATsiMbDhzY57luYAuyUrTXAT/Lnzsb+Ez+/SbAE8Bo4Bjg\nvNi/rzIu75dyVsZl+FLG7/zSjLxxJ5rZb4BfAZOArfPlq8mq1wF8EHjc3Z/xrHdc3ef1HwO+YmYP\nAneRdZj3NaXl6VDGzaGcw1PG4VU+Y31G3gAz2xuYDnzY3f/bzH5J9p8O8N95BxlwM8AB7v5Uv21P\nL7a1aVLGzaGcw1PG4SnjjGbkjRkPvJx3mB2AaetZ71FgWzPb3MwMOLjPc7eR3TwGADObkn/7GrBR\ngDanRhk3h3IOTxmHp4zRQN6om4ExZvYo8E1g4bpWcvfXgS8C/wl0AcuA5fnTXwfGmtnDZvYI2ec9\nAHcCu+QnW7T2iRVhKePmUM7hKePwlDEq0RqMmY1z9xX5u79LgIfd/cLY7SoTZdwcyjk8ZRxemTPW\njDycf8pPnngU2BD498jtKSNl3BzKOTxlHF5pM9aMXEREJGGakYuIiCSskgO5ZfdEb2T9j5rZ/IJ+\n9tfM7ORBvnZLMzu0iHaEYmZv5pWQHjGz35jZP5tZtH5mZk+b2bsG+doDzOyDRbdpqJRxcyjn8JRx\nMSo5kCdsS6ClB3Kyazfb3H0HYB/gE8C/RW7TYB1AVkii1Sjj5lDO4SnjIsQuLRfji6zgPsBHgbuB\n64ElwFX0njfw8XzZA8AFwPx8+VjgcuA+YDEwO1/+WeCGfHtPAv/W5+d9hazs3y/JKgqdnC/fmqwg\n/yKgE9guX35F/jP/C/g9MCdf/muySyYeBE6MnWOtbPs83gp4iazownDgHOB+4CHgf/X5f1hAdinJ\n48DFwLD8uY+RVWx6gKys4rh8+dNkl408ADzcJ7sJwO3AI8D3gGeAd+XPHZ7/vz1Idtbq8J42A2cA\nv8kz3gzYA3gZ+EO+/taxs1XGyrlsOSvjgnKM/R8Zs/PkHWI5WVm/YXkH+HuyykDPAdvkHepH9A7k\nZwKH599vTDZAjyUbyF/IO8aGwG+BqcCueccZA/wN8Dt6B/JfANvk338IuDP//oq8Ew4je4f3uz7t\nnR87v3qy7bdsWd7Z5wKn58s2ILuec3L+e63Kd+LhwB3AHOBd+Q47Nn/NKcBX8++fBubl3x8LfC//\n/oI+63yKrN7yu4DtgZuAkflz/wc4Iv/egf3y78/u08YryN9EtdKXMlbOZclZGRfzpRKtcJ+7dwPk\nlyZsSfaO6w/u/mS+/D/IOhVk7/j27/M5d9+6vHd4drs9zOwnZG8KAH7qWUECzOzG/N9xZO/irssu\nawSyztrjZ+7+FvComW1W3K8b1ceAnfsUVxhP9mZpNdn/w+8BzOxqsuxWkb2RuTfPaBTZm60eP8n/\nXQQcmH8/ved7d7/ZzF7Jl88ke1N1f76tDYEX8+dWA/P7bGufAn7XWJRxcyjn8JRxnTSQw1/7fP8m\nA2diwD+4++NvW2j2IbJ3an15vv66DAOWuXtbHe1a3zZanpltRZbri2S/xzx3v63fOh9l/dnd4e6H\nrGfzPRnV+//2A3f/8jqeW+P5W+o6t9VSlHFzKOfwlPHg6GS3dVsCbGlmPXfR6dsxbgPm5dWB+tbl\nBdjHzDa17F62BwD3kh3qOcDMNjSzjYD9ANz9VeAPZnZQvh0zs10GaFcytX8BzGwi2edXF+Ud/zay\nogwj8+c/YGZj89V3M7PJ+RmrB5OdT/BrYE8ze3++/lgz+8AAP3YB+QmBZvYJstsSQvYxxhwze3f+\n3KZmtsUA22r5vJVxcyjn8JTx4GkgXwd3X0V2KP1mM3uA3kMqAN8ARgIPWVaX9xt9nrsP+DHZiRk/\ndvcud38AuJbsxIhbyE7c6HEYcLRlt+B7BJg9QNMeAt7ML9M4cdC/YFgb9lxOQlbX+Hayk0wgO5nk\nUeABM/st2QkkPe9m7wcuAh4jO2Hkp+6+lOzcg6vN7CGyw2TbDfDzvw5Mz3/+gcCzAO7+KHA6cHu+\nrTuA9wywrWuAL1lWa3nrAdZtJmXcHMo5PGVcAFV2K4iZfRaY6u5fjN2W1OSHyk52931jt6WslHFz\nKOfwlPE7aUYuIiKSMM3IRUREEqYZuURnZm7ZJX49j0eY2VLLy+Ka2f5mdmq8FkpVWW8J0d+a2XVm\nNiZf3lCZ5/Vs+7NmdlG/ZXeb2dQBXrfOMs9m9l9DbVMsTcjZzWzvPssOyJe19H3G66WBXFrBSmBH\ny872h+y6zOd7nnT3G939rCgtk6rrKSG6I9n1w5+P3aD1cfc9YrdhCELn/DDwmT6PDyE7AbkUNJBL\nq/g5WWUlyHayq3ue6DtzMbMrzOwCM/svM/t9zztq63djGzO7KD8BETM7y8weNbOHzOzcZv1CUjqd\nwPv7LgjZ7/rORs1sjpldUc/6tdqUiBA5d5JdsjbSsmJc7ycrpdqzvafN7Gwze9jM7utzCdtkM/tV\nvvybfTIeZ2a/MLMH8ucGuuIoqOgXsovkrgG+mu+sO5PVs+9Yz7rvIavktB1wI1mt/HUyswnAp8lq\nK7uZbVxoq6USzGwE2Q09bq1z/Xr73cFm9vd9Hr9/PetVQsCcnezytllkFeJuJCv32tdyd9/JzI4A\nzgP2Bc4HvuvuV5rZF/qsuwr4tLu/atnd0n5tZjd6pJPONCOXluDuD5GVxz2EbHZey8/c/a38Ws+B\nytcuJ9vpLjOzA4HXh9pWqZQNLSvd3EV2jfFldb6u3n53bX5IuS2v8tg15BanKXTOkE0WPpN/Xb2O\n56/u8+/u+fd79ln+f/usa8CZ+TXm/wm8l4H/FgWjGbm0khuBc8luijChxnrrKl/7Bm9/YzoawN3f\nMLPdyGonzwG+COxVUHul/P67RhllCNvv+s7uRjfwunW2qcUFz9nd7zOznYDX3f0Js3dUvvY6vu9x\nGDAR2NXd15jZ00TMWTNyaSWXA19394cH8dpngA+a2Qb54bWZsPbmNOPd/efAicBAZXBFGhGy3/3Z\nzLa3rAzpp4fapsQVlfOpwGnree7gPv/23GzlXnpPkjusz7rjgRfzQXwGMFD51qA0I5eWkd+F7oJB\nvvY5M/sR2e1j/0B2r3jIah/fYGajyWbvJxXRVhEI3u9OJbvL1lKyQ87j+jx3upmd0Kcdk+poU7KK\nytndb6nx9Cb5ofK/0nt/jeOBH5rZKcANfda9CrjJzB4m+79Z0uCvVCgVhBERkUrLD41Pdfe/DLDe\nCncfV2udGHRoXUREJGGakYuIiCRMM3IREZGEaSAXEZFk9K+/buuoWb+O12xp2T3N+y//f/rWYE+V\nzloXEZFKcvevxm5DETQjFxGRUsjvxTCnz+Oad0/ru35eb/1d+fdTzezuoI0tkGbkIiKSkp5yrj02\nJasKWVkayEVEJCVvK+ea3wWt5j3cy06H1kVEpCzW1mTPS9uOGsxrSaM+/VoayEVEpCyeBnbNv98f\nGDnI1/5DcU0KTwO5iIiUxb8DHzGz35DdinRln+e2NbPuPl8H9Xvt14HzzawLeLNJ7S2EKruJiIgk\nTDNyERGRhGkgFxERSZgGchERkYRpIBcREUmYBnIREZGEaSAXqQAzczP7jz6PR5jZUjOb3+B21taj\nHso6IlIcDeQi1bAS2NHMNswf7wM8H7E9IlIQDeQi1fFz4FP594cAV/c8YWabmtnPzOwhM/u1me2c\nL59gZreb2SNm9j3A+rzmcDO7z8weNLNLzGx4M38ZEcloIBepjmuAz5jZaGBnYGGf574OLHb3nYHT\ngCvz5f8G/NLddwB+CrwPwMy2Bw4G9sxvYPEmcFhTfgsReRvd/UykItz9ITPbkmw2/vN+T/89eX1p\nd78zn4n/DTAdODBffrOZvZKvP5OsLvX9ZgawIfBi6N9BRN5JA7lItdwInAt8FJgwhO0Y8AN3/3IR\njRKRwdOhdZFquRz4urs/3G95J/mhcTP7KPAXd38VWAAcmi//BLBJvv4vgDlm9u78uU3NbIvwzReR\n/jQjF6kQd+8GLljHU18DLjezh4DXgSPz5V8HrjazR4D/Ap7Nt/OomZ0O3J7f93kN8AXgmbC/gYj0\np7ufiYiIJEyH1kVERBKmgVxERCRhGshFREQSpoFcREQkYRrIRUREEqaBXEREJGEayEVERBL2/wOQ\nl63opyluagAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbae4827ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_score_vectors = [(fill(model,10), score, vectors.strip('[]')) for model, score_vectors in vector_results.items() \n",
    "                       for score, vectors in score_vectors]\n",
    "vector_graph = defaultdict(list)\n",
    "for model_name, score, vectors in model_score_vectors:\n",
    "    vector_graph['Model'].append(model_name)\n",
    "    vector_graph['Accuracy %'].append(score)\n",
    "    vector_graph['Word Vector(s)'].append(vectors)\n",
    "ax = sns.barplot(x=\"Model\", y=\"Accuracy %\", hue=\"Word Vector(s)\", \n",
    "                 data=pd.DataFrame(vector_graph), palette=sns.color_palette(\"colorblind\"))\n",
    "# Sets the hatching\n",
    "hatches = ['+', '\\\\', '-']\n",
    "hatch_num = 0\n",
    "for i in range(len(model_score_vectors)):\n",
    "    ax.patches[i].set_hatch(hatches[hatch_num])\n",
    "    if ((i + 1) % 5) == 0:\n",
    "        hatch_num += 1\n",
    "ax.set(ylim=(58, 70))\n",
    "ax.figure.set_size_inches(8,5)\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the results above using the combination of the two word vectors is best accross all models which is the finding in the original paper. Also that **Target-Dep+** > **Target-Dep** > **Target-Dep-** > **Target-Ind** which is also what the original paper found. However un-like the original paper we found that using the *SSWE* word vectors to be generally worse than using the *Word2Vec* vectors showing that using just semantic information is more important than using a vector model that was created by reducing the semantic and sentiment loss. Also we found that using **Hu & Liu** lexicon to be better than any other and any other combination of lexicons compared to the original paper which found using the combination of **MPQA and Hu & Liu** to be the best. Finally we can see that we got similar results to the original.\n",
    "\n",
    "# Results of the final models on the test data\n",
    "Here we show the affect of the **Target-Ind**, **Target-Dep**, and **Target-Dep+** models on the test data as reported in the paper where each model uses the best parameters found in the previous tests.\n",
    "\n",
    "For the **Target-Dep+** we show using **Hu & Liu** lexicon and using the combination of **MPQA and Hu & Liu** for direct comparison with the original paper as they found using **MPQA and Hu & Liu** to be better than **Hu & Liu** however we did not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the default model parameters to use a combination of word 2 vec and SSWE\n",
    "# vectors\n",
    "for model in [target_ind, target_dep, target_dep_minus]:\n",
    "        model.model_parameters = {'word_vectors': [w2v, sswe]}\n",
    "# Create two new sentiment model version: 1. using Hu Liu 2. using MPQA and Hu Liu\n",
    "target_sent_hu = TargetDepPlus(word_vectors=[w2v, sswe], senti_lexicon=hu_liu_low)\n",
    "target_sent_mphu = TargetDepPlus(word_vectors=[w2v, sswe], senti_lexicon=mpqa_huliu_low)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit and predict with the models\n",
    "test_acc = {}\n",
    "test_f1 = {}\n",
    "for model in [target_ind, target_dep, target_dep_minus, target_sent_hu, target_sent_mphu]:\n",
    "    model.fit(train_data, train_y)\n",
    "    predictions = model.predict(test_data)\n",
    "    if model == target_sent_hu:\n",
    "        model_name = f'{str(model)} Hu Liu'\n",
    "    elif model == target_sent_mphu:\n",
    "        model_name = f'{str(model)} MPQA Hu Liu'\n",
    "    else:\n",
    "        model_name = f'{str(model)}'\n",
    "    test_acc[model_name] = round(accuracy_score(test_y, predictions) * 100, 4)\n",
    "    test_f1[model_name] = round(f1_score(test_y, predictions, average='macro') * 100, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target Dependent</th>\n",
       "      <th>Target Dependent Minus</th>\n",
       "      <th>Target Dependent Plus Hu Liu</th>\n",
       "      <th>Target Dependent Plus MPQA Hu Liu</th>\n",
       "      <th>Target Independent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Our Test Accuracy</th>\n",
       "      <td>69.65</td>\n",
       "      <td>69.36</td>\n",
       "      <td>70.81</td>\n",
       "      <td>69.94</td>\n",
       "      <td>66.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Paper Test Accuracy</th>\n",
       "      <td>69.70</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>71.10</td>\n",
       "      <td>67.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Test Macro F1</th>\n",
       "      <td>66.69</td>\n",
       "      <td>66.53</td>\n",
       "      <td>68.68</td>\n",
       "      <td>67.58</td>\n",
       "      <td>61.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Paper Test Macro F1</th>\n",
       "      <td>68.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>69.90</td>\n",
       "      <td>66.40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Target Dependent  Target Dependent Minus  \\\n",
       "Our Test Accuracy               69.65                   69.36   \n",
       "Paper Test Accuracy             69.70                     NaN   \n",
       "Our Test Macro F1               66.69                   66.53   \n",
       "Paper Test Macro F1             68.00                     NaN   \n",
       "\n",
       "                     Target Dependent Plus Hu Liu  \\\n",
       "Our Test Accuracy                           70.81   \n",
       "Paper Test Accuracy                           NaN   \n",
       "Our Test Macro F1                           68.68   \n",
       "Paper Test Macro F1                           NaN   \n",
       "\n",
       "                     Target Dependent Plus MPQA Hu Liu  Target Independent  \n",
       "Our Test Accuracy                                69.94               66.04  \n",
       "Paper Test Accuracy                              71.10               67.30  \n",
       "Our Test Macro F1                                67.58               61.95  \n",
       "Paper Test Macro F1                              69.90               66.40  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results = add_row(test_acc, 'Our Test Accuracy')\n",
    "paper_acc = {'Target Dependent': 69.7, 'Target Independent': 67.3, 'Target Dependent Plus MPQA Hu Liu': 71.1}\n",
    "test_results = add_row(paper_acc, 'Paper Test Accuracy', test_results)\n",
    "test_results = add_row(test_f1, 'Our Test Macro F1', test_results)\n",
    "paper_f1 = {'Target Dependent': 68.0, 'Target Independent': 66.4, 'Target Dependent Plus MPQA Hu Liu': 69.9}\n",
    "test_results = add_row(paper_f1, 'Paper Test Macro F1', test_results)\n",
    "test_results.astype(float).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from above our results are very close to those reported in the paper and are identical for **Target-Dep** model. Also as you can see that our results using the **Hu Liu** lexicon are much better and are closer to the results of **Target-Dep+** in the original paper "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine Tuning\n",
    "We want to see if instead of using the C-Value reported in the paper, we fine tune our-seleves the C-Value where we use the combination of **Word2Vec** and **SSWE** embeddings which was never done/shown in the paper to see if we get values closer to those reported in the paper. We only do this for the best model (**Target-Dep+ (Hu Liu)**)\n",
    "\n",
    "The C-Values to be selected from range from $2^{-15}$ to $2^1$ these values were chosen due to the suggestion in the Lib Linear SVM [guide](https://csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf), we also include the C value 1 as it is the default value for the SKlearn C value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "c_values = [1] # include the default sklearn C Value\n",
    "for c in range(-15, 3, 2):\n",
    "    c_values.append(math.pow(2, c))\n",
    "fine_values = grid_search_model(target_dep_plus, train_data, train_y, \n",
    "                                word_vectors=[[w2v, sswe]], senti_lexicon=[hu_liu_low], C=c_values)\n",
    "best_c = sorted(get_score(fine_values, 'C'), key=lambda x: x[0])[-1][1] \n",
    "best_c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_dep_plus = TargetDepPlus(word_vectors=[w2v, sswe], senti_lexicon=hu_liu_low, C=float(best_c))\n",
    "target_dep_plus.fit(train_data, train_y)\n",
    "fine_predictions = target_dep_plus.predict(test_data)\n",
    "model_name = f'{str(target_dep_plus)} Hu Liu'\n",
    "fine_acc = {model_name: accuracy_score(test_y, fine_predictions) * 100}\n",
    "fine_f1 = {model_name: f1_score(test_y, fine_predictions, average='macro') * 100}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target Dependent Plus Hu Liu</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tunned Result Accuracy</th>\n",
       "      <td>0.709538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Result Accuracy</th>\n",
       "      <td>70.809200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tunned Result F1</th>\n",
       "      <td>0.687001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Result F1</th>\n",
       "      <td>68.681900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Target Dependent Plus Hu Liu\n",
       "Tunned Result Accuracy                      0.709538\n",
       "Our Result Accuracy                        70.809200\n",
       "Tunned Result F1                            0.687001\n",
       "Our Result F1                              68.681900"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fine_tune_results = add_row(fine_acc, 'Tunned Result Accuracy')\n",
    "fine_tune_results = add_row({model_name: test_acc[model_name]}, 'Our Result Accuracy', fine_tune_results)\n",
    "fine_tune_results = add_row(fine_f1, 'Tunned Result F1', fine_tune_results)\n",
    "fine_tune_results = add_row({model_name: test_f1[model_name]}, 'Our Result F1', fine_tune_results)\n",
    "fine_tune_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see there is slight improvement but only marginal showing that optimising for different word vectors is not required. This mainly shows that there is probably no correlation between different word vectors and the C value.\n",
    "\n",
    "# Problems encountered when reproducing results\n",
    "When reproducing these methods the main error we came across was not **scaling/normalising** the features before inputting them into the SVM classifier.\n",
    "\n",
    "This was not stated in the paper. We show the affects of not doing this to the results accross all of the different models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the default model parameters to use a combination of word 2 vec and SSWE\n",
    "# vectors\n",
    "for model in [target_ind, target_dep, target_dep_minus]:\n",
    "        model.model_parameters = {'word_vectors': [w2v, sswe], 'scale': None}\n",
    "# Create two new sentiment model version: 1. using Hu Liu 2. using MPQA and Hu Liu\n",
    "target_sent_hu = TargetDepPlus(word_vectors=[w2v, sswe], senti_lexicon=hu_liu_low, scale=None)\n",
    "target_sent_mphu = TargetDepPlus(word_vectors=[w2v, sswe], senti_lexicon=mpqa_huliu_low, scale=None)\n",
    "\n",
    "scale_f1 = {}\n",
    "scale_acc = {}\n",
    "for model in [target_ind, target_dep, target_dep_minus, target_sent_hu, target_sent_mphu]:\n",
    "    model.fit(train_data, train_y)\n",
    "    predictions = model.predict(test_data)\n",
    "    if model == target_sent_hu:\n",
    "        model_name = f'{str(model)} Hu Liu'\n",
    "    elif model == target_sent_mphu:\n",
    "        model_name = f'{str(model)} MPQA Hu Liu'\n",
    "    else:\n",
    "        model_name = f'{str(model)}'\n",
    "    scale_acc[model_name] = round(accuracy_score(test_y, predictions) * 100, 4)\n",
    "    scale_f1[model_name] = round(f1_score(test_y, predictions, average='macro') * 100, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7AAAAFkCAYAAAAZjdfSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzs3Xu8lWWd///3JxERBVEyc1AGIvMQ\nchK10uxgxNh4yjGz0p1akWk2+c3GmvpONh1+lfWwMpvGMpVCm7TJ0zSmZX7NqVQQRIQco8ggS7II\nCQ2Fz++PvTYutntf617rPlz34fV8PPaDvdZe614X7/257mvf130ydxcAAAAAAGX3nNgNAAAAAAAg\nCTZgAQAAAACVwAYsAAAAAKAS2IAFAAAAAFQCG7AAAAAAgEpgAxYAAAAAUAkj8lqwme0r6T/annqB\npH+RNE7SOyStbT3/z+7+vbzaAQAA+jE2AwCqzoq4D6yZbSdpjaRDJZ0uaYO7fzb3DwYAAENibAYA\nVFFRhxAfKWmlu/+6oM8DAABhjM0AgMopagP2ZElXtz1+t5ktNbOvm9muBbUBAAA8g7EZAFA5uR9C\nbGYjJf1W0ovd/fdmtoekP0hySR+TtKe7nzHE++ZJmidJO+2000H77bdfru0EADTHokWL/uDuu8du\nRyyMzQCAskk6NhexAXucpLPd/bVD/GySpJvcfWpoGbNnz/aFCxfm00AAQOOY2SJ3nx27HbEwNgMA\nyibp2FzEIcRvUtshSma2Z9vPXi9pWQFtAAAAz2BsBgBUUm630ZEkM9tJ0hxJ72x7+jNmNkP9hymt\nGvQzAACQI8ZmAECV5boB6+5/kTR+0HOn5vmZAABgeIzNAIAqy3UDNk9PPfWUVq9erSeffDJ2U9Bm\n1KhR2muvvbT99tvHbgoAoGCMzeXE2AygTiq7Abt69WqNGTNGkyZNkpnFbg4kubsee+wxrV69WpMn\nT47dHABAwRiby4exGUDdFHUf2Mw9+eSTGj9+PANkiZiZxo8fz8w7ADQUY3P5MDYDqJvKbsBKYoAs\nIX4nANBsjAPlw+8EQJ1UegM2NjPT+973vq2PP/vZz+qCCy4Ivue6667T8uXLh/zZgw8+qFe+8pWa\nMWOG9t9/f82bN6+ndp122mm69tprE79+1apVmjo1eLs/AAAqgbEZAOqtsufADrbHOZdt/X7DXddL\nknY+9LhE7x3q9b+/+G0d37fDDjvoP//zP/XBD35Qz33ucxN91nXXXaejjz5aBxxwwLN+9p73vEfn\nnnuujjuuvx33339/omUCAFBG7WPzULodrxmbAQDsgU1hxIgRmjdvni666KJn/WzVqlV69atfrWnT\npunII4/Uww8/rJ/85Ce64YYb9P73v18zZszQypUrt3nPI488or322mvr4wMPPFCStHnzZp133nma\nOnWqpk2bposvvliS9K//+q86+OCDNXXqVM2bN0/u/qx2LFq0SK94xSt00EEHae7cuXrkkUe2Pj99\n+nRNnz5dl1xySWaZAAAQE2MzANQbG7ApnX322VqwYIH+/Oc/b/P8Oeeco7e+9a1aunSp3vKWt+g9\n73mPXvayl+nYY4/VhRdeqCVLlmjKlCnbvOfcc8/Vq1/9ah111FG66KKLtG7dOknSpZdeqlWrVmnJ\nkiVblydJ7373u3XPPfdo2bJleuKJJ3TTTTdts7ynnnpK55xzjq699lotWrRIZ5xxhj70oQ9Jkk4/\n/XRdfPHFuu+++/KKBgCAKBibAaC+2IBNaezYserr69MXv/jFbZ7/6U9/qje/+c2SpFNPPVV33nln\nx2WdfvrpWrFihd7whjfo9ttv10te8hL99a9/1Q9+8AO9853v1IgR/Ud877bbbpKkH/3oRzr00EN1\n4IEH6rbbbtMDDzywzfIefPBBLVu2THPmzNGMGTP08Y9/XKtXr9a6deu0bt06HXHEEVvbBwBAXTA2\nA0B91eYc2Jje+973atasWTr99NNTL+tv/uZvdMYZZ+iMM87Q1KlTtWzZsiFf9+STT+qss87SwoUL\ntffee+uCCy541iXy3V0vfvGL9dOf/nSb5wdmjwEAqCvGZgCoJ/bAZmC33XbTSSedpMsue+ZiFS97\n2cv0rW99S5K0YMECvfzlL5ckjRkzRo8//viQy7n55pv11FNPSZJ+97vf6bHHHtOECRM0Z84c/fu/\n/7uefvppSdIf//jHrQPic5/7XG3YsGHIKxvuu+++Wrt27dZB8qmnntIDDzygcePGady4cVtnnhcs\nWJBFDAAAlAZjMwDUExuwGXnf+96nP/zhD1sfX3zxxbr88ss1bdo0feMb39AXvvAFSdLJJ5+sCy+8\nUDNnznzWhSJuueUWTZ06VdOnT9fcuXN14YUX6vnPf77e/va3a+LEiZo2bZqmT5+uq666SuPGjdM7\n3vEOTZ06VXPnztXBBx/8rDaNHDlS1157rc4//3xNnz5dM2bM0E9+8hNJ0uWXX66zzz5bM2bMGPIC\nEwAAVB1jMwDUj1VhBTl79mxfuHDhNs+tWLFC+++/f6QWIYTfDYCyM7NF7j47djuqjLG5WvjdACi7\npGMze2ABAAAAAJXABiwAAAAAoBLYgAUAAAAAVAIbsAAAAACASmADFgAAAABQCWzAAgAAAAAqgQ3Y\nFLbbbjvNmDFDU6dO1THHHKN169YV+vmrVq3S1KlTe3rv7bffvvW+c5L0la98RfPnz8+qaQAARMHY\nDAD1NiJ2A7Ky+4Vnd3zNppVr9Pg3v68xp8zVyCkTgq9d+/5LOi5vxx131JIlSyRJb33rW3XJJZfo\nQx/6ULIGB2zevFnbbbdd6uWE3H777dp55531spe9TJJ05pln5vp5AIDmGRibuxl/QxibAQCN2gM7\ncsoEjTllrh7/5ve1aeWaTJf90pe+VGvWPLPMCy+8UAcffLCmTZumj3zkI5L6Z2X3228/veUtb9H+\n+++vE088URs3bpQkTZo0Seeff75mzZqla665RkuWLNFLXvISTZs2Ta9//ev1pz/9SZK0aNEiTZ8+\nXdOnT9cllzwzkF9xxRV697vfvfXx0Ucfrdtvv12SdPPNN2vWrFmaPn26jjzySK1atUpf+cpXdNFF\nF2nGjBn68Y9/rAsuuECf/exnJWnYz37lK1+p888/X4cccohe9KIX6cc//nGmGQIA6inP8TeEsRkA\n6qdRG7BSPoPo5s2b9cMf/lDHHnusJOmWW27RQw89pLvvvltLlizRokWLdMcdd0iSHnzwQZ111lla\nsWKFxo4dqy9/+ctblzN+/Hjde++9Ovnkk9XX16dPf/rTWrp0qQ488EB99KMflSSdfvrpuvjii3Xf\nffclatvatWv1jne8Q9/5znd033336ZprrtGkSZN05pln6txzz9WSJUv08pe/fJv3DPfZkvT000/r\n7rvv1uc///ltngcAIKTojVjGZgCop8ZtwErZDaJPPPGEZsyYoec///n6/e9/rzlz5kjqHyRvueUW\nzZw5U7NmzdLPf/5zPfTQQ5KkvffeW4cddpgk6ZRTTtGdd965dXlvfOMbJUl//vOftW7dOr3iFa+Q\n1H8I1B133KF169Zp3bp1OuKIIyRJp556asc2/uxnP9MRRxyhyZMnS5J222234OuH++wBJ5xwgiTp\noIMO0qpVqzp+PgAAA4rYiGVsXtXx8wGgyhq5AStlM4gOnGfz61//Wu6+9bAhd9cHP/hBLVmyREuW\nLNEvfvELve1tb5Mkmdk2y2h/vNNOO/X4v5FGjBihLVu2bH385JNP9ryskB122EFS/0Uynn766Vw+\nAwBQX3lvxDI2MzYDqLfGbsBK2Q2io0eP1he/+EV97nOf09NPP625c+fq61//ujZs2CBJWrNmjR59\n9FFJ0sMPP6yf/vSnkqSrrrpKhx9++LOWt8suu2jXXXfdeh7LN77xDb3iFa/QuHHjNG7cuK0zwwsW\nLNj6nkmTJmnJkiXasmWLfvOb3+juu++WJL3kJS/RHXfcoV/96leSpD/+8Y+SpDFjxujxxx9P/NkA\nkIdTZ47N5QvlVsSeWMZmAKin2lyFuFftg2iaqyPOnDlT06ZN09VXX61TTz1VK1as0Etf+lJJ0s47\n76xvfvOb2m677bTvvvvqkksu0RlnnKEDDjhA73rXu4Zc3pVXXqkzzzxTGzdu1Ate8AJdfvnlkqTL\nL79cZ5xxhsxMr33ta7e+/rDDDtPkyZN1wAEHaP/999esWbMkSbvvvrsuvfRSnXDCCdqyZYue97zn\n6dZbb9UxxxyjE088Uddff70uvvjiRJ8NAEBWshp/QxibAaB+zN1jt6Gj2bNn+8KFC7d5bsWKFdp/\n//0jtag3q1at0tFHH61ly5bFbkquqvi7ARDHrWfslcty53x9dfDnZrbI3Wfn8uENwdhcLVX83QBo\nlqRjc+P3wAKovyT3ie5kqPtYJrknJQAAALLT6HNgizZp0qTaz/ACdRXrPpYA8sXYDADVktsGrJnt\na2ZL2r7Wm9l7zWw3M7vVzB5q/btrXm0AgCyxEYuqY2wGAFRdbhuw7v6gu89w9xmSDpK0UdJ3JX1A\n0g/dfR9JP2w9BoBKaN+IBaqGsRkAUHVFHUJ8pKSV7v5rScdJurL1/JWSji+oDQCQiYGNWKDiGJsB\nAJVT1AbsyZKubn2/h7s/0vr+d5L2KKgNABoqj8N987jlB1AwxmYAQOXkfhViMxsp6VhJHxz8M3d3\nMxvyPj5mNk/SPEmaOHFirm3s1erVq3X22Wdr+fLl2rJli44++mhdeOGFGjly5Dav++1vf6v3vOc9\nuvbaa4PLe93rXqerrrpK48aN67otF1xwgXbeeWedd955Xb8XqLs87zPZFL1eyXmoqze3uyptw9AT\nxmbGZgCoqiJuo3OUpHvd/fetx783sz3d/REz21PSo0O9yd0vlXSp1H+vuU4fkvW9BDvdQ9DddcIJ\nJ+hd73qXrr/+em3evFnz5s3Thz70IV144YVbX/f000/rb/7mbzoOkJL0ve99L3W7ATzbwDmrbMQW\nr/2cYfIvFcZmxmYAGFKse7QnVcQG7Jv0zCFKknSDpLdK+lTr3+sLaEPmbrvtNo0aNUqnn366JGm7\n7bbTRRddpMmTJ2vy5Mm6+eabtWHDBm3evFlXXnnl1pukb9y4UaeddpqWLVumfffdV7/97W91ySWX\naPbs2Zo0aZIWLlyoDRs26KijjtLhhx+un/zkJ5owYYKuv/567bjjjvrqV7+qSy+9VJs2bdILX/hC\nfeMb39Do0aMjp1G8PDpWVp0K5cNGVFzkX0qMzYzNAFBJuW7AmtlOkuZIemfb05+S9G0ze5ukX0s6\nKYvPmr94fRaL2WpOh58/8MADOuigg7Z5buzYsZo4caKefvpp3XvvvVq6dKl22203rVq1autrvvzl\nL2vXXXfV8uXLtWzZMs2YMWPI5T/00EO6+uqr9dWvflUnnXSSvvOd7+iUU07RCSecoHe84x2SpA9/\n+MO67LLLdM4556T5r+aq18MOhzNwOOJb9tk+0+VKnX/nqDY2ouIaLv+s190D6M/DY2xmbAaAkLKP\nzblexMnd/+Lu4939z23PPebuR7r7Pu7+Gnf/Y55tiGXOnDnabbfdnvX8nXfeqZNPPlmSNHXqVE2b\nNm3I90+ePHnrAHrQQQdtHWiXLVuml7/85TrwwAO1YMECPfDAA/n8B0qKq78iDe7jGhf5lwNjM2Mz\nAFRZEYcQF6Jv5thCP++AAw541rkz69ev18MPP6wRI0Zop512SrX8HXbYYev32223nZ544glJ0mmn\nnabrrrtO06dP1xVXXKHbb7891efkbdPKNZnv6Ro5ZYL6NhX7+46pl73YnS6cI0lr339J2qZVEnti\n4xqcf9HrbhSLsRkAqqfsY3NRt9GpnSOPPFIbN27U/PnzJUmbN2/W+973Pp122mnB814OO+wwffvb\n35YkLV++XPfff39Xn/v4449rzz331FNPPaUFCxb0/h8oCHta4mBPVxj5xNWeP5AlxmYAqD82YHtk\nZvrud7+ra665Rvvss49e9KIXadSoUfrkJz8ZfN9ZZ52ltWvX6oADDtCHP/xhvfjFL9Yuu+yS+HM/\n9rGP6dBDD9Vhhx2m/fbbL+1/I3dsJMTDRloY+cTF6QDIA2MzANSfuXe8Cn50s2fP9oULF27z3IoV\nK7T//vtHalHvNm/erKeeekqjRo3SypUr9ZrXvEYPPvjgs+5PV2Xtv5vdLzw70eGs3bpqRfYXyCzr\nVYjTXghruPybdAhxKMM09dmkDHc58/hcDrfOoy9LnfuzmS1y99m5fHhDMDZXS1V/NwCKF+s2OknH\nZvbAFmzjxo06/PDDNX36dL3+9a/Xl7/85VoNkENhT1dc5B9GPsmQD+qsiWMzAFRVbS7iVBVjxozR\n4BnrJuDCOXGRfxj5dEY+qLOmjs1ATANHR2V9pF6Tjo5qKvbAojDs6YqL/MPIJ4x8AAB5YHxBtyq9\nAVuF83ebptPvhJVUXFz9NYz6DCMfJMHYXD78TlB2jC/oRmU3YEeNGqXHHnuMlXKJuLsee+wxjRo1\nKvg6VlJxcfXXMOozjHwQwthcPknHZiA2xhckVdlzYPfaay+tXr1aa9eujd0UtBk1apT22qvzlcs4\n5zC5TSvXZJ4PeYdRn2Hkg+EwNpdT0rEZiI3xBUlUdgN2++231+TJk2M3AymkWUnNX7w+8/bMyXyJ\n2WAlHgeDaFhW+eTRl6Xy9ue6Y2wGkBbjb3xlH5sruwGLemAl1Rn5xEN9hpEPgAGnzhyby3K/kdMf\n0ig3xheEsAGL6HpZSfXlNFCWESvxuMg/LG0+TerLQJ3Rl5E1xt94yt6fK3sRJ9QLJ+6HkU9c5B9G\nPgCAPDC+YCjsgUVpMNMWRj5xkX8Y+QAA8sD4MrzdLzxbUv8FP7PM56rUS8gXe2BRKsy0hZFPXOQf\nRj4AgDwwvoQ1LR/2wBbs1jPyuYz9nK+vzmW5MTDTFkY+cbXnr6/Ebk35UJ8AgDwwvoQ1KR/2wKKU\nmjaT1C3yiWsgfwyN+gQA5IHxJawp+bABi9JqSifsFfkkl0c+dZ7ZzAL1CQDIA+NLWBPyYQMWpdaE\nTpgG+SRDPnFQnwCAPDC+hNU9HzZgUXp174RpkU9n5BMP9QkAyAPjS1id8+EiTqiEwSemY1tNOnG/\nF+QTF/kD5TJw642kktyio+y33UA9Mb6E1TUfNmALNn/x+lyWOyeXpZZLeyecv8/2mS+/6hnWdSWV\nFfKJa7j8WScC5Zdk/dm0vjx4EiCr+3Cuff8laZvWOOzkCOvl75+y92c2YFEpW6/+etdtsZtSSmyk\nhZFPXOSPsjp15thclvuNnP4IjIH+G0Y+cXGLu7C61ScbsAXry2mQLKtNK9dk3klGTpmgvk3NyrEb\ndVtJZY184hqcf9PWiSgn6jCZ0PqTDBlfYuMWd2Hd1GfZ+zMXcUKu6njieBXU+cT9LJBPXNvMlAOo\nFNafYeSTHLe4K15d6pM9sAFJLnLQ7TkPTbvIATOR8TATHEY+cW2dKd+0MHZTAHSJ9WcY+SRDPnHU\noT7ZA5tSXWYy8kI+cZF/GPkkx0w5gHasP8PIpzPyiafq9ckGbAaqXgR5I5+4yD+MfJIhHwCDcTpA\nGONLGPnEVeX82YDNSJWLoAjkExf5h5FPZ+QDYChcOCeM8SWMfOKqav65bsCa2Tgzu9bMfm5mK8zs\npWZ2gZmtMbMlra/X5dmGIlW1CIpCPnGRfxj5hJFPfTRtbEb+OB0gjPVnGPnEVcX8894D+wVJN7v7\nfpKmS1rRev4id5/R+vpezm0oVBWLoEjkExeHe4VRn2HkUxuNG5uxLfpv8Vh/hpFPXFXLP7cNWDPb\nRdIRki6TJHff5O7r8vq8MqlaERSNfOLicK8w6jOMfKqtyWMznkH/jYP1Zxj5xFWlnRx57oGdLGmt\npMvNbLGZfc3Mdmr97N1mttTMvm5mu+bYhmjohGHkkxxXfy0e9RlGPpXW6LEZ/ei/8bD+DCOfuKqy\nkyPP+8COkDRL0jnufpeZfUHSByR9SdLHJHnr389JOmPwm81snqR5kjRx4sQcm5mfoe6zNH/x+lw+\na04uS81XmvtQ5ZFjWTOs8n26qqwO90nLU1b5sE4sXOPH5qE0rQ7zWL81LcM0GF/CyCe5TSvXZJ7P\nyCkTNP+q2zJd5oCs+nOee2BXS1rt7ne1Hl8raZa7/97dN7v7FklflXTIUG9290vdfba7z959991z\nbGa+mEkKI5/OyCce6jOMfCqJsRmS6L+xkX8Y+STT1Hxy2wPr7r8zs9+Y2b7u/qCkIyUtN7M93f2R\n1steL2lZXm0oi/ZO2HfS+NjNKZ1eZtr6Zo4toGXlwExkXOQfljafJvXlMmBsHlpT6zDL9VtTM0yD\n8SWMfDrLK5+y9+e8r0J8jqQFZrZU0gxJn5T0GTO7v/XcqySdm3MbSqEqx5THwkxbGPnERf5h5FM5\njM3Yiv4bF/mHkU9YU/PJdQPW3Ze0DjWa5u7Hu/uf3P1Udz+w9dyxbTO+tcfMUVhTO2FS5BMX+YeR\nT3UwNmMw+m9cVbr6awzUZ1gT88l7D2ylNaUIyqSJnbAb5BMX+YeRD1Bd9N+4OFIvjPoMa1o+bMAG\nNKUIyqZpnbBb5BMXM+Vh1CdQXfTf5LjFXfGoz7Am5cMGbEBTiqCMmtQJe0E+cTFTHkZ9AtVF/02G\nfOKgPsOakg8bsAFNKYKyIv8w8kmOmfLiUZ9AddF/OyOfeKjPsCbkwwZsB00ogjIj/zDySYZ84qA+\ngeqi/4aRT1zkH1b3fNiATaDuRVB25B9GPp2RTzzUJ1Bd9N8w8omL/MPqnA8bsAnVuQiqgPzDyCeM\nfOIif6C66L9h5BMX+YfVNZ8RsRtQJe1FMOaUuT2dAzd/8focWibNyWWp5dKe//x9ts98+VXPMIv6\nrDPyiWu4/Fknogyow7Ak688mZ8j4Etfg/LGtXuqz7P2ZPbBdqutMRlVw9dcw6jOMfOIif6C66L9h\n5BMXt7gLq1t9sge2B2lm2vpmjs2xZeWzaeWazGciR06ZoL5NzcqxG8wEh5FPXIPzb9o6EeVEHSYT\nWn+SIeNLbOzkCOumPsven9kD26O6zWTkhXzioD7DyCcuZsqB6mL9GUY+yXGLu+LVpT7ZgE2hLkWQ\nJ/KJh/oMI5+4mCkHqov1Zxj5JEM+cdShPtmATakORZAn8omL/MPIJzlmygG0Y/0ZRj6dkU88Va9P\nNmAzUPUiyBv5xEX+YeSTDPkAGIzTAcIYX8LIJ64q588GbEaqXARFIJ+4yD+MfDojHwBD4XSAMMaX\nMPKJq6r5swGboaoWQVHIJy7yDyOfMPIBMBxOBwhj/RlGPnFVMX82YDNWxSIoEvnExeFeYdRnGPkA\n9UD/LR7rzzDyiatq+bMBm4OqFUHRyCcuDvcKoz7DyAeoPvpvHKw/w8gnrirt5GADNid0wjDySY6r\nvxaP+gwjH6Da6L/xsP4MI5+4qrKTY0TsBtRZeyccc8pcjZwyQfMXr8/ls+bkstR8DZVPUnnkWNYM\ne8kH6aWpzybIKh/WiejW7heeneh1m1auSVyff9ewOsxj/UZfTo7xJYx8ktu0ck3m+YycMkHzr7ot\n02UOyKo/swc2Z8wkhZFPZ+QTD/UZRj4oM+ozjHziIv8w8kmmqfmwB7YA7Z2w76TxsZtTOr3MtPXN\nHFtAy8qBmci4yD8sbT5N6ssoXtL6bGodZrl+a2qGaTC+hJFPZ3nlU/b+zB7YglTlmPJYmGkLI5+4\nyD+MfFBm1GcY+cRF/mHkE9bUfNiALRAzR2FN7YRJkU9c5B9GPigz6jOMfOKq0tVfY6A+w5qYDxuw\nAU0pgjJpYifsBvnERf5h5IMyoz7DyCcujtQLoz7DmpZP4g1YM3uJmd1sZreb2fF5NqosmlIEZdO0\nTtgt8omLmfIw6hNlRn2GkU9y3OKueNRnWJPyGXYD1syeP+ip/yPp9ZJeJ+ljeTaqLJpSBGXUpE7Y\nC/KJi5nyMOoTZUZ9hpFPMuQTB/UZ1pR8Qntgv2Jm/2Jmo1qP10k6Uf0bsfnc7KtkmlIEZUX+YeST\nHDPlxaM+UWbUZxj5dEY+8VCfYU3IZ9gNWHc/XtJiSTeZWZ+k90raQdJ4SY04hFhqRhGUGfmHkU8y\n5BMH9Yky43SAMPpvGPnERf5hdc8neA6su98oaa6kXSR9V9L/uvsX3X1tEY0ri7oXQdmRfxj5dEY+\n8VCfKDNOBwij/4aRT1zkH1bnfELnwB5rZj+SdLOkZZLeKOk4M/uWmU1JsnAzG2dm15rZz81shZm9\n1Mx2M7Nbzeyh1r+7ZvNfyVedi6AKyD+MfMLIJy7yL5c6jc1Z4HSAMPpvGPnERf5hdc1nROBnH5d0\niKQdJX3f3Q+R9D4z20fSJySdnGD5X5B0s7ufaGYjJY2W9M+SfujunzKzD0j6gKTz0/wnitJeBGNO\nmdvToDd/cT6nD8/JZanl0p7//H22z3z5Vc8wi/qsM/KJa7j8WSdGUemxedPKNZn3X+owLMn6s8kZ\nMr7ENTh/bKuX+ix7fw4dQvxnSSdI+gdJjw486e4PuXvHjVcz20XSEZIua71vk7uvk3ScpCtbL7tS\nFTuftq4zGVXB4V5h1GcY+cRF/vHVYWymfuKg/4aRT1yc0x5Wt/oM7YF9vaQ3SXpK0pt7WPZkSWsl\nXW5m0yUtkvSPkvZw90dar/mdpD16WHZUaWba+maOzbFl5ZPHTPnIKRPUt6lZOXaDmeAw8olrcP5N\nWyeWQOXH5jz6L3WYTGj9SYaML7GxkyOsm/ose38OXYX4D+5+sbt/xd172Y88QtIsSf/m7jMl/UX9\nhyS1f4ZL8qHebGbzzGyhmS1cu7Z814yq20xGXsgnDuozjHziYqY8qsqPzfTfuMg/jHyS4xZ3xatL\nfQavQpzSakmr3f2u1uNr1T9o/t7M9pSk1r+PDvVmd7/U3We7++zdd989x2b2ri5FkCfyiYf6DCOf\nuJgpj6YWYzP9Ny7yDyOfZMgnjjrUZ24bsO7+O0m/MbN9W08dKWm5pBskvbX13FslXZ9XG4pQhyLI\nE/nERf5h5JMcM+X1UKexmf4bF/mHkU9n5BNP1euz4wasmZ2T4nL650haYGZLJc2Q9ElJn5I0x8we\nkvSa1uNKq3oR5I184iL/MPKwgOJWAAAgAElEQVRJhnxqpTZjM/03Lk4HCKM+w8gnrirnn2QP7B6S\n7jGzb5vZ35mZJV24uy9pHWo0zd2Pd/c/uftj7n6ku+/j7q9x9z/23vzyqHIRFIF84iL/MPLpjHzq\no25jM/03Lk4HCKM+w8gnrqrm33ED1t0/LGkf9V9y/zRJD5nZJ81sSs5tq5yqFkFRyCcu8g8jnzDy\nQZlRn3FxOkAY9RlGPnFVMf9E58C2rkj4u9bX05J2lXStmX0mx7ZVUhWLoEjkExeHe4VRn2HkgzKj\nPpMjn+JRn2HkE1fV8k9yDuw/mtkiSZ+R9D+SDnT3d0k6SNI/5Ny+SqpaERSNfOLicK8w6jOMfMrH\nzEaZWblv2lcQ6jMZ8omD+gwjn7iqtJMjyR7Y3SSd4O5z3f0ad39Kktx9i6Sjc21dhdEJw8gnOa7+\nWjzqM4x8ysPM3i7pOknfMbNPxm5PGVCfnZFPPNRnGPnEVZWdHCMSvOa/JW29mENrlnd/d7/L3Vfk\n1rIaaO+EY06Zq5FTJmj+4vW5fNacXJaar6HySSqPHMuaYS/5IL009dkEWeXDOrE7Znasu9/Q9tRr\n3P3vWj+7T9I/x2lZuXRbn02rwzzWb03LMA3GlzDySW7TyjWZ5zNyygTNv+q2TJc5IKv+nGQP7L9J\n2tD2eEPrOSTATFIY+XRGPvFQn2HkE8WBZna9mc1oPV5qZl8zs69KeiBmw8qG+gwjn7jIP4x8kmlq\nPkn2wFrrIk6S+g8dNrMk70NLeyfsO2l87OaUTi8zbX0zm3O6FzORcZF/WNp8mtSXs+DunzCz50v6\n19Zt7f6vpDGSdnT3pXFbVz5J67OpdZjl+q2pGabB+BJGPp3llU/Z+3OSPbC/NLP3mNn2ra9/lPTL\nvBtWN1U5pjwWZtrCyCcu8g8jn8L9RdJ7JX1J0qWS3iTpf6O2qMSozzDyiYv8w8gnrKn5JNmAPVPS\nyyStkbRa0qGS5uXZqLpi5iisqZ0wKfKJi/zDyKcYZvZxSd+RdJOkV7n7sZKWSPqemfVFbVyJUZ9h\n5BNXla7+GgP1GdbEfDpuwLr7o+5+srs/z933cPc3u/ujRTQutqYUQZk0sRN2g3ziIv8w8inE0e7+\nWklHSuqTpNZFnV6r/nu0YxjUZxj5xMWRemHUZ1jT8klyH9hRZna2mX3ZzL4+8FVE42JrShGUTdM6\nYbfIJy5mysOoz9wtM7NLJc2X9P8GnnT3p939C/GaVQ3UZxj5JMct7opHfYY1KZ8khxB/Q9LzJc1V\n/2C5l6TH82xUWTSlCMqoSZ2wF+QTFzPlYdRnftz9FEkXS/qEu58buz1VRH2GkU8y5BMH9RnWlHyS\nbMC+0N3/r6S/uPuVkv5e/efB1l5TiqCsyD+MfJJjprx41Gd+3P1+d/957HZUGfUZRj6dkU881GdY\nE/JJsgH7VOvfdWY2VdIukp6XX5PKpQlFUGbkH0Y+yZBPHNQnyozTAcLov2HkExf5h9U9nyQbsJea\n2a6SPizpBknLJX0611aVTN2LoOzIP4x8OiOfeKhPlBmnA4TRf8PIJy7yD6tzPsENWDN7jqT17v4n\nd7/D3V/QuhrxvxfUvtKocxFUAfmHkU8Y+cRF/sUws/1it6GKOB0gjP4bRj5xkX9YXfMZEfqhu28x\ns3+S9O2C2lNq7UUw5pS5PQ168xevz6Fl0pxcllou7fnP32f7zJdf9QyzqM86I5+4hsufdWKmbpE0\nMXYj8rZp5ZrM+y91GJZk/dnkDBlf4hqcP7bVS32WvT8nOYT4B2Z2npntbWa7DXxl9PmVU9eZjKrg\ncK8w6jOMfOIi//TM7IvDfF0saVzs9hWB+omD/htGPnFxTntY3eozuAe25Y2tf89ue84lvSD75lRD\nmpm2vpljc2xZ+eQxUz5yygT1bWpWjt1gJjiMfOIanH/T1okZOF3S+yT9dYifvangtkSRR/+lDpMJ\nrT/JkPElNnZyhHVTn2Xvzx33wLr75CG+GrvxOqBuMxl5IZ84qM8w8omLmfJU7pG0zN2vHPylhtyj\nnf4bF/mHkU9y3OKueHWpz44bsGbWN9RXEY0ru7oUQZ7IJx7qM4x84mKmvGcnSloy1A/cfXLBbYmG\n/hsX+YeRTzLkE0cd6jPJObAHt329XNIFko7NsU2VUociyBP5xEX+YeSTHDPlpbGzu2+M3YgyoP/G\nRf5h5NMZ+cRT9fpMcgjxOW1f75A0S9LO+TetOqpeBHkjn7jIP4x8kiGf0rhu4Bsz+07MhpQB/Tcu\nTgcIoz7DyCeuKuefZA/sYH+R1JjDlJKqchEUgXziIv8w8umMfErD2r5v/PUoJPpvbJwOEEZ9hpFP\nXFXNP8k5sDea2Q2tr5skPSjpu/k3rXqqWgRFIZ+4yD+MfMLIpzR8mO8bjfqMi9MBwqjPMPKJq4r5\nJ9kD+1lJn2t9/X+SjnD3D+TaqgqrYhEUiXzi4nCvMOozjHxKYbqZrTezxyVNa32/3sweN7N87jxf\nEdRncuRTPOozjHziqlr+STZgH5Z0l7v/P3f/H0mPmdmkXFtVcVUrgqKRT1wc7hVGfYaRT1zuvp27\nj3X3Me4+ovX9wONy37ivANRnMuQTB/UZRj5xVWknR5IN2GskbWl7vLn1HALohGHkkxxXfy0e9RlG\nPigz6rMz8omH+gwjn7iqspNjRJLXuPumgQfuvsnMRubYptpo74RjTpmrkVMmaP7ifI7wmpPLUvM1\nVD5J5ZFjWTPsJR+kl6Y+myCrfFgnIg/d1mfT6jCP9VvTMkyD8SWMfJLbtHJN5vmMnDJB86+6LdNl\nDsiqPyfZA7vWzLbe99XMjpP0h4w+v/aYSQojn87IJx7qM4x8UGbUZxj5xEX+YeSTTFPzSbIH9kxJ\nC8zsS63HqyX1JVm4ma2S9Lj6Dzt+2t1nm9kFkt4haW3rZf/s7t/rptFV094J+04aH7s5pdPLTFvf\nzOac6sVMZFzkH5Y2nyb15bJo0tictD6bWodZrt+ammEajC9h5NNZXvmUvT933APr7ivd/SWSDpB0\ngLu/zN1/0cVnvMrdZ7j77LbnLmo9N6MOA2QSVTmmPBZm2sLIJy7yDyOfSmrM2Ex9hpFPXOQfRj5h\nTc0nyX1gP2lm49x9g7tvMLNdzezjRTSubpg5CmtqJ0yKfOIi/zDyQZlRn2HkE1eVrv4aA/UZ1sR8\nkpwDe5S7rxt44O5/kvS6hMt3SbeY2SIzm9f2/LvNbKmZfd3Mdu2ivYVqShGUSRM7YTfIJy7yDyOf\nyqj02Nwr6jOMfOLiSL0w6jOsafkk2YDdzsx2GHhgZjtK2iHw+naHu/ssSUdJOtvMjpD0b5KmSJoh\n6RFJnxvqjWY2z8wWmtnCtWvXDvWS3DWlCMqmaZ2wW+QTFzPlYdRnJVR6bE6D+gwjn+S4xV3xqM+w\nJuWTZAN2gaQfmtnbzOxtkm6VND/Jwt19TevfRyV9V9Ih7v57d9/s7lskfVXSIcO891J3n+3us3ff\nffckH5e5phRBGTWpE/aCfOJipjyM+iy3qo/NaVGfYeSTDPnEQX2GNSWfJBdx+rSkj0vav/X1sdZz\nQWa2k5mNGfhe0mslLTOzPdte9npJy3ppeBGaUgRlRf5h5JMcM+XFoz7LqQ5jcxaozzDy6Yx84qE+\nw5qQT5I9sHL3m939PHc/T9JfzOySBG/bQ9KdZnafpLsl/Ze73yzpM2Z2v5ktlfQqSef22vgiNKEI\nyoz8w8gnGfKJg/ospVqMzVngdIAw+m8Y+cRF/mF1zyfRBqyZzTSzz7TuHfcxST/v9B53/6W7T299\nvdjdP9F6/lR3P9Ddp7n7se7+SKr/QQHqXgRlR/5h5NMZ+cRDfZZLncbmLHA6QBj9N4x84iL/sDrn\nM+wGrJm9yMw+YmY/l3SxpN9IMnd/lbtfXFgLS6LORVAF5B9GPmHkExf5o8w4HSCM/htGPnGRf1hd\n8xkR+NnPJf1Y0tHu/gtJMrPaH1IU0l4EY06Z29OgN3/x+hxaJs3JZanl0p7//H22z3z5Vc8wi/qs\nM/KJa7j8WSeiW5tWrsm8/1KHYUnWn03OkPElrsH5Y1u91GfZ+3PoEOIT1H8p/R+Z2VfN7EhJltHn\nVlZdZzKqgsO9wqjPMPKJi/yRBeonDvpvGPnExTntYXWrz2H3wLr7dZKua12l8DhJ75X0PDP7N0nf\ndfdbCmpj6aSZaeubOTbHlpVPHjPlI6dMUN+mZuXYDWaCw8gnrsH5N22diPTy6L/UYTKh9ScZMr7E\nxk6OsG7qs+z9OcltdP7i7le5+zGS9pK0WNL5ubes5Oo2k5EX8omD+gwjn7iYKUca9N+4yD+MfJLj\nFnfFq0t9JroK8QB3/1PrJuZH5tWgKqlLEeSJfOKhPsPIJy5mypEG/Tcu8g8jn2TIJ4461GdXG7B4\ntjoUQZ7IJy7yDyOf5JgpR9nQf+Mi/zDy6Yx84ql6fbIBm4GqF0HeyCcu8g8jn2TIB2VE/42L0wHC\nqM8w8omryvmzAZuRKhdBEcgnLvIPI5/OyAdlRf+Ni9MBwqjPMPKJq6r5swGboaoWQVHIJy7yDyOf\nMPJBmVGfcXE6QBj1GUY+cVUxfzZgM1bFIigS+cTF4V5h1GcY+aDMqM/kyKd41GcY+cRVtfzZgM1B\n1YqgaOQTF4d7hVGfYeSDMqM+kyGfOKjPMPKJq0o7OdiAzQmdMIx8kuPqr8WjPsPIB2VGfXZGPvFQ\nn2HkE1dVdnKMiN2AOmvvhGNOmauRUyZo/uL1uXzWnFyWmq+h8kkqjxzLmmEv+SC9NPXZBFnlwzoR\neei2PptWh3ms35qWYRqML2Hkk9ymlWsyz2fklAmaf9VtmS5zQFb9mT2wOWMmKYx8OiOfeKjPMPJB\nmVGfYeQTF/mHkU8yTc2HPbAFaO+EfSeNj92c0ullpq1v5tgCWlYOzETGRf5hafNpUl9G8ZLWZ1Pr\nMMv1W1MzTIPxJYx8Ossrn7L3Z/bAFqQqx5THwkxbGPnERf5h5IMyoz7DyCcu8g8jn7Cm5sMGbIGY\nOQpraidMinziIv8w8kGZUZ9h5BNXla7+GgP1GdbEfNiADWhKEZRJEzthN8gnLvIPIx+UGfUZRj5x\ncaReGPUZ1rR82IANaEoRlE3TOmG3yCcuZsrDqE+UGfUZRj7JcYu74lGfYU3Khw3YgKYUQRk1qRP2\ngnziYqY8jPpEmVGfYeSTDPnEQX2GNSUfrkIc0KSrn+1xzmXD/mzDXddLknY+9LhEy9rm9RN7b1OT\n8u8F+SSX133SMDzqE2U2uD6xLfpvZ+QTD/UZ1oR82IDtoAlFUGZVyj80CdCu2wmB0CRAlfKJiXzi\noD5RZtucDsAt7p6F/htGPnGRf1jd8+EQ4gSasju+rMg/jHw6I594qE+UGacDhNF/w8gnLvIPq3M+\nbMAmVOciqALyDyOfMPKJi/xRZnXbM5E1+m8Y+cRF/mF1zYdDiLuQxe74+YvX59AyaU7K9w8c1jqU\njffc2NWy2l8/euLePbdpsPb85++zfWbLHZBnhu26zTNphnU/XCQt8olruPzLuk5EeeVxTjt1GJZk\n/dnkDBlf4uKc9rBe6rPs/Zk9sF2q60xGVXC4Vxj1GUY+cZE/skD9xEH/DSOfuLjFXVjd6pM9sD1I\nM9PWN3Nsji3rXZILCiW+6FDb659ceWMuV3/t21S+HHvJJ4kturer5TITHEY+cQ3Ov6zrRJRXHv2X\nOkwmtP4kQ8aX2NjJEdZNfZa9P9diA3bw1V9T3falHVd/TY184qA+w8gnLq7+ijTov3GRfxj5JMct\n7opXl/rM9RBiM1tlZveb2RIzW9h6bjczu9XMHmr9u2uebchT3XbH54F84qE+w8gnLmbK46nD2Ez/\njYv8w8gnGfKJow71WcQ5sK9y9xnuPrv1+AOSfuju+0j6YetxZdWhCPJEPnGRfxj5JJdHPlWd+a2J\nyo/N9N+4yD+MfDojn3iqXp8xLuJ0nKQrW99fKen4CG3IVNWLIG/kExf5h5FPMuRTe5Ucm+m/cXHh\nnDDqM4x84qpy/nlvwLqkW8xskZnNaz23h7s/0vr+d5L2yLkNhahyERSBfOIi/zDy6Yx8aqVWYzP9\nNy5OBwijPsPIJ66q5p/3Buzh7j5L0lGSzjazI9p/6O6u/oH0WcxsnpktNLOFa9euzbmZ2ahqERSF\nfOIi/zDyCSOfWqnd2Ex9xsXpAGHUZxj5xFXF/HPdgHX3Na1/H5X0XUmHSPq9me0pSa1/Hx3mvZe6\n+2x3n7377rvn2cxMVbEIikQ+cXG4Vxj1GUY+9VDXsZn6TI58ikd9hpFPXFXLP7cNWDPbyczGDHwv\n6bWSlkm6QdJbWy97q6Tr82pDLFUrgqKRT1wc7hVGfYaRT7XVfWymPpMhnziozzDyiatKOzny3AO7\nh6Q7zew+SXdL+i93v1nSpyTNMbOHJL2m9bh26IRh5JMcV38tHvUZRj6VVvuxmfrsjHzioT7DyCeu\nquzkGJHXgt39l5KmD/H8Y5KOzPKzNty17UTxxntu7Or9w71+9MS9e26TNPTNgucvXp9qmcOZk/L9\ngzNslybPUIZpbqacR455Ztiu2zyfWH5rpW82XVV1udl3XrLKp6zrxLoqcmyOqdv6bFod5rF+a1qG\naTC+hJFPcptWrsk8n5FTJmj+VbdluswBWfXnGLfRaRRmksLIpzPyiYf6DCMflBn1GUY+cZF/GPkk\n09R8ctsDW6SdDz2uq+eTLmeL7u25Te3aO2HfSeMzWWbWkmTVS55JMuxlpq1v5tiu2lKEtPU2nC0T\n72UmMiJmgsPS5lPGvoz6SFqfTa3DLNdvTc0wDcaXMPLpLK98yt6f2QNbkKocUx4LM21h5BMX+YeR\nD8qM+gwjn7jIP4x8wpqaDxuwBWLmKKypnTAp8omL/MPIB2VGfYaRT1xVuvprDNRnWBPzYQM2oClF\nUCZN7ITdIJ+4yD+MfFBm1GcY+cTFkXph1GdY0/KpxTmweeGY+zg45yGMfOLaZqb8K7FbUz7UJ8qs\navW50yHHSkp+zYSBq+F3fP3EoZ+uWj4x5XX1VwyP+gxrUj7sgQ1o0kxG2TRtJqlb5BMXM+Vh1CfK\njPoMI59kyCcO6jOsKfmwARvQlCIoK/IPI5/k8sinzjObWaA+UWbUZxj5dEY+8VCfYU3Ihw3YDppQ\nBGVG/mHkkwz5xEF9osy4cE4Y/TeMfOIi/7C658M5sAk06ZjyMiL/MPLpjHzioT5RZltPB9i0MHZT\nSqkO/XePcy7b+n3ic4STvH5iPfKpMvIPq3M+7IFNqO4zGWVH/mHkE0Y+cZE/yqxOf9Tlgf4bRj5x\nkX9YXfOpxR7YgRmyARvvubGr9w/3+tET997mcRYzGfMXr+/6PUnMSfn+wRm2S5Pn4AzTaM9//j7b\nZ7bcAXlm2K7bPJNmWOeZtiyQT1zD5V/WdSLKK4+rv5a1DrP6e2awbsfmJOvPsmbYPjZnmWd7howv\ncQ3OH9vqpT7L2p8HsAe2S3WdyagKrv4aRn2GkU9c5I8sUD9x0H/DyCcuzmkPq1t91mIP7HDnMiQ9\nx2G412/RvUO+Ls1MW9/MsV21qShJsuolzydX3pjLfdL6NpUvx7T1Npzh6nA4zASHkU9cg/Mv6zoR\n5ZVH/y1rHY4++BhJ2Y8v3Y4rA0Lrz7JmOFQWWeQ5VIaML3GxkyOsm/osa38ewB7YHtVtJiMv5BMH\n9RlGPnExU4406L9xkX8Y+STHLe6KV5f6ZAM2hboUQZ7IJx7qM4x84mKmHGnQf+Mi/zDySYZ84qhD\nfbIBm1IdiiBP5BMX+YeRT3LMlKNs6L9xkX8Y+XRGPvFUvT7ZgM1A1Ysgb+QTF/mHkU8y5IMyov/G\nxekAYdRnGPnEVeX82YDNSJWLoAjkExf5h5FPZ+SDsqL/xsXpAGHUZxj5xFXV/NmAzVBVi6Ao5BMX\n+YeRTxj5oMyoz7g4HSCM+gwjn7iqmD8bsBmrYhEUiXzi4nCvMOozjHxQZtRncuRTPOozjHziqlr+\nbMDmoGpFUDTyiYvDvcKozzDyQZlRn8mQTxzUZxj5xFWlnRxswOaEThhGPslx9dfiUZ9h5IMyoz47\nI594qM8w8omrKjs5RsRuQBY23HX9No833nNjV+8f7vWjJ+7dc5ukbTvhmFPmauSUCZq/eH2qZQ5n\nTsr3D86wXZo8QxkOlU9SeeSYZ4btus3zieW3dp0P0ktTn02QVT5lXSei2rqtz7LWYVZ/zww2+g17\nZ75+K2uG7WNzlnmm+RuR8SWMfJLbtHJN5vmMnDJB86+6LdNlDshqbGYPbM6YSQojn87IJx7qM4x8\nUGbUZxj5xEX+YeSTTFPzqcUe2J0PPa6r55MuZ4vu7blN7do7Yd9J4zNZZtaSZNVLnkky7GWmrW/m\n2K7aUoS09TacLRPvZSYyImaCw9LmU8a+jPpIWp9lrcPRBx8jKfvxZWBsznL9VtYMh8oiizyz+BuR\n8SWMfDrLK5+y9ucBtdiArYKtx5RvWhi7KaXESiqMfOIi/zDywVD2OOeyjq8ZOLwz6QbFhruu1+g3\ndHfoJvUZRj5xDc4f26I+w5qaD4cQF6gpRdUrDhcJI5+4yD+MfFBm1GcY+cRVpau/xkB9hjUxHzZg\nA5pSBGXSxE7YDfKJi/zDyAdlRn2GkU9cVbn6ayzUZ1jT8mEDNqApRVA2TeuE3SKfuJgpD6M+UWbU\nZxj5JMct7opHfYY1KZ/cN2DNbDszW2xmN7UeX2FmvzKzJa2vGXm3oVdNKYIyalIn7AX5xMVMeRj1\nWX5VHpvToj7DyCcZ8omD+gxrSj5F7IH9R0krBj33fnef0fpaUkAbetKUIigr8g8jn+SYKS8e9Vl6\nlR2bs0B9hpFPZ+QTD/UZ1oR8ct2ANbO9JP29pK/l+Tl5akIRlBn5h5FPMuQTB/VZTnUYm7PA6QBh\n9N8w8omL/MPqnk/ee2A/L+mfJG0Z9PwnzGypmV1kZjvk3IbU6l4EZUf+YeTTGfnEQ32WUi3G5ixw\nOkAY/TeMfOIi/7A655PbBqyZHS3pUXdfNOhHH5S0n6SDJe0m6fxh3j/PzBaa2cK1a9fm1czE6lwE\nVUD+YeQTRj5xkX951G1szgKnA4TRf8PIJy7yD6trPiNyXPZhko41s9dJGiVprJl9091Paf38r2Z2\nuaTzhnqzu18q6VJJmj17toc+aOBG6AM23nNjVw0d7vWjJ257s/QsbhY8f/H6rt+TxJyU7x+cYbs0\neQ7OMI32/Ofvs31myx2QZ4btus0zaYZNvZl1UuQT13D5l3WdWGPRxuah9DK+jJh1XOb9t6x1mNXf\nM4N1OzYnWX+WNcP2Oswyz/YMGV/iGpw/ttVLfZa1Pw/IbQ+su3/Q3fdy90mSTpZ0m7ufYmZ7SpKZ\nmaTjJS3Lqw15qOtMRlVwuFcY9RlGPnGRf3x1GJupnzjov2HkExfntIfVrT7z3AM7nAVmtrskk7RE\n0plpF7jzocd19XzS5WzRvUO+Ls1MW9/MsV21qShJsuolzydX3pj5TOTIKRPUt6l8Oaatt+EMV4fD\nYSY4jHziGpx/WdeJDVTY2Jz2tSNmPSfz/lvWOhx98DGSsh9fuh1XBoTWn2XNcKgssshzqAwZX+Ji\nJ0dYN/VZ1v48oIjb6Mjdb3f3o1vfv9rdD3T3qe5+irtvKKINWavbTEZeyCcO6jOMfOJiprwcqjo2\n03/jIv8w8kmOW9wVry71WcgGbF3VpQjyRD7xUJ9h5BMXM+VIg/4bF/mHkU8y5BNHHeqTDdiU6lAE\neSKfuMg/jHySY6YcZUP/jYv8w8inM/KJp+r1yQZsBqpeBHkjn7jIP4x8kiEflBH9Ny5OBwijPsPI\nJ64q5x/jIk61xIn7YeQTF/mH1S2fPc65bOv3A7eQSHrRkuFeX6d8UC91679Vs/V0gE0LYzellKjP\nMPKJq6r5swc2Q1WeySgC+cRF/mHkE0Y+KDPqM66q/NEbC/UZRj5xVTF/NmAzVsUiKBL5xMXhXmHU\nZxj5oMyoz+TIp3jUZxj5xFW1/NmAzUHViqBo5BMXV38Noz7DyAdlRn0mQz5xUJ9h5BNXlXZysAGb\nEzphGPkkx9Vfi0d9hpEPyoz67Ix84qE+w8gnrqrs5KjFRZwGLjoyYOM9N3b1/uFeP3ri3j23SRr6\nxOj5i9enWuZw5qR8/+AM26XJM5RhmhPH88gxzwzbdZvnE8tvrdSJ9XVR1QsbSNvWYl7rw6zyKes6\nEeklWSf2Up+j33BWx9d1W59lrcOs+u9go9+wd+brt7JmmMf6UEr3N2KVx5cikE9ym1auyTyfkVMm\naP5Vt2W6zAFZjc3sgc0ZM0lh5NMZ+cRDfYaRD8qM+gwjn7jIP4x8kmlqPrXYAzvc7SGS3jZiuNdv\n0b09t6ldeyfsO2l8JsvMWpKseskzSYa9zLT1zRzbVVuKkLbehrNl4r3MREZUxZngoWorr/Vh2nzK\n2JeRjW5qrtv6TCppfZa1DkcffIyk7MeXgf6c5fqtrBnmsT6UsvkbsYrjS5HIp7O88ilrfx7AHtiC\nVOWY8liYaQsjn7jIP4x8UGbUZxj5xEX+YeQT1tR82IAtEDNHYU3thEmRT1zkH0Y+KDPqM4x84qrS\n1V9joD7DmpgPG7ABTSmCMmliJ+wG+cRF/mHkgzKjPsPIJy6O1AujPsOalg8bsAFNKYKyaVon7Bb5\nxMVMeRj1iTKjPsPIJzlucVc86jOsSfnU4iJOeeHE8Xg4cT+szvnsdMixkpJfZGPgFgnB109M3axt\nMFMeVuf6RPVRn2Hkkwz5xEF9hjUlH/bABjRpJqOMyD+MfJJjprx41CfKjPoMI5/OyCce6jOsCfmw\nAdtBE4qgzMg/jHySIZ84qE+UGacDhNF/w8gnLvIPq3s+bMAmUPciKDvyDyOfzsgnHuoTZcbpAGH0\n3zDyiYv8w+qcDxuwCdW5CKqA/MPIJ4x84iJ/lBmnA4TRf8PIJy7yD6trPrW4iNPARVwGbLznxq7e\nP9zrR0/ce5vHWZwYPcd0kzoAABS1SURBVH/x+q7fk8SclO8fnGG7NHkOzjCN9vzn77N9ZssdkGeG\n7brNM2mGdTlxP6v+2270xL1rk08S7bWY1/qwW8PlX9Z1ItJLsk7spT5HzDou8/5b1jrMY30odd+f\nk6w/y5phHutDadsMmzS+lNHg/LGtXuqzrP15AHtgu1TXmYyq4HCvMOozjHziIn9kgfqJg/4bRj5x\ncU57WN3qsxZ7YIe7fUbS23AM9/otunfI16WZaeubObarNhUlSVa95Pnkyhszn4kcOWWC+jaVL8e0\n9Tac4epwOFWfCR598DGSss2zPcOq55PEUFnktT7s1uD8y7pORHrd1Fw3rx0x6zmZ99+y1mEe60Op\n9/4cWn+WNcM81ofS0Bk2YXwpM3ZyhHVTn2XtzwPYA9ujus1k5IV84qA+w8gnLmbKkQb9Ny7yDyOf\n5LjFXfHqUp9swKZQlyLIE/nEQ32GkU9czJQjDfpvXOQfRj7JkE8cdahPNmBTqkMR5Il84iL/MPJJ\njplylA39Ny7yDyOfzsgnnqrXZy3OgY2Ncx7CyCcu8g8jn2TIB2VE/41rm9MBThofuzmlU6f63OOc\ny5713MAVnpOeUzz49XXKJ4mhMmzXS56j39D7HQKqnD97YDNS9ZmMvJFPXOQfRj6dkQ/Kiv4bF6cD\nhFGfYeQTV1XzZwM2Q1UtgqKQT1zkH0Y+YeSDMqM+46rSnpsYqM8w8omrivmzAZuxKhZBkcgnLq7+\nGkZ9hpEPyoz6TI58ikd9hpFPXFXLP/cNWDPbzswWm9lNrceTzewuM/uFmf2HmY3Muw1Fq1oRFI18\n4uJwrzDqM4x86qGuYzP1mQz5xEF9hpFPXFXayVHEHth/lLSi7fGnJV3k7i+U9CdJbyugDYWjE4aR\nT3Jc/bV41GcY+dRCbcdm6rMz8omH+gwjn7iqspMj16sQm9lekv5e0ick/R8zM0mvlvTm1kuulHSB\npH9L8zkDV+0asPGeG7t6/3CvHz2x9yt7SUNf3Wv+4vWpljmcOSnfPzjDdmnyDGWY5upneeSYZ4bt\nus3zieW3Vu7qcGlk1X/b9dKXq3x1vvZazGt9mFU+ZV0n1lmssXkovdTn6Dec1fF13dZnWeswj/Wh\nJI1+w96Zr9/KmmEe60Mp3d+IVRxfhurPafMcLsMq5pNEp3ViL3mOmHVc5vmMnDJB86+6LdNlDshq\nbM57D+znJf2TpC2tx+MlrXP3p1uPV0uqR1UOg5mkMPLpjHzioT7DyKeyGjE2U59h5BMX+YeRTzJN\nzcfcPZ8Fmx0t6XXufpaZvVLSeZJOk/Sz1iFKMrO9Jf23u08d4v3zJM1rPdxX0oO5NDQ7z5X0h9iN\nqDgyTI8M0yPD9KqQ4d+6++6xG1E0xmb0gAzTI8P0yDC9KmSYaGzO8xDiwyQda2avkzRK0lhJX5A0\nzsxGtGZ695I05LSBu18q6dIc25cpM1vo7rNjt6PKyDA9MkyPDNMjw1JjbEZXyDA9MkyPDNOrU4a5\nHULs7h90973cfZKkkyXd5u5vkfQjSSe2XvZWSclOHAQAAKkwNgMAqi7GfWDPV/9FI36h/vNuLovQ\nBgAA8AzGZgBAJeR6FeIB7n67pNtb3/9S0iFFfG7BKnNIVYmRYXpkmB4ZpkeGFcDYjITIMD0yTI8M\n06tNhrldxAkAAAAAgCzFOIQYAAAAAICuFXIIcdmZ2XhJP2w9fL6kzZLWth4f4u6bcvjMWZKe5+43\nZ73sGMgwPTLMBjmmR4YoA+owPTJMjwyzQY7pkeEz2ICV5O6PSZohSWZ2gaQN7v7ZpO83s+3cfXOX\nHztL0lRJpSqIXpFhemSYDXJMjwxRBtRhemSYHhlmgxzTI8NncAhxB2Z2o5ktMrMHzOztredGmNk6\nM/u8mS2VdIiZHWtmD7Zee7GZXdd67c5mdoWZ3W1mi83sGDPbUdK/SHqLmS0xsxMDTag8MkyPDLNB\njumRIcqAOkyPDNMjw2yQY3qNy9Dd+Wr7knSBpPPaHu/W+ne0pOWSdlX/nmuXdELbz1ZL+ltJJuka\nSde1fvYZSSe3vt9V0v+q/+bxb5f0+dj/XzIs5xcZkmNZvsiQrzJ8UYdkWIYvMiTHsnw1PUP2wHZ2\nrpndJ+mnkvaSNKX1/CZJ3219f4CkB939197/m7+67f2vlfQhM1ui/hvFj5I0sZCWlwcZpkeG2SDH\n9MgQZUAdpkeG6ZFhNsgxvUZlyDmwAWb2GklHSHqJuz9hZneq/xcqSU+0fvkdFyPpeHdfOWjZR2Tb\n2nIiw/TIMBvkmB4Zogyow/TIMD0yzAY5ptfEDNkDG7aLpD+2iuHFkg4e5nXLJe1rZnubmUl6Y9vP\nvi/pnIEHZjaz9e3jksbk0OayIcP0yDAb5JgeGaIMqMP0yDA9MswGOabXuAzZgA37L0mjzWy5pI9L\numuoF7n7RknvlvQDSQslrZP059aPPyppJzO738weUP8x65J0m6TprROly3NSdPbIMD0yzAY5pkeG\nKAPqMD0yTI8Ms0GO6TUuQ0u2VxmdmNnO7r6hNaPx75Lud/eLY7erSsgwPTLMBjmmR4YoA+owPTJM\njwyzQY7p1SVD9sBm513Wf+Lzckk7Svpq5PZUERmmR4bZIMf0yBBlQB2mR4bpkWE2yDG9WmTIHlgA\nAAAAQCWwBxYAAAAAUAmN2IA1sw1dvv6VZnZTRp99gZmd1+N7J5nZm7NoR6/MbLOZLTGzB8zsPjN7\nn5lFqxszW2Vmz+3xvceb2QFZtynB55JhSmSYHhmibBibe0d/To8M0yPD9MiwN43YgK2wSZKiDpLq\nv3/UDHd/saQ5ko6S9JHIberV8eq/iXPRyDA9MkyPDIFsTBJjc5ZYJ6ZHhumRYXrFZejutf+StKH1\n7ysl3S7pWkk/l7RAz5wH/Het5+6V9EVJN7We30nS1yXdLWmxpONaz58m6frW8h6S9JG2z/uQpP+V\ndKekqyWd13p+iqSbJS2S9GNJ+7Wev6L1mT+R9EtJJ7ae/5n6L2+9RNK5MbNre/wCSY+p/4bH20m6\nUNI9kpZKemdbzneo/7LeD0r6iqTntH72Wkk/beV8jaSdW8+vUv8lvO+VdH9bNuMl3SLpAUlfk/Rr\nSc9t/eyU1u9lifqvpLbdQJslfULSfa0M95D0Mkl/lPSr1uunkCEZkiEZ8hXvS4zNqbNre0x/JkMy\nJMPGZFjILyf2l7YdJP8saS/1733+qaTDJY2S9BtJ+7QK5tt6ZpD8pKRTWt+PU//gt5P6B8lHWr/4\nHSUtkzRb0kGtwhgtaaykX+iZQfKHkvZpfX+opNta31/RKrLnqH/m4hdt7b2pDNkNem5dq1jnSfpw\n67kd1H9Pqcmtdj/Z6oTbSbpV0omSntvqcDu13nO+pH9p61jntL4/S9LXWt9/se01fy/JW8vZX9KN\nkrZv/ezLkvpa37ukY1rff6atjVeo9QcIGZIhGZIhX3G/xNicOrtBz9GfyZAMybARGY5Q89zt7qsl\nqXUZ6Unqn0n4lbs/1Hr+m+ovGql/JuPYtnNlRkma2Pr+Vnd/rPWe/1T/gCtJ3/X+mwXLzG5o/buz\n+mcnrum/9ZKk/mIccJ27b5G03Mz2yO6/m6vXSppmz9zYeBf1/6GxSf05/1KSzOxq9WfzpPr/CPif\nVgYj1f+HyoD/bP27SNIJre+PGPje3f/LzP7Uev5I9f9Bck9rWTtKerT1s02Sbmpb1pwM/q95IcP0\nyDA9MkRsjM3ZoT+nR4bpkWF6ZDiMJm7A/rXt+83qnIFJ+gd3f3CbJ80OVf8MRDtvvX4oz5G0zt1n\nJGjXcMuIzsxeoP7cHlV/O89x9+8Pes0rNXw2t7r7m4ZZ/EAGSX8vV7r7B4f42VPemgpKuKxCkWF6\nZJgeGaJkGJtToD+nR4bpkWF6ZJgMF3Hq93NJk8xsSutx+y/++5LOsdb0g5nNbPvZHDPbzcx2VP+J\ny/+j/l33x5vZjmY2RtIxkuTu6yX9ysze0FqOmdn0Du16XNKYlP+3zJjZ7uo/zv5LrcL9vvpviLx9\n6+cvMrOdWi8/xMwmt66k9kb1n3P0M0mHmdkLW6/fycxe1OFj71DrYhlmdpSkXVvP/1DSiWb2vNbP\ndjOzv+2wrOh5kmF6ZJgeGaIiGJsToD+nR4bpkWF6ZJgcG7CS3P1J9R+W9F9mdq+e2UUuSR+TtL2k\npWb2QOvxgLslfUf9J1Z/x90Xuvu9kv5D/Sc2/7f6T7we8BZJbzOz+9R/svRxHZq2VNJm67+s9rk9\n/wfT2dFal/eW9AP1n+j90dbPviZpuaR7zWyZ+k/wHpiFuUfSlyStUP8J3d9197XqPz/pajNbqv7D\nGvbr8PkflXRE6/NPkPSwJLn7ckkflnRLa1m3Stqzw7K+Jen9Zra47Q+iIpBhemSYHhmiUhibg+jP\n6ZFhemSYHhn2YOAqf+iSmZ0maba7vzt2W8rG+g9tOM/dj47dlqoiw/TIMD0yRNUwNg+P/pweGaZH\nhumRIXtgAQAAAAAVwR5YAAAAAEAlsAcWKCEzc+u/ZcTA4xFmttbMbmo9PtbMPhCvheVjZptb55Es\nM7NrzGx06/kNGSz7NDP70qDnbjez2WmXXbQCcnIze03bc8e3njux9fh2M3uwdf7g/5jZvq3nR5rZ\n583sF62vm8xs4qDlDyyr0zk9AJA5xubuMTYnw9jcHTZggXL6i6Sp1n8VTan/HltrBn7o7je4+6ei\ntKy8nnD3Ge4+Vf33KDszdoNKKu+c7pd0ctvjN6n/wjnt3uLu0yVdKenC1nOfVP/VC/d19xeq/yI8\n11v/FRbbl3Wntr0aLQAUhbG5e4zNyTA2d4ENWKC8vifp71vfv0nS1QM/aJ91NLMrzOyLZvYTM/tl\n22zaKwdmhVuPv9S6wInM7FNmttzMlprZZ4v6DxXox5Je2P5Ennm0z5Ca2YlmdkXPLS9WHjn9WP2X\n99/ezHZuLX/JMK+9Q9ILWzPNp0s61903S5K7Xy5pg6TXtD57Z/XfqP1t2nYQBoAiMTb3jrE5Gcbm\nDkp1814A2/iWpH9prbCmSfq6pJcP89o91b8C2U/SDZKuHW6hZjZe0usl7efubmbjMm11ZGY2QtJR\nkm5O+PqkebzRzA5ve/zCYV5XCTnm5Oq/FcBcSbuovx4nD/PaY9Q/K/xCSQ+37snZbqGkA9R/W4Hj\nJN3s7v9rZo+Z2UHuvihJ2wEgQ4zNPWBsToaxORn2wAIl5e5LJU1S/wzv9zq8/Dp339K6b9ceHV77\nZ0lPSrrMzE6QtDFtW0tiRzNbov4V68OSLkv4vqR5/Efr8J4Z7j6j9TlVlHdOUv8feCe3vq4e4ucL\nWm04TNJ5CT//Ta3lDiyfw4gBFI6xuWuMzckwNneBPbBAud0g6bOSXilpfOB1f2373lr/Pq1tJ6lG\nSZK7P21mh0g6UtKJkt4t6dUZtTemJ1qD13DyzKP9cu6junxv0XLPyd3vNrMDJW1szcoOfslb3H3r\nHxn2/7dzL6FalHEcx78/7aKRRRrRpmxrFwtsI0UF0SKCoBLKinJZtEoKMoJyGUgLCboi3cAIpAgz\nCIroRpeFphm06li2KKOgshYV/xbvaKfTOXqO5z3OzHm/Hzgw88wzM/95FufPf+Z5n+Qn4NwkS6rq\n13H9VgHbkixt7nVRkgIWApXk/nIpfUnHn7l5+szN02NungG/wErdtgXYWFV7juHcfcD5SU5uppRc\nDYd/r3B6Ve0A7gUuHlq03TaX4/F9khUZLGpww9AibsewxukB4MHp3LCqDjJYNOKxJAub+93B4K3y\nhwyS8otVtbyqzquqc4CvmXraniTNJXPz8Jibp8fcPI5fYKUOq6r9wOZjPPfbJK8AXzD4h7KzObSE\nwQpyixi8EV4/jFi7bo7H4wFgO3CAwfSfU2cfcTuGNU5V9eYMb72BwaqHX2WwwucBYHXzm561wKMT\n+m9jMFXpvRneR5Jmxdw8PObm6TE3/1ecfSVJ6pIkZwNvAk9U1dNtxyNJ0qjrUm62gJUkSZIk9YK/\ngZUkSZIk9YIFrCRJkiSpFyxgJY2kJL9N0nZXs8LeyJs4PknWJXl8Ntdo2hxjSdKkzBtHZm4ecBVi\nSWpU1ZNtxzDfOcaSpJkwb8y9vo2xX2AlqZHkkST3NdvvJrm02T4zyVirwXVIkueSrBm3/7+3uUc4\n1zGWJE2beWN6Rik3+wVWkjSZxUl2jdtfCrzeVjCSJMncDBawkqTJ/VFVlxzaSbIOuLS9cCRJGnnm\nZpxCLElT+Yt//0cuajOQDjo8NkkWACfN9jo4xpKkozNvTG1kcrMFrCRNbgxY1WyvOUK/UTTGv2Nz\nPXDiEK7jGEuSjmYM88ZUxhiR3GwBK2lUnZJk/7i/9ROObwLuTrITOLOF+LrsGeDKJJ8Dq4GDU/Rz\njCVJM2HeOHYjk5tTVW3HIEmSJEnSUfkFVpIkSZLUCxawkiRJkqResICVJEmSJPWCBawkSZIkqRcs\nYCVJkiRJvWABK3VQkkry0rj9E5IcSLJ9htcZS3LEJdCn00eSpFFnbpa6wQJW6qaDwIVJFjf71wDf\ntRiPJEmjztwsdYAFrNRdO4Drmu21wNZDB5IsTfJakt1JPk6ysmlfluStJHuTPAtk3Dm3J/k0ya4k\nTyVZeDwfRpKkecDcLLXMAlbqrpeBW5IsAlYCn4w7thHYWVUrgQeBF5r2h4EPquoC4FXgXIAkK4Cb\ngcuq6hLgb+C24/IUkiTNH+ZmqWUntB2ApMlV1e4k5zF4w7tjwuHLgZuafu80b3dPA64Abmza30jy\nc9P/amAV8FkSgMXAD3P9DJIkzSfmZql9FrBSt70ObAKuApbN4joBnq+qDcMISpKkEWZullrkFGKp\n27YAG6tqz4T292mmGSW5Cvixqn4B3gNubdqvBc5o+r8NrElyVnNsaZLlcx++JEnzjrlZapFfYKUO\nq6r9wOZJDj0CbEmyG/gduLNp3whsTbIX+Aj4prnOl0keAt5KsgD4E7gH2De3TyBJ0vxibpbalapq\nOwZJkiRJko7KKcSSJEmSpF6wgJUkSZIk9YIFrCRJkiSpFyxgJUmSJEm9YAErSZIkSeoFC1hJkiRJ\nUi9YwEqSJEmSesECVpIkSZLUC/8A3JIoTXU3PNQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbadb3bd198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, axes = plt.subplots(1, 2)\n",
    "acc_data = (scale_acc, test_acc, paper_acc, 'Accuracy %')\n",
    "f1_data = (scale_f1, test_f1, paper_f1, 'F1 %')\n",
    "for i, data in enumerate([acc_data, f1_data]):\n",
    "    scale_score, test_score, paper_score, score_name = data\n",
    "    legend_data = [('Not Scaled', scale_score), ('Reproduction', test_score), ('Original', paper_score)]\n",
    "    scale_model_score = [(scale, fill(model,10), score) for scale, model_score in legend_data \n",
    "                         for model, score in model_score.items()]\n",
    "    scale_graph = defaultdict(list)\n",
    "    for scale, model, score in scale_model_score:\n",
    "        scale_graph['Scale'].append(scale)\n",
    "        scale_graph['Model'].append(model)\n",
    "        scale_graph[score_name].append(score)\n",
    "    ax = sns.barplot(x=\"Model\", y=score_name, hue=\"Scale\", \n",
    "                     data=pd.DataFrame(scale_graph), \n",
    "                     palette=sns.color_palette(\"colorblind\"),\n",
    "                     ax=axes[i])\n",
    "    # Sets the hatching\n",
    "    hatches = ['+', '\\\\', '-']\n",
    "    hatch_num = 0\n",
    "    for i in range(15):\n",
    "        ax.patches[i].set_hatch(hatches[hatch_num])\n",
    "        if ((i + 1) % 5) == 0:\n",
    "            hatch_num += 1\n",
    "    ax.set(ylim=(40, 75))\n",
    "    ax.figure.set_size_inches(16,5)\n",
    "    ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see not scaling the data (in this can we used [MinMax](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) scaling) affects the results quite a lot more so for the models that contain more parameters. This was not stated in the paper. They did use a different Support Vector Machine Library [LibLinear](https://www.csie.ntu.edu.tw/~cjlin/liblinear/) however we actually use this library just using the [Scikit-learn interface](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html). Even though in the [Practical guide to LibLinear](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf) it states that you should scale and shows the importance like we have done above but this is not stated or reiterated in the paper.\n",
    "\n",
    "This scaling effect can also be seen in the TDParse models of which that notebook can be found [here](./TDParse.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}