3_3_RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Data and Recurring Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "source(\"src/lib.R\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll summarise two related and important topics in Neural Networks:\n",
    "* Text processing and how to feed text data into Neural Networks\n",
    "* The Neural Networks models that have been developed to handle sequence data (i.e. text or timeseries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text vectorization\n",
    "In the previous notebooks we have seen that neural networks model must be fed with numeric tensors, the *text vectorization* is a process that transforms text data into quantitative data.\n",
    "\n",
    "There are different way to *vectorize* text data, as example:\n",
    "* Segment text into **words** and associate each word to a numerical vector.\n",
    "* Segment text into **characters** and transform each character into a numerical vector.\n",
    "* Extract **N-Grams** of words/characters and transform each N-Gram into a vector. A **N-gram** is a sequence of at most **N** consecutive words/characters in a text data: consider text *\"UniCredit is a pan European Winner\"*, it contains the following **3 grams**: *Unicredit, is, a, pan, European, Winner, Unicredit is, is a, a pan, pan European, European Winner, Unicredit is a, is a pan, a pan European, pan European Winner*.\n",
    "\n",
    "In the following we'll focus our attention on the first *vectorization* strategy: **words** vectorization.\n",
    "\n",
    "The **first step** in this process is to segment text data into words: *Tokenization*. Let's see how to tokenize text data in **KERAS**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'unicredit'</li>\n",
       "\t<li>'is'</li>\n",
       "\t<li>'a'</li>\n",
       "\t<li>'pan'</li>\n",
       "\t<li>'european'</li>\n",
       "\t<li>'winner'</li>\n",
       "\t<li>'cib'</li>\n",
       "\t<li>'fully'</li>\n",
       "\t<li>'plugged'</li>\n",
       "\t<li>'into'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'unicredit'\n",
       "\\item 'is'\n",
       "\\item 'a'\n",
       "\\item 'pan'\n",
       "\\item 'european'\n",
       "\\item 'winner'\n",
       "\\item 'cib'\n",
       "\\item 'fully'\n",
       "\\item 'plugged'\n",
       "\\item 'into'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'unicredit'\n",
       "2. 'is'\n",
       "3. 'a'\n",
       "4. 'pan'\n",
       "5. 'european'\n",
       "6. 'winner'\n",
       "7. 'cib'\n",
       "8. 'fully'\n",
       "9. 'plugged'\n",
       "10. 'into'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"unicredit\" \"is\"        \"a\"         \"pan\"       \"european\"  \"winner\"   \n",
       " [7] \"cib\"       \"fully\"     \"plugged\"   \"into\"     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "samples <- c(\"UniCredit is a pan European Winner\", \"CIB is fully plugged into UniCredit!\")\n",
    "tokenizer <- text_tokenizer() %>%\n",
    " fit_text_tokenizer(samples)\n",
    "\n",
    "tokenizer$word_counts %>% names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer found 10 different *words* in the input sentences. Note that the *tokenizer* makes also a normalization step: all the words (*tokens*) are lowercase and all the punctuaction marks have been removed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One-Hot encoding\n",
    "The next step require to to associate each *token* to a numeric vector, the more simple way to do this is called **One-Hot encoding**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This encoding strategy first requires to associate a unique integer index id $H_w$ to each word $w$. In **KERAS** this is done directly by the *tokenizer*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<dl>\n",
       "\t<dt>$unicredit</dt>\n",
       "\t\t<dd>1</dd>\n",
       "\t<dt>$is</dt>\n",
       "\t\t<dd>2</dd>\n",
       "\t<dt>$a</dt>\n",
       "\t\t<dd>3</dd>\n",
       "\t<dt>$pan</dt>\n",
       "\t\t<dd>4</dd>\n",
       "\t<dt>$european</dt>\n",
       "\t\t<dd>5</dd>\n",
       "\t<dt>$winner</dt>\n",
       "\t\t<dd>6</dd>\n",
       "\t<dt>$cib</dt>\n",
       "\t\t<dd>7</dd>\n",
       "\t<dt>$fully</dt>\n",
       "\t\t<dd>8</dd>\n",
       "\t<dt>$plugged</dt>\n",
       "\t\t<dd>9</dd>\n",
       "\t<dt>$into</dt>\n",
       "\t\t<dd>10</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description}\n",
       "\\item[\\$unicredit] 1\n",
       "\\item[\\$is] 2\n",
       "\\item[\\$a] 3\n",
       "\\item[\\$pan] 4\n",
       "\\item[\\$european] 5\n",
       "\\item[\\$winner] 6\n",
       "\\item[\\$cib] 7\n",
       "\\item[\\$fully] 8\n",
       "\\item[\\$plugged] 9\n",
       "\\item[\\$into] 10\n",
       "\\end{description}\n"
      ],
      "text/markdown": [
       "$unicredit\n",
       ":   1\n",
       "$is\n",
       ":   2\n",
       "$a\n",
       ":   3\n",
       "$pan\n",
       ":   4\n",
       "$european\n",
       ":   5\n",
       "$winner\n",
       ":   6\n",
       "$cib\n",
       ":   7\n",
       "$fully\n",
       ":   8\n",
       "$plugged\n",
       ":   9\n",
       "$into\n",
       ":   10\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "$unicredit\n",
       "[1] 1\n",
       "\n",
       "$is\n",
       "[1] 2\n",
       "\n",
       "$a\n",
       "[1] 3\n",
       "\n",
       "$pan\n",
       "[1] 4\n",
       "\n",
       "$european\n",
       "[1] 5\n",
       "\n",
       "$winner\n",
       "[1] 6\n",
       "\n",
       "$cib\n",
       "[1] 7\n",
       "\n",
       "$fully\n",
       "[1] 8\n",
       "\n",
       "$plugged\n",
       "[1] 9\n",
       "\n",
       "$into\n",
       "[1] 10\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer$word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we need to transform our sample sentences into list of indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'list'"
      ],
      "text/latex": [
       "'list'"
      ],
      "text/markdown": [
       "'list'"
      ],
      "text/plain": [
       "[1] \"list\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "2"
      ],
      "text/latex": [
       "2"
      ],
      "text/markdown": [
       "2"
      ],
      "text/plain": [
       "[1] 2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sequences <- texts_to_sequences(tokenizer, samples)\n",
    "class(sequences)\n",
    "length(sequences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence associated with the first sentence, *UniCredit is a pan European Winner*, is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>1</li>\n",
       "\t<li>2</li>\n",
       "\t<li>3</li>\n",
       "\t<li>4</li>\n",
       "\t<li>5</li>\n",
       "\t<li>6</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 1\n",
       "\\item 2\n",
       "\\item 3\n",
       "\\item 4\n",
       "\\item 5\n",
       "\\item 6\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 2\n",
       "3. 3\n",
       "4. 4\n",
       "5. 5\n",
       "6. 6\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 1 2 3 4 5 6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sequences[[1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence associated with *CIB is fully plugged into UniCredit* is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>7</li>\n",
       "\t<li>2</li>\n",
       "\t<li>8</li>\n",
       "\t<li>9</li>\n",
       "\t<li>10</li>\n",
       "\t<li>1</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 7\n",
       "\\item 2\n",
       "\\item 8\n",
       "\\item 9\n",
       "\\item 10\n",
       "\\item 1\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 7\n",
       "2. 2\n",
       "3. 8\n",
       "4. 9\n",
       "5. 10\n",
       "6. 1\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1]  7  2  8  9 10  1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sequences[[2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the two words *UniCredit* (coded by **1**) and *is* (coded by **2**) are the only ones contained in both sentences, and as a consequence their codes are repeated in both sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally** we need to transform each number $H_w$ into a $N$-dimensional binary vector (where $N$ is the number of in unique tokens found by the tokenizer) such that the only not-zero component is the one corresponding to $H_w$. This final step is currently done by **KERAS** in a more smart way (as we'll see soon).  Here is a simple base *R* code to compute this final step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "results <- array(0, dim = c(\n",
    "  length(sequences), # number of sentences\n",
    "  max(sapply(sequences, length)), # maximum number of words in a sentence\n",
    "  max(sapply(sequences, max)) # number of unique words in all sentences\n",
    "))\n",
    "\n",
    "for (i in 1:length(sequences)) {\n",
    "  sequence <- sequences[[i]]\n",
    "  for(h in 1:length(sequence)) {\n",
    "    results[i, h, sequence[h]] <- 1\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first sentences is **one-hot encoded** by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "\t<tr><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{llllllllll}\n",
       "\t 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]\n",
       "[1,] 1    0    0    0    0    0    0    0    0    0    \n",
       "[2,] 0    1    0    0    0    0    0    0    0    0    \n",
       "[3,] 0    0    1    0    0    0    0    0    0    0    \n",
       "[4,] 0    0    0    1    0    0    0    0    0    0    \n",
       "[5,] 0    0    0    0    1    0    0    0    0    0    \n",
       "[6,] 0    0    0    0    0    1    0    0    0    0    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results[1,,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the second one by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr>\n",
       "\t<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td></tr>\n",
       "\t<tr><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{llllllllll}\n",
       "\t 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\\\\n",
       "\t 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0\\\\\n",
       "\t 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0\\\\\n",
       "\t 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\\\\n",
       "\t 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | \n",
       "| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | \n",
       "| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | \n",
       "| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | \n",
       "| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]\n",
       "[1,] 0    0    0    0    0    0    1    0    0    0    \n",
       "[2,] 0    1    0    0    0    0    0    0    0    0    \n",
       "[3,] 0    0    0    0    0    0    0    1    0    0    \n",
       "[4,] 0    0    0    0    0    0    0    0    1    0    \n",
       "[5,] 0    0    0    0    0    0    0    0    0    1    \n",
       "[6,] 1    0    0    0    0    0    0    0    0    0    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results[2,,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Words Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One-hot encoding is very simple and immediate but have a great disadvantage: the dimension of the vector it produces increases with $N$, the size of the considered vocabulary. Is not **BIG DATA** working with a vocabulary with thousands of words: in these cases we need a more **dense representation**.\n",
    "\n",
    "The idea behind **words embedding** is to reduce the dimensionality of the binary sparse vectors produced by the **one-hot encoding** in to a more compact space of **numeric vectors**:\n",
    "\n",
    "<img src=\"fig/words_embedding.PNG\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This mapping should preserve some semantic structure of the vocabulary it represents, in some way the geometric relationships between the embedded vectors should reflect the semantic relationships between the corresponding words. \n",
    "\n",
    "Otherwise, if we'd use a random mapping between words and numeric vector, any **neural network** couldn't be very effective since two words with a similar meanings like *handsome* and *attractive* could be mapped into very different vectors:  word embeddings are meant to\n",
    "map human language into a geometric space (i.e. similar words should be \"near\"):\n",
    "\n",
    "<img src=\"fig/words_plot.PNG\" width=\"400\">\n",
    "\n",
    "How to find this type of mapping? There are mainly two ways:\n",
    "* Use one of the words embedding mapping already computed by other people, like the [Word2Vec](http://code.google.com/archive/p/word2vec) or the [Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/)\n",
    "* Learn the right embedding for your task while learning the rest of your model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Words embedding on IMDB dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll see only how to find the right embedding while solving a text oriented task. But first we need a task!\n",
    "\n",
    "Here we consider the [Internet Movie DataBase (IMDB)](https://www.imdb.com/) sentiment analysis dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_features <- 10000\n",
    "\n",
    "imdb <- dataset_imdb(num_words = max_features)\n",
    "x_train <- imdb$train$x\n",
    "y_train <- imdb$train$y\n",
    "x_test <- imdb$test$x\n",
    "y_test <- imdb$test$y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is a sample dataset provided by **KERAS** and as a consequence has been already preprocessed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'list'"
      ],
      "text/latex": [
       "'list'"
      ],
      "text/markdown": [
       "'list'"
      ],
      "text/plain": [
       "[1] \"list\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "25000"
      ],
      "text/latex": [
       "25000"
      ],
      "text/markdown": [
       "25000"
      ],
      "text/plain": [
       "[1] 25000"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>1</li>\n",
       "\t<li>14</li>\n",
       "\t<li>22</li>\n",
       "\t<li>16</li>\n",
       "\t<li>43</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 1\n",
       "\\item 14\n",
       "\\item 22\n",
       "\\item 16\n",
       "\\item 43\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 14\n",
       "3. 22\n",
       "4. 16\n",
       "5. 43\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1]  1 14 22 16 43"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "1"
      ],
      "text/latex": [
       "1"
      ],
      "text/markdown": [
       "1"
      ],
      "text/plain": [
       "[1] 1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train %>% class\n",
    "x_train %>% length\n",
    "x_train[[1]] %>% head(n = 5)\n",
    "y_train[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input features are the indexes associated with the words in a movie review, in the target binary class we have an indicator telling if the review is positive (1) or negative (0). Note that parameter *num_words* set to 10000 limits the considered dictionary to the 10000 words with the highest frequency in the IMDB reviews corpus. Using function **dataset_imdb_word_index** we can retrieve the original content of the review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you could just imagine being there robert &lt;UNK&gt; is an amazing actor and now the same being director &lt;UNK&gt; father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for &lt;UNK&gt; and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also &lt;UNK&gt; to the two little boy\\'s that played the &lt;UNK&gt; of norman and paul they were just brilliant children are often left out of the &lt;UNK&gt; list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don\\'t you think the whole story was so lovely because it was true and was someone\\'s life after all that was shared with us all'"
      ],
      "text/latex": [
       "'<START> this film was just brilliant casting location scenery story direction everyone\\textbackslash{}'s really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy\\textbackslash{}'s that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don\\textbackslash{}'t you think the whole story was so lovely because it was true and was someone\\textbackslash{}'s life after all that was shared with us all'"
      ],
      "text/markdown": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you could just imagine being there robert &lt;UNK&gt; is an amazing actor and now the same being director &lt;UNK&gt; father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for &lt;UNK&gt; and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also &lt;UNK&gt; to the two little boy\\'s that played the &lt;UNK&gt; of norman and paul they were just brilliant children are often left out of the &lt;UNK&gt; list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don\\'t you think the whole story was so lovely because it was true and was someone\\'s life after all that was shared with us all'"
      ],
      "text/plain": [
       "[1] \"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'Positive'"
      ],
      "text/latex": [
       "'Positive'"
      ],
      "text/markdown": [
       "'Positive'"
      ],
      "text/plain": [
       "[1] \"Positive\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "y_train\n",
       "    0     1 \n",
       "12500 12500 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "index_step <- 3  # word index offset\n",
    "data <- dataset_imdb_word_index()\n",
    "mapping_df <- data_frame(\n",
    "  word = names(data),\n",
    "  index = data %>% sapply(function(x)x[[1]]) + index_step\n",
    ") %>% rbind(\n",
    "  data_frame(\n",
    "    word = c(\"<PAD>\", \"<START>\",\"<UNK>\"),\n",
    "    index = 0:2\n",
    "  )\n",
    ")\n",
    "\n",
    "x_train[[1]] %>% sapply(function(x) mapping_df %>% filter(index == x) %>% pull(word)) %>% paste(collapse = \" \")\n",
    "ifelse(y_train[1] == 1, \"Positive\", \"Negative\")\n",
    "table(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This review is a **positive** one, the grammar is not perfect because we are removing from each review the words that don't belong to the 10000 most frequent ones in the IMDB review corpus. Note that we are using a balanced training dataset! (and we are going to use accuracy to assess its performance!)\n",
    "\n",
    "To further simplify the input dataset we consider only the first 20 words of each review: this simplification is done by function **pad_sequences** that also transform our sequences in matrix (or tensor) with one row for each review and one column for each word in the review. If the review has less than 20 words **pad_sequences** complete the sequence with the magic word < PAD >:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'list'"
      ],
      "text/latex": [
       "'list'"
      ],
      "text/markdown": [
       "'list'"
      ],
      "text/plain": [
       "[1] \"list\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you'"
      ],
      "text/latex": [
       "'<START> this film was just brilliant casting location scenery story direction everyone\\textbackslash{}'s really suited the part they played and you'"
      ],
      "text/markdown": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you'"
      ],
      "text/plain": [
       "[1] \"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "maxlen <- 20\n",
    "mtx_train <- pad_sequences(x_train, maxlen = maxlen, truncating = \"post\", padding = \"post\")\n",
    "mtx_test <- pad_sequences(x_test, maxlen = maxlen, truncating = \"post\", padding = \"post\")\n",
    "class(x_train)\n",
    "dim(x_train)\n",
    "mtx_train[1,] %>% sapply(function(x) mapping_df %>% filter(index == x) %>% pull(word)) %>% paste(collapse = \" \")                   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to learn our embedding while training a neural network on the imdb dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "________________________________________________________________________________\n",
      "Layer (type)                        Output Shape                    Param #     \n",
      "================================================================================\n",
      "embedding_4 (Embedding)             (None, 20, 8)                   80000       \n",
      "________________________________________________________________________________\n",
      "flatten_2 (Flatten)                 (None, 160)                     0           \n",
      "________________________________________________________________________________\n",
      "dense_4 (Dense)                     (None, 1)                       161         \n",
      "================================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "  layer_embedding(input_dim = max_features, output_dim = 8, input_length = maxlen) %>%\n",
    "  layer_flatten() %>%\n",
    "  layer_dense(units = 1, activation = \"sigmoid\")\n",
    "model %>% compile(\n",
    "  optimizer = \"rmsprop\",\n",
    "  loss = \"binary_crossentropy\",\n",
    "  metrics = c(\"acc\")\n",
    ")\n",
    "summary(model)\n",
    "history <- model %>% fit(\n",
    "  mtx_train, y_train,\n",
    "  epochs = 10,\n",
    "  batch_size = 32,\n",
    "  validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here all the magic is done by the **layer_embedding** it takes in input our sequences of 20 *hot-encoded* words (i.e. a 10000-diimensional binary vector) and embeds each word into a 8-dimensional numeric vector: as a consequences this layer has (8 X 10000) parameters. After the **layer_embedding** we remove the bidimensional structure (20 X 8) of the data by stacking them into a 160-dimensional numeric vector. Finally we feed them to a standard classification layer.\n",
    "\n",
    "Let's see the performance of this network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2di3bbOBJE6VGcONlJHI/1/9+61psU\nSRBAF9AFsOqczcrmlLvdrGs+RQ1HSZLMGrwbkKQeJJAkCSCBJEkACSRJAkggSRJAAkmSABJI\nkgSQQJIkgKAg/Q0ouHBTcld3szaODCxQAkluutICyWWychdxszaODCxQAkluutICyWWychdx\nszaODCxQAkluutICyWWychdxszaODCxQAkluutICyWWychdxszaODCxQAkluutICyWWychdx\nszaODCxQAkluutICyWWychdxszaODCxQAkluutICyWWychdxO5YehiHwkzklkORmKz28vLys\nk4QMLFACSW6y0ieOAiQhAwuUQJKbq/TLi0AqM1m5Xdw+pV9eBJJA6srtUfplmyOBlDVZud3c\n1UvfGNJZO4HUk7ty6QdGG25kYIESSHL7l36ZYCSQcJOV291drfQzRRtuZGCBEkhye5Z+WcBI\nICEmKzeJu0LpZYo23MjAAiWQ5HYqvUbRhhsZWKAEktwepQMUbbiRgQVKIMldvXSYog03MrBA\nCSS565ZePTCKrI0MLFACSe6KpWMo2qiNDCxQAknuaqXjKNqojQwsUAJJ7kqlYynaqI0MLFAC\nSe4apRMo2qiNDCxQAknu8qXTMBJIubOxTVbuIm5Y6egjo7jayMACJZDkLlk6g6KN2sjAAqVP\nNZcKaBhOwXqc7Pbup7y0RZIbbx5exsLWRgYWKIEkN9w8mCjaqI0MLFACSW642UbRRm1kYIES\nSHJjzXeKAg8wsdRGBhYogSQ30DzapzNwJJAyZ2ObrNxF3Onm0YFR8IFattrIwAIlkOTGmKcH\nRuUaRwYWKIEkN8A8O70gkCzKnY1tsnIXccebl07SCSSLcmdjm6zcRdxx5rUrRgLJotzZ2CYr\ndxF3hDlw3VUgWZQ7G9tk5S7i3jKH714QSBblzsY2WbmLuIPmzVuABJJFubOxTVbuIu6AOeIW\nIIFkUe5sbJOVu4h7zRx3I51AsshnsnIXcS+ao29HFUgW+UxW7iLumTnpvRECySKfycpdxD0x\nvyRRZC0tkNZV8C5GuYu47+aXZIqspQXSqobPz89S99XLXcJ9/cuXwZC5tEBa/eVPHJlIajWO\nzbpP7xYfMiEylt5wIwMLlECSe0FDzu4cpvSWGxlYoASS3AsaTBSZSm+5kYEFSsdIcs9kpchQ\netuNDCxQ1c7afX6WmqzcUPdjl47zLx8ysEDVAum0a1dosnLD3KMDI9rrFcjAAlXtgqyNpKbi\n2Kj76fQCa+PIwAJV784GE0msa7UX98JJOtbGkYEFquItQhaSWNdqF+7lM92sjSMDC1TNe+0M\nJLGu1ebd69eLWBtHBhaoqjet5pPEulbbdgcvurI2jgwsUHXv/hZINO7NWxdYG0cGFqjKb6PI\nJYl1rTbqjrn/h7LxvwLpDlIWSaxrtUF37F10dI3flnGq9hv7MkliXautuRNuReVqfLSMU9Xf\nIZtHEutabcmdeEM3T+NPyzhV/63mWSSxrtVm3GkQQUuD3cjAAuXwzIYckljXahPuxE0RsnQB\nNzKwQHk8/CSDJNa1Su0+33eaBZG5tEAyKXY26SSxrlVm9+RdrkxvFre5kYEFyudxXMkksa5V\nYrcFImPpom5kYIFyeq5dKkmsa5XUbdoU2UoXdyMDC5TXAyITSWJdq4zuyR5d/rvzWH9tZGCB\ncnvSahpJrGuVzD1lyMQR7a+NDCxQfo8sTiKJda3SuF+mOn/P9m5x1l8bGVigHJ/9nUIS61rl\ncC9RZK7N+msjAwuU50P0BRLAHTrHLZDqyfXTKOJJYl2rzu6Nk3MCqZ5cP40ifueOda06uiNO\ncQukeqr1pNXlM0jRJLGuVS933HUigVRPlZ79vXZNI5Yk1rXq4E648Ucg1ZMzSLEksa7Vqu61\nk3NlatP82s/LOOUNUiRJrGu1sPtxZPmSTJG1NuvQkIEFquIx0srajyKJda2WdV+OLHMQstem\nHRoysEBlgHT40tLr8Fm79RjEkMS6Vou6h3yEzLV5h4ZIfQGlg3S4/zN9fdy4jrQehwiSWNdq\nQbcRIlNts1kgbSsXpEc0Zsu2SWJdq2XcAIaya2PMAmlbM5Ae2p7NSjg2SWJdq2j38zkFx0/7\nYh2aOfJlZAXpdoz0z0kx9ltApt89kZTcSW96QujrO8MAPRkklZMRpMMxYdfuqsWt0sY2ifXP\nI8q9tjPn2Dnr0CCxx6vmMdJDCyyFSWJdq3Z3+MycQJov45QPSEsfnB0kiXWtGt1hisrWLmoW\nSNvCgDQO0fUbIZJY12q+O+7MnECaL+OUH0h/Z/cwB0hiXasZ7ufTcjVrVzMLpAjd7mY4jF5f\nlDGbSaLWSWJdq0nuZ4S2KELWrmwWSCblzWaUq1WSWNfqpu63naYiBKgtkCqKAKSTxiilu7fk\nfNtpDkGA2gKppkhA+jtmKccdlJc7byuEqW12s44cGVigeED6u3ROPMVtqw11LxwNNfcAbtaR\nIwMLFBVIC+fEk9y22hD3EkJtPu6UdeTIwAJFBtLflafjsK7VqeYbIeNjgwXSwjJO8YH0d+mC\nP+tavWttN8722GCBtLCMU5Qg/f38fGKJda1un1Dg7Zy3tECCTfZ6Qcl+tJ5RO9K9dDRUq3YN\nN2vjyMACRQrS+CYHO0uYS6rzdmI6Y80jc2mBBJzs5CYH45bJfEl13kR0N6x5ZC4tkJCTHZN0\n2iokR9hQ+6HnJ/kkl2fNI3NpgQSd7IOk21YhN9C5a9UCkLW2u5u1cWRggSIG6U7S03Na0zcQ\n+VeCDAQl1yZzszaODCxQzCDdSFp54HF82iNqhwhq9ZKqQKooapCuJAWeHP43hqfgRdHNbVC7\nl1QFUkVxg3Qm6TPqRpvgJiVauM6bd7M2jgwsUOQg3UiK3iogCbJ13rqbtXFkYIFiB+m6e5fq\nhhB0FWuiCrtZG0cGFih+kM4kma8EtflWBoG0sIxTDYCU8JnNi2r4rQwCaWEZp1oAyUzSTs+7\nCaSKagKk6ymHbLFmgtrN2jgysEC1AZKRJNZMULtZG0cGFqg2QPraObOQxJoJajdr48jAAtUE\nSCeKLCSxZoLazdo4MrBAtQDSmSELSayZoHazNo4MLFANgZR/8o41E9Ru1saRgQWqJZCyTzmw\nZoLazdo4MrBAtQDS5Rjp9CKTJNZMULtZG0cGFqgmQBpdUs0iiTUT1G7WxpGBBaoNkEbuHJJY\nM0HtZm0cGVigmgMphyTWTFC7WRtHBhao9kDKIIk1E9Ru1saRgQWqQZDSTzmwZoLazdo4MrBA\ntQhSMkmsmaB2szaODCxQTYKUShJrJqjdrI0jAwtUmyAlosSaCWo3a+PIwALVKkhJJLFmgtrN\n2jgysEA1C1IKSayZoHazNo4MLFDtgpSAEmsmqN2sjSMDC1TLIEWTxJoJajdr48jAAtU0SLEX\nZ1kzQe1mbRwZWKAaBymOJNZMULtZG0cGFqjWQYoiiTUT1G7WxpGBBap5kGJIYs0EtZu1cWRg\ngWofpIhTDqyZoHazNo4MLFAdgLRNEmsmqN2sjSMDC1QPIG3u3rFmgtrN2jgysED1AdIGSayZ\noHazNo4MLFCdgBQmiTUT1G7WxpGBBaoXkIIHSqyZoHazNo4MLFBQkFx1Jsm7CWmv6maLFNom\nsf5xpXazNo4MLFAdgbSOEmsmqN2sjSMDC1RXIK2RxJoJajdr48jAAtUXSBeSZiixZoLazdo4\nMrBAdQbS8kaJNRPUbtbGkYEFqjuQlkhizQS1m7VxZGCB6g+kBZRYM0HtZm0cGVigegRpRhJr\nJqjdrI0jAwtUlyA9k8SaCWo3a+PIwALVJ0hPKLFmgtrN2jgysED1CtKEJNZMULtZG0cGFqhu\nQRrfEM6aCWo3a+PIwALVMUgPklgzQe1mbRwZWKB6BulOEmsmqN2sjSMDC1TXIN0OlFgzQe1m\nbRwZWKD6BulKEmsmqN2sjSMDC1TnIF1IGgaX2m27WRtHBhao3kG6oeRTu2U3a+PIwALVP0iD\nlSTWRBV2szaODCxQewEp7WPQUbVbdrM2jgwsUKsg/Tocj7+Hw8+UH+Yz2Q3dQMomiTVRhd2s\njdszX0RrIP0ahuP7YRiGFJJ8JrslK0msiSrsZm0ckfoCWgPp2/D763+//gyHhB/mM9lNDaez\ndvkosSaqsJu1cUTqC2gNpK8N0r/Dt/P/x8tnspHubJLcO/dxszaOSH0BrXFyGN5/DH9OR0kJ\nP8xnsrHu3P07/85d3KyNI1JfQGsg/fzaHzqcNkhvCT/MZ7Lx7jyUGDp3cLM2jkh9Aa3uub0N\nh3+/NkwpHNGDlEcSRef13ayN2zNfRP1fR5q6M1Ai6by2m7VxZGCB2htIGSjRdF7Xzdo4MrBA\n9X9BduZOvarE03lVN2vj9swX0Q4uyM7daSQxdV7Rzdo4IvUFtIcLsgvuFJS4Oq/mZm0ckfoC\n2s8F2SfFo8TWeSU3a+OI1BfQji7IPin6UImu8zpu1sYRqS+gXV2QfVIkSYSd13CzNo5IfQHt\n64Lss6JQouy8vJu1cXvmi2h/15GmikCJtPPSbtbGkYEFau8gRRwqsXZe2M3aODKwQK2C9PH2\nbRi+vX2k/DCfyVrdWyTxdl7Uzdq4PfNFtAbS+WLs6YTDe8IP85ms3R1Gibnzgm7WxhGpL6A1\nkH4Mr18Ivb8OPxJ+mM9kEe4QStydF3OzNo5IfQEFLshO/j9KPpOFuAOHSuSdl3KzNm5NfCEJ\npJtWSaLvvIybtXFr4gtJu3YPraDUQOcl3KyNI1JfQDrZMNYiSk10jnezNo5IfQHp9PdES4dK\nbXQOd7M2bs98Ee3+guyz5iS10jnYzdo4MrBACaSZnlFqp3Oom7VxZGCBWgRpGCvhh/lMFu+e\notRS50A3a+Og4KMlkJY0OVRqqnOcm7VxUPDR0q7dsh4P3h92+nl/rI0jAwuUQFrT50O7/Lw/\n1saRgQVKIK0LQlKDv7d7aYHkMtmSboFE50YGFiiBFNaQ8YxjVG1fN2vjyMAClQHS4UvjLx8v\nfSZb1m39CNpWf2/axs2RL6N0kA73fy5fdg7S38GGUrO/N2vj9swXkRWkQ+9bpLPbQFKzvzdr\n44DQl5ARpEP3u3ZXdzZK7p03WHrPIP1zEqwrRl1Q8u5CIpUNpMNxL1ukv+kfB4Os7eBmbRyR\n+gIygfR03qFzkPJQ4ui8rdL7A+mi+yKfydZ0p6PE0nlLpXcH0vHppc9k67pTUeLpvJ3SAsll\nsrXdaSgxdd5K6X2AdL+zYXTC4SqfydZ3p5DE1XkbpXcC0rp8JuvhjkeJrfMWSgskl8n6uGNR\n4uucv7RAcpmskzvyUImwc/rSAsllsm7uzxiWKDsnLy2QXCbr6I5AibRz6tICyWWyru5NlGg7\nJy4tkFwm6+0Ok8TcOWtpgeQyWX93CCXuzjlLCySXyTK411Fi75yxtEBymSyFe/VQib5zwtIC\nyWWyJO6V0w4NdE5XWiC5TJbGvYhSE52TlRZILpMlci+g1EjnVKUFkstkudzPJLXTOU9pgeQy\nWTb3FKWWOmcpLZBcJsvnHqPUVuccpQWSy2QJ3aNDpcY6pygtkFwmS+m+kdTux5SxjhwZWKAE\nUin37Qxeqx9TxjpyZGCBEkjl3ACSBNJ8GacEUkG3QCrgRgYWKIFU0n37RJgWP12JdeTIwAIl\nkIq67yS19+lKrCNHBhYogVTWPQymrZJAmi/jlECq4c4mSSDNl3FKINVxZ6IkkObLOCWQarmz\nUBJI82WcEkjV3DmHSgJpvoxTAqmiO/20g0CaL+OUQKrqTkVJIM2XcUogVXanoSSQ5ss4JZDq\nuxNIEkjzZZwSSB7uaJQE0nwZpwSSjzsSJYE0X8YpgeTkjjtUEkjzZZwSSG7uGJIE0nwZpwSS\no3sbJYE0X8YpgeTq3kJJIM2XcUog+bo3DpUE0nwZpwSStzt4hVYgzZdxSiD5uwMoCaT5Mk4J\nJAb3KkoCab6MUwKJxL1MkkCaL+OUQKJxL6EkkObLOCWQiNxzlATSfBmnBBKTe3aoJJDmyzgl\nkLjcT6cdBNJ8GacEEpt7gpJAmi/jlEDicz9Q8vxQGNahIQMLlECidH/e5fahMKxDQwYWKIFE\n6kaQJJDqSSDRutMf3oWrbTULJIlHt8fve/chRUhbJGK39eOVtEWqJ4HE7D6dtTOgJJDqSSDR\nu/M3SwKpngRSA+7PTJYEUj0JpCbceSgJpHoSSK24M1ASSPUkkBpyp7IkkOpJIDXlTkNJINWT\nQGrMnYKSQKongdSeO5okgVRPAqlFdyRKAqmeBFKb7iiUBFI9CaRG3TGHSgKpngRSs+7t0w4C\nqZ4EUsPuLZQEUj0JpKbdYZQEUj0JpNbdAZYEUj0JpPbdqygJpHoSSD24V1ASSPUkkDpxL7Ek\nkOpJIHXjnqMkkOpJIHXkfkZJINWTQOrLPWFJINWTQOrN/UDJ9uRw1l8bGVigBFJ/7s+HvJ53\nLJBM8pms3HMBSGL9tZGBBUogderOeFYKqnRZNzKwQAmkXt2uzzsWSCb5TFbuZQ2PQ6XapQWS\nTT6TlXtFjyeHV3/esUAyyWeycgfduSi5N762jFMCaQfuLJYYGl9cximBtAt3BkocjS8s45RA\n2ok7GSWWxmfLOCWQ9uNOY4mo8ekyTgmkPblTUKJqfLyMUwJpX+54lMgafyzjlEDanTuSJb7G\nr8s4JZB26I5CibHx8zJOCaRduiNQ4mxcILGul926t1hibRwZWKAE0n7dYZRYG0cGFiiBtGd3\nCCXWxpGBBUog7dy9yhJr48jAAiWQdu9eedcSa+PIwAIlkOQePS2leul0NzKwQAkkuU+as8Ta\nODKwQAkkua96Qom1cWRggRJIcj80Ysn2dEmBZJLPZOVGuh/7eJanSwokk3wmKzfWDSFJIFnk\nM1m54e6Md6ajSm+6kYEFSiDJvaDh08iSQLLIZ7JyF3CPni5J9nhJZGCBEkhyL2r8dMkMlgTS\ntg5fWnotkLpyX8y5LAmkTR3u/0xfHwVSV+67OYslgbQpgbQP99iczpJA2tQTPAKpU/eTOZEl\ngbSpZZD+OQnTkkSqB0venRDKDpJONvTpXjRHb5a0RdqUQNqHe80cx5JA2tQTSOOdPJ/Jyl3E\nHTBHsCSQNjUFacyRQOrJHTRvnnoQSJuagDThSCD15N4yh+8hEkjbut3NcLi8POj0d5fuCHOA\nJYFkUe5sbJOVu4g7zrzGkkCyKHc2tsnKXcQdb15CSSBZlDsb22TlLuJOMs9YEkgW5c7GNlm5\ni7hTzROWCj46BRlYoASS3Cjz43Dp9LbAUo9OQQYWKIEkN9A8OvdQ6tEpyMACJZDkxprD15fs\ntZGBBUogyQ03f1phEkh5s7FNVu4ibot5/OiUHJaMIA2TVP86rP13SAkkuQuYz2ft8mGCgjRA\nM75aE/nDcmdjm6zcRdyY0nksCaS82dgmK3cRN6x0xoYpH6T31+H7BZ3f34fh8HbiaPJlKQkk\nuSuUTmQpG6SPwxc230/k/Duc9XYF6f5lKQkkueuUTtkwZYP0NrweP15P5Hwb/nc8/jm9Om+Q\nHl8WkkCSu17pWJayQfo2vH/t3l14ef/35+sdpMeXhSSQ5K5aOmrDlA3ShZTzv6+Xnbnb9+5f\nFpJAkrt66U2WACD9GL79+vf9DtLjy0ISSHJ7lA5vmAC7dmdmPu6vHl8WkkCS26v0OkvZIP0c\nXj+O1yOj39fTDleQbl8WkkCS27H0yoYJcPr7bXgcIx3GXxaSQJLbufQCS4YLst9vF2R/DMPr\n79OrXyeQHl8WkkCS27/004Yp+P5aZGCBEkhyc5T+nGidJGRggRJIctOUjiIJGVigBJLcVKUF\n0lEgdeV2Kz0IpEKTldvD7Vc6/AwiZGCBEkhy05XWWbtSk5W7vpu1cWRggRJIctOVFkguk5W7\niJu1cWRggRJIctOVFkguk5W7iJu18WAC/1sXMugLEkhy05UWSC6TlbuIm7XxYAIFkmGychdx\nszYeTKBAMkxW7iJu1saDCRRIhsnKXcTN2ngwgQLJMFm5i7hZGw8mUCAZJit3ETdr48EECiTD\nZOUu4mZtPJhAgWSYrNxF3KyNBxOYCdKw+DJJVT47RpLqaEzOMAwrIJUIvUCSOtKYo9PbAwWS\nJGXov/8+lzUG6fygyOF4eXjk5Tmsx/PXTy+Pl/8uTgJJ6khRIB3vDwMfrl8Nj6+nL+M/N1Mg\nSR0pAaTxV8OUqcfLeEAEktSR4o+RrsG/Ph5cIEnSSPFn7YbxK4EkSWPFXUcaHx4JJEmaKQOk\n8K6dz8kGn0vdchdxszYeTGAkSPfT2tdXyyD5nf72mazcRdysjQcTGAdSigSS3ALJBpLfMZLP\nZOUu4mZtPJhA7BYp5bMyBZLcdKVZQEqRQJKbrrRAcpms3EXcrI0HEyiQDJOVu4ibtfFgAgWS\nYbJyF3GzNh5MoEAyTFbuIm7WxpGBBUogyU1XWlskl8nKXcTN2ngwgQLJMFm5i7hZGw8mUCAZ\nJit3ETdr48EECiTDZOUu4nYsnf+p5gJpQ3JXd/uVPr9JfP0nhySQNiR3dbdb6cvTSlZJCiZQ\nIG1I7upuh9Ljx/4IJORk5fZz1y799PwsgQSbrNx7AWmKUPVjpOGBwTBbEiuBJLdf6efHOF6/\njTlrt/44rnUGll/GSCDJ7VB66WGoke5gAsccvby8rDwgcp0BgSQ32l2o9AZC27WDCfzvv5dl\nTR6if/t3/Aj9xzODrg9fnX75eLL+ivRcO6mengmCF0gCafoErtEz7h5fj74cPed4SdoiyV2l\n9MY2KL52MIFRIE0eoj8B6em7KY9dFUhyFy69tR+XWDuYwLhjpMeTVkeP0B89w/j2XYEkt92N\nuVsug6KN2sEERp61e+zWLWyRxodNAkluq9t8t1wWQtu1gwmMvI40TB7/PX/it0CSG+Y23y2X\nx9Bm7WACYy/Ijh7vvXSyQbt2csPcuWYzRRu1gwlMAunpEfqj893Xc+PT099hWgSS3DDzFKHA\n3QmW2sEExoKEl0CSG2B+2gyF75az1Q4mUCAZJit3EXe8eWFnLni3nK12MIECyTBZuYu4M247\nRZUWSJmzsU1W7iLuCHPgnIJAsih3NrbJyl3EvWHeODMnkCzKnY1tsnIXcYfM26e3BZJFubOx\nTVbuIu41c9xFIoFkUe5sbJOVu4h70Rx9pdUJJD8JJLmjzIn3Kwgki3wmK3cR98Occ9OPQLLI\nZ7JyF3FfzTkUWUsLJJfJyl3EfTanE4QoLZBcJit3CfcwDLkQWUsLJJ/Jyl3AbaLIVnrDjQws\nUAJJ7pkyjolQpbfdyMACJZDknmpEUaF3FNncyMACJZDkfujpFJ1AipdAkvuqyQ5dybfm2dzI\nwAIlkOT+u3QDXcG35tncyMACJZDkXj63wNo4MrBACaR9u9fvWmBtHBlYoATSjt3B09ysjSMD\nC5RA2qt762IRa+PIwAIlkHbpjrjiytm4QGJdL3t0R923wNj4eRmnBNLe3LF3/9A1flvGKYG0\nK3fCPXRcjY+WcUog7cadeCsqT+NPyzglkPbhTr+hm6Tx+TJOCaQduNMpgpXGu5GBBUog9eu+\n3C6XA5G5tEAyyWeyci9r+vmTVUsLJJt8Jiv3ooasHTpI6aJuZGCBEki9ugcLRbbSRd3IwAIl\nkDp1GzHi/bWRgQVKIPXovu/TUb5Z3OZGBhYogdSd+0GR6U2urL82MrBACaS+3JMTDKbarL82\nMrBACaSO3M+n6QRSPQmkbtzzk90CqZ4yQDp8aem1QPJ0L52lE0j1lA7S4f7P9PVRILm5V851\nC6R6Ekjtu1cvGQmkehJIrbsDV14FUj2hQPrnJFhXUqRuFHn3IdlBOmiL5OXeug1IW6R6soF0\nOmunXTsn9/bddAKpnowgTV/7THaX7qh7UgVSPelkQ4vuyFu7BVI9CaT23NHvkBBI9WS4s+Ew\nen2Rz2T35U55n5FAqifda9eUO+3tegKpngRSQ+7Ud70KpHoSSK24M947LpDqSSA14Z6/RaJ4\nbYZfe3EZpwRSA+48jARSTQkkdncuRdbarENDBhYogcTsHmwPpxNI9SSQiN2uz3hkHRoysEAJ\nJFq3kSJTbatZIJnkM9k+3fdDI69nPLIODRlYoAQSpfvzUyCtLeOUQCJ03/bpBhtHAqmiBBKf\n+3FoZHvosECqKIHE5p6cYXDsnHVoyMACJZC43E8n6gTSfBmnBBKRe37CWyDNl3FKING4l64b\nCaT5Mk4JJBL38uVXgTRfximBROFeu4lBIM2XcUogMbhX7wUSSPNlnBJI/u7ALXUCab6MUwLJ\n2x28M1UgzZdxSiD5ujfu8BZI82WcEkiu7q03Sgik+TJOCSRH9/b7jQTSfBmnBJKbO+ZtewJp\nvoxTAsnHHfn2V4E0X8YpgeThjn4XuUCaL+OUQKrvTngYg0CaL+OUQKrtTnqkiUCaL+OUQKrs\nTnsykECaL+OUQKrqTn3AlkCaL+OUQKroTn9OnUCaL+OUQKrmznneo0CaL+OUQKrlzsBIIC0t\n45RAquPOwkggLS3jlECq4c7ESCAtLeOUQCrrPj3iMefgCFFbIFWUQCrqvn8wS+ZnSgik+TJO\nCaSSbusHHAmkhWWcEkgl3YMNI4G0tIxTAqmc+7Y58vpgFoFUUQKplPsTwJFAWljGKYFUxn2n\nyPGDWQRSRQmkEu7RKYbGOqcovXuQpIsuGHl3IdWUtkhw9/RMXUuds5Te/RbJZ7Jc7ucT3u10\nzlNaILlMlsk9v/7aSudMpQWSy2SJ3AvXXxvpnKq0QHKZLI178TaGJjonKy2QXCZL4l65G6iB\nzulKCySXyVK4V29Ope+csLRAcpksg3v95lT2zhlLCySXyfq7Q/d4c3fOWVoguUzW2x1+qwRz\n56ylBZLLZH3dW+/c4+2ct7RAcpmsq3vznXu0nROXFkguk3V0R7wBlrRz6tICyWWybu6o95FT\ndk5eWiC5TNbJHflYE3GrQ+0AABH1SURBVMLO6UsLJJfJ+rhjH2vC1zl/aYHkMlkPd/zTgdg6\nb6G0QHKZbH13ykO2uDpvo7RAcplsbXfaMx+ZOm+ltEBymWxdd+qjU3k6b6e0QHKZbE13+hOI\nWTpvqbRAcplsPXfOg7w5Om+rtEBymWwtdw5GHJ23VloguUy2kjsLI4rOmystkFwmW8WdiRFB\n5w2WFkguk63gzsbIvfMmSwskl8kWd+cdHGFq+7lZG0cGFiiBtCXjh4U1+3uzNo4MLFACaUM2\njNr9vWkbRwYWKIEU1HWvrs3P3BNIFSWQAvr8FEh0bmRggRJI6/oUSIRuZGCBEkhruh4cDTaO\n2vu9CUoLJJfJFnE/zjG0+ymwAqmiBNKSJleOmuoc52ZtHBlYoATSgqanvFvqHOhmbRwZWKAE\n0kzPV47a6RzqZm0cGVigBNKT5hdgW+kc7GZtHBlYoATSREu31bXROdzN2jgysEAJpLEW7wdq\nonO8m7VxZGCBEkgPrdxW10DnJdysjSMDC5RAumn17lT6zsu4WRtHBhYogXRR4D1H5J2XcrM2\njgwsUALprNCbJbg7L+ZmbRwZWKAE0t9uP7xSIFWUQNp86x5v50XdrI0jAwvU7kHafiADa+eF\n3ayNIwMLVAZIhy8tvW4SpIh3kpN2XtrN2jgi9QWUDtLh/s/09bFBkKIeyEDZeXk3a+OI1BfQ\nnkGKfK4JYec13KyNI1JfQKsgfT8vGb69Py/oBaTop9XRdV7Hzdo4NP44rYH0NlxAGn48L1kG\n6Z+TCvRXTBeMvLuQetEaSIfh9+n//gyz/2CyFWr1ZEPK0+q4Oq/mZm0cm3+Y1kC6ARQGqdFd\nu7SHPjJ1XtHN2jgy/UCtgfR9+PFxPH68Da/PS9oHKfHhqUSd13SzNo4mAKQ1kN4Pw1mHP89L\nWgcp+RnENJ3XdbM2jiYApNWzdh9v34bh29vspF3rIKU/ypul88pu1sax+YfJcGfD4djgyYaM\nR+KTdF7bzdo4KPho7eteuwyMSDqv72ZtHBlYoNIvyAbkM9l4dxZHFJ07uFkbN0e+jNIvyAbk\nM9lodx5HDJ17uFkbt2e+iNIvyAbkM9lYdx5GDJ27uFkbt2e+iNIvyAbkM9lIdy5H/p37uFkb\ntya+kNIvyAbkM9k4dzZH7p07uVkbR6S+gNIvyAbkM9lNff0amYdH9trtulkbR6S+gNIvyAbk\nM9ktDdFvmMDXbtjN2rg980XU/3WkwYYRb6IKu1kbRwYWqL2AtMdPgRVIFbUK0ttwVcIP85ns\nhgRSc6V7AunOUfMgWTmiTVRhN2vjiNQX0PoF2T+vw/vH6+W6bKR8JhvWGaN9fpyyQKqowAXZ\nn8O/x4/WryOdzzOwZoLazdo4IvUFFADp3+FX83c2XM7XsWaC2s3aOCL1BbR+Z8P/3odvx99t\ng3Q9782aCWo3a+OI1BfQGicngl5P5xpavvv7dv2INRPUbtbGEakvoNUNzr/fjscfw/CW8sN8\nJruq+3VY1kxQu1kbt2e+iDq+IPu4n4E1E9Ru1saRgQWqX5BG9wWxZoLazdo4MrBAdQvS+P46\n1kxQu1kbRwYWqF5BmtynypoJajdr48jAAtUpSNP7vVkzQe1mbRwZWKD6BOnpfROsmaB2szaO\nDCxQXYL0/P4j1kxQu1kbRwYWqB5Bmr2PjzUT1G7WxpGBBapDkObvh2XNBLWbtXFkYIHqD6SF\n95WzZoLazdo4MrBAdQfS0vMZWDNB7WZtHBlYoHoDafE5J6yZoHazNo4MLFCdgbT8vCDWTFC7\nWRtHBhaovkBaee4Wayao3ayNIwMLVFcgrT2/jjUT1G7WxpGBBaonkFafA8maCWo3a+PIwALV\nEUjrz1NlzQS1m7VxZGCB6gskt9odulkbRwYWqH5ACjzfmzUT1G7WxpGBBaobkEIPymfNBLWb\ntXFkYIHqBaTgB06wZoLazdo4MrBAdQJS+INbWDNB7WZtHBlYoPoAaeMDkFgzQe1mbRwZWKC6\nAGnrg8RYM0HtZm0cGVigegBp8wP5WDNB7WZtHBlYoDoAafuDLVkzQe1mbRwZWKDaByniA2JZ\nM0HtZm0cGVigoCB56MSRdw+S1PoWKeoDy1n/uFK7WRtHBhaoxkGK4og2E9Ru1saRgQWqbZDi\nOKLNBLWbtXFkYIFqGqRIjmgzQe1mbRwZWKBaB8ngttXu3s3aODKwQLUMUixHtJmgdrM2jgws\nUA2DFLtjx5sJajdr48jAAtUuSPEc0WaC2s3aODKwQDUNksFtq70HN2vjyMAC1SxICRzRZoLa\nzdo4MrBAtQpSwo4dbyao3ayNIwMLVKMgJXFEmwlqN2vjyMAC1SZIaRzRZoLazdo4MrBAtQHS\nMAzjLxM5os0EtZu1cWRggWoCpOELnBFJqRzRZoLazdo4MrBAtQDSiaMRSckc0WaC2s3aODKw\nQLUHUjpHtJmgdrM2jgwsUM2BlMERbSao3ayNIwMLVAsgjY+RcjiizQS1m7VxZGCBagKkx1m7\nLI5oM0HtZm0cGVig2gDp5s7jiDYT1G7WxpGBBao5kAxuW+29uVkbRwYWqKZAyuSINhPUbtbG\nkYEFqiWQMnfseDNB7WZtHBlYoBoCKZsj2kxQu1kbRwYWqLZAMrhttXfoZm0cGVig2gEpnyPa\nTFC7WRtHBhaoZkDK37HjzQS1m7VxZGCBagUkC0e0maB2szaODCxQjYBk4og2E9Ru1saRgQWq\nHZAMbtZMULtZG0cGFqg2QLJxRJsJajdr48jAAtUESLYdO95MULtZG0cGFqgWQLJyRJsJajdr\n48jAAtUISKxrtWM3a+PIwALVAEin7RHrWu3Yzdo4MrBA8YP0aQbp6WFeqWJNVGE3a+PIwAJF\nD9LlAMmyXoaXlxcDSXvFkLVxZGCBYgfpeqLBsF5OHF1IegkoaLeQxJpH5tICCT/Z6wk7M0g2\nGUhizSNzaYEEn+ynDSQAQ1vaaoE1j8ylBRJ6svcrSKnu5MwXw6rZ0yQCKUn0ICW6F+NtiaNl\nz/CvEQXX4zOBlCRqkD4TQAptIEpuFQycrfZ7q3z+ntfxmUBKEjNID46CYd5OZL1MQLFa/F1K\ndY41CySTsJMdcbT+hzkqdV6ZKIFVAl62IyyBlKQMkA5fGr28f1EApMuLlV2c+GC5ZSJ15wxK\nl/EISyAlKR2kw/2f8TfOgk72cx2k1D/OfplA3FVhQiu/uEBKkh2k0UvkZEfvnZiAFPwjDKoN\ndBc7gZ2MFK60r9sc+TJiBun++vZ3PTcZrJmAu5E8sf7a5siXkRmk66t/TsK0dNZpg/T46uvv\n+nEUBmCdjhVzBcy7x36EAuks3J+opzfFGvZQ0mt3457uVqZuplh/bXPky4gXpPtrI0XJtft2\nR/PE1vh9GaesII05woF03yBl7d3bau/KHcKK9G1Y5siXESVIV47G+/IJblvtXbpTd/uApZPd\n5siXESNIZ46mq7KJODbvBvIkkLZ1u5lhdmUWB9Js/bUUxx7cm2f8ypXeciNSX0CE99p9Lqyt\nNuPYrnt0DTyPJ4FkUe5sxlpeR23GsWH37N6mNJ4KviMRGViguEBaXTONxrFhd87bsB7mZwwT\nJZDyZnNRaIeh1Tg27E77yzfhqeg7EpGBBYoFpMdqWHzOd7NxbNedYl7kSSBlK3c2o126lefl\nNxvHdt0Z5rjjJ2NtZGCBYgBpNPK1D55oNo7tuvPNizwlYSWQ0mczmfDqB7g0G8d23Sbz6VSF\ngSeBlDqb6WDXPwip2Ti260aVzuFJIMVMdj7i2zfWP1Cs2Ti26y5TOo4ngRQ/2YUpBj6Zr9k4\ntusuWjrMU/AaFjKwQPmAtPjHKPQJl83GsV13jdLLPIUv5yIDC5QDSGub9NAnxTYbx3bdFUsv\n4bRKEjKwQFUHaXXHOPiRy83GsV13/dIC6abtya4eXm58dHmzcWzX7VZaIG2CtErR3/COXcNx\nbNftV1rHSBuTDWC0sUFqN47tuh1L66xdaLIhjLY4ajeO7bpZG0cGFqhaIA0hjLZ27HjXasdu\n1saRgQWqEkjbHAkkMjdr48jAAlUHpK13em1xRLtWO3azNo4MLFAUIG1ukGjXasdu1saRgQWK\nAaRtjmjXasdu1saRgQWq4jFSaINkmey25G6rtEBaV+DSQMQGiXatduxmbRwZWKDc3yEbxRHt\nWu3Yzdo4MrBAUYBkm6zcRdysjSMDC5Q7SFEbJNq12rGbtXFkYIHyBimOI9q12rGbtXFkYIEi\nAMk6WbmLuFkbRwYWKGeQIjdItGu1Yzdr48jAAuUPknmychdxszaODCxQviDFbpBo12rHbtbG\nkYEFyhWkaI5o12rHbtbGkYEFyhskwGTlLuJmbRwZWKA8QYrfINGu1Y7drI0jAwuUI0gJHNGu\n1Y7drI0jAwuUL0iQycpdxM3aODKwQPmBlLJBol2rHbtZG0cGFig3kJI4ol2rHbtZG0cGFihP\nkECTlbuIm7VxZGCB8gIpbYNEu1Y7drM2jgwsUI4goSYrdxE3a+PIwALlBFLiBol2rXbsZm0c\nGVigfEBK5Yh2rXbsZm0cGVigoCBF64sjl7qSVEguW6TkDRLtn8eO3ayNIwMLlBdIwMnKXcTN\n2jgysEB5gJS+QaJdqx27WRtHBhYoB5AyOKJdqx27WRtHBhYoH5Cgk5W7iJu1cWRggaoPUs4G\niXatduxmbRwZWKBcQMJOVu4ibtbGkYEFqjpIWRsk2rXasZu1cWRggaoNUh5HtGu1Yzdr48jA\nAuUAEnqychdxszaODCxQlUHK3CDRrtWO3ayNIwMLVH2Q4JOVu4ibtXFkYIGqC1LuBol2rXbs\nZm0cGVigqoOEn6zcRdysjSMDC1RVkLI3SLRrtWM3a+PIwAJVE6R8jmjXasdu1saRgQWqMkgl\nJit3ETdr48jAAlURJMMGiXatduxmbRwZWKDqglRksnIXcbM2jgwsUPVAsmyQaNdqx27WxpGB\nBaoaSCaOaNdqx27WxpGBBaomSIUmK3cRN2vjyMACVQsk2waJdq127GZtHBlYoCpukUpNVu4i\nbtbGkYEFyvdTzSGTlbuIm7VxZGCBEkhy05UWSC6TlbuIm7VxZGCBqgXSMAylJit3ETdr48jA\nAlUJpOHz89NCEuta7djN2jgysEDVAenEkYkk1rXasZu1cWRggRJIctOVFkhrEkjtuVkbRwYW\nKB0jyU1XWiCtS2ftWnOzNo4MLFC6jiQ3XWmB5DJZuYu4WRtHBhYogSQ3XWmB5DJZuYu4WRtH\nBhYogSQ3XWmB5DJZuYu4WRtHBhYogSQ3XWmB5DJZuYu4WRtHBhYogSQ3XWmB5DJZuYu4WRtH\nBhYogSQ3XWmB5DJZuYu4WRtHBhYogSQ3XWmB5DJZuYu4WRtHBhYoKEjl9I9q76i0a+1MCSTV\npistkIppr2tVILUigaTadKUFkiTtVAJJkgASSJIEkECSJIAEkiQBJJAkCaAmQDp8ybO8W2G/\n39uv9MG5fqZaAOlw/8envFuY7//sqPThUbopkgTSdnWBVLOwQCorxzS7guRY2pthgVRCuwTJ\n90BFICWoFZA8twp+B/x+xZ1PNgikUnJfq/sqTlFaIOHleRaY4G/zPksLJLh8R6otkk9pgYSW\n80QFkk9pgQSW5+7Vub5bYd3Z0IxaAEmS6CWQJAkggSRJAAkkSQJIIEkSQAJJkgASSJIEkECS\nJIAEkiQBJJAkCSCBRKVB66NRacVRSSC1Kq04KgmkVqUVV0kfP4bhx8fxzMr34fX99L330/cu\nr74Ph7fLwrfLK6ktCaRKOgxf+nY8sfKFz3D4Yurj/L3Hq++nhd9Pr0RScxJIdfTzBMfb8OvE\nyuvH8fXy5evx9urH8fdpt+688OfQ1ltxpKNAqqVv50FfNjp/vnblThunb8P7/dXH5T8bTt/S\nkVKD0iqro+GqGyVLr47jb0ltSausjgRS59Iqq6Nv90EPlx2615Vdu8e/UkvSKqujt9M5hf+d\n8Bm+/vl4HX5OTza8Hf9MN1JSW9Iqq6PLCe7TeYYvkE4nvY/j09/vt5PjAqlVaZVV0uni6+vv\n43nX7vV2GfZ+QfbP6+WVQGpVWmW1JUq6lNZqbQmkLqW1WlsCqUtprdaWQOpSWquSBJBAkiSA\nBJIkASSQJAkggSRJAAkkSQJIIEkSQAJJkgD6P1E4XE3cRQmrAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.6732</li>\n",
       "\t<li>0.7032</li>\n",
       "\t<li>0.711</li>\n",
       "\t<li>0.7102</li>\n",
       "\t<li>0.705</li>\n",
       "\t<li>0.706</li>\n",
       "\t<li>0.7022</li>\n",
       "\t<li>0.6944</li>\n",
       "\t<li>0.6894</li>\n",
       "\t<li>0.6808</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.6732\n",
       "\\item 0.7032\n",
       "\\item 0.711\n",
       "\\item 0.7102\n",
       "\\item 0.705\n",
       "\\item 0.706\n",
       "\\item 0.7022\n",
       "\\item 0.6944\n",
       "\\item 0.6894\n",
       "\\item 0.6808\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.6732\n",
       "2. 0.7032\n",
       "3. 0.711\n",
       "4. 0.7102\n",
       "5. 0.705\n",
       "6. 0.706\n",
       "7. 0.7022\n",
       "8. 0.6944\n",
       "9. 0.6894\n",
       "10. 0.6808\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] 0.6732 0.7032 0.7110 0.7102 0.7050 0.7060 0.7022 0.6944 0.6894 0.6808"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "history$metrics$val_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An **accuracy** around 70% is not too bad considering we are using only 20 words for each review. Moreover the networks which we have seen so far do not takes into account the order of words and this order often contains important information for a text classification task.\n",
    "\n",
    "Let's see how the embedding works by extracting the partial output of the network on the first two test reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'array'"
      ],
      "text/latex": [
       "'array'"
      ],
      "text/markdown": [
       "'array'"
      ],
      "text/plain": [
       "[1] \"array\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>2</li>\n",
       "\t<li>20</li>\n",
       "\t<li>8</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 2\n",
       "\\item 20\n",
       "\\item 8\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 2\n",
       "2. 20\n",
       "3. 8\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1]  2 20  8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embedding_layer_name <- model$layers[[1]]$name\n",
    "intermediate_layer_model <- keras_model(inputs = model$input,\n",
    "                                        outputs = get_layer(model, embedding_layer_name)$output)\n",
    "\n",
    "pred <- mtx_test[1:2,] %>% predict(intermediate_layer_model, .)\n",
    "\n",
    "pred %>% class\n",
    "pred %>% dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first two words of each review are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'&lt;START&gt; please give'"
      ],
      "text/latex": [
       "'<START> please give'"
      ],
      "text/markdown": [
       "'&lt;START&gt; please give'"
      ],
      "text/plain": [
       "[1] \"<START> please give\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'&lt;START&gt; this film'"
      ],
      "text/latex": [
       "'<START> this film'"
      ],
      "text/markdown": [
       "'&lt;START&gt; this film'"
      ],
      "text/plain": [
       "[1] \"<START> this film\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mtx_test[1, 1:3] %>% sapply(function(x) mapping_df %>% filter(index == x) %>% pull(word)) %>% paste(collapse = \" \") \n",
    "mtx_test[2, 1:3] %>% sapply(function(x) mapping_df %>% filter(index == x) %>% pull(word)) %>% paste(collapse = \" \") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words *give* and *film* are respectively embedded in vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.408642649650574</li>\n",
       "\t<li>-0.360232323408127</li>\n",
       "\t<li>0.381942182779312</li>\n",
       "\t<li>-0.34835746884346</li>\n",
       "\t<li>0.114029869437218</li>\n",
       "\t<li>0.0238661095499992</li>\n",
       "\t<li>0.0243303719907999</li>\n",
       "\t<li>0.215956971049309</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.408642649650574\n",
       "\\item -0.360232323408127\n",
       "\\item 0.381942182779312\n",
       "\\item -0.34835746884346\n",
       "\\item 0.114029869437218\n",
       "\\item 0.0238661095499992\n",
       "\\item 0.0243303719907999\n",
       "\\item 0.215956971049309\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.408642649650574\n",
       "2. -0.360232323408127\n",
       "3. 0.381942182779312\n",
       "4. -0.34835746884346\n",
       "5. 0.114029869437218\n",
       "6. 0.0238661095499992\n",
       "7. 0.0243303719907999\n",
       "8. 0.215956971049309\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1]  0.40864265 -0.36023232  0.38194218 -0.34835747  0.11402987  0.02386611\n",
       "[7]  0.02433037  0.21595697"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>-0.0831379964947701</li>\n",
       "\t<li>-0.0105152847245336</li>\n",
       "\t<li>0.0675807818770409</li>\n",
       "\t<li>-0.0476522371172905</li>\n",
       "\t<li>0.0397259928286076</li>\n",
       "\t<li>-0.0381452701985836</li>\n",
       "\t<li>0.155477046966553</li>\n",
       "\t<li>-0.0162332933396101</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item -0.0831379964947701\n",
       "\\item -0.0105152847245336\n",
       "\\item 0.0675807818770409\n",
       "\\item -0.0476522371172905\n",
       "\\item 0.0397259928286076\n",
       "\\item -0.0381452701985836\n",
       "\\item 0.155477046966553\n",
       "\\item -0.0162332933396101\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. -0.0831379964947701\n",
       "2. -0.0105152847245336\n",
       "3. 0.0675807818770409\n",
       "4. -0.0476522371172905\n",
       "5. 0.0397259928286076\n",
       "6. -0.0381452701985836\n",
       "7. 0.155477046966553\n",
       "8. -0.0162332933396101\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] -0.08313800 -0.01051528  0.06758078 -0.04765224  0.03972599 -0.03814527\n",
       "[7]  0.15547705 -0.01623329"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred[1,2,]\n",
    "pred[2,3,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurring Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have seen networks that are called **feed forward networks** in which the data flows in only one direction: from input towards output. This type of **network** cannot model directly sequence data: in order to pass the IMBD review to the final classification layer we had to pack a whole review into a single vector (dimension #words x #embedding_vector_size = 160).\n",
    "\n",
    "However when we process a sequence (or a sentence) we process it step by step (or words by words), remembering the words we read (and eventually the decisions we made) in the past.\n",
    "\n",
    "**A Recurrent Neural Network** tries to mimic this type of learning process by introducing a **LOOP** in the network:\n",
    "\n",
    "<img src=\"fig/RNN.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **RNN** processes a sequence *step by step* maintaining a **state** related to the piece of sequence it has seen so far. How does it do that? By feeding in input to the node at step **T** a *state* computed as the *output* produced by the **RNN** node at step **T-1**\n",
    "\n",
    "<img src=\"fig/rnn_unrolled.png\" width=\"800\">\n",
    "\n",
    "Note that the state is initialized at zero at the beginning of each sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use them in **KERAS** we just need to insert in our network a **layer_simple_rnn**. Let's see how this works on the **IMDB** dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_embedding(input_dim = max_features, output_dim = 8) %>%\n",
    "    layer_simple_rnn(units = 8) %>%\n",
    "    layer_dense(units = 1, activation = \"sigmoid\")\n",
    "    model %>% compile(\n",
    "    optimizer = \"rmsprop\",\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"acc\")\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    mtx_train, y_train,\n",
    "    epochs = 10,\n",
    "    batch_size = 32,\n",
    "    validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "0.71"
      ],
      "text/latex": [
       "0.71"
      ],
      "text/markdown": [
       "0.71"
      ],
      "text/plain": [
       "[1] 0.71"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2di3bbSA5E6VGceGYniTPW/3/rWtaL\nEl/djUKjull1zmblcMqAwLqh+NRwlCTJrCG6AUnqQQJJkgASSJIEkECSJIAEkiQBJJAkCSCB\nJEkACSRJAggK0vuKVhduSu7qbtbGkYEFSiDJTVdaIIVMVm4XN2vjyMACJZDkpistkEImK7eL\nm7VxZGCBEkhy05UWSCGTldvFzdo4MrBACSS56UoLpJDJyu3iZm0cGVigBJLcdKUFUshk5XZx\nszaODCxQAkluutI7AenwqbnXAqkrN2vjiNQ7KB+kw+2Px9dHgdSVm7VxROodJJDkpis9DMPK\nb+aUQJKbrfTw8vKyTBIi9Q5CgfTXSbCupB3rxNEnSdFtZMoGkg429OuOKv1y1uImCZF6B+mj\nndxMpV9eBJJA6sodUfplmyOBVDRZucPc9UtfGNJRO4HUk7t66StGG25E6h1kuLLhcNTBho7d\nlUuPMNoLSMuCTlbuWHfV0g8YCSTgZOWOdtcr/fLyxJFAAk1WbgJ3rdJTjAQSZrJyU7jrlJ7D\nSCAhJis3ibtC6XmKNtzIwAIlkOQOKr2IkUAyTlZuJrdz6RWMBJJpsquSu7rbtfQaRRtuZGCB\nEkhy1y+9jpFAKp/sluSu7vYrvYWRQCqdzbbkru72Kr2NkUAqnc225K7udim9eoghrTYysEAJ\nJLmrlU7ESCCVzmZbcld3w0snYySQSmdjm6zcLm5s6QyKNmojAwuUQJLbv3QeRgKpdDa2ycrt\n4saVzsVIIJXOxjZZuV3coNL5FG3URgYWKIEkt2PpIowEUulsbJOV28VtMp+fA1SIkUAqnY1t\nsnK7uC3m4eUudG1kYIESSHLDzYMNI4FUOhvbZOV2cSNA8qiNDCxQAklusPlK0crTUi21kYEF\nSiDJDTW/ADgSSIWzsU1Wbhd3kflG0drTu221kYEFSiDJjTKP94z8GkcGFiiBJDfG/HiAQSBZ\nVDob22TldnHnmZ+P0wkki0pnY5us3C7uDPPM0W6BZFHpbGyTldvFnWyePWkkkCwqnY1tsnK7\nuBPNC+deBZJFpbOxTVZuF3eKefkKBoFkUelsbJOV28W9bV67EEggWVQ6G9tk5XZxb5nXr6cT\nSBaVzsY2Wbld3KvmzatSBZJFpbOxTVZuF/eKOeHiboFkUelsbJOV28W9aE66R0IgWRQzWbld\n3PPm1DuNBJJFMZOV28U9Z06/YU8gWRQzWbld3FNzzn2vAsmimMnK7eJ+MmfePS6QLIqZrNwu\n7rE5/1kmAsmimMnK7eK+mV/yMRJINsVMVm4X99lcRJG1tEAKmazcLu5jOUXW0gIpZLJye7iN\nD3kUSBbFTFZuB7eJIlvpDTcysEAJJLmnMlJkKb3pRgYWKIEk95Pun+mcnpVqcyMDC5RAkvtB\nLy8CqUQCSe6Rbp/pBhtHAsmkmMnKDXI/7Bk5PnTY5kYGFiiBJPdZzwcYWBtHBhYogST3++zR\nbtbGkYEFCgqS1KRuFEU30rK0Rdq5e/ECBtbGkYEFSiDt2b12GRBr48jAAiWQ9utevwyItXFk\nYIESSDt1b15Mx9o4MrBACaQ9ulMuSaVs/F0gsa6X/bkT74/ga/yyjFMCaV/u9LuMyBq/L+OU\nQNqTO+cuI6rGx8s4JZB24868V4+n8adlnBJI+3BnUoQsDXYjAwuUQNqBO33HCF4a70YGFiiB\n1K/7fCdEEUXW0gLJpJjJyj2vr3vzCikylhZINsVMVu5ZjZ6nRfZALZsbGVigBFKvbhNFttKu\nbmRggRJIXbptGyNTaW83MrBACaT+3KPPdJTPAbK5kYEFSiD15r5vi2zPL2F928jAAiWQunI/\nfqIz1WZ928jAAiWQ+nFP9osEUj0JpE7cc4cXBFI9CaQe3AsH6QRSPQmk9t2Lh7oFUj0JpMbd\nayeMBFI9CaSW3RunXQVSPQmkZt3bVy8IpHoSSG26k64BEkj1JJBadCdeSSeQ6kkgNedOvx5V\nINVTLZBov7eqMXfWVd0CqZ4qgTR8fHz0eAVlVXf2nXoCqZ7qgHTiyEQS61qt5n7Jpsham+Jt\nzy3jlEBidl8+EBdRZK3NOjRkYIESSMTux8eXVH5+CevQkIEFSvtIvG7r40sEUkXpqB2v2/As\nLXNt3qEhAwtUtfNIHx9ek+3UbaXIUttsFkgmrbz90z6S02Q7dI8+04V9IGYdGjKwQFXcIllI\nYl2rHu4xRYEfiFmHhgwsUPUuETKRxLpW0e7nA92BnbMODRlYoCpea2chiXWtQt0zB+kE0nQZ\np2petGogiXWt4tzzh7oF0nQZp6pe/V1OEutaBbkXzxcJpOkyTtW9jaKYJNa1CnGvHOgWSNNl\nnKp8P1IpSaxr1e5eP10kkKbLOFX7xr5CkljXqtG9edJVIE2Xcar6HbJlJLGuVYM76UI6gTRd\nxqn6t5oXkcS6VkvdqZejCqTpMk4FPLOhhCTWtVrifj7pWrN2LbNAMilxNgUksa7VbHfmnREC\nabqMUyFPEconiXWtZrlzNkXo2pXNAsmk5Nlkk8S6VtPd+RDhatc3CyST0meTSxLrWk1zF2yK\nYLVjzALJpIzZZJLEulY3NdjuFxdI02WcCnvSah5JrGt1Q+WbInttq5t15MjAAhX3yOIskljX\n6pqsEFlqA9ysI0cGFqgCkA6fmnud++zvHJJY1+qSxpsiy93iAmm6jFP5IB1ufzy+PmY/Regj\nHSXWtTqrlycJJKQbkXoHVQJpmM1TOkmsa3Wip72i+fftVBvrZh05IvUOAoB01+KbH5b+ZU4l\niXWtPmpmr6jd5/mxjtwceR9ZQbruI/110nKVM0gzS04kZbfAqPGmKLoXqb6MIB2OSR/thuWD\nV2nbJNZ/Hi9aOUBH3jll6T1ukY7p+0gLLCWRxLpW3zfPFRF3TltaIC1rWH6QdQpJrGt1+1wR\na+fMpQXS+myWQpdAEuNa3dgUudau4GZtHJF6B9UE6X3pX/BtkujWahJETrXruFkbR6TeQYYr\nGw6j12clzWYugpskhV52+ngA++VBvrUF0swyToVcazdN4hZJcWt1dEr15SUTImttgTS3jFNh\nF60+BXKDpLC1ejuVXESRrXasm7VxZGCBirv6+2nDtE5SMEglBNlrx7pZG0cGFqhIkE4aJXQN\npaC1amPIVjvazdo4MrBARYP0/pTWgsluq8wNoKi4NoGbtXFkYIEiAOmkrc9PldfqYzOtXnYq\nkCqKBKT3KUsPONVaq3MdsCbK2c3aODKwQPGA9KWPBZb81+oyx6yJcnazNo4MLFBkIF0P3j0n\n2nmtrn6yZE2Us5u1cWRggWIDaXzX7NaOU7LWam+XYE2Us5u1cWRggaID6fH+85kPXKinwyX/\nWtZEObtZG0cGFig+kGaucpjDKYerx9q5v4Q1Uc5u1saRgQWKEaRnkq6X6azyVKrtblgT5exm\nbRwZWKAoQXoi6enRKdUIKum8Hzdr48jAAsUJ0iNJi88g+pIfQUWdd+NmbRwZWKBIQXo85GB6\nOtw6hgliTZSzm7VxZGCBYgXpiSTD0+EEUnOlBRJ0smOSLOul4aedCqSZZZwiBmlMkmm9tPu0\nU4E0s4xTzCCNSGJdqx27WRtHBhYoapDuJLGu1Y7drI0jAwsUN0i3Qw6sa7VjN2vjyMACRQ7S\nlSTWtdqxm7VxZGCBYgfp8vGOda127GZtHBlYoPhB+iKJda127GZtHBlYoBoAKfML0LG19+tm\nbRwZWKBaAMlMEmsmqN2sjSMDC1QTIOV8bTO89l7drI0jAwtUGyAZSWLNBLWbtXFkYIFqBKSj\n6eMdayao3ayNIwML1CJI/xyOx5/D4e+cX+Y22dPVcgaSWDNB7WZt3J55Fy2B9M8wHH8fPgOc\nQ5LXZIdPhiwksWaC2s3aOCL1DloC6dvw8/N///waDgv/wZycJjt87SEN5TtKrJmgdrM2jki9\ng5ZA+twg/Tt8+/r/dDlN9gpSMUmsmaB2szaOSL2Dljg5DL9/DL9Oe0kZv8xpsjeQziTlo8Sa\nCWo3a+OI1DtoCaS/P3ePDqcN0lvGL/Oa7HDh6L0QJdZMULtZG0ek3kGLn9zehsO/nxumHI58\nj9pdVEISayao3ayN2zPvombOI92VTxJrJqjdrI0jAwtUgyDlo8SaCWo3a+PIwALVxgnZZ3cm\nSayZoHazNm7PvIuaOCE7484iiTUT1G7WxhGpd1ALJ2Rn3TkksWaC2s3aOCL1DmrhhOy8O4Mk\n1kxQu1kbR6TeQS2ckF1wp+8osWaC2s3aOCL1DmrihOyCO5kk1kxQu1kbR6TeQW2ckF1wp5LE\nmglqN2vj9sy7qMnzSHelocSaCWo3a+PIwALVOEhpJLFmgtrN2jgysEAtgvTn7dswfHv7k/PL\nIiabcu0dayao3ayN2zPvoiWQvk7Gng44/M74ZTGT3SaJNRPUbtbGEal30BJIP4bXT4R+vw4/\nMn5ZzGS3SWLNBLWbtXFE6h20ckL24f+TFDPZ902UWDNB7WZt3Jp4J/UB0gZJrJmgdrM2bk28\nk3r4aHfS6jEH1kxQu1kbR6TeQR0cbLhohSTWTFC7WRtHpN5B0MPfsfoiKboJaadq/YTsg5Y2\nSqz/uFK7WRtHBhaorkBaIok1E9Ru1saRgQVqFqRhrIxfFjPZseaPObBmgtrN2jgo+Gh1BtI8\nSqyZoHazNg4KPlp9fbQ7aYYk1kxQu1kbRwYWqP5AmkGJNRPUbtbGkYEFqkeQJiixZoLazdo4\nMrBA9QnSE0msmaB2szaODCxQnYL0eCScNRPUbtbGkYEFqluQxiixZoLazdo4MrBAdQzSnSTW\nTFC7WRtHBhaonkG67SmxZoLazdo4MrBAdQ3SFaX7tyvVrN24m7VxZGCB6hykC0kfFpJYE+Xs\nZm0cGVigegfptlEKqd2ym7VxZGCB6h+k4aPwK5wBtVt2szaODCxQ+wGpGCXWRDm7WRtHBhao\n/kEyk8SaKGc3a+PIwAK1A5DO34lejhJropzdrI0jAwvUHkC6uEtRiu88xM3aODKwQO0IpPfC\nD3gEnUe4WRtHBhaoPYFUuFGi6Ly+m7VxZGCB2hdIRSiRdF7bzdo4MrBA7Q2kApRoOq/rZm0c\nGVig9gdS9q4ST+dV3ayNIwML1A5BuqGUyBJT5xXdrI0jAwvULkHKQ4mr82pu1saRgQVqpyDl\noMTWeSU3a+PIwAK1W5Dekw87EHZew83aODKwQO0ZpESUKDv3d7M2jgwsUPsGKQkl0s693ayN\nIwML1N5BSjgYztq5s5u1cWRggdo9SNuHHXg7d3WzNo4MLFAC6X0LJebOHd2sjSMDC5RA+tIa\nStydu7lZG0cGFiiBdNUiS/Sd+7hZG0cGFiiBdNcCSg107uFmbRwZWKAE0lizKDXROd7N2jgy\nsEAJpCdNWWqlc7CbtXFkYIESSBM9o9RO51A3a+PIwAIlkGb0gNKw0yeHszaODCxQAmleN5ZO\nT8Xb5ZPDWRtHBhYogbSkj5H2+ORw1saRgQVKIC1LIDG6kYEFSiCtKu+mdGztYDdr48jAAiWQ\nNmRFqdX3Tds4MrBACaQtDTaUmn3frI0jAwuUQEpxG1hq9n2zNo4MLFACKc1djFJ45w2WFkgh\nk63k/ihjiaDz5koLpJDJ1nOXsMTReVulBVLIZGu681Fi6byl0gIpZLKV3ZkoEXXeTGmBFDLZ\n+u4clrg6b6O0QAqZbIQ7HSW2zlsoLZBCJhvjTkWJr3P+0jsB6fCp8Y/3lzGTDXMnsUTZOXnp\nfYB0uP1x/nG/ICUdxCPtnLr0HkE67HiLdNLHFku0nROX3iFIhz1/tLtonSXmzllL7xmkv06C\nddWabihFNyJFyQbS4agt0kVLmyX+zvlK726L9HTcYdcgvff2eEnWxhGpd5ANpLNui2ImS+Se\nQamRzqlK7w6k49PLmMlyuZ9ZaqdzntICKWSybO5HlFrqnKX0PkC6XdkwOuBwUcxk+dwjlNp9\nTivryBGpd5CutfNx308utfqcVtaRIwMLlEDyckNIEkjTZZwSSH7uzcuHHGvb3awjRwYWKIHk\n6R6sLAmk6TJOCSRX94ikIpQE0nQZpwSSr/vrqF05SwJpuoxTAqmOuxAlgTRdximBVMtdhJJA\nmi7jlECq6M5nSSBNl3FKIFV156IkkKbLOCWQKrvzUBJI02WcEkj13RksCaTpMk4JpAh38vFw\ngTRdximBFONOPE8rkKbLOCWQwtwpLAmk6TJOCaRI9yZKAmm6jFMCKdi9zpJAmi7jlECKd6+g\nJJCmyzglkBjciygJpOkyTgkkEvc8SwJpuoxTAonGPbe3JJCmyzglkIjc0wPiAmm6jFMCicv9\nxJJAmi7jlECic49QinwqHuvQkIEFSiAxuu/bpbin4rEODRlYoAQSqRtBkkCqJ4FE686/nxZX\n22oWSCbFTLZb9/BhZEkg1ZNAInZbny8pkOpJIDG7T0ftDDAJpHoSSA24PwphEkj1JJDacBeh\nJJDqSSC1485mSSDVk0Bqyp3HkkCqJ4HUmDvnM55AqieB1J47mSWBVE8CqUl3GkoCqZ4EUqvu\nBJYEUj0JpIbdWywJpHoSSE2713eXBFI9CaTW3SuXPQikehJIHbiXWBJI9SSQ+nDPsiSQ6kkg\n9eOeoCSQ6kkgdeV+ZEkg1ZNA6s19Z8n2DCLWt40MLFACqT/3aH8p6skpAsmkmMnKPdXKMXHv\n0t5uZGCBEki9uq0ssb5tZGCBEki9ukdPTiliifVtIwMLlEDq1j187SMVo8T6tpGBBUog9eu+\nHrUrY4n1bSMDC5RA2oW7gCWOxmeWcUog7cSdvbvE0vhkGacE0n7cecceiBp/XMYpKEgSu0Ys\nRbfSmbRF2ps79aA4XePXZZwSSHt0p7BE2fi7QGJdL3t1b26YWBtHBhYogbRj9ypLrI0jAwuU\nQNq3e3nDxNo4MrBACSS551libRwZWKAEktzvsxsm1saRgQVKIMl90SNLtttrBZJJMZOVG+b+\neBDnjerIwAIlkOR+FIYkgWRRzGTlRrtTL35wKL3tRgYWKIEk94wGK0sCyaKYycrt4H68UT2f\nJYFkUcxk5fZwX47alcIkkCyKmazcLu6buWjTJJAsipms3C7uB3M2SwLJopjJyu3inpizYBJI\nFsVMVm4X96w5mSWBZFHMZOV2cS+Z03aZBJJFpbOxTVZuF/ea+fHY+BxOAsmi0tnYJiu3i3vD\nvMGSQLKodDa2ycrt4k4yL8IkkCwqnY1tsnK7uNPNcywJJItKZ2ObrNwu7izz84bJ8W4mZGCB\nEkhyg8wjls5fhOFTOyXUD6n+54DM+GJN5C8rnY1tsnK7uEvMD7tMTnczpYR6WP7JSwJJbqx5\n+9C4rXZKqAWS3DRug9nMUjlIv1+H72d0fn4fhsPbiaOHH70kkOTGm4fLflIpTMUg/Tl8YvP9\nRM6/w5feLiDdfvSSQJLbwXw7alfGUjFIb8Pr8c/riZxvw/+Ox1+nV18bpPuPThJIcjuXLtgy\nFYP0bfj9+fHuzMvvf/9+vYF0/9FJAknuCqUz95qKQTqT8vXn6/nD3PXvbj86SSDJXad0DksA\nkH4M3/759/cNpPuPThJIctcsnQYT4KPdFzN/bq/uPzpJIMldvfQmS8Ug/T28/jle9ox+Xg47\nXEC6/ugkgSR3ROn1LRPg8PfbcN9HOox/dJJAkjuq9PJek+GE7PfrCdkfw/D68/TqnxNI9x+d\nJJDkDiw9z9LqtePIwAIlkOQOL/1E0/q148jAAiWQ5OYo/bRxWiQJGVigBJLcNKUF0kXwycod\n5w4rLZAEUk/uuNLaR/KarNwB7sDSOmrnNVm567tZG0cGFiiBJDddaYEUMlm5XdysjSMDC5RA\nkpuutEAKmazcLm7WxlcT+N+ykEGfkUCSm660QAqZrNwubtbGVxMokAyTldvFzdr4agIFkmGy\ncru4WRtfTaBAMkxWbhc3a+OrCRRIhsnK7eJmbXw1gQLJMFm5Xdysja8mUCAZJiu3i5u18dUE\nCiTDZOV2cbM2vprAQpCG2ZdZqvLdMZJUR2NyhmFYAMkj9AJJ6khjjk63BwokSSrQf/99zGsM\n0teDIofj+eGR5+ewHr9+fnp5PP93aRJIUkdKAul4exj4cPlpuP/8+DL9ezMFktSRMkAa/zQ8\nMnV/mQ6IQJI6Uvo+0iX4l8eDCyRJGin9qN0wfiWQJGmstPNI490jgSRJExWAtP7RLuZgQ8yp\nbrld3KyNryYwEaTbYe3Lq3mQ4g5/x0xWbhc3a+OrCUwDKUcCSW6BZAMpbh8pZrJyu7hZG19N\nIHaLlPNdmQJJbrrSLCDlSCDJTVdaIIVMVm4XN2vjqwkUSIbJyu3iZm18NYECyTBZuV3crI2v\nJlAgGSYrt4ubtXFkYIESSHLTldYWKWSycru4WRtfTaBAMkxWbhc3a+OrCRRIhsnK7eJmbXw1\ngQLJMFm5XdyBpcu/1VwgbUju6u640l83iS//5jUJpA3JXd0dVnr4elrJIkmrCRRIG5K7ujuo\n9PWxPwIJPVm5Y9z1Sz88P0sgAScr9z5AmnkOXc19pOGOwTBZkiqBJHdg6RmE3lFH7ZYfx7XM\nwPzLFAkkuaNKzyCU4F5N4Jijl5eXhQdELjMgkORGu11LLxGU4F5N4H//vczr4SH61z/Hj9C/\nPzPo8vDVxx/vT9ZfkECSu2rpxa1Qau3VBGaB9PgErtEz7u4/j34cPed4TgUgHT4191ogdeX2\nKJ1E0Ubt1WwmgfTwEP0HkJ7+Nuexq/kgHW5/PL4+CqSu3ODSiQht114NZ9o+0v1Jq6NH6I+e\nYXz9W4Ekt90NK526GUqsvRrOxKN29491M1uk8W6TQJLb6saULqBoo/ZqOBPPIw0Pj/+ePvE7\nEKS/Tsr+XVKnOn04ekSoTt3UE7Kjx3vPHWwI+Wh30BapS7fBXLYZSqy9Gs4skJ4eoT863n05\nNv54+HudFhtIp6N2+mjXp7vUbKZoo/ZqOFNBwssI0uPr0tnYJiu3i9vvslNb7dVwtgqSDjb0\n6842P22KBNKGBNI+3Dnm5w9z6/e42mqvhrMlkG5XMxxGr88qnY1tsnK7uNPMC3tEq9dv22qv\nZrMpkJZVOhvbZOV2cSeYV44r+DW+mkCBZJis3C7uDfPGoTmBZFHpbGyTldvFvWbePrwtkCwq\nnY1tsnK7uJfMaSeJgkCKk0CSO8OcfKZVIFkUM1m5XdwTc9b1CgLJopjJyu3ifjBnX/QjkCyK\nmazcHu77maCN43P40gIpZrJyO7iv1yYUMGQtveFGBhYogST3jIaVc63OpbfcyMACJZDkntFg\noshUesuNDCxQAknuiawUGUpvu5GBBUogyf0oO0TFpZPcyMACJZDkvmu8KfK7ftvmRgYWKIEk\n90VPmyLWxpGBBUogyf0+u1fE2jgysEAJJLnn94pYG0cGFiiBtG/38gE61saRgQVKIO3YvXqA\njrVxZGCBEkg7dW+eK2JtHBlYoATSHt0p54ooG38XSKzrZXfu1MsW6Bq/LuOUQNqVO+OyBa7G\nR8s4JZD24867+Ieo8cdlnBJIO3FnX0HH0vhkGacE0i7cBdehcjQ+s4xTAql/d9nV3ASNzy/j\nlEDq1/312IUyiqylBZJJMZOVe17Gu1xZ3zYysEAJpF7dJopspV3dyMACJZC6dFspMpT2diMD\nC5RA6s89/khXfpsr69tGBhYogdSb+86QiSPat40MLFACqSv3wwc629fmsb5tZGCBEkj9uCe7\nRabarG8bGVigBFIn7rmjCwKpngRSD+6FY3QCqZ4EUvvuxSPdAqmeBFLr7pXzRQKpngRS0+71\ns64CqZ4EUsPurYsXBFI9CaRm3dvXAAmkehJIbbqTrqQTSPUkkFp0J16QKpDqSSA1506/rlsg\n1ZNAasuddXeEQKongdSQO/cmI4FUTwKpFXfBrXoCqZ4EUhPuAoqstRne9uwyTgmkBtxFFFlr\nx7/thWWcEkjM7tOteaUUWWuzDg0ZWKAEErF7KPtEB6lNOzRkYIGCgiRBdecouhNpU9oi0bpN\nGyNjbatZWySTYibbp/u2NYp6EBDr0JCBBUogMbrv+0YWjgRSRQkkOvfoAIPtgVoCqaIEEpf7\n6ThdYOesQ0MGFiiBROSeHu0WSNNlnBJINO65c0YCabqMUwKJxD1/sFsgTZdxSiAxuBdPGQmk\n6TJOCaR498qZV4E0XcYpgRTtXr2AQSBNl3FKIIW6ty4DEkjTZZwSSIHu7avpBNJ0GacEUpg7\n5aJUgTRdximBFONOvLRbIE2XcUogRbiT75AQSNNlnBJI9d0ZNxoJpOkyTgmkyu682/UE0nQZ\npwRSVXfuXa8CabqMUwKpojv/5nGBNF3GKYFUy51PEa52dbNAMilmsk24izASSHPLOCWQargL\nMRJIc8s4JZB83cMwFFNkrS2QKkogubrvz3isX1sg1ZRA8nSbtkbG2lY368iRgQVKIPm5rxRF\nPeJRIFWUQPJy3z7UCSSoGxlYoASSi/vjA8KRQJpZximBhHePDzBEPitVIFWUQAK7P8YYVa4N\ndbM2jgwsUAIJ6X6mqGZttJu1cWRggRJIOPfsOaMmOicrLZBCJkviXjhl1EDndKUFUshkGdzL\nJ17ZO2csLZBCJhvvXrt+gbtzztICKWSywe6Nq+mIO6ctLZBCJhvp3r4mlbVz5tICKWSyYe6Z\ng93Vavu7WRtHBhYogVToTqLIqXYVN2vjyMACJZCK3Om3GbF13kJpgRQy2frunLuMuDpvo7RA\nCplsZXfmvXpEnTdTWiCFTLamO/0jHb52ZTdr48jAAiWQMtz5GLF03lZpgRQy2VruAopgtQPc\nrI0jAwuUQEp1l1CEqh3hZm0cGVigBFKauxAjgs4bLC2QQiZbwV2MUXjnTZYWSCGTdXYbn03X\n7PtmbRwZWKAE0oZsGLX7vmkbRwYWKIG0KitGrb7v2NICKWSybu77daltPuJRIFWUQFrSx4dA\nYnQjAwtUAUiHT8297id+bhkAABJQSURBVAqk22e6wcZRa++bovQ+QDrc/nh8fewHpIc9o3af\nlSqQKkogTfR8gKGdzqFu1sYRqXeQQHrS9DhdK52D3ayNI1LvIBRIf50E6ypKV4qi+5Cakw2k\nvg42LJw0aqBzDzdr44jUO0gf7a5aPPdK37mPm7VxROodJJC+tHYFA3fnbm7WxhGpd5BAet+6\nEIi5c0c3a+OI1DtIIG1eT8fbuaubtXFE6h1kuLLhcOzgYEPCVamknXu7WRsHBR+tfV9rl3Rx\nN2Xn/m7WxpGBBWrPICXeI0HYeQ03a+PIwAK1X5CS7zSi67yOm7VxZGCB2i1I6TfssXVeyc3a\nODKwQO0UpJz7Xrk6r+ZmbRwZWKB2CVLe7eNMnVd0szaODCxQewQp8zEMRJ3XdLM2jgwsUPsD\nKftpJjSd13WzNo4MLFC7Ayn/oUAsnVd2szaODCxQewOp4OFaJJ3XdrM2jgwsUPsCqQAjks7r\nu1kbRwYWqF2BVMQRRecBbtbGkYEFakcgZR9lANZu0M3aODKwQO0HpEKMCDqPcbM2jgwsULsB\nqZij8M6D3KyNIwML1B5AGoahHCPeRDm7WRtHBhaoHYB0/YKjiNoNu1kbRwYWqP5BsnJEmyhn\nN2vjyMACtReQ9vh9EgKpogSSZ+2W3ayNIwMLVP8gWTmiTZSzm7VxZGCB6h6kL4z2+cUsAqmi\negfp6zADayao3ayNIwMLVOcgnQ/XsWaC2s3aODKwQPUN0uWwN2smqN2sjSMDC9QiSN+/lgzf\nfmf8spjJLut6+og1E9Ru1sbNkffREkhvwxmk4UfGL4uZ7KJup2FZM0HtZm3cnnkXLYF0GH6e\n/u/XkPPZL2ayS7pfzsCaCWo3a+P2zLtoiZMrQO2CNLosiDUT1G7Wxq2Jd9ISJ9+HH3+Oxz9v\nw2vGL4uZ7LzGl9exZoLazdo4IvUOWgLp92H40uFXxi+LmeysHi5TZc0EtZu1cUTqHbT4ye3P\n27dh+PaWc9COCKTHy71ZM0HtZm3cnnkX9Xke6em2CdZMULtZG0cGFqguQXq+/Yg1E9Ru1saR\ngQWqxxOyk9v4WDNB7WZt3Bx5H3V4QnZ6OyxrJqjdrI3bM++i/k7IztxWzpoJajdr4/bMu6i/\nE7Izj2dgzQS1m7Vxa+Kd1N0J2bnHnLBmgtrN2jgi9Q7q7YTs7POCWDNB7WZtHJF6B3V2Qnb+\nuVusmaB2szZuz7yL+jqPtPD8OtZMULtZG0cGFqiuQFp6DiRrJqjdrI0jAwvUIkhvw0UZvyxm\nsjctPk+VNRPUbtbGAaH30MoJ2dZAWn4uMWsmqN2sjSNS76DlE7K/Xofff17P52UTFTPZq5af\n782aCWo3a+OI1Dto5YTs38O/xz/tnEdaeU4+ayao3ayNI1LvoBWQ/h3+aejKhrUvnGDNBLWb\ntXFE6h20fGXD/34P344/WwFp9YtbWDNB7WZtHJF6By1xciLo9XSsoY2rv9e/AIk1E9Ru1sYR\nqXfQ4gbn32/H449heMv5ZTGTfd/iiDYT1G7Wxu2Zd1EXJ2S3vpCPNRPUbtbGkYEFCgpSkE4c\nRfcg7Vw9bJE2vyCW9R9Xajdr48jAAtUBSNtftMyaCWo3a+PIwALVPkgJ31jOmglqN2vjyMAC\n1TxICRzRZoLazdo4MrBA9QCSwW2r3bWbtXFkYIFqHaQUjmgzQe1mbRwZWKAaBynlgx1vJqjd\nrI0jAwtU2yClcUSbCWo3a+PIwALVNEiJHNFmgtrN2jgysEC1DpLBbavdvZu1cWRggWoZpFSO\naDNB7WZtHBlYoBoGKfWDHW8mqN2sjSMDC1S7IKVzRJsJajdr48jAAtU0SAa3rfYe3KyNIwML\nVLMgZXBEmwlqN2vjyMAC1QZIwzA8/U3GBzveTFC7WRtHBhaoJkAaPql5JCmLI9pMULtZG0cG\nFqgWQDpx9EhSHke0maB2szaODCxQzYKU8wtYM0HtZm0cGVigmgQpkyPaTFC7WRtHBhaoFkB6\n3kfK/GDHmwlqN2vjyMAC1QRIj0ftsjmizQS1m7VxZGCBagOksTufI9pMULtZG0cGFqjmQCrg\niDYT1G7WxpGBBao1kEo4os0EtZu1cWRggWoMpCKOaDNB7WZtHBlYoNoCqYwj2kxQu1kbRwYW\nqOZAMrhttffmZm0cGVigmgKpcINEmwlqN2vjyMAC1RJIpRzRZoLazdo4MrBANQaSwW2rvT83\na+PIwALVEEjFGyTaTFC7WRtHBhaodkAq54g2E9Ru1saRgQWqGZAMHNFmgtrN2jgysEC1BJLB\nbau9Szdr48jAAtUKSJYNEm0mqN2sjSMDC1QjIJk4os0EtZu1cWRggWoHJIObNRPUbtbGkYEF\nqg2QbBsk2kxQu1kbRwYWqCZAMnJEmwlqN2vjyMAC1QpIBjdvJqjdrI0jAwtUCyBZN0i0maB2\nszaODCxQDYB04oh1rXbsZm0cGVig2gDJtl6mTw7PEmuinN2sjSMDCxQ/SB9WkIaXlxcLSayJ\ncnazNo4MLFBNgGRaLyeOTCSxJsrZzdo4MrBA0YP0gQPpZU7bv4A1Uc5u1saRgQWKHaTLETsz\nSHl68O90D4u1cWRggWoApBJ3ATwLPO12D4u1cWRggSIH6SMbpMTtTMp/n2LfFGsemUsLJPRk\nb6diE90LCGTXBvLEmkfm0gIJPdnbJQ3b7rW0l65VxAaKNY/MpQUSeLIfKSAlBBxzFqqMJ9Y8\nMpcWSNjJjq6xm3Unxxp6XUQmT6x5ZC69E5AOnxq9vP3gAdKyO2fL4LJWE3FizSNz6X2AdLj9\nMf6LL2EnO9ogPW0VcvdVPDOxxRJrHplL7xOk0UvsZEccvdzP5WQyVFg7271IE2semUsLJORk\n7xuk20U+uVui4tqF7rmtE2semUvvEqTLq79OwrR01tddSGdNLvJB1gFrZscpuiWphlAgfQn5\nT9ToSMMDSJm/pqi22b214+RZG+XWFilLrCCNjjTYAxmVCTtMAmm6jFNWkMYcAUG6cgT5Zz0u\nE08fSqvWtroFUpZ4QXqgiHWtbuhymKQcJoE0XcYpTpA+OXrMHuta3dLsBUaVagukmjJc2TA5\nMwsD6WMSOta1uqnxqeQClgTSdBmnGK+1m4kb61rNdmeiJJCmyzjFB9Js0ljXaok7hyWBNF3G\nKTKQljLGulYL3cksCaTpMk5RgXSN1/QJxaxrtdiduLskkKbLOEUE0h2j6ZO+WdeqxZ3CkkCa\nLuMUC0j3TM0+MZ91rRrdWyhFPgqMdWjIwALFAdIoT/NfPcG6Vu3uNZZCHwXGOjRkYIFiAOkh\nSvNf4cK6ViHuJZZiH7bMOjRkYIEKB+kpRAvfhcS6VkHu+d0lgTS7jFPBID3nZ+k7xVjXKs79\nMoVJIM0u41QoSNN/hpe+m491rULdE5Yi95FoH3mODCxQcSDNfZhZ/JLLRlAwu19eHmiKSzPv\nI8+RgQUqCqT5/evFL4ttBwW7++XlCaeKta8i/lIpZGCBigFpISTL37rcFgpmN4glgVRPASAt\nxmPl28ubQwHhNtMkkOqpOkgruVjmqFUUAO6XZ9WprX2kTFUGaS0NKxukplEwuycspfKko3b1\nVBWk9QyscBQf5nj3LE6O146TvO3pMk7VBGl91a9tkGjXaoB7HqjZ2QqkeqoF0jBs/Qu6xhHt\nWo1zp/AkkOqpEkibGK1vkGjXarh7lSeBVE91QNrmaH2DRLtWydwZH/vQpau5kYEFqiZIa4eB\n1jdItGuV073GUzJrrG8bGVigSEDa4Ih2rVK7S3kq3IThGhdIi9o6v7fBUbth5nHX5UkgWbTy\n9tfP721tkHji2LUbyJNAsqh4slscNRXHTtxz35mWDpRAsqh0NpsbpHbj2K77+eLIPKwEkkWl\ns9nkqN04tut+Mufx5HilHjKwQDGAtL1BajeOkW5bmhdKJ/Hkee04MrBAkYBkmey29uk2pjmp\n9DxPrnczIQMLFAFICRukZsMc6LamOaf0LE8CqVhls0ngqNUwR7prgnRR8qEIS21kYIGKByll\ng9RqmCPdASBdZKdJIBXMJoWjVsMc6q6xj7RYezBtnQRS/mySNkjNhrm/o3YZ7tUDe8W1kYEF\nKhqkNI7aDXO7blTpEpwEUvZs0jhqN47tusGls1gSSLmzSdwgtRvHdt0+pdNoEki5k03kqN04\ntut2LL25cRJImZNN3SC1G8d23b6l11kSSJmTTeWo3Ti2665ReokmgZQ32eQNUrtxbNddrfTM\nxkkg5U02maN249iuu2bpZ5YEUtZk0zdI7caxXXf90iOUVs4kIwMLVDBIqTNuNo7tuoNKb5KE\nDCxQgSBlbJDajWO77rDSG1fbIgMLVCxIycNtNo7tugVSluJAytkgtRvHdt0CKUthIGVx1G4c\n23XHlV6//wMZWKAiQcqYbbNxbNcdWHr1/g9kYIGKAilvg9RuHNt1szaODCxQUJAy9MlRUGVJ\nclDQFilzg0T7z2PHbtbGkYEFKg4k2GTldnGzNo4MLFAxIOVukGjXasdu1saRgQUqDCTcZOV2\ncbM2jgwsUCEgZW+QaNdqx27WxpGBBSoKJOBk5XZxszaODCxQESDlb5Bo12rHbtbGkYEFKggk\n5GTldnGzNo4MLFABIBVskGjXasdu1saRgQUqBiToZOV2cbM2jgwsUPVBKtkg0a7Vjt2sjSMD\nC1QISNjJyu3iZm0cGVigqoNUtEGiXasdu1kbRwYWqAiQwJOV28XN2jgysEDVBqlsg0S7Vjt2\nszaODCxQASChJyu3i5u1cWRggaoMUuEGiXatduxmbRwZWKDqgwSfrNwubtbGkYEFqi5IpRsk\n2rXasZu1cWRggaoOEn6ycru4WRtHBhaoqiAVb5Bo12rHbtbGkYEFqjZIDpOV28XN2jgysEDV\nBKl8g0S7Vjt2szaODCxQlUHymKzcLm7WxpGBBaoiSIYNEu1a7djN2jgysEDVBcllsnK7uFkb\nRwYWqHogWTZItGu1Yzdr48jAAlUVJJ/Jyu3iZm0cGVigqoFk2iDRrtWO3ayNIwMLVE2QnCYr\nt4ubtXFkYIGqBZJtg0S7Vjt2szaODCxQFbdIXpOV28XN2jgysEAFfqs5arJyu7hZG0cGFiiB\nJDddaYEUMlm5XdysjSMDC5RAkpuutEAKmazcLm7WxpGBBUogyU1XWiAtaxgGr8nK7eJmbRwZ\nWKAqgTR8fHxYSGJdqx27WRtHBhaoOiCdODKRxLpWO3azNo4MLFACSW660gJpSQKpPTdr48jA\nAqV9JLnpSgukZemoXWtu1saRgQVK55HkpistkEImK7eLm7VxZGCBEkhy05UWSCGTldvFzdo4\nMrBACSS56UoLpJDJyu3iZm0cGVigBJLcdKUFUshk5XZxszaODCxQAkluutI7AenwqbnXAqkr\nN2vjiNQ7KB+kw+2Px9dHgdSVm7VxROodJJDkpiu9U5Duipms3C5u1sbNkfeRFaTrPtJfJ+Ha\nkqS2ZATpcJxsllwUyehOa+/0bZcKuo/kp72uVYHUigSSatOVFkhu2utaFUitqBGQJIlbhisb\nDqPXkrRvQa+1k6S9SiBJEkACSZIAEkiSBFATIAUf0girHfi+40ofgusXqgWQgg+yx4X59seO\nSt+PCLd1XkUgbVcXSDULCyRfBaY5FKTA0tEMCyQP7RKk2B0VgZShVkCK3CrE7fDHFQ8+2CCQ\nvBS+VvdVnKK0QMIr8igwwb/N+ywtkOCKHam2SDGlBRJawRMVSDGlBRJYkR+vvuqHFdaVDc2o\nBZAkiV4CSZIAEkiSBJBAkiSABJIkASSQJAkggSRJAAkkSQJIIEkSQAJJkgASSFQatD4alVYc\nlQRSq9KKo5JAalVacZX058cw/Phz/GLl+/D6+/R3v09/d371fTi8nRe+nV9JbUkgVdJh+NS3\n44mVT3yGwydTf77+7v7q+2nh99MrkdScBFId/X2C423458TK65/j6/nH1+P11Y/jz9PHuq+F\nfw9t3YojHQVSLX37GvR5o/Pr86PcaeP0bfh9e/Xn/J8Np7/SnlKD0iqro+GiKyVzr47jv5La\nklZZHQmkzqVVVkffboMezh/oXhc+2t3/lFqSVlkdvZ2OKfzvhM/w+cef1+Hvx4MNb8dfjxsp\nqS1pldXR+QD36TjDJ0ing97H8eHv39eD4wKpVWmVVdLp5Ovrz+PXR7vX62nY2wnZX6/nVwKp\nVWmV1ZYo6VJaq7UlkLqU1mptCaQupbVaWwKpS2mtShJAAkmSABJIkgSQQJIkgASSJAEkkCQJ\nIIEkSQAJJEkC6P9TrlD1+yYwcQAAAABJRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(history)\n",
    "history$metrics$val_acc %>% max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ooops...what's going on? It arrives at a 71% accuracy and then it deteriorates. why? The *simple recurring network* are too much simple: although it should theoretically be able to retain at time **T** all the information associated with data seen many timesteps before, in practice, such long-term dependencies are impossible to learn due to a mathematical problem call the *vanishing (or exploding) gradient problem*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Long Short Term Memory (LSTM)\n",
    "The LSTM network is a way to overcome the difficulties caused by the *vanishing gradient problem* in the training of the simple *RNN* network\n",
    "\n",
    "<img src=\"fig/lstm_unrolled.png\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple enough, w.r.t. to the simple RNN, **LSTM** adds a **Carry Track** that should help to transfer the information from the previous steps to the current steps. The information carried in this track is then used in the *output* activation function.\n",
    "\n",
    "The complexity is hidden in the way in which the *carry* is computed. It involves three distinct transformations similar to the one computed to generate  the output in the simple RNN network.\n",
    "\n",
    "*y = activation(dot(state_t, U) + dot(input_t, W) + b)*\n",
    "\n",
    "But all three transformations have their own weight matrices *U*, *W* and *b*. The results of this transformation are used, along with the carry at time **T - 1**, to compute carry at time **T**. Abstractly these three transformations should mimic our *Long* and *Short* memory: but we are not neither *Daniel Kahneman* neither *Jürgen Schmidhuber*! \n",
    "\n",
    "So let's see how **LSTM** works on the IMDB problem using the **layer_lstm_network**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_embedding(input_dim = max_features, output_dim = 8) %>%\n",
    "    layer_lstm(units = 8) %>%\n",
    "    layer_dense(units = 1, activation = \"sigmoid\")\n",
    "    model %>% compile(\n",
    "    optimizer = \"rmsprop\",\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"acc\")\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    mtx_train, y_train,\n",
    "    epochs = 10,\n",
    "    batch_size = 32,\n",
    "    validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.6848</li>\n",
       "\t<li>0.7072</li>\n",
       "\t<li>0.7172</li>\n",
       "\t<li>0.7158</li>\n",
       "\t<li>0.7144</li>\n",
       "\t<li>0.7182</li>\n",
       "\t<li>0.7158</li>\n",
       "\t<li>0.712</li>\n",
       "\t<li>0.7102</li>\n",
       "\t<li>0.7088</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.6848\n",
       "\\item 0.7072\n",
       "\\item 0.7172\n",
       "\\item 0.7158\n",
       "\\item 0.7144\n",
       "\\item 0.7182\n",
       "\\item 0.7158\n",
       "\\item 0.712\n",
       "\\item 0.7102\n",
       "\\item 0.7088\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.6848\n",
       "2. 0.7072\n",
       "3. 0.7172\n",
       "4. 0.7158\n",
       "5. 0.7144\n",
       "6. 0.7182\n",
       "7. 0.7158\n",
       "8. 0.712\n",
       "9. 0.7102\n",
       "10. 0.7088\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] 0.6848 0.7072 0.7172 0.7158 0.7144 0.7182 0.7158 0.7120 0.7102 0.7088"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2dC3faSBaElSFOPLOTxB74/791QWAQ\nQo9+VKmrW1XnbBabKd/r2/W5hRDQnSzLylZXugHLakEGybIAMkiWBZBBsiyADJJlAWSQLAsg\ng2RZABkkywKIAtLHghbvXJPNW5o1+2YEFiCDZLNgaYPUq8R8bYabNftmBBYgg2SzYGmD1KvE\nfG2GmzX7ZgQWIINks2Bpg9SrxHxthps1+2YEFiCDZLNgaYPUq8R8bYabNftmBBYgg2SzYGmD\n1KvEfG2GmzX7ZgQWIINks2Bpg9SrxHxthps1+2YEFiCDZLNgaYPUa2FCXddx5msz3KzZNyOw\nAG0MUnc8HjNIklzZZs2afTMCC9C2IF04yiFJcmWbNWv2zQgsQAbJZsHSBqnX7BAMUk1mzb4Z\ngQXIj5FsFixtkHotTMhn7eoxa/bNCCxAfhchywLIT8jaLFjaO1KvEvO1GW7W7JsRWIAMks2C\npQ1SrxLztRlu1uybEViADJLNgqUNUq8S87UZbtbsmxFYgDYH6XjkzNdmuFmzb0ZgAdoapMsl\nQpT52gw3a/bNCCxABXYkg1SJWbNvRmABKgFSOkmSK9usWbNvRmAB2v5kg0GqxazZNyOwABkk\nmwVLG6ReiyPKObaTXNlmzZp9MwILUIHnkQxSJWbNvhmBBagMSKkkSa5ss2bNvhmBBajElQ0G\nqQ6zZt+MwAJUCKREkiRXtlmzZt+MwAJU5Fo7g1SFWbNvRmABMkg2C5Y2SL3WRpR8bCe5ss2a\nNftmBBagMi+jMEg1mDX7ZgQWoGIgJZEkubLNmjX7ZgQWoEIv7DNIFZg1+2YEFqByIKWQJLmy\nzZo1+2YEFqBSLzU3SPpmzb4ZgQWoIEgJJEmubLNmzb4ZgQWo2JufGCR5s2bfjMACZJBsFixt\nkHqFjCjp2E5yZZs1a/bNCCxA5d7XziCpmzX7ZgQWoKIgRZMkubLNmjX7ZgQWoILvtGqQxM2a\nfTMCC1BZkGJJklzZZs2afTMCC1DJ9/42SNpmzb4ZgQXIINksWNog9QocUfyxneTKNmvW7JsR\nWICKfqyLQZI2a/bNCCxApUGKI0lyZZs1a/bNCCxAZT9ozCApmzX7ZgQWoOIgRZEkubLNmjX7\nZgQWoMIffWmQhM2afTMCC1B5kGJIklzZZs2afTMCC1DpD2M2SLpmzb4ZgQVIAKQIkiRXtlmz\nZt+MwAJUGqS4LUlyZZs1a/bNCCxABslmwdIGqVfMiKKO7SRXtlmzZt+MwAJUHKSoLUlyZZs1\na/bNCCxAEiAFkyS5ss2aNftmBBag8iDFbEmSK9usWbNvRmAB0gAplCTJlW3WrNk3I7AACYAU\nsSVJrmyzZs2+GYEFSASkQJIkV7ZZs2bfjMACpABS+JYkubLNmjX7ZgQWIBWQwkiSXNlmzZp9\nMwILkARIwVuS5Mo2a9bsmxFYgGRACiJJcmWbNWv2zQgsQBoghW5JkivbrFmzb0ZgAdIBKYQk\nyZVt1qzZNyOwAImAFLglSa5ss2bNvhmBBcgg2SxY2iD1ShhR2LGd5Mo2a9bsmxFYgFRACtuS\nJFe2WbNm34zAApQB0uGsqdvJIK2TJLmyzZrLle66bv7naiodpMP9n+fbpzSQgrakShNZqblY\n6e7bt2+zJOWknSgpkFZJqjSRlZpLlb5wNE9STtqJAoL0UNp8DZKY2SBFCAXS/THSX2el/cDL\nlpTcjdWObiCVbiNOIJAOp+xDu5AtqdI/7ZWa/RgpQjqPkUIeJVWayErNPmsXISWQ1rekShNZ\nqblU6fN+tKfnkTggLZNUaSIrNW9f+tuXDFKv5PkaJCXzxqW/DbQnkO5XMxwGt69Knu/allRp\nIis1b1f625OWzZmBZ0nmWrurDJKQeYvS314YWjMzAguQHkhLJFWayErN9NIzFC2bGYEFSAyk\nlS2p0kRWamaWnkVo1cwILECCIC2QVGkiKzVzSs9vQ2GVGYEFSA2k5S2p0kRWal56WjSxdBBF\nc+bbXZraGqTVxVnckipNZJ3mxQt14ksHIjRtHt6lqY1BClgcg6RhXr4GO6L0t9BtaMo8cZem\ntgUpZHGWtqQqE1mpGQDSGKFAij4M0k2zQwhaHIMkYc4FKRGhqwxSr9khhII0R1KViazVnP4Y\nKYOgmwxSr/kpBC2OQdIwx5+1y9qFhjJIvRYmFLI481tSpYms1Bzhnng0lE7RcmVGYAHa/nmk\ngAEbJAlzkHsWIVbfjMACtDlIIX+rZrekShNZqXndvbQJGaR8rc5+bY4GScG86F49jDNI+Voc\nUcgB9NyWVGkiKzUHXeUzu5IGKV8rIwpAaYakShNZqXnkjjyhYJDytTaigNM6Bqm8+cs9RdD6\nKSODlK/1Ea0uxvSWVGkiKzUnEoQobZB6hYxobVUMUkFzFkF5pVfNjMACVPD1SMvLM7klVZfI\nesyT7CQRdBXhxUy3uzRV8oV9y+tkkMjmBXRuyrzkFfhipuFdmir7Ctmlv3lTW5JmIqsxr6Pz\nvB7ppWEvZpq4S1OlX2q+gJJByjcHozP5t8wghas0SAsoTWxJlcZZ5nzBKjqQ0h8GCaTYEc2T\nFD7fde3InIxOfumb/BgJofgRTS7065ZUV5yLmJ/R4V1rtyaftQMoZURTKBmkGPPU9lMOJD+P\nBFDSiCYOP162JMmVlTDPHMQZpK2kA9LUaYcxSZIryzWvHyItPQ4ySFtJCaQJlHYP0uKD9tVz\nCXkPVDQnxggsQFogvaA02pIkV5ZpXjiNHHBGLvPUmebEGIEFSA2kjxFLBmkKhlWElszhkpwY\nI7AACYL0hNLzliS5skzzFAshDM2aoyQ5MUZgAZIEaYjSvkEaHZ2F7URDr0HaSKIgfTxCMyRJ\ncmW55uv5gtEDosBrFPwYaTvpgjRAKcWcVVnHPCYonKKLfNZuMymD9Ho6XHJlOeYpguJfYufn\nkbaSNkgfI5YkVxZsxhCUUhnrNkj5ws53kCbJlUWZZwgq17bmxBiBBagCkD6O+X+ZEytvY17e\ngwzS6C5NVQHSWQCWhGIxic7M72eQRndpqgaQbs/K5qKkEIsYgrCVN3cbpHzB5/v1XFIWSkVj\nkUAQqHIpt0HKF3y+XxcKnXJYKhOLBHRAlbPNBilCdYD08QAp/RBv61jkEpReGWQ2SBGqB6Tj\nw5yWzM1iMUUQ5Q11yGaDFKFKQPp4AukjiSV+LCb3IOI7U5HNBilCFYF0HJljj5mIsVh6IGSQ\nsGZGYAGqBaR+S3q9BjPmIUjOys5d/blE0N2bxZFBGt+lqZpAOk4mMpiljJV9ZSGEoLub9RZv\nZLNBilA1IM2T1Csg0ekrOzw6iyEov3JRs2bfjMACVA9IH4sg9VrMd/K+MIlOGEE3SQZSurRB\n6sWZ73VLWqFhNu0Jj1TyCbpJMpDSpQ1SL9J8LyCFsTAX/7j3Wswn6CbJQEqXNki9SPOd/oDm\nBQWQEIhOrecLDNJWqgmkeJKuCoBlFTnJTJHNmn0zAgtQVSBNf9R5mLpYdJ4lmSmyWbNvRmAB\nqg6kdJK6qfPXoW7JTJHNmn0zAgtQXSCdMkDSjIWyWbNvRmABqg+kZJIkY6Fs1uybEViAKCAR\ndQapdAuW9arKdqScR0mSf1+VzZp9MwILUG0gZZy4k4yFslmzb0ZgAaoRpESSJGOhbNbsmxFY\ngKoDKZ0kyVgomzX7ZgQWoPpASj64k4yFslmzb0ZgAaoTpCSSJGOhbNbsmxFYgCoEKXVLkoyF\nslmzb0ZgAaoUpBSSJGOhbNbsmxFYgGoEKXFLkoyFslmzb0ZgAaoVpASSJGOhbNbsmxFYgKoE\nKY0kyVgomzX7ZgQWoDpBSjq4k4yFslmzb0ZgAaoXpGiSJGOhbNbsmxFYgCoFKWVLkoyFslmz\nb0ZgAaoKpMFbkCRsSZKxUDZr9s0ILEA1gdSd4ckgSTIWymbNvhmBBagikC4cPZMU55eMhbJZ\ns29GYAGqGqQ4kiRjoWzW7JsRWIBmQfrncDr96g5/p/xQznxHIEWTJBkLZbNm3+lZp2oOpH+6\n7vTncH50n0ISab5Pj5E+og/uJGOhbNbsOyftRM2B9L37df7fP7+7Q8IPZc139MbBkVuSZCyU\nzZp956SdqDmQzhvSv933/v/jtdV840iSjIWyWbPvnLQTNcfJofvzs/t9eZSU8EM3m69BYpo1\n+85JO1FzIP19Pow6XDak94Qfutl8o7YkyVgomzX7zkk7UbNHbu/d4d/zxpTC0XYgRZEkGQtl\ns2bf6VmnqqLnkSbNESRJxkLZrNk3I7AA1Q5SxMMkyVgomzX7ZgQWoIqekJ02h29JkrFQNmv2\nnZ51qmp6QnbaHEySZCyUzZp956SdqKqekJ02GySSWbPvnLQTVfUTsleFbkmSsVA2a/adk3ai\n6n5C9qpAkiRjoWzW7Dsn7UTV/YTsTQaJYtbsOyftRFX+hOxVYVuSZCyUzZp9p2edquqfR+oV\nRJJkLJTNmn0zAgtQGyAFkSQZC2WzZt+MwAI0C9Ln+/eu+/7+mfJDC8zXIOHNmn2nZ52qOZD6\nJ2MvJxz+JPzQAvMN2JIkY6Fs1uw7J+1EzYH0s3s7I/TnrfuZ8ENLzHedJMlYKJs1+85JO1EL\nT8g+/X+USsx3/eBOMhbKZs2+U5NOVlMgLZMkGQtls2bfqUknq5lDu3WSJGOhbNbsOyftRLVy\nsuGiFZIkY6Fs1uw7J+1EtXL6u5dBgpo1+07POlWNPCF70yJJkrFQNmv2zQgsQM2BNE+SZCyU\nzZp9MwIL0CRI3VAJP7TEfK9aIkkyFspmzb4zA89SYyAtHdxJxkLZrNl3ZuBZauvQ7mOJJMlY\nKJs1+2YEFqAWQZohSTIWymbNvhmBBag5kOZJkoyFslmzb0ZgAWoPpFmSJGOhbNbsmxFYgBoE\nae5hkmQslM2afTMCC1CLIM2QJBkLZbNm34zAAtQqSBMkScZC2azZNyOwADUJ0jRJkrFQNmv2\nzQgsQG2CNHlwJxkLZbNm34zAAtQoSFMkScZC2azZNyOwADUM0pgkyVgomzX7ZgQWoFZBmiBJ\nMhbKZs2+GYEFqFmQXkmSjIWyWbNvRmABahekF5IkY6Fs1uybEViAGgZpTJJkLJTNmn0zAgtQ\nBkiHs4a3H5+kVGK+U3omSTIWymbNvrPizlM6SIf7P4P/v6rEfCdlkHLMmn0nB5artkF6Ikky\nFspmzb6TA8sVBqTR52OWmO+0hgd3krFQNmv2nRxYrkAgPR4i/XVWdlc4XUgq3YO1AwF3JL2T\nDRc99iTJv6/KZs2+c9JOFOgx0vPtEvOd1Z0kyVgomzX7Tg4sV+2DdCdJMhbKZs2+kwPLVfOH\ndh93kiRjoWzW7Dsn7UThQBpsTiXmuySDVFvpPYF0v7LhMLh9VYn5LqonSTIWymbNvjMDz1LL\n19o91B/cScZC2azZNyOwAO0DpJ6kritRuWKzZt+MwAK0E5B6ko4ZJElmimzW7JsRWID2AlKX\nSZJkpshmzb4ZgQXIILEr12vW7JsRWID2BdLip56TKtdr1uybEViA9gLSF0mpKElmimzW7JsR\nWIB2A9JH13UZJElmimzW7JsRWID2A1JvTiZJMlNks2bfjMACtDOQkkmSzBTZrNk3I7AA7Q2k\nVJJKt13CrNk3I7AA7Q6kj7QHSsXbLmDW7JsRWID2B1IaSeXb3t6s2TcjsADtEKQkkgTa3tys\n2TcjsADtEaQUlCTa3tis2TcjsADtE6R4kjTa3tas2TcjsADtFKSP2MscRNre1KzZNyOwAO0V\npFiUZNre0KzZNyOwAO0XpDiSdNrezqzZNyOwAO0YpCiUlNreyqzZNyOwAO0apAiUtNrexqzZ\nNyOwAO0cpI/QF1eItb2JWbNvRmAB2jtIoZuSXNsbmDX7ZgQWIIMUhpJg23SzZt+MwAJkkC5a\nR0mybbJZs29GYAEySL1WHyppts01a/bNCCxABumm4zJLqm0zzZp9MwILkEG6axEl3bZ5Zs2+\nGYEFyCANtICSctsss2bfjMACZJCeNceSeNsUs2bfjMACZJDGmkZJvm2CWbNvRmABMkivmkKp\ngrbhZs2+GYEFyCBN6oWlOtrGmjX7ZgQWIIM0oxFKtbSNNGv2zQgsQAZpVkOUuj1+Splm34zA\nAmSQlvTF0uVdw/f3KWWafTMCC5BBWtbxod19uJJm34zAAmSQ1mSQtMyMwAJkkAK0ch0esXJR\ns2bfjMACZJBC1OWhVOXvXLS0QepVYr5cc5eFUp2/s2jfjMACZJDCzcksVfo7a/bNCCxABinG\nnIhS6bbrK22QepWY70bmJJTKt11baYPUq8R8tzPHsyTRdlWlDVKvEvPd0nyMZEmk7YpKG6Re\nJea7rfkYxZJM29WUNki9Ssx3c/MxHCaltusobZB6lZhvCXMoS2JtV1DaIPUqMd9C5mMITHpt\nq5c2SL1KzLegeZUlzbaVSxukXiXmW9a8vDHJti1b2iD1KjHf8uZ5lqTblixtkHqVmK+EeWZj\nUm9br7RB6lVivjLmCZZqaFurtEHqVWK+SuYxS5W0LVTaIPUqMV8184ClWt+CSHPcjMACZJB4\n5sHGVOVbEGmOmxFYgAwS1QwgySCN7tKUQWKbYy8Vx1XONWuOmxFYgCggWQN1gyO80r1YNHlH\nopuHJFX1hg+a42YEFiCDxDdfz9qlwmSQRndpyiBtaE5iySCN7tKUQdrWHH+YZ5BGd2nKIG1v\nPh5jcDJIo7s0ZZCKmCNYMkijuzRlkAqag2AySKO7NGWQSpvXWDJIo7s0ZZAEzIs7k0Ea3aUp\ngyRinn3UZJBGd2nKIOmYJ1kq+BoMzYkxAguQQVIzP8NU8mOgNSfGCCxABknS/Lw3FXoNhubE\nGIEFyCCpmqMvgYBVxrgNUr5KzLdJczZLBmkrGSRtc8nXYGhOjBFYgAySuPly1i7igiJgZdGJ\nMQILkEGqw5zIkkHaSgapJnM0TAZpKxmk6swxLBmkrWSQajQH70wGaSsZpFrNQY+aDNJWMkgV\nm9dZMkhbySBVb16CySBtJYPUhnmGJYO0lQxSM+aJnSnvNRiavzQjsAAZpKbMxxfV+YkyBqlX\nifnafBOQJMlfmhFYgAxSm+boayBwpclmRmABMkjtmrtcliR/aUZgATJIDZufX4MRT5PkL80I\nLEAGqWXzy+dgxOEk+UszAguQQdqFOY2l8n1P3aUpg7QjcyxMKn0/36Upg7Q3cwRLUn3f79KU\nQdqhOXRnUuv7epemDNJOzSGPmhT7Nkjk+docbV5lSbJvRmABMkh7Ny+wJNk3I7AAGSSbP4Y0\nbV461swILEAGyearJnYmyb4ZgQXIINl81+hBU8FPlDFIvUrM12aIeczShqUDzYzAAmSQbH5V\nyvVEoNKrZkZgATJINk8r90UYBilfJeZrM9g8fg1GLE0GKV8l5msz2nz71M1UnAxSvkrM12a4\n+X7WLoklg5SvEvO1GW4euSNhMkj5KjFfm+HmKXc4SwYpXyXmazPcPOMO3JkMUr4SR5Q1X5vh\n5gV3wKMmg5SvxBFlzddmuHnZvcaSQcpX4oiy5msz3BzinofJIOUrcURZ87UZbg52T7FEu+KV\nEViADJLNCPd4Z7o9m0uozAgsQAbJZpT7OFY6SXkgdU+p/ufAyPhLTcYPTRxR1nxthpsT3EnX\nQMRVDgl1N/8VSwbJZrg7HyaD1CtxRFnztRluznF3eSwlg/TnrftxRefXj647vF84evqSJYNk\nM8V9OWuXvjOlgvR5ONf9cSHn367X+w2k+5csGSSbqaXTHjWlgvTevZ0+3y7kfO/+dzr9vtzq\nN6THlyQZJJvZpRNYSgXpe/fnfHh35eXPv3+/3UF6fElSxk8+nDX88nEzcURZ87UZbsaWjoIp\nFaQrKf2/b9eDua/v3b8kKf0nH+7/XL80SK2ZCaVDWcoH6Wf3/Z9//9xBenxJEgqkg3ek5syc\n0kE7U/6hXc/M5/3W40uSQCAdfGjXnplXevVRUypIf3dvn6fbI6Nft9MON5C+viQJDtJfZ+U2\nZbWvEUugn/o4/f3ePR4jHYZfkoQB6XDyjtSeeYPScztT+hOyP76ekP3ZdW+/Lrf+uYD0+JIk\nCEij8w4GqQ3zVqVfD/SWXoORk3aiMCBddb8LM1+bC5s3LH180SxJmYFnCXb62ztSc+bNSweR\nlBxYrgySzVqldwfS/cqGwQmHmxjztXlzc6nS3c5AmhdnvjZvbC5WevF16ozAAmSQbNYrvauz\ndgtizdfmTc2afTMCC5BBslmwtEHqVWK+NsPNmn0zAguQQbJZsLRB6lVivjbDzZp9Lybvv3kx\ngj6QQbJZsLRB6lVivjbDzZp9LybPIGXP12a4WbPvxeQZpOz52gw3a/a9mDyDlD1fm+Fmzb4X\nk2eQsudrM9ys2fdi8gxS9nxthps1+15MnkHKnq/NcLNm34vJM0jZ87UZbtbsezF5iSB1kzej\ntMlnx1jWNhqS03XdDEiM0BskqyENObq8OtAgWVaC/vvv9f2Ieg1B6t8osjtd3zzy+j6sp/7r\n0c3T9b8Lk0GyGlIQSKf7m4F3t6+6x9fPN8M/N9MgWQ0pAqThV90zU4+b4YAYJKshhT9GugX/\n9vbgBsmyBgo/a9cNbxkkyxoq7Hmk4cMjg2RZL0oAafnQruzJhhJPeNsMN2v2vZi8QJDup7Vv\nt6ZBKn/6u8R8bYabNfteTF4YSDEySDbnmjX7XkweFKTyj5FKzNdmuFmz78XkYXekmM/KNEg2\nC5bWAClGBslmwdIGqVeJ+doMN2v2vZg8g5Q9X5vhZs2+F5NnkLLnazPcrNn3YvIMUvZ8bYab\nNftmBBYgXyJkNSTvSNl/qGyGmzX7XkyeQcqer81ws2bfi8kzSNnztRlu1ux7MXkGKXu+NsPN\n5Uonf6q5QQqTzVuai5XuXyU++3OXZJDCZPOW5lKlr29XMkfSYvIMUphs3tK8eemn9/0xSAap\nEfN2pafeQMsgGaRGzFuUnkJo68dI3QOD7uWeUBkkm4uVfiHoS5CzdvNvxzXS9GdRRJJhkGwu\nUXqGoADzYvKGHH379m3mDSJHMkg2U82U0rO7UGjlxeT999+3aT29if7Xv8O30H+8Z9DtzVef\nv3y8s/6MDJLNFPfL0dnko6GEyovJiwLp+R24Bu9x9/h68OXgfY6nZJBsZrgH5wumCJpHaLXy\nYvKCQHp6E/0nkEbfjXnbVYNkM8HdTdITQNB65cXkhT1GerzT6uAt9AfvYfz1XYNkM8Kc7M4g\naL3yYvICz9o9DusmdqThwyaDZHO+OcGdTdB65cXkBT6P1D29/ffrO34bJJuB5lj3mKD554Jy\nKi8mL/QJ2cHbe0+dbPChnc1Ac4T7dQ9aek41q/Ji8qJAGr2F/uB89+3c+PPp75NBspnmnj+O\nY/W9mLxQkPAySDYnuVcfDRmkfCWOKGu+NsPN0+7QU3IGKV+JI8qar81w88gdeVLbIOUrcURZ\n87UZbv5yRxKEKG2QepWYr81w8/TlCZuUNki9SszXZqA5i6C80qvmxeQZpOz52gwxAwhKLR1o\nXkyeQcqer8355gmI0p9VLQJSORkkm3u97EGdQYqRQdq7efY4bvEtSCClk8yMwAJkkHZrXn00\nRLtcLsvMCCxABml/5uBzcmJ93+7SlEFq2TzaU2LPakv+0ozAAmSQGjbfH+XEEpRf2iCt63DW\n+PbwewZJxDz3xgkblCaa8zNPUTxIh/s/j9vD750MkoI5i6C80lxzfuYpMkgtmnMJyihNN+dn\nniKD1Jh5zJDiGewsc37mKYKD9NdZmNasKI0Junxv+V12LaDyQPo6yeAdqaB56dFQVmXNXzo/\n8xT50K5ec8ApOYO0lQxShebwk9oGaSsZpJrM4QQhKov80uO7NGWQxM23q3xiCQJUFp1YfuYp\nyriy4TC87SsbOOb+Kp8EgvIr57oNUr5KzLdNc5dKUHblbLdByleJ+bZmTt+FciuD3AYpXyXm\n24w58dEQoDLWbZDyVWK+9ZsnT8mVfL235sQYgQXIIJU3L57ULvh6b82JMQILkEEqaF4kiFp5\nA7dByleJ+VZlDiKIUnk7t0HKV4n5VmKOP5VgkEZ3acogbWhOOhlnkEZ3acog8c2X8wXpp7MN\n0uguTRkktjllF8JUzjVrjpsRWIAMEtWcS1F6ZYBZc9yMwAJkkGjmJ4T03oxeurRB6lVivkrm\niTNzBgllZgQWIIOENr8czIl+qoN0aYPUq8R8FcwzD4g0P9VBurRB6lVivmXNyycVZNuWLW2Q\nepWYb0Hz6qk5zbaVSxukXiXmW8YcdnZbrm350gapV4n5bm2OeYZIqO1KShukXiXmu6k58nlW\nlbbrKW2QepWY73bm+EsVJNquqrRB6lVivlzz/Qx2LEPZlQ3S+C5NGaQQfT2nmkJRXmWD9HKX\npgxSgIq+u5xBGt2lKYMUoC6HoqzKRc2afTMCC5BBWlcmRlX+zoVLG6ReJebLMmdTlFy5uFmz\nb0ZgATJICxo+MKrzulODtJUM0pyet6Jq2gaaNftmBBYggzShiXN0NbSNNmv2zQgsQAZpqNGl\nP4NHRdJtk8yafTMCC5BBumqM0PjcgmjbVLNm34zAAkQBqTaNESrdj1WfvCOFXbMg1/YGZs2+\nGYEFaOcgBT9JpNX2NmbNvhmBBWjXIEU81arU9lZmzb4ZgQVoxyBFXbCg0/Z2Zs2+GYEFaLcg\nRV72o9L2lmbNvhmBBWivIMVePSfS9qZmzb4ZgQVonyDFX4Qq0fbGZs2+GYEFaI8gpVzLLdD2\n5mbNvhmBBWiHICW9JqJ829ubNftmBBag/YGUgpFA2wXMmn0zAgvQ7kBKwqh82yXMmn0zAgvQ\n3kBK5Kh020XMmn0zAgvQzkBK5UgzU2SzZt+MwAK0K5DSHh4hKldp1uybEViAdgdSmcpVmjX7\nZgQWoD2BlI6RaKbIZs2+GYEFaEcgZexHopkimzX7ZgQWoP2AlMWRZqbIZs2+GYEFaDcg5XGk\nmSmyWbNvRmAB2gtIF+XMOc4AABODSURBVI52+B6PBmkr7QSk6/V1GSRJZops1uybEViA9gHS\n14th00mSzBTZrNk3I7AAGSRu5ZrNmn0zAgvQLkC6IGSQaiptkHqVmO+C+hN2nR8jVVTaIPUq\nMd953U583z9PebvKlZs1+2YEFqD2Qbo/gSQZC2WzZt+MwALUPEiPJ2IlY6Fs1uybEViAWgdp\ncEGDZCyUzZp9MwILUOMgDS8MkoyFslmzb0ZgAWofpEKVGzBr9s0ILEBtg3Tc+4fuGaSt1DRI\nR4OUY9bsmxFYgFoGafTKCclYKJs1+2YEFqAEkA5nDW72XxwG31QCqVDlNsyafUNij1c8SIf7\nP4NvHJ7+kxLzfdX4lXySsVA2a/adk3ai8kF6AUsEpJeXxErGQtms2XdO2onCgPTMkQRIry8t\nl4yFslmz75y0E5UN0vXLx0Okv85CNJapC0ele7B2JBBIz98r8YdqpIm3OpH8+6ps1uw7M/As\nYUAa3Sox32dNvWWQZCyUzZp956SdqFyQxmcdepWY75Mm33tLMhbKZs2+c9JOFAYksUO76few\nk4yFslmz78zAs4QDabA3lZjvUNPvBSkZC2WzZt+ZgWcp48qGIVFPFzaUBmnmPVUlY6Fs1uw7\nP/MUNXit3dybE0vGQtms2TcjsAC1B9Lsm3xLxkLZrNk3I7AANQfS/JvlS8ZC2azZNyOwALUI\nUqHKzZk1+2YEFqDWQFr49BbJWCibNftmBBagxkBa+hQkyVgomzX7ZgQWoPZAKlS5QbNm34zA\nAtQWSIsfyycZC2WzZt+MwALUFEjLH28pGQtls2bfjMAC1BJIKx8TKxkLZbNm34zAAtQYSIUq\nt2nW7JsRWIAaAmntc8slY6Fs1uybEViA2gFpjSPNWCibNftmBBagpkBKN2dVbtWs2TcjsAA1\nA9LqhqQZC2WzZt+MwALUCkjrHGnGQtms2TcjsAA1BFK6Oatyu2bNvhmBBagRkAI2JM1YKJs1\n+2YEFqA2QArhSDMWymbNvhmBBagqkLqum74jhCPNWCibNftmBBagmkDqzrxMkhS0IWnGQtms\n2TcjsABVBNKFo2mSgjjSjIWyWbNvRmABagGksA1JMxbKZs2+GYEFqAGQAjnSjIWyWbNvRmAB\nqgikucdIgRxpxkLZrNk3I7AA1QTS9Fm70A1JMxbKZs2+GYEFqCqQpszBHGnGQtms2TcjsAC1\nAFK6Oaty82bNvhmBBah2kMI3JM1YKJs1+2YEFqDKQYrgSDMWymbNvhmBBah+kNLNWZV3YNbs\nmxFYgOoGKWZD0oyFslmzb0ZgAaoepHRzVuU9mDX7ZgQWIApIW+myIZXuwbIuqnlHijqwE/37\nqmzW7JsRWIAqByndnFV5H2bNvhmBBahikCI3JM1YKJs1+2YEFqC6QUo3Z1XeiVmzb0ZgAaoX\npNgNSTMWymbNvhmBBahakKI50oyFslmzb0ZgAaoZpHRzVuVtzbNv+EKvLDoxRmABqhWk+A1J\nMxZr6r59+5ZBkkHaShWDFOkt+Kc93XzhKIckg7SVZkH60d/Tff+T8EP5843fkEr+aTdIQHNy\n1LmaA+m9u4LU/Uz4ofT5pnGUkcgNt7Nv6wr+WXltG6QIzYF06H5d/u93l3LsR59vwoHdDEgB\nsc1P9tzvjCk+31HmLmyQIjTHyRdAkiAlnGl4gERMr7oSx50mg9TrR/fz83T6fO/eEn4oe77x\nHH2RVIECBsYpNDvuNBmkXn8OXa/D74QfSp5v9IaUlaaPgAdYtCh/pA4M0oRBCtfskdvn+/eu\n+/6ectJuA5AiPEnpHSnnwUbBE2/jtmOpMkjhqvB5pPAN6Tkkha4vKHkGe7Htdawkn3ljBBag\nOkFa/Q+n/tDu8eKEAPM6T0ub1vZ9MwILUIVPyK6CNLPye7xcLtYcS9UCVgapl+4TsstHdgur\nLHnEr2tOOc05tLP+eqRnnar6npBd4Gj5L2WVcS4NEuApbHDf6VmnqronZOc2pPWlqzLOJY9I\nJzkaiYPV0naWmnSyqntCdpKjoIWqM84FzfFHZ9FYTS/aIsI5aSeqtidkXzek4D93lca5nBl0\n1Xo0VstPGOSknaiqnpA9gz3iKBSi3Mq7NFPO+YfjNNeUpmp6HumyOAOQgrei/Mr7NG93zt8g\nTWtpZnmHC/cju0iI8irv1Fyq9CJH1YH03t2U8EM5871vSLFbUXblnZo1n8LOCDtTC0/ICoJ0\n5igNorzKOzVr9p2TdqLmn5D9/db9+Xy7Pi8bKdJ8u2/HZIryKu/TrNl3TtqJWnhC9u/u39On\n0vNIGRBlVt6nWbPvnLQTtQDSv90/Slc25FGUU3mvZs2+c9JO1PyVDf/7030//VIBKRsj0Vgo\nmzX7zkk7UXOcXAh6u5xrULj6+4uihLdqyKy8a7Nm3zlpJ2p2w/n3++n0s+veX+85nDW42X8x\n+N4JDdIDo6PkyjZr1uwbm3+Y4o/cDvd/Ht8YfQ853/sx3dEgbWzW7Dsz8Czlg3R4/R5svsNH\nRpfnYiVXtlmzZt+ZgWdJGaSnEwxHg7S1WbPvzMCzlA3SYfS9v84C9HU6fWF0+/JyYGdZqoKD\ndFH+H6qXs93Xq1Ul/0Q2a9bsOzPwLEmCNPGk0fWqb8mVbdas2Xdm4FnKBekw8b1MkKaeez0a\npO3Nmn1nBp4lNZBmrmC4vQxJcmWbNWv2nRl4lrRAmrsQ6GiQCpg1+84MPEsJl9J9XcUwpAdy\nZcP89XRfr4uVXNlmzZp952eeIpmXmi9dlXo0SCXMmn0zAguQCEjLF3ff3/FEcmWbNWv2zQgs\nQBIgrbxG4miQipg1+2YEFiABkFZfafR4Cy7JlW3WrNk3I7AAlQdp/QV7BqmMWbNvRmABKg1S\nwOtejwapjFmzb0ZgASoLUtDLxwdvriq5ss2aNftmBBaggiAFvg3D0SAVMmv2zQgsQMVACn43\nk+Hb5kuubLNmzb4ZgQWoEEjhbwp0NEilzJp9MwILUBGQgin6GH2wmOTKNmvW7JsRWIBKgBSB\n0eiDxSRXtlmzZt+MwAK0PUgxGI0/6VJyZZs1a/bNCCxAm4MUhdH4ky4lV7ZZs2bfjMACtDVI\nXRRH449ellzZZs2afTMCC9DGIAV95PxD449ellzZZs2afTMCC9C2IC1/XvWrxh9hLrmyzZo1\n+2YEFiBpkMYbkubKNmvW7JsRWIDUQQqe77psrqe0Qeo1P4W4x0gGqahZs29GYAHa/KzdwudV\nj/VyZKe5ss2aNftmBBag0q9HWtILR5or26xZs29GYAESBul1Q9Jc2WbNmn0zAguQNkjp5qzK\nNpcubZB6QeY7sSFprmyzZs2+GYEFSBqkdHNWZZuLlzZIvRDzndqQNFe2WbNm34zAAqQMUro5\nq7LN5UsbpF6A+U5uSJor26xZs29GYAESBindnFXZZoHSBqkXYL4GqbxZs29GYAESBWn6yE5z\nZZs1a/bNCCxAFJDydeaodAuWFSHNHWlmQ9L8E9msWbNvRmABkgUp3ZxV2WaJ0gapV+585zYk\nzZVt1qzZNyOwAKmClG7OqmyzRmmD1Ct3vgZJwqzZNyOwACmCNHtkp7myzZo1+2YEFiBRkNLN\nWZVtFiltkHrlzXd+Q9Jc2WbNmn0zAguQJkjp5qzKNquUNki9sua7sCFprmyzZs2+GYEFSBKk\ndHNWZZtlShukXlnzNUgqZs2+GYEFSA6kpSM7zZVt1qzZNyOwACmClG5elM31lDZIvTLmu7gh\naa5ss2bNvhmBBUgQpHTzsmyup7RB6pU+3+UNSXNlmzVr9s0ILEB6IKWbV2RzPaUNUq/0+Rok\nIbNm34zAAqQF0sqRnebKNmvW7JsRWIDkQEqd77psrqe0QeqVOt+1DUlzZZs1a/bNCCxAaiAl\nz3ddNtdT2iD1Sp2vQZIya/bNCCxASiCtHtlprmyzZs2+GYEFSAyk9Pmuy+Z6ShukXmkjWt+Q\nNFe2WbNm34zAAqQFUsZ812VzPaUNUq+0ERkkMbNm34zAAqQDUsCRnebKNmvW7JsRWICkQMqZ\n77psrqe0QeqVMqKQDUlzZZs1a/bNCCxASiBlzddmuFmzb0ZgATJINguWNki9EkYUdGSnubLN\nmjX7ZgQWoASQDmeNbx+G30wEKW++NsPNmn0DQs9QPEiH+z+D24en/yRhRAZJz6zZd2bgWRIB\nKezITnNlmzVr9p0ZeJYAIJ3GHKWBlDlfm+Fmzb5z0k5ULkjXh0aDh0h/nRX9My8bUrTJsnSU\nCdLtsG74vVPCjhS4IWn+iWzWrNk3IPQMgR4jDf/fIDVi1uw7M/AsSYAUeKpBdGWbNWv2nRl4\nljAgZR7ahXKkubLNmjX7zs88RTiQBmfuIkcUvCFprmyzZs2+8zNPUcaVDYfh7eGFDQkg5c/X\nZrhZs29I7PFSuNbOIGmaNftmBBYgAZDCj+w0V7ZZs2bfjMACpAESYL42w82afTMCC1B5kCI2\nJM2Vbdas2TcjsABJgISYr81ws2bfjMACZJBsFixtkHrFjCjmyE5zZZs1a/bNCCxACiBB5msz\n3KzZNyOwABkkmwVLG6ReESOKOrLTXNlmzZp9MwILkABImPnaDDdr9s0ILECFQYrbkDRXtlmz\nZt+MwAJUHiTQfG2GmzX7ZgQWIINks2Bpg9QreESRR3aaK9usWbNvRmABKg4Sar42w82afTMC\nC1BRkGI3JM2Vbdas2TcjsACVBgk2X5vhZs2+GYEFyCDZLFjaIPUKHFH0kZ3myjZr1uybEViA\nCoOEm6/NcLNm34zAAmSQbBYsbZB6hY0o/shOc2WbNWv2zQgsQGVBAs7XZrhZs29GYAEqB1LC\nhqS5ss2aNftmBBagoiAh52sz3KzZNyOwABkkmwVLG6ReISNKObLTXNlmzZp9MwILUEmQoPO1\nGW7W7JsRWIAMks2CpQ1Sr4ARJR3Zaa5ss2bNvhmBBaggSNj52gw3a/bNCCxAFJACdAapUGXL\nIqjQjpR2ZKf5J7JZs2bfjMACVA4k8Hxthps1+2YEFqAyICVuSJor26xZs29GYAEqBhJ6vjbD\nzZp9MwILkEGyWbC0Qeq1NqLUIzvNlW3WrNk3I7AAlQIJPl+b4WbNvhmBBcgg2SxY2iD1WhlR\n8pGd5so2a9bsmxFYgAqBhJ+vzXCzZt+MwAJkkGwWLG2Qei2PKP3ITnNlmzVr9s0ILEBlQCLM\n12a4WbNvRmABMkg2C5Y2SL0WR5RxZKe5ss2aNftmBBagIiAx5msz3KzZNyOwABkkmwVLG6Re\nSyPKObLTXNlmzZp9MwILUAmQKPO1GW7W7JsRWIAMks2CpQ1Sr4UJZR3Zaa5ss2bNvhmBBajA\njsSZr81ws2bfjMACVPTDmJHztRlu1uybEViADJLNgqUNUq8S87UZbtbsmxFYgAySzYKlDVKv\nEvO1GW7W7JsRWIAMks2CpQ1SrxLztRlu1uybEViADJLNgqUNUq8S87UZbtbsmxFYgAySzYKl\nDVKvEvO1GW7W7JsRWIAMks2CpXcB0uGs8e3h9wxSI2bNvvMzT1E8SIf7P4/bw++dDFIjZs2+\n8zNP0dYgdV3Hma/NcLNm3/mZpwgA0ikCpO54PGaQJLmyzZo1+84MPEu5IF0fGw2+99dZ89Uu\nL5A9Uk5wWFZJZYJ0oyh0R7qBxPhDZTPcrNk3IPQMbfsYySDVZNbsOz/zFG18ssGPkSoya/ad\nn3mKfNbOZsHSBqlXifnaDDdr9p2feYoyrmw4DG/7yob2zJp9Q2KPl6+1s1mwtEHqVWK+NsPN\nmn0zAguQQbJZsLRB6lVivjbDzZp9MwILkEGyWbC0QepVYr42w82afTMCC5BBslmwtEHqVWK+\nNsPNmn0zAguQQbJZsLRB6lVivjbDzZp9MwILkEGyWbC0QepVYr42w82afTMCC1BdL/uefxG7\nKzdVuuAvnSiD5MqCpQ0SV3tcWYNUhQySKwuWNkiWtUsZJMsCyCBZFkAGybIAMkiWBZBBsiyA\nagLp+T2/ti5eqGy537lU6UPR6qmqCKTRu1BuXLxM4YK/c6nSjzdNLLbYKTJIgbX3leZypQ8n\ng7SFioWqYJrLqFiUDdIW2h1IxR4qGKQoVQZSuX2hUOVypcuebDBIXBVeWZfetLBBYqnUtnAo\n/9d5P6UNEl0lB7uzNBukSNUEUtG57izNBilSFYFU7gCrr16orK9sqEMVgWRZujJIlgWQQbIs\ngAySZQFkkCwLIINkWQAZJMsCyCBZFkAGybIAMkiWBZBBklPnNalQXjQ5GaQa5UWTk0GqUV60\nDfX5s+t+fp56Vn50b38u3/tz+d711o/u8H698/16y6pHBmlDHbqzvp8urJzx6Q5npj777z1u\n/bjc+eNyyyRVJYO0nf6+wPHe/XNh5e3z9Hb98u30devn6dflsK6/8++uqlfjWAZpO33vh33d\ndH6fD+Uum9P37s/91uf1P+su3/Ijpcrk5dpO3U1flEzdOg2/ZdUjL9d2MkgNy8u1nb7fh91d\nD+jeZg7tHv9atcjLtZ3eL+cU/nfBpzv/8/nW/f18suH99Pt5k7LqkZdrO11PcF/OM5xBupz0\nPg1Pf//5OjlukGqUl2tDXZ58fft16g/t3r6ehr0/Ifv77XrLINUoL1cJmZLm5BUtIYPUnLyi\nJWSQmpNXtIQMUnPyiloWQAbJsgAySJYFkEGyLIAMkmUBZJAsCyCDZFkAGSTLAuj/fCfJzCF9\nwnAAAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(history)\n",
    "history$metrics$val_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the training process arrives over 71% and then stabilize! But the difference in performance between *LSTM* and simple *RNN* increases when the length of the sequences increases (Long Memory in action!). Let's what happens when we consider 100 words for each review in IMDB dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you could just imagine being there robert &lt;UNK&gt; is an amazing actor and now the same being director &lt;UNK&gt; father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for &lt;UNK&gt; and would recommend it to everyone to watch and the fly fishing was'"
      ],
      "text/latex": [
       "'<START> this film was just brilliant casting location scenery story direction everyone\\textbackslash{}'s really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was'"
      ],
      "text/markdown": [
       "'&lt;START&gt; this film was just brilliant casting location scenery story direction everyone\\'s really suited the part they played and you could just imagine being there robert &lt;UNK&gt; is an amazing actor and now the same being director &lt;UNK&gt; father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for &lt;UNK&gt; and would recommend it to everyone to watch and the fly fishing was'"
      ],
      "text/plain": [
       "[1] \"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "maxlen <- 100\n",
    "mtx_train <- pad_sequences(x_train, maxlen = maxlen, truncating = \"post\", padding = \"post\")\n",
    "mtx_test <- pad_sequences(x_test, maxlen = maxlen, truncating = \"post\", padding = \"post\")\n",
    "\n",
    "mtx_train[1,] %>% sapply(function(x) mapping_df %>% filter(index == x) %>% pull(word)) %>% paste(collapse = \" \")                  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_embedding(input_dim = max_features, output_dim = 8) %>%\n",
    "    layer_simple_rnn(units = 8) %>%\n",
    "    layer_dense(units = 1, activation = \"sigmoid\")\n",
    "    model %>% compile(\n",
    "    optimizer = \"rmsprop\",\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"acc\")\n",
    ")\n",
    "\n",
    "history_rnn <- model %>% fit(\n",
    "    mtx_train, y_train,\n",
    "    epochs = 10,\n",
    "    batch_size = 32,\n",
    "    validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.519</li>\n",
       "\t<li>0.5704</li>\n",
       "\t<li>0.7036</li>\n",
       "\t<li>0.7672</li>\n",
       "\t<li>0.7624</li>\n",
       "\t<li>0.748</li>\n",
       "\t<li>0.7468</li>\n",
       "\t<li>0.7728</li>\n",
       "\t<li>0.7536</li>\n",
       "\t<li>0.7492</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.519\n",
       "\\item 0.5704\n",
       "\\item 0.7036\n",
       "\\item 0.7672\n",
       "\\item 0.7624\n",
       "\\item 0.748\n",
       "\\item 0.7468\n",
       "\\item 0.7728\n",
       "\\item 0.7536\n",
       "\\item 0.7492\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.519\n",
       "2. 0.5704\n",
       "3. 0.7036\n",
       "4. 0.7672\n",
       "5. 0.7624\n",
       "6. 0.748\n",
       "7. 0.7468\n",
       "8. 0.7728\n",
       "9. 0.7536\n",
       "10. 0.7492\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] 0.5190 0.5704 0.7036 0.7672 0.7624 0.7480 0.7468 0.7728 0.7536 0.7492"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2di3bbxhZD6SpO3N4mcSr9/7de602K\n5HAeOHMwQ2CtJrJZeOAhdvgUNZwkSSrW4B1AknqQQJIkgASSJAEkkCQJIIEkSQAJJEkCSCBJ\nEkACSZIAgoL0GVBw4abkru5mDY4sLFACSW66oQWSy8zKbeJmDY4sLFACSW66oQWSy8zKbeJm\nDY4sLFACSW66oQWSy8zKbeJmDY4sLFACSW66oQWSy8zKbeJmDY4sLFACSW66oQWSy8zKbeJm\nDY4sLFACSW66oQWSy8zKbeJmDY4sLFACSW66oQWSy8zKbeJmDY4sLFACSW66oYdhCPxkTgkk\nudmGHt7e3tZJQhYWKIEkN9fQb1etkoQsLFACSW6mod/eBJJA6srtMfTbm0A6Cz+zcru5qw99\nR0jHSAKpJ3floR8YfeqsnUDqyV116DFGG25kYYESSHJ7D/1C0YYbWVig6oF0TJjaubvELDfz\n0HOMBFJovo5vJSQ1W8d23XWGXsJIIG3MWAFJzdaxXXeFoZcp2nAjCwtUTZDejtkoNVvHdt3m\nQ69iJJCCM3uZtFySmq1ju27joQMYCaTwzL4VbJSarWO7bsuhgxRtuJGFBarm6e+3/I1Ss3Vs\n12039BZGAmlzZrNJaraO7bqtht7GSCBtz03u7l2zdWzXXWReu8knhqKNsZGFBarWJ/Z9zez1\nxRWlSqNKPrrcdjr/9h2j+oHsVWmLNByPx2H0j1LqRqnZf9fbdReYh6U3QrxFbo22xkYWFqg6\nIJ05mpKUeKTUbB3bdUNBSqFoY2xkYYFyACnnnEOzdWzXjQPpLREjgbQ+s1OQ0nfvmq1ju+4S\n8+iteekUbYyNLCxQlUC6HCPNtvXxJDVbx3bdkLN2ORRtjI0sLFC1TjYMrydE0zZKzdaxXTdg\n6DyKNsZGFhaoWteR5hcWUs45BN97vK1my9wySNkUbYyNLCxQnu+QjZ7o8NMwttVsmdsFqQQj\ngZQ6N5EbpcXLEilqtsyNgpR5ZBQ3NrKwQDk/syFqvgWShzvXXEzRxtjIwgLl/vCTiEkXSB7u\nLHPWye60sZGFBcodpFiSdIxU2Z1uHlNkFxxZWKD8QbqtgODE6qxdfXeqebotEkglyp2bTZQE\nUn13knm2RyeQSpQ7N1skDZP7i9LVbJnbAGnpuEgglSh3bj7DR0qvd+olq9ky84P0PC6arjyB\nVKLcuTnruI6SQPJwR5jXKCodWiBlzs1F6yQJJA931C7E6joTSCXKnZurjqso6RjJwR0yhyEq\nHlogZc7NXaso6axdfffGhb+NE60CqUS5c/PQnaSKu9xyJ5ljb10QSCUqn9njykap2Tq2614y\nR1JUOrRAAszsMkrN1rFd98wcT1Hp0AIJMbPHJZSarWO77ok5do8OM7RAwszsAknN1rFd98O8\ncaLbYGiBBJrZx0Ypy102ttwTcw5ExUMLJNTMnkma3H/XbB3bdZ+yISoeWiDBZvaVpGbr2Ky7\nhKLCoQUScGaP03MOrdaxTXfWURFm6G03srBAsYJ0JelxU3iLdWzUXQ5R9tBRbmRhgaIFabx7\n99ZeHdt0jzdFrMGRhQWKGKQrSfFPv4OOvUP3y6aINTiysEAxg3QjqRwl1k7wuBeOiliDIwsL\nFDVIU5LyUWLtBIl7+aiINTiysEBxg3Q75VC6UWLthLE74s0nbwubIsTQAqlIBjN7I+lUhBJr\nJ2zd4acBvr0FICodWiCVyWRmLySd3YxXB4ndoefTblJUNrSpG1lYoPhBupB0cWcfKrF2wtS9\nAlIEQsVDm7qRhQWqAZDupxw+s1Fi7YSpewZSzGYIM7SpG1lYoFoAaURSHkqsnbB1T46REikq\nHFoglclsZo9lKLF2wtg9+yDXpHlj/bWRhQWqDZAmJKWjxNqJGu4chs6ifXATsrBANQLSabx7\nl3wCz7vMTu68LdFVvJ+kgywsUM2ANCUpDaVmUch1px8RvYj4s92QhQWqHZBeSEpBqUEUCtzF\nFH0KpHQ1BNIySS2fgMK7pwjljy2QUtUSSNNTDp/xG6WWUChwzzZDhVehdIyUoKZAmpEUiVIz\nKGS7l3fmiq5C6axdktoCabZ7F4dSCyjkukNHREVjs/7ayMIC1RpIayQFUWLtRLF747yCQKqn\nDJAOX1p6XQekOUnbGyXWThS5I07NCaR6Sgfp8Phj+vpUCaQFkrZQYu1Evjvu9LZAqqcGQVoi\nKYwSayey3BFbIsjYZL/2cxmnACA9VWtmZyfvPsOHSqyd2NT01NnGERF4bNZJK668jUpBuh8j\n/XUWLtaGLiS9fvNar2oZKuhyMef68gWirn7NLlQI0uHksGt31tLu3er+Hes/rht63F6QuCFC\njM07aZDa49XiMdJFiyStoMTaiQ0NyftyuLF5Jw3RegM1C1KQpEaeLLWhoQSiwrF5Jw3RegO1\nC9IKSZ8L1WPtxJbKMBJINdUwSGskzVCivW0sqFKKSsYuNgukCN3vZjiMXl9Ve2bXSJqixHsj\n87oK9+mKxkaYBVKRqs/sKkmj0w7Eb61Z05MixzYLpCS1DVKIpDtK7YE02hYJpPkyTjUOUpCk\n8fWXdkCa7NEJpPkyTrUOUiRJzRwjvRwYCaT5Mk41D1KYpKWT4cCx0e5ZWIE0X8ap9kHaIKkc\npXqNmgcVSPNlnOoApC2SSlGq1ailkAJpvoxTPYC0SVILH1O2nFAgzZdxqguQtkg6u/NRqtKo\nlXgCab6MU32AtEHS1Z2LUoVGrUYTSPNlnOoEpDBJN3fmoZJ9o9ZjCaT5Mk71AlKQpIc7iyTz\nRgVCCaT5Mk51A1KIpJE7AyXj5MFAAmm+jFP9gBQgaeJORsk0+UYagTRfxqmOQFonaepOPVSy\nTL4VRSDNl3GqJ5BWSXp1p5Fkl3w7h0CaL+NUVyCtkTR3p6BklTwmg0CaL+NUXyCtkLTkjkfJ\nKHlUAIE0X8apzkBaJmnRHU2SSfLI0QXSfBmnegNpkaRQmZFjx7tdIa5gFkhF8pnZFy2QFN69\nAo4d6/berSQfWiC5zOyr5iRtHPADx45yE5zo4B5aILnM7EwzkgLuiGMVcPKkc+8Cab6MUz2C\nNCMp6N4sNjY5yzUs5qEFksvMLuiFJJ67C1LvTxJI82Wc6hOkF5K23OF645Kn3zErkObLONUp\nSFOStt2hioOSv+W8HUogzZdxqleQJiRFuAMthyTPwkggLS3jVLcgjUiK+zSK1aKXJ3/LxEgg\nLS3jVL8gPUgavv6OetLqStkLk+dTVD62m1kgFclnZld1JWm4fHJz3DOLFxufn/xtpLyfIJDm\nyzjVM0hXklJAWjxUyhn7baqMn5A/NsgtkJLUNUgXkpJAgjw2+IWihj4Jg2RogeQysyEdb0rp\n8itKSWNPN0TtfTYTxdACyWVmg7pilFjl6Q5Z/NiznTmBhHcjCwtU7yBtPxh8SRMeosZeOSRq\n8fNr/YcWSC4zu6UckuJPuL0eEL0sbvMT1Z2HFkguM7upLJI2EXldvIIcax+ZhxZILjO7rUyS\nlmFZ1uqPYO0j89ACyWVmI9zZJH2ed87yCLqPnTuut5s1OLKwQO0DpPxt0uh0QSJBj7FLJJDm\nyzi1E5CySWr5BLZAqqi9gJRLkkBicyMLC9RuQMokSSCxuZGFBWo/IBWQ1OglVYFUUTsCKZek\nZi+pCqSK2hNIn3mHSQzJHdyswZGFBWpXIOWRRJG8vps1OLKwQO0OpHSSKJLXd7MGRxYWKChI\n/DqT5J1B6lH72iJlbZNIktd2swZHFhaovYGUcZjEkryymzU4srBA7Q6kdJJoktd1swZHFhao\nXYKURhJN8rpu1uDIwgK1P5CSSeJJXtXNGhxZWKB2CFLqzh1R8ppu1uDIwgK1R5ASSWJKXtHN\nGhxZWKD2ClICSUzJK7pZgyMLC9QuQUojiSp5PTdrcGRhgdonSEkkcSWv5mYNjiwsUDsFKeUw\niSx5LTdrcGRhgdorSAkksSWv5GYNjiwsUHsGKZIktuSV3KzBkYUFarcgxZNEl7yOmzU4srBA\n7Rek6J07vuRV3KzBkYUFascgxZJEmLyGmzU4srBA7RykGJIIk9dwswZHFhaoPYMUSRJj8gpu\n1uDIwgK1a5DiSKJMbu9mDY4sLFD7BinqMIkzubmbNTiysEDtHKQYkkiTW7tZgyMLC5RA2iSJ\nNLm1mzU4srBA7R2kCJJYkxu7WYMjCwvU7kHa3rmjTW7rZg2OLCxQAmmTJN7kpm7W4MjCAiWQ\nNnfueJObulmDIwsLlEDaJIk4uaWbNTiysEAJpM+tnTvm5IZu1uDIwgIlkD63NknMyQ3drMGR\nhQVKIJ0VJIk6uZ2bNTiysEAJpItCJHEnN3OzBkcWFqhVkP45nE4/h8PfKT/MZ2YhboFENHRP\nIP0zDKffh2EYUkjymVmIO7BJIk9u5WYNjmi9gdZA+jb8/Prvn1/DIeGH+cwsxr1OEntyIzdr\ncETrDbQG0tcG6d/h2+XvePnMLMgtkGiG7gmkw/D7x/DrfJSU8MN8ZhbkXt0k0Se3cbMGR7Te\nQGsg/f11eHQ4b5A+En6Yz8yi3Gsk8Sc3cbMGR7TeQKt7bh/D4d+vDVMKR42DtEZSA8kt3KzB\nyztvIl1HGkkgcQwtkFxmFuheJKmJ5Hg3a3BkYYHSBdmxFnfumkiOd7MGL++8iXRBdqIlktpI\nDnezBke03kC6IDuVQCIYuieQ9ndB9qY5Sa0kB7tZgyNabyBdkH3RfOeuleRgN2twROsNpAuy\nr5qR1ExyrJs1OKL1BtIF2ZleSWonOdTNGry88ybSdaS5XkhqKDnSzRocWVigBNKCpiS1lBzo\nZg2OLCxQqyD9+fg2DN8+/qT8MJ+ZNXBPSGoqOc7NGry88yZaA+lyMfZ8wuH3bNHhS+Mvny99\nZtbCLZBog0Prj9MaSD+G9y+Efr8PP16XHB5/XL/sEqQxSY0lR7lZg2P7D1Pgguzk76emIB36\n3CKNd+4aS45yswYHlh+pQpAOne7ajUlqLTnIzRoc2X6gCnftHiD9dZZJQi+dSfLOIDWj9JMN\nY5AOp263SM9tUnvJIW7W4GgCQEo//T0C6eW8Q2cg3UlqMDnCzRoc23+Y0i/IjkG66rHIZ2bt\n3FeSWkwOcLMGBxUfrSKQTi8vfWbW0C2Q+NzFlbfRIkjDWK8L9wTShaQ2kxe7WYOjCQApHaTH\nnQ2jEw43+cyspfty6s5pbGc3a3ADCBDSTathbX3ArOXYvm7W4MjCAiWQNlRKUqu/N21wZGGB\nEkhbKiSp2d+bNTiysEAJpE2VkdTs780aHFlYoATSloahiKRmf2/W4MjCAiWQNnTBSCDxuJGF\nBUoghXXjKJ+kRn9v3uDIwgIlkMJ6gJRLUqO/N29wZGGBEkhhXUEqOExq9PfmDY4sLFACaUNn\nhoaCU3et/t60wZGFBUogbWkYhvNfuSQ1+3uzBkcWFiiBFOvOJIkgeXNDCySXma3lzjvlwJC8\ntaEFksvMVnNnkUSRvLGhBZLLzNZz55DEkbytoQWSy8zWdKejxJK8paEFksvMVnUnk0STvKGh\nBZLLzNZ1p5LEk7ydoQWSy8zWdqeRxJS8laEFksvMVncnkUSVvJGhBZLLzNZ3p5DElbyNoQWS\ny8w6uBNIIkvexNACyWVmPdzxJLElb2FogeQysy7uaJLokjcwtEBymVkfdyxJfMn5hxZILjPr\n5I4kiTA5/dACyWVmvdxxl2YZk7MPLZBcZtbNHUUSZXLyoQWSy8w6uiNIIk1OPbRAcplZT/c2\nSazJmYcWSC4z6+reJIk2OfHQAsllZn3dWyTxJucdWiC5zKyze+OUA3Fy2qEFksvMervDJDEn\nZx1aILnMrL87RBJ3cs6hBZLLzBK4AySRJ6ccWiC5zCyDe50k9uSMQwskl5mlcK+SRJ+ccGiB\n5DKzHO61Uw78yfmGFkguM0viXiGpgeR0Qwskl5mlcS+S1ERysqEFksvM8riXSGojOdfQAsll\nZoncCyQ1kpxqaIHkMrNM7jlJrSRnGlogucwslXt2yqGZ5ERDCySXmeVyv5LUTnKeoQWSy8yy\nuacktZScZWiB5DKzdO4JSU0lJxlaILnMLJ97TFJbyTmG3j1I0k2XAyXvEFJNaYtk4n6ecmgt\nOcPQu98i+cwspftO0jAM1cfGuFmnHFlYoASSlfuC0nD+r/7YCDfrlCMLC5RAMnMf7yogSSDN\nl3FKINm5BZKFG1lYoASSpVsg4d3IwgIlkEzdcZ9YYTO2QKopgWTrHspQEkjzZZwSSNbuIpIE\n0nwZpwSSuftYgJJAmi/jlECq4M4nSSDNl3FKINVwZ5MkkObLOCWQ6rgzURJI82WcEkiV3Hkk\nCaT5Mk4JpFrurHMOAmm+jFMCqZ47AyWBNF/GKYFU0Z1OkkCaL+OUQKrqTkVJIM2XcUogVXan\noSSQ5ss4JZBqu48pKAmk+TJOCaT67mM8SwJpvoxTAsnDHY2SQJov45RA8nFHoiSQ5ss4JZDc\n3DEkCaT5Mk4JJEf3NkoCab6MUwLJ1b2FkkCaL+OUQPJ1bxwqCaT5Mk4JJG938LSDQJov45RA\n8ncHUBJI82WcEkgM7lWUBNJ8GacEEol7mSWBNF/GKYFE415CSSDNl3FKIBG55ygJpPkyTgkk\nLvcLSwJpvoxTAonNPUFJIM2XcUog8blHKAmk+TJOCSRK940kzw/OZJ00ZGGBEkik7vtmye2D\nM1knDVlYoAQSrbucJIFUTwKJ1530dAfw2LyThiwsUAKJ1z0cj2UsCaR6EkjE7idJDh8Kwzpp\nyMICJZCY3eezdgUsCaR6EkgNuHNZEkj1JJCacOehJJDqSSC14s5ASSDVk0BqyJ3KkkCqJ4HU\nlDsNJYFUTwKpMXcKSgKpngRSe+5olgRSPWWAdPjS0muBVM0diZJAqqd0kA6PP6avTwKpojsK\nJYFUTwKpWfc2SwKpnspAOk1f+8zsft1bKAmkekKB9NdZmEhSvO4oeeeQykHSyQZfd2CzpC1S\nPQmk9t2rN+IJpHoqBmm8k+czs3I/UZqyJJDqqRSkMUcCydG9xJJAqqdCkCYcCSRf9wwlgVRP\nBXc2HK4vDzr9zeOesFT2VDzWXxtWfax0r11v7gdK5yc+eD3LSyAVyWdm5X7RcaR8klh/bWRh\ngRJIXboFUm0JpE7dqxeX7Ie2dSMLC5RA6ta9cnGpxtACqUw+Myv3iobRAybTYWL9tZGFBUog\nde7OZsk7+OoyTgmk/t3HLJgIgi8v45RA2oc7nSWS4PNlnBJIu3EnssQT/GUZpwTSrtwJKHEF\nHy3jlEDamzuWJbrg92WcEkg7dEexxBj8soxTAmmX7ojDJc7gAol1vezWvXXugTU4srBACaQd\nu4MssQZHFhYogbRv9/rFWtbgyMICJZDkXmaJNTiysEAJJLk/FzdMrMGRhQVKIMl905Slsgc+\nCKQi+cys3DD3cSLOR6cgCwuUQJJ7KgxJAqlEPjMrN9q9firPfOhtN7KwQAkkuRc0lLIkkErk\nM7NyG7gnJGXQJJBK5DOzclu4b2ftcnkSSCXymVm5TdwPcxZLAqlEPjMrt4n7xZwIk0Aqkc/M\nym3iXjRHsySQSuQzs3KbuNfMcRsmgVSi3Lkpm1m5Tdwh8zZLAqlEuXNTNrNym7g3zBuHTAKp\nRLlzUzazcpu4I8zrp/MMb3mNKfWk1f8c1v4/pASS3CXmZZYKP+MMC9IA7fjqmMgfljs3ZTMr\nt4k7xfxC0/W2CKNbXmNKLZDkpnEnm18v3LqA9Pt9+H5F5+f3YTh8nDmafGklgSQ3zpxzD0Tq\n2MEG/jl8YfP9TM6/w0UfN5AeX1pJIMmNNpfTlA3Sx/B++vN+Jufb8L/T6df51WWD9PzSSAJJ\nbgPzUHb3eDZI34bfX7t3V15+//v3+wOk55dGEkhyWw39ylI8TtkgXUm5/Pl+3Zm7f+/xpZEE\nkty2Q+ewBADpx/Dtn39/P0B6fmkkgSR3laGTaALs2l2Y+fN49fzSSAJJ7npDx26dskH6e3j/\nc7odGf28nXa4gXT/0kgCSe6qQy8cOM15Apz+/hiex0iH8ZdGEkhyOwy9hNOIp4ILst/vF2R/\nDMP7z/Orf84gPb80kkCS22/oFZ6Ct7wiCwuUQJLbfegFmtZJQhYWKIEkN8vQcSQhCwuUQJKb\nbGiBJEkA3d6E4R0jUdoiyc02dPhtgcjCAiWQ5KYbWmftrGZW7vpu1uDIwgIlkOSmG1ogucys\n3CZu1uDIwgIlkOSmG1ogucys3CZu1uDBBv63LmTRFySQ5KYbWiC5zKzcJm7W4MEGCqSCmZXb\nxM0aPNhAgVQws3KbuFmDBxsokApmVm4TN2vwYAMFUsHMym3iZg0ebKBAKphZuU3crMGDDRRI\nBTMrt4mbNXiwgQKpYGblNnGzBg82MBOkYfFlklp7/5QkBTQmZxiGFZAsSi+QpI405uj89kCB\nJEkZ+u+/xSd8HY9jkC4PihxO14dHXp/Derp8/fLydP3/4iSQpI4UBdLp8TDw4fbV8Px6+jL+\nczMFktSREkAafzVMmXq+jAdEIEkdKf4Y6Vb82+PBBZIkjRR/1m4YvxJIkjRW3HWk8eGRQJKk\nmTJACu/a+Zxs8LnULbeJmzV4sIGRID1Oa99eLYPkd/rbZ2blNnGzBg82MA6kFAkkuQVSGUh+\nx0g+Myu3iZs1eLCB2C1SymdlCiS56YZmASlFAkluuqEFksvMym3iZg0ebKBAKphZuU3crMGD\nDRRIBTMrt4mbNXiwgQKpYGblNnGzBkcWFiiBJDfd0Noiucys3CZu1uDBBgqkgpmV28TNGjzY\nQIFUMLNym7hZgwcbKJAKZlZuE7fj0Pmfai6QNiR3dbff0Jc3ia//5JAE0obkru52G/r6tJJV\nkoINFEgbkru6u+LQS4/9EUiImZXb32069MpDswTSVQUzuyG5q7ttht4m6Fj9GGl4YjDMlsRK\nIMltYB6dd4tBZ929kCukMTnrj+NaZ2D5ZYwEktx482WbkoJOwtjBBo45ent7W3lA5DoDAklu\ntDvDnLHxSR872MD//ntb1uQh+vc/x4/Qfz4z6Pbw1emXzyfrr0ggyV1uLkcnfuxgA5NAmj6B\na/SMu+fXoy9HzzlekkCSu8AcIihwnFMydrCBUSBNHqI/AenluymPXRVIcueYwxuf8Hm3srGD\nDYw7Rno+aXX0CP3RM4zv3xVIcpe7V8xxu2/B827ZY1+XhRR51u65W7ewRRofNgkkuUvdL+bE\nAyC74MEGRl5HGiaP/54/8VsgyY1yP7YpeacQuEEaP9576WSD966d1I1WrwJ5BwsrCaSXR+iP\nznffzo1PT3+ftEWSO0lZmyDM0BHuYANjQcJLIMn9EIKgzKHj3cEGCqSCmZW72L2yG2d0JajM\nHWygQCqYWbnz3avbIMsrQWXuYAMFUsHMyp3h3tyLM7wSVOYONlAgFcys3Enu2OMguuD3ZSEJ\npIKZlTvenXAmgSv4aFlIAqlgZuWOcqeei6MJ/rosJIFUMLNyr+h2mJN5Ppv110YWFiiB1K17\n8eaEOkMLpDL5zKzci8pHqHhoUzeysEAJpA7dZQQVDW3vRhYWKIHUlXu2K5d/MYj110YWFiiB\n1I/7laEijmh/bWRhgRJIXbgXd+bKbk5g/bWRhQVKIDXuDp1SKBqb9ddGFhYogdSye+PEnECq\nJ4HUqDvmzJxAqieB1Jw7/vqQQKongdSWO+kqq0CqJ4HUjjv5MqtAqieB1IY761YFgVRPAonZ\nfb0SlLwlgozNOmnIwgIlkIjdL/dvVx2bdtKQhQVKIPG6hxKICsfmnTRkYYESSKzuUopKxi42\nC6Qi+cxsh+7xDp3Xfaesk4YsLFACic49PTByu++UddKQhQVKIFG5X84teD5cjnXSkIUFSiDx\nuBeOihyTs04asrBACSQO98ppboE0X8YpgUTgXj9BJ5DmyziVAdLhS0uvBVKOO3zFVSDNl3Eq\nHaTD44/p65NASnZvXisSSPNlnBJIXu6om38E0nwZpwSSizv2tgWBNF/GKRRIf50FS9W3RhR5\nR5FQKgfpoC1SijvtDjptkebLOFUG0vmsnXbtEtyp96EKpPkyThWCNH3tM7PNuFMpQo5d2SyQ\ntqWTDVnuDIpgY9c3C6RtCaR0d9qBEXZsF7NAitD9bobD6PVVPjNL786lCDG2k1kgFclnZtnd\n2RQBxvYyC6Qi+cwst7uAouKxBVI9CSRTdxlGAmlpGacEkp07/9CofOxyN+uUIwsLlECycj/P\n1LX5sXmsU44sLFACycY9vrdbICHdyMICJZAM3I99ukEgwd3IwgIlkODu8aFR4QciC6SFZZwS\nSGD3yxkGzwdqCaSKEkhI98J5ukaSUw0tkFxmlsa9eLq7ieRkQwskl5klca9cNWogOd3QAsll\nZhnc69de2ZMzDi2QXGbW3x26hYE7OefQAsllZr3d4TuBmJOzDi2QXGbW1b15Px1tcuKhBZLL\nzDq6I25LJU1OPbRAcplZN3fU3d2UycmHFkguM+vkjnyPBGFy+qEFksvM+rhj32rEl5x/aIHk\nMrMe7vh37LElb2FogeQys/XdKW985UrextACyWVma7vT3kDOlLyVoQWSy8zWdac+h4EneTtD\nCySXma3pTn+cCUvyloYWSC4zW8+djhFL8raGFkguM1vNnYERSfLGhhZILjNby52DEUfy1oYW\nSC4zW8mdhRFF8uaGFkguM1vHnckRQfL2ht49SB3rzJF3BolY2iLFuPMOjzBje7lZgyMLC5RA\n2tIwDAUctft7swZHFhYogbShIeukN2ZsgbS0jFMCKaxSjlr9vXmDIwsLlEAKq+XH4AukihJI\nYQkkNjeysEAJpKCOpRw1+nv7Di2QXGbW0H3FqNXPkxBIFSWQArqeZmgxOcDNGhxZWKAE0rpu\np+saTI5wswZHFhYogbSq+2nv9pJD3KzBkYUFSiCt6XH5qLnkGDdrcGRhgRJIK3pehm0tOcjN\nGhxZWKAE0rJGtzM0lgbvgF8AABUVSURBVBzlZg2OLCxQAmlR49uC2koOc7MGRxYWKIG0pMnt\ndU0lx7lZgyMLC5RAWtD0NtWWkgPdrMGRhQVKIM31crt3Q8mRbtbgyMICJZBmen3bRDvJoW7W\n4MjCAiWQXjV7+1EzybFu1uDIwgIlkF40fxtfK8nBbtbgyMICJZBeNH87bCvJwW7W4MjCAiWQ\nplp4W3kjydFu1uDIwgIlkCZaej5DG8nhbtbgyMICJZDGWnzOSRPJ8W7W4MjCAiWQxlp8XlAT\nyfFu1uDIwgIlkEZafu5WC8kN3KzBkYUFSiA9tfIAuwaSW7hZgyMLC5RAemjtQZD8yU3crMGR\nhQVKIN21+kBV+uQ2btbgyMICJZDuWn0wMX1yGzdrcGRhgRJIN60/4Js9uZGbNTiysEAJpKsC\nT8onT27lZg2OLCxQGSAdvrT0umWQQp84wZ3czM0aHNF6A6WDdHj8MX19ahik4Ce3UCe3c7MG\nR7TeQALpc+uDlpmTG7pZgyNabyCBtPmB5cTJLd2swRGtNxAKpL/OgqWqKn1guVSuMpC6ONmw\n9cmWvMlN3azBEa03kHbtNj8hlja5rZs1OKL1Bto9SNsftcya3NjNGhzRegPtHaSIjywnTW7t\nZg2OaL2BBNIWR6zJrd2swRGtN1DBnQ2HU/snGyI4Ik1u7mYNDio+Wvu+1y5ix440ub2bNTiy\nsEDtGqQojiiTV3CzBkcWFqg9gxTHEWPyGm7W4MjCArVzkNzG5nezBkcWFqgdgxTJEWHyKm7W\n4MjCArVfkCJ37AiT13GzBkcWFqjdghTNEV3ySm7W4MjCArVnkNzGbsLNGhxZWKD2ClI8R2zJ\na7lZgyMLC9ROQYrfsWNLXs3NGhxZWKD2CVIKR1zJq7mHYfAaWiD5zGy6O4kjquTV3MPb21sJ\nSQKpRD4zm+5O4ogqeS33maMikgRSiXxmNtmdxhFT8mpugZSqHYKUtmPHe7Bg6RZIqdofSKkc\nvRbqbaqksZOlY6T5Mk7tEqQE6/if5rdtbYydLJ21my/j1O5ASjxAeoAUQdEiTyy/d0tDCySX\nmU1yJ+7Y3UAKbHSy6MpJXttNssLmyzi1M5BSOZqStP1/o3kSSPNlnNoXSOkcfWYBAMNJIM2X\ncWp3IKU5S3bLPl/OIuewJJDmyzi1B5AeJ6BSOSqj6HP5ckwSSwJpvoxTOwBp+OLn0uXUHbtb\n2YvGnnE0/tHbKFWatYhDuwTZBkcWFqj+QTpzdCEpkaNHKTBbw5Wfv8GS2axh0dnCChhcIOVM\nKMI9BineNSqBXfLtsgHHrodOSBnBX5dxaj8g5XJkmnyrYUVjl9U7f+hyrARS4PfPXi+l7uHO\nUTRI07VsnDxYrMSxyyucPfRYwZOVMWEEksl6KXWfj1OSOQKNHeVe71UShqXopAdfUeDe8aiE\nwfv8kIUFag8gfbkLOKqTfLlTgUbFoON2+jv23vHl2GE3srBArYL0/bJk+Pa7XhZDxX/e8mVd\nmmbZGrxA05/2haHPr5E69NKv4hY9U2t5P64zMQw/En5Y6B8f1y1S/Pbo+m8icuxkRxI6gZ9T\n+Jai+itsCtLqT+bUGkiH4ef5r19J/7KgZxbljt+xWyqnQ/Iygq4qfZOr2wrrC6Q7QD2AlMYR\ndmyjY3Zzt+tp1mDu0sYbaY2T78OPP6fTn4/hPeGHWc1sOUhx/+Pyv/SsjYowtwlSV2ftfh+G\niw6/En6Y2cwWuQs5om3Uprm1Y6Q4N6L1Blrdc/vz8W0Yvn0knbTzmdkNxe7YrR55sDZqU2WP\nXWD9tcs7b6LuryOdOYq+KAoeu2k3a3BkYYHqHaTL9ijCHTgRxtooYzdrcGRhgYJekPWZ2aAu\n+3Xb7tAJZdZGGbtZgxdX3kadX5C9Hh9F3bEGH7txN2vw8s6bqO8Lssc4kMIXOFkbZexmDV7e\neRN1fUH2fsIu7N66UYC1UcZu1uCljTdSzxdkHye+g+7NG25YG2XsZg2OaL2BOr4g+7yAFHJv\n37fG2ihjN2twROsN1O8F2dGF2IA74v5P1kYZu1mDl3feRN1eRxrf0LDujrmPmrVRxm7W4MjC\nAtUzSNvuqPcjsDbK2M0aHFlYoFZB+hhuSvhhPjO7qGMMSFEc0TbK2M0aHFB6CwUuyLYM0vRO\n1RV3HEe0jTJ2swZHtN5A6xdkf70Pv/+8X6/LRspnZpc0veN72R3JEW2jjN2swRGtN1Dgguzf\nw7+nP21eRzpGgBTLEW2jjN2swRGtN1AApH+Hfxq9s+H1LUhL7miOaBtl7GYNjmi9gdbvbPjf\n7+Hb6WeLIM3eyrfgjueItlHGbtbgiNYbaI2TM0Hv53MNDd79PXtL7NydwBFto4zdrMERrTfQ\n6gbn32+n049h+Ej5YT4z+6r5e8tn7hSOaBtl7GYNXt55E/V3QXbhGQ2v7iSOaBtl7GYNjiws\nUF2CtOFO44i2UcZu1uDIwgLVHUhLzwyauhM5om2UsZs1OLKwQPUG0uLDtybuVI5oG2XsZg2O\nLCxQnYG0/BC7sTuZI9pGGbtZgyMLC1R/IIXd6RzRNsrYzRocWVig+gJpmaPnE0fjPschb+ze\n3KzBkYUFqiuQlnfsns/AzuKItlHGbtbgyMIClQHS4Uujl48v3EFa5+hKUhZGvI0ydrMGx/Qe\nrnSQDo8/xt+4yGdm71p5Wv4DpEyOaBtl7GYNXlx5G5WDNHrpM7N3rXzqxB2kXI5oG2XsZg1e\nXHkb9QPS6qe3DGUc0TbK2M0avLjyNioG6fbqr7MwkfJ0XP/g8vP75c8cVc0j7UsokC7y+Sfq\nouDHiZ0Ktke8/zQbu1mDF1feRh2BFHCXcETbKGM3a/DiytuoFKQxR44ghT/esogj2kYZu1mD\nF1feRn2AFP6c2DKOaBtl7GYNXlx5G3UBkilHtI0ydrMGL668jQrubJhdmfUCyZYj2kYZu1mD\nI1pvoB7utQtxdLkay9oJajdrcGRhgeoApNAG6XqbKmsnqN2swZGFBap9kLY4shy7ZzdrcGRh\ngeoCpLVF98Mj1k5Qu1mDIwsLVPMgBTZIj9MMrJ2gdrMGRxYWqNZBiuGIthPUbtbgyMIC1ThI\n6xyN3w3L2glqN2twZGGBah+k5QWTd5WzdoLazRocWVig2gZpdYM0vQrL2glqN2twZGGBahqk\nSI5oO0HtZg2OLCxQrYO0+P3Xu4JYO0HtZg2OLCxQLYO0tkGa3V3H2glqN2twZGGBahikFY4W\nHl7H2glqN2twZGGBahukhe8uPQSStRPUbtbgyMIC1S5IyxukxTdNsHaC2s0aHFlYoJoFKYEj\n2k5Qu1mDIwsLVKsgpXBE2wlqN2twZGGBahik+f+19mZY1k5Qu1mDIwsLVKMgLW6QVt9UztoJ\najdrcGRhgWoTpDSOaDtB7WYNjiwsUM2CNP9/1h9ywtoJajdrcGRhgWoSpESOaDtB7WYNjiws\nUC2ClLhjx9sJajdrcGRhgWoQpGSOaDtB7WYNjiwsUO2BlM4RbSeo3azBkYUFqkmQZos3nqbK\n2glqN2twZGGBag6kDI5oO0HtZg2OLCxQrYG0tGO3+XRv1k5Qu1mDIwsLVGMgZXFE2wlqN2tw\nZGGBagukPI5oO0HtZg2OLCxQUJDMtfCBy/qUZYlBTW2RFjZIUZ9+xPqPK7WbNTiysEC1BFIu\nR7SdoHazBkcWFqjGQHr5duSn8bF2gtrNGhxZWKAaAmm+QYr9VEvWTlC7WYMjCwtUOyAt79jF\nusvG3qObNTiysEA1BdLLN6M/ZZm1E9Ru1uDIwgLVDEj5O3a8naB2swZHFhaolkCafiueI9pO\nULtZgyMLC1QrIM02SAkc0XaC2s0aHFlYoBoBqYgj2k5Qu1mDIwsLVDsgTb5O4oi2E9Ru1uDI\nwgLVBkivG6Q0jmg7Qe1mDY4sLFBNgFTIEW0nqN2swZGFBaoVkMZfpnJE2wlqN2twZGGBagGk\nlw1SMke0naB2swZHFhaoBkA6czRyp3NE2wlqN2twZGGB4gfpsj16ujM4ou0EtZs1OLKwQDUB\n0sidwxFtJ6jdrMGRhQWKHqTjBKQsjmg7Qe1mDY4sLFDsIN1ONNzceRzRdoLazRocWVigGgBp\n5M7jiLYT1G7W4MjCAkUO0nECUiZHtJ2gdrMGRxYWKG6QHleQLu7MHTveTlC7WYMjCwsUPUhP\ndzZHtJ2gdrMGRxYWKGqQjmOQ8jmi7QS1mzU4srBAMYM0ujXoVMIRbSeo3azBkYUFihykh7uE\nI9pOULtZgyMLCxQxSON7VYs4ou0EtZs1OLKwQHGD9HhdxBFtJ6jdrMGRhQWKF6SXDVKqvWhs\nuXmDIwsLFC1IuB073k5Qu1mDIwsLFDNI95elHNF2gtrNGhxZWKAyQDp8aek1FKTRBunyUWJp\n7rKx5XYeeh8gHR5/TF+fkCC9cES7Vjt2swZHtN5AvCDdXl04GoYhyV40ttzeQ+8UpKdgM/vc\nIF05+vqyhCTWTlC7WYMXV95GpSDdj5H+OgsV6vj80OXz8dGZoy+SUD9dkvAqBOlwsti1m+zY\nfX7eQEr4AQVjy+0/9B63SCcDkF527ASSi5s1OKL1BiIF6frifgFJx0gObtbgiNYbiBCkxwbp\neSFWZ+3qu1mDI1pvID6QFjjiXasdu1mDI1pvoII7Gw6j11dBZnaJI9q12rGbNTio+GjR3Wt3\n3yBNb7BjXasdu1mDIwsLFBtI4x27dHfZ2HKTDC2Qymd2mSPatdqxmzU4srBAkYG0vGPHu1Y7\ndrMGRxYWKC6Q1jiiXasdu1mDIwsLFB1Il79n7+RjXasdu1mDIwsLFBVIow1ShrtsbLl5hhZI\nhTO7tmPHu1Y7drMGRxYWKCaQbhukpUc0sK7Vjt2swZGFBYoIpABHtGu1YzdrcGRhgeIC6fzX\n4iODWNdqx27W4MjCAsUD0nODlOMOSu62hhZIBTMb2rHT2ygc3KzBkYUFigmkz1WOvr6rN/ZV\ndrMGRxYWKBaQQjt2Z46KSGLtBLWbNTiysECRgBQ8QBJIHm7W4MjCAsUD0ufqQ74FkoebNTiy\nsEBxgHTdIK0+LF/HSA5u1uDIwgJFA1LwQyd01q6+mzU4srBAUYD02CBlubcld1tDC6S8udnm\niHatduxmDY4sLFAkIG19mhjrWu3YzRocWVigCEC6bJA2PpWPda127GYNjiwsUBwgbX7cMuta\n7djNGhxZWKD8QbpvkLJndltytzW0QMqYm5gdO9612rGbNTiysEAxgBTxseWsa7VjN2twZGGB\n8gbpvEHa5oh2rXbsZg2OLCxQBCBtHiBtzOy25G5raIGUPDe3DVLRzG5L7raGFkipcxO5Y8e7\nVjt2swZHFhYo388Kv3HkmkGSAHLdIsXu2PH+89ixmzU4srBAeYMUxxHtWu3YzRocWVigPEGK\nPUDamFm5TdyswZGFBcoRpASOaNdqx27W4MjCAuUKUjRHtGu1YzdrcGRhgfIDKf4AaWNm5TZx\nswZHFhYoN5BSdux412rHbtbgyMIC5QhSAke0a7VjN2twZGGB8gIpaceOd6127GYNjiwsUH4g\npWyQaNdqx27W4MjCAuUEUiJHtGu1YzdrcGRhgfIBKZUj2rXasZs1OLKwQDmChJpZuU3crMGR\nhQXKBaTkDRLtWu3YzRocWVigfEBK5Yh2rXbsZg2OLCxQHiAl79jxrtWO3azBkYUFygGkDI5o\n12rHbtbgyMIC5QFS8o4d71rt2M0aHFlYoOqDlMMR7Vrt2M0aHFlYoGqB9PyosIwdO9612rGb\nNTiysEBVAun54ZVZHNGu1Y7drMGRhQWqDkjPj1PO2rHjXasdu1mDIwsLVG2Q8jiiXasdu1mD\nIwsLVC2Qvo6RziBlckS7Vjt2swZHFhaoWsdIx+PxtkGCz6zcJm7W4MjCAlVpi3R+FOQXSbkb\nJNq12rGbNTiysEBVBSmbI9q12rGbNTiysEDVBumIn1m5TdyswZGFBarmMVL+Bol2rXbsZg2O\nLCxQFe9sKOCIdq127GYNjiwsUPXutXvL37HjXasdu1mDIwsLVFWQsjmiXasdu1mDIwsLVAZI\nhy+NXz++CE5e0QaJdq127GYNDqk9XukgHR5/jP6+KvDrl3FEu1Y7drMGL668jWqB9FnEEe1a\n7djNGry48jYqA2nKUejXP19HMppZuU3crMGLK2+jQpAeh0h/nRWyHY/JI0lSMwJskeJONrD+\nAyc339D72yKdpq99ZlZuEzdr8OLK20ggyU039P5A0q5dv27W4IjWG6gcpNHGyWdm5TZxswZH\ntN5ABXc2HEavr/KZWblN3KzBQcVHy+9TzWEzK7eJmzU4srBACSS56YYWSC4zK7eJmzU4srBA\nCSS56YYWSC4zK7eJmzU4srBACSS56YYWSC4zK7eJmzU4srBACSS56YYWSC4zK7eJmzU4srBA\nCSS56YYWSC4zK7eJmzU4srBAQUGyU/DNtxq7s6Fdx86UQNLYdEMLJDPtda0KpFYkkDQ23dAC\nSZJ2KoEkSQAJJEkCSCBJEkACSZIAEkiSBFATIE2fVVR/eLeB/X5vv6EPzuNnqgWQZs92rTy8\nW5kff+xo6Oej3hxXeYYE0vboAqnmwALJVo5tdgXJcWhvhgWShXYJku+BikBKUCsgeW4V/A74\n/QZ3PtkgkKzkvlb3NTjF0AIJL8+zwAT/Nu9zaIEEl++UaovkM7RAQst5RgWSz9ACCSzP3avL\n+G4D686GZtQCSJJEL4EkSQAJJEkCSCBJEkACSZIAEkiSBJBAkiSABJIkASSQJAkggSRJAAkk\nKg1aH41KK45KAqlVacVRSSC1Kq24SvrzYxh+/DldWPk+vP8+f+/3+XvXV9+Hw8d14cf1ldSW\nBFIlHYYvfTudWfnCZzh8MfXn8r3nq+/nhd/Pr0RScxJIdfT3GY6P4Z8zK+9/Tu/XL99P91c/\nTj/Pu3WXhX8Pbb0VRzoJpFr6dpno60bn19eu3Hnj9G34/Xj15/q/Dedv6UipQWmV1dFw052S\npVen8bektqRVVkcCqXNpldXRt8dED9cduveVXbvnn1JL0iqro4/zOYX/nfEZvv748z78PT3Z\n8HH6Nd1ISW1Jq6yOrie4z+cZvkA6n/Q+jU9//76fHBdIrUqrrJLOF1/ff54uu3bv98uwjwuy\nv96vrwRSq9Iqqy1R0qW0VmtLIHUprdXaEkhdSmu1tgRSl9JalSSABJIkASSQJAkggSRJAAkk\nSQJIIEkSQAJJkgASSJIE0P8BSWtcMFtiPkAAAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(history_rnn)\n",
    "history_rnn$metrics$val_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_embedding(input_dim = max_features, output_dim = 8) %>%\n",
    "    layer_lstm(units = 8) %>%\n",
    "    layer_dense(units = 1, activation = \"sigmoid\")\n",
    "    model %>% compile(\n",
    "    optimizer = \"rmsprop\",\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"acc\")\n",
    ")\n",
    "\n",
    "history_lstm <- model %>% fit(\n",
    "    mtx_train, y_train,\n",
    "    epochs = 10,\n",
    "    batch_size = 32,\n",
    "    validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.806</li>\n",
       "\t<li>0.823</li>\n",
       "\t<li>0.8218</li>\n",
       "\t<li>0.8384</li>\n",
       "\t<li>0.8352</li>\n",
       "\t<li>0.8328</li>\n",
       "\t<li>0.8372</li>\n",
       "\t<li>0.835</li>\n",
       "\t<li>0.8272</li>\n",
       "\t<li>0.8252</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.806\n",
       "\\item 0.823\n",
       "\\item 0.8218\n",
       "\\item 0.8384\n",
       "\\item 0.8352\n",
       "\\item 0.8328\n",
       "\\item 0.8372\n",
       "\\item 0.835\n",
       "\\item 0.8272\n",
       "\\item 0.8252\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.806\n",
       "2. 0.823\n",
       "3. 0.8218\n",
       "4. 0.8384\n",
       "5. 0.8352\n",
       "6. 0.8328\n",
       "7. 0.8372\n",
       "8. 0.835\n",
       "9. 0.8272\n",
       "10. 0.8252\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] 0.8060 0.8230 0.8218 0.8384 0.8352 0.8328 0.8372 0.8350 0.8272 0.8252"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2dC3fbxraD6Spu0t62SY70/3/rlUa2\nRNF8zAPgYIbAWidHtorsnT34PBRFU8PFsqxiDbUbsKweZJAsCyCDZFkAGSTLAsggWRZABsmy\nADJIlgWQQbIsgCgg/VrR6pNbsnlPs2bfjMACZJBsFixtkIJqzNdmuFmzb0ZgATJINguWNkhB\nNeZrM9ys2TcjsAAZJJsFSxukoBrztRlu1uybEViADJLNgqUNUlCN+doMN2v2zQgsQAbJZsHS\nBimoxnxthps1+2YEFiCDZLNgaYMUVGO+NsPNmn0zAguQQbJZsLRBClqZ0DAMnPnaDDdr9s0I\nLEA7gzScz+cCkiRXtluzZt+MwAK0L0g3jkpIklzZbs2afTMCC5BBslmwtEEKWhyCQWrJrNk3\nI7AA+TWSzYKlDwXS6aq5xz5r14lZs++StBOVD9Lp8cfr44vfR+rErNl3SdqJMkg2C5Y+NEhP\n1ZivzXCzZt/ZgeUKBdLjNdIfVxV3ZVmNCQTS6eJDu+7Mmn0XxZ0nv0ayWbC0QQqqMV+b4WbN\nvkvSTpRBslmwtEEKqjFfm+Fmzb5L0k4U4MqG0+jxXTXmazPcrNl3YeBZ8m/I2ixY2iAF1Ziv\nzXCzZt+MwAK0O0jnM2e+NsPNmn0zAgvQ3iDdfh+JMl+b4WbNvhmBBajCjmSQGjFr9s0ILEAG\nyWbB0gYpaG1ERcd2kivbrVmzb0ZgAdr/rJ1BasWs2TcjsAAZJJsFSxukoNURlRzbSa5st2bN\nvhmBBajCG7IGqRGzZt+MwAJkkGwWLG2QgtZHVHBsJ7my3Zo1+2YEFqAa19oZpDbMmn0zAgtQ\nJZAySZJc2W7Nmn0zAgtQlau/DVITZs2+GYEFyCDZLFjaIAVtjSj72E5yZbs1a/bNCCxAdX6x\nzyC1YNbsmxFYgAySzYKlDVLQ5ohyj+0kV7Zbs2bfjMACVOmeDQapAbNm34zAAlQPpBySJFe2\nW7Nm34zAAlTrLkIGSd+s2TcjsAAZJJsFSxukoIgR5R3bSa5st2bNvhmBBajaDSINkrxZs29G\nYAGqCVI6SZIr261Zs29GYAGqd8tig6Ru1uybEViADJLNgqUNUlDUiHKO7SRXtluzZt+MwAJU\n8dMoDJK4WbNvRmABMkg2C5Y2SEFxI8o4tpNc2W7Nmn0zAgtQzQ8aM0jaZs2+GYEFqDJIiSRJ\nrmy3Zs2+GYEFqOpHXxokabNm34zAAkQBKVZXkGqWtyyYau9IaVuS5I/Ibs2afTMCC1DdTzU3\nSMpmzb4ZgQWoOkhJJEmubLdmzb4ZgQWoLkipW5LkynZr1uybEViADJLNgqUNUlD8iBKP7SRX\ntluzZt+MwAJUGaTELUlyZbs1a/bNCCxAAiAlkCS5st2aNftmBBag2iClbUmSK9utWbNvRmAB\nMkg2C5Y2SEEpI0o6tpNc2W7Nmn0zAgtQdZCStiTJle3WrNk3I7AAGSSbBUsbpKCkEaUc20mu\nbLdmzb4ZgQWoPkgpW5LkynZr1uybEViANECKJUlyZbs1a/bNCCxAAiAlbEmSK9utWbNvRmAB\nMkg2C5Y2SEGJI4o/tpNc2W7Nmn0zAguQAkjxW5LkynZr1uybEViAVECKI0lyZbs1a/bNCCxA\nEiBFb0mSK9utWbNvRmABMkg2C5Y2SEHJI4o9tpNc2W7Nmn0zAguQBkixW5LkynZr1uybEViA\ndECKIUlyZbs1a/bNCCxAIiBFbkmSK9utWbNvRmABMkg2C5Y2SEEZI4o7tpNc2W7Nmn0zAguQ\nCkhxW5LkynZr1uybEViAlEDaJklyZbs1a/bNCCxAMiBFbUmSK9utWbNvRmABkgJpkyTJle3W\nrNk3I7AA6YAUsyVJrmy3Zs2+GYEFyCDZLFjaIAXljSji2E5yZbs1a/bNCCxAQiBFbEmSK9ut\nWbNvRmABEgNpgyTJle3WrNk3I7AAKYG0vSVJrmy3Zs2+GYEFqACk01Xjx48vDFIfZs2+i+LO\nUz5Ip8cfo/+/K3e+m8d2kivbrVmz7+zAciUF0uaWJLmy3Zo1+84OLFcYkF45KgJplSTJle3W\nrNl3dmC5AoH0fIn0x1X57VxByjdbVjUBd6Tykw2bx3aSPyK7NWv2XZJ2okCvkV4f589349hO\ncmW7NWv2nR1YrsRA2tiSJFe2W7Nm39mB5Urs0G5jS5Jc2W7Nmn2XpJ0oHEijzalkvgZJxqzZ\nd0naiQJc2XAaPb6rZL6rW5LkynZr1uy7MPAsSV1rF2SQVMyafTMCC5BBslmwtEEKKprv2rGd\n5Mp2a9bsmxFYgPRAWtuSJFe2W7Nm34zAAqQJ0hJJkivbrVmzb0ZgARIEaWVLklzZbs2afTMC\nC5AoSAskSa5st2bNvhmBBUgRpOUtSXJluzVr9s0ILECqIM2TJLmy3Zo1+2YEFiBJkBa3JMmV\n7das2TcjsAAZJJsFSxukoOL5Lh3bSa5st2bNvhmBBUgTpKUtSXJluzVr9s0ILEC6IM2RJLmy\n3Zo1+2YEFiBRkBa2JMmV7das2TcjsAAJgzRDkuTKdmvW7JsRWIBUQZrfkiRXtluzZt+MwAKk\nDNJXkiRXtluzZt+MwAIkC9LsliS5st2aNftmBBYgaZC+kCS5st2aNftmBBYgXZDmtiTJle3W\nrNk3I7AAaYM0JUlyZbs1a/bNCCxAwiDNbEmSK9utWbNvRmAB2hukYRiip/l1S5Jc2W7Nmn0z\nAgvQziANb29vSSRFz3dbNrdT2iAFLQ7hxlECSQapqlmzb0ZgAVIH6ZUkyZXt1qzZNyOwAEmD\n9GVLklzZbs2afTMCC5D0a6QvW5LkynZr1uybEViAlM/a/fqyJUmubLdmzb4ZgQVI+X2kX1+2\nJMmV7das2TcjsADtD9Jb0kQNkkGaPKWp3UG6vkhKmejrliS5st2aNftmBBagGiClkhQ1323Z\n3E5pgxS0OqK3NJRetiTJle3WrNk3I7AAVTjZkE5SzHy3ZXM7pQ1S0MaI3pJQGm9JkivbrVmz\nb0ZgAapz+juVpIj5bsvmdkobpEgFkiL/29uWRG3GsopV7Q3ZhE3puSVJ/ojs1qzZNyOwANW7\nsiGeJINUx6zZNyOwAFW8RCj6nMPzdIPkynZr1uybEViAql5rl0DS5ny3ZXM7pQ1SUPSIIkl6\nbEmSK9utWbNvRmABqn31dxxKBqmGWbNvRmABqg1SHEmfW5LkynZrrlb67c0gBSXNN+qcg0Gq\nYK5T+n43guVf/2QEFqD6IEWh9LElNZrIRs27l34baZEkRmABUgApkqQFc7Rs3tOddk+Bt69a\nsjMCC5AESBEo3bekRhPZqLnEnXKXmwlBG/eaYgQWIBGQtlEySLubC9yx912bIBRjZgQWIBmQ\nfr2toxS2pEYT2aiZCtJ0G5q6/Rrpkn0XoU2SWk1ko2YeSGsUBbvP2gWtjXj1yTWUwm9TrJm3\nZPOO7sVNZQOh7cqMwAKkBdIqSgZpZzP4rN3WNhRZmRFYgNRAWiFp9nPOE9RonBsFaWJOoWi9\nMiOwAMmBtHLRkEHa1wwqnYbQZmVGYAFaBOnv0+Xy73D6K+cvzRzRQwsTL9ySGo1zuyC9JW5D\ncZXzs07VEkh/D8Pl5+l6oJtDUuaInlomKcZdUtlmhHuKUBpF65VL0k7UEkjfhn+v//v7v+GU\n8Zdmjmik+eGXbUmNxrk5kMoQ2qxcknailkC6bkj/DN/C/6crc0Qvml0Eg7SnOd1dTtB25ZK0\nE7XEyWn4+X347/YqKeMvzRzRq+YWo2hLajTOTYA0swux+i5JO1FLIP11fXl0um1IPzL+0swR\nTfUVpYtB2tEcf2JobicySHf9GE7/XDemHI5QIH1F6VKyJTUaZ2WQ1l4OGaRyZY5oRpMFupS8\nSmo0zqIgbb4cMkjlyhzRrF6W6lLyKqnROKuBFHtSziDdVfMN2Ve9voA1SLuZ16/y2ToxZ5CC\n6r4hO9Fj2S4lJ+4ajbMASIkIAUr3A1LlN2S/6L56N7NB2st8dycThCjdD0jV35D9os9lNEh7\nmcveXzVIQfXfkP2q+3pmH9s1Guca5txdCFB601ySdqKU35D9ordPlLLczcW5gjnr1RCmdKy5\nJO1Eab8h+8X8iVKWuahy9+YZhCT7zs86VfLvI03N+T8mJWMhYp7fhCT7ZgQWoOZAykdJMhb1\nzSuHcZJ9MwIL0CJIv398G4ZvP37n/KXs+eaRJBmLiubN10Jpdx1OKV1izs86VUsghTdjbycc\nfmb8pez5ns85KGnGmWqeRyHyjELKXYdnZJCCvg/vV4R+vg/fM/5S+nzPOSi1yUKJeYrCHEGL\nI4y96/CSDNL9+8Pr/8/odNX4y+dD+nzPD5ISUGqShRLzE4Ukgr6482SQ7t/fBOn0+OP+5Z4g\n3a+4i0zEhzSP+JnmYZaf6HkZpBTlH9q9gnTadUf6vFAoIRqiR/wscwFBnxKdGCj4aOWfbHgB\n6bTvod3zIvDYjKj+fN1U6kYKICi39KsM0l2bp78XQPrjKlx7y7rdCfyuz7Ss/ucfIPH7AmuI\n7nqWoKxrjq0M5Q96DNLpssuONP4ROb52NeInb6M70mrbCy+BxpMA/BqFmrkk7URBQJqcd2CB\n9HrQ/nIV+PZRjOgR/4YmIK2hM/vvN0h7aRakYawl5xikux5PceY7/ek8+XWKLZY0j/g39Plv\nTiaouHKx2yClg3SZPOTMdwakyVXgxFDVMiejA6tc6jZIkaoO0uzvyq6kTDIWy8pEB1AZ4zZI\nsfo8mhudcPgQab7TlzkLvyu7FDvJWCyomKDsyjC3QSoXa77TlzlLv3U+H0DJWMxoQlC9tjUn\nxggsQC2BNFy5+ULSwn8789NcMhZTfe3bIE2e0lRDIN04mpC0eiOUaSYlYzHW/LGcQZo8pam2\nQdq6N9dLNCVj8dDiCyKDNHlKU+2DtHUjFMBL9l/kTK2eVjBIk6c01RBIM6+RIu9gDDgBxsvU\nVmsGafKUploCafbihNjbRRaeT+bEIqYjgzR5SlNNgTRnjr/x6vT66NLKpebIVgzS5ClNNQ9S\nPEkf5rc5ZVWO1hdzSm2DNHlKU+2DFH1wNzbP0rQWbVTb6buiQZo8pak+QIoiaWJeZ2kadshL\nu3SKCisbpN3UAUixJK1VjqYqDYKlvznavdE212yQEtQDSJEHd/GVk7FiEJjeNtpskBLUC0gR\nJOVWrklRQdvlZoOUoC5AituSoO9gxbPT6N0i6pY2SEG7zzdqSyq5pqIEhTbvFlG3tEEK2n++\nMSQVXOVXtqk0ebeIuqUNUlCF+SqDpBlI6dIGKajCfCO2JIPUTmmDFFRjvtsk1XqNJBpI6dIG\nKajGfLcP7qBn7ZIkGUjp0gYpqMZ8t7ckyVgomzX7ZgQWoH5A2iRJMhbKZs2+GYEFqCOQtg7u\nJGOhbNbsmxFYgDoDaY0kyVgomzX7ZgQWoJ5A2iBJMhbKZs2+GYEFqCuQ1kmSjIWyWbNvRmAB\n6guk1ZdJkrFQNmv2zQgsQP2BtEiSZCyUzZp9MwILUGcgrZEkGQtls2bfjMAC1BtIKwd3krFQ\nNmv2zQgsQD2CtECSZCyUzZp9MwILUHcgLZMkGQtls2bfjMAC1B9IiyRJxkLZrNk3I7AAdQjS\n0sskyVgomzX7ZgQWoB5BWiBJMhbKZs2+GYEFqFeQZkiSjIWyWbNvRmABooBUXTeSavdgHUpd\n7kjzB3eSP1+VzZp9MwILUKcgzZEkGQtls2bfjMAC1DFIU5IkY6Fs1uybEViAegVphiTJWCib\nNftmBBagbkH6SpJkLJTNmn0zAgtQvyB9eZkkGQtls2bfjMAC1DFIU5IkY6Fs1uybEViAOgdp\nTJJkLJTNmn0zAgtQzyBNSJKMhbJZs29GYAHqGqRfBqnErNk3I7AA9Q3SC0mSsVA2a/bNCCxA\n/YP0IEkyFspmzb4ZgQWoc5DGJEnGQtms2TcjsAD1DtKIJMlYKJs1+2YEFqDuQfplkBosbZCC\nasx3WZ8kScZC2azZNyOwAB0DpHOdyo2bNftmBBagA4D0SZJkLJTNmn0zAgvQEUD6ZZBaK22Q\ngmrMd1WBJMlYKJs1+2YEFqDDgHTWjIWyWbNvRmABOgZIgaRhqFG5YbNm34zAAnQQkAJJ5wKS\nJDNFNmv2zQgsQEcBaSgkSTJTZLNm34zAAmSQ2JXbNWv2zQgsQMcCaeUzz2mV2zVr9s0ILEBH\nAemDpBqVmzVr9s0ILECHAenXMAwFKElmimzW7JsRWICOA9LVXLApSWaKbNbsmxFYgA4F0uqH\nnpMrN2jW7JsRWIAOBlI2SZKZIps1+2YEFqCjgZRLUu22a5g1+2YEFqDDgZT5Qql62xXMmn0z\nAgvQ8UDKI6l+2/ubNftmBBagA4KURZJA27ubNftmBBagI4KUg5JE2zubNftmBBagY4KUTpJG\n2/uaNftmBBagg4L0K/XSO5G2dzVr9s0ILEBHBSkVJZm2dzRr9s0ILEAFIJ2umnvcCEhpJOm0\nvZ9Zs++StBOVD9Lp8cfr40srICWhpNT2XmbNvkvSTtShQfpEKYIlrbb3MWv2XZJ2ojAgXV4f\n15hvpvl8jmNJrO1dzJp9ZweWKzhIf1xV1NHeeqB0rt2J1bBQIDV4suGpiG1JsW22WbPvgrAz\ndfRDu09tsSTaNtWs2Xd2YLkySA+tsqTbNs+s2Xd2YLk6+Fm7V52XWVJum2XW7Lsk7UQZpFed\nF2ASb5ti1uy7JO1EAa5sOI0e31VjvjjzLEv6bePNmn0XBp6l415rt6YvLA1HvAO/Zt+MwAJk\nkJY0Rul2R7zj3YFfs29GYAEySCs6v+hwNw7X7JsRWIAM0roMkpqZEViADNKWDJKWmRFYgAxS\nhCIvbCVUrmrW7JsRWIAMUpRiLxLHVzZI06c0ZZAizStXPZArVzNr9s0ILEAGKd6czVKj/2bN\nvhmBBcggJZnzUKrednOlDVJQjfnuZs5gSaHttkobpKAa893TnMqSSNsNlTZIQTXmu6857eWS\nTNvNlDZIQTXmu7t56Rcu+JX3Mmv2zQgsQAapwBzLkljbDZQ2SEE15lvJfI6BSa9t9dIGKajG\nfCuaN1nSbFu5tEEKqjHfuubz6s4k27ZsaYMUVGO+9c3n8xJO0m1LljZIQTXmK2FeYEm9bb3S\nBimoxnyFzF9YaqNtpdIGKajGfNXMI5ZavXOK5rgZgQXIIPHMo42pyTunaI6bEViADBLVnHD5\nA7gywKw5bkZgATJIbHPUW7aUyqVmzXEzAguQQWKbh8Wz4uzKpWbNcTMCC5BBopvvd5fMpskg\nTZ7SlEHimx9n7ZbfsiVVNki7ySDta05nySBNntKUQaphTqHJIE2e0pRBqmaO3JwM0uQpTRmk\nmuYYlgzS5ClNGaT65nWaDNLkKU0ZJBHz4uZkkCZPacog6ZhnWap4xavmxBiBBcggqZlfaar5\nWYGaE2MEFiCDJGk+z+5Oe1SGuQ2SJaEpS1fVbslalnckafNQuDV5R9pLBknb/PEaKZcmg7SX\nDJK4eXTWLmN3Mkh7ySA1ZZ554bTKk0HaSwapQfMcToTri7T+0Y+nNGWQ2jVv81T2bq7kP9og\nked7XPMyT4Xv5mr+oxmBBcgg9WKe5amMJMl/NCOwABmkzszxr5/gpfcxMwILkEHq0wzhSfIf\nzQgsQAapY/OX6yISeZL8RzMCC5BB6tn8edYukyfJfzQjsAAZpAOZU3lS6fv1KU0ZpOOZo3kS\n6/vjKU0ZpMOat3GS7JsRWIAM0tHNyzxpfq5TTKhfUv33iZHxLzUZf2mN+dpcZp4/wSf4bm5M\nqIflr1gySDaPlHl6D1E61hwTaoOUP1+bgWYET3VA+vk+/HlH598/h+H048bRy5csGSSb57Xw\nbm40VlVA+n26YvPnjZx/hqAfHyA9vmTJINm8oMe143k8VQHpx/B++f1+I+fb8H+Xy3+3R2FD\nen5JkkGyeUmTs3ZpPNFO+a0m79vw83p4d+fl5z9/vT9Aen5JkkGyOcu9wRPvxpbrcR4ef77f\nD+Y+v/f4kiSDZHOJe/V1VD5J5SB9H779/c/PB0jPL0kySDYD3EWnJdIqrybveWgXmPn9ePT8\nkiSDZDPODeMpF6S/hvffl49XRv9+nHb4AOnzS5IMks0M98bJ8y2syk9//xier5FO4y9JMkg2\nU9zPs3Y5POW/Ifvn5xuy34fh/d/bo79vID2/JMkg2bxP6Y0t6gWrtXPnjMACZJBsrlF6lafV\nc+eMwAJkkGyuWHoFpyWSGIEFyCDZLFPaIE0Enq/Ndcy1ShukT3Hma/PO5qoXnvs10sUgdWKu\neOG5z9oFseZr865mzb4ZgQXIINksWNogBdWYr81ws2bfjMACZJBsFixtkIJqzNdmuFmz79Xk\n/W9ZjKCPVADS6aq5xwapE7Nm36uZbBGk0+OP18cXg9SJWbPv1VAapOL52gw3a/a9GsrWQbq8\nPq4xX5vhZs2+V0PZEUh/XFXUkWVlqweQfLKhN7Nm36uhNEjF87UZbtbsezWUHYA0PsirMV+b\n4WbNvldD2T5ILx/mVGO+NsPNmn2vhjITpGH2YZJAIO3yoWiWtaExOcMwLIDEuJwHcGXD6f7w\nZJis2hpzdPvtwCZAsiw1/e9/C7dSGYMUbhQ5XO43j7zfh/USvp48vNz/uzgZJKsjRYF0edwM\nfPj4anh+/fow/nMzDZLVkRJAGn81vDL1fBgPiEGyOlL8a6SP4H/cHtwgWdZI8WfthvEjg2RZ\nY8W9jzR+eWSQLOuLMkBaP7Sre7KhxhveNsPNmn2vJi8SpMdp7Y9H8yDVP/1dY742w82afa8m\nLw6kFBkkm0vNmn2vJg8KUv3XSDXmazPcrNn3avKwO1LKZ2UaJJsFS2uAlCKDZLNgaYMUVGO+\nNsPNmn2vJq8zkCyrjjoDqcYPKpvhZs2+V5NnkIrnazPcrNk3I7AAGSSbBUt7RwqqMV+b4WbN\nvleTZ5CK52sz3KzZ92ryDFLxfG2GmzX7Xk2eQSqer81wc73S2Z9qbpDiZPOe5mqlw2+JL/69\nazJIcbJ5T3Ot0vfblSyRtJo8gxQnm/c0M0sv3OxnJINkkDoxg17mbDNjkBbEWhybdzWvveRf\nVSY6n9r7NdLwxGD48kysDJLNC1qN80TpsKyXRpy1W74d10Tzn0WRSIZBsnlew+oBVgI64L5X\nkzfm6O3tbeEGkRMZJJuJ5i8gpaCzQWFJ36vJ+9//3ub1chP9zz/Ht9B/3jPo4+arr18+76y/\nIINk87yGeHBmdp2U48K0vleTlwTS6x24Rve4e349+nJ0n+M5GSSbRypAZ6LsMxVBVJBebqL/\nAtLkuym3XTVIRzfD0EkvnWVeTV7ca6TnnVZHt9Af3cP487sGyeYI8zY6kn2vJi/yrN3zsG5m\nRxq/bDJINs+YU3cdlb5fn1pT5PtIw8vtv7/e8dsg2Txrzjxiq9737FNrin1DdnR777mTDT60\ns/lDkNc8kv/o1eQlgTS5hf7ofPfHufHX098Xg3Qkcxk6RaX3Ma8mLxYkvAxS82bI5pNXuoZ5\nNXkGqXi+vZpX3o/ZQqf08gLNia0mzyAVz7dT8+sVAmmbj0EySDbf9clC1nGbQTJINt+1esFb\nlJt0oU4982rymgLpdNX08fh7BglhzkNnorIL3jQntprNlkA6Pf54Ph5/72KQiswQgrIqY90G\naUMGiWSeJ4h2FTXbXQWkejJI9c3re1C9tjUnVp55iuAg/XEVprX+NUtQ7aasLJWDdPKOlGxO\neR3kHWnylKbKQLqdrfOhXYI540yCQZo8palCkD4eG6Q1hfMF2efiDNLkKU35ZAPdPPOm6k6V\nDdJ+Mkhkcz5BpZWLzZrjLs88RQVXNpzGj31lwxcV7EKFlUFmzXFDYo+Xr7XDm+deDlHulUg2\na46bEViADBLSPHtCgXevRLJZc9yMwAJkkGDmxTMKFa/yMUh7ySAhzBuvhlTb1i1tkIJqzLeS\nOeqMgl7b6qUNUlCN+dYwx56XE2u7gdIGKajGfPc2p5zaFmq7kdIGKajGfHc1J749pNJ2O6UN\nUlCN+e5mzniTVaHttkobpKAa893HnHelQvW2myttkIJqzJdvztiKQJUrmTX7ZgQWIIMUpSEb\notLKBmn6lKYM0qZSzs9hK1c2a/bNCCxABmlZi9f80CtrmDX7ZgQWIIMUNGVmTi1ewG2Q9tLh\nQdoGqPwm2mr/Zv3SBimoxnyzzFsEfarZ34QwSHvpwCClvPxp9TchDNJeOixIiacQVNre06zZ\nNyOwAB0VpNQTcSJt72rW7JsRWICOCVL66WyJtnc2a/bNCCxARwQp510hgbZ3N2v2zQgsQAcE\nKYcjgbb3N2v2zQgsQMcDKQcjgbYrmDX7ZgQWoMOBlIVR/bZrmDX7ZgQWoKOBlMlR7barmDX7\nZgQWIApIurpxVLsHq0cdakfKe3mEqNykWbNvRmABOhxIdSo3adbsmxFYgI4D0jDkYySaKbJZ\ns29GYAE6DEhDwXFdWeVmzZp9MwIL0FFAOuivFBmkvWSQ2JXbNWv2zQgsQEcB6aC/LW6Q9tJB\nQCrlSDNTZLNm34zAAnQMkAJFR/wlV4O0lw4DkmgslM2afTMCC9AhQLqf95aMhbJZs29GYAE6\nAkhng9RaaYMUVGO+y/p8I1YyFspmzb4ZgQXoGCBVqty4WbNvRmAB6h+kx4VBkrFQNmv2zQgs\nQN2DdDZIDZY2SEE15rug0ZWqkrFQNmv2zQgsQJ2DNL7iWzIWymbNvhmBBah/kCpV7sCs2Tcj\nsAD1DdLLbyBJxkLZrNk3I7AAdQ3S2SCVmDX7ZgQWoJ5BmvxKrGQslM2afTMCC1DnIFWq3IdZ\ns29GYAHqGKTpLRokY6Fs1uybEViA+gXpy71OJGOhbNbsmxFYgLoF6es9gyRjoWzW7JsRWIB6\nBWnm3luSsVA2a/bNCCxAHYNUqXI/Zs2+GYEFqFOQ5m4GKRkLZbNm34zAApQB0umq6ePT+Jv1\nQZq9qapkLJTNmn0DQoVOpVYAABOPSURBVM9QOkinxx+jx6eX/6TGfF80e3NiyVgomzX7Lgw8\nS12CNH+Xb8lYKJs1+y4MPEsYkF45qg3Swt3yJWOhbNbsuzDwLIFAer5E+uMqUG+Z8qfyWfur\nDKTPkw0v36u9Iy19eovkz1dls2bf5ZmnCPQaafz/lUFa/BgkyVgomzX7Lgw8S92BtPxxYpKx\nUDZr9l0YeJaAJxtkQKpUuTuzZt/lmacIB9LozF2N+X5o5fMtJWOhbNbsuzzzFBVc2XAaPx5f\n2FARpLXPiZWMhbJZs29I7PHq7Fq7tc9bloyFslmzb0ZgAeoLpNUPLpeMhbJZs29GYAHqCqRV\njjRjoWzW7JsRWIB6A6lS5S7Nmn0zAgtQTyCtb0iasVA2a/bNCCxAnYHEqnzIT3LW7JsRWIA6\nAmljQyqpPLy9vRWQJBlI6dIGKajGfDc5Kqh846iEpGqBLNtIDVKCmgJpNRdbHB0QpMKN1CAl\nqCWQhisri7nY3JDqgVTrBVYp/wYpQQ2BdONokaRtjnb40f6WqKjasW2j6j2leYaFEViAOgJp\ny884qExFJyPmy21Ty8qeYWEEFqBOQIrYkEAri0mvqsbjDt/Qe2HICCxADYG08hophqPsylkx\nHHW9HEgWA8/arxsp8G/elkEqF2u+iwftMRylVwZlLPYQiRHwiH8zCyuDVK695xu1IYFCldZw\nUMGL9r2Pr4A8GaRy7T3fKI5WK29GptKZ4KovVD5/AORhZZDKlTmi3PnGbUjz5th0NPqeKuES\noVisDFK5MkeUOd9Ijibm2B+sa5Vj1ejlstl7+OY0SyozAgtQHyClmPPWXPJNfrI5wY3mySAF\nZY4ob76xG1ImQSuVo9WomfKGQXFlRmAB6gKkDRfi52SjLFT7NYphDabt8RukoMwRbWvGvLoh\noY40mmWhFkjjs405PBmkoLURg3OxwtF00RqNc/MgvSqKp7UzLIzAAtQBSDP/3ewyNRrnzkD6\n0CpPq+f8GYEFqCmQvv6gmm5Iaz/qGo1zkyAV/eLJOoWMwAJEAYmksDiv33p+ptjMguzeoPXU\n9Yde/H88ixOvN4Ya2pFmflB9bkhbx9zBLvlrasrm3Uu/grT092qqaZBuHG0T9HQL/pqasrlW\n6VWODFLxfKcgRexCy+ZUNcpCkyD5rN2nSPMdbypJFP0ySI2V9vtIQaz53n9QJSL0YTVILZU2\nSEG0+b4yFHuRXZBfI7VU2iAFceY73YmSOPJZu6ZKG6Qg/HxnDuair/ouq3xgs2bfjMAC1AJI\n86+IUjnSjIWyWbNvRmABEgdpcjQ3MidvSJqxUDZr9s0ILECyIM2e334FKfJvSq1sc/3SBimo\neL5L7xI9zekbkmYslM2afTMCC5AeSGvvEr2AtPUXJVe2Waa0QQrKn+/GO60Pc8aGpBkLZbNm\n34zAAiQEUsT1CmOQ1lciqbLNaqUNUlDGiOIu+vk052xImrFQNmv2zQgsQAogRWxFE3MOR5qx\nUDZr9s0ILEDVQYqGaGTO2pA0Y6Fs1uybEViAqoIUvxW9mrM40oyFslmzb0ZgAaoHUipET3Pe\nhqQZC2WzZt+MwAJUB6TkrehlvnkcacZC2azZNyOwAO0N0jAMmRA95pu5IWnGQtms2TcjsADt\nDFIJRb+eIGVYRWOhbNbsmxFYgPYFaSiA6DFfg7STWbNvRmABqgFS2Z0Tco/sNGOhbNbsmxFY\ngJoEKd9cVPloZs2+GYEFqMJrpLJbkGRvSJqxUDZr9s0ILEAVztqVzTebI81YKJs1+2YEFqAM\nkE5XTR+Pv0f9WJf8DUkzFspmzb7LM09ROkinxx/Px+PvXdgg5ZuLKh/PrNl3eeYpMkg2C5Y2\nSEG8+RYc2WnGQtms2Xd55imCg/THVZjW5vT8YDHLUlIZSJ8nGfbakUo2JM2fr8pmzb7LM09R\nW4d2JRxpxkLZrNl3eeYpagqkog1JMxbKZs2+yzNPUWsgFbglY6Fs1uy7PPMULYL0Z3hm+PZz\n+kQ9kMo2JM1YKJs1+waGH6klkH7cP919GL5/eepxNcP48R5XNpRxpBkLZbNm32AAUFoC6TT8\ne/u//4aci/E48y3ckDRjoWzW7Ds/61QtcfIJkBZIkivbrVmz79ykk7XEyZ/D99+Xy+8fw3vG\nX8qZr0Ha2azZd0naiVoC6edpCDr9l/GXUuZ7Nkg7mzX7Lkk7UYtHbr9/fBuGbz++nLSLEWW+\nt1dIkivbrVmz7/ysU1X9lsWxOhukvc2afTMCC1BLIImubLdmzb4ZgQUo/Q3ZCBHmezZIu5s1\n+86OOlcZb8huizDf+3tIkivbrVmz7/ysU9XIG7Jng7S/WbPv/KxT1cgbsh8XNUiubLdmzb5z\nk05WG2/Ing1SBbNm3yVpJ6qNN2Q/r7KTXNluzZp9l6SdqDbekDVINcyafednnaom3kc6G6Qa\nZs2+GYEFqBWQ8s1FlQ9t1uybEViAFkH6MXwo4y8Fz/dskKqYNfsuCDtTK2/IKoGUby6qfGyz\nZt8laSdq+Q3Z/96Hn7/f7+/LJgo737NBqmPW7Lsk7UStvCH71/DP5bfA+0ij3zCXXNluzZp9\nl6SdqBWQ/hn+lriywSBVMmv2XZJ2opavbPi/n8O3y7/1QTobpEpmzb5L0k7UEic3gt5v5xqq\nX/09vneQ5Mp2a9bsuyTtRC1uOP98u1y+D8OPnL8UOd+zQapl1uw7P+tUyb8h+3IzO8mV7das\n2TcjsACpg3Q2SNXMmn0zAgsQBSSg/MliVhNqYEfKNxdVtlmzb0ZgARIH6WyQDNLkKU3pg5Rv\nLqpss2jfjMACpA3S9AMoJFe2W7Nm34zAAiQPUr65qLLNVUsbpCDUfL98IpLkynZr1uybEViA\n1EHKNxdVtrluaYMUhJqvQapq1uybEViAlEH6+lmXkivbrVmzb0ZgARIHKd9cVNnmyqUNUhBm\nvjMfviy5st2aNftmBBYgbZDyzUWVba5d2iAFYeZrkCqbNftmBBYgXZBmjuw0V7Zbs2bfjMAC\nJA1Svrmoss3VSxukIMR85zYkzZXt1qzZNyOwACmDlG8uqmxz/dIGKQgxX4NU3azZNyOwAKmC\nNHtkp7my3Zo1+2YEFiBhkPLNRZVtFihtkILK5zu/IWmubLdmzb4ZgQVIF6R8c1FlmxVKG6Sg\n8vkaJAGzZt+MwAKkCdLCkZ3mynZr1uybEViAZEHKNxdVtlmitEEKKp6vQVIwa/bNCCxAkiAt\nHdlprmy3Zs2+GYEFSBWkfHNRZZs1ShukoML5Lm5ImivbrVmzb0ZgAcoA6XTV6GH44jT6JgKk\nfHNRZZtFSh8BpNPjj9E3Ti//SeF8DZKGWbPvkrQTVQ7SF7BKQVo+stNc2W7Nmn2XpJ0oDEiv\nHJWDlG8uqmyzSukDgnT/8vkS6Y+rijo6+0P6rPYEAun1e0U/qFY2JM0fkd2aNfsuDDxLGJAm\nj4rma5BUzJp9l6SdqFKQpmcdgkrmu3KqQXRluzVr9l2SdqIwIAEP7dY40lzZbs2afRcGniUc\nSKO9qWS+BknGrNl3YeBZKriyYUzUy4UNJSCtHtlprmy3Zs2+yzNPkdq1dqscaa5st2bNvhmB\nBUgMpPUNSXNluzVr9s0ILEB6IOWbN2RzO6UNUlD+fA2SkFmzb0ZgAdICaePITnNluzVr9s0I\nLEByIOXOd1s2t1PaIAVlz9cgKZk1+2YEFiApkLaO7DRXtluzZt+MwAKkBlL2fLdlczulDVJQ\n7nwNkpRZs29GYAFSAmnzyE5zZbs1a/bNCCxAYiDlz3dbNrdT2iAF5Y1oe0PSXNluzZp9MwIL\nkBZIBfPdls3tlDZIQXkjMkhiZs2+GYEFSAekiCM7zZXt1qzZNyOwAEmBVDLfbdncTmmDFJQ1\nIoOkZtbsmxFYgGRAijmy01zZbs2afTMCC5ASSEXztRlu1uybEViADJLNgqUNUlDGiKKO7DRX\ntluzZt+MwAIkBFLZfG2GmzX7ZgQWIINks2BpgxSUPqK4IzvNle3WrNk3I7AA6YBUOF+b4WbN\nvhmBBYgCUrr84WJW29LYkSKP7DR/RHZr1uybEViAZEAqna/NcLNm34zAAmSQbBYsbZCCUkcU\ne2SnubLdmjX7ZgQWIBWQiudrM9ys2TcjsAApgBS9IWmubLdmzb4ZgQVIBKTy+doMN2v2zQgs\nQAbJZsHSBikobUTxR3aaK9utWbNvRmAB0gAJMF+b4WbNvhmBBcgg2SxY2iAFJY0o4chOc2W7\nNWv2zQgsQBIgIeZrM9ys2TcjsAAZJJsFSxukoJQRpRzZaa5st2bNvhmBBUgBJMh8bYabNftm\nBBYgg2SzYGmDFJQwoqQjO82V7das2TcjsAAJgISZr81ws2bfjMACZJBsFixtkILiR5R2ZKe5\nst2aNftmBBag+iCB5msz3KzZNyOwABkkmwVLG6Sg6BElHtlprmy3Zs2+GYEFqDpIqPnaDDdr\n9s0ILEBVQUrdkDRXtluzZt+MwAJUGyTYfG2GmzX7ZgQWIINks2DpQ4B0umr0MHwx+t4lGqTk\nIzvNle3WrNk3IvUEpYN0evzx/Mbke5EjSuZIc2W7NWv2XRh4lspBOn39XuSIDJK2WbPvwsCz\nVBGk9CM7zZXt1qzZd2HgWSoG6TT53h9Xxf1F/nAxqx/BQbop7mdN+oak+SOyW7Nm34WBZ6ke\nSBlHdpor261Zs+/CwLNUCtJp5nvRICHnazPcrNl3YeBZMkg2C5Y2SEExI8o5stNc2W7Nmn0X\nBp6lgisbxvSkX9mQw5HmynZr1uy7PPMUVbvWziDJmzX7ZgQWoFogZR3Zaa5st2bNvhmBBagi\nSNj52gw3a/bNCCxABslmwdIGKWh7RHlHdpor261Zs29GYAGqBxJ4vjbDzZp9MwILkEGyWbC0\nQQraHpFBasCs2TcjsADVASnzJZLmynZr1uybEViAqoGEnq/NcLNm34zAAmSQbBYsbZCCtkaU\ne2SnubLdmjX7ZgQWoFogwedrM9ys2TcjsAAZJJsFSxukoI0RZR/Zaa5st2bNvhmBBagSSPj5\n2gw3a/bNCCxABslmwdIGKWh9RPlHdpor261Zs29GYAGqAxJhvjbDzZp9MwILkEGyWbC0QQpa\nH5FBasSs2TcjsADtD1LBSyTNle3WrNk3I7AAVQGJMV+b4WbNvhmBBcgg2SxY2iAFrY2o5MhO\nc2W7NWv2zQgsQDVAoszXZrhZs29GYAGigLQmf7yY1aP23pGKjuw0f0R2a9bsmxFYgCqAxJmv\nzXCzZt+MwAJU4TUSZ742w82afTMCC1C1T6NAz9dmuFmzb0ZgATJINguWNkhBNeZrM9ys2Tcj\nsAAZJJsFSxukoBrztRlu1uybEViADJLNgqUNUlCN+doMN2v2zQgsQAbJZsHSBimoxnxthps1\n+2YEFiCDZLNgaYMUVGO+NsPNmn0zAguQQbJZsLRBCqoxX5vhZs2+GYEFyCDZLFjaIAXVmK/N\ncLNm34zAAmSQbBYsbZCCaszXZrhZs29GYAEySDYLljZIQTXmazPcrNk3I7AA7Q3SMAyc+doM\nN2v2zQgsQDuDNJzP5wKSJFe2W7Nm34zAArQvSDeOSkiSXNluzZp9MwILkEGyWbC0QQpaHIJB\nasms2TcjsAD5NZLNgqUNUtDKhHzWrh2zZt+MwAKUAdLpqunj0/ibfh+pD7Nm34DQM5QO0unx\nx+jx6eU/qTFfm+Fmzb4LA8+SQbJZsPQhQbpMOTJIfZg1+y5JO1GlIN1fGo1eIv1xFao5y2pF\nhSB9HNaNv3fxjtSJWbNvQOgZAr1GGv+/QerErNl3YeBZMkg2C5Y+Kkg+tOvRrNl3eeYpwoE0\nOnNXY742w82afZdnnqKCKxtO48fjCxsMUh9mzb4hscfLv2pus2BpgxRUY742w82afTMCC5BB\nslmwtEHiqt41E0esfMx/dKYMkisLljZIXB1xZQ1SEzJIrixY2iBZ1iFlkCwLIINkWQAZJMsC\nyCBZFkAGybIAagmk10vM9y5eqWy9f3Ot0qeq1XPVEEiT3x7cuXidwhX/zbVKP39Hp9pi58gg\nRdY+VprrlT5dDNIeqhaqimmuo2pRNkh76HAgVXupYJCS1BhI9faFSpXrla57ssEgcVV5ZV16\n18IGiaVa28Kp/k/n45Q2SHTVHOzB0myQEtUSSFXnerA0G6RENQRSvQOsUL1SWV/Z0IYaAsmy\ndGWQLAsgg2RZABkkywLIIFkWQAbJsgAySJYFkEGyLIAMkmUBZJAsCyCDJKfBa9KgvGhyMkgt\nyosmJ4PUorxoO+r392H4/vsSWPlzeP95+97P2/fuj/4cTj/uT/64P7LakUHaUafhqm+XGytX\nfIbTlanf4XvPR3/envzz9sgkNSWDtJ/+usHxY/j7xsr778v7/cv3y+ej75d/b4d14cm/hqZ+\nG8cySPvpWxj2fdP573ood9ucvg0/H49+3/+z4fYtv1JqTF6u/TR86JOSuUeX8besduTl2k8G\nqWN5ufbTt8ewh/sB3fvCod3zT6sVebn204/bOYX/u+EzXP/4/T789Xqy4cflv9dNympHXq79\ndD/BfTvPcAXpdtL7Mj79/fPz5LhBalFerh11e/P1/d9LOLR7/3wb9vGG7H/v90cGqUV5uWrI\nlHQnr2gNGaTu5BWtIYPUnbyiNWSQupNX1LIAMkiWBZBBsiyADJJlAWSQLAsgg2RZABkkywLI\nIFkWQP8PEY3meEyORpwAAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "history_lstm$metrics$val_acc\n",
    "plot(history_lstm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally let's see what happens when using a *GRU*  (*Gated Recurrent Unit*) layer, that has performance similar to *LSTM* but with less parameters (more easy to train)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    layer_embedding(input_dim = max_features, output_dim = 8) %>%\n",
    "    layer_gru(units = 8) %>%\n",
    "    layer_dense(units = 1, activation = \"sigmoid\")\n",
    "    model %>% compile(\n",
    "    optimizer = \"rmsprop\",\n",
    "    loss = \"binary_crossentropy\",\n",
    "    metrics = c(\"acc\")\n",
    ")\n",
    "\n",
    "history_gru <- model %>% fit(\n",
    "    mtx_train, y_train,\n",
    "    epochs = 10,\n",
    "    batch_size = 32,\n",
    "    validation_split = 0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.7458</li>\n",
       "\t<li>0.8192</li>\n",
       "\t<li>0.825</li>\n",
       "\t<li>0.8368</li>\n",
       "\t<li>0.8302</li>\n",
       "\t<li>0.8344</li>\n",
       "\t<li>0.8352</li>\n",
       "\t<li>0.8336</li>\n",
       "\t<li>0.826</li>\n",
       "\t<li>0.8274</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.7458\n",
       "\\item 0.8192\n",
       "\\item 0.825\n",
       "\\item 0.8368\n",
       "\\item 0.8302\n",
       "\\item 0.8344\n",
       "\\item 0.8352\n",
       "\\item 0.8336\n",
       "\\item 0.826\n",
       "\\item 0.8274\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.7458\n",
       "2. 0.8192\n",
       "3. 0.825\n",
       "4. 0.8368\n",
       "5. 0.8302\n",
       "6. 0.8344\n",
       "7. 0.8352\n",
       "8. 0.8336\n",
       "9. 0.826\n",
       "10. 0.8274\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] 0.7458 0.8192 0.8250 0.8368 0.8302 0.8344 0.8352 0.8336 0.8260 0.8274"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2di3bbthZE6SpO3N62SWr9/7deS7Ik\nyuIDIAbABjizVlM5zugcH8w2SEqmh6NlWckaajdgWT3IIFmWQAbJsgQySJYlkEGyLIEMkmUJ\nZJAsSyCDZFkCSUH6taDFT67K7uJuauPKwAplkOzGlTZIVSZrdxY3tXFlYIUySHbjShukKpO1\nO4ub2rgysEIZJLtxpQ1SlcnancVNbVwZWKEMkt240gapymTtzuKmNq4MrFAGyW5caYNUZbJ2\nZ3FTG1cGViiDZDeutEGqMlm7s7ipjSsDK5RBshtX2iDNaxiGXJO1O4ub2rgysEIVAml4f39P\nIYm6qh27qY0rAytUGZBOHCWRRF3Vjt3UxpWBFcog2Y0rbZDmZJDac1MbVwZWKJ8j2Y0rbZDm\n5at2rbmpjSsDK5RfR7IbV9ogVZms3Vnc1MaVgRXKINmNK22QqkzW7ixuauPKwAplkOzGlTZI\nVSZrdxY3tXFlYIUqBtL7e67J2p3FTW1cGVihSt1E//TOhkKlLKu8Cu5IKVsS9dtjx25q48rA\nCmWQ7MaVNkgLek8iibqqHbupjSsDK1S5q3YGqS03tXFlYIUySHbjShukpa8/6diOuqodu6mN\nKwMrVMEXZA1SU25q48rACmWQ7MaVNkiLs0k5tqOuasduauPKwApV8r12BqklN7VxZWCFKgzS\nVpKoq9qxm9q4MrBCFX33t0FqyE1tXBlYoQyS3bjSBmllNtuP7air2rGb2rgysEKV/cE+g9SO\nm9q4MrBCGSS7caUN0tpsNh/bUVe1Yze1cWVghSp8zwaD1Iyb2rgysEKVB2kTSdRV7dhNbVwZ\nWKFK30XIILXipjauDKxQBsluXGmDtD6bjcd21FXt2E1tXBlYoYrfINIgNeKmNq4MrFA1QNpA\nEnVVO3ZTG1cGVqjytyw2SG24qY0rAyuUQbIbV9oghcxm07EddVU7dlMbVwZWqAq/jcIgNeGm\nNq4MrFB1QIomibqqHbupjSsDK1SN349kkFpwUxtXBlaoSiDFkkRd1Y7d1MaVgRWqym/sM0gN\nuKmNKwMrlEGyG1faIAXOJv7YjrqqHbupjSsDK1SdX8ZskPhuauPKwAq1AaTDh6Yex4EURxJ1\nVTt2UxtXpD6D4kE63P54fHyMACl6S6KuasduauOK1GeQQbIbV3qnIN0VPpvYYzvqqnbspjae\nHPk8SgXpeo70x0kRz/IBUnRly8IqEaTDcduhXeyWRP322LGb2rgk9npVOkeKPUuirmrHbmrj\nitRnkEGyG1faIEXMJu7YjrqqHbupjStSn0HVQIrbkqir2rGb2rgi9RmU8M6Gw+jxRVGzMUhs\nN7VxUfDVqvNeu5Oiju2oq9qxm9q4MrBC1QMpakuirmrHbmrjysAKVRekYJKoq9qxm9q4MrBC\nVQQpZkuirmrHbmrjysAKZZDsxpU2SHGziTi2o65qx25q48rAClUTpIgtibqqHbupjSsDK1Rt\nkAJJoq5qx25q48rAClUVpPAtibqqHbupjSsDK1R1kMJIoq5qx25q48rAClUXpOAtibqqHbup\njSsDK5RBshtX2iBFzyb02I66qh27qY0rAytUZZBCtyTqqnbspjauDKxQAJBCSKKuasduauPK\nwApVG6TALYm6qh27qY0rAysUAaQAkqir2rGb2rgysEJVBylsS6KuasduauPKwAplkOzGlTZI\nW2YTdGxHXdWO3dTGlYEVqj5IQVsSdVU7dlMbVwZWKAZIqyRRV7VjN7VxZWCFAoAUsiVRV7Vj\nN7VxZWCFgoC0RhJ1VTt2UxtXBlYoAkgBWxJ1VTt2UxtXBlYog2Q3rrRB2jib9WM76qp27KY2\nrgysUAiQ1rck6qp27KY2rgysUBiQlkmirmrHbmrjysAKxQBpdUuirmrHbmrjysAKZZDsxpU2\nSJsnu3ZsR13Vjt3UxpWBFUoKUoI+QKrdgmVtF2RHWtuSqN8eO3ZTG1cGVigKSCtnSdRV7dhN\nbVwZWKFIIC2QRF3Vjt3UxpWBFQoD0vKWRF3Vjt3UxpWBFcog2Y0rbZDmNQzDyvAWj+2oq9qx\nm9q4MrBCFQJpeHl5CSBp02TXZXdbpQ3SnE4crZK0tCVRV7VjN7VxZWCFAoG0tCVRV7VjN7Vx\nZWCFgoE0RxJ1VTt2UxtXBlYo0DnS0pZEXdWO3dTGlYEVinPV7tfSlkRd1Y7d1MaVgRWK8zrS\nSQaJ46Y2rgysUAbJblxpg5Q62dljO+qqduymNq4MrFAskGa3JOqqduymNq4MrFA8kCZJoq5q\nx25q48rACgUDaW5Loq5qx25q48rAClUMpJeXoBHObEnUVe3YTW1cGVihSoF0emdD0AwNEsRN\nbVwZWKEK7khhJE1vSdRV7dhNbVwZWKHKnSNFkBQ32XXZ3VZpg7Q4m5QtibqqHbupjSsDK1TJ\nq3YJWxJ1VTt2UxtXBlaowiCFkDS1JVFXtWM3tXFlYIUq+jpSOEkxk12X3W2VNkhrs9m8JVFX\ntWM3tXFlYIUq/M6GrVsSdVU7dlMbVwZWqPIgBZBkkABuauPKwApV+r12QSQ9H9tRV7VjN7Vx\nZWCFKv6m1VCSwie7LrvbKm2QQmYTQtLTlkRd1Y7d1MaVgRWqwo9RBJIUPNl12d1WaYMUNpsA\nkr5uSdRV7dhNbVwZWKGq/GBf/JZEXdWO3dTGlYEVagNIhw+NH98+CAYpgKQvWxJ1VTt2UxuX\nxF6veJAOtz9G/78oeDZhB3eBk12X3W2VNkjBs1kn6XFLoq5qx25q48mRz6M0kB45igApjKSw\nya7L7rZK7xCk2ynSHyfFPM2JpMV/cNqSopuzrDoS7EgbLjactLonjbck6rfHjt3UxhWpz6DE\nc6THx3GzWSNpfJZEXdWO3dTGkyOfRxVBWr0IbpAM0sTnmKp3aPdrlaTRlkRd1Y7d1MYVqc+g\ndJBGm1PsbAIO7gImuy672yq9D5Bu72w4jB5fFD2bFZLuWxJ1VTt2UxsXBV+tyjfRXydpfbLr\nsrut0gYpfjbLJN22JOqqduymNq4MrFC1QVonaXWy67K7rdIGactsFkm6bknUVe3YTW1cGVih\n6oO0fBHcINVyUxtXBlYoAEiLJH1uSdRV7dhNbVwZWKEgIC2StDLZddndVmmDtHE2SyRdtiTq\nqnbspjauDKxQCJDWSOKuasduauPKwArFAGmJpPOWRF3Vjt3UxpWBFQoC0gpJ2FXt2E1tXBlY\noSggLZB0/lHZFfey7G6rtEFKmewiSdRV7dhNbVwZWKE4IM2TNPnrmWNEzQTaTW1cGVihQCAt\nkrTuTqttN6i0QUqc7BxJqVsSNRNoN7VxZWCFQoG0RFKIPam23ZjSBil5sjMkJW5J1Eyg3dTG\nlYEVCgbSAklh/pTadkNKGyTBZF8mUUrbkqiZQLupjSsDKxQOpGmSjgaptJvauDKwQvFAmjy8\nOyZtSdRMoN3UxpWBFWoWpL8Ox+M/w+HPmCdTTfaZpGPSWRI1E2g3tfH0zGfRHEh/DcPx52EY\nhhiSZJN9IumYdJZEzQTaTW1ckfoMmgPp2/DPx39//TscZv7BlHST/UrSMenCHTUTaDe1cUXq\nM2gOpI8N6e/h2/n/4RJO9sslh2PShTtqJtBuauOK1GfQHCeH4eeP4d/TWVLEkykn+0jSyb2d\nJGom0G5q44rUZ9AcSH9+nB4dThvSW8STaSc7JunsNkgl3dTGFanPoNkjt7fh8PfHxhTDkRik\nMUlXkLaRRM0E2k1tPD3zWUR8HemmO0kXt0Eq6KY2rgysUGiQ7iTdQNpEEjUTaDe1cWVghUK+\nIHvX9ZLDp9sglXNTG0/PfBZJX5DNoDNJx+HzKvz5PiiWxRP0BdmRTiR98DOcP9h2cEf95op2\nUxtXpD6DqC/IjvSVpPhnoGYC7U4yfxxC5KqtSH0GYV+QHelCUsKW1GyYWwVp+FixFJI6Agnw\nguxNw5mkz4XZQlKzYW4UpPOCpZDUEUiEF2RvGsbvFzJIZdwGKUrs15GuGpO0YUtqNswGaeJz\nTLUB0oc7haRmw9woSD5HGun327dh+Pb2O+bJts4maLIPJEW702rv0e2rdlGaA+n8YuzpgsPP\niCfbOpuwyd5Iit6Smg1zsyD5daSrfgyvHwj9fB1+RDxZ5sneTpRiSaJmAu2mNq5IfQYtvCD7\n8P8g5Z7smKR4d1rt3bmpjacmPpOaAulC0kv0lkTNBNpNbTw18ZnU0qHdSZ8oGaTsbmrjitRn\nUEMXGy66kBS3JVEzgXZTG1ekPoPaufx90w2lTe602jtyUxtPz3wWNfOC7FifKG10p9XejZva\nuDKwQjUJUixJ1Eyg3dTGlYEVahKkYayIJys42ZfbpfAt7rTa+3BTGxcFX61WQYpCiZoJtJva\nuCj4arV5aHfWSzBK1Eyg3dTGlYEVqmGQfr2HkkTNBNpNbVwZWKGaBuk9ECVqJtBuauPKwArV\nBkgz78o/vZYUghI1E2i39hX0L0qorQysUE2ANPdzYu93khaXptkwtwLSJCxbtVJbGVihWgBp\n/ieXL+8UWiWp2TBzQZKi8ySDtHVdFrVwC4DPNwqtoNRsmEEgbYZi8skN0qIi1iVGyyC9j1ZG\nX3u/7mMSLAmlDVLxc6Rfox/xW/r+12yY67g37zPppQPcysAK1QRIC/fSGP08xfwKNxjmKu4U\nchJLh7uVgRWqDZAW3OOfTJpZ8oy3tOnFLUBna+lYtzKwQjUP0uMPy05FIOdN1rpwTxJU87vP\nYm1lYIXqAqQnksYoZb3tZ/PuuT2o5nef5drKwArVPkhf74TyNRUGaVbzx3E1h7ZSWxlYoToA\n6elOKI/hqAtS2hFSxuOr5bMhgxSrDSAdPjT+8P4w02TX3M93Qnk82scepeR1L3S+eknBIMUq\nHqTD7Y/Lh/VBmrzN3Sgo9c6b0/KYJ83LW9G4NvO7T3rmsygVpANgR5q58+o9LdXOU4JReAlX\nRPnnzsOfx1ft4pQI0oFwaDd759VrZlAgRTCTytND57Fu6lUSReozSAXSHyfJuorWiaTJT9xY\nKioRLOGa7OLzhhtB/9ZKVBpIhyNjR1q6G/gtQNlqP1TZpplnnTrGCX6aIaLQ1i+7gluR+gxK\nAunLdYeaIC3dDXxLjNZri5mJqr2lfFTdxdJ13YrUZ1AaSBfdPlVnshct3Q38GHipKscJTJk8\nKgmKLF3aLQq+WsmXvyk70hJJZ7eIkdi0Fs3jc1/bL70ZpCj1A9LCwd3VXQad+M7zuBNfCjJI\nUUp4Z8PogsOn6kz2qvkt6dEdSUe7P4SR1rlBilIP77W7apakit/XsXkklzZIVSZ7VwaQ9vve\ncWrjysAK1RtIkyQZpLZKG6Qqkx1phiSD1FZpg1RlsmNNk+RzpLZKG6Qqk32QHKSGr9oZpILq\nEKRnkqiZQLupjSsDK1RvIE2SRM0E2k1tXBlYoboDaergjpoJtJvauDKwQnUJ0leSqJlAu6mN\nKwMrVH8gTZBEzQTaTW1cGVihOgTpmSRqJtBuauPKwArVI0hPp0nUTKDd1MaVgRWqS5C+kkTN\nBNpNbVwZWKG6BWlMEjUTaDe1cWVgheoTpC8kUTOBdlMbVwZWqE5B+mWQUt3UxpWBFapXkB5I\nomYC7aY2rgysUD2DdCOJmgm0m9q4MrBCdQvSmCRqJtBuauPKwArVL0gjkqiZQLupjSsDK1TH\nIP0ySI2WNkhVJjuvK0nUTKDd1MaVgRWqd5Dea9Vu3k1tXBlYoboG6UoSNRNoN7VxZWCF6huk\nXwapxdIGqcpkF3UmiZoJtJvauDKwQu0ApHdsJtBuauPKwArVO0hnkqiZQLupjSsDK1T3IC3+\nBrLctVt2UxtXBlao/kFKJomaqMxuauPKwAq1A5BSSaImKrOb2rgysELtAaQhjSRqojK7qY0r\nAyuUFCSmzhy91+7C6lv970gXjhK2JOq35sxuauPKwAq1H5A2k0RNVGY3tXFlYIXaEUhbSaIm\nKrOb2rgysEL1D9KZpCHh0h01UZnd1MaVgRVqByB9/qqwzSRRE5XZTW1cGVih9gDSp3srSfU7\nr+KmNq4MrFA7AmkrSYDOa7ipjSsDK9SeQPq17ZIDofMKbmrjysAKtSuQtpGE6Ly8m9q4MrBC\n7QukTYd3kM5Lu6mNKwMr1N5A2kASpfPCbmrjysAKtTuQ4knCdF7WTW1cGVih9gdS9IkSp/Oi\nbmrjysAKtUOQYkkCdV7STW1cGVih9ghSJEmkzgu6qY0rAyvULkGKQ4nVeTE3tXFlYIXaKUgx\nJME6L+WmNq4MrFB7BelX+E9W0Dov5KY2rgysULsFKRwlXudF3NTGlYEVascghZIE7LyEm9q4\nMrBC7RmkQJSQned3UxtXBlaofYMUhBK089xuauPKwAq1d5B+rd/Pgdp5Zje1cWVghdo9SOub\nErfzrG5q48rACmWQfq2hRO48o5vauDKwQhmks5ZQYneezU1tXBlYoQzSRQunSvDOc7mpjSsD\nK5RBumr2NpL4zvO4qY0rAyuUQbprBqUGOs/hpjauDKxQBmmsSZSa6FzvpjauDKxQBumLnklq\npXOxm9q4MrBCGaQnfUWpnc6lbmrjysAKZZAm9IDS5c7hxWpj3NTGlYEVyiBNaXSqdPldFgVr\nU9zUxpWBFWoDSIcPTT3uCKT7ZYfLb1dKIKmxrxtReh8gHW5/PD4+dgXSHSWDxHIrUp9BBmle\nBonoVqQ+g9JAOj4+rjPZnO5kjhr9uuuW3jNIf5ykaYmlC0q1u7DoSgWp14sNI4Xfb0hf2zvS\n8+eY8qHdunv9h2jz1a7npjaeHPk8Mkgh7tl3hheoXctNbTw58nnkq3Zh7s0oVe+8wdIGqcpk\nC7nvF8PL167hpjauSH0GJbyz4TB6fFGdyZZzj1+jLV27uJvauCj4avm9dnHuaJYwnTdU2iBV\nmWxxdxRMqM4bKW2Qqky2ijuYJVznDZQ2SFUmW8sdtjERO6eXNkhVJlvTvc4StXNyaYNUZbKV\n3SsbE7hzbGmDVGWyAPcCS/DOkaUNUpXJMtxzGxO/c15pg1Rlshz3FEttdM4qbZCqTBbl/rox\ntXsPIurIlYEVyiDp3SOWGr4HEXXkysAKZZCyuMcbU6N3fKCOXBlYoQxSNveGd7jKagvc1JEr\nAyuUQcrpftiYCtdOdVNHrgysUAYpq3t4/6pytQ1SSRmkvO7zVbvtLBmk588xZZCKubfAZJCe\nP8eUQSrrjmTJID1/jimDVNwdszMZpOfPMWWQqrhDz5oM0vPnmDJItdxBLBmk588xZZDquldo\nMkjPn2PKIAHcs6811XzLK3VoysAKZZAY7qcXbk84VX3LK3VoysAKZZBI7gmcqr3llTo0ZWCF\nMkhAd+AlvSy1NWaDlKQ6k+3UPXviVKB2stkgJanOZPt1Tx3plaqNHZoysEIZJLJ76i2vESwZ\npHIySK24N9BkkMrJILXljtqdDFI5GaTm3OFnTgapnAxSo+4pnL7yZJDKSQqSVVyTPH1+bhi8\nusXkHakL9yRP763eCWz3O1Kdydp9U8jxXqbSxdzKwAplkDp0S3iiftnKwAplkHp1P98JLI4n\n6petDKxQBqlb9/WHMDbyRP2yQ0L9kOq/DsqMz9ZUPlmdydo9o8cfC4zlifplh4R6mP8olwzS\nztxz1/e+YoX9fTQhoTZIdhdyr/HE/X00yxH8+Tp8v6Dzz/dhOLydX00bf5hLBmnX7iWeoK9C\nLSbw9+EDm+8ncv4eznr7BOn2YS4ZJLtv2nhZQlE63L2YwLfh9fj79UTOt+F/x+O/p0fnDen+\nYSYZJLu/anGXyls6wL2YwG/Dz4/DuwsvP//+8/UG0v3DTDJIdk9p5lWoYJ4qgXQh5fzn6+Vg\n7vp3tw8zySDZPanbVbsVoqaxynjNbznON5B+DN/++vvnDaT7h5lkkOyOMgfxlPOa32IC74d2\nZ2Z+3x7dP8wkg2T3FnPANpXpmt9iAv8cXn8fP8+M/vm87PAJ0vXDTDJIdqebtxz9ba29mMD7\n5e+34X6OdBh/mEkGyW6ZWcZTwguy368vyP4Yhtd/To/+OoF0/zCTDJLdevP5HGk7VouXKpSB\nFcog2Z3B/IxCAFZXupYvVSgDK5RBsrtk6WCcZklSBlYog2R3vdIGaVrpk7Ub465W2iAZpJ7c\nFd9U4XOkTJO1u4K7Ymlftcs1WbvLu6mNKwMrlEGyG1faIFWZrN1Z3NTGlYEVyiDZjSttkKpM\n1u4sbmrjiwn8b17KoE/IINmNK22QqkzW7ixuauOLCTRICZO1O4ub2vhiAg1SwmTtzuKmNr6Y\nQIOUMFm7s7ipjS8m0CAlTNbuLG5q44sJNEgJk7U7i5va+GICDVLCZO3O4qY2vphAg5QwWbuz\nuKmNLyZwI0jD5MMo+RfIWx1pTM4wDDMg5Qi9QbI60pij048HGiTL2qD//pu5BcQYpPONIofj\n5eaRl/uwHs8ff3l4vPy7MBkkqyMFgXS83Qx8+PxouH/8+DD892YaJKsjRYA0/mh4ZOr+MBwQ\ng2R1pPBzpM/gf94e3CBZ1kjhV+2G8SODZFljhb2OND49MkiW9aQNIC0f2tW52FDnpW67s7ip\njS8mMBCk22Xtz0fTINW7/F1nsnZncVMbX0xgGEgxMkh2G6Q0kOqdI9WZrN1Z3NTGFxOo3ZFi\nflemQbIbV5oCUowMkt240gapymTtzuKmNr6YQIOUMFm7s7ipjS8m0CAlTNbuLG5q44sJNEgJ\nk7U7i5vauDKwQhkku3GlvSNVmazdWdzUxhcTaJASJmt3Fje18cUEGqSEydqdxU1tfDGBBilh\nsnZncVcsvf23mhukFdld3F2v9PmHxOefeUkGaUV2F3dXK325W8ksSYsJNEgrsru4O9/B2a8r\nLEsySHNf/8YlsbuSO/ngbB0WgzSjXOtidwV3jDmNmS8qfo403DEYnj4TKoNk96TmDs5EsGyq\nffmiljQmZ/52XPMMTD8MkUGye0r3XWEzLMtnSZsbX0zgmKOXl5eZG0TOM2CQ7Ba4ZRvM8sFZ\nWuOLCfzvv5dpPdxE//rn+Bb693sGfd589fHD+531Z2SQdu1OZ2Zaa1ftEhpfTGAUSI934Brd\n4+7+8ejD0X2Op2SQ+nVPpjkTOdLGM4P0cBP9B5C+/G3MbVcNUrfu6/HVJmZyHpyluRcTGHaO\ndL/T6ugW+qN7GF//1iDt3K3YbjIenKW5FxMYeNXuflg3sSONT5sM0s7cGY7UqF/2YgIDX0ca\nHm7//XzHb4O0O7cSncjSldyLCQx9QXZ0e++piw1lDu0OH5p6bJDk7qnjq2B0Ek9zqENbzGYU\nSF9uoT+63v15bfzx8vcyLfEgHW5/PD4+GiS1+wGF+F0n7TSHOrTFcIaCpJdB4rovb/tcfvNn\nrtrcoS2G0yAlTLZbd/Ipj0FqD6Q/TpJ1tXNNAlS7qSbUMkgH70gyd+oWlFJbavaOtK6Hw7nT\nRTuDlOBePv+p+aoodWiL4WwWpMfHdSbbpjvsCkLFzolDO39uSa2C5IsN8e4wgvLULmY2SOsy\nSFvdG06ADNLz55hKeGfDYfT4ojqTbcO97RKCQXr+HFN+r11e9+lyQcJFOIP0/DmmDFJG97Zd\nSFM72U0duTKwQhmkHO4NZ0Oy2io3deTKwAplkKTuyWtxbb4FmzpyZWCFMkga9/TV7MEgyd3K\nwAplkBLd0wRdxb31Abm0Qaoy2UruZYKuwt76gFzaIFWZbHF31KUEVOeNlDZIVSZb1B19MQ7T\neUOlDVKVyRZzb7qcjei8sdIGqcpky7jjEdLVruKmNq4MrFAGaU3DMGyFKLm2QZr4HFMGaUVJ\nFCXWNkhTn2PKIC1oyzmRqnZ1N7VxZWCFMkhT+npprs33JhikgjJID5p8ldUgkdzKwAplkGbY\nuR7NNfwmH4NUUDsHaYGgq9p9k49BKqgdg7RGUM7aDbipjSsDK9ReQYq4HAfrvJSb2rgysELt\nEqS4i9qkzgu6qY0rAyvUDkGKfWWI03lRN7VxZWCF2h9IcRRpa7fkpjauDKxQewMpliJl7bbc\n1MaVgRVqXyBtwAjSeXk3tXFlYIXaFUibOEJ0XsFNbVwZWKH2BNImjBCd13BTG1cGVqgdgbQN\nI0LnVdzUxpWBFWo/IG3lqH7nddzUxpWBFWo3IG3mqHrnldzUxpWBFWonIG08PZLUbtRNbVwZ\nWKF2BFKt2o26qY0rAyvUHkA6372kUu123dTGlYEVagcgDSnHdYm1G3ZTG1cGVqj+QWr5N0IY\npInPMWWQctZu2U1tXBlYofoH6d0gtVbaIFWZ7LKSOcImKrOb2rgysEL1DtKZop3evsQgFdQO\nQOJmAu2mNq4MrFCdg3S57k3NBNpNbVwZWKH6BundILVY2iBVmey8ri/EUjOBdlMbVwZWqN5B\nqla7eTe1cWVgheoZpNsbg6iZQLupjSsDK5QUJJZOB3a1e7D2on53pNE7VanfXNFuauPKwArV\nLUjjd3xTM4F2UxtXBlaonkGqVrsLN7VxZWCF6hWkd4NkkEqqU5Aef5SPmgm0m9q4MrBC9QtS\ntdqduKmNKwMrVJ8gvRukZDe1cWVgheoSpK/3aKBmAu2mNq4MrFA9gvR0rxNqJtBuauPKwArV\nKUjC2nv9sUBq48rACtUhSM8330qpPby8vOzyB9WpjSsDK1R/IE3cxC6h9omjJJIq5jFtLzVI\nUeoSJGHtuiDFovAypTKlv8ggpajOZB81dVfVVkEKOKycRCdAgtKLMkgpqjPZB03enbjWOVJi\noB8h3orMJqzAR7TKwArVIUhr7vBg5cxuFo2/zBsL25/OIIWrM5Cmb5d/dYviStLCLBb30hwF\np0a+TQapxmRHmvm1E2e3LrwRoVs9MIx5sliFXy9I/0K/yiClaOts0iZ719yvbznGJvLhX1a8\nVFHx0tkw6OmSNK4MrFC9gTTxt6lr2+o1vzwvyJbAavH7hzKwQvUE0sSGpPgOaZBWpcVq+XhY\nGVihOgLpiSMBRKG157zR5RoAABJNSURBVMR9OSar+fxlb6Vr5buPMrBC9QXS6KPHNaoWZuwb\nBHKap1CIpcog6dclzD3ekMbf5Vo+vOoIpEcZpEUtDTd3JEYcPR4rGKTi5o0jDzuEUAZWqJ5A\nujx4oOiXQaphTjw19FW7BWWOxOeG9JWikxo+4W8VpJynhsrACtUGSKvrcuFoCqMQ97KaRcE/\n2FdQTYC0vqecOJrBKLH2bt3UxpWBFaoFkNbPct7f5ylKq71fN7VxZWCF6gOkRYy4mUC7qY0r\nAytUDyCtYMTNBNpNbVwZWKFaAGnxHGmVosTau3VTG1cGVqgmQJq/7haCETcTaDe1cWVghWoD\npBn3FaPpn0LKW7t7N7VxZWCFahik62Y089N8WWvvwE1tXBlYodoF6XpMN/djsTlr78FNbVwZ\nWKFaBel2ahTAETYTaDe1cWVghWoDpK8XG0ZXGAI4wmYC7aY2rgysUE2ANHzQMiLp5ZEjg5TF\nTW1cGVihWgDpxNGdpMcL3iEcYTOBdlMbVwZWqNZA+vK6UdCGhM0E2k1tXBlYodoC6evLr2Ec\nYTOBdlMbVwZWqBZAup4jPb+LIYwjbCbQbmrjysAK1QRIl6t2z28GCtyQsJlAu6mNKwMrVBsg\nfbin3lMXyBE2E2g3tXFlYIXaANLhQ1OPc4I0+dbU0A0Jmwm0m9q4IvUZFA/S4fbH4+NjRpCm\n3+IdyhE2E2g3tXFF6jOoBZBmflIieEPCZgLtpjauSH0GNQDSzA8chXOEzQTaTW1ckfoMUoH0\nx0myrsY6czTx9x8cZalnWRuUBlL+iw2f29GzO2JDwn5zRbupjStSn0HsQ7vb2dEkSMFPQ80E\n2k1tXJH6DEKDdL/K8OSO2ZCwmUC7qY0rUp9BYJDGF+u+uqM4wmYC7aY2rkh9BmFBerzmPQFS\nxHNRM4F2UxtXpD6DEt7ZcDhmvNjw5bWjL+64DQmbCbSb2rgo+Gox32v39BLsM0gxT0fNBNpN\nbVwZWKGQID2/BPvojtyQsJlAu6mNKwMrFBGkibcyPIEU/mxxte0GlDZIislOviPowR27IWEz\ngXZTG1cGVigcSNPvrBu7oznCZgLtpjauDKxQNJBm3qH6BaSw54qtbTektEFKnuzcb5YYueM3\nJGwm0G5q48rACsUCafY3tDyCFPJU0bXtxpQ2SImTnf9NR3f3hg0Jmwm0m9q4MrBCgUBa+o1h\nDyCtPlN8bbtJpQ1SymQXf/Pezb1lQ8JmAu2mNq4MrFAUkBYxurs3cYTNBNpNbVwZWKEgIK1w\nNAZp8Xm21LabVtogbZ3sGkdX97YNCZsJtJvauDKwQhFAWsVoDNLyv4uubTewtEHaNJsAjj7d\nGzckbCbQbmrjysAKVR+kEI7uIK39w7jadiNLG6QNswnB6NO9dUPCZgLtpjauDKxQtUEK4+gG\nUsA/Da9tN7S0QYqeTSBHZ/fmDQmbCbSb2rgysEJVBSno9Ojm3swRNhNoN7VxZWCFqglSOEcn\n9/YNCZsJtJvauDKwQlUEKRyjK0ih/3q9tt3k0gYpZjYxHH24EzYkbCbQbmrjysAKVQ2kKI4u\nIIX/85XadrNLG6Tw2cRx9OuYsiFhM4F2UxtXBlaoSiBFcnQGKebfL9W2m17aIIXOJpajpDMk\nbibQbmrjysAKVQWkaI6SzpC4mUC7qY0rAytUDZA2cWSQCrupjSsDK1QFkOI5StyQsJlAu6mN\nKwMrVHmQtnFkkEq7qY0rAytUcZA2cJS6IWEzgXZTG1cGVqjCIEW8vW4kg1TBTW1cGVihyoK0\nmSODVNxNbVwZWKFmQfp+/szw7WfEk63NZhNGZ5Coq9qxm9p4cuTzaA6kt+EC0vAj4slWZrOd\nI4NU3k1tPD3zWTQH0mH45/S/f4eYY7/l2Wzk6HyGRF3Vjt3UxtMzn0VznFwB0oGUwJFBquCm\nNp6a+Eya4+T78OP38fj7bXiNeLKl2Wzl6HLJjrqqHbupjStSn0FzIP08DGcd/o14soUvfzNH\nBqmSm9q4IvUZNHvk9vvt2zB8e4u5aLeyI22b6btBquOmNp6e+SyqfV+7VV1eQ6KuasduauPK\nwApFB+ndIFVyUxtXBlYo6QuyGXR6Dcmy8Cr5guwGXd8dRP322LGb2nh65rOo1AuywzBsmej1\nXXbUVe3YTW08PfNZVOgF2eHl5WULSQapmpvaeGriM6nMC7LD+W3f8SS9G6RqbmrjitRnUJkX\nZBNAWp/suuxuq3RPIGlfkN0I0rtBquemNp6e+Swq9DrStnOk+w/0UVe1Yze1cWVghSr1guyW\nq3bvBskgPX+OqVmQ3oZPRTyZeLKjnzCnrmrHbmrjgtDn0MILsgZp125q44rUZ9D8C7L/vg4/\nf79eXpcNlHay7wbJIE18jqmFF2T/HP4+/pb9YN8mkBLcabXtxjauSH0GLYD09/CX8kfNY4f5\nbpAM0tTnmJp/Z8P/fg7fjv9UBSnBnVbbbm7jitRn0BwnJ4JeT9caqr372yDVdVMbV6Q+g2Y3\nnL+/HY8/huEt5smUk303SHXd1MbTM59F2J+QfbxNMXVVO3ZTG1cGVigqSO8GqbKb2rgysEKB\nQUpwp9W2u3Jpg6SbrEGq7aY2rgysUFCQvv4mF+qqduymNq4MrFBckBLcabXtrl3aIKkm+/Sr\nxair2rGb2rgysEJhQUpwp9W2u3ppgySa7PPvuqSuasduauPKwApFBSnBnVbb7vqlDZJosgYJ\n4KY2rgysUESQJn6LOXVVO3ZTG1cGVigoSAnutNp2A0obJMlkJzYk7Kp27KY2rgysUEyQEtxp\nte0mlDZIkskaJISb2rgysELxQJo6ssOuasduauPKwAqFBCnBnVbbbkRpgySY7OSGhF3Vjt3U\nxpWBFYoIUoI7rbbdjNIGSTBZgwRxUxtXBlYoGkjTR3bYVe3YTW1cGVihgCAluNNq2w0pbZCS\nJzuzIWFXtWM3tXFlYIXigZTgTqttN6W0QUqerEHCuKmNKwMrFAukuSM77Kp27KY2rgysUDiQ\nEtxpte3GlDZIqZM1SBw3tXFlYIVCgTR7ZIdd1Y7d1MaVgRWKBlKCO6223ZzSOwHp8KHRw9sH\nySDNb0jYVe3YTW1ck3u54kE63P4Y/8VZiZOd5wi7qh27qY0nRz6P0kEaPUycrEEiuamNJ0c+\nj0AgLRzZYVe1Yze18eTI51EySJ+P/jgprZUPjtKewLKqSQXSWUnfopY2JOy3x47d1MaTI59H\nKJAS3Iuyu63SewRpzJFB6slNbTw58nmEAWnxyA67qh27qY0nRz6PSCAluJdld1ul9wHS7Z0N\nT6/MGqSe3NTGFanPIMp77ZaP7LCr2rGb2rgysEKBQEpwr8jutkobpM2TXdmQsKvasZvauDKw\nQnFA2jzZddndVmmDtHmyBgnnpjauDKxQDJDWjuywq9qxm9q4MrBCYUDaPtl12d1WaYO0dbIG\nieemNq4MrFAIkFaP7LCr2rGb2rgysEJRQEqY7Lrsbqu0Qdo4G4MEdFMbVwZWKAJI60d22FXt\n2E1tXBlYoSAgpUx2XXa3VdogbZpNwIaEXdWO3dTGlYEVigFS0mTXZXdbpQ3SptkYJKSb2rgy\nsELVBynkyA67qh27qY0rAysUAqS0ydqdxU1tXBlYoQyS3bjSBmnDbIKO7LCr2rGb2rgysEIR\nQEqcrN1Z3NTGlYEVyiDZjSttkOJnE3Zkh13Vjt3UxpWBFQoAUupk7c7ipjauDKxQBsluXGmD\nFD2bwCM77Kp27KY2rgysUPVBSp6s3Vnc1MaVgRXKINmNK22QYmcTemSHXdWO3dTGlYEVqjpI\n6ZO1O4ub2rgysEIZJLtxpQ1S5GyCj+ywq9qxm9q4MrBC1QZJMFm7s7ipjSsDK1RNkMI3JOyq\nduymNq4MrFCVQVJM1u4sbmrjysAKZZDsxpU2SFGziTiyw65qx25q48rAClUXJMlk7c7ipjau\nDKxQBsluXGmDFDObmCM77Kp27KY2rgysUFVB0kzW7ixuauPKwAplkOzGlTZIEbOJOrLDrmrH\nbmrjysAKVRMk0WTtzuKmNq4MrFBSkGL0AVKt0pYlV60dKe7IDvvtsWM3tXFlYIWqCJJqsnZn\ncVMbVwZWKINkN660QQqeTeSRHXZVO3ZTG1cGVqh6IMkma3cWN7VxZWCFMkh240obpNDZxB7Z\nYVe1Yze1cWVghaoGkm6ydmdxUxtXBlYog2Q3rrRBCpxN9JEddlU7dlMbVwZWqFogCSdrdxY3\ntXFlYIUySHbjShuksNnEH9lhV7VjN7VxZWCFqgSScrJ2Z3FTG1cGViiDZDeutEEKms2GIzvs\nqnbspjauDKxQdUCSTtbuLG5q48rACmWQ7MaVNkghs9lyZIdd1Y7d1MaVgRWqCkjaydqdxU1t\nXBlYoQyS3bjSBilgNpuO7LCr2rGb2rgysELVAEk8WbuzuKmNKwMrlEGyG1faIK3PZtuRHXZV\nO3ZTG1cGVqgKIKkna3cWN7VxZWCFMkh240obpNXZbDyyw65qx25q48rAClUeJPlk7c7ipjau\nDKxQBsluXGmDtDobg9SMm9q4MrBClQVp6ykSdlU7dlMbVwZWqOIg6SdrdxY3tXFlYIUySHbj\nShukldlsPrLDrmrHbmrjysAKVRqkDJO1O4ub2rgysEIZJLtxpQ3S8my2H9lhV7VjN7VxZWCF\nKgxSjsnancVNbVwZWKEMkt240gZpcTYJR3bYVe3YTW1cGVihyoKUZbJ2Z3FTG1cGViiDZDeu\ntEFanI1BaspNbVwZWKHKgZRyioRd1Y7d1MaVgRWqKEh5Jmt3Fje1cWVghdoA0uFDU48NUldu\nauOK1GdQPEiH2x+Pj4/LICUd2WFXtWM3tXFF6jOoJEiZJmt3Fje1cUXqM0gA0l1LszFIjbmp\njSdHPo9SQbqeI/1x0oLrdGQXXcqyWlEiSIdj4KFd2oaE/fbYsZvauCT2ehU8R8o1WbuzuKmN\nK1KfQcVAoq6L3bzSBqnKZO3O4qY2rkh9Bhkku3Gl9wHS7d0Mh9Hji+pM1u4sbmrjouCrVf6X\nMcsna3cWN7VxZWCFMkh240obpCqTtTuLm9q4MrBCGSS7caUNUpXJ2p3FTW1cGVihDJLduNIG\nqcpk7c7ipjauDKxQBsluXGmDVGWydmdxUxtXBlYog2Q3rrRBqjJZu7O4qY0rAyuUQbIbV9og\nVZms3Vnc1MaVgRXKINmNK22Q5jUMQ67J2p3FTW1cGVihCoE0vL+/p5BEXdWO3dTGlYEVqgxI\nJ46SSKKuasduauPKwAplkOzGlTZIczJI7bmpjSsDK5TPkezGlTZI8/JVu9bc1MaVgRXKryPZ\njSttkKpM1u4sbmrjysAKZZDsxpU2SFUma3cWN7VxZWCFMkh240obpCqTtTuLm9q4MrBCGSS7\ncaUNUpXJ2p3FTW1cGVihDJLduNIGqcpk7c7ipjauDKxQBsluXGmDVGWydmdxUxtXBlYog2Q3\nrrRBqjJZu7O4qY0rAyuUFKR8+sO1d1S6au2NMkiujSttkLJpr6tqkFqRQXJtXGmDZFk7lUGy\nLIEMkmUJZJAsSyCDZFkCGSTLEqgJkA4fqlm+WuF6X3e90ofK9TeqBZAOtz/qlK8W5tsfOyp9\nuJduiiSDtF7dIJUsbJDyqmKaq4JUsXRthg1SDu0SpLonKgYpQq2AVHNXqHfCX6945YsNBimX\nqq/qvoojShskvWpeBQZ8b95naYMkV92RekeqU9ogqVV5ogapTmmDJFbNw6tz/WqF/c6GZtQC\nSJaFl0GyLIEMkmUJZJAsSyCDZFkCGSTLEsggWZZABsmyBDJIliWQQbIsgQwSSoPXo1F54VAy\nSK3KC4eSQWpVXrhC+v1jGH78Pp5Z+T68/jz93c/T310efR8Ob5dPvl0eWW3JIBXSYfjQt+OJ\nlQ98hsMHU7/Pf3d/9P30ye+nRyapORmkMvrzBMfb8NeJldffx9fLh6/H66Mfx39Oh3XnT/45\ntPWjONbRIJXSt/OgL5vOvx+HcqfN6dvw8/bo9+WfDae/8plSg/KSldHwqSslU4+O47+y2pKX\nrIwMUufykpXRt9ugh8sB3evMod39T6slecnK6O10TeF/J3yGjz9+vw5/Pl5seDv++7hJWW3J\nS1ZGlwvcp+sMHyCdLnofx5e/f14vjhukVuUlK6TTi6+v/xzPh3av15dhby/I/vt6eWSQWpWX\nrLRMSZfyqpaWQepSXtXSMkhdyqtaWgapS3lVLUsgg2RZAhkkyxLIIFmWQAbJsgQySJYlkEGy\nLIEMkmUJ9H+PDGWdLFZxAwAAAABJRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "history_gru$metrics$val_acc\n",
    "plot(history_gru)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}