9_vanilla-LSTM-with-cupy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla LSTM with CuPy\n",
    "\n",
    "This is a port of [Vanilla LSTM with numpy](http://blog.varunajayasiri.com/numpy_lstm.html) that shows how to run the numpy-based machine learning code on GPU using [CuPy](https://github.com/cupy/cupy). The all contents below is basically copied from the article: [Vanilla LSTM with numpy](http://blog.varunajayasiri.com/numpy_lstm.html).\n",
    "\n",
    "This is inspired from [Minimal character-level language model with a Vanilla Recurrent Neural Network](https://gist.github.com/karpathy/d4dee566867f8291f086), in Python/numpy by [Andrej Karpathy](https://github.com/karpathy).\n",
    "\n",
    "The model usually reaches an error of about 45 after 5000 iterations when tested with [100,000 character sample from Shakespeare](http://cs.stanford.edu/people/karpathy/char-rnn/shakespear.txt). However it sometimes get stuck in a local minima; reinitialize the weights if this happens.\n",
    "\n",
    "You need to place the input text file as `input.txt` in the same folder as the python code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 99993  100 99993    0     0  70182      0  0:00:01  0:00:01 --:--:--  149k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -L http://cs.stanford.edu/people/karpathy/char-rnn/shakespear.txt -o input.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython import display\n",
    "\n",
    "np.random.seed(2017)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above imports are from the original article as is. But in this article, you can run almost all computation on GPU by just replacing `np` with `cp`. Well, `cp` is just another name of `cupy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cupy as cp\n",
    "\n",
    "cp.random.seed(2017)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show the difference of computational time by switching CPU and GPU simply, let's use `xp` instead of `np` and switch the referenced package between `numpy` and `cupy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# If you use CPU\n",
    "# xp = np\n",
    "\n",
    "# If you use GPU: almost 2 times faster\n",
    "xp = cp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Because CuPy has high compatibility with NumPy, the same code using NumPy is easily converted for CuPy by just replacing `numpy` with `cupy`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:**\n",
    "\n",
    "If the `H_size` is larger than 1000 or so, training on GPU, namely, using CuPy is several times faster than NumPy. But when the array size is small, the speed gain is often small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read and process data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = open('input.txt', 'r').read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Process data and calculate indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data has 99993 characters, 62 unique\n"
     ]
    }
   ],
   "source": [
    "chars = list(set(data))\n",
    "data_size, X_size = len(data), len(chars)\n",
    "print(\"data has %d characters, %d unique\" % (data_size, X_size))\n",
    "char_to_idx = {ch:i for i,ch in enumerate(chars)}\n",
    "idx_to_char = {i:ch for i,ch in enumerate(chars)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "H_size = 100 # Size of the hidden layer\n",
    "T_steps = 25 # Number of time steps (length of the sequence) used for training\n",
    "learning_rate = 1e-1 # Learning rate\n",
    "weight_sd = 0.1 # Standard deviation of weights for initialization\n",
    "z_size = H_size + X_size # Size of concatenate(H, X) vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation Functions and Derivatives\n",
    "\n",
    "### Sigmoid\n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "\\sigma(x) &=& \\frac{1}{1 + e^{-x}} \\\\\n",
    "\\frac{d \\sigma(x)}{d x} &=& \\sigma(x) \\cdot (1 - \\sigma(x))\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "### Tanh\n",
    "\n",
    "$$\n",
    "\\frac{d \\tanh(x)}{dx} = 1 - \\tanh^2(x)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + xp.exp(-x))\n",
    "\n",
    "\n",
    "def dsigmoid(y):\n",
    "    return y * (1 - y)\n",
    "\n",
    "\n",
    "def tanh(x):\n",
    "    return xp.tanh(x)\n",
    "\n",
    "\n",
    "def dtanh(y):\n",
    "    return 1 - y * y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize weights\n",
    "\n",
    "We use random weights with normal distribution `(0, weight_sd)` for `tanh` activation function and `(0.5, weight_sd)` for `sigmoid` activation function.\n",
    "\n",
    "Biases are initialized to zeros.\n",
    "\n",
    "Formulae for LSTM are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W_f = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n",
    "b_f = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "\n",
    "W_i = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n",
    "b_i = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "\n",
    "W_C = (xp.random.randn(H_size, z_size) * weight_sd).astype(xp.float32)\n",
    "b_C = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "\n",
    "W_o = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n",
    "b_o = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "\n",
    "#For final layer to predict the next character\n",
    "W_y = (xp.random.randn(X_size, H_size) * weight_sd).astype(xp.float32)\n",
    "b_y = xp.zeros((X_size, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dW_f = xp.zeros_like(W_f, dtype=xp.float32)\n",
    "dW_i = xp.zeros_like(W_i, dtype=xp.float32)\n",
    "dW_C = xp.zeros_like(W_C, dtype=xp.float32)\n",
    "\n",
    "dW_o = xp.zeros_like(W_o, dtype=xp.float32)\n",
    "dW_y = xp.zeros_like(W_y, dtype=xp.float32)\n",
    "\n",
    "db_f = xp.zeros_like(b_f, dtype=xp.float32)\n",
    "db_i = xp.zeros_like(b_i, dtype=xp.float32)\n",
    "db_C = xp.zeros_like(b_C, dtype=xp.float32)\n",
    "\n",
    "db_o = xp.zeros_like(b_o, dtype=xp.float32)\n",
    "db_y = xp.zeros_like(b_y, dtype=xp.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward pass\n",
    "\n",
    "![](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png)\n",
    "\n",
    "Image taken from [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). Please read the article for a good explanation of LSTMs.\n",
    "\n",
    "### Concatenation of $h_{t-1}$ and $x_t$\n",
    "\n",
    "$$\n",
    "z = [h_{t-1}, x_t]\n",
    "$$\n",
    "\n",
    "### LSTM functions\n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "f_t &=& \\sigma(W_f \\cdot z + b_f) \\\\\n",
    "i_t &=& \\sigma(W_i \\cdot z + b_i) \\\\\n",
    "\\bar{C}_t &=& \\tanh(W_C \\cdot z + b_C) \\\\\n",
    "C_t &=& f_t \\ast C_{t-1} + i_t \\ast \\bar{C}_t \\\\\n",
    "o_t &=& \\sigma(W_O \\cdot z + b_t) \\\\\n",
    "h_t &=& o_t \\ast \\tanh(C_t)\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "### Logits\n",
    "\n",
    "$$\n",
    "y_t = W_y \\cdot h_t + b_y\n",
    "$$\n",
    "\n",
    "### Softmax\n",
    "\n",
    "$$\n",
    "\\hat{p}_t = {\\rm softmax}(y_t)\n",
    "$$\n",
    "\n",
    "$\\hat{p}_t$ is `p` in code and $p_t$ is `targets`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def forward(x, h_prev, C_prev):\n",
    "    assert x.shape == (X_size, 1)\n",
    "    assert h_prev.shape == (H_size, 1)\n",
    "    assert C_prev.shape == (H_size, 1)\n",
    "\n",
    "    z = xp.concatenate((h_prev, x))\n",
    "    f = sigmoid(xp.dot(W_f, z) + b_f)\n",
    "    i = sigmoid(xp.dot(W_i, z) + b_i)\n",
    "    C_bar = tanh(xp.dot(W_C, z) + b_C)\n",
    "\n",
    "    C = f * C_prev + i * C_bar\n",
    "    o = sigmoid(xp.dot(W_o, z) + b_o)\n",
    "    h = o * tanh(C)\n",
    "\n",
    "    y = xp.dot(W_y, h) + b_y\n",
    "    y -= y.max()\n",
    "    p = xp.exp(y) / xp.sum(xp.exp(y))\n",
    "\n",
    "    return z, f, i, C_bar, C, o, h, y, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backward pass\n",
    "\n",
    "### Loss\n",
    "\n",
    "$$\n",
    "\\mathcal{L} = - \\sum p_{t,j}log \\hat{p}_{t,j}\n",
    "$$\n",
    "\n",
    "### Gradients\n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "dy_t &=& \\hat{p}_t - p_t \\\\\n",
    "dh_t &=& dh'_{t+1} + W_y^T \\cdot d_y \\\\\n",
    "do_t &=& dh_t \\ast \\tanh (C_t) \\\\\n",
    "dC_t &=& dC'_{t+q} + dh_t \\ast o_t \\ast (1 - \\tanh^2(C_t)) \\\\\n",
    "d \\bar{C}_t &=& d C_t \\ast i_t \\\\\n",
    "d i_t &=& d C_t \\ast \\bar{C}_t \\\\\n",
    "d f_t &=& d C_t \\ast C_{t-1} \\\\\n",
    "d z_t &=& W_f^T \\cdot df_t + W_i^T \\cdot di_t + W_C^T \\cdot \\bar{C}_t + W^T_o \\cdot do_t \\\\\n",
    "[dh'_t, dx_t] &=& dz_t \\\\\n",
    "dC'_t &=& f \\ast dC_t\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "- `target` is target character index $p_t$\n",
    "- `dh_next` is $dh_{t+1}$ (size $H \\times 1$)\n",
    "- `dC_next` is $dC_{t+1}$ (size $H \\times 1$)\n",
    "- `C_prev` is $C_{t-1}$ (size $H \\times 1$)\n",
    "- Returns $dh_t$ and $dC_t$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def backward(target, dh_next, dC_next, C_prev, z, f, i, C_bar, C, o, h, y, p):\n",
    "\n",
    "    global dW_f, dW_i, dW_C, dW_o, dW_y\n",
    "    global db_f, db_i, db_C, db_o, db_y\n",
    "\n",
    "    assert z.shape == (X_size + H_size, 1)\n",
    "    assert y.shape == (X_size, 1)\n",
    "    assert p.shape == (X_size, 1)\n",
    "\n",
    "    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:\n",
    "        assert param.shape == (H_size, 1)\n",
    "\n",
    "    dy = xp.copy(p)\n",
    "    dy[target] -= 1\n",
    "\n",
    "    dW_y += xp.dot(dy, h.T)\n",
    "    db_y += dy\n",
    "\n",
    "    dh = xp.dot(W_y.T, dy)\n",
    "    dh += dh_next\n",
    "    do = dh * tanh(C)\n",
    "    do = dsigmoid(o) * do\n",
    "    dW_o += xp.dot(do, z.T)\n",
    "    db_o += do\n",
    "\n",
    "    dC = xp.copy(dC_next)\n",
    "    dC += dh * o * dtanh(tanh(C))\n",
    "    dC_bar = dC * i\n",
    "    dC_bar = dC_bar * dtanh(C_bar)\n",
    "    dW_C += xp.dot(dC_bar, z.T)\n",
    "    db_C += dC_bar\n",
    "\n",
    "    di = dC * C_bar\n",
    "    di = dsigmoid(i) * di\n",
    "    dW_i += xp.dot(di, z.T)\n",
    "    db_i += di\n",
    "\n",
    "    df = dC * C_prev\n",
    "    df = dsigmoid(f) * df\n",
    "    dW_f += xp.dot(df, z.T)\n",
    "    db_f += df\n",
    "\n",
    "    dz = xp.dot(W_f.T, df) \\\n",
    "        + xp.dot(W_i.T, di) \\\n",
    "        + xp.dot(W_C.T, dC_bar) \\\n",
    "        + xp.dot(W_o.T, do)\n",
    "    dh_prev = dz[:H_size, :]\n",
    "    dC_prev = f * dC\n",
    "\n",
    "    return dh_prev, dC_prev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward Backward Pass\n",
    "\n",
    "Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.\n",
    "\n",
    "- `input`, `target` are list of integers, with character indexes.\n",
    "- `h_prev` is the array of initial `h` at $h_1$ (size $H \\times 1$)\n",
    "- `C_prev` is the array of initial `C` at $C_1$ (size $H \\times 1$)\n",
    "- Returns loss, final $h_T$ and $C_T$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def forward_backward(inputs, targets, h_prev, C_prev):\n",
    "    # To store the values for each time step\n",
    "    x_s, z_s, f_s, i_s, C_bar_s, C_s, o_s, h_s, y_s, p_s = {}, {}, {}, {}, {}, {}, {}, {}, {}, {}\n",
    "\n",
    "    # Values at t - 1\n",
    "    h_s[-1] = xp.copy(h_prev)\n",
    "    C_s[-1] = xp.copy(C_prev)\n",
    "\n",
    "    loss = 0\n",
    "    # Loop through time steps\n",
    "    assert len(inputs) == T_steps\n",
    "    for t in range(len(inputs)):\n",
    "        x_s[t] = xp.zeros((X_size, 1), dtype=xp.float32)\n",
    "        x_s[t][inputs[t]] = 1 # Input character\n",
    "\n",
    "        z_s[t], f_s[t], i_s[t], C_bar_s[t], C_s[t], o_s[t], h_s[t], y_s[t], p_s[t] \\\n",
    "            = forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass\n",
    "\n",
    "        loss += -xp.log(p_s[t][targets[t], 0]) # Loss for at t\n",
    "\n",
    "\n",
    "    for dparam in [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y]:\n",
    "        dparam.fill(0)\n",
    "\n",
    "    dh_next = xp.zeros_like(h_s[0], dtype=xp.float32) #dh from the next character\n",
    "    dC_next = xp.zeros_like(C_s[0], dtype=xp.float32) #dh from the next character\n",
    "\n",
    "    for t in reversed(range(len(inputs))):\n",
    "        # Backward pass\n",
    "        dh_next, dC_next = backward(target = targets[t], dh_next = dh_next, dC_next = dC_next, C_prev = C_s[t-1],\n",
    "                 z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t], C = C_s[t], o = o_s[t],\n",
    "                 h = h_s[t], y = y_s[t], p = p_s[t])\n",
    "\n",
    "    # Clip gradients to mitigate exploding gradients\n",
    "    for dparam in [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y]:\n",
    "        xp.clip(dparam, -1, 1, out=dparam)\n",
    "\n",
    "    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample the next character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample(h_prev, C_prev, first_char_idx, sentence_length):\n",
    "    x = xp.zeros((X_size, 1))\n",
    "    x[first_char_idx] = 1\n",
    "\n",
    "    h = h_prev\n",
    "    C = C_prev\n",
    "\n",
    "    indexes = []\n",
    "    for t in range(sentence_length):\n",
    "        _, _, _, _, C, _, h, _, p = forward(x, h, C)\n",
    "        assert xp.all(p.ravel() >= 0)\n",
    "        idx = xp.random.choice(xp.arange(X_size), size=(1,), p=p.ravel())[0]\n",
    "        x = xp.zeros((X_size, 1), dtype=xp.float32)\n",
    "        x[idx] = 1\n",
    "        indexes.append(int(idx))\n",
    "\n",
    "    return indexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training (Adagrad)\n",
    "\n",
    "$$\n",
    "w = w - \\eta \\frac{dw}{\\sum dw^2_{\\tau}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def update_status(inputs, h_prev, C_prev):\n",
    "    #initialized later\n",
    "    global plot_iter, plot_loss\n",
    "    global smooth_loss\n",
    "\n",
    "    # Get predictions for 200 letters with current model\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    sample_idx = sample(h_prev, C_prev, inputs[0], 200)\n",
    "    txt = ''.join(idx_to_char[idx] for idx in sample_idx)\n",
    "\n",
    "    # Clear and plot\n",
    "    plt.clf()\n",
    "    plt.plot(plot_iter, plot_loss)\n",
    "    display.display(plt.gcf())\n",
    "\n",
    "    #Print prediction and loss\n",
    "    print(\"----\\n %s \\n----\" % (txt, ))\n",
    "    print(\"iter %d, loss %f\" % (iteration, smooth_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Memory variables for Adagrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "mW_f = xp.zeros_like(W_f, dtype=xp.float32)\n",
    "mW_i = xp.zeros_like(W_i, dtype=xp.float32)\n",
    "mW_C = xp.zeros_like(W_C, dtype=xp.float32)\n",
    "mW_o = xp.zeros_like(W_o, dtype=xp.float32)\n",
    "mW_y = xp.zeros_like(W_y, dtype=xp.float32)\n",
    "\n",
    "mb_f = xp.zeros_like(b_f, dtype=xp.float32)\n",
    "mb_i = xp.zeros_like(b_i, dtype=xp.float32)\n",
    "mb_C = xp.zeros_like(b_C, dtype=xp.float32)\n",
    "mb_o = xp.zeros_like(b_o, dtype=xp.float32)\n",
    "mb_y = xp.zeros_like(b_y, dtype=xp.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exponential average of loss\n",
    "# Initialize to a error of a random model\n",
    "smooth_loss = (-xp.log(1.0 / X_size) * T_steps).astype(xp.float32)\n",
    "\n",
    "iteration, p = 0, 0\n",
    "\n",
    "# For the graph\n",
    "plot_iter = np.zeros((0))\n",
    "plot_loss = np.zeros((0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8FGX+B/DPN53eEooECCV0qZEuglQBxbPCqYeCx09O\nPe84DuGwoCeK5dTTU++wYgNRUBCUIlJEapBeQi+hht4JSZ7fHzu72TK7szWbmXzerxcvdmdnZ58n\n2Xznme9TRpRSICIi64qJdgGIiCiyGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJ\niCyOgZ6IyOLiol0AAEhOTlZpaWnRLgYRkamsXbv2hFIqxWi/YhHo09LSkJmZGe1iEBGZiojs92c/\npm6IiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJiCzO9IF+5vpDOHflWrSLQURUbJk6\n0GcdPY8npq7H6K83RrsoRETFlqkD/aXcPADAkXNXolwSIqLiy9SBnoiIjJk60KtoF4CIyARMHejt\nJNoFICIqxiwR6PefvBjtIhARFVuWCPSnL3F4JRGRN6YO9IpJeiIiQ6YO9EREZMzUgV7YC0tEZMgw\n0IvIRyJyXEQ2u21/XES2i8gWEXnFaftYEdklIlki0icShSYiIv/5c8/YTwD8B8Cn9g0i0h3AQAAt\nlVJXRaSqtr0pgEEAmgG4DsBPItJQKZUf7oIDzNETEfnDsEWvlFoK4JTb5hEAJiqlrmr7HNe2DwQw\nVSl1VSm1F8AuAO3CWF4iIgpQsDn6hgBuFJFVIrJERG7QttcEcNBpv2xtmwcRGS4imSKSmZOTE2Qx\niIjISLCBPg5AZQAdAPwdwDSRwLpGlVKTlFIZSqmMlJSUIItBRERGgg302QBmKJvVAAoAJAM4BKCW\n036p2rYIYZKeiMhIsIH+OwDdAUBEGgJIAHACwCwAg0QkUUTqAkgHsDocBSUiouAYjroRkSkAugFI\nFpFsAM8C+AjAR9qQy1wAQ5RSCsAWEZkGYCuAPACPRmrEjVa6yB2aiMgiDAO9Umqwl5fu97L/BAAT\nQikUERGFj6lnxjJHT0RkzOSBnoiIjDDQExFZnKkDPZdAICIyZupAT0RExhjoiYgsztSBnuvRExEZ\nM3WgZ46eiMiYqQM9EREZY6AnIrI4BnoiIoszdaBnip6IyJipAz0RERljoCcisjhTB3oOoyciMmbq\nQM8cPRGRMVMHeiIiMsZAT0RkcaYO9FwCgYjImGGgF5GPROS4diNw99f+JiJKRJKdto0VkV0ikiUi\nfcJdYCIiCow/LfpPAPR13ygitQD0BnDAaVtTAIMANNPe866IxIalpEREFBTDQK+UWgrglM5LbwAY\nDdfBLwMBTFVKXVVK7QWwC0C7cBSUiIiCE1SOXkQGAjiklNrg9lJNAAednmdr24iIKEriAn2DiJQG\n8A/Y0jZBE5HhAIYDQO3atUM5FBER+RBMi74+gLoANojIPgCpAH4TkeoADgGo5bRvqrbNg1JqklIq\nQymVkZKSEkQxiIjIHwEHeqXUJqVUVaVUmlIqDbb0TBul1FEAswAMEpFEEakLIB3A6rCWmIiIAuLP\n8MopAFYAaCQi2SIyzNu+SqktAKYB2ApgLoBHlVL54SqszudF6tBERJZhmKNXSg02eD3N7fkEABNC\nKxYREYWLqWfGEhGRMQZ6IiKLY6AnIrI4BnoiIotjoCcisjhTB3oOriQiMmbqQE9ERMYY6ImILI6B\nnojI4hjoiYgsztSBnkvdEBEZM3WgJyIiYwz0REQWZ+pArziSnojIkKkDPeM8EZExcwd6IiIyxEBP\nRGRxDPRERBZn6kDPFD0RkTFTB3oiIjJmGOhF5CMROS4im522vSoi20Vko4h8KyIVnV4bKyK7RCRL\nRPpEquAAZ8YSEfnDnxb9JwD6um1bAKC5UqoFgB0AxgKAiDQFMAhAM+0974pIbNhK64bj6ImIjBkG\neqXUUgCn3LbNV0rlaU9XAkjVHg8EMFUpdVUptRfALgDtwlheIiIKUDhy9EMB/Kg9rgngoNNr2do2\nDyIyXEQyRSQzJycnDMWInGPnriBtzBysO3A62kUhIgpYSIFeRMYByAPwRaDvVUpNUkplKKUyUlJS\ngvr8osrR/7rrBADg0xX7i+YDiYjCKC7YN4rIgwAGAOihlCPkHgJQy2m3VG1bRDBDT0RkLKgWvYj0\nBTAawG1KqUtOL80CMEhEEkWkLoB0AKtDLyYREQXLsEUvIlMAdAOQLCLZAJ6FbZRNIoAFIgIAK5VS\njyiltojINABbYUvpPKqUyo9U4SVSByYishDDQK+UGqyz+UMf+08AMCGUQvmLqRsiImOmnhmrOGOK\niMiQqQM9EREZM3WgZ3ueiMiYqQM9Iz0RkTFzB3oiIjLEQE9EZHEM9EREFmfqQM9liomIjJk60BMR\nkTFTB3rOlyIiMsZAT0RkcaYO9EREZMzUgZ4NeiIiY6YO9EREZIyBnojI4hjoiYgsztSBnuvRExEZ\nM3WgJyIiY6YO9M7t+e1Hz+FqXsRuT0tEZFqGgV5EPhKR4yKy2WlbZRFZICI7tf8rOb02VkR2iUiW\niPSJVMEB1wlTfd/8BeO+3ex9ZyKiEsqfFv0nAPq6bRsDYKFSKh3AQu05RKQpgEEAmmnveVdEYsNW\nWgNr958GAMxcfwgHTl4qqo8lIirWDAO9UmopgFNumwcCmKw9ngzgdqftU5VSV5VSewHsAtAuTGX1\nUDrB9Rxi75x9Yup69H/7l0h9LBGRqQSbo6+mlDqiPT4KoJr2uCaAg077ZWvbIiK9Wlmvr52/khep\njyUiMpWQO2OVrRkd8DhHERkuIpkikpmTkxNqMWxlAbDj2PmwHIuIyCqCDfTHRKQGAGj/H9e2HwJQ\ny2m/VG2bB6XUJKVUhlIqIyUlJchieOr9xtKwHYuIyAqCDfSzAAzRHg8BMNNp+yARSRSRugDSAawO\nrYj+i/T8KU7QIiIz8md45RQAKwA0EpFsERkGYCKAXiKyE0BP7TmUUlsATAOwFcBcAI8qpSI2uN09\n7hboBOLNh84iv4ABmohKrjijHZRSg7281MPL/hMATAilUMHKPn3Z5fnWw+cw4O1leKx7A4zq0yjk\n44tIyMcgIipqpp4Za+TY+SsAgE2Hzka5JERE0WPqQF8uyfCChIioxDN5oI8PaP9j567gs5X7I1Qa\nsrKCAoW0MXPw3yW7o10UooCZOtAHatjkNXj6u804cvay8c5ETvK1jv7X5mVFuSREgbN0oH/o4zUu\nz09dyAUAjsKhoPGbQ2Zk6UDvDUfPUKD4jSEzK5GBnihYnDRHZsRAT0RkcSUq0LMtRkQlUYkK9CXR\nz9uP4XIub7FIVJIx0FtY1tHzGPpJJsZ9tynaRbEMXhWSGZk+0I/r1yTg95SUERTnr1wDAOznbRVD\nxgBPZmb6QF8mkcsgeMPgRESABQJ9TADNc18j4y7l5jlawEREVmKBQB94IkbvLZ0m/ozrx88PQ4mK\nj5KSoipKHEZPZmT6QB+uSa5nLlmvNc+YFD4M8GRmFgj0gUf6w2euYNWekwGnasw6K5Ite6KSzfQ9\nmYHk6I+es92I5M73lgMAbmqYgslD20WiWERExYbpW/T+5OiX7MjBhat5Htu3Hz0X0GfZTxRERGZi\n+kC/cs9Jv/bbonM7QTFIaizafhxpY+bgpLa88co9pwIvYBSZNNNERGEWUqAXkb+KyBYR2SwiU0Qk\nSUQqi8gCEdmp/V8pXIXVc+BU8JOBjp67gk9+3ev19Y+017Yc1r/n7J6cCyiw2Nr2B05eQs75q2E5\nVm5eAS7qXEmZkWLXNrkZ+dV6/LDpSLSL4ZegA72I1ATwZwAZSqnmAGIBDAIwBsBCpVQ6gIXa84gJ\nZnils/HfbzXcR6/Dd8vhs7j5X0sw6Zc9AX2eUgqbi+hm5cH8aLq+ugg3TPgpLJ9/z/9WoNmz88Jy\nLG/yCxTOXrbeiKmisvnQWWw4eCbaxTClGesO4U9f/BbtYvgl1NRNHIBSIhIHoDSAwwAGApisvT4Z\nwO0hfoZP4RhemZdfEPB7Dp6y3Y7wt/2nA3rfrA2HMeDtZZiz8QiUUliUdTxiVwXRTt2sL4IAMvHH\nbWj53HxOdgvSgLeXYeA7v0a7GBRhQQd6pdQhAK8BOADgCICzSqn5AKoppezXM0cBVNN7v4gMF5FM\nEcnMyckJthh+u3fSSq+vzd1yVHe7PVCGc3jiruMXAAC7cy5g3pajeOjjNfhwmff0UaRtO3IOn63Y\nF7XPD9Xsjbav2rkr1kgREUVCKKmbSrC13usCuA5AGRG533kfZRt4rtuuVEpNUkplKKUyUlJSgi1G\nWFqti7P0TzSRzssePWsbxZN9OnqLjt3y71/w9MwtUfv8UBXVHIFoXx0RhSKU1E1PAHuVUjlKqWsA\nZgDoBOCYiNQAAO3/46EXM7K8pU7sf9znDNICpy/mhpR+iXQMKQm3yDXrZDbyz8FTl/DAh6t0h0mT\nsVAC/QEAHUSktNh6K3sA2AZgFoAh2j5DAMwMrYi+5RUEnl/34BYI8wuUS+D4aVvhuWrsjE1IH/eD\n4/n8rcfQ+p8L0O+tX5BvEOzz8gtwLd/7PsfOXfE6SuWHTUfwvyW7fR7fXUkIfrzRe8nw2vws/LLz\nBH7aeizaRTGlUHL0qwB8A+A3AJu0Y00CMBFALxHZCVurf2IYyulVaqXSIR/jrNM6NycuXEX9f/yA\nT5bv071cn7L6gG6w3n70PD72MVQTsKVJ/usjWLd/cSHueHe57mt/+uI3vPTjdp/Ht5u35Sg6vrQQ\neRYb+ulLCTinEQUtpFE3SqlnlVKNlVLNlVIPKKWuKqVOKqV6KKXSlVI9lVIRnWV0X/vaIR9j4fbC\nFnv2adtomglztmGFn5Ox7HbnXPD5+s7jrq9fzfO8Gsk6dj6gz7TbdfwCTl6wjX9/+rvNOHL2Ck5e\nzPX7/Veued5ucPOhs45jUmC+WnMA244ENvOajHE+Q3BMPzM2Uo3WYFrDU1YfdNyfdcex81i++4TX\nfV9fsMPRQg9Ha7Tn60tw87+W6L62Zp/xENDGT8/12Dbg7WW47T++h95tOHgGI79ab7mJY6F6cvom\n3PLvX6JdDCIAFgj0bWpXDOvxQs1rX8y15dh7v7EUv39/VTiK5OL1+VleX4vExKFDZ2xXOHtyLmDt\nfs+Ls2GT12DGukMBXT2Q/y5ezcNLP27D1Tze4B0wXraE9Jk+0Be3zrhIl+atn3f5fP1ybj6Oa0sY\nhLMsN/9rCe58b4Xf+xdFCz/r6HnHiai45+gnzNmqe6I08s6iXfjfkj34ctWBCJQqfLJPX8LnK/dH\n/HOYugmO6QN9uP3OS2eov8Jx4kkbMwc7gszVN3nGMwUTKm9XORPmbMUJbcE39z/Au/4b+M/x7OVr\nhiOXnPV5c2nAn+GPvPwCxxwHABj19YaQ1zR5/5e9AZ0o7XK1fpw8H6O1ioPB76/EU99tjtgM5eLV\nnDMfBvowm7n+kMvzRduP4+ftx7DvxMWAjuPvqpy+hCs0eIu97//ifZTRbwcCW/7gyrV8tHxuPp77\nPvqTt56fvRUdXlqIM5dsJ7Fv1mZj5LQNEfmsggKF5bu89+WEw6wNhzH6m8iU3+7MRVuA9/ad23ns\nPD4IcF0oZ8X7NFf8MdCH2XNui6Q99MkaDP0kE91eW+z1Pb/uPoFF2wOfV/bMzM3o/YZ+BywQvlZQ\nUYzHt3diz9pwOKj3KyiXEULHz13B8E8z/Zpg897i3Xh3cWFK7Gftd3HucuQn53z06178/oNVXseH\nh+Mn/+cp6zAtMzsMR9KXm1eA8wY/5wFvL8MLc7Zh5Z6TSBszx5Fy88el3DzMXB/c9yKc8vILHFdY\nZmOJQP9gp7RoFyEke3IuYsQXa122+QrS9pm6n67Yjx3HvA/pDFf3xSfL97k8X7H7JEZOW++yzdsy\nEr7szrmAGb+FJwDN+O0Q2r7wk2MlxtcX7MD8rcfwvR8njpfnbscrcws7ue0nh3Dlg32dKPdoV3pH\n/Lipzfkr1wLulC2Kk/R7i40n8tmHEk9ZbetrWLPXd39F9ulLjvTZZysin/v3x+/eXY6GT/0Y1HuV\nUlGdwGiJQB/qUsXh9uQ3GwN+z5Vr/rcU/J0hq/e9WrIjx7He/FkfN0T/ak1h598Lc7Y5Hn+6Yh8G\nv78SM35zTVGN/majz3Xs7/7vcjQc9yP2OM016Pn6EoyctgEz1x8KeXLXit22VFfWUde+jeW7A0+B\n2W8UH66/S3+OY/8GK6XwwuytjlSf870Qrh8/H3f/Vz/Pf/byNaSNmYP3l7qmRxYUwUzS05cCH3Fl\ndBLt8vIidHhpIQAgv4gC5Nr9p3wG403a8uJ7A0zDArYrmrpjf8DZy9fw3bpDxm8IM0sE+p5Nqhb5\nZ7750w6vr32VeTAsn3HqYi6u6Syh/HVmtstCaHqTnQDPFv3pi7kY8tFqx3rz7q1yZ09O36S7/Rkf\nC6DN2ei99bxm32nk5hc4xvrPXH/IEQCfmLre7/ytUgqLs4773WnrT4ve62cFuP9nK/ZhzHTPk3wg\nx9l1/AI+WLbXkeqz39XssvY73pitfy+DnPO21u8U7QQ96usNSBszJ6ggrCc3r8BwQmCkFEWcn7/l\nKO58bwW+8GN0U3cfaVhvthy2TZ67/Z1f8Zev1ns0SCLNEoG+U4PkIv/M7RH+RRUooM0/F2D0Nxs9\nLtePn7+KLi8vcjzXm+wEAC86tcSnZR7EFbfjHDkb3nvgBvL3+MRU15OMfUjoGR9XGe8u3oWMF37C\ngx+v8bqURDiH3wXagfn0zC2YusbzJK/XSjxzKVd31VJvpTcOdq5n9W/W2lJi9vsmhGrct5vQ419L\ncCqI+RLhGFgQafY71e3JCby1Hgj71YC3xlmkWCLQW5G9xfr9hsNBt2gOOwXy0d9s9DiO+2EjNTRu\n4bbA0gfuefuc81dxKTcPr8zNckzMOuh2C8nV+2wt35e1XLvz1cz6g2ew63jgJ2Z/ZhTr+drtik7v\n19fttcXo8vIij99JIEnIn7Yaj+aavGJfAEfU9/iUdfhaO3Fc8GPd/7z8ApeT26KswoEGgX6XjVaO\nJf9YJtC/eW+raBchrKZrwS6cC5O5H8m9pXn9+Plh+yxnwyZnujz/VWc4oXOAGznNlnZ4a+FOAMAN\nE37CQLelGPRazgB0W5y3v/Mrer7uOea+4bgfA14R1B9/d+uj0QtuhVcutheNupk2ZHsOV334U53R\nXEF+XZRSyC9QuqunBpr+ajDuRzz13ebgCuKmxfj5OHCy8KTu7UTx7bpspI2ZE/Ds8PwCVeSt62iw\nTKAvZv2xIbPn9IDCy/BQRbrX/1Juvl9XBfd94Lk0hHN97SY7jfZxXxAuHHLzC/xeEdSXq3n5juGh\nepzTSfY+Bnfjvt3scxG0nw2G39q//8H+hg+cuoSnZ25Gs2fnBTRpzZ39K/bFqgMuk87cXblW4Pfs\n6X0n9a9ajpy9jNfnZ0EphUlLbXM63K/0vPl2XTY6vrQQf/pirUvq88Cp0FM3D3y4Cn/4aHXIxwkn\nywT64jbyJpzC1TpyjvO2NffDcliHV+dl4frx83HMj6GC7o6fD29/QVHOpezzxlKfM5Kdf85fZ2bj\nwY/X6O7nawlrd5dyC1vewdzz2J1ShSknfwO9UgppY+Z4DL+1s4+acbZWu8fy2Bmb8NTM0L7Xj3+5\nDm/9vAtbDp9zNGL8DQNjpm/CkbNXMG+La1rR+d4Tgcg6et4xdPSXnSewdEeOz4ZVUQ+0tEyg79ow\nBTUqJEW7GKZx/PyViK0b0v+twFdt1Lvna1H/MZzWSfv4sxTFvpPeW5HnrlxzSb9lu00UmrK6MAUl\n8D9QNX1mnuPx3f8rHHK598RFrHLq/Dzv9HO9lJvnNU2hULhgmK/vhYLC5OX7MHbGJt0Tgm75nXZz\nnig1VQuM2acv4ZW5273f6c39uVIY/c0GZGonDed4GuyiZ+5Lhuecv4q0MXNcfpa+9HlzKcbO2IRp\nTinFb9cdwpVr+Y4Z1tFkmUBfoVQ8VoztEe1iFGt/+7pwFMl9H6zyOdkqFPb1bwKhFzTOXb7m9x+a\nP75acwBnLuV6Xbdm8PsrPS79e7/hmdt/fMo6DPHz0rzF+Plo/uw84x0BfBfk7M91B864hLd7J63U\n3a/pM/NcRms5u3g1z3ERZHSl9+ysLZiy+oDuiqXBXCV2eXkR3l28G8u8LQXhdszL1/JdZvo6n1yC\nvbB/dZ7rqrCZWuf+x7/uC+g4o52G146ctgGNn56LVs8vCK5QYRQX7QJEkkjxX9WwKK12mo0Y6WFk\n4ZBXoLwGrWA8OX0T1h88gymrD+LHJ270eH370fN+debZOycLChSGf5ZpsHdg3vxpp+Oxt3SW3to4\n/i6md8LLjWT+PHWd06Stwu2/HfA+8qj9i56pmdk+5lK4cy9zKLcFtZfZfkj7bN0R3eoDAD5fuR/l\nkuIwsFVN/47n9nyjTme4mVimRW83fURHx+Mm1ctHsSQULScvXHXkS93ZUyXebgqi11HqzZQ1B4LO\n6Xoze2Ph1catby/T3efrEDvnF247ht05F5A2Zo5j256ci04durYwV1CgPG5teclHpzNg61R29v7S\nPX7n/Id+kokvVu13mQ3sXB47vfSMfZ8Jc7Zh/8mLeHnudrw819bRvvXwOTz13WaPuRv+sP9MfN2A\nJ5i5BUXNcoG+bZ3K6FCvMgDg3htqRbk0FA17gpiibvfafO8znt25B7VwO3ZOv/Ud6uipYZMz8foC\nz3raA+gzM7dg/cEzeGeR570P3JdYMDLhh20BnZjGfbsZ/d9yPcE5z2cYOW2Dbsvf/iP5ZecJ3PTq\nYpfXbv1P4fHOXr6mO4Q0FG3+Gf3UjJGQUjciUhHABwCaw3a1MxRAFoCvAKQB2AfgHqVUcDNPQpRe\nrSz2Tezv0nIh64vGWiL+cl/GOlz8HVZop9dBaG+9frM2G99vOIyWqZ53b5sRxM/WWzos2HFR7yzy\nHJ3k69TnfEXR8rn5KJ/kO+yt2nPS59BQMwq1Rf9vAHOVUo0BtASwDcAYAAuVUukAFmrPi1RalTIA\ngPJJ8QCAzg2qFHURKIr8Wa8kWvb7GKHjL71O20DHbeulP5y3iAAnLhbPG8O7n6RsfXH+X+XojfBy\ndu+klXh+9laf+4Rq06GzLgv8RVrQgV5EKgDoCuBDAFBK5SqlzgAYCGCytttkALeHWshAjb+tGT4c\nkoHmNSsAAF65q2VRF4HIdC465d8FUmw77N37nX/ddQK7i2lZvXn6u82OBf6KQigt+roAcgB8LCLr\nROQDESkDoJpSyt6jdBRAtVALGaik+Fj0aFL4sddVSMKdbVKLuhhEVARe/MH77ObGT+uvH+9v+//H\nzUd1t4dr2YSiupFJKIE+DkAbAO8ppVoDuAi3NI2yXU/p/kxFZLiIZIpIZk5O4DetCISIYMLvmkf0\nM4jMxGgm8uUiWv9lzsbA78UbyBINgdznIRDeVowNlF6HdyRIsD34IlIdwEqlVJr2/EbYAn0DAN2U\nUkdEpAaAxUqpRr6OlZGRoTIzwzse2V1efgEajAvu7jBEZC2JcTEes2Gj5cMhGS4ZiECIyFqlVIbR\nfkG36JVSRwEcFBF7EO8BYCuAWQCGaNuGAJgZ7GeEU2yMddfCIaLAFJcgDwBv+LiJUbiEOjP2cQBf\niEgCgD0AHoLt5DFNRIYB2A/gnhA/Iyz8nTlIRFSUNh/yvmppuIQU6JVS6wHoXTZw0RkiomLCcjNj\nfVny924uz2c/3iU6BSEiKkIlKtDXqVIGpeJjHc9FgKf6N4liiYiIIq9EBXrAdbKFUsADHetErzBE\nREWgxAV6d8HeqICIyCxKdKAXsd69ZomI3JXoQA8AsYz0RGRxJT7Qx8QI9k3sH+1iEBFFTIkL9C5L\nsTI/T0QlQIkL9H2aVQ9o/yY1yiM+1nZCMLphARFRcVTiAv3Ld7XAm/e2Qq3KpVAvpYxj+/8eaKu7\nv/NNpGc+5jrB6tHu9SNTSCKiMCpxgT4+Nga3t66JX0bfjCSnyVPOLf0143rqvtd9XbSKpRL8+sxX\n7mqBoZ3rBl5YIqIwKHGB3h8p5RJ1t7vn9NvUqeTzODUrlgIA3JNRC/k6NzQmIioKTDo7+eLh9sjN\ntwXkH5+40ZGbt7OPxKxaLhFLR3dHUnws6iaXwd4Tnrcxe6x7A/yhYx3s0u4LGakbIETTsC518eGy\nvdEuBhEZYIveSecGyejeqCoAWydsg6rlABSmdaqUtaVq/tqroSPt88yApgCATvWroK+233v3tcGo\nPo1QtXwSOtVPBgBcKqI79hSlHk2qRrsIROQHBno/vH5PK6z6Rw+UTojDvon9MbhdbcdrLWtVBAA8\nfKPvHPyfutVHWpXSLu8lIioKDPR+SIiLQbXySbqvVS6TgH0T++Pmxr5vBdakRnks/nt3/HNgs6DK\nYM/3FydxMfz6EJkB/1LDyH5D5YQ47z/WuNgYvHtfm4CPfWfb1KDLFSlcPYLIHBjow2h413oonRCL\njDqVfe7X7/oaAR/73htqBbR/zzDmzxtULRu2YxFR0WOgD6PODZKx9fm+qFA6PizHcx7PX7NiKXw2\nrJ3j+RcPt3c8rq6bVpKwzeQd0ML/E1MwVyvh4HxDmXDwNsSWyIwY6IuJ7o1SAAC1K5c23Ld+Shl0\nbpDsmLVbJtEzyHlLq/Rs4tqX8K+7Wxp+nlL62/U+olSCZ1kSfaSywmHPi/2w9fk+hvuVC+DEN+fP\nvM0kWUfIf4EiEisi60Rktva8sogsEJGd2v++ZxURAODjh9ph+ohO+PZPnfDFw+0x4XfNHUHT3hFr\nD7g1Ktiely9lu3Kom+w7tTKw1XWOxx8Mcb2XezC5/4TYGAxuVwuta/v3q133TC+M6t3QcL+bG1f1\nuZLo7U71cBYTIxA/Ogzi3Kc2e5EQF+P3rGerua6C/qADMrdwNLWeALDN6fkYAAuVUukAFmrPyYtX\n72qB17RWdds6lVClbCI6N0jGfe3roGxiHBaP6oafR93k8h57TKtZsRQ+fugGvHFvS7x5bytULuMa\nnDrWrwJzf+YkAAAPeUlEQVTA8yphcLta+L+u9fDz32zHva2lfgAFbC1b9wb9jgm34KU7WiDWj8CZ\nEBeD0glxePjGeniwU5pj+196pnvs+/o9vq8u9IamDuvi/9ISBV6uTNzteOEWnx3qzj50O3FGyh8N\nhu+Gy9AAfp5kHiEFehFJBdAfwAdOmwcCmKw9ngzg9lA+w+ruzqiFu3y0qtOSyyAxznv+uXujqiiX\nFI/bW9fEsie7O7YLgH8Pao0Ff+2KCqVc+wxeuqMFxvZrgnopnlcCbw1u7Xj89SMd0ey6Co5LiTpV\nSuPJvo39rZqLpPhYjO1X+N6/9GyIzKdc1xSqWNp7K3rfxP6opZPWelqbsOYP5ZaDapfm2Wk+7y9d\nPba5z5Cu6NQHU1Qjj8om2j5zeNd62P1iP+yacIvuft8/1gUbx/cO+nP8veohcwm1Rf8mgNEAnOf3\nV1NKHdEeHwWgO8BcRIaLSKaIZObk5IRYDHJXv2pZJMXHIr1aOcMboNuD1Zv3tnJp3d+gBUJ7eLyj\ndSpGdPO9YucNaZWRXrUspo/o6PGarxOWs3VP98LkobaO54c6p2Hmo539ep+31nVy2UTExwrKJrrm\n6Kc94lnGRtXLeWx79z7XlU1Xju3h6OhOjIvF58Pae7wnUhLjYhAbI4iLjcE7v3ft+I6LEVyfWgHl\nk4IfDJDg5+/ISh7qnBbtIkRc0IFeRAYAOK6UWuttH2VrQuleMCulJimlMpRSGSkpKcEWw7Tm/uXG\nsHf4lYqPxV1tU/G3Xg0xsldhTtwowN7awhbc7bN83dlb0rUqe5+0Nbhdbfxf13oomxiHBSNvsl0J\nGPDWdqxUJgE3NUzBnD93wdP9mzrKZdR67tGkGhrrBOrHutfHzgn9EBsbXGu1V9PCtsrnw9ojKT4W\n9bUhp0nxseiSnhzUcf3l3EJ3rkF/t9FQf+1l3A9i5M62NUM+Rri4X4lGSmUfV5KRdGebopsbE8r4\nu84AbhORfgCSAJQXkc8BHBORGkqpIyJSA8DxcBTUahpXLx/we4xSzCLiyPcHomfTai6doOWT4nDu\nSp7j+d1tU5FaqRQ61qvi8d6RvRoiNkbwaPcGARfauQN11T96eLzufrLwNvrH2Vwt9ZI2Zg4A27pD\n9rWKfC0g2qhaOb9+dvagfmebVKw7cAZ1k8v43P/BTmn4ZPk+44LrKJ0Qi0u5+SifFA9l8Nt378SO\njxVcy/ezU0ITFyNIjItFzYqlcOjM5YDLGw7JZRNw4kIuAGBcvyYYPX1jwMeoUCoeZy9fC3fR/FKr\ncikcPOXfz64oJxwG3aJXSo1VSqUqpdIADALws1LqfgCzAAzRdhsCYGbIpSQAQBWtszW9qmerNZwW\njLzJJfUiIuhUP1l3ZMufe6R7D/I6Zj7aGYtGdbMd12m7tyUmQnXL9TUQo+Wd83Qi/VfDO6BFagV8\nPaIjrk81vgqxu79DHex9qZ9HB7i7GjqjWAa0qIHFo7qhTe3CKyj3voatz/fBr0/ejF/H3AygsHVb\n3q2VW6VMAh65yTOdtvk54+Gm7uynhS4NPK9QXrmrRUDHMlrqIyG2MPTsfrGf4/HXj3RyPI6PCy4S\nfvV/HQLaP5wB19vf5t6X+nlsc+/7iaRIDHCeCKCXiOwE0FN7TmHQvGYFTB3eAWNuCbxDdO1TPbF6\nnGerWU+18kloazC7N1gta1V0tIJD/QNrGUBgBoB8nWE37etVwazHugSV1/ZnSKfeqKAneqQjLbmM\nox8C8BzWWDohDpXKJDiG1j7QoQ7+eXtzl5FLALD26V6634fEuFjsfakfRnSrj9mPB5Yi1KvWPRm1\n8PjNthN6p/qFV3aj+zby2LdUfCy6NkzR3qefnnC+QomNEWwc3xurx/VA3eQymP14F1xXIQldGgSX\n0m1cvTz+3qewXGlVjOemhMNfe+qnzpLiYzy+K18+XHT9OkCYAr1SarFSaoD2+KRSqodSKl0p1VMp\ndSocn0E2HepV8Xvon7MqZRNRtVzxGiMdTD2cud/a0dmiUd1cZg8DCDiVEarB7WojLta1ji/dcT3S\nq9lafeWS4rF8zM149tamqG4wfj0uNgYPdKjjcTxfRARP9m2M5jX9OyH21voivKXI/ta7ESbecT3e\nu78t6iWXwZN9G2No57oY0a0+bkgrnFPRsHo51KlSBnte7IdX7vJMh7WtUwkNtZ+BfcJe+aR4x/ez\nec0KWD62B5LLJnic2LzxtV/H+p4pR2fOQdh56ZCsF/r69dkA0LdZdTzRM91jZJc3nRok44831gMA\n9A9iSZRAcWYsRU3phDhMH9ExpOGA3tRNts0edpaXH7mbv7xyVwt8+XB7l45bPT0au65BdF3FUnio\nc120TK2IR26qj5fuuB5v3tsqImV0XwF12ZPdHSfbxaO64c1Bts+1t7b1UjiD2tVGhVLx+HlUN4zo\nVh9J8bF4sm9jR4d/i9QKjmUw7Ckze99OsnY/h8S4GEz43fVoXbuiR4eyMxHB+Nv0U0B/vLGuS3rx\nbi9XDrc0r+71GHomPVA4cisxLhZtalfEf37f2sc7XDlfNNrvSe0t9tdLKYsNz/Z2GdIcKbzDFEWE\nv2mZQFJE1con4cb0ZIzoVh+tvIwQ8mVg65r4ctWBgN8HANNHdESMj0rdk2FbdK5Tg2R8vnI/nvpu\ns8c+vmb9xsRIUCk5f/00siuSyyai1fMLHNtSK5V29JNUr5DkCNa3trwO0zKzMf62Zuj5+pKAPufv\nfRp5nFAmD22HK3n52HjwLO7/cBVEgFa1KuLbP/k3bNbdp0PbOVJDdp4d97boWi+ljEt/gJ4HO6Xh\n1XlZAGy/h43jeyNPu/qboZXxsS/X+TyG/avhHNPtP1tfjfyiGlnEFj1FREJsDNrWqYS3A2gNGYmN\nEXw2rD061U9G6YTA2yj/HNgcG8f39po39qVtncp+L/lQHDWoWk53Qlp6NdswUedz2I3pKdg3sX9A\nq5b6GhWUEBeD8knxKF/K9jurXdn3SCUj7kFetzxacQT6y2M8e6ut87t93coo4za/onxSvEcn+1P9\nm+C525oZ5vv/2jPdMelsQIvrUKtyKbx81/Uu+6wc619fWTixRU8RISKYPqKT8Y5FKDZGUD4pHq/c\n1VI3d+zu60c6Irls4KtYmmmd/k+HtsfmQ2f9nszmjXNg9aZFakV88IeMkOYd3Nc+uDu0tUurjNX7\nCrsL7RPj7L+r1EqlkH3a+7DIh7V8+se/2u6RvGhUN9SsWAoNn/oRQGHqqHXtStj1Yj+cuZSL8knx\nLnMbWteuiHZ1Kxv2x0QCAz2RFzfoLJFgdrMf74IBby9zPK9cJsFnC3lcvyZ4dX6W4XH7NKuO5btP\noo5Bi7enQR+Gng71KqNJjfJ49lbPXPuXD7dHvh8doE2vK+8I9Hc7LTlif+vsx7s4xu/7Ukq7khTY\nrlRWjL0Zl3LzUd9tORG9q6dgU1XhwEBPFGb22bkZdYpfqsffETh2f+xaD3/sWs9wvz90rIM726Z6\nLDMRDlOHey5VYdfJqcP4H/0a48Nltha3t9D/zICmGNqlLpbvPuGyvWLpBJ9rLdm9/4e2mLn+sOOE\nZl9JtrhjoCcKs7Z1KmPl2B4ul+jF7UYmwaSkfBHxXEuoqA3vWh/Du7pOHnNPoxk9N5JaqXRAEwSL\nCwZ6oghwDvK/Pd0r5DkD4fT9Y12ikicuSvHaSBv7DeztN7+xb7f/H0ynvhmVjFoSRZHRMglFLZCl\nHszqoc5pOHMpF8O1tNOfe6QjNkYcw2Az6lTCqN4Nde9xYEXi70yuSMrIyFCZmZnRLgYRkamIyFql\nlOHdb4rP9SQREUUEAz0RkcUx0BMRWRwDPRGRxTHQExFZHAM9EZHFMdATEVkcAz0RkcUViwlTIpID\nYH8Ih0gGcMJwr+LPKvUAWJfiyCr1AFgXuzpKKcMF+otFoA+ViGT6MzusuLNKPQDWpTiySj0A1iVQ\nTN0QEVkcAz0RkcVZJdBPinYBwsQq9QBYl+LIKvUAWJeAWCJHT0RE3lmlRU9ERF6YOtCLSF8RyRKR\nXSIyJtrl0SMiH4nIcRHZ7LStsogsEJGd2v+VnF4bq9UnS0T6OG1vKyKbtNfeEgn0Jmgh16OWiCwS\nka0iskVEnjBxXZJEZLWIbNDq8pxZ66KVIVZE1onIbJPXY59WhvUikmnyulQUkW9EZLuIbBORjlGt\ni1LKlP8AxALYDaAegAQAGwA0jXa5dMrZFUAbAJudtr0CYIz2eAyAl7XHTbV6JAKoq9UvVnttNYAO\nsN2A/kcAtxRxPWoAaKM9Lgdgh1ZeM9ZFAJTVHscDWKWVx3R10cowEsCXAGab9fullWEfgGS3bWat\ny2QAD2uPEwBUjGZdirTyYf5BdgQwz+n5WABjo10uL2VNg2ugzwJQQ3tcA0CWXh0AzNPqWQPAdqft\ngwH8L8p1mgmgl9nrAqA0gN8AtDdjXQCkAlgI4GYUBnrT1UP73H3wDPSmqwuACgD2QusDLQ51MXPq\npiaAg07Ps7VtZlBNKXVEe3wUQDXtsbc61dQeu2+PChFJA9AatpawKeuipTvWAzgOYIFSyqx1eRPA\naAAFTtvMWA8AUAB+EpG1IjJc22bGutQFkAPgYy2l9oGIlEEU62LmQG8JynaqNs3QJxEpC2A6gL8o\npc45v2amuiil8pVSrWBrEbcTkeZurxf7uojIAADHlVJrve1jhno46aL9Tm4B8KiIdHV+0UR1iYMt\nXfueUqo1gIuwpWocirouZg70hwDUcnqeqm0zg2MiUgMAtP+Pa9u91emQ9th9e5ESkXjYgvwXSqkZ\n2mZT1sVOKXUGwCIAfWG+unQGcJuI7AMwFcDNIvI5zFcPAIBS6pD2/3EA3wJoB3PWJRtAtnaVCADf\nwBb4o1YXMwf6NQDSRaSuiCQAGARgVpTL5K9ZAIZoj4fAlu+2bx8kIokiUhdAOoDV2uXeORHpoPW6\n/8HpPUVC+9wPAWxTSr3u9JIZ65IiIhW1x6Vg62vYDpPVRSk1VimVqpRKg+37/7NS6n6z1QMARKSM\niJSzPwbQG8BmmLAuSqmjAA6KSCNtUw8AWxHNuhR1h0uYOz36wTb6YzeAcdEuj5cyTgFwBMA12M70\nwwBUga0DbSeAnwBUdtp/nFafLDj1sAPIgO2LvxvAf+DW0VME9egC26XmRgDrtX/9TFqXFgDWaXXZ\nDOAZbbvp6uJUjm4o7Iw1XT1gGz23Qfu3xf73bMa6aGVoBSBT+459B6BSNOvCmbFERBZn5tQNERH5\ngYGeiMjiGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMji/h9rDZqjFxSujQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0a4f11dcc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----\n",
      " outhant her abnuck a casthantes Pandors,\n",
      "Are you corpore; surt is. in\n",
      "Is boo all you\n",
      "\n",
      "KLUS:\n",
      "Goke to knowt but, he have be pedaen lost'd Cold led. Ravert was, soperen, but brous.\n",
      "\n",
      "KIN:\n",
      "-flak\n",
      "Thym;,\n",
      "Wit \n",
      "----\n",
      "iter 6000, loss 49.692995\n",
      "CPU times: user 12min 4s, sys: 15.6 s, total: 12min 19s\n",
      "Wall time: 12min 21s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8FGX+B/DPN53eEooECCV0qZEuglQBxbPCqYeCx09O\nPe84DuGwoCeK5dTTU++wYgNRUBCUIlJEapBeQi+hht4JSZ7fHzu72TK7szWbmXzerxcvdmdnZ58n\n2Xznme9TRpRSICIi64qJdgGIiCiyGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJ\niCyOgZ6IyOLiol0AAEhOTlZpaWnRLgYRkamsXbv2hFIqxWi/YhHo09LSkJmZGe1iEBGZiojs92c/\npm6IiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJiCzO9IF+5vpDOHflWrSLQURUbJk6\n0GcdPY8npq7H6K83RrsoRETFlqkD/aXcPADAkXNXolwSIqLiy9SBnoiIjJk60KtoF4CIyARMHejt\nJNoFICIqxiwR6PefvBjtIhARFVuWCPSnL3F4JRGRN6YO9IpJeiIiQ6YO9EREZMzUgV7YC0tEZMgw\n0IvIRyJyXEQ2u21/XES2i8gWEXnFaftYEdklIlki0icShSYiIv/5c8/YTwD8B8Cn9g0i0h3AQAAt\nlVJXRaSqtr0pgEEAmgG4DsBPItJQKZUf7oIDzNETEfnDsEWvlFoK4JTb5hEAJiqlrmr7HNe2DwQw\nVSl1VSm1F8AuAO3CWF4iIgpQsDn6hgBuFJFVIrJERG7QttcEcNBpv2xtmwcRGS4imSKSmZOTE2Qx\niIjISLCBPg5AZQAdAPwdwDSRwLpGlVKTlFIZSqmMlJSUIItBRERGgg302QBmKJvVAAoAJAM4BKCW\n036p2rYIYZKeiMhIsIH+OwDdAUBEGgJIAHACwCwAg0QkUUTqAkgHsDocBSUiouAYjroRkSkAugFI\nFpFsAM8C+AjAR9qQy1wAQ5RSCsAWEZkGYCuAPACPRmrEjVa6yB2aiMgiDAO9Umqwl5fu97L/BAAT\nQikUERGFj6lnxjJHT0RkzOSBnoiIjDDQExFZnKkDPZdAICIyZupAT0RExhjoiYgsztSBnuvRExEZ\nM3WgZ46eiMiYqQM9EREZY6AnIrI4BnoiIoszdaBnip6IyJipAz0RERljoCcisjhTB3oOoyciMmbq\nQM8cPRGRMVMHeiIiMsZAT0RkcaYO9FwCgYjImGGgF5GPROS4diNw99f+JiJKRJKdto0VkV0ikiUi\nfcJdYCIiCow/LfpPAPR13ygitQD0BnDAaVtTAIMANNPe866IxIalpEREFBTDQK+UWgrglM5LbwAY\nDdfBLwMBTFVKXVVK7QWwC0C7cBSUiIiCE1SOXkQGAjiklNrg9lJNAAednmdr24iIKEriAn2DiJQG\n8A/Y0jZBE5HhAIYDQO3atUM5FBER+RBMi74+gLoANojIPgCpAH4TkeoADgGo5bRvqrbNg1JqklIq\nQymVkZKSEkQxiIjIHwEHeqXUJqVUVaVUmlIqDbb0TBul1FEAswAMEpFEEakLIB3A6rCWmIiIAuLP\n8MopAFYAaCQi2SIyzNu+SqktAKYB2ApgLoBHlVL54SqszudF6tBERJZhmKNXSg02eD3N7fkEABNC\nKxYREYWLqWfGEhGRMQZ6IiKLY6AnIrI4BnoiIotjoCcisjhTB3oOriQiMmbqQE9ERMYY6ImILI6B\nnojI4hjoiYgsztSBnkvdEBEZM3WgJyIiYwz0REQWZ+pArziSnojIkKkDPeM8EZExcwd6IiIyxEBP\nRGRxDPRERBZn6kDPFD0RkTFTB3oiIjJmGOhF5CMROS4im522vSoi20Vko4h8KyIVnV4bKyK7RCRL\nRPpEquAAZ8YSEfnDnxb9JwD6um1bAKC5UqoFgB0AxgKAiDQFMAhAM+0974pIbNhK64bj6ImIjBkG\neqXUUgCn3LbNV0rlaU9XAkjVHg8EMFUpdVUptRfALgDtwlheIiIKUDhy9EMB/Kg9rgngoNNr2do2\nDyIyXEQyRSQzJycnDMWInGPnriBtzBysO3A62kUhIgpYSIFeRMYByAPwRaDvVUpNUkplKKUyUlJS\ngvr8osrR/7rrBADg0xX7i+YDiYjCKC7YN4rIgwAGAOihlCPkHgJQy2m3VG1bRDBDT0RkLKgWvYj0\nBTAawG1KqUtOL80CMEhEEkWkLoB0AKtDLyYREQXLsEUvIlMAdAOQLCLZAJ6FbZRNIoAFIgIAK5VS\njyiltojINABbYUvpPKqUyo9U4SVSByYishDDQK+UGqyz+UMf+08AMCGUQvmLqRsiImOmnhmrOGOK\niMiQqQM9EREZM3WgZ3ueiMiYqQM9Iz0RkTFzB3oiIjLEQE9EZHEM9EREFmfqQM9liomIjJk60BMR\nkTFTB3rOlyIiMsZAT0RkcaYO9EREZMzUgZ4NeiIiY6YO9EREZIyBnojI4hjoiYgsztSBnuvRExEZ\nM3WgJyIiY6YO9M7t+e1Hz+FqXsRuT0tEZFqGgV5EPhKR4yKy2WlbZRFZICI7tf8rOb02VkR2iUiW\niPSJVMEB1wlTfd/8BeO+3ex9ZyKiEsqfFv0nAPq6bRsDYKFSKh3AQu05RKQpgEEAmmnveVdEYsNW\nWgNr958GAMxcfwgHTl4qqo8lIirWDAO9UmopgFNumwcCmKw9ngzgdqftU5VSV5VSewHsAtAuTGX1\nUDrB9Rxi75x9Yup69H/7l0h9LBGRqQSbo6+mlDqiPT4KoJr2uCaAg077ZWvbIiK9Wlmvr52/khep\njyUiMpWQO2OVrRkd8DhHERkuIpkikpmTkxNqMWxlAbDj2PmwHIuIyCqCDfTHRKQGAGj/H9e2HwJQ\ny2m/VG2bB6XUJKVUhlIqIyUlJchieOr9xtKwHYuIyAqCDfSzAAzRHg8BMNNp+yARSRSRugDSAawO\nrYj+i/T8KU7QIiIz8md45RQAKwA0EpFsERkGYCKAXiKyE0BP7TmUUlsATAOwFcBcAI8qpSI2uN09\n7hboBOLNh84iv4ABmohKrjijHZRSg7281MPL/hMATAilUMHKPn3Z5fnWw+cw4O1leKx7A4zq0yjk\n44tIyMcgIipqpp4Za+TY+SsAgE2Hzka5JERE0WPqQF8uyfCChIioxDN5oI8PaP9j567gs5X7I1Qa\nsrKCAoW0MXPw3yW7o10UooCZOtAHatjkNXj6u804cvay8c5ETvK1jv7X5mVFuSREgbN0oH/o4zUu\nz09dyAUAjsKhoPGbQ2Zk6UDvDUfPUKD4jSEzK5GBnihYnDRHZsRAT0RkcSUq0LMtRkQlUYkK9CXR\nz9uP4XIub7FIVJIx0FtY1tHzGPpJJsZ9tynaRbEMXhWSGZk+0I/r1yTg95SUERTnr1wDAOznbRVD\nxgBPZmb6QF8mkcsgeMPgRESABQJ9TADNc18j4y7l5jlawEREVmKBQB94IkbvLZ0m/ozrx88PQ4mK\nj5KSoipKHEZPZmT6QB+uSa5nLlmvNc+YFD4M8GRmFgj0gUf6w2euYNWekwGnasw6K5Ite6KSzfQ9\nmYHk6I+es92I5M73lgMAbmqYgslD20WiWERExYbpW/T+5OiX7MjBhat5Htu3Hz0X0GfZTxRERGZi\n+kC/cs9Jv/bbonM7QTFIaizafhxpY+bgpLa88co9pwIvYBSZNNNERGEWUqAXkb+KyBYR2SwiU0Qk\nSUQqi8gCEdmp/V8pXIXVc+BU8JOBjp67gk9+3ev19Y+017Yc1r/n7J6cCyiw2Nr2B05eQs75q2E5\nVm5eAS7qXEmZkWLXNrkZ+dV6/LDpSLSL4ZegA72I1ATwZwAZSqnmAGIBDAIwBsBCpVQ6gIXa84gJ\nZnils/HfbzXcR6/Dd8vhs7j5X0sw6Zc9AX2eUgqbi+hm5cH8aLq+ugg3TPgpLJ9/z/9WoNmz88Jy\nLG/yCxTOXrbeiKmisvnQWWw4eCbaxTClGesO4U9f/BbtYvgl1NRNHIBSIhIHoDSAwwAGApisvT4Z\nwO0hfoZP4RhemZdfEPB7Dp6y3Y7wt/2nA3rfrA2HMeDtZZiz8QiUUliUdTxiVwXRTt2sL4IAMvHH\nbWj53HxOdgvSgLeXYeA7v0a7GBRhQQd6pdQhAK8BOADgCICzSqn5AKoppezXM0cBVNN7v4gMF5FM\nEcnMyckJthh+u3fSSq+vzd1yVHe7PVCGc3jiruMXAAC7cy5g3pajeOjjNfhwmff0UaRtO3IOn63Y\nF7XPD9Xsjbav2rkr1kgREUVCKKmbSrC13usCuA5AGRG533kfZRt4rtuuVEpNUkplKKUyUlJSgi1G\nWFqti7P0TzSRzssePWsbxZN9OnqLjt3y71/w9MwtUfv8UBXVHIFoXx0RhSKU1E1PAHuVUjlKqWsA\nZgDoBOCYiNQAAO3/46EXM7K8pU7sf9znDNICpy/mhpR+iXQMKQm3yDXrZDbyz8FTl/DAh6t0h0mT\nsVAC/QEAHUSktNh6K3sA2AZgFoAh2j5DAMwMrYi+5RUEnl/34BYI8wuUS+D4aVvhuWrsjE1IH/eD\n4/n8rcfQ+p8L0O+tX5BvEOzz8gtwLd/7PsfOXfE6SuWHTUfwvyW7fR7fXUkIfrzRe8nw2vws/LLz\nBH7aeizaRTGlUHL0qwB8A+A3AJu0Y00CMBFALxHZCVurf2IYyulVaqXSIR/jrNM6NycuXEX9f/yA\nT5bv071cn7L6gG6w3n70PD72MVQTsKVJ/usjWLd/cSHueHe57mt/+uI3vPTjdp/Ht5u35Sg6vrQQ\neRYb+ulLCTinEQUtpFE3SqlnlVKNlVLNlVIPKKWuKqVOKqV6KKXSlVI9lVIRnWV0X/vaIR9j4fbC\nFnv2adtomglztmGFn5Ox7HbnXPD5+s7jrq9fzfO8Gsk6dj6gz7TbdfwCTl6wjX9/+rvNOHL2Ck5e\nzPX7/Veued5ucPOhs45jUmC+WnMA244ENvOajHE+Q3BMPzM2Uo3WYFrDU1YfdNyfdcex81i++4TX\nfV9fsMPRQg9Ha7Tn60tw87+W6L62Zp/xENDGT8/12Dbg7WW47T++h95tOHgGI79ab7mJY6F6cvom\n3PLvX6JdDCIAFgj0bWpXDOvxQs1rX8y15dh7v7EUv39/VTiK5OL1+VleX4vExKFDZ2xXOHtyLmDt\nfs+Ls2GT12DGukMBXT2Q/y5ezcNLP27D1Tze4B0wXraE9Jk+0Be3zrhIl+atn3f5fP1ybj6Oa0sY\nhLMsN/9rCe58b4Xf+xdFCz/r6HnHiai45+gnzNmqe6I08s6iXfjfkj34ctWBCJQqfLJPX8LnK/dH\n/HOYugmO6QN9uP3OS2eov8Jx4kkbMwc7gszVN3nGMwUTKm9XORPmbMUJbcE39z/Au/4b+M/x7OVr\nhiOXnPV5c2nAn+GPvPwCxxwHABj19YaQ1zR5/5e9AZ0o7XK1fpw8H6O1ioPB76/EU99tjtgM5eLV\nnDMfBvowm7n+kMvzRduP4+ftx7DvxMWAjuPvqpy+hCs0eIu97//ifZTRbwcCW/7gyrV8tHxuPp77\nPvqTt56fvRUdXlqIM5dsJ7Fv1mZj5LQNEfmsggKF5bu89+WEw6wNhzH6m8iU3+7MRVuA9/ad23ns\nPD4IcF0oZ8X7NFf8MdCH2XNui6Q99MkaDP0kE91eW+z1Pb/uPoFF2wOfV/bMzM3o/YZ+BywQvlZQ\nUYzHt3diz9pwOKj3KyiXEULHz13B8E8z/Zpg897i3Xh3cWFK7Gftd3HucuQn53z06178/oNVXseH\nh+Mn/+cp6zAtMzsMR9KXm1eA8wY/5wFvL8MLc7Zh5Z6TSBszx5Fy88el3DzMXB/c9yKc8vILHFdY\nZmOJQP9gp7RoFyEke3IuYsQXa122+QrS9pm6n67Yjx3HvA/pDFf3xSfL97k8X7H7JEZOW++yzdsy\nEr7szrmAGb+FJwDN+O0Q2r7wk2MlxtcX7MD8rcfwvR8njpfnbscrcws7ue0nh3Dlg32dKPdoV3pH\n/Lipzfkr1wLulC2Kk/R7i40n8tmHEk9ZbetrWLPXd39F9ulLjvTZZysin/v3x+/eXY6GT/0Y1HuV\nUlGdwGiJQB/qUsXh9uQ3GwN+z5Vr/rcU/J0hq/e9WrIjx7He/FkfN0T/ak1h598Lc7Y5Hn+6Yh8G\nv78SM35zTVGN/majz3Xs7/7vcjQc9yP2OM016Pn6EoyctgEz1x8KeXLXit22VFfWUde+jeW7A0+B\n2W8UH66/S3+OY/8GK6XwwuytjlSf870Qrh8/H3f/Vz/Pf/byNaSNmYP3l7qmRxYUwUzS05cCH3Fl\ndBLt8vIidHhpIQAgv4gC5Nr9p3wG403a8uJ7A0zDArYrmrpjf8DZy9fw3bpDxm8IM0sE+p5Nqhb5\nZ7750w6vr32VeTAsn3HqYi6u6Syh/HVmtstCaHqTnQDPFv3pi7kY8tFqx3rz7q1yZ09O36S7/Rkf\nC6DN2ei99bxm32nk5hc4xvrPXH/IEQCfmLre7/ytUgqLs4773WnrT4ve62cFuP9nK/ZhzHTPk3wg\nx9l1/AI+WLbXkeqz39XssvY73pitfy+DnPO21u8U7QQ96usNSBszJ6ggrCc3r8BwQmCkFEWcn7/l\nKO58bwW+8GN0U3cfaVhvthy2TZ67/Z1f8Zev1ns0SCLNEoG+U4PkIv/M7RH+RRUooM0/F2D0Nxs9\nLtePn7+KLi8vcjzXm+wEAC86tcSnZR7EFbfjHDkb3nvgBvL3+MRU15OMfUjoGR9XGe8u3oWMF37C\ngx+v8bqURDiH3wXagfn0zC2YusbzJK/XSjxzKVd31VJvpTcOdq5n9W/W2lJi9vsmhGrct5vQ419L\ncCqI+RLhGFgQafY71e3JCby1Hgj71YC3xlmkWCLQW5G9xfr9hsNBt2gOOwXy0d9s9DiO+2EjNTRu\n4bbA0gfuefuc81dxKTcPr8zNckzMOuh2C8nV+2wt35e1XLvz1cz6g2ew63jgJ2Z/ZhTr+drtik7v\n19fttcXo8vIij99JIEnIn7Yaj+aavGJfAEfU9/iUdfhaO3Fc8GPd/7z8ApeT26KswoEGgX6XjVaO\nJf9YJtC/eW+raBchrKZrwS6cC5O5H8m9pXn9+Plh+yxnwyZnujz/VWc4oXOAGznNlnZ4a+FOAMAN\nE37CQLelGPRazgB0W5y3v/Mrer7uOea+4bgfA14R1B9/d+uj0QtuhVcutheNupk2ZHsOV334U53R\nXEF+XZRSyC9QuqunBpr+ajDuRzz13ebgCuKmxfj5OHCy8KTu7UTx7bpspI2ZE/Ds8PwCVeSt62iw\nTKAvZv2xIbPn9IDCy/BQRbrX/1Juvl9XBfd94Lk0hHN97SY7jfZxXxAuHHLzC/xeEdSXq3n5juGh\nepzTSfY+Bnfjvt3scxG0nw2G39q//8H+hg+cuoSnZ25Gs2fnBTRpzZ39K/bFqgMuk87cXblW4Pfs\n6X0n9a9ajpy9jNfnZ0EphUlLbXM63K/0vPl2XTY6vrQQf/pirUvq88Cp0FM3D3y4Cn/4aHXIxwkn\nywT64jbyJpzC1TpyjvO2NffDcliHV+dl4frx83HMj6GC7o6fD29/QVHOpezzxlKfM5Kdf85fZ2bj\nwY/X6O7nawlrd5dyC1vewdzz2J1ShSknfwO9UgppY+Z4DL+1s4+acbZWu8fy2Bmb8NTM0L7Xj3+5\nDm/9vAtbDp9zNGL8DQNjpm/CkbNXMG+La1rR+d4Tgcg6et4xdPSXnSewdEeOz4ZVUQ+0tEyg79ow\nBTUqJEW7GKZx/PyViK0b0v+twFdt1Lvna1H/MZzWSfv4sxTFvpPeW5HnrlxzSb9lu00UmrK6MAUl\n8D9QNX1mnuPx3f8rHHK598RFrHLq/Dzv9HO9lJvnNU2hULhgmK/vhYLC5OX7MHbGJt0Tgm75nXZz\nnig1VQuM2acv4ZW5273f6c39uVIY/c0GZGonDed4GuyiZ+5Lhuecv4q0MXNcfpa+9HlzKcbO2IRp\nTinFb9cdwpVr+Y4Z1tFkmUBfoVQ8VoztEe1iFGt/+7pwFMl9H6zyOdkqFPb1bwKhFzTOXb7m9x+a\nP75acwBnLuV6Xbdm8PsrPS79e7/hmdt/fMo6DPHz0rzF+Plo/uw84x0BfBfk7M91B864hLd7J63U\n3a/pM/NcRms5u3g1z3ERZHSl9+ysLZiy+oDuiqXBXCV2eXkR3l28G8u8LQXhdszL1/JdZvo6n1yC\nvbB/dZ7rqrCZWuf+x7/uC+g4o52G146ctgGNn56LVs8vCK5QYRQX7QJEkkjxX9WwKK12mo0Y6WFk\n4ZBXoLwGrWA8OX0T1h88gymrD+LHJ270eH370fN+debZOycLChSGf5ZpsHdg3vxpp+Oxt3SW3to4\n/i6md8LLjWT+PHWd06Stwu2/HfA+8qj9i56pmdk+5lK4cy9zKLcFtZfZfkj7bN0R3eoDAD5fuR/l\nkuIwsFVN/47n9nyjTme4mVimRW83fURHx+Mm1ctHsSQULScvXHXkS93ZUyXebgqi11HqzZQ1B4LO\n6Xoze2Ph1catby/T3efrEDvnF247ht05F5A2Zo5j256ci04durYwV1CgPG5teclHpzNg61R29v7S\nPX7n/Id+kokvVu13mQ3sXB47vfSMfZ8Jc7Zh/8mLeHnudrw819bRvvXwOTz13WaPuRv+sP9MfN2A\nJ5i5BUXNcoG+bZ3K6FCvMgDg3htqRbk0FA17gpiibvfafO8znt25B7VwO3ZOv/Ud6uipYZMz8foC\nz3raA+gzM7dg/cEzeGeR570P3JdYMDLhh20BnZjGfbsZ/d9yPcE5z2cYOW2Dbsvf/iP5ZecJ3PTq\nYpfXbv1P4fHOXr6mO4Q0FG3+Gf3UjJGQUjciUhHABwCaw3a1MxRAFoCvAKQB2AfgHqVUcDNPQpRe\nrSz2Tezv0nIh64vGWiL+cl/GOlz8HVZop9dBaG+9frM2G99vOIyWqZ53b5sRxM/WWzos2HFR7yzy\nHJ3k69TnfEXR8rn5KJ/kO+yt2nPS59BQMwq1Rf9vAHOVUo0BtASwDcAYAAuVUukAFmrPi1RalTIA\ngPJJ8QCAzg2qFHURKIr8Wa8kWvb7GKHjL71O20DHbeulP5y3iAAnLhbPG8O7n6RsfXH+X+XojfBy\ndu+klXh+9laf+4Rq06GzLgv8RVrQgV5EKgDoCuBDAFBK5SqlzgAYCGCytttkALeHWshAjb+tGT4c\nkoHmNSsAAF65q2VRF4HIdC465d8FUmw77N37nX/ddQK7i2lZvXn6u82OBf6KQigt+roAcgB8LCLr\nROQDESkDoJpSyt6jdBRAtVALGaik+Fj0aFL4sddVSMKdbVKLuhhEVARe/MH77ObGT+uvH+9v+//H\nzUd1t4dr2YSiupFJKIE+DkAbAO8ppVoDuAi3NI2yXU/p/kxFZLiIZIpIZk5O4DetCISIYMLvmkf0\nM4jMxGgm8uUiWv9lzsbA78UbyBINgdznIRDeVowNlF6HdyRIsD34IlIdwEqlVJr2/EbYAn0DAN2U\nUkdEpAaAxUqpRr6OlZGRoTIzwzse2V1efgEajAvu7jBEZC2JcTEes2Gj5cMhGS4ZiECIyFqlVIbR\nfkG36JVSRwEcFBF7EO8BYCuAWQCGaNuGAJgZ7GeEU2yMddfCIaLAFJcgDwBv+LiJUbiEOjP2cQBf\niEgCgD0AHoLt5DFNRIYB2A/gnhA/Iyz8nTlIRFSUNh/yvmppuIQU6JVS6wHoXTZw0RkiomLCcjNj\nfVny924uz2c/3iU6BSEiKkIlKtDXqVIGpeJjHc9FgKf6N4liiYiIIq9EBXrAdbKFUsADHetErzBE\nREWgxAV6d8HeqICIyCxKdKAXsd69ZomI3JXoQA8AsYz0RGRxJT7Qx8QI9k3sH+1iEBFFTIkL9C5L\nsTI/T0QlQIkL9H2aVQ9o/yY1yiM+1nZCMLphARFRcVTiAv3Ld7XAm/e2Qq3KpVAvpYxj+/8eaKu7\nv/NNpGc+5jrB6tHu9SNTSCKiMCpxgT4+Nga3t66JX0bfjCSnyVPOLf0143rqvtd9XbSKpRL8+sxX\n7mqBoZ3rBl5YIqIwKHGB3h8p5RJ1t7vn9NvUqeTzODUrlgIA3JNRC/k6NzQmIioKTDo7+eLh9sjN\ntwXkH5+40ZGbt7OPxKxaLhFLR3dHUnws6iaXwd4Tnrcxe6x7A/yhYx3s0u4LGakbIETTsC518eGy\nvdEuBhEZYIveSecGyejeqCoAWydsg6rlABSmdaqUtaVq/tqroSPt88yApgCATvWroK+233v3tcGo\nPo1QtXwSOtVPBgBcKqI79hSlHk2qRrsIROQHBno/vH5PK6z6Rw+UTojDvon9MbhdbcdrLWtVBAA8\nfKPvHPyfutVHWpXSLu8lIioKDPR+SIiLQbXySbqvVS6TgH0T++Pmxr5vBdakRnks/nt3/HNgs6DK\nYM/3FydxMfz6EJkB/1LDyH5D5YQ47z/WuNgYvHtfm4CPfWfb1KDLFSlcPYLIHBjow2h413oonRCL\njDqVfe7X7/oaAR/73htqBbR/zzDmzxtULRu2YxFR0WOgD6PODZKx9fm+qFA6PizHcx7PX7NiKXw2\nrJ3j+RcPt3c8rq6bVpKwzeQd0ML/E1MwVyvh4HxDmXDwNsSWyIwY6IuJ7o1SAAC1K5c23Ld+Shl0\nbpDsmLVbJtEzyHlLq/Rs4tqX8K+7Wxp+nlL62/U+olSCZ1kSfaSywmHPi/2w9fk+hvuVC+DEN+fP\nvM0kWUfIf4EiEisi60Rktva8sogsEJGd2v++ZxURAODjh9ph+ohO+PZPnfDFw+0x4XfNHUHT3hFr\nD7g1Ktiely9lu3Kom+w7tTKw1XWOxx8Mcb2XezC5/4TYGAxuVwuta/v3q133TC+M6t3QcL+bG1f1\nuZLo7U71cBYTIxA/Ogzi3Kc2e5EQF+P3rGerua6C/qADMrdwNLWeALDN6fkYAAuVUukAFmrPyYtX\n72qB17RWdds6lVClbCI6N0jGfe3roGxiHBaP6oafR93k8h57TKtZsRQ+fugGvHFvS7x5bytULuMa\nnDrWrwJzf+YkAAAPeUlEQVTA8yphcLta+L+u9fDz32zHva2lfgAFbC1b9wb9jgm34KU7WiDWj8CZ\nEBeD0glxePjGeniwU5pj+196pnvs+/o9vq8u9IamDuvi/9ISBV6uTNzteOEWnx3qzj50O3FGyh8N\nhu+Gy9AAfp5kHiEFehFJBdAfwAdOmwcCmKw9ngzg9lA+w+ruzqiFu3y0qtOSyyAxznv+uXujqiiX\nFI/bW9fEsie7O7YLgH8Pao0Ff+2KCqVc+wxeuqMFxvZrgnopnlcCbw1u7Xj89SMd0ey6Co5LiTpV\nSuPJvo39rZqLpPhYjO1X+N6/9GyIzKdc1xSqWNp7K3rfxP6opZPWelqbsOYP5ZaDapfm2Wk+7y9d\nPba5z5Cu6NQHU1Qjj8om2j5zeNd62P1iP+yacIvuft8/1gUbx/cO+nP8veohcwm1Rf8mgNEAnOf3\nV1NKHdEeHwWgO8BcRIaLSKaIZObk5IRYDHJXv2pZJMXHIr1aOcMboNuD1Zv3tnJp3d+gBUJ7eLyj\ndSpGdPO9YucNaZWRXrUspo/o6PGarxOWs3VP98LkobaO54c6p2Hmo539ep+31nVy2UTExwrKJrrm\n6Kc94lnGRtXLeWx79z7XlU1Xju3h6OhOjIvF58Pae7wnUhLjYhAbI4iLjcE7v3ft+I6LEVyfWgHl\nk4IfDJDg5+/ISh7qnBbtIkRc0IFeRAYAOK6UWuttH2VrQuleMCulJimlMpRSGSkpKcEWw7Tm/uXG\nsHf4lYqPxV1tU/G3Xg0xsldhTtwowN7awhbc7bN83dlb0rUqe5+0Nbhdbfxf13oomxiHBSNvsl0J\nGPDWdqxUJgE3NUzBnD93wdP9mzrKZdR67tGkGhrrBOrHutfHzgn9EBsbXGu1V9PCtsrnw9ojKT4W\n9bUhp0nxseiSnhzUcf3l3EJ3rkF/t9FQf+1l3A9i5M62NUM+Rri4X4lGSmUfV5KRdGebopsbE8r4\nu84AbhORfgCSAJQXkc8BHBORGkqpIyJSA8DxcBTUahpXLx/we4xSzCLiyPcHomfTai6doOWT4nDu\nSp7j+d1tU5FaqRQ61qvi8d6RvRoiNkbwaPcGARfauQN11T96eLzufrLwNvrH2Vwt9ZI2Zg4A27pD\n9rWKfC0g2qhaOb9+dvagfmebVKw7cAZ1k8v43P/BTmn4ZPk+44LrKJ0Qi0u5+SifFA9l8Nt378SO\njxVcy/ezU0ITFyNIjItFzYqlcOjM5YDLGw7JZRNw4kIuAGBcvyYYPX1jwMeoUCoeZy9fC3fR/FKr\ncikcPOXfz64oJxwG3aJXSo1VSqUqpdIADALws1LqfgCzAAzRdhsCYGbIpSQAQBWtszW9qmerNZwW\njLzJJfUiIuhUP1l3ZMufe6R7D/I6Zj7aGYtGdbMd12m7tyUmQnXL9TUQo+Wd83Qi/VfDO6BFagV8\nPaIjrk81vgqxu79DHex9qZ9HB7i7GjqjWAa0qIHFo7qhTe3CKyj3voatz/fBr0/ejF/H3AygsHVb\n3q2VW6VMAh65yTOdtvk54+Gm7uynhS4NPK9QXrmrRUDHMlrqIyG2MPTsfrGf4/HXj3RyPI6PCy4S\nfvV/HQLaP5wB19vf5t6X+nlsc+/7iaRIDHCeCKCXiOwE0FN7TmHQvGYFTB3eAWNuCbxDdO1TPbF6\nnGerWU+18kloazC7N1gta1V0tIJD/QNrGUBgBoB8nWE37etVwazHugSV1/ZnSKfeqKAneqQjLbmM\nox8C8BzWWDohDpXKJDiG1j7QoQ7+eXtzl5FLALD26V6634fEuFjsfakfRnSrj9mPB5Yi1KvWPRm1\n8PjNthN6p/qFV3aj+zby2LdUfCy6NkzR3qefnnC+QomNEWwc3xurx/VA3eQymP14F1xXIQldGgSX\n0m1cvTz+3qewXGlVjOemhMNfe+qnzpLiYzy+K18+XHT9OkCYAr1SarFSaoD2+KRSqodSKl0p1VMp\ndSocn0E2HepV8Xvon7MqZRNRtVzxGiMdTD2cud/a0dmiUd1cZg8DCDiVEarB7WojLta1ji/dcT3S\nq9lafeWS4rF8zM149tamqG4wfj0uNgYPdKjjcTxfRARP9m2M5jX9OyH21voivKXI/ta7ESbecT3e\nu78t6iWXwZN9G2No57oY0a0+bkgrnFPRsHo51KlSBnte7IdX7vJMh7WtUwkNtZ+BfcJe+aR4x/ez\nec0KWD62B5LLJnic2LzxtV/H+p4pR2fOQdh56ZCsF/r69dkA0LdZdTzRM91jZJc3nRok44831gMA\n9A9iSZRAcWYsRU3phDhMH9ExpOGA3tRNts0edpaXH7mbv7xyVwt8+XB7l45bPT0au65BdF3FUnio\nc120TK2IR26qj5fuuB5v3tsqImV0XwF12ZPdHSfbxaO64c1Bts+1t7b1UjiD2tVGhVLx+HlUN4zo\nVh9J8bF4sm9jR4d/i9QKjmUw7Ckze99OsnY/h8S4GEz43fVoXbuiR4eyMxHB+Nv0U0B/vLGuS3rx\nbi9XDrc0r+71GHomPVA4cisxLhZtalfEf37f2sc7XDlfNNrvSe0t9tdLKYsNz/Z2GdIcKbzDFEWE\nv2mZQFJE1con4cb0ZIzoVh+tvIwQ8mVg65r4ctWBgN8HANNHdESMj0rdk2FbdK5Tg2R8vnI/nvpu\ns8c+vmb9xsRIUCk5f/00siuSyyai1fMLHNtSK5V29JNUr5DkCNa3trwO0zKzMf62Zuj5+pKAPufv\nfRp5nFAmD22HK3n52HjwLO7/cBVEgFa1KuLbP/k3bNbdp0PbOVJDdp4d97boWi+ljEt/gJ4HO6Xh\n1XlZAGy/h43jeyNPu/qboZXxsS/X+TyG/avhHNPtP1tfjfyiGlnEFj1FREJsDNrWqYS3A2gNGYmN\nEXw2rD061U9G6YTA2yj/HNgcG8f39po39qVtncp+L/lQHDWoWk53Qlp6NdswUedz2I3pKdg3sX9A\nq5b6GhWUEBeD8knxKF/K9jurXdn3SCUj7kFetzxacQT6y2M8e6ut87t93coo4za/onxSvEcn+1P9\nm+C525oZ5vv/2jPdMelsQIvrUKtyKbx81/Uu+6wc619fWTixRU8RISKYPqKT8Y5FKDZGUD4pHq/c\n1VI3d+zu60c6Irls4KtYmmmd/k+HtsfmQ2f9nszmjXNg9aZFakV88IeMkOYd3Nc+uDu0tUurjNX7\nCrsL7RPj7L+r1EqlkH3a+7DIh7V8+se/2u6RvGhUN9SsWAoNn/oRQGHqqHXtStj1Yj+cuZSL8knx\nLnMbWteuiHZ1Kxv2x0QCAz2RFzfoLJFgdrMf74IBby9zPK9cJsFnC3lcvyZ4dX6W4XH7NKuO5btP\noo5Bi7enQR+Gng71KqNJjfJ49lbPXPuXD7dHvh8doE2vK+8I9Hc7LTlif+vsx7s4xu/7Ukq7khTY\nrlRWjL0Zl3LzUd9tORG9q6dgU1XhwEBPFGb22bkZdYpfqsffETh2f+xaD3/sWs9wvz90rIM726Z6\nLDMRDlOHey5VYdfJqcP4H/0a48Nltha3t9D/zICmGNqlLpbvPuGyvWLpBJ9rLdm9/4e2mLn+sOOE\nZl9JtrhjoCcKs7Z1KmPl2B4ul+jF7UYmwaSkfBHxXEuoqA3vWh/Du7pOHnNPoxk9N5JaqXRAEwSL\nCwZ6oghwDvK/Pd0r5DkD4fT9Y12ikicuSvHaSBv7DeztN7+xb7f/H0ynvhmVjFoSRZHRMglFLZCl\nHszqoc5pOHMpF8O1tNOfe6QjNkYcw2Az6lTCqN4Nde9xYEXi70yuSMrIyFCZmZnRLgYRkamIyFql\nlOHdb4rP9SQREUUEAz0RkcUx0BMRWRwDPRGRxTHQExFZHAM9EZHFMdATEVkcAz0RkcUViwlTIpID\nYH8Ih0gGcMJwr+LPKvUAWJfiyCr1AFgXuzpKKcMF+otFoA+ViGT6MzusuLNKPQDWpTiySj0A1iVQ\nTN0QEVkcAz0RkcVZJdBPinYBwsQq9QBYl+LIKvUAWJeAWCJHT0RE3lmlRU9ERF6YOtCLSF8RyRKR\nXSIyJtrl0SMiH4nIcRHZ7LStsogsEJGd2v+VnF4bq9UnS0T6OG1vKyKbtNfeEgn0Jmgh16OWiCwS\nka0iskVEnjBxXZJEZLWIbNDq8pxZ66KVIVZE1onIbJPXY59WhvUikmnyulQUkW9EZLuIbBORjlGt\ni1LKlP8AxALYDaAegAQAGwA0jXa5dMrZFUAbAJudtr0CYIz2eAyAl7XHTbV6JAKoq9UvVnttNYAO\nsN2A/kcAtxRxPWoAaKM9Lgdgh1ZeM9ZFAJTVHscDWKWVx3R10cowEsCXAGab9fullWEfgGS3bWat\ny2QAD2uPEwBUjGZdirTyYf5BdgQwz+n5WABjo10uL2VNg2ugzwJQQ3tcA0CWXh0AzNPqWQPAdqft\ngwH8L8p1mgmgl9nrAqA0gN8AtDdjXQCkAlgI4GYUBnrT1UP73H3wDPSmqwuACgD2QusDLQ51MXPq\npiaAg07Ps7VtZlBNKXVEe3wUQDXtsbc61dQeu2+PChFJA9AatpawKeuipTvWAzgOYIFSyqx1eRPA\naAAFTtvMWA8AUAB+EpG1IjJc22bGutQFkAPgYy2l9oGIlEEU62LmQG8JynaqNs3QJxEpC2A6gL8o\npc45v2amuiil8pVSrWBrEbcTkeZurxf7uojIAADHlVJrve1jhno46aL9Tm4B8KiIdHV+0UR1iYMt\nXfueUqo1gIuwpWocirouZg70hwDUcnqeqm0zg2MiUgMAtP+Pa9u91emQ9th9e5ESkXjYgvwXSqkZ\n2mZT1sVOKXUGwCIAfWG+unQGcJuI7AMwFcDNIvI5zFcPAIBS6pD2/3EA3wJoB3PWJRtAtnaVCADf\nwBb4o1YXMwf6NQDSRaSuiCQAGARgVpTL5K9ZAIZoj4fAlu+2bx8kIokiUhdAOoDV2uXeORHpoPW6\n/8HpPUVC+9wPAWxTSr3u9JIZ65IiIhW1x6Vg62vYDpPVRSk1VimVqpRKg+37/7NS6n6z1QMARKSM\niJSzPwbQG8BmmLAuSqmjAA6KSCNtUw8AWxHNuhR1h0uYOz36wTb6YzeAcdEuj5cyTgFwBMA12M70\nwwBUga0DbSeAnwBUdtp/nFafLDj1sAPIgO2LvxvAf+DW0VME9egC26XmRgDrtX/9TFqXFgDWaXXZ\nDOAZbbvp6uJUjm4o7Iw1XT1gGz23Qfu3xf73bMa6aGVoBSBT+459B6BSNOvCmbFERBZn5tQNERH5\ngYGeiMjiGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMji/h9rDZqjFxSujQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0a4f11dcc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%time\n",
    "while True:\n",
    "    # Try catch for interruption\n",
    "    try:\n",
    "        # Reset\n",
    "        if p + T_steps >= len(data) or iteration == 0:\n",
    "            g_h_prev = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "            g_C_prev = xp.zeros((H_size, 1), dtype=xp.float32)\n",
    "            p = 0\n",
    "\n",
    "        inputs = [char_to_idx[ch] for ch in data[p: p + T_steps]]\n",
    "        targets = [char_to_idx[ch] for ch in data[p + 1: p + T_steps + 1]]\n",
    "\n",
    "        loss, g_h_prev, g_C_prev =  forward_backward(inputs, targets, g_h_prev, g_C_prev)\n",
    "        smooth_loss = smooth_loss * 0.999 + loss * 0.001\n",
    "\n",
    "        # Print every hundred steps\n",
    "        if iteration % 100 == 0:\n",
    "            update_status(inputs, g_h_prev, g_C_prev)\n",
    "\n",
    "        # Update weights\n",
    "        for param, dparam, mem in zip([W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y],\n",
    "                                      [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y],\n",
    "                                      [mW_f, mW_i, mW_C, mW_o, mW_y, mb_f, mb_i, mb_C, mb_o, mb_y]):\n",
    "            mem += dparam * dparam # Calculate sum of gradients\n",
    "            #print(learning_rate * dparam)\n",
    "            param += -(learning_rate * dparam / xp.sqrt(mem + 1e-8))\n",
    "\n",
    "        plot_iter = np.append(plot_iter, [iteration])\n",
    "        if isinstance(loss, cp.ndarray):\n",
    "            loss = loss.get()\n",
    "        plot_loss = np.append(plot_loss, [loss])\n",
    "\n",
    "        p += T_steps\n",
    "        iteration += 1\n",
    "        if iteration > 6000:\n",
    "            break\n",
    "    except KeyboardInterrupt:\n",
    "        update_status(inputs, g_h_prev, g_C_prev)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Check\n",
    "\n",
    "Approximate the numerical gradients by changing parameters and running the model. Check if the approximated gradients are equal to the computed analytical gradients (by backpropagation).\n",
    "\n",
    "Try this on `num_checks` individual paramters picked randomly for each weight matrix and bias vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import uniform\n",
    "\n",
    "\n",
    "def gradient_check(inputs, target, h_prev, C_prev):\n",
    "    global W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y\n",
    "    global dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y\n",
    "\n",
    "    num_checks = 10 # Number of parameters to test\n",
    "    delta = 1e-5 # The change to make on the parameter\n",
    "\n",
    "    # To calculate computed gradients\n",
    "    _, _, _ =  forward_backward(inputs, targets, h_prev, C_prev)\n",
    "\n",
    "    for param, dparam, name in zip([W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y],\n",
    "                                 [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y],\n",
    "                                 ['W_f', 'W_i', 'W_C', 'W_o', 'W_y', 'b_f', 'b_i', 'b_C', 'b_o', 'b_y']):\n",
    "        assert param.shape == dparam.shape\n",
    "        dparam_copy = xp.copy(dparam) #Make a copy because this will get modified\n",
    "\n",
    "        # Test num_checks times\n",
    "        for i in range(num_checks):\n",
    "            # Pick a random index\n",
    "            rnd_idx = int(uniform(0,param.size))\n",
    "\n",
    "            # evaluate cost at [x + delta] and [x - delta]\n",
    "            old_val = xp.ravel(param)[rnd_idx]\n",
    "            xp.ravel(param)[rnd_idx] = old_val + delta\n",
    "            loss_plus_delta, _, _ = forward_backward(inputs, targets, h_prev, C_prev)\n",
    "            xp.ravel(param)[rnd_idx] = old_val - delta\n",
    "            loss_mins_delta, _, _ = forward_backward(inputs, targets, h_prev, C_prev)\n",
    "            xp.ravel(param)[rnd_idx] = old_val\n",
    "\n",
    "            grad_analytical = xp.ravel(dparam_copy)[rnd_idx]\n",
    "            grad_numerical = (loss_plus_delta - loss_mins_delta) / (2 * delta)\n",
    "            # Clip numerical error because grad_analytical is clipped\n",
    "            grad_numerical = xp.clip(grad_numerical, -1, 1)\n",
    "\n",
    "\n",
    "            err_sum = abs(grad_numerical + grad_analytical) + 1e-09\n",
    "            rel_error = abs(grad_analytical - grad_numerical) / err_sum\n",
    "\n",
    "            # If relative error is greater than 1e-06\n",
    "            if rel_error > 1e-06:\n",
    "                print('%s (%e, %e) => %e' % (name, grad_numerical, grad_analytical, rel_error))    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_f (-5.788818e-03, -1.894525e-03) => 5.068488e-01\n",
      "W_f (-4.424725e-03, -2.570756e-03) => 2.650237e-01\n",
      "W_f (-1.574892e-02, 3.109146e-04) => 1.040279e+00\n",
      "W_f (2.522772e-02, -2.049634e-05) => 1.001626e+00\n",
      "W_f (-2.772584e-02, -3.596031e-04) => 9.743922e-01\n",
      "W_f (-9.696586e-02, -3.551726e-02) => 4.638221e-01\n",
      "W_f (-1.919746e-02, -4.452238e-05) => 9.953723e-01\n",
      "W_i (-2.128421e-02, -1.987336e-03) => 8.292046e-01\n",
      "W_i (-9.618537e-03, -1.770927e-03) => 6.890236e-01\n",
      "W_i (-5.882980e-02, 1.477409e-03) => 1.051520e+00\n",
      "W_i (-4.811230e-02, -8.973669e-03) => 6.856085e-01\n",
      "W_i (-2.069689e-02, 2.397710e-03) => 1.262057e+00\n",
      "W_C (-9.150008e-03, -4.846476e-03) => 3.074723e-01\n",
      "W_C (-5.384253e-02, 3.501519e-03) => 1.139112e+00\n",
      "W_C (0.000000e+00, 1.724286e-05) => 9.999420e-01\n",
      "W_C (-3.551228e-02, -4.914250e-03) => 7.568799e-01\n",
      "W_C (0.000000e+00, 2.751392e-05) => 9.999637e-01\n",
      "W_C (-2.206686e-02, 1.656087e-04) => 1.015123e+00\n",
      "W_C (-4.734427e-03, 3.258538e-02) => 1.339983e+00\n",
      "W_C (5.200395e-03, 4.812866e-02) => 8.049695e-01\n",
      "W_o (1.664602e-02, 1.216984e-02) => 1.553371e-01\n",
      "W_o (-3.094253e-03, -1.786075e-02) => 7.046765e-01\n",
      "W_y (5.069651e-05, 9.411758e-05) => 2.998380e-01\n",
      "W_y (1.354294e-06, 3.190190e-06) => 4.038944e-01\n",
      "W_y (-1.707993e-01, -3.533963e-01) => 3.483377e-01\n",
      "W_y (-3.616201e-05, -6.439822e-05) => 2.807863e-01\n",
      "W_y (1.045780e-04, 2.551565e-04) => 4.185810e-01\n",
      "W_y (-7.008616e-05, -6.961373e-05) => 3.381729e-03\n",
      "W_y (-1.320173e-03, -3.049548e-03) => 3.957632e-01\n",
      "W_y (-2.135536e-05, -5.128610e-05) => 4.120281e-01\n",
      "W_y (-3.736993e-05, -3.475788e-05) => 3.621374e-02\n",
      "W_y (4.692321e-03, 1.760141e-02) => 5.790457e-01\n",
      "b_f (9.625265e-02, 1.281972e-01) => 1.423236e-01\n",
      "b_f (-1.711278e-02, -8.734714e-03) => 3.241346e-01\n",
      "b_f (-6.363861e-02, -9.571880e-02) => 2.013097e-01\n",
      "b_f (-1.353493e-01, -2.709278e-01) => 3.337093e-01\n",
      "b_f (1.295448e-01, 2.294842e-01) => 2.783604e-01\n",
      "b_f (0.000000e+00, 1.190781e-05) => 9.999160e-01\n",
      "b_f (-1.097183e-02, 2.446750e-02) => 2.625979e+00\n",
      "b_f (6.660086e-03, 1.548321e-01) => 9.175182e-01\n",
      "b_f (-3.765070e-02, -1.929383e-03) => 9.025073e-01\n",
      "b_f (8.172772e-03, -1.733225e-02) => 2.784549e+00\n",
      "b_i (-1.797925e-02, 3.960431e-03) => 1.565016e+00\n",
      "b_i (2.327358e-04, 1.566012e-02) => 9.707118e-01\n",
      "b_i (3.910756e-05, 1.577477e-02) => 9.950540e-01\n",
      "b_i (-1.699091e-01, -2.288668e-01) => 1.478467e-01\n",
      "b_i (-7.337276e-02, -9.197736e-02) => 1.125164e-01\n",
      "b_i (-3.058013e-02, 9.644181e-03) => 1.921303e+00\n",
      "b_i (6.721953e-02, 1.486464e-01) => 3.772104e-01\n",
      "b_i (-8.904879e-02, -5.259708e-02) => 2.573440e-01\n",
      "b_i (-8.196457e-02, -5.133832e-02) => 2.297493e-01\n",
      "b_i (-1.127150e-02, 2.934633e-03) => 1.704013e+00\n",
      "b_C (-8.878277e-02, -9.282753e-02) => 2.227166e-02\n",
      "b_C (-1.053689e-01, -2.198471e-01) => 3.520066e-01\n",
      "b_C (-1.254594e-01, -1.306037e-01) => 2.008990e-02\n",
      "b_C (-8.069183e-03, 4.495833e-02) => 1.437483e+00\n",
      "b_C (1.315191e-01, 3.286199e-01) => 4.283507e-01\n",
      "b_C (-2.857717e-01, -5.199677e-01) => 2.906598e-01\n",
      "b_C (-3.367491e-01, -5.167727e-01) => 2.109186e-01\n",
      "b_C (-8.336149e-02, -2.002720e-01) => 4.121887e-01\n",
      "b_C (-2.030515e-01, -2.748539e-01) => 1.502439e-01\n",
      "b_C (-2.615591e-01, -5.192239e-01) => 3.300082e-01\n",
      "b_o (-1.828830e-01, -3.566602e-01) => 3.220820e-01\n",
      "b_o (-6.164580e-02, -6.660104e-02) => 3.863832e-02\n",
      "b_o (3.400145e-03, 4.486300e-02) => 8.590997e-01\n",
      "b_o (-8.065856e-02, 1.755721e-02) => 1.556476e+00\n",
      "b_o (-3.057445e-02, -1.059484e-02) => 4.853036e-01\n",
      "b_o (-6.164580e-02, -6.660104e-02) => 3.863832e-02\n",
      "b_o (-7.878454e-03, 8.931704e-03) => 1.596026e+01\n",
      "b_o (-3.220456e-02, 8.119028e-03) => 1.674183e+00\n",
      "b_o (-1.044374e-01, -1.323533e-01) => 1.178925e-01\n",
      "b_o (-1.055425e-01, -1.247781e-01) => 8.351679e-02\n",
      "b_y (4.384235e-02, 8.768427e-02) => 3.333312e-01\n",
      "b_y (7.502261e-03, 1.500445e-02) => 3.333311e-01\n",
      "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n",
      "b_y (2.693329e-01, 5.386631e-01) => 3.333312e-01\n",
      "b_y (3.615790e-03, 7.231543e-03) => 3.333311e-01\n",
      "b_y (-2.748632e-01, -5.497285e-01) => 3.333350e-01\n",
      "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n",
      "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n",
      "b_y (8.233201e-05, 1.646628e-04) => 3.333287e-01\n",
      "b_y (8.127466e-02, 1.625486e-01) => 3.333313e-01\n"
     ]
    }
   ],
   "source": [
    "gradient_check(inputs, targets, g_h_prev, g_C_prev)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}