math_ds_train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Math Dataset Training sample\n",
    "\n",
    "> You can contact me on twitter as [@mandubian](http://twitter.com/mandubian)\n",
    "\n",
    "This notebook demonstrates a Pytorch training of a Multi-Head Self-Attention Transformer model on Mathematical Reasoning Dataset v1.0 as explained in this very cool paper:\n",
    "\n",
    "> Analysing Mathematical Reasoning Abilities of Neural Models\n",
    ">\n",
    "> David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli\n",
    ">\n",
    "> (Submitted on 2 Apr 2019)\n",
    ">\n",
    ">http://arxiv.org/abs/1904.01557\n",
    "\n",
    "**On top of paper, the current project provides a few Numpy/Pytorch helpers to manipulate Mathematical Reasoning Dataset in a simpler way and train very fast different kinds of models on it. Without loading everything in memory, it tries to manage data in a streamed and lazy-loading way (as much as Python allows it).**\n",
    "\n",
    "The idea of the paper is to provide a robust toolkit (https://github.com/deepmind/mathematics_dataset) to randomly and heterogenously generate mathematical datasets among multiple math problem categories:\n",
    "    - algebra\n",
    "    - numbers\n",
    "    - polynomials\n",
    "    - arithmetic\n",
    "    - measurement\n",
    "    - comparison\n",
    "    - probability\n",
    "    - calculus\n",
    "    \n",
    "All problem are constituted of a textual question like `what is 30 + 535?` (max 160 chars) and a textual response `565` (max 30 chars). So it's not an abstract representation of math problem but a very human one and it mixes Natural Language Processing with math concepts.\n",
    "\n",
    "For each of those categories, it provides multiple operation modules. For ex, in `algebra` category:\n",
    "    - mul\n",
    "    - add_or_sub_in_base\n",
    "    - simplify_surd\n",
    "    - mul_div_multiple\n",
    "    - mixed\n",
    "    - nearest_integer_root\n",
    "    - div\n",
    "    - add_or_sub\n",
    "    - add_sub_multiple\n",
    "    - add_sub_multiple_longer\n",
    "    - mul_div_multiple_longer\n",
    "    - div_big\n",
    "    - mul_big\n",
    "    - mixed_longer\n",
    "    - add_or_sub_big\n",
    "\n",
    "Problem can be generated with different difficulties:\n",
    "    - train-easy\n",
    "    - train-medium\n",
    "    - train-hard\n",
    "\n",
    "It also provides test datasets for `interpolation` tests mixing all kinds of problem per category and `extrapolation` tests to measure generalization capabilities of models.\n",
    "\n",
    ">An implementation of dataset generator is provided at >https://github.com/deepmind/mathematics_dataset and a v1.0 of the dataset is >available there\n",
    "\n",
    "\n",
    "The main purpose of this dataset generator is to provide a playground to people who want to study how neural networks can learn to solve mathematical problems or even better learn math abstractions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install --user tensorboardX\n",
    "#!pip install --user --force https://github.com/chengs/tqdm/archive/colab.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "0m8YROdDpRAk",
    "outputId": "e6bb48eb-1c2b-4a99-9558-be452345ba23"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Torch Version 1.3.0\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils import data\n",
    "import torch.optim as optim\n",
    "\n",
    "import math_dataset\n",
    "from math_dataset import MathDatasetManager\n",
    "\n",
    "from transformer.Models import Transformer\n",
    "from math_dataset import (\n",
    "    random_split_dataset,\n",
    "    question_answer_to_position_batch_collate_fn\n",
    ")\n",
    "import model_process\n",
    "import utils\n",
    "from tensorboard_utils import Tensorboard\n",
    "from tensorboard_utils import tensorboard_event_accumulator\n",
    "\n",
    "import checkpoints\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook  \n",
    "\n",
    "print(\"Torch Version\", torch.__version__)\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Math Dataset Manager\n",
    "\n",
    "This class is just a Numpy/Pytorch helper to manage all files in Math Dataset v1.0 and select different parts of it by categories or modules to generate a Pytorch dataset for training. Pytorch Datasets created doesn't mount all questions/answers in memory and use Pandas limited streaming features to bufferize data. It allows loading huge files quite fast while keeping memory print reasonable. It also caches lazy datasets and allows fast re-using previously created ones.\n",
    "\n",
    "Here are the main features provided right now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Math Dataset Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initialized MultiFilesMathDataset with categories ['comparison', 'calculus', 'arithmetic', 'polynomials', 'numbers', 'algebra', 'measurement', 'probability'] and types ['train-easy', 'train-medium', 'train-hard', 'interpolate', 'extrapolate']\n"
     ]
    }
   ],
   "source": [
    "mdsmgr = MathDatasetManager(\n",
    "  \"/home/andrew/git/hs-math-nlp/mathematics_dataset-v1.0\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check availables types (difficulties + interpolate + extrapolate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "types ['train-easy', 'train-medium', 'train-hard', 'interpolate', 'extrapolate']\n"
     ]
    }
   ],
   "source": [
    "print(\"types\", list(mdsmgr.get_types()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check availables problem categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "categories ['comparison', 'calculus', 'arithmetic', 'polynomials', 'numbers', 'algebra', 'measurement', 'probability']\n"
     ]
    }
   ],
   "source": [
    "print(\"categories\", list(mdsmgr.get_categories()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check availables modules for a problem category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "modules of arithmetic dict_keys(['nearest_integer_root', 'add_or_sub', 'div', 'mixed', 'add_or_sub_in_base', 'simplify_surd', 'mul_div_multiple', 'add_sub_multiple', 'mul', 'add_sub_multiple_longer', 'div_big', 'mixed_longer', 'mul_big', 'add_or_sub_big', 'mul_div_multiple_longer'])\n"
     ]
    }
   ],
   "source": [
    "print(\"modules of arithmetic\", mdsmgr.get_modules_for_category('arithmetic'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset from a single module in a category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size 666666\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/andrew/.local/lib/python3.7/site-packages/pandas/core/frame.py:4238: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  return super().rename(**kwargs)\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_module('arithmetic', 'add_or_sub', 'train-easy')\n",
    "print(\"size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset from a single module in a category with limited number of elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size 1000\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_module('arithmetic', 'add_or_sub', 'train-easy', max_elements=1000)\n",
    "print(\"size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset from several modules in a category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size 1333332\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_modules('arithmetic', ['add_or_sub', 'add_sub_multiple'], 'train-easy')\n",
    "print(\"size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset from all modules in a category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adding category arithmetic/../train-easy\n",
      "added module arithmetic/nearest_integer_root/train-easy\n",
      "added module arithmetic/add_or_sub/train-easy\n",
      "added module arithmetic/div/train-easy\n",
      "added module arithmetic/mixed/train-easy\n",
      "added module arithmetic/add_or_sub_in_base/train-easy\n",
      "added module arithmetic/simplify_surd/train-easy\n",
      "added module arithmetic/mul_div_multiple/train-easy\n",
      "added module arithmetic/add_sub_multiple/train-easy\n",
      "added module arithmetic/mul/train-easy\n",
      "size 5999994\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_category('arithmetic', 'train-easy')\n",
    "print(\"size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset from all modules in several categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adding category arithmetic/../train-easy\n",
      "added module arithmetic/mul/train-easy\n",
      "added module arithmetic/add_or_sub_in_base/train-easy\n",
      "added module arithmetic/simplify_surd/train-easy\n",
      "added module arithmetic/mul_div_multiple/train-easy\n",
      "added module arithmetic/mixed/train-easy\n",
      "added module arithmetic/nearest_integer_root/train-easy\n",
      "added module arithmetic/div/train-easy\n",
      "added module arithmetic/add_or_sub/train-easy\n",
      "added module arithmetic/add_sub_multiple/train-easy\n",
      "adding category polynomials/../train-easy\n",
      "added module polynomials/add/train-easy\n",
      "added module polynomials/coefficient_named/train-easy\n",
      "added module polynomials/evaluate_composed/train-easy\n",
      "added module polynomials/compose/train-easy\n",
      "added module polynomials/collect/train-easy\n",
      "added module polynomials/simplify_power/train-easy\n",
      "added module polynomials/evaluate/train-easy\n",
      "added module polynomials/expand/train-easy\n",
      "size 11333322\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_categories(['arithmetic', 'polynomials'], 'train-easy')\n",
    "print(\"size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pytorch Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "device cuda\n"
     ]
    }
   ],
   "source": [
    "seed = 1\n",
    "torch.manual_seed(seed)\n",
    "device = torch.device(\"cuda\")\n",
    "print(\"device\", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train on Algebra Linear_1d in Easy mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an experiment with a name and a unique ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_name = \"math_ds_algebra_linear_1d_easy\"\n",
    "unique_id = \"2019-05-25_0900\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Dataset for training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train-easy dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train-easy dataset size 666666\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:4025: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  return super(DataFrame, self).rename(**kwargs)\n"
     ]
    }
   ],
   "source": [
    "ds = mdsmgr.build_dataset_from_module(\n",
    "    'algebra', 'linear_1d', 'train-easy'\n",
    ")\n",
    "print(\"train-easy dataset size\", len(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interpolate dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "interpolate dataset size 10000\n"
     ]
    }
   ],
   "source": [
    "ds_interpolate = mdsmgr.build_dataset_from_module(\n",
    "    'algebra', 'linear_1d', 'interpolate'\n",
    ")\n",
    "print(\"interpolate dataset size\", len(ds_interpolate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create default Transformer model\n",
    "\n",
    "Here we test the best model found in the paper: a multi-head self-attention transformer to give a default sample.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = utils.build_transformer()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create basic optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters(), lr=6e-6, betas=(0.9, 0.995), eps=1e-9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Pytorch dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# here we split data in 90/10% for train/validation and use interpolate for test\n",
    "train_ds, val_ds = math_dataset.random_split_dataset(ds, split_rate=0.9)\n",
    "\n",
    "# we provide the function question_answer_to_position_batch_collate_fn that collates\n",
    "# all questions/answers into transformer format enhanced with char positioning\n",
    "train_loader = data.DataLoader(\n",
    "    train_ds, batch_size=128, shuffle=True, num_workers=4,\n",
    "    collate_fn=question_answer_to_position_batch_collate_fn)\n",
    "\n",
    "val_loader = data.DataLoader(\n",
    "    val_ds, batch_size=128, shuffle=False, num_workers=4,\n",
    "    collate_fn=question_answer_to_position_batch_collate_fn)\n",
    "\n",
    "interpolate_loader = data.DataLoader(\n",
    "    ds_interpolate, batch_size=128, shuffle=False, num_workers=4,\n",
    "    collate_fn=question_answer_to_position_batch_collate_fn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Tensorboard metrics logger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing TensorBoard events locally to runs/math_ds_algebra_linear_1d_easy_2019-05-25_0900\n"
     ]
    }
   ],
   "source": [
    "tb = Tensorboard(exp_name, unique_name=unique_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.to(device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_process.train(\n",
    "    exp_name, unique_id,\n",
    "    model, \n",
    "    train_loader, val_loader, interpolate_loader,\n",
    "    optimizer, device,\n",
    "    epochs=20, tb=tb, log_interval=100,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting Training from Tensorboard data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Restore best model for this experience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting state from checkpoints/math_ds_algebra_linear_1d_easy_2019-10-27_2300_validation_best.pth\n",
      "Loading model state_dict from state found in checkpoints/math_ds_algebra_linear_1d_easy_2019-10-27_2300_validation_best.pth\n"
     ]
    }
   ],
   "source": [
    "# build default transformer model\n",
    "model = utils.build_transformer()\n",
    "model_exp_name = 'math_ds_algebra_linear_1d_easy'\n",
    "model_unique_id = '2019-10-27_2300'\n",
    "# restore best validation model from checkpoint\n",
    "_ = checkpoints.restore_best_checkpoint(model_exp_name, model_unique_id, \"validation\", model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading tensorboard events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_ea = tensorboard_event_accumulator(\n",
    "    \"./runs/math_ds_algebra_linear_1d_easy_2019-05-25_0900_eval/events.out.tfevents.1556176200.c77692173f6a\"\n",
    ")\n",
    "\n",
    "interpolate_ea = tensorboard_event_accumulator(\n",
    "    \"./runs/math_ds_algebra_linear_1d_easy_2019-05-25_0900_interpolate/events.out.tfevents.1556176208.c77692173f6a\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'valid_ea' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-20-8777cdec10c1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mvalid_accuracy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalid_ea\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScalars\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"epoch/accuracy\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mvalid_loss_per_char\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalid_ea\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScalars\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"epoch/loss_per_char\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0minterpolate_accuracy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minterpolate_ea\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScalars\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"epoch/accuracy\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0minterpolate_loss_per_char\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minterpolate_ea\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScalars\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"epoch/loss_per_char\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'valid_ea' is not defined"
     ]
    }
   ],
   "source": [
    "valid_accuracy = valid_ea.Scalars(\"epoch/accuracy\")\n",
    "valid_loss_per_char = valid_ea.Scalars(\"epoch/loss_per_char\")\n",
    "\n",
    "interpolate_accuracy = interpolate_ea.Scalars(\"epoch/accuracy\")\n",
    "interpolate_loss_per_char = interpolate_ea.Scalars(\"epoch/loss_per_char\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot loss per char\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'valid_loss_per_char' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-21-b3e34773abf2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m ax1.plot(\n\u001b[0;32m----> 6\u001b[0;31m     \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_loss_per_char\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m     \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_loss_per_char\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     \u001b[0mmarker\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'+'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Validation Loss per Char'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'valid_loss_per_char' is not defined"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAFpCAYAAACrn+1KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAW1UlEQVR4nO3dX6jdd5nv8c9jY0eoVWGyByRJbWHi1Iwj1NmUHrywoHNIe5FcOCMNFEcp5mYqnqMIFaVKvVI5DgjxT4aRjoJ2qheywUgunA6CWMkunSkmpbKJjk0VGmunN0Vrz3nOxV7Kcncne3V3fZOs9PWCwPr91nev9cCXtO/81tprVXcHAIAxXnGxBwAAuJyJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhoy9iqqq9U1ZNV9eNz3F9V9fmqWquqR6rqrfMfEwBgMc1yZeveJPvPc/8tSfZO/hxO8sWXPhYAwOVhy9jq7u8n+fV5lhxM8tVe92CS11XV6+c1IADAIpvHe7Z2JXl86vjM5BwAwMvejgv5ZFV1OOsvNeaqq6766+uvv/5CPj0AwLY89NBDv+rupe387Dxi64kke6aOd0/OvUB3H01yNEmWl5d7dXV1Dk8PADBWVf3Xdn92Hi8jriR5z+S3Em9K8kx3/3IOjwsAsPC2vLJVVd9IcnOSnVV1JsknkrwySbr7S0mOJbk1yVqSZ5O8b9SwAACLZsvY6u5DW9zfSf5hbhMBAFxGfII8AMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADDRTbFXV/qp6rKrWququTe6/pqoeqKqHq+qRqrp1/qMCACyeLWOrqq5IciTJLUn2JTlUVfs2LPt4kvu7+4YktyX5wrwHBQBYRLNc2boxyVp3n+7u55Lcl+TghjWd5DWT269N8ov5jQgAsLhmia1dSR6fOj4zOTftk0lur6ozSY4l+cBmD1RVh6tqtapWz549u41xAQAWy7zeIH8oyb3dvTvJrUm+VlUveOzuPtrdy929vLS0NKenBgC4dM0SW08k2TN1vHtybtodSe5Pku7+YZJXJdk5jwEBABbZLLF1Isneqrquqq7M+hvgVzas+XmSdyRJVb0p67HldUIA4GVvy9jq7ueT3JnkeJJHs/5bhyer6p6qOjBZ9uEk76+q/0zyjSTv7e4eNTQAwKLYMcui7j6W9Te+T5+7e+r2qSRvm+9oAACLzyfIAwAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEAzxVZV7a+qx6pqraruOsead1fVqao6WVVfn++YAACLacdWC6rqiiRHkvxNkjNJTlTVSnefmlqzN8lHk7ytu5+uqj8bNTAAwCKZ5crWjUnWuvt0dz+X5L4kBzeseX+SI939dJJ095PzHRMAYDHNElu7kjw+dXxmcm7aG5O8sap+UFUPVtX+zR6oqg5X1WpVrZ49e3Z7EwMALJB5vUF+R5K9SW5OcijJP1XV6zYu6u6j3b3c3ctLS0tzemoAgEvXLLH1RJI9U8e7J+emnUmy0t2/6+6fJvlJ1uMLAOBlbZbYOpFkb1VdV1VXJrktycqGNd/O+lWtVNXOrL+seHqOcwIALKQtY6u7n09yZ5LjSR5Ncn93n6yqe6rqwGTZ8SRPVdWpJA8k+Uh3PzVqaACARVHdfVGeeHl5uVdXVy/KcwMAvBhV9VB3L2/nZ32CPADAQGILAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYaKbYqqr9VfVYVa1V1V3nWfeuquqqWp7fiAAAi2vL2KqqK5IcSXJLkn1JDlXVvk3WXZ3kg0l+NO8hAQAW1SxXtm5Mstbdp7v7uST3JTm4ybpPJfl0kt/McT4AgIU2S2ztSvL41PGZybk/qKq3JtnT3d+Z42wAAAvvJb9BvqpekeRzST48w9rDVbVaVatnz559qU8NAHDJmyW2nkiyZ+p49+Tc712d5M1J/r2qfpbkpiQrm71JvruPdvdydy8vLS1tf2oAgAUxS2ydSLK3qq6rqiuT3JZk5fd3dvcz3b2zu6/t7muTPJjkQHevDpkYAGCBbBlb3f18kjuTHE/yaJL7u/tkVd1TVQdGDwgAsMh2zLKou48lObbh3N3nWHvzSx8LAODy4BPkAQAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYKCZYquq9lfVY1W1VlV3bXL/h6rqVFU9UlXfq6o3zH9UAIDFs2VsVdUVSY4kuSXJviSHqmrfhmUPJ1nu7rck+VaSz8x7UACARTTLla0bk6x19+nufi7JfUkOTi/o7ge6+9nJ4YNJds93TACAxTRLbO1K8vjU8ZnJuXO5I8l3N7ujqg5X1WpVrZ49e3b2KQEAFtRc3yBfVbcnWU7y2c3u7+6j3b3c3ctLS0vzfGoAgEvSjhnWPJFkz9Tx7sm5P1JV70zysSRv7+7fzmc8AIDFNsuVrRNJ9lbVdVV1ZZLbkqxML6iqG5J8OcmB7n5y/mMCACymLWOru59PcmeS40keTXJ/d5+sqnuq6sBk2WeTvDrJN6vqP6pq5RwPBwDwsjLLy4jp7mNJjm04d/fU7XfOeS4AgMuCT5AHABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAPNFFtVtb+qHquqtaq6a5P7/6Sq/nVy/4+q6tp5DwoAsIi2jK2quiLJkSS3JNmX5FBV7duw7I4kT3f3nyf5xySfnvegAACLaJYrWzcmWevu0939XJL7khzcsOZgkn+Z3P5WkndUVc1vTACAxTRLbO1K8vjU8ZnJuU3XdPfzSZ5J8qfzGBAAYJHtuJBPVlWHkxyeHP62qn58IZ+fudqZ5FcXewi2xd4tNvu3uOzdYvuL7f7gLLH1RJI9U8e7J+c2W3OmqnYkeW2SpzY+UHcfTXI0SapqtbuXtzM0F5/9W1z2brHZv8Vl7xZbVa1u92dneRnxRJK9VXVdVV2Z5LYkKxvWrCT5+8ntv03yb93d2x0KAOByseWVre5+vqruTHI8yRVJvtLdJ6vqniSr3b2S5J+TfK2q1pL8OutBBgDwsjfTe7a6+1iSYxvO3T11+zdJ/u5FPvfRF7meS4v9W1z2brHZv8Vl7xbbtvevvNoHADCOr+sBABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAANtGVtV9ZWqerKqfnyO+6uqPl9Va1X1SFW9df5jAgAsplmubN2bZP957r8lyd7Jn8NJvvjSxwIAuDxsGVvd/f0kvz7PkoNJvtrrHkzyuqp6/bwGBABYZPN4z9auJI9PHZ+ZnAMAeNnbcSGfrKoOZ/2lxlx11VV/ff3111/IpwcA2JaHHnroV929tJ2fnUdsPZFkz9Tx7sm5F+juo0mOJsny8nKvrq7O4ekBAMaqqv/a7s/O42XElSTvmfxW4k1JnunuX87hcQEAFt6WV7aq6htJbk6ys6rOJPlEklcmSXd/KcmxJLcmWUvybJL3jRoWAGDRbBlb3X1oi/s7yT/MbSIAgMuIT5AHABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgcQWAMBAYgsAYCCxBQAwkNgCABhIbAEADCS2AAAGElsAAAOJLQCAgWaKraraX1WPVdVaVd21yf3XVNUDVfVwVT1SVbfOf1QAgMWzZWxV1RVJjiS5Jcm+JIeqat+GZR9Pcn9335DktiRfmPegAACLaJYrWzcmWevu0939XJL7khzcsKaTvGZy+7VJfjG/EQEAFtcssbUryeNTx2cm56Z9MsntVXUmybEkH9jsgarqcFWtVtXq2bNntzEuAMBimdcb5A8lube7dye5NcnXquoFj93dR7t7ubuXl5aW5vTUAACXrlli64kke6aOd0/OTbsjyf1J0t0/TPKqJDvnMSAAwCKbJbZOJNlbVddV1ZVZfwP8yoY1P0/yjiSpqjdlPba8TggAvOxtGVvd/XySO5McT/Jo1n/r8GRV3VNVBybLPpzk/VX1n0m+keS93d2jhgYAWBQ7ZlnU3cey/sb36XN3T90+leRt8x0NAGDx+QR5AICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADDQTLFVVfur6rGqWququ86x5t1VdaqqTlbV1+c7JgDAYtqx1YKquiLJkSR/k+RMkhNVtdLdp6bW7E3y0SRv6+6nq+rPRg0MALBIZrmydWOSte4+3d3PJbkvycENa96f5Eh3P50k3f3kfMcEAFhMs8TWriSPTx2fmZyb9sYkb6yqH1TVg1W1f14DAgAssi1fRnwRj7M3yc1Jdif5flX9VXf/9/Siqjqc5HCSXHPNNXN6agCAS9csV7aeSLJn6nj35Ny0M0lWuvt33f3TJD/Jenz9ke4+2t3L3b28tLS03ZkBABbGLLF1Isneqrquqq5McluSlQ1rvp31q1qpqp1Zf1nx9BznBABYSFvGVnc/n+TOJMeTPJrk/u4+WVX3VNWBybLjSZ6qqlNJHkjyke5+atTQAACLorr7ojzx8vJyr66uXpTnBgB4Marqoe5e3s7P+gR5AICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICBxBYAwEBiCwBgILEFADCQ2AIAGGim2Kqq/VX1WFWtVdVd51n3rqrqqlqe34gAAItry9iqqiuSHElyS5J9SQ5V1b5N1l2d5INJfjTvIQEAFtUsV7ZuTLLW3ae7+7kk9yU5uMm6TyX5dJLfzHE+AICFNkts7Ury+NTxmcm5P6iqtybZ093fOd8DVdXhqlqtqtWzZ8++6GEBABbNS36DfFW9Isnnknx4q7XdfbS7l7t7eWlp6aU+NQDAJW+W2HoiyZ6p492Tc793dZI3J/n3qvpZkpuSrHiTPADAbLF1Isneqrquqq5McluSld/f2d3PdPfO7r62u69N8mCSA929OmRiAIAFsmVsdffzSe5McjzJo0nu7+6TVXVPVR0YPSAAwCLbMcui7j6W5NiGc3efY+3NL30sAIDLg0+QBwAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIFmiq2q2l9Vj1XVWlXdtcn9H6qqU1X1SFV9r6reMP9RAQAWz5axVVVXJDmS5JYk+5Icqqp9G5Y9nGS5u9+S5FtJPjPvQQEAFtEsV7ZuTLLW3ae7+7kk9yU5OL2gux/o7mcnhw8m2T3fMQEAFtMssbUryeNTx2cm587ljiTf3eyOqjpcVatVtXr27NnZpwQAWFBzfYN8Vd2eZDnJZze7v7uPdvdydy8vLS3N86kBAC5JO2ZY80SSPVPHuyfn/khVvTPJx5K8vbt/O5/xAAAW2yxXtk4k2VtV11XVlUluS7IyvaCqbkjy5SQHuvvJ+Y8JALCYtoyt7n4+yZ1Jjid5NMn93X2yqu6pqgOTZZ9N8uok36yq/6iqlXM8HADAy8osLyOmu48lObbh3N1Tt98557kAAC4LPkEeAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYSGwBAAwktgAABhJbAAADiS0AgIHEFgDAQGILAGAgsQUAMJDYAgAYSGwBAAw0U2xV1f6qeqyq1qrqrk3u/5Oq+tfJ/T+qqmvnPSgAwCLaMraq6ookR5LckmRfkkNVtW/DsjuSPN3df57kH5N8et6DAgAsolmubN2YZK27T3f3c0nuS3Jww5qDSf5lcvtbSd5RVTW/MQEAFtMssbUryeNTx2cm5zZd093PJ3kmyZ/OY0AAgEW240I+WVUdTnJ4cvjbqvrxhXx+5mpnkl9d7CHYFnu32Ozf4rJ3i+0vtvuDs8TWE0n2TB3vnpzbbM2ZqtqR5LVJntr4QN19NMnRJKmq1e5e3s7QXHz2b3HZu8Vm/xaXvVtsVbW63Z+d5WXEE0n2VtV1VXVlktuSrGxYs5Lk7ye3/zbJv3V3b3coAIDLxZZXtrr7+aq6M8nxJFck+Up3n6yqe5KsdvdKkn9O8rWqWkvy66wHGQDAy95M79nq7mNJjm04d/fU7d8k+bsX+dxHX+R6Li32b3HZu8Vm/xaXvVts296/8mofAMA4vq4HAGCg4bHlq34W1wx796GqOlVVj1TV96rqDRdjTja31f5NrXtXVXVV+S2pS8gs+1dV7578HTxZVV+/0DOyuRn+23lNVT1QVQ9P/vt568WYkxeqqq9U1ZPn+miqWvf5yd4+UlVvneVxh8aWr/pZXDPu3cNJlrv7LVn/5oDPXNgpOZcZ9y9VdXWSDyb50YWdkPOZZf+qam+SjyZ5W3f/ZZL/dcEH5QVm/Lv38ST3d/cNWf+Fsi9c2Ck5j3uT7D/P/bck2Tv5czjJF2d50NFXtnzVz+Lacu+6+4HufnZy+GDWP4ONS8Msf/eS5FNZ/wfOby7kcGxplv17f5Ij3f10knT3kxd4RjY3y951ktdMbr82yS8u4HycR3d/P+ufqnAuB5N8tdc9mOR1VfX6rR53dGz5qp/FNcveTbsjyXeHTsSLseX+TS5/7+nu71zIwZjJLH//3pjkjVX1g6p6sKrO969xLpxZ9u6TSW6vqjNZ/03/D1yY0ZiDF/v/xiQX+Ot6uDxV1e1JlpO8/WLPwmyq6hVJPpfkvRd5FLZvR9Zfyrg561eVv19Vf9Xd/31Rp2IWh5Lc293/p6r+R9Y/p/LN3f3/LvZgjDH6ytaL+aqfnO+rfrjgZtm7VNU7k3wsyYHu/u0Fmo2tbbV/Vyd5c5J/r6qfJbkpyYo3yV8yZvn7dybJSnf/rrt/muQnWY8vLq5Z9u6OJPcnSXf/MMmrsv69iVz6Zvp/40ajY8tX/SyuLfeuqm5I8uWsh5b3i1xazrt/3f1Md+/s7mu7+9qsv+fuQHdv+7u/mKtZ/tv57axf1UpV7cz6y4qnL+SQbGqWvft5knckSVW9KeuxdfaCTsl2rSR5z+S3Em9K8kx3/3KrHxr6MqKv+llcM+7dZ5O8Osk3J7/T8PPuPnDRhuYPZtw/LlEz7t/xJP+zqk4l+b9JPtLdXhW4yGbcuw8n+aeq+t9Zf7P8e11kuDRU1Tey/o+YnZP31H0iySuTpLu/lPX32N2aZC3Js0neN9Pj2l8AgHF8gjwAwEBiCwBgILEFADCQ2AIAGEhsAQAMJLYAAAYSWwAAA4ktAICB/j/FQtcjleAB1gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.rcParams['figure.figsize'] = [10, 6]\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)\n",
    "\n",
    "ax1.plot(\n",
    "    list(map(lambda l: l.step, valid_loss_per_char)),\n",
    "    list(map(lambda l: l.value, valid_loss_per_char)),\n",
    "    marker='+', label='Validation Loss per Char'\n",
    ")\n",
    "ax2.plot(\n",
    "    list(map(lambda l: l.step, interpolate_loss_per_char)),\n",
    "    list(map(lambda l: l.value, interpolate_loss_per_char)),\n",
    "    marker='+', label='Interpolate Loss per Char', color=\"orange\"\n",
    ")\n",
    "plt.title('Algebra/Linear_1d Loss per Char')\n",
    "ax1.legend(loc='upper left', frameon=False)\n",
    "ax2.legend(loc='upper left', frameon=False)\n",
    "plt.xticks(np.arange(0, 20, step=1.0))\n",
    "#plt.yticks(np.arange(0.3, 1.0, step=0.1))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As we can see, loss per char on validation dataset has a nice optimization curve but for interpolate, it's not the case. It's quite normal, interpolate contains more difficult and general cases. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy Evolution during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAF1CAYAAAAqdaQaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl4VdWh///3yjzPQRIIhElAAgEMg6AIIharYp1FrVVrvfprba+tt9W2V7mtHa+1autXr/U63hacrlWvIlWEOitBBZlkDBISIAmEzGQ46/fH3jk5CRlhZ+Tzep79nD2ss/faJyc5n6y9ztrGWouIiIiIHL+g3q6AiIiIyEChYCUiIiLiEQUrEREREY8oWImIiIh4RMFKRERExCMKViIiIiIeUbAS6QbGmDOMMduNMRXGmPN7uz59iTHmHmPMkx7sZ7QxplfGizHGfMsYs7w3ju0VY8z/GGOW9NCx/sMY82evy4r0RQpWMmC4IaZx8hljqgOWr+7h6twD/NFaG2Ot/b8ePnaf4H5w1xljTurtunSFG4rfMcaEGGOsMSazZRlr7VPW2nN7vnZtM8ZkG2P+YYwpMcbUH8d+/j3g96bGGNMQsLzuWPZprb3bWvs9r8uK9EUKVjJguCEmxlobA3wFXBCw7q8tyxtjQrqxOsOBjcfyxG6uV7fv3z1GLHARUAZc1d3H64wunPd5wOvdWZdj1cE51ALLgO8czzGstb8M+D36HvBuwO9RdhfrJHLCUbCSE4Z7CepZY8xSY0w5cI0x5jRjzEfGmFJjTKEx5kFjTKhbvrHF4l/cy3qHjDEPBuzvZLdl47AxptgY8zd3fR4wDFju/pcfbIwZaoz5P2PMQWPMNmPMDR3U6x5jzDJ3XYUxZp0xZpQx5ufGmCJjzFfGmLMD9pFgjHnCPYd8Y8wvjDFB7rYb3Xo+aIw5CPy8ldcmwt1eaIzZa4y5zxgT5m472xiTZ4z5sXvsAmPMtR283JcBB4BfA9/q4OdyvXs+xcaYn7r1n+tuC3LX7XC3LzPGJLZ4/nfcOhUYY2471p93gK/TQbByX9PV7ny775OA8lvcbcuNMRkB2/7snnOZMWaNMWZWe+fQVp2stZuttY8Dm9qo86nGmM+NMeXGmKVAeHvn2M65R7jne4sxZgewwV3/cMB5fGKMmRnwnN8aYx5z58cZY+rdn3u++576t2MsG2OM+Zv789xgjLnTGLP9WM5LxCsKVnKiuQj4GxAPPAvUAz8AUoDZwELgX1o85+vAqcAUnA/nxkDzK+A1IBEYCjwEYK3NBAqAc93/8hvcY+0C0oErgN8bY85sp14AFwL/DSTgtH695dY3DfgN8HDA858BqoFRbl3PA64P2D4L2AykAr9r5XW5C8gBJrnnORu4M2D7UCDSrf/NwMPGmLhW9tPoW8BSd5pojDmqpQPAGDMReBC4Ehji1m9wQJHb3HOZ49ahwi0faA4wGjgX+HljKHN16eftBp4Ea+36ds6tLa2+T4wxlwD/hvPzTAU+duvU6GOc1z0JeAF43hgTGHpae290ibu/l4HH3eO8DHzjWPYV4HyazhfgQ2AikOzu//lWQmujYJz322ic1+1XxpiRx1D2HpzXdDjO++Sbx3NCIp6w1mrSNOAmIA84u8W6e4C3O3je7cDz7nwIYIGZAdv/F7jdnf8bTrgZ0sp+8oG57vwIoA6IDtj+n8BjbdXLXbc8YPki4DAQ5C4nunWLwQkk1UB4QPlvAm+68zcCOzs4793AOQHL5wHb3fmzcQJNcMD2g0BOG/saAfiALHd5JfCHFuf2pDv/C+CZgG3ROOGn8bXbBpwZsD0DqMH5p3C0+xqMDth+H/Bfx/Lzdpf/JeD5jT//zFaedyOwupPvkzeBbwVsCwGOtPG+MUA5MKGz59DKPsYB9S3WnQXsAUzAuk+AJR3sy3+eAesi3POd1c7zDFAFjHWXfxvwfh/nPj8loPx64BvHULagxfvje43vW02aemtSi5WcaPYELriXGl4zxuwzxpThfNCntHjOvoD5KpwwA/AjIBTINcZ8YYxp65JXOlBsra0MWLcbJxC1Wi/X/oD5aqDIWusLWMaty3Ccyzr73UsipTitZ4Gdxlvbf8s67m6nfsXWaXlrFPg6tHQt8IW1doO7/FfgatN6X5z0wLq5r9GhgO3DgFcDzusLd/2ggDKB57bb3Wdr2zrz8+7wMmA72nqfDAceCjiHYpzgOdSt04/dy4SHcc49ukWdOvrZdUY6kG+tDfwW5e62CndSy9f2TmPMlwHnEcHRv0uNGqy1xQHL7b2fWi1rnEvdg1vUw4vXSuS4KFjJiabl1/P/C6ePyGhrbRzOJTHTqR1ZW2itvdFamwZ8F3jUGDOilaIFQIoxJjpg3TBgbzv16oo9OB82SdbaBHeKs9ZO6sL+C3ACQFv16xRjjMEJVie74WUf8HuckPe1Vp5SiBsw3OdH47TGNcoHFgScV4K1NsJaGxhiMgLmh7nn0qjTP2/j9Ck7HeeSq5f2AN9ucQ6R1tqPjTHzgB8Cl+Bc8k3EaR0MfA96MaREs9fZNew49+mvlzFmAXArTstqAs7lxmo6+bt0TAd3/snYT/PzymijuEiPUbCSE10sziW2SmPMeI7uX9UmY8zlxpjGVp1SnA+ahpblrLW7gFzg18aYcGPMZJz+T/9zvJV3978H+CdwrzEmzjgdvkcbY+Z0YTdLgbuMMSnGmFTg34+xfqfjfLjlAJPdKQt4DidwtfQ88A1jzEw32PyixfZHcF63YQDGmEHGmEUtyvy7MSbS7a/1Ldrvh9Tez/tMYG2LlkWAcLfDduMU3M7+W/MI8DP3eI1fNLg0oD71OK1YocASnBarLjOOCKDxSwcR7msK8B4QZIz5nnE6218OTD2W47QhFudyd5F7/F/gtFh1t+dwXtt49z1ySw8cU6RdClZyovsRzodxOU5rRlc6B88A1hhjKnH61HzXWvtVG2WvAMbgXC56AfiptXb1sVa6FdfgfCBvwrkM8zzNO4F35D+AdTitOetxOlT/5hjq8S3gJWvtRmvtvsYJeABYZIxJCCxsnU7it7n1LQBK3OmIW+Q+4A1gpfutuA+AaS2O+R6wE/gH8Btr7dvt1K+9n3dbwyxswWl9aZy61EHaWvu8ex7Pu5cf19PUevc6TgvZNpx+gWU4rUvHYpRbv3U4Hb6rcb8haK09gtOa9B2c98dFwN+P8TiteRV4B9iB87MoxglZ3e3nOOezG1iOE7SOtPsMkW5mml9yFxHpPe43DUuB4W5LXE8eeytwvrV2a08eV7xjnOE2FlprW7vsLNIj1GIlIr3KGLPIGBNljIkB/gB82guhKgL4b4Wq/sUYk+FeRg4yxkzAGUrjpd6ul5zYOhWsjDEL3W97bDfG3NHK9uHGmJXGmPXGmNXGmJadJEVE2nIRzmXAfCATWNzTFbDW1lhrWxvfq08xzi1rKlqZftzbdesl4Thjc5XjXDJeBjzWqzWSE16HlwLdjppbgQU4f/jWAIuttZsCyjwP/J+19iljzFnA9dZaDdQmIiIiJ5TOtFhNxxlwbae1tvFeVBe2KHMK0NhhdFUr20VEREQGvM4EqyE0H3Qtn+YDB4LzLZSL3fmLgFhjTPLxV09ERESk//DqruS3A382xlyH85XbvbQyno8x5ibgJoDo6OhTx40b59HhRURERLrP2rVri621qR2V60yw2kvz0WyH0mJEZmttAW6LlfvNnkustaUtd2StfRR4FCAnJ8fm5uZ24vAiIiIivcsY06nbQHXmUuAaYIwxZoQ7iu+VwCstDpbi3rcJ4E6cb2mIiIiInFA6DFbW2nqcO4avADYDz1lrNxpjfhFwa4m5wJfuAHsnAb/qpvqKiIiI9Fm9NvK6LgWKiIhIf2GMWWutzemonEZeFxEREfGIgpWIiIiIRxSsRERERDyiYCUiIiLiEQWrAPPmzWPFihXN1t1///3ccsst7T4vJiYGgIKCAi699NJWy8ydO5eOOuvff//9VFVV+Ze//vWvU1p61HBgx2zy5MlceeWVnu1PREREmhsQweqPb271ZD+LFy9m2bJlzdYtW7aMxYsXd+r56enpvPDCC8d8/JbB6vXXXychIeGY9xdo8+bNNDQ08O6771JZWenJPltTX1/fbfsWERHp6wZEsHpg5TZP9nPppZfy2muvUVtbC0BeXh4FBQWcccYZVFRUMH/+fKZOncrEiRN5+eWXj3p+Xl4eWVlZAFRXV3PllVcyfvx4LrroIqqrq/3lbrnlFnJycpgwYQJ33303AA8++CAFBQXMmzePefPmAZCZmUlxcTEA9913H1lZWWRlZXH//ff7jzd+/Hi+853vMGHCBM4555xmxwm0dOlSvvnNb3LOOec0q/v27ds5++yzyc7OZurUqezYsQOA3/3ud0ycOJHs7GzuuOMOoHmrW3FxMZmZmQA8+eSTLFq0iLPOOov58+e3+1o9/fTTTJo0iezsbL75zW9SXl7OiBEjqKurA6CsrKzZsoiISH/i1b0CPfcfr25kU0FZp8tf8V8fdljmlPQ47r5gQpvbk5KSmD59OsuXL+fCCy9k2bJlXH755RhjiIiI4KWXXiIuLo7i4mJmzpzJokWLMMa0uq+HH36YqKgoNm/ezPr165k6dap/269+9SuSkpJoaGhg/vz5rF+/nu9///vcd999rFq1ipSUlGb7Wrt2LU888QQff/wx1lpmzJjBmWeeSWJiItu2bWPp0qX85S9/4fLLL+fFF1/kmmuuOao+zz77LG+++SZbtmzhT3/6E1dddRUAV199NXfccQcXXXQRNTU1+Hw+li9fzssvv8zHH39MVFQUBw8e7PC1/fTTT1m/fj1JSUnU19e3+lpt2rSJe+65hw8++ICUlBQOHjxIbGwsc+fO5bXXXuMb3/gGy5Yt4+KLLyY0NLTDY4qIiPQ1/bbFKv9QFR/vOsjHu5wP/cb5/ENVHTyzfYGXAwMvA1pr+elPf8qkSZM4++yz2bt3L/v3729zP++8844/4EyaNIlJkyb5tz333HNMnTqVKVOmsHHjRjZt2tRund577z0uuugioqOjiYmJ4eKLL+bdd98FYMSIEUyePBmAU089lby8vKOen5ubS0pKCsOGDWP+/Pl89tlnHDx4kPLycvbu3ctFF10EQEREBFFRUbz11ltcf/31REVFAU7g7MiCBQv85dp6rd5++20uu+wyf3BsLH/jjTfyxBNPAPDEE09w/fXXd3g8ERGRvqjPtli117LUUuYdr5H32/M8Oe6FF17IbbfdxqeffkpVVRWnnnoqAH/9618pKipi7dq1hIaGkpmZSU1NTZf3v2vXLu69917WrFlDYmIi11133THtp1F4eLh/Pjg4uNVLgUuXLmXLli3+S3dlZWW8+OKLXe7IHhISgs/nAziqztHR0f75rr5Ws2fPJi8vj9WrV9PQ0OC/nCoiItLf9NsWq+4SExPDvHnzuOGGG5p1Wj98+DCDBg0iNDSUVatWsXt3+ze5njNnDn/7298A2LBhA+vXrwecUBMdHU18fDz79+9n+fLl/ufExsZSXl5+1L7OOOMM/v73v1NVVUVlZSUvvfQSZ5xxRqfOx+fz8dxzz/HFF1+Ql5dHXl4eL7/8MkuXLiU2NpahQ4fy97//HYAjR45QVVXFggULeOKJJ/wd6RsvBWZmZrJ27VqAdjvpt/VanXXWWTz//POUlJQ02y/Atddey1VXXaXWKhER6dcGRLD6wfwxnu5v8eLFrFu3rlmwuvrqq8nNzWXixIk8/fTTjBs3rt193HLLLVRUVDB+/Hjuuusuf8tXdnY2U6ZMYdy4cVx11VXMnj3b/5ybbrqJhQsX+juvN5o6dSrXXXcd06dPZ8aMGdx4441MmTKlU+fy7rvvMmTIENLT0/3r5syZw6ZNmygsLOSZZ57hwQcfZNKkScyaNYt9+/axcOFCFi1aRE5ODpMnT+bee+8F4Pbbb+fhhx9mypQp/k71rWnrtZowYQI/+9nPOPPMM8nOzuaHP/xhs+ccOnSo09/AFBER6Yt0E2bpE1544QVefvllnnnmmd6uioiIyFE6exPmPtvHSk4ct956K8uXL+f111/v7aqIiIgcFwUr6XV/+tOfersKIiLSj/3xza3ctuDk3q4GMED6WImIiMiJy6uBwr2gFisRERHpd+oafOw5WEVeSffdpu1YKFiJiIhIn9TgsxSUVpNXUklecSU7i53HXcWV7C6pIvDrd5l3vAY4IwX05mVBBSsRERHxXGf7PVlrKSo/0iw0NU67D1ZRW+/zl40MDWZESjQT0uM5b1IaI1JiGJESxSUPf+jZQOHHS8GqhZiYGCoqKtotc//993PTTTf5b/nSXZ588klyc3P585//3GaZ1atXExYWxqxZs7q8/3/913/l+eefZ8+ePQQFqbudiIh454GV25oFq0OVtf7wlFfS1PqUV1xJZW2Dv1xYcBDDkqMYkRLNvHGDGJESTWZyNCNToxkUG97mPXr7ioERrFb9Bubd2WOHu//++7nmmmu6FKwaGhoIDg72vC6rV68mJiamy8HK5/Px0ksvkZGRwT//+c+jBiX1irUWa62Cm4jICaKspo5Pdjp31rjt2c/9rU+Hq+v8ZYIMZCRFkZkczbTMJEakRPun9IRIgoO6Fp68Hij8eAyMT7t//tbzXa5evZq5c+dy6aWXMm7cOK6++mqstTz44IMUFBQwb948fxj5xz/+wWmnncbUqVO57LLL/C1emZmZ/OQnP2Hq1Kk8//zzzJ07lx/84AdMnjyZrKwsPvnkE8C5tcs3vvENJk2axMyZM/23vwn06quvMmPGDKZMmcLZZ5/N/v37ycvL45FHHuGPf/wjkydP5t1336WoqIhLLrmEadOmMW3aNN5///02z2/ChAnccsstLF261L++oqKC66+/nokTJzJp0iRefPFFAN544w2mTp1KdnY28+fPB2DJkiX+UdkBsrKy/LfNGTt2LNdeey1ZWVns2bOHW265hZycHCZMmMDdd9/tf86aNWuYNWsW2dnZTJ8+nfLycubMmcPnn3/uL3P66aezbt26Y/o5iohI96qpa+CD7cX854otzPj1W0xa8g9ufNoZAPylz/by+Z5SMpIi+fl543ns2hxW/uhMtvzyXP75b/N46obpLFk0gW/NymTOyalkJEV1OVQBfWaoBejLLVbL74B9X3S+/BOduLY6eCKc2/kQ9tlnn7Fx40bS09OZPXs277//Pt///ve57777WLVqFSkpKRQXF3PPPffw1ltvER0dze9+9zvuu+8+7rrrLgCSk5P59NNPAXjkkUeoqqri888/55133uGGG25gw4YN3H333UyZMoW///3vvP3221x77bXNggU44eKjjz7CGMNjjz3G73//e/7whz9w8803ExMTw+233w7AVVddxW233cbpp5/OV199xde+9jU2b9581LktXbqUxYsXc+GFF/LTn/6Uuro6QkND+eUvf0l8fDxffOG89ocOHaKoqIjvfOc7vPPOO4wYMaLZPf7asm3bNp566ilmzpwJwK9+9SuSkpJoaGhg/vz5rF+/nnHjxnHFFVfw7LPPMm3aNMrKyoiMjOTb3/42Tz75JPfffz9bt26lpqaG7OzsTv/cRESk+9Q3+Phi72E+2FHCBzuKyc07xJF6H8FBhklD47ns1AxmjU7mqr983Gf6PfWkvhusOlK6Gw7vaVre/Z7zGJ8BCcM9OcT06dMZOnQoAJMnTyYvL4/TTz+9WZmPPvqITZs2+e/5V1tby2mnnebffsUVVzQr33gvvDlz5lBWVkZpaSnvvfeev2XorLPOoqSkhLKysmbPy8/P54orrqCwsJDa2lpGjBjRap3feustNm3a5F8uKyujoqKCmJgY/7ra2lpef/117rvvPmJjY5kxYwYrVqzg/PPP56233mLZsmX+somJibz66qvMmTPHf8ykpKQOX7vhw4f7QxXAc889x6OPPkp9fT2FhYVs2rQJYwxpaWlMmzYNgLi4OAAuu+wyfvnLX/Kf//mfPP7441x33XUdHk9ERLqHtZZtByp4f3sx728v4eOdJZQfqQdg3OBYrp4xnNmjk5k+IonYiNBerm3v67vBqgstSyyJhyWHPa9CeHi4fz44OJj6+vqjylhrWbBgQbPLaYGio6ObLbfsdNfZTni33norP/zhD1m0aBGrV69myZIlrZbz+Xx89NFHREREtLmvFStWUFpaysSJEwGoqqoiMjKS888/v1N1aRQSEoLP1/RtjZqaGv984Hnv2rWLe++9lzVr1pCYmMh1113XrGxLUVFRLFiwgJdffpnnnnuOtWvXdqleIiJyfPYcrOKDHU6Q+mBHCcUVRwAYlhTF+dlpnDYqhVmjkkmJCW9zH32p31NPGhh9rHpYbGws5eXlAMycOZP333+f7du3A1BZWcnWrVvbfO6zzz4LwHvvvUd8fDzx8fGcccYZ/PWvfwWcvk8pKSn+1ptGhw8fZsiQIQA89dRTrdYF4Jxzzml2i5iWlxTBuQz42GOP+ftD7dq1izfffJOqqioWLFjAQw895C976NAhZs6cyTvvvMOuXbsA/JcCMzMz/Zc5P/30U//2lsrKyoiOjiY+Pp79+/ezfPlyAMaOHUthYSFr1qwBoLy83B9eb7zxRr7//e8zbdo0EhMT23w9RUSk8/74ZuufT8UVR3h1XQF3/u965vx+FWf8fhU/efELPthRwqxRyfzukom8++N5vPPjefzm4kksyk5vN1RB3+r31JP6botVV5x5R48e7qabbmLhwoWkp6ezatUqnnzySRYvXsyRI06iv+eeezj55NbfUBEREUyZMoW6ujoef/xxwOkEfsMNNzBp0iSioqKaBadGS5Ys4bLLLiMxMZGzzjrLH2IuuOACLr30Ul5++WX+9Kc/8eCDD/Ld736XSZMmUV9fz5w5c3jkkUf8+6mqquKNN95oti46OprTTz+dV199lZ///Od897vfJSsri+DgYO6++24uvvhiHn30US6++GJ8Ph+DBg3izTff5JJLLuHpp59mwoQJzJgxo81zzs7OZsqUKYwbN46MjAz/ZdOwsDCeffZZbr31Vqqrq4mMjOStt94iJiaGU089lbi4OK6//vpj+AmJiEhrGodAKK+p45NdB90WqWK27HP+QY8ND2HGyGSun53J7NEpjBkU0+eHN+hrjLW241LdICcnx+bm5vbKsXvL3Llzuffee8nJyentqvR5BQUFzJ07ly1btmioBhGR41B5pJ68Emek8v/vr58ydVgC6/IP0+CzhIcEkZOZyKxRKcwenUJWehwhwfqb2xpjzFprbYcf4AOjxUoGlKeffpqf/exn3HfffQpVIiKdUFVbT15xFbtLKtlV0jjwpnMfvQPlR5qV/fSrUgAunjKEX188kYhQ78dYPJGpxUpERKSXdeb2L9W1DW7LUyW7iqv8I5jnlVSyv6x5eEqNDScz2RmAM9MdeHN4chTnPfjeCTkEghfUYiUiItJPNPZ9qqlrYHdJFbsaQ5P/sYp9Zc2/TZ0S44SnM8ak+oNTY5CKCdfHe2/RKy8iItLDGnyWbQfKWbenlPX5znBBp/1mJYWHm4en5OgwMlOimT06hREpUQxPbmp9OpYxo07UIRB6koKViIhIN7LWUni4hnV7Svncnb7Ye5iqgBsPA/5QtTBrMN+dO5rhKVHEeTzg5ok6BEJPUrASERHxUFlNHV/kH/aHqM/3lFLkdiAPCw7ilPQ4Ls/JIDsjnskZiWQmRzHiztfV92mAULASERE5RrX1PrbsK3Nbow7z+Z5D7Ciq9G8fmRrNGaNTyM5IYHJGAuPSYgkP0bfwBjIFKxERkVa0/KaetZbdJVWsyy/ls69KWZdfysaCMmrrnVt7pcSEMTkjgW9MHsLkYQlMGpJAfFTnLuWp79PAoWAlIiLSigdWbmNyRgKf7Sll3R4nSJVW1QEQGRrMxCHxXDcrk+yhCWRnxDMkIfKYRylX36eBQ8FKREQEOFBew0c7D/LhjmI+2uncE/X6J9cQZODkk2JZOGEw2RkJZA9N4OSTYjRCubRKwUpERE5IBytr+WhnCR/uKOHDnSVsP1DRajmfha9NGKxWJekUBSsRETkhHK6u4+OdToj6cEeJ/8bDUWHBTMtM4tJThzJrVDIT0uMJDjJk3vGavqknXaZgJSIiA1LFkXrW7DroD1IbCw7js/hvPHz7OSdz2qgUJg2NJ1SX9cQjClYiIjIgVNc2kLv7oP/S3vr8wzT4LGHBQUwelsCtZ41h1qhkJg9L6NSQB/qmnhwLBSsREek3AodAqKlr4LOvSvlwZwkf7Sjhsz2HqGuwhAQZJg2N55YzR3HaqGSmDkskMqzrY0epT5UcCwUrERHpF2rqGnhg5TZCggwf7Cjh068OcaTeR5CBrCHx3HD6CE4bmcy0zCSidRNi6SV654mISJ9TXdvApsIyNhYcZsPew3yxt4xt+53O5n94cyvj0+K4ZuZwJ0iNSCI+0tt76okcKwUrERHpVeU1dWwqKGNDQRkb9jpBakdRBT7rbI8IDaKmztfsOZsLyzjnlJM4+5STeqHGIm3rVLAyxiwEHgCCgcestb9tsX0Y8BSQ4Ja5w1r7usd1FRGRfq60qpaNboD6Yu9hNhaUsau46d56J8WFk5Uez7kT08hKj2Pi0HgGx0X4RzTXEAjS13UYrIwxwcBDwAIgH1hjjHnFWrspoNjPgeestQ8bY04BXgcyu6G+IiLSB7W8rx5AccURfwvUhr1lbCg4TP6hav/2IQmRZA2J45KpQ5gwJJ4J6XEMio3o6aqLeKozLVbTge3W2p0AxphlwIVAYLCyQJw7Hw8UeFlJERHpu6y1PLByGxOHxLutUE6Q2ldW4y+TmRxFdkYC18wcTla6E6ISo8O6fCwNgSB9XWeC1RBgT8ByPjCjRZklwD+MMbcC0cDZre3IGHMTcBPAsGHDulpXERHpI3w+yyd5B3l1XQH/2LQfgBufzsUYGJUaw2mjkpmQHkfWkHhOSY8jLsKbzuUaAkH6Oq86ry8GnrTW/sEYcxrwjDEmy1rbrLehtfZR4FGAnJwc69GxRUSkB1hrWZ9/mFfWFfB/6wvYX3aklTJw3sQ0BSA5YXUmWO0NQ6ndAAAgAElEQVQFMgKWh7rrAn0bWAhgrf3QGBMBpAAHvKikiIj0nq37y3nl8wJeXV/A7pIqwoKDOHNsKhdkp3P2+EFEhYWoU7mIqzPBag0wxhgzAidQXQlc1aLMV8B84EljzHggAijysqIiItJzviqp4tX1BbzyeQFf7i8nyMDs0Sl8d+5ovpY1WONGibShw2Blra03xnwPWIEzlMLj1tqNxphfALnW2leAHwF/McbchtOR/TprrS71iYj0I/vLanh1XQGvri9k3Z5SAHKGJ/KLCydwblYaqbHhbT5XncpFHKa38k9OTo7Nzc3tlWOLiIjjUGUtr28o5NV1BXy86yDWwoT0OBZlp3N+djpDEiJ7u4oifYIxZq21Nqejchp5XUTkBFNeU8ebm/bzyroC3ttWTL3PMjI1mh/MH8MF2emMSo3p7SqK9FsKViIiA1TgoJ01dQ2s2nKAV9YV8PaWAxyp9zEkIZIbzxjJBdlpnJIW5x/dXESOnYKViMgA9cDKbUzOSPCPNVVxpJ6UmHAWTx/GBdlpTB2WqDAl4jEFKxGRAaSuwceHO0pYvmEfANc/uYa4iBDOm5jGosnpzBiRREhwUC/XUmTgUrASEennauoaeG9bMcs37OP/1hdwpL7Z2MyU1dQzOD6C2aNTeqmGIicOBSsRkX6oqraeVVuKWL6hkFVbDlBZ2+C0TE1K49ysNM4Yk8K4f39Dg3aK9DAFKxGRfqKspo6Vm/ez/It9/HNrEUfqfSRHh7FocjoLs9I4bWQyYSG6zCfSmxSsRET6sIOVtby5aR/LN+zj/e3F1DVYTooL58ppGSzMSmP6iCSCg1rvgK5BO0V6noKViEgfc6CshhUbnTD18a6DNPgsQxMjuW5WJguz0piSkUBQG2EqkG6ELNLzFKxERPqA/ENVvLFhH29s2Mfarw5hLYxMjebmM0dyblYaE9I1zpRIf6BgJSLSgwIH7dxVXMnyDYW8sWEf6/MPAzA+LY7bzj6Zc7MGM+ak2N6sqogcAwUrEZEeYq3lgZXbMAbe2LCPLfvKAcjOSOCOc8excMJgMlOie7mWInI8FKxERLpZ5ZF6Xv68gGc+2g04I6JPG57EXeefwsKswaTrRsciA4aClYhIN9l+oIL/+Wg3f/t4N7UN1r/eWvgk7yCnjUpWqBIZYBSsREQ8VNfg461N+3nmo918sKOEsOAgzpuUzjUzhzN1WAIj7nxdg3aKDGAKViIiHjhQVsPST/bwt092s7/sCEMSIvnxwrFcnpNBSkx4b1dPRHqIgpWIyDGy1vLRzoP8z0e7WbFxH/U+y5knp/Lri4Yzd+ygVgfu1KCdIgObgpWISBeV19Tx0md7eebD3Ww7UEFCVCg3nD6Cq2cMY3hy+9/q06CdIgObgpWISCdt2VfGMx/u5qXP9lJV20D20HjuvSyb8yelEREa3NvVE5E+QMFKRKQdtfU+3ti4j//5cDef5B0kPCSIRdlOZ/TsjITerp6I9DEKViIirdhbWs3Sj79i2ZqvKK6oZXhyFD/7+nguyxlKQlRYb1dPRPooBSsROeE13mbG57O8v6OYZz7czVub92OB+eMG8c3TMjljdEqnbnwsIic2BSsROeE9sHIbsREh/PXjr9hVXElydBg3nzmKxdOHkZEU1dvVE5F+RMFKRE5I1lo+3nWQ53L3AHDPa5s5dXgi/3r2GBZmDSY8RJ3RRaTrFKxE5ISyv6yGF9bm8+g7OzhcXd9s29rdhzh9dIpClYgcMwUrERnw6hp8rNx8gOdy97D6ywP4LMwYkcQV0zI4NyuN8Xe9odvMiIgnFKxEZMDafqCcZ9fs4aXP9lJcUcug2HBuPnMUl+dkkJnS/kCeIiLHQsFKRAaUiiP1vLa+gGfX7OHTr0oJCTLMHz+IK6ZlMGdMKiHBQUc9R7eZERGvKFiJSL9nrWXt7kM8u2YPr31RSFVtA6NSo/np18dx0ZShpMa2fxNk3WZGRLyiYCUi/daB8hr+99O9PJe7h51FlUSHBXPBpHQun5bB1GEJGKNxp0SkZylYiUi/Ut/gY/WXRTybu4e3txygwWfJGZ7IzZeO4ryJaUSH68+aiPQe/QUSkT6rcUR0gJ1FFTyXm8+Ln+ZTVH6ElJgwbjx9BJflZDB6UEwv11RExKFgJSJ91gMrtzE0MZLnc/P5JO8gwUGGeWNTuTwng3njBhHaSkd0EZHepGAlIn1GeU0dW/aVs7mwjM/3lALwby+sZ0RKND9eOJZLpw5lUFxEL9dSRKRtClYi0uOstRQcrmFTQRmbC8ucx31l7C6parX8ruJKjtT5FKpEpM9TsBKRblVb72PbgXI3RJWzqfAwmwvLOVxdB4AxkJkcTVZ6PJedOpRT0uM4JS2ek+LCGXHn6xoRXUT6FQUrEemSwA7lLR2qrHVaoBqngjJ2FFVQ12ABiAwNZuzgWM6blMb4tDhOSYtj3OBYfZNPRAYM/TUTkS55YOU2fjB/DF8drPKHp8YwVXi4xl/upLhwxqfFcda4QU6ISo8jMzma4KDOjy2lEdFFpL9RsBKRNjX4LHsPVbOjqMKdKgGYuGQFlbUNAAQHGUanxjBjRJL/Mt74tFiSY9of7bwzNCK6iPQ3ClYiQnlNHTuLKtlRVNHscVdJJbX1vqPKN4aqxdMzuPuCCUSEBvd0lUVE+iQFK5EBoL1+T40afJaC0mp/y9NOtxVqZ1ElB8qP+MsFBxmGJ0UxMjWGuWNTGZUaw8jUaEamxpAUHUbmHa+pQ7mISBsUrEQGgAdWbvMHq4oj9exs0fK0o6jCGbIgoPUpPjKUUanRnHlyKiNTYxjlhqdhSVGEhWjgTRGRY6FgJdKNOtOS1B5rLTV1Pspr6iirqafiSD3lNXWU1zQ9ltXUA3DVXz5iR1EF+8uaWp+CDAxLimJUagxnjElhVGoMowbFMDIlmqTosGO6SbE6lIuItE3BSqSbWGt5YOU2rpye4Q9CZTX1VNTUNwtGgQGp4kjjuqb19T7bqeN9sKMEgFmjkrn2tExGpUYzLDmK8BBv+z+pQ7mISNsUrESOQ229j/xDVewuqSKvpJLdJVXsdh/3HHJGET/tN2+3+fwgA7ERocSEhxAbEUJcRCiD4yIYMyiE2IhQYiNCiIlw5uMinDL+9eEh/ueO+qkG0hQR6Qs6FayMMQuBB4Bg4DFr7W9bbP8jMM9djAIGWWsTvKyoiFe6enmuuraBrw42BqfG8OQsF5RWE9igFBMeQmRoEEUVtUft58LJ6Vx7WqYbkJxwFBUWfEyX40REpG/qMFgZY4KBh4AFQD6wxhjzirV2U2MZa+1tAeVvBaZ0Q11FPBHY0btRWU0dX7VodcpzHwP7LAEkRoUyPDmaU4cncvHUoWQmRzE8OYrhydEkt+i31FPfoFO/JxGRvqEzLVbTge3W2p0AxphlwIXApjbKLwbu9qZ6It6w1lJaVceuEmeAy/vf2trs8t3ByuYtTINiwxmeHMUZY1Ld4BTthKekaOKjQnvjFNqlfk8iIn1DZ4LVEGBPwHI+MKO1gsaY4cAIoNVOJcaYm4CbAIYNG9aliop0xFrLwcpa8koqySt2Wpt2ua1OWwrLqG1oumZ3/1vbABiaGMnXJgxu1uo0LCnKs3vXqSVJROTE4nXn9SuBF6y1Da1ttNY+CjwKkJOT07mvOskJobP9nqy1FFUccVqbip3Wpl2NfZ+Kqyg/Uu8vG2RgaKITmK6YNozMlGgyk6P49lO5fHnPQs+/LdcatSSJiJxYOhOs9gIZActD3XWtuRL47vFWSk48gf2erLUcKD/iD055JZXNWqEab6cCzijhGYmRTp+nYYlueHIu2w1NbHugy54IVSIicuLpTLBaA4wxxozACVRXAle1LGSMGQckAh96WkMZsMpq6th+oILtByoAuPmZtf4+T9V1TeEpJMiQkRRFZnIU00ckkZkc5Q9QQxIjCQ3u2ijhujwnIiLdpcNgZa2tN8Z8D1iBM9zC49bajcaYXwC51tpX3KJXAsustbrEJ36Nl+62769ge1GFP0htP1DR7P50AG9s3AfA5IwELp46hMxkJzylJ0QQ0sXw1B5dnhMRke5ieisH5eTk2Nzc3F45tnjP57PkH6pme1F5s/C0/UCF/5Yr4IzzNGpQDKNTYxg9qGmad+9qDXApIiJ9ljFmrbU2p6NyGnld2tWyU3ltvY+8ksqjwtPO4gpq6ppu8JsSE8ao1BguyE5nzKAYRg+KZfSgGE6KC9eAmCIiMmApWMlRrLUcqqpjz8EqHli5jboGnxOgiirYXVJFQ8BQ40MSIhlzUgyzRiU3a4FKiArr0jHV70lERAYCXQo8AdU1+Nh3uIa9pdXsPVRNQWm1M+9OBaXVzVqfQoIMmSnRR12+G5kaTVSYsrmIiAx8uhQ4gHU05lNZTZ0TltzQlF9aTUFpjX/d/vIaWubplJhwhiREMG5wLPERoXy2p9S/rd5n2X6ggvMmpqnjt4iISDsUrPqZBp/lgZXbmHNyCnsDwlJgq1N5QGdxgLDgINISIkiPj+T0MSmkJ0QyNCGS9IRIhiRGkhYfQURo6+M69dS97kRERAYCBas+qqymjp1FlexwO4bvLKpkZ1Gl/153lzzcNFxYXEQIQxKjGJoYyYwRSf7A1BigUmLCCQpSh3EREZHupmDVixp8lvxDVewocoLTjqJKdhZVsKOokuKKpjGegoMMseEhlFbXHbWPW84cyU/OHd9tdVSnchERkc5TsPJIe/2eDlfX+QPTTn+Icr5hV9vQ1Ek8ISqUUakxzBubysjUGEalRjMyNYZhSc1vzdKTl+fUp0pERKTzFKw88sDKbVw0ZYi/9WlncQU7DjiPxRW1/nIhQYZhyVGMTInhrHGDGJXqfLtuZGoMSdFdG6JARERE+hYFq2NwsLKWzYVlbCoocx4LywCYe+9qf5mk6DBGpkQHhCcnQA1Liuryve1a0uU5ERGRvknBqh0NPsvukko2FZYFBKly9pXVtPu8f5kzkju/3n39nnR5TkREpG9SsHJVHqlny75yfwvU5sIythSWU13XADgdyEenxnDaqGTGp8VySlo849NiSY4JBzQsgYiIiAzwYNVah3JrLfvKavyX8TYXlrOpsIy8kkr/oJlxESGMT4vjyukZjE+L45S0OMacFEN4SOtjPYmIiIjAAA9WD6zcxtcmDG7WCrWpsIzSqqZhC4YnRzF+cBwXTRnC+LQ4xqfFMiQhsss3Cla/JxERERmwweqNDfsA+PqD7wIQHhLEuMGxnJs12N8KNXZwLLERoZ4cT/2eREREZMAFqz++uZUHVm47av1Nc0byo3PG9kKNRERE5ERhbMu78faQnJwcm5ub263HUIdyERER8YIxZq21Nqejcsc3oJKIiIiI+A3oYKUO5SIiItKTBnSwUodyERER6UkDOliJiIiI9CQFKxERERGPKFiJiIiIeETBSkRERMQjClYiIiIiHlGwEhEREfGIgpWIiIiIRxSsRERERDyiYCUiIiLiEQUrEREREY8oWImIiIh4RMFKRERExCMKViIiIiIeUbASERER8YiClYiIiIhHFKxEREREPKJgJSIiIuIRBSsRERERjyhYiYiIiHhEwUpERETEIwpWIiIiIh5RsBIRERHxiIKViIiIiEcUrEREREQ8omAlIiIi4pFOBStjzEJjzJfGmO3GmDvaKHO5MWaTMWajMeZv3lZTREREpO8L6aiAMSYYeAhYAOQDa4wxr1hrNwWUGQPcCcy21h4yxgzqrgqLiIiI9FWdabGaDmy31u601tYCy4ALW5T5DvCQtfYQgLX2gLfVFBEREen7OhOshgB7Apbz3XWBTgZONsa8b4z5yBiz0KsKioiIiPQXHV4K7MJ+xgBzgaHAO8aYidba0sBCxpibgJsAhg0b5tGhRURERPqGzrRY7QUyApaHuusC5QOvWGvrrLW7gK04QasZa+2j1toca21OamrqsdZZREREpE/qTLBaA4wxxowwxoQBVwKvtCjzd5zWKowxKTiXBnd6WE8RERGRPq/DYGWtrQe+B6wANgPPWWs3GmN+YYxZ5BZbAZQYYzYBq4B/s9aWdFelRURERPoiY63tlQPn5OTY3NzcXjm2iIiISFcYY9Zaa3M6KqeR10VEREQ8omAlIiIi4hEFKxERERGPKFiJiIiIeETBSkRERMQjClYiIiIiHlGwEhEREfGIgpWIiIiIRxSsRERERDyiYCUiIiLiEQUrEREREY8oWImIiIh4RMFKRERExCMKViIiIiIeUbASERER8YiClYiIiIhHFKxEREREPKJgJSIiIuIRBSsRERERjyhYiYiIiHhEwUpERETEIwpWIiIiIh5RsBIRERHxiIKViIiIiEcUrEREREQ8omAlIiIi4hEFKxERERGPKFiJiIiIeETBSkRERMQjClYiIiIiHlGwEhEREfGIgpWIiIiIRxSsRERERDyiYCUiIiLiEQUrEREREY8oWImIiIh4RMFKRERExCMKViIiIiIeUbASERER8YiClYiIiIhHFKxEREREPKJgJSIiIuIRBSsRERERjyhYiYiIiHhEwUpERETEIwpWIiIiIh7pVLAyxiw0xnxpjNlujLmjle3XGWOKjDGfu9ON3ldVREREpG8L6aiAMSYYeAhYAOQDa4wxr1hrN7Uo+qy19nvdUEcRERGRfqEzLVbTge3W2p3W2lpgGXBh91ZLREREpP/pTLAaAuwJWM5317V0iTFmvTHmBWNMRms7MsbcZIzJNcbkFhUVHUN1RURERPourzqvvwpkWmsnAW8CT7VWyFr7qLU2x1qbk5qa6tGhRURERPqGzgSrvUBgC9RQd52ftbbEWnvEXXwMONWb6omIiIj0H50JVmuAMcaYEcaYMOBK4JXAAsaYtIDFRcBm76ooIiIi0j90+K1Aa229MeZ7wAogGHjcWrvRGPMLINda+wrwfWPMIqAeOAhc1411FhEREemTjLW2Vw6ck5Njc3Nze+XYIiIiIl1hjFlrrc3pqJxGXhcRERHxiIKViIiIiEcUrEREREQ8omAlIiIi4hEFKxERERGPKFiJiIiIeETBSkS6ZtVversGIiJ9loJVf6QPNmmpJ94TDXVQXQr//G33H0tEpJ/qcOR16YP++VuYd2dv10L6ksD3hK8BaiubprrK5sv+qQLqqprmawPnK4/e1nCk6Xi/SoPoVGeKGQTRKRA9qPl847aIBAg6xv/hVv1G73UR6VcUrPqL2krYsQq2v+ksb/hfyJgO8UN7t17SvuMNBrWVUFnsTFXFUFnkLhdBVUnTPMDvRzrl62s6v38TBGExEBoFYdFNU1QShGU0bdu3HvZ83PS8uioo3Q0YqD8Ce9c6dbENRx8jKASiUiDGDWLRg5rPR6e6y24oCw5teq7+iRCRfkbBqi87nA9b34Av34Adbzf/0HrheucxLAZGneWErKHTIS0bQiN6p77HY6C2TLQMBnXVLYJRy6BU1DxI1VW1vt+QSCeAHClrWldV4jwOPx3GngthUc77ozEshUY3D09h0RASAcZ07ZyWxMOSw0ev9/mg+hBUHnDOo8J9bDlfvN0p01YAjExsCl0AH/4/GDTemWJO6np9RUR6kIJVX+LzQeFnTpDauhz2feGsTxoJM26GsQth2GnwyxT4ztuwZw3kr4H8T2DzK07ZoFBIm+SErIxpzmP80L7/YdSfWyashaqDULYXygudx7JCKC9wtv9lflNwqq1ofR/B4W6YSHYeU052L6mlOMtRKU3bo1KcUBT4M20r7PSkoCC3/snA+PbLWuu8FhUH3CDZGMaKnPd+wWdQvNUpuyLgfRGZCKnjm4LWoPHOcnRyt52W53ryn4iB+g+LSB+mYNXbaqtg1z/hy+WwdQVU7HMuz2TMgLP/A8Z+HVLGHB2MhpzqTNzsLJfvdwJW/honcK19Ej5+2NkWM7gpZGU0tmpF9uRZNldX7QaQQuexfJ+z/sP/BxHxEJng9MtpfIyIPzpIHI+ufNg01Dn1KytwglJZYUCAapzf17z/UUt73ZuND5kG4849OihFpzotS309/DY6847j34cxEB7rTMmjmm+b+5Om+SXxcPt2OLAJirY4jwe2wIYXoCYgSEYPgkHjmoeu1HHOe6gvaahz/omY/X0wwc7vepD72B0///78D4tIV/ShfyKMtbZXDpyTk2Nzc3N75di9rqzQucS39Q3Yudq5JBIWC6PnO5dwRi9o/z/wzryBGuqcFq/8XCdw7fnE7ROD06o1eKJ7+XCaMyUMa/0Pe5dCSD1U7HeCRnlhwNQYTPY54aTmGFpWgkICwlZ88+DV2rrAgBYe17zzdGPrzpHyppalMncqL2w+X3EAaPE7EhIBcekQmw5xaS3mh0BsmnPJKjik51qS+tAfFU+19fpZ6/x8Dmx2pqLGxy+btwrGph/dupU6FsJjmu+vK6+ftc4/B9WHoKbUeWxzKm3+WFvezo6NG7ICA1ew8941Qe58cMB8e+vddflrYM6/OSEzdRwkj+6fXQVEOtIDf2uNMWuttTkdllOw6gHWOp1/v3wDvnwdCj931icMg5PPdcLU8NkQEta99ag44LZouS1bez+F+mpnW8xJTsBqDFvpU5xWrSXxcHepc6mrzbBU2HYIMcEQO9id0txpsBtGAtb9bjj8eJfzQVVz2Pkgqil1Hw8HzLexvbVO0/46BDnhqjFsFa5zlgP7JzWKTHRDkhuU/PPpTj3j0p0ynW1d6AuX6PqzrgZGnw8O72neunVgk3NZMbBPV8IwGHSKEzYGjYeX/gW++VIroaiN4NReC2VQqPMe8U8JTn/J/RuOLpt5hjNZn/Me9jUEzLuP1ueub5xvrWzjvM8Jlwd3tF43EwSJI9ygNdYNm2MheYzTJ0+kv6kpcz7T/nqJgtWAC1YtPwDqamDXO05/ka0rnEtGGCe0jF3oBKpB43v38k9DHezf2DxsHdrlbAsKcfr5HNgEwWHQUHv086OSm4JSY0CKCwhPselOH6Gg4I7rcjwBpLG/TnvBa/tbUPDp0c8ddwHMvLkpNHl9iXSgtiT1N74GOJTXvHXrwBYncPnqWn9OaHTzcNTsscUUEbC+o8vWPRm2l8TDz/Y7QauxRa9oS1P48tW7BQ0kDm9q0WsMXiknH9261xa914+PXr/OqToIX30Eu9+HL15wus+0dOYd3fJaKlhBz75Rl8TD7duavsW3c5Xzja7QaBg1z+krNeYc52vlfVlFESz/CWx88ehtp1wIM/+/puAUEu7dcXv6Z6VWJFn1m9YHOz3tezD/Lm/f34F6Oli1daz6Wji40wmagYGreFvzsJkwrClopbr92FJPdvrHdfZY0jG9fq2rOOCEqN0fQN77cGCjsz443GmoGD4LMmfD0xf2mRargdt5vf6I80dzyjVOa4uv3mmh8dU5fYEaapvmfXXOttbK+dz1geX85QPKAdx7MmAhbihMvspplco8vX/1aYhJhcsedybouV92/acmPW3enU3vu578UPOi878XxwoJczr8DxrXfH1DHRzc1RS0Gh93/rP5JdC4oQFh62Rn3d61TWOT9ae/e72hoc4JDRX7oNIdKqXoS+eLLZGJxz6obn93ON8JUbvfd4JUyTZnfWiU86WuCRc5QSp9ap99jw3cYLV1hfN4f5a3+w0KdcYPCgp1AldjHyXA379oytUw76feHle805MfbCIt9eQ/EcdyrOBQJyg1hqVGjZdTi7YEhK3VsGNlU5m/nNU0HxYbMBBsavMR+aNT3FH63W1dGZ2/rw9XUVfjfImnYr/b/3S/E55aPlYWc1Sf1IemN837vzmc4nS7iE5x17WyHJnkfFmmu86pu1jrdD9pbI3a/X7Tl6zC45zhhaZc4zRQpGU3Hzy4pT70d33gXQpsq3l//CKYeKkbjMKcN6E/JIW46xrnQ9su11bfiYHajNuXfglFuove58fO1+D0Ib1/IixeFjAobFHTfONUVeJ0sG+pcXT+o0JXizAWner8s/yz/c2/NdldAv+uH6lo+tZzs6DkTo3bakqP3o8Jds4p5qSmL/PEDIbYk5zH6FT477Phkv9uGiS4qnGg4JKm5epDbdc1MrHt4BWV0jQG3n+dAXd85Qzx0pn+r8ej5e+VtU6/xsbWqN0fNI33F5nkXtY73Xk8Kav769dF6mMFfacvg4jIQNeZv4G+BiccBI7EHzhAbGVxwLZi5z6XHXKHqQgKcYNWiBO2Wi775xvLBrVYdoeo8Ae2EOfLR0mjnNDU2uC+wWEBAemkgMDUGJ7cdVHJHYeEzrx+DfVQfbDt4NVyua0g2ygk0vmyRXiM08Lon3cn/3y006fOP99amZijv9m+JB5ufq+pNWr3B069wHlths92LusNnw0pY/v85U/1seppfagZUkSkx3Xmb2BQcNMdBTqjttIJWe/8AT57+ujtw93WDdvg9I/1NTQNTdHmcr07bEV9i20+p29u6VdNrSjQNHTFiDMhe7ETohrHquvK0Csd6czrFxzitnwN6tw+fT6nBW3Vr2HNX47ePngiDM5yWuNq3anqoPMa1Fa668vbD2fN6hfm3i4rtunbpI+c7jzGD4PRZzcFqaSR/WdQ5C4a2C1Wat4XERlYdCXi+HT1nKx1xoBrDFn+wOVOLed3vdP60DbdNARCT1KLFfT7H6KIiEivMsYZ3y80EujicEEDMZh2Qt++oCkiIhKorwxX0V8NxHPqYxSsRESk/+jrw1X0dT15TidoiFOwEhEREe8NxGDaCQpWIiIiIh5RsBIRERHxiIKViIiIiEcUrEREREQ8omAlIiIi4hEFKxERERGPKFiJiIiIeETBSkRERMQjClYiIiIiHlGwEhEREfGIgpWIiIiIRxSsRERERDyiYCUiIiLiEQUrEREREY8oWImIiIh4RMFKRERExCMKViIiIiIeUbASERER8UingpUxZqEx5ktjzHZjzB3tlLvEGGONMTneVVFERESkf+gwWBljgoGHgAjej2sAAAskSURBVHOBU4DFxphTWikXC/wA+NjrSoqIiIj0B51psZoObLfW7rTW1gLLgAtbKfdL4HdAjYf1ExEREek3OhOshgB7Apbz3XV+xpipQIa19rX2dmSMuckYk2uMyS0qKupyZUVERET6suPuvG6MCQLuA37UUVlr7aPW2hxrbU5qaurxHlpERESkT+lMsNoLZAQsD3XXNYoFsoDVxpg8YCbwijqwi4iIyImmM8FqDTDGGDPCGBMGXAm80rjRWnvYWptirc201mYCHwGLrLW53VJjERERkT6qw2Blra0HvgesADYDz1lrNxpjfmGMWdTdFRQRERHpL0I6U8ha+zrweot1d7VRdu7xV0tERESk/9HI6yIiIiIeUbASERER8YiClYiIiIhHFKxEREREPKJgJSIiIuIRBSsRERERjyhYiYiIiHhEwUpERETEIwpWIiIiIh5RsBIRERHxiIKViIiIiEcUrEREREQ8omAlIiIi4hEFKxERERGPKFiJiIiIeETBSkRERMQjClYiIiIiHlGwEhGR/7+9O4+Vq6zDOP59bKlA2UEQaAVEIBDUsoioLNoioUioYDAgJhA0BKTK4hIQQkBiwq7/EAgCatiRzUJQCsqiiRTa0sItLQWkQCttcUUkYf35x3lvmFzu7R1y39+09+T5JJN7ZrnzzDtzzju/Oe9ZzKwSF1ZmZmZmlbiwMjMzM6vEhZWZmZlZJS6szMzMzCpxYWVmZmZWiQsrMzMzs0pcWJmZmZlV4sLKzMzMrBIXVmZmZmaVuLAyMzMzq8SFlZmZmVklLqzMzMzMKnFhZWZmZlaJCyszMzOzSlxYmZmZmVXiwsrMzMysEhdWZmZmZpW4sDIzMzOrxIWVmZmZWSUurMzMzMwqcWFlZmZmVokLKzMzM7NKXFiZmZmZVeLCyszMzKwSF1ZmZmZmlbiwMjMzM6vEhZWZmZlZJV0VVpIOkvS0pGclnT7I/SdIelLSPEl/lrRL/ZdqZmZmtmYbtrCSNAa4DJgK7AIcNUjhdENEfDIiJgEXApdWf6VmZmZma7hu1ljtBTwbEX+NiDeBm4BpnQ+IiFc7ro4Hot5LNDMzMxsdxnbxmK2BlzquLwU+O/BBkk4CTgPGAZOrvDozMzOzUaSbwqorEXEZcJmkbwBnAccMfIyk44Hjy9XXJD1dK38ImwF/T85w1ujKaWtWG9vUy6w2tqmtWW1sUy+z2timXmVt082DuimslgETO65PKLcN5Sbg8sHuiIgrgSu7eWE1SJodEXs6a83PamObepnVxjb1MquNbWprVhvb1MusNrap11nD6WYbq8eAHSRtJ2kccCQwo/MBknbouPoV4Jl6L9HMzMxsdBh2jVVEvC1pOnAvMAa4JiIWSPoJMDsiZgDTJR0AvAX8i0GGAc3MzMzarqttrCLiHuCeAbed3TF9cuXXVUvPhh2dNWpy2prVxjb1MquNbWprVhvb1MusNrap11mrpAgfGcHMzMysBp/SxszMzKyS1hZWw52Gp2LONZJWSurLyig5EyU9IOkpSQskpQ2/Slpb0qOS5pesc7OyOjLHSHpc0t3JOUs6Tr80OzFnI0m3SlokaaGkzyXl7FTa0n95VdIpGVkl79QyT/RJulHS2kk5J5eMBbXbM9gyK2kTSfdJeqb83Tgx64jSrnclVduLaYisi8o8+ISkOyRtlJRzXsmYJ2mmpK1GmjNUVsd935cUkjbLypJ0jqRlHcvXwRk55fbvls9qgaQLR5ozVJakmzvas0TSvMSsSZIe6e9vJe2VlPNpSX8pfftdkjYYac6IRETrLjQb2T8HfJzmgKXzgV2SsvYDdgf6ktu0JbB7mV4fWJzYJgHrlem1gFnA3sntOw24Abg7OWcJsFlmRsn5NfDtMj0O2KgHmWOA5cA2Sc+/NfA8sE65fgtwbELOrkAfsC7NdqD3A5+o+PzvW2ZpTsV1epk+HbggMWtnYCfgQWDP5HYdCIwt0xfUaNcQORt0TH8PuCKrTeX2iTQ7VL1Qa3keol3nAD+o9RmtIudLZT7/cLm+eeb713H/JcDZie2aCUwt0wcDDyblPAbsX6aPA86r+Zl90Etb11gNexqeWiLiYeCfGc89IOfliJhbpv8LLKT5osvIioh4rVxdq1zSNsaTNIHmMB1XZWX0kqQNaRb+qwEi4s2I+HcPoqcAz0XEC4kZY4F1JI2lKXz+lpCxMzArIl6PiLeBh4DDaz35EMvsNJpimPL3q1lZEbEwIqofHHmIrJnlPQR4hOY4hBk5Kac1W0X/+jPgR7VyhsmqaoicE4HzI+KN8piViVkASBLwdeDGxKwA+tcebUiF/mKInB2Bh8v0fcDXRpozEm0trAY7DU9KEbI6SNoW2I1mTVJWxpiyinglcF9EpGUBP6fpJN9NzOgXwExJc9ScCSDDdsArwC/L8OZVksYnZXU6kkqd5GAiYhlwMfAi8DLwn4iYmRDVB+wraVNJ69L80p04zP+M1BYR8XKZXg5skZy3OhwH/C7rySX9VNJLwNHA2cM9fgQ504BlETE/K2OA6WWY85paQ8SD2JFmnp8l6SFJn0nK6bQvsCIiMo87eQpwUZkvLgbOSMpZwHsrT44gv79YpbYWVq0laT3gNuCUAb8Sq4qIdyJiEs0v3L0k7ZqRI+kQYGVEzMl4/kHsExG7A1OBkyTtl5AxlmZV9eURsRvwP5rhpTRqDt57KPCbxIyNaTqv7YCtgPGSvlk7JyIW0gxbzQR+D8wD3qmds4r8oGUnkpd0JvA2cH1WRkScGRETS8b0jIxSaP+YxMJtgMuB7YFJND8mLknKGQtsAuwN/BC4paxRynQUiT/EihOBU8t8cSplLX6C44DvSJpDs6nMm0k5XWlrYfVBT8MzKkhai6aouj4ibu9FZhnCegA4KCniC8ChkpbQDNlOlnRdUlb/Wpf+Ve130Awb17YUWNqxlu9WmkIr01RgbkSsSMw4AHg+Il6JiLeA24HPZwRFxNURsUdE7Edz0OHFGTkdVkjaEqD8rTIUsyaQdCxwCHB0KRqzXU/eUMz2NIX9/NJnTADmSvpoRlhErCg/Mt8FfkFOfwFNn3F72QzjUZq191U2yh9MGco/HLg5K6M4hqafgOZHX8r7FxGLIuLAiNiDplh8LiOnW20trIY9Dc9oU369XA0sjIhLk7M+0r/3kKR1gC8DizKyIuKMiJgQEdvSfE5/jIjqa0EAJI2XtH7/NM2GvdX35oyI5cBLknYqN00BnqqdM0Avfn2+COwtad0yP06h2davOkmbl78fo/kCuCEjp8MM3jtjxDHAb5PzekLSQTTD7IdGxOuJOZ2nNZtGXn/xZERsHhHblj5jKc1OPcsz8vqL7eIwEvqL4k6aDdiRtCPNDi+ZJxQ+AFgUEUsTM6DZpmr/Mj2ZpNPddfQXHwLOAq7IyOna6txyPvNCs13GYprK9czEnBtpVhG/RbOQfyspZx+a4YknaIZG5gEHJ2V9Cni8ZPVRaa+RLnK/SOJegTR7ic4vlwXJ88UkYHZ5D+8ENk7MGg/8A9iwB5/RuTRfmn3AtZS9mBJy/kRTjM4HplR+7vcts8CmwB9oOv77gU0Ssw4r028AK4B7E7OepdnetL/PGPHeekPk3FbmiSeAu4Cts9o04P4l1NsrcLB2XQs8Wdo1A9gyKWcccF15D+cCkzPfP+BXwAk1MoZp1z7AnLIczwL2SMo5meb7fjFwPuXg56vr4iOvm5mZmVXS1qFAMzMzs55zYWVmZmZWiQsrMzMzs0pcWJmZmZlV4sLKzMzMrBIXVmZmZmaVuLAyMzMzq8SFlZmZmVkl/wfPwVCxYBR1twAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.rcParams['figure.figsize'] = [10, 6]\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(\n",
    "    list(map(lambda l: l.step, valid_accuracy)),\n",
    "    list(map(lambda l: l.value, valid_accuracy)),\n",
    "    marker='+', label='Validation Accuracy'\n",
    ")\n",
    "ax.plot(\n",
    "    list(map(lambda l: l.step, interpolate_accuracy)),\n",
    "    list(map(lambda l: l.value, interpolate_accuracy)),\n",
    "    marker='+', label='Interpolate Accuracy'\n",
    ")\n",
    "plt.title('Algebra/Linear_1d Accuracy')\n",
    "ax.legend(loc='upper left', frameon=False)\n",
    "plt.xticks(np.arange(0, 20, step=1.0))\n",
    "plt.yticks(np.arange(0.3, 1.0, step=0.1))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Accuracy for validation dataset is growing constantly up to 85% while for interpolate dataset, it doesn't change much. Interpolate dataset contains too complicated and generic problems compared to training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'resp': '5', 'score': 0.0}]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_process.predict_single(\"Solve 5 * d = 20 for d.\", model, device, n_best=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "mathematics_dataset_transformer.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}