8-translation-transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is adapted from [this one](https://github.com/fastai/fastai_docs/blob/master/dev_course/dl2/translation_transformer.ipynb) created by Sylvain Gugger.\n",
    "\n",
    "See also [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html) from Harvard NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention and the Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nvidia AI researcher [Chip Huyen](https://huyenchip.com/) wrote a great post [Top 8 trends from ICLR 2019](https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html) in which one of the trends is that *RNN is losing its luster with researchers*.\n",
    "\n",
    "There's good reason for this, RNNs can be a pain: parallelization can be tricky and they can be difficult to debug. Since language is recursive, it seemed like RNNs were a good conceptual fit with NLP, but recently methods using *attention* have been achieving state of the art results on NLP.\n",
    "\n",
    "This is still an area of very active research, for instance, a recent paper [Pay Less Attention with Lightweight and Dynamic Convolutions](https://arxiv.org/abs/1901.10430) showed that convolutions can beat attention on some tasks, including English to German translation.  More research is needed on the various strenghts of RNNs, CNNs, and transformers/attention, and perhaps on approaches to combine the best of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/giga-fren/cc.en.300.bin'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/data_save.pkl'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/giga-fren.release2.fixed.en'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/giga-fren.release2.fixed.fr'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/questions_easy.csv'),\n",
       " PosixPath('/home/jhoward/.fastai/data/giga-fren/cc.fr.300.bin')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = Config().data_path()/'giga-fren'\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reuse the same functions as in the translation notebook to load our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_collate(samples, pad_idx=1, pad_first=True, backwards=False):\n",
    "    \"Function that collect samples and adds padding. Flips token order if needed\"\n",
    "    samples = to_data(samples)\n",
    "    max_len_x,max_len_y = max([len(s[0]) for s in samples]),max([len(s[1]) for s in samples])\n",
    "    res_x = torch.zeros(len(samples), max_len_x).long() + pad_idx\n",
    "    res_y = torch.zeros(len(samples), max_len_y).long() + pad_idx\n",
    "    if backwards: pad_first = not pad_first\n",
    "    for i,s in enumerate(samples):\n",
    "        if pad_first: \n",
    "            res_x[i,-len(s[0]):],res_y[i,-len(s[1]):] = LongTensor(s[0]),LongTensor(s[1])\n",
    "        else:         \n",
    "            res_x[i, :len(s[0])],res_y[i, :len(s[1])] = LongTensor(s[0]),LongTensor(s[1])\n",
    "    if backwards: res_x,res_y = res_x.flip(1),res_y.flip(1)\n",
    "    return res_x, res_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqDataBunch(TextDataBunch):\n",
    "    \"Create a `TextDataBunch` suitable for training an RNN classifier.\"\n",
    "    @classmethod\n",
    "    def create(cls, train_ds, valid_ds, test_ds=None, path='.', bs=32, val_bs=None, pad_idx=1,\n",
    "               dl_tfms=None, pad_first=False, device=None, no_check=False, backwards=False, **dl_kwargs):\n",
    "        \"Function that transform the `datasets` in a `DataBunch` for classification. Passes `**dl_kwargs` on to `DataLoader()`\"\n",
    "        datasets = cls._init_ds(train_ds, valid_ds, test_ds)\n",
    "        val_bs = ifnone(val_bs, bs)\n",
    "        collate_fn = partial(seq2seq_collate, pad_idx=pad_idx, pad_first=pad_first, backwards=backwards)\n",
    "        train_sampler = SortishSampler(datasets[0].x, key=lambda t: len(datasets[0][t][0].data), bs=bs//2)\n",
    "        train_dl = DataLoader(datasets[0], batch_size=bs, sampler=train_sampler, drop_last=True, **dl_kwargs)\n",
    "        dataloaders = [train_dl]\n",
    "        for ds in datasets[1:]:\n",
    "            lengths = [len(t) for t in ds.x.items]\n",
    "            sampler = SortSampler(ds.x, key=lengths.__getitem__)\n",
    "            dataloaders.append(DataLoader(ds, batch_size=val_bs, sampler=sampler, **dl_kwargs))\n",
    "        return cls(*dataloaders, path=path, device=device, collate_fn=collate_fn, no_check=no_check)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqTextList(TextList):\n",
    "    _bunch = Seq2SeqDataBunch\n",
    "    _label_cls = TextList"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to notebook 7-seq2seq-translation for the code we used to create, process, and save this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos quelles questions devraient être traitées respectivement au niveau international et au niveau national , ou quelle xxunk devrait être établie entre la réglementation internationale et la réglementation nationale ?</td>\n",
       "      <td>xxbos which issues should be dealt with internationally and which nationally , or what division should be made between international regulation and national regulation ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos comment la culture et les arts y vivent - ils , et comment la société civile les prend - elle en considération dans le développement de la ville ?</td>\n",
       "      <td>xxbos where do art and culture fit in , and how is civil society taking them into consideration in developing the city ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos qu’arrivera - t - il si les entreprises canadiennes et les gouvernements ne se xxunk pas sur la question du conflit entre le travail et la vie personnelle ?</td>\n",
       "      <td>xxbos what will likely happen if canadian organizations and governments do not deal with the issue of work – life conflict ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos qu'adviendra - t - il de l'examen de rendement et de la rémunération au rendement de xxunk et comment se xxunk - t - il à cet égard ?</td>\n",
       "      <td>xxbos what happens to xxunk 's evaluation review and performance pay and how will this make him feel ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos quels avantages prévoit - on en général pour la région du delta de beaufort par suite de la signature de xxunk / d'une entente future sur l'autonomie gouvernementale ?</td>\n",
       "      <td>xxbos what benefits to the beaufort delta region generally are expected as a result of this aip / future self - government ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transformer model](images/Transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shifting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add a transform to the dataloader that shifts the targets right and adds a padding at the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = data.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.stoi['xxpad']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shift_tfm(b):\n",
    "    x,y = b\n",
    "    y = F.pad(y, (1, 0), value=1)\n",
    "    return [x,y[:,:-1]], y[:,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.add_tfm(shift_tfm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input and output embeddings are traditional PyTorch embeddings (and we can use pretrained vectors if we want to). The transformer model isn't a recurrent one, so it has no idea of the relative positions of the words. To help it with that, they had to the input embeddings a positional encoding which is cosine of a certain frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0000, 0.0667, 0.1333, 0.2000, 0.2667, 0.3333, 0.4000, 0.4667, 0.5333,\n",
       "        0.6000, 0.6667, 0.7333, 0.8000, 0.8667, 0.9333])"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = 30\n",
    "torch.arange(0., d, 2.)/d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    \"Encode the position with a sinusoid.\"\n",
    "    def __init__(self, d):\n",
    "        super().__init__()\n",
    "        self.register_buffer('freq', 1 / (10000 ** (torch.arange(0., d, 2.)/d)))\n",
    "    \n",
    "    def forward(self, pos):\n",
    "        inp = torch.ger(pos, self.freq)\n",
    "        enc = torch.cat([inp.sin(), inp.cos()], dim=-1)\n",
    "        return enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXd4W9XdgN+j7b1HbMfxyCR7sMNKSspI2CvMtkDYBdp+hbChQKFQWlpmmGXPFsJIQiCEHTLIDpke8V7ylGzN+/1xdW3Z1pZspY3e5/HjRHeda91zf+e3hSRJxIgRI0aMGAqqaA8gRowYMWIcWMQEQ4wYMWLE6EdMMMSIESNGjH7EBEOMGDFixOhHTDDEiBEjRox+xARDjBgxYsToR0wwxIgRI0aMfsQEQ4wYMWLE6EdMMMSIESNGjH5ooj2AUMjMzJSKioqiPYwYMWLE+K9iw4YNzZIkZfnb779SMBQVFbF+/fpoDyNGjBgx/qsQQlQGsl/MlBQjRowYMfoREwwxYsSIEaMfMcEQI0aMGDH6ERMMMWLEiBGjHzHBECNGjBgx+hERwSCEeFEI0SiE2OZluxBC/EMIsVcIsUUIMcNt22VCiD2un8siMZ4YMWLEiBE6kdIYXgZO8rH9ZGCM62cR8DSAECIduBs4HDgMuFsIkRahMcWIESNGjBCISB6DJElfCyGKfOxyOvCKJPcRXSOESBVCjACOB1ZKkmQEEEKsRBYwb0ZiXOHSZrby5a5GzpiWjxAijBPthz0rIWs8jDoKwjnXEOKUnNR21bKndQ/GHiMmmwmz3YxWpSXNkEaqPpX8xHxKU0vRqP4rU2CGju5W6GwAqwnsPZA3HXTxQ3pJSZJwmsw42tpwdnbg6OjE2dWJ09yNs9uM1NODZLMh2exINlv/g9UqhEaL0GgQeh2quHhUcQZUCQmoEpNQJyWiSklBk5bGjiYzVruT6YX/vWu2drON/UYzkwtSwjuRrRu2/wcSsqHwcNAnRWaABxjDNbvzgSq3/1e7PvP2+SCEEIuQtQ0KCwuHZpQDePSzXby2Zj+F6QnMHBXkpJAkWP8CbHoDajb0fT7ycJj9Oxj7y2EREBsqW7nl/S08c/EMRmcPfojL28tZXbWar6u/5mfjz5hsJr/nNKgNjE8fz4ycGcwbNY9DMg4JT3D+t7P5bVh6PTisfZ9ljIbzXuGeH6E4M4HLjioK6pSSJOFoacFaVYWtthZbTS32+jpsjY3YG5uwNzXhMBqRLJbI3osHenRxtBmSKB83Cl1ODpqcbLR5eWhH5KEtyEc3ciSquLghH0c4/O6dTXyzt5n1d/yCZIM2+BM4nbDtPfj8Xuiolj8TKhgxFebeBaVzIjtgD9S397Dk6zIWnzIerXpo3cPDJRg8vTUkH58P/lCSlgBLAGbNmuVxn0jSarLy3gb5AfhwU03wguGrh2H1nyF3Csy9G8adAhXfwHePw5vnw5Tz4cxnh1Q4VLaYuPKV9RhNVpZtreeGubJgMNlMfLj3Q97e9TZl7WUAjEsbx2mlpzE2bSxj0saQE59DnCaOeG08NoeNdks7RouR8vZytjdvZ3vLdl7Z/govbnuR/MR8Tik+hYXjF5IV7zfbPiAkSeKd9VXsN5rp7LHTY3NwxTEljM05gFZoTiesfhC+fgSKjoFZvwZdIlg6YcVtSM/Npav7Uu52HIfdKXH57OJBp5DsdqyVlVj27MWyby/WfWVYKsqxVe7HaeovpFUpKWizs9FkZ6MvKUGdkY4mLQ11Whqq5GTUScmoEhNRxcejio9D6PWodDqEVgtaba/wliQJHA4ku13+sVhwdvcgdZtxmkw4OrtkDaS9ndqKOj7/egeplk6S2syk12/GXl8/SAPR5OSgGzUKXUkx+pJS9KNL0Y8bhyYjY+j+/gHy/b5mvtjZCMDqXU2cNjUvuBOYjfD6uVCzXhYEpz8hf175PWz/N7xxAVz0LpQcF+GR97F8Wz23/nsLVruTM6fnh6/5+GG4BEM1MNLt/wVArevz4wd8vnqYxuSTN9bup8fmZEpBCh9truXO+YcELqV/ekUWCtMugtOf7Hv5Z4+Hmb+St33zVxgxDY68dkjG32qy8uuX1iFJEiPT4/h+XwsLj0rjha0v8J+9/8FkMzElcwq3HX4bxxccz4jEEV7PpVVpidfGMyJxBBMzJjK/ZD4A7ZZ2Vu1fxYqKFTy/9Xle3v4yC0oXcNnEyyhJKQlr/Hsbu7jl/a2oVYIkg4aObhvxOg33nDYxrPNGDIcd/rMItr0P0y+GU/8GGl3f9uJjaf7XJTxqf5bpKSZu/xjicXBGSg8927fRvW0blp93Ytm7t2/VLwTa/Hx0JcXEz5yFrrAQ7cgCdPn5aPPyUCUkRGToQgjQaBAa1/RPTPS67zPLfuaFzvGMzUnilW4bq//veDQCHC0t2GprsVZVY6vaj7VyP9aKCjo+XYazo6P3eHVmJoZx4zAccgiGSZMwTJyINj9v2DRMp1PiwU9/Jj81DovdwWfb64MTDJIEH1wD9Vvg9Kdg6kJQud4DpSfAEdfAy6fCmwvhkn9D4RERHX+PzcG9H+3gzbX7mVKQwuMXTKc4MzLPgS+GSzAsBa4XQryF7GhulySpTgixAnjQzeE8D1g8TGPyisXu4OXvKzh2bBaXHTmKy/+1nq93NzF3Qo7/g3d/Bh/dBKVzYcHjgzUCtRbm3AlNu+CzO2DEFCiaHfHxL3p1PdVt3bxxxeF8vLWSt3e/zqn//garw8q8onlcPOFiJmdNDus6KfoUzhxzJmeOOZP9Hft5ZccrfLD3Az7Y+wHnjT2P66dfT4o+tJXNlup2AJbfeAxjcpI495nv2VLdFtZ4I8raJbJQmHMnHPP7wd9zYjbPJC7mmA13M7flIwo7dpGytJ4KpwMAdUoK+kMmkLZwIfrx4zCMHYuuuPiAMsk4nRIfb65j9phMLjlCngcfb6nlzOkFaLKy0GRlETd1ar9jJEnC0dyMZe9eenbtwrJrNz07d9Ly0ktgtwOysIibOpW4qVOJnzkDw+TJqHQ6T0MImw8317CtpoO/nT+VH8uMfLylDovdgV6jDuwEa56G3cvhpIdh+kWDt8enwyUfwMunyFrFrz6WtYoI8c76Kt5cu5+rjivh9yeOQ6cZngyDiAgGIcSbyCv/TCFENXKkkRZAkqRngE+BU4C9gBn4tWubUQjxJ2Cd61T3KY7oaPLx5jqaOi389dxijizNIC1ey3821vgXDG1V8O6vIHcSnPcvWQh4Qgg442l4bo68/1VfQ3KQ6q0PvtzZyLqKVv567lTMmm180XUPmswmxiTP5v5jb6EopShi11IoTC7kjiPu4Npp1/L0pqd5Z/c7LKtYxg3TbuDcceeiEsE90Ftr2onXqSnJklezUwpSeW1NJXaHE80Q21f90lELXz4Ao3/RTyjYm5owrfkR049r6F63nnMq5Xplbep4RuY08NHEE2grGsstN541rKvmUNlY1UpNWze/nzeWE8ZlMy4niadX7+P0qfmoVJ7HLoToFRoJRx7Z+7nTYsGyezfdW7fSs3kL3Zs30/XFF/IxOh1xU6YQf/jhJBxxOIapUyMiKHpsDh5dsZtJ+cmcPjWflDgtb62r4od9LRw/Ltv/CWp+gpV3wbhT4fCrvO+XlAOXfgjPnwgfXg+LvurTKsJkS3U7WUl6Fp88ISLnC5RIRSUt9LNdAq7zsu1F4MVIjCMSSJLE89+WMy4niWPGZCKEYMHUPN5eV0Vnj40kX46rrx4Cpw3Of81/tIIhWd7v+bmw9Aa4+P2I3cPPdZ0IdQ8bTM/w0RcfUpIymoa9ZzEl5xdDIhTcSTekc/sRt3PuuHN5eO3D3P/j/Xy+/3PuP/p+chIC0LhcbK1pZ2JeMmrXC2hKQQoWu5PdDV0ckpc8VMMPjOW3gtOOc+4DmL/7HtO332L67lsse/YCyPb+qdN4PnEysxacwAXTuxH/+Q0lOfHcY5zAnQUe4ysOOJZuqkWvUXHiITmoVIKrjy/h5rc38+WuxsC0ZzdUej1xkycTN3kyXHghAPbWVrp/+gnz+g2Y162j+emnaX7ySYTBQPysWSTMPprE2bPRlZaGJERfW1NJTVs3j5wzBZVKcFRpJvE6NSt3NPgXDJZOeO83kJgj+xT8XT+lAE68D/59BWx9B6ZeEPR4PbGzvoPxucPvV4tlPg/gh30t/FzXweXHFPc+jGdMz8did7J8W733A5t2yxFIh14BqQFGTWWPh2P/AHs/h+rIlRHf0PATyaWP80nFR1wx+QreXfA2kzNn8N2+lohdwx9j08by/LznuevIu9jctJmzlp7FZxWfBXSs3eFkR20Hk/L7zFCTXf/eWhNdc5JtzXu0Ll1J1eap7D55IVVXXEHrG2+gycom+w+/p+i99xj7w/dsu/Yu3h9zPJNPPBox9WyYsIBfNr1EYlcFrSar/wtFGbvDySdb65gzPrt3MTR/Sh75qXE8/015RK6hSUsjae5ccm75I8XvvcvYNT9Q8NSTpJ5zDraaGhofepiy+QvY94sTqf/T/XR98y1Oa+B/ux/2tTA2J5GjRmcCYNCqOW5sFit3NOB0+olfWfsctJbDWUtkc1EgTDpb9ht+8Sc5rDVM7A55ITRhxPAvhGKCYQCrdzeh06j6Oaimj0xlVEY8H2yq8X7gl/eDNl42LQTDoVdAXBp89ZcQR9yHJEm8s+sdNtsfQqvW8OrJr3LjjBvRqXUcXZrB1uo2Onps/k8UIYQQnDv2XN6Z/w6FSYX8/qvf89f1f8XhsrN7Y1+TiW6bgylukRdFGQkkGTRsdvkehhNLWRnNzzxL+dlns/dXd1K/PhWL0UHq2WczcsmzjP1xDYUvvkDGFVcQN2kiQq1mTVkLyQZN36Q+5VEktZ7bNK+zu6Fz2O8hWNaUGWnusrLAbR5o1bL2sLm6TY5sijDq5GSS5swh947bKV32KaO/+Jzce+9FP24cbe+/T9WVV7LniCOpvvlm2j/5BEdXl8/zlTWbGJ3d37E+b2IOjZ0WNvvyV9m6Yc1Tcghq0dGB34BKBfPul8NZf3wm8OO8UNEi549EQ2OIZSkNoLzZRFFGPAZtn3NKCMEZ0/L5x6o9NHb2kJ1k6H9Q7UbY8SEcdwskZAZ3QX0SHHkdrLpfPk/e9JDGbXVYefDHB3l/z/s4TOM4vfhWpmRN6d1+ZGkm/1i1lx/LjJx4SHBmgHApSinilVNe4S9r/8LL219mT9se/nLsX0jWeV4Jba2RX/6T3TQGlUowOT+FrcMkGCxlZXQsW0bn8uW9JiJD6QiypnSQdPXD6E64yKd544eyFg4vyeg1hZGUi2XmIub+8FeWlm0b0tDGSLB0cw0JOjVzxvc3uZRkJWC2OmjstJCTbPBydGTQ5ueTdv55pJ1/Hs6eHsw//kjnF6voXLWKzmXLETodCbNnk3zSL0mcMwe1W3SV1e5kv9HM/Cn9o+3mjMtBrRJ8tqPBe8LextfA1BT8Ig+g+BgYexJ88xhMvxQSQg/X3VkvR3eNi5mSok9Fs4mijMHhYEeUZCBJsLvewyrli/vkVf+RHt0o/jlsERhS4KtHQjq8y9rFtZ9fy/t73ufsksswV13G5BH9J8SMUanoNSq+39cc2hjDRKvScvsRt3PnEXfyY+2PXPTJRdR0edbAtla3Ea9TU5zZf7U3pSCVnfUdWOy+NY5QsdXV0fLCC5SdcSZlp5xK8xNPokpJIef22xn95SqKF1jJPKEIvR+hUNvWTWWLmSNK+r8UEo6+EodQkbPzlSEZfyRZV9HK7DGZ/RZIQO/cKG/2nwgZSVQGA4nHHceI++5lzFerGfXG66QtvICe7dup/eMt7Dl6NtU33kTn55/jtFrZbzThcEqUZPWfyynxWo4oSWfljgbPF3LY4ft/QMFhMCoIbcGdX9wL1i749rHQjnexs64TtUoM0nqGg5hgcMPplKg0mj3GCRdmyOUNKo0DJkTNBti3CmbfLL/cQ8GQAkdcB7s+gbotQR3a3N3Mb1b8hg0NG3hw9oPMSrkQUDEmp//DpNeoObQone/3Dp+fwRPnjTuP5+Y9h7HHyKXLLqWsrWzQPltr2pmUl9K32nYxpSAFm0NiZ13kTDFOk4m2Dz6g8rJfsXfOXBofeRSh15Fz22JGr15N0WuvkX7JxWitFVC/FQ670q8j8geXL+fIAYJBJOXyneF4prd8DN0HUOjtABxOiepW8yDBDPTOjeEWDO4ItZr4GTPIWbyY0V+uYtQbr5N6zjmY162j+vob2HPMsRgfuJ/xxkpKPCzyjhubxd7GLoyefD3b3pdL2Bzzu9CTT7PHwyFnyJqHrSe0cyBrDKVZCYGH1kaQmGBwo7a9G6vdSZEHwZCbbECnVrHfaO6/4adXQRMHM38d3sUPvwr0yUGtMmq6arhs2WWUt5fzjzn/YEHpAnY3dKESUJo1eFIfNTqDXQ2dNHUOfRkFX8zKncWLv3wRh9PBZcsvY3vL9t5tdoeTHXX9Hc8Kis9hS0145iRJkjD/tJHa229n9zHHUnfrYmy1tWRedx2lK5ZT/PbbpF96KdocNzPK2iWyAJ98rt/z/1DWQlq81qNteGvBhRikHqSNr4Z1D0NJXXs3NodEYfrgWk95qXHo1CoqoigY3BEqFfEzZpB75x2M+Wo1I59bQuIxxxC/agV/+/qfxC+6kJYXXsDe3KcpK1pP1cC57HTK8y/7EBjzy/AGNvMy6GmDn5eGfIqf6zoZnxudCLyYYHCjoll+UEZlDJ4QapWgID2O/S1uD5PVLK8wDjldDj8Nh7hUmH4J/PyxnILvh7quOi5fcTltljaem/ccxxQcA8Cehk4K0+MHmQCgbwW7riLqqSKMSx/HKye/QrwmnstXXM7mps0A7G3q6s04H0h+ahzpCTq2VIW22na0tWH8178oW7CAygsvpGPZcpJPPolRr79G6WcryLr+OnSjRg0+sKNOnuDTLwGd/6zTNWUtHF6c4THWP6l4Jj86x+Nc86xstjgAURY/ngSDWiUYlRFP2QEiGNwRWi2JxxxD/qOP8MYfn+b5Iy5Em55O4yOPsuf4E6i+4bd0ffsdI1Nl30hV6wDBsO8LaNopa//h5iEUHQtpRXIVhBDo6LFR09bN+BHRKQETEwxulLfID7u3lPPC9Hgq3QXDzo/B0iGXRIgEU8+X8yC2/9vnbg2mBi7/7HI6LB0smbeEadnTerftbuhkjJd6Qsrn/e4hihQmF/Kvk/9FuiGda1Zew88tP/c6lz1pDEIIphSk9DqnA6V782Zqb13MnuOOp+HPD6FOSGTE/X9izNdfk/fAA8TPnOk7Tn7Dy+B0wKzf+L1Wl8VOdWs3U0Z6NiuOzUniRftJqDuqYNenQd3HcKGspD0tkACKMhMOGI3BG7u6JKqP/AVFb7xOyaefkH7ppZjXr6fqiitQX3Yu5+z5kvr9A8LPN78Fcekw8czwB6BSwYxL5fpoLfuCPnx3vWwunRDTGKJPRbMJg1ZFzsCoIxej0uOpMpr7QvU2viqvCkJ1Ug0kd4qsxm5+2+suzd3NXPHZFRh7jDxz4jNMzOirHWSxO6hoMTM2x7OzKlGvISVOS03bgSEYAHITcnl+3vMk6BK4auVVfLd/Owk6NSVehPOU/BR2N3RitvpebTutVto++IDyc8+j4vwL6PzsM1LOPIPiD/5D0dtvkXrOOagTA6g5Y7fChpdgzImQUep395pWOX69IM3zS3VsbhIrnbPoNOTBuuf9Xz8K7DeaUasEI1I8z4OSzAQqW8w4/OUCRJGypq7erHl9SQk5f/w/Rn+1mrxHH0U/YgSXb/+Ew2/9FbW33U739u1g6ZIF9cQzvVcsCJZpF4FQh6Q1/OwSDDGN4QBAiUjylu5fmJFAp8VOq9kGrRVQ/rX85Uco/R0h5Kqr1Ws9rjJMNhPXfn4tDeYGnpr7VL9wVHn88mT1VYE0PzWO6tbwk28iSV5iHs/Pex6VULG6/X7GFNi8fgdTClJxSrCjtsPjdntLC01PPMneE+ZQd+tinCYTOXfeweivv2bEPfdgGD8+uMHtWQFdDXDolQHtrgjd/FTPNY8yE/WkJRhYm3yivJrsagxuPMNAZYuZ/NQ4r6VHijITsDqc1LYdWM+RQqvJSqvZRumAiCSVTkfK/FMZ9eorPLbwHrZMnE3HsmVUnH0OlRecQ0e5E2niOZEbSFKuHLq66Q1wBJc/tLOug2SDhtwhDgn2RkwwuFHe4jlUVUGxuVa2mGDTm4CQqy1Gksnnyufd0l9rsDlt/H7179ndupvHjn+MGTkzBh2qJE6N8dB3QaEgLa53VXsgMSp5FM/84lnskpXG+Cdot3g2Fym+h4GJbpayMmrvuIO9J8yh+YknMEyayMgXnqfkk49Jv+iiwLQDT2z/QDYvBFhvv6ZNjkIpSPNeDG9MTiJL7YeD5AzLOTlUVBnNXs1IcGBEJvmirFkOKR8YquqObvQYnj/0PMZ8tZrsW27BVltLzbfp7LvyboyvvY7THCGteuZlYGqEXcuCOmxnfSfjRyRHrZ5WTDC4sDucVBnNHiOSFJTJsr+lCza9LpfdTR3pdf+QSMmXk5+2vC2X/EWOornvh/v4rvY77j7ybmbne67GuqehE5XwPSHy0+KoaesekszVcFHb8+iuvgSzs5GbV9+MzcMqKzvZQFq8ln1N8uQ3//QTVddeR9kpp9Lx0ceknHUmJZ9+QuGzz5J49NHhTSxbj1xZc8J8UAeWC1rT2o1WLchK1HvdZ1xOEl+0ZCBljpMFzwHGfqOZkR4czwqKYKhoOTAFw75GeVwlHsJtFUamx1PdaobEJDLOPZXSk6rJv/xoNBkZNNx/P3tPmEPj449jN4YZqFE6F5Ly5PdFgDidErvqO5kQhcQ2hZhgcFHb1oPNIVGc6X1CjHTZjR1l30J7lWxGGgqmXCCbqqp+BODZLc/ywd4PuGbqNZw5xrtjbHdDF0UZCR4jkhTyU+MwWx20mYevNEag1LR14zCXcPn4W1lXv467v7/bowAbkWwgbuNaKi6+mMoLL6L7p5/IvO46Rn+5ihH33IO+JLxeEL3sWyUnKh1yelD3MCIlzqspDOQggC6Lg87S+VDxrdwS9ACho8dGq9nmMSJJITtJT7xOTVnTASoYmrvQqoVPrW1kehw2h0RDRw9s/zdCOEi+7PcUvf0Wo954nbhZs2h5+hn2zplL/QMPYqutDW0wag1MPAP2fSn7MQKgpq2bLoud8VGokaQQEwwulIgkX6akOJ2a7CQ9mTWfg1oP404emsFMWCDXXdr8Jqv2r+LJTU+yoGQB10y9xudhuxs7/WZJKpOl5gC0D9e3y2aYs8eexvXTruejso94fmufg1ZyOulYuZI//udBznnnUWzVNeTcdhujV31B1g3Xo0kPsNhZoPy8FAypUBx4+Yratm6v/gUFpcTBz+lzAemAMicp4di+BIMQgqKMhANWYyhrkk3CvsqzK4u8KqMZtrwDOZMhWy5tHT9jBiOffIKSTz4m+aSTaH3zTfbO+yW1d9yBdf/+4Ac0/lRwWORw2AD4uU72n0WjRpJCTDC4UMLv/HVHGpUex7j2b6Hk+IBi2kNCnwjjT2Xv7o9Z/M1iJmVM4u6j7vZpFrHYHVS2mP22vsxPlSfEgeaABqht70ElICtJz6Ipizil+BT+ufGffFv1NR3LllF++hnU3PBb4uwWnj18IaM/W0H6pZegivf+EgsZuxV2fipP6iCiVGpau8n3sVIFGOvyAW3syYWsCXJz+QOEKh85DO4UZyUcuD6Gpi6f5lSg11RmrN4pt+ycMjhxUV9aSt5Df2b0iuWknX8+HUs/Yt/Jp1B7y61YKyoCH9DII2Q/1c5PAtp9T6OsWUSzjW1EBIMQ4iQhxC4hxF4hxK0etv9NCLHJ9bNbCNHmts3hti1qS6fyZhMJOjVZSd5twwCHJTSS46iH8acM6XjaS4/nxhQtcSoNfzvhb+jVvsdV1iTXhhlYCmMg+Qe0xtBNVpIerVqFEIK7j7iLM6pz6b7oGmpu/h2Sw0HeI39hw73P8MGImVjEEJYKKP8KLO1BmZGsdicNnT3k+dEYUuK15CTr5WCBiWfKvYM76sIdcUToTW7z4XwGKM5IoLpVrhRwIGF3yMXzSjxk/ruTl2pACEjc58olmXS21321+fnk3nkHpZ+vJP3ii+lYsYJ9p86n9tbFgWkQao1sXdi9PKDopJq2btITdCToo1fjNGzBIIRQA08CJwOHAAuFEIe47yNJ0s2SJE2TJGka8E/APYOrW9kmSdJp4Y4nVCpaTIzKSPDrrDzasRaAnuITh2wsTsnJ4savqNVo+HvKTHITcv0es9e1yvAVkQSQFq8lTqs+ICOT6tp7yE2JQ5IkOlevpv78S1j4ahVqB7y9MI/cf79FyoIF5KYn9O4/ZOz4QC5RUnJ8wIfUt/cgSVDgRzCA7BitaDbJ9ucDyJxUaTSTGq8l2VdDKmTN2uGUBmcPR5mqVrmch7c8GAW9Rk1usoG8pm/k/KGUAr/n1mZnk7P4Vkav/EwWEMuWse/kU6i7805sdX4E+7hToKcdKr/ze53Gjp4hr1zrj0hoDIcBeyVJKpMkyQq8BfhaZi0E3ozAdSNKRbMpoCbb49u/YZOzlGr70DmGXt7+Mt/UreGPUjrTKjcEdExdu/yiH5nu+6UkhHBFJh1YExrkF+us1jIqF15I9dXX4DSZyPvLwxjeepb/FDfzwLo/A5DrSrxS7jniOGyy2j/uZND41tTcUbQwf6YkkO+hocMCWePkpMYDJDqpymhmlB8zEtAbvXegZUCXNSmhqv4rko5LcVJk3gpj5gV1DU1WFjmLb6V05WekLVxI+wcfsm/eL6l/8EHsLV6KVJbOkWuqBWBOqu/oITc58OduKIiEYMgHqtz+X+36bBBCiFFAMbDK7WODEGK9EGKNEOKMCIwnaGwOJ1Wt3RT5iEgCoLOe9LatrHTMHLKyEpsaN/GPn/7BvFHzuGDM2dCwTe4l7YeGDgvxOjWJAaif+alxB5wpqefnn/nN0r9zzmsPYqurI/feeyn95GNSTjuNo0bO5uopV/NR2Ud8uPdDRqTIL976odIYKr6F7tagzEjQJxj8mZIAspP1NHb2yFFX406RI9DIcggvAAAgAElEQVR6hr8J0UD8haoqlByguQxKpNTA5DZPnKDbjhpn0IJBQZudLTcVWrGc5NNPo/W119l34jya/vkEjq4BfxddvCwcdn7SG4bujfr2oe914Y9ICAZPthdvd34B8J4kSe4F9QslSZoFXAj8XQjhse6AEGKRS4Csb2pqCm/EA6hu7cbhlHxGJAGyjRD43DljSARDW08bf/jqD4xIGME9R92DGHeSvGGP/5aYDS71M5C4/fwDKMnNWl1Dzf/9kfIzz6K0pZKys39D6YrlpJ1/HkLbZ85YNGURh+YeygM/PoAFWW0fMlPSXlfUWckJQR2m/E29lZJwJyfJgM0hyVn0o+eC5ICyr0IabqSwO5zUtHb7dTwDpCXoSInTHniCobmL9AQdqfE6v/seal1Lq5SIJTe05lgK2rw88u6/n5KPPyZh9myan3ySffPmYXztdSSbm09h/KnQUQN1m7yey+Zw0mL63xAM1YB7llcB4C3o9wIGmJEkSap1/S4DVgMevyVJkpZIkjRLkqRZWVlZ4Y65H4FGJLFrGVJqIdXaosHlt8NEkiTu/O5OjD1GHj3+UZJ0SZA5Vq7FtHuF3+MbOyxk+3GcK+SnxtFqtvmtNzSUODo6aHjkEcpOPpnOzz5DddFl/PrE27CfeyEqw+BJoVapeeiYhzCoDdz+/S2kJgyhKWnfKhh1pLzKC4KaNjNZSXqfeSQKijmsoaMHCg4FXVLA4YxDRV17D3an53LbnijOPPAik6pbuwPSeHA6KWlbw9fOKdR2RCanR19STME/HqfonbfRjx5Nw/33U7bgNDpWrpQ1w7EngVD5NCc1dVqQpL7nI1pEQjCsA8YIIYqFEDrkl/8gT5oQYhyQBvzg9lmaEELv+ncmcDSwIwJjCgrl4faV9YzVBGWrEeNOpTAjMeKC4b0977G6ejU3z7y5rzCeEHJd+PKv5BLfPmjoDNxh1ZvLEAWtQbLZML76GvtOnIfxxZdIPvVUSlcsp/GCyzHp4nrNRJ7Ijs/m/tn3s7t1N4acZUNjSuqog8YdAZfAcKe2rcdvDoNCjsuG3NDRI4fDlhwHe1f5NTMMJYFGJCkUH4BVVhs7LOQEskCq24TB2sKXjmkRn8txU6ZQ+K+XKXjmaVCrqbnht1Recgnd5fVy6Oqu5V6Pre+Qn+lo1UhSCFswSJJkB64HVgA/A+9IkrRdCHGfEMI9ymgh8JbUP5V1ArBeCLEZ+BJ4SJKkYRcM+41mEnRqMhJ8qJ9lq8HeA+NOojA9Tq6XFCEqOyp5ZN0jHDHiCC6aMCCbeuwv5etWfOP1eEmSXKakwDUGgOph9DNIkkTXV19RdvoZNDzwAPpDJlD87/fJe+jPaHNze81C/swwxxYcy8UTLsak/4qyro2RH+g+l/srBMFQE0Bym4LSN7yxw9U0afRcaN8PzXuCvm6k8NWHwRN5qQYaOi0HVJXVxs4esgOZB3tWIiH4yjllcMOeCCCEIOn44yn58ANy77kba1k5FeecQ+03GmwV28HkucVug2seBHQPQ0hE8hgkSfpUkqSxkiSVSpL0gOuzuyRJWuq2zz2SJN064LjvJUmaLEnSVNfvFyIxnmBp7OwhJ8WPfb7sKzmqoPAoRmUkUNXajTMCE8LutHPbN7ehVWm5/+j7UYkBX0nRbNAm9Po3PNHRY6fH5gxYY8gfZo3BUlZG1aKrqLrqanA6KXjqKQpffBHDhAm9+9QHMSFunHEjiao8mgyveC22FzL7VkFCNmRP9L+vG06nJAuGACKSoO8+G1wrRErnuq4fPXPSfqMZrVr41NrcyU4y4HBKnltkRgGr3Umr2dYrdH2yZwXkz8KkTh3SkFuh0ZB2wQWUrlhOxhWX07G+gn2fZNP86L04LYM7KTb8r2gM/ws0dVp8Fj0DZHPOqCNBo6MwPb43mSlcntvyHFuat3DXkXeRk5AzeAeNXi7Wt/szr2aGxg7lpRrYw5SdZECjEkMemeTo7KThoYcpO+10ujduJPvWWyhZ+iFJc04YJITr2nvITNQF1N/WoDEwL+smJHUn9695MHIDdjqh7Ev57x1kKfUWkxWr3RmwxqDXqEmL1/aaDkgbBRmjYW8UBUOLmYK0+EG9tr2hJINGu1WsQlOXPA6/vrauJqj5CTF2HvlpcVQbh36BpE5KIvsPf6Dko6Uk5jlpevNzyk6dT+fnn/erB1bfYUGnVpHuy3oxDMQEA/KD7fOl2tkgt/xz1cxRnFtVYT5QO407WbJlCfNL5vPLIh89ZkuOh45qMJZ53NzgMkcEZFtFbs84ItUwZBqD5HTS9sEH7Dv5FIz/+hepZ54hr5h+9SuEzvMDX9/eHZTDbXLWZKzNc1he8SkrKvw75wOifguYW0I2I4H3PgyeyEk29H53gKw1VHwbVgP5cAg0VFVBeQE3RmCBFAn6Fkh+5sG+LwAJxpzIyPT4YU3S0xUVU/DrQymcr0YVF0f19TdQdeUiLGXlgKwxZCfro1ZuWyEmGIBGfxpD+dfy7+JjgT7HYTgrJZvTxp3f3UmqIZVbDxtURaQ/rut68zMo6mcwIW5DlcvQ8/PPVF50MXW3Lkabn0fRO+8w4k9/QpOR4fO4uvYecpMDf6nmpRiwNp9AUeJ4HvzxQVp7WsMdep9/oeT4oA9VhGwgOQwKOcmG/i/V0b8Aezfs/z7o60eCuvZu8lMDf4YONI2h0TWOrEQ/91D+tVy7KHcqI9PihsTH4JOS40lIrKL4hcfIue02ujdtouz002n8619paW6PeqgqxAQDJosds9Xhu0ZS+VdgSIERUwF6hUhTGCulF7e+yE7jTu444g5S9J77A/eSORYSc+TVpAcUk1YwDqv81PiIagyOri7qH3yQ8rPPwVpZyYgHHqDozTeJmzwpoOPr2nsCiv9XkLULNafm3UiHtYOH1j4U4sjd2LcKcibJnbeCpLdzW4A+BpAXGL0+BoCio0Gti4o5ye5w0mKykhWIfd5Fr2DoOrAEg995UPGN7LtTqRiZHk+r2UZnzzCWoS+RLQ+i6lvSL72E0uXLSJk/n5bnnueqFxdzVO3WqPdLOegFg7La8WmXLP8Kio4BlWz/TovXoVaJkCfEntY9PLPlGU4uOpm5hXP9HyCE/CCXf+PRz9DYYSHJoCFeF3jRrYK0OBo6e8IugiZJEh3LllF28im0vvoaqeefR+myT0k9+yxEgHZ6s9VOe7ctKFOS4iB19OSyaPIiPi3/lK+qwkgQs5pg/xrZvxACtW09JLl6agdKTrKBJveoHl0CFB7Zp7kMI0aTFUmCrMTAbdvxOg2Jek1fZFWUaeroQQh8Rxe2VkLbfnk+0xcF109ADzWZYyFpRG9CoyYzk7w/P8ioN16nU2PglHf/TtXVV2Otrh6+MQ3goBcMveqnN8HQWiE/SG41+VUqQUaCLiQV2uF0cNd3d5GsS2bx4YsDP7DoGOiqh5a9gzY1dvYEnNymkJ8WhySFV1bCWlVF1ZWLqLn5d2iysih6521G3H036hQ/GtAA6gMMVXUnTqcmNV5LfXsPV0y+gjFpY7hvzX10WjuDunYvFd+B09YXHRQk1QGU2x5IdrIBpwQt7guMkuPkPAqTl5o7Q4SyyPFXXXgg2Un6A0pjyEjQ++zD0FvErkjugtin/Q9jZJUQ8vuk/Cs54MGF45DJXHfcTZSdeznd69ZTNn8BzUue6589PUwc9IKhyZ/6qZQpUOz8LrKS9DR3Bf8wvbXrLba1bOPWw24lzZAW+IHK9RV/hxsNHcGn0Bf05jIEb1+VbDaan11C2fwFdG/cSM5tt1H07jvETZ4c9LmgTzAEm+2Zm2ygrr0HrVrLfUfdR3N3M49teCykMVDxtWzGKTwipMNr2rqD8i9AX7BAPwf0qKPl3/t/8HDE0KHMg0x/0XkDyEzS03SAaAyNnQFk/1d8C/EZkDUekMcP0Dzcwq3keDnQoXF770cNHT04VGrsZy+k5JOPSZh9NE2PPUb5WWdj3jgEOTs+iAkGl33eq/O5/GvZvp81rt/HWUn6oDWGelM9/9z4T47OP5qTik4KbqDpJXLvWA8O6IYQyvQqq9tgG/Z0b95M+dnn0PS3v5F43HGUfPoJ6ZdeglCH3huhL7ktuBfriBQD9R3y+CdlTuLiCRfz3u732NTovRaNVyq/h/yZoA1uDAqBdG4biPKd9TNj5E0HjUEezzCiLHL+uzWGAJLbKr6Rha/LzKnM++EXDC4LRNnq3o96owuTDWhHjGDkE09Q8NSTOLq6qLzwIurv+xOOrsDag4bLQS8YGjstaFSCNE9FtyRJFgzFx8rqnxuZicELhofXPozD6eCOw+8IPhxNCCg+Rl7xuPkZJEmS6yQFmSmpvJQCvQenyUT9Aw9SccFCHO3tFDz1JAX/eBxtjofciyAJtQzAiNQ46tr6XqrXTbuOnPgc7ltzHzZnEOq3pQvqNsOoo4K6voLJIvtIgtYYXPdb7y4YNHq5dlIAdfsjSagaQ1aSvjdMNNr4rRc2wL8AkBKnRaMSwx9ZlZwn+xrcBEOv5uw2D5LmzKHko49Iu/hiWt98k7JT59Oza9eQD++gFwxNnRYyE/Wem7c37QRTo8eev1lJelpMloCzn7/c/yWf7/+cq6deTUGS/6YgHik6BkxN8rhctJltWB1OcoKIJgEwaNUk6TUBTYiub7+jbMFptL72GmkLZTU3aU7wsf7eqGvvJjVeS5wuOK1jRLKBFpOVHptcrDdeG8/iwxezp3UPr+14LfATVa8Dpx0KQxMMAQUweCAzUYdKMPjFOupoOadiGMtwN3fJZduD7RqWnWTAZHVgskSvICOAwynR3GXxnfU8wL8ALn9hom74NQaQF5z714BD/tvVewk7VycmkHv7bRS9/RaGiRPRFRYO+dBigqHL4l19LneZbYqPGbQpK1GPzSHR3u1/ZWq2mXlw7YOMTh3NpRMvDX2wyjjK+8xJSqhqKLHPGYk6WnyUM3C0t1O7+DaqrrgCodcz6vXXyL3rTtSJ/pugBEN9e09IJQAUn4R7VMzcwrkcP/J4nt78NLVd3or8DmD/D3LVy5GHBT0G6DNDZAYpGDRqFZmJ+v4+BpA1F8kJVWtDGk8oNHX6mAc+OFByGVpMFpySn1DVAf4FhczE0PyFYVN4JFi7ev0MDR09JBs0XhdIcVOmMPKpJ1HFhWbuDIaDXjA0dviYEPt/gOR8SB01aFNmEDHcz219jnpTPXcdeRdaVeDhjINIK4KUQtlR6qLPLhn8pM5M1NPsZUJ3rlpF2fwFtC9dSsZVV1H8wX+InzEjpGH7I9gcBgXFJ1E7oPz24sPkaK8/r/1zYCeq/B5yJ4MhtK58ykvFZ5ikF3KSDYNLqxQcCirNsJqTFM05WLIPkFwGZXHgU2sb4F9QkAVDFMavBDrsXwPIgiHa5bYVDnrB0NTlwy5Z9SOMPHyQfwHcnFZ+VkoV7RW8vP1lTis9jenZ4TUEAVx+hu96w9xCyXpWkDWG/uO3t7ZS84f/o/ra61BnZFD87jtk33wTKv3QVXusd/V6DpYRrizdgSG3eYl5XDXlKlZXreabau9VaQGwW2VTUohmJOjTGEJZcctJbgOeIV085M0YVgd0c1cA9cI8oNxztHMZmnrDzr3MAw/+BYWsJO8LpCElpQBSRvZGoNWHEF04VBzUgsHhlGjxZkpqq5K7LXkJXwwk61OSJB5aKzeXuXnmzREZM6OOgm5jbz6DYp8O5aU0UIXu/OILubHI8uVkXn89xe+8jeGQQyIzbi/02By0mKwhaQyK+clTJ7dLDrmEouQiHl73MFaHDzNB3Sa5rHmIjmeAFtffMJTCZ9nJBs/O21FHQc1PfvtwRIqmLguZSSGMv9eUFF0HtFJaxOsiz4N/QUGZB1HJNi48QtYYJImG9uCjC4eKg1owKHZJjy/Vqh/l3yMP93hsX2KMd8GwqmoV39V+x3XTriMzLjPs8fYbj2t8DR0WUuO1AXUNG0hmop5WsxWLsZXaW26h+rrr0WRmUvzeu2Rdf53XgneRRFlphqJCJ+g1JBs0Hju56dQ6bj3sVio7KnllxyveT6KsyguPDPr6Ci0m+TvQ+kqs8kJOkqG3Mms/Rh0tJ9zVrA95XIFitTtpM9v81xjyQFq8Do1K9CaKRgvlOfK6QKr8DuLSBvkXQA4CsDqcdHRHwYFeeAR01uForaSpyxL1ctsKB7Vg8BlNsn+N3Achx3Otn+Q4DTq1yqvG0GPv4S9r/8KYtDFcMP6CiI2ZjNHyA17VZ5cMNiJJITNRx/T6XVSccQbtH39C5rXXyFrC+MGTZ6hQXuqhTohsV1kJTxydfzRzRs5hyZYl1JvqPZ+g8ntXLarQ28U2d1lC8i8A5KZ40TwLD5cd4sNgTlLMiaFoDCqVCCl0O9I0dlpIifOxQKpaKy+qPJRpiWrNp5GyRaJr97c4nBI5/0s+BiHESUKIXUKIvUKIQaVChRC/EkI0CSE2uX6ucNt2mRBij+vnskiMJ1CafJXDqFoDBbNA7Tl8TwjhM8nt5e0vU2uqZfFhi9GoggsB9IkQ8gPuilhp6Aw+hwHkvISxbzzFAz88hyMugaK33ybrt78dFi3BnVATqxQyEnxHVv3fof+HU3Ly2HoPGdFOh7wACENbAPkeQnHcQl8PjUGlSQwpskPcS+HESNLsKgcRio8BXLkMURcMPsrCmI3QvNtr1FlmtJLcALIngD4Fe4W8APif0RiEEGrgSeBk4BBgoRDCk2H6bUmSprl+nncdmw7cDRwOHAbcLYQIok5EePRWYxy44u7pgIbtfl8YmYme6yU1mBp4cduLnDjqRA7NPTRi4+1l5GHyg2420hhC1nP3pk2UnXUWSZ9/zPujj6PlsSXETQquY1mkMLpWqxlBFG9zJzNR37/W0AAKkgr49aRfs6xi2eCM6MYdYGkPy78A8gslVMGQ09vi04ONvvAoqF4PjqGtldPUJV872HBbhewQqgBEmkZf4bbVLnNcwQEoGFRqGHkY+lp5oRdKdOFQEAmN4TBgryRJZZIkWYG3gNMDPPaXwEpJkoySJLUCK4Ega0WEjtdsz+p1chx5oWf/goK3ekmP//Q4dqed3838XcTG2g+Xn8G5fy2NnZaAHybJZqPx8cepuPAisNnRPf40z09aQHMUOzMqfz+PmecBkO5HYwD49cRfkx2XzcNrH8YpudnyK131iMIUDC1dVjJDFGw5A1t8ujPyMLk/Q8O2cIbnl/8JjcFX1nP1WhBqyPccbh31XIzCI0js2EMKXf87GgOQD1S5/b/a9dlAzhZCbBFCvCeEGBnksUNCU6eFJL2HhJKqH2X7boHv1b4nU9KWpi18VPYRlx5yaegZzv7ImwEqDd3lP8h2yQAeJktZORULL6Tl6WdIOe00ipd+SNYx8guxJRrJPS7CcdyCrGm0mW3YHd7Lh8dr47lp5k1sa9nGJ2Wf9G2oXivXn0oNPZPUanfS3m0jI8SXalq8Dq1a0ODppaQ8f1XrQh5fIIRaWVUhO0mP0eRWPnyYkSRJDjv3Ng+qfoTcSXJZcw+kxmlRq0R0NAbotUwcqt4T8nMUaSIhGDwV/Rn4hHwEFEmSNAX4HPhXEMfKOwqxSAixXgixvqmpKeTButPUaSHL02p7/xrImQj6JJ/HZyb2nxCSJPHwuofJjMvkyilXRmSMHtHFQ+6U3sQYX2UAJEmi9a23KD/rLGxVVeT/43HyHvoz6sREkg2+HejDgdFkDdlxC/ROJKPZt3A7teRUJmZM5O8//R2zzRUCWr1O9iOFgdGlrYRqClOpBNlJBs8aQ0qBXLe/eogFg2uBFEpkG0CWp/Lhw0hHtx2r3elZY3DYoXqD1+hC6Cuj3zycpbfdyZ+BXWiYrd8bcL/toSYSgqEaGOn2/wKgXy0CSZJaJElSnprngJmBHut2jiWSJM2SJGlWVlboESTuNHlq6emwyzbJkf7LL2cl6eUJ4bKTL69YzpamLfx2+m9J0HpenUSMkYdjaNyEBrtXU5K9pYXqa66l/p57iZ85k+KlS0meN693uxBynZhoagzNXVYyEkJfJSlCxd89qISKWw67hUZzIy9vf1luCN9a4Vcr9EdvOYwwVnrZyXrPCWJCyIKremhLY/gsCxMAyhyKljlJyWHweA+NO8Bm8ikYIIrZzwDaOCp1Y5ilGvrieIESCcGwDhgjhCgWQuiAC4Cl7jsIIUa4/fc04GfXv1cA84QQaS6n8zzXZ8OCxwnRsE1+kAKoy9+X/WzF6rDy+E+PMy5tHKeVnjYUw+3PyMNQO3qYIPZ7NCV1ffMNZaefgen778m5bTEjn1uCNid70H5RnRC4NIYQV9vQJxiMfvwMANOzp3NS0Um8tO0lGspd7TMjJhhCv4fsJH3/3s/uFBwmC7CuyGjJngi1HIaCEhUXLc3TaxAJ9OUj+fmeM6NcPnybagJj7XvkTPwDgLAFgyRJduB65Bf6z8A7kiRtF0LcJ4RQ3pC/FUJsF0JsBn4L/Mp1rBH4E7JwWQfc5/psWGjs6Bn8MPlJbHPHPf75zZ1vUtNVw+9n/R61KvTeBAHjGt9M1e5+L1an1UrDn/9M1ZWL0KSlUfTuu6RfeqnXNpuZUdYYWros4QmGICNKbpxxI3bJzpM7X5frEbn6eIeK8rcL58WanqD3LtiUF9oQJro1R0hjiFbDnkZfPc+r10Firl8/UmaiLjplMVxscI5Ghw0atkZtDO5EJI9BkqRPJUkaK0lSqSRJD7g+u0uSpKWufy+WJGmiJElTJUk6QZKknW7HvihJ0mjXz0uRGE8gmCx2TFbH4AlRvU6266aO9HygG8rLoLK1iWe3PMvs/NkcmRdeTHzApOTTpsvhcM1e9BpZEFn27aPivPMx/usV0i66iKJ338EwbqzP02REUWOwO5y0mm2kD4MpSaEgqYCF4xfygbmC3bnjZX9NGCh/u3CchhkJOlrNNs8l3POmyQJsCCutyhpD6MI5qgli+CmgV/UjjDzUY70zd5QIw6iUxQC+7ymS/1G9ISrXH8hBm/nstfBZzQa5k1cAKMd+VvsaJptp6MJTvbBPP5EZqt1IkkTbe+9Rfs652OvrKXjqKXLvvAOVwX+0kpwHEJ0J0Wq2ucYQ+kspxRVRMrAYoC+umnQFiU4nfwvjugotJisGrYqEIHtJuJORqMPh9FLCXRsnJ7oNkQO6x+ags8celsZg0KpJNmii1rCnsdNCnFZN4sBeEl2NshkuEO0/US+XxegZ/rIYFruDvZYUTNqMYSmBEggHrWBo9FQOw2wEY1nAkSoJeg3x8W1s7fiUM0efyZi0MUMxVK9sV40j09JC7Q3XUnfHncRNnUrxhx+SNOeEgM/RWycmChNCeZmHUnxOQaUSpCfoAvIxKKR01LGotZ1vbc2sqVsT8rVBrq6bkaAPviOfG8r9e83HKDhMLqjniPx3FAnnObhKk0TRx5Cd7OE7ULSsAARDNJPcWk02QGBMm9KXjBdlDlrB4LEcRs1P8u8ANQYAQ/ZKQM21066N4OgCo7o+gfIVWXSs+pqsm2+m8IXnPTqYfRHNCWHs7WMQ3kspI0EXXKOV6nUs7OwkLy6Lv67/a/+ktyBpNllDzhhWUO7fa7hnwaFyQETTz563h4HPsjBBkJXoJbJqGGjyVg6j6kdQ6wLyIynzIBpJbsrc686eDsZ98gI1yhz0gqHfA1WzHhByQ/YA2NGyA1vcBjIcvyA7PrgXcjhITictL7zAaUvfQkIw6qYTyLxqEUIdvDlDcfxGwwHdbFIct+GZdOSQ2yAmdPU69IZ0bpj5O3Yad7KsfFnI127utJAZhsYDfRqDdwe0S4MdAj9DuLWqFLKToxfV0+It5LnmJ9kMp/F/b0oBwagskFzfu1NZkNb+NOxjGMhBLRjUKtG/FEPNBrksr5/ENoXHf3ocDQmI9sBNN+FiNxqpuvpqGh95lDV5k4g/M414fWXI54umxqC8zMMxJYG84g7GlET1eiiYxSklpzAubRxPbHwCW4j1iFpM4YV6gptw9nYPaUWQkDUkZgavZWGCJJoag9FkJW3gM+R0QO3GgLX/3nkQBY1BeXZ1hTMBcUA4oA9awdDY2SM3Y1cyDSXJ9cII7EFaU7eG72u/Z7zhTFo6h+fPaF63jvIzzsS85kdSbrud+2ddQlvOFKjZ2NvRLViUCRGNrFWjyYpKQGqIdZIU0hOCCLntboOmnVBwKCqh4sYZN1LdVc27u98N+rpOpySvVsPUeJTFiVfhJoRsThqCRLe+qKrw7iEzSU+3zYHZOry+KqdTotXsIXu+ebdsfgtQMKTF61xlMYZfc1YWBBlpmfLC9ABwQB+0gkEuxeC2SmotlzujBfAgOSUnf9vwN0YkjGBW+nzazLbBjVYiiOR00vzMM1Re9itUcXEUvf0W1lPOBCEwZ00Faye07Anp3GnxWoSApmiYkrqsvRMyHDITdXRa7FjsDv87K2q6yzwzO382s3Jm8eyWZ/tKZQRIR48Nu1MKe7Wt06hIMmh8az0Fh8pd+yJsf25y9TFQQp5DJT0+OibJ9m4bTsmD1lnjWnXnBdanXO0KYoiOKcmCRiVIjtPIC9Pq9fJCNYoctIKhxWTt/zD1Op79RyStrFzJjpYdXDftOkYkJ7nONzQPlN1opOrKRTT9/XGSTz6ZovffxzBhQu8qw6n4Q2pCUz81ahXp8UHa6COE0RRecptCb72kQMxJ1S4/kmsBIITgppk3Yewx+u705gFldRmRe/BXJVapDFq3yfs+IRBucpuCMpda/dSsijTK38yjYNAny42tAiRaDYdaumRTmBBCfv90G+WFahQ5aAVD60DBUL0eNHGQ7bvHsd1p54mNTzA6dTTzS+b3Ok6H4oEyr18vm47WrSP33nvJe/QR1IlyDSblJRiXOx50SSELBpBfbNHxMYRXJ0khPZgkt5oNkDlGboTjYmrWVOYWzuXl7S/T2tMa8HUjFeoJuEJufXwHI6bJv2si65gMN3x5Pc4AACAASURBVLlNId2fn2SIUATRYMHwkys5MPBXXGa05oF7IUkl0CDKfoaDVjAM1hg2yA+Sl45tCh/t+4iKjgqun349apV6SGq5S04nzc89R+Vlv0LEGSh6+y3Szj+vX5y28hJJT4qTxx2GYFCS3IabFpO194USDsqLze+kliTZIenBvHDD9Bvotnfz4rYXA75uJMphKKQn+PkO4lIhvVQefwSRNYbwewAopqTWYRYMyt+s31y29cg1z4IIOwfZgR4NH4PR/V2UNQG08VH3MxyUgsHmcNLZY+/7MuxWqNvs90GyOqw8vflpJmVMYs7IOUBfDHpQUTE+cLS1UX3tdTT99TGSTjyRYpfpaCC9DqsEvTzu+m3yhAiBaJXFaOkKP9QTgvgOOmqhq8Fjw5bS1FLml8znzZ1v0mhuDOi6kXLcgmxK8jv+/BkHvMYQqXkQKEZPpqSGbeC0B+xfUMhyFdIb7ioAciFJ1+JCrZG1wygnuh2UgqF14MPUuB0cFr+C4d3d71JnquO3M37bu3qP5ITo3rqV8rPOpuu778i54w7y//YY6sREj/sau6zEadVyk6H8meC0hdzpKxqF9Kx2Ods6Eo1J0gPNxVBW217yVK6eejUOp4MlW5YEdN2WLgsqEXr3OXfSE3W0mv2UJsmbAZ210Fkf9vVALodhsjrC6oehkKTXoFWLKAgGDyHPivYcpMaQmajHanfSaRneyKqWLkv/76BgJtRviWql1YNSMAxyWPX2hPXueDbbzCzZsoTDcg/jiBF9JbkTdGp0apXfRjG+kCQJ4xtvUHnhRUhIFL3+GukXX+SzzEI/9VOZACGakzIT9XRa7PTYAojqiRBebcMhkKSXGw41+wsAqP1JLkiXO9nj5pFJIzl77Nm8v/t9qjqrPO7jTlOX/B1EorlKRoIOm0PyXZpE0XQipDX0fQfhC2ch5Jyg4RcMNhJ06v5Nhmo2QGIOJOcFdS4lyW04HdDKAqnfPMifCQ6r3EsiShyUgmGQxlC7EeIzIcV7RdU3dr6BscfIDdNv6PfCFsJVqyfEFbfTZKL2//5Iw31/Iv6oIyn597+JmzLF73Et7n0MkvPk0sIhvjAyo+A4jEQfAwWl4ZDf76B2I2RPkAvTeWHRlEWoVWqe2fyM3+u2dIWf3KbgN/sZZIEmVBHzM/SZYbQROV8g/bcjjdFkGeynqvlJfrkGWb8q0mbhQPC4QFI02gj7k4LhoBQMgzSG2o3yaszLg9Rp7eSlbS9xbMGxTMueNmh7WoIupDA9S1kZ5eefT8enn5J1042MfPpp1KmpAR3bT2MQrvDLMDQGGN6szz6nYeRerD5fSr2OZ9/lTrLjs1k4fiEf7fuIfW37fO7bHGYvCXf6BIOP70CXIDsnI1QyQS7eFhlTGMj3MOzOZ5O11/ENyAmMLXuC9i9AgMI5wrT01gtzu4fUURCXFhMMw00/KW01yZmwPl4Yr+54lQ5rB9dNu87j9vQEbdAPU8fyFVSccy6OFiOFzz9H5tVXe22m4wnjwKiq/BnyhOhuC2ocEHyzm0gQbq/kgWQk6n3nYrRWQHdrQC+M30z6DXGaOJ7a9JTP/VpM1ohpDH2F9Pw5oKfLK+IIOEiV3JtICrfhNiW1mgfMAyXPw0OAgT96czGG8R765oHbcyRc9dr+2wWDEOIkIcQuIcReIcStHrb/TgixQwixRQjxhRBilNs2hxBik+tn6cBjhwJl8qXGaeVoHsnpVTC09bTxyo5XOHHUiRyS4TnHwWcHrgFINhsNDz1MzU03oR8zhuL//JuEo44K/h5MAxxWYSRAZUahkF6vKSlCGkOmP41BWWUHUCAxzZDGRRMu4rPKz9hl9N6HN1J5GBBEEEPeDDkBqi30+lgKygswkhpDOL62UDB2Wftrnb0Zz4EVwnRH+TsMpznMa+n5vOmuftXR6XERtmAQQqiBJ4GTgUOAhUKIgW/QjcAsSZKmAO8Bf3Hb1i1J0jTXzzA0S5ZXGanxWjRqN3vtiMEmIoCXt7+M2Wbm2qney2qnxwemMdibmqj89a8xvvwyaRddxKhXX0Gbmxv0+M1WOz02Z/8JoYy/NhTBMPwduIwma18ZgAjgt15S7UZQ6/0mMCpcNvEykrRJPLnpSY/be2wOuiz2XodluPR2ogskZBUi4oA2mm2ICNSqUkhP0NFmtmF3DF15GHckSXLlI7n5SGp+grRiiE8P+nxxOjVxWvWwagweTUkgCwanHRq2D9tY3ImExnAYsFeSpDJJkqzAW8Dp7jtIkvSlJElKIZo1QEEErhsy/eyStRtlx23yiEH7NXc388bONzi5+GRGp3lPrU9P0NPRY8fmY0KYN2yg/Kyz6dm+g7xHHiH3zjsQutAmpMeHKT5d7msbgvpp0KpJ0A3/hEhXygBEgIxEP0XcajZC7iTQBPY3T9GncOnES/my6ku2Nw+enL05DBGIqgL5O4jXqf0vMLInyj0GIuBnMJospLo64EUCZdXb5qkT3RDQbXNgsQ9YINVuCsmMpDDcWo/RZEWtEqTEDQgA6HVAR6cEdyQEQz7gHttX7frMG5cD7gXwDUKI9UKINUKIM7wdJIRY5NpvfVNTU1gD7lcOw4dD8qVtL2FxWLhm6jU+z6esWDw5oCVJwvjKK3IBvPh4it5+i5QF88Mav8ekHpDvI8RaOmnDbB9uMVkiEqqq4LOvhNMp/12CdEhePOFiUvQpPLHpiUHbem3DETIlgaL1+NHaNDrImSQLujBpNdkGl6sOg+F23g5aIJmaoaPaq/YfCMPtJ2kxyYUkVQOFc3K+XGo9BAtAJIiEYPC03PDoGRNCXAzMAh5x+7hQkqRZwIXA34UQpZ6OlSRpiSRJsyRJmpWVlRXWgHvrt1u65PK8/8/eeYe3Vd2N/3O0reG948TZeyfsUcqmBUJbSKGlpewOCt1v+2vfDgotlL7lpYVCodDS8bZlj1LKagJlJiGELJM4O957SbZkS+f3x9WVZVuSr6R7ZafR53n8yL5LR74693u+O4ZgaPW18rddf+P8meczPW96wusVRJxWI1dKIZ9PCUX9yU9xf+hDTH/sURxz56Y1dnX8wNgwvYrlw07WJMn0SklPxy2MY4ppr4VAX9IrSbfNzZWLruT1+tfZ0jJygqr3QM8H67iF9FSmrFQEXSi9vJOO0RE9aZLpCqtj7oH6EK1MXTAUZDiyqmO0r1Blgh3QegiGOiA6AaAKaBh9kBDiTOC7wIVSysiySErZEH7dB6wHkvcaJUmkaFXTVkDG/CI9tP0hhkJDfH7p58e93nDP3uHVXuDgQQ5cehk9zz1HyVe+QtWvfonZo60BkJbxQxy7JKS0yihwZnZC6NHHIJqiRH0lxsl4TsRl8y+j0FHIve/fO2K7ngl6KppXq5UrFUHXviet9xsT2ZYm6kIlUxVWO0bfg0bVXzh+K894FDqtGTclxb0HlSuUdq6B5MrB64EegmEjMEcIMUMIYQMuBUZEFwkhVgC/QREKLVHbC4QQ9vDvxcBJgKHpflJKOlWNIY7jucXXwiO7HuHCWRcyNTd+0ptK4SiNoXf9evZffAlDzc1MfeABij9/fVKhqOMRswwADE+IFMxJE2Fb1dWUlEhjqN8MVhcUJ6+tOa1Orlx0JW82vMl7LcOrt47wvdZXMGiMbqtMPdAgmo7RoZ5pUpjoHhhAx2hTUsMWpdBgVOXcZCl02cdo/kbS3pegkGTlCiVismlbxsajkvbTSko5BNwAvADUAI9IKXcIIW4WQqhRRncAbuDRUWGpC4BNQoj3gXXAbVJKQwVDz8AQQyGpfJkatii2PE/ZiGN+u+23hGSI65Zep+maqgrd0TdA6933UPf5L2CdWsX0xx/HffJJun+Gdm8Am9mE2z4qosdZqLSBTEH9VJKTMjMhIhE9epqSEvkYGrdAxVIwpdaMZu28tRQ6CkdEKHWGnYa5Dn2iqiDcu9o7Tr0kgOJ5YHGk1ZtBXSDpKRgKMlxhdYxJtWFLWmYkUPyFfVqbPunAiJLbo4lEGmbenKTLt1pK+Q/gH6O2fT/q9zPjnPcmELtwjUGMKIcRw/Hc5G3isd2PsWb2Gqo82oKnClw2nIP9VP/iB7Rt30jemgsp/9GPMDnSL2cci45EET0Vy1MWDOqESLeb13jENYWlgdNmIcdqHmtKCgWVFdfKK1K/ttXJVYuv4uebfs67ze+yqmxVxGmoV1QVKP+PwFAIbyA4VuhHY7YoDujG91N+L3WBpKdgsJpN5I7XiU5H2r0BrGaBx26Jcjxfn9Y1o/2F5XnGzoPBYIju/sH4AQy5FeCpmBDBcNRlPqsPpWKbX3FKjjIjPbjtQaSUXLv0Ws3XDO3by69e+yVFOzdT9r3vUXHbbYYJBRjPLrlcSX5KsgXk8GrPeK2hI1YNfR1QV9wjaNsNg760V5Jr562lyFEUyYbuHB0/rwORqB4tztvK5dC4NeVe33ont6lkMqqnM1o46+B4hijtPwOfIeKnSuRrmyAH9FEnGNQJMcW3W9kQpTE0eZt4vPZxLppzEVPciSJuh+l54UX2f/JSXEN+nv7sd8etiqoHIwrojUb9PEmaGdSHXCYmRFukFIN+piSIE9WjPjDSCGEEyLHkcPWSq9nQtIGNTRvp8AV0f6hGzGFa2sRWLFd6fXckrucUj/bRZhidyKRgGNFsqyF9xzNktkVphxbNuWK5srjx9xo+nmiOOsEQuRk9NcqGqBVGRFtYMr62IINBWn5xJ/U33YRjzhzuvey/2Vk805AxjyahxqBOjCQdkxGNIRMTwiCNIWaoYeMWpSNW8Zy0r3/J3Esozinmvvfv090+D8MFBTPhgI6YVA3QGDLmfI7OhWlM3/EMmXWgx+w+N5rK5YDMuAP66BMM4Qefu2ObUmbbVQyM1BYq3YnruAe7ujh8/edpv/9+8teuZdof/4CltCxjK6WEgiGnQCkJkLTGMAEqtN4P1lj9ABq2KOWqU3Q8R+OwOLhq8VVsaNpAe7BGf1NYMg+lkvlKiY8UHdBjQj11IpMVVjt9g1EaQ/qOZ4j2MWRAMGjVGCAtf1IqHH2CwRvAbjFhbt46Qu3Uqi0M7NrN/kvW4n3nHcp/9CMqbv4RJpstYyq0f0iJ6En4ZapM3gGdaRVa74geiFH+XHU8p2lGiuaSuZdQ5ChiwPVPQx6qoFE4m61KiY8UNYa42fNpoobcZqI9ZqTzmQ4Zzyr5OVaEyMwCqaNPg0nVU6aU7MlwBvRRKRimOocQ7XsiKwyt2kLP889z4NJLkQMDTP/jHyj45NrIPjUPwOgJMTyhE3yZKldA16GkHNB54QmRiazVTp/+ET2g3ANfIDjcia59Dwx6dVlJqjgsDi6bdwVm1168ola36wI4bWbsFpP2h1LFcmUlmYIDutMbwGYx4bTpG3lT6LISCCqRVUYyGFQ6nxWoYeeQUgLjaCxmE3k5yZfRT4UObwCTCFd5TkTl8rRCk1PhqBQMK+11yh/hFYaqLVyz5JqY58hgkJb/+R/qv/o1HPPnM/3xx8hZPvJhUxAONfQZPCE02SVTiH9WJ0SmNAY9Q1VVxmg9OjmeR3Nq+QWEhty82/M3Xa8rhFAc6FqFc2XYAd25P+n3Usth6C+cw34SgxcY6j0uGpGoOn7nQy0UOjOT7Bm3TtJoVAd0wGv4mFSOOsHQ7g2wzByeSBXLaPY283jt46yZvSZmJFLEn/DAb8n/5Cepfvj3WEtLxxyXKRu9pgY36gRp2prUtWPa6A1AKd6mb6gnDDvQI5+hcQtYclLKeE6Ez28i0P4h9vVtYXOzvtUvC922xF3cokkjAUrvzHOVSHSbwQ/WEZqzTo5nlUzVS9J8DyqWhTOgtxs+JpWjTjB0egPMDe0DTyW4S3lo+0NxtYWB3bvZv/aTij/h5h9R8aMfxi2Vnan4Z0224ZyCcAZ0kpFJKbYoTRa9SzGojBHODVsUO7xZX19Gh3eQwc7jyLUWaOoNnQzJNH1SHNC2lMwMxt0DNbLK2N4eqkZS4LIq5jQdzYWZ8hdqFgyVmXdAH3WCocMbYHqgFiqW0epr5bHdj3HBrAvGZDn3vPgiBy69jFC/j+qHH6Zg7do4V1TQ3IErTTRnDVcsS/qLpEyIDCS4efXPAYBRuRihkKIx6WxGgnDEirSxdu5neKvxrTGVV9NBc4VVCJfgXpSSYzJSL0xnIpFVBpuSVI2k1OSF7sO63ueMac5ahbOnQinBnUE/w1ElGPxDQYL+PooHDkLlch7a/hBBGRyR5SxDIVruuov6G2/CPmc2Mx57HOfK8Z1amdMY/OGInnFMMRXLFNtzEj2gCzNQYTUYknQZtFodUaunY69SgVTHlaSK+lD6zMLLKLAXcN9W/bSGpMM9K8IZ0EkGPSSs0ZMGBaP9PAahzrPivnA+UpqJbdGomnMmAkk0CWchhgMNMsRRJRg6vYMsEIcQSNqKZvLo7kc5f+b5TPUoFVSDvb3UffFLtN97H3mf+DjVf/wj1rKx/oRYZG5CDGp3WEFSfga1WY+RE6Knf5CQ1L8UAwxHVnX4Bg1zPIMyoR1WE4VON1csuoI36t9gW6s+CUiFLhve6Miq8ahcDv5u6Nin+T0GgyF6B4YMuQcumxmbxWR4gpiqkXg6w931dHI8g6L1DAYlff443QB1IBSSSh6G1ntQsQxaamCw37AxRXNUCYYOb4AlJsXx/PueGgZDgxFtwb9vPwfWfpK+119X6h3dcgumJFpv5josWEzC8AmhZHtqcNymkBiTiVBDoxKrICqyyhtQ1G6zHUrm6f4+0Q1uLpt/Gfn2/DH9GlIl6Qz0FO6zpho9KSKEyIjm2eENkJdjxdz0PuRXK341nSjIQCBJ78AQwZDUbs6rXA4ymLEe0EedYFgs9tPkLOaRA3/nozM+SnVuNb3r1nFg7VqC3d1Me+jBlOodCSEyEs3QGdYYxsVVpGR2J2F/zkTZ5BHVbQ0g0lei8f2w41n/6Kdo+7zT6uSKRVfw7/p/x+wNnSxJ16wqXQAma1KCIRLAYIDGAJlx3nb4wqYwnR3PkJm6YcMLJI3fzzR6raTC0SUYfAEWm/bzcEkl/mCAaxZfTdu991L3xS9hmzaNGY89iuvYY1O+flGGJoTmh2qSDuhMhNwalXGrUui00dU3oHxuA8xIMPYeXDrvUnJtubr4GtSoHs1Vbi12KFuYkmAwImQYMiQY+gJMzfErrWx19C9AZuqGaUpUjSZvKuQUZiwDWhfBIIQ4VwixSwixRwjx7Rj77UKIv4X3vyOEmB617zvh7buEEOfoMZ54dHd3U2xu5HFzL+eXn4nt+3fRetcvyT3/fKr/789YKxPXSBqPggxEMyRVvK1imZL9q7EyY0QwGDgh1MlmRESMel1b7yHw9+j+wFAZfQ/cNjefXfhZ1h9eT017TVrXVleQmiqsqlQsU1aSGn1DqtCJ2wcgTTIhGDp9AZZYDil/6HyfiyIht8ZF6CWttQmRUqRhqqQtGIQQZuAe4DxgIXCZEGLhqMOuBjqllLOBO4Hbw+cuRGkFugg4F/h1+HqGYG7dyV/yXeR1BPnMXTvofeUVSr/1LSp/drsu/ROMbo+pOKySEQzJVWZMqh9AirQbbcZw2qjw7VL+MEgwxAq3/dSCT+GxevjN1t+kde2UzHkVy6C/Uwnb1ICaY2CkxmC489kbYIEMO9x11gwLIqYk43IxOlPR2iqXKw7oIWNzREAfjeFYYI+Ucp+UMgD8FVgz6pg1wMPh3x8DzhCKEX8N8FcppV9KuR/YE76eIZg632NLu5s7/gCirYupD9xP0VVX6lYWwOiVUneyET0Ru6S2VUYmIqs6wxE9OTrX6FEpcNmYHqhFmqxQOnp9kj5qjZ7Rwtlj83D5wst55dAr7OrYlfL185224cgqrSTpgFZXwkZEJYEyD3oHhhgMptZEaDzUtqQzBvdAblWkQrJeuO0WrGZhrMaQShBGxTIIDSrCwWD0EAxTgOilSl14W8xjwj2iu4EijefqgpQS/zsv8vVHJfbyKcx47FHcJ+nbj7nAZaO7f5AhgyZE0l8mtTKjxgeGx65EVhnrY0giRC8FCl1WFrCfUMkCJQFMZxKZwj694NO4rW7u33p/ytc3mwT5amSVVsoWgTBrtj93+gLkOixYzca4GCM1qwz6HqltSSv7dxuiFQohKDA4sqozXOU5x5rEAmn2WfD1XYbk5oxGj29GrOX2aGNnvGO0nKtcQIjrhBCbhBCbWltbkxyicrOdPUPsmWVh3qOPY5s6NelrjEeh04qUysreCFKK6KlcrvmBEYmsMtjHYJR/AaAgx8pi0wH6ixYbcn3VPh9LuOXZ8/jUgk/x0sGX2NO5J+X3KEhW87TmKOUxNC4A2pPxU6WA0b6qTm8AF/3k+Q4a9pA02iysFpJMylphd4On3LAxRaOHYKgDop+yVUBDvGOEEBYgD+jQeC4AUsr7pZSrpZSrS0pKUhroxX/ewEeffg+Ty5XS+eNR6E6iA1cKtKciGCqWQdsuCPg0HW50ZJVRxdtUppjaKBB9dOYvMuT640X0fGbBZ8ix5KTla0ipJEMSDmijymGoGO2ravcGWCgOIpCG+ZGMNgtrznqeIPQQDBuBOUKIGUIIG4oz+ZlRxzwDXBH+/WLgX1JJr30GuDQctTQDmANs0GFMMREWCxadC6pFY3RZjGGHVZKCQYagWVtlRkWFNs62mpTzPAXKvYp9v9mlf2IbRJd7jh3Rk+/I57L5l/HCgRfY1609GzmalLS2yuXgbYXepnEPNaochkomNIbFpuEKyUZgdE5SKkUMQzJEs7fZoBGNJG3BEPYZ3AC8ANQAj0gpdwghbhZCXBg+7EGgSAixB/ga8O3wuTuAR4CdwD+BL0kpjW1oYCBGd0GL+BiSsdEn6ZhUIkqMi3owqoCeSlFPDUPSxGHrDEOuryUH4LOLPovD4kjZ15CS1pZEAlSnwffA6HwYNR8p6CozzLRSZLApKZV78PLBlznvifN0SaQcD128T1LKf0gp50opZ0kpbw1v+76U8pnw7wNSykuklLOllMdKKfdFnXtr+Lx5Usrn9RjPRGF0I/FOb4Acqzm5iJ7cSnAWa/YzFLisdCYTEZMEgSGlRo+RGoOzYwe1cgptA8Y4ViOCIcGkLnQUcum8S3l+//Mc6D6Q9HukVMStbDEgxl0ASCkNK7mtonYkM0wweAMsFgeQ5frVRxpNgdNGl8/AQJIkTaohGeK+rfdR5alifuF8Q8YUzVGV+Ww0kfhnA22rSU9oIcKtATVqDE4bXb4AwZD+hfS6DE5uQ0osze+zQ840bLXX4Q3g0RDR89lFn8VmsvHAtgeSfo9Cp1LErTeZIm52t9KQaJz77A0ECQyFDLVvW8wm8p3Gtcfs6+lmtqjHPMW46Bx1nnUZEEgSL+Q5EesOraO2s5brll6H2WRYqleErGDQEbvFjNtuMVRjSCkpqWIZtNbA4MC4hxa4bISkUgVVb1IyhSVDbyPC28p+62zD7MNafSTFOcVcMu8Sntv3HId7tCWeqUTySVIxJ42jGXZoaQ2rA0b2NLB37MQsJEKHHs/xSPkeaKArrJFrFc5SSu7beh/VudWcO/1c3ccTi6xg0JlCA8M9O3waC+iNpmI5hIagZXzbpJGOQ6Nr9Kir5YacuYaaMbTeg6sWX4XFZElaayhK1UZfsQx6G6CvJe4hqv/ISOczGBvVU9it9mAwTmMoMtAsnGw5jPWH1/NBxwdcu+RaLCbjgmeiyQoGnTFyQnSmGk2ixnpr8DMYmZwUyQEw6qHUsAUQdHrmGyacO33a70FxTjEXz72YZ/c+S11vneb3SLnss4ZM985UMm5TIOlcjCQo931AtylP8Z8ZxJj+4TqSzAJJSsm9799LlbuKj878qO5jiUdWMOhMkctmWFvDlGOf86Yq9eo1RKyoE8KQlZLRD6XG96F4Li5ProHCeTCpe3DV4qswCRO/3fZbzeekHPasNqtJsABQv5tGFdBTMTIfZrq/lvqceYr/zCCK3MbNg2SE82t1r1HTUcN1S6/LmLYAWcGgO0atlPxDQfr8Q6nZ55NoDWisxjB+RE9aNG6BimWGRla1e/1JCbZSZymfmPsJnt7zNPV99ZrOUVeSSWs9jjwonJVwARAxYxjQpCcaw9pjDvZTLQ/T5lmg73VHYWRvEq2l51VtYYp7CufPOl/3cSQiKxh0Ro1/1ntCqA6rlCd0xTJo3jluZUajfQxaInpSorcZehuhcrlhkVX9gSADg6GkBdtVi69CCKFZa3DbLdjMptSKuI1TAqXDG8BmNuEyqIihitoeM6nIKg0MNmzDQojeAmMy21VsFhMeh8WQRZ7WBdK/6//NjvYdXLvkWqwmg/xyccgKBp0pdNkIDOnfHlM1AaQc0VO5XFNlRofVjNNmNiTk1tByGOoquWJ5JLJK75pVSXfdClPuKufjcz7OU3ueorGvcdzjlZpV1tTKPlcsh5468LbF3K3eA70qCsfDqBX3wMF3AfCXGpfDoFJkUPnwdg0LJCkl971/H1PcU7hw1oVxjzOKrGDQmYjjUOcHa9oNbiIZ0Nr8DEZoDJ0+AzNuw45nKpYalnmbjinsmiXXAGiOUFKaPqWoMUBcrcHoWlUqhQbZ6IMNW+iQbhxF1bpeNxZKIIn+VQC0hDy/Xv8629q2cc2Sa7Aa0J52PLKCQWeGw9z0/UKl3RKzYLpig9YQmVTkNsaBbrjGUDwH7B7DWjO29YVDPVMw55W7yvnEnE/w5J4nNWkNKYc9RyKT3ou5u90bSGn8yVJokMZgad7K9tAMCgx2noPSdtOoeZBocaFqCxWuCtbMGt3aJjNkBYPOGFUvKe0wwyRaAxoVUWJojZ6G9yJakVEag3q9VCN6VK1Bi68h5SJujjwonJlQYzCyTpKKIeVhBgdwdu1mu5yREa3HsHkwTsjzGw1vsLVt64RpC5AVDLqjPjT0Xmmo11Pr0KRExTJo3gHBxCaKQpfdmPhtn0GrbYRHEwAAIABJREFU1SjHMxgXWZVuRI+qNTyx5wka+mJWl4+QVhG3BBFoGTMlGXEPWnZikkNsC2VGMBS6jYmsShTyLKXk3i33Uu4q52OzP6br+yZDVjDozHC/WP01hrwcK5Z0InoqlkPQD60fJDysyG2jrc+v64RINaJHE1GOZ4hKTtLdlBTAahZ47KnHk2vVGtIq4la5XOn/7G0fsVkNeTY66xnAaTNjt5j0nQfh+7xNziDfafxKWo2s6hnQN7IqUcizqi1cu+TaCdMWICsYdCcSaqjzQ0mXlZ5aW6Yhtv1ZpdBlwz8UwqdjZJXqc0k2okcTUY5ngBybmRyr2QCNwU+Ry55WRE+0ryGR1pBWEbdIoMHI+xzJPM+Aj0EIoX8VgIYt+Mweeu2VhrUljcaI7OdECyQpJb/e8msqXZUTqi1AVjDoTmRCGBCVlLZgKJgB9vEd0CnX6klAp5EN6Bu3QNFssHsimwoNCDXUywyjJUIprSJuqgN61H3OVJ0kFd0FQ+P7HLLPiXRKNBpVgOoZmZQo5FmNRLp26cRqC5AVDIZgRL2k9j4dnIYmE1QuG1djMKIcgKHlMBq2jOn9W+Cy6q4xtPXp4yNRtYanap+Kmw2dlnDOyVei0EaFJmvpJaEnuvZNHvJD8w5qTbMoyIAZCaIiDHVc5A33bR8p3FRtYYp7yoRFIkWTlmAQQhQKIV4SQtSGXwtiHLNcCPGWEGKHEGKrEOKTUft+L4TYL4TYEv4xrlxiBjFitapoDDpMiMoVSpvPofjjK4w40PVbKaXUllQLvc1KRdFRlTaNcKCrDdz14Jol1yCE4IGtsbWGtM0YFcuhYaQDOhJVlQFTEui8QGreAaFB3g/NGvNQNQojotuGw85HzuV/1/+b7e3bJ9y3oJKuxvBt4BUp5RzglfDfo/EBn5VSLgLOBf5XCJEftf+bUsrl4R9tbcYmOXqX3pZSJl28LS6VKyAYSFiC24iSw+q1dO/FoK6KR9XmLzbMlKTPQ6ncVc4lcy/hqT1PxezXkHZpksrl0H0IfB2RTR1xVqtGUaBnT4aGzQBsGqw2xk8Vg0iEoQGCIVprk1Jyz5Z7lCzn2ZnPco5FuoJhDfBw+PeHgYtGHyCl3C2lrA3/3gC0ACVpvu+kRm8fgzcQJBAM6bNa1eCALnLrv1Jq7/NjMQny0gm3jcUox7OK3kl6A4PhiB4dV9tXL7kai8nCb7b+Zsy+SCG9dDQGGHGfO7wBTAL970Ecilw2egeGCAzp0B6z4T1kTiE7fHkZE2xqEIMxGsPw92jd4XXsbN/J55d9PuM1keKRrmAok1I2AoRfSxMdLIQ4FrABe6M23xo2Md0phMjMHTeYIpeNXv8Q/iF9onpUIaOLbTi/WinBnUAwOG0WHFZ9Qw3b+xTHrcmkc42eGI5ngCK3nf7BIL6APqGGw8lt+gmGUmcpl8y9hL/v+zsHew6O2Kd2A0ypLAZElcYYvs/t3gD5Thtmve9BHFQNt0sP7bn+PYIVKxgMGhTZFge9/YWdPkU45zqUzxCSIX695ddM80zj/JmZraCaiHEFgxDiZSHE9hg/SXlIhBAVwB+BK6WU6hLiO8B84BigEPivBOdfJ4TYJITY1NramsxbZ5zhiBJ9irjp6rgVQtEaxnNAu+yREhB60O71U2RENEkMxzPo7zhMuyRJHK5ecjVWk5X73r9vzL6UC+mBIvwLZ47UGPoyk9ymoptJMuCD1hq8RUuAzDnPQdE89dYYCpzDC6SXD77Mrs5dfH7Z5zPab2E8xhUMUsozpZSLY/w8DTSHH/jqgz9mT0EhRC7wHPA9KeXbUddulAp+4HfAsQnGcb+UcrWUcnVJyeS2ROkd7qm747ZyhVJldbA/7iF6T4i2vgDFejs9exoVx3PlyjG7isNCSC/hlk6dpEQU5xRz6fxL+cf+f7Cva9+IfYVOGx3p9JWoXAn1myN/ZirrWUW3vslN20CGaMtTSm1nynkOxmgM6j0IhoL8esuvmZE3g4/M+Ihu76EH6ZqSngGuCP9+BfD06AOEEDbgSeAPUspHR+1ThYpA8U9sT3M8kwK9oxl0N2NUrlB6QDfHd0DrPSHavf7Iw1o3wg5JpowVDJGQW501BiM6n125+EocZgf3bLlnxPaU6yWpTFmlCM7eJiBckuRI1BjC97nRNR/IrMag9zxo6xvuwvjCgRfY272XLy77ImaTsf0xkiVdwXAbcJYQohY4K/w3QojVQgg1538tcCrwuRhhqX8WQmwDtgHFwC1pjmdSMFxATJ/VaofuGkP4QZrAnFSoc4vS9j4DHkr1m0GYoXxsbX7VbKX3PTAia7jQUcjlCy/nxYMv8kHHcLmStB9KqsAMaw0TpjGk62NoeA88FTQGlWh4o9uSRqP0ZNDPpNrW56fEbWcoNMS979/L7PzZnD39bN2urxdpCQYpZbuU8gwp5Zzwa0d4+yYp5TXh3/8kpbRGhaRGwlKllKdLKZeETVOXSyn70v9IE4/uGoMvgMWUXo2eEeRWgqt0hJlhNHpOCF9gCF8gqL+PoWEzlC4Em3PMLlUItekk3PSok5SIKxZdgcfm4Z73hrWGwnTDPcuXKoKzYTPBkNQnez4J8nOsCKGD1la/GSpXRPUkyaTz2c7AYEi3IIb2sEn12b3PcqDnADesuAGTmHx5xpNvRP8B5DttCKFfZcmOsPqpW9ctDQ7oIrd+EyLSgF7P1baUygNjyoqYux1WJapHP1NS+nWSEpFry+Vziz7H+rr1bG3dCgxHVnlTbY9pc0LpAqjfTJcvgJQGZZ7HwWI2kZdjTU9jGOiB9lqoXEGHdxCrWeA2SDjHQs8ghsBQiO7+QfJdJu59/14WFy3m9Kmnp31dI8gKBgMwmwQFTv0SrNr6DLDPV66Atl3gj62kFeo4IVTHra7O5459MNCl2NHjUOTWT+vJhBnm0ws+TYG9gF+99ytg+P+VlgO9cgU0bKYzUsQwc4JBfb+05oFaPrxyJR3hqqRGtyWNpkBH7V/9Lh4eWkejt5Evr/hyRj9LMmQFg0EUOK26mZIUwaDzhK5cATKkRHzEQM/s54jGoKdtWNV2YkQkqRTp6CfRq05SIlxWF1cvuZq3G99mQ+MGij06RFZNWQn9nfQ1KalDmbTPg2IOS0tzVgMMwhpDJh3PoK9ZuL0vACLA2x2PsKpsFSdUnpD2NY0iKxgMoshl11FjCFBihMYAwxNvFMMTIv0Vd6Sqp54P1vp3wZKjmEriUOTWLxdDzzpJifjkvE9S6izlrs13URx+v9beNL5HYcEpw/6kidAY0nqoNrwH+dPAVaSY8zIYqgr6LpBa+/zYCt6id7CDG1fcOGm1BcgKBsMoTDfUMIyUktY+PyUenQWDpwxyq6BuU8zdqulKH1OSARpD/WalDEaCgmPFbv3MeXrWSUqEw+LgC8u+wNa2rdT2vQOkqTGULQKzHXuLUlPqiBMMYcczQKdvAjQGHUtv13W1Yytez6qSE1hZFl/TnQxkBYNBFOqUINbrV2rN6O5jAKhaDfWxBYOePXvb+wK4bGZybDrFageHFNtzAjMSKIKowxsgFEqvE50RdZIScdHsi6jOreaPu+4DQukJBrMVypeQ16GkCGUyogeGC0qm1A3Q2wZdByP3ua3Pn9E8DACP3YLVLHSZB680PoIw9/OVVTfpMDJjyQoGgyh0KhMi3YdSW2/YcesxYEJUrYauQ9A3NmFdz9aM7V5/xF6uC601MNSf0PEMiukqGJJ0p9IFLQoj6iQlwmKycMPyG9jTtYfcku3pm8OmrKSkr4Y8uwm7JbOJVIXh9pi9qURW1b+rvFYdw8BgkN6BIf0153HQq/FWq6+VLd3PEupdzvKyRTqNzjiygsEgCl02QjLF1oxRtKqCwQiNYcpq5TWGOUkIQbHbrospSffktvr4Gc/R6JXkZlSdpEScPf1s5hfOx1TwAi09vvQuVrkSW6if5c6YFWsMJa3otrqNSh5G5fKIcMy0YAB9env8ZutvCDFEnn/yFMpLRFYwGIRepatV+7whgqFimTLxEpiT9LCttvXpXECvYTM48pQicQnQxXmLcXWSEmESJm5aeRNBczt7/S+ld7GwAF1l2TfOgfqTVs2quo1QthBsrsgCaSIEQ1GaIbeHew7z+O7HKQydQqmjSseRGUdWMBiEXmFuwzkABkwIm1NxTsZxQOvViU73Anr1mxW78zhRHXprDJkO9Typ8iTyxUJaLc/RF0ijKEDRHHqFi6Vyj36D00hprvI/Ux/smgmFlPtcdcyI80vcDl3Hp4V0Hei/2vIrLCYL1p6zjZnHBpAVDAYx3JoxvYdSW58fkzDQjFF1jBISGBrbTEWPPIBQSEayhnUh4IOWneOakUC/QnpG1klKhBCCle5PI019/H7H71O/kMnEduYwd7BGt7FpRQ2zTlowtO0Gf0/E3NnaZ6CvbRzSiTDc0b6D5/c/z2cWfobO3hxKJmD8qZAVDAYReSjpoDEUugxsrlK1WpmAbbvH7NKj9HZX/yAhqaMZpuE9pTLs1OPGPbQgXJok3d7VRtdJSsTcgoUMdi/l4R0P0+pLrQ+JlJJNQ7Mo9+8Hf6/OI0xMQbgxUEvvQHInqubNsMbQ1jsxWpvynqk13pJS8otNv6DAXsAVCz+n7wLJYLKCwSD0KinR2hswVv1UHdAx/AyFrvS7oLVH7PM6fYbDSmy/+sBIhNkkKHTaaEtTuBldJykRxW4b/tazGQwNxmzmo4U+/xAbg7MxERqO9MkQJpOg2G1LXmOo26j4kYpmA9DaN0CB04rNkvlHlqopJtt46/X619nQtIHrl13P0JCdkNS5LIyBZAWDQdgtZvJyrMmvlEZhSJ2kaIpmgz1PmYijd+kg3CLOc71MYXUblTE7CzUdrvR+Tt/HkOnEMJVitx05WMyHK9fweO3j7OtO3oHc1hdgS0h5wMa6z0ZT4rGnIBg2KeHIJuUR1dprQJKnRoazn7V/hmAoyC/e/QVTPVNZO3ft8DyYoM+QLFnBYCClHjstPen7GAxdZZhMir2+buxKUg8H+nA5DB0mhJRweANUxW30N4YiV/oht5mokxQP9f92asllOCwO7tx0Z9LXaOkZoAcXvtxZcDjzgqHU44j4CDTh71P8SFFa4UQKBjXjPZl58MzeZ9jTtYebVt6E1Wwd1pyzpqQsZbkOWpJdKUUhpTReYwBlArbsgIB3xGY9Qm7bI+G2OjxYO/eDrw2mJiEYdCiLkak6SbFQ/29+v5NrllzD+rr1bGjckNQ1msPfwUDlakVjSCULOQ1K3ElqDA3vKQUeowVDJuZBHNQFktaQW9+gj7u33M2S4iWcXa004WmN5GEcBaYkIUShEOIlIURt+LUgznHBqO5tz0RtnyGEeCd8/t/CbUD/YyhNRYWOwhsIMjAYMl79rFqtTMSGLSM2q6ubdDJv28NRVfl61Lg5HH4gJiEYinUopNfe589InaRYROcBXL7gcipcFfx8088JybFRZPFo6VHMmbbq46C/A9r3GjLWeJR47LT1BQhqrQKg+rvCme1SStp6DSgkqZGycMitVu3/4R0P0+Jr4RurvxHxSxmaj2QA6WoM3wZekVLOAV4J/x2L/qjubRdGbb8duDN8fidwdZrjmVSU5CqCIaU6MUSVwzD6yxTJgB5pZijUQWNoC9vndYmqOrwBbB4oma/5lCKXjd6B5CNKVAYGg3gDwQkzJTmsZjx2C219ARwWBzeuvJGajhr+vu/vmq/R2uvHZjGRMzNc5jnDfoYSjz3SQU4TdZugcFbEj+QNBOkfDE6YKcltt+C0mTVp/83eZn6343ecXX32iEJ57X1+LCZBriOztapSJV3BsAZ4OPz7w8BFWk8Uiig9HXgslfOPBEo9DgLBEF2+1MpiGNLgJhauImUiqhE/6mabGVua9ZLa+3QM0avbAFWrIInG6aqNPtXPkOk6SbEo9tgjpoiPzPgIi4oWcdfmu+gf6td0fkuv0mdYlMwHe67yf8wg6gNdk/YspSK4RvkXoq+TaYQQlHrsNPeMH0jyy/d+yVBoiK+s+sqI7Ur2vw2TUWHnOpOuYCiTUjYChF9L4xznEEJsEkK8LYRQH/5FQJeUUo2FrAOmpDmeSUVp+Iucqp/B0Kzn0VSfAIfeHpHoJoSg2GVLq2+ybo5bfx8079CUvxBNuklu6nkTFZUEysJA1R5NwsQ3j/kmLb4Wfrf9d5rOb+4ZUMwhJpNinsmwA7o0GcHQdRD6mhXzZpiJFgwApbmOcU1JO9p38MzeZ7h84eVM9UwdsU/J/j8yzEigQTAIIV4WQmyP8bMmifeZJqVcDXwK+F8hxCwgluiMa3MRQlwXFi6bWltTS/TJNMOCIbWQ1dbwQ6k0ExNi2glh+3PtiM0lHntaIbftetVJqn837JDU7l+A9NtjGtJkKElG+0lWla3i3Onn8tD2h2jsaxz3/JZeP6WecCmJqccqgQYZTHQrSWaBdPAt5XXacHczQwtJaqQs10FzgnkgpeRnG35GoaOQ65ZcN2a/bvMgQ4wrGKSUZ0opF8f4eRpoFkJUAIRfY5ZvlFI2hF/3AeuBFUAbkC+EUNNJq4CGBOO4X0q5Wkq5uqSkJImPOHGU5iqTMdWQ1bZeP8LIchjRqBPx0FsjNpflOjSp0PHQrbKqav6oSlxqezSqGStVjUH97JEH6wSgCIaR4//aqq8B8It3fzHu+S09A5GaRVQdqwjY+tid+4ygOJmyGIfeVBLbShdGNrWGH8gTqTGUhUPP4/kLXzjwAptbNnPDihtw29xj9uteL8xg0jUlPQNcEf79CuDp0QcIIQqEEPbw78XAScBOqfyH1wEXJzr/SEZd6SdaaSSirc9PgdOGxZyBqOLCmeAqGV6xhSnPc9DUndr4BwaD9PqH9JkQhzdC8TzIiRn4Fpfh0iSpCeembuW8styJFQzd/YMEhobNfBXuCq5afBX/PPBP3m2On808MBikZ2BoWOtUBevhzPkZXHYLLptZo2B4G6YeH0lsA+WhajaJjHdvi6Ys10F/+Ps8Gt+gj59v+jkLChfw8dkfH7M/Y2HnOpLuE+c24CwhRC1wVvhvhBCrhRC/DR+zANgkhHgfRRDcJqXcGd73X8DXhBB7UHwOD6Y5nkmFy27BbbekrjEYndwWjRCK1hBDY+gZGKI/kHxUT8Rxm+6EUB2SSYSpqrjtFmwWU8oaQ1PPAMVu24SUYlBRC8eNFm5XLr6SMmcZt2+4nWAo9v1Rv3uq9kpOgbIaP/SmcQOOQUmUAz0u3jalZlf1CSM2t/YqndsMqxemgdJIyOrYRdKD2x+k2dfMd477DuYYgRF9/iH8Q6GjR2OQUrZLKc+QUs4Jv3aEt2+SUl4T/v1NKeUSKeWy8OuDUefvk1IeK6WcLaW8REqpT+f2SUQ6uQytvRleZUw7QXH+9Qxb9MrDD5SmFMxJ7XrFbrfuUvwf045P+tR0HeiK43bitAWIymUY1Vcix5LD11d/nZqOGh6vfTzmuap/aISfqvokOPQOBNNrIpUMSlmMcb5D6qJk2okjNhvS8zxJVFNi86hF3uHew/x+++/56MyPsqJ0Rcxzj7QcBshmPhtOOs7bjEcyqA/eQ29HNpXnhQVDCuakNr0ctwf+rbxWn5TS6UVue8qmpMbugYhwnCgSNbs5d/q5HFN+DHdtvouOgY4x+1WH7wgfyfSTYNCr9M3OEKUeDVUADr4FFgdUjnzATmQ5DBU1yW20v+2OjXdgNpn56sqvxj1X90KSGSArGAymNI2yGBm3S5YvBatrhDlJXS2nItwiGkO6eQwH34DcKiiYntLpRW5bylFJzT0DlOVNrGCI9DSI8RmEEHz3uO/iG/Rx1+a7xuxXH2Tqgw0YXpEffEP/wcZBUyG9Q28qyZaWkQuJjGvOMYgEkkR9htfqXmPd4XVct/Q6ylxlcc/NWD6SjmQFg8GUjhPNEA9fYAhfIJjZxiRmixI/HiUY0tIYwhMirQY3UsKB15VVboplr1MtZugfCtLhDUy8xuBJHHI7K38Wn1n4GZ6ofYL3W0dqAS29SsbtCMetpwyK5sCBzAqG3oEhBgbj+Kr8fdC4dYx/IRSStHsnXmNwh/2FqqDtH+rnJ+/8hJl5M7li4RUJz1XDzieqpEcqZAWDwZR6lJ4GfTGiGRKh2pMzvlKqPlFJJBvoBoYnRCo+hqbuATzh81OmrRa8rTD95JQvUZGXQ2uff0RUjxZUYTLRgsFpU0oyJHKgX7/sekqdpdz69q0jHNEtPcpDdUzG7fSTwgmNqZUKSZZxO7nVbQAZHJG/ANDdP8hgUE6Kh2r0AuOBrQ9Q31fP947/HlZz4jIXqimpYAKTJJMlKxgMJhLNkKQ5KVKNMdMTYtrxSpx7VD2dslxt5QBG09DVT0V+mg/VNP0LAJX5DqQcax8eD1UYTrQpCcYvBuiyuvjmMd+kpqOGv3zwl8j2lt6B2AmS1SeBvxuatxsx3DGMm+R28C0QpjGRZ8NVSSeBYAjPg33d+/jdjt9xwcwLOKZ8/IZRSti5FWsmws514sgZ6RGK6vRL1pSR0XIY0UxZDcI8Ip8h1VyGhu5+KvNz0hvPwTfAU6nkWaSIOoaGLm21hVTUz1wxKQTD+H6Sc6rP4aQpJ/HL934ZyYhu6fEPh6pGowraDJmTxq2XdOgtxcdl94zYPBnKYaio2c8/efsnkYgwLbQZ3YXRALKCwWBSLYsREQyZrt9udyuNe/a/FtlU5nGMCdPTQmPXABV5aQgGHfwLQGQMjUkKt2HH7WQQDPYx4aqjEULwveO+B8Ct79yKlDK+xpA3RXHmZ8gBHamXFEu4DQUUDbX6xDG7JkM5DJWyXAdt8g3eaXqHr6z8CkU5RZrOUwvoHUlkBYPBqKu1ZHMZJrL5OTNPU2oThf0MZXlKWYyQ1nr6KBm37d4AU9IxJbXvVQqqpeFfAMWUBIoGkwyN3QPkWM3kOtLwkehEsUdbX4kqTxVfWv4lXq17lef3v0inbzB+OY/qk+DgmyMKJxpFocuGEHHmQf0mGBoY41+AyaUxuHL6MZc8y9Li5Vw89+LxTwjT2J3mAmkCyAoGg8l1WLBbTEnbt9v6/OTlTEzzc2Z+WHEEHngdUJyvQyGZVCc0dXWelikp4l9ITzA4bRbycqzJm5J6BijPc0SarUwkxW47Hb4AQ8HxH+KfXvBpFhQu4LYNPwVT/8hQ1WiqT1ISB1s/0Hm0Y7GYTRS5bLEFw951in9hxqljdrX1Kb0kJoNwfqfnQRABrp7/LUxC27wcCoZo6hlgSrom1QyTFQwGI4SgNNeetPO5pXdg4lZJVceA1Qn71gPDppRkhJv6EE5rpXTgdXCXQ9Gs1K8RpjI/h8auJE1J3QPxH6oZpiJPcaBriQ6zmCz84MQf0OXvxF723HABvdFMD/sZMmROKnbHyX7et04pB56TP2ZXq9pLYoKF87pD69jW9RqBtjOwhso1n9fUM0AwJKkqyAqGLKMo9Yxfy300dZ39E/dlstiU1WRYMKSSy1AfFgwpr5SkVB5YafoXVCrzHDQk6WNo6pn4rGeVqQVOQPleaGFR0SJOK1+LLX8TDf73Yh+UXw15UyP32WhKcx1jNYb+TsVsOfPDMc+ZDOUwegO93PL2LUz3zCLQ/qGk5rJ6v6ZkBUOW0ZSmUBbjcIdvYlcZM09TCpp116dUL0ldnZflpTip22qhtzFt/4JKRb6DxiR8DFJKWnr8lE8S27D6XTjc4dN8zgr3WoL+Uh784A56Aj1jDxACZp8B+17NSN2kEneM7Of9/1bCo2edHvOcyVAO47YNt9E+0M4PTvgRYE6qWnJ9Z5oLpAkiKxgygCIYtK8yuvsH6RkYiqwSJ4SZpymv+1+l2G3DJJI3JZV47Ngt2ttwjmDPS8rrrDNSO38Ulfk5dPkG8QW0JRp2eAMEgiHKJ4kpqTI/ByG0awwAHd4QgcZL6PC38fONP4990OwzIdCbkTLcaoXVEVUA9q1T+nhHdWyLZqLLYfzr0L94Zu8zXL3kalZXLMPjSK5asqo5px22nWGygiEDlOY6EpcDGEVdp7IqnFo4gYKhbJHSn2HfeixmEyUee1KmpIbufirTif+vfUnpv1BQnfo1oqjMU3MZtH0G1XlePglyGABsFhPluQ4Od2rXGJp7Bii0zObKRVfy5J4nea3utbEHzTgVTBbY87KOo41NicfOYFCO7IG+91+KVhgje9g/FKTDF5gwP0/HQAc/eutHzC+cz+eXfh5Ac+9nlbpOHyUeOw5rigukCSIrGDJAJOtT40rjcIeyyphQjUEIRWvYtx6kpDzXkZQpqaErjeS2gFfxL8w5K7XzY6AmqWk1J02mHAaVqoKcpDSGll4/pbl2vrj8i8zOn8333/g+7f3tIw9y5Cl9tDMkGCAql6FjP3QeiGtGOtzRj5QwbQIWSFJKbnn7FnoDvdx68q2RshdlSRbFrO+aQF9hGmQFQwZItkKpqjFM+Bdq5mlKHkFLTVItPqWUNHYPpC4Y9r8GwYBi5tCJZLOfVSE4WTQGUBYK9ckIhh4/ZR4HNrON20+9nd5ALz948wdjCzrOPgOatkJvs84jHokqnFXzCvvWKa+zYjueD3V4Aaguchk6rlg8u+9ZXjr4El9c/kXmFsyNbE+21W19Z/8R51+ANAWDEKJQCPGSEKI2/Dqm76IQ4sNCiC1RPwNCiIvC+34vhNgftW95OuOZrJSOVydmFHWd/bjtFvKdiYtzGc6MDymv+9YnVRaju38QXyCYeimJ2peU8t8xMmFTRclH0G5Kau4ewCQmV0XMqoIcGrv7GdSQywDDGgPA3IK5fHXVV3m17lUe3f3oyANVAbz3X3oOdwzVRcrK/2Cbd/j9cqugaHbM4w+2+0aclykOdB/glrdvYVXZKq5cdOWIfaW52qslh0KShq6BIy4iCdLXGL4NvCKlnAO8Ev57BFLKdVLK5VLK5cDpgA94MeqQb6r7pZRb0hzPpCQiGDTtTQX4AAAbQ0lEQVSuNNSIpImO3SZ/qlKeec9LSbX4VB++Ka2UpFQEw8wPgUW/h7LVbKLEbddsSlJaetoz029bI1WFTkISTfkYQ8FQuFz1sHD+1IJPcWLlidyx8Q72de0bPrhsCbhKDTcnlbjtuGxmDrT7lKqu+19TtIU43/OD7T5cNjNFGaxKGggG+NZr38JmtnHbKbeNadVZ5nEQCIZG+kni0NrnJxAMUTWRJuEUSfdbvwZ4OPz7w8BF4xx/MfC8lFK7B+0/gAKnDatZ0KTVx9Dpm1jHczTzPwL7X2NqjpL1rMXPEEluS0UwtO2G7kO6mpFUKvJzNNdLaurxT4riedFEQlY1OKDb+gJIObKlp0mYuOWkW5QCcK9+nf6hsJA0mRRz0t5/GVqGWwhBdZGLA+1eqNuklFyJY0YCONjuZVqRK6MLpLs230VNRw03n3gz5a6xiWyqBqYlZDViEj7aTElAmZSyESD8WjrO8ZcCfxm17VYhxFYhxJ1CiLhLRCHEdUKITUKITa2tremNOsOYTIKphU4OqCp0AqSUHO7on1jHczTzL4DQEPN6lXafWuyrak2iylTqJNWGw1R1dDyrTMl3DNu3x0HJep5cgmE4yW18wVDfpRwz+h6UOEu47ZTb2Nu1l1vfvnV4x+wzlfIYDcYq7TOKXco8qHkGTNaEC4CDHT6qM7hAevXwq/xh5x+4dN6lnD4ttkM84i/UsMhTAwUm3FeYAuMKBiHEy0KI7TF+1iTzRkKICmAJ8ELU5u8A84FjgELgv+KdL6W8X0q5Wkq5uqSkJJm3nhTMKnGzt7Vv3OM6vAH6B4OT58s0ZRW4y5nSpJgZNAmGrgGsZpFaS8/aF6FkPuRPS/7ccajIU8piaLEPN3b3TyrHMyjOW7NJaIpM2tuiLEJmlbjH7Dtxyolct/Q6nt77NE/WPqlsnPlhQAznjxjE9GIndZ0+ZM0zirbgyIt5XDAkqevoz5h/4VDPIb7z7++woHBBwnLaZR7tyZ6R7P/JMpeTYFzBIKU8U0q5OMbP00Bz+IGvPvhbElxqLfCklDJinJNSNkoFP/A74Ni4Zx/hzCpxc6DdO24RtMPhST9pTEkmE8z/CK5D67ET0OSAbujqpyIvZ2zXsPHw9yl1+Q0wI4HyYO0fDNLdn9g+3B8I0jMwNOk0Bos5nMugIft5b2sfNrMprn37C8u+wHHlx3HrO7eyq2MXuIqUJjk1f9d72COoLnIxTx5AdB2CBRfEPa6pZ4BAMJSRiKT+oX6+uv6rCCH4xWm/wGGJf9/L8xxYTEKT9l/X2U+B04rTNvEFAJMlXVPSM4Da8PQK4OkEx17GKDNSlFARKP6JzLSTmgBmlrgYDMpxV3vqpJ9aOIlWGfPPRwx6OdO+U9NKqbG7P0Uz0gtKmOrcc1MY5PiozvDxzEmRUNVJJhhA+V5o0hha+5hR7MIcRzibTWZuO/U2cm253LTuJjoHOmHRx6B5G7Tu1nvYEWYUuzjXvAEpzDDvo3GPO9iuhqoau0CSUnLzWzdT21nLbafcRpWnKuHxNouJ6cUualvG1/7rO/uPSMczpC8YbgPOEkLUAmeF/0YIsVoI8Vv1ICHEdGAq8Oqo8/8shNgGbAOKgVvSHM+kRVXpxzMnHY7kMEyiL9T0U8Cex/nWdzWbkipTqTG07THwVOgaphqN6gwfL6qnaZJlPUdTVeDU5Hze09LH7NKxZqRoinOK+d8P/y+tvla+/urXGZx/PiBgxxM6jXYs1UVOzjNtoLFglaKlxOFQOFTV6OS2P9f8mb/v+ztfWP4FTqk6RdM5c0rd7NEiGLqOzBwGSFMwSCnbpZRnSCnnhF87wts3SSmviTrugJRyipQyNOr806WUS8KmqcullOP/t49QZpUoKvG4gqFDUT/d9kmkflpsMPccTg5t5EBLjGJsUQRDkqaeFJLb+jsVx/Oij4PJmPIBlRqzn9WoqskoGKYWOGnu8eMfih895B8KcqjDF/nOJWJpyVJ+eOIP2di0kds/eFgRytufUMKGDaCkfz+zTQ2850r8ED7Y4cNqFobWGHqt7jXu2HQHp089neuXXq/5vDllHg62exOWuJFSUtc5wYUw02DyBGn/h5PvtFHksrGvNbFtsm4yhapGs+B8PKEeCtrfJTAU30/S0qvUn69I1pS08xkIDcIS7Z2xkqXYbcdqFuOW397V3IvNYspoRIxW1AdNokS9g+0+QhJmjaMxqFww6wKuXHwlf9v1N/5SOQvadkHLTl3GOxoR9mG8FIpdNE/lULuPqgJnXFNYuuzq2MU3X/0m8wrm8dNTfqq58Q4oGkNIwv4EfoYOb4CBwdAR6XiGrGDIKFoik+o6J1GoajSzziBosnEmGxJ+hoZUq0lufwwKZ0LlinRGmRCTSVCe5xi3LEZNYw/zyjyTKrlNRUv5bdXMESsiKR43rbiJ06pO46dNr/KyywnbH09voPGoeZq9jkW83534O36g3WuYGanV18qXXvkSbpubu8+4G6c1ufeZU6b8XxP5GYZDVSfhXNbA5Pvm/wczq9TF3gQaQygkFYfVZHI8q9jd9FefwQXmt/igvi3uYepKNikfQ0+jUpd/ySW6NOVJhBqyGg8pJTsbelhQ4TF0HKmiapOJHNB7ww+smRpMSSpmk5mffehnLClZwn+VlrCp5nH9zUkd+6FpGwfLzuRwhy9uhJ6UkkPtPkMcz93+bq5/+Xp6Aj3cffrdlDrHS70ay4xiFyYBtc29cY9Ju1HVBJMVDBlkZrGbDm+Azji9k5t7lRC9SakxADnHX0Wx6EEmCGkc1hiSMCXteBKQsNg4M5JKZV7iJLeWXj/t3gALK3INH0sqlOU6sJpFQgf03tY+puTnJB0mmWPJ4Z7T76HSXsCNziF273423eGO5L0/gTDhm30+QyEZ9z50+gbp9Q/pHqrqG/TxxVe+yIHuA9z14btYULQgpevYLWamF7mobY6vMdQfoZ3bVLKCIYPMKlW+6PvaYn+hJnumpHnOGTSbSplXF9/MsL/NS77TiseRRAHAbY9C+RIomTv+sWkyp8xDfVc/Xb7Ywnlno+JcXzBJBYPZpDhkE2oMrd6ktIVo8h35/OaMe8mRkus23DyyplI6DAVg88Mw91xKq5SiefFs9JFQVR1NSYFggK+s+wrb27Zzx6l3cELlCWldb3apm9qW+BpDXacPj8NCXs4EF8JMkaxgyCCRkNWW2BNiOIdhcmoMmMxsLr6QRf4t0L435iGbD3WyYurYpu5xad8LDZsVM1IGWDFNGdt7h7pi7t/ZEBYMlZNTMIDalyG2xhAKSfa2jh+qmojKkoU8kDMfBvu56oUr9REONc+AtxWOuZrpxeEqq+2xP8OhDn2rqvqDfr62/mu81fgWN594M2dUp98VcG6ZhwPtvriBGEdyqCpkBUNGqSpwYjOb2BtHY1Ab9EzmL1TXvLUMSRPetx4cs6/bN8ju5j5WVY+pvh6ft+9VauYsWavjKOOzrCofs0mw+VBnzP01jT1UFeSQm4zGk2Gq8p2R78pomnoG8AWCSTmeYzHzhK/yUGMTDA1w1QtXpS8cNj4IBTNg5umRKqvxNQb9Fki+QR83vHIDr9a9yn8f/9+smZ1UJZ+4zClzEwxJpSBgDOqO4OQ2yAqGjGI2CaYXO+NrDJ0+ynIndxvA6umzeDm0Cuu2/4OhkYXE1IftqupCbRfztil256WfhNwKvYcaE5fdwvxyT1zBsLOxZ9L6F1SmFubQ1uePWQJdjRhLVzAw/WRmFi/ioW6lR/bn/vk5trVuS+1azTvg0Juw+iowmSJVVg/GeagebPdRnutIex70Bfr4wstfYEPTBm49+VbWztNv8aFqZLtjOKB7BgapbembtAEMWsgKhgwzq8TNvjjhnrUtfRPSxjAZFlbk8n/B07H5O+GDkU7oTQc7MJsEy6bGLow2hg33w1A/nHSjASONz8ppBWw51EUwNDLqxhcYYn+bl4WT2IwEip8E4P26seYwNSIpHVMSoESHnXADM1v28Pv51+K0Orn6xat59fDo4gUa2PQQmO2w4vLIpunFTqUvQwwOdXiZlqYZqcnbxOf++Tm2tm7lZ6f+jAtnXZjW9UYzq8SNEMR0QL+9t51gSHLy7GJd3zOTZAVDhplV4uZgx1jbZFufn611XZw0yb9M+U4be9zH0G4th3fuHxHS+O7BThZV5mqLhgl4FcEw7yNQMs/AEY9lZXU+3kCQXU0jV3u7mnqRcvI6nlVOml2M1SxY98HYmpV7WvvIdVgoduvQ3GbhReCpZPqWv/Gnj/yJGXkzuHHdjTyy6xHt1/D3wvt/hcWfAOewJjm9yBU3ZPVAu4/paQiG7W3buey5y6jvq+dXZ/yKc6afk/K14uGwmplW6IxZGuP1PW04bWZWTEvCpDrJyAqGDDOzxEUwJCMONpV1H7QgJZy5oGyCRqadeRV5/J/5Ijj8dkRrGAyG2HK4S7t/4b0/KWUwTrrJwJHGZtU05QE12pykRiRNdlOS227h2BmFrNs1VjDsbfEyq9StT3Mbiw2Ouw72v0pxdyO/O+d3nFh5Ij9++8f84M0fMDCkoenRO7+BQB8cc82IzdOLXDFDVrt8AVp7/SmHqj6//3mu/OeV2M12/njeHzl5yskpXUcLc0o9MSOTXt/TxnEzCrFZjtzH65E78iOUeMX0XqlpoTzXwaJJbsYAZUV9d8/JhIrnwYv/DUN+djb0MDAY0iYYgoPw5t0w9XiYdrzxAx7F1MIcit02Nh8cKRhqGnvwOCyTNlw4mg/PK2V3c9+Y6KS9rX3p+xeiWfU5pf/2G3fhtDq5+/S7uXbJtTxR+wSfff6zHO49HP/c7jr49//A/POhatWIXTPC4bQ1jSNrbz27tRGAD81NrueKb9DHD9/8Id967VssKFrAnz/yZ2YXxO4lrRdzytzsb/OO6MHd0NXPvlbvpNf8xyMrGDLMzBjF9AYGg7xW28oZC0onvs+zBhZU5OIPmTh8zHehcz9suJ93ww/Z1Vocz5seUtp3ToC2AEqLyRXTCsZqDA09LKjIPSLuwYfnKxm70eaknoFBWnr96fsXoskpgOOuV3JN9ryC2WTmxpU3cs8Z91DfV88lz17CI7seISRjhG2+8F2QITjnJ2N2LavKp8Rj568bRwqWx9+tY365J6kF0gcdH3Dpc5fyRO0TXLvkWh4850GKcuJXbtWLOaVuBoNyRNjt63uUqgCnzDnymolFkxUMGcbjsDIlP4dXd7VGOom9va8dXyB4RJiRgEi0xUbLKqWpzqt3sGvvfqbk54xfkbSlBl76Psw+C+adl4HRxmbltAIOtPto71Miq0IhyQdNvZPejKQys9hFdZGTf0UJhi3h3AxdNQaAD/0XFM+FZ76s9GkGTq06lUcueITFxYv58ds/5poXr+FwT9RDft962PkUnPJ1KKgec0mbxcSnjp3G+l2tkbDVPS29bDncxcWrqjQJ575AHz/b+DMu/ful9AX6uP/s+7lx5Y1YTZkJNZ5TqsyD6NIYr9e2UeKxM7dM53uQYbKCYQK4/kMzeWd/B//c3gQoZqQcq5kTZhm/ytGD6UUu7BaTYgY4+1ZkoI/jDt47vhlpyA+PXwM2N1z0a8PrIiVCHevm8MP0YIcPXyB4xAgGIQQfnlfKm3vbGRgMMhgMcetzNVTmOThR7++R1QEX3Qu9jYoWEGaKewoPnPUAPzzhh9S013DR0xdxx8Y76PS2wD++BQXT4cT4EWefPm4aFpPgj28dBOCxd+sxmwRrlk9JOJxgKMize5/lwqcu5E87/8TH5nyMJy58guMrMmuWnFXqwmISPPN+A1JKQiHJG3vaOHl28RGhdSYiLcEghLhECLFDCBESQsStoyuEOFcIsUsIsUcI8e2o7TOEEO8IIWqFEH8TQugQSjH5+dSx05hf7uGW52roDwR5paaZk+cUT+r8hWgsZhPzyj28uruVels1fcuu4uOhF7ky+FjiE1+5GZq3w5p7wJ188TI9WVqVhyWc6Cal5KWdipCe7BFJ0Xx4fin+oRBv7W3n/tf2sau5l5vXLMZlRC+PqtWK6e+9Pyp9M8IIIfjE3E/w1Jqn+OjMj/Knmj9x3mNnc89QE22nf1cRKnEozXVw3pIKHt10mJ6BQZ58r47T5pZQ4ondK3wwOMiTtU+y5uk1/L/X/x/FOcX8+SN/5gcn/IB8RxLZ9jrhtFn46llzeX57Ew++vp8Pmnpp9waOeP8CpK8xbAc+DrwW7wAhhBm4BzgPWAhcJoRYGN59O3CnlHIO0AlcneZ4jggsZhM/vHAR9V39fO2RLTR0D3Dmgol9UCbLNafMpK7Tx5n/8ypf77qYJ4MnsWLP3bD+9rEHh4Lw1q/hrbuV6JR5xrTuTAaH1czCylxe2N7EBXe/zk/+8QHzyz3MLT9yTADHzSgkx2rmD28d4Jev1HLe4nLOXGigOfK070DJfHjsKtg6MmS1zFXGzcd+hyctsznB28t9BXmc9d5P+dr6r/Fmw5sMhmL32f7cidX0+of4xiPv09zj5+JVI1trSin5oOMDfr7x55zz+Dl8/83v47Q4ufO0O/nr+X9lSckSwz6uFr542izOWVTGT5//gLteUVqiHsn5CyppLS2klDXAeGrTscAeKeW+8LF/BdYIIWqA04FPhY97GPghcG86YzpSOH5mERcsq+TZ9xuAYWfikcKFyypZMTWfH/99Jy/ubOYt2w2sWToV0/qfKDVxFpwP5Uuh6yA8+xVo3AKzzoCzfjzRQ4+wurqQh97YT3WRk59dvJSPrZiCdRL2YIiHw2rmpNnFvFzTjMdu4YcXLjL2DS12+PSjijnwiWth9wtw9i0w6IO+ZnjlZmYeeps7z7udA/PP4bHdj/HU3qd46eBLuKwujis/jhMqT2BOwRyqc6spchSxcloBi6fk8uLOZvKdVo6f7WZv1162tW1jS8sW3m1+lwM9B7CYLJwy5RQunnsxp0w5ZdKYaoQQ/PySZVx0zxu8sKOZ2aXuSdn5L1ky0T9yChAdelAHHAcUAV1SyqGo7YmNi/9h/L+PzOflnc3MLfdQ6jnyvkxTC53c/9nVvLa7lcFgCNO8cxTTwcYHlB8Vdxlc/JDStnOSTGiAL58+m1PmFHPKnOJJ2ZRHC2csKOXlmma+de48ynIz8B3Knwaf+we8fies/6nSYEnFbFPu8+KPMx34xjHf4Msrv8zrda/zRsMbvFH/Bv86/K/I4S6rC6fFSaBI4HIEETYfpz4ynNfgsXlYXrKcyxdczjnTz5kQc5EWPA4rv/nMaj52zxuccYQt8OIh5DjNOIQQLwPlMXZ9V0r5dPiY9cA3pJSbYpx/CXCO2gNaCPEZFC3iZuAtKeXs8PapwD+klDF1QyHEdcB1ANOmTVt18OBBTR9wsrNhfwe5ORbmlx85tu1x8bZB0zblJ+iHY68Dh8YyGVmSIjAU4tXdrZwxvxSTQW0w49KwRYk+cpcqwr9kPuTFX9tJKWnwNrC/ez8Hug9Q11fHwNAAA0N+ttV3smLKFGYUVFLqLGVB4QJm5s9MquXmRNPpDeCyWyZ1YpsQ4l0pZeK+qmgQDBrfbD3xBcMJwA+llOeE//5OeNdtQCtQLqUcGn1cIlavXi03bRrzVlmyZMmSJQFaBUMmRNtGYE44AskGXAo8IxWJtA5Q23ZdATydgfFkyZIlS5YEpBuu+jEhRB1wAvCcEOKF8PZKIcQ/AMI+hBuAF4Aa4BEp5Y7wJf4L+JoQYg+Kz2Fskf8sWbJkyZJRdDElZZqsKSlLlixZkmcymZKyZMmSJcsRRFYwZMmSJUuWEWQFQ5YsWbJkGUFWMGTJkiVLlhFkBUOWLFmyZBnBERmVJIRoBVJNfS4G2nQczpHC0fi5j8bPDEfn585+Zm1USynH7SJ0RAqGdBBCbNISrvWfxtH4uY/GzwxH5+fOfmZ9yZqSsmTJkiXLCLKCIUuWLFmyjOBoFAz3T/QAJoij8XMfjZ8Zjs7Pnf3MOnLU+RiyZMmSJUtijkaNIUuWLFmyJOCoEgxCiHOFELuEEHuEEN+e6PEYgRBiqhBinRCiRgixQwhxU3h7oRDiJSFEbfi1YKLHqjdCCLMQ4j0hxN/Df88QQrwT/sx/C5d9/49CCJEvhHhMCPFB+J6f8J9+r4UQXw1/t7cLIf4ihHD8J95rIcRDQogWIcT2qG0x761Q+GX42bZVCLEynfc+agSDEMIM3AOcBywELhNCLJzYURnCEPB1KeUC4HjgS+HP+W3gFSnlHOCV8N//adyEUtpd5Xb4/+2dTYiNURjHf0+GYiSxEDM0piYWykcWQpqGhY/JWJAFmUSWspBiIws7+ShlMz5GicTErKwoNuRrodhoiMsYymeUUf4W59zm3tvcaNzrzbnPr273Pee+9T6n/3vf//s+59z7cCSO+QOwLZOoqssx4Jqk2cBcwviT1drMGoCdwEJJc4BRhBovKWp9BlhZ0ldO21VAS3ztAE78zYFrxhgI5USfSuqTNAhcADoyjqniSOqX9CBufyFcKBoIY+2Ou3UD67KJsDqYWSOwBuiKbQPagHxR4hTHPAFYRqxjImlQ0kcS15pQq36smdUB44B+EtRa0k3gfUl3OW07gLMK3AYmmtnUkR67loyhAXhZ0M7FvmQxsyZgPnAHmCKpH4J5AGlULR/iKLAH+Bnbk4GPsVAUpKl3M6E87umYQusys3oS1lrSK+AQ8IJgCJ+A+6SvdZ5y2lb0+lZLxjBcpfRkl2SZ2XjgMrBL0ues46kmZtYOvJV0v7B7mF1T07sOWACckDQf+EpCaaPhiDn1DmAmMA2oJ6RRSklN699R0fO9lowhB0wvaDcCrzOKpaqY2WiCKZyT1BO7B/KPlvH9bVbxVYElwFoze05IEbYRniAmxnQDpKl3DshJuhPblwhGkbLWK4Bnkt5J+gH0AItJX+s85bSt6PWtlozhLtASVy+MIUxY9WYcU8WJufWTwBNJhws+6gU643YncPVfx1YtJO2V1CipiaDrdUmbgBvA+rhbUmMGkPQGeGlms2LXcuAxCWtNSCEtMrNx8VzPjzlprQsop20vsCWuTloEfMqnnEZCTf3AzcxWE+4kRwGnJB3MOKSKY2ZLgVvAI4by7fsI8wwXgRmEL9cGSaUTW/89ZtYK7JbUbmbNhCeIScBDYLOk71nGV2nMbB5hwn0M0AdsJdzwJau1mR0ANhJW4D0EthPy6UlpbWbngVbCv6gOAPuBKwyjbTTJ44RVTN+ArZLujfjYtWQMjuM4zu+ppVSS4ziO8we4MTiO4zhFuDE4juM4RbgxOI7jOEW4MTiO4zhFuDE4juM4RbgxOI7jOEW4MTiO4zhF/AJV1So3c5pfFwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tst_encoding = PositionalEncoding(20)\n",
    "res = tst_encoding(torch.arange(0,100).float())\n",
    "_, ax = plt.subplots(1,1)\n",
    "for i in range(1,5): ax.plot(res[:,i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.8415,  0.3877,  0.1578,  0.0631,  0.0251,  0.0100],\n",
       "        [ 0.9093,  0.7147,  0.3117,  0.1259,  0.0502,  0.0200],\n",
       "        [ 0.1411,  0.9300,  0.4578,  0.1882,  0.0753,  0.0300],\n",
       "        [-0.7568,  0.9998,  0.5923,  0.2497,  0.1003,  0.0400],\n",
       "        [-0.9589,  0.9132,  0.7121,  0.3103,  0.1253,  0.0500]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res[:6,:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerEmbedding(nn.Module):\n",
    "    \"Embedding + positional encoding + dropout\"\n",
    "    def __init__(self, vocab_sz, emb_sz, inp_p=0.):\n",
    "        super().__init__()\n",
    "        self.emb_sz = emb_sz\n",
    "        self.embed = embedding(vocab_sz, emb_sz)\n",
    "        self.pos_enc = PositionalEncoding(emb_sz)\n",
    "        self.drop = nn.Dropout(inp_p)\n",
    "    \n",
    "    def forward(self, inp): \n",
    "        pos = torch.arange(0, inp.size(1), device=inp.device).float()\n",
    "        return self.drop(self.embed(inp) * math.sqrt(self.emb_sz) + self.pos_enc(pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Feed forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The feed forward cell is easy: it's just two linear layers with a skip connection and a LayerNorm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def feed_forward(d_model, d_ff, ff_p=0., double_drop=True):\n",
    "    layers = [nn.Linear(d_model, d_ff), nn.ReLU()]\n",
    "    if double_drop: layers.append(nn.Dropout(ff_p))\n",
    "    return SequentialEx(*layers, nn.Linear(d_ff, d_model), nn.Dropout(ff_p), MergeLayer(), nn.LayerNorm(d_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-head attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Multi head attention](images/attention.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, n_heads, d_model, d_head=None, p=0., bias=True, scale=True):\n",
    "        super().__init__()\n",
    "        d_head = ifnone(d_head, d_model//n_heads)\n",
    "        self.n_heads,self.d_head,self.scale = n_heads,d_head,scale\n",
    "        self.q_wgt,self.k_wgt,self.v_wgt = [nn.Linear(\n",
    "            d_model, n_heads * d_head, bias=bias) for o in range(3)]\n",
    "        self.out = nn.Linear(n_heads * d_head, d_model, bias=bias)\n",
    "        self.drop_att,self.drop_res = nn.Dropout(p),nn.Dropout(p)\n",
    "        self.ln = nn.LayerNorm(d_model)\n",
    "        \n",
    "    def forward(self, q, kv, mask=None):\n",
    "        return self.ln(q + self.drop_res(self.out(self._apply_attention(q, kv, mask=mask))))\n",
    "    \n",
    "    def create_attn_mat(self, x, layer, bs):\n",
    "        return layer(x).view(bs, x.size(1), self.n_heads, self.d_head\n",
    "                            ).permute(0, 2, 1, 3)\n",
    "    \n",
    "    def _apply_attention(self, q, kv, mask=None):\n",
    "        bs,seq_len = q.size(0),q.size(1)\n",
    "        wq,wk,wv = map(lambda o: self.create_attn_mat(*o,bs),\n",
    "                       zip((q,kv,kv),(self.q_wgt,self.k_wgt,self.v_wgt)))\n",
    "        attn_score = wq @ wk.transpose(2,3)\n",
    "        if self.scale: attn_score /= math.sqrt(self.d_head)\n",
    "        if mask is not None: \n",
    "            attn_score = attn_score.float().masked_fill(mask, -float('inf')).type_as(attn_score)\n",
    "        attn_prob = self.drop_att(F.softmax(attn_score, dim=-1))\n",
    "        attn_vec = attn_prob @ wv\n",
    "        return attn_vec.permute(0, 2, 1, 3).contiguous().view(bs, seq_len, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The attention layer uses a mask to avoid paying attention to certain timesteps. The first thing is that we don't really want the network to pay attention to the padding, so we're going to mask it. The second thing is that since this model isn't recurrent, we need to mask (in the output) all the tokens we're not supposed to see yet (otherwise it would be cheating)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_output_mask(inp, pad_idx=1):\n",
    "    return torch.triu(inp.new_ones(inp.size(1),inp.size(1)), diagonal=1)[None,None].byte()\n",
    "#     return ((inp == pad_idx)[:,None,:,None].long() + torch.triu(inp.new_ones(inp.size(1),inp.size(1)), diagonal=1)[None,None] != 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Example of mask for the future tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.triu(torch.ones(10,10), diagonal=1).byte()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder and decoder blocks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to regroup these layers in the blocks we add in the model picture:\n",
    "\n",
    "![Transformer model](images/Transformer.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderBlock(nn.Module):\n",
    "    \"Encoder block of a Transformer model.\"\n",
    "    #Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        super().__init__()\n",
    "        self.mha = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff  = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "    \n",
    "    def forward(self, x, mask=None): return self.ff(self.mha(x, x, mask=mask))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderBlock(nn.Module):\n",
    "    \"Decoder block of a Transformer model.\"\n",
    "    #Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        super().__init__()\n",
    "        self.mha1 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.mha2 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff   = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "    \n",
    "    def forward(self, x, enc, mask_out=None): return self.ff(self.mha2(self.mha1(x, x, mask_out), enc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### The whole model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class Transformer(Module):\n",
    "    def __init__(self, inp_vsz, out_vsz, n_layers=6, n_heads=8, d_model=256, d_head=32, \n",
    "                 d_inner=1024, p=0.1, bias=True, scale=True, double_drop=True, pad_idx=1):\n",
    "        self.enc_emb = TransformerEmbedding(inp_vsz, d_model, p)\n",
    "        self.dec_emb = TransformerEmbedding(out_vsz, d_model, 0.)\n",
    "        args = (n_heads, d_model, d_head, d_inner, p, bias, scale, double_drop)\n",
    "        self.encoder = nn.ModuleList([EncoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.decoder = nn.ModuleList([DecoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.out = nn.Linear(d_model, out_vsz)\n",
    "        self.out.weight = self.dec_emb.embed.weight\n",
    "        self.pad_idx = pad_idx\n",
    "        \n",
    "    def forward(self, inp, out):\n",
    "        mask_out = get_output_mask(out, self.pad_idx)\n",
    "        enc,out = self.enc_emb(inp),self.dec_emb(out)\n",
    "        enc = compose(self.encoder)(enc)\n",
    "        out = compose(self.decoder)(out, enc, mask_out)\n",
    "        return self.out(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Bleu metric (see dedicated notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class NGram():\n",
    "    def __init__(self, ngram, max_n=5000): self.ngram,self.max_n = ngram,max_n\n",
    "    def __eq__(self, other):\n",
    "        if len(self.ngram) != len(other.ngram): return False\n",
    "        return np.all(np.array(self.ngram) == np.array(other.ngram))\n",
    "    def __hash__(self): return int(sum([o * self.max_n**i for i,o in enumerate(self.ngram)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_grams(x, n, max_n=5000):\n",
    "    return x if n==1 else [NGram(x[i:i+n], max_n=max_n) for i in range(len(x)-n+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_correct_ngrams(pred, targ, n, max_n=5000):\n",
    "    pred_grams,targ_grams = get_grams(pred, n, max_n=max_n),get_grams(targ, n, max_n=max_n)\n",
    "    pred_cnt,targ_cnt = Counter(pred_grams),Counter(targ_grams)\n",
    "    return sum([min(c, targ_cnt[g]) for g,c in pred_cnt.items()]),len(pred_grams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class CorpusBLEU(Callback):\n",
    "    def __init__(self, vocab_sz):\n",
    "        self.vocab_sz = vocab_sz\n",
    "        self.name = 'bleu'\n",
    "    \n",
    "    def on_epoch_begin(self, **kwargs):\n",
    "        self.pred_len,self.targ_len,self.corrects,self.counts = 0,0,[0]*4,[0]*4\n",
    "    \n",
    "    def on_batch_end(self, last_output, last_target, **kwargs):\n",
    "        last_output = last_output.argmax(dim=-1)\n",
    "        for pred,targ in zip(last_output.cpu().numpy(),last_target.cpu().numpy()):\n",
    "            self.pred_len += len(pred)\n",
    "            self.targ_len += len(targ)\n",
    "            for i in range(4):\n",
    "                c,t = get_correct_ngrams(pred, targ, i+1, max_n=self.vocab_sz)\n",
    "                self.corrects[i] += c\n",
    "                self.counts[i]   += t\n",
    "    \n",
    "    def on_epoch_end(self, last_metrics, **kwargs):\n",
    "        precs = [c/t for c,t in zip(self.corrects,self.counts)]\n",
    "        len_penalty = exp(1 - self.targ_len/self.pred_len) if self.pred_len < self.targ_len else 1\n",
    "        bleu = len_penalty * ((precs[0]*precs[1]*precs[2]*precs[3]) ** 0.25)\n",
    "        return add_metrics(last_metrics, bleu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_x_vocab,n_y_vocab = len(data.train_ds.x.vocab.itos), len(data.train_ds.y.vocab.itos)\n",
    "\n",
    "model = Transformer(n_x_vocab, n_y_vocab, d_model=256)\n",
    "learn = Learner(data, model, metrics=[accuracy, CorpusBLEU(n_y_vocab)], loss_func = CrossEntropyFlat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNX9//HXJ/tCEkIIEBIg7AiILAHFFde2at1oK7bWvda1VWuX77e/+u3XfmvVahdrq+KuVVt3XBH3HTAgm8q+ZoGEJSEJ2XN+f8ygaUwgCXNzZ5L38/GYBzN3zsy8D1k+Offce6455xAREWlNlN8BREQkfKlIiIhIm1QkRESkTSoSIiLSJhUJERFpk4qEiIi0SUVCRETapCIhIiJtUpEQEZE2xfgdoKP69u3rcnNz/Y4hIhJRFi1atN05l9nR10VckcjNzSU/P9/vGCIiEcXMNnXmddrdJCIibVKREBGRNqlIiIhIm1QkRESkTSoSIiLSJk+LhJn91MxWmNlnZnZNK8+bmd1hZmvNbJmZTfYyj4iIdIxnRcLMxgM/AqYBhwCnmtnIFs2+BYwM3i4F7vIqj4iIdJyX50kcBMx3zu0BMLN3gTOBW5u1OR14xAWuoTrfzHqbWZZzrjjUYVZtreClZUVEmREd9dUtJniLjo4iNsqIiY4iNtqIi44iJjoq8NzedtFRJMdHk5oQS0pCDL3iY4iJ1h47Eem+vCwSK4Dfm1kGUA2cDLQ8Cy4b2NLscUFw238UCTO7lMBIg8GDB3cqzNqSSu58ey2hvqR3Ymw0vRJiSEmIISUhlt6JsaQnxdI7KY7eSbGkJMSSEv/V8+nJsWQkx9MnOY64GBUYEQlvnhUJ59wXZnYL8DpQCSwFGlo0s9Ze2sp7zQZmA+Tl5XXq1/wpE7I4ZcIpOOdobHI0Bv9taHI0Njrqm5poaHQ0NDrqGptoaGqivsHR0NREkwtsr290VNbWU1HT8OWtsraeytoGdtc0sLu6nl176li/vZKyqnoqalt29z+lxMeQmhhLWvDWOymWfinx9EtNIDMlnn4p8fTtFU9Grzj6JMcRHxPdma6LiHSap8tyOOfuB+4HMLObCIwUmisABjV7nAMUeZnJzIiJti5Zj6ShsYmq2kYqgoVlbxHZUVXHzsrAv7ur6ykP3lZvq+DDtdvZXdN6cUlNiCEnPYlBfRIZ3CeJwX2SGN6vFyP69SKzVzxmrdVcEZHO8/R3pZn1c86VmNlg4CxgeosmLwBXmdm/gEOBci/mI/wSEx1FWlIUaUmxHXpdTX0jpRW1lFTUsL2yjp1VdeyorKWkopaCXdWsK63inVWl1DY0ffmatMRYhmcmk9U7kf4pCfRPjWdAWgKH5PRmSEaSCoiIdIrXf1A/E5yTqAeudM7tMrPLAJxzdwOvEJirWAvsAS70OE9ESIiNZlCfJAb1SWqzTVOTo6SilrUllawtqWBNSSXrS6v4omg3b+8uYU9d45dt+/aKY/LgdKYMSWfswFRGD0jRyENE2sVcqGdyPZaXl+e0Cuz+VdY2sGXnHj7dXEb+pp0s3rSLjTv2fPl8n+Q4xmalcsakbE6dkEVCrOY7RLozM1vknMvr8OtUJHqOnVV1rNy6m1VbK1i1tYKFG3ayfnsV6UmxnD11MD84dPA+Ry8iErlUJKTDnHN8vG4Hj3y8iXmfb8UBU3P7cOqELL45fgD9UhL8jigiIaIiIQekqKyap/ILeHl5Eau3VWIWKBhTc9MZPzCN8dlp5KQnah5DJEKpSEjIrN5WwcvLinn9822s2lZBY1PgeyQjOY6LjxrKRUcM1RyGSIRRkRBP1NQ3smprBcsLy3lrZQlvrSxhYFoC139jNGdMzCYqSiMLkUigIiFd4uN1O7jplS9YXljO+OxUfnPKWA4dluF3LBHZj84WCS0eJB0yfXgGc648gr/OmsjOyjrOnj2fKx9fTGFZtd/RRMQDKhLSYVFRxukTs3nzZzP46fEjeePzbRx/+zv85Y3VlFfX+x1PREJIu5vkgBWWVXPTK1/w8rJikuKimTk5h/MPH8KIfil+RxORIM1JiO9WFJbz8EcbmbO0iLqGJo4a2ZefHj+SvNw+fkcT6fFUJCRs7Kis5V+fbOGhjzZSWlHLiWP784tvjGZkf40sRPyiIiFhZ09dAw9+uJG731lHVV0DMyfncPKELKYMSSc1oWMr44rIgVGRkLC1s6qOv7+9lkc/3kRdYxNmMGZAKtNy0/ne1EGMG5jmd0SRbk9FQsLenroGlmwu45ONu8jftJP8jbuorm/kyBF9ufToYRw1sq+W/RDxiIqERJzy6noeX7CZBz/cQElFLWMGpHDhEbmcdkg2iXFa9kMklFQkJGLVNjTywpIi7nt/A6u2VZCWGMv38nI497AhDMlI9jueSLegIiERzznHwg07eeTjTcz9bCtNznHpUcO47qRRxMdoZCFyIDpbJLy+fKlIu5kZhw7L4NBhGWwtr+Gvb67hnvfW896a7fzl7ImMHqBDaEW6mpblkLA0IC2BP5x1MPedl0dpRQ3fvvMD7v9gA01NkTXyFYl0KhIS1k4Y25+51xzN0SP78ruXPufMuz5iWUGZ37FEegwVCQl7fXvFc+95efz57EMoKqvm9L9/yH89u5xdVXV+RxPp9jQnIRHBzDhzUg4nHNSfv7yxhoc+2sirK4o5f3ou5x42hMyUeL8jinRLOrpJItKqrRXcMnclb60sIS46itMmDuTCI3J19rZIG3QIrPRI60sreeijjTyVX0B1fSPDMpM54aD+HDemH1OGpBMbrT2qIqAiIT1c+Z56nl9SyBtfbGP++h3UNzrSEmOZOTmHCw7PZXBGkt8RRXylIiESVFnbwAdrSnl5+VZeXV5Mo3OccFB/LjpiKIcN66P1oaRHUpEQacXW8hr+OX8Tjy/czM6qOsZmpXLxkUP59iEDiYvRrijpOVQkRPahpr6ROUsKuf+DDazeVklmSjznTx/ChUcMJTleB/lJ96ciIdIOzjneX7Od+z/YwLurS8lJT+TWmRM4fERfv6OJeKqzRULjbelRzIyjR2Xy8EXTePqy6cRGR/H9+xbw388tp6Km3u94ImFHRUJ6rLzcPrz606P40VFDeWLhZr75l/dZsH6H37FEwoqnRcLMrjWzz8xshZk9YWYJLZ6/wMxKzWxJ8HaJl3lEWkqIjebXp4zlmcsPJy4minPunc+db63RQoIiQZ4VCTPLBn4C5DnnxgPRwKxWmv7bOTcxeLvPqzwi+zJ5cDovXn0kp04YyG3zVnP+gwspraj1O5aI77ze3RQDJJpZDJAEFHn8eSKd1is+hr/OmsgfzjqYhRt2cvId7/Pi0iIi7eAOkVDyrEg45wqB24DNQDFQ7pyb10rTmWa2zMyeNrNBXuURaQ8z45xpg5lz1RFkJMdx9ROfMvOuj1i0aZff0UR84eXupnTgdGAoMBBINrNzWzR7Ech1zk0A3gAebuO9LjWzfDPLLy0t9SqyyJfGDEjl5Z8cxS0zD2bLrmpm3vURVz2+mJKKGr+jiXQpz86TMLPvAt90zl0cfHwecJhz7oo22kcDO51z+1zGU+dJSFerqm3gnnfXcc9760lJiOWOWRN1XoVEnHA8T2IzcJiZJVlgsZzjgS+aNzCzrGYPT2v5vEg4SI6P4bqTRvPCVUeSlhjDufcv4K9vrKFRR0BJD+DlnMQC4GlgMbA8+FmzzexGMzst2OwnwUNklxI4EuoCr/KIHKjRA1J44aojOX1iNn9+YzXnP7CQ7ZU6Akq6Ny3LIdJBzjmezN/CDXM+Iz0pjr//YDJThqT7HUtkn8Jxd5NIt2RmnD11MM9eETgBb9bsj3nk4406VFa6JRUJkU4aNzCNF686kqNGZnLDnM+47sml1NQ3+h1LJKRUJEQOQFpSLPedl8fPThzF80sKufKxxTQ0NvkdSyRkVCREDlBUlHH18SO58fTxvLmyhP9+brl2PUm3oautiITIDw8bQmlFLXe8uYbMlHh+/o0xfkcSOWAqEiIhdO0JIymtqOXvb6+jb694LjxiqN+RRA6IioRICJkZ/3fGeHZU1nLjS5+TlhjLWZNz/I4l0mmakxAJsego445zJjF9WAbXP7WU5z8t9DuSSKepSIh4ICE2mvvPn8qhQzO47sklzFmiQiGRSUVCxCOJcdHcf0EeU3P7cO2/l/DiUl1ORSKPioSIh5LiYnjwwqnkDenDNf9ewmufbfU7kkiHqEiIeGxvoTg4O42rn/iUhRt2+h1JpN1UJES6QHJ8DA9cMJWc9EQuefgTVm2t8DuSSLuoSIh0kT7JcTxy0TQS46I5/4GFFJZV+x1JZL9UJES6UE56Eg9fNI2qugbOu38Bu6rq/I4ksk8qEiJdbMyAVO47L48tO6u54rHF1GtBQAljKhIiPjh0WAY3nXUwH6/fwf+99LnfcUTapGU5RHzynSk5rNq6m3vf38CYrFTOmTbY70giX6ORhIiPfvWtgzhmVCY3zFmhQ2MlLKlIiPho7zpPg9KTuPyfi3TEk4QdFQkRn6UlxnLv+XnUNTRx5WOLqWvQRLaEDxUJkTAwPLMXt3xnAku2lHHzqyv9jiPyJRUJkTBx8sFZXHB4Lg98uIG5K7TGk4QHFQmRMPLfJx/EITlp/PzppWzaUeV3HBEVCZFwEhcTxZ3fn4wBVz6+mJr6Rr8jSQ+nIiESZgb1SeL2701kReFu7n1vvd9xpIdTkRAJQyeO7c+JY/sz+/31lO3R+k7iHxUJkTD1s5NGUVnbwD0aTYiPVCREwtSYAamcdshAHvxwAyUVNX7HkR5KRUIkjF17wijqGx3/eHud31Gkh1KREAljuX2T+V5eDo8t2ETBrj1+x5EeSEVCJMxdfdxIDOOON9f4HUV6IE+LhJlda2afmdkKM3vCzBJaPB9vZv82s7VmtsDMcr3MIxKJBvZO5NzDhvD0ogLWlVb6HUd6GM+KhJllAz8B8pxz44FoYFaLZhcDu5xzI4A/A7d4lUckkl1x7HASYqO5fd4qv6NID+P17qYYINHMYoAkoKjF86cDDwfvPw0cb2bmcSaRiNO3VzyXHDWMV5ZvZemWMr/jSA/iWZFwzhUCtwGbgWKg3Dk3r0WzbGBLsH0DUA5keJVJJJL96Kih9EmO45a5K3HO+R1HeggvdzelExgpDAUGAslmdm7LZq289Gvf/WZ2qZnlm1l+aWlp6MOKRICUhFiuOnYEH63bwftrtvsdR3oIL3c3nQBscM6VOufqgWeBw1u0KQAGAQR3SaUBX7uGo3NutnMuzzmXl5mZ6WFkkfD2g8MGk5OeyC1zV9LUpNGEeM/LIrEZOMzMkoLzDMcDX7Ro8wJwfvD+d4C3nMbRIm2Kj4nmuhNH8VnRbl5aXux3HOkBvJyTWEBgMnoxsDz4WbPN7EYzOy3Y7H4gw8zWAtcBv/Iqj0h3cfrEbMYMSOH2eat0qVPxnEXaH+55eXkuPz/f7xgivnp7ZQkXPvQJvzl1LBcfOdTvOBIBzGyRcy6vo6/TGdciEWjG6EyOHpXJn19fTcluLf4n3lGREIlAZsaNp42jrrGJ/3u55VSfSOioSIhEqNy+yVx2zHBeWFrER2t1SKx4o11FwsyGm1l88P4MM/uJmfX2NpqI7M8VM4YzuE8Sv5mzQpPY4on2jiSeARrNbASBI5KGAo97lkpE2iUhNpr/PW0c60qruO8DXcFOQq+9RaIpuGzGmcBfnHPXAlnexRKR9jp2TD++Ma4/d7y5RteckJBrb5GoN7NzCJz49lJwW6w3kUSko2749jgamxwPfLDR7yjSzbS3SFwITAd+75zbYGZDgX96F0tEOiK7dyInjRvAM4sLqKlv9DuOdCPtKhLOuc+dcz9xzj0RXLgvxTl3s8fZRKQDvj9tMOXV9by6Qst1SOi09+imd8ws1cz6AEuBB83sT95GE5GOmD4sg9yMJB5fsNnvKNKNtHd3U5pzbjdwFvCgc24KgVVeRSRMREUZ50wbzCcbd7F6W4XfcaSbaG+RiDGzLOB7fDVxLSJhZuaUHGKjjScWajQhodHeInEj8Bqwzjn3iZkNA9Z4F0tEOqNvr3i+MW4AzyzSBLaERnsnrp9yzk1wzl0efLzeOTfT22gi0hnfnzaY3TUNvKLrTUgItHfiOsfMnjOzEjPbZmbPmFmO1+FEpOOmD89gaN9kTWBLSLR3d9ODBK4iNxDIBl4MbhORMGNmnDNtEPmbNIEtB669RSLTOfegc64heHsI0MWmRcLUzMk5xEQZT+Vv8TuKRLj2FontZnaumUUHb+cCO7wMJiKdl9ErnmPH9OP5JUU0NGp1WOm89haJiwgc/roVKAa+Q2CpDhEJUzMnZ1NaUcuH6/T3nHRee49u2uycO805l+mc6+ecO4PAiXUiEqaOHdOPtMRYnl1c4HcUiWAHcmW660KWQkRCLj4mmlMnZPHaZ1uprG3wO45EqAMpEhayFCLiibMm51BT38SrOmdCOulAioQLWQoR8cTkwb3JzUji2cWFfkeRCLXPImFmFWa2u5VbBYFzJkQkjJkZZ03O4eP1O3TVOumUfRYJ51yKcy61lVuKcy6mq0KKSOedOSkbgDlLinxOIpHoQHY3iUgEGNQniWm5fXh2cQHOaS+xdIyKhEgPcNbkbNaVVrG0oNzvKBJhVCREeoCTJ2SRHBfNgx9u8DuKRBgVCZEeIDUhlnMPG8KLS4vYuL3K7zgSQVQkRHqIi48cSkx0FHe/u87vKNIJxeXV1DV0/TpcKhIiPUS/1ATOzhvEM4sLKCqr9juOdIBzjuNue5db5q7s8s9WkRDpQX58zDCcg3vfX+93FOmA8up6qusbyUpL6PLP9qxImNloM1vS7LbbzK5p0WaGmZU3a3ODV3lEBHLSkzh9YjZPLNzM9spav+NIOxWV1QAwsHdil3+2Z0XCObfKOTfROTcRmALsAZ5rpen7e9s55270Ko+IBFxx7HBqG5p44AMd6RQpissDuwe71UiiheOBdc65TV30eSLShuGZvTh5fBaPfryJ8up6v+NIOxSVd8ORRAuzgCfaeG66mS01s1fNbFxrDczsUjPLN7P80tJS71KK9BBXHDucitoGHvloo99RpB22llcTE2X07RXf5Z/teZEwszjgNOCpVp5eDAxxzh0C/A14vrX3cM7Nds7lOefyMjN1aW2RAzVuYBrHj+nH/R9u0LUmIkBxWQ39UxOIjur6KzR0xUjiW8Bi59y2lk8453Y75yqD918BYs2sbxdkEunxrj5+JGV76vnnfO0FDndF5dW+zEdA1xSJc2hjV5OZDTAzC96fFsyjC/KKdIGJg3pz1Mi+3PveevbUaTQRzorLa8jyYT4CPC4SZpYEnAg822zbZWZ2WfDhd4AVZrYUuAOY5bRMpUiX+enxI9lRVcfjCzb7HUXa4JyjuLyGgT6NJDy9JoRzbg+Q0WLb3c3u3wnc6WUGEWlbXm4fpg/LYPZ76zn3sCEkxEb7HUla2FFVR11DU7fe3SQiYezq40dQUlHLk/lb/I4irdgaPPx1QFo33N0kIuFv+rAMpuamc9c766htaPQ7jrSwd52tgb01khARH5gZVx83kuLyGh6br7mJcFMcHElkaSQhIn45amRfjh6Vye3zVmmF2DBTVF5NXHQUGclxvny+ioSIYGb8/ozxNDm4Yc4KXQs7jBSX1TAgLYEoH06kAxUJEQka1CeJ604cxRtflPDqiq1+x5GgYh9PpAMVCRFp5sIjchmfncr/vPAZ5Xu0+F84KCqrUZEQkfAQEx3FzWdNYGdVHTfP/cLvOD1eU5Nj227/zrYGFQkRaWF8dhoXHzmUJxZu4d3VWnXZT9sra2locr6dbQ0qEiLSimtOGMmYASlc9ugi8jfu9DtOj1Xk8+GvoCIhIq1Iiovh0YsPJSstgQsf/ITlBeV+R+qRioOHI2f5dCIdqEiISBsyU+J57EeHkpYUyw8fWMCqrRV+R+pxNJIQkbCWlZbIY5ccSnxMFD+4bwGbd+zxO1KPUlxWTXxMFOlJsb5lUJEQkX0akpHMY5ccSl1DI9c/vZSmJp1o11WKd9cwsHciwcvu+EJFQkT2a0S/FP7fKWNZuGEn//pEq8V2leIyf0+kAxUJEWmn7+blcPjwDP7wyhds213jd5weobi8xtf5CFCREJF2MjNuOvNg6hqbuGHOCr/jdHsNjU2BE+k0khCRSJHbN5lrTxzFa59tY+6KYr/jdGslFbU0OX8PfwUVCRHpoEuOHMq4gan8Zs5nlFdrfSevFJcHLzak3U0iEklioqO4ZWZgfadfP7dcy4p7pKgseI6ERhIiEmnGZ6fxs5NG8dKyYh5fqKvZeWFrGJxIByoSItJJlx09nGNGZfK/L37OZ0VatiPUisqrSY6LJjUhxtccKhIi0ilRUcafvncI6UmxXPX4p1TWNvgdqVvZe0U6P0+kAxUJETkAGb3iuWPWJDbtqNL8RIgVl1cz0MfrSOylIiEiB+TQYRlcd+Io5iwp4sl8nY0dKkXl/p8jASoSIhICV8wYwREjMvjtC5+zrrTS7zgRr7ahke2Vtb5PWoOKhIiEQFSUcft3J5IQG8VP//UpdQ1NfkeKaMVlNTgH2ekqEiLSTQxIS+CWmRNYUbib2+et8jtORCsMXmwoR3MSItKdnDRuAOceNph73lvPB2u2+x0nYhXuChaJ9CSfk6hIiEiI/frksYzo14vrnlzCzqo6v+NEpIKyaswCozO/qUiISEglxkVzx6xJ7NpTx61zV/odJyIV7qqmf0oCcTH+/4r2LIGZjTazJc1uu83smhZtzMzuMLO1ZrbMzCZ7lUdEus7Ygan88LBcnszfwpptujZ2RxWW7QmLSWvwsEg451Y55yY65yYCU4A9wHMtmn0LGBm8XQrc5VUeEelaVx83guT4GG7RaKLDCsuqyQ6DSWvout1NxwPrnHObWmw/HXjEBcwHeptZVhdlEhEPpSfHccWMEbzxRQnz1+/wO07EaGxyFJfVdP+RRAuzgCda2Z4NND9FsyC4TUS6gQuPyCUrLYE/vPIFTU1asqM9SipqaGhyPWckYWZxwGnAU6093cq2r30nmdmlZpZvZvmlpaWhjigiHkmIjeZnJ41maUE5Ly/XlezaY+/hrz1pJPEtYLFzblsrzxUAg5o9zgGKWjZyzs12zuU55/IyMzM9iikiXjhzUjZjBqRw62srqW1o9DtO2AunE+mga4rEObS+qwngBeC84FFOhwHlzjn9uSHSjURHGb/61hi27KzmD6+s1Eqx+1EQZiMJT69mYWZJwInAj5ttuwzAOXc38ApwMrCWwNFPF3qZR0T8ccyoTC44PJeHPtpIdJTx/045yPfrJISrwrJq0pNiSYrz92JDe3mawjm3B8hose3uZvcdcKWXGUTEf2bG/3x7LAD3f7AB5+A3p6pQtKZgV3VYLMexV3iUKhHp9vYWCjN44MMNOBw3nDpWhaKFwl17GNkvxe8YX1KREJEuY2aBwoDxwIcb6BUfw89OGu13rLDhnKOwrJoZo/v5HeVLKhIi0qXMjN+cehBVtQ387a21DMtM5sxJOX7HCgs7q+qoqW8Km3MkQAv8iYgPzIzfnTGeQ4f24ZdPL2fRpp1+RwoLew9/DZcjm0BFQkR8EhcTxd3nTmFg7wQufWQRW3bu8TuS7748kU4jCRGRwPpO950/lbrGJi55OJ+Kmnq/I/nqyxPpNJIQEQkY0a8Xd/1gCmtLK7n8n4t79PWxC3ZVkxwXTVpirN9RvqQiISK+O3JkX24+62A+WLud659a2mMXAywsqyY7PTGsDgvW0U0iEha+mzeI0spabp27ir694nvkyXaFu8LnOhJ7qUiISNi4/JjhlOyu5YEPN9AvNZ7Ljhnud6QuVVhWzeQhvf2O8R9UJEQkbOw92W57ZS03v7qSAakJnDGpZ1xiprK2gfLq+rBakgNUJEQkzERFGbd/7xBKK2r5xTPLGJKRxKTB6X7H8lw4Hv4KmrgWkTAUHxPNXedOoX9qPD9+dBFby2v8juS5wrLAeSLhdCIdqEiISJjqkxzHfedNpaq2gUsfzaemvntfsGjvdSTC5WJDe6lIiEjYGj0ghT+fPZHlheX88pll3fqCRYW7qomLjqJvr3i/o/wHFQkRCWsnjRvA9SeNZs6SIq54bHG3Xb6joKyagb0TiIoKr8N+NXEtImHvihnDcc7x97fX8ebKEi4+cihXzBhOSkL4nJl8oAp3VYfdfARoJCEiEcDMuOq4kbx1/TGcenAWd72zjmNve4e3Vm7zO1pIOOcoCMMT6UBFQkQiSFZaIn86eyLPX3kE/VISuOThfB5bsMnvWAfs72+vZXtlbVge6qsiISIRZ+Kg3jx12XSOGZXJr59bwa1zV0bspPZznxZw27zVnDUpm1lTB/kd52tUJEQkIiXHx3DveXmcM20w/3hnHdf+ewklu2u+ViwamxzrSiuZu6KYtSWVPqVt3UfrtvOLp5cxfVgGN8+cEJZrVWniWkQiVkx0FDedOZ6c9ET++Noqnl9SRHJcNEMzkxncJ4mt5TV8UVxBdbNzLEb178XJB2dxysFZjOyf0iU5d1bV8ctnltHQ2MTkwelMHpJOcnwMP350EbkZydz9wynExYTn3+wWaUO0vLw8l5+f73cMEQkzS7eUsbSgjPWlVWzYXsXmnXvITIln3MBUxmalMqJfL5ZuKeOVFVv5ZONOnIOzJmVz01kHkxAb3aHPqqpt4Pv3zic9OY6Zk3M4cWz/Nt+jvLqe7987n7UllQzuk8SaZqOZfinxPHflEV0yYW1mi5xzeR1+nYqEiPQ0JbtreOTjTdz59lom5KRxzw+nkJXW/l/Uf3p9NXe8uYb+qfFs211LSkIMp07I4pxpg5mQ89UqrpW1DZx73wI+L9rN7POmMGN0P8qr61mypYyVxbs5cWx/hmX28qKLX6MiISLSQfM+28q1/15CYlwM9/xwMlOG9Nnva4rLqzn2tnc44aD+3DFrEvPX7+DpxQXMXbGVPXWN5A1J56Ijh3LkyL5c8lA+izfv4h8/mMxJ4wZ0QY/apiIhItIJq7dV8KNH8ikuq+Gq40Zw8ZFDSY5ve7r2uieX8NKyYt687hgG9flqWe/dNfU8lV9tcAyyAAAKQ0lEQVTAQx9tYMvOauJiomhobOKOcyZx6oSBXdGVfVKREBHppLI9dfzXs8t5dcVWMpLjuPLYEXz/0MFfm2dYVlDGaXd+yOUzhvPLb45p9b0amxxvfLGNJz/ZwumTsjntEP8LBKhIiIgcsE837+KPr63io3U7GJiWwOUzhvPdvEEkxEbjnOPse+azrrSSd34+I+KWBOlskdAhsCIiQZMGp/P4jw7jw7XbuW3eKn4z5zP+8sYazj88l/6p8SzcuJPfnzk+4grEgVCREBFp4YgRfTl8eAYLN+zk7nfX8afXVwOBcyzOzgu/s6K9pCIhItIKM+PQYRkcOiyDlVt386+FWzhrcjYx0eF50ptXVCRERPZjzIBUfnvaOL9j+MLTkmhmvc3saTNbaWZfmNn0Fs/PMLNyM1sSvN3gZR4REekYr0cSfwXmOue+Y2ZxQFIrbd53zp3qcQ4REekEz4qEmaUCRwMXADjn6oA6rz5PRERCz8vdTcOAUuBBM/vUzO4zs+RW2k03s6Vm9qqZ9cydfiIiYcrLIhEDTAbucs5NAqqAX7VosxgY4pw7BPgb8Hxrb2Rml5pZvpnll5aWehhZRESa87JIFAAFzrkFwcdPEygaX3LO7XbOVQbvvwLEmlnflm/knJvtnMtzzuVlZmZ6GFlERJrzrEg457YCW8xsdHDT8cDnzduY2QALXorJzKYF8+zwKpOIiHSM10c3XQ08FjyyaT1woZldBuCcuxv4DnC5mTUA1cAsF2mLSYmIdGMRt8CfmZUCm1p5Kg0o7+Tjvff3/tsX2N7JiC0/p6NtwqUf+8u5v+dD2Q/w9mvSkX60tq217M3vqx/tz7m/NupH5/sxxDnX8f31zrlucQNmd/bx3vvN/s0PVY6OtgmXfrSnL13VD6+/Jh3pR3uzqx+d78e+2qgfoe/H/m7daRGSFw/g8YtttAlFjo62CZd+tOd9emI/WtvWWvbm99WP/Wdpbxv1I/T92KeI293UFcws33Vi3fVw0136Ad2nL+pHeFE/9q87jSRCabbfAUKku/QDuk9f1I/won7sh0YSIiLSJo0kRESkTd2+SJjZA2ZWYmYrOvHaKWa23MzWmtkde0/8Cz53tZmtMrPPzOzW0KZuNUvI+2FmvzWzwmZLtZ8c+uRfy+LJ1yP4/PVm5lo7az/UPPp6/M7MlgW/FvPMbGDok38tixf9+GPw8gDLzOw5M+sd+uRfy+JFP74b/PluMjNP5y0OJH8b73e+ma0J3s5vtn2fP0Ot8uqwqXC5EViJdjKwohOvXQhMBwx4FfhWcPuxwBtAfPBxvwjtx2+B6yP96xF8bhDwGoFzaPpGYj+A1GZtfgLcHaH9OAmICd6/BbglQvtxEDAaeAfIC8f8wWy5Lbb1IXDych8gPXg/fV993det248knHPvATubbzOz4WY218wWmdn7Zjam5evMLIvAD+3HLvC/+whwRvDpy4GbnXO1wc8o8bYXnvWjy3nYjz8DvwC6ZJLNi34453Y3a5pMF/TFo37Mc841BJvOB3K87YVn/fjCObfK6+wHkr8N3wBed87tdM7tAl4HvtnZ3wXdvki0YTZwtXNuCnA98I9W2mQTWKRwr4LgNoBRwFFmtsDM3jWzqZ6mbduB9gPgquBugQfMLN27qPt0QP0ws9OAQufcUq+D7scBfz3M7PdmtgX4AeDXlRpD8X2110UE/mL1Qyj74Yf25G9NNrCl2eO9fepUX3vcNa7NrBdwOPBUs91x8a01bWXb3r/sYggM4w4DpgJPmtmwYHXuEiHqx13A74KPfwfcTuCHusscaD/MLAn4NYFdHL4J0dcD59yvgV+b2X8BVwH/E+Ko+xSqfgTf69dAA/BYKDO2Ryj74Yd95TezC4GfBreNAF4xszpgg3PuTNruU6f62uOKBIHRU5lzbmLzjWYWDSwKPnyBwC/Q5sPkHKAoeL8AeDZYFBaaWROBtVO68mIXB9wP59y2Zq+7F3jJy8BtONB+DAeGAkuDP0w5wGIzm+YCKxF3lVB8XzX3OPAyXVwkCFE/gpOlpwLHd+UfT82E+uvR1VrND+CcexB4EMDM3gEucM5tbNakAJjR7HEOgbmLAjrTVy8nY8LlBuTSbEII+Aj4bvC+AYe08bpPCIwW9k7ynBzcfhlwY/D+KAJDO4vAfmQ1a3Mt8K9I/Hq0aLORLpi49ujrMbJZm6uBpyO0H98kcFmAzK7I7/X3FV0wcd3Z/LQ9cb2BwN6O9OD9Pu3pa6u5uvKL6McNeAIoBuoJVNKLCfzlORdYGvxmvqGN1+YBK4B1wJ18dfJhHPDP4HOLgeMitB+PAsuBZQT+qsqKxH60aLORrjm6yYuvxzPB7csIrMuTHaH9WEvgD6clwVtXHKXlRT/ODL5XLbANeC3c8tNKkQhuvyj4dVgLXNiRn6GWN51xLSIibeqpRzeJiEg7qEiIiEibVCRERKRNKhIiItImFQkREWmTioR0C2ZW2cWfd5+ZjQ3RezVaYOXXFWb24v5WTTWz3mZ2RSg+W2R/dAisdAtmVumc6xXC94txXy1S56nm2c3sYWC1c+73+2ifC7zknBvfFfmkZ9NIQrotM8s0s2fM7JPg7Yjg9mlm9pGZfRr8d3Rw+wVm9pSZvQjMM7MZZvaOmT1tgesjPLZ3/f3g9rzg/crgwnxLzWy+mfUPbh8efPyJmd3YztHOx3y1cGEvM3vTzBZb4BoApwfb3AwMD44+/hhs+/Pg5ywzs/8N4X+j9HAqEtKd/RX4s3NuKjATuC+4fSVwtHNuEoGVVm9q9prpwPnOueOCjycB1wBjgWHAEa18TjIw3zl3CPAe8KNmn//X4Ofvd42c4LpCxxM4+x2gBjjTOTeZwDVMbg8WqV8B65xzE51zPzezk4CRwDRgIjDFzI7e3+eJtEdPXOBPeo4TgLHNVtFMNbMUIA142MxGElgFM7bZa153zjVf13+hc64AwMyWEFhf54MWn1PHV4sjLgJODN6fzlfr9T8O3NZGzsRm772IwPr/EFhf56bgL/wmAiOM/q28/qTg7dPg414EisZ7bXyeSLupSEh3FgVMd85VN99oZn8D3nbOnRncv/9Os6erWrxHbbP7jbT+M1Pvvprca6vNvlQ75yaaWRqBYnMlcAeBa0pkAlOcc/VmthFIaOX1BvzBOXdPBz9XZL+0u0m6s3kErskAgJntXXY5DSgM3r/Aw8+fT2A3F8Cs/TV2zpUTuGzp9WYWSyBnSbBAHAsMCTatAFKavfQ14KLgNQgws2wz6xeiPkgPpyIh3UWSmRU0u11H4BduXnAy93MCS7wD3Ar8wcw+BKI9zHQNcJ2ZLQSygPL9vcA59ymBVT9nEbhYT56Z5RMYVawMttkBfBg8ZPaPzrl5BHZnfWxmy4Gn+c8iItJpOgRWxCPBq+ZVO+ecmc0CznHOnb6/14mEE81JiHhnCnBn8IikMrr40rAioaCRhIiItElzEiIi0iYVCRERaZOKhIiItElFQkRE2qQiISIibVKREBGRNv1/RMDi/MT8mrkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.388738</td>\n",
       "      <td>2.521896</td>\n",
       "      <td>0.608056</td>\n",
       "      <td>0.440603</td>\n",
       "      <td>02:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.901857</td>\n",
       "      <td>1.958518</td>\n",
       "      <td>0.687528</td>\n",
       "      <td>0.504890</td>\n",
       "      <td>01:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.565219</td>\n",
       "      <td>1.652850</td>\n",
       "      <td>0.723345</td>\n",
       "      <td>0.536689</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.306575</td>\n",
       "      <td>1.455323</td>\n",
       "      <td>0.747950</td>\n",
       "      <td>0.565299</td>\n",
       "      <td>02:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.137455</td>\n",
       "      <td>1.342423</td>\n",
       "      <td>0.764108</td>\n",
       "      <td>0.587329</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.949592</td>\n",
       "      <td>1.282115</td>\n",
       "      <td>0.774509</td>\n",
       "      <td>0.601252</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.808173</td>\n",
       "      <td>1.269965</td>\n",
       "      <td>0.778810</td>\n",
       "      <td>0.607137</td>\n",
       "      <td>01:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.767959</td>\n",
       "      <td>1.274921</td>\n",
       "      <td>0.778822</td>\n",
       "      <td>0.607116</td>\n",
       "      <td>01:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_predictions(learn, ds_type=DatasetType.Valid):\n",
    "    learn.model.eval()\n",
    "    inputs, targets, outputs = [],[],[]\n",
    "    with torch.no_grad():\n",
    "        for xb,yb in progress_bar(learn.dl(ds_type)):\n",
    "            out = learn.model(*xb)\n",
    "            for x,y,z in zip(xb[0],xb[1],out):\n",
    "                inputs.append(learn.data.train_ds.x.reconstruct(x))\n",
    "                targets.append(learn.data.train_ds.y.reconstruct(y))\n",
    "                outputs.append(learn.data.train_ds.y.reconstruct(z.argmax(1)))\n",
    "    return inputs, targets, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='149' class='' max='149', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [149/149 00:28<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "inputs, targets, outputs = get_predictions(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj pendant que xxunk les activités requises pour maintenir mon xxunk physique , est - ce que je xxunk de la protection d’un régime d’assurance ou de pension ?,\n",
       " Text xxbos xxmaj while i go about maintaining this high degree of fitness , am i protected under an insurance or pension plan ?,\n",
       " Text xxbos xxmaj while i do to the my physical physical of physical , do i aware by the pension plan service plan ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[10],targets[10],outputs[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les conséquences sur la recherche , la mise en pratique et les politiques en ce qui a trait à l'ac ?,\n",
       " Text xxbos xxmaj what are the xxunk for xxup kt research , practice / policy ?,\n",
       " Text xxbos xxmaj what are the implications implications research kt , , policy and policies in)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[700],targets[700],outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelle est la position des xxmaj états - xxmaj unis , du xxmaj canada et de la xxup xxunk à ce propos ?,\n",
       " Text xxbos xxmaj where do the xxup us , xxmaj canada and xxup xxunk stand ?,\n",
       " Text xxbos xxmaj what is xxmaj xxup us xxmaj xxmaj united and the xxunk fit in)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[701],targets[701],outputs[701]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quels sont les atouts particuliers du xxmaj canada en recherche sur l'obésité sur la scène internationale ?,\n",
       " Text xxbos xxmaj what are the unique xxmaj canadian strengths in obesity research that set xxmaj canada apart on an international front ?,\n",
       " Text xxbos xxmaj what are xxmaj specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[2500],targets[2500],outputs[2500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?,\n",
       " Text xxbos xxmaj what are some of the long - term policy implications of this global knowledge revolution ?,\n",
       " Text xxbos xxmaj what are the long the long - term policies implications of this global scientific ? ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[4002],targets[4002],outputs[4002]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label smoothing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They point out in the paper that using label smoothing helped getting a better BLEU/accuracy, even if it made the loss worse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Transformer(len(data.train_ds.x.vocab.itos), len(data.train_ds.y.vocab.itos), d_model=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, model, metrics=[accuracy, CorpusBLEU(len(data.train_ds.y.vocab.itos))], \n",
    "                loss_func=FlattenedLoss(LabelSmoothingCrossEntropy, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.281034</td>\n",
       "      <td>3.357356</td>\n",
       "      <td>0.621848</td>\n",
       "      <td>0.458009</td>\n",
       "      <td>01:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.872045</td>\n",
       "      <td>2.923340</td>\n",
       "      <td>0.690921</td>\n",
       "      <td>0.510376</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.598603</td>\n",
       "      <td>2.653438</td>\n",
       "      <td>0.729291</td>\n",
       "      <td>0.545735</td>\n",
       "      <td>01:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.407944</td>\n",
       "      <td>2.514847</td>\n",
       "      <td>0.748187</td>\n",
       "      <td>0.567057</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.195246</td>\n",
       "      <td>2.403729</td>\n",
       "      <td>0.766409</td>\n",
       "      <td>0.592165</td>\n",
       "      <td>01:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.095695</td>\n",
       "      <td>2.362098</td>\n",
       "      <td>0.776127</td>\n",
       "      <td>0.604666</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.999303</td>\n",
       "      <td>2.358647</td>\n",
       "      <td>0.779535</td>\n",
       "      <td>0.609675</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.923621</td>\n",
       "      <td>2.359421</td>\n",
       "      <td>0.780211</td>\n",
       "      <td>0.610871</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quels sont les atouts particuliers du Canada en recherche sur l'obésité sur la scène internationale ?\n",
      "What are Specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?\n",
      "Quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?\n",
      "What are the long the long - term policies implications of this global scientific ? ?\n"
     ]
    }
   ],
   "source": [
    "print(\"Quels sont les atouts particuliers du Canada en recherche sur l'obésité sur la scène internationale ?\")\n",
    "print(\"What are Specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?\")\n",
    "print(\"Quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?\")\n",
    "print(\"What are the long the long - term policies implications of this global scientific ? ?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelle distance y a - t - il entre le point le plus rapproché de la surface à xxunk et la position d’utilisation habituelle du tube radiogène ?,\n",
       " Text xxbos xxmaj what is the distance between the nearest point of the area to be shielded and the usual operational position of the x - ray tube ?,\n",
       " Text xxbos xxmaj what is the xxmaj between the xxmaj and of the xxmaj ? the ? and the most ? ? of the xxmaj - ray tube ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[10],targets[10],outputs[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quels types de présentations xxmaj santé xxmaj canada xxunk - t - il dans le format ectd à compter du 1er septembre ?,\n",
       " Text xxbos xxmaj what kind of submission types will xxmaj health xxmaj canada accept on xxmaj september 1 , 2004 in ectd format ?,\n",
       " Text xxbos xxmaj what is of information is of be canadian xxmaj canada take ? the canadian ? , and ? the format ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[700],targets[700],outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les trois caractéristiques qui vous incitent le plus à investir dans votre région ( xxup nommez - xxup les ) ?,\n",
       " Text xxbos xxmaj what are the three most attractive features about investing in your region ( xxup name xxup it ) ?,\n",
       " Text xxbos xxmaj what is the main main important concerns of the in the country ? xxup xxunk , xxunk ) ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[701],targets[701],outputs[701]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles actions avez - vous prises et quel en a été le résultat ?,\n",
       " Text xxbos xxmaj what were your actions and the outcomes ?,\n",
       " Text xxbos xxmaj what is the targets ? how main of)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[4001],targets[4001],outputs[4001]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test leakage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we change a token in the targets at position n, it shouldn't impact the predictions before that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xb,yb = data.one_batch(cpu=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp1,out1 = xb[0][:1],xb[1][:1]\n",
    "inp2,out2 = inp1.clone(),out1.clone()\n",
    "out2[0,15] = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y1 = learn.model(inp1, out1)\n",
    "y2 = learn.model(inp2, out2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0., device='cuda:0', grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y1[0,:15] - y2[0,:15]).abs().mean()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}