16_accel_sgd.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "! [ -e /content ] && pip install -Uqq fastbook\n",
    "import fastbook\n",
    "fastbook.setup_book()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "from fastbook import *"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "[[chapter_accel_sgd]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Training Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now know how to create state-of-the-art architectures for computer vision, natural language processing, tabular analysis, and collaborative filtering, and you know how to train them quickly. So we're done, right? Not quite yet. We still have to explore a little bit more the training process.\n",
    "\n",
    "We explained in <<chapter_mnist_basics>> the basis of stochastic gradient descent: pass a mini-batch to the model, compare it to our target with the loss function, then compute the gradients of this loss function with regard to each weight before updating the weights with the formula:\n",
    "\n",
    "```python\n",
    "new_weight = weight - lr * weight.grad\n",
    "```\n",
    "\n",
    "We implemented this from scratch in a training loop, and also saw that PyTorch provides a simple `nn.SGD` class that does this calculation for each parameter for us. In this chapter we will build some faster optimizers, using a flexible foundation. But that's not all we might want to change in the training process. For any tweak of the training loop, we will need a way to add some code to the basis of SGD. The fastai library has a system of callbacks to do this, and we will teach you all about it.\n",
    "\n",
    "Let's start with standard SGD to get a baseline, then we will introduce the most commonly used optimizers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establishing a Baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll create a baseline, using plain SGD, and compare it to fastai's default optimizer. We'll start by grabbing Imagenette with the same `get_data` we used in <<chapter_resnet>>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide_input\n",
    "def get_data(url, presize, resize):\n",
    "    path = untar_data(url)\n",
    "    return DataBlock(\n",
    "        blocks=(ImageBlock, CategoryBlock), get_items=get_image_files, \n",
    "        splitter=GrandparentSplitter(valid_name='val'),\n",
    "        get_y=parent_label, item_tfms=Resize(presize),\n",
    "        batch_tfms=[*aug_transforms(min_scale=0.5, size=resize),\n",
    "                    Normalize.from_stats(*imagenet_stats)],\n",
    "    ).dataloaders(path, bs=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = get_data(URLs.IMAGENETTE_160, 160, 128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create a ResNet-34 without pretraining, and pass along any arguments received:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_learner(**kwargs):\n",
    "    return vision_learner(dls, resnet34, pretrained=False,\n",
    "                    metrics=accuracy, **kwargs).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the default fastai optimizer, with the usual 3e-3 learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.571932</td>\n",
       "      <td>2.685040</td>\n",
       "      <td>0.322548</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.904674</td>\n",
       "      <td>1.852589</td>\n",
       "      <td>0.437452</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.586909</td>\n",
       "      <td>1.374908</td>\n",
       "      <td>0.594904</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = get_learner()\n",
    "learn.fit_one_cycle(3, 0.003)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try plain SGD. We can pass `opt_func` (optimization function) to `vision_learner` to get fastai to use any optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = get_learner(opt_func=SGD)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to look at is `lr_find`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(0.017378008365631102, 3.019951861915615e-07)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhc9X3v8fdXI412a7HlfTfGYDYbGxNKYxxCEhIIJCHNpU3a0NBCkiYkN0/TNM29tOU+Cbmla5qnpQQuoQFSEghroEAbtkAM2JjF7LaxZEm2JWvfpdF87x8zkoUsyZLRmTmj+byeZx7PnDkz58NIzFe/81uOuTsiIpK9ctIdQERE0kuFQEQky6kQiIhkORUCEZEsp0IgIpLlVAhERLJcbroDTNWcOXN8+fLl6Y4hIpJRtm/ffsjdq8Z6LuMKwfLly9m2bVu6Y4iIZBQzqx7vOZ0aEhHJcioEIiJZToVARCTLqRCIiGQ5FQIRkSynQiAikuVUCEREMsCjrx1kV0NnIO+tQiAiEnLuzpdv285dL9QG8v4qBCIiIdfRF2Ng0Kksigby/ioEIiIh19LVD0BFsQqBiEhWakoWgtkqBCIi2UktAhGRLKcWgYhIllOLQEQkyzV39RPNzaE4Ggnk/VUIRERCrrmrn8qiKGYWyPurEIiIhFxzVz+VAZ0WAhUCEZHQa+5WIRARyWpqEYiIZDkVAhGRLDYwGKejN6ZCICKSrYKeQwAqBCIiodbcHeysYlAhEBEJtebOZIsgoCWoQYVARCTUhlsEJSoEIiJZqblLLQIRkax2uBDkBXYMFQIRkRBr7uqnrDCP3EhwX9cqBCIiIdbc1R/oiCFQIRARCbXmrv5A5xCACoGISKgFvbwEqBCIiITa0LUIgqRCICISUu5OS3c/lQHOIQAVAhGR0OrsizEw6GoRiIhkq+YULDgHkBvkm5vZXqADGARi7r5x1PNbgHuBd5KbfuHu1wSZSUQkUwwVgqCHjwZaCJI+4O6HJnj+KXe/MAU5REQySqpaBDo1JCISUqlqEQRdCBx4xMy2m9kV4+xzlpm9ZGYPmdlJAecREckYM6KPADjb3evNbC7wqJm94e5Pjnj+BWCZu3ea2ceAe4DVo98kWUSuAFi6dGnAkUVEwqG5u59obg7F0Uigxwm0ReDu9cl/G4C7gU2jnm93987k/QeBPDObM8b73ODuG919Y1VVVZCRRURCo7kzMZnMzAI9TmCFwMyKzax06D7wYWDnqH3mW/K/0Mw2JfM0BZVJRCSTtHQHv7wEBHtqaB5wd/J7Phe43d3/08y+CODu1wOfBr5kZjGgB7jU3T3ATCIiGaMpBesMQYCFwN33AKeNsf36Efd/CPwwqAwiIpmspaufJRVFgR9Hw0dFREIqVS0CFQIRkRAaGIzT0RtTIRARyVYtKZpDACoEIiKh1NydmlnFoEIgIhJKzZ3JFkHAS1CDCoGISCg1dPQBUFWaH/ixVAhERELoYHsvAPPLCgI/lgqBiEgIHWjvpTgaoSQ/+KsFqBCIiIRQQ3sf81LQGgAVAhGRUDrQ3sv8WSoEIiJZ62B7L/NUCEREspO7J04NqRCIiGSn5q5++gfjzJsV/NBRUCEQEQmdg+2JOQTqIxARyVJDcwjmqhCIiGSnVE4mAxUCEZHQOZAsBFUl6iMQEclKB9t7mVMSJZqbmq9oFQIRkZA5mMKho6BCICISOgfaUjeZDFQIRERCp6FDhUBEJGv1x+Ic6uxP2WQyUCEQEQmVxs7UTiYDFQIRkVA50JYYOqpTQyIiWWpoMpkKgYhIljpcCNRHICKSlQ609xKN5FBZHE3ZMVUIRERCpKG9j7mz8jGzlB1ThUBEJERSPZkMVAhERELlYAqvVTxEhUBEJEQOtvcyN4UdxaBCICISGh29A3T1D6pFICKSrYYuUak+AhGRLJWOyWQQcCEws71m9oqZvWhm28Z43szsB2a2y8xeNrPTg8wjIhJmQ8tLpOoSlUNyU3CMD7j7oXGe+yiwOnk7E/jX5L8iIlnnYEfqZxVD+k8NXQz8uydsBcrNbEGaM4mIpMXBtl5K83Mpiqbib/TDgi4EDjxiZtvN7Ioxnl8E7BvxuDa57V3M7Aoz22Zm2xobGwOKKiKSXnWtvSwsL0z5cYMuBGe7++kkTgH9iZltHvX8WHOo/YgN7je4+0Z331hVVRVEThGRtKtv7WFRxQwrBO5en/y3Abgb2DRql1pgyYjHi4H6IDOJiIRVXWsPC8tT21EMARYCMys2s9Kh+8CHgZ2jdrsP+IPk6KH3AW3uvj+oTCIiYdXZF6OtZ4BF5UUpP3aQPRLzgLuTK+jlAre7+3+a2RcB3P164EHgY8AuoBv4wwDziIiEVn1rD0BaWgSBFQJ33wOcNsb260fcd+BPgsogIpIp6pKFYPFM6yMQEZHJqWsZahGoEIiIZKX61h5yc4y5pTOos1hERCavrrWHBeUFRHJSd2WyISoEIiIhUNfSw8Ky1J8WAhUCEZFQSNdkMlAhEBFJu4HBOAfae1mUho5iUCEQEUm7g+29xB0VAhGRbJXOoaOgQiAiknb1bYlCoD4CEZEsNdwi0KghEZHsVNfay+ziKIXRSFqOP6lCYGarzCw/eX+LmV1lZuXBRhMRyQ51aRw6CpNvEdwFDJrZccBNwArg9sBSiYhkkfrW9E0mg8kXgri7x4BPAv/o7v8T0LWFRUTeI3enriUzWgQDZva7wOeBB5Lb8oKJJCKSPVq7B+gZGEzb0FGYfCH4Q+As4Lvu/o6ZrQBuDS6WiEh2GLoOQbomk8EkL0zj7q8BVwGYWQVQ6u7fDzKYiEg2CEMhmOyoocfNbJaZVQIvATeb2d8HG01EZOYbmkOQCX0EZe7eDnwKuNndNwDnBRdLRCQ71Lf2UJCXQ0VR+rpdJ1sIcs1sAfAZDncWi4jIe1TX2sOi8kLMUn9BmiGTLQTXAA8Du939eTNbCbwdXCwRkeyQmExWlNYMkyoE7v5zdz/V3b+UfLzH3S8JNpqIyMzm7uw91MXiNPYPwOQ7ixeb2d1m1mBmB83sLjNbHHQ4EZGZbM+hLtp7Y5y2uCytOSZ7auhm4D5gIbAIuD+5TUREjtGOmlYA1i+tSGuOyRaCKne/2d1jyduPgaoAc4mIzHg7aloozc/luKqStOaYbCE4ZGafM7NI8vY5oCnIYCIiM92OmlbWLS0nJyd9I4Zg8oXgCySGjh4A9gOfJrHshIiIHIOuvhhvHGhn/ZL0r+g/2VFDNe5+kbtXuftcd/8EicllIiJyDF6ubSPu6e8fgPd2hbJvTFsKEZEss2NfCwDrMqVFMI70ntQSEclgO2paWTmnmIriaLqjvKdC4NOWQkQki7g7O2paWLc0/a0BOMoy1GbWwdhf+AakdyqciEiGqm3p4VBnfyj6B+AohcDdS1MVREQkW7xQk+gfOD0kLYL3cmpoUpLzDnaY2RGrlprZZWbWaGYvJm9/FHQeEZF021HTSmFehDXzwvG39qSuUPYefQ14HZg1zvN3uPtXUpBDRCQUdtS0cOriMnIjgf8tPimBpkguTHcBcGOQxxERyRS9A4O8Wt8emv4BCP7U0D8CfwbEJ9jnEjN72czuNLMlY+1gZleY2TYz29bY2BhIUBGRVHjjQAexuLNuSXpXHB0psEJgZhcCDe6+fYLd7geWu/upwH8Bt4y1k7vf4O4b3X1jVZXWuhORzFXd1AXAyjQvNDdSkC2Cs4GLzGwv8B/AuWZ268gd3L3J3fuSD38EbAgwj4hI2lU3dQOwtDK9VyUbKbBC4O7fdvfF7r4cuBT4lbt/buQ+yesgD7mIRKeyiMiMVd3UzbxZ+RTkRdIdZVgqRg29i5ldA2xz9/uAq8zsIiAGNAOXpTqPiEgq1TR3sayyON0x3iUlhcDdHwceT96/esT2bwPfTkUGEZEwqG7qZvPx4errDMcgVhGRLNDTP0hDRx/LQtQ/ACoEIiIpU9Oc7CierUIgIpKVhguBWgQiItlpaA7Bstnh6ixWIRARSZGa5m5K83OpKMpLd5R3USEQEUmR6qZuls4uwixcF3hUIRARSZGa5m6WhayjGFQIRERSYjDu1LZ0szRkk8lAhUBEJCXqW3sYGHS1CEREstXQ0NGwTSYDFQIRkZQYXnVULQIRkexU09xNXsRYUFaY7ihHUCEQEUmBmuYuFlcUEckJ19BRUCEQEUmJ6qbu0C0tMUSFQEQkYO5OTVM45xCACoGISOBaugfo6IupRSAikq3CutjcEBUCEZGADc8h0KkhEZHsVJOcQ7CkQoVARCQrVTd3M7c0n8JoJN1RxqRCICISsDCPGAIVAhGRwNU0h3PV0SEqBCIiAeodGORAe29oh46CCoGISKBqW8I9YghUCEREAjW06ugStQhERLJT2OcQgAqBiEigqpu6KYpGmF0cTXeUcakQiIgEaF9zYtVRs/AtPz1EhUBEJEDVzeFdfnqICoGISEDicWdfc7gnk4EKgYhIYBo6+uiLxdUiEBHJVkMjhpaGdPnpISoEIiIBGb4OQba3CMwsYmY7zOyBMZ7LN7M7zGyXmT1rZsuDziMikir7mrvJMVhYXpjuKBNKRYvga8Dr4zx3OdDi7scB/wD83xTkERFJiermbhaWFxLNDffJl0DTmdli4ALgxnF2uRi4JXn/TuCDFubBtiIiU1CTAUNHIfgWwT8CfwbEx3l+EbAPwN1jQBswe/ROZnaFmW0zs22NjY1BZRURmVZhvw7BkMAKgZldCDS4+/aJdhtjmx+xwf0Gd9/o7hurqqqmLaOISFA6+2I0dfWHerG5IUG2CM4GLjKzvcB/AOea2a2j9qkFlgCYWS5QBjQHmElEJCWGrlO8LMQXpBkSWCFw92+7+2J3Xw5cCvzK3T83arf7gM8n7386uc8RLQIRkUwzPIcgA1oEuak+oJldA2xz9/uAm4CfmNkuEi2BS1OdR0QkCDXNiTkESzOgjyAlhcDdHwceT96/esT2XuB3UpFBRCSVapq7KSvMo6wwL91Rjircg1tFRDJUdYaMGAIVAhGRaefuvHmgg5Vzwt9RDCoEIiLTbm9TNw0dfZyxojLdUSZFhUBEZJo9u6cJgDNXHDE/NpRUCI7RM7sO8YP/fpvY4HiTpkUkWz37TjNzSvJZVZUZp4ZSPnw0jHY1dNLTP8gpi8uOum9XX4xrH3qdW7fWANDc1c9fXXRS0BFFJEO4O8/uaeLMFZWhvk7xSFlfCNydK3+yjeqmbv7ls6fz4ZPmj7vvM7sP8a27Xqa2pYc/+u0VxOLOj5/Zy/LZRVx29ooUphaRsKpt6aG+rZcvrsyM/gFQIWB7dQu7G7uoKMrjy7e9MGYxqG/t4XsPvs4DL+9n2ewifnblWZyxvJLBuFPb0sM1D7zGstnFnLGikmd2HeKZ3U3MKYly8bpFGbHOiIhMn60Z1j8AWVYIDrT1Mr+s4F3b7nh+H8XRCA99bTNX3rqdL9/2An/z6VNZUFZIY2cfb+xv5+an9xJ352sfXM0Xz1lFYTQCQCTH+KdL1/GZf/sNV966nXjcicWdgrwcegfi/O0jb3Hmiko+eOJcygujFOVHKC+McubKSvIi6p4RmYm27mmmoiiP1XNL0h1l0rKmENz7Yh3f/PnL3PuVszlxwSwgsTrgL1/Zz8dPXcj8sgJ+cvkmfv+m5/jGz15612vPP2k+37ngxDH/ui/Oz+Wmz5/B1ffuZGVVCZuPn8OGZRU0tPdxz446frGjju89+Ma7XrNsdhFfPXc1n1i3kNxIDu7OvuYeork5RxQqEcksz77TxKYVleTkZEb/AIBl2hpvGzdu9G3btk35dS1d/XzoH55gYXkhd3/5bCI5xh3P1/Ctu17hri/9FhuWVQCJzuCn3m6ktCCPuaX5zJ1V8J6miLs7bT0DdPUP0tUXY1dDJz/81S5e29/OijnFVJXk8/r+djr6YkRyjE+uX8RV564ed32Srr4YDpTkZ00NF8kYda09nP39X3H1hWv5wm+Hq9/QzLa7+8axnsuab5OK4ih/+fGT+OpPd3Dz0+/wR+9fyR3P72NVVTGnLy0f3q84P5fzT14wbcc1M8qLopQnv9ePn1fKR0+ez8OvHuRHT+1h0J1PrF/E2oWz2NXQya1bq7lnRx0fP20hK+cUU1aUR1E0l7cOdvDsniZ21rcDsGFpBeesqeKc46tYu2BWRv31ITJTDc8fyKCOYsiiQgBw4akLuGdHHX/3yFusqirhhZpW/uJjJ6R8iJeZcf7J8zn/5CNHKF25eSX/8vhu7nqhlo7e2PD2aCSHdUvK+fKWVQzGnSffbuS6h9/kuoffZE5JPuccX8WWNYlbaUH4F7kSmYme3dNMWWEeJ86fle4oU5I1p4aG1Lf28KG/f4KBQSfuzta/+CBzSvKnMeH06Y/Fae8doKM3xoKyAgryIu96vrGjjyffauTxtxp58q1G2noGiObm8IE1VVxw6kLOO3EuRdGsqvUiabXlusc4bm4pN35+zDMwaaVTQyMsLC/kWx89gavvfZWPnDQvtEUAIJqbw5yS/HEzVpXmc8mGxVyyYTGxwTg79rXyy5f38+Ar+3n41YPMKsjlMxuX8AdnLR/uc4jHnY6+GLMKcjNmsotIJth7qIu9Td187n3L0h1lyrKuEAB87sxlNHf189Fp7AtIt9xIDmcsr+SM5ZX87wvX8tw7zdz2bDU/fmYvNz39DqcsKqOlu5+DbX30D8ZZWFbA+1bO5n2rZrOgrID+WJyBwTgVRVE2TTAjsrW7n7cOdtLWM8DskihVJflUFkeJJPsozCA/NzLma0Vmsuuf2E00N4ePn7Yw3VGmLCsLQU6O8fXzjk93jMBEcoyzVs3mrFWzOdDWy61bq9le3cKKOcXML0uMgnq1rp0n3mrkFzvqjnj9mnmlXP7+FVy8biH7mnt48q1Gfr3rEK/Wt3Gwve+ox1+/tJzLfms5HztlwTHPl3B3Drb3sauhk5wcKIrmUhyNMKswj/KivGkvNv2xOF19MXpjg/QOxDESlxgc3QnfOzCIO8NzSYIy1HIryc8dLrISXvuau7lzey2fPXMp82Zl3hDwrOsjkMPcnV0Nib/u8yI5RHNzeLW+nRuf2sMbBzqI5ubQH0ssqrdyTjHrl1awZn4Jx88rpaIoSlNXH4c6+mnp7iee/DXq6Y9x30v17G3qZt6sfNYtKWd/Wy/1rT209QywtLKIVVUlrKgqpiA3gieC0DcYp7M3RldfjIaOPl7f305L98C42YuiEeaW5nPc3FKOn1fCqqoSZhXmUZCXQ0FehKbOfmqau6hp7qatJ0ZhXg6FeREiOTm0dvfT2NlHU2c/rd39tPYM0N0/eMQxSvJzOXVxGSctnEVjRx+v7W9nd2MXg3GnrDCPBWUFLCovZNXcEo6rKmHV3GLmzSpgTkn+Ef05Y+mPxbn/pXrebuiktTvxOTZ19rO/rZeD7b3E4k6OQWVxlMriKHmRHOKe+Ln1x+J09sXo7IvhDicvmsX6pRWsW1LOovJCKoqilBfnUZQXwcwwEq01nQ4Mxl/c/Qp3bqvliT/bwoKywnTHGdNEfQQqBHIEd+fpXU08/OoBTlhQyubVVVNaKiMed554q5FbfrOX2pYeFpYXsqi8gNKCPKqbutjd2EV1UxcDg4d/96K5OZTm51Kcn0tFcZQT5pWyduEsVs8rwTC6+2N09Q/S3jNAS1c/Ld0D7G/r4e2GTvYe6iIWH/v3uLwoj4qiKL0Dg/QMDDIQi1NRHGV2ST5ziqNUFEcpT7YyivNzKciLUJCXKICv1LXx0r42Xt/fTlVpPmsXzGLtwlkU5EU40NbL/rZealu62XOoa7hgDinJz2VheQHLZhezYk7idsL8UtbMLyWSY/x8Wy3/+vhu6lp7yIskhhhXFOVRWRxlYVkh88sKqCyO0tYzwKHOfpo6+xiMO2ZGjkHeiM9rMO68XNvKzrp2+idYDTdx2i6H/NwIxdEIpy0pZ/PxVWw+vopF5eH88soE9a09nHPdY3xm4xK++8lT0h1nXCoEEjpDv3fT8RdqfyxObUs33f2JL/vegUEqiqIsqSyaluvFxuM+4TyNwbizr7mbPYc6aezo41BnP4c6+6ht6aG6KdGBOFQozKAkmktHX4z1S8v5+nnHs3n1nGn5HPpig7x5oIOG9j5auvtp7R6gJ3kqy3EG44mWRO/AIG09A2zd08yB9l4g0aL49OmLuXjdIiqKo+85Sza5+t6d/PS5Gh770y0srgjv2mIqBCJpFE8uTvj6gXZe399OXUsPHz9tIe+fpgJwrNydtxs6eeLNRu55sY5X69vJixjvWzmbxRWFzC0tYH5ZAacsKuOE+aXkan2sIxxo62Xz3zzGJRsWce2nTk13nAmpEIjIUb1W386d22vZuqeJho4+mrr6GPp6KMyLcNqSMtYuKGNlVTErq4pZM6+U2SEefh00d+eKn2znsTcaeOxPt4R+pWHNIxCRo1q7cBZXL1w7/Dg2GGd/Wy8v7mtle3ULL9S08NPnaugZONyxvnJOMRuXV7BpxWzOO3Eu5UXZc1rp5qf38uhrB/lf4yxImUlUCERkTLmRHJZUFrGksmh4bHw87hxo72VPYxc769vYtreZR147yM+21ZIXMbasmcsn1i3inDVVM3phxJf2tXLtQ69z3olzuTxki8sdC50aEpH3JB53Xq1v594X67jvpXoaOvqI5BgnLyrjfSsqOX1ZBSfML2VJxZHzMjJRW88AF/zgKdzhl1f9dsa0gtRHICIpMRh3nnunmWd2H2LrniZe3Nc6PEy4KBphzfxSNiytYOPySjYurwj1Ei9jOdTZx1dv38Hze5u548qzhpevzwQqBCKSFj39g7xxoJ03D3TwxoEOXq1v46XatuHhtMtmF7F+SXlysmIpC8sKmVeWH8plSp7edYiv3/EibT0DXPvJU7hkw+J0R5oSdRaLSFoURiOsX1rB+qWH/3Luiw2ys66d7dXNvFDdyjO7m7jnxfp3vW5BWQEbllWwaUUlm1ZUsmZeadqG2vbFBvnBf7/Nvzy+m5Vzivn3L2wavsrhTKFCICIplZ8bYcOyiuHTKu5OfVsvexo72d+amLG9q7GT599p5oGX9wOwem4Jv3fmUj61fjFlRam73sZvdjfxnXteYU9jF7+zYTF/ffFJM3Jpd50aEpFQck9MxHvq7UPc8XwNL9W2UZCXw4fWzudDa+exZU0VswK6CNNbBzv4tyf2cNcLtSypLOT/XHwyW9bMDeRYqaI+AhHJeDvr2vjpczU8/OoBDnX2kxcxfmvVHD51+iI+ctL8SS30N5H61h7ueH4fD76yn7cbOsnNMa7YvJKvnrs68NVmU0GFQERmjMG4s6OmhUdfO8gDL++nrrWH0oJcLjx1AZtWVA7Pfp7KEuhPvd3IV27fQXvvAJuWV3LBqQs4/6T5zM3AJaXHo0IgIjNSPO5sfaeJO7fV8tDOA8OznqORHE5eNIszV85m04pKTl9SMWbfgrtz06/f4XsPvs7quaVc//sbWDGnONX/GSmRlkJgZgXAk0A+iU7pO939L0ftcxlwHTB0dZQfuvuNE72vCoGIjCU2GOedQ128tr+dnXVtbK9u4eXatuElyiuLoyybXcSi8sLh1sLB9l6e2d3ER0+ez9/+zmkUz+DZ0OkaPtoHnOvunWaWB/zazB5y962j9rvD3b8SYA4RyQK5kRxWzytl9bxSLl63CIDu/hg7alrZWddGdXM31U1d7KxrG76QUo7BNz+yhi+ds2pGzHo+VoEVAk80NTqTD/OSt8w6DyUiGa0omsvZx83h7OPmpDtKqAW6wLiZRczsRaABeNTdnx1jt0vM7GUzu9PMlozzPleY2TYz29bY2BhkZBGRrBNoIXD3QXdfBywGNpnZyaN2uR9Y7u6nAv8F3DLO+9zg7hvdfWNVVVWQkUVEsk5KLjnk7q3A48D5o7Y3uXtf8uGPgA2pyCMiIocFVgjMrMrMypP3C4HzgDdG7bNgxMOLgNeDyiMiImMLctTQAuAWM4uQKDg/c/cHzOwaYJu73wdcZWYXATGgGbgswDwiIjIGTSgTEckCE80jSEkfgYiIhJcKgYhIlsu4U0Nm1ghUJx+WAW0T3B+9LQ84NMVDjnyPyTw3ett4jyfKO2eKOSfKeCw5J8p2rBmPlnM6Mw5t0897cjkz9ec9Vt7p/Cxn2s+73N3HHn/v7hl7A26Y6P7obSQ6qY/5GJN5bvS28R5PlHeqOSfKeCw5j5LtmDJO92epn7d+3kF/ljP15z3WLdNPDd1/lPvjPX+sx5jMc6O3jff4aHmn4mivm2rOibIda8ajvXY6Mx7tWBPRz3vsf49F0D/vkff1855424TvkXGnht4LM9vm4/Sah0km5FTG6ZMJOTMhI2RGzjBmzPQWwVTdkO4Ak5QJOZVx+mRCzkzICJmRM3QZs6pFICIiR8q2FoGIiIyiQiAikuVUCEREspwKQZKZvd/MrjezG83smXTnGYuZ5ZjZd83sn83s8+nOMx4z22JmTyU/zy3pzjMeMys2s+1mdmG6s4zHzE5Mfo53mtmX0p1nLGb2CTP7kZnda2YfTneesZjZSjO7yczuTHeWkZK/g7ckP7/PpivHjCgEZvb/zKzBzHaO2n6+mb1pZrvM7M8neg93f8rdvwg8wDgXyEl3RuBiYBEwANROd8ZpzDl0mdKCIHJOU0aAbwE/m+58I/JMx+/l68nfy88A0z7kcJoy3uPuf0xi9eD/EdKMe9z98unONpYp5v0UcGfy87soFfnGNNXZgmG8AZuB04GdI7ZFgN3ASiAKvASsBU4h8WU/8jZ3xOt+BswKY0bgz4Erk6+9M6yfJZCTfN084LaQZjwPuJTEl9eFYf0sk6+5CHgG+L2wZky+7u+A00OeMZD/b95D3m8D65L73B50tvFuQV6PIGXc/UkzWz5q8yZgl7vvATCz/wAudvdrgTFPBZjZUqDN3dvDmNHMaoH+5MPB6c44XTlHaAHyw5jRzD4AFJP4n7HHzB5093jYcibf5z7gPjP7JXB72DKamQHfBx5y9xemM990ZUylqeQl0WJeDLxIGs/QzIhCMI5FwL4Rj2uBM4/ymsuBmwNLdKSpZvwF8M9m9n7gySCDjTKlnGb2KQeorNkAAARLSURBVOAjQDnww2CjDZtSRnf/DoCZXQYcmu4iMIGpfpZbSJw+yAceDDTZYVP9vfwqiRZWmZkd5+7XBxkuaaqf42zgu8B6M/t2smCk0nh5fwD80Mwu4L0t5/GezORCYGNsm3D2nLv/ZUBZxjOljO7eTaJYpdpUc/6CRNFKpSn/vAHc/cfTH2VCU/0sHydxve9UmmrGH5D4QkulqWZsAr4YXJyjGjOvu3cBf5jqMKPNiM7icdQCS0Y8XgzUpynLeDIhI2RGzkzICJmRUxmnX6jzzuRC8Dyw2sxWmFmURMfgfWnONFomZITMyJkJGSEzcirj9At33nT1Uk9zL/1Pgf0cHlZ5eXL7x4C3SPTWf0cZZ0bOTMiYKTmVUXndXYvOiYhku5l8akhERCZBhUBEJMupEIiIZDkVAhGRLKdCICKS5VQIRESynAqBzAhm1pni491oZmun6b0GzexFM9tpZvebWflR9i83sy9Px7FFQBevlxnCzDrdvWQa3y/X3WPT9X5HOdZwdjO7BXjL3b87wf7LgQfc/eRU5JOZTy0CmbHMrMrM7jKz55O3s5PbN5nZM2a2I/nvmuT2y8zs52Z2P/CIJa609rglrg72hpndllxymeT2jcn7nZa4ctxLZrbVzOYlt69KPn7ezK6ZZKvlNyRWqsTMSszsv83sBTN7xcwuTu7zfWBVshVxXXLfbyaP87KZ/fU0foySBVQIZCb7J+Af3P0M4BLgxuT2N4DN7r4euBr43ojXnAV83t3PTT5eD3ydxHULVgJnj3GcYmCru59GYnnwPx5x/H9KHv+oC4yZWQT4IIfXoOkFPunupwMfAP4uWYj+HNjt7uvc/ZuWuDzkahJr3q8DNpjZ5qMdT2TITF6GWuQ8YG3yj3iAWWZWCpQBt5jZahJLF+eNeM2j7t484vFz7l4LYGYvAsuBX486Tj+JK2EBbAc+lLx/FvCJ5P3bgb8dJ2fhiPfeDjya3G7A95Jf6nESLYV5Y7z+w8nbjuTjEhKFIZXXrJAMpkIgM1kOcJa794zcaGb/DDzm7p9Mnm9/fMTTXaPeo2/E/UHG/n9mwA93to23z0R63H2dmZWRKCh/QmJ9/88CVcAGdx8ws70krgM9mgHXuvu/TfG4IoBODcnM9gjwlaEHZrYuebcMqEvevyzA428lcUoKEssOT8jd24CrgD81szwSORuSReADwLLkrh1A6YiXPgx8wcyGOpwXmdncafpvkCygQiAzRZGZ1Y64fYPEl+rGZAfqaxy+QtXfANea2dMkLioelK8D3zCz54AFQNvRXuDuO0hc2PxS4DYS+beRaB28kdynCXg6Odz0Ond/hMSpp9+Y2SvAnby7UIhMSMNHRQJiZkUkTvu4mV0K/K67X3y014mkmvoIRIKzgcSFyQ1oBb6Q5jwiY1KLQEQky6mPQEQky6kQiIhkORUCEZEsp0IgIpLlVAhERLKcCoGISJb7/3JpPASwEr3ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like we'll need to use a higher learning rate than we normally use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.969412</td>\n",
       "      <td>2.214596</td>\n",
       "      <td>0.242038</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.442730</td>\n",
       "      <td>1.845950</td>\n",
       "      <td>0.362548</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.157159</td>\n",
       "      <td>1.741143</td>\n",
       "      <td>0.408917</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(3, 0.03, moms=(0,0,0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because accelerating SGD with momentum is such a good idea, fastai does this by default in `fit_one_cycle`, so we turn it off with `moms=(0,0,0)`. We'll be discussing momentum shortly.)\n",
    "\n",
    "Clearly, plain SGD isn't training as fast as we'd like. So let's learn some tricks to get accelerated training!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Generic Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build up our accelerated SGD tricks, we'll need to start with a nice flexible optimizer foundation. No library prior to fastai provided such a foundation, but during fastai's development we realized that all the optimizer improvements we'd seen in the academic literature could be handled using *optimizer callbacks*. These are small pieces of code that we can compose, mix and match in an optimizer to build the optimizer `step`. They are called by fastai's lightweight `Optimizer` class. These are the definitions in `Optimizer` of the two key methods that we've been using in this book:\n",
    "\n",
    "```python\n",
    "def zero_grad(self):\n",
    "    for p,*_ in self.all_params():\n",
    "        p.grad.detach_()\n",
    "        p.grad.zero_()\n",
    "\n",
    "def step(self):\n",
    "    for p,pg,state,hyper in self.all_params():\n",
    "        for cb in self.cbs:\n",
    "            state = _update(state, cb(p, **{**state, **hyper}))\n",
    "        self.state[p] = state\n",
    "```\n",
    "\n",
    "As we saw when training an MNIST model from scratch, `zero_grad` just loops through the parameters of the model and sets the gradients to zero. It also calls `detach_`, which removes any history of gradient computation, since it won't be needed after `zero_grad`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The more interesting method is `step`, which loops through the callbacks (`cbs`) and calls them to update the parameters (the `_update` function just calls `state.update` if there's anything returned by `cb`). As you can see, `Optimizer` doesn't actually do any SGD steps itself. Let's see how we can add SGD to `Optimizer`.\n",
    "\n",
    "Here's an optimizer callback that does a single SGD step, by multiplying `-lr` by the gradients and adding that to the parameter (when `Tensor.add_` in PyTorch is passed two parameters, they are multiplied together before the addition): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sgd_cb(p, lr, **kwargs): p.data.add_(-lr, p.grad.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pass this to `Optimizer` using the `cbs` parameter; we'll need to use `partial` since `Learner` will call this function to create our optimizer later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Optimizer, cbs=[sgd_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if this trains:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.730918</td>\n",
       "      <td>2.009971</td>\n",
       "      <td>0.332739</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.204893</td>\n",
       "      <td>1.747202</td>\n",
       "      <td>0.441529</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.875621</td>\n",
       "      <td>1.684515</td>\n",
       "      <td>0.445350</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = get_learner(opt_func=opt_func)\n",
    "learn.fit(3, 0.03)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's working! So that's how we create SGD from scratch in fastai. Now let's see what \"momentum\" is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Momentum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As described in <<chapter_mnist_basics>>, SGD can be thought of as standing at the top of a mountain and working your way down by taking a step in the direction of the steepest slope at each point in time. But what if we have a ball rolling down the mountain? It won't, at each given point, exactly follow the direction of the gradient, as it will have *momentum*. A ball with more momentum (for instance, a heavier ball) will skip over little bumps and holes, and be more likely to get to the bottom of a bumpy mountain. A ping pong ball, on the other hand, will get stuck in every little crevice.\n",
    "\n",
    "So how can we bring this idea over to SGD? We can use a moving average, instead of only the current gradient, to make our step:\n",
    "\n",
    "```python\n",
    "weight.avg = beta * weight.avg + (1-beta) * weight.grad\n",
    "new_weight = weight - lr * weight.avg\n",
    "```\n",
    "\n",
    "Here `beta` is some number we choose which defines how much momentum to use. If `beta` is 0, then the first equation becomes  `weight.avg = weight.grad`, so we end up with plain SGD. But if it's a number close to 1, then the main direction chosen is an average of the previous steps. (If you have done a bit of statistics, you may recognize in the first equation an *exponentially weighted moving average*, which is very often used to denoise data and get the underlying tendency.)\n",
    "\n",
    "Note that we are writing `weight.avg` to highlight the fact that we need to store the moving averages for each parameter of the model (they all have their own independent moving averages).\n",
    "\n",
    "<<img_momentum>> shows an example of noisy data for a single parameter, with the momentum curve plotted in red, and the gradients of the parameter plotted in blue. The gradients increase, then decrease, and the momentum does a good job of following the general trend without getting too influenced by noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hU1dbA4d8mBEhACEhNAEFRBGnRWDCCFAUuIgYVBXvnqlgRDVe5dkWxtw8VvIKiwBUMCCgXBKQjwURRBKkKoYUSigQIyf7+2DNhMpkWppwp630enpCZkzkrgaw5Z++111Zaa4QQQkS/SlYHIIQQIjQk4QshRIyQhC+EEDFCEr4QQsQISfhCCBEjKlsdgDt169bVzZo1szoMIYSIKCtXrtytta7n6rmwTfjNmjUjOzvb6jCEECKiKKX+dPecDOkIIUSMkIQvhBAxQhK+EELECEn4QggRIyThCyFEjJCEL4QQMUISvhBCxIiwrcMXIhpl5eQxctZathUUkpyUwNCeLclITbE6LBEjJOELESJZOXkMm7KKwqJiAPIKChk2ZRWAJH0REjKkI0SIjJy1tjTZ2xUWFTNy1lqLIhKxRhK+ECGyraCwQo8LEWiS8IUIkeSkhAo9LkSgScIXIkSG9mxJQnxcmccS4uMY2rOlRRGJWCOTtkKEiH1iVqp0hFUk4YuoECnljhmpKWEZl4gNkvBFxPO13DFS3hSECBZJ+CLieSp3tCf0k62BlzcJEU1k0lZEPF/KHU+mBt7+JpFXUIjmxJtEVk5eQOIWItTkCl9EvOSkBPJcJH3Hckdf3hScr+b/Pnrc452DXP2LSCNX+CLi+VLu6K0G3tXVfEFhkcuv2VZQyFNZq3hkYu7JXf0fOQK//+7T9yZEIEnCFxEvIzWFl69uS0pSAgpISUrg5avblrna9vam4GrIx51aCfGMX/YX2ulxd0NEWTl5pI+YS/PMGXR6aTbrOnaH1q358MJrSH/5exkiEiETkCEdpdQnQB9gl9a6jYvnFfA20Bs4DNymtf4pEOcWkSHYwx+O5Y72cz0yMbfcudzF4Gt7g4T4OJSiXLK3c34d58nigd98xJm5S1jWpA2DfpxCXs16DD3Yl2e/+Y2Cw0Vefzb27y2voJA4pSjWmhQZThI+CtQY/qfAe8A4N8//AzjT9udC4P9sH0UMCGWXSFfnemRiLtl/7uWFjLZuz+duHqB2YjyJVSqXeZN4ZGKu2/M7Dx053jn0XrOI+5Z9xfgOvXiqx318PPl5npo7hl8anUVucsvSeN39bJy/t2KtvX6NEI4CMqSjtV4A7PVwyFXAOG0sA5KUUo0CcW4R/nytkHEc+kgfMfekhjpcnUsD45f95fH13A35PH3lOSzO7MamEVewOLMbGakppUn9lKN/8/ScD2m6bzsAyvY6juxX/Gflb2bkzLdYmXw2z3YfhFaVGHLFo+w85VTezxpB2tbfuGRTDmjtdmjI07CTdN0UvgjVGH4KsMXh8622x8pQSt2jlMpWSmXn5+eHKDQRbL5WyASiBNLduTR4TIi+zAPYDe3ZkobHD/PPZV9x+8pv+O4/g2l0YDc3XtTUHF9cDFu2wKJF3LZpEfcvmcjoyc/zd5UE7s0YxrHK8QDsTziF+67KpO7hfXw1/gk+nzSct6a/RtXjx1x+H96GnVzdoZyMQLzxivAUqrJM5eKxcsOgWuuPgI8A0tLS3A2TigjjS9mkL4un3HGcH6hkG9d2pWhrHgweDNdfD506lXveY9uDbdtg4UJYsICMBQvI+PVX85qV4tCV4pgzdTjV17SARzebZH/8OABP2758a816/DPjX+w65dQyL7uq0Zncct3zNDi0h9P2bWfIovEkH8jnmdteKBeCu5+jXZxy9WtWMbJJS3QL1RX+VqCJw+eNgW0hOrewmC9lkyfbK975zsBdsu+yIZtZnz4I778PXbvC66+D87EbNsBNN5k3hG0O/z0ffhhSUmDAABg3DpKT4YUXYMEC4g//TfXZ31G9/qlw9Ch07AhDh8KHH8J338GaNUxbso7rh00gp3ErUpISuOmipqV3EkkJ8fzUvB3TWnfh3fSBDO77OO23r2PS2Edh3TqPP8fKxcepd+jESKq7793xZ+Xtyl02aYluobrCnwYMVkpNwEzW7tdabw/RuYXFfOkS6ctdgCveyinji4t4bMFnDPpxCvtbnA2ffQuvvgqPPQbLlsEnn5hE/fzz8H//B/Hx5o1g1ix480247TaYNw/S0szzHTpAZadfm06dICfHbQx9gb4dW7h93vEOJadjT5ZndKTz0Lvgootg6lS45BLgxM9xyKSfKdaaN6e/zpVrFnLuA+PZm1iLFA8/K1+v3GWTlugWqLLML4EuQF2l1FbMnWw8gNZ6FDATU5K5HlOWeXsgzisih7cukUN7tiyTkMC3XvGeElFaSQHDxz9L++3r2Nj/Fk4fOwoSEmDyZHjtNcjMhJUrYc8eOHQI7roLnnnG/P3OO+GOO2DiRNi0CW65xSR9J4EoN3X5s+nUBq64Arp3h/Hj4dprS48FmD7yU65csxCAW1dOZ1S3Wzz+rHwdMjvZN14RGQKS8LXWA708r4H7A3EuEZ1Otle8Y4KqVXiQ/dVqgFK0jj/GV58/BYUFMHkyp1999YkvUsoMu5x/vknkl14KI0ZA69Ynjpk/Hz74wLwp/P03NG1a7txBHe9u0QKWLIG+feGGG6BOHejWzbx2w0r0mv0OGxs0Iy+hNrflTOf0kc9ypYdz+nrlfrJvvCIyKO1l3M8qaWlpOjs72+owRJizJ924QwdZ/v4tvN/xOj7tdD3zZ79M/Z9XmInW888/+RNs2mSGdh59FJo1K/NU+oi5Lq+GU5ISWJzZzef4Pb7JFRRAejrk5cGiRVC/PnTpAn/9Zd4QDh40Qz7vvAMPPOD2PPZYqxYdJb6kmENVE93GKj2CIptSaqXWuvztKNI8TUQ4eyKa8XEW1YuOcP/yr+jcrwv1f1xkkqA/yR6geXPzOi74O97t0x1CUhJ8+60Zz+/d23y+ebN5rF07c8zFF8Mbb0CbNrBrl/mzc+eJv+/axawt22DXLmocK6QExZLT2jG9XXcuefyecnHJJi3RS67wRXQYO9ZMsIJJiocOQX6++XuQ+HuFX6Gvz8mBzp2hqAimT4fLLjvx3NSpkJFR9vhKlaBePXNHYPuzgUTm7NEcP3iIvmsW0mTfdg5Urc7ETv1p9O8n6NPpbJ++bxHe5ApfRL81a0yFzYAB8NlnJiEGMdmD/+PdFbpDSE2FxYtNff+555Z9rm9fmD3bVA/ZE3ydOibpOzjD9icrJ48ek3+h9eZfuefHKdw951P2LZ7CvDGT6Dqwp0+xi8gkCV9EhzVr4MwzTX381KlmotMLX8aqPR3j76bkFa6IsQ/hOFOq7BW/FyNnraXweAkrG7dmUOPWtNmxntGTn6PRg4PosmEUD1/hvueQiGyS8EVA+ZtET9qaNabKpmlTM25dpYrXOL2Nn/tyjD/j3VZVxDjfQfzasAWZvR7g06+epf+MTxh29A5AVtZGI+mHLwLGl344Ad828PBhePBBk/BTU81jVauaq14PfFlRGuxVpxXp3xNIru4g5p9xPhPbXs4/l0/mrD9X8/0HE6B6dbNGQUQNucIXAePL4h6vx+zfbxJ2tWreT7h8uamj/+MPk/SHDPE5Vl/Gz0Ox6tTTHUKwyiNd3VkAvND9Li7ZnMvrM94kr1Z982b6/fdw3XV+n1OEB7nCFwHjdxI9cAAaNYLERDjjDLPSdMgQGD3a1KDv3m0OPnYMnnzSlCMeOWKS0ttvm1W0PvK25aGvxwSLqzuhRybm0iwAHSwd7ywcHaxanSf+8SAt9m7l0k22/Yk2bvTjuxDhRhK+CBi/k+iuXVBYCH36mPr5vDyz2vXuu02/mnr1oG5dswr1pZfg1lvhl19KV6BWhC8N3Xw5Jljc9fWHAAyDYZL+4sxuvHV9hzLf46LmqUw4t/eJAyXhRxVJ+CJg/E6iBw6YB+66CyZMgNxcU0+/cSPMnAlvvMGmS3uxJDGZu68eTvpZN5G18VCZ1/K1l7sv4+dWjbGD92GjQM0luPoea7z9xolVxfPnw759fp9HhAcZwxcB40uZYkZqCmjNitc/5mjBAZZ2urL0DWHwqPm8BzzwzTq6N8kzx8bFmdWuzZuT1bAdw/a2ovBM25WvU9VMRXvb+FJhY9WqU2+97+HEm4K/Y/0uv8cpU0wfofnzTcnn7Nmmtl9ENEn4MS7QE4NeE+QPP5Dx+ONk/Pij+XzIlWQBw6asIn1PAQCbiiq7TNTeJnz92UQl3LibWHWUnJQQvAZuqammRfTMmdCvn0n6c+ZI0o9wMqQTwwJeIunN3Xebxl/btpmJ2BYt4K67yH/8Ke79/lOu/+V/AByqkuByyMLbpHA09XJ3nlh1LjK1D4MFfcOS3r0hKwtWrzZJf6+nratFuJOEH8NCtbtRVk4enV+cDaNHM6ddF6ZNmmf6zY8ZA7t3c/ecTxm8dBKdN/3EptqN2GHbBtA5UXubFLayqiYY7BOrm0dcwZvXd3A5lxCSN7l//ONE0u/eXWrzI5gM6cSwUCQL+11Etf3mynBh/ZZMmrmekmoJZHTuDHv30unVeWw5cKzc1zonam8rU6O5l7u7obKQbVjSq5dpWXHVVSbpz5ljKqZERJEr/BgWiiti+11E0hFTTVNQrUbpXURWTh7pry1gy4FjbocsHHmrmrGyqsYqIS0d7dkTpk2DtWvhyisD//oi6OQKP4aF4orYfreQVHgQgP3VTgFOzBfYz60x49Qak6jdTR57mxSOtV7u/jZwc8ftZH6PHmYbyMxM2LEDGjYMwHchQkUSfgwLVrJwZB9yqHXEJPyCBJPw45RyubCoIrtFCSPQb3JeK38uvtgcuHKlWQ0tIoYk/BgX7Cti+12E45BOQnyc23LDSKyoiTZey1tTU02v/RUrJOFHGBnDF0FlH1dvhknk1RrWd9nHxS5SK2qiidfJ/Bo1oFUrkB3pIo4kfBF0GakpPHxuXVCK756+kozUFEv71AjPfJrM79jRrL59+WWz7aKICDKkI1zyZwWu89eOPGU7F48bZyb44kySD8X8gTg5Pk3mv/SS6bHzr3+ZvkejR/u/YbwIOtnEXJTjPGkH5hfelxJHx69NPrCLp74fTe8/lnCoaXNqjPsPXHppsMMXAeDzG35WFtx/v6nYefZZ07bay+YzIrhkE3NRIf70pBk5ay2df13IwJ9nccHWX1EaXu18CzMvH8h8SfYRoUJ3dxkZ0LUr3HcfDB9uupu+/LIk/TAlCV+U427SLq+gkPQRcz0mgk4/ZPHSd++xtVZ9Zp3Zkdc630JerfqoQ+6bgInwcVLN2GrVgs8+g5o14ZVXzKY0b77pNekHa0cv4Z4kfFGOu+X6Ckofd5kI3nqLEd+9y/zm5/HPfsM4En9im0KpvokMJ3N3V5q4a17BiPR8rn/7bfMG0LAhNGgADRqwQVVn9l7ITWjAhg4dubD9aUxemRf4Lp/CI0n4ohxXk3b2VbCOShNBh2R48UUYPpy87r158PxBHNEnKnCk+iZyVLS/Upk7AqV4Iv02fq7dlLsS9nB68SHYuZNDi5fRYOdO/nnMvMbRbyqz5LT2HG2ZzldtL0MrUywYqa2sI4kkfFGOqwoad5txbNt32Cyzf/VVuOUWUsaM4blVO+VWPUJ5a8bmPAxz+NjxsncESvHFOd35wWHFdM8Rc8krKKRa0RHab1/H5euW0XPdMkZ++w6/NWjB6ganl365LLwLLkn4wiXnFbjptl9aR0qX8PoPo2H5NDNp9+67UKlSzPWziSaeSjJdje+742rj+iPx1VjetC3Lm7Zl1lkd+e8XmdQ5vL/M18nQX3BJwo9SzldiXc+ux7w1+Sd91T20Z0vGffA1PX6Zx28NzuDblum8NutdMlZ9D48/DiNGSGVGFPC0PiJ9xFyPO3A50piLhKE9W7q8a9hfrQYA1/z6PU327+TLDr0A6Hp2vcB9M6IcqcOPQq7q6J15ras/doyfXnyXw5O+4ufaTbl06yrabPmdEqWopDV7ayRR51ABvPCCWXwjyT7qNc+cUW4ex5uE+DiuOS+lzAQtQP2De/jxg1tLP7+/7xPMaNVJmucFgKc6fGmtEIVcVVo4c7uz1aFD8MYbFDY5jXOfe4xL1izj/qWTSPj7AC/1GMS33//CT0+/zoGEmjxz2SDS4zqSlbstSN+JCCeehlvi3LzhFxYVM29Nfrn+SQUJNTkcX5Ufmp/LqgZn8PJ379KkYIeM4QeZDOlEIV9/acoct3u3GYN/913Yt4/fT2/PW5feyx91T6PJ/h2saHwOKEXSgm0cLW5N4R0fmK+TcrqYMbRnSx6emOvyuRKtXVZygfl/Zp/XKb37BC69ZzR7EmvS6OAeZv7nAd6b+goP3v9uML+FmCdX+FHI14mv0uPefReaNoXnnjOtD5Yu5Zr+L7Lg9PPYUbMuK5q0KR2yKSgsCsk+uCL8ZKSmUDsx3uVzyUkJPjVds3dPTUqIJ79GbUoqxZFXqz5Dez9M+x3rGLNmclBiF4Yk/CjkqhOls9LaeK3NDkYdOphNqr/+Gi66qMLVEnIrHhuevvIct11Ofe2AmpGaQu7TPXjLYWP23y7oxoYBt9Ni/MfwzTfB/jZilgzpRCFXlRZuq3Q2b4a9e+GWW0yPcxt35XnV4iux73D5drhSThcbfOly6usajHLlu0fTYW0u3HYb5OZCkybB/FZiklTpxLrJk+Haa2H5crjggjJPuep1Apx0J00hvFq3Ds491wwxzpgBzZpZHVHECXq3TKVUL+BtIA4YrbUe4fT8bcBIIM/20Hta69GBOLc4OfZkfsO0SQyqVImZJafS1+kYTwuoZCWtCIozz4Rp0+Dqq+HCC0375Y4drY4qavid8JVSccD7wOXAVmCFUmqa1nq106ETtdaD/T2f8J9jnX6bHetZd2pTnpixjpKq1XxK3LKSVvjLY6fMrl1h6VLo08f8/dNPYcAAS+ONFoGYtL0AWK+13qi1PgZMAK4KwOuKICmt09eac3ZuYFXDFlJpI0LGfsGRV1CI5kSnzKycvBMHnX02LFtmhhkHDjQVZGE6/BxJApHwU4AtDp9vtT3m7Bql1C9Kqa+UUi5nY5RS9yilspVS2fn5+QEITbhir6hpdHA3dQ/v59cGZ5R5XIhg8tSCuYy6dc2+uTffDE8/DY89FsIoo1MgEr6rJXbOb8XfAM201u2AOcBYVy+ktf5Ia52mtU6rV096agSLvaKm7Y71APzasEWZx4UIpgq1YK5aFcaOhUGD4I03YMmSIEcX3QKR8LcCjlfsjYEya+211nu01kdtn34MnBeA8woPsnLySB8xl+aZM0gfMbfM7fLTbRO5M3cGDy6ZQLGqxOr6zaVnvQgZXxZolaEUvPaaKdO85x44diyI0UW3QCT8FcCZSqnmSqkqwABgmuMBSqlGDp/2BX4PwHmFG+7GSH/4dBq0bUuPPh0ZPuv/qFl8lFcvvZVT69WWskoRMr4u0CqjRg14/3347TeT/MVJ8btKR2t9XCk1GJiFKcv8RGv9m1LqOSBbaz0NeFAp1Rc4DuwFbvP3vMI9V2OkiQV7OOfBh6BODXNrfMUVND3rLIYBw6wJU8QoXxZvuXTllWbNyHPPQf/+poRTVIgsvIpCzm1slS5h7KSnuWDrb1RbuQLatbMsNiH8sm2bWRGelgZz5khbbhekPXKMcR4LvW/pf+m8OYe3+g6WZC8iW3Ky2Wxn7lyzUbqoELnCD1P2hSl5BYXEKUWx1qT4eOvruLDqwr9W8cWEJ5nZujPF4z4j49zGIfoOhAiSkhLo1AnWroU1a0z5pigV9NYKIrCcd6wqtr0p5/nYe97+3KivV/DONyPJOzUZNWqUJHsR9tz1byo33v/hh5CaCkOGmLJN4RO5wg9DrjYMd+TzNnD2xmjz5kGXLoELUIggcLU1Z3wlBQqKik/kqdJmfV99AC+9BD/8AJ07WxFyWJIx/AjjbcWrzytit241H9u29TMiIYLPVXVZUYkuk+zBYVXuk0+arpoPPADHj4cy1IglCT8MeVvx6ul5xwVXX0xeTHHVqlCnTqBDFCLgKtLaY1tBISQmwptvwi+/wKhRQYwsekjCD0OedqzytEDFecHVKfk72Jp4qmwyLiJCRVp7lB7brx9cfjkMHw67dgUpsughCT8M2ff9TLH9p46z1RqnJCV4XBHrfEvc6OBu8k6pK10wRURwdaETX0kRH1e21r7MRY9S8M47cOgQPPVUqEKNWFKlE6Yq2nM+Kyev3ERvw4O7Wd6kjXTBFBHB3QpcV4+V+d04+2y4/Xb44guT/KtVsyL8iCAJPwrYh3IcVSoppsGhvWyvWU+6YIqI4e5Cx+vFT0YGfPyxqdjp2TNI0UU+GdKJAq6qG5IP5BNfUszupPrSBVNEv65dISHB7IMr3JKEHwVcDdn0XrsYgIv/OUC6YIrol5AA3bvD9OmyM5YHkvADyFMP+mByNWTT77d5/NqkFT36pockBiEs16cPbNpk2i0IlyThB4hP+3QGiXN1w9m7NtEqfzPFN9wY9HMLETZ69zYfp0+3No4wJgk/QHzepzMIHMs4FXDThkWUVK5M+yH3BP3cQoSNJk2gfXv4739lWMcNqdIJkArt0xkEpdUNR47A6XebSgXZF1jEmvvvN9sgTpgAAwdaHU3YkSv8AKnwPp3B8vHHsH07PPpoaM8rhEUc58465Tej4Oy2MHSoWYwlypCEHyAntU+nr/bt8+24wkLTPfDSS02ZmhBRznnubMuBY9x70e2Ql2d+F0QZkvADxHkc3VsbBJ/Nm2ean82Z4/6YzZtNE6kuXWDHDnj2Wdn6TcQEV3NnSxucxbepl8Prr8Mff1gUWXiSMfwAqmg7BJ/MnGk+LlkCl11m/q41rFoFWVnw9deQm2seb9sW3n7bXOELEQPczZH9++Jb+MefK+G668zvTmJiiCMLT5Lww92ePeZjUREsXmwSfFYWbNhgruI7doSRI83S8hYtrI1ViBCx74zlrhanSuNkGD/elGrefTd8+CHUqBHSGMOR7HhlMVdbupW5S+jYEZYtO/F5fLxZUdivH/TtCw0bhj5oISzkamcsR6U7YqWmwAsvmNbJtWrBXXfB4MHQrFloAw4xTzteScK3kKv/uGX+s5aUQM2a8Pff5tY0I8NcsdSqZWHUQljL0xagKa4umpYuNUOdX31lhkP79oWHHjJDn1E41yVbHIahrJw8hkz62fNirU2bTLIfPRomTjR1xZLsRYxzN26vgMWZ3UqTfWm55tS9pHe4h++mL4XMTFi40FSxdegAY8aY6rYYIQnfAvYr+2I3d1el/6Htdzjt2oUoMiHCny9rXly1Onlk8R6yrr0PtmwxF1FghnmaNIGpU0MQufUk4VvAVSmZo+SkBFN7/8QTZrxREr4QpXxZ8+Kx1UlCAtx5p6lumzfPDPNMnhyS2K0mVToW8NRuISE+jqE9zjKVBXl5sGgRVK0awuiECG/udsZyHLd3N8Zf5ndPKbN2pVkz2Ls3iBGHD0n4FkhOSnD5HzJOKTNhu/wbc8Xx6qtkVWnMyBFz3VfxCBGDXK15sVe8uUv24GY4qE6dmEn4MqRjAXe3pK9f156MuD3w8MPQqxdZ3QZY1nJZiEjiOGbvjgLXrU4k4YtgctuG4awkuP56qF0bxo5l5Ox1lrVcFiKSeJsXA9C42Rs3hhK+DOlYxGUbhjvvhLVrYfZsqF/f8pbLQkQKX34nUtx1rrUnfK2jsi7fkVzhh4svvoBPPoEnnzQraQmjlstChDlvvxPuOtdm5eTxbu4eKC7m8menR/1wqST8IPJ5j9v162HQIEhPh6efLn04qC2XhYgirn5X7Nfq7jrX2sf9/9LVACjcmR/1c2QypBMkzm0T7BOu4DSOqDXccIPpkfPFF1D5xD+JL+VnQoiT+12xj/sXJJwCQK3Cg2y1zZFF6++YJPwg8bTwo8x/ppwcWLECRo2Cpk3LvU5QWi4LEYUq+rtiH/ffX8100Uw6cqjM49FIhnSCxOcJ18mTIS4OrrkmBFEJIezs4/4FtoRfy5bwo3mOTBJ+kPg84TpliunaV7duCKISQtjZx/1PXOEfjPo5Mkn4QeLThOuqVbBmDVx9dYijE0LY18NUTW7E7sRa3PHzTF7p3SKqh1ADkvCVUr2UUmuVUuuVUpkunq+qlJpoe365UqpZIM4bzrzucbt6NfTpY3bhkeEcISyRkZrCgid7UHfyBFrs2ETf/7xqdUhB5fekrVIqDngfuBzYCqxQSk3TWq92OOxOYJ/WuoVSagDwCnC9v+cOd24nkRYuNJswVK0KP/wgu1YJYbVevWDYMHj5ZTPEeuONVkcUFIGo0rkAWK+13giglJoAXAU4JvyrgGdsf/8KeE8ppXS4brcVQM5bGL5ZaR0XDH8QmjeH776L+u3WhIgYzz1nLsYGDYK0NGgZfWP5gRjSSQG2OHy+1faYy2O01seB/cCpzi+klLpHKZWtlMrOz88PQGjWct6EodfsL7ngiX+yp3V7WLJEkr0QYaB0geRTs8i4ZDBH46tC//5RuRNWIBK+q+YTzlfuvhyD1vojrXWa1jqtXr16AQjNWvZafKVLGP79xwyfO5qZZ13MtVc/Z/p3CCEs5XxRlqtrMPgfj5iCiocesjq8gAtEwt8KNHH4vDGwzd0xSqnKQC0g6tvTbSso5Ibcb5k1ZjB3Zk9lTNpV3J+Ryea/PXf1E0KEhqsFkrObpjKuy0D4+GMYP96iyIIjEAl/BXCmUqq5UqoKMACY5nTMNOBW29+vBebGwvh9Zs4UXpr1PoerVOOhPkN4vvvdaFUpqhd2CBFJ3C2QfO78AXDJJWY8f230tCP3O+HbxuQHA7OA34FJWuvflFLPKaX62g4bA5yqlFoPPAqUK92MOm+9xaD/fcLUtt3pd/NrTD2nKyDNz4QIJ+4uvhrUqQFffmn2v+3fHw4fDnFkwRGQOnyt9Uyt9Vla6zO01i/aHvu31nqa7e9HtNb9tdYttNYX2Ct6ooljZ8xXrn4UHnnE1NePGUNy7equa/GFEJbyuECycWMYO9aM548bZ1GEgVKGykgAABLXSURBVKXCdWQlLS1NZ2dnWx2GTxw7Y/Zes4j3pr7CwhZpFIyfyFUXNLc6PCGEB86l02W6bGoNLVpAq1Ywfbq1gfpIKbVSa53m6jnplhkAjhM/N+bOZHPtRtzTN5O6czdJwhcizHnssqmUWRH/0UdmWCcxMbTBBZj00gkAx4mf6scK+SupEUfjq0Z1m1UhYkafPnDkCMyda3UkfpOEHwCOEz+Jx45yOL5quceFEBGqc2fT8ypChnQ8kYQfAI4TP4lFhRRWqSbVOEJEi6pVTa+dSZNg926ro/GLJPwAcOyMmVh0FGrUkGocIaLJM8/AwYPw+ONWR+IXSfgBkpGawuLMbtQpOco1nWTfWSEihWNJdfqIua43MT/nHHjsMfjPf2DBgtAHGSCS8AOpuBiOmit8IUT4c+6lk1dQyLApq1wn/eHDTcPDe++FY8dKv97rm0UYkYQfSH//bT5Wr25tHEIIn7jqpVNYVMzIWS7aKSQmwnvvmc2L/v1v5n8+g8/en0LVDevQWnt+swgTUodfAR4XaAAcMpsgyxW+EJHBXem025LqK64wW5K+8gpdeIUutof73zCCFU3alL5ZhOuQriR8HzmupoUTt37Zf+5l3pp8thUUcsHxPUwEucIXIkIkJyWQ5yK5eyyp/vJLmD+fOz5eQq3Cg7w54w1a7v6LFU3aAB7eLMKADOn4yN2t3/hlf5WO/x3aUwDA8l1HLYhQCFFRHnvpuFOlCvTowdrzOpN1TheOxsXTpGBH6dPhvP5GEr4PsnLyXF4FQNldXBKKjgDw5W97QhCVEMJfjiXVFW1wOLRnS6pViWdrrQalCT/c19/IkI4X9qEcXyQeMwn/r2PyPipEpPDYSwf3c3f2r9k1qRFN9u8kxdW8XpiRhO+Fq6EcO0XZK/xE2xV+jTq1gh+YECLo3M3dgcMbRfc0mDCBxZndrAzVJ3Ip6oWnCZgbL2paZvyvTuEBAG66vE3Q4xJCBJ9PZZunnw779kFBQYijqzhJ+F64m4BJSUrghYy2Zcb/rtz0I4cbptCj94WhDVIIERQ+lW02t7VA37QpBBH5RxK+F95m8e0tFTY9fB4d168k8Y5boZL8WIWIBu4u+Mo8Lgk/evg8i//ll1BSAjfdZEmcQojA63p2PZTTYwozll/aSuH0080TG8N/51ZJ+D6wX8W/eX0HAB6ZmFu+b8bnn8N555mt0IQQES8rJ4/JK/Nw3gTW/nlpK4VNf5uk//33oQ6xwiTh+8hjk6XVq+Gnn+Dmm60OUwgRIJ4q9OxKJ3CvucYk/H37QhTdyZGE7yOPs/Wffw5xcTBwoEXRCSECzdcWCdsKCuHaa6GoCL75JshR+UcSvo/c/ePnFRSy6pNJrDytLVl5RSGOSggRLL62SEhOSoDzz4cmTWDy5CBH5R9J+D5y949fregIrXZtYkmDlmHfGlUI4TtXFXrOSiv2lDLDOrNmwYEDIYqw4iTh+8jdP36bnRuorEvITT7LfR9tIUTEcVWhd9NFTd1X7F17rdkAado0K8P2SFor+Mj+j2rvqaG1pse6ZTy68HOKVSVyG5m6/HBujSqEqBhvfXbK6NgRTjsNxo0L2/JsSfgVkJGaQkaHZJgxgzWDHuXsbevYWDuZ+67KZE/1JCC8W6MKIYKoUiW49VZ4/nnYssWM6YcZGdKpCK2hWze48kqaVDpGZt8hXH7X/zGr5cVA+LdGFUIE2W23QeXK0K8f7N1rdTTlSMKnAhsRHz8O8+fD7bdTfeM6LnrmERrWqVHhPtpCiCjVvDlMmQKrVpmLw/x8qyMqI+aHdLy1Py1D29bYtWgB8fEVG98TQsSGPn1MPf5VV0HXrjBnDjRsaHVUgFzhV2zXeu28yFoIIVzo0QNmzDAN1bp0gbzwKNeO+YRfoV3r7QlfObdTEkIIp+HhH2HhO5+ZZH/ppfDXX1aHJwnfp/anziThCyGcuOq3dc/mRH54bzzs3m2SvsUtlGM+4Vdo1/oVK8zH+vVDEJkQIpK4Gx7+1/YaprHa/v0m6a9fb1GEkvArtmv9iy9CvXowYEDI4xRChDePw8PnnQfz5kFhIXTuDGvWhDg6I+ardMDH1XTZ2aZPxssvQ2JiaAITQkSMWgnxFBSWb6BYOjzcvr0p6+7e3Vzp//QTpIS2yi/mr/B99vrrUKsW3Hef1ZEIIcJMVk4efx87Xu7x+Eqq7PDwOefA3LlmeGfo0BBGaEjC98XBg5CVZfpj1KxpdTRCiDAzctZaiorLl23XqFa5/OhB69bw+ONmW9QFC0IUoeFXwldK1VFKzVZKrbN9rO3muGKlVK7tT/i2knMnKwuOHIEbbrA6EiFEmMnKySPPzfh9wWE3e2RkZkLTpvDAA2YFf4j4e4WfCXyvtT4T+N72uSuFWusOtj99/Txn6I0fD82amW54QghhYy/FdMdteXdiohkm/uUXGDUqSNGV52/CvwoYa/v7WCDDz9cLGZ/75+zcaZZGDxwo9fdCiDI87XvrtZniNdeYCdzhw6GgIEgRluVvwm+gtd4OYPvorkC9mlIqWym1TCnl9k1BKXWP7bjs/CA2HfK4Ibmz//4XiovhxhuDFo8QIjJ52v/CazNFpUzVX0EBTJwYhOjK85rwlVJzlFK/uvhzVQXO01RrnQbcALyllDrD1UFa64+01mla67R69epV4OUrpkL9c774Atq1M7PrQgjhwN2QTUpSgm+NFdPSzCTuuHEBjsw1rwlfa32Z1rqNiz9TgZ1KqUYAto+73LzGNtvHjcB8IDVg38FJKPeurDUZv83jWN62so9v3AhLl8pkrRDCpQqt1HdFKbNpypIlIVmB6++QzjTgVtvfbwWmOh+glKqtlKpq+3tdIB1Y7ed5/eL8rpz+58+8Nf11bt60pOyBEyaYj7KyVgjhQoVW6rtz440m8YfgKt/flbYjgElKqTuBv4D+AEqpNOCfWuu7gFbAh0qpEswbzAittaUJf2jPlmV64N+77L8AXN60+omDtDbVOZdcYvapFEIIF/zeFyMlBS67DD77DJ55xmyVGCR+JXyt9R6gu4vHs4G7bH9fArT15zyB5rghed3ff+aSP38GoFVS/ImDVq2C1avhgw+sCFEIEUtuuQVuvhmWLYOLLw7aaWJ2pW1GagqLM7sx9dAiSEqChAQ4fPjEAV98Yfam7N/fuiCFELGhVy/zMcgrb2M24QOmY93XX8P995uWCYW2ydySErPsuUcPqFvX2hiFENGvbl1o2RIWLQrqaWI74b/6KlSrBg89ZK7w7Ql/yRKzO43U3gshQiU93eSekpKgnSJ22yPv2gWffw6DBpke94mJJ4Z0xo83n/eNvC4QQojIkJWTx8hZa9lWUEhyUgLvNjmHc/d9YkYeWrcOyjlj9wp/6VIoKjItE+DEFf6xYzBpktlxvkYNa2MUQkQlV6v9n9xl68S7eHHQzhu7CX/lSlP+1KEDWTl5/LznKEt/3cJj97wGe/fKYishRNC4Wu3/+ykN2Vc9Kajj+LGb8LOz4ZxzyFq7j2FTVrGfeKoVHeOSFf+jIOEUptaXVgpCiMCyN2102U5ZKVYkny1X+IFS2iHziensXbiUP5u1Kn2nPRJfldqFB+ixbhkzWqbz6lxrd5cXQkQXx2Ecd/5o0R42bDBdeoMgZhK+4w+74cHd1DlUwLhjp5b+8AsrV6VZwXYSi44ytXUXj13whBCiojy1UgbTg6ftdb3NJ0G6yo+ZKh3HH3bbHaZJ0U/1ziBOKYq15kjlKgDknVKPFY1bu9+4QAghToKni8iUpASG9mzJpa3rwrPNzJ63QRAzV/iOP+w2O9ZzXFVidf3mFGtNQnwcR+JNwv+mdWeqVYn3vdudEEL4wFMr5aE9WzJy1lqaPz2H9EFjyOrQIygxxEzCd/xht9uxnnV1m3I0vmppd7vKNUzjtKUX9qx4tzshhPDCXSvlrmfX831DJj/FTMIv/WFrTZud61nVsEVp3+qM1BRufO0xeOopxr51lyR7IUTAuWulPG9Nvu8bMvkpZsbw7Ul87MSF1D28ny3NW5W9kj//fPNHCCGCxFUr5Ucm5ro8NhiFIzGT8MH2w95cFYAh/7oR5EpeCGGx5KQEl6WawSgcicohndJ6+8wZpI+YW3YsbOVKiIsz+9QKIYTF/N4msQKi7grfXm9vHxOzT4CAbVhn5UqzIXmClF0KIaznuCGTvZGafW4x0KIu4bta3GCfAMlIrmx2lOnXz6LohBCiPL+3SfRR1A3puJvoOLx9J1x+uemGee+9IY5KCCGsF3UJ39VER80jh/hy8jPwxx8wbZpU4wghYlLUJXznCZDqRw8zdvKznLVzE0yZAt3L7bkuhBAxIerG8B0nQPbm7+OzaS/SfvsfVJo0CXr3tjg6IYSwTtQlfLBNgLSua7Yo3PSL2crw6qutDksIISwVlQkfrc3Whf/7H4wZI7tXCSEEUTiGD8COHfD11zBsGNxxh9XRCCFEWIjOhL9nj/mYmmptHEIIEUaiM+Hv3Ws+1qljbRxCCBFGJOELIUSMiM5JW0n4QogIlJWTF9SeOpLwhRAiDHht/BgA0TukU7ky1KhhdSRCCOETT40fAyXqEn5WTh5Z369id5UapL8yLyj7QgohRKC5a/wYyJ2voirh22+JKu8voKBajaBuBiyEEIHkboerQO58FVUJ335LlHTkIAUJpwDB2wxYCCECKRQ7X0XVpK391mfUhddSueR4uceFECJchWLnq6hK+PbNgBc1Ty33uBBChLtg73wVVUM6odwMWAghIk1UXeGHcjNgIYSINH4lfKVUf+AZoBVwgdY6281xvYC3gThgtNZ6hD/n9SRUmwELIUSk8XdI51fgamCBuwOUUnHA+8A/gNbAQKVUaz/PK4QQooL8usLXWv8OoJTydNgFwHqt9UbbsROAq4DV/pxbCCFExYRi0jYF2OLw+VbbY+Uope5RSmUrpbLz8/NDEJoQQsQOr1f4Sqk5QEMXTz2ptZ7qwzlcXf5rVwdqrT8CPgJIS0tzeYwQQoiT4zXha60v8/McW4EmDp83Brb5+ZpCCCEqKBRDOiuAM5VSzZVSVYABwLQQnFcIIYQDpfXJj5wopfoB7wL1gAIgV2vdUymVjCm/7G07rjfwFqYs8xOt9Ys+vHY+8OdJhFUX2H0SXxds4RoXhG9sElfFhWtsElfFnWxsp2mt67l6wq+EH46UUtla6zSr43AWrnFB+MYmcVVcuMYmcVVcMGKLqtYKQggh3JOEL4QQMSIaE/5HVgfgRrjGBeEbm8RVceEam8RVcQGPLerG8IUQQrgWjVf4QgghXJCEL4QQMSKqE75S6jGllFZK1bU6FgCl1PNKqV+UUrlKqf/Z1itYTik1Uim1xhbb10qpJKtjslNK9VdK/aaUKlFKWV4+p5TqpZRaq5Rar5TKtDoeO6XUJ0qpXUqpX62OxZFSqolSap5S6nfbv+NDVscEoJSqppT6USn1sy2uZ62OyZFSKk4plaOUmh7I143ahK+UagJcDvxldSwORmqt22mtOwDTgX9bHZDNbKCN1rod8AcwzOJ4HHltwR0qYd7q+1Ogl9VBuHAcGKK1bgVcBNwfJj+zo0A3rXV7oAPQSyl1kcUxOXoI+D3QLxq1CR94E3gcN43arKC1PuDwaXXCJDat9f+01vZd35dh+h2FBa3171rrtVbHYVPa6ltrfQywt/q2nNZ6AbDX6jicaa23a61/sv39ICaJWb5DkTYO2T6Nt/0Ji99HpVRj4ApgdKBfOyoTvlKqL5Cntf7Z6licKaVeVEptAW4kfK7wHd0BfGt1EGHK51bfojylVDMgFVhubSSGbdgkF9gFzNZah0VcmDY0jwMlgX7hiN3T1lPbZuBfQI/QRmR4ayettX4SeFIpNQwYDDwdDnHZjnkScws+PhQxVSS2MOFzq29RllKqBjAZeNjpTtcyWutioINtzuprpVQbrbWlcyBKqT7ALq31SqVUl0C/fsQmfHdtm5VSbYHmwM+2nbgaAz8ppS7QWu+wKi4XvgBmEKKE7y0updStQB+guw7x4owAtOAOFWn1fRKUUvGYZD9eaz3F6nicaa0LlFLzMXMgVk96pwN9bQ0nqwE1lVKfa61vCsSLR92QjtZ6lda6vta6mda6GeaX9NxQJHtvlFJnOnzaF1hjVSyObJvMPwH01VoftjqeMCatvitImauuMcDvWus3rI7HTilVz16NppRKAC4jDH4ftdbDtNaNbblrADA3UMkeojDhh7kRSqlflVK/YIacwqJEDXgPOAWYbSsZHWV1QHZKqX5Kqa1AR2CGUmqWVbHYJrYHA7Mwk4+TtNa/WRWPI6XUl8BSoKVSaqtS6k6rY7JJB24Gutn+b+Xarl6t1giYZ/tdXIEZww9oCWQ4ktYKQggRI+QKXwghYoQkfCGEiBGS8IUQIkZIwhdCiBghCV8IIWKEJHwhhIgRkvCFECJG/D93auf9SR44sAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide_input\n",
    "#id img_momentum\n",
    "#caption An example of momentum\n",
    "#alt Graph showing an example of momentum\n",
    "x = np.linspace(-4, 4, 100)\n",
    "y = 1 - (x/3) ** 2\n",
    "x1 = x + np.random.randn(100) * 0.1\n",
    "y1 = y + np.random.randn(100) * 0.1\n",
    "plt.scatter(x1,y1)\n",
    "idx = x1.argsort()\n",
    "beta,avg,res = 0.7,0,[]\n",
    "for i in idx:\n",
    "    avg = beta * avg + (1-beta) * y1[i]\n",
    "    res.append(avg/(1-beta**(i+1)))\n",
    "plt.plot(x1[idx],np.array(res), color='red');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It works particularly well if the loss function has narrow canyons we need to navigate: vanilla SGD would send us bouncing from one side to the other, while SGD with momentum will average those to roll smoothly down the side. The parameter `beta` determines the strength of the momentum we are using: with a small `beta` we stay closer to the actual gradient values, whereas with a high `beta` we will mostly go in the direction of the average of the gradients and it will take a while before any change in the gradients makes that trend move.\n",
    "\n",
    "With a large `beta`, we might miss that the gradients have changed directions and roll over a small local minima. This is a desired side effect: intuitively, when we show a new input to our model, it will look like something in the training set but won't be *exactly* like it. That means it will correspond to a point in the loss function that is close to the minimum we ended up with at the end of training, but not exactly *at* that minimum. So, we would rather end up training in a wide minimum, where nearby points have approximately the same loss (or if you prefer, a point where the loss is as flat as possible). <<img_betas>> shows how the chart in <<img_momentum>> varies as we change `beta`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAHiCAYAAAAuz5CZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3iTVRvH8e9pKTRllQ0tG9kgFCpDcLBBVkV8AVFBRUDEiSjgHkABxYELRMEByrSioAwBlVFZBZElG9qyocxC13n/eJKSpkm60qRp78919Wrz5El6CumvJ+c55z5Ka40QQgghhBAi83w83QAhhBBCCCG8jXSihRBCCCGEyCLpRAshhBBCCJFF0okWQgghhBAii6QTLYQQQgghRBZJJ1oIIYQQQogskk608ApKqSNKqY6ebocQQoiMSWaLgkA60SJfy60gV4ZJSqlz5o/JSinl4Ny7lVIpSqkrVh+DXN0mIYTwdnkks8fZ5HW8OcPLurpdwrsV8nQDhPBSQ4EwoAmggZXAIeBzB+fHaq0ru6ltQggh0sp0ZmutJwATLLeVUm8Ad2qtz7qlpcJryEi08Ca3KaV2K6UuKKVmKaX8AZRSPZRS25VScUqpDUqpW83HvwWqAj+bRxNeNB9foJQ6qZS6qJT6UynVMBttGQS8p7WO1lrHAO8Bg13yUwohRP7g9ZltHq1+CPg6G99T5HPSiRbeZCDQBagF1AFeUUo1A74ChgFlgOnAEqVUEa31Q8AxoKfWupjWerL5eX4FagPlgW3AHMs3UEqNMQe73Q+rtjQEdljd3mE+5kh5pdQppdRhpdT7SqmiOfh3EEIIb+DNmW1xB1ABWJTln17ke0pr7ek2CJEhpdQRIFxr/bn59j3ANGAFcFZr/arVufuAoVrrP8yPG6K1XuXgeQOBC0Cg1vpiFtqTDDTUWu81364N/Af4aJtfKqVURaA0sBeohjGisUdrPSyz308IIbyJN2e2zeO+BHy11oMz+71EwSEj0cKbHLf6+igQhNEpHWUz8lDFfF86SilfpVS4UuqgUuoScMR8V1YXjFwBSljdLgFcsRfGWuuTWuvdWusUrfVh4EWgbxa/nxBCeBuvzGyr720C7kemcggHpBMtvEkVq6+rArEYIT1eax1o9RGgtf7efJ5tQD4A9AY6AiWB6ubjCuyuyk7zYfU8uzAWqFg0MR/LDG35fkIIkY95e2b3Ac4DazPzw4qCRzrRwps8qZSqrJQqDYwD5gFfAMOVUi2VoahSqrtSqrj5MaeAmlbPURy4AZwDArBagQ3GqmzzXDy7H1anfgM8r5QKVkoFAaOA2fYarYwSd1XN7asChAM/5fQfQwgh8jivzGwrg4BvnI1Wi4JNOtHCm8zFmE93yPzxjtZ6C/A48DHGPLkDpF1xPRFjMUucUuoFjCA9CsQAu4HIbLZlOvAzsBP4F1hqPgaAeRTkDvPNZsBG4CqwwXz+09n8vkII4S28NbNRSgUD7c3fXwi7ZGGhEEIIIYQQWSQj0UIIIYQQQmSRdKKFEEIIIYTIIpd0opVSXymlTiul/nVw/0Cl1D/mjw1KqSZW9x1RSu1Uxu5FW1zRHiGEEI5JZgshRM65aiR6NtDVyf2Hgbu01rcCbwMzbO5vp7VuqrUOdVF7hBBCODYbyWwhhMiRQq54Eq31n0qp6k7u32B1MxKo7IrvK4QQIusks4UQIudc0onOoseAX61ua2CFUkoD07XWtiMe6ZQtW1ZXr149l5onhBC5Z+vWrWe11uU83Y4skMwWQhRYzjLbrZ1opVQ7jEBua3W4jdY6VilVHliplNqrtf7TzmOHAkMBqlatypYtMhVPCOF9lFJHPd2GzJLMFkIUdM4y223VOZRStwIzgd5a63OW41rrWPPn08CPQAt7j9daz9Bah2qtQ8uV86ZBHCGE8D6S2UII4ZxbOtFKqarAYuAhrfV/VseLWrb6VEoVBTpj7CQkhBDCQySzhRAiYy6ZzqGU+h64GyirlIoGXgf8ALTWnwOvAWWAT5VSAEnmVd0VgB/NxwoBc7XWv7miTUIIIeyTzBZCiJxzVXWOARncPwQYYuf4IaBJ+kcIIYTILZLZQgiRc7JjoRBCCCGEEFnkiRJ3QniViKgYpizfR2xcPEGBJkZ3qUtYSLCnmyWEEMIOyWzhLtKJFsKJiKgYxi7eSXxiMgAxcfGMXbwTQEJZCCHyGMls4U4ynUMIJ6Ys35caxhbxiclMWb7PQy0SQgjhiGS2cCfpRAvhRGxcfJaOCyGE8BzJbOFO0okWwomgQFOWjgshhPAcyWzhTtKJFsKJ0V3qYvLzTXPM5OfL6C51PdQiIYQQjkhmC3eShYVCOGFZiOJopbesAhdCiLxDMlu4k3SihchAWEiw3ZCVVeBCCJH3SGYLd5FOtBDZ5GwVeFYCWUZGhBAi90lmC1eTTrQQ2eSKVeAyMiKEEO4hmS1cTRYWigIpIiqGNuGrqTFmKW3CVxMRFeP8AbNmwe7daQ65YhW41DQVQojMyXJu25DMFq4mnWhR4FhGEmLi4tHcHElwGMjR0fDoo9CkCbzwAly6BLhmFbjUNBVCiIxlObftkMwWruaSTrRS6iul1Gml1L8O7ldKqY+UUgeUUv8opZpZ3TdIKbXf/DHIFe0RBVdmRiqyPJKQmGh8rlcPpk41Ps+ZQ1jTICb2aUxwoAkFBAeamNincZYu6UlNU+EJktkir8js6LIrRoDDQoIls4VLuWpO9GzgY+AbB/d3A2qbP1oCnwEtlVKlgdeBUEADW5VSS7TWF1zULlGAZHauWpZHErQ2Pr/wAtSvD08+CQ8+CNOnE/bxx4SNaZ/tNo/uUjdNm0Fqmgq3mI1ktvCwrMwvdtUIsKPKHZklmS2suWQkWmv9J3DeySm9gW+0IRIIVEpVAroAK7XW580hvBLo6oo2iYInsyMVWR5JSEkxPvv4QIsW8PffMGOGMUe6SRMoXhyWLs1Wm10xMiJEVklmi7wgK6PLWc7t8eNhwAB45x1YvBj27YOkpBy3WTJbWHNXdY5g4LjV7WjzMUfHhXDKXomhzI5U2BtJUBijIG3CV6crV7Ry1wk6Ac/N38GmmGDj/scfZ1Xp2nTs2w6uXCGpVy92vhxOyFujM91ey/fI6ciIELlAMlu4VE4yG7KW2xFRMbSe8iGBV+Io8sMPqecn+xXmcJlg9pUMJrJZO0Kfe4zet1XLUpsteS2ZLcB9nWhl55h2cjz9Eyg1FBgKULVqVde1TOR5tkHWrl45Fm2NSXcJMDDAjwvXEtM93nakwnpHq5i4eBQ3X3S2lxMjomKYvnwfnYBkpVLv33L0PKs2nqGj+XG7ytci5O0X2XsylnrTp4JSqW1/Y8ku4uJvtismLp7n5m1ny9HzvBPW2DhomTKi7P1KCOF2ktkiR6xzu6TJj6sJSSQmGy+VrGY2ZD63AcYu3sma5GQWN2zHWx2G0vBiDP2LXSZu83aGbFzILScP033fOmKXfs6/gx+n0RsvQKlSadpt73uky21R4LmrOkc0UMXqdmUg1snxdLTWM7TWoVrr0HLlyuVaQ0XeYm9F9pzIY3YvAWptzE2rcPksShtTMBzNVQsLCWb9mPYEB5rS9QCsLydOWb6PhETLJUCVev/3fx/nrCqc+pgH+o9nfuOO1PviA3jsMThwgF9XbeeNeVuIu5aQ7vtrYE7ksZuLaEaMgHbtICH9uUJ4gGS2yDbb3I6LT0ztQFtYZ7Y1Z/OLM5PblikiPlqjlQ/xhf3ZUq4WLwWE8M6dg1PP/7RVXw6XqkSjjyZA5crQtSvH7+nDpWFP0veXL6l17ni675Eut0WB565O9BLgYfOK71bARa31CWA50FkpVUopVQrobD4mCoDsVtKwO+wFXIxPZFqzANZNH8KgbUszNVct9sI1msTuY8yar5iy9APKXI0zjpsvJ549e4mmsf+l+77JWpPke/NCztUiAbzY7Rk+aDPAqClduzbdOoWwfVIY/717L712/wGAb0oyQZdOpz7flOX74MQJmDkT/vgD3n7b2T+ZEO4imS3SyUklDXsuxidma36xs2kglvt8dArJPje7OMk67V+OL0PDGNh/Avc88pExd/r8eXz+jiTsn1U8t34usxa8gU+K8TOUvnYx9XGpuR0ZCcOHQ4x0qAsyl0znUEp9D9wNlFVKRWOs3vYD0Fp/DiwD7gEOANeAR8z3nVdKvQ1sNj/VW1prZ4tdRD6R00oaALecPUbn/ZEsq9uGI6WDCQo00fG3OZCcxBtHV/PGi58aiwEdWbWKDTOGUCnuFAk+hdBKERK7l6d7vUiby8ehz5dELf2VgITrXPXz51Dpm+3yVSpdKKMUszsNZk/j1hQ7fpiAhHiKJcTz8NZfCNu1hmV12zBj8TvceXgbYQ9P5d+Ktxg/31dfGQteOneGCROgRw9o2TIb/6pCZI5ktsgqV1TSsBUUaDLmF9csBgEB4OeX6cedPXuRRicPsLdcda4WCQCgnl8CdU4coOLB3RRLiCfFanqcJbM/at2PfjtXcq5ooNHWanVpU6QhsWXvRZsLLXXdt57PIyYybNNiWh/9hzuPRPF5y/sIv2swKEXKsePQazScOQPz58Pnn8P//peptov8RWnbjoAXCA0N1Vu2bPF0M0QOhLy1wu5cuOBAE+utSsa1CV9NjINAfmf5Jzy4/VcAIqs04tfG7Ri3/HPOlihL8IUTsGIFdOpkvwF//QVdunCpUhUmNOzBspotqHvmCF8tfJPiCcb3iy9fkZN3dGRSodqsDW7IdT9/wLjceF/zYBZtjWHPO90AqP7SL/j5KFCku2z5+qrp9N+xgpm3hfHUxnlcLmziaKkgej88laBAE39NHwK1a8OiRXDrreDvD1FRxh8VnC9KFN5HKbVVax3q6Xa4k2S298tsZoPz3LYW4OeD3+VLrJw1Er8qwZTatCE19+zSGtau5egH0ym9/GeK37jGDV8/tgXXo0rcKSqbr/IBRJcoz5sdh7Kydqs0mW09Qu4os31Sklk7YyhVL57ign9xooLq0v7QFuY07cpb7R9n8YJXaHjhOMyda1T/2LQJRo+GSZNAKcnsfMZZZrtrYaEQqSKiYuyGMWS+koYGgi+d5kDpyixq3IF+O1bw5rJpAAwJG8t3817lxsT3CLLXid6yBbp3h6pVKfHnn7SKSWT5z7vY7F+Mvg9Ooe2R7URWbcyh4NpMvO9WugI77QRiaLXSvHB8AgeTCxMcaOJaQpLdn+uPGs15ZOvPPLVxHr/Uu4Olddvw2U/hDNu2hDt63wnHjsG77xJx6Aq/d36aaTNfYH6nByn8sfHzZHb0RwghckNWMhvs5zYAWlPjQizJyocrRQK4UjiA11d/SenL5/HZdY7osP5U/i3C/hXErVuNWv1r11KteHG23d6Rr4rXo3nMHm6L3k1UUF1+CO1B6/s7EVevERMiTxMbF0+wTWZbd24dZXaKjy+vdB5By+P/8kWLe4nzL87oP7/hycgFtDu0zZiSt3Ah9OzJTxVvJfmpp+kzZQq/rtrOxnHhLNhxWjK7gJCRaOF2zkYp7I1qRETF8MmSKO788yfqx58l8HQMwZfOUOtcNGtqhTKszyugNa2O76T4jWusrN2KF/+YzbC/F+N77KixaMTi33/hrrugRAljNNp8n6M22WuPIzXGLLU7X9s/8To7PhxAoq8ffZ6ayX7f4nz78wRaHY6iUJMmcOQIPy3ZyJifjXmEr6+aziNbf2bQg+H8Uzsk06M/wjvISLTwNlnNbEh7Bc2Si20PR/Hd/FfTnftZy75cMBVn3NpZMG6cMbprmYpx/Di8/DJ8+y2ULQuvvw6PPUabDzfmWmaDMVhjqQa1Zu8ZYuPiGb0jghG/zYQXX4RJk25OcUlI4qkNPzBq3RzW1GzOiN5jiS/sn+12ibxFRqJFnpJm5ELrNGXdHFXSCJv2KqyeBSVLst9UhiMlKxBZtTELG3UwTlKKyKq3pj5mbtNuDI9cBB9/DOHhxsH9+6FjR2O6xO+/p+lcu2I3rKBAk91QL1OuFEU+mEqRKlVY0bu3cXBkE2jQwNi4ZexYJq8+nDpyMemuQdx1aCvjl0yl66MfQ5H0lzezukuXEEJkl7O8cVZJwzLyaumE1z99GIAxXUbil5JE8RvXuF6oCHNCunHD149a52PoN2FC6tU5PvoIpk41/k689BKMHQslSzptkysy23GHtzscfy31b0fqAkqlmNZmAGeKlmL8ik/5/odxvNjtac4WLcX5AOftFd5NOtHC7ayDa/aCNzAl3aDfA+EEmvzsX+46ftwYhRg5EqZNY5fNAhd7oktWYFWT9nSePNnYZbB5c+jQAZKTYc0aqFnTYZtsjzuSUf1qMOZPt6tXjjZ7GxC7MZ6gPZZNAaoYfySefx6GDiX2812pj7nu588L3Z9jwZyXeGX1TMZ0ezpL7RJCCFdylI8OM9vixg3YupXpJ37n1G+rue3oTs6ZSvBDU/ubXE7rN5p+fdsao81z5hid54EDjd0Hq6XdFCXXMzt8tf05zVVuVni07Rj/0LQr5wNKMm3JZFZ8NZLzphI80/MF/qrRTDI7n3JXiTshUo3uUheTny+B8Ze4+/BWWh7/F5OfL2/0apjmPEs5pS/7PEVSimZ51weB9NuulgrwMxaIWDH5+XLj08+NKhcPPGBM4bh0yVhsWL++wzbZPoejURZ79asXbY3hvubBaco1WRazWJ83dvFOozTUsGFw/jxUr54uYLcF12d6yz70/2cFvQ5GsnTW0zy5YV6G7RJCCFdzlI+OMrvGmKUMHfExiWXLQZs2NPpoAi1unObPRnfwfI9RDjP7hW714dVX4ddfjbJzmzbBd9+l60A7a1OuZrYNex3jFXVaM7D/eJbVuZ1rfv58Pf91noucz+hOte22S3g3GYkWbmd5R79v3PjUY1N/mkRQUidW/3cbrx+E45cSUEDgtYsM2PEbPzW4i1ciLzAxKMbutqv2VkP3DAlm6YQvaNTvHsrGnuK5xyZzD+UJc9KmzK6otlcHNT4xmTV7z6SrLmLvvDd/3pXhiMjndz/EwLP/8v5Pk/FNTKDh6UMklq9Ag5eflQUqQgi3seSN9e6r/n7GGJy9Hf6K37jKq3Pf4ZRvADHvvU/LB3tSvHx5egA9zM/pqIJFRFQMU6L8iK3yAEG/X2Z0oRi7eefuzLbegCujkex6fbsyvlFzLpy+wPtrPuOZP76BN87DN9+k7owo8gdZWCg8ImJbNPW73UG187GsqtWC22J2U+GKUW72UpGi7CtbjeuFClPq+mUanTpIp0c/YX+5aplenGG93XaxG9coceMKsSXKY/LzzVQx/4w4WpCigMPh3TM8z5alBJNlAUvqHwR1Bm67zVitfscdsHYtLFtm1JQWXkkWFgpvZFsnGhyUiNOaaUsm023feu4fOJnTDUOynNnW8lJmm/x803WY7ea2dVu1hk8/heeegwoVjFJ4jz4KxYrl6OcR7iMLC0WeYD3y0Dx2LwtPH+HlziOYE3IPaE3li6e4LXo3t0XvpsaFGAISr6OB6S36sL+ccTkvM4szbMP+SpEArpgX51lGFHIayJmdj+foPFv2RkQMwTB9Opw6BU8+aXSk+/aFdeuMmtJCCJFLrDPbx7xZScVLZxkRuYDCyWk7u34pSZS7coFKl89S+9xxJt/5MFHB9VDZyGxreSWzfZXK1Eh2OkoZ2R0aCqNGwTPPGHO+n3gCnnoKKlXK3g8k8gTpRAu3sA3Jh7cs4VKRovzYsJ1xglJEB1YkOrAiPzZyHEiZWZyR0Zazrlglba8Oqr35eA7rpWalXY8+evPrpUuNed7duxvbzgan/8Mihf6FEDllm9mWHVqfX/cd/9u5ipPFSqO5Oa852ceHs0VLcbBMZX5qcBeft7wPyB+ZbTsCna22tWxpDH5ERsJ77xkbs7z3HgwcyOouA3j1kA+xF69LZnsZ6UQLt7AOyQqXz9Jt33pmN+/JtcKZX7Gc2QV1GYWaK1ZJZ3Y+nr3zrt5ISnfJMtPtqlzZ6EjfcYcRyq+/DoMHp26Xm5WteYUQwhF7HdsS16/Qc89fzG3SlXFdR2b4HPkls0d3qZs67zvHbWvVChYsgIMH4YMPSJr5Je1nzSK0cADfNbuHyXcNlsz2ItKJFm5hHZIPRS3DR2u+btYjzTn25tel7k6YhXfnzqZQuLKyhe3ixsyeZ+/SZZba1bSpUef6mWdg6FCYMgXeegv+9z+Hi2dccTlUCFFw2OvY9t35O6akG3wXcg9QcDLbIke5batWLZg2jZ6B7Wm6eTXtD25iRORC/q7SmD9qNpfM9hLSiRY5Zq/2pu1CC0tIFkm8wQPbf2PVLS2IDqyIr1KkaJ16HmR+tbUjjqZQlArw4/WeDT0eTFlZVe5wakaLFrBhA/zyi7Gb14ABMHEi9WqHUa1QYcJ/m8aMFn34rpmxYEYK/QshLLKS2RaFkpN4eNsvbAuux54KNVM7yZD/Mxsyn9tZnU63N7Ewe5p2ZVGjDiyb9RTjl39Ml0c/ITYuV38c4SJSnUPkiLMFIRaWFcxLIw8yfM23DNu0mAH9J7D9lhCXrLp21C5vnxfsaMQ63WrwTrUJ+2+dUV/14MHUc08VK03b4V+S6OuXYVUT6zJVvuYFRFkZSRKZJ9U5hCdlJbOty7cN3rKEN36fwcYPZtP6mUG50q4Ck9lWP5v1lurNo3ezYM5LfN28BzP7PiuZnUfkenUOpVRX4EPAF5iptQ63uf99wLyCjACgvNY60HxfMrDTfN8xrXUvV7RJuJ69kMtoQQiAvnaV8l98ysa/F+F/7gzL6rbh2K0tmNi1Xq79smf2sl1e5mhqxpzIY6klmGLi4hkbsQv63Imau4od73xA+dPR7KhYm89+CueevetY0bSj00uOjhYQyXzq/Esyu2DIbmZbqk5M7NOYKcv3ce3EKZ7f8D2nW91J66cfzpW2FqjMXrwz9f5rCUmpX2+t3IBvmnVn0NZfqDXyMYffRzI778jxSLRSyhf4D+gERAObgQFa690Ozn8KCNFaP2q+fUVrnaWCiTKq4X6O3mE7Hc1IuM7A7csY9vdiyl2Lg44djYVwbdu6o8leL7M1psHYfvdGUkrq/4fSKayaOYLr/gHs//l3wppVdvhY65EQezJbm1tkjqdHoiWzC4bsZLa1NPWTn3rKqHW8Ywc0apQLrc0fcpLZFkE+iayaNZKAiuUgKsruYyWz3ctZZrti2+8WwAGt9SGtdQLwA9DbyfkDgO9d8H2FGzl6h+2rVLpzCyUn8ejmn/hz+hBeWfMVe8tV54lhH8DKldKBzoKsrPqOi09M8/+jlQ+zQnvRMPY/wq4fc/rYjOZLy3zqfEcyuwDISmbbExRogosX4aOP4LPPYNgw6UBnICeZbaFKlCBgyCPwzz8QZ39itGR23uGKTnQwcNzqdrT5WDpKqWpADWC11WF/pdQWpVSkUsrejszCTSKiYmgTvpoaY5bSJnw1EVExqfc5+qVM1tpYoW3W+ugOfp31FK+t/oJ95arRd+Akhj40kS7D+uZ6+/Ob0V3qYvLzTXMsc3/+DIsatudikaLwwQdOz0sNfq0plJxEu4ObqXYhNv39Ir+QzM4nXJHZ9jQ9f5TvNn9l1KF/5hljIfNbb7m07flRTjMbzP9vtWtDSgqULm1sqjV8OHz9NezfD+aF+FUvnOD5P7/lzZWfUfXCCeqcOZL6HJLZ7uOKOdH2XiOOrmj0BxZqra3fflXVWscqpWoCq5VSO7XWB20fqJQaCgwFqFq1ak7bLMysFydYShNB+nlblt2qbAWa/LiakETn/zbSf8dy2h/awtHAijz9wJv8XKU5QYEmJspCh2yxtxq8Xb1yaRb7gHGJ1t/PhwvX0taeji/sz4/N7+HBRYu4a8QsqFrN7qKT0V3qMmHuRr78dgw3ChWmaew+tlZuQL8Hwl1aXkrkGZLZXsxVmW3NR0FZP83tUWt5dOdv3Hp0F/j7wwMPGDvrhRaodbDZltPMBihp8qNtdBDV+73D3ef20/PqESr88IOxcy1A2bIsLVuRwL3/pj6m/44VJPoWotfD73OiYjXJbDdyxZzo1sAbWusu5ttjAbTWE+2cGwU8qbXe4OC5ZgO/aK0XOvueMr/ONV6J2JlmwYM9juZtwc0gKHX8MKtnDgdgatuBTG95H2XLlpQ5WbnE3mIhSF/D1M9HUenSaVZ/+hizQnsxvv0QTH6+6SuiXL/O2bbtKLs1Ms33eWzEJ/Qc0lveALlYHpgTLZntpVyV2dadN5+UZMJ2r+XFdXOoePG0MQr6xBMwaJAxEipyLCuZbVt32+Tny8SwhoT5XzLKmm7cCPv3s7tJG94wNWLa588Q7+dP8etXuFCyDHsWraDn7be4/WfMz3K7OsdmoLZSqgYQgzFy8YCdRtQFSgEbrY6VAq5prW8opcoCbYDJLmhTgZPV8kARUTEZhjGQZmc9/8TrhP82jU1VGjG/aVcm9mnMc/O28/a6OVwvVJgOQz4npmR5QOZk5SZnq9itXwPXEpI4llKOv2qE8PjmCHrs+YvDpYM5tbIa9GvHxkJleP+oYvAv07lnXyQbuz9A66VzWVa3DXce2c6bB5ZTOWSEm3864QaS2XmAOzLbmq9SqZltceehrYxdO4v6Z46wo2JtKs77Bjp1Ah9XzPQUFlnJ7HRXFBOTmbJyP2Fj2hOREMiUc7WILX9zlHtI4ekcjFd0vbCf974exy1fhcPtM93xYwlc0InWWicppUYCyzHKJX2ltd6llHoL2KK1XmI+dQDwg0479F0fmK6USsGYnx3uaIW4cCw7Wz1PWb4v06uILZ6IXETY7j8I2/0H9+38neYPLiDiWgw99v7FR637pXagQeZkeYJtUNcYsxSA1zsOZ/Pev6h1Ppoa52Not2MN/P0zrYHW5nPfbj+Erxr24tFrJpY0uJvHNv/I4ysjWLE0ks7dW7n/hxG5RjLb89yV2dZStCYsJDh1Kkj7A5v4atFbHAmsxJO9XmJHy46s69IxB99BZJWjzLYVGxdv9zXzXeQxwASFYXGFxtS5vR/Dv/wS7rwTHs6dUoQiLdlsJR9wVO7GWZmbzJTisb70VyXuJKtmPsGK2q1YUyuU19Z8SWDCNa5WDCbx7DnuHDCyqVgAACAASURBVDaTS/7FUh+XW5uoiMxz+Loo6U/Ja5cwHT5AzfMxXPQvxoo6rdOcU/HSWf6a/hg/turN/9YvcleTCwRPT+fwBMnstNyR2Y6eOyIqhtfmb2Xel89QLCGe9o9/TiF/f8nsPMDZ6wJwWtYOwDclmQWLXqPZyf2waRM0bJgr7SxocrvEnfAwR1MnnE2pyGikODjQxMQ+jXm9Z0NMhXx4dfVMknx8Gd/uMX4N6cyGn9dxtFsfikYf5dNW93PVVDzN4ySMPc/eSnGTny+ju9ZjT2JhtlZuwIJbO6XrQAOcLFGWn+vfSffNy+DCBXc1WYgCIdcz297vfZe6qVNInljzLfXPHOG1TsMpX6aEZHYe4TCzu9TN1BTJZB9fhncbBcWLw/33w5UrudVUYSad6HzAUbg6C11HpXgebFWVI+HdWT+mvXGpqZqJVZs+pfP+SKbd3h/fqlWY2KcxCaVK07XJI7QcMZsZLfqQrHXqL7uEcd4QFhLMfc2DU+vC+irFfc2Ny4eZmW4z87Z7KZp4HWbMyO2mClGg5GpmhwQzsU9jggNNKOAWE3zSyJeiRw8x7atV1Nq2jmF/L2Juky5E1mslmZ2H5DSzAfwqB8PcubB3L4wYAV4428CbyHSOfMDRzlQZjS5kuLBlzRp46CE4dcqoEfrii+BrhHh2LkcK93L2uoD0K8Ntmfx8+WPlBMofPwiHD0Phwrne5oJApnOIXMtsW1evGnWGDx1Kc/hYyQp0e2QaV4sESGbnIa7I7NTX0FtvGTsEz5gBjz+e623Pz3K7OofwMHu1KTMzuuBwxXBCArzyCrz7rlHuaOPGdHVCs3M5UriXox3Lpizfl/pH07ae6Zq9Z9K8hsq3GQfdusEPPzhcqJLlP+xCFHAuz2xHJk40OtDTpvHMiqMUTkqgcHISa2qFcrVIACCZnZe4IrNTXx8vvwzr1xuj0TVrQocOqc8pme060onOJ7IcrtYiImDWLPjpJzh40JhLFRUFQ4fC1KlQtGi6hwQFmuyOREtVjrwjozc6mXrN6CBjq9933zWuSthsGZydKgNCiBxmdmbs3w9TpsCDD8LIkWy5Yv/qoWR23uGSzLbw9YV586BtW+jTx6gx3bChZLaLyZxoAQsWwJIlxqW/554zRi4iIowdkux0oMH5AgiRN2Rn3mU6SsGoUbBzJ6xcme5uZyMnQggPOXsWhgyBIkVgslHGWzI773NJZlsLDIRlyyA5GaZNAySzXU060QL+NW8fGhtrzIMeMAB693b6ENvFK1KVI+9x2R/NAQOgYkV47710d8m0HiHymIULjdJmGzYYHadKlQDJbG+QK290qlY1dp68cQOQzHY1mc5R0CUmwp49xtdLlxolcazmTjmT65cjRY5kd95lOkWKwNNPw7hxrJnzK68cL5L6fIEBfnZr0solYiHc7NQpePJJWLQImjUzrhzdemuaUySz8zaXZbaViKgYWl5OYP2W47wfvtrI7KsJVLp8lktFiqbOjZfMzh7pRBd0//1ndKTBWDwGcPfdHmuOcC2X/dEcPpyEiZNoPTiMh5r14N07HyYmLh4/H4WfryIx+WaVH7lELIQbaW2UNHv6aWMQZMIEGD0aCsmfd2/kyjc6lvnPyzVUuHyODr/Pp0XMHppH76bS5bMsr92KYX1ekczOAfktK+gsUzkA/v4bmjaFsmU91x6RN5UqxQPDP+GhX75g+KbF7KxYm6X17yAxRVO6iA8Ddq/kp4qN0dWqy0pvIdzp66/hkUegVSv46iuoX9/TLRJ5hGX+c6KvH3cc3c4dR7cTW7ws26s14tL5aBqeOkSwVOfIEelEF3Q7d4KPD6SkGLczOZVDFDxbfQI5cdcgeu/5g4BEY/6cKeE6kxdNpuOBTYx++WUY86iHWylEAaI1fPihMW1j3brUOv5CwM15zmO6jqTS5bNsqdyA2BLlUcBh0xZ4803WP3s7+Pt7tqFeTBYWFmRaG/PmGja8WYVDOtHCgaBAEynmEndKQ7krF/jh+7G0O7jF2Ijl5EkPt1CIAuLwYbjjDvjlF9i+HYYNkw60SMcyz3lzlUYsaXA3sSXK3zxep47RBzh40JNN9HrSiS7IVq+GTZuMYuzlyxtz6O64w9OtEnnU6C51KWJeOV773DEWf/cCtc8dY/PUL41a0idOeLiFQhQQH35ojDz36gUmEwwc6OkWiTzIabWPOnWMA//954GW5R/SiS7I3nkHgoJg8GCoXt0oyl6smKdbJfKosJBgxnRvAMDjmyMISE5g06zFtHp2sFECT0aihXCPixdvft2/P5Qs6bm2iDzLaVnD2rWNk6QTnSMumROtlOoKfAj4AjO11uE29w8GpgAx5kMfa61nmu8bBLxiPv6O1vprV7RJZGDdOli7Ft5/35gPNWeOMTdaCCe6Nq9ufFG/PmWWLePu6ubbFSsal5WF15Dc9mJRUTe/HjbMc+0QeZ7Dah8lSkCFCsbOliLbctyJVkr5Ap8AnYBoYLNSaonWerfNqfO01iNtHlsaeB0IBTSw1fzYCzltV4E0YQL8/DMsX278gjgzfjyUKwePP27cNhfkF8KpMmVgxQq47TZjNyyLihWNOrUpKfJmzAtIbnux69dh1y64/Xbjo0ULT7dIeKs6dWQkOodc8deuBXBAa31Ia50A/AA43+7upi7ASq31eXMArwS6uqBNBY9lW8/ISBg06Ga1DXu2bIHffoPnn3e4rbcQDnXqlLYDDUYnOjnZ2G5YeAPJbW+1cyckJRn5PWUKmBf7CpFldeoYZW6vXvV0S7yWKzrRwcBxq9vR5mO27lNK/aOUWqiUqpLFx6KUGqqU2qKU2nLmzBkXNNs7RUTF0CZ8NTXGLKVN+GoiosxXWjdsMOakduwIEREwaZLjJxk/3ugEjRjhnkaLfMPh689yJUPmRXuLXM9tyWyDw9+Z7PrlF+Nzs2Y5b5zI95y+/gYPhgsX4LXXPNY+b+eKTrS9t8Ha5vbPQHWt9a3AKsAyfy4zjzUOaj1Dax2qtQ4tV65cthvrzSy7D8XExaOBmLh4xi7eafxSLFxozG1evBgGDICXXzYuu9vaudPoZD/zTMZTPoSw4vT1V7GicZJ0or1Frue2ZHYGvzPZ8eefxiDI/fdDjRoubavIfzJ8/bVtC8OHwwcfwObNHm2rt3JFJzoaqGJ1uzIQa32C1vqc1vqG+eYXQPPMPlbcZNl9yFp8YjLv/roHFi2Crl2heHH44guj5NiAAUY9UWsTJhgVOJ5+OvWQy0dKRL4TERXDqPk77L7+pizfJ51o7yO57QaOMnvK8n1Zf7LTp41KHDVr8stTb0lmC6cyzGyL8HAjv4cMgcREN7fS+7miE70ZqK2UqqGUKgz0B5ZYn6CUsl611gvYY/56OdBZKVVKKVUK6Gw+Juyw7D5kq/ye7RATA337GgeKFoUffzTmRffpA9euGcf37YN58+DJJ6F0aSAXRkpEvmN5jSRruxeJjNeldKK9jeS2GzjKbEfHHUpOhgcegAsXWP32J4xecUQyWziUqcy2KFkSPvsM/vkHJk92Uwvzjxx3orXWScBIjBDdA8zXWu9SSr2llOplPu1ppdQupdQO4GlgsPmx54G3MQJ9M/CW+Ziww7L7kK37j/xt7BjXo8fNg7VqGWXrduzgaP9HaBO+mgV9R3K9UGGWdRqQeppLR0pEvmTvNWItKNBkXN0oWlQ2XPESktvu4SizHR23sL06uPeJF+D33+Hjj3n1sK9ktnAqU5ltrVcv+N//4K23YO/eXG5d/uKSWlRa62Va6zpa61pa6/HmY69prZeYvx6rtW6otW6itW6ntd5r9divtNa3mD9muaI9+ZXd3YcK+dD7YCR07py+4P4993D4vgep9vN8fA8f4t5dq5l7axdGrT2ROmrhspESkW85ey2k7n4FxuLCWLmq7y0kt3Of0x3jHLC9Olgzaj11vviQYz3vh0cflcwWGcp0Zlv76CNjIOTxx51X9xJpSEFXL2Jv96HP6msCTkTfnMphY8UlPwCeXT+XZB8fZrTok2bUIrsjJaLgcPRa8FXq5u5XAKGh8OuvUuZOCDOnO8Y5YD2KWPHSWT74+V3+K1uVwaGDQSnJbJGhTGe2tQoVYOpUYyO2r2XvpMySTrSXCQsJZv2Y9hwO7876Me25e+efUKiQcTnGjgsJxjvKPrvWMLdpN06WKAvcfKeanZESUbA4eo28978macP41VeNeqMTJ7q5hULkXbaZneZ3JiXFqNNrWbfCzWwulJzEx0smUSQ5kRFhYzl8zZjfKpktMpLpzLY1aJAxGDJ+vFGLXGTIJdt+Cw/R2iht16EDlCqV7u6IqBi2VGnEumpNmH9rJ36pd0fqfZZ3qpZfqCnL9xEbF09QoInRXeo6/0UTBUqmXyMNGsDDD8Mnn8Czz0KVKnaeTQiR6sMPjU1TfHygbl1o2pRnzgbwb/Egeuz9k9CYPYzs9SKHylQmWDJbZFK2XyNKwdixcN99Rt+if383tNa7Ke1g9WZeFhoaqrds2eLpZnje9u0QEmKUtBsyJM1dEVExjF6wg8SU9P+/Jj/fDC8pCpEtR48au2A99BDMnOnp1uRJSqmtWutQT7fDnSSz7bh+3aj1XK2aUZ50+3biNmwi8MzNxblT2w7kozYD8PNVTOmbwSiiEK6QkgINGxrFCrZvlx0xcZ7ZMhLtzRYuBF9fCAtLd9cbS3bZ7UArhXSgRe6pVg2eeMLYgv6FF6BePU+3SIi86euvjZKQc+ZA+/ZERMXw3LztlLt8jmfXz2X+rZ3ZHmRM0ShauJBktnAPHx9jNHrQIFi2DLp393SL8jSZE+2ttIYFC+Duu6Fs2XR3x8XbL5quNRLGIneNGwcmkzFHWgiRXlISTJoELVtCu3aAceldA6eLl2Fc16dSO9AAFx3kuRC5YsAAY0Bk/Hij0yAckk60t0lIgN69jbl0//3nsCqHEB5TvjyMGmVcKdm61dOtESLvmTfP2E123LjUy+XOypJJ5Q3hVn5+8OKLsHGjsdW8cEg60d7mt99gyRJ47jkjfO+91+5ppQL8snRcCJcaNQrKlDE6CUKIm1JSjAo2jRql2SDLUUdZgVTeEO73yCPGgMiECZ5uSZ4mneg8znbnqpiPvyDJ3wjbyMoNaTNrl93tXl/v2RA/37QLAvx8Fa/3bOiWdosCrkQJowO9YoUxmiFEAWGb2RFRMWmOvTR4POzaBWPGGPNPzeyVJVPAwFZVZQqecD+TyRisW7FCdjF0QhYW5mGWnasshfcvnjpHmTUr+L5JZ/aVqcru8jWIiYtn7OKdQNq5zlIGSXjc0KHwxhvw6afQurWnWyNErrPN7Ji4eEYv2AEKEpM1aM2AVd9xPLAi2+q0pbfVYyWzRZ4zcKCxyDAiwnjTJ9KRTnQeZr1zFUDX/zbgn5TAj/XvYltw/dTjlh0IbcM2LCRYAli4RURUjP0//pZSd++/b3cBrBD5iW1mA2mqJN1+dAdNT/zHuC5P8sfvB+l9W7U050pmC3dxmNnWqlQxNl+RTrRDMp0jD7NdaNJ711qOBlZkW1D6smHOFqUIkZsso28xcfFoSL06EhEVY5S7S0iA2bM93Uwhcl1GOfxk5HxOFSvNokYdJLOFxzjNbFthYfD333DsmNvb6Q2kE52HWS80KXflPLcf+4eIBnfbLX4uq7eFp9gbfbNcHaFRI2jbFj7/3FhQJUQ+5iyHm8buo83Rf/jitjBuFCosmS08xmlm2xo40OhzfPWVm1rnXVzSiVZKdVVK7VNKHVBKpRvzV0o9r5TarZT6Ryn1u1KqmtV9yUqp7eaPJa5ojzeytxjFeqFJrz1/4qtTWNqoXboFgyY/X1m9LTzG0Yha6vEnnoCDB2HVKje2SjgjmZ1zGWW2hZ+PIjApnjdWfU6cfzG+b9JVMlt4VIaZba16dejcGb780qhvLtLIcSdaKeULfAJ0AxoAA5RSDWxOiwJCtda3AguByVb3xWutm5o/euW0Pd7I0aUVMHYXDA400Xv3WvYG12HE8O5M6duE4EATCggONMkOhMKjHI2o+ShFjTFLuftQaW6UKgOffebmlgl7JLNzLjOZbcnnKfc34detM2l06iAv3vMsgRXKSGYLj3KU2SVNfuneGALGIvHoaKPErkjDFQsLWwAHtNaHAJRSPwC9gd2WE7TWa6zOjwQedMH3zTecXVpZP6Y9YabLMPYATJ1KPXPwSgCLvGJ0l7ppKhJYJJt3ujpyJZlv6rdnyJJFqOhoqFzZE80UN0lm51CGmW2dz1u3wtrlMH48M6RuusgD7GW2n4/iakJS6m7HaSp/9ewJFSrAjBlpapsL10znCAaOW92ONh9z5DHgV6vb/kqpLUqpSKVUmKMHKaWGms/bcubMmZy1OA+wvhQYk9GllblzjXqi/fu7sYVCZE5YSHCa0TdfO3P2v27cBZ2i+WrQuPSjHMLdJLOzIUuZbW3SJChZEkaOzOUWCpE5tpkdHGiimH8howyjlfjEZEbN30GNV1fwTf326KVL4cgRj7Q5r3LFSHT6v5hgd7N1pdSDQChwl9XhqlrrWKVUTWC1Umqn1vpguifUegYwAyA0NNSrN3O3rSXqiOVy+OpZ31CsWSvKVarkphYKkTXWpblqjFma7v7owIqsrdmc/n/NZ3vRCixpcLfd+ubCLSSzsyirmZ1aMqx4PCxaZGyhXKKEm1orRMZsyynay224eUXxs/qd6bt+MZcHPkKFdavtFjgoiFwxEh0NVLG6XRmItT1JKdUReBnopbW+YTmutY41fz4ErAVCXNCmPM3epUB7krWm6oVYapw+yhelG8vInfAKjubbje06kj3lahD+2zSK3rjmeDW4yG2S2VmUlcy2niN9eMyb4OcHzzyT+40UIgcyqhZzokQ5Jt85iAob1sK337qlTd7AFZ3ozUBtpVQNpVRhoD+QZsW2UioEmI4RxqetjpdSShUxf10WaIPVvLz8yll9UNvL4Z32RwKwrGYL6XAIr2CvQgHAqeJleaf9EAISb9B97zpA6pt7iGR2FmUlsy2KXThL0E/zYNAgqFgxF1snRM45ym1rXzfvwebgBvDss3DypJtalrfluBOttU4CRgLLgT3AfK31LqXUW0opy8rtKUAxYIFNWaT6wBal1A5gDRCutc73gezoHV9woInD4d1J0TevfHba/ze7y9cgumQF6XAIr2Bvvl2pAD8AooLqcqB0Ze7faZS7k1q57ieZnXVZyWyLwVuX4JeUBC+8kNvNEyLHMrO2RSsfpvZ7Ea5dgxEjwM7rvqBxybbfWutlwDKbY69Zfd3RweM2AI1d0QZvYm9lrHXd0KBAEzFx8ZS6dpHQmD183Lpf6nEhvIHtfDvrOaULbu3I2LWzqX/pBMP6dcvS82Zqq1qRIcnsrMlsZlsUu3GNh6KWsbbRHbSvXdvt7RUiO6xz2946AJOfL/36dYTybxrbgC9cCPff7/Q583tmy46FHmBvpM66bqjlskrHA5vw1SmsqN1SivMLr2b9mv+xYXuSfHz48Pr2LIVplraqFcKFMpvZFn13rqLEjav4vPSih1osRM44fc2PGgXNm8Pw4fD11w53oy0Ima20Fw7Hh4aG6i1btni6GbnC8q7tRkwsv8x+liuFTQx6YTaju9bLV+/eRAHXowds22bsZGjK3BWWNuGr7ZYWCw40sX5Me1e3MNcopbZqrUM93Q53KgiZHRMXj69SJKeksPKbZyhbujildm33dPOEyB3798ODD8KmTRASAu+9B+3apTmlIGS2jETnIZZ3bSfPX2HaksmUvH6FUfeNkw608Hq2WyT/FTYYTpyAiRMz/RxZ2qpWCDewHmkDozpH83NHqH3yEKWeHOrh1gmRffa2tU+jdm3YuNHYx+LcOWjfHnr1gr17U08pCJntkjnRIuts5wm92rQ4tz3Qm1VJSST7+FD14ime7/4cO0pXZcryfdKJFl7Ldm5dTFw8Q68WY0W3e6kyaRI89JARyBmwnXdqfVyI3GZvbqe90nc9o5Zzo5AfRQYM8FBLhcgZe5ltt66/jw8MGAD33gsffggTJkDjxvDNNzBgQGpmF79xlat+/qT4GFOe8lNmy0i0B9ibJ7Rr8mcEXzjB31Ubsad8Dd5uP4TFjToA+etdmyh4HG2R3KdmH64oP049/HimVnnbK8EkawWEOzia22n7pq5IUgJhu9eyvHZrKFXKM40VIoccZfaz87bbH5X294eXXoIDB6BtWxg4EGbOZFKps8z8cTzbPxzAY5t/AvJfZstItAfYe4He8+8atgXV4/keo9Kdn5/etYmCx9GbwDPFSjHljgd5c9V0Nr07gxajhzl9nrCQYIof2Eul559lePfRJFevke9Weou8yVGnwlep1B3dAPrvWE7g9SusadWdXrZPIoSXcDZw53BUGqBcOVi6FO67Dx5/nLbAjcDSXCtiolnsXoKlOodwBcsL9Jazx1A6hVvOHqP+mSMsqX+njLSJfMfZm8DvQu5hV/maVH/nFbh82fkTXb9Oh7efo0H0Pv68K4D1Y9rnqzAWeZejTkWy1qmZXevcccasnc2ftUK5a+QD7myeEC6V0cCd091mAwIgIgJefRVmzaLIiRiKd+1EN98L+TKzpRPtAUGBJqqfj2HVlyO469A2eu3+g2Tlw9aWnZyWURLCGznbCSvZx5dXOz9B+UtnjZ3dliyBU6fsL2oZNw52GiMgxOSfEkki73O22crEPo2pWtyPqb9MJcGvMFc/nU5Ys8pubqEQrpOZ3Qtt31imyez31xNx7zAYPNiY6lG/vlHNIyEhF1vtGTKdw80iomK4eiOJkAuxAJS5dpFee/7k7+pNeKxv63SbVAjh7SyvZ0sZMFvbguvzzd0DeHjJfPjxRwBCS1ZgTKU6/F21Mb/Ua8ui8A2EzX+fRa1603vHSgrFxrr1ZxAFW7t65ZgTeQzrmfuWq4Rht5QgbPsXcHI/LFxIt87NPNZOIVwho8yGtG8s7S1EfG7edp6dt53gQBMfBlQkNCnJmDPdoEHu/wBuJCPRbmR5ocXFJxJ86QwAodG7qB53Av+HBkrnWeRbYSHBrB/Tng/6NbU7ZanE1Clw6RL89RfT7hnG9oq1aRazl3dWfMqmjx/mk4iJ7C9ThXG3P0y0qRTRuw566CcRBU1EVAyLtsak6UAr4L7mwYSZLkPLlrB4Mbz7rjEXVIh8IKPMtp5mam/NgOX3JSYunolHzF3NPXtys8keISPRbhIRFcOo+TtSF6EEXToNwP07V5Hg60ez5x7zZPOEcAvrEQ6728C2bcvUxhfRjXsCUP/0Ie7b+Tutj+1k9D3PcsOvCCeLlsL/v8PIBXORm6w3UbGlgcgdx2DoI0ZlmZUrjTq5QuQzGWY2GVcQ212ikvmL3fnujaZ0ot3AMgJtvYo7yDwSXUinsKJmCzoHBnqqeUK4le2UJctcOktABwb4ceFaIgB7ytfknQ410zz+VLEyNDnxn1vbLAoW28vT9rRfuxjOnDE2nGjVyo2tE8K9spLZ9sQX9ud4yQpU2b3bHc11K5nO4Qb2LnVYOtEA60I7urtJQuQJ9urvXrmehJ+vcviYU8VKU+HKeWq89Iv9mqVC5JC9zLZWJPEGw7b8CB06SAdaFCjZyWyAA2Wr8N/KDfkus13SiVZKdVVK7VNKHVBKjbFzfxGl1Dzz/X8rpapb3TfWfHyfUqqLK9qT19i71GHdib5t5CB3NkeIPMNeZyUxRVO0cCGCzQtXbKP5VLHSmJJuUPzG1dSapfkplN1FctuxjC5PP7j7d8pcuQAvv+ymFgmRN2QnswHWVW1CnXPHqLVtXb7K7Bx3opVSvsAnQDegATBAKWW7/PIx4ILW+hbgfWCS+bENgP5AQ6Ar8Kn5+fIV2/JIvinJVLx8LvV2z9tvcXeThMgTHHVWLsYnsn5Me46Ed+f9fk1Tyz76KsXpYqUBqGD+HXJas1TYJbntnLM6uVWL+zFq+0/QujXcfbf7GiVEHpCdzAb4pnkPThQrw4jIBfkqs10xEt0COKC1PqS1TgB+AHrbnNMb+Nr89UKgg1JKmY//oLW+obU+DBwwP1++YltzsfyV8xTSKR5skRB5g6POivVxyyrxw+HdSdGak8XLAFDhyvnUczIaORTpSG474WiL+Q/6NeXPoBgCTsbAK6+Acn4JW4j8JjuZDZDo68fMFvfS6vi/NIvZk28y2xWd6GDguNXtaPMxu+dorZOAi0CZTD4WAKXUUKXUFqXUljNnztg7Jc8KCwlOs4lKk5RLnm6SEHmCo86Ko106gwJNnCpmdKKnLp1qVEYg4x22RDq5ntv5KbNTN74qD7z2GoSEQLdunm6mEG6Xncy2+L5JFy74F+eJyIX5JrNdUZ3D3ltxnclzMvNY46DWM4AZAKGhoXbPycvSrG6dPRume7Q5QuQJmSmfZG10l7q8Oc94E1r+6gUanD7M4cq1HQa4cCjXcztfZTYYb9h8zONOU6fKKLQokLKT2ZZKN9cKm/i6eQ+eXf89PvlkFqsrOtHRQBWr25UB2+3ELOdEK6UKASWB85l8bP6zdClUqgQnTni6JUJ4XFZ26Uw9b5Lx6X9HNxH4dB/ZqCjrJLez6vzN6UO0aeO5dgjhYdnJbEune3m7+xm5NYIOS2ZDX++vre6K6RybgdpKqRpKqcIYC06W2JyzBLCUoOgLrNZaa/Px/uZV4DWA2sAmF7Qp74qPh19/hd620w+FEJlhHd6DT26VDnT2SG5n1YEDN78uXNhz7RDCy1jPkf717XspNHwYzJ0LR496umk5luNOtHmu3EhgObAHmK+13qWUeksp1ct82pdAGaXUAeB5YIz5sbuA+cBu4DfgSa214+Kc+cGqVXD1Ktx7r6dbIoT3unABwsONbWTz4VayuU1yOxsOWm01L51oIbJv1ChjatS773q6JTmmtPa6qWqEhobqLVu2eLoZ2fPoo7B4MZw+DSNHwrVr8N13nm6VMGyuXwAAIABJREFUEN4nJgYqV4Z33vGqer1Kqa1a61BPt8OdvDqzLd5+21hUCEZum/LHwighPOLRR+H7743R6PLlPd0ap5xltuxY6E5JSbBkCfToYYxkzJghHWghsis42KjVu2iRp1siCgLrkWg/P8+1Q4j8YMwYSEiA997zdEtyRDrR7rRuHZw7J1M5hHCVPn0gKgoOH/Z0S0R+d/Ag1K8Py5ZBIVesyReiAKtTB/r1g08+gbNnPd2abJNOtDvNnWtcAuza1dMtESJ/uO8+4/PixZ5th8j/DhyAVq2kPrQQrvLyy8bUqPff93RLsk060e5y4QLMmQMPPABFi3q6NULkDzVqGBtfyJQOkZuuXoWTJ+GWfFLcVoi8oGFD42riZ58Z0129kHSi3WXWLOMd18iRnm6JEPlLnz6wcaOx0FCI3LBhg/G5cWPPtkOI/Ob++41BRi9deCyd6FwSERVDm/DV1BizlDsmrOTq+x9B27bQtKmnmyZE/mKZ0hER4dl2CK9mndltwlcTEWX1pmzRIuMKYseOnmugEPlRhw7G7p8rVni6JdkinehcEBEVw9jFO4mJi0cDt0Stp2j0UTb1GOjppgmR/9Svb3z88IOxNbMQWWSb2TFx8YxdvNPoSCcnG3Pue/SQsnZCuFrZstC8uXSixU1Tlu8jPvHm3gODt/7MqWKleSGppgdbJUQ+NmyYUf3myy893RLhhWwzGyA+MZkpy/fBX3/BmTPQt6+HWidEPte5M0RGwsWLnm5JlkmdnlwQGxef+nWN8zHcdXgbU9sO5PjlRA+2Soj8ISIqhinL9xEbF09QoInRXeoS9tRTRg32Z54xpk3Vq5e1x8vW4QWadWanO77wV2MEWqpyCJEtGWZu584wYQKsXQu9e2f98R4knegcsvefGxRoIsYcyg9tW0qCTyG+b9KVoEC5FChETlguu1tGDS2X3QH8X5rM7WHtWBc2hPGD37IbtM4en1dCWeSujDLbWnCJIvD1IrjnHqmqJEQ2OMtcMK4CnTl3ie2F/Tk550dq2nSi83pmy3SOHHA0j65dvXKY/HwJSIin785VLKvXhiulyjK6S900j3W4iEUIYZejy+5vLNnFc3+dYXWNZoTE7E07pzUTj5+yfF+ut114XkaZbc3k58ukUmeM0nb33SeZLUQ2OMtsy+9igq8fG6o0xnfVSq/LbBmJzgFH/7lr9p5hYp/G7H9zCiUSrrHsrr5M7NM49V1TXn9nJURe5eiye1y8MVXqn4p1CNv9B+WunOdMsdJMWb4vze+U08v2It/LKLPTjFB3vIU2Q8OgalWWVL9NMluIbMgosy3WVQ+h48HNLHhjMnStB5cvw6VLDPntH4rdiKdowjXiTMV5ucuTaOWTZzJbOtE54OwPcljTINi/Epo3Z8YnTxolXMycvbOSQBbCMUeX3S3+qWRshnHryf38fkvLdL+jjh4vU60KBqeZHRKcNn9nzTK2lP/+eyb9cUwyW4hsyCizLdbUCuXV1TN5YclHsMR8UCn6Fgngsp8Jv5Qkyl2N4/22AzlTrHSeyewcTedQSpVWSq1USu03fy5l55ymSqmNSqldSql/lFL9rO6brZQ6rJTabv7Is0WU7V3Kc/SfGBRogilTYPdueOqpNB1okNEwIbJrdJe6di+7lwrwA2BX+VokKx9uPXEASN85dvR466lW+V1Bye0sZ7a1K1dg3Dho3Rr69ZPMFiKbMspsi6Olgrhr6Az6Pz8bYmON38HkZH7fsI8Oz3zLpLsGA1AkOTFPZXZO50SPAX7XWtcGfjfftnUNeFhr3RDoCnyglAq0un+01rqp+WN7DtuTK7I6j27GiVXw0kvwv//BwPS1oTMd5EKINMJCgpnYpzHBgSYUEBxoYmKfxrzesyEmP1/iC/uzv0wVbj35n92gdfT4AjaamO9zO6uZne4P8qRJxlzo998HpSSzhcimjDLb2rlywfR/sCNUqmQs5FUq9fFFixsLe6sE+OapzM7pdI7ewN3mr78G1gIvWZ+gtf7P6utYpdRpoBwQl8Pv7TZZmUc3I2YlDT8Oh/794dtvoVD6f+LRXeqmmV8HBW80TIjsSnfZ3cqU5fv4p1JtOh3azMR7G9k9LywkmLDEGEg2/Z+9Ow+PsrrfP/7+EIIGUQKKKGF1Q0FUNKIWN3BBqyJ112rVn4haW5daFNzXSkVrXb6tW9XWXasiiooLWK2KAiKrYhVlCS7IJkuEEM7vjzOByWRmMssza+7Xdc2V5JlnZg4w3DlznnM+x480Nj1Fn9tJzX2OrOIybx7cfjucdhrssw+gzBZJR2OZ3VjpukG9Kxh00p7wDDx1Zm/YLT860JB+J7q9c+5bAOfct2a2dbyTzawP0AL4KuzwLWZ2LaEREefcmhiPHQIMAejcuXOazU5OwvPobrkF7h3hw/ef/4zagYaNC1Hyte6hSCGJLFnW/ah+tBnxFoO2rG148qpV/jL93Xf7XQ5nzcp+g3MvK7mddmY//zy8/DI8+mjSD01q7nOk4cP911tv3XBImS0SnMjMvvPkPRr/v7TJJv7rmqhdxJxptBNtZm8B20S566pkXsjMtgUeA850zq0PHR4OfIcP6AfwoyE3Rnu8c+6B0DlUVlZmdW/fhBYj3XgjXHcdnH66D/2Skgbnh2s0yEWkUdEq3dy8aDOeA5g4Ebp02Xjyf/8LZ58NX37pt5pdvTonbc6GfMjttDN77lw/GHHTTdCpU1IPTXkB6UcfwZNPwlVXQUTHX5ktkr6Uq5PlaSe60TnRzrlDnXO7Rrm9BHwfCtm6sP0h2nOY2RbAGOBq59yEsOf+1nlrgEeAPkH8oYLW6GKkm27yHejf/CahDrSIBCPaZfupbbpQU9IcJk3yB6qr4bLL4MADobYWxo+HgQOhpnh3EC2K3D7kEP913LikH5rSAtJVq/xC8G22gWHRpomLSLpSrvtcqJ3oRowGzgx9fybwUuQJZtYCeBH4l3PuuYj76oLcgEHAjDTbkxFxFyPNn+870KeeCg8/rA60SBZFu2y/tnkpn7Xr6keiP/wQ9tgD/vIXuOACmDYNDj6Yr5evZeny1U1144zCyO1evfwVg7ffTvqhSS8gnT8fDjgAJk+Gu+6CVq3Sa7uIRJVqpZt3vlkOwNn3vZdXmZ3unOgRwLNmdg4wDzgRwMwqgfOdc4OBk4ADgS3N7KzQ484Kreh+wszaAQZ8CpyfZnsyJualvKeeAuf8aLQ60CJZFeuy/ZzOO7Pbe2Nh//2hY0d4660NI5ujplSxcu5yjq5dV69yAzSZjTMKI7ebNYN+/fxItHMNSoU2JuHpFx99BIMG+ZHoV16BI49MscEi0phUplqNmlLFo+/N42Bgq1XLGJ9HmZ3WSLRzbrFz7hDn3I6hr0tCxyeFghjn3OPOudKwckgbSiI55/o753qFLjOe7pxbmf4fKcsef9yv8N9++1y3RKTJiXXZvuJXv/TTNQYPhunTN04NwF9O/JlmNF+/8ZJiPm0jm2kFlduHHAJVVfDFF42fm4onn4SDDoKyMn/VQh1okYxKZarVyLGzmb9Ja1a0KOPW1+9hn3nT8yaz053O0bRNm+Z/QUepBS0imRfrsv3eV5wP334L998PW2xR7zELl1WzrqQ5pbXrGhyXPNO/v/+awpSOuNavh6uv9tm9zz7w8cfQs2ewryEiDaRSq3/hsmoWb1bOQec9RFXrrRnx+t1sWvNzXmS2tv1Ox+OP+zJ2J52U65aINFkxL9tvE604hb9sWNOseb2R6Lrjkmd22MFX5hg3Dn772+Ce98IL4b774Jxz4G9/gxYtgntuEYkr2Uo3dVNAlrRszbAjfs9TT1/FJf99ksd+dWEGW5kYjUSnYNSUKvb/01t8+/eH+e8OlYxasDbXTRKRBA0d0B1r3pwStx4LVW3Txhl5ysxP6Rg/3o8epyh8C/AzL37Qd6B//3t48EF1oEXyXPgUkA+77M5Tux3OuRNHcUvH3I9EqxOdpLoah52mT2TblYt5eqcDGf7C9LxZKSoi8Q3qXUH/Xn4U5Ia37qei9aZ5tY2sROjfH5YsgalTU3p45BbgZ4x5kOWbtmLM8eclvVhRRLIvcgrII7+6kLVbtePg24bD2twOYmo6RzI++4za3w7j5hUr2PX7r1jZooy3dujDz6EJ7volLFIYenTZEoDffDKG39w9HPR/N3/VLQp9+23o3Tvph4fXpd1zwWcc+tVEbjvwN7z04fccdZDmQYsUggZTQPYyX+9/xAi49tqctUsj0YmqqYETTuCoSa/TZ8EszMH/7XcSP5duCmhRkkhBCS9HeeeduWuHNK5DB9h555QXF27IZue4/N1/smizch7Za6AyW6SQHXOM35/j5pthRu62GNFIdKLuuw9mzeK6M27kmQ57Nrhbi5JECsjsUGmk8nJ48UX4+mvo1i23bZLY+vf3W4CvXZv0HOYO5WWs+G4RJ057k33nz+DaQ8+jusWmVCizRQrbXXfBm2/6BcIffJCTvTo0Eh0SvvCkwW44P/7oLxcceij7XXJ28tvJikh+2W47//WZZ/ymHvfem9v2SHyHHOI3Q/n443qH4+Y2wNdf89wbtzHp3tO5Zvw/+HTbHXl69yOU2SLFoF07uPtunwt33ZWTJphzLicvnI7Kyko3adKkwJ6vbuFJ+H7uZaUlGxcbXXihrzc7dSr07MmoKVWMHDubhcuq6VBextAB3TUfWiRPJPT/c906mDvXb5J02mkwZgwsWACbb57x9pnZZOdcZcZfKI+kndlLlkD79nDyyb60KAnk9qxZcNhhsGoVXx51IiPKe/N2q850aNNSmS2SZ1LuVzkHxx7rd6WdNs2XxQxYvMxuOp3oF17w85pPPrnBXX1HjIu6DWVFeRnvH9UO9tjDd6TvvjvVJotIFjTasYpm4kTo0wf++le4+OKMt1Gd6BRdey3cdBOMHg3HHBM/tw/dAo44AkpL4Y03oFev9F5bRDImpdwOV1UFPXrAnnv6mvIBV92Jl9lNZzrHfffBn/4U9a5YC0wWLl3tf6mWl8P112ewcSIShPBKDHUa3R52772hb19/ObC2NvZ5kltXXw277QZDhsB338XM7U7TPvZzqDffHN57Tx1okTyXUm6Hq6iA22+Hd96Bhx8OvoFxNJ1O9AEH+C26ly1rcFesRYGnLPzEF/m/6SZo2zbTLRSRNMX8QNxYJYZLL/WLC0ePzkCrJBAtWsC//gU//QRHHcX2UWK731cT+edz10HHjvDf/2bk0q6IBCvl3A43eDDssw/ccktWB0OaTid6//393JkPPmhwV/huOHVaWy1Xj/8H7LqrH/kQkbwX6wNxo9VzBg2Crl1V7i7f7b47PPccTJ3KE2/czg4rF/Hg8zdy+BcfMnDWf3jghZtZvePO8O67fnRKRPJeyrkdzgyGDvWDIU88EVDLGpdWJ9rM2prZm2b2v9DXNjHOqzWzT0O30WHHu5nZR6HHP2Nmmdt/dZ99/Py4995rcFfkbjgV5WU8ufJDNqua5+dJNlclQJFCEO0DcUKVGEpK4KKLfD5MnpzBFuZeQeV2NL/8Jfz977T/4B3GPng+h335MX8bdSt/feV2lvfemzYT3oOttspqk0QkdSnndqRBg2Dfff2Vxe+/D7CFsaU7Ej0MeNs5tyPwdujnaKqdc3uEbgPDjv8ZuDP0+KXAOWm2J7aWLWGvvTZ0oiNLIwG8P6w/dw3ama1++pGu9/+V//Tcn1Ftd85Yk0QkWNE+ECe8OOWcc/w82jvvbLx0WmErnNwOU+/fZPH2fHbBHylpvzXvPfgcH+20N692358Tj72WUV+tyEZzRCQgaeV2mFHTvuPX+57LmuU/8fZhJ2clt9OqzmFms4GDnXPfmtm2wDvOuQYfHcxspXOuVcQxAxYB2zjn1pnZfsD1zrkBjb1uyiu9hw6Fu+9m9H9mccWY/zVYCfrHTRZy9jWDWdB6a9qvXMyhg+/jx3YVKf1jikgBuvRS1t97L/1++whzyzYO0Ca1UrwRua7OkYvcTrc6R6zV+8fv2YHnP1mY+qp+ESkK4Rnx2w+f5fJ3/8VFJ1xN/yvPTzsLMlmdo71z7luA0NetY5y3qZlNMrMJZjYodGxLYJlzbl3o5wVAzD+pmQ0JPcekRYsWpdbaAw6AtWt5/ZGXo64EdS+/Qm2zZmyzYjH37XMi88u3SW6FqIgUtosuwtWu56SP6y8wLLIcyEpuB5LZIbFW7z/18YL0VvWLSFEIz4gH+hzHjPbbc81r93LfixMz+rqNdqLN7C0zmxHldmwSr9M51Is/DfirmW0PRCvkF3NY3Dn3gHOu0jlX2a5duyReOkzfvgB0+3xK1Lsr58/kkw47s8fFT3Hn/qdtOJ7UClERKVzduvHmjvtw2qevs2nNz34xckgh5UA+5HYgmR0S6+++NsaV1EL6txKR9IX/n19X0pzLj7yY8uoVnPviPRl93UY70c65Q51zu0a5vQR8H7ocSOjrDzGeY2Ho6xzgHaA38CNQbmZ1q/Y6AgvT/hPFs+WW0KMH+3/3WYO7Wq6tZtfvvmRip56sblFWr1h3UitERSSvNTbfefTBJ9Hm5xX8bdQIptx9GhXLfawVUg4UVW4T++++JMamCoX0byUijWsstyP/z89qvx1/3+cEjp8xDl57LWPtSnc6x2jgzND3ZwIvRZ5gZm3MbJPQ91sBfYFZzk/GHg+cEO/xgTvgACoXfs5m9ReCsufC2TR365nYsWe94ymtEBWRvFQ3b65qWTUOqFpWzfAXptcL5MPPO4EZ2+5I/zmTaPPzCi7575PFlgMFl9vRVu8bfiQ6shtdZP9WIk1eIrkdLSMeOug0ftpuJzj33Kh7hAQh3U70COAwM/sfcFjoZ8ys0sweCp2zCzDJzKbiw3eEc25W6L4rgD+Y2Zf4uXb/SLM9jTvgAEpXruDeXqVc8MXbjP7nJdwx5i8M+eh5aq0Zkyt22RDKqa4QFZH8lMjOWIP27MjPF/4egMVlW3DczHHcu3uLYsqBgsvt8NX74DvQdRM5HCizRYpYQrkdpcLHjSftxRbPPAHffedLmGZAWgWQnXOLgUOiHJ8EDA59/wEQdd/V0GXCPum0IWn77w9Av4Uz6DfhaX5ctZb2K5fQfuUSPu7Yg5WbtAT8P8D7w/pntWkiklmJ7oxVecUFsOlatjz6aOjTh0OeuAdOKI48KMjcxv+SHNS7gr4jxlEV8e/lUGaLFKtEc7suI+qrgKuvhhtu8HWkjzsu0LY1vV1EunSBTp1gxAhYtIgrjr+Wt3foQ9vVy/m5+cY9A7QwRaT4dCgva9ABqzteT/PmcNll/vuhQ+Gaa2DCBF/IX3IqkC2CRaRgJJzbsVx1FbzyCpx3nq/SluZC53BNZ9vvcAccAIsWQZcufLGnH5le0rK1X1AYooUpIsUnpZ2xLrkEtt4ahg2rV61DciOQLYJFpGCkvaNhaSn8619+QKRt20Db1nQ70QBDhnDZkT2C2W5SRPJeSjtjtWrlR6L/8x94442stVWiC2yLYBEpCIHsaNijB1x+OZSUNH5uEtLasTBX0t39ikWLYPhwuO02aNuWUVOqGDl2NguXVdOhvIyhA7prYYqIbLR2LXTvDuXlMHkyNEt9/CHXOxbmQtqZHUGZLSLZEi+zm2YnWkQkWY8/DmecAfffD0OGpPw06kSLiBSOTG77LSLSNJx2GvziF3Deeby61wB2v+SZqEX/RUQkfzS2UUs61IkWEUlEs2aM/stj3Nf3FA6f8hZPPTWcqqWrGxT9FxGR/JDIRi3paHol7kREUvTn8d9Qtf/pvLzDvrSt/gnMNhT915xcEZH8Em+jliAyW51oEZEE1dUinrnNDlGPi4hI/sh0XXlN5xARSZBqFIuIFI5MZ7Y60SIiCVKNYhGRwpHpzNZ0DhGRBNTVJq6uqaXEjFrnqFCNYhGRvJSNzFYnWkSkEXUrvOsWqNQ6t2E0Qx1oEZH8kq3MTqsTbWZtgWeArsA3wEnOuaUR5/QD7gw7tDNwinNulJk9ChwELA/dd5Zz7tN02iQikq7IHfFWr12X0RXe2aTcFpFik6vMTndO9DDgbefcjsDboZ/rcc6Nd87t4ZzbA+gPrAbeCDtlaN39CmIRybVodUWXrq6Jem6BVuVQbotI0chlZqc7neNY4ODQ9/8E3gGuiHP+CcBrzrnVab5uyiI/rehyrIiEi1ZXNJYCrcqh3BaRopHLzE53JLq9c+5bgNDXrRs5/xTgqYhjt5jZNDO708w2SbM9cWV65xoRKXyJjlQUcFUO5baIFI1cZnajnWgze8vMZkS5HZvMC5nZtkAvYGzY4eH4uXZ7A22JMxpiZkPMbJKZTVq0aFEyL71BvJ1rREQg9khFeVkpFeVlGFBRXsatx/XK29HQfMjtIDIblNsiEl8uM7vR6RzOuUNj3Wdm35vZts65b0Nh+0OcpzoJeNE5t2GiSt1oCLDGzB4B/hinHQ8ADwBUVla6xtodTaZ3rhGRwjd0QPd6q7rBj2BcP7Bn3naaI+VDbgeR2aDcFpH4cpnZ6U7nGA2cGfr+TOClOOeeSsQlwVCAY2YGDAJmpNmeuLTbmIg0ZlDvCm49rlfBjDqnQLktIkUjl5md7sLCEcCzZnYOMA84EcDMKoHznXODQz93BToB/4l4/BNm1g4w4FPg/DTbE1esTysFOq9RRDJkUO+KYuo0R1Jui0hRyVVmp9WJds4tBg6JcnwSMDjs52+ABn8651z/dF4/WXV/wVrlLSJNlXJbRCQYTW7HwiIfYRIRKTrKbRHJR02iE60aoyIihUOZLSKFoOg70ZH7p9fVGAUUyiIieUaZLSKFIt3qHHlPNUZFRAqHMltECkXRd6JVY1REpHAos0WkUBR9J1o1RkVECocyW0QKRdF3oocO6E5ZaUm9Y6oxKiKSn5TZIlIoin5hoWqMiogUDmW2iBSKou9Eg2qMiogUEmW2iBSCop/OISIiIiISNHWiRURERESSpE60iIiIiEiS1IkWEREREUmSOtEiIiIiIklSJ1pEREREJEnmnMt1G5JmZouAuUk+bCvgxww0J1n50g7In7aoHQ3lS1vUjobSbUsX51y7oBpTCJTZgcmXtuRLOyB/2qJ2NJQvbclYZhdkJzoVZjbJOVepdmyUL21ROxrKl7aoHQ3lU1uKWb78PedLOyB/2pIv7YD8aYva0VC+tCWT7dB0DhERERGRJKkTLSIiIiKSpKbUiX4g1w0IyZd2QP60Re1oKF/aonY0lE9tKWb58vecL+2A/GlLvrQD8qctakdD+dKWjLWjycyJFhEREREJSlMaiRYRERERCUST7ESb2R/NzJnZVjl6/ZvMbJqZfWpmb5hZhxy1Y6SZfR5qy4tmVp6LdoTacqKZzTSz9WaW9dW8ZnaEmc02sy/NbFi2Xz+sHQ+b2Q9mNiNXbQi1o5OZjTezz0L/LhfnqB2bmtnHZjY11I4bctGOsPaUmNkUM3sll+1oapTZ9dqSF7mtzN7QDmV2/XY0qcxucp1oM+sEHAbMy2EzRjrndnPO7QG8Alybo3a8CezqnNsN+AIYnqN2AMwAjgPezfYLm1kJ8H/AkUAP4FQz65HtdoQ8ChyRo9cOtw64zDm3C7AvcGGO/k7WAP2dc7sDewBHmNm+OWhHnYuBz3L4+k2OMruBfMltZbb3KMrscE0qs5tcJxq4E7gcyNlkcOfcT2E/bpartjjn3nDOrQv9OAHomIt2hNrymXNudo5evg/wpXNujnNuLfA0cGwuGuKcexdYkovXjmjHt865T0Lfr8CHUEUO2uGccytDP5aGbjn5/2JmHYGjgIdy8fpNmDK7flvyIreV2Z4yu0E7mlRmN6lOtJkNBKqcc1PzoC23mNl84NfkdlSjzv8DXst1I3KkApgf9vMCchA++crMugK9gY9y9PolZvYp8APwpnMuJ+0A/orvzK3P0es3OcrsRjXV3FZmx6HM3iDjmd08U0+cK2b2FrBNlLuuAq4EDs91O5xzLznnrgKuMrPhwO+A63LRjtA5V+EvBT2RiTYk05YcsSjHVLYGMLNWwPPAJRGjcVnjnKsF9gjN/XzRzHZ1zmV1/qGZHQ384JybbGYHZ/O1i50yO/m2hM7JeG4rswuPMtvLVmYXXSfaOXdotONm1gvoBkw1M/CXwD4xsz7Oue+y1Y4ongTGkKFAbqwdZnYmcDRwiMtwvcMk/k6ybQHQKeznjsDCHLUlb5hZKT6Mn3DOvZDr9jjnlpnZO/j5h9lexNMXGGhmvwQ2BbYws8edc6dnuR1FR5mdfFuyldvK7MKizK4nK5ndZKZzOOemO+e2ds51dc51xf8n3DMTYdwYM9sx7MeBwOfZbkOoHUcAVwADnXOrc9GGPDER2NHMuplZC+AUYHSO25RT5nst/wA+c879JYftaFdXfcDMyoBDycH/F+fccOdcx1B2nAKMUwc6s5TZMdui3FZmN6DMri9bmd1kOtF5ZoSZzTCzafhLlTkpRQPcC2wOvBkq3XRfjtqBmf3KzBYA+wFjzGxstl47tEjnd8BY/GKMZ51zM7P1+uHM7CngQ6C7mS0ws3Ny0Q78p/gzgP6h98anoU/02bYtMD70f2Uifn6dystJtuVLZkOe5LYy21NmN9CkMls7FoqIiIiIJEkj0SIiIiIiSVInWkREREQkSepEi4iIiIgkSZ1oEREREZEkqRMtIiIiIpIkdaJFRERERJKkTrSIiIiISJLUiZaCYGbfmFm+bkErItKkKaOlKVInWopapoLdvD+b2eLQ7bbQtquxzr3KzOaZ2U9m9rSZbRF0m0RECk0hZLSZVZjZS2a2JLQr4flBt1cKkzrRIqkZAgwCdgd2A44Gzotx7m/w27H2BToAZcA9WWijiEhTFWRGPw58DbQHjgL+ZGb9MtNsKSTqREsh2dvMZpnZUjN7xMw2BTCzo83sUzNbZmYfmNluoeOPAZ2Bl81spZldHjr+nJl9Z2bLzewBUw9nAAAgAElEQVRdM+uZQlvOBO5wzi1wzlUBdwBnxTj3GOAfzrn5zrmVwJ+Bk82sZQqvKyKSr4ouo82sFXAwcItzrsY5NxX4N/D/UmiTFBl1oqWQ/BoYAGwP7ARcbWZ7Ag/jRxi2BO4HRpvZJs65M4B5wDHOuVbOudtCz/MasCOwNfAJ8ETdC5jZsFDQR72FtaUnMDXs56mhY9FY6Bb+8yahNoiIFItizGgLOxZ+/64J/H1IkVMnWgrJvaGRgiXALcCpwLnA/c65j5xztc65fwJrgH1jPYlz7mHn3Arn3BrgemB3M2sdum+Ec6481i3saVoBy8N+Xg60ijHn7jVgsJl1Db3OFaHjGokWkWJSdBntnFsBvA9cY2abhj4UHI/yW1AnWgrL/LDv5+LnrnUBLosYiegUuq8BMysxsxFm9pWZ/QR8E7prqyTbshIIXxy4BbDSOeeinPsw8BTwDjATGB86viDJ1xQRyWfFmtG/Brrh/3x/x4+MK79FnWgpKJ3Cvu8MLMSH2i0RoxEtnXNPhc6LDMzTgGOBQ4HWQNfQcQMwsytDc/Oi3sKeZyZ+wUqd3UPHGnDOrXfOXeec6+qc6xg6ryp0ExEpFkWZ0c65uc65o51z7Zxz++CnpXycxN+LFCl1oqWQXGhmHc2sLXAl8AzwIHC+me0TKlO0mZkdZWabhx7zPbBd2HNsjr+UuBh/Oe5P4S/gnPtTaG5e1FvYqf8C/hAqfdQBuAx4NFqjzaytmW0fal8P4C/Ajc659en+hYiI5JGizGgz28XMNjezFmZ2OnB46Bxp4tSJlkLyJPAGMCd0u9k5Nwk/5+5eYCnwJfVXYN+KX9yyzMz+iA/WufgRhlnAhBTbcj/wMjAdmAGMCR0DIDQqckDox62AV4FV+Ll3DzvnHkjxdUVE8lWxZvSA0J9nKXA+cIRzblGK7ZIiYtGnB4mIiIiISCwaiRYRERERSZI60SIiIiIiSVInWkREREQkSYF0os3sYTP7wcxmxLj/12Y2LXT7wMx2D7vvGzObbn5L0ElBtEdERGJTZouIpC+okehHgSPi3P81cJBzbjfgJiCyMkE/59wezrnKgNojIiKxPYoyW0QkLc2DeBLn3Ltm1jXO/R+E/TgB6JjO62211Vaua9eYLycikrcmT578o3OuXS7boMwWEUlMvMwOpBOdpHPwdRjrOOANM3PA/bHq55rZEGAIQOfOnZk0SVcRRaTwmNncXLchScpsEWmy4mV2VjvRZtYPH8j7hx3u65xbaGZbA2+a2efOuXcjHxsK6gcAKisrVdxaRCTDlNkiIrFlrTqHme0GPAQc65xbXHfcObcw9PUH4EWgT7baJCIi0SmzRUTiy0on2sw6Ay8AZzjnvgg7vpmZbV73PX4/+qirxUVEJDuU2SIijQtkOoeZPQUcDGxlZguA64BSAOfcfcC1wJbA38wMYF1oVXd74MXQsebAk86514Nok4iIRKfMFhFJX1DVOU5t5P7BwOAox+cAuzd8hIiIZIoyW0QkfbmoziFSUEZNqWLk2NksXFZNh/Iyhg7ozqDeFbluloiIRKHMlmxRJ1okjlFTqhj+wnSqa2oBqFpWzfAXpgMolEVE8owyW7Ipa9U5RArRyLGzN4RxneqaWkaOnZ2jFomISCzKbMkmdaJF4li4rDqp4yIikjvKbMkmdaJF4uhQXpbUcRERyR1ltmSTOtEicQwd0J2y0pJ6x8pKSxg6oHuOWiQiIrEosyWbtLBQJI66hSixVnprFbiISP5QZks2qRMt0ohBvSuihqxWgYuI5B9ltmSLOtEiKYq3CjyZQNbIiIhI5imzJWjqRIukKIhV4BoZERHJDmW2BE0LC0VSFMQqcNU0FRHJDmW2BE2daGmSRk2pou+IcXQbNoa+I8YxakpV0s8RxCpw1TQVEUlMurmtzJagaTqHNDlBXY5rbBV4IjqUl1EVJXxV01REZKMgcluZLUELpBNtZg8DRwM/OOd2jXK/AXcBvwRWA2c55z4J3XcmcHXo1Judc/8Mok3SNCWy4COoxSUQexV4ooYO6F7vFwOopqlknjJb8kWii/SCym1ltgQpqOkcjwJHxLn/SGDH0G0I8HcAM2sLXAfsA/QBrjOzNgG1SZqYupGKqmXVODaOVERe8suny3GDeldw63G9qCgvw4CK8jJuPa6XFqhIpj2KMltyLNHMhvzJbWW2hAtkJNo5966ZdY1zyrHAv5xzDphgZuVmti1wMPCmc24JgJm9iQ/2p4JolxSvaKMXiY5UxLoc54C+I8Y1GAkJopxRvOdId2REJFnKbMm2dDIbksvtoErQxXoeZbbUydac6ApgftjPC0LHYh1vwMyG4EdE6Ny5c2ZaKXkpMsj67dyO5ydXNZgbFxnGdSJHKqJdjqsTOc8u3jw8aHxu3agpVVw/eibLqmvqvcalz3zKpLlLuHlQrxT+RkQyTpktaQnP7dZlpaxau46aWgckn9mQeG4DaWf2yLGzqVpWjeE76XXPo9yWSNnqRFuUYy7O8YYHnXsAeACgsrIy6jlSfKJ1Yp+YMK/Bm6S6ppYSM2pdw7dG5IKP8MUl0UY2wkdCYo2U3PDyTH6uWR93kUtk28M54IkJ86js0lYjGpKPlNmSssjsCx9EqJNMZkPiuV33feR9qWR2ZMuU2xIpWyXuFgCdwn7uCCyMc1yagETKFUXrxMb6bVzrXMLliwb1ruD9Yf2j9ghg40hIrPl2S1fXNForNFrbw7nQOSJ5SJktDSRaYq6x7KuTTGZDYrmdycwG5bbUl61O9GjgN+btCyx3zn0LjAUON7M2ocUph4eOSZFLdxFgNHULPJJZ8NFY8f1kyxZVLave8Esm2mhJJNUWlTylzJZ6glgEGCmVzIb4uZ3pzAbltmwUSCfazJ4CPgS6m9kCMzvHzM43s/NDp7wKzAG+BB4EfgsQWpxyEzAxdLuxbsGKFLcbXp6Z0K5PyQTi0lVruPSZTwG48+Q9eH9Y/0bDOFrxfYBVa9YxakpVzOL85WWlUZ/PYMMvmUSotqjkgjJbkpVoZkPiuZZKZkP03Dag387tMp7ZoNyWjYKqznFqI/c74MIY9z0MPBxEO6QwjJpSxdLVDefIQWKLAMMXe4RbXbMeSK4If939N7w8s16bllXXMPyF6dx6XC9uPa5Xg8UoQMLtClex/Ad2//YLZrbfjh/adWTo4TvBuHFw990sn/k5s5ptwRet2vNZj73Z/9wTOfrAXQJbaS5SR5ktyUgmsyH+IsBwqWR23TmT5i6ptz7GAc9PrqKyS9tAMztS5HSTaPkM6W3oIoVDOxZK1sWbTxZvEWBdICVyyS2ZIvx1Cwgjf0nUPUe80ZE7XptF3/depjPVrF71M6W1tZSur6Ht6p9ov3Ix7VcuYcvVy5m2zY60Wl/D3t98SvP1/hfHz223YtPn2sIXX7CmzZZM3moH2i3/kRO+ns5mn7zCuiduYG733Zm/dU923Hp71rbfniraprS7oohIqpLJbGiY24l0VJPdOGX854uiLjBvLLMT/V1isKEa1PjPF0XtEEdb+D70ualgNKhEAsrsYqROtGRdvPlksRYBhodP3xHjAp9vnEoh/0G92jNo4hh4/Z56x9c2a87Sllvwfau2LNx8K+ZW7MCR38+Eli3hiivgmGNg2jQ2ff99qKqCK67giKoOfL3KB3FpbQ29F87mgK+ncOA3U7jw8ydoFvp1Mb399ryzXSWv/diXQff+FkoaTkUREQlSspkN9XM7bzI7wd8lFeVlvD+sf6NtiLYQsWZ9w48Mqe6IK/lPnWjJulgjAOVlpQnPh0vkUmEy89ZitWnDczz2GEybBvPnb7itX7iQZrW1rGhRxq+ufI5f9NyW5z79nup16zc8vqy0hOP3quDmupGMLcoYumlnBp23H5x33obzvhk2ZsP3NSWlfNxpVz7utCt3HHgG5dU/scPi+fSZP5OD50ziggnP0fzDZ+Cx4bDnnlBZCXvv7b/usANYrLXrIiLJK8jMjqKxPQfAZ3a/ndvRd8S4RqdjBNHpl8KmTrRkXbRALSst4fqBPeudF2+3KNh4Wa68ZSkrf15XbwQgXpmkZNo0dEB3P1r8m9/AJptAp07QqRPz9tiXMV2as652PR936smX1VA19QeOr+xY79JfrI1hoP6lvVi/EErMWFa2BZM69mRSx578bb+T2OLnlRz7w0xuarccJk6Ev/8d7rzTP6C8HH7xC/j1r+H4432bRUTSUHCZHUW0qRfPT67i+L0qUspsiJ3b0WgxYnEyF6XQeb6rrKx0kyZNynUzJA2RO/m1aVnKdcf4QI62WxT4gIxV/ihWeCezKC/muV9+CTvu6EejTz8dSPwyYKzz2rQspWWL5o2OiBy/V0XU4/X+Htatg5kzYdIk36l+7TWYNw922QUefhj23Tf6P4LkhJlNds5V5rod2aTMLnwFldlRpJvZFWFblcfL7dJmVm9OdGN/D5L/4mW2OtGSE9F284sWPpESnasWbbttSDDMampg1izfIZ04ET78EKZPh2eegZNOAqDbsDFRF8sY8PWIozb8HOu8SHUd5mgLWJKuzrF+PYwZAxde6EfRL70UbroJyjQSkg/UiZZClNeZnYAgMrustCTqQEdkboOqcxSTeJmt6RySNeGdwWZRtnuNtiAjUiLzyuJttx1zgUd1Nbz0Ejz5JLz5Jvz8sz9eXu7nGh97LBxxxIbTE52Pl0w1kfGfL4r6yyZyMUyjmjXzixcPOgguvxzuuAOWL4cHH0z8OUSkycvrzE5SupldYha1Tna83Jbip060ZEVkSEaGcaISmVfW2Nat9UJ93jw/p/jBB2HxYqiogHPP9VMg9t475kK9ROfjJbqgpkG70lBv5LrrSTzf9wu2ef55Pz969939bdddfbUQEZEo8jazU5ROZkeOQAfdNu0FULjUiZasaCwkE5HowpPGQm3v2qV+Id7o0fDuu/7gscf66Q/9+vmR3EZEq18dLfiinbdqzboGlywhmIUn0RbPXN/xYG5ZuYIt//lPWLnSn9ismZ/nvfvusNtu0KOHn0O9444qmycieZXZQWRjOpldNxc62WogiYiW2aorXTjUiZasSOTTerT5dXULVSqS+HQeeTmu2fpadv/2fxz65UcM+PIjdvhxnr+jVy+48koYPBi6dEn2j5TwNIvI86Jdukx2ZXos0X7xvd51L6bvsT/vX34wfP01TJ268fbxx/DssxtPPvJIP59aZfJEmrRcZna4oLIRUs/sOpnI7WiZrbrShUOdaElbtNqbkQst4s0zW+9coAsybt16OR+9NYbOP8yj+49z2enHubSsWcO6Zs1Yutd+cPUfYOBA6NYtkD9/shIdEYHkL/PF3YCgWTPYfnt/O+64jXeuWAGffw7PPQcjR8KAAXDOOXDiiQmNyotIYcm3zI417a2uAkg+dCYTze1AM1vynqpzSFriLQipk3CptiC88IKvjwwsbtWG2W07Mb/Tjmx3VD/2Pv80aNMmuNfKsFgj1rGqeEDqO3CNmlLFX16dxakvP8AvZ/+XLsu+Y3aHHVh85fX84renaWQ6QKrOIbmUd5kd1q5Cnxec7cyum2JSElr0mczovyROJe4kYxLdzjVajc3A/7PPmgX77+9HWseOhbZtg3vuHIj1dxurFivADS/PZOnq5EpERQZ/s/W1DPzsXS5773E6Lf+eRZX70e7eO2GffQL5czV16kRLLuVVZheZXGV2Mo+V5GW8xJ2ZHQHcBZQADznnRkTcfyfQL/RjS2Br51x56L5aYHrovnnOuYFBtEmCF22kINFLTguXVSdfqi0R//ufn4bw73/DlCmw+ebw1FMF34GG2JfzIj/2VtfUcv3omaxZt75BqJaXlXL9wPiXQyPn5K1vVsKonv14tfv+nDr1dS758BlfreRXv/ILMlOYPy75RZndNORlZhexXGV25HNrPnX2pN2JNrMS4P+Aw4AFwEQzG+2cm1V3jnPu0rDzfw/0DnuKaufcHum2QzIr1gri8palDT5FR5ORLU/vvRd+/3v//b77wu23w8knQ8eOwb9WDiSzpWy0ah8Am23SvNEwjRX8a5uX8s+9juH5XQ9hRtvP4bbbYL/94I03fIk8KUjK7KYhLzO7yOU6sxO9X4ITxKqhPsCXzrk5zrm1wNPAsXHOPxV4KoDXlYCNmlJF3xHj6DZsDH1HjGPUlKoN98VaQexcaIV2HEGurq7n3XehUydf6/nDD+Gyy4qmAw1+sU1Zaf1yc8nOTE4kTBv7Zdm6/ZZwzTUwYYKfG33AAfDVV0m2RPKIMrtIFFxmF7l8yWx9AMqeIKZzVADzw35eAESdPGlmXYBuwLiww5ua2SRgHTDCOTcqxmOHAEMAOnfuHECzBeovTgiftxVeq7Lu52iWVddQWlI/JpoZtC4rZdnqmszMo/vpJz9lY/x4vyFKp07BPXceibYavN/O7aIu9tm0tFnU0aVEwjTehjD1fpn27Akvvwx77QX/+Y+fey6FSJldwAoys5uIvMtsybggOtHRPmjFWq14CvBv51z4v3xn59xCM9sOGGdm051zDYa5nHMPAA+AX6SSbqMFrh41nScmzNvwjxVv3lYsJWb1aoQCrHfQskVzplx7eHCNdQ4mTYIHHvAd6FWrfJ3na64J7jXyULQ5iZVd2jaY5wjRa5j227kdfUeMS3hzgUZXem+9tf+6enUG/rSSJcrsAlVQmd1E5V1mS0YF0YleAIQPBXYEFsY49xTgwvADzrmFoa9zzOwd/Nw7XStOUrLlgUZNqaoXxrHEmrcFmd8KdQPn/FSNO+/0W1WfeqrfmrtPnyZZei3eYp94IyCRO2HFqxUbM4w328x//f3v4cYb/eh0jx5+jvQppxRUCcEmTJmdB4o6s6WenGa2ZFQQneiJwI5m1g2owofuaZEnmVl3oA3wYdixNsBq59waM9sK6AvcFkCbmpRUtg0dOXZ2o2HcmFuP65WxrVDr+dOffAf6ggtgxAjYYovgnruIRAZ13xHjYu6EBTR4zzw+Yd6G82K+h9q08fPPP/zQlxScORMef9xPsfniC//vJPlOmZ1jRZ/ZkpCsZLZkVNoLC51z64DfAWOBz4BnnXMzzexGMwsvfXQq8LSrX5h6F2CSmU0FxuPn181CkhJv29BYEhl1KCstoU3L0qj3VZSXMah3RdSFFIHNyVq7Fm65Ba6+Gk4/3VfjUAc6YfF2wopXIqlOzPfQvvvCpZfCgw/CBx/AsmVw9NHw/PP+qoHkNWV27hVtZktaMpbZkjGB1Il2zr0KvBpx7NqIn6+P8rgPgF5BtKEpS2Xb0MZK8VQ0Mm9r6IDuGy4tVdfUBjsnyzl49VX4wx/86OYJJ8DDD2sL6iTF+jfuUF6WVK3YRpn5f6NXXoHJk6GySe0jUpCU2blVdJktgchaZktg1CspArEuw8W7PBerFM/p+3bmmxFH8f6w/hsuNd16XC8qysswfFDX7bQ0/IXpG/7D1zq3IahTDuNvvoE//xl69/Yjm+A7Zs8+C6XRR1cktqEDujdYhV9aYgwd0D3hS7cJX+I95hgoKYEnnki2mSJNTtFktgQqq5ktgQhkJFpyK1q5m8Yuz0UrxRMrTKMtiog3dyupQF6xAh55xFfcmDDBH9t3X/jb32DwYHWe0xU5uyL0c7wSSXWSusTbti2cdBL89a/QqhXccIOuHIjEUNCZLZmVrcyWQKgTXQSSCdfIx6Uanqlcjozqz3/285732MMvGjzpJOjWLaU2SX0jx86mZn39RK5Z7xg5djbvD+u/4ZxoK72TqRVbd4l4UcVJ3FG5lGNuvtlPw3n0USjTqIhIpILObMmYbGd2so+ThtSJLhLphGsq4s3dSsp330GHDjBlSkAtkzqN/dIM4j1Tr8pASSm/738hX2yxLZc9+wjMnQsvvQTt26f1GiLFqGAzWzIm65mNqnqkS9dbJSWBrPB2DiZOLKqtuvNJKvMuk9VgxbgZ9+x9PFf++nqYNs3X8p47N7DXE5HUqCpH/stJZqOqHulQJ1pSEmvxSlKfZF991Xe0fvvbjLWzKcvGL81YIydPdayEd9+F5cv9osMVKwJ7TRFJXiCZLRmVy8zWtJ7UaDqHpCytS0vOwU03QdeucFqDfR4kAKnOu0xG3EvElZXw3HNw5JF+l8mXXvIVPEQkJ7I9hUSSk/PMlqSpEy258cAD8NFHcP/9qsCRQUH/0oy27Wz4NrUQMXJy2GFwzz3+asPll8MddwTWFhGRYpPzzJakaDqHZN+kSXDRRX6EcvDgXLdGElS3IKVqWTUOvyDl+clVHL9XRfxLxBdc4P+9//IXv8uhiIhkXMqZLQnTSLRk1+LFfne7bbaBxx5TLeECEmtByvjPF20ovxTTHXfAzJnwxz/CmWdCixYZbKmIiKSV2ZIQ9WAkOz7/HC65BHbYARYu9HNlt9wy162SJKS1IKV5c7j4YvjpJ3jnnWAbJiIiDWgRYeapEy2ZU1PjO8v9+8Muu/hdCI84Av77X1/6TApK2uWXDj3Ub74yenSArRIRkWiyUTKvqVMnWoK3Zo3fhbBzZ78D4Zw58Kc/wfz5fntvdaALUtrll8rKYMAAePZZWLkyAy0UEZE6qg2eeYF0os3sCDObbWZfmtmwKPefZWaLzOzT0G1w2H1nmtn/Qrczg2iP5NCUKbD33nD11bDnnvDKK/DVVzB8uHauK3CB1JkdOhQWLYI778xYOyUxym2R4qba4JlnzrnGz4r3BGYlwBfAYcACYCJwqnNuVtg5ZwGVzrnfRTy2LTAJqAQcMBnYyzm3NN5rVlZWukmTJqXVbglYTY0fbb75ZmjXzpewO/roXLdK8tFxx8Gbb/oPV1tvnevWZJ2ZTXbOVea4DVnNbWW2iBSqeJkdxEh0H+BL59wc59xa4Gng2AQfOwB40zm3JBTAbwJHBNAmybbbb4frr4dTToEZM9SBlthuvRWqq/1mO5Irym0RkTQFUeKuApgf9vMCYJ8o5x1vZgfiRz8udc7Nj/HYqNcZzGwIMASgc+fOATS7MEUWTg96N6OUffgh9Ozpy9ZJ0Qrk/de9OwwZ4heaHnIIDBqUmcZKPBnPbWW2l7eZLU2C3n+ZFcRItEU5FjlH5GWgq3NuN+At4J9JPNYfdO4B51ylc66yXbt2KTe2kEUrnD78hemMmlKV24atWgUTJ0KvXrlth2RUoO+/kSP9tuCnngrvvRd4W6VRGc9tZXYeZ7Y0CXr/ZV4QnegFQKewnzsCC8NPcM4tds6tCf34ILBXoo+VjWIVTh85dnZazztqShV9R4yj27Ax9B0xLvn/YBddBN9/D+eem1Y7JH+NmlLFZc9ODe79t9lmMGYMdOkCxxwD06cH1FJJkHI7C/I2s6XoBZ7ZElUQneiJwI5m1s3MWgCnAPUKwZrZtmE/DgQ+C30/FjjczNqYWRvg8NAxiSIThdPT/qT6+OPw8MNw5ZW+HrQUnbr3SG2MRcgpv/+22greeANatfKl7775JvVGSrKU21mQl5ktRS9jmS0NpN2Jds6tA36HD9HPgGedczPN7EYzGxg67SIzm2lmU4GLgLNCj10C3IQP9InAjaFjEkWqhdPjjVqkPFLy009www1+busBB/hFhVKUor1HwqVVuL9zZ3j9db/QcP/94bXXUn8uSZhyOzvyKrOlychoZks9gdSJds696pzbyTm3vXPultCxa51zo0PfD3fO9XTO7e6c6+ec+zzssQ8753YI3R4Joj3FKpXC6Y2NWiQ9UrJ6tZ/Put12vuN8xBF+84zmQaxRlXwUb9QikML9u+4K48ZB69bwy1/CWWfB0rhVLiUAyu3My4vMliYn45ktG2jHwgKSSuH0xkYtkhopWbQIevSAyy/3i8ImToQXXoBttkn5zyT5L9Z7pMQsuML9vXvDJ5/AVVf5KUI9esBLL6X/vCI5lPPMliYpK5ktQDAl7iSLBvWuSOo/QGOjFkMHdGf4C9PrhXbMT6qXXAILF/qNMg49NLmGS8GK9R4JPIw32cRv1nPccXD22b703dlnw0MPQTN93pfClNPMliYpa5ktGokuZqOmVNHMolWj2vhJNeGRktdegyef9AsI1YFuUrK+deyee/qrHMOGwSOP+I18RJqI8palUY8nndnSZOk9kj1pb/udC9pCtnGjplQx9Lmp1Kxv+O+b9CfSGTPgqKN8WbIpU/yIoUimOQcnnwwvvuhrSe+7b65bFIh82PY725TZibl61HQenzCvwfHSEmPkCburEySSA5ne9lvy0PWjZ0btQJuReAd62TI/hWOPPWDFCj8qqA60ZIsZPPAAdOzot5OfMyfXLRLJmFFTqngiSgcaYLMWzdWBFslD6kQXqWXVNVGPO0fjYbx+va/9vNNOcPfdfhOV//0P9om2K7BIBpWXwzPPwOLFflv5W26BNWsaf5xIgRk5dnb07XqB5THyXERyS51oqW/mTNhvPzjnHNhxR5g0Cf7+d9hyy1y3TJqqPn3gs8/g6KPh6qt9JY/vv891q0QCFa8smSpviOQnVecoUm1alrJ0dcPRizYxFq0AsHatr4ywdCn8619w+un+krpIrnXsCM89B6NHw7HH+kWul16a61aJBKZDeRlVUTrSBqq8IfE5539/r1u38VZT478uWQITJvireStWbLytXOkf17IllJZuvLVo4Y9tvrm/tW4NFRXQoQNsu63fYVb9gg3Uic5zo6ZUMXLsbBYuq6ZDedmGMI08FjlF47pjejL031Opqd14gbC0xLjumJ6xX+zuu+GLL2DMGL/phUi+GTjQT+t49VV1oiUvpZrZ0cqSGfDrfTtrPrQ0tGyZn2Y5erSvrf/NN40/pnlz3zFu1cp/Bb9bbE3NxtvatbBqle9gR1Na6q9Mt23rb61a1b9tvrnvbHfoAG3abDy2ww7+sUVG1TnyWN3OVeGhWtrMwKjXOY5VbSNamMcM4+++83OgDzwQXnklI38ekUBcfjn89a9+hJS3kGUAACAASURBVKVVq1y3JmmqzlG8sprZUnyc81eCZ83yVYkWL/YjxpG3pUt9/oEfFR4wAPr29aPIpaW+s1z3tXVrfyWvZ0/fmU1kFNk5vztx3WtVVcG33/rb4sUbb0uX+g53XbtWrYKffvIj4JHatPELxI86CrbfHrp1K5hCBfEyW53oPNZ3xLiol/eiqSgv4/1h/VN/sbPPhiee8HOid9wx9eeRJimrv/zHj4f+/WHUKD+1o8CoE128sprZUlhqazd2PJct81/rvp87F8aO9Z3ntWs3PqZz540jx5Gjvd26+d/Ve+zhv09SxjJ7/Xr48Ue/Mdvy5b5zvWSJ32vi6ac3jnCbQZcucMABPs8PO8xPG8lD8TJb0znyWLyFJumc28DEifDoozB0qDrQkrTI0beqZdUMf2E6kEAlmFT07et/ibz2WkF2oqV4ZS2zJb+tWePLc44du3EE9/vvfQczmpIS2H9/X1J2221hm218NawUOseJyGhmN2sGW2/tb+HOOAPuucdPGf3qK3+bMQNefx0ee8yf0707dOrkR8+32GLj17rvt9vOd7jziDrReSzWQpNY56bs73/3b9Crr079OaTJGjl2dr3L1wDVNbWMHDs7M53oFi38roYzZgT/3CJpyFpmS+6tX+8z6D//8Qv3qqr8tMjvv/ejywA9ekDXrr6iUIcOvmPZtq2f2lBe7r/W3Vq0yFrTs57Zdbbc0lf/2m+/jcec83+Pb7zh/y5//NF/6Fi+3E8NWbGi/vzsceOgX7/MtTFJgXSizewI4C6gBHjIOTci4v4/AIOBdcAi4P855+aG7qsFpodOneecGxhEmwpNtEsr0RaaxJpfl/Lqbef8J8EBA/ynPZEkxRpRy+hIW7duPnQlJcrs9OUssyU3Fi/20x2nTYN33vG3xYv9fR07+kzq1ctPS2jfHnbZxVe7ysNKFjnJ7FjM/N9br15w2WUN71+/3k8J+fJLOOQQPxK9xRZ+KkjdrWvXjd/37p3VBYxpz4k2sxLgC+AwYAEwETjVOTcr7Jx+wEfOudVmdgFwsHPu5NB9K51zSa0OKrb5ddEWo9QtPIGGq7qjHUv50+O0abD77n5zlbPPTvvPIk1PrHmgJWasdy4zc6TvvBP+8Af497/h+OODe94syPWcaGV2+nKa2ZI9zvnSmn/8I8yfv/F4p06+M9evHxx8sO+8FZBYmV1eVspmmzTP3/fpjz/6qR9z5vh55HW35cs3ntOnD1xxhe9Md+0ayIeYjC4sNLP9gOudcwNCPw8HcM7dGuP83sC9zrm+oZ+bfCDHekNnZeHJiBEwfLi/FNWhQ2ZfS4pStA5FpFjVCFK2Zo3/5TVjBnz0kb9sWiDyoBOtzE5TTjNbMm/1ar9O6O67YfZs2GsvOPVUX+GiZ08/8pyHI8yJSreKTN6pW5z50Ud+cGXVKn+8vNx3pvff38/JTnHNV6Y70ScARzjnBod+PgPYxzn3uxjn3wt855y7OfTzOuBT/GXDEc65UTEeNwQYAtC5c+e95s6dm1a7cy38UmCsfwEDvh5xVGYasH69LxM2bJgPiA8/zMzrSJMQ/n5uZkZtlFwJfJSjqsq/d7fYwi+Obd06jT9B9uRBJ1qZnYKcZ7YEyzk/svnFF/72+ed+Z9TPPvMjnevX+1HNCy/0pdmyOGc5GyKnI61euy7qBm0ZvaKYCdXVMH06TJnib598ApMn+3/PX/wCXn7Zz0tPQqarc0T7OBY1Y8zsdKASOCjscGfn3EIz2w4YZ2bTnXNfNXhC5x4AHgA/qpF+s3MnkZE7yODCk+++g7PO8iuHBw2Chx7KzOtIkzGod8WGcO02bEzUc5ZV17Cs2od0IKvBKyr8pdb+/f0ow6hRfmW4NEaZnaScZ7YEZ/VqvxvvuHH1pwG0aOH3Sujd2486H3qoL79WwCPO8YRnNsTO7boBkYxXXQpKWZn/8NOnz8ZjCxf6Er7vv+8XcQYoiE70AqBT2M8dgYWRJ5nZocBVwEHOuTV1x51zC0Nf55jZO0BvoEEgF5NoK2MjlTYzVq9dR7dhY4L9BPjaa74D/dNPvirHeecVbUhIbiRaoSCQ1eAHHAB/+QtcdBHccgtcc03qz9V0KLOTlNPMlvSsW+c7UfPn+9v99/sqEIMH+6kZO+3kb126+M1JmqhEcjsrFTwyoUMHX8J36NDAnzqIYZuJwI5m1s3MWgCnAKPDTwjNqbsfGOic+yHseBsz2yT0/VZAX2AWRS7eCljDX/bGYOnqGhwbPwGOmlKV+ouuWeNrUP7yl37l8OTJcP756kBL4IYO6E5ZaUlC5wayGvx3v/MjRzfdBAsWpP98xU+ZnaScZLak5+KL/dzlTTbxHeT99/c58c47cO21vo7zxRfDkUf6HfSacAcaEs9t1TevL+1OtHNuHfA7YCzwGfCsc26mmd1oZnWlj0YCrYDnzOxTM6sL7F2ASWY2FRiPn19X9IEc65JfRXkZX484is02aV5vcj9s/ASYkrpFWHfd5UfsPv64oBZiSWEZ1LuCW4/rRUV5GYZ/X7dpGb3kUCCXv83g1lv9HMfbbkv/+YqcMjt5Wc9sSdyqVf4K6z33+EVlgwb5ilN33+3XTVx1le8wv/aanyu7bBlcf32uW513InO7JMYAm6Ys1RfIRy/n3KvAqxHHrg37/tAYj/sA6BVEGwpJtFqi4XVDA6/heP31vhj8k0/6T+IiGRY53y5WSbBka+XG3Kq2Sxf4zW/gwQfhyiv9jl8SkzI7OVnPbInNOb8I8N134e23YcwYP88ZoGVLX6+5Wzc46CC44AJfr1kSEp7bGc/sItG0r1/kSN0bKNYbK9bcpJQ+AU6c6EfnBg9WB1pyprH3fCIa3ap2+HBfluqOO2DkyMD/DNJ0ZTWzxc9jnjkTvvnG38JrAs+ZA0uX+vO22QbOPNPXit91V78joKYoBiIrmV0E1InOkciRujqjplSxas26BsdT3uFq/Hhf2mXEiMbPFcmgWO/5RDW6Ve0OO8BJJ/mFQ9dcox04JVBZy+ymas4ceOstvxPpW2/Vr5xRVrZxR7rKSl954YAD/P95dZozJuOZXQTUic4jscootWlZynXH9Ez+Tbdsma+TCOpQSE4FcUkvoUvml10GTz/tyzb+4Q/pNFmkUYFndlOzerXvNF9/PUyd6o917AgnnuhLV+6wg991bqut1FnOsqxldoFTJzpHor1BY5VRatmieXJv3k8+8eXrnnjCFx4fMKDJrzyW3Anqkl5Cl8wrK+HAAzcuotX7XgKS0cxuKqqr4fbbfXWoGTP86LNzfie5u+6Cww+H7t3VYc6xrGZ2gdPOBDlQ9watCu18VfcGjVWjMeFPbe++C/vt53dxe/JJX1D+k0/g9dcVSpIz8S7pJSNaCaaol8z/8AeYNw+efz6l9opEylhmF7tVq/zV0Kef9qPNe+/ty8vNmgV77umPvfSSn/980UWw8876XZUHsp7ZBUzDNDkQ6w1aEmO75IQ+tb33HhxxhF9ocdddvlJBeXlQTRZJWawORdWyavqOGJfwZcKEF7occ4wf2Ro5Eo4+GjbbLO0/gzRtGcnsYrVggZ+i8cwzvnpGbejvzcxXzbjqKrj55ty2UeLKemYXMHWicyDWG7TWOcpKS5IvKTNpEhx1lF908e670K5dkM0VSUu8nbCSvUyY0EKXZs3giit8RZqtt4aBA+G00/y0phYtkm6/SOCZXeicgx9+8NMx5syBr77yZefef99fBQLYbjv44x/9FKvu3f0H2003zW27JSFZz+wCpukcORCvcH/kJhW3Htcr/htw5kw/Ar3llvDmm+pAS95pbCesjGxKcc45fmvfM87wo2IDB/qrNOeeC+PGbRwdE0lAoJld6D75BNq29f+ffvELP23wuuv81dA+feDOO/05X37pq0KdcAL06qUOdAHJSWYXKI1EZ1lj5ZCS+tQ2Zw4cdpgfXXvrLb+qWSTPhF/SS3QOafgirvKWpTgHy6trkrsceOCB/nbPPf4D5lNP+dtDD8G228KwYX4epkgj+u3cjicmzCN84kZKmV3IvvkG/v1vuOUWX/nphhv8+pvttvMVNMqa8BSWIpOzzC5A6kRnUaDlkKqq4JBDYO1aP+K2/fYBt1YkOHUdjb4jxjW6Wjvy/8nS1TUb7ktplXhpKfzyl/62erXf4ezaa+Gvf1UnWho1akoVz0+uqteBNuD4vYqs81xTA/Pn+znNVVX1b59/7rfMBr8T4IgRsO++uW2vZFROM7uAqBOdRYGVQ6qt9XOgFy/2m6n07BlgK0Uyp7HtkyH2/5M6aRXrb9nS16AdNw5eeCH5x0uTE+396IDxny/KTYOC8vPPcN99fk3N9Onw2We+Ix2uZUt/hbNLFzjrLDj2WA3YNDE5z+w8p050loyaUhVcOaQXX/SF6Z96yl9OEykQiazWTuT/Q9olxFq1gpUr03sOKWp1l6eLqoxdTY2fljFuHPzjHzBxIlRUwG67wZFHwk47+U5zRYW/tW6tknNNXN5kdp5SJzoL6i51xJJUOaT16+HWW33YnXhiAK0Tya7IOaSjplTRd8S4DQFd3rK03uXAaJqZ0W3YmNTn27Vq5ad2rF/vq3mIhIk19S5c3pexq62FCRN8HeZPP/UVNObO3biotls3P6Xp4otz207Je3mR2XkqkE60mR0B3AWUAA8550ZE3L8J8C9gL2AxcLJz7pvQfcOBc4Ba4CLn3Ngg2pRP4l3qSLoc0vDhfuXzY49BSezVsyKFINrOWKXNjNISo6a2Yf3dOnW1eVOeb1dXO3r1at+hboKU27E1dnk6b8rYOec3Lvnvf/185ro5zXVfq6v9moDddvOVM0491U/H6NMHevTQKLMkLWeZnafS7kSbWQnwf8BhwAJgopmNds7NCjvtHGCpc24HMzsF+DNwspn1AE4BegIdgLfMbCfnXFHVn4p3GSOpckgPPwy33QYXXAC//nVArRPJnWidlZr1jvKyUjbbpHmDld7NomxukdJ8u80391+XLGmSnWjldnzxMrsiH0bSVq70m2o99JCfngF+UKVDBz8do3dvv+lQnz6+BGrr1rlrqxSVnGV2ngpiJLoP8KVzbg6AmT0NHAuEh/GxwPWh7/8N3GtmFjr+tHNuDfC1mX0Zer4PA2hX3ohVuLyivCzxN9E778B558Hhh8Pdd2sEQYpCrM7K8uoaPr3u8AbHuw0bk9TzxFS3luC995rqB1LldhzxMvv9Yf2z04iff/ZTML79FhYu3Hj79lv/vv3+e//7YNgwX+q0c2dorhmaklk5y+w8FcT/uApgftjPC4B9Yp3jnFtnZsuBLUPHJ0Q8tvA/mkRIZHVrXP/7Hxx3nN/x6ZlnFJRSNGJ1VmLNN032/Jj23BO22grGjm2qnej8z+333/cLqHv39tMRsrh9e9qZnSjnfM3lJUtg6VJ/mzMH/u//YPZsX8I0XOvWvsb5Pvv4qX0qMydZlrPMzlNB9MaiDYlGToyJdU4ij/VPYDYEGALQuXPnZNqXc2ntH19b67csbtYMXnkFyssz3FqR7Em2sxJY56ZZMz+K99JL8PHH/rJ305Lx3E47s0eNgttv9983a+a3ju7b19cp3m03//MmmyT/vAlIK7MjOQc//ugX+U2b5hf31d3mzfPzlqM57TQ/JaOiwnect902qx8kRKLJWWbnqSA60QuATmE/dwQWxjhngZk1B1oDSxJ8LADOuQeAB+D/t3f3UVbV9R7H31950MF0DSoiDKLY9VIqCDK37LJuElpoGiD5eK1LZcvL8mFpGQmXHoyWS5LlxcryRmqWV1NbKlIuQwWpVsvMURBBRb2WykCBISiC6MD3/vHbA2dmzvOZc/be53xea501c87ZZ/aXOcxnvrP37/fb0Nramnv0ekKVfVWrhQvDOp533hmuDCVSR0ptVnq1uZk7NzQ2EybAHXfAmWeW+89Io6rndsWZfd11cNllsGJFmEz91FPwq1+FccCwt7E+6SQ44wxobQ2Xlt5333CrcMhbUZm9ezds3br3aPKGDWFS39q18OKLYbzyq6+GCaydDj00DL047rhwAaBhw+Dgg2HgwL23wYNh0KCK6hephlgzO4HMvbJ+NArXF4GTgXbgSeDf3X1NxjaXAKPcfUY0QWWau59jZscCdxLG0w0FlgJHF5qg0tra6m1tbRXVnQobN4ZfEmPHwtKlGgct0ts2bYLJk+GJJ+D66+GKK6r+c2ZmT7l7a1V3UriGmuZ2r2X2rl2wZk1YkeK550Jj/Yc/wNtv99y2f//QTA8YEI7itrSE5vSAA8Jtv/3CyhV9+4Zb5+fu0NERbrt2hbWV33kH3norNPObNoXn/vGPMPwi2+/QAQPCMqQjRoRLYh95JIwaFYZhDBhQ+fdBRGomX2ZXfCQ6Git3KbCEsFTSre6+xszmAm3uvhi4Bbg9moCymTCzm2i7ewiTWTqAS+pphndFOjrgi18M4f2jH6mBFqmGQYPChSc+/3n46lfDRK4bbqj7eQepze0+fcJQjtGj9z723nvhPXzlFdi5M0zIy/y4bVs4QtzeHo5qb9sWmu5SDyDtv39oivv2Dfs/+OBwO+igMMxu4MDQrA8dGm5af1yk7lV8JDoOdX8k2h0uvjhckvWmm2DGjLgrEqlvu3eHVQ7mz4fTT4e77qra0ndJOBJda4nLbPdwhLmjY+/Hzs/32Wfv0enO2377qSkWaVBVPRItvcwdrr46NNCzZqmBFqmFffYJY3CPOgouuQQ+/vEwkXfo0Lgrk2owC8M9+vePuxIRSbHG+NPaHRYsCGMek+zdd2H69DDhafp0uOaauCsSaSwzZoTm+aWXwvjV116LuyIREUmoxmiizeDxx+Gb30zuL8XNm+Hkk8PlvL/7XfjZz3T6UCQOp50WLm60bh3cc0/c1YiISEI1Tpc2f344Ij1zZtyV9LR5c7jiVFtb+KX9jW9oIqFInMaNC8uQJWkcr4iIJErjNNFHHAFXXRWa1N/9Lu5q9upsoFevDhcXOPvsuCsSEQjrDquJFhGRHBqniQb4+tfh8MPh8svD+p9VtGhFO+PnLWPErAcZP28Zi1a099yoewN92mlVrUlEStDaGpa8e/bZuCuRGigqs0VEMjRWEz1gQLiM7DPPwE9/WrXdLFrRzuz7nqV9yw4caN+yg9n3PdszlL/znfALWg20SPJMnx7W/T399LDOsNStojNbRCRDYzXREIZLnHRSGHf85ptV2cX8JWu7XCceYMf7u5i/ZG3XDV94AT74QTXQIkk0dGhYqWPz5nBZ6W3b4q5IqqTozBYRydB460Sbwfe/Hy6lfe21YW3YXrZ+y47Cjz/3HDz8cK/vW6TeLVrRzvwla1m/ZQdDm5uYOWkkU8e2VOf1J5wQLrwyZQqcf344a9SnTy/9SyQpispsESlLTTO7xhqviQY4/vhwmd8f/AAuuyyMky5Ttjd3aHMT7VnCd2hz0947d90VPs6eXfa+RRpN52n3zqOGnafdOxUK2nyvzxnKZ5wBP/xhuAjLFVeE3NDqOalVdmaLSMliyewaarzhHJ3mzg1L3s2ZU/aXyDWO7hMfGkRTv65Hq5r69WHmpJF77r91x908PeJ4Ruwer0ksIkXKddr96sVrihrTWvZp+4svhiuvhBtvhJ/8pDf+KRKDSjJbEw9FShdbZtdI4zbRRxwRfinefjv8+MdlfYlcb+5jL2zi2mmjaGluwoCW5iaunTZqz19Nj967nANfeZEHPniiJrGIlCDX6fUtO94vKmgrOm1/3XXhgkhz5sDWrcUVLIlSbmZr4qFIeWLN7BpozOEcnebODatjXHYZDBsGkyeX9PJ8b+7UsS3ZTzXs3MmhV17Ku33789A//+uehzv/8yTh9IRIUuU67Z5L95/Rik7b77NPaKTHjYMFC+Dqq4uuQ5KhrMwm/9EwZbZIbrFmdg1UdCTazA4ys0fM7KXo48As24wxs8fNbI2ZrTKzczOeu83M/mJmK6PbmErqKVnfvmFs8rhxcN558MQTOTfNdiov15uY8811h4suYvSra7jy019h4wEHd3k6KX9ZiSTVzEkjs552HzigX9btu/8s5np95mn7vE44AT77Wbj+enjjjeILT5DU53aReiWzI0k/GiaSVLFndpVVOpxjFrDU3Y8Glkb3u9sO/Ie7HwucCtxgZs0Zz8909zHRbWWF9ZRu//3h178O68FOmQLtPU/PVTKObo9168Jkxl/8gptPmc6DH/63Hpsk5S8rkaSaOrYl62n3b3/m2KJ+FnO9vqSjiXPnwjvvwPe+V/k/KB7pz+0CeiWzM5TbfIs0ukRkdhVVOpxjCjAh+vznwHLgqswN3P3FjM/Xm9lGYBCwpcJ9957Bg2HxYvjoR+Gss8Jlwfv33/N0oXF0eWeXbt0aftkuWBCORH/jGxxy5n/SdP/qLl8zSX9ZiSRZodPuhZZByvf6ohxzDHzuc/Dqq+FnOn0rddRHbudRUWZnMXPSyC4rBIAyW6RYsWd2FVXaRA929w0A7r7BzA7Nt7GZfQToD/xfxsPXmNm3iI6IuPvOCmsqz7HHwq23wrnnwk03hUuDR0oeR9fRAS+/DL/9LVxzTTjte8EF4fMjjmAqgFli1z0USZPuS5YtOHdM9X+WbrkF+mU/HZkC9ZPbOZQ79jmXzu2V2SKViyWzq6RgE21mjwKHZXmqpLXhzGwIcDsw3d13Rw/PBv5GCOiFhKMhc3O8/iLgIoDhw4eXsuvinX023HxzuBz3mDGwaxds384Frz/Jji1vs1/HTt7p38Tyo8axpenAcCpv584wlvrpp2HVqnBbswbefTd8zYkTYf78MJYyQ5L/shJJi9jWEE14A52E3K5JZudQjclIymyRyiV93edSmbuX/2KztcCE6GjGEGC5u/c4v2VmBxJOGV7r7r/K8bUmAF9z9zMK7be1tdXb2trKrjuvVavC1Qx37865SYftQ9vwUYxoGcjglX+G7dvDE4MHw+jRe29jx8Jxx6XxdK9IKoyftyxrs9TS3MQfZ02MoaLCzOwpd2+Ncf81z+2qZnYW3X9RQxh+kaSxlCKNqN4yu9LhHIuB6cC86OMDWXbeH7gf+EX3IDazIVGQGzAVWF1hPZUbPTocWX7jDRgwAJqaYMAAHn5lKzc+3g7t7Ux7vY1pr7Vx4Oa/wZe+BKecAieeGJpoEakZrZpQlvrL7W40/EIkmeotsyttoucB95jZhcBrwNkAZtYKzHD3LwPnAB8HDjazL0Sv+0I0o/sOMxsEGLASmFFhPb2jtecfHJ86Fj71mRhqEZGcyj1tn+3Szw3UYNVnbnej4RciyVNvmV3RcI641PrUoIgkUzmn7eM+1R/3cI44KLNFBOovsxv3st8iknrlrCGa7+pzIiJSPfWW2Y192W8RSb1ST9vX25g8EZE0qafMVhNdhqSOzRGRwqqx/JkkmzJbJL2SnNkazlGiXJeTXbSi5+XCRSR5Zk4aWdalnyWdlNki6ZbkzFYTXaIkj80RkcLKGZMn6aXMFkm3JGe2hnOUKMljc0SkOFr+rHEos0XSL6mZrSPRJco1BicJY3NERKQrZbaIVIua6MiiFe2Mn7eMEbMeZPy8ZTnHyyV5bI6ISCMpJreV2SJSLRrOQc+FvDsnngA9Th/ocrIiIvErNreV2SJSLWqiyT/xJFvQJnVsjohoObNGUUpuK7NFki2tua0mGk08EakXpZxVknRTbovUhzTntsZEo4knIvVCy5k1DuW2SH1Ic26riUYTT0TqhY5ONg7ltkh9SHNuV9REm9lBZvaImb0UfRyYY7tdZrYyui3OeHyEmT0Rvf5uM+tfST3lSvJC3iJSPB2dLEy5LSJJkubcrvRI9CxgqbsfDSyN7mezw93HRLfJGY9/D1gQvf5N4MIK6yla96WRAP44ayILzh0DwFfuXpl3qTsRSZ7eOjpZ7JKXKZXK3M72nkwd28LMSSMZ2tzE+i07mL9kbb29VyJ1L825be5e/ovN1gIT3H2DmQ0Blrt7j3+1mW1z9w90e8yATcBh7t5hZh8Drnb3SYX229ra6m1tbWXX3X0QO4Q37LPjWrj3qfYej+vohkh6VDrLO1c+9FYOmNlT7t5a8Rcqf/81z21ltojkk+TczpfZlTbRW9y9OeP+m+7e49SgmXUAK4EOYJ67LzKzQ4A/ufs/RdscDjzk7scV2m+lgTx+3jLas4y16WPGrizfj5bmJv44a2LZ+xOR9MiVD72VAwloomue28psEammauZ2vswuuMSdmT0KHJblqTkl1DDc3deb2VHAMjN7Fngry3Y5O3ozuwi4CGD48OEl7LqnXIPVs4Vxvu1FpP6keZJLpyTktjJbRGolrtwu2ES7+ym5njOzv5vZkIzTghtzfI310cdXzGw5MBa4F2g2s77u3gEMA9bnqWMhsBDCUY1CdecztLmppKMaaRjcLiLFKXTaMFc+pCkHkpDbymwR6S1Jze1KJxYuBqZHn08HHui+gZkNNLN9o88PAcYDz3kYR/IYcFa+11dDtkHsRjiqYd221ZJJIvWjc9xc+5YdOHsX9c+cgNIAS6elLreV2SKNK8m5XWkTPQ/4pJm9BHwyuo+ZtZrZzdE2HwbazOwZQvjOc/fnoueuAr5qZi8DBwO3VFhPUTKXRoIQxp3HMjy6D1oySaTeFLOofwMsnZa63FZmizSuJOd2RRML41LpJJVM1Z5EJCLJMWLWg1kH8Brwl3mn16SGuCcWxkGZLSLliju382V2w1+xsB4mEYlIcdK8qL8EymyRxpLk3G74JjrJb46I9K4GGO9c95TZIo0lybnd8E10kt8cEeldDTDeue4ps0UaS5Jzu+ASd/Wu802o5Eo5IpIeU8e26Oc7xZTZIo0nqbnd8E00JPfNERGRnpTZIpIEaqJFREpQaNF/ERFJjmpmtppoEZEidS7637lmaeei/4AaaRGRhKl2Zjf8xEIRkWIVs+i/iIgkQ7UzW020iEiR998rQgAACEhJREFUtEaxiEh6VDuz1USLiBRJaxSLiKRHtTNbTbSISJG0RrGISHpUO7M1sVBEpAidM7x3vL+LPmbscqdFq3OIiCRSLTJbTbSISAHdZ3jvct9zNEMNtIhIstQqsytqos3sIOBu4Ejgr8A57v5mt20+ASzIeOhDwHnuvsjMbgNOArZGz33B3VdWUpOISKW6ryu6/b2OnDO809ZEK7dFpN7EldmVjomeBSx196OBpdH9Ltz9MXcf4+5jgInAduDhjE1mdj6vIBaRuHUewWjfsgMnrCv65vb3s26b0lU5lNsiUjfizOxKh3NMASZEn/8cWA5clWf7s4CH3H17hfstm642JiL5ZFtXNJeUrsqh3BaRuhFnZld6JHqwu28AiD4eWmD784BfdnvsGjNbZWYLzGzfCuvJK9tfK7Pve5ZFK9qruVsRSZFij1SkeFUO5baI1I04M7tgE21mj5rZ6iy3KaXsyMyGAKOAJRkPzyaMtfsX4CDyHA0xs4vMrM3M2jZt2lTKrvfQ1cZEpJBcRyqam/rR0tyEAS3NTVw7bVRij4YmIbd7I7NBuS0i+cWZ2QWHc7j7KbmeM7O/m9kQd98Qhe3GPF/qHOB+d98zUKXzaAiw08x+BnwtTx0LgYUAra2tXqjubHS1MREpZOakkV1mdUM4gnH15GMT2zR3l4Tc7o3MBuW2iOQXZ2ZXOpxjMTA9+nw68ECebc+n2ynBKMAxMwOmAqsrrCcvXW1MRAqZOraFa6eNSs1R5zIot0WkbsSZ2ZVOLJwH3GNmFwKvAWcDmFkrMMPdvxzdPxI4HPhdt9ffYWaDAANWAjMqrCevXH+tpHRco4hUydSxLfXUNHen3BaRuhJXZlfURLv7P4CTszzeBnw54/5fgR7/OnefWMn+S9X5DdYsbxFpVMptEZHe0XBXLKzzI0wiInVHuS0iSdQQTbTWGBURSQ9ltoikQd030d2vn965xiigUBYRSRhltoikRaWrcySe1hgVEUkPZbaIpEXdN9FaY1REJD2U2SKSFnXfRGuNURGR9FBmi0ha1H0TPXPSSJr69enymNYYFRFJJmW2iKRF3U8s1BqjIiLpocwWkbSo+yYatMaoiEiaKLNFJA3qfjiHiIiIiEhvUxMtIiIiIlIiNdEiIiIiIiVSEy0iIiIiUiI10SIiIiIiJVITLSIiIiJSInP3uGsomZltAl4t8WWHAG9UoZxSJaUOSE4tqqOnpNSiOnqqtJYj3H1QbxWTBsrsXpOUWpJSBySnFtXRU1JqqVpmp7KJLoeZtbl7q+rYKym1qI6eklKL6ugpSbXUs6R8n5NSBySnlqTUAcmpRXX0lJRaqlmHhnOIiIiIiJRITbSIiIiISIkaqYleGHcBkaTUAcmpRXX0lJRaVEdPSaqlniXl+5yUOiA5tSSlDkhOLaqjp6TUUrU6GmZMtIiIiIhIb2mkI9EiIiIiIr2iIZtoM/uambmZHRLT/r9rZqvMbKWZPWxmQ2OqY76ZvRDVcr+ZNcdRR1TL2Wa2xsx2m1nNZ/Oa2almttbMXjazWbXef0Ydt5rZRjNbHVcNUR2Hm9ljZvZ89L5cHlMd+5nZn83smaiO78RRR0Y9fcxshZn9Js46Go0yu0stichtZfaeOpTZXetoqMxuuCbazA4HPgm8FmMZ8919tLuPAX4DfCumOh4BjnP30cCLwOyY6gBYDUwDfl/rHZtZH+BHwGnAMcD5ZnZMreuI3AacGtO+M3UAV7r7h4ETgUti+p7sBCa6+/HAGOBUMzsxhjo6XQ48H+P+G44yu4ek5LYyO7gNZXamhsrshmuigQXA14HYBoO7+1sZd/ePqxZ3f9jdO6K7fwKGxVFHVMvz7r42pt1/BHjZ3V9x9/eAu4ApcRTi7r8HNsex7251bHD3p6PP3yaEUEsMdbi7b4vu9otusfy8mNkw4HTg5jj238CU2V1rSURuK7MDZXaPOhoqsxuqiTazyUC7uz+TgFquMbPXgQuI96hGpy8BD8VdRExagNcz7q8jhvBJKjM7EhgLPBHT/vuY2UpgI/CIu8dSB3ADoZnbHdP+G44yu6BGzW1ldh7K7D2qntl9q/WF42JmjwKHZXlqDvBfwKfirsPdH3D3OcAcM5sNXAp8O446om3mEE4F3VGNGkqpJSaW5TEtWwOY2QeAe4Eruh2Nqxl33wWMicZ+3m9mx7l7TccfmtkZwEZ3f8rMJtRy3/VOmV16LdE2Vc9tZXb6KLODWmV23TXR7n5KtsfNbBQwAnjGzCCcAnvazD7i7n+rVR1Z3Ak8SJUCuVAdZjYdOAM42au83mEJ35NaWwccnnF/GLA+ploSw8z6EcL4Dne/L+563H2LmS0njD+s9SSe8cBkM/s0sB9woJn9r7t/rsZ11B1ldum11Cq3ldnposzuoiaZ3TDDOdz9WXc/1N2PdPcjCT+EJ1QjjAsxs6Mz7k4GXqh1DVEdpwJXAZPdfXscNSTEk8DRZjbCzPoD5wGLY64pVha6lluA5939v2OsY1Dn6gNm1gScQgw/L+4+292HRdlxHrBMDXR1KbNz1qLcVmb3oMzuqlaZ3TBNdMLMM7PVZraKcKoylqVogBuBA4BHoqWb/iemOjCzM81sHfAx4EEzW1KrfUeTdC4FlhAmY9zj7mtqtf9MZvZL4HFgpJmtM7ML46iD8Ff854GJ0f+NldFf9LU2BHgs+ll5kjC+TsvLSa0lJbMhIbmtzA6U2T00VGbrioUiIiIiIiXSkWgRERERkRKpiRYRERERKZGaaBERERGREqmJFhEREREpkZpoEREREZESqYkWERERESmRmmgRERERkRKpiRYRERERKdH/AxLAPez3VQ2aAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide_input\n",
    "#id img_betas\n",
    "#caption Momentum with different beta values\n",
    "#alt Graph showing how the beta value influences momentum\n",
    "x = np.linspace(-4, 4, 100)\n",
    "y = 1 - (x/3) ** 2\n",
    "x1 = x + np.random.randn(100) * 0.1\n",
    "y1 = y + np.random.randn(100) * 0.1\n",
    "_,axs = plt.subplots(2,2, figsize=(12,8))\n",
    "betas = [0.5,0.7,0.9,0.99]\n",
    "idx = x1.argsort()\n",
    "for beta,ax in zip(betas, axs.flatten()):\n",
    "    ax.scatter(x1,y1)\n",
    "    avg,res = 0,[]\n",
    "    for i in idx:\n",
    "        avg = beta * avg + (1-beta) * y1[i]\n",
    "        res.append(avg)#/(1-beta**(i+1)))\n",
    "    ax.plot(x1[idx],np.array(res), color='red');\n",
    "    ax.set_title(f'beta={beta}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see in these examples that a `beta` that's too high results in the overall changes in gradient getting ignored. In SGD with momentum, a value of `beta` that is often used is 0.9.\n",
    "\n",
    "`fit_one_cycle` by default starts with a `beta` of 0.95, gradually adjusts it to 0.85, and then gradually moves it back to 0.95 at the end of training. Let's see how our training goes with momentum added to plain SGD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to add momentum to our optimizer, we'll first need to keep track of the moving average gradient, which we can do with another callback. When an optimizer callback returns a `dict`, it is used to update the state of the optimizer and is passed back to the optimizer on the next step. So this callback will keep track of the gradient averages in a parameter called `grad_avg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def average_grad(p, mom, grad_avg=None, **kwargs):\n",
    "    if grad_avg is None: grad_avg = torch.zeros_like(p.grad.data)\n",
    "    return {'grad_avg': grad_avg*mom + p.grad.data}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use it, we just have to replace `p.grad.data` with `grad_avg` in our step function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def momentum_step(p, lr, grad_avg, **kwargs): p.data.add_(-lr, grad_avg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Optimizer, cbs=[average_grad,momentum_step], mom=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Learner` will automatically schedule `mom` and `lr`, so `fit_one_cycle` will even work with our custom `Optimizer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.856000</td>\n",
       "      <td>2.493429</td>\n",
       "      <td>0.246115</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.504205</td>\n",
       "      <td>2.463813</td>\n",
       "      <td>0.348280</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.187387</td>\n",
       "      <td>1.755670</td>\n",
       "      <td>0.418853</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = get_learner(opt_func=opt_func)\n",
    "learn.fit_one_cycle(3, 0.03)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAD4CAYAAAAwyVpeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3yV9fn/8deVTfYkjEz2JpAQQETcolVRBAGto9pqtWqdraNaZx1UqbXan7TiqBNwoYI4EBVkJIywAyEJJKwkZJKdnM/vjxz8pjRAgJzcZ1zPx+M8PLnPfd95H4STK/d9fT4fMcaglFJKKaWUchwvqwMopZRSSinl7rToVkoppZRSysG06FZKKaWUUsrBtOhWSimllFLKwbToVkoppZRSysF8rA7QGaKjo01SUpLVMZRS6oStWbOmxBgTY3WOzqSf2UopV3Wsz2yPKLqTkpLIzMy0OoZSSp0wEdlldYbOpp/ZSilXdazPbG0vUUoppZRSysG06FZKKaWUUsrBtOhWSimllFLKwbToVkoppZRSysG06FZKKaWUUsrBHFp0i8hEEckWkRwRub+N1/1F5AP766tEJMm+PV1E1tsfWSJyeXvPqZRSSimllLNxWNEtIt7Ay8CFwCBghogMOmK3G4EyY0wfYBbwrH37JiDNGJMCTAReFRGfdp5TKaWUUkopp+LIebrTgRxjTC6AiLwPTAK2tNpnEvCo/fl84B8iIsaYmlb7BADmBM6pOlh9UzMrc0vJ3l9JTUMzgX7eBPn7EOzvQ3igHwmRgfQM74Kfj3YrKaWsMevr7UQG+TEsLoyU+HBExOpISikXUVHTyP7KOg5U1lFUVU9pdT01Dc3MSE8gNjSgw76PI4vunkBBq68LgdFH28cY0yQiFUAUUCIio4E5QCJwjf319pwTABG5CbgJICEh4dTfjQdqthneWbWLv32zg9LqhmPu6yXQPawL/buFMCI+nBEJEQyPDyMkwLeT0iqlPFVjs43ZP+RS29gMwIBuITx66WDG9IqyOJlSytkUVdWxMreUzXsr2LK3kq37qig5VN/mvuP7xrhM0d3WZQbT3n2MMauAwSIyEHhTRBa185zYj58NzAZIS0trcx91dBW1jdz5/jq+yy5mbK8ofnNGMqmJkQT7+1Db2Ex1fRPV9U2UHGpgd2kNuw9Ws6u0hs17K1myrQgAEUiJD+fcgbGcNyiWvl2D9eqTUqrD+Xp7seXxCyiuqmfJtiJeXprD9Nkruff8fvzurD76uaOUB2tqtrEi9yDfZxezLKeEbfurAPDz9qJvbDBn9o+hf2wI3cMDiA0NoGuIP5FBfnTx9cbHu2Pv4Duy6C4E4lt9HQfsPco+hSLiA4QBpa13MMZsFZFqYEg7z6lOUWVdI9e8toqt+yp54rIh/HJ0wn/90Aq2t5YA9IqB9OTI/zq+oraRrIJyMneVsTS7iJmLs5m5OJuEyEAuS+nB1LR44iMDO/U9KaXcm4jQNTSA6ekJXJrSg4c+3sRfv9pOVV0TD1w00Op4SqlOZIxh7e5yFqzfwxcb91FyqAE/Hy9GJUXwx4kDOL1PNAO6h+DbwUX18Tiy6M4A+opIMrAHmA5cdcQ+C4DrgBXAFGCJMcbYjymwt5QkAv2BfKC8HedUp6Cx2cZNb2WyZW8lr16TyjkDY0/4HGFdfDmjXwxn9Ivh7vP6caCyjm+3FrFo0z5e+i6Hl77LYVzvaKaNimfikG6d/pdeKeXeAv18eOHK4YQE+PDqD7l0Dwvg+nHJVsdSSjlYdX0TH64t5I3l+eSWVOPv48W5A2O5ZHgPJvSLoYuft6X5HFZ02wvm24DFgDcwxxizWUQeBzKNMQuA14D/iEgOLVe4p9sPPx24X0QaARtwqzGmBKCtczrqPXiiZxdtY2VuKS9cOfykCu62xIYGcNXoBK4ancCe8lrmZRYwL7OQ299bR1xEF246oxdTU+Mt/8eglHIfIsKfLxnMvoo6nvhiK8PiwxmZEGF1LKWUA+wtr+X15Xm8n1FAVV0Tw+PDmTllGBOHdHOqsWVijPu3O6elpZnMzEyrYzi9H7YXc+2c1Vw3NpHHJg1x6Pey2Qzfbivin0tzWLu7nKggP244PZkbxiVr8a1UKyKyxhiTZnWOztSRn9mVdY1c9OKPiMCXvz+DIH9H3uBVSnWmoso6Xlm6k3dX7abZGC4c0o0bTk+29BfsY31m66ePAqCqrpH7P9xA75igTul/9PISzhsUy7kDu5KRX8YrS3OYuTibt1bkc/d5/ZiSGo+3lw5+UkqdmtAAX2ZNS2Hq/1vB377ZzkO/0KUdlHJ1FTWNvLI0hzdX5NPYbJiaGsdtZ/chLsK5x4tp0a0AeGlJDvsq65j/29MI8O28K80iQnpyJOnJ6WTkl/KXhVv544cbmbMsn4d+MZAz+sV0WhallHsalRTJ9FHxzFmezxWpcQzoFmp1JKXUSbDZDHMzC3hucTZlNQ1cltKT35/Tl6ToIKujtYuOYFPklVTz+vI8poyMIzXRulsyo5Ii+eiW03jl6pHUNTVz7ZzV3PHeOoqr2p4/Uyml2uuPEwcQ7O/D0wu3WR1FKXUSsgrKufyV5dz/0UZ6xwTxxe3jmTUtxWUKbtCiWwEzF2/Dz9uL+yb2tzoKIsJFQ7vz1V1ncOe5ffly037OeX4p76/ejc3m/uMPlFKOERHkx61n9ub77cX8tLPE6jhKqXaqa2zmic+3cNkry9lbUcesacOZe/NYBvVwvTtWWnR7uM17K1i4cT83np5M15COW3XpVPn7eHPnuf1Y+PvxDOweyv0fbeSaOavYV1FrdTSllIu67rQkuocF8PxX2/GESQSUcnVrd5dx0Ys/8tqyPK5KT2DJPRO4fEScyy54pUW3h/vbNzsIDfDhxvG9rI7Spj5dg3n/pjE8PXko63aXM/FvP/LFhn1Wx1JKuaAAX29+O6E3a3aVsTqv9PgHKKUs0dBk45lF25jyz5+ob7Lx9o2jeeryoU41/d/J0KLbg+UWH+LrLQe4flwyYV2c9y+yiDAjPYGFd4wnOTqI3727lrvnrudQfZPV0ZRSLmbaqHiig/34x3c5VkdRSrWhoLSGqa+u4P99v5Mr0+L58s7xnN432upYHUKLbg/2+vJ8/Ly9uGZMotVR2iUpOoh5vx3L78/pyyfr9nDpP5ax40CV1bGUUi4kwNebX41L5scdJeQU6eeHUs7kq837+cXffyS36BCvXD2SZ64Y5vJXt1vTottDVdQ0Mn9NIZem9CAmxN/qOO3m6+3FXef1451fj6GytpFJLy/ns6y9VsdSSrmQaaPi8fP24u2Vu62OopQCmpptPPn5Fm76zxoSo4L4/I7TuWhod6tjdTgtuj3Uexm7qW1s5oZxyVZHOSlje0fx+e0tgyxvf28dj3+2haZmm9WxlFIuIDrYnwuHduPDNYXUNGibmlJWqqhp5FdvZPDvZXlcOzaR+beMJTHKdaYBPBFadHugxmYbb/6Uz9heUS455c5h3cICeO83Y7j+tCTmLM/jV29kUFnXaHUspZQLuGZMIlX1TXy6Xu+UKWWVnKIqJr28jJW5B3nuimE8PmkI/j6dt0BfZ9Oi2wN9veUA+yrquPF017zK3ZqfjxePXjqYZyYPZcXOg1zxyk8UlNZYHUsptyIiE0UkW0RyROT+Nl5PFJFvRWSDiCwVkbgjXg8VkT0i8o/OS31sqYkRDOgWwn9W7NLpA5WywNLsIi5/+ScO1Tfx3m/GcOWoeKsjOZwW3R5oXmYB3cMCOGtAV6ujdJjp6Qm8dUM6ByrruPyV5azdXWZ1JKXcgoh4Ay8DFwKDgBkiMuiI3f4KvGWMGQY8Djx9xOtPAN87OuuJEBF+OSaRLfsqWVdQbnUcpTzK3IwCbnwzk/jIQD697XTSkiKtjtQptOj2MEWVdXy/vZjJI3vi7eWak8sfzWl9ovno1nEE+fswY/ZKvtlywOpISrmDdCDHGJNrjGkA3gcmHbHPIOBb+/PvWr8uIqlALPBVJ2Q9IZeN6EmgnzdzMwqsjqKURzDG8OI3O/jDhxs4rXcUc387lp7hXayO1Wm06PYwn6zfg83A5JFxx9/ZBfXpGszHt45jQLcQbn57DR+uKbQ6klKurifQuiottG9rLQu4wv78ciBERKJExAt4HrjveN9ERG4SkUwRySwuLu6A2McX7O/DBYO7sWjTfuqbmjvleyrlqZqabTz48UZmfbOdySN7Muf6UQT7+1gdq1Np0e1BjDHMX1PIyIRwescEWx3HYSKD/HjnN2MY0yuSe+Zl8e8fc62OpJQra+uW2JFN0PcCE0RkHTAB2AM0AbcCC40xx72UbIyZbYxJM8akxcTEnGrmdrs0pQcVtY18n905hb5SnqihycZt767jvdUF3HZWH56fOhxfb88rQT3vHXuwjXsq2H7gEFNS3X+wQrC/D3OuH8VFQ7vx5Bdbmbl4mw6WUurkFAKtPzTigP+a8sMYs9cYM9kYMwJ4yL6tAhgL3CYi+bT0fV8rIs90Sup2Or1PNJFBfnyq8/0r5RB1jc3c9J9Mvty8n0cuHsS9F/RHxL3aW9vLs67re7gP1xTi5+PFL4a534TzbfH38ealGSMJ67KJl7/bSWOz4YELB3jsP3alTlIG0FdEkmm5gj0duKr1DiISDZQaY2zAA8AcAGPM1a32uR5IM8b8z+wnVvL19uIXQ7szN7OAQ/VNHne7WylHqq5v4tdvZrIy7yBPTx7KjPQEqyNZSq90e4jGZhsLsvZy/qBYwrq4z5Kqx+PtJfzl8iFcNzaR2T/k8sTnW/WKt1InwBjTBNwGLAa2AnONMZtF5HERudS+25lAtohsp2XQ5FOWhD1Jl43oQX2Tja8277c6ilJuo7KukWteW8Xq/FJmXZni8QU36JVuj7Eqt5SymkYuHd7D6iidTkR49NLBeHkJc5bn0Wyz8eilg/WKt1LtZIxZCCw8YtsjrZ7PB+Yf5xxvAG84IN4pG5kQQVxEFz5Zv9dtB5kr1Zmq65v41esZbCis4OWrRjBxiGfcYT8eLbo9xBcb9xHk580Z/TpvgJIzEREeuXgQPl7Cv37MA9DCWykFtHw+XDK8B7N/yKW0uoHIID+rIynlsmobmrnxzQzWF5TzjxlacLem7SUeoKm55bbp2QNjCfB13+VVj0dEePCigfxmfDJvrtjFzMXZVkdSSjmJC4d0o9lmWLKtyOooSrmsw4MmV+WV8sKVw7lwqBbcrWnR7QFW55dysLqBi4Z0szqK5Q4X3leNTuCVpTt5+bscqyMppZzA0J5hdAsN4Ost2tet1MloaLJx6ztr+XFHCc9dMYxJKUdO56+0vcQDLNq4ny6+3pzZ332WfT8VIsKTk4ZQU9/EzMXZhAT4cO3YJKtjKaUsJCKcNyiW+WsKqWts9ui7gkqdqMZmG7e/t5Yl24r4y+VDmZrm/lMTnwyHXukWkYkiki0iOSLyP9NEiYi/iHxgf32ViCTZt58nImtEZKP9v2e3Omap/Zzr7Q+tJI+h2Wb4cvN+zhoQQxc//SFymJeXMHPqcM4bFMsjn25mvq5cqZTHO39wLLWNzSzbUWJ1FKVchjGG+z/cyOLNB/jzJYO4arTOUnI0Diu6RcQbeBm4EBgEzBCRQUfsdiNQZozpA8wCnrVvLwEuMcYMBa4D/nPEcVcbY1LsD23AO4bM/FKKq+q5SPuq/oevtxcvzRjB6X2i+cP8LBZt3Gd1JKWUhUYnRxHi78NX2mKiVLvNXJzNh2sLuevcfvxqXLLVcZyaI690pwM5xphcY0wD8D4w6Yh9JgFv2p/PB84RETHGrDPGHF4ebDMQICL+Dszqtr7acgA/Hy/O0taSNgX4ejP72lRGJETw+w/Wk5FfanUkpZRF/Hy8OGtAV77dWkSzTefzV+p43liexytLd3LV6ATuOKeP1XGcniOL7p5AQauvC+3b2tzHvgBDBRB1xD5XAOuMMfWttr1uby15WI4y55uI3CQimSKSWVxcfCrvw6Ut2VbE2F5RBOkqa0cV6OfDv69NIy6iC79+M5OcokNWR1JKWeT8wbEcrG5g7e4yq6Mo5dQ+37CXxz7fwvmDYnli0hCdgrcdHFl0t/Wnf+Slg2PuIyKDaWk5ubnV61fb207G2x/XtPXNjTGzjTFpxpi0mBjPnJs6t/gQeSXVnDNQr3IfT0SQH2/+Kh1fby+um7Oaoqo6qyMppSwwoV8Mft5efL3lgNVRlHJaP+0s4e4PshiVGMnfZ4zA20sL7vZwZNFdCLQevhoH7D3aPiLiA4QBpfav44CPgWuNMTsPH2CM2WP/bxXwLi1tLKoNh+eb1daS9omPDGTO9WmU1TRwwxsZVNc3WR1JKdXJQgJ8GZUcwffZnnuHVKlj2bqvkpveWkNSdCD/ujZNZ/o5AY4sujOAviKSLCJ+wHRgwRH7LKBloCTAFGCJMcaISDjwBfCAMWb54Z1FxEdEou3PfYGLgU0OfA8u7dutRfSPDSE+MtDqKC5jWFw4L181kq37qrj1nbU0NtusjqSU6mRn9utK9oEq9lXUWh1FKadSVFnHjW9kEOzvw5s3pBMW6Gt1JJfisKLb3qN9G7AY2ArMNcZsFpHHReRS+26vAVEikgPcDRyeVvA2oA/w8BFTA/oDi0VkA7Ae2AP8y1HvwZVV1DaSkV/K2dpacsLOGtCVpy4bwvfbi3n4k00YowOqlPIkE/q3tCTq1W6l/k9dYzO/eSuT8tpGXrs+je5hXayO5HIcOrrOGLMQWHjEtkdaPa8DprZx3JPAk0c5bWpHZnRXP+4opslmOGeAFt0nY3p6AoVltfzjuxz6xoZw4+k6DZJSnqJv12C6hwXw/fZipqfrnMNK2WyGe+ZmsWFPBbOvSWNwjzCrI7kkXQbeTS3ZWkR4oC8jEiKsjuKy7j6vHxcMjuWpL7bwXbZOB6+UpxARJvSLYdmOEm0xUwqY9c12vti4jwcvHMh5g2KtjuOytOh2Q802w3fZRZzVv6uOKD4FXl7CrGkpDOgWyh3vriOnqMrqSEqpTjKhXwxV9U2sLyi3OopSlvp4XSEvLclh+qh4fj1e7/qeCi263dCmPRWU1TQyoZ9nTpXYkQL9fPjXdWn4+3pz45uZlFU3WB1JKdUJxvWNxttLtK9bebQ1u0r54/yNjO0VxeM6F/cp06LbDf24o+WHxOl9oy1O4h56hnfh1WtS2Vdexy3vrNHbzUp5gNAAX1ITIli6XVvLlGfaX1HHzf9ZS4/wAP75y5H4+WjJeKr0T9AN/bCjhME9QokO9rc6ittITYzgmSuGsjK3lEcXbLY6jlKqE0zoH8OmPZWUHKo//s5KuZG6xmZufnsNtQ1N/OvaNMID/ayO5Ba06HYzh+qbWLurjPF9tbWko00eGcfNE3rxzqrdfJCx2+o4SikHG9en5W7hTzsPWpxEqc5jjOGRTzeRVVDO81em0Dc2xOpIbkOLbjezcudBmmyGM/ppa4kj/OGCAZzeJ5qHP91Mlg6wUsqtDe0ZRkiADyt2llgdRalO8/bKXczNLOT2s/swcUg3q+O4FS263cyPO4rp4utNaqJOFegI3l7C32eMICbYn1veXsNBve2slNvy9hLG9IpieY5e6VaeYXVeKY99toWzB3TlrnP7WR3H7WjR7WZ+3FHCmF6R+Pt4Wx3FbUUG+fHqNakcrG7g9vfW0aQDK5VyW+N6R7G7tIaC0hqroyjlUPsqarn1nTXERwYya1oKXjrlcIfTotuNFJTWkFtSrf3cnWBIzzCeunwoP+08yMzF2VbHUUo5yGk/93Vri4lyXw1NNm59Zy21Dc3MviaVsC6+VkdyS1p0u5FlOS0/FLSfu3NMSY3jmjGJvPpDLl9s2Gd1HKWUA/TtGkxMiL8OplRu7ZlF21i3u5znpgzXgZMOpEW3G1mWU0K30AB6xwRbHcVjPHzxIEYmhHPf/Cx2Fh+yOo5SqoOJCKf1juKnnQcxxlgdR6kOt2jjPuYsz+P605L4xbDuVsdxa1p0uwljDKtyDzK2d5SuGNWJ/Hy8ePnqkQT4evO7d9ZS19hsdSSlVAcb1zua4qp6dhTpL9bKveSXVPOH+RsYHh/OgxcNtDqO29Oi203sKDpEyaEGxvaKsjqKx+ke1oUXrhzOtv1VPPaZLpyjlLsZ27vlc3V5jvZ1K/dR19jMLe+sxctLePmqEbriZCfQP2E3scLeb3j4h4PqXGf278qtZ/bmvdUFfLJuj9VxlFIdKD4ykITIwJ8/Z5VyB48u2MzWfZXMmjacuIhAq+N4BC263cSKnQfpGd6F+Ej9h2OVu8/rR3pSJA9+vJEcvQ2tlFsZnRxJRn4pNpv2dSvX9+GaQt7PKODWM3tz9oBYq+N4DC263YDNZliZd1CvclvMx9uLv88YQRd7f3dtg/Z3K+Uu0pMjKatpJEcHTCsXl72/ioc+2cjo5EjuPk8XwOlMWnS7gW37qyivaeQ0Lbot1y0sgFnTUtheVMWjC7S/W7kHEZkoItkikiMi97fxeqKIfCsiG0RkqYjE2beniMgKEdlsf21a56fvGKOTWz5fV+WVWpxEqZNX09DEre+sIdjfl5dmjMDHW8vAzqR/2m7g8KINeqXbOZzRL4bfndmHDzIL+GhtodVxlDolIuINvAxcCAwCZojIoCN2+yvwljFmGPA48LR9ew1wrTFmMDAR+JuIhHdO8o4VH9mFbqEBrNaiW7mwxxZsIbekmr9PT6FraIDVcTyOFt1uYGXuQZKiAuke1sXqKMruznP7Mjo5koc+3qT93crVpQM5xphcY0wD8D4w6Yh9BgHf2p9/d/h1Y8x2Y8wO+/O9QBHgkkvmigjpyZGsytX5upVr+ixrLx9ktvRxH15pVXUuLbpdXLPNsCqvVK9yO5mf+7v9vLnjvXXUN2l/t3JZPYGCVl8X2re1lgVcYX9+ORAiIv/1oSQi6YAfsNNBOR1udK9Iiqrq2XWwxuooSp2QgtIaHvxoIyMSwrnzXO3jtooW3S5u894KquqaGKPzczud2NAAnrtiGFv2VfLcl9lWx1HqZLW12taRl3rvBSaIyDpgArAHaPr5BCLdgf8AvzLG2Nr8JiI3iUimiGQWFxd3TPIONjo5EkBbTJRLaWy2ccf76wD4+/QR+Goft2X0T97F/Tw/txbdTuncQbFcNzaR15blsTS7yOo4Sp2MQiC+1ddxwN7WOxhj9hpjJhtjRgAP2bdVAIhIKPAF8CdjzMqjfRNjzGxjTJoxJi0mxjk7UHrHBBMZ5KeDKZVLefGbHazbXc5fJg/VaYUtpkW3i1uRe5DeMUE6IMKJPXDRQPrHhnDvvCyKq+qtjqPUicoA+opIsoj4AdOBBa13EJFoETn88+QBYI59ux/wMS2DLOd1YmaHEBHSkyJZna+L5CjX8FNOCS8vzWFaWjyXDO9hdRyP59Ciux3TTPmLyAf211eJSJJ9+3kiskZENtr/e3arY1Lt23NE5O8i0tatT4/Q2GwjQ/u5nV6Arzd/nzGCqrom7p2XpYtrKJdijGkCbgMWA1uBucaYzSLyuIhcat/tTCBbRLYDscBT9u1XAmcA14vIevsjpXPfQcdKT46koLSWveW1VkdR6phKqxu484P1JEcH8edLj5xwSFnBYUV3O6eZuhEoM8b0AWYBz9q3lwCXGGOGAtfR0gt42D+Bm4C+9sdER70HZ7dxTwXVDc2c1ltHITu7/t1C+NMvBvL99mJe/ynf6jhKnRBjzEJjTD9jTG9jzFP2bY8YYxbYn883xvS17/NrY0y9ffvbxhhfY0xKq8d6K9/LqUrXvm7lAowx3Dcvi/KaRl6aMYJAPx+rIykce6W7PdNMTQLetD+fD5wjImKMWWefXgpgMxBgvyreHQg1xqwwLXM2vQVc5sD34NRW5bZ86B/+IaCc2y/HJHLuwFieXbSNzXsrrI6jlDoJA7uHEhLgo33dyqm9vXIX324r4oGLBjC4R5jVcZSdI4vu9kwz9fM+9luYFcCRvRJXAOvsV0562s9zrHMCrjES/lRl5pfSKyaI6GB/q6OodhARnpsyjPBAX+54bx01DU3HP0gp5VS8vYRRSZGsztO+buWccooO8eQXW5nQL4brT0uyOo5qxZFFd3ummTrmPiIymJaWk5tP4JwtG11gJPypsNkMmbvKSE/Sq9yuJDLIj1nTUsgtqebJL7ZaHUcpdRLSkyPZWVxNySEdGK2cS0OTjTs/WEegnzczpwzDg4e9OSVHFt3HnWaq9T4i4gOEAaX2r+NoGfV+rTFmZ6v9445zTo+wo+gQFbWNpGnR7XLG9YnmN+N78e6q3SzZdsDqOEqpE6R93cpZvfjtdjbtqeTpycN0VjMn5Mii+7jTTNm/vs7+fAqwxBhjRCSclnldHzDGLD+8szFmH1AlImPss5ZcC3zqwPfgtFbn2/u5teh2Sfec348B3UL4w/yNHNSrZUq5lKE9wwjw9SIzv8zqKEr9LCO/lH8u3cmVaXFMHNLN6jiqDQ4ruts5zdRrQJSI5AB3A4enFbwN6AM83Gqaqa72124B/g3k0LKc8CJHvQdnlplfStcQf+Iju1gdRZ0Efx9vZk1LobK2kQc+2kjLuGCllCvw9fZiWFw4a3bplW7lHKrqGrnrg/XERQTyyCWDrY6jjsKhc8gYYxYCC4/Y9kir53XA1DaOexJ48ijnzASGdGxS15ORV8qo5Ejt13JhA7uHcu8F/fjLwm3MW1PIlWnxxz9IKeUU0hIjmP1DLrUNzXTx87Y6jvJwj322hb3ltcz77WkE++v0gM5KV6R0QXvKa9lbUceoxAiro6hTdOPpvRidHMljCzZTUFpjdRylVDulJkbQZDNkFZZbHUV5uEUb9zF/TSG3ndWHVK0LnJoW3S4owz54Z5TOz+3yvL2E568cjpcId89dT7OuVqmUSxiZ0FLcrNmlfd3KOgcq63jg440Mjwvj9nP6Wh1HHYcW3S4oI7+UEH8fBnQLtTqK6gBxEYE8NmkwGfllzP4h1+o4Sql2iAjyo3dMkBbdyjI2m+HeeVnUNTbzwrQUfL21pHN2+n/IBWXklzIyMQJvL+3ndheXj+jJRUO78cLX2Wzao6tVKuUK0hIjWbOrDJveoVIWeGtFPj/uKOFPvxhE75hgq+OodtCi2/LRC9gAACAASURBVMWU1zSw/cAhRiVp35Y7ERGeumwoEYF+3PXBeuoam62OpJQ6jtTECCpqG8ktOWR1FOVhdhYf4ulF2zirfwxXj06wOo5qJy26XczheWFH6fzcbiciyI/npgxjR9EhZi7OtjqOUuo4Uu0XP3S+btWZmppt3DM3iwBfb569QleddCVadLuYjF2l+HoLw+PDrY6iHODM/l25Zkwiry3L46edJVbHUUodQ6/oICICfbWvW3Wq2T/msr6gnCcuG6KrTroYLbpdTEZeKcPiwgnw1Xlh3dWDFw0kKSqQ++ZtoKqu0eo4SqmjEBFSEyO06FadJnt/FX/7egcXDe3GJcO6Wx1HnSAtul1IXWMzG/dUkKb93G6ti583z185nH0Vtfxl4Var4yiljmFkYgS5JdWUVjdYHUW5ucZmG3fPXU9IgA9PTBqibSUuSItuF7K+oJzGZkO69nO7vdTESH4zvhfvrS5gaXaR1XGUUkeRltjyeaxXu5WjvfxdDpv3VvLU5UOJCva3Oo46CVp0u5DDi+LoilOe4a7z+tG3azB//HADFTXaZqKUMxoWF4avt2jRrRxq054K/rEkh8tSejBxSDer46iTpEW3C8nYVUb/2BDCA/2sjqI6QYBvS5tJyaEGHvtss9VxlFJtCPD1ZnCPMNbsKrU6inJT9U3N3D13PVHBfjx26RCr46hToEW3i2i2GdbuKtN+bg8zLC6c353Zm4/W7WHx5v1Wx1FKtSE1MYKswgoammxWR1Fu6G/f7GD7gUM8c8UwwgJ9rY6jToEW3S5i675KDtU3kZ6s/dye5raz+zKoeygPfbxRB2sp5YTSEiNoaLKxaa+uJqs61trdZbz6/U6mpcVzVv+uVsdRp0iLbheRkd9y6zJNB1F6HD8fL16YNpyK2kYe/mST1XGUUkc4PM5mrfZ1qw5U29DMvXOz6B7WhT9dPNDqOKoDaNHtIjLzy+gZ3oWe4V2sjqIsMKBbKHee248vNu7js6y9VsdRSrXSNTSA+MguujKl6lAzF2eTW1LNc1OGERKgbSXuQItuF2CMYXV+qfZze7ibz+jF8PhwHv50E0VVdVbHUUq1kpoQwZrdZRhjrI6i3MCq3IO8/lMe145NZFyfaKvjqA6iRbcL2F1aQ3FVPaO0tcSj+Xh78fzU4dQ2NPPgRxv1h7tSTiQ1KZLiqnoKSmutjqJcXHV9E/fOzyIhMpD7LxxgdRzVgbTodgGr7fNz6yBK1adrMPdd0J9vthbx4do9VsdRLkhEvEXkUhG5Q0TuPvywOperS01ouRO5ZrdOHahOzV8WbqWwrJa/Th1OoJ+P1XFUB9Ki2wVk5pcR1sWXPjHBVkdRTuCGccmkJ0Xy2ILN7C3Xq2rqhH0GXA9EASGtHuoU9O8WQrC/j/Z1q1Py445i3lm1m1+fnqx3t92Q/grlAjLySxmVFIGXl1gdRTkBLy9h5tRhTPzbj/zxww28dUM6Ivp3Q7VbnDFmmNUh3I23lzAiIZy1u8utjqJcVGVdI3+Yv4HeMUHcc35/q+MoB9Ar3U6uuKqe3JJqnSpQ/ZfEqCAe/MVAftxRwrurd1sdR7mWRSJyvtUh3NHIhAiy91dSVddodRTlgp74bAsHKut4/soUAny9rY6jHECLbid3eGlhvc2kjvTL0Qmc3ieap77Yyu6DNVbHUa5jJfCxiNSKSKWIVIlIpdWh3EFqYgQ2A+sL9Gq3OjHfbj3AvDWF3HJmb1Liw62OoxxEi24ntzqvDH8fL4b2DLM6inIyIsKzU4bhLcK987Ow2XQ2E9UuzwNjgUBjTKgxJsQYE2p1KHeQkhCOCKzRRXLUCSirbuD+jzYyoFsId5zT1+o4yoEcWnSLyEQRyRaRHBG5v43X/UXkA/vrq0Qkyb49SkS+E5FDIvKPI45Zaj/nevvDrddFzdxVSkp8OH4++vuR+l89w7vw8CWDWJ1Xyhs/5VsdR7mGHcAmcwJzTrbjszxRRL4VkQ32z+i4Vq9dJyI77I/rOug9OKXQAF/6x4Zo0a1OyJ8XbKasuoHnrxyOv4+2lbgzh1VyIuINvAxcCAwCZojIoCN2uxEoM8b0AWYBz9q31wEPA/ce5fRXG2NS7I+ijk/vHKrrm9i8t1JbS9QxTU2N45wBXXlu8TZyiw9ZHUc5v33AUhF5oD1TBrbzs/yvwFv2AZqPA0/bj40E/gyMBtKBP4uIW6/ylZoYwfrd5TTrnSfVDos27mNB1l7uOKcvg3voHW13d9yiW0S8RGTTSZw7HcgxxuQaYxqA94FJR+wzCXjT/nw+cI6IiDGm2hizjJbi22Ots39wj9L5udUxiAhPTx6Kv48398zL0h/26njygG8BP9o3ZWB7PssH2c8J8F2r1y8AvjbGlBpjyoCvgYkd8i6cVGpiBFX1TewoqrI6inJyJYfqeeiTTQztGcYtZ/a2Oo7qBMctuo0xNiBLRBJO8Nw9gYJWXxfat7W5jzGmCaigZe7Y43nd3lrysBxlrjQRuUlEMkUks7i4+ASjO4fV+aV4CYxM0EEV6ti6hgbw+KTBrNtdzuwfcq2Oo5yYMeYxY8xjwAvA862+Ppr2fJZnAVfYn18OhIhIVDuPBdzjMxtaim7Qvm51bMYYHvp4I4fqmnj+yuH4emsLqSdo7//l7sBme8/egsOP4xzTVjF85CW49uxzpKuNMUOB8fbHNW3tZIyZbYxJM8akxcTEHOeUzikzv5SB3UMJCfC1OopyAZcO78GFQ7ox6+vtZO/Xq2yqbSIyRETWAZto+VxfIyKDj3VIG9uO/Jy+F5hgP+8EYA/Q1M5jWza6wWc2QEJkINHBflp0q2P6dP1eFm8+wD3n96NfrK5N5SnaW3Q/BlxMS6/e860ex1IIxLf6Og7Ye7R9RMQHCAOOuYauMWaP/b9VwLu03Pp0O43NNtbtLtd+btVuIsKTlw0hJMCHe+atp7HZZnUk5ZxmA3cbYxKNMYnAPcC/jrH/cT/LjTF7jTGTjTEjgIfs2yrac6y7ERFGJkSwVotudRQHKut45NNNpCZG8OvxvayOozpRu4puY8z3bT2Oc1gG0FdEkkXED5gOHHl1fAFweDT7FGDJsUbUi4iPiETbn/vS8ovAyfSbO71NeyqobWwmLcmtxxypDhYV7M9Tlw9h055KXvlup9VxlHMKMsZ8d/gLY8xSIOgY+x/3s1xEokXk8M+TB4A59ueLgfNFJMI+gPJ8+za3lpoYQf7BGoqr6q2OopyMMYb7P9xAQ7ONv04djreuNO1Rjll0H140oY3HcRdTsPdo30bLB+xWYK4xZrOIPC4il9p3ew2IEpEc4G7g56moRCSflp7D60Wk0D5a3h9YLCIbgPW03MI81hUal5WZ33KVJF2vdKsTNHFIdyal9OClJTvYtKfC6jjK+eTax8Mk2R9/omVwZZva+Vl+JpAtItuBWOAp+7GlwBO0FO4ZwOP2bW7tcF/32t16tVv9t7mZBXyXXcwfJw4gOfpYv+sqd+RzrBeNMafUaGSMWQgsPGLbI62e1wFTj3Js0lFOm3oqmVzF6vxSEqMC6RoaYHUU5YIeu3QwK3Ye5J65WSy4fZzO/apau4GWlsEPaem5/gG4/lgHtOOzfD4tM1C1dewc/u/Kt0cY0jMMP28v1u4q44LB3ayOo5xEYVkNT3y+lTG9IrlubJLVcZQFdLisE7LZDJn5pdrPrU5aeKAfz1wxlOwDVbz4zQ6r4yjn0puWPmsvwBc4h5bCW3WQAF9vhvQM1cGU6mc2m+EP8zdgjGHmlOF4aVuJR9Ki2wntLD5EWU2jtpaoU3L2gFiuTIvj/32/k3V6m1v9n3doufI8mZZxMRcDl1iayA2lJkawYU8F9U3NVkdRTuDtVbv4aedB/nTxIOIjA62OoyyiRbcTWp3f0vKoi+KoU/WniwfRLTSAe+ZlUdeoP/wVAMXGmM+MMXnGmF2HH1aHcjepiRE0NNnYvPeYw5+UB8gvqebphduY0C+G6aPij3+AcltadDuhjLxSooP9SYrS34bVqQkN8OW5KcPJLa5m5uJsq+Mo5/BnEfm3iMwQkcmHH1aHcjcjE+yDKbXFxKM12wz3zc/Cx1t45oqhHGU9P+UhtOh2Qhn5ZaQnR+g/TtUhTu8bzS/HJDBneR6r89x+4gh1fL8CUmhZjv0S++NiSxO5oa6hAcRHdtG+bg83Z1keGfllPHbpYLqHdbE6jrKYFt1OZk95LXvKa3UQpepQD1w4kPiIQO6dl0V1fZPVcZS1httXfrzOGPMr++MGq0O5o9SECDJ3lXGM5SeUG8spqmLmV9mcNyiWy0f0tDqOcgJadDuZDPuVSC26VUcK8vfhr1OHU1BWwzOLtlkdR1lrpX3dA+VgqYkRFFfVU1hWa3UU1cmamm3cMzeLID9v/nK5tpWoFlp0O5nV+aWE+PswsHuo1VGUm0lPjuSGccn8Z+Uulu0osTqOss7pwHoRyRaRDSKy0b7gmOpgI+2L5GiLief5f9/vJKuwgicvG0pMiL/VcZST0KLbyWTklTIyMUKXhlUOcd8F/ekVE8Qf5mdRWddodRxljYlAX1qWZD/cz61TBjrAgG6hBPl5a9HtYbbsreTFb3dw8bDu/GJYd6vjKCeiRbcTKatuYEfRIdJ1qkDlIAG+3jw/dTj7K+t48vMtVsdRFmg9TaBOGehY3l7CiIQILbo9SEOTjXvmZRHWxY8nJg2xOo5yMlp0O5GMfO3nVo43IiGC307ozdzMQpZsO2B1HKXc2sjECLbtr+SQDmD2CC8t2cHWfZU8M3koEUF+VsdRTkaLbieSkV+Kn7cXw+LCrI6i3Nzvz+3LgG4h3P/hRsprGqyOo5TbSk2MwGYgq6Dc6ijKwbIKynll6U6uGBnHuYNirY6jnJAW3U4kI7+M4fFhBPh6Wx1FuTl/H2/+OnU4pdUNPLpgs9VxlHJbKfHhiOhgSndX19jMPfOy6BrizyOX6ORAqm1adDuJmoYmNu2pIE1bS1QnGdIzjNvO7sMn6/fy5aZ9VsdRyi2FdfGlX9cQLbrd3Atfbyen6BDPXjGMsC6+VsdRTkqLbiexfnc5TTZDuhbdqhP97qw+DOkZykMfb+LgoXqr4yjllkYmRrB2dxk2my6S445W55Xyrx9zuWp0Amf0i7E6jnJiWnQ7idX5pYj837yuSnUGX28vnp+aQlVdE3/6ZJOunKeUA6QmRlBV18SOokNWR1EdrKqukbvnrichMpCHLhpodRzl5LTodhIZ+aUM6Baqt6VUp+vfLYS7zuvHok37+XT9XqvjKOV2UnWRHLf1+Gdb2FteywtXphDk72N1HOXktOh2Ao3NNtbuKic9Sa9yK2vcdEYvUhMjePjTTewp1yWrlepISVGBRAX5adHtZr7ctI95awr53Vl9fv7FSqlj0aLbCWzeW0ltYzOjdFEcZRFvL2HWlSnYbIZ75q7X3lOlOpCI/NzXrdxDUVUdD3y0kaE9w7jjnL5Wx1EuQotuJ7Ay9yCArkSpLJUQFcifLx3MytxS/r0s1+o4SrmV1MQI8kqqdcCyGzDG8Mf5G6hpaGbWtBR8vbWUUu2jf1OcwMrcg/SOCaJrSIDVUZSHm5oaxwWDY5m5OJsteyutjqOU2zjcfrB2ty6S4+reXb2b77KLefCigfTpGmx1HOVCtOi2WGOzjYy8Usb2jrI6ilKICE9PHkZ4oB93frCOusZmqyMp5RaG9gzD11u0r9vF5RYf4snPtzK+bzTXjEm0Oo5yMVp0W2zjngqqG5oZ2yva6ihKARAZ5MfMKcPYfuAQMxdnWx1HKbcQ4OvN4B5hrNWi22U1Ndu4a24Wfj5ezJwyHC8vsTqScjFadFvscD/36F7az62cx5n9u3Lt2EReW5bHsh0lVsdRyi2kJkaQVVhOQ5PN6ijqJLz83U6yCsp56vIhdAvTdlB14hxadIvIRBHJFpEcEbm/jdf9ReQD++urRCTJvj1KRL4TkUMi8o8jjkkVkY32Y/4uIi79q+aKnQfpFxtMdLC/1VGU+i8PXDiQ3jFB3Dsvi/KaBqvjKOXyUhMjqG+ysXlvhdVR1AnKKijn70t2cFlKDy4e1sPqOMpFOazoFhFv4GXgQmAQMENEBh2x241AmTGmDzALeNa+vQ54GLi3jVP/E7gJ6Gt/TOz49J2joclGZn4ZY3tpP7dyPl38vHlx+ghKDtXzkK5WqdQpS7MPpszM1xYTV1Jd38SdH6wnNsSfxyYNsTqOcmGOvNKdDuQYY3KNMQ3A+8CkI/aZBLxpfz4fOEdExBhTbYxZRkvx/TMR6Q6EGmNWmJYK4C3gMge+B4faUFhObWOzDqJUTmtIzzDuOq8fX2zYxyfr91gdRymX1jU0gMSoQFbnl1odRZ2ARxdsJv9gNbOmpeiq0eqUOLLo7gkUtPq60L6tzX2MMU1ABXCsCrSn/TzHOicAInKTiGSKSGZxcfEJRu8cK3MPIgKjk7XoVs7rtxN6k5YYwSOfbKagtMbqOEq5tNHJkWTkl+oCVC7is6y9zFtTyG1n9WG03pVWp8iRRXdbvdZHfsq0Z5+T2t8YM9sYk2aMSYuJiTnGKa2zIvcgA7qFEhHkZ3UUpY7K20uYNS0FBG5/bx2NzToITKmTNTo5ivKaRrYXVVkdRR1HQWkND368kZEJ4fxeV51UHcCRRXchEN/q6zhg79H2EREfIAw41n23Qvt5jnVOl1Df1ExmfhljdNYS5QLiIwN5evJQ1heUM+vr7VbHUcplHV55eFWutpg4s6ZmG3d+sB4MvDh9BD666qTqAI78W5QB9BWRZBHxA6YDC47YZwFwnf35FGCJOcZoLWPMPqBKRMbYZy25Fvi046M7XlZBBfVNNh1EqVzGxcN6MH1UPP/8fifLc3QaQU/TjtmoEuyzTq0TkQ0icpF9u6+IvGmfdWqriDzQ+emdR3xkID3Du7A6T4tuZ/bSkhzW7CrjycuHEB8ZaHUc5SYcVnTbe7RvAxYDW4G5xpjNIvK4iFxq3+01IEpEcoC7gZ8/yEUkH3gBuF5EClvNfHIL8G8gB9gJLHLUe3CkFTu1n1u5nkcuGUSv6CDu+mA9Bw/VWx1HdZJ2zkb1J1o+50fQcpHlFfv2qYC/MWYokArcfHh6WE+VnhzJqryDOiOQk1qdV8pLS3YweWRPJqW0OWxMqZPi0PslxpiFxph+xpjexpin7NseMcYssD+vM8ZMNcb0McakG2NyWx2bZIyJNMYEG2PijDFb7NszjTFD7Oe87VhXxp3ZitwSBvcIJSxQR0Ir1xHo58NLM0ZSXtvIffM3aNHgOdozG5UBQu3Pw/i/1j8DBNlbCLsADUCl4yM7r9HJkZQcaiC3pNrqKOoIFTWN3Pn+OuIjA3lcpwdUHUyblCxQ19jM2t3ljNGr3MoFDeoRykMXDWTJtiJeX55vdRzVOdozG9WjwC9FpBBYCNxu3z4fqAb2AbuBvxpjPLq3Qvu6nZMxhgc/3khRVT0vTh9BsL+P1ZGUm9Gi2wKZ+WU0NNk4rY8W3co1XTs2kXMHduWZRdvYtEdX1/MA7Zk5agbwhjEmDrgI+I+IeNFylbwZ6AEkA/eISK//+QYuMM1rR0mODiImxJ/VeQetjqJaeWfVbr7YuI+7z+9HSny41XGUG9Ki2wI/5hTj6y2M0UGUykWJCM9NGU5EkC93vLeO6vomqyMpx2rPbFQ3AnMBjDErgAAgGrgK+NIY02iMKQKWA2lHfgNXmOa1o4iIva+7VFu0nMSmPRU8/vkWJvSL4bdn9LY6jnJTWnRb4MftJaQmRhDop7eulOuKDPLjb9NGkHewmod1mXh3157ZqHYD5wCIyEBaiu5i+/azpUUQMAbY1mnJndSY5Ej2VdRRWFZrdRSPV1XXyG3vriUy0I8XrhyOl1dbN3aUOnVadHeykkP1bNlXyfi+7n0lR3mGsb2juPOcfny0bg/vZxQc/wDlkto5G9U9wG9EJAt4D7jePtD9ZSAY2ERL8f66MWZDp78JJ5NuH9OzMldbTKxkjOH+jzZSUFbLS1eNICrY3+pIyo3ppdZOdnh+4/F9oy1OolTHuP3sPmTuKuXPCzYztGcYQ3qGWR1JOYAxZiEtAyRbb3uk1fMtwLg2jjtEy7SBqpW+XYOJCPRldV4pU9Pij3+Acoi3V+7iiw37+OPEAYxK0sXqlGPple5O9uOOEsIDfRncQwsT5R68vIS/TUshMtCP3727lsq6RqsjKeX0vLyEUUktfd3KGpv2VPDE51s5q38MN5/xP2N7lepwWnR3ImMMP+4oZlyfaLy1Z0y5kahgf/5x1QgKy2q5b16W9ncr1Q6je0Wxu7SGfRXa193ZKusaufWdtUQF+/H8lSnax606hRbdnSin6BAHKusZ30dbS5T7SUuK5P6JA1i8+QCvLcuzOo5STm9Mr5Z2hhU7ta+7MxljuP/DDewpr+WlGSOIDPKzOpLyEFp0d6IfdrT0c5+u/dzKTf16fDLnD4rlmUXbWLOrzOo4Sjm1gd1CiQzyY5l9rI/qHK8vz2fhxv384YL+pGkft+pEWnR3ou+3F9MrJoi4iECroyjlECLCzKnD6RHehdveXUtpdYPVkZRyWl5ewtheUfyUc1BbsjrJqtyD/GXhVs4dGMtvxmsft+pcWnR3kpqGJlbmHuSs/l2tjqKUQ4V18eWVq0dy8FADv39/Hc02LSaUOppxfaLZX1lHbkm11VHc3v6KOn737joSIgN5YZrOx606nxbdnWR5zkEammycPUCLbuX+hvQM4/FJg/lxRwnPLfb4dVCUOqpxfVrm616uLSYOVd/UzC3vrKGmoYlXr0klNMDX6kjKA2nR3UmWbCsi2N9H5wFVHmN6egJXjU7g1e9z+SzryBXDlVIACZGB9AzvokW3gz3x+RbW7S5n5pTh9I0NsTqO8lBadHcCYwzfbStifN9o/Hz0j1x5jkcvGUxaYgR/mL+BLXsrrY6jlNMREcb1iWLFzoPaiuUgczMLeHvlbm4+oxe/GNbd6jjKg2kF2Am27Ktkf2UdZ2lrifIwfj5evPLLkYR28eHmtzMp04GVSv2PcX2iqaxrYtOeCqujuJ0NheX86ZNNnNY7ivsu6G91HOXhtOjuBN9tKwLgzP4xFidRqvN1DQng//0ylQMV9dzx/jqamm1WR1LKqZzWu2Ua2eU7tcWkIx2orOM3b2USE+zPSzNG4OOtJY+ylv4N7ARLthUxLC6MriEBVkdRyhIjEiJ44rLDAyuzrY6jlFOJCfGnf2wIy3Zo0d1R6hqbuemtTKrqmvjXtWlEBftbHUkpLbodrbS6gXUF5TpVoPJ400YlcM2YRGb/kMvcjAKr4yjlVM7oF01mfhnV9U1WR3F5xhj+MH8DWYUVzJqWwqAeoVZHUgrQotvhvtlyAGPg3IGxVkdRynKPXDKI8X2jefDjjbr0tVKtnNm/Kw3NNn7Sfxen7JWlO1mQtZf7LujPBYO7WR1HqZ9p0e1gizbtIy6iC0N66m/aSvl6e/GPq0aSGBXILe+sIU8XBFEKgLSkCAL9vPl+e5HVUVzal5v2M3NxNpel9ODWM3tbHUep/6JFtwNV1jWyLKeEiYO7IaIrXykFLStWzrl+FALc+EYG5TU6o4lS/j7enNY7mqXZxbok/EnavLeCuz5YT0p8OM9cMUx/7iqno0W3Ay3ZWkRjs+HCoXp7S6nWEqOCmH1tGoVltdzy9loamnRGE6XO7B9DYVktO4v1DtCJ2lNeyw1vZBAe6Mvsa1IJ8PW2OpJS/0OLbgdatGkfsaH+jIiPsDqKUk5nVFIkT08eyorcg/zpk416dU95vMPTyi7N1haTE1FR08j1c1ZT09DMG79Kp2uozhSmnJNDi24RmSgi2SKSIyL3t/G6v4h8YH99lYgktXrtAfv2bBG5oNX2fBHZKCLrRSTTkflPRU1DE99vL+aCwd3w8tJbXEq15YrUOG47qw9zMwv52zc7rI6jlKXiIgLp0zWY77cXWx3FZdQ3NXPTfzLJP1jNq9ek0r+bLvGunJePo04sIt7Ay8B5QCGQISILjDFbWu12I1BmjOkjItOBZ4FpIjIImA4MBnoA34hIP2NMs/24s4wxTj2h6dLsYuoabUwcoq0lSh3LPef3Y39lHS9+u4OYEH9+OSbR6khKWWZCvxj+s2IXNQ1NBPo57Ee0W7DZDPfO28CqvFJenJ7y8yJDSjkrR17pTgdyjDG5xpgG4H1g0hH7TALetD+fD5wjLSMfJgHvG2PqjTF5QI79fC5j4cZ9RAb5kZ4UaXUUpZyaiPD05KGc1T+GRz7dxJeb9lsdSSnLnGWfOnB5jk4deDzPfrmNz7L28seJA5iU0tPqOEodlyOL7p5A6xUwCu3b2tzHGNMEVABRxznWAF+JyBoRuelo31xEbhKRTBHJLC7u3Ft1VXWNfLP1ABcN7abLzirVDr7eXrx89UiGxYVzx/vryMgvtTqSUpZIT44kJMCHrzbrL5/H8vryPF79IZdfjkngtxN6WR1HqXZxZEXYViPzkSOljrbPsY4dZ4wZCVwI/E5EzmjrmxtjZhtj0owxaTExMe3N3CEWbdpPXaONySPjOvX7KuXKAv18mHP9KOIiunDjGxlk76+yOpJSnc7Px4uzB3Tlm60HaGrWWX3aMi+zgMc+28L5g2J59JLBOjWgchmOLLoLgfhWX8cBe4+2j4j4AGFA6bGONcYc/m8R8DFO2Hby0dpCkqODGBEfbnUUpVxKZJAfb/4qnQBfb66ds4rdB2usjqRUp7tgcDfKahrJ3FVmdRSn88WGffzxww2M7xvNS1eN0LvJyqU48m9rBtBXRJJFxI+WgZELjthnAXCd/fkUYIlpmTdsATDdPrtJMtAXWC0iQSISAiAiQcD5wCYHvocTVlhWw8rcUiaP6Km/fSt1EuIjA3nrxnTqm2zM+NdKCsu08FaeZUK/GPx8vFisLSb/5bttRdz5wTpGJETw6jWp+PvoXNzKtTis6Lb3aN8GtR0GdwAAEytJREFULAa2AnONMZtF5HERudS+22tAlIjkAHcD99uP3QzMBbYAXwK/s89cEgssE5EsYDXwhTHmS0e9h5Pxybo9AFw2Qgd1KHWyBnQL5e0bR1NZ18jV/17F/oo6qyMp1WmC/H0Y3yearzYf0Pnr7VbmHuS3b6+hX2wIc64fpTO7KJfk0PsyxpiFxph+xpjexpin7NseMcYssD+vM8ZMNcb0McakG2NyWx37lP24/saYRfZtucaY4fbH4MPndBbGGD5au4f05EjiIwOtjqOUSxvSM4y3bkinpKqeq/69kuKqeqsjKdVpLvj/7d15eBX1vcfx95esBAgQwhKSsAgBwr7EyFptsQh4FdEioBa0WrRUra2tt9Y+bZ8+2tv23tba4gIFRdRKuW1RKhRXKlZZlR0Ewh6IBBJ2QkKS3/3jjNc0JIiQk0nOfF7Pc54zZ87MOd8vM+fHNzO/+U2PNuw/WsSmA8f9DsV3a/cd5c7Zq0Jnwb6RTdOGMX6HJHJR1BmqBn209yg7D5/ipv46yi1SE/q1a85zd2STd/QMt81cQeGpEr9DEqkVwzNb0cAI/Cgma/cd5euzVpDUOJYX77yCFo3j/A5J5KKp6K5Bc5btpklcNNf2but3KCIRI7tjErMmZ7G74JQKbwmMFo3juLxDEos2fhLYLiYf7jnC12euoHlCLHOnDKJNU93eXeo3Fd015ODxMyxcn8e4rHQax6mvmUhNGtw5mRmTsthx6CQTZiwj/7j6eEvku65PW3LyT7I5L3hdTFbtLmTSrBW0aBzLn+8eSGqzhn6HJHLJVHTXkJdW7KXMOSYN0i2sRcLhyi4tee72y8k9UsT4Gcs5cLTI75BEwuraXinERNn/X6AfFMt2FDD52ZW0Toznz3cPIqWpCm6JDCq6a0BxaRl/WrGHr3RtRYfkRn6HIxKxBndO5oU7QxdXjntmGXsKTvkdUiCY2Ugz22pmOWb2wyreb2dmS8xsjZmtN7PRFd7rbWbLzGyTmW0wM/URuEDNG8VyZZdWvLr2AGXlwehi8vqmT5j83EpSmzVk7pSBtE7U7iKRQ0V3DVi4Po/DJ0u4fUgHv0MRiXgD2ifx8pSBnC4p5WvPLGPTgWN+hxTRzCwKeJLQXYC7AxPNrHulxX5MaFjYfoTuyfCUt2408CJwj3OuB3AVcLaWQo8IY/ulkn+imOU7C/wOJezmrdrHt178kO4picy7exCtVHBLhFHRfYmcc8x8bxedWjZiaOdkv8MRCYSeqU2Zd/cgYhoY46cv51/bD/sdUiTLBnK8IVtLgLnAmErLOCDRm27KZ3cfHgGsd86tA3DOFXj3XJALNDyzFU3iopkf4V1Mnnl3Bw/9dT1DOifz0l1X0LxRrN8hidQ4Fd2X6K0t+WzOO849V3bSHShFalFG6yb8beoQ0po35PbnVjJ/Ta7fIUWqVGBfhde53ryKfgbcZma5wCLgPm9+F8CZ2etm9pGZPVTdl5jZFDNbbWarDx06VHPR13PxMVGM6tWGxRs/oagk8v5eKS0r56evbuSX//iY6/q0Zdbky2mkwQgkQqnovgTOOZ54exvtWyQwVnegFKl1bZrGM++eQVzeIYnv/nkd097ZHtjh1cKoqqMJlf+RJwKznXNpwGjgBTNrAEQDQ4FbveexZja8qi9xzs1wzmU557JatmxZc9FHgBv6pXKyuJQ3NkfWmN0nzpzlrjmreX7ZHr45rCNPjO9LbLTKEolc2rsvwWvr89i4/zj3frkz0VH6pxTxQ2J8DLO/cTlj+rblf97Yxv1z10bkEUEf5QLpFV6n8Vn3kU/dCcwDcM4tA+KBZG/dd51zh51zpwkdBe8f9ogjzMCOLWjfIoEXlu3xO5Qas/9oEeOeWcZ72w/zi7G9eOTa7jRooLPFEtlUKV6kM2fL+NXij+nWpgk39k/zOxyRQIuLjuJ34/vy0MiuvLb+ADdPX0beMQ0pWENWARlm1tHMYgldKLmg0jJ7geEAZpZJqOg+BLwO9DazBO+iyiuBzbUWeYRo0MD4+sD2rN5zhI376/+Fwyt3FTJm2vvsP1rE83dkc8sV7fwOSaRWqOi+SDOW7iT3SBE/vrY7UfrrXMR3ZsbUqzozc1IWuw6f4ro/vM+q3YV+h1XvOedKgXsJFdBbCI1SssnMfm5m13uLPQh808zWAS8Dt7uQI8BvCRXua4GPnHMLaz+L+m/cgHQaxkTV66PdoYEHdjLxj8tJjI9m/tTBDM3QAAQSHCq6L0JO/gmmvZPDdX3aqsEQqWOGZ7Zm/tTBNI6LYsKM5Tz1zxzKAzLGcbg45xY557o45zo55x7z5v3EObfAm97snBvinOvjnOvrnHujwrovOud6OOd6OueqvZBSzq9pQgw39EvllbX7OXKqxO9wvrCTxaXc+6c1PLpwC1dntuLVe4fQuVUTv8MSqVUqur+g4tIyvjdvHQlxUfzkPyoPVSsidUFG6yb8/b6hjOrZhl8v3sods1dRcLLY77BELsmkQe0pLi1n3up9n79wHbIh9xjXT/sX/9iYx8OjuvHMbQNoEh/jd1gitU5F9xf02MItrM89xq9v6k3LJnF+hyMi1WgSH8MfJvbj0Rt6smxnAaN//x5Lt2koOqm/MlMSye6YxJxlezhbVu53OJ+rrNzx5JIcxj71PqeLy3jxriu4W8PrSoCp6P4Cpr+7gzne0EYjerTxOxwR+Rxmxm0D23vdTaKZ9OxKfjR/AyeLS/0OTeSiTBl2GfuPFvFKHb9Zzr7C00ycsZz/fn0r1/Row+IHhjG4k7pjSrCp6L4Azjkef3Mb/+UN3v/wqEy/QxKRL6BH26YsvH8YU750GS+v3Ms1jy/lgxzdxVLqn+GZreieksiTS3IorYNHu0vLypn53k5GPL6UzXnH+e3NfZh2Sz+aJegOkyIquj/Hxv3HuHXmCp54eztfG5DGb8b10ViiIvVQfEwUPxqdyV/uGURsdANumbmCB+auIf/4Gb9DE7lgZsZ3rs5gd8Fp/vJh3boL6/rco4x58n0eXbiFQZ1asPiBYdzYP03dSUQ8utdqFbZ+coJHF25mT8Fp9haeJjE+msfG9uSW7HZqPETquQHtk1h0/zCe+mcO09/dyZubD/LA1V24fUgHYnSTK6kHRnRvzYD2zfnNm9u4rk9b32+bnn/iDL97aztzV+4luXEcT9/an5E92+j/S5FKVHRXISbKOF50lp6pidw1rCPX92mrU2MiEaRhbBQPjujKTf3T+Plrm3ls0RZeXrmX7361C9f2StHZLKnTzIxHrs3kxqc+YNqSHP5zZDdf4jhdUsofl+5i+tIdlJSWM2lQB743oguJGplEpErmXOSPX5uVleVWr17tdxgiUke9veUgv1r8MdsOnqRbmyY8OKIrV2e2qhNH6szsQ+dclt9x1Ca12Rfm+/+7jlfW7Ofv9w0lMyWx1r73VHEpc1ftY/q7O8g/Ucyonm14aGQ3OiY3qrUYROqq87XZOtItIoE3PLM1V3VtxWvrD/C7t7bzzTmryUxJ5I4hHbi+T1viY6L8DlHkHI+MzmTJx/l8b9465k8dHPb9tPBUCbM/2M3zH+zmWNFZBl6WxNO39WdA+6Swfq9IpNCRbhGRCkrLyvnbmv3Mem8XWw+eoEWjWG69oh3js9uR2qxhrcejI91yPu98fJBvzF7NxOx2/GJszxo/O+OcY8WuQuat2seijXmcOVvOV7u35ltXdaJ/u+Y1+l0ikUBHukVELlB0VANuzkpn3IA0PthRwHPv7+IPS3L4/Ts5ZHdI4vq+bRndK4WkRrrOQ/z3lW6hAvjpf+4gPakhU6/qfMmf6Zxj28GTvLHpE/76US67C07TJC6aG/unccfgDmS01u3bRS5GWItuMxsJPAFEATOdc7+s9H4cMAcYABQA451zu733HgbuBMqA+51zr1/IZ4qI1AQzY0jnZIZ0TmZf4WkWrDvAK2v28+NXNvLTBZvol96MoRnJDMtIpndaM418Ir75wYiuHDhaxK8Xb6WopIzvXt3lC18MfPR0CWv2HmXp9kO8teUg+wqLAMjumMR9X8lgdK8UGsaqm5XIpQhb9xIziwK2AV8FcoFVwETn3OYKy0wFejvn7jGzCcBY59x4M+sOvAxkA22Bt4Au3mrn/cyq6FSliNQE5xxb8k6wcMMB3tt+mA37j+EcJMRG0a1NEzJTEslMSSSjVWNaJ8bTKjGOhNhLO7ah7iVyIc6WlfPI/A3MW51Ldockvn9NV7LaN/+34ts5x6mSMvYWnGZv4Sn2Fp5m6ycnWbP3CDsPnwIgLroBQzsnMzyzNcMzW9E6Md6vlETqJb+6l2QDOc65nV4Qc4ExQMUCeQzwM2/6L8A0C3VIGwPMdc4VA7vMLMf7PC7gM0VEwsLM6N42ke5tE/nBNaGjgx/sKGDlrkI25x1nwdoDvLRi77+t0yQumqYJMTSOi+Yf3xlWJ0ZEkcgTE9WAX93Um6z2Sfxy8cfcPH0ZifHRJDWKpaS0nJPFpZwuKaO0/N8PtCU3jqVvenNuGpBGv3bN6Jve7JL/UBSRqoXzl5UK7KvwOhe4orplnHOlZnYMaOHNX15p3VRv+vM+EwAzmwJMAWjXrt3FZSAich7NEmIZ3SuF0b1SgNCRxNwjRew6fIr8E8XknzhD/vFijhWdpbTcqeCWsDIzbr48ndG9U3h7y0FW7irk+JlS4qIb0DgumkZxUSTGx5DWPIH2LRJIb55A0wSNqS1SW8JZdFf1v0vlvizVLVPd/Ko6TVbZP8Y5NwOYAaFTldWHKSJSM8yM9KQE0pMS/A5FAqxxXDRj+qYypm/q5y8sIrUmnFf+5ALpFV6nAQeqW8bMooGmQOF51r2QzxQRERERqVPCWXSvAjLMrKOZxQITgAWVllkATPamvwa840JXdi4AJphZnJl1BDKAlRf4mSIiIiIidUrYupd4fbTvBV4nNLzfs865TWb2c2C1c24BMAt4wbtQspBQEY233DxCF0iWAt92zpUBVPWZ4cpBRERERKQmhPUSZefcImBRpXk/qTB9BhhXzbqPAY9dyGeKiIiIiNRlupuDiIiIiEiYqegWEREREQkzFd0iIiIiImGmoltEREREJMwsNEJfZDOzQ8Cei1g1GThcw+HUF8o9uIKcf13Mvb1zrqXfQdQmtdkXJci5Q7DzD3LuUPfyr7bNDkTRfbHMbLVzLsvvOPyg3IOZOwQ7/yDnHgmCvP2CnDsEO/8g5w71K391LxERERERCTMV3SIiIiIiYaai+/xm+B2Aj5R7cAU5/yDnHgmCvP2CnDsEO/8g5w71KH/16RYRERERCTMd6RYRERERCTMV3SIiIiIiYaaiuwpmNtLMtppZjpn90O94ws3MdpvZBjNba2arvXlJZvammW33npv7HWdNMbNnzSzfzDZWmFdlvhbye29fWG9m/f2LvGZUk//PzGy/tw+sNbPRFd572Mt/q5ld40/UNcPM0s1siZltMbNNZvYdb35gtn8kClqbDcFqt9Vmq82OlDZbRXclZhYFPAmMAroDE82su79R1YovO+f6Vhjr8ofA2865DOBt73WkmA2MrDSvunxHARneYwrwdC3FGE6zOTd/gMe9faCvc24RgLfvTwB6eOs85f1G6qtS4EHnXCYwEPi2l2OQtn9ECXCbDcFpt2ejNlttdgS02Sq6z5UN5DjndjrnSoC5wBifY/LDGOB5b/p54AYfY6lRzrmlQGGl2dXlOwaY40KWA83MLKV2Ig2PavKvzhhgrnOu2Dm3C8gh9Bupl5xzec65j7zpE8AWIJUAbf8IpDb7MxHZbqvNVpvtTdf7NltF97lSgX0VXud68yKZA94wsw/NbIo3r7VzLg9COz3Qyrfoakd1+QZpf7jXOx33bIXT0hGbv5l1APoBK9D2r8+Cuo2C3m7rN6s2u95tfxXd57Iq5kX6uIpDnHP9CZ2W+baZfcnvgOqQoOwPTwOdgL5AHvAbb35E5m9mjYG/Ag84546fb9Eq5tX7/CNMULeR2u2qBWV/UJtdzaJVzKsz+avoPlcukF7hdRpwwKdYaoVz7oD3nA/MJ3Qq6uCnp2S853z/IqwV1eUbiP3BOXfQOVfmnCsH/shnpyMjLn8ziyHUeL/knPubNzvQ27+eC+Q2Ursd7N+s2uz6uf1VdJ9rFZBhZh3NLJbQBQkLfI4pbMyskZk1+XQaGAFsJJTzZG+xycCr/kRYa6rLdwEwybsieiBw7NNTWpGkUp+3sYT2AQjlP8HM4sysI6GLU1bWdnw1xcwMmAVscc79tsJbgd7+9Vyg2mxQu+0J9G9WbXY93f7OOT0qPYDRwDZgB/CI3/GEOdfLgHXeY9On+QItCF0RvN17TvI71hrM+WVCp+POEvqr+M7q8iV0qupJb1/YAGT5HX+Y8n/By289oUYrpcLyj3j5bwVG+R3/JeY+lNCpxvXAWu8xOkjbPxIfQWqzvXwD1W6rzVabHSlttm4DLyIiIiISZupeIiIiIiISZiq6RURERETCTEW3iIiIiEiYqegWEREREQkzFd0iIiIiImGmoltEREREJMxUdIuIiIiIhNn/AZgklY9og79ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_sched()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're still not getting great results, so let's see what else we can do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSProp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RMSProp is another variant of SGD introduced by Geoffrey Hinton in Lecture 6e of his Coursera class [\"Neural Networks for Machine Learning\"](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). The main difference from SGD is that it uses an adaptive learning rate: instead of using the same learning rate for every parameter, each parameter gets its own specific learning rate controlled by a global learning rate. That way we can speed up training by giving a higher learning rate to the weights that need to change a lot while the ones that are good enough get a lower learning rate.\n",
    "\n",
    "How do we decide which parameters should have a high learning rate and which should not? We can look at the gradients to get an idea. If a parameter's gradients have been close to zero for a while, that parameter will need a higher learning rate because the loss is flat. On the other hand, if the gradients are all over the place, we should probably be careful and pick a low learning rate to avoid divergence. We can't just average the gradients to see if they're changing a lot, because the average of a large positive and a large negative number is close to zero. Instead, we can use the usual trick of either taking the absolute value or the squared values (and then taking the square root after the mean).\n",
    "\n",
    "Once again, to determine the general tendency behind the noise, we will use a moving average—specifically the moving average of the gradients squared. Then we will update the corresponding weight by using the current gradient (for the direction) divided by the square root of this moving average (that way if it's low, the effective learning rate will be higher, and if it's high, the effective learning rate will be lower):\n",
    "\n",
    "```python\n",
    "w.square_avg = alpha * w.square_avg + (1-alpha) * (w.grad ** 2)\n",
    "new_w = w - lr * w.grad / math.sqrt(w.square_avg + eps)\n",
    "```\n",
    "\n",
    "The `eps` (*epsilon*) is added for numerical stability (usually set at 1e-8), and the default value for `alpha` is usually 0.99."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add this to `Optimizer` by doing much the same thing we did for `avg_grad`, but with an extra `**2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def average_sqr_grad(p, sqr_mom, sqr_avg=None, **kwargs):\n",
    "    if sqr_avg is None: sqr_avg = torch.zeros_like(p.grad.data)\n",
    "    return {'sqr_avg': sqr_mom*sqr_avg + (1-sqr_mom)*p.grad.data**2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can define our step function and optimizer as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rms_prop_step(p, lr, sqr_avg, eps, grad_avg=None, **kwargs):\n",
    "    denom = sqr_avg.sqrt().add_(eps)\n",
    "    p.data.addcdiv_(-lr, p.grad, denom)\n",
    "\n",
    "opt_func = partial(Optimizer, cbs=[average_sqr_grad,rms_prop_step],\n",
    "                   sqr_mom=0.99, eps=1e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.766912</td>\n",
       "      <td>1.845900</td>\n",
       "      <td>0.402548</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.194586</td>\n",
       "      <td>1.510269</td>\n",
       "      <td>0.504459</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.869099</td>\n",
       "      <td>1.447939</td>\n",
       "      <td>0.544968</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = get_learner(opt_func=opt_func)\n",
    "learn.fit_one_cycle(3, 0.003)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better! Now we just have to bring these ideas together, and we have Adam, fastai's default optimizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adam mixes the ideas of SGD with momentum and RMSProp together: it uses the moving average of the gradients as a direction and divides by the square root of the moving average of the gradients squared to give an adaptive learning rate to each parameter.\n",
    "\n",
    "There is one other difference in how Adam calculates moving averages. It takes the *unbiased* moving average, which is:\n",
    "\n",
    "``` python\n",
    "w.avg = beta * w.avg + (1-beta) * w.grad\n",
    "unbias_avg = w.avg / (1 - (beta**(i+1)))\n",
    "```\n",
    "\n",
    "if we are the `i`-th iteration (starting at 0 like Python does). This divisor of `1 - (beta**(i+1))` makes sure the unbiased average looks more like the gradients at the beginning (since `beta < 1`, the denominator is very quickly close to 1).\n",
    "\n",
    "Putting everything together, our update step looks like:\n",
    "``` python\n",
    "w.avg = beta1 * w.avg + (1-beta1) * w.grad\n",
    "unbias_avg = w.avg / (1 - (beta1**(i+1)))\n",
    "w.sqr_avg = beta2 * w.sqr_avg + (1-beta2) * (w.grad ** 2)\n",
    "new_w = w - lr * unbias_avg / sqrt(w.sqr_avg + eps)\n",
    "```\n",
    "\n",
    "Like for RMSProp, `eps` is usually set to 1e-8, and the default for `(beta1,beta2)` suggested by the literature is `(0.9,0.999)`. \n",
    "\n",
    "In fastai, Adam is the default optimizer we use since it allows faster training, but we've found that `beta2=0.99` is better suited to the type of schedule we are using. `beta1` is the momentum parameter, which we specify with the argument `moms` in our call to `fit_one_cycle`. As for `eps`, fastai uses a default of 1e-5. `eps` is not just useful for numerical stability. A higher `eps` limits the maximum value of the adjusted learning rate. To take an extreme example, if `eps` is 1, then the adjusted learning will never be higher than the base learning rate. \n",
    "\n",
    "Rather than show all the code for this in the book, we'll let you look at the optimizer notebook in [fastai's GitHub repository](https://github.com/fastai/fastai) (browse the *nbs* folder and search for the notebook called optimizer). You'll see all the code we've shown so far, along with Adam and other optimizers, and lots of examples and tests.\n",
    "\n",
    "One thing that changes when we go from SGD to Adam is the way we apply weight decay, and it can have important consequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoupled Weight Decay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weight decay, which we discussed in <<chapter_collab>>, is equivalent to (in the case of vanilla SGD) updating the parameters\n",
    "with:\n",
    "\n",
    "``` python\n",
    "new_weight = weight - lr*weight.grad - lr*wd*weight\n",
    "```\n",
    "\n",
    "The last part of this formula explains the name of this technique: each weight is decayed by a factor `lr * wd`. \n",
    "\n",
    "The other name of weight decay is L2 regularization, which consists in adding the sum of all squared weights to the loss (multiplied by the weight decay). As we have seen in <<chapter_collab>>, this can be directly expressed on the gradients with:\n",
    "\n",
    "``` python\n",
    "weight.grad += wd*weight\n",
    "```\n",
    "\n",
    "For SGD, those two formulas are equivalent. However, this equivalence only holds for standard SGD, because we have seen that with momentum, RMSProp or in Adam, the update has some additional formulas around the gradient. \n",
    "\n",
    "Most libraries use the second formulation, but it was pointed out in [\"Decoupled Weight Decay Regularization\"](https://arxiv.org/pdf/1711.05101.pdf) by Ilya Loshchilov and Frank Hutter, that the first one is the only correct approach with the Adam optimizer or momentum, which is why fastai makes it its default.\n",
    "\n",
    "Now you know everything that is hidden behind the line `learn.fit_one_cycle`!\n",
    "\n",
    "Optimizers are only one part of the training process, however when you need to change the training loop with fastai, you can't directly change the code inside the library. Instead, we have designed a system of callbacks to let you write any tweaks you like in independent blocks that you can then mix and match. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you need to change how things work a little bit. In fact, we have already seen examples of this: Mixup, fp16 training, resetting the model after each epoch for training RNNs, and so forth. How do we go about making these kinds of tweaks to the training process?\n",
    "\n",
    "We've seen the basic training loop, which, with the help of the `Optimizer` class, looks like this for a single epoch:\n",
    "\n",
    "```python\n",
    "for xb,yb in dl:\n",
    "    loss = loss_func(model(xb), yb)\n",
    "    loss.backward()\n",
    "    opt.step()\n",
    "    opt.zero_grad()\n",
    "```\n",
    "\n",
    "<<basic_loop>> shows how to picture that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"Basic training loop\" width=\"300\" caption=\"Basic training loop\" id=\"basic_loop\" src=\"images/att_00048.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usual way for deep learning practitioners to customize the training loop is to make a copy of an existing training loop, and then insert the code necessary for their particular changes into it. This is how nearly all code that you find online will look. But it has some very serious problems.\n",
    "\n",
    "It's not very likely that some particular tweaked training loop is going to meet your particular needs. There are hundreds of changes that can be made to a training loop, which means there are billions and billions of possible permutations. You can't just copy one tweak from a training loop here, another from a training loop there, and expect them all to work together. Each will be based on different assumptions about the environment that it's working in, use different naming conventions, and expect the data to be in different formats.\n",
    "\n",
    "We need a way to allow users to insert their own code at any part of the training loop, but in a consistent and well-defined way. Computer scientists have already come up with an elegant solution: the callback. A callback is a piece of code that you write, and inject into another piece of code at some predefined point. In fact, callbacks have been used with deep learning training loops for years. The problem is that in previous libraries it was only possible to inject code in a small subset of places where this may have been required, and, more importantly, callbacks were not able to do all the things they needed to do.\n",
    "\n",
    "In order to be just as flexible as manually copying and pasting a training loop and directly inserting code into it, a callback must be able to read every possible piece of information available in the training loop, modify all of it as needed, and fully control when a batch, epoch, or even the whole training loop should be terminated. fastai is the first library to provide all of this functionality. It modifies the training loop so it looks like <<cb_loop>>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"Training loop with callbacks\" width=\"550\" caption=\"Training loop with callbacks\" id=\"cb_loop\" src=\"images/att_00049.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The real effectiveness of this approach has been borne out over the last couple of years—it has turned out that, by using the fastai callback system, we were able to implement every single new paper we tried and fulfilled every user request for modifying the training loop. The training loop itself has not required modifications. <<some_cbs>> shows just a few of the callbacks that have been added."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"Some fastai callbacks\" width=\"500\" caption=\"Some fastai callbacks\" id=\"some_cbs\" src=\"images/att_00050.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason that this is important is because it means that whatever idea we have in our head, we can implement it. We need never dig into the source code of PyTorch or fastai and hack together some one-off system to try out our ideas. And when we do implement our own callbacks to develop our own ideas, we know that they will work together with all of the other functionality provided by fastai–so we will get progress bars, mixed-precision training, hyperparameter annealing, and so forth.\n",
    "\n",
    "Another advantage is that it makes it easy to gradually remove or add functionality and perform ablation studies. You just need to adjust the list of callbacks you pass along to your fit function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, here is the fastai source code that is run for each batch of the training loop:\n",
    "\n",
    "```python\n",
    "try:\n",
    "    self._split(b);                                  self('before_batch')\n",
    "    self.pred = self.model(*self.xb);                self('after_pred')\n",
    "    self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')\n",
    "    if not self.training: return\n",
    "    self.loss.backward();                            self('after_backward')\n",
    "    self.opt.step();                                 self('after_step')\n",
    "    self.opt.zero_grad()\n",
    "except CancelBatchException:                         self('after_cancel_batch')\n",
    "finally:                                             self('after_batch')\n",
    "```\n",
    "\n",
    "The calls of the form `self('...')` are where the callbacks are called. As you see, this happens after every step. The callback will receive the entire state of training, and can also modify it. For instance, the input data and target labels are in `self.xb` and `self.yb`, respectively; a callback can modify these to alter the data the training loop sees. It can also modify `self.loss`, or even the gradients.\n",
    "\n",
    "Let's see how this works in practice by writing a callback."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you want to write your own callback, the full list of available events is:\n",
    "\n",
    "- `before_fit`:: called before doing anything; ideal for initial setup.\n",
    "- `before_epoch`:: called at the beginning of each epoch; useful for any behavior you need to reset at each epoch.\n",
    "- `before_train`:: called at the beginning of the training part of an epoch.\n",
    "- `before_batch`:: called at the beginning of each batch, just after drawing said batch. It can be used to do any setup necessary for the batch (like hyperparameter scheduling) or to change the input/target before it goes into the model (for instance, apply Mixup).\n",
    "- `after_pred`:: called after computing the output of the model on the batch. It can be used to change that output before it's fed to the loss function.\n",
    "- `after_loss`:: called after the loss has been computed, but before the backward pass. It can be used to add penalty to the loss (AR or TAR in RNN training, for instance).\n",
    "- `after_backward`:: called after the backward pass, but before the update of the parameters. It can be used to make changes to the gradients before said update (via gradient clipping, for instance).\n",
    "- `after_step`:: called after the step and before the gradients are zeroed.\n",
    "- `after_batch`:: called at the end of a batch, to perform any required cleanup before the next one.\n",
    "- `after_train`:: called at the end of the training phase of an epoch.\n",
    "- `before_validate`:: called at the beginning of the validation phase of an epoch; useful for any setup needed specifically for validation.\n",
    "- `after_validate`:: called at the end of the validation part of an epoch.\n",
    "- `after_epoch`:: called at the end of an epoch, for any cleanup before the next one.\n",
    "- `after_fit`:: called at the end of training, for final cleanup.\n",
    "\n",
    "The elements of this list are available as attributes of the special variable `event`, so you can just type `event.` and hit Tab in your notebook to see a list of all the options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at an example. Do you recall how in <<chapter_nlp_dive>> we needed to ensure that our special `reset` method was called at the start of training and validation for each epoch? We used the `ModelResetter` callback provided by fastai to do this for us. But how does it work? Here's the full source code for that class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelResetter(Callback):\n",
    "    def before_train(self):    self.model.reset()\n",
    "    def before_validate(self): self.model.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, that's actually it! It just does what we said in the preceding paragraph: after completing training or validation for an epoch, call a method named `reset`.\n",
    "\n",
    "Callbacks are often \"short and sweet\" like this one. In fact, let's look at one more. Here's the fastai source for the callback that adds RNN regularization (AR and TAR):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNNRegularizer(Callback):\n",
    "    def __init__(self, alpha=0., beta=0.): self.alpha,self.beta = alpha,beta\n",
    "\n",
    "    def after_pred(self):\n",
    "        self.raw_out,self.out = self.pred[1],self.pred[2]\n",
    "        self.learn.pred = self.pred[0]\n",
    "\n",
    "    def after_loss(self):\n",
    "        if not self.training: return\n",
    "        if self.alpha != 0.:\n",
    "            self.learn.loss += self.alpha * self.out[-1].float().pow(2).mean()\n",
    "        if self.beta != 0.:\n",
    "            h = self.raw_out[-1]\n",
    "            if len(h)>1:\n",
    "                self.learn.loss += self.beta * (h[:,1:] - h[:,:-1]\n",
    "                                               ).float().pow(2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> note: Code It Yourself: Go back and reread \"Activation Regularization and Temporal Activation Regularization\" in <<chapter_nlp_dive>> then take another look at the code here. Make sure you understand what it's doing, and why."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both of these examples, notice how we can access attributes of the training loop by directly checking `self.model` or `self.pred`. That's because a `Callback` will always try to get an attribute it doesn't have inside the `Learner` associated with it. These are shortcuts for `self.learn.model` or `self.learn.pred`. Note that they work for reading attributes, but not for writing them, which is why when `RNNRegularizer` changes the loss or the predictions you see `self.learn.loss = ` or `self.learn.pred = `. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When writing a callback, the following attributes of `Learner` are available:\n",
    "\n",
    "- `model`:: The model used for training/validation.\n",
    "- `data`:: The underlying `DataLoaders`.\n",
    "- `loss_func`:: The loss function used.\n",
    "- `opt`:: The optimizer used to update the model parameters.\n",
    "- `opt_func`:: The function used to create the optimizer.\n",
    "- `cbs`:: The list containing all the `Callback`s.\n",
    "- `dl`:: The current `DataLoader` used for iteration.\n",
    "- `x`/`xb`:: The last input drawn from `self.dl` (potentially modified by callbacks). `xb` is always a tuple (potentially with one element) and `x` is detuplified. You can only assign to `xb`.\n",
    "- `y`/`yb`:: The last target drawn from `self.dl` (potentially modified by callbacks). `yb` is always a tuple (potentially with one element) and `y` is detuplified. You can only assign to `yb`.\n",
    "- `pred`:: The last predictions from `self.model` (potentially modified by callbacks).\n",
    "- `loss`:: The last computed loss (potentially modified by callbacks).\n",
    "- `n_epoch`:: The number of epochs in this training.\n",
    "- `n_iter`:: The number of iterations in the current `self.dl`.\n",
    "- `epoch`:: The current epoch index (from 0 to `n_epoch-1`).\n",
    "- `iter`:: The current iteration index in `self.dl` (from 0 to `n_iter-1`).\n",
    "\n",
    "The following attributes are added by `TrainEvalCallback` and should be available unless you went out of your way to remove that callback:\n",
    "\n",
    "- `train_iter`:: The number of training iterations done since the beginning of this training\n",
    "- `pct_train`:: The percentage of training iterations completed (from 0. to 1.)\n",
    "- `training`:: A flag to indicate whether or not we're in training mode\n",
    "\n",
    "The following attribute is added by `Recorder` and should be available unless you went out of your way to remove that callback:\n",
    "\n",
    "- `smooth_loss`:: An exponentially averaged version of the training loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks can also interrupt any part of the training loop by using a system of exceptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback Ordering and Exceptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, callbacks need to be able to tell fastai to skip over a batch, or an epoch, or stop training altogether. For instance, consider `TerminateOnNaNCallback`. This handy callback will automatically stop training any time the loss becomes infinite or `NaN` (*not a number*). Here's the fastai source for this callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TerminateOnNaNCallback(Callback):\n",
    "    run_before=Recorder\n",
    "    def after_batch(self):\n",
    "        if torch.isinf(self.loss) or torch.isnan(self.loss):\n",
    "            raise CancelFitException"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line `raise CancelFitException` tells the training loop to interrupt training at this point. The training loop catches this exception and does not run any further training or validation. The callback control flow exceptions available are:\n",
    "\n",
    "- `CancelBatchException`:: Skip the rest of this batch and go to `after_batch`.\n",
    "- `CancelTrainException`:: Skip the rest of the training part of the epoch and go to `after_train`.\n",
    "- `CancelValidException`:: Skip the rest of the validation part of the epoch and go to `after_validate`.\n",
    "- `CancelEpochException`:: Skip the rest of this epoch and go to `after_epoch`.\n",
    "- `CancelFitException`:: Interrupt training and go to `after_fit`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can detect if one of those exceptions has occurred and add code that executes right after with the following events:\n",
    "\n",
    "- `after_cancel_batch`:: Reached immediately after a `CancelBatchException` before proceeding to `after_batch`\n",
    "- `after_cancel_train`:: Reached immediately after a `CancelTrainException` before proceeding to `after_train`\n",
    "- `after_cancel_valid`:: Reached immediately after a `CancelValidException` before proceeding to `after_valid`\n",
    "- `after_cancel_epoch`:: Reached immediately after a `CancelEpochException` before proceeding to `after_epoch`\n",
    "- `after_cancel_fit`:: Reached immediately after a `CancelFitException` before proceeding to `after_fit`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, callbacks need to be called in a particular order. For example, in the case of `TerminateOnNaNCallback`, it's important that `Recorder` runs its `after_batch` after this callback, to avoid registering an `NaN` loss. You can specify `run_before` (this callback must run before ...) or `run_after` (this callback must run after ...) in your callback to ensure the ordering that you need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter we took a close look at the training loop, exploring different variants of SGD and why they can be more powerful. At the time of writing, developing new optimizers is a very active area of research, so by the time you read this chapter there may be an addendum on the book's website that presents new variants. Be sure to check out how our general optimizer framework can help you implement new optimizers very quickly.\n",
    "\n",
    "We also examined the powerful callback system that allows you to customize every bit of the training loop by enabling you to inspect and modify any parameter you like between each step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questionnaire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What is the equation for a step of SGD, in math or code (as you prefer)?\n",
    "1. What do we pass to `vision_learner` to use a non-default optimizer?\n",
    "1. What are optimizer callbacks?\n",
    "1. What does `zero_grad` do in an optimizer?\n",
    "1. What does `step` do in an optimizer? How is it implemented in the general optimizer?\n",
    "1. Rewrite `sgd_cb` to use the `+=` operator, instead of `add_`.\n",
    "1. What is \"momentum\"? Write out the equation.\n",
    "1. What's a physical analogy for momentum? How does it apply in our model training settings?\n",
    "1. What does a bigger value for momentum do to the gradients?\n",
    "1. What are the default values of momentum for 1cycle training?\n",
    "1. What is RMSProp? Write out the equation.\n",
    "1. What do the squared values of the gradients indicate?\n",
    "1. How does Adam differ from momentum and RMSProp?\n",
    "1. Write out the equation for Adam.\n",
    "1. Calculate the values of `unbias_avg` and `w.avg` for a few batches of dummy values.\n",
    "1. What's the impact of having a high `eps` in Adam?\n",
    "1. Read through the optimizer notebook in fastai's repo, and execute it.\n",
    "1. In what situations do dynamic learning rate methods like Adam change the behavior of weight decay?\n",
    "1. What are the four steps of a training loop?\n",
    "1. Why is using callbacks better than writing a new training loop for each tweak you want to add?\n",
    "1. What aspects of the design of fastai's callback system make it as flexible as copying and pasting bits of code?\n",
    "1. How can you get the list of events available to you when writing a callback?\n",
    "1. Write the `ModelResetter` callback (without peeking).\n",
    "1. How can you access the necessary attributes of the training loop inside a callback? When can you use or not use the shortcuts that go with them?\n",
    "1. How can a callback influence the control flow of the training loop?\n",
    "1. Write the `TerminateOnNaN` callback (without peeking, if possible).\n",
    "1. How do you make sure your callback runs after or before another callback?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Look up the \"Rectified Adam\" paper, implement it using the general optimizer framework, and try it out. Search for other recent optimizers that work well in practice, and pick one to implement.\n",
    "1. Look at the mixed-precision callback with the documentation. Try to understand what each event and line of code does.\n",
    "1. Implement your own version of the learning rate finder from scratch. Compare it with fastai's version.\n",
    "1. Look at the source code of the callbacks that ship with fastai. See if you can find one that's similar to what you're looking to do, to get some inspiration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Foundations of Deep Learning: Wrap up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, you have made it to the end of the \"foundations of deep learning\" section of the book! You now understand how all of fastai's applications and most important architectures are built, and the recommended ways to train them—and you have all the information you need to build these from scratch. While you probably won't need to create your own training loop, or batchnorm layer, for instance, knowing what is going on behind the scenes is very helpful for debugging, profiling, and deploying your solutions.\n",
    "\n",
    "Since you understand the foundations of fastai's applications now, be sure to spend some time digging through the source notebooks and running and experimenting with parts of them. This will give you a better idea of how everything in fastai is developed.\n",
    "\n",
    "In the next section, we will be looking even further under the covers: we'll explore how the actual forward and backward passes of a neural network are done, and we will see what tools are at our disposal to get better performance. We will then continue with a project that brings together all the material in the book, which we will use to build a tool for interpreting convolutional neural networks. Last but not least, we'll finish by building fastai's `Learner` class from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}