explanations/rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNNs\n",
    "\n",
    "*You can watch a [video](https://youtu.be/4wuIOcD1LLI) of this lesson if you prefer.*\n",
    "\n",
    "In the [last lesson](https://github.com/VikParuchuri/zero_to_gpt/blob/master/explanations/classification.ipynb), we learned how to use a neural network to perform classification.  This is one important piece of using a neural network to do NLP (natural language processing).\n",
    "\n",
    "In this lesson, we'll learn another important piece - how to use neural networks to take in sequences of input and make predictions.  A sequence of input can be a sentence made up of words, or a series of weather observations.\n",
    "\n",
    "Each element in a sequence has a position.  We can think of those positions as being steps in time.  Here is a sequence of temperatures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1970-01-01    60.0\n",
       "1970-01-02    52.0\n",
       "1970-01-03    52.0\n",
       "1970-01-04    53.0\n",
       "1970-01-05    52.0\n",
       "1970-01-06    50.0\n",
       "1970-01-07    52.0\n",
       "1970-01-08    56.0\n",
       "1970-01-09    54.0\n",
       "1970-01-10    57.0\n",
       "Name: tmax, dtype: float64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Read in our data, and fill missing values\n",
    "data = pd.read_csv(\"../data/clean_weather.csv\", index_col=0)\n",
    "data = data.ffill()\n",
    "\n",
    "# Display a sequence of temperatures\n",
    "data[\"tmax\"].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence has 10 elements.  The first sequence element (at time step 0) is `60`.  The second sequence element (at time step 1) is `52`, and so on.\n",
    "\n",
    "Let's say we want to predict the next element in the sequence (at time step `10`).  With a normal neural network, you have to treat each element in a sequence as a separate input feature, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[60., 52., 52., 53., 52., 50., 52., 56., 54., 57.]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Turn our sequence into a single row of data\n",
    "data[\"tmax\"].head(10).to_numpy()[np.newaxis,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could feed the above sequence as a single training example into a neural network to predict the next temperature.  But what if we don't know the length of the sequence beforehand?  For example, what if we wanted the neural network to predict the next word that came after a prompt?\n",
    "\n",
    "We could have prompts with different lengths, like:\n",
    "\n",
    "- `Write me a song`\n",
    "- `Tell me a story about dinosaurs`\n",
    "- `Add 1 and 2`\n",
    "\n",
    "We want our network to be able to handle all of these inputs, even though they have different lengths.  One way to do this is to add extra zeros to the end of the sequence to make all of the sequences the same length.  Then our network will handle a fixed-sized input.\n",
    "\n",
    "But this introduces the issue that the neural network will have to learn parameters for each position separately.  For example, if we're passing in two weather observations of different lengths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[-0.72725587, -1.68779357, -1.68779357, -1.56772636, -1.68779357,\n        -1.92792799]])"
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"tmax\"].head(6).to_numpy()[np.newaxis,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[-0.24698703, -0.36705424, -0.48712145, -0.60718866, -0.72725587,\n        -0.48712145,  0.11321461, -0.0068526 ,  0.47341624, -0.48712145]])"
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"tmax\"].tail(10).to_numpy()[np.newaxis,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These temperatures are all related - the temperature today affects the temperature tomorrow.  But if we train a neural network to evaluate each sequence position separately, it won't be able to efficiently learn the relationship between one sequence position and the next.  The network will effectively have to re-learn this relationship between each pair of time steps.\n",
    "\n",
    "Instead of the network being able to say \"all time steps are related to the previous time step\", it will have to say \"time step `0` is related to time step `1`, time step `1` is related to time step `2`\", and so on.  This is a lot less efficient.\n",
    "\n",
    "RNNs are a kind of neural network that solve this problem by processing an entire sequence at once.  In this lesson, we'll learn how to build one in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN Architecture\n",
    "\n",
    "RNNs work by sharing parameters (weights and biases) across steps in a sequence.  At a very high level, a recurrent neural network looks like this:\n",
    "\n",
    "![Rolled RNN](images/rnn/rnn_rolled.svg)\n",
    "\n",
    "What we see above is called a compact representation of an RNN.  We pass input (values from a single time step) into the bottom of an RNN.  It is then passed through the input step (I) and the hidden step (H) before getting to the output step (O) where you get a prediction for the next element in the sequence.\n",
    "\n",
    "The arrow connecting H to itself is called recurrence.  It means that the H layers are connected to each other across sequence steps.  To see how this works, let's take a look at an expanded representation of an RNN:\n",
    "\n",
    "![Unrolled RNN](images/rnn/rnn_unrolled.svg)\n",
    "\n",
    "Each sequence element is labeled at the bottom with the time step of the element.  The first element in the sequence is `t0`, the second is `t1`, and so on.  You can see that each element is passed into the input step, then to the hidden step.  But the output of the hidden step is passed both to the output, and to the next hidden step.\n",
    "\n",
    "This means that at each step, an RNN knows about the inputs at previous steps.  The \"memory\" of the network is stored in the hidden step (H), and represented as an internal matrix of values.  This matrix is updated at each time step with new information about the sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN Operations\n",
    "\n",
    "It can help to look at the exact operations that happen at each step in an RNN:\n",
    "\n",
    "![Unrolled RNN](images/rnn/rnn_operations.svg)\n",
    "\n",
    "An RNN has 3 steps:\n",
    "\n",
    "- Input - take in a sequence item, and multiply it by the input weight\n",
    "- Hidden - Take the previous hidden state, and multiply it by the hidden weight.  Add in the input, then apply a nonlinear activation function.\n",
    "- Output weight - Take the hidden state, and multiply by the output weight.\n",
    "\n",
    "We also add in bias terms at the hidden and output steps, but the biases are left off the diagram to make it easier to read.\n",
    "\n",
    "The trickiest part of an RNN is the hidden step.  It's this step that gives an RNN its power, by enabling it to have memory.  The hidden state of the RNN stores information about previous sequence elements.  The hidden weights enable the RNN to remember or forget certain information about past sequence elements.\n",
    "\n",
    "This way, the RNN can have knowledge of past elements in the sequence without having separate parameters for each sequence item.  This is called parameter sharing - the RNN shares the same set of input, hidden, and output weights for every sequence element.\n",
    "\n",
    "This enables the RNN to be used for sequences of any length - we don't need a fixed size input sequence length.\n",
    "\n",
    "## Forward Pass\n",
    "\n",
    "Let's go through an example to see how this works.  We'll initialize each weight matrix, then do a sample forward pass with 3 sequence elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([66., 70., 62.])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set a random seed so the random numbers are the same every time\n",
    "np.random.seed(0)\n",
    "# Take the input number and turn it into 2 features\n",
    "i_weight = np.random.rand(1,2)\n",
    "# Hidden to hidden weight connection - 2 features to 2 features\n",
    "h_weight = np.random.rand(2,2)\n",
    "# Output weight connection - turn 2 features into one prediction\n",
    "o_weight = np.random.rand(2,1)\n",
    "\n",
    "# Get 3 temperature values from our data\n",
    "temps = data[\"tmax\"].tail(3).to_numpy()\n",
    "temps"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Above, we initialized our network parameters.  We used the numpy `random.rand` function to randomly create parameter matrices of a certain shape.  This network will take in a single input feature, turn it into `2` hidden features, and output one prediction.\n",
    "\n",
    "We can then setup our input to the forward pass.  We'll reshape each input value into a `1x1` matrix to make multiplication more convenient:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign the sequence input at each time step to a different variable.\n",
    "# x0 means input at time step 0\n",
    "# Ensure that each element is a 1x1 matrix, so we can multiply it\n",
    "x0 = temps[0].reshape(1,1)\n",
    "x1 = temps[1].reshape(1,1)\n",
    "x2 = temps[2].reshape(1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can feed the element at time step 0 into our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[57.94406231]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate xi at time step 0\n",
    "xi_0 = x0 @ i_weight\n",
    "\n",
    "# There is no previous time step, so there is no hidden state\n",
    "# apply relu over the input to get the hidden state for time step 0 xh_0\n",
    "xh_0 = np.maximum(0, xi_0)\n",
    "\n",
    "# Get the output at time step 0 xo_0\n",
    "xo_0 = xh_0 @ o_weight\n",
    "\n",
    "xo_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`xo_0` is our prediction for the next step in the sequence.\n",
    "\n",
    "We can then move the network forward to time step `1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[124.54916092]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We feed the input in the same way as the previous time step\n",
    "xi_1 = x1 @ i_weight\n",
    "\n",
    "# This time, we do have a previous time step, so we calculate xh\n",
    "# This is multiplying the previous hidden state xh_0 by the hidden weights\n",
    "xh = xh_0 @ h_weight\n",
    "\n",
    "# We add the previous hidden state (times h_weight) to the input at time step 1\n",
    "xh_1 = np.maximum(0, xh + xi_1)\n",
    "\n",
    "# We again find the output by multiplying xh_1 by the output weight\n",
    "xo_1 = xh_1 @ o_weight\n",
    "\n",
    "xo_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`xo_1` is our prediction for the next sequence element.  Now we can do the same for our final time step, 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[190.94853131]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We feed the input in the same way as the previous time step\n",
    "xi_2 = x2 @ i_weight\n",
    "\n",
    "# This time, we do have a previous time step, so we calculate xh\n",
    "# This is multiplying the previous hidden state xh_1 by the hidden weights\n",
    "xh = xh_1 @ h_weight\n",
    "\n",
    "# We add the previous hidden state (times h_weight) to the input at time step 2\n",
    "xh_2 = np.maximum(0, xh + xi_2)\n",
    "\n",
    "# We again find the output by multiplying xh_1 by the output weight\n",
    "xo_2 = xh_2 @ o_weight\n",
    "\n",
    "xo_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now passed through 3 forward steps of our RNN!  The output `x0` at each time step is the prediction for the next element in the sequence.\n",
    "\n",
    "The hidden state of the RNN allows the network to have information about all past sequence elements.  So when we're processing the sequence item at time step 2, the hidden state of the RNN stores information about the sequence elements at time step `0` and `1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonlinearity\n",
    "\n",
    "You may have noticed that the output values increased in each time step of our RNN.  That's because the hidden states kept getting larger and larger.  Unlike the sigmoid or softmax activation functions, relu doesn't change the scale of the inputs at all (it just sets some to 0).  This means that some values get repeatedly multiplied and grow larger and larger.\n",
    "\n",
    "We can see this below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[36.22169126, 47.20249818]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xh_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[124.88411227, 152.84252918]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xh_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our hidden state at time step 2 is much larger than the initial hidden state.\n",
    "\n",
    "To fix this, we usually use the $\\tanh$ activation function for RNNs.  The equation for $\\tanh$ is:\n",
    "\n",
    "$$\\tanh = \\dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$$\n",
    "\n",
    "As x gets large, $e^{x}$ gets very big, and $e^{-x}$ gets very small.  This pushes the output of $\\tanh$ towards `1`.  When x gets very small (negative), $e^{-x}$ gets very large, and $e^{x}$ gets very small.  This results in `-1`.\n",
    "\n",
    "We can graph the $tanh$ function to see how this works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x11e598410>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABDBUlEQVR4nO3deXxU9b3/8fdMlkmCTBYSMgkECIsssgolBq0buSRKW+m1FiwW5SJUBS2GqqSPCiJVXLjWavkV24LoFet21Wr1ohhFq0TQACoIFDAQlkwChGSyQLY5vz9CBkaSkEAms72ej8d5kDnzPWc+Xw5J3nzP95xjMgzDEAAAQAAxe7sAAACAjkbAAQAAAYeAAwAAAg4BBwAABBwCDgAACDgEHAAAEHAIOAAAIOAQcAAAQMAJ9XYB3uB0OnXo0CF17dpVJpPJ2+UAAIA2MAxDFRUVSk5Oltnc+hhNUAacQ4cOKSUlxdtlAACAc7B//3717Nmz1TZBGXC6du0qqfEvyGq1erkaAADQFg6HQykpKa7f460JyoDTdFrKarUScAAA8DNtmV7CJGMAABBwCDgAACDgEHAAAEDAIeAAAICAQ8ABAAABh4ADAAACDgEHAAAEHAIOAAAIOAQcAAAQcDwacD755BP9+Mc/VnJyskwmk958882zbrNu3TpdfPHFslgs6t+/v1atWnVGm2XLlqlPnz6KiIhQWlqaNm7c2PHFAwAAv+XRgFNVVaURI0Zo2bJlbWpfUFCgiRMn6qqrrtKWLVs0d+5c3XrrrXrvvfdcbV5++WVlZ2dr4cKF2rRpk0aMGKHMzEyVlJR4qhsAAMDPmAzDMDrlg0wmvfHGG5o0aVKLbe677z6988472rp1q2vdlClTVFZWpjVr1kiS0tLS9IMf/EB/+tOfJElOp1MpKSm68847NX/+/DbV4nA4FB0drfLycp5FBQCAn2jP72+fethmXl6eMjIy3NZlZmZq7ty5kqTa2lrl5+crJyfH9b7ZbFZGRoby8vJa3G9NTY1qampcrx0OR8cWDgB+xjAM1dQ7daKuwfXniTqnauob/2xaX9/gVL3TkNMw1OA8bTEMOZ2G6k++dhqNXzudhhqcJz9Dhpr+C200fuipr0++NE6+Or3dqa9PrTy1zff2+b39wHeM7h2rHw1P9trn+1TAsdvtSkxMdFuXmJgoh8Oh48eP69ixY2poaGi2zY4dO1rc75IlS7Ro0SKP1AwAvqKsulZF5SdkLz+hQ+XHZS8/oaNVtSo/XifH8TqVH69TWXXjnxUn6uQkE8CDauqdBBxPy8nJUXZ2tuu1w+FQSkqKFysCgHN3vLZBm/cf046iCu0qqdSekkrtKqnQseq6c9pfqNkkS6hZEWEhp/48+XVYiEkh5sbFbDIp9PSvQxr/bHo/5OTXZrNJppP7NpmkpleNXzetN7k+v+nL5tudau/a4nv7PLlKp+0SPmBEzxivfr5PBRybzabi4mK3dcXFxbJarYqMjFRISIhCQkKabWOz2Vrcr8VikcVi8UjNAOBp1bX1+vy7o9pYcEwbC47q6wPlqm9h+KVbl3DZoiOUFB0hW3SEunWxKCYqTNGRjUvT19bIMEWFhyoi1KzQEO4YgsDjUwEnPT1d7777rtu6tWvXKj09XZIUHh6u0aNHKzc31zVZ2el0Kjc3V3PmzOnscgHAY2rqG/T+tmK9/dUhffzvw6qpd7q9b7NGaHjPaF2Y2FUDEi9Qv4TGJTI8xEsVA77FowGnsrJSu3fvdr0uKCjQli1bFBcXp169eiknJ0cHDx7U888/L0m67bbb9Kc//Un33nuv/uu//ksffvihXnnlFb3zzjuufWRnZ+vmm2/WmDFjNHbsWD355JOqqqrS9OnTPdkVAOgUh8qOa+WnBXpt0wGVnXbKqWdspMb166axqd2UlhqnnrGRbqd5ALjzaMD58ssvddVVV7leN82Dufnmm7Vq1SoVFRWpsLDQ9X5qaqreeecd3X333frjH/+onj176m9/+5syMzNdbSZPnqzDhw9rwYIFstvtGjlypNasWXPGxGMA8CcHy47rjx/8W29sPqi6hsbTTzZrhK4f3UPXDkvSkCQrgQZoh067D44v4T44AHzF8doG/fnjPXrm4z2u01Dpfbtp5uWpuuLC7goxE2qAJn57HxwACCabC4/p7pe3aO/RaklSWmqc7s0apNG9Y71cGeD/CDgA0MmcTkN/+mi3/pi7Sw1OQ0nREbr/R0N0zVAbp6GADkLAAYBOdLy2QfNe3aJ3v7FLkn4yIlmLrxuq6KgwL1cGBBYCDgB0ktKqWk1/dqO+OlCusBCTHvrpMP18DDcdBTyBgAMAnaCsulZT/7ZB24sciokK0zM3jVZa327eLgsIWAQcAPCw8uN1umlFY7iJv8Cil2alqX/3rt4uCwho3J8bADyorsGp21/I19aDDnXrEq4XZxJugM5AwAEAD3rone1av+eoosJD9D8z0nRhIuEG6AwEHADwkFe+2K9V6/dKkp74+UgNSebGokBnIeAAgAd8d7hSC97aKkm6O+NCZQ21ebkiILgQcACggzU4Dc179SudqHPq0v7ddOfV/b1dEhB0CDgA0MGe+WSPNheWqaslVI/9bITMPE8K6HQEHADoQHuPVOnJtbskSQt+PEQ9YiK9XBEQnAg4ANCBHnp3u2obnPrhgHj9bHRPb5cDBC0CDgB0kM92H9Hab4sVYjZpwY+G8OBMwIsIOADQARqchhb/81tJ0k1pvTSA+90AXkXAAYAO8Mbmg9phr1B0ZJjmZlzo7XKAoEfAAYDz1OA09P8+2i1Juu2KfortEu7ligAQcADgPP3f1iJ9d6RK0ZFh+mV6b2+XA0AEHAA4L4ZhaNlHeyRJt4zrowssoV6uCIBEwAGA8/LRzhJtL3IoKjxE0y/t4+1yAJxEwAGA8/C3fxVIkm66pLdioph7A/gKAg4AnKM9hyu1fs9RmUzSNObeAD6FgAMA5+jvGwolSVcP7K6esVFergbA6Qg4AHAOTtQ16LVNByRJUy/p5eVqAHwfAQcAzsG73xSprLpOPWIidcWF3b1dDoDvIeAAwDlYffL01C/SeinEzDOnAF9DwAGAdtpfWq38fcdkNkk38MRwwCcRcACgnd766pAkKb1fN3W3Rni5GgDNIeAAQDu9fTLg/GREspcrAdASAg4AtMO/iyu0w16hsBCTsi5K8nY5AFrQKQFn2bJl6tOnjyIiIpSWlqaNGze22PbKK6+UyWQ6Y5k4caKrzS233HLG+1lZWZ3RFQBBrmn05ooLuys6KszL1QBoicefCvfyyy8rOztby5cvV1pamp588kllZmZq586d6t79zEsrX3/9ddXW1rpeHz16VCNGjNANN9zg1i4rK0vPPvus67XFYvFcJwBAjQ/WbJp/8+MRjN4AvszjIzhPPPGEZs6cqenTp2vIkCFavny5oqKitHLlymbbx8XFyWazuZa1a9cqKirqjIBjsVjc2sXGxnq6KwCC3LZDDu07Wq3IsBD9x5BEb5cDoBUeDTi1tbXKz89XRkbGqQ80m5WRkaG8vLw27WPFihWaMmWKunTp4rZ+3bp16t69uwYOHKjbb79dR48ebXEfNTU1cjgcbgsAtNeHO0okST8cEK+ocI8PgAM4Dx4NOEeOHFFDQ4MSE93/p5OYmCi73X7W7Tdu3KitW7fq1ltvdVuflZWl559/Xrm5uXr00Uf18ccf65prrlFDQ0Oz+1myZImio6NdS0pKyrl3CkDQyj0ZcMYP5s7FgK/z6f+CrFixQsOGDdPYsWPd1k+ZMsX19bBhwzR8+HD169dP69at0/jx48/YT05OjrKzs12vHQ4HIQdAuxyuqNFX+8skSVcNIuAAvs6jIzjx8fEKCQlRcXGx2/ri4mLZbLZWt62qqtJLL72kGTNmnPVz+vbtq/j4eO3evbvZ9y0Wi6xWq9sCAO3x0cnRmxE9o9W9Kzf3A3ydRwNOeHi4Ro8erdzcXNc6p9Op3Nxcpaent7rtq6++qpqaGt10001n/ZwDBw7o6NGjSkriqgYAnpG7o/E/alcPYnIx4A88fhVVdna2/vrXv+q5557T9u3bdfvtt6uqqkrTp0+XJE2bNk05OTlnbLdixQpNmjRJ3bp1c1tfWVmpe+65R59//rn27t2r3NxcXXfdderfv78yMzM93R0AQaimvkH/2nVEEvNvAH/h8Tk4kydP1uHDh7VgwQLZ7XaNHDlSa9ascU08LiwslNnsnrN27typTz/9VO+///4Z+wsJCdHXX3+t5557TmVlZUpOTtaECRO0ePFi7oUDwCM2fFeq6toGJVotuiiZU9yAPzAZhmF4u4jO5nA4FB0drfLycubjADirh975Vn/9V4Emj0nRoz8b7u1ygKDVnt/fPIsKAM5i/Z7G+2xdOiDey5UAaCsCDgC04lhVrb4tarw5aHrfbmdpDcBXEHAAoBUbCo7KMKQLEy9QQlfm+QH+goADAK1oOj01rh+npwB/QsABgFZ8trvx8vD0fpyeAvwJAQcAWlDsOKE9h6tkMkmXpBJwAH9CwAGAFuSdPD01NDla0VFhXq4GQHsQcACgBev3NJ6eGsfpKcDvEHAAoAVf7jsmSRqbGuflSgC0FwEHAJpRWlWr7w5XSZIu7hXr5WoAtBcBBwCasbmwcfSmX0IXxXYJ93I1ANqLgAMAzcg/eXpqdG9GbwB/RMABgGZsKiTgAP6MgAMA31PX4NRX+8slEXAAf0XAAYDv2VFUoeN1DbJGhKpv/AXeLgfAOSDgAMD35O8rlSRd3DtWZrPJy9UAOBcEHAD4nvzCMknSaC4PB/wWAQcAvmcTV1ABfo+AAwCnOVJZo4Nlx2UyScNTYrxdDoBzRMABgNN8c7Dx6qm+8V10gSXUy9UAOFcEHAA4zdYDjQFnWI9oL1cC4HwQcADgNF+fHMEZSsAB/BoBBwBOs/VkwBneM8a7hQA4LwQcADjpcEWNispPyGSSLkq2erscAOeBgAMAJ209bYJxFyYYA36NgAMAJ33D6SkgYBBwAOCkb5hgDAQMAg4AnPQNl4gDAYOAAwBqnGBsdzDBGAgUBBwAkLTtUOPoTSoTjIGAQMABAEk77BWSpCFJjN4AgaBTAs6yZcvUp08fRUREKC0tTRs3bmyx7apVq2QymdyWiIgItzaGYWjBggVKSkpSZGSkMjIytGvXLk93A0AA21HkkCQNJuAAAcHjAefll19Wdna2Fi5cqE2bNmnEiBHKzMxUSUlJi9tYrVYVFRW5ln379rm9/9hjj+mpp57S8uXLtWHDBnXp0kWZmZk6ceKEp7sDIEA1jeAMsnX1ciUAOoLHA84TTzyhmTNnavr06RoyZIiWL1+uqKgorVy5ssVtTCaTbDaba0lMTHS9ZxiGnnzySf3ud7/Tddddp+HDh+v555/XoUOH9Oabb3q6OwACUG29U7tLKiVJgxjBAQKCRwNObW2t8vPzlZGRceoDzWZlZGQoLy+vxe0qKyvVu3dvpaSk6LrrrtO2bdtc7xUUFMhut7vtMzo6WmlpaS3us6amRg6Hw20BgCZ7Dleq3mmoa0SokqMjzr4BAJ/n0YBz5MgRNTQ0uI3ASFJiYqLsdnuz2wwcOFArV67UP/7xD73wwgtyOp0aN26cDhw4IEmu7dqzzyVLlig6Otq1pKSknG/XAASQHfaT829sVplMJi9XA6Aj+NxVVOnp6Zo2bZpGjhypK664Qq+//roSEhL0zDPPnPM+c3JyVF5e7lr279/fgRUD8Hfbi07Ov0li/g0QKDwacOLj4xUSEqLi4mK39cXFxbLZbG3aR1hYmEaNGqXdu3dLkmu79uzTYrHIarW6LQDQZPvJK6gG2fjZAAQKjwac8PBwjR49Wrm5ua51TqdTubm5Sk9Pb9M+Ghoa9M033ygpKUmSlJqaKpvN5rZPh8OhDRs2tHmfAHA61xVUjOAAAcPjt+vMzs7WzTffrDFjxmjs2LF68sknVVVVpenTp0uSpk2bph49emjJkiWSpAcffFCXXHKJ+vfvr7KyMj3++OPat2+fbr31VkmNV1jNnTtXv//97zVgwAClpqbq/vvvV3JysiZNmuTp7gAIMEcqa3S4okaSNDCRgAMECo8HnMmTJ+vw4cNasGCB7Ha7Ro4cqTVr1rgmCRcWFspsPjWQdOzYMc2cOVN2u12xsbEaPXq01q9fryFDhrja3HvvvaqqqtKsWbNUVlamyy67TGvWrDnjhoAAcDY7T47e9O4WxSMagABiMgzD8HYRnc3hcCg6Olrl5eXMxwGC3N/+9Z1+/852ZV1k0/JfjvZ2OQBa0Z7f3z53FRUAdKZdxY03+Lsw8QIvVwKgIxFwAAS1XSWNp6gGMP8GCCgEHABByzAM7Tr5iIYBjOAAAYWAAyBoHa6oUcWJeplNUmp8F2+XA6ADEXAABK2m0Zs+3brIEhri5WoAdCQCDoCgtau4cf5N/+6cngICDQEHQNBi/g0QuAg4AIKWK+B05woqINAQcAAErd0nAw6nqIDAQ8ABEJSOVtaotKpWJpPUL4GAAwQaAg6AoNR0eiolNkqR4VxBBQQaAg6AoHRq/g2jN0AgIuAACEq7my4R5woqICARcAAEpaYRnP7MvwECEgEHQFDac7gx4PTjFBUQkAg4AIJOZU29ih01kqR+8QQcIBARcAAEnYLDVZKkbl3CFR0V5uVqAHgCAQdA0PnuSOPpqb4JPEEcCFQEHABBZ8/JEZy+nJ4CAhYBB0DQ+e4wIzhAoCPgAAg63zWN4HCJOBCwCDgAgophGCo40hRwGMEBAhUBB0BQsTtO6Hhdg0LNJvWKi/J2OQA8hIADIKg0nZ7qFRelsBB+BAKBiu9uAEGFCcZAcCDgAAgqe5hgDAQFAg6AoPJd0wTjeEZwgEBGwAEQVE6domIEBwhkBBwAQeNEXYMOlh2XJKUyggMENAIOgKBRWFotw5C6WkIVf0G4t8sB4EEEHABBY+/J+Td94rvIZDJ5uRoAntQpAWfZsmXq06ePIiIilJaWpo0bN7bY9q9//at++MMfKjY2VrGxscrIyDij/S233CKTyeS2ZGVlebobAPzc3qONAad3N27wBwQ6jwecl19+WdnZ2Vq4cKE2bdqkESNGKDMzUyUlJc22X7dunW688UZ99NFHysvLU0pKiiZMmKCDBw+6tcvKylJRUZFr+fvf/+7prgDwcwVHqiUx/wYIBh4POE888YRmzpyp6dOna8iQIVq+fLmioqK0cuXKZtuvXr1ad9xxh0aOHKlBgwbpb3/7m5xOp3Jzc93aWSwW2Ww21xIbG+vprgDwc65TVN0IOECg82jAqa2tVX5+vjIyMk59oNmsjIwM5eXltWkf1dXVqqurU1xcnNv6devWqXv37ho4cKBuv/12HT16tMV91NTUyOFwuC0Ags++o6fm4AAIbB4NOEeOHFFDQ4MSExPd1icmJsput7dpH/fdd5+Sk5PdQlJWVpaef/555ebm6tFHH9XHH3+sa665Rg0NDc3uY8mSJYqOjnYtKSkp594pAH7pRF2DDpWfkCT1YQ4OEPBCvV1Aax555BG99NJLWrdunSIiIlzrp0yZ4vp62LBhGj58uPr166d169Zp/PjxZ+wnJydH2dnZrtcOh4OQAwSZfUcb5990jQhVXBcuEQcCnUdHcOLj4xUSEqLi4mK39cXFxbLZbK1uu3TpUj3yyCN6//33NXz48Fbb9u3bV/Hx8dq9e3ez71ssFlmtVrcFQHApODn/JpVLxIGg4NGAEx4ertGjR7tNEG6aMJyent7ido899pgWL16sNWvWaMyYMWf9nAMHDujo0aNKSkrqkLoBBB7X/BsmGANBweNXUWVnZ+uvf/2rnnvuOW3fvl233367qqqqNH36dEnStGnTlJOT42r/6KOP6v7779fKlSvVp08f2e122e12VVY2Pj+msrJS99xzjz7//HPt3btXubm5uu6669S/f39lZmZ6ujsA/NReJhgDQcXjc3AmT56sw4cPa8GCBbLb7Ro5cqTWrFnjmnhcWFgos/lUzvrzn/+s2tpa/exnP3Pbz8KFC/XAAw8oJCREX3/9tZ577jmVlZUpOTlZEyZM0OLFi2WxWDzdHQB+qsB1iTgTjIFgYDIMw/B2EZ3N4XAoOjpa5eXlzMcBgsQlD+fK7jih1+8Yp4t7cd8swB+15/c3z6ICEPCO1zbI7mi8RDyVOThAUCDgAAh4+0obT09FR4YplkvEgaBAwAEQ8PaefAYV82+A4EHAARDwCk+O4PTi9BQQNAg4AAJe012Me8cxggMECwIOgIBXWNoYcHpxigoIGgQcAAGPERwg+BBwAAS0uganDpYdlyT1Zg4OEDQIOAACWlHZCTU4DVlCzerelbudA8GCgAMgoDXdAyclLkpmM08RB4IFAQdAQGP+DRCcCDgAAhpXUAHBiYADIKDtO9p4iooRHCC4EHAABDTXKSquoAKCCgEHQMAyDINTVECQIuAACFhHKmtVXdsgk0nqGRvp7XIAdCICDoCA1TR6k2SNkCU0xMvVAOhMBBwAAevUU8Q5PQUEGwIOgIB16h44TDAGgg0BB0DAKjzKBGMgWBFwAASsfaVNl4gTcIBgQ8ABELA4RQUELwIOgIBUVVOvI5U1kjhFBQQjAg6AgLT/WOPoTUxUmKIjw7xcDYDORsABEJCaTk/14hlUQFAi4AAISIUEHCCoEXAABKR9J2/yxxVUQHAi4AAISFxBBQQ3Ag6AgMRTxIHgRsABEHDqG5w6eOy4JE5RAcGKgAMg4BwqO6F6p6HwULMSu0Z4uxwAXtApAWfZsmXq06ePIiIilJaWpo0bN7ba/tVXX9WgQYMUERGhYcOG6d1333V73zAMLViwQElJSYqMjFRGRoZ27drlyS4A8COu01NxUTKbTV6uBoA3eDzgvPzyy8rOztbChQu1adMmjRgxQpmZmSopKWm2/fr163XjjTdqxowZ2rx5syZNmqRJkyZp69atrjaPPfaYnnrqKS1fvlwbNmxQly5dlJmZqRMnTni6OwD8QNMVVFwiDgQvk2EYhic/IC0tTT/4wQ/0pz/9SZLkdDqVkpKiO++8U/Pnzz+j/eTJk1VVVaV//vOfrnWXXHKJRo4cqeXLl8swDCUnJ2vevHn6zW9+I0kqLy9XYmKiVq1apSlTppy1JofDoejoaJWXl8tqtXZQTwH4iiXvbtczn3ynW8b10QM/ucjb5QDoIO35/e3REZza2lrl5+crIyPj1AeazcrIyFBeXl6z2+Tl5bm1l6TMzExX+4KCAtntdrc20dHRSktLa3GfNTU1cjgcbguAwOW6RJwJxkDQ8mjAOXLkiBoaGpSYmOi2PjExUXa7vdlt7HZ7q+2b/mzPPpcsWaLo6GjXkpKSck79AeAf9pUScIBgFxRXUeXk5Ki8vNy17N+/39slAfAQwzBUeLRpDg43+QOClUcDTnx8vEJCQlRcXOy2vri4WDabrdltbDZbq+2b/mzPPi0Wi6xWq9sCIDAdrapVVW2DTCYpJS7S2+UA8BKPBpzw8HCNHj1aubm5rnVOp1O5ublKT09vdpv09HS39pK0du1aV/vU1FTZbDa3Ng6HQxs2bGhxnwCCR9Ml4knWCFlCQ7xcDQBvCfX0B2RnZ+vmm2/WmDFjNHbsWD355JOqqqrS9OnTJUnTpk1Tjx49tGTJEknSr3/9a11xxRX67//+b02cOFEvvfSSvvzyS/3lL3+RJJlMJs2dO1e///3vNWDAAKWmpur+++9XcnKyJk2a5OnuAPBxrqeIM/8GCGoeDziTJ0/W4cOHtWDBAtntdo0cOVJr1qxxTRIuLCyU2XxqIGncuHF68cUX9bvf/U6//e1vNWDAAL355psaOnSoq829996rqqoqzZo1S2VlZbrsssu0Zs0aRURwx1Ig2DVdQcU9cIDg5vH74Pgi7oMDBK7sV7bo9U0HdU/mQM2+qr+3ywHQgXzmPjgA0NkKGcEBIAIOgADDPXAASAQcAAGkurZehytqJEm9uQcOENQIOAACRtMl4tGRYYqOCvNyNQC8iYADIGAU8gwqACcRcAAEjKYRnBQmGANBj4ADIGC4niJOwAGCHgEHQMDgCioATQg4AAIGTxEH0ISAAyAg1Dc4deDYcUmM4AAg4AAIEEXlJ1TvNBQeYpbNynPpgGBHwAEQEPY2nZ7qFiWz2eTlagB4GwEHQEDYe/IKqj6cngIgAg6AALHvSOMITu9uTDAGQMABECAYwQFwOgIOgICw7+QcnD7xjOAAIOAACABOp+G6yV8fTlEBEAEHQACwO06ott6psBCTkqK5RBwAAQdAAGi6RDwlNkqhIfxYA0DAARAAXA/ZZIIxgJMIOAD8XtMIDpeIA2hCwAHg9/Yd4RJxAO4IOAD8nmsEh0vEAZxEwAHg1wzDcM3B4RJxAE0IOAD82uGKGh2va1CI2aQeMZHeLgeAjyDgAPBrTY9o6BETqfBQfqQBaMRPAwB+7dQVVEwwBnAKAQeAX3M9g4r5NwBOQ8AB4Nf2cpM/AM0g4ADwa4zgAGgOAQeA3zIM49RN/uIZwQFwikcDTmlpqaZOnSqr1aqYmBjNmDFDlZWVrba/8847NXDgQEVGRqpXr1666667VF5e7tbOZDKdsbz00kue7AoAH3S0qlYVNfUymaSesQQcAKeEenLnU6dOVVFRkdauXau6ujpNnz5ds2bN0osvvths+0OHDunQoUNaunSphgwZon379um2227ToUOH9Nprr7m1ffbZZ5WVleV6HRMT48muAPBBTaenkqMjFREW4uVqAPgSjwWc7du3a82aNfriiy80ZswYSdLTTz+ta6+9VkuXLlVycvIZ2wwdOlT/+7//63rdr18/PfTQQ7rppptUX1+v0NBT5cbExMhms3mqfAB+YO8RJhgDaJ7HTlHl5eUpJibGFW4kKSMjQ2azWRs2bGjzfsrLy2W1Wt3CjSTNnj1b8fHxGjt2rFauXCnDMFrcR01NjRwOh9sCwP/t4yniAFrgsREcu92u7t27u39YaKji4uJkt9vbtI8jR45o8eLFmjVrltv6Bx98UFdffbWioqL0/vvv64477lBlZaXuuuuuZvezZMkSLVq06Nw6AsBn7T3KU8QBNK/dIzjz589vdpLv6cuOHTvOuzCHw6GJEydqyJAheuCBB9zeu//++3XppZdq1KhRuu+++3Tvvffq8ccfb3FfOTk5Ki8vdy379+8/7/oAeB8jOABa0u4RnHnz5umWW25ptU3fvn1ls9lUUlLitr6+vl6lpaVnnTtTUVGhrKwsde3aVW+88YbCwsJabZ+WlqbFixerpqZGFovljPctFkuz6wH4N9cIDpeIA/iedgechIQEJSQknLVdenq6ysrKlJ+fr9GjR0uSPvzwQzmdTqWlpbW4ncPhUGZmpiwWi9566y1FRESc9bO2bNmi2NhYQgwQRMqqa1V+vE6S1CuOgAPAncfm4AwePFhZWVmaOXOmli9frrq6Os2ZM0dTpkxxXUF18OBBjR8/Xs8//7zGjh0rh8OhCRMmqLq6Wi+88ILbhOCEhASFhITo7bffVnFxsS655BJFRERo7dq1evjhh/Wb3/zGU10B4IOaRm8SrRZFhXv0jhcA/JBHfyqsXr1ac+bM0fjx42U2m3X99dfrqaeecr1fV1ennTt3qrq68QfVpk2bXFdY9e/f321fBQUF6tOnj8LCwrRs2TLdfffdMgxD/fv31xNPPKGZM2d6sisAfAzzbwC0xqMBJy4ursWb+klSnz593C7vvvLKK1u93FuSsrKy3G7wByA4Nd0DhyuoADSHZ1EB8Et7GcEB0AoCDgC/9N3hxufa9Y0n4AA4EwEHgN8xDEPfHW4cwembcIGXqwHgiwg4APzO4coa11PEeQ4VgOYQcAD4nabRm56xPEUcQPMIOAD8juv0VDynpwA0j4ADwO+4JhgnMMEYQPMIOAD8zndHmGAMoHUEHAB+p2kEpx+XiANoAQEHgF+prXdq/7HjkhjBAdAyAg4Av1JYWqUGp6Eu4SFKtFq8XQ4AH0XAAeBX9py8gio1oYtMJpOXqwHgqwg4APwKl4gDaAsCDgC/wiXiANqCgAPAr3CJOIC2IOAA8BuGYWgPTxEH0AYEHAB+42hVrcqq62QySf27M4IDoGUEHAB+Y1dx4+hNr7goHrIJoFUEHAB+Y3dJhSRpAKM3AM6CgAPAb+wqaRzB6d+9q5crAeDrCDgA/EbTKSpGcACcDQEHgN9oGsEZkEjAAdA6Ag4Av3CsqlZHKmskSf24Bw6AsyDgAPALu0/e/6ZHTKS6WEK9XA0AX0fAAeAXXPNvOD0FoA0IOAD8wi4uEQfQDgQcAH5hd9MEYy4RB9AGBBwAfqHpFFV/TlEBaAMCDgCf5zhRJ7vjhCSeQQWgbQg4AHzeruLG+Tc2a4SsEWFergaAPyDgAPB53xY1BpzBScy/AdA2Hg04paWlmjp1qqxWq2JiYjRjxgxVVla2us2VV14pk8nkttx2221ubQoLCzVx4kRFRUWpe/fuuueee1RfX+/JrgDwoh1FDknSoCSrlysB4C88eresqVOnqqioSGvXrlVdXZ2mT5+uWbNm6cUXX2x1u5kzZ+rBBx90vY6KinJ93dDQoIkTJ8pms2n9+vUqKirStGnTFBYWpocffthjfQHgPTvsjSM4g2yM4ABoG48FnO3bt2vNmjX64osvNGbMGEnS008/rWuvvVZLly5VcnJyi9tGRUXJZrM1+97777+vb7/9Vh988IESExM1cuRILV68WPfdd58eeOABhYeHe6Q/ALzD6TS00950iooRHABt47FTVHl5eYqJiXGFG0nKyMiQ2WzWhg0bWt129erVio+P19ChQ5WTk6Pq6mq3/Q4bNkyJiYmudZmZmXI4HNq2bVuz+6upqZHD4XBbAPiHg2XHVVlTr/AQs1Lju3i7HAB+wmMjOHa7Xd27d3f/sNBQxcXFyW63t7jdL37xC/Xu3VvJycn6+uuvdd9992nnzp16/fXXXfs9PdxIcr1uab9LlizRokWLzqc7ALxk+8n5N/27X6CwEK6LANA27Q448+fP16OPPtpqm+3bt59zQbNmzXJ9PWzYMCUlJWn8+PHas2eP+vXrd077zMnJUXZ2tuu1w+FQSkrKOdcIoPO45t9wBRWAdmh3wJk3b55uueWWVtv07dtXNptNJSUlbuvr6+tVWlra4vya5qSlpUmSdu/erX79+slms2njxo1ubYqLiyWpxf1aLBZZLJY2fyYA37HD3jiCM9jG/BsAbdfugJOQkKCEhISztktPT1dZWZny8/M1evRoSdKHH34op9PpCi1tsWXLFklSUlKSa78PPfSQSkpKXKfA1q5dK6vVqiFDhrSzNwB83Y4iRnAAtJ/HTmgPHjxYWVlZmjlzpjZu3KjPPvtMc+bM0ZQpU1xXUB08eFCDBg1yjcjs2bNHixcvVn5+vvbu3au33npL06ZN0+WXX67hw4dLkiZMmKAhQ4bol7/8pb766iu99957+t3vfqfZs2czSgMEmOO1DSo4WiVJGsQIDoB28OiMvdWrV2vQoEEaP368rr32Wl122WX6y1/+4nq/rq5OO3fudF0lFR4erg8++EATJkzQoEGDNG/ePF1//fV6++23XduEhITon//8p0JCQpSenq6bbrpJ06ZNc7tvDoDA8O/iChmGFH9BuBK68h8YAG1nMgzD8HYRnc3hcCg6Olrl5eWyWvlfIeCrXtpYqPmvf6NL+3fT6lsv8XY5ALysPb+/ueYSgM/65mC5JGlocrSXKwHgbwg4AHzW1qaA04OAA6B9CDgAfFJtvVPbT94DZ3hPAg6A9iHgAPBJ/y6uUG29U9aIUPWKizr7BgBwGgIOAJ/UdHpqWM9omUwmL1cDwN8QcAD4pG+YfwPgPBBwAPikpoAzjIAD4BwQcAD4nNp6p+sRDcN7xHi3GAB+iYADwOf8u7hCtQ1ORUeGKSUu0tvlAPBDBBwAPufU/BsrE4wBnBMCDgCfc2r+TYx3CwHgtwg4AHzOlsIySdIIbvAH4BwRcAD4lMqaeu2wOyRJF/eO9XI1APwVAQeAT/lqf5mchtQzNlKJ1ghvlwPATxFwAPiU/H3HJEkX92L0BsC5I+AA8ClNAWc0p6cAnAcCDgCf4XQa2lRIwAFw/gg4AHzG7sOVqjhRr8iwEA2ydfV2OQD8GAEHgM9oOj01MiVGoSH8eAJw7vgJAsBnuCYY947xbiEA/B4BB4DP2MQEYwAdhIADwCcUO07ouyNVMpm4RBzA+SPgAPAJeXuOSpIuSrYqJircy9UA8HcEHAA+Yf2eI5Kkcf3ivVwJgEBAwAHgE9afHMFJ79fNy5UACAQEHABet7+0WgeOHVeo2aSxfeK8XQ6AAEDAAeB1TaenRqbEqIsl1MvVAAgEBBwAXtd0emocp6cAdBACDgCvMgzjtPk3TDAG0DEIOAC8aldJpQ5X1MgSataoXjHeLgdAgCDgAPCqj3aUSJLS+nZTRFiIl6sBECg8GnBKS0s1depUWa1WxcTEaMaMGaqsrGyx/d69e2UymZpdXn31VVe75t5/6aWXPNkVAB6SezLgZAzu7uVKAAQSj16uMHXqVBUVFWnt2rWqq6vT9OnTNWvWLL344ovNtk9JSVFRUZHbur/85S96/PHHdc0117itf/bZZ5WVleV6HRMT0+H1A/Cssupa1wM2rxpIwAHQcTwWcLZv3641a9boiy++0JgxYyRJTz/9tK699lotXbpUycnJZ2wTEhIim83mtu6NN97Qz3/+c11wwQVu62NiYs5oC8C/fPzvw2pwGhqY2FUpcVHeLgdAAPHYKaq8vDzFxMS4wo0kZWRkyGw2a8OGDW3aR35+vrZs2aIZM2ac8d7s2bMVHx+vsWPHauXKlTIMo8X91NTUyOFwuC0AvC93e+Ppqas5PQWgg3lsBMdut6t7d/cfWqGhoYqLi5Pdbm/TPlasWKHBgwdr3LhxbusffPBBXX311YqKitL777+vO+64Q5WVlbrrrrua3c+SJUu0aNGic+sIAI+ob3Bq3c7GgDN+EAEHQMdq9wjO/PnzW5wI3LTs2LHjvAs7fvy4XnzxxWZHb+6//35deumlGjVqlO677z7de++9evzxx1vcV05OjsrLy13L/v37z7s+AOcnf98xOU7UKzYqTKN6xXq7HAABpt0jOPPmzdMtt9zSapu+ffvKZrOppKTEbX19fb1KS0vbNHfmtddeU3V1taZNm3bWtmlpaVq8eLFqampksVjOeN9isTS7HoD3rNnWOJJ71cDuCjGbvFwNgEDT7oCTkJCghISEs7ZLT09XWVmZ8vPzNXr0aEnShx9+KKfTqbS0tLNuv2LFCv3kJz9p02dt2bJFsbGxhBjATzQ4Db3zdeMVk9cOS/JyNQACkcfm4AwePFhZWVmaOXOmli9frrq6Os2ZM0dTpkxxXUF18OBBjR8/Xs8//7zGjh3r2nb37t365JNP9O67756x37ffflvFxcW65JJLFBERobVr1+rhhx/Wb37zG091BUAH21BwVCUVNbJGhOryC8/+nxgAaC+P3gdn9erVmjNnjsaPHy+z2azrr79eTz31lOv9uro67dy5U9XV1W7brVy5Uj179tSECRPO2GdYWJiWLVumu+++W4ZhqH///nriiSc0c+ZMT3YFQAd6+6tDkqRrhiYpPJQbqgPoeCajteurA5TD4VB0dLTKy8tltVq9XQ4QVGrrnRr78Acqq67T6lvTdGl/HrAJoG3a8/ub/zoB6FSf7j6ssuo6JXS16JK+3bxdDoAARcAB0Kn+saXx9NTEYUlcPQXAYwg4ADrNsapa/d/WxsvDJ43q4eVqAAQyAg6ATvO/mw6ott6pi5KtGtEz2tvlAAhgBBwAncLpNLR6Q6Ek6aZLestk4vQUAM8h4ADoFHnfHVXBkSpdYAnVT0Yke7scAAGOgAOgU6zesE+S9NNRPdTF4tFbcAEAAQeA5+0vrdZ724olSb9I6+XlagAEAwIOAI975pM9anAauqx/vAYncXNNAJ5HwAHgUSWOE3rlywOSpNlX9fdyNQCCBQEHgEf99V/fqbbeqTG9Y3VJ3zhvlwMgSBBwAHhMaVWt69Lw2Vf359JwAJ2GgAPAY57K3aXq2gYN7WHVlRcmeLscAEGEgAPAI3aXVOh/Pm+8NPy31w5m9AZApyLgAPCIxf/crganoQlDEjWuX7y3ywEQZAg4ADrchzuK9fG/DyssxKTfXjvY2+UACEIEHAAdqvx4nX77+lZJ0vRLU9UnvouXKwIQjAg4ADrUore3ye44odT4Lro740JvlwMgSBFwAHSY97bZ9fqmgzKbpKU3DFdkeIi3SwIQpAg4ADpEwZEq3fPqV5KkWZf30+je3NQPgPcQcACct4oTdZr5/JdynKjXqF4xuvs/Bni7JABBjoAD4LzUNTj165e2aHdJpWzWCD1z02hZQjk1BcC7CDgAzll9g1N3v7xFH+4oUXioWc/8crS6WyO8XRYAEHAAnJsGp6F7Xvta//y6SGEhJv2/X1ysESkx3i4LACRJod4uAID/qThRp7v+vlkf7TysULNJf/rFxcoYkujtsgDAhYADoF32l1br1ue+1M7iCkWEmfXHKaOUeZHN22UBgBsCDoA2MQxDr+Uf0KK3v1VlTb0Sulr0t2ljOC0FwCcRcACc1d4jVfr9O9/qg+0lkqQxvWP11I2jlBwT6eXKAKB5BBwALTpcUaO//us7PftZgeoaDIWFmHT3f1yoX13eTyFmk7fLA4AWEXAAnGF3SYX+J2+fXvpiv2rqnZKkyy9M0IIfDVb/7l29XB0AnB0BB4AkqcRxQu99W6x/bD6oL/cdc60fkRKjueMH6MqBCTKZGLUB4B88dh+chx56SOPGjVNUVJRiYmLatI1hGFqwYIGSkpIUGRmpjIwM7dq1y61NaWmppk6dKqvVqpiYGM2YMUOVlZUe6AEQ2Mqr6/TpriN6/L0dum7ZZ0pbkqv739yqL/cdU4jZpP8YkqjVt6bpzTvG6apB3Qk3APyKx0ZwamtrdcMNNyg9PV0rVqxo0zaPPfaYnnrqKT333HNKTU3V/fffr8zMTH377beKiGi8O+rUqVNVVFSktWvXqq6uTtOnT9esWbP04osveqorgF+rqqnX/mPVKjxard2HK7X1YLm+OViu/aXHz2g7MiVG1wy16aejenBHYgB+zWQYhuHJD1i1apXmzp2rsrKyVtsZhqHk5GTNmzdPv/nNbyRJ5eXlSkxM1KpVqzRlyhRt375dQ4YM0RdffKExY8ZIktasWaNrr71WBw4cUHJycptqcjgcio6OVnl5uaxW63n1D+gstfVOHa9r0PHaBh2va1B1bb1O1DWo/HidjlbWqrSqcTna9GdljQ6WHdeRytoW95kSF6nRvWJ12YAEXdY/XrZoQg0A39We398+MwenoKBAdrtdGRkZrnXR0dFKS0tTXl6epkyZory8PMXExLjCjSRlZGTIbDZrw4YN+ulPf9rsvmtqalRTU+N67XA4PNKH/H2levuroja1bSlXtpQ2m2tutNC6+bZt329LW7TU9nxra/7TWmvb9p203O9m+tdi2zZ/XMvH1Wh8tEG901CD06l6p6H6BuPkOudp753+p1M1dU5XoKl3nvv/RaIjw9S7W5R6d+uii5KtGpocraE9rIqJCj/nfQKAL/OZgGO32yVJiYnut3tPTEx0vWe329W9e3e390NDQxUXF+dq05wlS5Zo0aJFHVzxmXbaK7Vq/V6Pfw6CW4jZpKiwEEWGNy5dI0IV18Wibl3CFXdyafo6OSZSKbFRio4K83bZANCp2hVw5s+fr0cffbTVNtu3b9egQYPOq6iOlpOTo+zsbNdrh8OhlJSUDv+ci5KtmnNV/2bfa25+ZotTNluYzNnc2pbmfZqaad1y27aX0Z6Jpu2prcXP64B9t2dubEv9a9ffUQttQ0PMCjWbFGI2KTTEpBDzaa9df5pd75tNJllCzYo6GWSiwkIVGR6isBATE34B4CzaFXDmzZunW265pdU2ffv2PadCbLbGZ9kUFxcrKSnJtb64uFgjR450tSkpKXHbrr6+XqWlpa7tm2OxWGSxWM6prvYYkRLDbesBAPAB7Qo4CQkJSkhI8Eghqampstlsys3NdQUah8OhDRs26Pbbb5ckpaenq6ysTPn5+Ro9erQk6cMPP5TT6VRaWppH6gIAAP7HY/fBKSws1JYtW1RYWKiGhgZt2bJFW7ZscbtnzaBBg/TGG29Iajw1MHfuXP3+97/XW2+9pW+++UbTpk1TcnKyJk2aJEkaPHiwsrKyNHPmTG3cuFGfffaZ5syZoylTprT5CioAABD4PDbJeMGCBXruuedcr0eNGiVJ+uijj3TllVdKknbu3Kny8nJXm3vvvVdVVVWaNWuWysrKdNlll2nNmjWue+BI0urVqzVnzhyNHz9eZrNZ119/vZ566ilPdQMAAPghj98HxxdxHxwAAPxPe35/e+wUFQAAgLcQcAAAQMAh4AAAgIBDwAEAAAGHgAMAAAIOAQcAAAQcAg4AAAg4BBwAABBwCDgAACDgeOxRDb6s6ebNDofDy5UAAIC2avq93ZaHMARlwKmoqJAkpaSkeLkSAADQXhUVFYqOjm61TVA+i8rpdOrQoUPq2rWrTCZTh+7b4XAoJSVF+/fvD8jnXAV6/yT6GAgCvX8SfQwEgd4/qeP7aBiGKioqlJycLLO59Vk2QTmCYzab1bNnT49+htVqDdh/sFLg90+ij4Eg0Psn0cdAEOj9kzq2j2cbuWnCJGMAABBwCDgAACDgEHA6mMVi0cKFC2WxWLxdikcEev8k+hgIAr1/En0MBIHeP8m7fQzKScYAACCwMYIDAAACDgEHAAAEHAIOAAAIOAQcAAAQcAg47fTQQw9p3LhxioqKUkxMTLNtCgsLNXHiREVFRal79+665557VF9f3+p+S0tLNXXqVFmtVsXExGjGjBmqrKz0QA/aZ926dTKZTM0uX3zxRYvbXXnllWe0v+222zqx8vbp06fPGfU+8sgjrW5z4sQJzZ49W926ddMFF1yg66+/XsXFxZ1Ucdvt3btXM2bMUGpqqiIjI9WvXz8tXLhQtbW1rW7n68dw2bJl6tOnjyIiIpSWlqaNGze22v7VV1/VoEGDFBERoWHDhundd9/tpErbb8mSJfrBD36grl27qnv37po0aZJ27tzZ6jarVq0643hFRER0UsXt98ADD5xR76BBg1rdxp+OYXM/U0wmk2bPnt1se384fp988ol+/OMfKzk5WSaTSW+++abb+4ZhaMGCBUpKSlJkZKQyMjK0a9eus+63vd/LbUXAaafa2lrdcMMNuv3225t9v6GhQRMnTlRtba3Wr1+v5557TqtWrdKCBQta3e/UqVO1bds2rV27Vv/85z/1ySefaNasWZ7oQruMGzdORUVFbsutt96q1NRUjRkzptVtZ86c6bbdY4891klVn5sHH3zQrd4777yz1fZ333233n77bb366qv6+OOPdejQIf3nf/5nJ1Xbdjt27JDT6dQzzzyjbdu26Q9/+IOWL1+u3/72t2fd1leP4csvv6zs7GwtXLhQmzZt0ogRI5SZmamSkpJm269fv1433nijZsyYoc2bN2vSpEmaNGmStm7d2smVt83HH3+s2bNn6/PPP9fatWtVV1enCRMmqKqqqtXtrFar2/Hat29fJ1V8bi666CK3ej/99NMW2/rbMfziiy/c+rZ27VpJ0g033NDiNr5+/KqqqjRixAgtW7as2fcfe+wxPfXUU1q+fLk2bNigLl26KDMzUydOnGhxn+39Xm4XA+fk2WefNaKjo89Y/+677xpms9mw2+2udX/+858Nq9Vq1NTUNLuvb7/91pBkfPHFF651//d//2eYTCbj4MGDHV77+aitrTUSEhKMBx98sNV2V1xxhfHrX/+6c4rqAL179zb+8Ic/tLl9WVmZERYWZrz66quuddu3bzckGXl5eR6osGM99thjRmpqaqttfPkYjh071pg9e7brdUNDg5GcnGwsWbKk2fY///nPjYkTJ7qtS0tLM371q195tM6OUlJSYkgyPv744xbbtPQzyVctXLjQGDFiRJvb+/sx/PWvf23069fPcDqdzb7vb8dPkvHGG2+4XjudTsNmsxmPP/64a11ZWZlhsViMv//97y3up73fy+3BCE4Hy8vL07Bhw5SYmOhal5mZKYfDoW3btrW4TUxMjNuISEZGhsxmszZs2ODxmtvjrbfe0tGjRzV9+vSztl29erXi4+M1dOhQ5eTkqLq6uhMqPHePPPKIunXrplGjRunxxx9v9bRifn6+6urqlJGR4Vo3aNAg9erVS3l5eZ1R7nkpLy9XXFzcWdv54jGsra1Vfn6+29+92WxWRkZGi3/3eXl5bu2lxu9LfzhWUuPxknTWY1ZZWanevXsrJSVF1113XYs/c3zFrl27lJycrL59+2rq1KkqLCxssa0/H8Pa2lq98MIL+q//+q9WH/Dsb8fvdAUFBbLb7W7HKDo6WmlpaS0eo3P5Xm6PoHzYpifZ7Xa3cCPJ9dput7e4Tffu3d3WhYaGKi4ursVtvGXFihXKzMw868NKf/GLX6h3795KTk7W119/rfvuu087d+7U66+/3kmVts9dd92liy++WHFxcVq/fr1ycnJUVFSkJ554otn2drtd4eHhZ8zDSkxM9Llj9n27d+/W008/raVLl7bazleP4ZEjR9TQ0NDs99mOHTua3aal70tfP1aS5HQ6NXfuXF166aUaOnRoi+0GDhyolStXavjw4SovL9fSpUs1btw4bdu2zeMPFz4XaWlpWrVqlQYOHKiioiItWrRIP/zhD7V161Z17dr1jPb+fAzffPNNlZWV6ZZbbmmxjb8dv+9rOg7tOUbn8r3cHgQcSfPnz9ejjz7aapvt27efdQKcPzmXPh84cEDvvfeeXnnllbPu//T5Q8OGDVNSUpLGjx+vPXv2qF+/fudeeDu0p4/Z2dmudcOHD1d4eLh+9atfacmSJT57G/VzOYYHDx5UVlaWbrjhBs2cObPVbX3hGEKaPXu2tm7d2ur8FElKT09Xenq66/W4ceM0ePBgPfPMM1q8eLGny2y3a665xvX18OHDlZaWpt69e+uVV17RjBkzvFhZx1uxYoWuueYaJScnt9jG346fPyDgSJo3b16ryVqS+vbt26Z92Wy2M2aAN11ZY7PZWtzm+xOq6uvrVVpa2uI25+tc+vzss8+qW7du+slPftLuz0tLS5PUOHrQWb8cz+e4pqWlqb6+Xnv37tXAgQPPeN9ms6m2tlZlZWVuozjFxcUeO2bf197+HTp0SFdddZXGjRunv/zlL+3+PG8cw+bEx8crJCTkjCvWWvu7t9ls7WrvK+bMmeO66KC9/4sPCwvTqFGjtHv3bg9V17FiYmJ04YUXtlivvx7Dffv26YMPPmj3yKe/Hb+m41BcXKykpCTX+uLiYo0cObLZbc7le7ldznsWT5A62yTj4uJi17pnnnnGsFqtxokTJ5rdV9Mk4y+//NK17r333vOpScZOp9NITU015s2bd07bf/rpp4Yk46uvvurgyjzjhRdeMMxms1FaWtrs+02TjF977TXXuh07dvjsJOMDBw4YAwYMMKZMmWLU19ef0z586RiOHTvWmDNnjut1Q0OD0aNHj1YnGf/oRz9yW5eenu6zE1SdTqcxe/ZsIzk52fj3v/99Tvuor683Bg4caNx9990dXJ1nVFRUGLGxscYf//jHZt/3t2PYZOHChYbNZjPq6uratZ2vHz+1MMl46dKlrnXl5eVtmmTcnu/ldtV43nsIMvv27TM2b95sLFq0yLjggguMzZs3G5s3bzYqKioMw2j8Rzl06FBjwoQJxpYtW4w1a9YYCQkJRk5OjmsfGzZsMAYOHGgcOHDAtS4rK8sYNWqUsWHDBuPTTz81BgwYYNx4442d3r+WfPDBB4YkY/v27We8d+DAAWPgwIHGhg0bDMMwjN27dxsPPvig8eWXXxoFBQXGP/7xD6Nv377G5Zdf3tllt8n69euNP/zhD8aWLVuMPXv2GC+88IKRkJBgTJs2zdXm+300DMO47bbbjF69ehkffvih8eWXXxrp6elGenq6N7rQqgMHDhj9+/c3xo8fbxw4cMAoKipyLae38adj+NJLLxkWi8VYtWqV8e233xqzZs0yYmJiXFcv/vKXvzTmz5/vav/ZZ58ZoaGhxtKlS43t27cbCxcuNMLCwoxvvvnGW11o1e23325ER0cb69atczte1dXVrjbf7+OiRYuM9957z9izZ4+Rn59vTJkyxYiIiDC2bdvmjS6c1bx584x169YZBQUFxmeffWZkZGQY8fHxRklJiWEY/n8MDaPxl3WvXr2M++6774z3/PH4VVRUuH7nSTKeeOIJY/Pmzca+ffsMwzCMRx55xIiJiTH+8Y9/GF9//bVx3XXXGampqcbx48dd+7j66quNp59+2vX6bN/L54OA004333yzIemM5aOPPnK12bt3r3HNNdcYkZGRRnx8vDFv3jy39P7RRx8ZkoyCggLXuqNHjxo33nijccEFFxhWq9WYPn26KzT5ghtvvNEYN25cs+8VFBS4/R0UFhYal19+uREXF2dYLBajf//+xj333GOUl5d3YsVtl5+fb6SlpRnR0dFGRESEMXjwYOPhhx92G3H7fh8NwzCOHz9u3HHHHUZsbKwRFRVl/PSnP3ULDb7i2Wefbfbf7OkDuP54DJ9++mmjV69eRnh4uDF27Fjj888/d713xRVXGDfffLNb+1deecW48MILjfDwcOOiiy4y3nnnnU6uuO1aOl7PPvusq833+zh37lzX30diYqJx7bXXGps2ber84tto8uTJRlJSkhEeHm706NHDmDx5srF7927X+/5+DA2jcSRekrFz584z3vPH49f0u+v7S1M/nE6ncf/99xuJiYmGxWIxxo8ff0bfe/fubSxcuNBtXWvfy+fDZBiGcf4nugAAAHwH98EBAAABh4ADAAACDgEHAAAEHAIOAAAIOAQcAAAQcAg4AAAg4BBwAABAwCHgAACAgEPAAQAAAYeAAwAAAg4BBwAABBwCDgAACDj/HxueHWVsXhMUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "# Define some temperature values\n",
    "temps = np.arange(-10, 10, .1)\n",
    "\n",
    "# Plot the tanh of the values\n",
    "plt.plot(temps, np.tanh(temps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\tanh$ outputs values between `-1` and `1`.  It also has a very steep slope before and after `0` on the x-axis.  The tanh activation function is better for our purposes than sigmoid because it enables us to have negative values.  This enables the gradient to be steeper, which aids in gradient descent.\n",
    "\n",
    "Let's take a look at the derivative of tanh using sympy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{\\left(- e^{x} + e^{- x}\\right) \\left(e^{x} - e^{- x}\\right)}{\\left(e^{x} + e^{- x}\\right)^{2}} + 1$"
      ],
      "text/plain": [
       "(-exp(x) + exp(-x))*(exp(x) - exp(-x))/(exp(x) + exp(-x))**2 + 1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sympy import diff, symbols, exp\n",
    "\n",
    "x = symbols(\"x\")\n",
    "sympy_tanh = (exp(x) - exp(-x)) / (exp(x) + exp(-x))\n",
    "diff(sympy_tanh, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sympy doesn't quite simplify the derivative all the way.  Like the derivatives of the sigmoid and softmax functions, we can express the derivative of $\\tanh{x}$ in terms of the output - $1 - \\tanh^2{x}$.  We can then graph this derivative:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x12e54cad0>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/S0lEQVR4nO3de3xU9Z3/8fdMLpMLuRBCEoKRq4KIXARJg7XWNZWqpXV7War+issqrRa6VtyuYitU24pab1uXFmtFffysK+qvtd1qscqKLoIXblURVOQOuRJyJ5lk5vv7Y3ImiSSQSWbmzJm8no/HPCJnzjnzOTMmeed7Oy5jjBEAAIBN3HYXAAAABjfCCAAAsBVhBAAA2IowAgAAbEUYAQAAtiKMAAAAWxFGAACArQgjAADAVol2F9AXfr9fR44cUUZGhlwul93lAACAPjDGqKGhQYWFhXK7e2//cEQYOXLkiIqKiuwuAwAA9MPBgwd12mmn9fq8I8JIRkaGpMDFZGZm2lwNAADoi/r6ehUVFQV/j/fGEWHE6prJzMwkjAAA4DCnGmLBAFYAAGArwggAALAVYQQAANiKMAIAAGxFGAEAALYijAAAAFsRRgAAgK0IIwAAwFaEEQAAYKuQw8gbb7yhuXPnqrCwUC6XSy+88MIpj1m/fr3OPfdceTwejR8/Xk888UQ/SgUAAPEo5DDS1NSkqVOnauXKlX3af+/evbr88st10UUXafv27frhD3+o6667Ti+//HLIxQIAgPgT8r1pLr30Ul166aV93n/VqlUaM2aM7r//fknSWWedpQ0bNujBBx/UnDlzQn15AAAQZyI+ZmTTpk0qLS3ttm3OnDnatGlTr8e0traqvr6+2wNAfPL7jZ56a782fFJtdykAbBLxMFJeXq78/Pxu2/Lz81VfX6/jx4/3eMyKFSuUlZUVfBQVFUW6TAA2+b9v7ddPXvhA//Lku/qkosHucgDYICZn0yxdulR1dXXBx8GDB+0uCUAE7K1u0oq/7pQkedv9uvm5v6vN57e5KgDRFvEwUlBQoIqKim7bKioqlJmZqdTU1B6P8Xg8yszM7PYAEF98fqObn92ulja/ZowaqsyURL13qE6/Wf+p3aUBiLKIh5GSkhKtW7eu27ZXXnlFJSUlkX5pADHsj9sOa+uBWg3xJOpXV07XnV+bLEn61bpPVFHfYnN1AKIp5DDS2Nio7du3a/v27ZICU3e3b9+uAwcOSAp0scyfPz+4//XXX689e/bo3//937Vr1y79+te/1rPPPqubbropPFcAwJH+95MqSdKC80drZHaqvjatUGcXZqrdb/TWnqM2VwcgmkIOI5s3b9b06dM1ffp0SdKSJUs0ffp0LVu2TJJUVlYWDCaSNGbMGL344ot65ZVXNHXqVN1///363e9+x7ReYJDbvO+YJKl4zDBJksvl0qwxOd2eAzA4hLzOyBe/+EUZY3p9vqfVVb/4xS9q27Ztob4UgDhVVndch2uPK8Ht0rTTs4Pbzxudo8ff3KfN+wkjwGASk7NpAMQ3q+XjrBEZGuLp/Jto5qihkqRd5fWqb2mzpTYA0UcYARB1WzpaPmaOyum2PS8zRafnpMkYaduBWhsqA2AHwgiAqHt3X40kaebooSc8Z23b3LEPgPhHGAEQVY2t7dpZFrjFw2dbRrpuYxArMHgQRgBE1fYDtfIb6bShqSrISjnh+fM6Wka2H6xlNVZgkCCMAIiqYBfNqBO7aCRp3PAhykpN0vE2nz48wk0ygcGAMAIgqrYfrJUkzegljLjdruBz1r4A4hthBEBU7alulCRNKOj9nlNn5mcE9q1qjEpNAOxFGAEQNa3tPh0+dlySNCY3vdf9xnY8t6e6KSp1AbAXYQRA1BysaZbfSEM8icodktzrfqM7wsi+o4QRYDAgjACImj1VgXAxJjddLper1/2sVpNDx46rtd0XldoA2IcwAiBq9lZ3hpGTyR2SrAxPooyRDhxtjkZpAGxEGAEQNVa3y+hThBGXyxXcZy/jRoC4RxgBEDVWN83YU4QRqbP1hDACxD/CCICosYLFqVpGuu5DGAHiH2EEQFQ0tbarsqFVkjRm2KnDyFjCCDBoEEYARIUVKoalJysrLemU+9NNAwwehBEAURFKF03X/SobWtXY2h6xugDYjzACICr29XFaryUrNUnD0pO7HQsgPhFGAERFX9cY6YquGmBwIIwAiIo9/QgjzKgBBgfCCICosBY8o2UEwGcRRgBEXENLm2qb2yRJRTlpfT7u9I59Dx1jSXggnhFGAERcWV2LJCkzJVFDPIl9Pq4wO0WSdKS2JSJ1AYgNhBEAEXek9rgkqTA7NaTjRmQF9q+ob5HPb8JeF4DYQBgBEHFWy8iIrJSQjsvL8Mjtktr9RtWNrZEoDUAMIIwAiLiyjpaRESG2jCQmuJWfaXXVHA97XQBiA2EEQMQd6WgZKQyxZUTqbE2xWlcAxB/CCICIK6vraBnJCq1lROpsTSGMAPGLMAIg4so6ZsOMyA69ZcRqTSmjmwaIW4QRABFljNGRjpaRwv60jGTRMgLEO8IIgIiqbW5TS5tfklTQjzEjwbVG6mgZAeIVYQRARFkhYlh6slKSEkI+PtgywsJnQNwijACIqIGMF+l6XGVDi9p9/rDVBSB2EEYARNRAZtJIUm66R0kJLvmNVNHAwmdAPCKMAIiogawxIklutys41oQZNUB8IowAiKj+rr7aldWqcoQZNUBcIowAiKgj/bwvTVesNQLEN8IIgIiyxoyEesferliFFYhvhBEAEeP3G5WHsWWEm+UB8YkwAiBiqpta1eYzcrkUvPtuf7AKKxDfCCMAIsZaYyQvw6OkhP7/uLHWGiljFVYgLhFGAERMeX0gjBQMoFWk6/HVjV5521n4DIg3hBEAEVPVsUjZ8IyBhZGhaclKdLskSdWNLHwGxBvCCICIqewII3mZngGdx+12aXiGp9s5AcQPwgiAiKlq6BwzMlDWOSrrGcQKxBvCCICIqazvaBkZYDeN1NnVQ8sIEH8IIwAiJthNE46WkUy6aYB4RRgBEDGVVjfNAMeMSJ2BpoowAsQdwgiAiPD5jaobvZLC001jncMahwIgfhBGAERETZNXPn9g9dXcIckDPl8es2mAuEUYARARVhfNsPRkJQ5g9VVLcMxIPWEEiDeEEQARURmmBc8sVjdNdWOr/H4TlnMCiA2EEQARUVUfvpk0UqCrx+WS2v1GNc3esJwTQGwgjACIiMowLngmSYkJbg1LD4w9oasGiC+EEQAREa6l4LvqXPiMGTVAPCGMAIiIcK6+amFGDRCf+hVGVq5cqdGjRyslJUXFxcV65513Trr/Qw89pAkTJig1NVVFRUW66aab1NLCXzZAPAt3N03Xc7HwGRBfQg4ja9as0ZIlS7R8+XJt3bpVU6dO1Zw5c1RZWdnj/k8//bRuvfVWLV++XDt37tRjjz2mNWvW6Lbbbhtw8QBiVyS6aTqn9/LHDBBPQg4jDzzwgBYuXKgFCxZo0qRJWrVqldLS0rR69eoe99+4caPOP/98XXXVVRo9erQuueQSXXnlladsTQHgXMaYYOtFeLtpuFkeEI9CCiNer1dbtmxRaWlp5wncbpWWlmrTpk09HjN79mxt2bIlGD727Nmjl156SZdddlmvr9Pa2qr6+vpuDwDOUd/SrtZ2vyRpeBi7aYYzZgSIS4mh7FxdXS2fz6f8/Pxu2/Pz87Vr164ej7nqqqtUXV2tz3/+8zLGqL29Xddff/1Ju2lWrFihO+64I5TSAMQQ6/4xGSmJSklKCNt5Owew0k0DxJOIz6ZZv3697rrrLv3617/W1q1b9Yc//EEvvviifvazn/V6zNKlS1VXVxd8HDx4MNJlAgijyjAveGYJdtPUt8oYVmEF4kVILSO5ublKSEhQRUVFt+0VFRUqKCjo8Zjbb79d3/nOd3TddddJks455xw1NTXpu9/9rn784x/L7T4xD3k8Hnk84f0hBiB6KiMwXkTqHMDa2u5XfUu7slKTwnp+APYIqWUkOTlZM2bM0Lp164Lb/H6/1q1bp5KSkh6PaW5uPiFwJCQEmm35ywaIT8FpvWGcSSNJKUkJykgJ/A1VRVcNEDdCahmRpCVLluiaa67RzJkzNWvWLD300ENqamrSggULJEnz58/XyJEjtWLFCknS3Llz9cADD2j69OkqLi7W7t27dfvtt2vu3LnBUAIgvnTOpAl/C2dehkcNLe2qbGjV+LyMsJ8fQPSFHEbmzZunqqoqLVu2TOXl5Zo2bZrWrl0bHNR64MCBbi0hP/nJT+RyufSTn/xEhw8f1vDhwzV37lz94he/CN9VAIgp1Y2BG9nlDgl/GMkd4tGnVU3B1wDgfCGHEUlavHixFi9e3ONz69ev7/4CiYlavny5li9f3p+XAuBA1Y2BlpGIhBFWYQXiDvemARB2VqvFsCHJYT/38I6AYwUeAM5HGAEQdhFtGekIONW0jABxgzACIKz8fqOapkDLSDhXX7Xk0jICxB3CCICwOtbslc8fmLafkx7+bprOMMIAViBeEEYAhJUVErLTkpSUEP4fMdYAVlpGgPhBGAEQVkcjOF4kcN7kjtfxsnAiECcIIwDCqioYRsLfRRM4byDkeH1+1R9vj8hrAIguwgiAsIrkgmfSZ5aEp6sGiAuEEQBhFclpvRbWGgHiC2EEQFgdjXA3TeDchBEgnhBGAIRVpLtpJCk3g4XPgHhCGAEQVtHopmGtESC+EEYAhJXVWhGJ+9JY6KYB4gthBEDYGGNU3RSFbhrCCBBXCCMAwqahtV3edr+kyNyXxmINjq2imwaIC4QRAGFjddEM8SQqJSkhYq8TXBKeAaxAXCCMAAgba0BpJMeLSN3XGWFJeMD5CCMAwibS96WxWOdvbfersZUl4QGnI4wACJvqKCx4JkmpyQlKTw50A1XRVQM4HmEEQNhURWHBM0tw3AiDWAHHI4wACBurZWRYNMII03uBuEEYARA21piR4RHuppE6u4III4DzEUYAhE007ktjCbaMMGYEcDzCCICwCQ5gjeCCZxYrjLDwGeB8hBEAYWO1UkR3ACstI4DTEUYAhMVxr09NXp+kyC96JnWOSyGMAM5HGAEQFlYoSE50K8OTGPHXYzYNED8IIwDCojo4k8Yjl8sV8dcbHrw/DWNGAKcjjAAIi86ZNJHvogm8TiCMHG/zqYkl4QFHI4wACIujUVzwTJLSPYlK7bgzMF01gLMRRgCERbTuS9NVbgaDWIF4QBgBEBbRXPDMElxrhHEjgKMRRgCERVVj9NYYsTCjBogPhBEAYdE5ZiSK3TSEESAuEEYAhIXVTTM8ii0jLHwGxAfCCICwiOZ9aSy5rDUCxAXCCIABa/P5VdvcJokxIwBCRxgBMGA1TYGWiQS3S9mpSVF73c479xJGACcjjAAYsKqOu/XmpCfL7Y78UvAWa00T627BAJyJMAJgwKptmNYrdY4ZafL6dLzjjsEAnIcwAmDAon1fGkuGJ1HJie6OGmgdAZyKMAJgwI7a1DLicrmCU4kZNwI4F2EEwIDZcV8aC+NGAOcjjAAYMDvuS2PpnN7LWiOAUxFGAAyYXQNYu74mY0YA5yKMABgwq1UimvelseRmsCQ84HSEEQADZmfLyHBaRgDHI4wAGBC/3wRXYB0exfvSWLg/DeB8hBEAA3Ks2Suf30gKrMAabYwZAZyPMAJgQKzxIkPTkpSUEP0fKdyfBnA+wgiAAbHuS2NHF43UOWakoaVdLW0sCQ84EWEEwIDYOXhVkjJTE5Xc0SJztIlxI4ATEUYADIjdYcTlcgWnFLMKK+BMhBEAA2J3N43EIFbA6QgjAAakyuaWkcBrs/AZ4GSEEQAD0nlfmuhP67VwfxrA2foVRlauXKnRo0crJSVFxcXFeuedd066f21trRYtWqQRI0bI4/HozDPP1EsvvdSvggHElpjopul47SrGjACOlBjqAWvWrNGSJUu0atUqFRcX66GHHtKcOXP00UcfKS8v74T9vV6vvvSlLykvL0/PP/+8Ro4cqf379ys7Ozsc9QOwmd0DWLu+NmuNAM4Uchh54IEHtHDhQi1YsECStGrVKr344otavXq1br311hP2X716tWpqarRx40YlJSVJkkaPHj2wqgHEBJ/NS8FbcplNAzhaSN00Xq9XW7ZsUWlpaecJ3G6VlpZq06ZNPR7z5z//WSUlJVq0aJHy8/M1efJk3XXXXfL5el+cqLW1VfX19d0eAGKPtRS8y2XPUvAWbpYHOFtIYaS6ulo+n0/5+fndtufn56u8vLzHY/bs2aPnn39ePp9PL730km6//Xbdf//9+vnPf97r66xYsUJZWVnBR1FRUShlAogS65f/0LRkW5aCtwRvlscAVsCRIv7Tw+/3Ky8vT7/97W81Y8YMzZs3Tz/+8Y+1atWqXo9ZunSp6urqgo+DBw9GukwA/WANGLVzJk3g9QNhpO54m7ztfltrARC6kMaM5ObmKiEhQRUVFd22V1RUqKCgoMdjRowYoaSkJCUkJAS3nXXWWSovL5fX61Vy8ok/xDwejzwe+/qfAfSN1TJi53gRScpOTVKC2yWf3+hoU6tGZKXaWg+A0ITUMpKcnKwZM2Zo3bp1wW1+v1/r1q1TSUlJj8ecf/752r17t/z+zr9WPv74Y40YMaLHIALAOaobrDVG7A0jbrdLw9KtQax01QBOE3I3zZIlS/Too4/qySef1M6dO3XDDTeoqakpOLtm/vz5Wrp0aXD/G264QTU1Nbrxxhv18ccf68UXX9Rdd92lRYsWhe8qANjCmko73OYwIrEkPOBkIU/tnTdvnqqqqrRs2TKVl5dr2rRpWrt2bXBQ64EDB+R2d2acoqIivfzyy7rppps0ZcoUjRw5UjfeeKNuueWW8F0FAFtYU2lzbe6mCdZQxlojgBOFHEYkafHixVq8eHGPz61fv/6EbSUlJXrrrbf681IAYlgs3JfGwvRewLm4Nw2AfouFpeAtuRmMGQGcijACoN9i4SZ5FlpGAOcijADol8BS8AxgBTBwhBEA/VLT5JXfyPal4C2EEcC5CCMA+sX6pZ+TlqxEG5eCtwTHjLAkPOA49v8EAeBI1TE0k0bqrONYs1ftPpaEB5yEMAKgX2JpJo0UuFmf2yUZE+hCAuAchBEA/dLZMmL/eBFJSnC7lJMeCEYsfAY4C2EEQL903rE3NlpGpM5gZNUGwBkIIwD6xRooGivdNFJnLQxiBZyFMAKgX2JtAKvE9F7AqQgjAPqlKoZukmexummq6aYBHIUwAqBfrNaHWFh91ULLCOBMhBEAIQssBd9xX5qM2JhNI3UNI4wZAZyEMAIgZEebWuU3ktslDUuPoZaRDFpGACcijAAIWXVDoOUhJz1ZCW6XzdV0Co4ZIYwAjkIYARCyWJxJI3WOX6lp8srnNzZXA6CvCCMAQhZrS8FbctKT5XJJfpaEBxyFMAIgZLHaMpKY4FZOGl01gNMQRgCELNbuS9MV03sB5yGMAAhZrHbTSJ1TjQkjgHMQRgCEzFrHI9a6aaQuLSMNjBkBnIIwAiBksXjHXgvdNIDzEEYAhCy4FHwsdtN0hJEqwgjgGIQRACFp9/lV0xzL3TTWmBG6aQCnIIwACElNk1emYyn4nPQYnE1jLQnPnXsBxyCMAAiJ1f2Rk+6JqaXgLcMZMwI4DmEEQEg6Z9LEXquI1Nl1dLTJKz9LwgOOQBgBEJJYXmNEkoZ1hCSf3+hYM+NGACcgjAAISXAmTQwOXpWkpAS3stOSJDGIFXAKwgiAkFgDQ3NjtGVEYq0RwGkIIwBCUtkRRvJiOIxYtVUxowZwBMIIgJBU1LdIkvIyU2yupHf5HbVZtQKIbYQRACGxWkbyY7llJDNQW0U9LSOAExBGAPSZMcYRLSN5GR0tIw20jABOQBgB0GeNre1q9vokxfaYkfyOlpFKumkARyCMAOgzq4smw5OodE+izdX0zhozUskAVsARCCMA+qyziyZ2W0UkKT+jcwCrMazCCsQ6wgiAPqvsGBCaH8PjRaTOsNTS5ld9S7vN1QA4FcIIgD6zWkZiPYykJCUoMyXQjcS4ESD2EUYA9Jk1VTaWB69aOtcaYdwIEOsIIwD6rLIh9qf1WjoHsdIyAsQ6wgiAPuscMxL7LSNW6w0tI0DsI4wA6DNrEbFYHzMidbbesCQ8EPsIIwD6pOvqq9bU2VgWXPiMbhog5hFGAPRJQ2u7Wtr8kmJ/nRGpy5gRummAmEcYAdAn1hTZzJREpSQl2FzNqVktI9yfBoh9hBEAfVLhkAXPLMGb5dW3sgorEOMIIwD6xCkLnlmGd8ym8bb7VXe8zeZqAJwMYQRAnwQXPHPAeBEpsAprdlqSJKb3ArGOMAKgTyodNK3XYs36YUYNENsIIwD6pNJBS8FbrFYcWkaA2EYYAdAnThszInUdxErLCBDLCCMA+qQ8GEac0zISnN5LGAFiGmEEwCn5/Z2rrxZkpdpcTd+NyAq0jJTVEUaAWEYYAXBKR5u8avMZuV3OGjMyoiM4lRNGgJjWrzCycuVKjR49WikpKSouLtY777zTp+OeeeYZuVwuXXHFFf15WQA2Kas7LimwdkdSgnP+hikItowct7kSACcT8k+VNWvWaMmSJVq+fLm2bt2qqVOnas6cOaqsrDzpcfv27dO//du/6YILLuh3sQDsYXVzjHBQF40kFWYH6q1u9Kq13WdzNQB6E3IYeeCBB7Rw4UItWLBAkyZN0qpVq5SWlqbVq1f3eozP59PVV1+tO+64Q2PHjh1QwQCir6w20LJgjcFwiqFpSfIkBn7MccM8IHaFFEa8Xq+2bNmi0tLSzhO43SotLdWmTZt6Pe7OO+9UXl6err322j69Tmtrq+rr67s9ANinLDh41VlhxOVyBQPUkVq6aoBYFVIYqa6uls/nU35+frft+fn5Ki8v7/GYDRs26LHHHtOjjz7a59dZsWKFsrKygo+ioqJQygQQZmW1gTBS6LBuGqkzQJUzvReIWREdidbQ0KDvfOc7evTRR5Wbm9vn45YuXaq6urrg4+DBgxGsEsCpWLNRnNYyInWOczlSSxgBYlViKDvn5uYqISFBFRUV3bZXVFSooKDghP0//fRT7du3T3Pnzg1u8/v9gRdOTNRHH32kcePGnXCcx+ORx+Oc6YNAvCurd+aYEamz5nJm1AAxK6SWkeTkZM2YMUPr1q0LbvP7/Vq3bp1KSkpO2H/ixIl6//33tX379uDjq1/9qi666CJt376d7hfAAfx+E2wZGZHtvG4aFj4DYl9ILSOStGTJEl1zzTWaOXOmZs2apYceekhNTU1asGCBJGn+/PkaOXKkVqxYoZSUFE2ePLnb8dnZ2ZJ0wnYAscla8MzlsAXPLFY3DWEEiF0hh5F58+apqqpKy5YtU3l5uaZNm6a1a9cGB7UeOHBAbrdzFkUCcHLWgmF5DlvwzFJAywgQ80IOI5K0ePFiLV68uMfn1q9ff9Jjn3jiif68JACblNU57540XXUufNYqb7tfyYnOC1RAvOO7EsBJBRc8y3Te4FWp+8Jn3L0XiE2EEQAnZS14NiLbmWGk68JndNUAsYkwAuCkgjNpHDit18IN84DYRhgBcFLW6qtOu0leV8yoAWIbYQTASTl5wTNL58JnhBEgFhFGAPSq64JnTlwK3sLN8oDYRhgB0KuuC57lO3Q2jUQ3DRDrCCMAenW4oyUhPyPFkQueWay1RmgZAWKTc3+6AIi4Q8eaJUmnDXXu4FVJGtlR/9Emr5q97TZXA+CzCCMAenXoWKAlwelhJCs1SZkpgQWnDx+jdQSINYQRAL3qbBlJs7mSgbOu4RBhBIg5hBEAvYqXlhGp8xqsgAUgdhBGAPTqcDCMOL9lZGQwjNAyAsQawgiAHhlj4qxlhG4aIFYRRgD0qKbJq+NtPrlczr1JXlfBbhqm9wIxhzACoEdWC0JehkeexASbqxk4K4wcZswIEHMIIwB6dCiOxotInddR3ejVca/P5moAdEUYAdCjw7XxseCZJSs1SRnWWiO1tI4AsYQwAqBH8TR41TKyY1n4gwxiBWIKYQRAj+Ktm0ZiRg0QqwgjAHoUL/el6YqFz4DYRBgBcILua4zEU8sIC58BsYgwAuAEx5rb1Nwx46QwDtYYsVjBipvlAbGFMALgBNYv6/zM+FhjxELLCBCbCCMATmCNqbBmn8QLK4xUN7ay1ggQQwgjAE6wvyYQRopy4me8iNR9rZGDDGIFYgZhBMAJ9h9tkiSNHpZucyXh5XK5gte0r7rJ5moAWAgjAE6wt+MX9ejc+GoZkaRRwwLXtO8oYQSIFYQRACfYfzTQhRFvLSOSNCa3o2XkKN00QKwgjADopqXNp7K6FknxGUZGdVzTflpGgJhBGAHQjdUqkpmSqOy0JJurCb/RVjdNNS0jQKwgjADoxhpLMTo3XS6Xy+Zqwm90RzfNkbrjamljei8QCwgjALqxui9GxWEXjSQNS0/WEE+ijOEeNUCsIIwA6GZvR/fFmGHxN5NGCkzvHUVXDRBTCCMAuon3lhGpc2Au03uB2EAYAdBNcFpvHK4xYrGujTACxAbCCICgljafjtQFbiIXzy0jndN76aYBYgFhBEDQwZpmGSNleBI1LD3Z7nIihm4aILYQRgAEWauSjspNi8tpvRarm+bwsePytvttrgYAYQRAkHXzuHjuopGk4UM8SktOkN9w914gFhBGAAQFFzyL02m9lsD0Xu7eC8QKwgiAoE+rGiVJY3KH2FxJ5I0dHggje6oII4DdCCMAgnZXBsLIGXnxH0bGDw9c4yeVDTZXAoAwAkCSdKzJq+pGryRp3CAII2fkB67RCmAA7EMYASBJ2t3RRTMyO1VDPIk2VxN5Z+RlSJI+qWyUMcbmaoDBjTACQJL0SUUgjIwfBK0iUmB6b4LbpYaWdlU2tNpdDjCoEUYASOocOzEYxotIkicxIXjDPCuIAbAHYQSApM6xE4OlZURiECsQKwgjACR1tg5YAzsHA+taP2EQK2ArwggANbS0qby+RZI0fniGzdVEjzWIlRk1gL0IIwCCv4zzMjzKSkuyuZrosbqkCCOAvQgjAILdFIOpi0aSxg0fIpdLqmny6mgjM2oAuxBGAHRZeXXwdNFIUmpygoqGdsyooXUEsA1hBIA+qQjMJhkMK69+ltVVQxgB7EMYAdDZTTMIw4h1zVYgAxB9hBFgkKs73qZDx45LkibkD65uGkmaUBC45l1lhBHALv0KIytXrtTo0aOVkpKi4uJivfPOO73u++ijj+qCCy7Q0KFDNXToUJWWlp50fwDRtbOsXlLgnjRD05Ntrib6zi7MkiR9WFYvv5971AB2CDmMrFmzRkuWLNHy5cu1detWTZ06VXPmzFFlZWWP+69fv15XXnmlXnvtNW3atElFRUW65JJLdPjw4QEXD2DgdhwJhJGzRmTaXIk9xg5PV3KiW42t7TpQ02x3OcCgFHIYeeCBB7Rw4UItWLBAkyZN0qpVq5SWlqbVq1f3uP/vf/97ff/739e0adM0ceJE/e53v5Pf79e6desGXDyAgfuwI4ycXTg4w0hSglsTO7pqrGAGILpCCiNer1dbtmxRaWlp5wncbpWWlmrTpk19Okdzc7Pa2tqUk5PT6z6tra2qr6/v9gAQGTuO1EkavGFE6rz2D8vqbK4EGJxCCiPV1dXy+XzKz8/vtj0/P1/l5eV9Osctt9yiwsLCboHms1asWKGsrKzgo6ioKJQyAfRRa7svuMbI2SOzbK7GPpM6xo3QMgLYI6qzae6++24988wz+uMf/6iUlJRe91u6dKnq6uqCj4MHD0axSmDw+KSiUe1+o6zUJBVm9f49Ge8mdYyXIYwA9kgMZefc3FwlJCSooqKi2/aKigoVFBSc9Nj77rtPd999t1599VVNmTLlpPt6PB55PJ5QSgPQD127aFwul83V2OesERlyuaSqhlZVNrQoL2PwBjPADiG1jCQnJ2vGjBndBp9ag1FLSkp6Pe7ee+/Vz372M61du1YzZ87sf7UAwmqwD161pCUnamxuuqTO9wRA9ITcTbNkyRI9+uijevLJJ7Vz507dcMMNampq0oIFCyRJ8+fP19KlS4P733PPPbr99tu1evVqjR49WuXl5SovL1djI0svA3bbEQwjg3e8iOVsxo0Atgmpm0aS5s2bp6qqKi1btkzl5eWaNm2a1q5dGxzUeuDAAbndnRnnN7/5jbxer775zW92O8/y5cv105/+dGDVA+g3v98EFzybNMhbRqTAe/Dnvx+hZQSwQchhRJIWL16sxYsX9/jc+vXru/173759/XkJABG272iTmrw+eRLdwS6KwczqqvrgCNN7gWjj3jTAILXtQK0kafLILCUm8KNgyshsSdL+o82qafLaWwwwyPATCBikth44Jkk69/RsewuJEVlpSRo3PNBCtK3jvQEQHYQRYJDasj/wC3fGqKE2VxI7rPfCem8ARAdhBBiEGlvb9XFFgyTp3NMJIxbrvdhKywgQVYQRYBD6+8Fa+Y00MjtVeZks8GU5t6Nl5O8H69Tu89tcDTB4EEaAQWhrRzfEuXTRdDN++BBlpCTqeJtPu8ob7C4HGDQII8AgtIXBqz1yu12aVpQtia4aIJoII8Ag4/eb4LReBq+eyHpPtjKIFYgawggwyOypblLd8TalJLl11ghWXv2szkGstfYWAgwihBFgkLH+4p8yMltJLHZ2gmmnZ8vlkg7UNKuyocXucoBBgZ9EwCCzac9RSdJ5Y+ii6UlmSpImFgRajN7aU2NzNcDgQBgBBhFjjDbsrpYknT8+1+ZqYtfnxw+TJL35SbXNlQCDA2EEGER2VzaqqqFVnkQ3i52dxOyOoLZhd7WMMTZXA8Q/wggwiFitIrPG5CglKcHmamLXrNE5Skpw6XDtcR2oaba7HCDuEUaAQeRNumj6JN2TqOkdLUdWgAMQOYQRYJBo9/mDAzLPH0cYORXrPdq4+6jNlQDxjzACDBJ/P1SnxtZ2ZaclaVIh64ucyufPCAxi3fhptfx+xo0AkUQYAQaJjR3dDbPHDVOC22VzNbFvymnZGuJJ1LHmNn1YVm93OUBcI4wAg8TrH1dJkmbTRdMnSQlufW5sjqTO9w5AZBBGgEGgqqE1eHO8i8/Ks7ka5/iHifmSpL/tKLe5EiC+EUaAQeDVnRUyRpp6WpZGZKXaXY5jlE7Kk8sVGG9TVnfc7nKAuEUYAQaBlzv+sr/k7AKbK3GWvIyU4OJwr3xYYXM1QPwijABxrqGlLTg9dc7Z+TZX4zzWe/a3HYQRIFIII0Cce/3jKnl9fo3NTde44UPsLsdxLpkUaE16a89R1TW32VwNEJ8II0Cce7njL/pLzi6Qy8WU3lCNzk3XhPwMtfuN/ucjWkeASCCMAHHsuNen13ZVSpIuoYum36z37qX3mVUDRAJhBIhjf/uwXI2t7TptaKqmnZZtdzmONXdqoSTptV2VOtrYanM1QPwhjABx7PkthyRJ35xxmtysutpvZ+ZnaOppWWr3G/1p+xG7ywHiDmEEiFNHao8H7zj7jXNPs7ka5/vmjMB7aAU8AOFDGAHi1B+2HpIx0ufG5qgoJ83uchxv7tRCJSe49WFZvXYcqbO7HCCuEEaAOGSM6dJFU2RzNfEhOy1ZX5oUGMhK6wgQXoQRIA69tadG+442Kz05QZedw6qr4fLNmYGumhe2HVZLm8/maoD4QRgB4tBjG/ZIkq6YPlJpyYk2VxM/vnDGcJ02NFXHmtv0/7bSOgKEC2EEiDO7Kxv16s5KuVzStZ8fY3c5cSXB7dK/nB94Tx/7373y+43NFQHxgTACxJnHNuyVJJWela+xLP8edv90XpEyUxK1p7pJ6zoWlAMwMIQRII5UN7bqDx3dBwsvGGtzNfFpiCdRVxWPkiQ9+r97bK4GiA+EESCOrN6wV63tfk09LUvnjR5qdzlx659nj1ai26V39tZo874au8sBHI8wAsSJ8roWrX4z0EXz/YvGc1O8CCrIStG3OmbWrPjrLhnD2BFgIAgjQJx46NWP1dLm14xRQ3XJJG6KF2k/LD1TKUlubdl/TH/7kLv5AgNBGAHiwCcVDXp280FJ0m2XTaRVJAryM1OCs5XuXbtL7T6/zRUBzkUYARzOGKOfv7hTfiNdMilfM0bl2F3SoPG9C8dpaFqSPq1q0u/fPmB3OYBjEUYAh3th+2G9/nGVkhPcuuXSiXaXM6hkpiRpyZfOlBRoHTl0rNnmigBnIowADlbV0Ko7/vtDSdK/Xjxe41hXJOquLh6lmaOGqsnr021//IDBrEA/EEYAhzLGaNmfPlBtc5smjcjU9y4cZ3dJg5Lb7dI935yi5ES33vi4Ss9tZpl4IFSEEcChnty4T3/9oFwJbpfu/eYUJSXw7WyXccOHBLtrlv95h3aW1dtcEeAs/PQCHOidvTX6+Ys7JUlLL52oySOzbK4ICy8YqwvOyNXxNp++93+3qK65ze6SAMcgjAAOc+hYs77/+61q9xvNnVrIzfBiRILbpV99e7pGZqfqQE2zfvDMNrUx3RfoE8II4CAV9S266tG3Vd3YqokFGbrnG+ewpkgMGZqerEe+M0OejvEjN63ZLh939gVOiTACOER1Y6uu/t3bOlDTrNNz0vTEgllKS060uyx8xuSRWVr1f2YoKcGlv7xXplv+33sEEuAUCCOAA+yubNTXf71RuysbNSIrRb+/rlgFWSl2l4VeXDQxTw9fOV0Jbpee33JINzy1Rc3edrvLAmIWYQSIcW/urtbXf/2mDtQ0qygnVU8v/JyKctLsLgun8OXJI/Srb09XcqJbf/uwQvMeeUvldS12lwXEJMIIEKO87X7d/ddd+j+Pva36lnade3q2Xvj++RqTm253aeijy6eM0NPXFSsnPVnvH67Tl//jDf31/TK7ywJiDmEEiEGb99XoipVvatXrn8oYad7MIj298HMaNsRjd2kI0czROfrj92frnJFZqm1u0w2/36obn9lGKwnQhcs4YO3i+vp6ZWVlqa6uTpmZmXaXA0TM3uomPfTqx/rT9iOSpOy0JN399XP05ckjbK4MA+Vt9+uhVz/WbzoCZlpygq6/cJz++fzRykxJsrs8ICL6+vubMALYzBijbQdrtXrDXr30fpn8RnK5Aq0h/zZngnJpDYkr7x2q1U//vENbD9RKkjJSEjW/ZJSuKh6lkdmp9hYHhBlhBIhxB442668flOkPWw/ro4qG4PaLJ+bppi+dyaqqccwYoz///Yj+839265PKRkmBAHrhmcP1tWmF+oeJ+cpKpbUEzhfRMLJy5Ur98pe/VHl5uaZOnaqHH35Ys2bN6nX/5557Trfffrv27dunM844Q/fcc48uu+yyPr8eYQTxoLqxVe8dqtXG3Ue1YXe1dpV3BpCUJLcuPyewmuqkQv4fHyz8fqO/fVihJzfu06Y9R4PbE90uzRg1VMVjcnTemByde/pQpXtYUwbOE7EwsmbNGs2fP1+rVq1ScXGxHnroIT333HP66KOPlJeXd8L+Gzdu1Be+8AWtWLFCX/nKV/T000/rnnvu0datWzV58uSwXgwQC+pb2nTgaLMO1jTr44pGvX+4TjuO1KnsMwMWE9wufW5sji6dPEJzpxbyl/Agt6+6SX/YdlhrPyjTxxWN3Z5LcLs0aUSmJhRkaHzeEJ2RN0Tj84ZoZHaqErlBImJYxMJIcXGxzjvvPP3nf/6nJMnv96uoqEg/+MEPdOutt56w/7x589TU1KS//OUvwW2f+9znNG3aNK1atSqsFwOEizFGXp9fLV6/mtvaddzr0/E2n457fTrW3KZjTV4dbfKqpqlVR5u8OtbkVXWjVwePNau2lxukuVzSmNx0zRqdo/PH5+rz43M1ND05ylcGJ9hb3aS39hzVO3tr9M7eGh2uPd7jfm6XlJeRooKsFBVkBr5mpSZ1f6QFvmakJColMUGeJLc8iQlKcHMbAUReX39/h9Tu5/V6tWXLFi1dujS4ze12q7S0VJs2berxmE2bNmnJkiXdts2ZM0cvvPBCr6/T2tqq1tbW4L/r6yNzO+7HNuzVwZrmU+7XU17rKcF9djfTw149Rb++nKu3PXs8X4+vEd5aejpfHzf1+/0M6XxGavf75fMbtftN4Kuv4+tnt3d8bfP51druDwaPgSzhPSw9WUU5aRqbm66zR2bpnJFZmlSYqSE0taMPxuSma0xuuq6cdbok6XDtcf39YK12Vzbqk8pG7a5s1J6qRrW2+1Ve36Ly+tCnCScluLqFk86QIiW4XEpwBx5ul0uJCYGvCW6XEt2d/911H0kKxhuX9cUll6vbpi7/7vJclwP7tH/wOQJVOF37+TG2LagY0k/G6upq+Xw+5efnd9uen5+vXbt29XhMeXl5j/uXl5f3+jorVqzQHXfcEUpp/fLie0eCI9qB3iS4XUpLSlBqcuCRlZqknPRk5aQna1h6snLSPR1fk3VaTqqKhqbRv4+wGpmdesJMG7/fqLqpVeV1LSqra1F5XYsq6ltUe7xNdcfbVN/x1Xo0trSrvUvAbvMZtfna1dD62VfDYPXVaYXOCCPRsnTp0m6tKfX19SoqKgr763xjxmmaPS73hO09he0e83cPO/aW03s+Zw/H9/G1e9wvhL8SolFPKOfs8dg+vr+91ZiY4FZix19uiQkuJbi7/Dv41R183u1yKSXJrbTkRKVa4SMpQcmJ9Mkj9rjdLuVlpCgvI0VTTuvbMe0drX+t7X61tPnU0uYL/re13d/RUugzHV/9Rn7T2Yro73jO36VlUepsyTQyXf5bwees1tSuDZlWq2bgeZ3yPDLmhP0QPgWZ9t3vKqQwkpubq4SEBFVUVHTbXlFRoYKCgh6PKSgoCGl/SfJ4PPJ4Ir+2wtXFoyL+GgAQKxIT3EpMcCudpWsQY0L6ky85OVkzZszQunXrgtv8fr/WrVunkpKSHo8pKSnptr8kvfLKK73uDwAABpeQu2mWLFmia665RjNnztSsWbP00EMPqampSQsWLJAkzZ8/XyNHjtSKFSskSTfeeKMuvPBC3X///br88sv1zDPPaPPmzfrtb38b3isBAACOFHIYmTdvnqqqqrRs2TKVl5dr2rRpWrt2bXCQ6oEDB+R2dza4zJ49W08//bR+8pOf6LbbbtMZZ5yhF154oc9rjAAAgPjGcvAAACAi+vr7m2kCAADAVoQRAABgK8IIAACwFWEEAADYijACAABsRRgBAAC2IowAAABbEUYAAICtCCMAAMBWIS8Hbwdrkdj6+nqbKwEAAH1l/d4+1WLvjggjDQ0NkqSioiKbKwEAAKFqaGhQVlZWr8874t40fr9fR44cUUZGhlwuV9jOW19fr6KiIh08eDBu73nDNTpfvF+fxDXGg3i/Pin+rzES12eMUUNDgwoLC7vdRPezHNEy4na7ddppp0Xs/JmZmXH5P1ZXXKPzxfv1SVxjPIj365Pi/xrDfX0naxGxMIAVAADYijACAABsNajDiMfj0fLly+XxeOwuJWK4RueL9+uTuMZ4EO/XJ8X/Ndp5fY4YwAoAAOLXoG4ZAQAA9iOMAAAAWxFGAACArQgjAADAVnEfRn7xi19o9uzZSktLU3Z2do/7HDhwQJdffrnS0tKUl5enH/3oR2pvbz/peWtqanT11VcrMzNT2dnZuvbaa9XY2BiBKwjN+vXr5XK5eny8++67vR73xS9+8YT9r7/++ihW3nejR48+oda77777pMe0tLRo0aJFGjZsmIYMGaJvfOMbqqioiFLFodm3b5+uvfZajRkzRqmpqRo3bpyWL18ur9d70uNi/TNcuXKlRo8erZSUFBUXF+udd9456f7PPfecJk6cqJSUFJ1zzjl66aWXolRp6FasWKHzzjtPGRkZysvL0xVXXKGPPvropMc88cQTJ3xeKSkpUao4ND/96U9PqHXixIknPcZJn5/U888Vl8ulRYsW9bi/Ez6/N954Q3PnzlVhYaFcLpdeeOGFbs8bY7Rs2TKNGDFCqampKi0t1SeffHLK84b6vdwXcR9GvF6vvvWtb+mGG27o8Xmfz6fLL79cXq9XGzdu1JNPPqknnnhCy5YtO+l5r776au3YsUOvvPKK/vKXv+iNN97Qd7/73UhcQkhmz56tsrKybo/rrrtOY8aM0cyZM0967MKFC7sdd++990ap6tDdeeed3Wr9wQ9+cNL9b7rpJv33f/+3nnvuOb3++us6cuSIvv71r0ep2tDs2rVLfr9fjzzyiHbs2KEHH3xQq1at0m233XbKY2P1M1yzZo2WLFmi5cuXa+vWrZo6darmzJmjysrKHvffuHGjrrzySl177bXatm2brrjiCl1xxRX64IMPolx537z++utatGiR3nrrLb3yyitqa2vTJZdcoqamppMel5mZ2e3z2r9/f5QqDt3ZZ5/drdYNGzb0uq/TPj9Jevfdd7td3yuvvCJJ+ta3vtXrMbH++TU1NWnq1KlauXJlj8/fe++9+tWvfqVVq1bp7bffVnp6uubMmaOWlpZezxnq93KfmUHi8ccfN1lZWSdsf+mll4zb7Tbl5eXBbb/5zW9MZmamaW1t7fFcH374oZFk3n333eC2v/71r8blcpnDhw+HvfaB8Hq9Zvjw4ebOO+886X4XXnihufHGG6NT1ACNGjXKPPjgg33ev7a21iQlJZnnnnsuuG3nzp1Gktm0aVMEKgy/e++914wZM+ak+8TyZzhr1iyzaNGi4L99Pp8pLCw0K1as6HH/f/qnfzKXX355t23FxcXme9/7XkTrDJfKykojybz++uu97tPbz6RYtHz5cjN16tQ+7+/0z88YY2688UYzbtw44/f7e3zeSZ+fMcZIMn/84x+D//b7/aagoMD88pe/DG6rra01Ho/H/Nd//Vev5wn1e7mv4r5l5FQ2bdqkc845R/n5+cFtc+bMUX19vXbs2NHrMdnZ2d1aGkpLS+V2u/X2229HvOZQ/PnPf9bRo0e1YMGCU+77+9//Xrm5uZo8ebKWLl2q5ubmKFTYP3fffbeGDRum6dOn65e//OVJu9W2bNmitrY2lZaWBrdNnDhRp59+ujZt2hSNcgesrq5OOTk5p9wvFj9Dr9erLVu2dHv/3W63SktLe33/N23a1G1/KfB96aTPS9IpP7PGxkaNGjVKRUVF+trXvtbrz5xY8Mknn6iwsFBjx47V1VdfrQMHDvS6r9M/P6/Xq6eeekr/8i//ctKbszrp8/usvXv3qry8vNvnlJWVpeLi4l4/p/58L/eVI26UF0nl5eXdgoik4L/Ly8t7PSYvL6/btsTEROXk5PR6jF0ee+wxzZkz55Q3Grzqqqs0atQoFRYW6r333tMtt9yijz76SH/4wx+iVGnf/eu//qvOPfdc5eTkaOPGjVq6dKnKysr0wAMP9Lh/eXm5kpOTTxgzlJ+fH3OfV092796thx9+WPfdd99J94vVz7C6ulo+n6/H77Ndu3b1eExv35dO+Lz8fr9++MMf6vzzz9fkyZN73W/ChAlavXq1pkyZorq6Ot13332aPXu2duzYEdEbg/ZHcXGxnnjiCU2YMEFlZWW64447dMEFF+iDDz5QRkbGCfs7+fOTpBdeeEG1tbX653/+5173cdLn1xPrswjlc+rP93JfOTKM3HrrrbrnnntOus/OnTtPOcDKSfpzzYcOHdLLL7+sZ5999pTn7zre5ZxzztGIESN08cUX69NPP9W4ceP6X3gfhXJ9S5YsCW6bMmWKkpOT9b3vfU8rVqyI6WWa+/MZHj58WF/+8pf1rW99SwsXLjzpsXZ/hghYtGiRPvjgg5OOqZCkkpISlZSUBP89e/ZsnXXWWXrkkUf0s5/9LNJlhuTSSy8N/veUKVNUXFysUaNG6dlnn9W1115rY2WR8dhjj+nSSy9VYWFhr/s46fNzAkeGkZtvvvmkiVWSxo4d26dzFRQUnDAS2JplUVBQ0Osxnx2s097erpqaml6PGaj+XPPjjz+uYcOG6atf/WrIr1dcXCwp8Fd5NH6RDeQzLS4uVnt7u/bt26cJEyac8HxBQYG8Xq9qa2u7tY5UVFRE7PPqSajXeOTIEV100UWaPXu2fvvb34b8etH+DHuTm5urhISEE2Yvnez9LygoCGn/WLF48eLggPZQ/zpOSkrS9OnTtXv37ghVFz7Z2dk688wze63VqZ+fJO3fv1+vvvpqyC2KTvr8pM7fbxUVFRoxYkRwe0VFhaZNm9bjMf35Xu6zAY04cZBTDWCtqKgIbnvkkUdMZmamaWlp6fFc1gDWzZs3B7e9/PLLMTWA1e/3mzFjxpibb765X8dv2LDBSDJ///vfw1xZ+D311FPG7XabmpqaHp+3BrA+//zzwW27du2K6QGshw4dMmeccYb59re/bdrb2/t1jlj6DGfNmmUWL14c/LfP5zMjR4486QDWr3zlK922lZSUxOwASL/fbxYtWmQKCwvNxx9/3K9ztLe3mwkTJpibbropzNWFX0NDgxk6dKj5j//4jx6fd9rn19Xy5ctNQUGBaWtrC+m4WP/81MsA1vvuuy+4ra6urk8DWEP5Xu5zfQM62gH2799vtm3bZu644w4zZMgQs23bNrNt2zbT0NBgjAn8DzR58mRzySWXmO3bt5u1a9ea4cOHm6VLlwbP8fbbb5sJEyaYQ4cOBbd9+ctfNtOnTzdvv/222bBhgznjjDPMlVdeGfXr682rr75qJJmdO3ee8NyhQ4fMhAkTzNtvv22MMWb37t3mzjvvNJs3bzZ79+41f/rTn8zYsWPNF77whWiXfUobN240Dz74oNm+fbv59NNPzVNPPWWGDx9u5s+fH9zns9dnjDHXX3+9Of30083//M//mM2bN5uSkhJTUlJixyWc0qFDh8z48ePNxRdfbA4dOmTKysqCj677OOkzfOaZZ4zH4zFPPPGE+fDDD813v/tdk52dHZzF9p3vfMfceuutwf3ffPNNk5iYaO677z6zc+dOs3z5cpOUlGTef/99uy7hpG644QaTlZVl1q9f3+3zam5uDu7z2Wu84447zMsvv2w+/fRTs2XLFvPtb3/bpKSkmB07dthxCSd18803m/Xr15u9e/eaN99805SWlprc3FxTWVlpjHH+52fx+Xzm9NNPN7fccssJzznx82toaAj+zpNkHnjgAbNt2zazf/9+Y4wxd999t8nOzjZ/+tOfzHvvvWe+9rWvmTFjxpjjx48Hz/EP//AP5uGHHw7++1Tfy/0V92HkmmuuMZJOeLz22mvBffbt22cuvfRSk5qaanJzc83NN9/cLRW/9tprRpLZu3dvcNvRo0fNlVdeaYYMGWIyMzPNggULggEnFlx55ZVm9uzZPT63d+/ebu/BgQMHzBe+8AWTk5NjPB6PGT9+vPnRj35k6urqolhx32zZssUUFxebrKwsk5KSYs466yxz1113dWvF+uz1GWPM8ePHzfe//30zdOhQk5aWZv7xH/+x2y/3WPL444/3+P9s14ZMJ36GDz/8sDn99NNNcnKymTVrlnnrrbeCz1144YXmmmuu6bb/s88+a84880yTnJxszj77bPPiiy9GueK+6+3zevzxx4P7fPYaf/jDHwbfj/z8fHPZZZeZrVu3Rr/4Ppg3b54ZMWKESU5ONiNHjjTz5s0zu3fvDj7v9M/P8vLLLxtJ5qOPPjrhOSd+ftbvrs8+rOvw+/3m9ttvN/n5+cbj8ZiLL774hGsfNWqUWb58ebdtJ/te7i+XMcYMrKMHAACg/wb9OiMAAMBehBEAAGArwggAALAVYQQAANiKMAIAAGxFGAEAALYijAAAAFsRRgAAgK0IIwAAwFaEEQAAYCvCCAAAsBVhBAAA2Or/A0e/031f58X6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(temps, 1 - np.tanh(temps) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the derivative has a steep slope, making it ideal as an activation function - this will enable gradient descent to descend effectively.  We can now use $\\tanh$ instead of relu in our network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full Forward Pass\n",
    "\n",
    "We now know enough to put together a complete RNN forward pass.  We'll first initialize our weights and biases.  We'll add in bias terms in the hidden and output steps.\n",
    "\n",
    "We'll also scale the weights and biases to work properly with the $\\tanh$ nonlinearity.  We'll make our input and hidden weights small, so $\\tanh$ doesn't squash all the values to `1` or `-1`.  We'll make the output weight large, since the output of the hidden step will be small (between `1` and `-1`).  Of course, in a full RNN, the network would eventually learn the correct parameters.  But initializing weights and biases to the correct ranges helps with gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "\n",
    "# Define our weights and biases\n",
    "# Scale them down so values get through the tanh nonlinearity\n",
    "i_weight = np.random.rand(1,5) / 5 - .1\n",
    "h_weight = np.random.rand(5,5) / 5 - .1\n",
    "h_bias = np.random.rand(1,5) / 5 - .1\n",
    "\n",
    "# Tanh pushes values to between -1 and 1, so scale up the output weights\n",
    "o_weight = np.random.rand(5,1) * 50\n",
    "o_bias = np.random.rand(1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then we can write the forward pass as a for loop.  This loop will process sequence elements one by one.  We'll store the output prediction and the hidden state:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "# An array to store the output predictions\n",
    "outputs = np.zeros(3)\n",
    "# An array to store hidden states for use in backpropagation\n",
    "hiddens = np.zeros((3, 5))\n",
    "\n",
    "# This will store the previous hidden state, since we'll need it to calculate the current hidden step\n",
    "prev_hidden = None\n",
    "sequence = data[\"tmax\"].tail(3).to_numpy()\n",
    "\n",
    "for i in range(3):\n",
    "    # Get the input sequence at the given position\n",
    "    x = sequence[i].reshape(1,1)\n",
    "\n",
    "    # Multiply input by input weight\n",
    "    xi = x @ i_weight\n",
    "    if prev_hidden is not None:\n",
    "        # Add previous hidden to input\n",
    "        xh = xi + prev_hidden @ h_weight + h_bias\n",
    "    else:\n",
    "        xh = xi\n",
    "\n",
    "    # Apply our activation function\n",
    "    xh = np.tanh(xh)\n",
    "    prev_hidden = xh\n",
    "    hiddens[i,] = xh\n",
    "\n",
    "    # Multiply by the output weight\n",
    "    xo = xh @ o_weight + o_bias\n",
    "    outputs[i] = xo"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Afterwards, we can take a look at our outputs and hidden states.  As we can see, the hidden states don't increase constantly anymore:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([74.31470595, 80.66149404, 77.67852446])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[ 2.74338471e-04, -2.89009929e-04, -3.20562379e-05,\n         3.67012717e-04,  2.85825121e-04],\n       [-6.63092480e-02,  7.42736825e-02, -6.91914532e-03,\n        -1.20488975e-02, -1.15884296e-01],\n       [-2.94826242e-02,  4.33377272e-02, -2.63269775e-02,\n         3.78706198e-02, -7.86427108e-02]])"
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hiddens"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Calculating loss\n",
    "\n",
    "After we do our forward pass, we can calculate the gradient with respect to the network outputs using `mse_grad`:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse(actual, predicted):\n",
    "    return np.mean((actual-predicted)**2)\n",
    "\n",
    "def mse_grad(actual, predicted):\n",
    "    return (predicted - actual)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([-69.61482716, -66.14775128, -65.93075892])"
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Actual next day temperatures\n",
    "actuals = np.array([70, 62, 65])\n",
    "\n",
    "loss_grad = mse_grad(actuals, outputs)\n",
    "loss_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Backward Pass\n",
    "\n",
    "Now that we've written a forward pass, let's think about the backward pass to update our model parameters.  The main complication in the backward pass is that parameters impact both the current output and future outputs.\n",
    "\n",
    "Let's visualize this, starting with the last sequence item:\n",
    "\n",
    "![last hidden](images/rnn/gradient_last.svg)\n",
    "\n",
    "As you can see above, the last hidden step is only used by the output of the last time step.  But this is different for other hidden steps:\n",
    "\n",
    "![prev hidden](images/rnn/gradient_inside.svg)\n",
    "\n",
    "The hidden step at time step 2 is connected to both the output and the next hidden state.  So it affects not just the current output, but all subsequent outputs, too.\n",
    "\n",
    "We have to consider this fact when we do backpropagation - some parameters will impact multiple outputs.  This means that they need to get gradients from multiple outputs to be properly adjusted."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We'll have to send the gradient with respect to each hidden step backwards to the previous sequence position.  This is called backpropagation through time, and it's how we train the parameters of an RNN."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the operations we will follow to do a backward pass.  They're essentially a reverse of the forward pass:\n",
    "\n",
    "![rnn backward](images/rnn/rnn_operations_bw.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's implement the backward pass one by one for each sequence element.\n",
    "\n",
    "We start out by creating variables to store the gradients with respect to each parameter.  We do this because we want to sum the gradients across time positions before making updates with gradient descent:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "outputs": [],
   "source": [
    "# These will keep a running total of the gradients\n",
    "o_weight_grad, o_bias_grad, h_weight_grad, h_bias_grad, i_weight_grad = [0] * 5"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then, we can run backpropagation for the last sequence element (we start at the end).  In this case, there is no next hidden state, so we only need to worry about the gradient wrt the output at this position:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "outputs": [],
   "source": [
    "# Get the loss wrt the output at the current time step\n",
    "l2_grad = loss_grad[2].reshape(1,1)\n",
    "\n",
    "# Add to the output weight gradient\n",
    "# Multiply the output of the hidden step (hiddens[2]) transposed by the l2 grad\n",
    "# np.newaxis creates a new size 1 axis, effectively transposing the hiddens\n",
    "o_weight_grad += hiddens[2][:,np.newaxis] @ l2_grad\n",
    "# Add to the bias gradient.  Similar to a dense neural network, this is just the mean of the l2_grad.\n",
    "o_bias_grad += np.mean(l2_grad)\n",
    "\n",
    "# Find the gradient wrt the hidden step output\n",
    "h2_grad = l2_grad @ o_weight.T\n",
    "\n",
    "# Derivative of the tanh function\n",
    "tanh_deriv = 1 - hiddens[2,:][np.newaxis,:] ** 2\n",
    "# Multiply each position in the h_grad by the tanh derivative - this \"undoes\" the tanh in the forward pass\n",
    "h2_grad = np.multiply(h2_grad, tanh_deriv)\n",
    "\n",
    "# Now, find how much we need to update the hidden weights.\n",
    "# We take the input to the hidden step (the output of the previous hidden step in the forward pass) @ h2_grad\n",
    "h_weight_grad += hiddens[1,:][:,np.newaxis] @ h2_grad\n",
    "h_bias_grad += np.mean(h2_grad)\n",
    "\n",
    "# This multiples the sequence value at time step 2 by the gradient\n",
    "# We don't need the .T here, but I left it here in case you have a larger input size\n",
    "i_weight_grad += sequence[2].reshape(1,1).T @ h2_grad"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "A lot of the above step is very similar to backpropagation in a dense neural network.  The main difference comes in the next sequence position (1) where we need to consider multiple gradients at the hidden step:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "outputs": [],
   "source": [
    "l1_grad = loss_grad[1].reshape(1,1)\n",
    "\n",
    "o_weight_grad += hiddens[1][:,np.newaxis] @ l1_grad\n",
    "o_bias_grad += np.mean(l1_grad)\n",
    "\n",
    "h1_grad = l1_grad @ o_weight.T\n",
    "\n",
    "# We do have a next sequence position (2), so we need to include that gradient\n",
    "# We multiply the h2 gradient by the weight to pull it back to the current sequence position\n",
    "h1_grad += h2_grad @ h_weight.T\n",
    "\n",
    "# The rest of the operation is the same\n",
    "tanh_deriv = 1 - hiddens[1,:][np.newaxis,:] ** 2\n",
    "h1_grad = np.multiply(h1_grad, tanh_deriv)\n",
    "\n",
    "h_weight_grad += hiddens[1,:][:,np.newaxis] @ h1_grad\n",
    "h_bias_grad += np.mean(h1_grad)\n",
    "\n",
    "i_weight_grad += sequence[1].reshape(1,1).T @ h1_grad"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, we can do the final sequence position, 0.  The main difference here is that we don't update the hidden gradient, since there is no previous sequence position that gave us hidden state input in the forward pass:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "outputs": [],
   "source": [
    "l0_grad = loss_grad[0].reshape(1,1)\n",
    "\n",
    "o_weight_grad += hiddens[0][:,np.newaxis] @ l0_grad\n",
    "o_bias_grad += np.mean(l0_grad)\n",
    "\n",
    "h0_grad = l0_grad @ o_weight.T\n",
    "\n",
    "h0_grad += h1_grad @ h_weight.T\n",
    "\n",
    "tanh_deriv = 1 - hiddens[0,:][np.newaxis,:] ** 2\n",
    "h0_grad = np.multiply(h0_grad, tanh_deriv)\n",
    "\n",
    "# We don't update the hidden weight, since there was no previous hidden state\n",
    "# We can update the hidden bias if you want\n",
    "\n",
    "i_weight_grad += sequence[0].reshape(1,1).T @ h0_grad"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can now look at our gradient updates:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[-154.9774425 ,  -61.87971495,  -11.63213862,  -81.30882966,\n          17.37148576]])"
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i_weight_grad"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We've now completed backpropagation across 3 time steps!  We'll see how to make the gradient updates in the next section."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full Backward Pass\n",
    "\n",
    "Similar to a forward pass, we can implement the full backward pass as a loop.  Here it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "next_hidden = None\n",
    "\n",
    "o_weight_grad, o_bias_grad, h_weight_grad, h_bias_grad, i_weight_grad = [0] * 5\n",
    "\n",
    "for i in range(2, -1, -1):\n",
    "    l_grad = loss_grad[i].reshape(1,1)\n",
    "\n",
    "    o_weight_grad += hiddens[i][:,np.newaxis] @ l_grad\n",
    "    o_bias_grad += np.mean(l_grad)\n",
    "\n",
    "    o_grad = l_grad @ o_weight.T\n",
    "\n",
    "    # Only add in the hidden gradient if a next sequence exists\n",
    "    if next_hidden is not None:\n",
    "        h_grad = o_grad + next_hidden @ h_weight.T\n",
    "    else:\n",
    "        h_grad = o_grad\n",
    "\n",
    "    tanh_deriv = 1 - hiddens[i,:][np.newaxis,:] ** 2\n",
    "    h_grad = np.multiply(h_grad, tanh_deriv)\n",
    "\n",
    "    next_hidden = h_grad\n",
    "\n",
    "    # Don't update the hidden weights for the first sequence position\n",
    "    if i > 0:\n",
    "        h_weight_grad += hiddens[i-1,:][:,np.newaxis] @ h_grad\n",
    "        h_bias_grad += np.mean(h_grad)\n",
    "\n",
    "    i_weight_grad += sequence[i].reshape(1,1).T @ h_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use gradient descent to make parameter updates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-6\n",
    "# We'll divide the learning rate by the sequence length, since we were adding together the gradients\n",
    "# This makes training the model more stable\n",
    "lr = lr / 3\n",
    "\n",
    "i_weight -= i_weight_grad * lr\n",
    "h_weight -= h_weight_grad * lr\n",
    "h_bias -= h_bias_grad * lr\n",
    "o_weight -= o_weight_grad * lr\n",
    "o_bias -= o_bias_grad * lr"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Full Implementation"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now know enough to do a full implementation of a complete network.  This code will mostly be the same as the forward and backward passes we already implemented.\n",
    "\n",
    "We'll first load in and scale our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "import math\n",
    "\n",
    "# Define predictors and target\n",
    "PREDICTORS = [\"tmax\", \"tmin\", \"rain\"]\n",
    "TARGET = \"tmax_tomorrow\"\n",
    "\n",
    "# Scale our data to have mean 0\n",
    "scaler = StandardScaler()\n",
    "data[PREDICTORS] = scaler.fit_transform(data[PREDICTORS])\n",
    "\n",
    "# Split into train, valid, test sets\n",
    "np.random.seed(0)\n",
    "split_data = np.split(data, [int(.7*len(data)), int(.85*len(data))])\n",
    "(train_x, train_y), (valid_x, valid_y), (test_x, test_y) = [[d[PREDICTORS].to_numpy(), d[[TARGET]].to_numpy()] for d in split_data]"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then we can initialize our weights and biases.  We'll scale our parameters so they are relatively small.  This helps the network descend better:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_params(layer_conf):\n",
    "    layers = []\n",
    "    for i in range(1, len(layer_conf)):\n",
    "        np.random.seed(0)\n",
    "        k = 1/math.sqrt(layer_conf[i][\"hidden\"])\n",
    "        i_weight = np.random.rand(layer_conf[i-1][\"units\"], layer_conf[i][\"hidden\"]) * 2 * k - k\n",
    "\n",
    "        h_weight = np.random.rand(layer_conf[i][\"hidden\"], layer_conf[i][\"hidden\"]) * 2 * k - k\n",
    "        h_bias = np.random.rand(1, layer_conf[i][\"hidden\"]) * 2 * k - k\n",
    "\n",
    "        o_weight = np.random.rand(layer_conf[i][\"hidden\"], layer_conf[i][\"output\"]) * 2 * k - k\n",
    "        o_bias = np.random.rand(1, layer_conf[i][\"output\"]) * 2 * k - k\n",
    "\n",
    "        layers.append(\n",
    "            [i_weight, h_weight, h_bias, o_weight, o_bias]\n",
    "        )\n",
    "    return layers"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then we'll write a forward pass:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(x, layers):\n",
    "    hiddens = []\n",
    "    outputs = []\n",
    "    for i in range(len(layers)):\n",
    "        i_weight, h_weight, h_bias, o_weight, o_bias = layers[i]\n",
    "        hidden = np.zeros((x.shape[0], i_weight.shape[1]))\n",
    "        output = np.zeros((x.shape[0], o_weight.shape[1]))\n",
    "        for j in range(x.shape[0]):\n",
    "            input_x = x[j,:][np.newaxis,:] @ i_weight\n",
    "            hidden_x = input_x + hidden[max(j-1,0),:][np.newaxis,:] @ h_weight + h_bias\n",
    "            # Activation.  tanh avoids outputs getting larger and larger.\n",
    "            hidden_x = np.tanh(hidden_x)\n",
    "            # Store hidden for use in backprop\n",
    "            hidden[j,:] = hidden_x\n",
    "\n",
    "            # Output layer\n",
    "            output_x = hidden_x @ o_weight + o_bias\n",
    "            output[j,:] = output_x\n",
    "        hiddens.append(hidden)\n",
    "        outputs.append(output)\n",
    "    return hiddens, outputs[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "And a backward pass:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward(layers, x, lr, grad, hiddens):\n",
    "    for i in range(len(layers)):\n",
    "        i_weight, h_weight, h_bias, o_weight, o_bias = layers[i]\n",
    "        hidden = hiddens[i]\n",
    "        next_h_grad = None\n",
    "        i_weight_grad, h_weight_grad, h_bias_grad, o_weight_grad, o_bias_grad = [0] * 5\n",
    "\n",
    "        for j in range(x.shape[0] - 1, -1, -1):\n",
    "            # Add newaxis in the first dimension\n",
    "            out_grad = grad[j,:][np.newaxis, :]\n",
    "\n",
    "            # Output updates\n",
    "            # np.newaxis creates a size 1 axis, in this case transposing matrix\n",
    "            o_weight_grad += hidden[j,:][:, np.newaxis] @ out_grad\n",
    "            o_bias_grad += out_grad\n",
    "\n",
    "            # Propagate gradient to hidden unit\n",
    "            h_grad = out_grad @ o_weight.T\n",
    "\n",
    "            if j < x.shape[0] - 1:\n",
    "                # Then we multiply the gradient by the hidden weights to pull gradient from next hidden state to current hidden state\n",
    "                hh_grad = next_h_grad @ h_weight.T\n",
    "                # Add the gradients together to combine output contribution and hidden contribution\n",
    "                h_grad += hh_grad\n",
    "\n",
    "            # Pull the gradient across the current hidden nonlinearity\n",
    "            # derivative of tanh is 1 - tanh(x) ** 2\n",
    "            # So we take the output of tanh (next hidden state), and plug in\n",
    "            tanh_deriv = 1 - hidden[j][np.newaxis,:] ** 2\n",
    "\n",
    "            # next_h_grad @ np.diag(tanh_deriv_next) multiplies each element of next_h_grad by the deriv\n",
    "            # Effect is to pull value across nonlinearity\n",
    "            h_grad = np.multiply(h_grad, tanh_deriv)\n",
    "\n",
    "            # Store to compute h grad for previous sequence position\n",
    "            next_h_grad = h_grad.copy()\n",
    "\n",
    "            # If we're not at the very beginning\n",
    "            if j > 0:\n",
    "                # Multiply input from previous layer by post-nonlinearity grad at current layer\n",
    "                h_weight_grad += hidden[j-1][:, np.newaxis] @ h_grad\n",
    "                h_bias_grad += h_grad\n",
    "\n",
    "            i_weight_grad += x[j,:][:,np.newaxis] @ h_grad\n",
    "\n",
    "        # Normalize lr by number of sequence elements\n",
    "        lr = lr / x.shape[0]\n",
    "        i_weight -= i_weight_grad * lr\n",
    "        h_weight -= h_weight_grad * lr\n",
    "        h_bias -= h_bias_grad * lr\n",
    "        o_weight -= o_weight_grad * lr\n",
    "        o_bias -= o_bias_grad * lr\n",
    "        layers[i] = [i_weight, h_weight, h_bias, o_weight, o_bias]\n",
    "    return layers"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can end by setting up a training loop and measuring error:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 train loss 3122.594400144508 valid loss 2171.3186862102025\n",
      "Epoch: 50 train loss 30.593193275313595 valid loss 30.568271740103427\n",
      "Epoch: 100 train loss 25.263986813543738 valid loss 24.435517510355645\n",
      "Epoch: 150 train loss 22.9567624295313 valid loss 22.177010971976852\n",
      "Epoch: 200 train loss 22.306774327704215 valid loss 21.557992202834164\n"
     ]
    }
   ],
   "source": [
    "epochs = 250\n",
    "lr = 1e-5\n",
    "\n",
    "layer_conf = [\n",
    "    {\"type\":\"input\", \"units\": 3},\n",
    "    {\"type\": \"rnn\", \"hidden\": 4, \"output\": 1}\n",
    "]\n",
    "layers = init_params(layer_conf)\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    sequence_len = 7\n",
    "    epoch_loss = 0\n",
    "    for j in range(train_x.shape[0] - sequence_len):\n",
    "        seq_x = train_x[j:(j+sequence_len),]\n",
    "        seq_y = train_y[j:(j+sequence_len),]\n",
    "        hiddens, outputs = forward(seq_x, layers)\n",
    "        grad = mse_grad(seq_y, outputs)\n",
    "        params = backward(layers, seq_x, lr, grad, hiddens)\n",
    "        epoch_loss += mse(seq_y, outputs)\n",
    "\n",
    "    if epoch % 50 == 0:\n",
    "        sequence_len = 7\n",
    "        valid_loss = 0\n",
    "        for j in range(valid_x.shape[0] - sequence_len):\n",
    "            seq_x = valid_x[j:(j+sequence_len),]\n",
    "            seq_y = valid_y[j:(j+sequence_len),]\n",
    "            _, outputs = forward(seq_x, layers)\n",
    "            valid_loss += mse(seq_y, outputs)\n",
    "\n",
    "        print(f\"Epoch: {epoch} train loss {epoch_loss / len(train_x)} valid loss {valid_loss / len(valid_x)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see above, the network reduces training and validation loss over successive epochs.  It will perform similarly to the RNN implementation in Pytorch, but it will descend more slowly.\n",
    "\n",
    "## Wrap-up\n",
    "\n",
    "We learned a lot in this lesson! You should now have a good idea of how to train a recurrent neural network.  Don't worry if you don't get all of the concepts immediately.  RNNs are tricky, and take multiple readings to really understand.\n",
    "\n",
    "In the next lesson, we'll learn about how we can reduce test and validation error with regularization."
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}