cs236756_tutorial_13_deep_learning_intro_backprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/dusk/64/000000/mind-map.png\" style=\"height:50px;display:inline\"> CS 236756 - Technion - Intro to Machine Learning\n",
    "---\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 13 - Deep Learning Introduction & Backpropagation\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_12_intro.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [Motivation](#-Motivation)\n",
    "    * [Demo - Solving the XOR Problem](#-Demo---Solving-the-XOR-Problem)\n",
    "* [Problem Formulation](#-The-Optimization-Problem)\n",
    "    * [Optimization Problem](#-The-Optimization-Problem)\n",
    "    * [Multi-Layer Perceptron (MLP)](#-Multi-Layer-Perceptron-(MLP) )\n",
    "* [Modular Approach](#-Modular-Approach)\n",
    "    * [Building Blocks & Backpropagation](#-Backpropagation)\n",
    "* [Regression Problem](#-Example---Neural-Networks-for-Regression---Housing-Prices)\n",
    "* [Backpropagation By Hand Exercise](#-Backpropagation-By-Hand--Exercise)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/confetti.png\" style=\"height:50px;display:inline\"> Motivation\n",
    "---\n",
    "* From the biological perspective, it seems logical to look at the brain's architecture for inspiration on how to build an intelligent machine. \n",
    "* This is the main idea that inspired *artificial neural networks* (ANN) which later developed into *deep neural networks* (DNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/100/000000/lightning-bolt.png\" style=\"height:50px;display:inline\"> The Perceptron Expressive Power\n",
    "---\n",
    "#### Reminder (Tutorial 9) - The Perceptron\n",
    "\n",
    "* Based on a *linear threshold unit* (LTU): the input and output are numbers (not binary values), and each connection is associated with a weight.\n",
    "* The LTU computes a weighted sum of its inputs: $z = w_1x_1 + w_2x_2 +....+w_nx_n = w_Tx$, and then it applies a **step function** to that sum and outputs the result: $$ h_w(x) = step(z) = step(w^Tx) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Illustration: <img src=\"./assets/tut_09_perceptron.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The most common step function used is the *Heaviside step function* but sometimes the *sign function* is used (as is the illustration).\n",
    "* **Perceptron Training** draws inspiration from biological neurons: the connection weight between two neurons is increased whenever they have **the same output**. Perceptrons are trained by considering the error made.\n",
    "    * At each iteration, the Perceptron is fed with one training instance and makes a prediction for it.\n",
    "    * For every output that produced a wrong prediction, it reinforces the connection weights from the inputs that would ahve contributed to the correct prediction.\n",
    "    * Criterion: $ E^{perc}(w) = - \\sum_{i \\in D_{miss}}w^T(x^iy^i) $\n",
    "* **Perceptron Learning Rule (weight update)**: $$ w_{t+1} = w_t +\\eta(y_i -sign(w_t^Tx_i))x_i $$\n",
    "    * $\\eta$ is the learing rate\n",
    "* The decision boundary learned is linear, the Perceptron is incapable of learning complex patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Perceptron Convergence Theorem**: If the training instances are **linearly seperable**, the algorithm would converge to a solution.\n",
    "    * **There can be multiple solutions (multiple hyperplanes)**\n",
    "* Perceptrons do not output a class probability, they just make predicitons based on a **hard threshold** (they output an un-normalized score)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **What mappings can a Perceptron represent perfectly?**\n",
    "    * The Perceptron is a **linear classifier**.\n",
    "    * Thus, it can represent any mapping that is linearly separable.\n",
    "        * Some Boolean functions like *AND* and *OR* (left figure)\n",
    "        * But not Boolean functions like *XOR* (right figure)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* This was demonstrated in the work of Minsky and Papert named *Perceptron*, 1969:\n",
    "\n",
    "<img src=\"./assets/tut_12_boolean.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  <img src=\"https://img.icons8.com/dusk/64/000000/classroom.png\" style=\"height:50px;display:inline\"> Demo - Solving the XOR Problem\n",
    "---\n",
    "In the XOR problem we have 2 classes such that:\n",
    "\n",
    "| Input | Output   |\n",
    "|------|------|\n",
    "|   [-1,-1]  | 0|\n",
    "|   [-1,1]  | 1|\n",
    "|   [1,1]  | 0|\n",
    "|   [1,-1]  | 1|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot_xor_problem():\n",
    "    x1 = [-1, 1]\n",
    "    y1 = [-1, 1]\n",
    "    x2 = [-1, 1]\n",
    "    y2 = [1, -1]\n",
    "    fig = plt.figure(figsize=(5,5))\n",
    "    ax = fig.add_subplot(1,1,1)\n",
    "    ax.plot(x1, y1, 'ro')\n",
    "    ax.plot(x2, y2, 'bo')\n",
    "    ax.grid()\n",
    "    ax.set_xlabel('x')\n",
    "    ax.set_ylabel('y')\n",
    "    ax.set_title('The XOR problem')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVkAAAFNCAYAAACqtRxWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdbklEQVR4nO3df5QdZZ3n8fdngxkmNCvESCdB7LC70RHdAWnkx4y7ocMEQ45MZMYfZLORUdlePIZRZ2AnWWYYzjg4iMcfxwFkESJhNtDDWUSyGgkQOjDIoklYCAkYCTGBkIACEW2CspHv/lHVM8XNvd33dtfT1eR+XufU6ap6nqe+T/XN/aS6+t6+igjMzCyNf1X1BMzMDmQOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFryUm6RNL/rHoe44mkkPTvGrT9iaT7xnpOloZD1kZN0kBheVXSy4XthSXXukXSNTX7vi3pisL2WyStkPS8pJck/VDS+2vGRN42IOlpSV+WNKHMuZqBQ9ZKEBEdgwvwJHBmYd+Kkst9CvhjST0Akj4CvBtYkm9PBu4DXgHeCUwBvgLcKOmDNcc6Np/zLOAjwMfLmKCkg8o4jh0YHLI2ViZKukHSLyVtlnTCYIOk6fkV6s8k/UTSnzY6SEQ8A/w58A1JbwW+BvzXiBjIu3wWGAA+ERHPRMTLEXETcCnwJUmqc8ytwPeB4xrVlbRd0lJJj0raI+mbkg7O206VtFPSX0h6Bvhmvv+/SNoq6QVJKyVNrznsPEnbJD0n6YuS6j4fJf2OpDvz42yR9OFC2/WSrpL0vfyq/PuSpkr6aj7PH0l6d6PzsvQcsjZW/hDoAw4DVgJXAOTB8r+Bh4EjgdOAz0h6X6MDRcT1wBPAg8DtEXF7oXkOcEtEvFoz7GbgrcDbao8n6XeA/wBsHeYcFgLvA/5tfpy/LLRNBSYDXUCvpNnA3wEfBqYBO/LzLzoLOAE4HphPnStpSYcAdwI3AkcAC4CrJL2z0O3D+VymAL8G/g/Z92YK8L+ALw9zXpaQQ9bGyn0RsSoifgP8A3Bsvv89wJsj4m8i4pWI2AZ8Azh7mOP9E/AmoPYXalOA3XX67y60D3pQ0kvAY8Ba4Kphal4REU9FxAtkV8YLCm2vAn8dEb+OiJfJAnlZRDwYEb8GlgKnSJpRGPOFiHghIp4EvlpzvEHvB7ZHxDcjYl9EPAjcAhRvfdwaERsi4lfArcCvIuKG/Hv9j2S3U6wiDlkbK88U1vcCB+f3LruA6ZJ+PrgA/x3obHQgSTOBC8hC8UuS3lBofo7syrHWtEL7oOOBDrL7sScBhwxzDk8V1ncAxR//f5aH3KDpeR8A8tsZz5NdrTdzvEFdwEk135+FZFfOg54trL9cZ7uj4RlZcg5Zq9pTwE8i4rDCcmhEzKvXOb+nei3Zld/5wEvAXxS63EX2i7Haf9sfzmv9uLgzMjeT/Yh98TBzPaqw/lZgV/FQNX13kQXk4LwPIbvyfrrJ4w16Crin5vvTERGfHGauNk44ZK1qPwR+kf/S6LclTZD0LknvadD/k2Q/8n8+v+/6CeC/5fdVIXslwb8Grst/AXSwpAXARcCF0fhve15Gdi91aoN2gE/lLw+bTHa1/Y9D9L0R+Jik4yT9FvB54AcRsb3Q50JJh0s6Cvh0g+N9B3ibpEWS3pAv75H0jiFq2zjikLVK5fcNzyT7zf5PyH6cvxZ4Y23fPIw+T/bKgVfy8Y8CXyJ7tYEi4nngvcDBwKNkP6L/GbAoIhqGYkQ8AtwDXDjEdG8E7gC25cvfDnG8NcBfkd0/3U32y7La+8y3ARuAh4DvAtfVOc4vgdPzsbvIbrt8AfitIeZp44j8R7vNhidpO3BuRNxV9Vzs9cVXsmZmCTlkzcwS8u0CM7OEfCVrZpaQQ9bMLKG2+mtBU6ZMiRkzZrQ05qWXXuKQQ4Z7I1AaVdZu9/rtfO7tXn8ktTds2PBcRLy5bmNEtM3S3d0drerv7295TFmqrN3u9dv53Nu9/khqA+ujQe74doGZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpZQpSEraZmkn0ra1KBdkr6Wf+LnRknHF9rm5p/cuVXSkrLntmIFzJgBs2fPYsaMbNvMDmD5k37W7NmU+aSv+kr2emDuEO1nADPzpRf4OoCkCcCVefsxwAJJx5Q1qRUroLcXduyACLFjR7btoDU7QBWe9IqgzCd9pSEbEfcCLwzRZT5wQ/6migeAwyRNA04EtkbEtsj+Qn5f3rcUF10Ee/e+dt/evdl+MzsAJXzSj/e/XXAkr/1Ez535vnr7T6p3AEm9ZFfBdHZ2snbt2mGLPvnkLEB19gdr197T3MxLMDAw0NR8Xf/Aqu36Y19/1pNP1nnGQzz5JPeMdh6N3m87VgswA9jUoO27wHsL22uAbuBDwLWF/YuAvx+uVrN/u6CrKwL2X7q6mhpemnZ+/3jV9dv53Nuy/iif9LyO/3bBTl77sclvIfswuUb7S3HppTBp0mv3TZqU7TezA1DCJ/14D9mVwEfzVxmcDLwYEbuBdcBMSUdLmkj2SZ4ryyq6cCFccw10dYEUdHVl2wsXllXBzMaVwpM+JMp80ld6T1bSTcCpwBRJO4G/Bt4AEBFXA6uAecBWYC/wsbxtn6TFwGpgArAsIjaXObeFC7Nl7dp7OPXUU8s8tJmNR/mT/p61a0t9zlcashGxYJj2AD7VoG0VWQibmY1b4/12gZnZ65pD1swsIYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZglVGrKS5kraImmrpCV12i+U9FC+bJL0G0mT87btkh7J29aP/ezNzIZX2UeCS5oAXAnMAXYC6yStjIhHB/tExBeBL+b9zwQ+GxEvFA7TExHPjeG0zcxaUuWV7InA1ojYFhGvAH3A/CH6LwBuGpOZmZmVpMqQPRJ4qrC9M9+3H0mTgLnALYXdAdwhaYOk3mSzNDMbBUVENYWlDwHvi4hz8+1FwIkRcX6dvh8B/nNEnFnYNz0idkk6ArgTOD8i7q0zthfoBejs7Ozu6+traZ4DAwN0dHS0NKYsVdZu9/rtfO7tXn8ktXt6ejZExAl1GyOikgU4BVhd2F4KLG3Q91bgPw1xrEuAC4ar2d3dHa3q7+9veUxZqqzd7vXb+dzbvf5IagPro0HuVHm7YB0wU9LRkiYCZwMraztJeiMwC7itsO8QSYcOrgOnA5vGZNZmZi2o7NUFEbFP0mJgNTABWBYRmyWdl7dfnXc9C7gjIl4qDO8EbpUE2TncGBG3j93szcyaU1nIAkTEKmBVzb6ra7avB66v2bcNODbx9MzMRs3v+DIzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlCDlkzs4QcsmZmCVUaspLmStoiaaukJXXaT5X0oqSH8uXiZseamY0HB1VVWNIE4EpgDrATWCdpZUQ8WtP1nyLi/SMca2ZWqSqvZE8EtkbEtoh4BegD5o/BWDOzMVNlyB4JPFXY3pnvq3WKpIclfU/SO1sca2ZWqcpuFwCqsy9qth8EuiJiQNI84NvAzCbHZkWkXqAXoLOzk7Vr17Y0yYGBgZbHlKXK2u1ev53Pvd3rl147IipZgFOA1YXtpcDSYcZsB6aMZGxE0N3dHa3q7+9veUxZqqzd7vXb+dzbvf5IagPro0HuVHm7YB0wU9LRkiYCZwMrix0kTZWkfP1Estsbzzcz1sxsPKjsdkFE7JO0GFgNTACWRcRmSefl7VcDHwQ+KWkf8DJwdv6/Rt2xlZyImdkQqrwnS0SsAlbV7Lu6sH4FcEWzY83Mxhu/48vMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJVRpyEqaK2mLpK2SltRpXyhpY77cL+nYQtt2SY9IekjS+rGduZlZcw6qqrCkCcCVwBxgJ7BO0sqIeLTQ7SfArIjYI+kM4BrgpEJ7T0Q8N2aTNjNrUZVXsicCWyNiW0S8AvQB84sdIuL+iNiTbz4AvGWM52hmNipVhuyRwFOF7Z35vkY+AXyvsB3AHZI2SOpNMD8zs1FTRFRTWPoQ8L6IODffXgScGBHn1+nbA1wFvDcins/3TY+IXZKOAO4Ezo+Ie+uM7QV6ATo7O7v7+vpamufAwAAdHR2tnVxJqqzd7vXb+dzbvf5Iavf09GyIiBPqNkZEJQtwCrC6sL0UWFqn3+8CTwBvG+JYlwAXDFezu7s7WtXf39/ymLJUWbvd67fzubd7/ZHUBtZHg9yp8nbBOmCmpKMlTQTOBlYWO0h6K/AtYFFE/Liw/xBJhw6uA6cDm8Zs5mZmTars1QURsU/SYmA1MAFYFhGbJZ2Xt18NXAy8CbhKEsC+yC7JO4Fb830HATdGxO0VnIaZ2ZAqC1mAiFgFrKrZd3Vh/Vzg3DrjtgHH1u43Mxtv/I4vM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpZQpSEraa6kLZK2SlpSp12Svpa3b5R0fLNjzczGg8pCVtIE4ErgDOAYYIGkY2q6nQHMzJde4OstjDUzq9ywIStpsaTDE9Q+EdgaEdsi4hWgD5hf02c+cENkHgAOkzStybFmZpVr5kp2KrBO0s35j+gqqfaRwFOF7Z35vmb6NDPWzKxyBw3XISL+UtJfAacDHwOukHQzcF1EPDGK2vXCOprs08zY7ABSL9mtBjo7O1m7dm0LU4SBgYGWx5SlytrtXr+dz73d65deOyKaWoBjga8CPyK7N/p/gcubHV/neKcAqwvbS4GlNX3+B7CgsL0FmNbM2HpLd3d3tKq/v7/lMWWpsna712/nc2/3+iOpDayPBrnTzD3ZP5W0Abgc+D7w7yPik0A38MejyPd1wExJR0uaCJwNrKzpsxL4aP4qg5OBFyNid5NjzcwqN+ztAmAK8EcRsaO4MyJelfT+kRaOiH2SFgOrgQnAsojYLOm8vP1qYBUwD9gK7CW7XdFw7EjnYmaWSjP3ZC8eou2x0RSPiFVkQVrcd3VhPYBPNTvWzGy88Tu+zMwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlClYSspMmS7pT0eP718Dp9jpLUL+kxSZslfbrQdomkpyU9lC/zxvYMzMyaU9WV7BJgTUTMBNbk27X2AX8eEe8ATgY+JemYQvtXIuK4fFmVfspmZq2rKmTnA8vz9eXAB2o7RMTuiHgwX/8l8Bhw5JjN0MysBFWFbGdE7IYsTIEjhuosaQbwbuAHhd2LJW2UtKze7QYzs/FAEZHmwNJdwNQ6TRcByyPisELfPRFRNygldQD3AJdGxLfyfZ3Ac0AAnwOmRcTHG4zvBXoBOjs7u/v6+lo6j4GBATo6OloaU5Yqa7d7/XY+93avP5LaPT09GyLihLqNETHmC7CFLBgBpgFbGvR7A7Aa+LMhjjUD2NRM3e7u7mhVf39/y2PKUmXtdq/fzufe7vVHUhtYHw1yp6rbBSuBc/L1c4DbajtIEnAd8FhEfLmmbVph8yxgU6J5mpmNSlUhexkwR9LjwJx8G0nTJQ2+UuD3gUXA7Dov1bpc0iOSNgI9wGfHeP5mZk05qIqiEfE8cFqd/buAefn6fYAajF+UdIJmZiXxO77MzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUKVhKykyZLulPR4/vXwBv22S3pE0kOS1rc63sysalVdyS4B1kTETGBNvt1IT0QcFxEnjHC8mVllqgrZ+cDyfH058IExHm9mNiaqCtnOiNgNkH89okG/AO6QtEFS7wjGm5lVShGR5sDSXcDUOk0XAcsj4rBC3z0Rsd99VUnTI2KXpCOAO4HzI+JeST9vZnze1gv0AnR2dnb39fW1dB4DAwN0dHS0NKYsVdZu9/rtfO7tXn8ktXt6ejbU3NL8FxEx5guwBZiWr08DtjQx5hLggpGOjwi6u7ujVf39/S2PKUuVtdu9fjufe7vXH0ltYH00yJ2qbhesBM7J188BbqvtIOkQSYcOrgOnA5uaHW9mNh5UFbKXAXMkPQ7MybeRNF3SqrxPJ3CfpIeBHwLfjYjbhxpvZjbeHFRF0Yh4Hjitzv5dwLx8fRtwbCvjzczGG7/jy8wsIYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpklVEnISpos6U5Jj+dfD6/T5+2SHiosv5D0mbztEklPF9rmjf1ZmJkNr6or2SXAmoiYCazJt18jIrZExHERcRzQDewFbi10+cpge0SsGpNZm5m1qKqQnQ8sz9eXAx8Ypv9pwBMRsSPprMzMSlZVyHZGxG6A/OsRw/Q/G7ipZt9iSRslLat3u8HMbDxQRKQ5sHQXMLVO00XA8og4rNB3T0TUDUpJE4FdwDsj4tl8XyfwHBDA54BpEfHxBuN7gV6Azs7O7r6+vpbOY2BggI6OjpbGlKXK2u1ev53Pvd3rj6R2T0/Phog4oW5jRIz5AmwhC0aAacCWIfrOB+4Yon0GsKmZut3d3dGq/v7+lseUpcra7V6/nc+93euPpDawPhrkTlW3C1YC5+Tr5wC3DdF3ATW3CiRNK2yeBWwqdXZmZiWpKmQvA+ZIehyYk28jabqkf36lgKRJefu3asZfLukRSRuBHuCzYzNtM7PWHFRF0Yh4nuwVA7X7dwHzCtt7gTfV6bco6QTNzErid3yZmSXkkDUzS8gha2aWkEPWzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrZpaQQ9bMLCGHrJlZQg5ZM7OEHLJmZgk5ZM3MEnLImpkl5JA1M0vIIWtmlpBD1swsIYesmVlCDlkzs4QqCVlJH5K0WdKrkk4Yot9cSVskbZW0pLB/sqQ7JT2efz289EmuWAEzZjBr9myYMSPbNrMDVv6UZ/bsWaU+5au6kt0E/BFwb6MOkiYAVwJnAMcACyQdkzcvAdZExExgTb5dnhUroLcXduxAEbBjR7btoDU7IBWe8kSo1Kd8JSEbEY9FxJZhup0IbI2IbRHxCtAHzM/b5gPL8/XlwAdKneBFF8Heva/dt3dvtt/MDjgpn/KKiNEfZaTFpbXABRGxvk7bB4G5EXFuvr0IOCkiFkv6eUQcVui7JyLq3jKQ1Av0AnR2dnb39fUNO69Zs2dnV7A1QuKeu+9u6tzKMDAwQEdHx5jVc/3xUdv1x77+7NmziNB++6Xg7rvvGXZ8T0/Phoiof+szIpIswF1ktwVql/mFPmuBExqM/xBwbWF7EfD3+frPa/ruaWZO3d3d0ZSurgjYf+nqam58Sfr7+8e0nuuPj9quP/b1R/uUB9ZHg9xJdrsgIv4gIt5VZ7mtyUPsBI4qbL8F2JWvPytpGkD+9aflzRy49FKYNOm1+yZNyvab2QEn5VN+PL+Eax0wU9LRkiYCZwMr87aVwDn5+jlAs8HdnIUL4ZproKuLkKCrK9teuLDUMmY2PhSe8khR6lO+qpdwnSVpJ3AK8F1Jq/P90yWtAoiIfcBiYDXwGHBzRGzOD3EZMEfS48CcfLtcCxfC9u3ZPdjt2x2wZge4/CnP3XffU+pT/qByDtOaiLgVuLXO/l3AvML2KmBVnX7PA6elnKOZWRnG8+0CM7PXPYesmVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS6jSPxAz1iT9DNjR4rApwHMJpjPea7d7/XY+93avP5LaXRHx5noNbRWyIyFpfTT66zoHcO12r9/O597u9cuu7dsFZmYJOWTNzBJyyA7vmjat3e712/nc271+qbV9T9bMLCFfyZqZJdT2IVv1x5M3M17S2yU9VFh+Iekzedslkp4utM3bv8ro6uf9tkt6JK+xvtXxozj3oyT1S3osf5w+XWgb0bk3eiwL7ZL0tbx9o6Tjmx1bUv2Fed2Nku6XdGyhre7jUGLtUyW9WPieXtzs2JLqX1iovUnSbyRNzttGe+7LJP1U0qYG7Wke90afS9MuC/AO4O0M/XljE4AngH8DTAQeBo7J2y4HluTrS4AvtFi/pfH5XJ4he10ewCVkH0Y50vNvqj6wHZgy2vm3OhaYBhyfrx8K/LjwvW/53Id6LAt95gHfAwScDPyg2bEl1f894PB8/YzB+kM9DiXWPhX4zkjGllG/pv+ZwN1lnHs+/j8CxwObGrQnedzb/ko2qv948lbHnwY8ERGtvqmirPpljh92bETsjogH8/Vfkn1KxpEtzrFoqMeyOK8bIvMAcJiyz5JrZuyo60fE/RGxJ998gOzz7cowmvmPybnXWADc1GKNhiLiXuCFIbokedzbPmSbdCTwVGF7J//yRO+MiN2QBQJwRIvHbnX82ez/D29x/uPNslZvV7RQP4A7JG1Q9jHrI53/SGoDIGkG8G7gB4XdrZ77UI/lcH2aGVtG/aJPkF1dDWr0OJRZ+xRJD0v6nqR3jnDeo6mPpEnAXOCWwu7RnPto5jeqc6/k42fGmqS7gKl1mi6K5j49d/8PZM8e8FHXb/YY+XEmAn8ILC3s/jrwuXw+nwO+BHw8Qf3fj4hdko4A7pT0o/zKYLg5l3XuHWRPuM9ExC/y3cOee71D1dlX+1g26jOqfwct1M86Sj1kIfvewu4RPQ4t1H6Q7FbUQH6P+9vAzFbmPcr6g84Evh8RxSvP0Zz7aOY3qnNvi5CNiD8Y5SGG/XjyiNitBh9PPlR9ScOOLzgDeDAini0c+5/XJX0D+E6K+pF9/hoR8VNJt5L9CHUvw5x/GbUlvYEsYFdExLdaOfc6hnosh+szsYmxZdRH0u8C1wJnRPaZdsCQj0MptQv/gRERqyRdJWlKs/Mebf2C/X5iG+W5j2Z+o3rcfbugOSk/nryV8fvdo8rDadBZQN3fnI6mvqRDJB06uA6cXqgzmvNvpraA64DHIuLLNW0jOfehHsvivD6a/7b5ZODF/HZGM2NHXV/SW4FvAYsi4seF/UM9DmXVnpp/z5F0IllGPN/M2DLq53XfCMyi8O+hhHNvRprHfaS/qTtQFrIn507g18CzwOp8/3RgVaHfPLLfbD9BdpthcP+bgDXA4/nXyS3Wrzu+Tv1JZP/Y31gz/h+AR4CN+QM/rez6ZL9VfThfNpd1/k3Wfi/Zj2YbgYfyZd5ozr3eYwmcB5yXrwu4Mm9/hMKrThr9O2jxez5c/WuBPYXzXT/c41Bi7cX5sR8m+6Xb743luefbfwL01Ywr49xvAnYD/4/sOf+JsXjc/Y4vM7OEfLvAzCwhh6yZWUIOWTOzhByyZmYJOWTNzBJyyJqZJeSQNTNLyCFrViDpPfkfnDk4f5fRZknvqnpe9vrlNyOY1ZD0t8DBwG8DOyPi7yqekr2OOWTNauTvT18H/IrsbaW/qXhK9jrm2wVm+5sMdJB9EsPBFc/FXud8JWtWQ9JKsr9+fzTZH51ZXPGU7HWsLf6erFmzJH0U2BcRN0qaANwvaXZE3F313Oz1yVeyZmYJ+Z6smVlCDlkzs4QcsmZmCTlkzcwScsiamSXkkDUzS8gha2aWkEPWzCyh/w+mj4Hf1Hq5qwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the xor problem\n",
    "plot_xor_problem()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <a href=\"https://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=xor&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=5&seed=0.35101&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false\"> Online Demo (@ TensorFlow) </a>\n",
    "* Parameters to play with:\n",
    "    * Number of layers\n",
    "    * Feature construction (add $x_1x_2$ and see what happens)\n",
    "    * Problem Type: change from classification to regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/100/000000/maintenance.png\" style=\"height:50px;display:inline\"> The Optimization Problem\n",
    "---\n",
    "<img src=\"./assets/tut_12_optim_general.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The Design Variables (Parameters)\n",
    "    * The network weights\n",
    "* The problem:\n",
    "    * Search for the weights' values that minimize the **loss function**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/100/000000/stocks.png\" style=\"height:50px;display:inline\"> Reminder - Regression (Tutorial 8)\n",
    "---\n",
    "To find the value of $\\theta$ that minimizes the cost function, there is a *closed-form solution* - a mathemtical equation that gives the result directly. It is also called the **Normal Equation**. We will now derive it.\n",
    "\n",
    "* We wish to find a solution for $\\hat{y} = X \\theta$\n",
    "* The parameters $\\theta$ are obtained by minimzing the *sum of squared* errors or residuals (SSE): $$ SSE(\\theta) = \\sum_{i=1}^n (\\theta^T x_i - y_i)^2 = ||X \\theta -y ||_2^2 = (X\\theta -y)^T(X\\theta - y) = \\theta^TX^TX\\theta -\\theta^Tx^Ty - y^TX\\theta +y^Ty$$\n",
    "* Minimizing w.r.t to $\\theta$: $$\\nabla_{\\theta}SSE(\\theta) = 2X^TX\\theta - 2X^Ty = 0 \\rightarrow \\theta^{*} = (X^TX)^{-1}X^Ty $$\n",
    "* The matrix $(X^TX)^{-1}X^T$ is the *Pseudo Inverse* of $X$\n",
    "* **How do we solve the multi layer model?**\n",
    "    * The inputs of every layer is now the output of the previous layer ($X$ is not just the original input)\n",
    "    * We have weights for every layer, that is, $w$ is now a matrix $W$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/layers.png\" style=\"height:50px;display:inline\"> Multi-Layer Perceptron (MLP)\n",
    "---\n",
    "* An MLP is composed of one input layer, one or more hidden layers and a final output layer. \n",
    "* Every layer, except the output layer includes a **bias neuron** which is fully connected to the next layer. \n",
    "* When the number of hidden layers is larger than 2, the network is usually called a deep neural network (DNN)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The algorithm is composed of two main parts: **forward pass and backward pass**.\n",
    "* In the *forward pass*, for each training instance, the algorithm feeds it to the network and computes the output of every neuron in each consecutive layer (using the network for prediction is just doing a forward pass). \n",
    "* Then, the output error (the difference between the desired output and the actual output from the network) is computed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* After the output error calculation, the network calculates how much each neuron in the last hidden layer contributed to the output error (using the **chain rule**).\n",
    "* It then proceeds to measure how much of these error contributions came from each neuron in the previous layers until reaching the input layer. \n",
    "* This is the *backward pass*: measuring the error gradient across all the connection weights in the network by propagating the error gradient backward in the network (this is the backpropagation process)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In short: for each training instance the **backpropagation algorithm** first makes a prediction (forward pass), measures the error, then goes in reverse to measure the error contribution from each connection (backward pass) and finally, using Gradient Descent, updates the weights in the direction that reduces the error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_12_mlp.jpg\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For example, if: $$ X \\in \\mathbb{R}^2 $$ $$ W_1 \\in \\mathbb{R}^{2 \\times 4} $$ $$ W_2 \\in \\mathbb{R}^{4 \\times 3} $$ $$ W_3 \\in \\mathbb{R}^{3 \\times 1} $$ $$ b_1 \\in \\mathbb{R}^4 $$ $$ b_2 \\in \\mathbb{R}^3 $$ $$ b_3 \\in \\mathbb{R} $$ Then: $$ F(X,W) = W_3^T \\phi_2(W_2^T\\phi_1(W_1^TX + b_1) + b_2) + b_3 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The key change made to the Perceptron that brought upon the era of deep learning is the addition of **Activation function** to the output of each neuron. These allow the learning of non-linear functions. We will use three popular activation functions:\n",
    "1. **Logistic function (sigmoid)**: $\\sigma(z) = \\frac{1}{1 + e^{-z}}$. The output is in $[0,1]$ which can be used for binary clssification or as a probability (why?)\n",
    "2. **Hyperbolic tangent function**: $tanh(z) = 2\\sigma(2z) - 1$. The output is in $[-1,1]$ which tends to make each layer's output more or less normalized at the beginning of the training (which may speed up convergence).\n",
    "3. **ReLU (Rectified Linear Unit) function**: $ReLU(z) = max(0,z)$. Continuous but not differentiable at $z=0$. However, it is the most common activation function as it is fast to compute and does not bound the output (which helps with some issues during Gradient Descent)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**The Activation functions derivatives (for the backpropagation)**:\n",
    "1. $\\frac{d\\sigma(z)}{dz} = \\sigma(z)(1-\\sigma(z))$\n",
    "2. $\\frac{d tanh(z)}{dz} = 1 - tanh^2(z)$\n",
    "3. We define the derivative at 0 to be zero: $\\frac{d ReLU(z)}{dz} = 1$ if $x>0$ else $0$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# activation functions\n",
    "def sigmoid(z, deriv=False):\n",
    "    output = 1 / (1 + np.exp(-1.0 * z))\n",
    "    if deriv:\n",
    "        return output * (1 - output)\n",
    "    return output\n",
    "\n",
    "def tanh(z, deriv=False):\n",
    "    output = np.tanh(z)\n",
    "    if deriv:\n",
    "        return 1 - np.square(output)\n",
    "    return output\n",
    "\n",
    "def relu(z, deriv=False):\n",
    "    output = z if z > 0 else 0\n",
    "    if deriv:\n",
    "        return 1 if z > 0 else 0\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot_activations():\n",
    "    x = np.linspace(-5, 5, 1000)\n",
    "    y_sig = sigmoid(x)\n",
    "    y_tanh = tanh(x)\n",
    "    y_relu = list(map(lambda z: relu(z), x))\n",
    "    fig = plt.figure(figsize=(8, 5))\n",
    "    ax1 = fig.add_subplot(2,1,1)\n",
    "    ax1.plot(x, y_sig, label='sigmoid')\n",
    "    ax1.plot(x, y_tanh, label='tanh')\n",
    "    ax1.plot(x, y_relu, label='relu')\n",
    "    ax1.grid()\n",
    "    ax1.legend()\n",
    "    ax1.set_xlabel('x')\n",
    "    ax1.set_ylabel('y')\n",
    "    ax1.set_ylim([-2, 2])\n",
    "    ax1.set_title('Activation Functions')\n",
    "\n",
    "    y_sig_derv = sigmoid(x, deriv=True)\n",
    "    y_tanh_derv = tanh(x, deriv=True)\n",
    "    y_relu_derv = list(map(lambda z: relu(z, deriv=True), x))\n",
    "    ax2 = fig.add_subplot(2,1,2)\n",
    "    ax2.plot(x, y_sig_derv, label='sigmoid')\n",
    "    ax2.plot(x, y_tanh_derv, label='tanh')\n",
    "    ax2.plot(x, y_relu_derv, label='relu')\n",
    "    ax2.grid()\n",
    "    ax2.legend()\n",
    "    ax2.set_xlabel('x')\n",
    "    ax2.set_ylabel('y')\n",
    "    # ax2.set_ylim([-2, 2])\n",
    "    ax2.set_title('Activation Functions Derivatives')\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAFgCAYAAABUnBiIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXwV9fX4/9e5Nzf7AiQQVgmoKEsE2cS6EBGp4k7dtW5Va7+1tfaD4vLR4m4r+unmvpT2V8WNqqhYLWoAcWETFUUqOyGyBEjIntx7z++PmYQbSEKAJJPknufjcb0z8555z5n3vXJP3vOeGVFVjDHGGGOihc/rAIwxxhhjWpMlP8YYY4yJKpb8GGOMMSaqWPJjjDHGmKhiyY8xxhhjooolP8YYY4yJKpb8GGOaTEQuFZH3W6juJ0Xkzpao22sicoKIrPQ6DmOMQ+w+P8ZEBxHJBYYC3VW1sgnrZwFrgYCqBps5liuBa1T1+Oast4F9TQXuACKP+R5V/UML7lOBw1V1VUvtwxhz4Kznx5go4CYyJwAKnOVpMN54WVWTI14tlvgYY9o+S36MiQ6XA58B04ErIgtEJEFEHhGR9SJSJCIfi0gCMM9dpVBESkTkWBG5UkQ+drd7UkSm7VHXmyLyW3f6VhFZLSLFIvKtiJzrLh8IPAkc69Zb6C6fLiL3RdR1rYisEpEdIjJLRHpGlKmIXC8i34vIThF5TERkfxtFRNaJyPiI+aki8k93OsvdzxUiskFECkTkjoh1/SJye8QxLhGRPiJS025fusd3oYjkiEhexLYDRSRXRApF5BsROSuibLp7PO+49X4uIoe6ZSIi/yciW93P6isRGbK/x21MtLPkx5jocDnwgvv6sYhkRpRNA0YAPwK6ALcAYeBEt7yT21vy6R51vghcWJN0iEhnYALwklu+Gqe3KQ24G/iniPRQ1RXA9cCnbr2d9gxWRMYBDwIXAD2A9RH11jgDGIVzKu8C4MdNb479cjxwBHAycJebvAH8FrgYmAikAlcDZapa025D3eN7ObIyEQkAbwHvA92AXwEviMgREatdjNNmnYFVwP3u8gk4n8sAoBNwIbC9+Q7VmOhgyY8xHZyIHA/0BV5R1SU4ScklbpkP50f7RlXdpKohVf2kKWOCgPk4p9FOcOfPw0lo8gFU9VVVzVfVsJsAfA+MbmLYlwLPq+pSN5bbcHqKsiLWeUhVC1V1A/ARMKyR+i5we1lqXj0bWXdPd6tquap+CXyJk2wBXAP8r6quVMeXqtqURGQMkOzGX6WqHwJv4yQ8Nf6lqgvdsVYvRBxbNZACHIkzZnOFqv6wH8dijMGSH2OiwRXA+6pa4M6/yO5TXxlAPE5CtF/UuVriJXb/aF+C80MNgIhcLiLLahIOYIi7v6boidPbU7OvEpwejl4R62yOmC7DSSga8oqqdop45Tcxjsb204cDaDecY9uoquGIZetpwrG5idJfgceALSLytIikHkAMxkQ1S36M6cDcsTsXAGNFZLOIbAZuAoaKyFCgAKgADq1n86ZcCjoDOE9E+gLHADPd/fYFngFuANLdU1vLgZpxOfuqOx+nt6rmOJKAdGBTE2LaH6VAYsR89/3YdiP1t9u+5AN93F63GofQxGNT1T+r6ghgMM7pr5sPIAZjopolP8Z0bOcAIWAQzqmTYcBAnFNWl7u9D88Dj4pIT3cQ77EiEgdswxn707+hylX1C3e9Z4H3VLXQLUrCSXC2AYjIVTg9PzW2AL1FJLaBql8ErhKRYW4sDwCfq+q6/W2AfVgGXCQiAREZiXPqrqmeBe4VkcPdgchHiUi6W7aFhtvtc5yk6xZ3vznAmew9pmkvIjJKRI5xxw2V4iSuof2I2RiDJT/GdHRXAH9T1Q2qurnmhXPq5FIRiQEmA18Di4AdwO8Bn6qW4Qy0XeCeuhrTwD5mAONxEhYAVPVb4BHgU5xEIBtYELHNh8A3wGYRKWAPqvoBcCdOT9IPOD0sFx1gGzTmTrfunTgDjF9sfPU6HgVewRm4vAt4Dkhwy6YCf3fb7YLIjVS1Cud2A6fh9Lw9jpOIfteEfabi9KjtxDlVth1nwLoxZj/YTQ6NMcYYE1Ws58cYY4wxUaXNJj/uzcI+EpEV7k3AbvQ6JmOMMca0f232tJeI9AB6qOpSEUkBlgDnuGMJjDHGGGMOSJvt+VHVH1R1qTtdDKyg7n0wjDHGGGP2W4zXATSFe1fXo3EuEd2z7DrgOoCEhIQRffr0adXY2oNwOIzP12bz3A7L2t0b1u6tqyxcxs7QTsIaJtmfTJo/DV/b/bu6w7Hve8P++9//Fqhq1/rK2uxprxoikgzMBe5X1X81tu7IkSN18eLFrRNYO5Kbm0tOTo7XYUQda3dvWLu3jo3FG7n/s/tZkL+AQemDOD1wOpf/+HKvw4o69n1vmIgsUdWR9ZW16Z4f90ZeM4EX9pX4GGOMaXnVoWqmfzOdp756ihhfDLeOvpWLjriI+fPmex2aMU3WZpMf90nRzwErVPVRr+Mxxphot3jzYu797F7WFK3hlL6nMGXUFDKTMr0Oy5j91maTH+A44KfA1yKyzF12u6rO9jAmY4yJOoUVhTyy5BHeWPUGPZN68tjJj3Fi7xO9DsuYA9Zmkx9V/ZjdD0E8YNXV1eTl5VFRUdEMUbVPaWlprFixotF14uPj6d27N4FAoJWiMsa0darKm6vf5JHFj1BSVcLVQ67m50f9nMRA4r43NqYNa7PJT3PJy8sjJSWFrKwsnDNp0ae4uJiUlJQGy1WV7du3k5eXR79+/VoxMmNMW7WmaA33fnovi7csZljXYdx57J0M6DzA67CMaRYdPvmpqKiI6sSnKUSE9PR0tm3b5nUoxhiPVQQreObrZ3h++fMkxCTwu2N/x6TDJ+ETu5zadBwdPvkBLPFpAmsjY8wn+Z9w32f3sbF4I2f0P4PJIyeTnpDudVjGNLuoSH6MMcY0rKC8gD8s+gPvrn2Xvql9eWbCM4zpMcbrsIxpMdaP6YFrrrmGb79t2UeUTZw4kcLCwr2WT506lWnTprXovo0x7UNYw7yy8hXOev0s5qyfwy+G/oKZZ820xMd0eNbz44Fnn322xfcxe7bdEcAY07CVO1Zyz6f38FXBVxzT/RjuGHMH/dLsggcTHaznp4WVlpZy+umnM3ToUIYMGcLLL79MTk4ONY/heO655xgwYAA5OTlce+213HDDDQBceeWV/OIXv+Ckk06if//+zJ07l6uvvpqBAwdy5ZVX1tY/Y8YMsrOzGTJkCFOmTKldnpWVRUFBAQAPP/wwRxxxBOPHj2flypWtd/DGmDanrLqMaYumceHbF5JXkscDxz/AMxOescTHRJWo6vm5+61v+DZ/V7PWOahnKr87c3CD5f/+97/p2bMn77zzDgBFRUU88cQTAOTn53PvvfeydOlSUlJSGDduHEOHDq3ddufOnXz44YfMmjWLM888kwULFvDss88yatQoli1bRrdu3ZgyZQpLliyhc+fOTJgwgTfeeINzzjmnto4lS5Ywc+ZMvvjiC4LBIMOHD2fEiBHN2gbGmPbhow0f8cDCB9hcupmfHP4TbhpxE2lxaV6HZUyrs56fFpadnc2cOXOYMmUK8+fPJy1t9z80CxcuZOzYsXTp0oVAIMD5559fZ9szzzwTESE7O5vMzEyys7Px+XwMHjyYdevWsWjRInJycujatSsxMTFceumlzJs3r04d8+fP54wzziAxMZHU1FTOOuusVjluY0zbsbl0Mzd+eCO//ujXJAeS+cdp/2Dqj6Za4mOiVlT1/DTWQ9NSBgwYwJIlS5g9eza33XYbEyZMqC1T1Ua3jYuLA8Dn89VO18wHg0FiYpr28dll7MZEp2A4yIsrXuSxZY8R1jC/Gf4bLh98OQGf3cndRDfr+Wlh+fn5JCYmctlllzF58mSWLl1aWzZ69Gjmzp3Lzp07CQaDzJw5c7/qPuaYY5g7dy4FBQWEQiFmzJjB2LFj66xz4okn8vbbb1NeXk5xcTFvvfVWsxyXMaZt+3rb11z8zsU8vPhhRmSO4PWzX+dn2T+zxMcYoqznxwtff/01N998Mz6fj0AgwBNPPMHkyZMB6NWrF7fffjvHHHMMPXv2ZNCgQXVOi+1Ljx49ePDBBznppJNQVSZOnMjZZ59dZ53hw4czadIkhg0bRt++fTnhhBOa9fiMMW1LcVUxf176Z15e+TJdE7ryyNhHOKXvKdYDbEwE2depl/Zk5MiRWnMVVY0VK1YwcOBAjyLat5KSEpKTkwkGg5x77rlcffXVnHvuuc26j30926tGW2+r9iY3N5ecnByvw4g60druqsp769/jDwv/QEF5ARcfeTG/OvpXJMcmt8r+o7XdvWbt3jARWaKqI+srs54fj02dOpU5c+ZQUVHBhAkT6lypZYwxTbGxeCP3f34/CzYtYGCXgfxl3F8YnNH6YxyNaS8s+fGY3W3ZGHOgqkPVTP9mOk999RR+8TNl1BQuOvIiYnz2T7sxjbH/Q4wxph1asmUJ9356L6uLVjP+kPFMGT2F7kndvQ7LmHbBkh9jjGlHCisKeXTJo7y+6nV6JvXkr+P+ytg+Y/e9oTGmliU/xhjTDqgqs1bP4pHFj7CrahdXDbmK64+6nsRAotehGdPuWPJjjDFt3JqiNdz32X0s2ryIoV2HctexdzGg8wCvwzKm3bKbHLawwsJCHn/88QPePvIhqMaY6FIRrOCvX/yVn8z6Cd/t+I67jr2Lf5z2D0t8jDlIbTb5EZHnRWSriCz3OpaDcbDJjzEmOn2S/wmTZk3iqa+e4sdZP2bWObM4f8D5+KTN/rNtTLvRlv8vmg6c6nUQB+vWW29l9erVDBs2jJtuuomTTz6Z4cOHk52dzZtvvgnAunXrGDhwINdeey2DBw9mwoQJlJeX19bx6quvMnr0aAYMGMD8+fO9OhRjTCsoKC/glnm38PP//Byf+Hj6lKd56ISHyEjI8Do0YzqMNjvmR1XniUhWs1b67q2w+etmrZLu2XDaQw0WP/TQQyxfvpxly5YRDAYpKysjNTWVgoICxowZU/uU9e+//54ZM2bwzDPPcMEFFzBz5kwuu+wyAILBIAsXLmT27NncfffdzJkzp3mPwRjjubCGee2/r/HHJX+kIlTB9UOv55rsa4jzx+17Y3NwVJ0XEe81y6mnrNF3DmDbBgNrPGYgvnwL7Fzf9O0itj3Qfe63xrZL6w2B+AOr9yC02eSnI1JVbr/9dubNm4fP52PTpk1s2bIFgH79+jFs2DAARowYwbp162q3mzRpUr3LjTEdw8odK7nns3v4attXjOo+ijvH3Em/tH5eh9U0oSAEywlUFcLOdVBdDtVl7nvNdIXzHqyAUJX7CkZMV+/x7k6HI5aHgxAOgYbcd42YjnwPg4brWRZylkeuv68koR0YA/C511EcuFVnv8VhR5/Y6vtt98mPiFwHXAeQmZlJbm5unfK0tDSKi4udmePvaJkgauqvR0lJCeFwmOLiYl544QV++OEHcnNzCQQCDBkyhIKCAgACgUBtnMFgkNLSUoqLiwmFQgSDQYqLiykvL6e6unr38TRRKBRq0jYVFRV7tZ85cCUlJdaeHmhP7V4ZruTdonf5aNdHJPoS+Wn6TxkVO4r1X6xnPXv+Nd9yJFxNoHoXsVVFBKp3EaguJFBdTEyw1H2VERMsxR8qqzMfEyzDp9UAHAfwyf7tNywxqMQQ9jX+7rx8qPgI4yNELCF8hFScd3yE1EcQ9yU+QghBnxBUH0FfRLmKM63O9kGFsEIIqTuvQigMoZppFZy+msh3cdMnqbOcetZjj232nG7ws6l5ScS7OGNWnO0Uv8+313rUri8I4HN3UbMt7vL6ton8b+TzcMVdSerMR64XsV1kfY2Uddu4g7yi3AaPv6W0++RHVZ8GngbnwaZ7PuBtxYoVTXqoZ0vp0aMHpaWlpKSkUFlZSc+ePenSpQsfffQRGzZsIDnZeeigz+erjTMuLo7q6mpSUlLw+/0kJSXVbi8i+308TX2waXx8PEcfffT+H6Splz1w0Bvtpd1zN+byyOeP8EPpD/zk8J9w04ibSItLa96dhENQshV2bXJeRZt2T+/6AUq3QWkBVBY1XEcgCeLTID4VklIhvivEpTrzcakQlwKBBP67No8Bg4dCIAECiRBIQGMSqJA4ysKxlIQDlAT9FAd9FFcLxVVQWhWiuDJISUWQ0srg7umqICWVIcqrgpRXhyivClNRHaK8OkQovP+9NX6fEOv3ERfwEev3ERvjIzbgIy7GT2yMj7iaZTE+4tz3mvVq1nGWCTF+HzE+IeD3EeMXYnxCjM+ZDvh9+H1CwL97WW2ZL2L9+urw+/CL4POBXwS/T5DIzKMB7eX73ta0++SnrUtPT+e4445jyJAhjBo1iu+++46RI0cybNgwjjzySK/DM8a0ss2lm3lo4UN8sOEDDut0GH8/9e8Mzxx+4BWGQ1CUBztWw3b3VTNduN45XRQpJh5Se0FqT+g5DBIzIMl9JWZAUld3Oh3iUlGfn9KqEEXl1RSWVVFUXs2u8moKy6opKndehTuqWbVtE3FLOkcsr6C4opim5Cp+n5AcF7P7FR9DWkKAHqnxJMT6iQ/4SQj4SYz115lPiPWREIghIdadr1kWG0NCwE+8m+D4fftOIkx0abPJj4jMAHKADBHJA36nqs95G9WBefHFF/e5zvLlu6/onzx5cu10ZPd9RkaGjfkxpp0KhoPM+G4Gf/3ir4Q1zI3Db+SKQVcQ8AeaXkn5TtjyDWxeDlvc19YVzliaGoFE6HIodB8Cg85yBpSm9naSnbTeaHwnSqpCbC+pYntpFdtLKtlRWsX24iq2/1DF9tJKdpTuoKBkMztKnbLqUMMZTIxPSEsIECBMj9gg6cmxHNo1ibSEAKkJAZIikxo3samZToqLISU+hrgYX5N6OYxpLm02+VHVi72OwRhjmsPyguXc8+k9rNixguN7Hc8dx9xB75TejW8UqnauTs1bDHkLYeNCpyenRmI6ZA6BkT+DrgMIdz6UnQmH8EMoja0llWzdVcmWXZVsyatg665KthbvYlvxF2wvqaIqFK53l0mxftKT4+iSFEuvTvFk90qlS1IcXZICpCXUvGKd98QAnRICJMb6ERH39MtxzdhqxrScNpv8GGNMe1dcVcyfl/6Zl1e+TEZCBtPGTmNC3wn193IEq2DTYliTC+s+hk1LIejc70uTu1OeOZyC/heyMbY/KzWL1eVJbCmuYtuqCrYsrWRbSTGh8N73hE1PiqVbajzdUuIYkJlCenIsGUlOgpOeHEt6UhxdkmNJT4olPuBv4RYxpm2w5McYY5qZqvLe+vf4w8I/UFBewEVHXsSvjv4VKbF7XHiw7b/o9+8R/P4jfBs/xR8sI4yPzUlH8l3y6SwJH8b88iy+3p6CFkQmTDvonFhMZmo83VLjGZCZQrfUOGc+JZ7M1Di6pcbTNTmO2Ji2fC9bY7xhyY8xxjSjjcUbuf/z+1mwaQEDuwzkz+P+zGFpA8nbWc7i7ZspW/MZnTbMof+OXHpUb0SA9eGeLAgfx4LwED4LD6IimEKvTgn07BTPEX0SOKlTgjvvLOvZKcF6aYw5CJb8GGPMQVJVNheX8vSXf+ONtdMRfBwZuAzNP45rntpMz5L/cI5/Aaf6F9FNCqlWP18Fsvkw/Ry29RxHSmY/enWK54ZOCTzQKYH0pFgbAGxMC7LkxxhjmqiorJq120tZs62EtQWlrCkoZe22UtaXLkczXsMft5XqXYOp3HIWJFRzYewL5JBLRlw+IV8cRYeMZ9egM0nJPpURCZ0Z4fUBGROlLPlpI3Jycpg2bRojR470OhRjolpFdYj128tYW1BSm9ysLXBe20uratfz+4ReXcL4M2bj6zSflJiuXJL1O87TUrqt/Du+jZ9B0Af9xsJRd+I/8gy6xKd6eGTGmBqW/LQiVUVV8flsAKIxXgqFlfzCctYU7O7FWVtQypptpeQXldd5DmO3lDj6ZSQxYXAm/TKS6JeRTFZ6Il8Xfcgflz7CzqpdXNXvHK4vrSbx45uhfAd06Q+n3APZF0BqD+8O1BhTL0t+Wti6des47bTTOOmkk/j000/5zW9+w5NPPkllZSWHHnoof/vb32ofcVEjOTmZkpISAF577TXefvttpk+f7kH0xrRvhWVVrN7mJDg172sLSlm/vazOvW5S4mLo1zWJkVmd6ZfRm/5dk+mfkURWRhLJcXX/mVxTtIb7PruNRZsXcVRqf+4KpXPEB38Bnx+OmAgjr3Z6e+yPHGParKhKfn6/8Pd8t+O7Zq3zyC5HMmX0lEbXWblyJX/729+45557mDRpEnPmzCEpKYnf//73PProo9x1113NGpMx0SQYCrNxZzmrt5awpqCEj5dX8th3n7BmW93TVAG/cEiXRPp3TWbcwG70d3tx+mUkkZG87wHGlaFKnvnqGZ5f/jzx4ufOUArnfZmLL6EznHgzjPoZpHRv6cM1xjSDqEp+vNK3b1/GjBnD22+/zbfffstxxzl3Qa2qquLYY4/1ODpj2of6enHWFJSyfntpnccvpMTCkT3hlEGZ9O+aRP+MZA7tlkyfzgnE+A+sN+bT/E+577N72VC8kYlVws0/rCUj7RCYOA2GXQKxSc11mMaYVhBVyc++emhaSlKS8w+jqnLKKacwY8aMRteP/Au0oqKikTWN6Vj27MVZs62U1dtK6u3F6ZueRP+MJMYPzOTQrkn075rMoV2TWLbwE3JyftQs8RSUF/Dwwj8we927HBKCp7Zu5UedBsBP7oaBZzmnuowx7U5UJT9eGzNmDL/85S9ZtWoVhx12GGVlZeTl5TFgwIA662VmZrJixQqOOOIIXn/9dVJSUhqo0Zj2JxxWNu+qYF1BKeu2l7Fue81g4xI27Cir04uTnhRL/65Jtb04h3ZNpn/Xg+vFaVKMGua1la/yx8XTKA9W8PPCIq6N60PcOQ8543rsHjzGtGuW/LSirl27Mn36dC6++GIqKysBuO+++/ZKfh566CHOOOMM+vTpw5AhQ2oHPxvTXtQmONtLWVdQxno3wVm33RlsXBncPdg4NsbHIV0SObRrMhMGd6d/xu5enE6Jsa0e+8odK7k3dzJfFq9jVHkF/6td6D/hfjjyDBvEbEwHYclPC8vKymL58t0PGxw3bhyLFi3aa73c3Nza6fPOO4/zzjuvNcIz5oBVBcPkF5azcWcZG3eU1yY467eXsX5HKRXVEQmO38ch6YlkpSdx4uFdycpIol9GEn3TE+mRloDf531PSll1GU9+ci//WPc2qaEQ91f4OfP4B5AhP7Gkx5gOxpIfY0y9Qm7vzcYdZeTtLGfjjjI27iwjb4eT8GzeVVHnfji7E5xETjg8g6yMJLLSk8jKaDsJTkNyV/6LBz5/gB+0knPLqvjtkGvpdOyvICbO69CMMS3Akh9jolRZVZD8wgo2F1WQX1TOD4UV/FBU7iQ6O8vILyyvM/5GBDJT4unTJYFj+6fTu0sifTon0LtzIn26JLT5BKc+m3dt5KH3rueDsg0cWlXN9O45jLjgQUjo7HVoxpgWFBXJj6raQwL3QSP/hDftXkllkK27KvihyH0VlpNfVMHmovLaZUXl1Xttl54US+8uiQzplcZpQ3rQp0sCfTon0qdLIj07xRMX0zGubgqGg8xYcB9/XT2TkIa5MdCTKyY9TiD9cK9DM8a0gg6f/MTHx7N9+3bS09MtAWqAqrJ9+3bi4+O9DsU0IhRWtpdUsrW4km01r5JKtu6qcN+d+W3FlZRVhfbavktSLD3S4undOYFRWV3o0SmeHmnx9EhLoEdaPJmp8cQHOkZy05jl6+dyz7xbWBEu47gg3DH6DvocdYnXYRljWlGHT3569+5NXl4e27Zt8zoUz1RUVOwzsYmPj6d3796tFJEB5wGaO0qr2FFaxc6yqt3TpVXsKKtiZ2l1bVlBSRU7SisJ19NBlxIfQ7eUOLqmxDG0dye6utPdUuJqE5vuadGR2DSmuHwnf3n/l7y08yvSw2Ee7nYiP57wKBKb6HVoxphW1uGTn0AgQL9+/bwOw1O5ubkcffTRXofR4agqFdVhdlVUs6u82n0P1s4vW13Fp2Ur6izfGZHUlFfv3TsDztiaTgkBOifF0iUxlkO6JDKsT6faBKdrSnxtctM1JS7qk5p9UVXeX/oEv//6SQoIc6F04tcTHyel+1Feh2aM8UiHT36MqaGqVAbDlFYGKasKUVIZpKwqSGllqPa9dI/5sqqgu16oznY1yU7kgOD6xK9bR2p8gNSEAKnxMWQkxzEgM4X0pNja5KZzUixdkmLpnOi8pyUE2t3A4bYqb9s33P/+/+Pj4A6ODCl/GvJzskffYDcpNCbKtenkR0ROBf4E+IFnVfUhj0MyByAcVqrDYapDSnUwTHUoTFXInQ+FqXKX1c6Hwu56u+crg2Eqq0NUVIeoqA4778Hd05VB97067C6PWM8tj7yx3r74BJJiY0iM89d575oSR1ZGEmkJMRFJTYDUOvMxpCYE+GLhJ5wy7qQWbFnTkOrqSv7+wW95avNcRJVb0rK5+LQniLGruIwxtOHkR0T8wGPAKUAesEhEZqnqt60dy9ZdFRRXBlF1eg/CCmFVNOJdiVxeU+bMh50Vdpeze71wmDrze23nloXDu7etiUEj9hfeI7ZQWAmpEgopq9dWsaRqJcGwEg4rwbBbHrFOMOzsb/c6YUJhCIXDu8tCe64TUY877yQxNQmNMx+sb6DKQfAJxAf8xAf8xMX46rzHB3x0SYolPsZPXMBHfIyzLD7gJ84tT4qNITHWT3JcDIlxMSTF+kmMjXHnnSQnPuA76AHyAeu98cSmnR9zwQs3sUqCjCOB206aRvessV6HZYxpQw46+RGRG4AXVHVnM8QTaTSwSlXXuPt5CTgbaPXk55I3bmJzcGFr77b5+HDSx9rfYqGhn2Uhcj132h1SIvV9W6T+uvyAXyB+j+pA9jrjIHvMyB4lkTHtua8wUOa+9hJyX5X1Fba8UCiE/wUbj9N6FA1VUa4heoTD/PnQCzjpxLvsFJcxZi/N0fPTHadXZinwPPCeNs9NY3oBGyPm84Bj9lxJRK4DrgPngaCRj4loLkfF9VkPfgAAACAASURBVKWz+8sv7P63NCKXqJ2OfI/8N1fqW0/2zjNqlu1VX6N1yV51+SLqqQ5WExsINPVwTTOprq4mYO3eKhLLNtGpcDn+cCVx8f0Y1PeXiGaQO3eu16FFjZKSkhb599c0ztr9wEhz5CninB+YAFwFjAReAZ5T1dUHUef5wI9V9Rp3/qfAaFX9VUPbjBw5UhcvXnygu+ywcnNzycnJ8TqMqGPt3goKvod3/gfWzoUew+CMR8n9vtja3QP2ffeGtXvDRGSJqo6sr6xZntbn9vRsdl9BoDPwmoj84SCqzQP6RMz3BvIPoj5jTEdRVQpz7obHj4X8ZTBxGlz7IfQa4XVkxph2oDnG/PwauAIoAJ4FblbVahHxAd8Dtxxg1YuAw0WkH7AJuAiw27AaE81UYcUs+PftsCsPjroIJtwLyd28jswY0440x5ifDGCSqq6PXKiqYRE540ArVdWgO5j6PZzxs8+r6jcHF6oxpt0qWAXv3gyrP4TMIfCTZ6HvsV5HZYxphw46+VHVuxopW3GQdc8GZh9MHcaYdq6qFOZNg0/+AoEEOPX3MOoa8LfZO3UYY9o4+9fDGNM2qcI3/4L373JOcQ29GMbfDSmZXkdmjGnnLPkxxrQ9GxfBe7dD3kL3FNcz0PdHXkdljOkgLPkxxrQdhRucq7iWvwZJ3eCsv8CwS8FnN4s0xjQfS36MMd4rL4QFf4RPH3fu6HnizXDcjRCX4nVkxpgOyJIfY4x3Kkvg8yfhkz9DRREcdSGcfBek9fY6MmNMB2bJjzGm9VVXwOLnYP6jUFYAA06Fk+6AHkd5HZkxJgpY8mOMaT1VpbD0/4MFf4LifOg3FsbdCX1GeR2ZMSaKWPJjjGl5ZTtg4TPOKa7yHXDIsTDpKeh3oteRGWOikCU/xpiWU7gBPnsSlkyH6lIYcBoc/xs4ZIzXkRljopglP8aY5qUKa3Kdnp7/vgsIZJ/vXL2VOcjr6IwxxpIfY0wzKdsBX7/qJD3bv4fEdDj+JhhxFXTq43V0xhhTy5IfY8yBC1XDqg9g2Qvw339DqAp6jYRzn4JB50Ag3usIjTFmL5b8GGP2TzgMGz+Hb9907sRcug0SM5yHjQ692C5XN8a0eZb8GGP2LRSE9R/Dt7Pgu7ehZAv442DABBh6CRx+CvgDXkdpjDFNYsmPMaZ+RZtg9Yew+gNnAHP5TggkOonOwLNgwI/t8RPGmHbJkh9jjKO0ADZ8Bus/cRKebd85y5O7O5eoHzkRDj0ZYhO9jdMYYw6SJT/GRKNwCLavcsbubPgcNn7mzINzOqvvsc7T1A87GboNch42aowxHYQlP8Z0dFVlsG0F/PAVbP4aNn8FW76B6jKnPKEz9BkDR1/m3Hm5xzC7SssY06FZ8mNMRxAKOs/K2r4KCr53Xtu/h4JVsCtv93pxadA9G4Zf4VyV1WsEpB8OPp93sRtjTCuz5MeYti5Y6VxOXroNSrZCUR4UbXTf3deufNDQ7m3iUiH9MMg6zkluug6A7kdB5yw7hWWMiXptMvkRkfOBqcBAYLSqLvY2ImOaQSgIFUVQUei+1/cqdJOcbVC61ZmuKNq7Ll8MpPaCtD6QdTyk9XZe6Yc5yU5yN0tyjDGmAW0y+QGWA5OAp7wOxLQDqqBh5xUOudOhiGXhusvCIWc6FHTuSByqgnDEdKjafVVFvNezTrDCGU9TXQbV5e67O11Vyuii7bBUdi8PVjR+HOKD+DRI6gpJ3SBziJPEJHWDpIzd02m9IDkTfP7WaV9jjOlg2mTyo6orAKSt/OWa+xCsnedMq0YUaMPL9nd5c9RRZ3L3spElxfBtsvfx7TNmd6ZOEhOZtNSXxIT3rKD1iB9ikyCQ4Nz/JpDoTMcmQmpPSkKpJPbK2r0skAQJnZwEp75XbLL11hhjTCtok8nP/hCR64DrADIzM8nNzW32ffRdt45ORZGnHqSBaYdK4+V7Lt+9/p7vja3bcH010+ouCgXiKQ/HNLDubrV1Sz3r7LHswI+x8ZiddX3u+j5UfIBELPOjInWWO+tFTkduE7meoOKvrTvsi0ElZq/3+pbtvY4f9TX+v09JagnJycl1F5a7LwCqgG3uyzSXkpKSFvl3wDTO2t0b1u4HxrPkR0TmAN3rKbpDVd9saj2q+jTwNMDIkSM1JyeneQKsoyXqbD25ubm0TLuYxli7e8Pa3RvW7t6wdj8wniU/qjreq30bY4wxJnrZzT2MMcYYE1XaZPIjIueKSB5wLPCOiLzndUzGGGOM6Rja5IBnVX0deN3rOIwxxhjT8bTJnh9jjDHGmJZiyY8xxhhjooolP8YYY4yJKpb8GGOMMSaqWPJjjDHGmKhiyY8xxhhjooolP8YYY4yJKpb8GGOMMSaqWPJjjDHGmKhiyY8xxhhjooolP8YYY4yJKpb8GGOMMSaqWPJjjDHGmKhiyY8xxhhjooolP8YYY4yJKpb8GGOMMSaqWPJjjDHGmKhiyY8xxhhjooolP8YYY4yJKpb8GGOMMSaqtMnkR0QeFpHvROQrEXldRDp5HZMxxhhjOoY2mfwA/wGGqOpRwH+B2zyOxxhjjDEdRJtMflT1fVUNurOfAb29jMcYY4wxHUeM1wE0wdXAyw0Vish1wHXubImIrGyVqNqXDKDA6yCikLW7N6zdvWHt7g1r94b1bahAVLU1A9m9Y5E5QPd6iu5Q1Tfdde4ARgKT1KtAOwARWayqI72OI9pYu3vD2t0b1u7esHY/MJ71/Kjq+MbKReQK4AzgZEt8jDHGGNNc2uRpLxE5FZgCjFXVMq/jMcYYY0zH0SYHPAN/BVKA/4jIMhF50uuA2rmnvQ4gSlm7e8Pa3RvW7t6wdj8Ano35McYYY4zxQlvt+THGGGOMaRGW/BhjjDEmqljyE2VEZLKIqIhkeB1LNLBHtbQuETlVRFaKyCoRudXreKKBiPQRkY9EZIWIfCMiN3odUzQREb+IfCEib3sdS3tiyU8UEZE+wCnABq9jiSL2qJZWIiJ+4DHgNGAQcLGIDPI2qqgQBP5HVQcCY4BfWru3qhuBFV4H0d5Y8hNd/g+4BbBR7q3EHtXSqkYDq1R1japWAS8BZ3scU4enqj+o6lJ3uhjnh7iXt1FFBxHpDZwOPOt1LO2NJT9RQkTOAjap6pdexxLFrgbe9TqIDqwXsDFiPg/7EW5VIpIFHA187m0kUeOPOH/Qhr0OpL1pkzc5NAemsUeGALcDE1o3ouiwH49qCQIvtGZsUUbqWWa9nK1ERJKBmcBvVHWX1/F0dCJyBrBVVZeISI7X8bQ3lvx0IA09MkREsoF+wJciAs6pl6UiMlpVN7diiB2SPaqlzcgD+kTM9wbyPYolqohIACfxeUFV/+V1PFHiOOAsEZkIxAOpIvJPVb3M47jaBbvJYRQSkXXASFW1JwG3MPdRLY/iPKplm9fxdGQiEoMzqPxkYBOwCLhEVb/xNLAOTpy/qP4O7FDV33gdTzRye34mq+oZXsfSXtiYH2Nalj2qpZW4A8tvAN7DGXT7iiU+reI44KfAOPc7vsztjTCmzbKeH2OMMcZEFev5McYYY0xUseTHGGOMMVHFkh9jjDHGRBVLfowxxhgTVSz5McYYY0xUseTHGGOMMVHFkh9jjDHGRBVLfowxHYKIjBKRr0QkXkSSROQbERnidVzGmLbHbnJojOkwROQ+nOccJQB5qvqgxyEZY9ogS36MMR2GiMTiPNOrAviRqoY8DskY0wbZaS9jTEfSBUjGeZ5avMexGGPaKOv5McZ0GCIyC3gJ6Af0UNUbPA7JGNMGxXgdgDHGNAcRuRwIquqLIuIHPhGRcar6odexGWPaFuv5McYYY0xUsTE/xhhjjIkqlvwYY4wxJqpY8mOMMcaYqGLJjzHGGGOiiiU/xhhjjIkqlvwYY4wxJqpY8mOMMcaYqGLJjzHGGGOiiiU/xhhjjIkqlvwYY4wxJqpY8mOMMcaYqGLJjzHGGGOiiiU/xrRhInKpiLzfQnU/KSJ3tkTdXhORE0RkpddxHCwROUREStyn1Dd33S323TKmrbOnuhvTQkQkFxgKdFfVyiasnwWsBQKqGmzmWK4ErlHV45uz3gb2NRW4A4g85ntU9Q8tuE8FDlfVVS21jybEkAuMAaoBBb4HXgX+rymff0tqye+WMe2R9fwY0wLcH5sTcH4Ez/I0GG+8rKrJEa8WS3zamBtUNQXoAfwPcBEwW0RkfysSkZjmDs4Y47Dkx5iWcTnwGTAduCKyQEQSROQREVkvIkUi8rGIJADz3FUK3VMdx4rIlSLysbvdkyIybY+63hSR37rTt4rIahEpFpFvReRcd/lA4EngWLfeQnf5dBG5L6Kua0VklYjsEJFZItIzokxF5HoR+V5EdorIYwf4g75ORMZHzE8VkX+601nufq4QkQ0iUiAid0Ss6xeR2yOOcYmI9BGRmnb70j2+C0UkR0TyIrYdKCK5IlIoIt+IyFkRZdPd43nHrfdzETnULRMR+T8R2ep+Vl+JyJB9HaeqlqpqLk7ieyxwulufL+Jz2i4ir4hIlz2O/2cisgH4MGJZjIhcJCKL92jPm0Rkljt9uoh8ISK7RGSj2wNX42C/Wz1FZKaIbBORtSLy64j1RovIYne/W0Tk0X21jzFes+THmJZxOfCC+/qxiGRGlE0DRgA/AroAtwBh4ES3vJPbW/LpHnW+CFxYk3SISGdgAvCSW74ap7cpDbgb+KeI9FDVFcD1wKduvZ32DFZExgEPAhfg9Fqsj6i3xhnAKJxTeRcAP256c+yX44EjgJOBu9zkDeC3wMXARCAVuBooU9WadhvqHt/LkZWJSAB4C3gf6Ab8CnhBRI6IWO1inDbrDKwC7neXT8D5XAYAnYALge1NPRBV3QAsxvlcAH4NnAOMBXoCO4HH9thsLDCQvdt3FnCEiBwesewSnO8FQCnO964TTrL1CxE5xy074O+WiPhw2u9LoBfO5/IbEamJ70/An1Q1FTgUeKXRRjGmDbDkx5hmJiLHA32BV1R1CU5Scolb5sP50b5RVTepakhVP2nimJD5OKfRan5Iz8NJaPIBVPVVVc1X1bCbAHwPjG5i2JcCz6vqUjeW23B6irIi1nlIVQvdH/SPgGGN1HeB28tS8+rZyLp7ultVy1X1S5wf3KHu8muA/1XVler4UlWbkoiMAZLd+KtU9UPgbZyEp8a/VHWhOx7mhYhjqwZSgCNxxkiuUNUf9uNYAPJxklyAnwN3qGqe285TgfOk7imuqW7PUXlkJapaBrxZE7ebBB2JkxShqrmq+rX7+X8FzMBJpJqise/WKKCrqt7jtt8a4BmcU3rgtNFhIpKhqiWq+lkT92mMZyz5Mab5XQG8r6oF7vyL7D71lQHE4yRE+0WdqxNeYveP9iU4P9QAiMjlIrKsJuEAhrj7a4qeOL09Nfsqwenh6BWxzuaI6TKchKIhr6hqp4hXfhPjaGw/fTiAdsM5to2qGo5Ytp4mHJubKP0Vp3dmi4g8LSKp+7n/XsAOd7ov8HrEZ7QCCAGRPYMbG6nrRep+/m+4SREicoyIfOSemirC6e1r0ue/j+9WX6BnZDIL3B4R889wesa+E5FFInJGU/ZpjJcs+TGmGYkzducCYKyIbBaRzcBNwFARGQoUABU4pwf21JRLL2fg9BT0BY4BZrr77Yvz1/gNQLp7ams5UDMuZ1915+P8yNUcRxKQDmxqQkz7oxRIjJjvvh/bbqT+dtuXfKCP2+tW4xCaeGyq+mdVHQEMxvmRv7mpOxaRPjinOOe7izYCp+2RGMaramQsjX1W7wMZIjIMJ1F5MaLsRZxeoD6qmoYzzqupnz808N1yY167R8wpqjoRQFW/V9WLcU4p/h54zf3+GNNmWfJjTPM6B+cv+UE4p06G4YzfmA9c7vY+PA886g4i9buDT+OAbThjf/o3VLmqfuGu9yzwnqoWukVJOD9w2wBE5Cqcnp8aW4DeIhLbQNUvAleJyDA3lgeAz1V13f42wD4sAy4SkYCIjMQ5vdJUzwL3isjh7kDko0Qk3S3bQsPt9jlO0nWLu98c4Ez2HtO0FxEZ5faoBNw6KnA+331tlygiY3FOUy0EZrtFTwL3uwkGItJVRM7eV3013NNyrwEP45xK+09EcQqwQ1UrRGQ07qlW18F8txYCu0RkijiD9f0iMkRERrnHcJmIdHW/2zXb7LONjPGSJT/GNK8rgL+p6gZV3Vzzwjl1cqk7tmMy8DWwCOd0yO8Bn3v64n5ggXt6YUwD+5gBjCfir35V/RZ4BPgUJxHIBhZEbPMh8A2wWUQK2IOqfgDcifPX/g84PSwX7bleM7jTrXsnzgDjFxtfvY5HcQbTvg/sAp4DEtyyqcDf3Xa7IHIjVa3CuerqNJyet8dxEtHvmrDPVJwetZ04p8q24wxYb8hfRaQY5zP4I057nhpxyu1POL0z77vrfYbTy7I/XsT5/F/d4549/w+4x633LiIGHh/kdyuEkywOw7lXUAFOgpTmrnIq8I2IlLjHd5GqVuznMRnTquwmh8YYY4yJKtbzY4wxxpioYsmPMcYYY6KKJT/GGGOMiSqW/BhjjDEmqnSoB+dlZGRoVlaW12G0OaWlpSQl2W03Wpu1uzes3b1h7e4Na/eGLVmypEBVu9ZX1qGSn6ysLBYvXrzvFaNMbm4uOTk5XocRdazdvWHt7g1rd29YuzdMRNY3VNZip71E5HlxnoS8vIFyEZE/i/MU6a9EZHhE2akistItu7WlYjTGGGNM9GnJMT/TcW5+1ZDTgMPd13XAEwAi4sd5js5pOHfJvVhEBrVgnMYYY4yJIi122ktV5+3xROg9nQ38w32g3mci0klEegBZwCr3ycGIyEvuut+2VKzGmHautAA2LSW4K49vN7zPhv+8CcndILk7+PxeRxcVVu1axYZvNngdRtRp7+0+sf9EMhKa+vzl5uPlmJ9e1H16cZ67rL7lDd7+XUSuw+k5IjMzk9zc3D3LSUpKwu+P3n8AU1NT+eKLLxpdJxQKUVpait3xu/mUlJTs9X00zccXqqL75g/ovvkDUou/B2B5XCxP9OwO+d94HF2UsiGX3mjH7a55St+4vvtesZl5mfxIPcu0keX1UtWngacBRo4cqXsO/Fq7di0pKSmkp6cjUl/VHV9xcTEpKSkNlqsq27dvp7i4mH79+rViZB2bDURsIarwxT8h90HYtQkys2HUnXDIsVSHiuDjKfxlxK2MCAmsmQvfvQNVJXD4BBj/O+h0iNdH0CF9/PHHHH/88V6HEXXae7snxCQQ42v9VMTL5CcP6BMx3xvIB2IbWH5AKioqyMrKitrEpylEhPT0dLZt2+Z1KMY0rmgTzPoVrP4Aeo+Gc56AfidCzf/fmxcBkJQxgJTuo2DoJVCxCxY9A/MegafHwYT7YNQ1u7cxzSLBl0BKbMN/ZJmWYe1+YLy8yeEs4HL3qq8xQJGq/oDzpOvDRaSfiMTiPFl61sHsyBKffbM2Mm3ehs/h6bGw4VOYOA2ufg/6j62TxIRrH54eIT4VTvgfuGERZJ0AsyfDa1dDVVkrBm+MaUtarOdHRGYAOUCGiOQBvwMCAKr6JDAbmAisAsqAq9yyoIjcALwH+IHnVdVO4BsTzZb/C17/OaT1hivfga5H1LuaumfIpb6z52m94JJXYMH/wYf3QVEeXPIyJHZpyciNMW1Qi/X8qOrFqtpDVQOq2ltVn1PVJ93EB3X8UlUPVdVsVV0cse1sVR3glt3fUjF65ZprruHbb1v24rWJEydSWFi41/KpU6cybdq0Ft23Mc3qy5dh5s+g10i45oMGEx+gdsB+gz2ZPp/TC3T+dPhhGfxtIpTY6V5jok2HusNze/Hss8+2+D5mz57d4vswpsV9/ZrT49PvBLj4ZYhNbHT1Rnt+Ig06G+LT4MWL4J/nwhVvQ0Kn5oraGNPG2YNNW1hpaSmnn346Q4cOZciQIbz88svk5OTUPobjueeeY8CAAeTk5HDttddyww03AHDllVfyi1/8gpNOOon+/fszd+5crr76agYOHMiVV15ZW/+MGTPIzs5myJAhTJkypXZ5VlYWBQUFADz88MMcccQRjB8/npUrV7bewRtzMNbOg9evh74/alLiA9ReF9qkMWz9c+DCf8LW7+DFC6C64mCiNca0I1HV83P3W9/wbf6uZq1zUM9Ufnfm4AbL//3vf9OzZ0/eeecdAIqKinjiiScAyM/P595772Xp0qWkpKQwbtw4hg4dWrvtzp07+fDDD5k1axZnnnkmCxYs4Nlnn2XUqFEsW7aMbt26MWXKFJYsWULnzp2ZMGECb7zxBuecc05tHUuWLGHmzJl88cUXBINBhg8fzogRI5q1DYxpdltXwEuXQfqhcNGLTUt82I+enxqHj4efPAOvXulcRTbpabsKzJgoYD0/LSw7O5s5c+YwZcoU5s+fT1paWm3ZwoULGTt2LF26dCEQCHD++efX2fbMM89ERMjOziYzM5Ps7Gx8Ph+DBw9m3bp1LFq0iJycHLp27UpMTAyXXnop8+bNq1PH/PnzOeOMM0hMTCQ1NZWzzjqrVY7bmANWtsPpiQkkwKWv7dfpqNrkZ38SmMHnwrj/ha9fgY8f3d9ojTHtUFT1/DTWQ9NSBgwYwJIlS5g9eza33XYbEyZMqC3b192U4+LiAPD5fLXTNfPBYJCYmKZ9fHYZu2k3wmH413VQvBmu+jd06rPvbSI3dy91b3LPT40TJsO2lfDBPZA5BAb8eP+2N8a0K9bz08Ly8/NJTEzksssuY/LkySxdurS2bPTo0cydO5edO3cSDAaZOXPmftV9zDHHMHfuXAoKCgiFQsyYMYOxY8fWWefEE0/k7bffpry8nOLiYt56661mOS5jWsT8abDqP3DqQ9D7wE/P7nfyIwJn/QW6ZzsDrIs2HfC+jTFtX1T1/Hjh66+/5uabb8bn8xEIBHjiiSeYPHkyAL169eL222/nmGOOoWfPngwaNKjOabF96dGjBw8++CAnnXQSqsrEiRM5++yz66wzfPhwJk2axLBhw+jbty8nnHBCsx6fMc1m7Xz46AE46iIYefUBVbHPS90bE0iA86Y7N1Kc+TPnCjC//RNpTEckHelBliNHjtSaq6hqrFixgoEDB3oU0b6VlJSQnJxMMBjk3HPP5eqrr+bcc89t1n3s69leNdp6W7U39myv/VBRBI//CGLi4Pr5EJt0QNXkbszlVx/+ipdOf4nBGQd4mvurV+Ff18CJNztjgUyT2PfdG9buDRORJao6sr4yO+3lsalTpzJs2DCGDBlCv3796lypZUzUmH0zFP8Ak5454MQHIsbRHcwwt6POh2GXwvxHIG/JQVRkjGmrrE/XY3a3ZRP1lv8LvnoZcm47qHE+sPtqL9/B/l136oPOE+HfuB5+Ps85JWaM6TCs58cY452SrfD2TdBrhHPF1UE6oEvd6xOfBmf/BQr+6zwHzBjToVjyY4zxzru3QHUZnPNkswwurh3wfFDnvVyHjnMGXn/6GKz/5ODrM8a0GZb8GGO8sfJd+OZ1GHsLdB3QLFXW9Pw0m1Pude419NaNEKxs3rqNMZ6x5McY0/oqdsE7/wPdBsGPbmy2ag/qUvf6xCXD6Y86p78W/Kl56jTGeM6SnxZWWFjI448/fsDbRz4E1ZgO48N7YVe+c2PBmNhmq3a/n+3VFIefAoMnwbxpULCq+eo1xnjGkp8WdrDJjzEdTt5iWPgMHHM99K73FhwHrEWSH3Cu/oqJh3dugg50bzRjopUlPy3s1ltvZfXq1QwbNoybbrqJk08+meHDh5Odnc2bb74JwLp16xg4cCDXXnstgwcPZsKECZSXl9fW8eqrrzJ69GgGDBjA/PnzvToUYw5eOAyzJ0NKdxh3R/PXX3Obn+Z+nl1Kdxj/O1g7z7ks3xjTrkXXfX7evRU2f928dXbPhtMearD4oYceYvny5SxbtoxgMEhZWRmpqakUFBQwZsyY2qesf//998yYMYNnnnmGCy64gJkzZ3LZZZcBEAwGWbhwIbNnz+buu+9mzpw5zXsMxrSWZS9A/hcw6VmI2/ddx/dXs13qXp8RV8GXL8F7d8CAU/frafPGmLalRXt+RORUEVkpIqtE5NZ6ym8WkWXua7mIhESki1u2TkS+dss6xKAXVeX222/nqKOOYvz48WzatIktW7YA0K9fP4YNGwbAiBEjWLduXe12kyZNqne5Me1KeSHMmQp9xkD2eS2yi2a91H1PPh+cPg3KtkPug81fvzGm1bRYz4+I+IHHgFOAPGCRiMxS1W9r1lHVh4GH3fXPBG5S1R0R1ZykqgXNFlQjPTSt4YUXXmDbtm0sWbKEQCBAVlYWFRUVAMTFxdWu5/f765z2qinz+/0Eg8HWDdqY5jL3D07i8NN/OU9RbwFhwkALJT8APYbCyKucMUvDr4DMQS2zH2NMi2rJnp/RwCpVXaOqVcBLwNmNrH8xMKMF4/FESkoKxcXFABQVFdGtWzcCgQAfffQR69ev9zg6Y1rJ1u9g4VMw4gongWghzX6pe33G3Qnxqc4NGm3wszHtUkuO+ekFbIyYzwOOqW9FEUkETgVuiFiswPsiosBTqvp0A9teB1wHkJmZSW5ubp3ytLS02uTDC7GxsYwePZpBgwYxfPhw/vv/t3fn8XGV9eLHP9+ZSWayTLYmzdKk+74DpRSKEApls4p4QUFcQL1c4Irw+/1wF1TUK3h/14sLilxFFLiiImDlllWJyl4KtHSndKFpk25plsk2y3nuH88kmaRpm7aZnGTm+369zussc86c75ymM9/zPM95ns2buxs8T506lVAoBIDjON1xdnZ20tnZSUtLC7FYjNbWVlpaWgiFQhhjjvnzxGKxAR3T0dFxyPVTxy8UCun1BDCGuWu+SdDj5zX/EiJJvCYbQhsAeO3V19iWsS1p56mo/ChT37mHdX/4LvtGn5m084wk+vfuDr3ux0dMku5cRORy4AJjVRGYxgAAIABJREFUzGfj658AFhpjbuxn348CHzfGfCBhW4UxZreIjAaeBW40xvz9SOdcsGCB6dsnzoYNG5gxY8aJf6ARrKWlhWDw6I1L9VoNrpqaGqqrq90Ow30b/wce/hhc9H047V+Seqrl7y7nay98jRWXrqAqryp5J3JicO/Z0NYAn1t5QiPRpwr9e3eHXvfDE5FVxph++9NIZrVXLZD47VMJ7D7MvlfQp8rLGLM7Pt8LPIatRlNKjSSRDnjqK1AyAxZ8Jumn676ZS2KtFwAeL1z079C8C/7xgySfTCk12JKZ/KwEpojIBBHJxCY4y/vuJCL5wNnAnxK25YhIsGsZOB9Ym8RYlVLJ8PKPoXEHXHTnoAxcejRdj7p7ZAi6MBt3Osz5CLz0I2jYmvzzKaUGTdK+IYwxUWwbnqeBDcDvjTHrROQ6EbkuYddLgWeMMa0J20qBF0RkNfAa8D/GmKeSFatSKgmaam2pyMxLYOLZQ3LKpD7q3p+lt4M30/b9o5QaMZJ6K2aMWQGs6LPtnj7r9wP399m2FUjeIyFKqeR79jYwDpz/nSE7ZdKGtzicvHI46wvw3Ddgy3Mw+byhOa9S6oTo8BZKqcG3/UVY+0dYfDMUjB2y0w7Jo+59Lboeiibatk2xyNCdVyl13DT5UUoNLicGT34J8qtg8U1Deuqukp8h5fPDhXfA/s3wWr89ciilhhlNfoaJ6upq+j6mr9SItOp+2PO2re7KzB7SUw95tVeXqRfA5KVQcweE9g7tuZVSx0yTnyFkjMFxHLfDUCp52hrgr9+G8e+zDZ2HmCvVXl0u/B5E2uEv3xr6cyuljokmP0m2fft2ZsyYwQ033MDJJ5/MAw88wOmnn87JJ5/M5Zdf3t3Dc6Lc3Nzu5UceeYSrr756CCNW6gQ8/2/Q0WQfbXcjAYkbkkfd+yqeAouugzcfgl2rhv78SqkBS37HG8PIna/dycaGjYP6ntOLpvOlhV864j6bNm3iV7/6Fbfffjsf/vCHee6558jJyeHOO+/kBz/4AbfddtugxqSUK+rXwuu/hFM/C6WzXAkhWT3WD9hZX4TVv7Ntnj79jB0JXik17KRV8uOWcePGsWjRIp544gnWr1/P4sWLAQiHw5x++ukuR6fUIDDG/uAHCqD6K66FkfRR3Y8mkAdLvwWPXw9v/x7mXeFOHEqpI0qr5OdoJTTJkpNjx/0xxrB06VJ++9sjD16f2F6ho6MjqbEpNSjWPQo7XoD3/wCyi1wLw9U2P13mXgErf2H7OZr+fvAffVw9pdTQ0jLZIbRo0SJefPFFtmzZAkBbWxubN28+ZL/S0lI2bNiA4zg89thjQx2mUscm3ArP3Aplc+GUq10NxbWnvRJ5PHbcr9Ae+Pv/dy8OpdRhafIzhEpKSrj//vu58sormTt3LosWLWLjxkPbIN1xxx0sW7aMJUuWUF5e7kKkSh2Df/yHHeDz4n+3A34OA64mPwCVp8D8q+Dlu+HAu+7GopQ6RFpVe7lh/PjxrF3bMybrkiVLWLly5SH71dTUdC9fdtllXHbZZUMRnlIn5sC78NKPYe5HYewit6MZHtVeXc79Bqxfbnt+vur3bkejlEqgJT9KqeP39FftwJ5Lb3c7EiCh2ms4JD/BUjj7i/DO07D5GbejUUol0ORHKXV8Nj8Dm5+yP/DBMrejAVwY1f1oTrsORk2Gp78C0bDb0Sil4tIi+XG9748RQK+ROibRTnjqSzBqCpx2vdvRdBsWDZ4T+TLtuF8HtsCr97gdjVIqLuWTn0AgwIEDB/TH/QiMMRw4cIBAIOB2KGqkePluaNgKF91hf+CHiWHV5qfLlKUw9UL42/ehZY/b0SilSIMGz5WVldTW1rJv3z63Q3FNR0fHURObQCBAZWXlEEWkRrSmXfYR7mnvh8nnuR1NL8Ou5KfLBf8GP11kx/360E/djkaptJfyyU9GRgYTJkxwOwxX1dTUcNJJJ7kdhkoVT30ZTAwu+K7bkRyiK/kZdkZNgkU3wIt3wSnXQNWpbkekVFpLarWXiFwoIptEZIuIfLmf16tFpElE3opPtw30WKWUCzY9BRuWw1lfgKLhd1MxLKu9upx1C+SNgSduhljE7WiUSmtJS35ExAvcDVwEzASuFJGZ/ez6D2PM/Ph0+zEeq5QaKuFWWHELlEyHMz7vdjT9GrbVXmCHubjo+7BnLbyiVV9KuemoyY+IfE5ECo/jvRcCW4wxW40xYeBh4JIhOFYplQw1d0DTTlh217Bq5NwfjwzTZzlmLIPpy+D578HB7W5Ho1TaGkibnzJgpYi8AdwHPG0G9ujUGGBnwnotcFo/+50uIquB3cAtxph1x3AsInItcC3YMbESe0pWVigU0uviglS67jmhbSx4/SfUl53Hpm2dsK3G7ZD69W6THUri73//Oz4Znk0a/QUf4lTzHE0PXMPbc26D4VhFdxxS6e99JNHrfnyO+u1gjPm6iNwKnA9cA/xERH4P/NIYc6RBa/r7H903aXoDGGeMCYnIxcDjwJQBHtsV373AvQALFiww1dXVR/o4aammpga9LkMvZa6748B934HsIso/+V+Uuzhq+9FsWr0J3oKzzz6bDE+G2+EcXsEBRj31JapLGmD2P7kdzaBImb/3EUav+/EZUNlwvKSnPj5FgULgERH5/hEOqwWqEtYrsaU7ie/bbIwJxZdXABkiUjyQY5VSQ2TVfVC70j7dNYwTHxjmbX4SLfxnKJ8PT34Z2hvdjkaptDOQNj+fF5FVwPeBF4E5xpjrgVOAI92yrASmiMgEEckErgCW93nvMok/liEiC+PxHBjIsUqpIXBwBzz7DZhYbQcvHeZGTPLj8cIHfght++HZW92ORqm0M5BK8WLgw8aYHYkbjTGOiCw73EHGmKiIfA54GvAC9xlj1onIdfHX7wEuA64XkSjQDlwRL2Xq99jj+HxKqeNlDCy/0S5/8Mcjo21KvHJ8WD7q3lfFfDjjRnjxhzDzkmHXYaRSqWwgbX5uO8JrG45y7ApgRZ9t9yQs/wT4yUCPVUoNoVX3w7a/wbL/hIKxbkczICOm5KdL9Vdt30nLPw83vAyBfLcjUiotDNPnQZVSrmp8D575Okw4y/ZIPEJ0Jz8joeQHICMAl/4MWurh6a+6HY1SaUOTH6VUb8bYkgiAD/5kZFR3xRljRk6pT5cxp8CZN8ObD8LmZ9yORqm0oMmPUqq3138JW5+HpbdD4Ti3ozkmjnHcDuH4nP0lGD0T/vx5aD/odjRKpTxNfpRSPfZuhKe/BpPOHVHVXYlGXMkPgM9vR3tv3Qd/vtmWvimlkkaTH6WUFe2EP34WMnPhQz8Dz8j7ehi2o7oPRMVJsOTrsP5xePMBt6NRKqWNvG83pVRyPPct2PM2XHI3BEvdjua4jMg2P4nOuAkmnA1Pfgn2bXY7GqVSliY/SinY8hy8cjec+s8w7UK3ozluhhGe/Hg8cOnPISMLHvk0RDrcjkiplKTJj1LprmkXPHotlMyA87/tdjQnxGBGzmPuh5NXbqsd97wNz33T7WiUSkma/CiVzqJh+MOnbHufj/zGljiMZCO4yU8vUy+A066HV38G6x53OxqlUo4mP0qls2dvtYOWXvITKJnqdjQnzDHOyK72SrT0W1B5KvzpX+1TeEqpQaPJj1Lpau0f4dV7YNENMOtSt6MZFCP6aa++fP6e0rjfXQUdTW5HpFTK0ORHqXRUtwb+dCNUnWY7M0wRI77Bc195FXD5r6FhGzx2PTgjtBNHpYYZTX6USjct9fDbKyCrwJYseDPcjmjQjPhH3fszfjFc8F3Y9D/wtzvcjkaplHDUUd2VUikk0g4Pf8wOofDppyBY5nZEaiBOuw7q18Lf7oTCCTD/SrcjUmpE0+RHqXRhjG08u+sN+OiDUD7P7YgGXUo86t4fEVj2n9D0Hiy/EfIrYcL73I5KqRFLq72UShfP3mYbOZ/3DZixzO1okiIlq726+DLhIw9A0UTbAFp7gFbquGnyo1Q6eOEueOlHtgfnxTe7HU3SjNhR3QcqqwCu+gN4M+HBD0NTrdsRKTUiafKjVKp74zfw3Ddg9mVw0fdtFUqKSrmnvfpTOA6uesQ++v6bSyC01+2IlBpxkpr8iMiFIrJJRLaIyJf7ef0qEVkTn14SkXkJr20XkbdF5C0ReT2ZcSqVstY9Dn++CSadO2JHaj9WKZ/8AFTMtyVAzbvhgUttA3al1IAl7ZtQRLzA3cBFwEzgShGZ2We3bcDZxpi5wLeBe/u8fo4xZr4xZkGy4lQqZa191A6OWbkQPvqAbTOS4oxJoU4Oj2bsIrjiIdi/GR68DNob3Y5IqREjmU97LQS2GGO2AojIw8AlwPquHYwxLyXs/wpQmcR4lEofa/4Aj11rOzG86g+QmeN2RAPW2hllX0sn+0N2amqP0NIRJdQZJdQR7V7uiMSIOIZozCEaM0Qch/qMOlq98KG7X8TnEXxewefx4PUImT4PQb+PHL+P3ICPXL+d8rJ8jMrxMyo3k5JcP0U5mfi8I6SEbNISuOxX8Ier4TcfhI8/Bjmj3I5KqWEvmcnPGGBnwnotcNoR9v8M8GTCugGeERED/NwY07dUCAARuRa4FqC0tJSampoTiTklhUIhvS4ucOu6l9Y/z/SNP6IpfyZvj72J2MurhjyGw+mMGQ60G/a3O+xvN/HJoaHD0NRpaAobwrHDH+/3QpZPCPjA7xW8gp08di7BMOKFaHsLYWOIORAz4BiIONARNbRHDe1Ru/1wcjMgzy8U+oXiLA+jsuLzgFCcJRQGBM+waTuVS9GsLzN77R20313N6nm3E/YXDnkU+j3jDr3uxyeZyU9/3wz9ft2IyDnY5OfMhM2LjTG7RWQ08KyIbDTG/P2QN7RJ0b0ACxYsMNXV1ScceKqpqalBr8vQG/Lrbgy89GPYeBdMOIuCKx/mfS6U+MQcQ+3BNrbsDfHuvhDv7m1ly74QOw60sj8U7rVvptfDmMIsykcFmBX0U5LrpzjopzjXT0nQT3FuJgXZmd2lNF7PkROOb770Ks9u3cATt1x41Dg7ozFCHVGa2iMcaA1zINTJvlCY/QmlTvVNHaxrbGd/be+4M7zC2KJsJpbkMrEkh0nFuUwancPE4lwKc9yoXqyGk04l57dXcsamb8MnHrMNo4eQfs+4Q6/78Ulm8lMLVCWsVwK7++4kInOBXwAXGWMOdG03xuyOz/eKyGPYarRDkh+lFODE4Omv2oFKZ34ILv05ZASSekpjDLubOli/u5kNdc1sqm/h3X0htu5vJRzteeS8ODeTiSW5nDejlKqibCoLs+JTNiW5fjxHSWiO1UAbPPt9Xvy5Xkbl+plYcuR928MxdjW22+lgOzsPtrF1X4it+1r526Z9hGM9n7cwO4Mpo4NMLw8yozyP6WVBppUFyc5Mcp+yE8+2Sc9/Xw6/OA8+9jCMOSW551RqhErm/8aVwBQRmQDsAq4APpa4g4iMBR4FPmGM2ZywPQfwGGNa4svnA6kz+qJSgyncBo9fB+v/ZEdoP/+7g/5UVzjqsGVviPV1zazf3cz6uiY21LXQ1B4B7NPzY4uymVySy1lTS5hcYktCJpXkUpA9dCUhjnGS8rRXVqaXyaNzmTw695DXojGHXY3tNvHb18q7+1rZvKeFP66qpTVehycC40flML2sJyGaW1lAaZ5/cHukHnsafOZZeOgy+NX74Z9+kbIdWip1IpKW/BhjoiLyOeBpwAvcZ4xZJyLXxV+/B7gNGAX8NP4FEI0/2VUKPBbf5gP+2xjzVLJiVWrEOrgdfvdxO+7T+d+BM2484beMOYZ39rawemcjb+1sYvXORt7Z20Ik3kgmK8PLtLIg759bzszyvO4f8xy/+6PlmP5r1pPK5/UwblQO40blsGR6z3bHMexqbGd9XTMb61rYUGdLyJ5aV0/XQ2klQT9zx+QzpzKfuZX5zBlTQEnQf2IBlUyDz/7FDl77u4/bHr0X35zS/TspdayS+m1ljFkBrOiz7Z6E5c8Cn+3nuK1A6g08pNRgevd5eOQacBz42O9h6vnH/BZdVVerdzbGk51G3t7VRFu8xCIv4GNeVQFnT5vIzPI8ZlbkMX5UzlHb3rhlOA1v4fEIVUXZVBVlc8GsngFkWzujbKxvYe2uJlbXNvJ2bRN/3bS3OyGqyA/Ek6EC5ozJZ86Y/GNvR5Q7Gj71BPzpBnjum1D7OnzopxDIH7wPqNQI5v6tmlLq2Dgx+Md/QM33oHia7etl1KQBHdrUHmFNbWNPqU5tI/taOgHb+HhmRR4fWVDF/KoC5lUVMH5U9ogaKNSNkp9jleP3ccq4Qk4Z1/NEVmtnlHW7m1lTa5PPt2ubeHrdnu7Xx43KZm5lAfMq85lXVcDsinyyMr1HPlFmtn0MvnIhPHsr3FttxwYrm52kT6bUyKHJj1IjSeN78Oi18N7LMOdyWHYX+A9thwL2aaYNdS09pTq1jWzd19r9+qSSHN43pZiT4onO9LI8Mn0jpH+bIxguJT/HIsfvY+GEIhZOKOre1tQeYd2uJlbXNrGmtpFV2xv482r7zIjXI0wtDXYnQ/MqC5hamnto/0QicPoNUHGS7Qvov5bYarDTrk+L3r6VOhxNfpQaCYyBNb+HFV8A48Cl98K8j3a/7DiG7Qdaeas70Wliw+7m7qeQSoJ+5lcV8E8nVzK/qoA5lfnkBTLc+jRJY4wZUSVVR5KflcEZk4s5Y3Jx97a9LR2siZfYra5t4sm19Ty80nanFsjwMLvCVpfNq8pnflUBY4viJXfjTofrXoA/f94+FbjpSVsNVjDWrY+nlKs0+VFquDu4HZ74v/DuX2yPzZf+nH0ZFaxev8cmO/FqrOaOKAA5mV7mVOZzzZnju0t1yvICKZMUHEmqD2w6OhjgvJkBzptZCthkb8eBtvjfgC0heujVHdz3ok16C7IzeqrLKguY+/77GD3tEXjqy/CzxbDkVjj1M+A5ShWaUilGkx+lhqtoGF79Geb57+Hg4ZWpX+S3zvm8ee82djXaUWK8HmF6WZBl8yqYX2kTncmjc4dtg+Rkc4xz9J1SiIgwvjiH8cU5XDJ/DACRmMPmPS2sqbVP6q2ubeKnNe8Sc2x7qIr8CpaU/5x/af4xVU9+gdibD+L9wF0w5mQ3P4pSQ0qTH6WGkUjM4Z36Fhpef4Tp6/6D4vAu/hI7mVsj11C3ZhRVRS2cNLaAaxaPZ35VAbMG0vA1jaR6yc9AZHg9zKrIZ1ZFPlcutNVabWHboLorGfpHbSMPHriRD3gWcGvdgxT/1xJWFlzMznk3MWXyNKaXB/H79O9KpS5NfpRySUckxqb6FtbubmLtrmbW7W7CX/8GX/A8yJmeTWyhivtL/g3P1KV8t8pWW4zKPcE+YFLd8H/YyxXZmT5OHV/EqeN7GlQfbA2zunYhj26/jInrfsQ5jcuZV/MM9/3lIj5tLqGyory7umxeVT4Ti3MHvTdupdyiyY9SQ6AtHGVDXTNrdzWzdlcTa3c3886eFqLxqojqwBa+63+cOb436PSPYv/p32fS+z7NLd7Ua5ScTFryM3CFOZlUTxtN9bTRcMGvMA3biDzzbW7Y+Eeu8dbw545LuGtVNb95OQuAoN/X3f/QvMp8ZlbkUVWYrQmRGpE0+VFqEMUcw3sNbWyqb+apLWF+V7uKTfUtbD/QSjzPoTg3k9lj8jl3WjHV3tXM3n4/gV0vg68Ezrod/4LP4D/M4+vqyEZCPz/DlRRNIPuK+6DuJrKe/x4f2fwAl/sfpXH+x3ix5KO8ciDAmtomfvnC1u7evrMzvUwtDTKjPIi3JULW1gNML8sjP1uTdjW8afKj1HEwxrAv1Mmm+paeaU8Lm/e00BGxjW4FGF/cwrTSIB+cX8Hsinxmj8mnNKMdeeshWPkLOLgNghVw4Z1w8idtx3TquBlj8Ij2X3NCyufZQVH3rENe/CGFb9/HMvkVy6ZdDBd/ho7KpWza08rG+mY21LWwsb6ZJ9fW09gW4cENr9i3yA/EB3TNY0Z5kCmjg0wsySGQoe2I1PCgyY9SRxCOOrzX0Ma7+0IJA1faedegngDFuX6mlwW56rRxTCsLMr0sSN2mN7ng3Gq7QywKW2vguYdhwxMQbYexZ8C5t8GMD4BWbw0KLfkZRKWz4MP3wjlftYn6mw/BhuUEiiYx7+RPMG/2ZXDqLMAmnY8//TyFE2azsb6FjXXNbKxv4YUt+7tLiURgTEEWk0fnMqnETnY5h6KczLToikENH5r8qLQXcwz1zR28d6CN9xpaeyU4Oxrauh8RBhgd9DOpJJdlc8uZVJIbv7sN9tsQueEdB3a+Buseg7cfgda9ECiA+VfCgk9D2Zyh/JhpYTiN7ZUyCsfbQXPP+TpsWA4rf2nHC3vum1C1COZchsz8EIUBT08borhw1GHr/hBb9oZ4d28rW/aFeHdviFe2HuguIQXbH9HkeEI0rjibcUU5jC3KZuyobPKz9MZADT5NflRaCHVG48lNGzsb7LxrufZge3dPyGDHuJpQnMO0siAXzyln0ugcJhbnMrEkh+DRekUOt9kSnk0rOGPtcvhbE3gzYeoFMPcKmLIUfPrEVrJoyU8SZQRg7kfs1LAN1v7RJvUrboEVX+CkvKng+QhMvdCWGomQ6fMwvSyP6WV5vd7KcQy7m9ptUhS/2diyN8RfNu5hfyjca9+C7AzGxQeIHTcqnhiNymZsUTaleYG07dNKnRhNftSI1xmNsaepk91N7dQ3dbC7qZ26xg7qmtrZHZ8fbIv0OiY/K4OxRdnMKM/j/Fll9i4zPo0pzBr4F2o0DLteh21/t9PO18CJgD+Pg4XzKH3f1TD5PMgqGPwPrg6hJT9DpGgCnHWLnfasg/V/wrPqEfjrt+2UNwYmnAXjFsP4M23pUUK1lscjVBZmU1mYTfW03m/dc6PSynsNbeyI37SsiQ/nkVgS6/UIZXkBKgoClOdnUVGQRUVBgIqE5fysDK1SU4fQ5EcNW53RGPtDYfa1dPaa9oc62dPcQX1zB7sbO9gf6jzk2PysDMrzA5TnB5g/toCqwuxeCc5xPY1iDDTthF2r4tMbdoq2A2Ibii66HiadA+POZMMLL1E6p/qEr4MaOC35cUHpLCidxSo5g+pTpsM7z8A7z9r56t/affLGwNjTbS/SFSdB2dzDDsib6/cxsyKPmRV5h7wWjTnsbuxgRzwxqmvsYHdjO7sa23lrZyNPrq3rbmPUJTvTS1l+gNFBP6ODAUqCfkYH/fF5z3pBtiZJ6USTHzUkjDG0R2IcbItwsDVMY1uEg21hGtvCdltbOJ7odHQnOV1jVfVVmJ1BSdBPWX4WM8vzKM/PorwgEE92sijPD5DjP8E/7c4W2LcJ9q6HvRth3waoWwNt++3r3kz7BX7Kp2D8+2D8YsgqPLFzqhOm/fy4LFhmn1o8+ZP2ZmHfRtj+Aux4EXa8BGsfie8oUDINyufD6OlQMh2Kp9oSoiOMM+bzemyV16j+n4p0HMP+1k52x5MiO3VQ39zOvpZOVtc2sre5k/ZI7JBjM7xCSa5Niopz/RTmZFKUk0lhdiZFORnxeSYF8Xl+VoZWuY1gmvyoAYvGHEKdUVo6ojR3RGjpsMuhzp7lru2NbWEOttqk5mA8wQlHDz/uUtDvozjopyTXz/SyPM6cnElJ/O6s68uoJOhnVI6fTN8gPMpsDLQ12EFDD26z88Yddt6wzZbwdPEF7BfzlPPtneuYU6B0NvgyTzwONbgMevc+XIjA6Bl2WvjPdlvLHqh7C3a/aadtf4M1D/cc4/VD8RQYNQkKxkHhODsvGGunjKwjntLjEUYHA4wOBphf1X9VszGG1nCMvc32Rmtv/GarZ95BXVMH6+uaaWgN03mY7y0RKMjKsElStk2G8rIyCAZ85AXi86yMXstdr+Vl+XT4EJclNfkRkQuBHwJe4BfGmDv6vC7x1y8G2oCrjTFvDORYdXgxx9AZjdEWjtEejlHb4vDmewdpD9ttbZEY7eGoXY7v0xaO0R7pvc0mOj2JTX93S31l+jwE/T4KczIpzM6gqiibuZX5FGbbO6bC7IzuedddVEF2BhneQUpowiFoP2intgP2yzZUb+ctdRDaAy31dh5p6318zmh75zn2dCi52n5pl0w/6t2oGj605GeYC5ZC8AL7AECX9kbY/44tJdq/yZa47lkHm56EWO/Gz+SMtqVLuaX2vXK7ptF2nlNin6jMKjhs9xEiQq7fR25JLhNLjt6ZaHs4RkNbmIOtYRpa7c1cQ6tdP9gWoaEtTEMoTH1zB+/sDdHcEaG5PYJzlBrYTJ+HvICPHL+PrAwvOX4f2ZlecjJ9ZPu9PcuZPnL8XrK613v2zcr0Ut/qUNfUjt/nJZDhwe/zaonUACQt+RERL3A3sBSoBVaKyHJjzPqE3S4CpsSn04CfAacN8NghZ4wh5hiijiESc4jGDBHHsdti8W0Jr0Udh0jMdO8XjRmiMYeIY+eJ2yMxh3DMoTPSM++MxghHHTqjfZed+HLMrncfY9ej/f2ve/Glw34uEcjO8JIV/4/V9Z8qGPBRURAg6Ld3LMFA19zXZ71n+zHdzTgxiHZCuN1+yUXaIdxqk5JwyC6HW+PLbQnL8Xl7Y0+i034QOhrB6b+qjMxg/Iu33JbcBMsgv9ImNl13mJk5A49dDUvpNqp7SsgqgKpT7ZTIcexNSuN78Wm7nbfssdv3rLPdRxz2/3xuPBEqtOfIKrDr/qD9v56RbffJzLGdi3Ytd23PyAKfnyxvJmNy/YzJz+vVaPtIjDG0hWPdJeHN7ZE+y9F4khSlPRylNeGGc29zJ63xm9PWzuhhS556+cdfe61meKVXMuTP8BA4wjzT5yHD6yHDJ2R4epYzvR58HiEj/nqm14PPK93LGX3XfYLP07OfzyP4vB68HrucOHe7hDaZJT8LgS3GmK0AIvIwcAmQmMBcAvzGGGOAV0SkQETKgfEDOHbIfPXXd/N3vjyDAAAHgElEQVRO7SZsOzqDABJvWNn1z9ezbuLbehKQxP179ul9PAnrPiH+B2KfZvAJBL0euy7g9Qo+r+DNEHxd27r/sMArglds/XiGF5oPHqS0pAivx0OG1+7jE0OGB7wYvOIgxoBx4lMsPjd22XEgHINOBxrj+zgO4NDuOLSbGHu7jnei9mmnWCQ+j8aXoz3bnKhNdo7nh8qbaauhMrJsg8nMIOSOBf8su+zPjX+5xedZhZBVZB/TPRynBQ6sPfZYjmJT+yay6o5cTK8GV2Nno5b8pAqPB/LK7TT2tP73cRxob7DJUGgPtO63N0UdXTdGCcv7t9jlrpun4/r+8duuKryZCfOArQLvek08iMdHjsdHjsdLuccLHp+dpGs5Pg94ITthPfF1BERwECIOhGOGcKxrHp8c2LV7D0UlpUQcQ8QRIjHTvRyOGSIOROLHRqKGcKdd74xBxLHzTsfQ6hhiMYeIA47pO0Zw71810+uXq+8vXuK2Q1/rei8RweMR/s81VzF74thj/7c4QclMfsYACQ0nqMWW7hxtnzEDPBYAEbkWuBagtLSUmpqaEwq6P/s7fs3GyvZBf9+kMfHJASJAFhAawvML/fxlCZAZnwZLyE6d2Gk4esbtANLPJN+kpHwPqCMLhUIuX3cPMDo+AYH41B9j8DhhvLGOPlM7HqcTX7QdjxPG40QQE8Hj9Ex91z1OBAlH4/u3IsbeGIqJxSenzzxxez/b6J2UeQB/fOrPLIB9J3zxLG98GkLPv1FEzXvTjr7jIEtm8tPf7Vff+pjD7TOQY+1GY+4F7gVYsGCBqa6uPoYQB2bsuDtpaNmVUOQpvSMUW7bTa6MkrEufeTyjT9j50G2HLB/j+yfsu3btWmbPmdvnPT327kI88ck74CJdNTBvvfUW8+fPdzuMtFO3ro5kfA+oI6upqdHrPhiMsU0CMPHSdye+7PS7/uILL7B48Rn9vH64Y+m93qsEzPTEcNhtCa8NZFuvX+5D3/+c0TMO2+1BMiUz+akFqhLWK4HdA9wncwDHDpmJE85holsnHwShHT4WjK92O4y0EwqEWFC2wO0w0k7Nxhq3Q1Dq+ImAd+A/zZHMPMgpTmJAqSmZwx+vBKaIyAQRyQSuAJb32Wc58EmxFgFNxpi6AR6rlFJKKXXMklbyY4yJisjngKextYj3GWPWich18dfvAVZgH3Pfgn3U/ZojHZusWJVSSimVPpLaz48xZgU2wUncdk/CsgH+daDHKqWUUkqdKDGm33bEI5KI7AN2uB3HMFQM7Hc7iDSk190det3dodfdHXrdD2+cMaakvxdSKvlR/ROR140x2vJ2iOl1d4ded3fodXeHXvfjk8wGz0oppZRSw44mP0oppZRKK5r8pId73Q4gTel1d4ded3fodXeHXvfjoG1+lFJKKZVWtORHKaWUUmlFkx+llFJKpRVNftKMiNwiIkZEdDCYISAi/y4iG0VkjYg8JiIFbseUykTkQhHZJCJbROTLbseTDkSkSkSeF5ENIrJORG5yO6Z0IiJeEXlTRJ5wO5aRRJOfNCIiVcBS4D23Y0kjzwKzjTFzgc3AV1yOJ2WJiBe4G7gImAlcKSIz3Y0qLUSB/2eMmQEsAv5Vr/uQugnY4HYQI40mP+nlP4EvAtrKfYgYY54xxkTjq68AlW7Gk+IWAluMMVuNMWHgYeASl2NKecaYOmPMG/HlFuwP8Rh3o0oPIlIJvB/4hduxjDSa/KQJEfkgsMsYs9rtWNLYp4En3Q4ihY0Bdias16I/wkNKRMYDJwGvuhtJ2rgLe0PruB3ISJPUgU3V0BKR54Cyfl76GvBV4PyhjSg9HOm6G2P+FN/na9jqgYeGMrY0I/1s01LOISIiucAfgZuNMc1ux5PqRGQZsNcYs0pEqt2OZ6TR5CeFGGPO62+7iMwBJgCrRQRs1csbIrLQGFM/hCGmpMNd9y4i8ilgGXCu0Y61kqkWqEpYrwR2uxRLWhGRDGzi85Ax5lG340kTi4EPisjFQADIE5EHjTEfdzmuEUE7OUxDIrIdWGCM0ZGAk0xELgR+AJxtjNnndjypTER82Ebl5wK7gJXAx4wx61wNLMWJvaP6NdBgjLnZ7XjSUbzk5xZjzDK3YxkptM2PUsn1EyAIPCsib4nIPW4HlKriDcs/BzyNbXT7e018hsRi4BPAkvjf+Fvx0gilhi0t+VFKKaVUWtGSH6WUUkqlFU1+lFJKKZVWNPlRSimlVFrR5EcppZRSaUWTH6WUUkqlFU1+lFJKKZVWNPlRSimlVFrR5EcplRJE5FQRWSMiARHJEZF1IjLb7biUUsOPdnKolEoZIvId7DhHWUCtMeZ7LoeklBqGNPlRSqUMEcnEjunVAZxhjIm5HJJSahjSai+lVCopAnKx46kFXI5FKTVMacmPUipliMhy4GFgAlBujPmcyyEppYYhn9sBKKXUYBCRTwJRY8x/i4gXeElElhhj/up2bEqp4UVLfpRSSimVVrTNj1JKKaXSiiY/SimllEormvwopZRSKq1o8qOUUkqptKLJj1JKKaXSiiY/SimllEormvwopZRSKq38L/L9xz+d9m9BAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot\n",
    "plot_activations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/drill.png\" style=\"height:50px;display:inline\"> Optimizing MLPs\n",
    "---\n",
    "* Optimization is performed by stochastic gradient descent (SGD) and the chain rule - **backpropagation**.\n",
    "* Reminder - **The Chain Rule**:\n",
    "    * Given a function: $F(X)=f(g(x))$\n",
    "    * Denote: $y=g(x), z=f(y)$\n",
    "    * The derivative is given by: $$ \\frac{dz}{dx} = \\frac{dz}{dy} \\cdot \\frac{dy}{dx} = f'(y)g'(x) = f'(g(x))g'(x) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/doodle/96/000000/link.png\" style=\"height:30px;display:inline\"> Optimizing MLPs -  Simple Example\n",
    "---\n",
    "* Demonstration for a simple 1D regressor: $$ f(x) = w_2 \\cdot \\sigma(w_1x + b_1) + b_2 $$ $$ E(x,y) = (f(x) - y)^2 $$\n",
    "    * $\\sigma(x)$ is the *sigmoid* function\n",
    "* Denote: $$ u = w_1x + b_1 $$ $$ v=\\sigma(u) $$ $$ z=f(x)=w_2v + b_2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The derivatives:\n",
    "    * $\\frac{\\partial E}{\\partial w_2} = \\frac{\\partial E}{\\partial z} \\frac{\\partial z}{\\partial w_2} = 2(f(x)-y) \\cdot \\sigma(w_1x + b_1)$\n",
    "    * $\\frac{\\partial E}{\\partial b_2} = \\frac{\\partial E}{\\partial z} \\frac{\\partial z}{\\partial b_2} = 2(f(x)-y) \\cdot 1$\n",
    "    * $\\frac{\\partial E}{\\partial w_1} = \\frac{\\partial E}{\\partial z} \\frac{\\partial z}{\\partial v} \\frac{\\partial v}{\\partial u} \\frac{\\partial u}{\\partial w_1} = 2(f(x)-y) \\cdot w_2 \\sigma(u)(1-\\sigma(u)) \\cdot x$\n",
    "    * $\\frac{\\partial E}{\\partial b_1} = \\frac{\\partial E}{\\partial z} \\frac{\\partial z}{\\partial v} \\frac{\\partial v}{\\partial u} \\frac{\\partial u}{\\partial b_1} = 2(f(x)-y) \\cdot w_2 \\sigma(u)(1-\\sigma(u)) \\cdot 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/drill.png\" style=\"height:50px;display:inline\"> Optimizing MLPs - Cont.\n",
    "---\n",
    "* Once we calculated all the derivatives we're done! We can now train MLPs!\n",
    "* Calculating the derivatives for every network is time consuming and becomes difficult for large networks.\n",
    "* Rather than explicitly deriving the gradient of the loss function with respect to every variable, we will take a modular approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/lego-head.png\" style=\"height:50px;display:inline\"> Modular Approach\n",
    "---\n",
    "* We code **layers**, not networks.\n",
    "* Layer Specification - each layer needs to provide 3 functions:\n",
    "    1. The layer output given its input (forward pass)\n",
    "    2. Derivative with respect to the input\n",
    "    3. Derivative with respect to parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Illustration: <img src=\"./assets/tut_12_mod_app_1.jpg\" style=\"height:350px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Zoom-in: <img src=\"./assets/tut_12_mod_app_2.jpg\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/plasticine/100/000000/serial-tasks.png\" style=\"height:50px;display:inline\"> Backpropagation\n",
    "---\n",
    "We now establish a common language when it comes to neural networks architecture:\n",
    "* **Forward Pass**: $Z^{(k+1)} = f(Z^{(k)}) $\n",
    "* **Backward Pass**: $\\delta^{(k+1)} = \\frac{\\partial E}{\\partial Z^{(k+1)}}$\n",
    "* Applying the **chain rule** for a single layer: $$ \\frac{\\partial E}{\\partial Z^{(k)}} = \\frac{\\partial E}{\\partial Z^{(k+1)}} \\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = \\delta^{(k+1)}\\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = \\delta^{(k+1)}\\frac{\\partial f(Z^{(k)})}{\\partial Z^{(k)}} $$\n",
    "* The **gradient with respect to layer parameters** (if it has any): $$ \\frac{\\partial E}{\\partial W^{(k)}} = \\frac{\\partial E}{\\partial Z^{(k+1)}} \\frac{\\partial Z^{(k+1)}}{\\partial W^{(k)}} = \\delta^{(k+1)} \\frac{\\partial Z^{(k+1)}}{\\partial W^{(k)}}  $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> Extension to Multi-Dimensions\n",
    "---\n",
    "* $f: \\mathbb{R}^n \\to \\mathbb{R}^m$ is a vector function of a vector variable: $$ f(x) = \\begin{bmatrix} f_1(x) \\\\ \\vdots \\\\ f_m(x) \\end{bmatrix}, x \\in \\mathbb{R}^n, f(x) \\in \\mathbb{R}^m $$\n",
    "* The **gradient** is given by: $$ \\frac{\\partial f_i}{\\partial x} = \\big[ \\frac{\\partial f_i(x)}{\\partial x_1}, ..., \\frac{\\partial f_i(x)}{\\partial x_n} \\big] $$\n",
    "* The **Jacobian**, $J_f(x) \\in \\mathbb{R}^{m \\times n}$, is given by: $$ J_f(x) = \\begin{bmatrix} \\frac{\\partial f_1(x)}{\\partial x}\\\\ \\vdots\\\\ \\frac{\\partial f_m(x)}{\\partial x} \\end{bmatrix} = \\begin{bmatrix} \\frac{\\partial f_1}{\\partial x_1} && \\cdots && \\frac{\\partial f_1}{\\partial x_n} \\\\ \\vdots && \\ddots && \\vdots \\\\ \\frac{\\partial f_m}{\\partial x_1} && \\cdots && \\frac{\\partial f_m}{\\partial x_n} \\end{bmatrix} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **The Chain Rule**:\n",
    "    * Given: $$ F: \\mathbb{R}^n \\to \\mathbb{R}^m $$ $$ \\phi: \\mathbb{R}^m \\to \\mathbb{R}^k $$ $$ \\psi(x)= \\phi(F(x)) $$\n",
    "    * The Jacobian is given by: $$ J_{\\psi} = J_{\\phi} J_F $$ $$ J_{\\phi} \\in \\mathbb{R}^{k \\times m}, J_F \\in  \\mathbb{R}^{m \\times n} \\to J_{\\psi} \\in \\mathbb{R}^{k \\times n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/popular-topic.png\" style=\"height:50px;display:inline\"> Commonly Used Layers (as Modular Blocks)\n",
    "---\n",
    "* Linear Layer (linear combination of the inputs)\n",
    "* Activation Layer (usually together with the linear layer, apply a function on the linear combination of weighted inputs): ReLU, Binary Step, Sigmoid, TanH...\n",
    "* Softmax Layer (Sigmoid for for than 2 classes, outputs the probability of each class)\n",
    "* Loss Function Layer (MSE, Cross Entropy...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/home.png\" style=\"height:50px;display:inline\"> Example - Neural Networks for Regression - Housing Prices\n",
    "---\n",
    "* The Housing Prices Dataset:\n",
    "    * Two input features: *Size* and *Floor*\n",
    "    * One output: *House Price*\n",
    "    * **Loss function**: MSE\n",
    "* Suggested **network architecture**: 2 hidden layers\n",
    "    * Two inputs, one for each feature\n",
    "    * Four neurons in the *first hidden layer*\n",
    "    * Three neurons in the *second hidden layer*\n",
    "    * One output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Layout: <img src=\"./assets/tut_12_mlp.jpg\" style=\"height:300px\"> $$ F(X,W) = W_3^T \\phi_2(W_2^T\\phi_1(W_1^TX + b_1) + b_2) + b_3 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Where: $$ X \\in \\mathbb{R}^2 $$ $$ W_1 \\in \\mathbb{R}^{2 \\times 4} $$ $$ W_2 \\in \\mathbb{R}^{4 \\times 3} $$ $$ W_3 \\in \\mathbb{R}^{3 \\times 1} $$ $$ b_1 \\in \\mathbb{R}^4 $$ $$ b_2 \\in \\mathbb{R}^3 $$ $$ b_3 \\in \\mathbb{R} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/office/80/000000/baby-footprints-path.png\" style=\"height:50px;display:inline\"> Step-by-Step Solution\n",
    "---\n",
    "* The MSE loss function over all the training examples $x_i$ and the corresponding training targets: $$ Error = \\frac{1}{N} \\sum_{i=1}^N (F(x_i, W) - y_i)^2 = \\frac{1}{N} ||F(X, W) - Y||_2^2 $$\n",
    "* Linear Layer: $$ u_{out} = W^Tu_{in} + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Activation Layer:\n",
    "    * $\\phi_1$ and $\\phi_2$ are multivariate vector *nonlinear* functions, such that: $$ \\phi(U) = \\phi(\\begin{bmatrix} u_1 \\\\ \\vdots \\\\ u_n \\end{bmatrix}) = \\begin{bmatrix} \\phi(u_1) \\\\ \\vdots \\\\ \\phi(u_n) \\end{bmatrix} $$\n",
    "    * For **ReLU**: $$ \\begin{bmatrix} \\phi(u_1) \\\\ \\vdots \\\\ \\phi(u_n) \\end{bmatrix} = \\begin{bmatrix} \\max(0, u_1) \\\\ \\vdots \\\\ \\max(0, u_n) \\end{bmatrix} $$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot_relu():\n",
    "    x = np.linspace(-5, 5, 1000)\n",
    "    y_relu = list(map(lambda z: relu(z), x))\n",
    "    y_relu_derv = list(map(lambda z: relu(z, deriv=True), x))\n",
    "    fig = plt.figure(figsize=(8,5))\n",
    "    ax1 = fig.add_subplot(1,1,1)\n",
    "    ax1.plot(x, y_relu, label='relu')\n",
    "    ax1.plot(x, y_relu_derv, label='relu derivative')\n",
    "    ax1.grid()\n",
    "    ax1.legend()\n",
    "    ax1.set_xlabel('x')\n",
    "    ax1.set_ylabel('y')\n",
    "    ax1.set_ylim([-1, 2])\n",
    "    ax1.set_title('ReLU Activation Function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAFNCAYAAADo2q2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV9Z3/8deHkBA22QlLgIQRF0RUGhEEbNzFutVq1Vp3xHZsO+2j0/6cmbZO5+e0ttNHpz9rR0eC4lbB0dqqxbpNUwKCAoobCFLCEnYCIgECWT6/P+6FhpiEJPfce+49eT8fjzy4955zz/ncL0ne+Z7l+zV3R0RERKKpU9gFiIiISPIo6EVERCJMQS8iIhJhCnoREZEIU9CLiIhEmIJeREQkwhT0Ih2UmT1oZj9M0rY/NLPiZGw7bGb2z2ZWEnYdIq2loBc5CjNba2b7zazKzLaY2Swz69HK9xabWUUzy0rNbFpr12+wTvd4LXPb8BluNrP5DV9z96+5+/9t7TZa2PYsM7un0bZPcvfSRLfdxL5Kzaw6/vkPfU0Mej8N9veZ/w93/4m7T2vuPSLpRkEv0jqXunsP4FTgNOCfQqzlKuAAcIGZDQ6xjrB8w917NPhaGHZBIulMQS/SBu6+BXiZWOADYGZdzOwXZrbezLbGD4l3TWIZNwEPAu8B1zdcYGbDzOx3ZrbdzCrN7H4zOzG+/sR4D/iT+LqHe+JmtsLMLmmwnc5mtsPMxsWf/0/8aMZuM5tnZifFX58er+H78W2/EH99rZmd16B9fmVmm+JfvzKzLvFlxWZWYWbfNbNtZrbZzG5pa4OYWYGZuZl1bvDa4SMmh45oxP+fdplZuZlNbbBuXzN7JF7fLjP7vZl1B14ChjQ4ejDEzP7VzJ5o8N7L4qcqPonv88QGy9aa2T+a2XvxtptjZrlt/XwiiVDQi7SBmeUDU4HVDV7+GXAcsfA/FhgK/ChJ+x8OFANPxr9ubLAsC3gRWAcUxOuY7e4rgK8BC+M94N5NbPop4LoGzy8Edrj72/HnLwGjgIHA2/F94+4PxR//PL7tS5vY9r8AE4i1zynAeOAHDZYPAnrF670N+I2Z9WlFc7TVGcBKoD/wc2CmmVl82eNAN+AkYp/xP919L7H/600Njh5sarhBMzuOWNt9GxgAzAVeMLOcBqt9GbgIKATGAjcn4bOJNEtBL9I6vzezPcAGYBtwN0A8KG4HvuPuO919D/AT4Nok1XEj8J67LycWMCeZ2WnxZeOBIcD33H2vu1e7+/zmNtTIb4HLzKxb/PlX4q8B4O4Pu/sedz8A/Ctwipn1auW2rwf+zd23uft24MfADQ2W18SX17j7XKAKOL6F7d0X7z1/YmZvt7BeY+vcfYa71wGPAoOBvPjpj6nA19x9V7yOv7Rym9cAf3T3V929BvgF0BU4s2G97r7J3XcCL9DgaJBIKijoRVrnCnfvSaw3fQKxXiHEenHdgKWHwgf4U/z1o6kFshu9lk0s+JpzI3/rTW8C/kLsUD7AMGJhVtuKfR/B3VcDK4BL42F/GfGgN7MsM7vXzP5qZp8Ca+Nv69/kxj5rCLGjDIesi792SGWjmvcBLV3s+C137x3/GtfKGgC2HHrg7vviD3sQa7ed7r6rDds65IjP5u71xP4YHNrUfjn6ZxMJnIJepA3iPb1ZxHpuADuA/cBJDcKnV/zCvaNZT+wQe0OFHBmKh5nZmcQOn/9T/Hz5FmKHo6+Ln5veAAxveJ66YemtqOfQ4fvLgeXx8IdY7/5y4Dxih9gP1XzosPfRtr0JGNHg+fD4a0HaG/+3W4PXBrXyvRuAvmbW1CmNNn22+BGeYcDGVu5bJOkU9CJt9yvgfDM7Nd6DmwH8p5kNBDCzoWZ2YcM3mFluoy8D5gC3mNl4izkO+A4wu5n93gS8Cowmdvj3VGAMsXCbCrwFbAbutdgteLlmNin+3q1AfqNzx43NBi4Avk6Dw/ZAT2JX+VfG9/WTRu/bCoxsYbtPAT8wswFm1p/Y9QtPtLB+m8VPCWwEvho/AnEr8HetfO9mYtcg/JeZ9TGzbDM7K754K9CvhdMUTwNfMLNzzSwb+C6xtnojkc8jEiQFvUgbxUPlMeDQYDP/h9jFeYvih7Zf48hzzEOJ9fobfv2du78M3AU8AuwmdiHXo8BDjfcZv1L7y8Cv3X1Lg69yYheS3RQ/93wpsQsC1wMVxM4hA/wv8CGwxcx2NPO5NgMLiZ1fntNg0WPEjjJsBJYDixq9dSYwOn7q4vdNbPoeYAmxuwTeJ3Yx3z1NrJeo24HvEfuD5CTaFrY3EDtl8hGxazC+DeDuHxH7Q2VN/PM1POWAu68Evgr8mtjRnUuJ3Yp5MLGPIhIcc2/NET0RERHJROrRi4iIRFhoQW+xgT3+bLGBOj40s39oYh0zs/vMbHV8wIm2XGErIiLS4TV1dW6q1ALfdfe3zawnsduTXo3fH3zIVGJXGY8idnXxA/F/RUREpBVC69G7++ZDo27FBxlZwZH3nkLslp7HPGYR0Ns65tjeIiIi7ZIW5+jNrIDYRCFvNlo0lNg9rodU8Nk/BkRERKQZYR66B8Bi030+C3zb3T9tvLiJtzR5m4DFJteYDtC1a9fPDRs2LNA601l9fT2dOqXF32wZTe2YOLVh+9XWQ0VVPcdkQ9+uasNEdMTvw1WrVu1w9yZH5Aw16OMDTDwLPOnuv2tilQpio0wdkk8zI2rFJ9d4CKCoqMiXLFkScLXpq7S0lOLi4rDLyHhqx8SpDdvv315YzmML1/Lzs3K58qJzwi4no3XE70Mza3JETQj3qnsjNtDGCnf/ZTOrPQ/cGL/6fgKwOz6oh4hIZOzeX8Ocxeu5ZOxg+uZ2rJ6oJF+YPfpJxEajet/MlsVf+2di42Dj7g8SGynsYmKjju0D2jxPtYhIupuzeD17D9YxbcpIdnz8TtjlSMSEFvTx6TObOgffcB0H7kxNRSIiqVdTV88jC9YyYWRfxgztRenHYVckURP6xXgiIh3Z3Pc3s3l3NfdcMSaU/dfU1FBRUUF1dXUo+0+GXr16sWLFirDLSIrc3Fzy8/PJzm48w3XzFPQiIiFxd2aUrWHkgO6cffzAUGqoqKigZ8+eFBQUELt0KvPt2bOHnj17hl1G4NydyspKKioqKCwsbPX7dNWHiEhI3izfyQcbP+W2yYV06hROyFZXV9OvX7/IhHyUmRn9+vVr89EXBb2ISEhKytbQt3sOXxqXH2odCvnM0Z7/KwW9iEgI1myv4rUV2/jqhBHkZmeFXU5GKC4upiONkRIUBb2ISAhmzi8np3MnbpgwIuxS0oq7U19fH3YZkaKgFxFJsZ17D/LM0gq+eOpQBvTsEnY5oVu7di0nnngif//3f8+4ceN4/PHHmThxIuPGjePqq6+mqqrqM+/p0aPH4cfPPPMMN998cworziwKehGRFHty0ToO1NZz25TWXzkddStXruTGG2/k1VdfZebMmbz22mu8/fbbFBUV8ctfNjd4qrSGbq8TEUmh6po6Hl24js8fN4Dj8tLrFrAfv/Ahyzc1nlssMaOHHMPdl5501PVGjBjBhAkTePHFF1m+fDmTJk0C4ODBg0ycODHQmjoaBb2ISAo9v2wTO6oOcPuUkWGXkla6d+8OxM7Rn3/++Tz11FMtrt/w6vMoDfaTDAp6EZEUcXdK5q/hhEE9mXRsv7DL+YzW9LyTbcKECdx5552sXr2aY489ln379lFRUcFxxx13xHp5eXmsWLGC448/nueeey6SA+QERefoRURSZN7HO1i1tYppU0bq3vVmDBgwgFmzZnHdddcxduxYJkyYwEcfffSZ9e69914uueQSzjnnHAYPHhxCpZlDPXoRkRQpKVvDwJ5duOyUIWGXklYKCgr44IMPDj8/55xzWLx48WfWKy0tPfz4qquu4qqrrkpFeRlPPXoRkRT4aMunlH28g5vOLCCns371Surou01EJAVKysrpmp3F9WcMD7sU6WAU9CIiSbbt02r+sGwjVxfl07tbTtjlSAejoBcRSbLHFq6jtt65dZIGyJHUU9CLiCTRvoO1PPHmOs4/MY+C/t3DLkc6IAW9iEgSPbu0gk/21XD7WRogR8KhoBcRSZL6emfm/HJOGdabohF9wi4n4yU6TW1BQQE7duxo03umTZvG8uXL27W/n/zkJ0c8P/PMM9u1nUQp6EVEkuS1FVtZW7mPaZMLNUBOK6TbFLV1dXWUlJQwevTodr2/cdC/8cYbQZTVZgp6EZEkKSkrZ2jvrkwdMyjsUtJW4ylqN2zYwCuvvBLINLWVlZVccMEFnHbaadxxxx24++FlTzzxBOPHj+fUU0/ljjvuoK6u7vB2f/SjH3HGGWewcOHCw0cRHnjgAb7//e8ffv+sWbP45je/CcAVV1zB5z73OU466SQeeughAO666y7279/PqaeeyvXXX39Ezddccw1z5849vK2bb76ZZ599lrq6Or73ve9x+umnM3bsWP77v/+7vc16BAW9iEgSvLvhE95au5NbJhXQOUu/altyaIrad955h+7du3PPPfcEMk3tj3/8YyZPnsw777zDZZddxvr16wFYsWIFc+bMYcGCBSxbtoysrCyefPJJAPbu3cuYMWN48803mTx58uFtXXXVVfzud787/HzOnDlcc801ADz88MMsXbqUJUuWcN9991FZWcm9995L165dWbZs2eFtH3LttdcyZ84cIDY73+uvv87FF1/MzJkz6dWrF4sXL2bx4sXMmDGD8vLydn32hjQErohIEpTML6dnl85cc/qwsEtpvZfugi3vB7vNQSfD1HtbXOXQFLUAixYtCmya2nnz5h0O5y984Qv06RO7TuL1119n6dKlnH766QDs37+fgQMHApCVlcWXvvSlz2xrwIABjBw5kkWLFjFq1ChWrlx5uMb77ruP5557DoANGzbw8ccf069f85MWTZ06lW9961scOHCAP/3pT5x11ll07dqVV155hffee49nnnkGgN27d/Pxxx9TWJjYbZkKehGRgG38ZD9z39/MrZMK6JmbHXY5ae/QFLUQ/DS1TV0b4e7cdNNN/PSnP/3MstzcXLKysprc1jXXXMPTTz/NCSecwBe/+EXMjNLSUl577TUWLlxIt27dKC4uPuq0ubm5uRQXF/Pyyy8zZ84crrvuusN1/frXv+bCCy9s8f1tpaAXEQnYrAWxw603Z9oAOUfpeadCkNPUnnXWWTz55JP84Ac/4KWXXmLXrl0AnHvuuVx++eV85zvfYeDAgezcuZM9e/YwYsSIFmu78sor+fd//3dGjBjBz372MyDW6+7Tpw/dunXjo48+YtGiRYfXz87Opqamhuzsz/6xd+2111JSUsKSJUuYNWsWABdeeCEPPPAA55xzDtnZ2axatYqhQ4ce8YdQe+jEkYhIgPZU1zD7rQ1cfPJghvbuGnY5GSfIaWrvvvtu5s2bx7hx43jllVcYPjw2z8Do0aO55557uOCCCxg7diznn38+mzdvPmptffr0YfTo0axbt47x48cDcNFFF1FbW8vYsWP54Q9/ePgUBMD06dMZO3bs4YvxGrrggguYN28e5513Hjk5sWGRp02bxujRoxk3bhxjxozhjjvuoLa29uiNdhTW8CrEqCgqKvJE7rXMNKWlpRQXF4ddRsZTOyZObRibivaeP67g+W9MYmx+7za/P9VtuGLFCk488cSU7S8V9uzZ02QPPyqa+j8zs6XuXtTU+urRi4gEpLaunkcWrGV8Yd92hbxIMijoRUQC8tIHW9j4yX6mTc6wc/MSaQp6EZEAuDslZWso7N+d807MC7sckcMU9CIiAViybhfvVuzm1smFdOqUWcPdRvFarahqz/+Vgl5EJAAz5q2hd7dsrhqXH3YpbZKbm0tlZaXCPgO4O5WVleTm5rbpfbqPXkQkQeU79vLqiq3cWXwsXXOaHmwlXeXn51NRUcH27dvDLiUw1dXVbQ7DTJGbm0t+ftv+mFTQi4gk6JEF5WR36sSNZ7Y84Eo6ys7OTniI1XRTWlrKaaedFnYZaUOH7kVEEvDJvoP8z5IKLjt1CAN7RrMXKZkt1KA3s4fNbJuZfdDM8mIz221my+JfP0p1jSIiLXnyzfXsr6lj2pRo9YolOsI+dD8LuB94rIV1ytz9ktSUIyLSegdq65j1xlqmjOrPCYOOCbsckSaF2qN393nAzjBrEBFprxfe3cz2PQeYNmVk2KWINCsTztFPNLN3zewlMzsp7GJEROBvA+Qcn9eTs0b1D7sckWaFPqmNmRUAL7r7mCaWHQPUu3uVmV0M/D93H9XMdqYD0wHy8vI+N3v27OQVnWaqqqro0aNH2GVkPLVj4jpSG364o47/WFLNbWNymJIf3JzzHakNk6UjtuHZZ5/d7KQ2aR30Tay7Fihy9x0trafZ66Q91I6J60hteNPDb/Hhpk9ZcNfZdOkc3L3zHakNk6UjtmHGzl5nZoPMzOKPxxOrtzLcqkSko1u1dQ9/WbWdmyaOCDTkRZIh1KvuzewpoBjob2YVwN1ANoC7PwhcBXzdzGqB/cC1HvYhCBHp8GaWlZOb3YnrJ2TeADnS8YQa9O5+3VGW30/s9jsRkbSwfc8BnntnI1cX5dO3e07Y5YgcVVofuhcRSTePL1xLTX09t2nOeckQCnoRkVaqrqnj8UXrOPeEPEYO6FhXdUvmUtCLiLTSs29XsGtfjYa7lYyioBcRaYX6emdmWTknD+3FGYV9wy5HpNUU9CIirfDnldtYs2Mv06YUEr/rVyQjKOhFRFphRtkaBvfK5eKTB4ddikibKOhFRI7ig427WbRmJ7dMKiA7S782JbPoO1ZE5ChKytbQPSeLa8cPD7sUkTZT0IuItGDz7v28+N5mrjl9OMfkBjd5jUiqKOhFRFowa8Fa6t25ZVJB2KWItIuCXkSkGVUHavntW+uZevJghvXtFnY5Iu2ioBcRacbTizewp7qWaRruVjKYgl5EpAm1dfU8vKCcohF9OG14n7DLEWk3Bb2ISBNeWb6Vil37mTZlZNiliCREQS8i0oQZZWsY0a8b54/OC7sUkYQo6EVEGlm6bifvrP+EWycVktVJw91KZlPQi4g0UlJWTq+u2VxdlB92KSIJU9CLiDSwvnIfL3+4ha+cMZxuOZ3DLkckYQp6EZEGHl5QTlYn4+YzC8IuRSQQCnoRkbjd+2p4eskGLj1lCHnH5IZdjkggFPQiInG/fWs9+w7WMW2ybqmT6FDQi4gAB2vrmfVGOZOO7cfoIceEXY5IYBT0IiLAH9/fxNZPD2iAHIkcBb2IdHjuzox55Rw7sAefHzUg7HJEAqWgF5EOb+FfK1m++VOmTS6kkwbIkYhR0ItIh1cyv5x+3XO44rShYZciEjgFvYh0aKu37eF/P9rGDRNHkJudFXY5IoFT0ItIhzZzfjldOnfihgkjwi5FJCkU9CLSYVVWHeDZtzdy5bh8+vXoEnY5IkmhoBeRDuvxRes4WFvPbZMLwy5FJGkU9CLSIVXX1PH4wnWcc8JAjh3YI+xyRJJGQS8iHdLv39lI5d6DTJui3rxEm4JeRDqc+nqnZH45owcfw8SR/cIuRySpFPQi0uH8ZdV2Vm+r4vazCjHTADkSbQp6EelwSuavYdAxuXzh5CFhlyKSdAp6EelQPty0mwWrK7npzAJyOutXoESfvstFpEOZWVZOt5wsvjJ+eNiliKREqEFvZg+b2TYz+6CZ5WZm95nZajN7z8zGpbpGEYmOLburef7dTXy5aBi9umWHXY5ISoTdo58FXNTC8qnAqPjXdOCBFNQkIhH16MK11Ltz6yTdUicdR+cwd+7u88ysoIVVLgcec3cHFplZbzMb7O6bU1KgiLReTTVd922EHavDrqRJ+w7WMn/RQm4Y1Yfhvgl2hF1R09K5DTNFRrRh937QtU9KdhVq0LfCUGBDg+cV8dcU9CLp5n9u5oxVL8FbYRfStG7ACwDrgfvDraUlZ0DatmGmyIg2vOhemPD1lOwq3YO+qRtcvckVzaYTO7xPXl4epaWlSSwrvVRVVXWoz5ssasfEjNv8V+iaT0XBl8Mu5TPcnac+qqFrZ/jiqJywy2lRdXU1ubm5YZeR0TKhDasqe7IvRb9v0j3oK4BhDZ7nA5uaWtHdHwIeAigqKvLi4uKkF5cuSktL6UifN1nUjgn6uCc793Vj9DV3h13JZ/zpg808sext/uv6cYw+eXDY5bSotLSUcfo+TIja8EhhX4x3NM8DN8avvp8A7Nb5eZE05Y5bev5KKSkrZ1jfrlx40qCwSxFJuVB79Gb2FFAM9DezCuBuIBvA3R8E5gIXA6uBfcAt4VQqIkfnpGPf4Z31u1iybhc/umQ0WZ003K10PGFfdX/dUZY7cGeKyhGRRHg96Rj0JWXl9MztzJdPH3b0lUUiKP1+KkUkM7nT9PWz4dmwcx8vfbCZr5wxnB5d0v2SJJHkUNCLSEAcT7OZ4B5ZsJZOZtx8ZkHYpYiERkEvIsFo8sbX8OzeX8Ocxeu5ZOxgBvfqGnY5IqFR0ItIQNLrYrw5i9ez92Ad06aMDLsUkVClz0+liGQ2dzxNjtzX1NXzyIK1TBjZlzFDe4VdjkioFPQiEpD0uRhv7vub2by7mtvVmxdR0ItIQLyedAh6d2dG2RpGDujO2ccPDLsckdAp6EUkGJ4eV+O9Wb6TDzZ+ym2TC+mkAXJEFPQiEpT0uL2upGwNfbvn8KVx+WGXIpIWFPQiEow0GDBnzfYqXluxja9OGEFudlaotYikCwW9iAQk/KCfOb+cnM6duGHCiFDrEEknCnoRCYaHe+h+596DPLO0gi+eOpQBPbuEVodIulHQi0hAwu3RP7loHQdq67ltSmFoNYikIwW9iATD60PbdXVNHY8uXMfnjxvAcXk9Q6tDJB0p6EUkGCFejPf8sk3sqDqgAXJEmqCgF5GAhHOO3t0pmb+GEwb1ZNKx/VK+f5F0p6AXkWA4hNGjn/fxDlZtrWLalJFYGtzHL5JuFPQiEpBwDt2XlK1hYM8uXHbKkJTvWyQTKOhFJBghzF730ZZPKft4BzedWUBOZ/06E2mKfjJEJCCp79GXlJXTNTuL688YntL9imQSBb2IBCPFs9dt+7SaPyzbyNVF+fTulpOy/YpkGgW9iAQjxbPXPbZwHbX1zq2TNECOSEsU9CISEMctNb9S9h2s5Yk313H+iXkU9O+ekn2KZCoFvYgEI4UD5jy7tIJP9tVw+1kaIEfkaBT0IhKQ1By6r693Zs4v55RhvSka0Scl+xTJZAp6EQlGimave23FVtZW7mPa5EINkCPSCgp6EQlIag7dl5SVM7R3V6aOGZT0fYlEgYJeRIKRgtnr3t3wCW+t3cktkwronKVfXyKtoZ8UEQlGCi7GK5lfTs8unbnm9GFJ3Y9IlCjoRSQgyT1Hv/GT/cx9fzPXjh9Gz9zspO1HJGoU9CISjCTPXjdrQTkAN2uAHJE2UdCLSECSd+h+T3UNs9/awMUnD2Zo765J2YdIVCnoRSQYSZy9bs7iDew5UMvtU9SbF2krBb2IBCNJk9rU1tXzyIK1jC/sy9j83oFvXyTqFPQiEpDkHLp/6YMtbPxkP9Mmqzcv0h4KehEJRhJGxnN3SsrWUNi/O+edmBfotkU6ilCD3swuMrOVZrbazO5qYnmxme02s2Xxrx+FUaeItEbwPfol63bxbsVubp1cSKdOGu5WpD06h7VjM8sCfgOcD1QAi83seXdf3mjVMne/JOUFikjbJGHAnBnz1tC7WzZXjcsPdLsiHclRe/Rm9g0zS8YUUeOB1e6+xt0PArOBy5OwHxFJiWBnryvfsZdXV2zlq2eMoGtOVqDbFulIWnPofhCx3vbT8UPtQf3JPhTY0OB5Rfy1xiaa2btm9pKZnRTQvkUkaAGfo39kQTnZnTpx45kjAtumSEd01EP37v4DM/shcAFwC3C/mT0NzHT3vyaw76Z+IzTuErwNjHD3KjO7GPg9MKrJjZlNB6YD5OXlUVpamkBpmaWqqqpDfd5kUTsm5vP1dRw8WBNIG1YddGa/tY/xgzqzfOkiGp/PizJ9HyZObXikVp2jd3c3sy3AFqAW6AM8Y2avuvv327nvCqDhzBT5wKZG+/20weO5ZvZfZtbf3Xc0UeNDwEMARUVFXlxc3M6yMk9paSkd6fMmi9oxQX+BnJycQNrwN39ezcG6lfzg6omcMOiYxGvLIPo+TJza8EitOUf/LTNbCvwcWACc7O5fBz4HfCmBfS8GRplZoZnlANcCzzfa96BDpwrMbHy83soE9ikiyeKOW+I38hyorWPWG2uZMqp/hwt5kWRoTY++P3Clu69r+KK715tZu6+Gd/daM/sG8DKQBTzs7h+a2dfiyx8ErgK+bma1wH7gWncP9oofEQlIMD+aL7y7me17DvCLq08JZHsiHV1rztE3e++6u69IZOfuPheY2+i1Bxs8vh+4P5F9iEgqJXYx3qEBco7P68lZo/oHVJNIx6aR8UQkcYcPtCUW9AtWV/LRlj3cNqWQ4G7wEenYFPQikrh40Cd6e92MsjX079GFy08dEkRVIoKCXkSC4PUJb2LV1j38ZdV2bpo4gi6dNUCOSFAU9CISgEOH7tv/K2VmWTm52Z24foIGyBEJkoJeRBJ3+NB9+96+fc8BnntnI18al0/f7jkBFiYiCnoRCUBiF+M9vnAtNfX13KY550UCp6AXkcQlcNV9dU0djy9ax7kn5DFyQI9g6xIRBb2IBKH9g+U8+3YFu/bVMG2KevMiyaCgF5HEtfP2uvp6Z2ZZOScP7cUZhX2TUZlIh6egF5HEHb69rm1B/+eV21izYy/TNECOSNIo6EUkAO07Rz+jbA2De+Vy8cmDgy9JRAAFvYgEoR2H7j/YuJtFa3Zyy6QCsrP0q0gkWfTTJSIBaHuPvqRsDd1zsrh2/PDklCQigIJeRILQxtmjN+/ez4vvbeaa04dzTG52kooSEVDQi0gg2tajn7VgLfXu3DKpIGkViUiMgl5EEteGc/RVB2r57VvrmXryYIb17ZbsykQ6PAW9iCSuDSPjPb14A3uqa5mm4W5FUkJBLyIBaF3Q19bV8/CCcopG9OG04X2SX5aIKOhFJACtnL3u5Q+3Ukj26vEAAArsSURBVLFrP9OmjExBUSICCnoRCUTrevQl89cwol83zh+dl/ySRARQ0ItIEFpxjn7pup28s/4Tbp1USFYnDXcrkioKehEJwNHvo58xr5xeXbO5uig/BfWIyCEKehFJ3OFz9E3/SllXuZeXl2/hK2cMp1tO51RWJtLhKehFJHFHmb3ukQVr6dzJuPnMgpSVJCIxCnoRCUDzh+5376vh6SUbuPSUIeQdk5vCmkQEFPQiEoQWRsb77Vvr2XewjmmTdUudSBgU9CISgKavuj9YW8+sN8qZdGw/Rg85JvVliYiCXkQC0MzsdS++t4mtnx7QADkiIVLQi0gAPnvVvbtTUlbOsQN78PlRA8IqTKTDU9CLSOIOX3X/Nwv/WsnyzZ8ybXIhnTRAjkhoFPQikrgmRsabUbaGft1zuOK0oeHUJCKAgl5EAhUL+tXb9vDnldu5YeIIcrOzQq5JpGNT0ItI4hrNXjdzfjldOnfihgkjQixKREBBLyKB+Nuh+x1VB3j27Y1cOS6ffj26hFqViCjoRSQIDc7RP7FoHQdr67ltcmGoJYlIjIJeRAIQC/oaNx5fuI5zThjIsQN7hFyTiEDIQW9mF5nZSjNbbWZ3NbHczOy++PL3zGxcGHWKyFHEb69btaueyr0HmTZFvXmRdBFa0JtZFvAbYCowGrjOzEY3Wm0qMCr+NR14IKVFikjrxA/dL9tex+jBxzBxZL+QCxKRQ8Ls0Y8HVrv7Gnc/CMwGLm+0zuXAYx6zCOhtZoNTXaiIHE0s6HdWw+1nFWJNTG4jIuHoHOK+hwIbGjyvAM5oxTpDgc3JLe1vFj7yf7Dq3anaXbvU7NnDohWzwy4j46kd269b7S7GAt2zjS+cPCTsckSkgTCDvqk/+RvPjNGadWIrmk0ndnifvLw8SktLEyrukIJ1zzPAdwayraTaG3YBEaF2bLedHMPx+Xm8MX9e2KVktKqqqsB+f3VUasMjhRn0FcCwBs/zgU3tWAcAd38IeAigqKjIi4uLg6myeEUw20mi0tJSAvu8HZjaMTHdgVFqw4Tp+zBxasMjhXmOfjEwyswKzSwHuBZ4vtE6zwM3xq++nwDsdveUHbYXERHJdKH16N291sy+AbwMZAEPu/uHZva1+PIHgbnAxcBqYB9wS1j1ioiIZKIwD93j7nOJhXnD1x5s8NiBO1Ndl4iISFRoZDwREZEIU9CLiIhEmIJeREQkwhT0IiIiEaagFxERiTAFvYiISIQp6EVERCJMQS8iIhJhCnoREZEIU9CLiIhEmIJeREQkwhT0IiIiEaagFxERiTAFvYiISIQp6EVERCJMQS8iIhJhCnoREZEIU9CLiIhEmIJeREQkwhT0IiIiEaagFxERiTAFvYiISIQp6EVERCJMQS8iIhJhCnoREZEIU9CLiIhEmIJeREQkwhT0IiIiEaagFxERiTAFvYiISIQp6EVERCJMQS8iIhJhCnoREZEIU9CLiIhEmIJeREQkwhT0IiIiEaagFxERibDOYezUzPoCc4ACYC3wZXff1cR6a4E9QB1Q6+5FqatSREQk84XVo78LeN3dRwGvx58352x3P1UhLyIi0nZhBf3lwKPxx48CV4RUh4iISKSFFfR57r4ZIP7vwGbWc+AVM1tqZtNTVp2IiEhEmLsnZ8NmrwGDmlj0L8Cj7t67wbq73L1PE9sY4u6bzGwg8CrwTXef18z+pgPTAfLy8j43e/bsID5GRqiqqqJHjx5hl5Hx1I6JUxsmTm2YuI7YhmefffbS5k5xJy3oW2JmK4Fid99sZoOBUnc//ijv+Vegyt1/cbTtFxUV+ZIlS4IpNgOUlpZSXFwcdhkZT+2YOLVh4tSGieuIbWhmzQZ9WIfunwduij++CfhD4xXMrLuZ9Tz0GLgA+CBlFYqIiERAWEF/L3C+mX0MnB9/jpkNMbO58XXygPlm9i7wFvBHd/9TKNWKiIhkqFDuo3f3SuDcJl7fBFwcf7wGOCXFpYmIiESKRsYTERGJMAW9iIhIhCnoRUREIkxBLyIiEmEKehERkQhT0IuIiESYgl5ERCTCFPQiIiIRpqAXERGJMAW9iIhIhCnoRUREIkxBLyIiEmEKehERkQhT0IuIiESYgl5ERCTCFPQiIiIRpqAXERGJMAW9iIhIhCnoRUREIkxBLyIiEmEKehERkQhT0IuIiESYgl5ERCTCFPQiIiIRpqAXERGJMAW9iIhIhCnoRUREIkxBLyIiEmEKehERkQhT0IuIiESYgl5ERCTCFPQiIiIRpqAXERGJMAW9iIhIhCnoRUREIkxBLyIiEmGhBL2ZXW1mH5pZvZkVtbDeRWa20sxWm9ldqaxRREQkCsLq0X8AXAnMa24FM8sCfgNMBUYD15nZ6NSUJyIiEg2dw9ipu68AMLOWVhsPrHb3NfF1ZwOXA8uTXqCIiEhEpPM5+qHAhgbPK+KviYiISCslrUdvZq8Bg5pY9C/u/ofWbKKJ17yF/U0HpsefVpnZylbsIyr6AzvCLiIC1I6JUxsmTm2YuI7YhiOaW5C0oHf38xLcRAUwrMHzfGBTC/t7CHgowX1mJDNb4u7NXtQoraN2TJzaMHFqw8SpDY+UzofuFwOjzKzQzHKAa4HnQ65JREQko4R1e90XzawCmAj80cxejr8+xMzmArh7LfAN4GVgBfC0u38YRr0iIiKZKqyr7p8Dnmvi9U3AxQ2ezwXmprC0TNUhT1kkgdoxcWrDxKkNE6c2bMDcm72+TURERDJcOp+jFxERkQQp6CPGzP7RzNzM+oddS6Yxs/8ws4/M7D0ze87MeoddU6bQcNWJM7NhZvZnM1sRHyL8H8KuKROZWZaZvWNmL4ZdS7pQ0EeImQ0DzgfWh11LhnoVGOPuY4FVwD+FXE9G0HDVgakFvuvuJwITgDvVju3yD8Qu4JY4BX20/CfwfVoYWEia5+6vxO/2AFhEbOwGObrDw1W7+0Hg0HDV0gbuvtnd344/3kMsrDQaaBuYWT7wBaAk7FrSiYI+IszsMmCju78bdi0RcSvwUthFZAgNVx0wMysATgPeDLeSjPMrYp2d+rALSSeh3F4n7dPSsMLAPwMXpLaizNOaoZnN7F+IHUZ9MpW1ZbA2DVctLTOzHsCzwLfd/dOw68kUZnYJsM3dl5pZcdj1pBMFfQZpblhhMzsZKATejc8ImA+8bWbj3X1LCktMe0cbmtnMbgIuAc513XvaWm0arlqaZ2bZxEL+SXf/Xdj1ZJhJwGVmdjGQCxxjZk+4+1dDrit0uo8+gsxsLVDk7h1tUoeEmNlFwC+Bz7v79rDryRRm1pnYxYvnAhuJDV/9FY1k2TYW+yv9UWCnu3877HoyWbxH/4/ufknYtaQDnaMX+Zv7gZ7Aq2a2zMweDLugTKDhqgMzCbgBOCf+/bcs3jsVSYh69CIiIhGmHr2IiEiEKehFREQiTEEvIiISYQp6ERGRCFPQi4iIRJiCXkREJMIU9CIiIhGmoBeRhJjZ6Wb2npnlmln3+FzqY8KuS0RiNGCOiCTMzO4hNr54V6DC3X8ackkiEqegF5GEmVkOsTHuq4Ez3b0u5JJEJE6H7kUkCH2BHsTmCsgNuRYRaUA9ehFJmJk9D8wmNl3yYHf/RsgliUic5qMXkYSY2Y1Arbv/1syygDfM7Bx3/9+waxMR9ehFREQiTefoRUREIkxBLyIiEmEKehERkQhT0IuIiESYgl5ERCTCFPQiIiIRpqAXERGJMAW9iIhIhP1/Oc44uqqAN6AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot\n",
    "plot_relu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/color/96/000000/layers.png\" style=\"height:20px;display:inline\"> The Linear Layer\n",
    "---\n",
    "* **Forward Pass**: $$Z^{(k+1)} = f(Z^{(k)}) = (W^{(k)})^T Z^{(k)} + b^{(k)}$$\n",
    "    * $k$ denotes the $k^{th}$ layer with the corresponding weights and bias $W^{(k)}, b^{(k)}$\n",
    "* **Derivative** with respect to *input* $Z^{(k)}$: $$ \\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = \\frac{\\partial ((W^{(k)})^T Z^{(k)} + b^{(k)})}{\\partial Z^{(k)}} = (W^{(k)})^T $$ <br> $$ \\delta^{(k)} = \\delta^{(k+1)} (W^{(k)})^T  $$\n",
    "* **Derivative** with respect to the *parameters* $W^{(k)}, b^{(k)}$: $$ \\frac{\\partial Z^{(k+1)}}{\\partial W^{(k)}} = Z^{(k)} ,\\frac{\\partial E}{\\partial W^{(k)}} = \\delta^{(k+1)} Z^{(k)} $$ <br> $$  \\frac{\\partial Z^{(k+1)}}{\\partial b^{(k)}} = I,  \\frac{\\partial E}{\\partial b^{(k)}} = \\delta^{(k+1)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  <img src=\"https://img.icons8.com/color/96/000000/layers.png\" style=\"height:20px;display:inline\">  The ReLU Layer\n",
    "---\n",
    "* **Forward Pass**: $$ Z^{(k+1)} = \\begin{bmatrix} max(0,Z_1^{(k)}) \\\\ \\vdots \\\\ max(0,Z_n^{(k)}) \\end{bmatrix}, ReLU(Z): \\mathbb{R}^{n} \\to \\mathbb{R}^n $$\n",
    "* **Derivative** with respect to *input* $Z^{(k)}$: $$ \\phi = max(0,Z^{(k)}), \\phi' = heaviside(Z^{(k)}) $$ <br> $$ \\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = diag(\\phi') $$ <br> $$ \\delta^{(k)} = \\delta^{(k+1)} diag (\\phi') $$\n",
    "* **Derivative** with respect to the *parameters*: **NO PARAMETERS!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  <img src=\"https://img.icons8.com/color/96/000000/layers.png\" style=\"height:20px;display:inline\">  The MSE Layer \n",
    "---\n",
    "* **Forward Pass**: $$ E = Z^{(k+1)} = (Z^{(k)} - y)^2 $$\n",
    "*  **Derivative** with respect to *input* $Z^{(k)}$: $$ \\delta^{(k+1)} = \\frac{\\partial E}{\\partial Z^{(k+1)}} = \\frac{\\partial E}{\\partial E} = 1 $$ <br> $$ \\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = 2(Z^{(k)} - y) $$ <br> $$ \\delta^{(k)} = \\delta^{(k+1)} 2(Z^{(k)} - y) =  2(Z^{(k)} - y) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/fast-forward.png\" style=\"height:50px;display:inline\"> Forward Pass\n",
    "---\n",
    "$$ F(X,W) = W_3^T \\phi_2(W_2^T\\phi_1(W_1^TX + b_1) + b_2) + b_3 $$\n",
    "<img src=\"./assets/tut_12_forward.jpg\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/rewind.png\" style=\"height:50px;display:inline\"> Backward Pass\n",
    "---\n",
    "<img src=\"./assets/tut_12_backward.jpg\" style=\"height:400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/color/96/000000/python.png\" style=\"height:50px;display:inline\"> Scikit-Learn Example\n",
    "---\n",
    "We will now demonstrate how to build an MLP in Scikit-Learn, but it is worth noting:\n",
    "* Every ML researcher has to implement Backpropagation at least once in order to get the deep understanding of how this algorithm works.\n",
    "* It is very uncommon working with neural network using Scikit-Learn. The are frameworks dedicated for NNs, which allow taking advantage of GPUs. Such frameworks (=libraries) are **TensorFlow** (maintained by Google) and **PyTorch** (maintained by Facebook) for example.\n",
    "<img src=\"http://www.freelogovectors.net/wp-content/uploads/2018/07/tensorflow-logo.png\" style=\"height:50px\">\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/9/96/Pytorch_logo.png\" style=\"height:50px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _boston_dataset:\n",
      "\n",
      "Boston house prices dataset\n",
      "---------------------------\n",
      "\n",
      "**Data Set Characteristics:**  \n",
      "\n",
      "    :Number of Instances: 506 \n",
      "\n",
      "    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\n",
      "\n",
      "    :Attribute Information (in order):\n",
      "        - CRIM     per capita crime rate by town\n",
      "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
      "        - INDUS    proportion of non-retail business acres per town\n",
      "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
      "        - NOX      nitric oxides concentration (parts per 10 million)\n",
      "        - RM       average number of rooms per dwelling\n",
      "        - AGE      proportion of owner-occupied units built prior to 1940\n",
      "        - DIS      weighted distances to five Boston employment centres\n",
      "        - RAD      index of accessibility to radial highways\n",
      "        - TAX      full-value property-tax rate per $10,000\n",
      "        - PTRATIO  pupil-teacher ratio by town\n",
      "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
      "        - LSTAT    % lower status of the population\n",
      "        - MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
      "\n",
      "This is a copy of UCI ML housing dataset.\n",
      "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/\n",
      "\n",
      "\n",
      "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
      "\n",
      "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
      "prices and the demand for clean air', J. Environ. Economics & Management,\n",
      "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
      "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
      "pages 244-261 of the latter.\n",
      "\n",
      "The Boston house-price data has been used in many machine learning papers that address regression\n",
      "problems.   \n",
      "     \n",
      ".. topic:: References\n",
      "\n",
      "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
      "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# let's load the boston housing dataset\n",
    "boston_dataset = load_boston()\n",
    "# print description of the features\n",
    "print(boston_dataset.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>311</td>\n",
       "      <td>0.79041</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.90</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.544</td>\n",
       "      <td>6.122</td>\n",
       "      <td>52.8</td>\n",
       "      <td>2.6403</td>\n",
       "      <td>4.0</td>\n",
       "      <td>304.0</td>\n",
       "      <td>18.4</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.98</td>\n",
       "      <td>22.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>328</td>\n",
       "      <td>0.06617</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.460</td>\n",
       "      <td>5.868</td>\n",
       "      <td>25.8</td>\n",
       "      <td>5.2146</td>\n",
       "      <td>4.0</td>\n",
       "      <td>430.0</td>\n",
       "      <td>16.9</td>\n",
       "      <td>382.44</td>\n",
       "      <td>9.97</td>\n",
       "      <td>19.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>470</td>\n",
       "      <td>4.34879</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.580</td>\n",
       "      <td>6.167</td>\n",
       "      <td>84.0</td>\n",
       "      <td>3.0334</td>\n",
       "      <td>24.0</td>\n",
       "      <td>666.0</td>\n",
       "      <td>20.2</td>\n",
       "      <td>396.90</td>\n",
       "      <td>16.29</td>\n",
       "      <td>19.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>46</td>\n",
       "      <td>0.18836</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.91</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.448</td>\n",
       "      <td>5.786</td>\n",
       "      <td>33.3</td>\n",
       "      <td>5.1004</td>\n",
       "      <td>3.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>17.9</td>\n",
       "      <td>396.90</td>\n",
       "      <td>14.15</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>294</td>\n",
       "      <td>0.08199</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.92</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.437</td>\n",
       "      <td>6.009</td>\n",
       "      <td>42.3</td>\n",
       "      <td>5.5027</td>\n",
       "      <td>4.0</td>\n",
       "      <td>289.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>10.40</td>\n",
       "      <td>21.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>160</td>\n",
       "      <td>1.27346</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.58</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.605</td>\n",
       "      <td>6.250</td>\n",
       "      <td>92.6</td>\n",
       "      <td>1.7984</td>\n",
       "      <td>5.0</td>\n",
       "      <td>403.0</td>\n",
       "      <td>14.7</td>\n",
       "      <td>338.92</td>\n",
       "      <td>5.50</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>87</td>\n",
       "      <td>0.07151</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.449</td>\n",
       "      <td>6.121</td>\n",
       "      <td>56.8</td>\n",
       "      <td>3.7476</td>\n",
       "      <td>3.0</td>\n",
       "      <td>247.0</td>\n",
       "      <td>18.5</td>\n",
       "      <td>395.15</td>\n",
       "      <td>8.44</td>\n",
       "      <td>22.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>44</td>\n",
       "      <td>0.12269</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.91</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.448</td>\n",
       "      <td>6.069</td>\n",
       "      <td>40.0</td>\n",
       "      <td>5.7209</td>\n",
       "      <td>3.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>17.9</td>\n",
       "      <td>389.39</td>\n",
       "      <td>9.55</td>\n",
       "      <td>21.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>105</td>\n",
       "      <td>0.13262</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.56</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.520</td>\n",
       "      <td>5.851</td>\n",
       "      <td>96.7</td>\n",
       "      <td>2.1069</td>\n",
       "      <td>5.0</td>\n",
       "      <td>384.0</td>\n",
       "      <td>20.9</td>\n",
       "      <td>394.05</td>\n",
       "      <td>16.47</td>\n",
       "      <td>19.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>418</td>\n",
       "      <td>73.53410</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.679</td>\n",
       "      <td>5.957</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1.8026</td>\n",
       "      <td>24.0</td>\n",
       "      <td>666.0</td>\n",
       "      <td>20.2</td>\n",
       "      <td>16.45</td>\n",
       "      <td>20.62</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         CRIM   ZN  INDUS  CHAS    NOX     RM    AGE     DIS   RAD    TAX  \\\n",
       "311   0.79041  0.0   9.90   0.0  0.544  6.122   52.8  2.6403   4.0  304.0   \n",
       "328   0.06617  0.0   3.24   0.0  0.460  5.868   25.8  5.2146   4.0  430.0   \n",
       "470   4.34879  0.0  18.10   0.0  0.580  6.167   84.0  3.0334  24.0  666.0   \n",
       "46    0.18836  0.0   6.91   0.0  0.448  5.786   33.3  5.1004   3.0  233.0   \n",
       "294   0.08199  0.0  13.92   0.0  0.437  6.009   42.3  5.5027   4.0  289.0   \n",
       "160   1.27346  0.0  19.58   1.0  0.605  6.250   92.6  1.7984   5.0  403.0   \n",
       "87    0.07151  0.0   4.49   0.0  0.449  6.121   56.8  3.7476   3.0  247.0   \n",
       "44    0.12269  0.0   6.91   0.0  0.448  6.069   40.0  5.7209   3.0  233.0   \n",
       "105   0.13262  0.0   8.56   0.0  0.520  5.851   96.7  2.1069   5.0  384.0   \n",
       "418  73.53410  0.0  18.10   0.0  0.679  5.957  100.0  1.8026  24.0  666.0   \n",
       "\n",
       "     PTRATIO       B  LSTAT  MEDV  \n",
       "311     18.4  396.90   5.98  22.1  \n",
       "328     16.9  382.44   9.97  19.3  \n",
       "470     20.2  396.90  16.29  19.9  \n",
       "46      17.9  396.90  14.15  20.0  \n",
       "294     16.0  396.90  10.40  21.7  \n",
       "160     14.7  338.92   5.50  27.0  \n",
       "87      18.5  395.15   8.44  22.2  \n",
       "44      17.9  389.39   9.55  21.2  \n",
       "105     20.9  394.05  16.47  19.5  \n",
       "418     20.2   16.45  20.62   8.8  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the target is the MEDV field - median value of owner-occupied homes in 1000$\n",
    "boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)\n",
    "boston['MEDV'] = boston_dataset.target\n",
    "boston.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total training samples: 404, total test samples: 102\n"
     ]
    }
   ],
   "source": [
    "# splitting to train and test (no need for valdiation, sklearn takes it from the train set)\n",
    "# we will use 2 features\n",
    "x = boston[['RM', 'LSTAT']].values  # RM-num rooms, LSTAT-% lower status of the population\n",
    "y = boston['MEDV'].values\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state=5)\n",
    "# scaling\n",
    "x_scaler = StandardScaler()\n",
    "x_scaler.fit(x_train)\n",
    "x_train = x_scaler.transform(x_train)\n",
    "x_test = x_scaler.transform(x_test)\n",
    "print(\"total training samples: {}, total test samples: {}\".format(len(x_train),len(x_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# train an MLP\n",
    "# scoring is using the R^2 score -> the best possible score is 1\n",
    "mlp_reg = MLPRegressor(hidden_layer_sizes=(4, 3,),\n",
    "                       activation='relu',\n",
    "                       solver='sgd',\n",
    "                       batch_size=32,\n",
    "                       learning_rate_init=0.001,\n",
    "                       max_iter=200,\n",
    "                       shuffle=True,\n",
    "                       random_state=5,\n",
    "                       early_stopping=True,\n",
    "                       validation_fraction=0.1,\n",
    "                       verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, loss = 311.17377473\n",
      "Validation score: -5.025490\n",
      "Iteration 2, loss = 268.72251319\n",
      "Validation score: -4.090056\n",
      "Iteration 3, loss = 221.29791391\n",
      "Validation score: -3.238986\n",
      "Iteration 4, loss = 181.01058667\n",
      "Validation score: -2.534500\n",
      "Iteration 5, loss = 148.16609236\n",
      "Validation score: -1.938828\n",
      "Iteration 6, loss = 116.34661767\n",
      "Validation score: -1.140537\n",
      "Iteration 7, loss = 69.69816627\n",
      "Validation score: 0.051736\n",
      "Iteration 8, loss = 25.77497218\n",
      "Validation score: 0.540522\n",
      "Iteration 9, loss = 15.90205835\n",
      "Validation score: 0.577223\n",
      "Iteration 10, loss = 13.61650701\n",
      "Validation score: 0.606377\n",
      "Iteration 11, loss = 12.96205861\n",
      "Validation score: 0.609462\n",
      "Iteration 12, loss = 12.55515383\n",
      "Validation score: 0.623108\n",
      "Iteration 13, loss = 12.32375027\n",
      "Validation score: 0.634032\n",
      "Iteration 14, loss = 12.08402963\n",
      "Validation score: 0.645485\n",
      "Iteration 15, loss = 11.87488789\n",
      "Validation score: 0.648403\n",
      "Iteration 16, loss = 11.43187277\n",
      "Validation score: 0.658779\n",
      "Iteration 17, loss = 11.26966664\n",
      "Validation score: 0.663497\n",
      "Iteration 18, loss = 11.19208310\n",
      "Validation score: 0.674298\n",
      "Iteration 19, loss = 11.13472957\n",
      "Validation score: 0.666873\n",
      "Iteration 20, loss = 11.15699608\n",
      "Validation score: 0.676877\n",
      "Iteration 21, loss = 11.08427796\n",
      "Validation score: 0.675492\n",
      "Iteration 22, loss = 10.88977348\n",
      "Validation score: 0.681250\n",
      "Iteration 23, loss = 10.76642854\n",
      "Validation score: 0.680605\n",
      "Iteration 24, loss = 10.71300689\n",
      "Validation score: 0.686791\n",
      "Iteration 25, loss = 10.80547190\n",
      "Validation score: 0.683455\n",
      "Iteration 26, loss = 10.69159824\n",
      "Validation score: 0.685877\n",
      "Iteration 27, loss = 10.56056444\n",
      "Validation score: 0.687389\n",
      "Iteration 28, loss = 10.42521632\n",
      "Validation score: 0.688858\n",
      "Iteration 29, loss = 10.48946053\n",
      "Validation score: 0.689039\n",
      "Iteration 30, loss = 10.32832853\n",
      "Validation score: 0.696769\n",
      "Iteration 31, loss = 10.18603586\n",
      "Validation score: 0.704198\n",
      "Iteration 32, loss = 10.05947410\n",
      "Validation score: 0.706582\n",
      "Iteration 33, loss = 10.19044429\n",
      "Validation score: 0.706211\n",
      "Iteration 34, loss = 10.01684032\n",
      "Validation score: 0.708056\n",
      "Iteration 35, loss = 10.01955770\n",
      "Validation score: 0.708487\n",
      "Iteration 36, loss = 10.00801058\n",
      "Validation score: 0.710032\n",
      "Iteration 37, loss = 9.87250764\n",
      "Validation score: 0.708921\n",
      "Iteration 38, loss = 9.89419338\n",
      "Validation score: 0.712440\n",
      "Iteration 39, loss = 10.06426791\n",
      "Validation score: 0.714254\n",
      "Iteration 40, loss = 9.93772435\n",
      "Validation score: 0.719554\n",
      "Iteration 41, loss = 9.94274857\n",
      "Validation score: 0.720450\n",
      "Iteration 42, loss = 9.82609378\n",
      "Validation score: 0.722752\n",
      "Iteration 43, loss = 9.64512640\n",
      "Validation score: 0.718780\n",
      "Iteration 44, loss = 9.75976480\n",
      "Validation score: 0.722973\n",
      "Iteration 45, loss = 9.65591169\n",
      "Validation score: 0.723112\n",
      "Iteration 46, loss = 9.78115195\n",
      "Validation score: 0.724884\n",
      "Iteration 47, loss = 9.69174573\n",
      "Validation score: 0.727692\n",
      "Iteration 48, loss = 9.60698201\n",
      "Validation score: 0.721403\n",
      "Iteration 49, loss = 9.61016930\n",
      "Validation score: 0.726065\n",
      "Iteration 50, loss = 9.60675077\n",
      "Validation score: 0.721118\n",
      "Iteration 51, loss = 9.83545758\n",
      "Validation score: 0.726615\n",
      "Iteration 52, loss = 9.54714247\n",
      "Validation score: 0.726878\n",
      "Iteration 53, loss = 9.60241415\n",
      "Validation score: 0.729196\n",
      "Iteration 54, loss = 9.60992152\n",
      "Validation score: 0.729096\n",
      "Iteration 55, loss = 9.42023675\n",
      "Validation score: 0.730153\n",
      "Iteration 56, loss = 9.46561184\n",
      "Validation score: 0.724883\n",
      "Iteration 57, loss = 9.51904526\n",
      "Validation score: 0.729530\n",
      "Iteration 58, loss = 9.39911317\n",
      "Validation score: 0.729139\n",
      "Iteration 59, loss = 9.55070730\n",
      "Validation score: 0.727462\n",
      "Iteration 60, loss = 9.44000212\n",
      "Validation score: 0.729576\n",
      "Iteration 61, loss = 9.28364486\n",
      "Validation score: 0.724464\n",
      "Iteration 62, loss = 10.01713384\n",
      "Validation score: 0.730879\n",
      "Iteration 63, loss = 9.44379694\n",
      "Validation score: 0.731703\n",
      "Iteration 64, loss = 9.31913354\n",
      "Validation score: 0.734320\n",
      "Iteration 65, loss = 9.33178344\n",
      "Validation score: 0.737758\n",
      "Iteration 66, loss = 9.42650623\n",
      "Validation score: 0.735222\n",
      "Iteration 67, loss = 9.40870130\n",
      "Validation score: 0.730186\n",
      "Iteration 68, loss = 9.21574291\n",
      "Validation score: 0.732758\n",
      "Iteration 69, loss = 9.17898517\n",
      "Validation score: 0.731925\n",
      "Iteration 70, loss = 9.26299215\n",
      "Validation score: 0.729684\n",
      "Iteration 71, loss = 9.12722023\n",
      "Validation score: 0.732696\n",
      "Iteration 72, loss = 9.25974393\n",
      "Validation score: 0.731122\n",
      "Iteration 73, loss = 9.21044211\n",
      "Validation score: 0.724620\n",
      "Iteration 74, loss = 9.26834345\n",
      "Validation score: 0.728877\n",
      "Iteration 75, loss = 9.24952838\n",
      "Validation score: 0.729408\n",
      "Iteration 76, loss = 9.29803778\n",
      "Validation score: 0.725133\n",
      "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MLPRegressor(activation='relu', alpha=0.0001, batch_size=32, beta_1=0.9,\n",
       "             beta_2=0.999, early_stopping=True, epsilon=1e-08,\n",
       "             hidden_layer_sizes=(4, 3), learning_rate='constant',\n",
       "             learning_rate_init=0.001, max_iter=200, momentum=0.9,\n",
       "             n_iter_no_change=10, nesterovs_momentum=True, power_t=0.5,\n",
       "             random_state=5, shuffle=True, solver='sgd', tol=0.0001,\n",
       "             validation_fraction=0.1, verbose=True, warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlp_reg.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "regression using mlp - MSE loss: 14.537\n",
      "original price: $27.9\n",
      "predicted price: $29.250\n"
     ]
    }
   ],
   "source": [
    "# MSE loss\n",
    "y_pred = mlp_reg.predict(x_test)\n",
    "loss = np.mean(np.square(y_test - y_pred))\n",
    "print(\"regression using mlp - MSE loss: {:.3f}\".format(loss))\n",
    "# example\n",
    "sample = 1\n",
    "print(\"original price: ${}\".format(y_test[sample]))\n",
    "print(\"predicted price: ${:.3f}\".format(y_pred[sample]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/office/80/000000/pullups--v2.png\" style=\"height:50px;display:inline\"> Backpropagation By Hand  Exercise\n",
    "---\n",
    "We will now solve a simple backpropagation exercise from <a href=\"https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/\"> A Not So Random Walk </a>. Consider the following network:\n",
    "<img src=\"./assets/tut_12_ex_1.png\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 0 - Define Dataset and Network Architecture\n",
    "\n",
    "We will work with one sample for this example, but it can be extended to mini-batches.\n",
    "\n",
    "* Input: $x = \\begin{bmatrix} 1 \\\\ 4 \\\\ 5 \\end{bmatrix} \\in \\mathbb{R}^3$\n",
    "* Output (target): $ t = \\begin{bmatrix} 0.1 \\\\ 0.05 \\end{bmatrix} \\in \\mathbb{R}^2 $\n",
    "* Number of Hidden Layers: 1\n",
    "* Activation: Sigmoid for both hidden and output layers\n",
    "* Loss Functions: MSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 1 - Initialization\n",
    "\n",
    "We initialize the weights and biases to random values as follows:\n",
    "<img src=\"./assets/tut_12_ex_2.png\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 2 - Forward Pass\n",
    "\n",
    "We denote:\n",
    "\n",
    "* $ z_{h_1}, z_{h_2}$ - the weighted output of the hidden neurons **before** activation\n",
    "* $ h_1 = \\sigma(z_{h_1}),  h_2 = \\sigma(z_{h_2})$ - the weighted output of the hidden neurons **after** activation\n",
    "* $ z_{o_1}, z_{o_2}$ - the weighted output of the output neurons **before** activation\n",
    "* $ o_1 = \\sigma(z_{o_1}),  o_2 = \\sigma(z_{o_2})$ - the weighted output of the output neurons **after** activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Input -> Hidden Layer\n",
    "\n",
    "* $z_{h_1} = w_1x_1 + w_3x_2 + w_5x_3 + b_1 = 0.1 * 1 + 0.3 *4 + 0.5 * 5 + 0.5 = 4.3$\n",
    "* $h_1 = \\sigma(z_{h_1}) = \\sigma(4.3) = 0.9866$\n",
    "* $z_{h_2} = w_2x_1 + w_4x_2 + w_6x_3 + b_1 = 0.2 * 1 + 0.4 *4 + 0.6 * 5 + 0.5 = 5.3$\n",
    "* $h_2 = \\sigma(z_{h_2}) = \\sigma(4.3) = 0.9950$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Hidden Layer -> Output Layer\n",
    "\n",
    "* $z_{o_1} = w_7h_1 + w_9h_2 + b_2 = 0.7 *0.9866 + 0.9 * 0.9950 + 0.5 = 2.0862$\n",
    "* $o_1 = \\sigma(z_{o_1}) = \\sigma(2.0862) = 0.8896$\n",
    "* $z_{o_2} = w_8h_1 + w_{10}h_2 + b_2 = 0.8 * 0.9866 + 0.1 * 0.9950 + 0.5 = 1.3888$\n",
    "* $o_2 = \\sigma(z_{o_2}) = \\sigma(1.3888) = 0.8004$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 3 - Calculate Error\n",
    "\n",
    "* MSE: $E = \\frac{1}{2}[(o_1 - t_1)^2 + (o_2 - t_2)^2]$\n",
    "* Derivative w.r.t. the outputs\n",
    "    * $\\frac{\\partial E}{\\partial o_1} = o_1 - t_1 = 0.7896 $\n",
    "    * $\\frac{\\partial E}{\\partial o_2} = o_2 - t_2 = 0.7504 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 4 - Backropagation\n",
    "\n",
    "* Reminder: $\\sigma'(x) = \\sigma(x) (1 - \\sigma(x))$\n",
    "\n",
    "It is basically applying the chain rule!\n",
    "* $\\frac{\\partial E}{\\partial w_7} = \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial w_7} = (o_1 - t_1)(o_1(1-o_1))h_1 = 0.0765$\n",
    "* $\\frac{\\partial E}{\\partial w_8} = \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial w_8} = (o_2 - t_2)(o_2(1-o_2))h_1 = 0.1183$\n",
    "* $\\frac{\\partial E}{\\partial w_9} = \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial w_9} = 0.0772$\n",
    "* $\\frac{\\partial E}{\\partial w_{10}} = \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial w_{10}} = 0.1193$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The second bias is a little more complicated as it is affected by both outputs:\n",
    "* $\\frac{\\partial E}{\\partial b_2} = \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial b_2} + \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial b_2} = 0.1975 $\n",
    "    * $\\frac{\\partial z_{o_1}}{\\partial b_2}, \\frac{\\partial z_{o_2}}{\\partial b_2} = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For our convenience, we will calculate the derivatives w.r.t. $h_1, h_2$ first:\n",
    "* $\\frac{\\partial E}{\\partial h_1} = \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial h_1} + \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial h_1} = 0.1502 $\n",
    "    * $\\frac{\\partial z_{o_1}}{\\partial h_1} = w_7, \\frac{\\partial z_{o_2}}{\\partial h_1} = w_8$\n",
    "* $\\frac{\\partial E}{\\partial h_2} = \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial h_2} + \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial h_2} = 0.0818 $\n",
    "    * $\\frac{\\partial z_{o_1}}{\\partial h_2} = w_9, \\frac{\\partial z_{o_2}}{\\partial h_2} = w_{10}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's continue:\n",
    "* $\\frac{\\partial E}{\\partial w_1} = \\frac{\\partial E}{\\partial h_1} \\frac{\\partial h_1}{\\partial z_{h_1}} \\frac{\\partial z_{h_1}}{\\partial w_1} = 0.0020 $\n",
    "    * $\\frac{\\partial h_1}{\\partial z_{h_1}} = h_1 (1-h_1)$\n",
    "    * $\\frac{\\partial z_{h_1}}{\\partial w_1} = x_1 $\n",
    "* $\\frac{\\partial E}{\\partial w_3} = \\frac{\\partial E}{\\partial h_1} \\frac{\\partial h_1}{\\partial z_{h_1}} \\frac{\\partial z_{h_1}}{\\partial w_3} = 0.0079 $\n",
    "* $\\frac{\\partial E}{\\partial w_5} = \\frac{\\partial E}{\\partial h_1} \\frac{\\partial h_1}{\\partial z_{h_1}} \\frac{\\partial z_{h_1}}{\\partial w_5} = 0.0099 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similarly:\n",
    "* $\\frac{\\partial E}{\\partial w_2} = \\frac{\\partial E}{\\partial h_2} \\frac{\\partial h_2}{\\partial z_{h_2}} \\frac{\\partial z_{h_2}}{\\partial w_2} = 0.0004 $\n",
    "    * $\\frac{\\partial h_2}{\\partial z_{h_2}} = h_2 (1-h_2)$\n",
    "    * $\\frac{\\partial z_{h_2}}{\\partial w_2} = x_1 $\n",
    "* $\\frac{\\partial E}{\\partial w_4} = \\frac{\\partial E}{\\partial h_2} \\frac{\\partial h_2}{\\partial z_{h_2}} \\frac{\\partial z_{h_2}}{\\partial w_4} = 0.0016 $\n",
    "* $\\frac{\\partial E}{\\partial w_6} = \\frac{\\partial E}{\\partial h_2} \\frac{\\partial h_2}{\\partial z_{h_2}} \\frac{\\partial z_{h_2}}{\\partial w_6} = 0.0020 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The first bias:\n",
    "* $\\frac{\\partial E}{\\partial b_1} =  \\frac{\\partial E}{\\partial o_1} \\frac{\\partial o_1}{\\partial z_{o_1}} \\frac{\\partial z_{o_1}}{\\partial h_1} \\frac{\\partial h_1 }{\\partial z_{h_1}} \\frac{\\partial z_{h_1} }{\\partial b_1} + \\frac{\\partial E}{\\partial o_2} \\frac{\\partial o_2}{\\partial z_{o_2}} \\frac{\\partial z_{o_2}}{\\partial h_2} \\frac{\\partial h_2 }{\\partial z_{h_2}} \\frac{\\partial z_{h_2} }{\\partial b_1} = 0.0008$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 5 - Update Weights\n",
    "\n",
    "We will use learning rate $\\alpha = 0.01$:\n",
    "* $w_1 := w_1 - \\alpha \\frac{\\partial E}{\\partial w_1} = 0.1 - 0.01 * 0.0020 = 0.1$\n",
    "* $w_2 := w_2 - \\alpha \\frac{\\partial E}{\\partial w_2} = 0.2 - 0.01 * 0.0004 = 0.2$\n",
    "* $w_3 := w_3 - \\alpha \\frac{\\partial E}{\\partial w_3} = 0.3 - 0.01 * 0.0079 = 0.2999$\n",
    "* $w_4 := w_4 - \\alpha \\frac{\\partial E}{\\partial w_4} = 0.4 - 0.01 * 0.0016 = 0.4$\n",
    "* $w_5 := w_5 - \\alpha \\frac{\\partial E}{\\partial w_5} = 0.5 - 0.01 * 0.0099 = 0.4999$\n",
    "* $w_6 := w_6 - \\alpha \\frac{\\partial E}{\\partial w_6} = 0.6 - 0.01 * 0.0020 = 0.6$\n",
    "* $w_7 := w_7 - \\alpha \\frac{\\partial E}{\\partial w_7} = 0.7 - 0.01 * 0.0765 = 0.6992$\n",
    "* $w_8 := w_8 - \\alpha \\frac{\\partial E}{\\partial w_8} = 0.8 - 0.01 * 0.1183 = 0.7988$\n",
    "* $w_9 := w_9 - \\alpha \\frac{\\partial E}{\\partial w_9} = 0.9 - 0.01 * 0.0772 = 0.8992$\n",
    "* $w_{10} := w_{10} - \\alpha \\frac{\\partial E}{\\partial w_{10}} = 0.1 - 0.01 * 0.1193 = 0.0988$\n",
    "* $b_1 := b_1 - \\alpha \\frac{\\partial E}{\\partial b_1} = 0.5 - 0.01 * 0.0008 = 0.5$\n",
    "* $b_2 := b_2 - \\alpha \\frac{\\partial E}{\\partial b_2} = 0.5 - 0.01 * 0.1975 = 0.4980$\n",
    "\n",
    "#### Repeat the steps until convergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "\n",
    "* Deep Learning - <a href=\"https://www.youtube.com/watch?v=kPXxbmBsFxs\">Machine Learning Lecture 35 \"Neural Networks / Deep Learning\" -Cornell CS4780</a>\n",
    "    * <a href=\"https://www.youtube.com/watch?v=zmu9wR2c7Z4\">Machine Learning Lecture 36 \"Neural Networks / Deep Learning Continued\" -Cornell CS4780</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com\n",
    "* Datasets from <a href=\"https://www.kaggle.com/\">Kaggle</a> - https://www.kaggle.com/\n",
    "* Examples and code snippets were taken from <a href=\"http://shop.oreilly.com/product/0636920052289.do\">\"Hands-On Machine Learning with Scikit-Learn and TensorFlow\"</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}