daily_lab_notebooks/day09-LogisticRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS 136 day09: MAP estimation for Logistic Regression\n",
    "\n",
    "## Outline\n",
    "\n",
    "* **Part 1: Understanding sigmoids and Logistic Regression as a model**\n",
    "* **Part 2: Computing the MAP objective**\n",
    "* **Part 3: Gradient descent for the MAP: Comparing 1st and 2nd order GD**\n",
    "\n",
    "## Takeaways\n",
    "\n",
    "* First-order methods are cheap but require many iterations\n",
    "* Second-order methods are awesome, but still require careful step-size selection\n",
    "* For all gradient descent methods, selecting step sizes is super important. Line search is needed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=3, suppress=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.float_format = '{:,.3g}'.format  # show 3 digits of precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.set_context(\"notebook\", font_scale=1.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: The Probabilistic view of logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task: Binary classification\n",
    "\n",
    "Given $N$ observations of *paired* feature-outcome observations: $\\{ x_n, t_n \\}$.\n",
    "\n",
    "* Each input feature $x_n$ is a scalar real: $x_n \\in \\mathbb{R}$\n",
    "* Each output or \"label\" or \"outcome\" $t_n$ is a scalar binary value: $t_n \\in \\{0, 1\\}$\n",
    "\n",
    "We're also given a feature transform function $\\phi$ which maps each $x_n$ to a vector in $M$-dimensional space. This function is known in advance.\n",
    "\n",
    "We want to make good predictions of new outcomes $t_*$ given new features $x_*$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature transformation\n",
    "\n",
    "For now, we'll assume that the \"feature transform\" $\\phi(x_n)$ just simply passes along the features $x_n$, while adding an additional offset or \"intercept\" feature that is always 1. This is a *simplifying* assumption for today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_features(x_N1, M=2):\n",
    "    ''' Transform raw features into complete features useful for prediction\n",
    "    \n",
    "    Could do any non-linear transformations thought relevant for the problem.\n",
    "    Here we'll just do an identity transform with an extra intercept feature.\n",
    "    \n",
    "    Args\n",
    "    ----\n",
    "    x_N1 : 2D array, shape (N, 1) = (n_examples,)\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    phi_NM : 2D array, shape (N, M) = (n_examples, n_transformed_features)\n",
    "        First column will contain all ones (a bias or intercept feature)\n",
    "        Second column will just include the raw features\n",
    "    '''\n",
    "    assert x_N1.ndim == 2\n",
    "    assert x_N1.shape[1] == 1\n",
    "    N = x_N1.shape[0]\n",
    "    phi_NM = np.zeros((N, M))\n",
    "    phi_NM[:,0] = 1\n",
    "    phi_NM[:,1] = x_N1[:,0]\n",
    "    return phi_NM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. , -1. ],\n",
       "       [ 1. , -0.5],\n",
       "       [ 1. ,  0. ],\n",
       "       [ 1. ,  0.5],\n",
       "       [ 1. ,  1. ]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_N1 = np.linspace(-1, 1, 5)[:,np.newaxis]\n",
    "\n",
    "# Get transformed features using our \"calc_features\" function\n",
    "# * first column will be all 1s, an \"intercept\"\n",
    "# * second column will be the x values\n",
    "calc_features(x_N1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the logistic sigmoid function\n",
    "\n",
    "As discussed in your pre-recorded lectures, the *logistic sigmoid function* is:\n",
    "\n",
    "\\begin{align}\n",
    "\\sigma(r) = \\frac{1}{1 + e^{-r}}\n",
    "\\end{align}\n",
    "\n",
    "It maps real inputs $r \\in (-\\infty, +\\infty)$ to the probability interval $(0, 1)$.\n",
    "\n",
    "We call it a \"sigmoid\" function because it has an S-shaped curve, which you'll plot below.\n",
    "\n",
    "This function is also sometimes called the \"expit\" function. \n",
    "\n",
    "We can use an existing implementation of this function available in SciPy:\n",
    "\n",
    "* expit: https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.expit.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import expit as sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.01798620996209156"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9820137900379085"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.002, 0.018, 0.119, 0.5  , 0.881, 0.982, 0.998])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(np.asarray([-6, -4, -2, 0, 2, 4, 6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1a: Plot the logistic sigmoid function \n",
    "\n",
    "We give you an array of G candidate $r$ values below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "G = 101\n",
    "r_G = np.linspace(-8, 8, G)\n",
    "sigmoid_of_r_G = np.zeros(G) # TODO evaluate sigmoid at each r value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.001, 1.001)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAETCAYAAADOPorfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAa4klEQVR4nO3df1BVdf7H8RcgcC+CgZE1Kg6R1zJpFS/qFFtm2W6JpVDXypp0W6dl1qxlzGq3Zkt3dsYvNeaiZmVthrKz5Y9VtN12ZbatXTMRojXcDQRXUqkUsVKEyw3O948dGfFehE96Ofeuz8cMM5fD+97zms/cuS/uufeeG2FZliUAAAxE2h0AABB+KA8AgDHKAwBgjPIAABijPAAAxigPAICxkCiP/fv3a/z48Tp48GC3M+vXr9fNN9+sMWPGaObMmaqtre3DhACA09leHu+++65mzpypr7/+utuZnTt3avHixVqyZInKyso0btw4zZs3T3xEBQDsYWt5rFy5Us8//7zy8/PPOrdu3TplZ2dr9OjRiomJ0SOPPKKjR49qx44dfZQUAHC6fnbuPDc3V3l5eTp06NBZ52pra3XXXXd1/h4VFaXU1FRVV1fruuuu63E/HR0dam5uVnR0tCIiIs45NwBcCCzLks/nU//+/RUZ2fW5hq3lcemll/Zqrrm5WQ6Ho8s2h8OhlpaWXl+/pqbGOB8AQBoxYoQSEhK6bLO1PHrL6XTK6/V22dba2qr4+PheXT86OlrSfxcgJibmvOc7paqqSunp6UG7/WAK5+xSeOcP5+xSeOcP5+xS8PO3tbWppqam8zH0dGFRHi6XS/v27ev8vb29Xfv375fL5erV9U8dqoqJiVFsbGxQMp4S7NsPpnDOLoV3/nDOLoV3/nDOLvVN/kCH+21/t1Vv5ObmqqSkRBUVFWpra1NhYaEGDBigzMxMu6MBwAUpJMujoaFBGRkZKi8vlyRlZWXpySef1C9+8QtNmDBBFRUVevnllwM+lQIABF9IHLYaOnSoqqurO38fPHiwKisru8zceeeduvPOO/s6GgAggJB85gEACG2UBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMCYreXR2NiovLw8ud1uZWVlafny5QHnOjo69MILL+iGG26Q2+3WjBkzVFZW1sdpAQCn2Foe+fn5SkpK0vbt27V69Wpt2LBBW7du9Ztbt26dSktLtW7dOu3atUtTpkzRT37yE3m9XhtSAwBsK4/6+nqVlZVp/vz5cjgccrlcmjVrloqLi/1m6+rq1NHRIcuyZFmWIiIi5HA4ZFmWDckBAP3s2vHevXuVmJio5OTkzm1paWmqrq72m7377rtVWlqqiRMnKioqSrGxsVq1apUcDofRPquqqs45d08qKiqCvo9gCefsUnjnD+fsUnjnD+fskn35bSuP5uZmOZ3OLtucTqdaW1v9Ztva2jR27Fi9+uqrGjx4sF577TXNmzdPJSUluuSSS3q9z/T0dMXGxp5z9u5UVFTI7XYH7faDKZyzS+GdP5yzS+GdP5yzS8HP7/V6u/2n27bDVnFxcX5F0dLSovj4eL/Zxx9/XJMmTVJaWpocDofmzp2rhIQEvfPOO30VFwBwGtvKw+Vy6dixY2pqaurcVldXp+HDh/vNNjQ0yOfzddnWr18/xcTEBD0nAMCfbeWRmpqqjIwMFRQU6OTJk6qtrVVRUZFycnL8Zm+++WatXLlSBw4ckM/n0xtvvKHGxkbdeOONfR8cAGDvW3ULCwt14sQJTZo0SbNnz5bH45HH41FDQ4MyMjJUXl4uSXrmmWd0/fXX67777tN1112n0tJS/fa3v9Wll15qZ3wAuGDZ9oK5JA0aNCjgBwMHDx6sysrKzt/79++vp59+Wk8//XRfxgMAdIPTkwAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACM2VoejY2NysvLk9vtVlZWlpYvX97tbGlpqaZPny63263s7Gz99a9/7cOkAIDT2Voe+fn5SkpK0vbt27V69Wpt2LBBW7du9Zt7//339fjjj+uJJ55QeXm5Hn74YT3yyCP64osvbEgNALCtPOrr61VWVqb58+fL4XDI5XJp1qxZKi4u9ptds2aNZs2apWuvvVYRERG67bbb9NZbbyk+Pt6G5AAA28pj7969SkxMVHJycue2tLQ0VVdX+81WVVUpPj5eDz74oCZMmCCPx6Pjx49THgBgk3527bi5uVlOp7PLNqfTqdbWVr/Zr7/+WkVFRVq2bJlGjhypTZs2KS8vTyUlJUpJSen1Pquqqs45d08qKiqCvo9gCefsUnjnD+fsUnjnD+fskn35bSuPuLg4v6JoaWkJ+GwiNjZWHo9H3/ve9yRJHo9Ha9eu1fvvv6/77ruv1/tMT09XbGzsuQU/i4qKCrnd7qDdfjCFc3YpvPOHc3YpvPOHc3Yp+Pm9Xm+3/3TbdtjK5XLp2LFjampq6txWV1en4cOH+81eccUV8nq9Xba1t7cHPSMAIDDbyiM1NVUZGRkqKCjQyZMnVVtbq6KiIuXk5PjN3n333SouLtauXbvU3t6uN998U4cOHdLkyZNtSA4AsO2wlSQVFhZq0aJFmjRpkqKjozVz5kx5PB41NDQoOztbq1atUmZmpjwejyTp2WefVUNDg1JTU/XSSy/p0ksvtTM+AFywbC2PQYMGBfxg4ODBg1VZWdllm8fj6SwRAIC9OD0JAMAY5QEAMEZ5AACMUR4AAGPGL5hXV1dr9+7damxslNfrVWJiYufbbi+66KJgZAQAhJhelceBAwf0u9/9Tlu2bNHRo0cVGRmphIQExcTE6Pjx42ppaVFkZKTGjRsnj8ejKVOmKDKSJzUA8L+qx/J46qmntGXLFrndbs2dO1cZGRlyuVyKiorqnGlqatInn3yif/zjH3ruuee0bNky/frXv1ZmZmZQwwMA7NFjecTGxupPf/qThgwZ0u3MwIEDNXHiRE2cOFE///nP9c477+jw4cPnNSgAIHT0eGzpl7/8pYYMGSKfz6epU6eqvLz87DcYGakpU6ZoypQp5y0kACC09PqFiejoaB09elQ+ny+YeQAAYcDoVe2pU6dq27ZtwcoCAAgTRuUxdOhQlZaWqrCwUG1tbcHKBAAIcUaf81i6dKlaWlr04osv6vXXX9eECRN09dVX66qrrtJVV12lYcOGBSsnACCEGJXHRx99pAMHDqi6ulrV1dWqqanR22+/rZdeekkdHR1yOp1+Z8MFAPzvMSqPiIgIDRs2TMOGDdMtt9zSud3r9aqmpkZ79+497wEBAKHnvHyfR2xsrK655hpdc8015+PmAAAhrscXzDdt2mT8feH19fU9fh4EABC+eiyP119/XbfccouWLl2qTz/9tNu5Y8eOqaSkRHl5eZo+fTqfMAeA/2E9HrbavHmz/vjHP2rNmjV66aWXFBcXpyuuuEJJSUmKiYnRN998o4MHD+rzzz/XgAEDdMcdd2jhwoV8vzgA/A/r1Wsep0438tlnn2nHjh3as2ePjhw5opaWFiUnJ2vcuHEaO3asxo8fr+jo6GBnBgDYzOgF8/z8fP34xz/WokWLgpUHABAGjMpjz5492rlzp7799lsdP35cgwYN0rXXXqv4+Phg5QMAhCDjt+q+9dZbevPNNxUZGamOjg5FR0crOztbTz31lBISEoKREQAQYoy/7i8zM1Pbtm3Tv/71L33wwQf61a9+pY8//ljTp09XY2NjMDICAEKMUXlERERozpw5SklJkfTfL4GaPn26Nm/erKSkJL3wwgtBCQkACC1G5TFw4EB99dVXfttjY2M1Z84cvfvuu+ctGAAgdBmVx/e//30tW7ZMR44cCfj35ubm8xIKABDajF4wX7BggWbPnq0pU6ZoxowZuvbaa3XxxRfrP//5j55//nnObQUAFwij8khOTtbGjRu1YsUKbd26Va+99poiIiJkWZZSUlK0cOHCYOUEAIQQ47fqxsTEKD8/X/n5+dq3b5++/PJLJSQkaOTIkYqKigpGRgBAiDmnU7KnpaUpLS3tfGUBAIQJ4895AABAeQAAjNlaHo2NjcrLy5Pb7VZWVpaWL1/e43W2b9+ukSNH6uDBg32QEAAQyHn5GtrvKj8/X0OHDtX27dt14MABPfTQQ0pNTdXUqVMDzh85ckRPPPGEOjo6+jgpAOB0tj3zqK+vV1lZmebPny+HwyGXy6VZs2apuLg44HxHR4cee+wxzZgxo4+TAgDOZFt57N27V4mJiUpOTu7clpaWpurq6oDzL774ohITEykPAAgBth22am5ultPp7LLN6XSqtbXVb7asrEybN2/Wxo0bz+kUKFVVVd/5ur1VUVER9H0ESzhnl8I7fzhnl8I7fzhnl+zLb1t5xMXF+RVFS0uL3xdLNTU16cknn9SSJUuUkJBwTuWRnp6u2NjY73z9nlRUVMjtdgft9oMpnLNL4Z0/nLNL4Z0/nLNLwc/v9Xq7/afbtsNWLpdLx44dU1NTU+e2uro6DR8+vMvc3//+dx09elRz5sxRZmamsrOzJUl33HGHtmzZ0qeZAQD/ZVt5pKamKiMjQwUFBTp58qRqa2tVVFSknJycLnPTpk3TP//5T5WXl6u8vFxvv/22JKmkpES33367HdEB4IJn6+c8CgsLdeLECU2aNEmzZ8+Wx+ORx+NRQ0ODMjIyVF5ebmc8AEA3bP2cx6BBgwJ+MHDw4MGqrKwMeJ3LLrus23dkAQD6BqcnAQAYozwAAMYoDwCAMcoDAGCM8gAAGKM8AADGKA8AgDHKAwBgjPIAABijPAAAxigPAIAxygMAYIzyAAAYozwAAMYoDwCAMcoDAGCM8gAAGKM8AADGKA8AgDHKAwBgjPIAABijPAAAxigPAIAxygMAYIzyAAAYozwAAMYoDwCAMcoDAGCM8gAAGKM8AADGKA8AgDHKAwBgjPIAABijPAAAxmwtj8bGRuXl5cntdisrK0vLly8PONfe3q4lS5boxhtvlNvtlsfj0YcfftjHaQEAp9haHvn5+UpKStL27du1evVqbdiwQVu3bvWbe/nll/Xuu+9qzZo1Kisr04wZM5SXl6eGhgYbUgMAbCuP+vp6lZWVaf78+XI4HHK5XJo1a5aKi4v9Zk+cOKG5c+cqJSVFUVFR8ng8iomJ0Z49e2xIDgDoZ9eO9+7dq8TERCUnJ3duS0tLU3V1td/s448/3uX3nTt36vjx4xo5cmTQcwIA/NlWHs3NzXI6nV22OZ1Otba2nvV6n3zyiR599FHNmzdPQ4cONdpnVVWVcU5TFRUVQd9HsIRzdim884dzdim884dzdsm+/LaVR1xcnF9RtLS0KD4+vtvrbNq0SQsXLtTcuXM1Z84c432mp6crNjbW+Hq9VVFRIbfbHbTbD6Zwzi6Fd/5wzi6Fd/5wzi4FP7/X6+32n27bysPlcunYsWNqamrSwIEDJUl1dXUaPny436xlWVq8eLE2btyopUuXauLEiX0dFwBwGtteME9NTVVGRoYKCgp08uRJ1dbWqqioSDk5OX6zy5cv19atW/XWW29RHAAQAmx9q25hYaFOnDihSZMmafbs2fJ4PPJ4PGpoaFBGRobKy8vV1tamVatW6auvvlJubq4yMjI6f0pKSuyMDwAXLNsOW0nSoEGDAn4wcPDgwaqsrOz8fffu3X0ZCwDQA05PAgAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBGeQAAjFEeAABjlAcAwBjlAQAwRnkAAIxRHgAAY5QHAMAY5QEAMEZ5AACMUR4AAGOUBwDAGOUBADBma3k0NjYqLy9PbrdbWVlZWr58ebez69ev180336wxY8Zo5syZqq2t7cOkAIDT2Voe+fn5SkpK0vbt27V69Wpt2LBBW7du9ZvbuXOnFi9erCVLlqisrEzjxo3TvHnzZFmWDakBAFHPPvvss3bsuL6+XosXL9arr76qAQMG6OKLL5Yk/eEPf9Bdd93VZXbp0qUaNWqU7rnnHkVFRWn8+PFatmyZ0tPTlZKS0uO+2tvbdfjwYQ0aNEj9+vUzyrljxw6tXbtW/fr108GDB896ubm5Wa2trb2e7+3llJQUoxzf5fKePXv0zjvvBO32g3H59HUJ1tr3xeWioiIlJSWF1f0lHO87gdblfK99KN93evNYafLYGWHZ9O97aWmpnnrqKe3cubNz2/vvv6+f/exn+uijj7rMTp8+XXfddZfuv//+zm0zZszQbbfdph/96Ec97qu1tVV79uzRiBEjFBMT0+uMH330ke655x75fD5FRERIkizL6vayJEVERJx1xvRyZGSkhg0bps8++0wdHR3n7XbD/fKZ6xKMtQ/Hy9xfWJczL0dERCgmJkbFxcUaO3asTLS1tammpkajRo2Sw+Ho8jezf8PPo+bmZjmdzi7bnE6nWltbA86eGdzhcKilpaVX+/L5fJKkmpoao4wxMTHauHGj0XUAIFRVVVV9p+v5fL7QKY+4uDi/omhpaVF8fLzfrNPplNfr7bKttbU14Gwg/fv314gRIxQdHd3ZxgCAs7MsSz6fT/379/f7m23l4XK5dOzYMTU1NWngwIGSpLq6Og0fPjzg7L59+zp/b29v1/79++VyuXq1r8jISCUkJJyf4ABwATnzGccptr3bKjU1VRkZGSooKNDJkydVW1uroqIi5eTk+M3m5uaqpKREFRUVamtrU2FhoQYMGKDMzEwbkgMAbHvBXJIOHz6sRYsWadeuXYqOjtbMmTP105/+VA0NDcrOztaqVas6C2LDhg165ZVXdPjwYY0aNUoLFy7UFVdcYVd0ALig2VoeAIDwxOlJAADGKA8AgDHKAwBgjPIAABiz7XMe4aykpETPPPNMl22nPsXe3Sc4p0+frn379ikqKqpz2/r16215x1hpaanmzZvX5f3bkydP1nPPPRdwfv369Vq5cqWOHj2qq6++WosWLQr4eZy+snv3bhUUFOjTTz+V0+nUD3/4Q82fP9/vjAWn2L32jY2Nevrpp7Vr1y45HA7de++9evjhhwPOhtpaS2brbfdan8nkvh5qa2/6ONPna2/hnH3xxRdWVlaWtWnTpoB/93q91qhRo6y6uro+ThbYCy+8YOXn5/dq9sMPP7Tcbrf18ccfW16v11qyZIl16623Wh0dHUFOGdg333xjjR8/3nr99dctn89nff7551Zubq61aNGigPOhsPb333+/9eSTT1otLS1WTU2NdeONN1pbtmzxmwu1tbYss/UOhbU+U2/v66G49mc62+OMHWvPYatzZFmWFixYoJtuuknTpk0LOPPvf/9bsbGxuvzyy/s4XWBVVVVKT0/v1ey6deuUnZ2t0aNHKyYmRo888oiOHj2qHTt2BDllYIcOHVJmZqZmz56tfv366bLLLtO0adO0a9eugPN2r319fb3Kyso0f/58ORwOuVwuzZo1S8XFxX6zobbWktl6273WgfT2vh6Ka3+6nh5n7Fh7Dlt1w+v16siRIwH/dskllyg2NlaStHnzZtXV1WnlypXd3lZVVZXi4uL0wAMPqKamRkOGDNHDDz+sm266yZbse/bskWVZWrt2rXw+nyZOnKgFCxbooosu8puvra3tcor8qKgopaamqrq6Wtddd12f57/88su1YsWKzt8ty9K2bdu6fYDo67U/0969e5WYmKjk5OTObWlpaaqurvabtWOte3LVVVf1er3tXutAentfD8W1P11PjzN2rD3l0Y2PP/5YDzzwQMC/FRUVacKECero6NDKlSuVl5cX8MRhp1iWpWuuuUaPPfaYhg4dqm3btunRRx/VmjVrNGbMmD7NvmLFCg0fPlyTJ0/WtGnT9M033+iJJ57QggUL9Morr/jNn+sZjb+L3qy9JH377bd69tlndeDAAT3//PMB5/t67c/Ul2ePDrae1tvutT7T8ePHe31fD+W1783jjC1r32cHyP4HffDBB1ZGRobV3NxsfN2HHnrIWrx4cRBSmdu9e7d15ZVXWsePH/f72+23326tXbu2yzaPx2O98cYbfRUvoMbGRuuBBx6wpk6dah08eNDoun259n/5y1+sCRMmdNn23nvvWePGjfObDdW1tqzvvt6hdD+3rO7v66G89t/1cSbYa89rHufgz3/+syZPnqy4uLizzhUXF+u9997rss3r9XYe+upL+/bt0//93/+pvb29S5bIyMiAX5R1rmc0Dobq6mrl5ubqoosu0u9//3sNGTKk21m71/70s0efEqyzRwdLb9fb7rU+k8l9PVTXXurd44wda095nIPKykqNGzeux7lTJ4Dcv3+/vv32W23YsEGVlZUBzyAcbImJiVq3bp1Wrlwpn8+nzz//XM8995xycnIClkeondG4sbFRDz74oG699VYVFhae9XChZP/ah/vZo03W2+61PpPJfT0U1/6U3jzO2LL2QXtOcwEYO3as9be//c1v+6FDh6wxY8ZYu3btsizLstra2qyCggLr+uuvt0aPHm3l5uZaH374YV/H7VRZWWnde++91tixY60JEyZYixYtslpbWwNmtyzLWr9+vfWDH/zAGjNmjHXfffdZtbW1dkW3li1bZo0YMcIaPXq0NWbMmM6fKVOmWJYVmmv/5ZdfWnPnzrXGjx9vZWVlWStWrAiY1bJCa60t6+zrHYprfabu7uvhsPanBHqcCYW156y6AABjHLYCABijPAAAxigPAIAxygMAYIzyAAAYozwAAMYoDwCAMcoDAGCM8gAAGKM8ABtdeeWV+s1vfqMbbrhBN9xwQ7ffYwKEGr7PA7DZunXrtGrVKvl8Pl1yySV2xwF6hfIAbJaTk6ORI0faHQMwwmErwGYpKSl2RwCMUR6Azez6siTgXFAeAABjlAcAwBjlAQAwxjcJAgCM8cwDAGCM8gAAGKM8AADGKA8AgDHKAwBgjPIAABijPAAAxigPAIAxygMAYOz/AdBdccb1mxRHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(r_G, sigmoid_of_r_G, 'k.-');\n",
    "plt.xlabel('r'); plt.ylabel('$\\sigma(r)$');\n",
    "plt.ylim([-0.001, 1.001])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Define the Likelihood\n",
    "\n",
    "Each observation (indexed by $n$) is drawn iid from a Bernoulli as follows:\n",
    "\n",
    "$$\n",
    "t_n | w \\sim \\text{BernPMF}\\left( t_n | \\sigma(w^T \\phi(x_n)) \\right)\n",
    "$$\n",
    "\n",
    "where $w \\in \\mathbb{R}^M$ is a weight vector, the same size as our feature vector $\\phi(x_n) \\in \\mathbb{R}^M$\n",
    "\n",
    "The key properties here are:\n",
    "* The *mean* of $t_n$ is a *non-linear activation* of a linear function of the transformed features.\n",
    "\n",
    "## Define the  Prior\n",
    "\n",
    "For now, we'll assume that weights come from a zero mean prior with some covariance determined by a scalar parameter $\\alpha$:\n",
    "\n",
    "$$\n",
    "w \\sim \\mathcal{N}( 0, \\alpha^{-1} I_M )\n",
    "$$\n",
    "\n",
    "A zero mean prior makes sense if we don't know if the slope should be negative or positive.\n",
    "\n",
    "### Parameter we'll treat as a random variable: $w$\n",
    "\n",
    "* Weights vector: $w = [w_1, w_2, \\ldots w_M]^T$, so $w \\in \\mathbb{R}^M$\n",
    "\n",
    "### Parameters we'll treat as fixed: $\\alpha$ \n",
    "\n",
    "* Prior precision $\\alpha > 0$\n",
    "\n",
    "The larger $\\alpha$ is, the more confident we are in the weight values before seeing any data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a simple toy data for analysis\n",
    "\n",
    "Just execute the cells below to get the sense of how to generate toy data from this model\n",
    "\n",
    "We'll manually intervene to set the weight vector to a known value. This makes it easy to tell if our learning is working later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 10  # Number of examples we observe\n",
    "M = 2     # Number of transformed features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the weight vector we'll use to generate our dataset. Set an intercept of 1.2 and a slope of -0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_w_M = np.asarray([0.1, -0.25])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a \"true\" alpha value which controls the prior precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_alpha = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create observed features $x$ and observed outputs $t$ manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_N1 = np.asarray([-5, -0.8, -0.7, -0.6, -0.4, 0.5, 0.8, 0.9, 4.3, 4.1]).reshape((N, 1))\n",
    "phi_NM = calc_features(x_N1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "prng = np.random.RandomState(101) # reproducible random seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_N = (prng.rand(N) < sigmoid(np.dot(phi_NM, true_w_M))).astype(np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the toy dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAFACAYAAABNz1D8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeViU9fr48few77uACIIILqkoggJauWWbWWnSpqallUfLXPrlkpmWHdE6WmqLlWUaWlrqMbWOZZamCIKk4oKC7IuyibIOMPP7I50vI+iMCsPi/bour6v5PJ95nnsegnuez6pQq9VqhBBCiBswauoAhBBCNH+SLIQQQugkyUIIIYROkiyEEELoJMlCCCGETpIshBBC6GTS1AEI0VRmz57N1q1br3u8Xbt2/P777w1+3XvvvZfz589rXpuamuLu7s7999/PlClTsLa21vtcZ86cIScnhwEDBjR4nELUppB5FuJOdfnyZSoqKgDIyckhPDycTz75hICAAACMjY1xcnJq8Ovee++9jBgxgjFjxgBQXl7OsWPHiIiIwNvbm7Vr12JqaqrXuQYPHsyoUaOYPHlyg8cpRG3yZCHuWLa2ttja2gJQWVkJgL29PW3atGn0a1tbW2tdp3379rRv357w8HC2bdtGeHh4o8cgxM2QPgshdCgrK2Pp0qUMGjSIHj168NRTTxEdHQ3AsWPH6Ny5MydPntR6z1NPPcWSJUtu6joBAQEEBgayc+dOTdmuXbsYOXIkAQEB9OzZk2eeeYaEhAQAnnnmGbKysvjoo48YOnQoABkZGUydOpWQkBC6devG4MGD+frrr2/n4wsBSLIQQqdp06bx66+/smjRIrZt20a3bt2YMGECx48fJyAgAH9/f7Zv366pn5aWxt9//82IESNu+lqdOnXizJkzAMTHxzNz5kzCw8PZtWsX69evp7q6mvnz5wPw6aef4u7uzosvvsj333+PWq3mpZdeorq6mvXr17Nr1y6GDx9OREQEiYmJDXMzxB1LkoUQN3D69Gn+/PNP3nnnHfr370/Hjh2ZP38+nTp10nxjf/zxx9mxYwcqlQqA7du3061bNzp16nTT17Ozs6OkpAQAMzMzFi5cyDPPPIOnpycBAQGEh4drkomDgwPGxsZYWVnh5OREeXk5o0aNYsGCBXTq1Alvb29eeeUVAM6ePdsQt0PcwaTPQogbuPpHtnfv3lrlwcHBREVFAfDYY4+xbNkyoqKi6N+/P9u3b+e55567peuVlJRgZ2cHQLdu3bCzs2P16tUkJSWRmprK6dOnNUnpWlZWVowZM4Zdu3Zx7Ngx0tLSOH36NAA1NTW3FI8QV0myEOIGzM3NUSgUXDtoUKVSYWLyz69PmzZtuPvuu9mxYwdWVlbk5OTwyCOP3NL1Tp48yV133QXAwYMHeemll7j//vsJDAxk1KhRJCUl8d5779X73tLSUp5++mkAHnjgAcLCwujRowcDBw68pViEqE2ShRA34Ofnh1qtJj4+nrCwME35kSNH6Nixo+b1iBEjWLBgAfb29gwcOBBHR8ebvlZCQgJHjx7l/fffByAyMpKBAweybNkyTZ0//vijTuK6KioqirNnz3L48GHNKK+rT0YyQl7cLumzEOIGfH19eeCBB5g/fz4HDx4kOTmZRYsWkZiYqNXUNGTIEAA2btyoV8d2aWkpeXl55OXlkZ6eztatW5k0aRKhoaGapxInJydOnjzJ0aNHycjIYN26dXz77beoVCqqq6uBf4bgpqSkcP78eZycnFCr1Wzfvp2srCwOHjzIjBkzAKiqqmroWyPuMDIpTwggMzOTIUOGEBkZSXBwsNaxkpISli5dyu7duykvL6dbt25MmzaNvn37atV7++232b17N/v27bvhpLprZ3BbW1vTrl07HnnkEcaNG4eFhQUABQUFzJs3j+joaExMTOjSpQtPPfUUM2bM4LvvviMwMJCtW7fy73//G1NTUw4cOMAXX3zBt99+S3FxMR4eHoSHh/P777/j6+vLO++804B3TNxpJFkI0UAmT56Ml5cXc+bMaepQhGhw0mchxG3av38/p0+fZt++fezYsaOpwxGiUUiyEOI2fffdd0RHRzN37lx8fHyaOhwhGoU0QwkhhNBJRkMJIYTQSZKFEEIInSRZiGYtJydHaxXW5igzM5POnTsTGxvb1KHU8ddff/HYY48REBDA8OHD+fPPP29YPy0tjcmTJxMSEkJoaChTp04lOztbq05YWBidO3fW+vfJJ5805scQzYAkC9GszZ07l/379zd1GC1SUlIS//rXv3jwwQfZunUrQ4YMYcqUKdddVLCsrIwJEyagUqn45ptvWLNmDUVFRbz44osolUoA8vPzKSwsJDIykr/++kvzb/z48Qb8ZKIpyGgo0azJ+Itbt27dOnr16sW//vUv4J+l1uPi4li3bh3vvvtunfoHDhwgJyeHbdu2YWNjA8DSpUsZOHAgR48epU+fPpw9exYTExMCAgIwMzMz6OcRTUueLESzNXv2bKKioti6dSudO3cGYOzYscyfP5+RI0fSp08ffv/9d8aOHcubb76p9d5ry2JjY3n66acJCAhgyJAh/Oc//9HsjqePc+fO8cILL9C7d2+CgoKYPHkymZmZ9datrq7miy++4P7776dHjx4MHz6cXbt2aY6vXLmS8ePH8+GHH9KnTx9CQkJYtGiR5ts7/NP8NnXqVHr37k2/fv2YPn261qxvfcTGxtaZZR4SEnLd5rKAgAA+//xzTaIAMDL6509EcXEx8M+e315eXpIo7kCSLESz9eabbxIcHMxDDz3EX3/9pSnfvHkzL730EuvXr6/zx7A+p06dYsKECQwdOpSffvqJRYsWsXfvXhYsWKB3LK+//joeHh5s3bqVyMhIioqKmDt3br11IyIiWLNmDTNmzGD79u0MGzaMGTNm8L///U9TJzY2liNHjrB+/XqWLl3KL7/8wqJFi4B/moPGjh2Lubk53333HWvWrKGqqopx48ZpJRRdcnNzcXNz0ypzdXUlNze33vpubm70799fq+zzzz/H0tKSoKAgAM2Txcsvv0z//v0ZOXIk27Zt0zsm0XJJshDNlq2tLaamplhYWGjtVx0QEMCDDz5Ily5dtL4FX8+aNWsYMGAAEyZMwNvbm7CwMBYuXMiWLVu4cOGCXrGkpaXh6OhIu3bt6NKlC++//75mkb7aSkpK2LhxI9OnT+fBBx+kQ4cOTJo0iQcffJDPP/9cU8/Y2Jjly5fTpUsXBgwYwLRp09iyZQslJSXs3LmT8vJyIiIi6NSpE127dmXZsmWcP3+e3bt36xUvQEVFRZ0nADMzM72fqDZs2MC3337L66+/rllFNykpiYsXLzJq1CjWrFnDgw8+yNy5c/nxxx/1jku0TNJnIVocT0/Pm6p/6tQp0tLSCAwM1JRd7QtJTk7G1dVV5zlee+01lixZwoYNGwgNDWXgwIEMHz68Tr1z585RXV1dZ7Okq01mV/n6+uLs7Kx53atXL6qqqkhJSeHkyZMUFhbWWdCwvLyc5OTkOtfMzs5m2LBhmtceHh7s3LkTc3PzOqvNKpVKLC0tdX7eTz/9lA8//JCXX36ZMWPGaMrXrVuHUqnUJOkuXbqQlZXF2rVreeKJJ3SeV7RckixEi3N1VdYbubqEN4CpqSmPP/44L774Yp16tZ9YbuS5557j4YcfZu/evRw8eJDFixezYcMGvv/+e61612vLr6mp0WyWBGj999Xj8E8fgampKX5+fqxatarOea7uU1Gbq6urVlPQ1XO3bdu2zpPThQsX6jRN1aZSqViwYAHff/89r7/+ep17ZmZmVuczdurUqdkPbxa3T5qhRLOmUCh01jE1NdXsWw3//MHLyMjQvPbz8yM5ORlvb2/Nv8LCQpYsWUJpaanO8xcVFfHuu+9SXV1NeHg4y5cvZ+3atZw8eVKzbelVPj4+mJqaEhcXp1UeFxeHn5+f5nVKSgplZWWa10ePHsXCwgJfX1/8/f3JzMzEwcFBE6+zszOLFy/W7L9dm4mJidZna9euHQBBQUEcPnxYq250dHSdJ5ba3nnnHX744QcWL15cJ1FUV1czYMAA1q5dq1WekJCg9dlE6yTJQjRr1tbWZGZmkpWVdd06vXr1Yv/+/ezfv5/U1FQWLlzIpUuXNMdffPFFjh07xuLFi0lOTiYmJoZZs2Zx+fJlvZ4s7O3t2bdvH/Pnz+f06dOkpaWxZcsW7Ozs6NChg1ZdCwsLnn/+eT788EN++eUXUlNT+fzzz9m9ezfPP/+8pt7ly5eZM2cOSUlJ7N27lw8//JBnn30WS0tLhg8fjqOjI9OmTeP48eOcOXOGmTNncvToUfz9/fW+d2PGjCE2NpYVK1aQnJzMRx99xNGjRxk3bpymTmFhIZcvXwbgzz//ZOPGjfzrX//innvu0WzOlJeXR2VlJSYmJgwaNIhPP/2UPXv2kJaWxpo1a9i+fTuvvPKK3nGJlkmaoUSzNnr0aF5//XUefvhhfvvtt3rrvPDCC6SnpzN16lTMzMwYNWqUVht+586dWb16NR999BEbNmzA1taWQYMG8cYbb+gVg5GREatXryYiIoKxY8eiVCrp0aMHa9aswdbWVjOs9KrXXnsNIyMj/v3vf1NUVETHjh1ZtmwZDz30kKaOp6cn7du358knn8TKyoqnnnpK8wfXwsKCr7/+moiICMaNG4dCoaBXr1588803Wv0cunTu3JlVq1bx/vvv88UXX+Dr68tnn32mtR3sqFGj6Nu3LxEREWzfvh2AVatW1WkCW7p0KY899hhz587F3t6e9957jwsXLuDr68uHH37I3XffrXdcomWSVWeFMLCVK1eyfft2fv3116YORQi9STOUEEIInaQZStzR4uPjeeGFF25YZ+LEiUyZMsVAEQnRPEkzlLijVVZWXndG81X29vY4ODgYKCIhmidJFkIIIXSSPgvRrDTV/hWDBw++qT0Z4uPj68ylaG7UajXbtm2joKDgunXKy8t56623CAkJITg4mHnz5uk19wT+mTvSvXv3OuUFBQW89tprBAcHExYWxvvvv681SVK0TJIsRLPSVPtX/PDDDze1J8OYMWNIS0trvIAawJEjR5g1axbl5eXXrTN//nzi4uJYvXo1n332GTExMcyfP1/nuY8ePcrkyZNRqVR1jr366qvk5+fz7bffEhERwZYtW1i5cuVtfRbR9CRZiGalqVpFnZycsLKy0rt+S2i91RXj+fPn2bFjB2+//Ta9evUiODiYRYsWsXPnzhsuh/7BBx8wevRoPDw86hy7+sQVERGhWSTxjTfeYP369Te1Yq5ofiRZiGajKfevqN0MtXLlSiZMmMDHH3/M3XffTZ8+fZg0aZLmD+jgwYOpqalhzpw5jB07Fvhnv4c5c+YQEhJC3759efHFFzl37pzWZ5s2bRpjx44lKCiIDRs2ALBt2zaGDx+uWUl369atmvfo2tNi7NixLF26lFdffZWAgAAGDx7Mxo0bgX+2eh09ejQAQ4YMqfebfVxcHEZGRlqLHvbu3RtjY+MbNrFFRUXx+eefaz57bbGxsbRr1w4vLy9NWd++fSktLeXUqVPXPado/iRZiGajOe1fER0dTWJiIl9//TXLly8nPj6eFStWAP80WRkbGzN37lxWrlyJWq3mpZde4sKFC3z55Zds2LABDw8Pnn32WYqKijTn/Pnnnxk6dCibNm1i6NCh7Nq1izfffJNRo0bx008/MWHCBObNm8dff/2l954W69atw93dna1btzJhwgTeffddduzYQdu2bTXJb/PmzfUODz5//jxOTk6YmppqykxMTHByciInJ+e69+bHH3+kX79+9R47f/58nVV8r76+0TlF8yfzLESzoWv/Cn3V3r8CwNvbm4ULF/Lss88yffp0vZYkV6vV/Pvf/8bGxgZ/f38effRRDh48CPzTZHU1XgcHBw4ePMjx48eJiYnRLN29cOFCDh06xKZNm3j55ZeBf1a4fe655zTX+Oabbxg+fLhmrSZvb29KS0tRqVRae1oYGxsDsGzZMkJCQti9ezePPPII8M+Kr1efqDp27MjRo0dZv349jzzyCPb29pp4ra2t63zG8vJyzM3N65TfzJ4X+pzT1NQUhUJxy+cUzYMkC9HsNcX+FS4uLlobK9nZ2dXZG+KqkydPUlNTwz333KNVXllZqbX/xLWf48yZMzz66KNaZVc72RcuXKjXnhZ9+vTROt6zZ0+9lxGxsLCotx9BqVTeVP+NrnNWVVWhVqtv+ZyieZBkIZq9pti/or59Ka7XYWxqaoqDgwObNm2qc6z2H8hrP8e1e1pce0599rS49hwqlUqvZd0B3N3dKSwspKamRvP0Ul1dTWFhoV4J9Xrn/PPPP7XKru6pcaN9NETzJ30WollpDvtX3Gyc/v7+XLx4EUBzPU9PTz788MM6+0nU1rFjRxISErTK3njjDRYtWqT3nhYnTpzQev/ff//NXXfdVSfG+gQFBVFdXU18fLymLC4uDpVKpdlz+2YFBQWRkZGh1T8RHR2NtbU1Xbp0uaVziuZBkoVoVprD/hX6xpmUlERBQQFhYWH06tWLadOmERsbS0pKCvPmzWPv3r106tTpuueYOHEiP/30Exs3biQ9PZ1Nmzaxc+dOBg8erPeeFlFRUXz22WekpKSwbt06fv75Z01n9tV+ilOnTmn2rKjNzc2Nhx56iDfffJO4uDhiY2N56623eOyxxzRPARUVFeTl5Wl28tMlMDCQXr16MX36dE6cOMGff/7JBx98wPPPP3/dXQRFyyDJQjQro0ePJiUlhYcffpi8vLx667zwwgsMHjyYqVOn8tRTT2FjY1Pv/hVHjhzh8ccfZ9q0afTp06feJp1b9eKLL7Jx40YmTJiAQqHg448/xs/Pj8mTJzNixAhSU1P58ssvb7iD3H333cf8+fNZu3Ytw4YN45tvvmHp0qX069dPs6eFhYUF48aN45lnnqG6urrOnhb3338/x44d47HHHmPjxo28//77DB48GPjnCeuBBx5g+vTpmpFc11q0aBG9e/fmpZdeYsqUKYSGhmqNGtu1axd333233iOZFAoFq1atwtnZmdGjRzN37lxGjRolCzG2ArI2lBAt1NixY2nfvj3vvfdeU4ci7gDyZCGEEEInSRZCCCF0kmYoIYQQOrXKeRYqlYrS0lLNzFEhhBC6qdVqqqqqsLa2xshIu+GpVSaL0tJSrbHoQggh9NepUyetyZ/QSpPF1YXROnXq1OzHdickJNS7gUxz0xLibAkxgsTZ0CTOhqNUKjlz5ozW4pJXtcpkcbXpyczMrN6F0pqblhAjtIw4W0KMIHE2NImzYdXXfC+joYQQQugkyUIIIYROkiyEEELoJMlCCCGETpIshBBC6CTJQgghhE6SLIQQQugkyUIIIYROkiyEEELoJMlCCCGETpIshBBC6NQkySI1NZW+ffuSmZl53To//PADQ4YMoVevXjz77LMkJSUZMEIhhBC1GTxZ7N27l2effZbi4uLr1omOjiYiIoJly5YRExNDnz59ePXVV5F9moQQomkYdNXZTz/9lB07djB9+nTmzZt33XqbN29m2LBh9OzZE4CpU6eyceNGoqKi6Nevn6HCbZGioqL4448/GDhwIGFhYU0dTqtS+94C/PHHHzg7O1NQUGDQ+y0/47rknjQ+gyaLkSNHMmnSJLKysm5YLykpiVGjRmleGxsb4+PjQ2JioiSLG4iKimLIkCEolUrMzMzYs2eP/OI0kNr31tjYGIVCQVVVFSqVCiMjI8zNzQ1yv+VnXNexY8eYMmWK3JNGZtBk4ebmple90tJSLCwstMosLCwoLy+/qeslJCTcVP2mEhcX1yDniYyMpLKyEpVKhVKpJDIyskE3f2qoOBtTY8VY+96qVCoATbPordzvW42zsX/G12opP3ND3pPb0RLu5/U0y82PLC0tqays1CqrqKjAxsbmps7TvXv3Zr/ZSFxcHEFBQQ1yLqVSyVdffaX5hjV69OgGO3dDxtlYGjPG2ve2vieLm7nftxNnY/6Mr9USfubwz5OFubm5Qe7J7WgJ97OysvK6X7KbZbLw9/fn3Llzmtc1NTWkpqbi7+/fhFE1f2FhYezZs0fabhvBtfcWmqbPQn7GdQUEBMg9MYBmmSxGjhzJtGnTePjhh+nRowcff/wxdnZ2BAcHN3VozV5YWJj8sjSSa+9tU91n+RnXJfek8TWLSXnZ2dkEBgYSGxsLQP/+/Zk9ezZz584lJCSEuLg4Vq9eXe8m4kIIIRpfkzxZeHp6kpiYqHnt4eFBfHy8Vp0nnniCJ554wtChCSGEqEezeLIQQgjRvEmyEEIIoZMkCyGEEDpJshBCCKGTJAshhBA6SbIQQgihkyQLIYQQOkmyEEIIoZMkCyGEEDpJshBCCKGTJAshhBA6SbIQQgihU6tOFmtzczlUXExFTU1ThyKEEC1as9zPoqEsTE0lR63GVKEg0MaGMDs7wuztCbOzo/0127YKIYS4vladLK6qUquJuXyZmMuX+SgrCwBPc3P62dnRz96eu+3t6WltjYlRq37QEkKIW9aqk8Xjzs78WlpKUnl5nWOZlZVsystjU14eANZGRoTZ23OvvT33OjjQ19YWS2NjQ4cshBDNUqtOFsv9/TE3NydfqST68mUOXbpEVHExhy5dolSl0qpbqlLxW1ERvxUVAWCmUNDXzo6BDg4MdHAgzM4OK0keQog7VKtOFle5mJkxzNmZYc7OAFSrVBwvLeVAcTEHLl3ir+JiMisrtd6jVKv5q7iYv4qLWZSWhqlCQaidHYMdHBjs6EiInR3m0mwlhLhD3BHJ4lomRkYE2toSaGvLK1fK0isq2HfxIvuLi9lXXMzpsjKt91Sp1ewvLmZ/cTEL09KwNDLiXnt7hjg6cp+jIz1tbDBSKAz/YYQQwgDuyGRRn/YWFoxxd2eMuzsA55VK9l28yB9X/p28JnmUq1T8r6iI/11ptmpjaspQR0eGOjrygJMTbc3NDf4ZhBCisUiyuA43MzPCXV0Jd3UFILeykj8uXmTvxYvsKSoiuaJCq35eVRUbLlxgw4ULAARYW/OgkxMPOjnR394eM2myEkK0YJIs9ORubs7Tbm487eYGQGp5OXsuXtR0iudXVWnVP1ZayrHSUpZmZGBrbMxQR0cednbmIScnPOSpQwjRwkiyuEU+lpZMsLRkQtu2qNRq/i4pYXdhIbuLiviruJgqtVpT93JNDVvy89mSnw9AbxsbHnF2ZrizM+pa9YQQormSZNEAjBQKetva0tvWltne3lyuruaPixf5ubCQnwsLSb2myepISQlHSkp4Jy2NNgoFIxMTedTFhcEODljI8FwhRDMkyaIR2JqYMNzFheEuLqjVak6XlbGrsJBdBQXsKy6mutbTRJ5azeqcHFbn5GBjbMxDTk487uLCMGdn7E3kxyOEaB7kr1EjUygUdLW2pqu1NTO9vCiurmZ3YSE7CgrYUVBAYXW1pm5JTQ2b8/LYnJeHqULBEEdHRrq48LiLC23MzJrwUwgh7nSSLAzM3sREM8qqWqVi7eHDnHZ2Zlt+vtYIqyq1ml8KC/mlsJBJZ84wwMGBUW3aMNLFBXfpIBdCGJhBx3Pm5+czadIkgoKC6N+/P6tWraq3nkqlYvny5dx7770EBQXx5JNPEhMTY8hQDcLEyIhAExM+8PPjbEgICX36sKhDB4JsbLTqqYC9Fy8y5exZPKKiGPT333yalcV5pbJpAhdC3HEM+mQxffp0PD09OXDgABkZGbz00kv4+PjwyCOPaNXbvHkzv/32G5s3b6ZNmzasW7eOl19+mUOHDmHeSr9VKxQKullb083amje9vUmrqGBbfj4/5OVxoLiYq70catBMFHzl7FkGOTjwjJsbI11ccDQ1bcqPIIRoxQz2ZJGWlkZMTAwzZ87EwsICf39/xo0bR2RkZJ26ycnJqFQq1Go1arUahUKBhYXFHTXM1NvCgtc8PdkfGEhmWBir/P0ZYG9P7QVFVMCeixeZmJiI28GDPHr8ON9fuECZbPYkhGhgCrWB/gL/9ttvvPnmm0RHR2vK9u3bx7Rp0zhy5IhW3eTkZF588UWysrIwNjbG3NycL774guDgYL2uVVlZSUJCQoPG31zkq1T8Xl3Nr9XV/F1TQ30/PCtgoIkJD5ua0sfYGGNZs0oIcRO6d+9epxXHYM1QpaWlWFpaapVZWlpScc0cBAClUknv3r358ssv8fDwYM2aNbz66qts376dNm3a6H3N+j5wcxMXF0dQUNBNvecBYAmQWVHB5rw8Nl64wOHLlzXHy4Bd1dXsqq6mrZkZz7i68py7Oz2v6Qtp7DgNrSXECBJnQ5M4G86NvmgbrBnKysqqTmIoLy/Hpp4/YG+88QaDBg3C19cXCwsLpkyZgq2tLb/88ouhwm0RPC0smO7lRUxQEGf79uUdHx86X5OQc5RKlmVm0is2ll6HD7MsI0M6xoUQN81gycLf35+ioiIKCws1ZcnJyfj5+dWpm52dTdU1ay2ZmJhgJnMNrsvPyoq3fHw41bcvsUFBTPP0xO2aDu+jpaXMTE6m3cGDPHb8OFvz8lBeswmUEELUx2DJwsfHh8DAQJYuXUpZWRlJSUmsW7eOESNG1Kk7ZMgQPv30UzIyMqiqquKbb74hPz+fgQMHGircFkuhUBBka8tyPz8yw8L4uUcPnnF1xaLWqrc1wPaCAkaeOEG7qChmJCVxorS06YIWQjR7Bp1nsWLFCkpKShg0aBDjx48nPDyc8PBwsrOzCQwMJDY2FoC3336be+65h9GjR9OvXz9+++03vvrqK9yurPgq9GNiZMSDzs5suOsucvv144tOnbjb3l6rTn5VFcszM+l++DBhR46wJieHklqzyoUQAgw8z8LV1bXeiXgeHh7Ex8drXltbWzNv3jzmzZtnyPBaNXsTEyZ6eDDRw4OzZWWszc3lm9xcsmr1Xxy6dIlDly4xLSmJZ11decnDgyBb2yaMWgjRXMiOPHcgfysr3vP1JS0sjF09evCEiwumtYbXltTU8HlODsFxcQTHxvJFdjZld9AcFyFEXbI21B3MWKHgIWdnHnJ2Jk+pZN3583yRnU1iebmmTlxJCS+dOYM1MPAyNnkAACAASURBVO7MGf7l4UH32xiCK4RomeTJQgDQxsyMmV5enOrbl329ejHWzQ3zWk8bpcAn2dn0iI3l3vh4vjt/XkZSCXEHkWQhtCgUCu5xcGBd165k9+vH8o4d6WJlpVVnf3Exz5w6hfehQ7ydkkJWZWUTRSuEMBRJFuK6nExNmeblxck+ffjM0pLwNm0wqfW0katU8k5aGt5RUTx14gR/Xbx4R63fJcSdRJKF0EmhUBBsYsKmbt1IDw1loY8PHrUmSNYAm/LyuOfvvwmKi2NtTg4VspihEK2KJAtxU9qamzPfx4fU0FA23XUXA66ZtxFfUsLziYl4HzrEgpQUcqWJSohWQZKFuCWmRkaEu7ryR2AgR4ODmdi2rdYs8QtVVSxMS8P70CGeP32aYyUlTRitEOJ2SbIQty3AxoYvOncmMyyMCF9fPGut9KtUq1mbm0vP2FiG/P03uwoKUEm/hhAtjiQL0WCcTU2Z1b4950JC+P6uuwi1s9M6/vvFiww7fpwehw+zRvo1hGhRJFmIBmdqZMSTrq5E9e5NVGAgT7Zpo/U/2smyMiYmJuJz6BD/Tkuj6JoVhoUQzY8kC9GoQu3t+b5bN86FhjLd0xMbY2PNsfNVVbyZkoJXVBTTzp4lvZ6NsIQQzYMkC2EQ3hYWLPPzIyM0lKW+vrSrNfS2VKXio6wsOkZH89ypUyRIZ7gQzY4kC2FQDqam/L/27TkXGso3XbrQ3dpac6xarWb9+fP0iI1l+PHjHCwubsJIhRC1SbIQTcLMyIjn3N05FhzMzh496szX2FFQQP/4eAbEx/O/wkKZGS5EE5NkIZqUQqHgYWdn/ggMJCowkMddXLSO7ysu5sFjx+gTF8eWvDwZditEE5FkIZqNUHt7tnbvzsk+fRjn5qa1DlVcSQlPnDhBj8OH+TY3l2pZ8VYIg5JkIZqdrtbWrO3alaSQEF5t105rZvjJsjLGnj5Nl5gY1uTkyDLpQhiIJAvRbHlbWLDC35/U0FBmt2+Pba1ht8kVFUxMTMQ/OppPs7KolKQhRKOSZCGaPTczMxb7+pJ2ZcVbR5P/2+AxvbKSyWfP0vHQIb5TKmVWuBCNRJKFaDEcTU2Z7+NDWmgoEb6+tDE11RzLUir5oLIS3+hoVmRmUi5JQ4gGJclCtDi2JibMat+elNBQ/tOxI+61JvjlKJW8lpSEb3Q0H0nSEKLB6JUsnnvuOS5dulSnvLCwkJEjRzZ4UELow9rYmBleXpwLCeEjPz9crtnFb1pSEh2vPGlI85QQt8fkegeOHDlCeno6AIcPH2b79u3Y2Nho1UlKSiI1NbVRAxRCF0tjY6Z6ehKcm8sRd3ci0tPJUiqB/3vSWJKezlxvbya2bYu5kTxQC3GzrpssjIyMmDdvnmbm7OLFi7WOKxQKrK2tmTx5cuNGKISezBUKXvH0ZGLbtnyZk8Pi9HSyrySNbKWSV86eZUl6Om96e/O8uztmkjSE0Nt1k0WvXr1ISEgAYPDgwfzwww84OTkZLDAhbpWFsbEmaXyRk8O/09PJvZI0MiormXTmDBHp6cz39masmxsmkjSE0Emv35Lff/+9QRJFfn4+kyZNIigoiP79+7Nq1arr1v3tt994/PHHCQoKYtiwYfz++++3fX1xZ7EwNuZVT0/OhYSwrGNHXGuNnkqtqOCFxETuOnyYDefPyzIiQuhg0K9U06dPx9HRkQMHDrB27Vp+/PFHduzYUafevn37eOONN5g1axaxsbG88sorTJ06ldzcXEOGK1oJS2Njpnt5cS40lCW+vjjXmqdxtryc0adO0TM2lm15ebJgoRDXYbBkkZaWRkxMDDNnzsTCwgJ/f3/GjRtHZGRknbrr169n3LhxhIWFoVAoeOihh9i0aVOdDnYhboa1sTFvXBly+66PD/a1ZoQnlJYy4sQJ+h45wm5Z5VaIOhRqA/1W/Pbbb7z55ptER0dryvbt28e0adM4cuSIVt2wsDAmTpzIgQMHOHHiBO3bt+f1118nJCREr2tVVlZq+luEuJ5LajXfKpVsVCopv+ZYb2NjJpuZ0cvkut16QrRa3bt3x9zcXKvMYL8JpaWlWFpaapVZWlpSUc9WmsXFxaxbt46VK1fStWtXtm3bxqRJk9i+fTteXl56X7O+D9zcxMXFERQU1NRh6NQS4ryVGAcBi5VKlqSn83FWFpVXvjsdqalhYnk5Dzs58V6HDvSytW3SOJuCxNmwWkKcN/qirVcz1KpVqygvv/a7F5SVld2wk7o2KyurOomhvLy83qYlc3NzwsPDCQgIwNTUlPDwcNq3b8++ffv0upYQN8PVzIz/+PmRFBLCy23bai2NvquwkMC4OJ45eZKzZWVNGKUQTUvvZFFWzy/KzSQLf39/ioqKKCws1JQlJyfj5+dXp27Hjh2prKzUKquRGbiikXlaWPBZ586c7tuXMW5uKGod++7CBbrGxPByYiJZ1/y/KcSdQK9kcfr0aZydneuUu7i4cPr0ab0u5OPjQ2BgIEuXLqWsrIykpCTWrVvHiBEj6tR96qmniIyM5PDhw9TU1PD999+TlZXFfffdp9e1hLgdHS0tWd+1K8eCg7V27qsBPs/JwS86mjeSkymsqmq6IIUwML2SxbZt21BemdRUW1lZGWvXrtX7YitWrKCkpIRBgwYxfvx4wsPDCQ8PJzs7m8DAQGJjYwEIDw9nzpw5LFiwgODgYL777js+++wz3Nzc9L6WELeru40NW7t351Dv3gxycNCUV6hUvJ+Rge+hQyxOS6NUnnrFHUCvDu45c+Zw77331pmYd+7cOf7zn/8wfvx4vS7m6upab7OVh4cH8fHxWmVXE4kQTS3Ezo49PXvyW1ERc86dI66kBIDimhrmpqSwIiuL+VfWnTKV2eCilbpusvjmm2+IiIgAQK1W079//3rrBQcHN05kQjQjCoWCoU5O3OfoyI95ecxLSSHxyqCPXKWSyWfPsiwzk0UdOhDepg1GCoWOMwrRslw3WYwZMwYnJydUKhWzZs1i3rx52NYaPnh1IUF95z4I0RooFApGubryuIsLa3NzWZCaqlnhNqm8nKdPnmSpjQ0Rvr4MlbXURCty3WRhbGzM8OHDAWjbti29e/fGRCYoCQGAiZEREz08GO3mxqqsLBanp1NUXQ3AkZIS7j92jCEODkT4+hJsZ9fE0Qpx+/T663/kyJE6s6xrmzRpUoMFJERLYmlszP9r356JbduyNCPjn935VCoA9ly8SJ8jR3iyTRve69ABPyurJo5WiFunV7LYtGmT1uuamhoKCgowMTGhd+/ekizEHc/R1JTFvr682q4dC1NTWZOTw9UxUpvy8tiSn89Lbdvylrd3k8YpxK3SK1nUtzx4SUkJc+bMafbT14UwJA9zc1Z37swMLy/mnjvHlvx8AKrVaj7Jzuab3FyeNTHhP9XV2EqzrmhBbnmcn42NDVOnTuWrr75qyHiEaBU6W1nxY/fuRAUGcq+9vaa8VKXiC6WSjtHRrMrMRHmlyUqI5u62BoWXlpZy+fLlhopFiFYn1N6eP3r1YmePHvSwttaU51VV8WpSEl1jYvhONl8SLYBez8GfffZZnbKSkhJ27twpQ2eF0EGhUPCwszMPODkRef48sxITyb2SHM5VVPDMqVN8kJHBko4dGeLo2MTRClG/W+rgBjA1NSUkJITp06c3eFBCtEbGCgXPubvjl5nJITc33ktLo/DKcNu4khLuO3qUBxwdifD1bdAl0YVoCLfcwS2EuDXmCgUzvLx4wd2dJRkZfJiZScWVvov/FRXxv7g4Rru6sqhDB3yu2QNGiKaid5+FSqXiwIEDfP3116xdu5bDhw83ZlxCtHoOV4bbJoWEMMHdXeuXMfLCBTrHxDAjKYkCWd1WNAN6PVnk5uYyceJEkpKScHJyoqamhuLiYnr27Mnq1atxqLUipxDi5rQzN+fLLl2Y7uXFnHPn+KmgAAClWs3yzEzW5OQwu317XvP0xKrWvuFCGJJeTxYLFizAxsaGX3/9lYMHDxIdHc3PP/+MWq3mvffea+wYhbgjdLO2ZnuPHuzr1YuwWkuEXLqyuq1/dDRfZmdTLcNtRRPQK1lER0ezYMECrf2vO3TowFtvvcXevXsbLTgh7kT3ODhwIDCQrd260blWn0W2UsmLZ84QEBvLf/PzUctwW2FAeiULR0dHLl68WKe8uroa61pjx4UQDUOhUPB4mzYk9OnD6k6dcDcz0xw7VVbG4wkJ3B0fz1/1/F4K0Rj0ShazZs3i7bff5s8//6S0tBSlUsmRI0d4++23GT9+POfPn9f8E0I0HBMjI17y8CApJIR3fXywrdVncfDSJe75+28ePX6cE6WlTRiluBPo1cE9Y8YMampqePnll1HU2tRFrVazZMkSli5dilqtRqFQcOrUqUYLVog7lbWxMfN8fJjk4cF76el8nJVF1ZVmqJ8KCthZUMA4d3cW+vjgZWHRxNGK1kivZPH11183dhxCCD24mJmx3M+Pqe3a8VZKChsuXEANqICvc3PZcP48r3p6Mqd9e5xMTZs6XNGK6JUsYmJimDBhApbXTBAqKSlh5cqVzJkzp1GCE0LUr4OlJd/edReve3kxJyWFXwoLAahUq/kgI4MvsrOZJcNtRQO6bp9FYWEh2dnZZGdn8/HHH3Pu3DnN66v/Dh06xMaNGw0ZrxCill62tvwcEMCenj3pU2uJkOIrw239oqNZnZ1NlQy3Fbfpuk8W+/btY/bs2Zo+ilGjRtVbb+jQoY0TmRBCb4MdHYnu3Zsf8/J4MyWFM+XlAOQolUw6c4ZlGRks6tCBJ9q0wahWv6MQ+rpusnj88cdp3749KpWKMWPG8Mknn2Bfa11+hUKBtbU1fn5+BglUCHFjCoWCUa6uPObiwte5uSxITSVHqQTgTHk5T548SZCNDYt9fRnq5NTE0YqW5oZ9Fr179wZgz549eHh4aI2EEkI0T6ZXhtuOcXNjRWYmEenpFNf8s8lrXEkJ9x87xiAHBxb7+hJSa6a4EDdyy/tZ1Pbuu+82SDBCiIZjZWzMbG9vXvLwYEl6OiuysjSr2+69eJHQI0d43MWFRR060E0m1wod9EoWqampWq9rampIT0+npKSEYcOGNUZcQogG4mRqypKOHZnq6ck7qamsycmh5sqxbfn5/Dc/n7Fubiz08ZEl0cV16ZUs1q9fX6dMrVazcOFCbG9ik5b8/HzmzZvH4cOHsbCw4JlnnuGVV1654XsOHDjAxIkT+fXXX/H09NT7WkIIbe3MzVnduTMzvbyYn5LC93l5AKiBdefPs/HCBV5q25Z53t64m5s3bbCi2bnlPbgVCgXPP/88P/zwg97vmT59Oo6Ojhw4cIC1a9fy448/smPHjuvWz8vLY9asWahk2J8QDaaTlRXfdevGkaAgHqzV0V2lVvNxdja+0dHMTk6mWBYqFLXccrIAyMjIQHlltIUuaWlpxMTEMHPmTCwsLPD392fcuHFERkbWW1+lUvH666/z5JNP3k6IQojrCLwyR2Nfr170r9XRXa5SsSQjg0dLSng3NZVLV7Z+FXc2vZqh3nrrrTplJSUl7N+/nyFDhuh1obNnz+Lg4ICLi4umzNfXl8TExHrrf/LJJzg4OPDkk0/y8ccf63UNIcTNu8fBgf2BgfxcWMi8lBTiS0oAKAXmp6byUWYms9q3Z0q7djIb/A52Sx3cAGZmZowbN47nn39erwuVlpbWWS7E0tKSioqKOnVjYmL473//y5YtWyi9jdU0ExISbvm9hhQXF9fUIeilJcTZEmKE5hmnG7Aa+N3Cgs+USlKvNP8WVFfzxrlzLElJ4XkzM0aammLWzIbRN8f7WZ+WEmd9brmD+2ZZWVnVSQzl5eXY2NholRUWFjJ79myWLVuGra3tbSWL7t27Y97MO+ri4uIICgpq6jB0aglxtoQYofnH2QeYqVLxXkwMa4HUK7+3BWo1H1RW8j0wz9ub593dMTW6rZbsBtHc7+dVLSHOysrK637J1itZAGRnZxMZGcnZs2cxMTHB39+fJ598knbt2un1fn9/f4qKiigsLMTpSqdacnJynRng+/fvp6CggIkTJwJodgN79NFHWbhwIcOHD9c3ZCHELTIxMuIRU1PmBAbyVU4O76alkX2lfzKjspKXz5whIj2dt7y9GevmhkkzSBqicen1Ez516hTDhw9n165dWFpaYmxszH//+18effTR6/Y5XMvHx4fAwECWLl1KWVkZSUlJrFu3jhEjRmjVe+yxxzh69CixsbHExsayc+dOALZv3y6JQggDMzMyYlK7diSFhLC8Y0dcay17nlJRwQuJiXQ9fJh1ubmyN3grp1eyWLJkCffeey+7d+/mo48+YuXKlfz6668MHDiQpUuX6n2xFStWUFJSwqBBgxg/fjzh4eGEh4eTnZ1NYGAgsbGxt/xBhBCNx9LYmGleXpwLDSXC1xdnk/9rlEgqL2fc6dN0O3yYb3NzqZEht62SXs1Qf//9N5s3b8a01rcKU1NTXn75ZZ5++mm9L+bq6sqqVavqlHt4eBAfH1/ve9zd3fV+ehFCNC5rY2NmtW/PZA8PVmZl8UFGBkVXhtaeKS9n7OnTvJuWxlve3jzt6irNU62IXj9JOzu7ejuaS0pKMDHRu9tDCNFK2JqYMNfbm5TQUBb6+GBfa0jt1aRx1+HDrJfmqVZDr2QxcOBA3nnnHdLT0zVlqampvPfeewwYMKDRghNCNG/2JibM9/EhNTSUBdckjbPl5Tx3+jRdDx/m65wc2YCphdMrWcyYMQO1Ws0DDzxAv3796NevHw899BCmpqaypaoQAgdTU96ulTQcrunTeCExkU4xMXyenU2lJI0WSa82JAcHB3788Uf279/P2bNnMTc3x8/Pj7CwsMaOTwjRglxNGtM8PVmRmcnyzExNn0ZqRQUvnznDu2lpvOHlxcS2bbGUGeEtht4dDkZGRgwYMECanYQQOtmbmPCWjw+veXrycVYW/8nIoOBK0sisrGRqUhLvpaUxw8uLSR4e2EnfZ7MnQxWEEI3GzsSEOd7epIaG8r6vL261RlSer6pi1rlzeB86xNspKRRUVTVhpEIXSRZCiEZnY2LC6+3bkxIayko/PzxrLcNzsbqad9LS8I6KYkZSEpn1rBcnmp4kCyGEwVgaG/OKpyfJISF80akTHS0sNMdKVSqWZ2biGx3NxNOnSSwra8JIxbUkWQghDM7MyIiJHh6c7tuXDV27ElBrD/AqtZo1ubl0jYlhZEICh4qLmzBScZUkCyFEkzExMuIZNzf+Dg5mR48eWpswqYGt+fmExcczID6eHfn5qGQpkSYjyUII0eQUCgXDnJ35q3dv9vfqxSPOzlrH9xUXMzwhge6HD/NVTo7M1WgCkiyEEM3K3Q4O/NSjB8eDg/9Z/rzWRkunysqYkJiIz6FDvJeWJiOoDEiShRCiWepuY8O6rl05FxLCTE9PbGtN4MtVKpmXkoJXVBQRFRXSGW4AkiyEEM2al4UFH/j5kR4ayhJfX9qZmWmOlatU/FBVRZeYGB45dow9RUWaDdNEw5JkIYRoERxMTXmjfXvOhYayvksXel2zJfPOwkLuO3qUgNhYvszOprympokibZ0kWQghWhQzIyPGuLtzJCiI33v25B5jYxS1jieUlvLimTN4RUUx59w50mWSX4OQZCGEaJEUCgWDHB1ZbmXF6b59eaVdO6xrbbZUUF1NRHo6HQ4d4omEBPZKE9VtkWQhhGjxOllZsdLfn8ywMP7TsSM+tWaGq4At+fkMPnqU7ocP80lWFpevLGoo9CfJQgjRajiYmjLDy4ukkBC2de/OEAcHreMny8qYcvYsHlFRTD5zhuMlJU0UacsjyUII0eoYKxQ85uLCb716kdCnD5M9PLCpNfS2pKaGT7OzCYiN5e4jR/g2N5cK6RC/IUkWQohWrZu1NR936kRWWBgr/fzoamWldfzApUuMPX2adldWvT1dWtpEkTZvkiyEEHcEOxMTXvH05ESfPuzt2ZPwNm20ZocXVlezPDOTrocPc098POtyc2X4bS2yPZUQ4o6iUCgY6OjIQEdHcisr+So3ly9yckitNcT2r+Ji/iouZurZszzr5sbEtm3pbWvbhFE3PXmyEELcsdzNzZnr7U1SSAg/9+jBCBcXau8KXnylbyMoLo7A2FhWZGaSr1Q2WbxNSZKFEOKOZ6xQ8KCzM1u6dyczLIzFHTpobcwE8HdJCa8lJeERFcUTCQn8lJ9P1R20+q0kCyGEqMXd3JzZ3t6cCQlhb8+ejHFzw6LWZL8qtZot+fk8mpBAu6gopp09S/zly61+wp9Bk0V+fj6TJk0iKCiI/v37s2rVqnrr1dTUsGzZMgYOHEhQUBDh4eEcOnTIkKEKIe5wRlf6NtZ37UpOWBifdepEyDX9FnlVVXyUlUXvuDh6HD5MRFpaq11exKDJYvr06Tg6OnLgwAHWrl3Ljz/+yI4dO+rUW716NXv37mX9+vXExMTw5JNPMmnSJLKzsw0ZrhBCAP9M9nvZw4NDQUGc7NOHWV5eeNRa/RbgRFkZc1JS8D50iAHx8XyenU1hK9pvw2DJIi0tjZiYGGbOnImFhQX+/v6MGzeOyMjIOnVLSkqYMmUKXl5eGBsbEx4ejpmZGSdOnDBUuEIIUa+u1tZEdOxIelgYuwMCGO3qipWR9p/SfcXFvHzmDO4HD/Lo8eNsOH+eshbeTGWwobNnz57FwcEBFxcXTZmvry+JiYl16r7xxhtar6Ojo7l8+TJdu3Zt9DiFEEIfxgoFQ52cGOrkREl1NVvz81l//jx7ioq42u1dpVbzU0EBPxUUYA48euIET7Zpw8POzlgZG9/o9M2OwZJFaWkplpaWWmWWlpZU6GjfO378OK+99hqvvvoqnp6eN3XNhISEm46zKcTFxTV1CHppCXG2hBhB4mxozSHOu4DFQL61Nb9VV/NzVRUnao2WqgQ25+WxOS8PS+AeExPuMzGhn4kJFgrFdc7afBgsWVhZWdVJDOXl5dhcs4FJbdu2bWPhwoVMmTKFiRMn3vQ1u3fvjrm5+U2/z5Di4uIICgpq6jB0aglxtoQYQeJsaM0xzgeA94Hk8nK+v3CB7y5c4HitZUTKgd3V1eyursbKyIhhzs480aYNDzs5YWvSdHOlKysrr/sl22BR+fv7U1RURGFhIU5OTgAkJyfj5+dXp65arSYiIoItW7bw4YcfMmDAAEOFKYQQDaajpSVzvb2Z6+3NppgYTrZpw/d5eZyutWd4mUqleeIwVyi4z9GRkW3aMNzZmTbXdKI3JYN1cPv4+BAYGMjSpUspKysjKSmJdevWMWLEiDp1V61axY4dO9i0aZMkCiFEq9DR2JgFHTpwsk8fjgUH85a3N12uWdSwUq1mZ2EhExITcT94kHvj41mWkUFSreTSVAw6dHbFihWUlJQwaNAgxo8fT3h4OOHh4WRnZxMYGEhsbCxKpZIvvviCixcvMnLkSAIDAzX/tm/fbshwhRCiwSkUCnrY2PBOhw6c6tuXE336sNDHp86e4ipgf3ExM5OT8Y+J4a6YGGYnJ3OguJiaJhhZZdDGMVdX13on4nl4eBAfH695fezYMUOGJYQQTeYua2vmW1sz38eHc+XlbMnL478FBRwoLqZ2SjhVVsapsjKWZGTgZGLCQ05ODHN25gEnJ5xMTRs9Tll1VgghmglfS0teb9+e19u357xSyY6CAv6bn89vRUWU1xpZVVhdTeSFC0ReuIAREGpnx0NOTjzk7EygjQ1GjTC6SpKFEEI0Q25mZkxo25YJbdtSVlPDnqIifiooYEdBATm1Vr5VAQcvXeLgpUu8lZpKG1NT7nd05AEnJ4Y6OuLeQCNCJVkIIUQzZ2VszHAXF4a7uKBWq4kvKWFHQQG7CgqIuXxZq7kqr6pK89QB0MPamvsdHbnP0ZF7HBywvsXJgJIshBCiBVEoFPS2taW3rS3zfXzIVyr5X1ERPxcUsLuoiLxr1qM6XlrK8dJS/pOZialCQZidHUMcHRns4EBfOzvMjPQb5yTJQgghWjAXMzNGu7kx2s0N1ZWnjl8KC/m1sJCDly5RVWvkVJVazb7iYvYVF/M2YGVkxN329gx0cGCggwMBN5jXIclCCCFaCSOFgiBbW4JsbXnT25uS6mr+LC7m18JC9ly8SEKtWeTwz4TA3UVF7C4qAsDX2JhN18z9uEqShRBCtFI2JiYMc3ZmmLMzALmVlfx+8SJ7L17k96Iizl27BNMNdv6TZCGEEHcId3NznnVz41k3NwBSy8v5s7iYPy5e5M+LF6msrLzueyVZCCHEHcrH0hIfS0vGubsDUFpezumTJ+utK3twCyGEAMDkBiOjJFkIIYTQSZKFEEIInSRZCCGE0EmShRBCCJ0kWQghhNBJkoUQQgidJFkIIYTQSZKFEEIInSRZCCGE0EmShRBCCJ0kWQghhNBJkoUQQgidJFkIIYTQSZKFEEIInSRZCCGE0EmShRBCCJ0kWQghhNDJoMkiPz+fSZMmERQURP/+/Vm1atV16/7www8MGTKEXr168eyzz5KUlGTASIUQQtRm0GQxffp0HB0dOXDgAGvXruXHH39kx44ddepFR0cTERHBsmXLiImJoU+fPrz66quo1WpDhivETYuKimLx4sVERUU1+rka8lqtjdybhmdiqAulpaURExPD8uXLsbCwwN/fn3HjxhEZGckjjzyiVXfz5s0MGzaMnj17AjB16lQ2btxIVFQU/fr1M1TIQtyUqKgohgwZglKpxMzMjD179hAWFtYo52rIa7U2cm8ah8GSxdmzZ3FwcMDFxUVT5uvrS2JiYp26SUlJjBo1SvPa2NgYHx8fEhMTbypZJCQk3F7QBhIXF9fUIeilJcTZlDFGRkZSWVmJSqVCqVQSGRmJmZlZvXV1xanrXDdzrdvREn7moB2noe7NrWgp97M+BksWpaWlWFpaapVZWlpSUVFRb10LCwutMgsLY6GDKAAACcRJREFUC8rLy2/qmt27d8fc3PzmgzWguLg4goKCmjoMnVpCnE0do1Kp5KuvvtJ8ox09enS98egTp65z6Xut29HU91Nf18ZpiHtzK1rC/aysrLzul2yDJQsrK6s6iaG8vBwbG5s6dS0tLamsrNQqq6ioqLeuEM1FWFgYe/bs4Y8//mDgwIG31fSh61wNea3WRu5N4zBYsvD396eoqIjCwkKcnJwASE5Oxs/Pr966586d07yuqakhNTUVf39/Q4UrxC0JCwtrsD9Ous7VkNdqbeTeNDyDjYby8fEhMDCQpUuXUlZWRlJSEuvWrWPEiBF16o4cOZLt27cTFxeHUqlkxYoV2NnZERwcbKhwhRBC1GLQobMrVqygpKSEQYMGMX78eMLDwwkPDyc7O5vAwEBiY2MB6N+/P7Nnz2bu3LmEhIQQFxfH6tWrMTU1NWS4QgghrjBYMxSAq6trvRPxPDw8iI+P1yp74okneOKJJwwVmhBCiBuQ5T6EEELoJMlCCCGETpIshBBC6CTJQgghhE6SLIQQQugkyUIIIYROkiyEEELoJMlCCCGETpIshBBC6CTJQgghhE4GXe7DUK5uv6pUKps4Ev1cuxx7c9US4mwJMYLE2dAkzoZx9W9mfVtYK9StcGPry5cvc+bMmaYOQwghWqROnTpha2urVdYqk4VKpaK0tBRTU1MUCkVThyOEEC2CWq2mqqoKa2trjIy0eylaZbIQQgjRsKSDWwghhE6SLIQQQugkyUIIIYROkiyEEELoJMlCCCH+f3v3HtLU+8cB/J1+nblCTJL8S8iUTOwyE0eaUmoYJek/EkHpksr6IyxNnaZiYPeyKDPMLiaO0i4wtTCiP4LCFEnRBE0iIqTIe215nfv+0c/Rcu4YrZ3j9/d+gbD7897D8+xzzjN3DglisSAiIkEsFkREJIjFgoiIBLFYSMDg4CCysrIQHh4OpVKJjIwM6PV6sWNZZDAYsGvXLqjVarGjWNTa2oqdO3ciKCgIYWFhKCgowPDwsNixAAC9vb3Yv38/1q5di9DQUBQVFYkdySIp9+GvpD4e59LcFsJiIQHJyckYGRnB48ePUVdXh+7ubpw9e1bsWBYVFRWhqalJ7BgWffv2DXv37kVUVBRevXqFe/fuobm5GefOnRM7GgDg8OHDWLRoEV6+fImysjI8ePAAtbW1YscyI/U+/JWUxyMwt+a2kP/kUWfnktbWVnR0dODWrVuQy+UAgIsXL0py66O+vh51dXXYvHmz2FEs6u7uRlBQEFQqFQDA09MTsbGxuH//vrjBAHz48AGNjY24cOEC5s+fD19fXyQmJkKj0SAmJkbseCZS7sNfSX08zqW5PRssFnYwOjqKnp4ei/e1tbVh2bJlqKioQGVlJcbGxhAdHY20tDTJZPTw8IBOp8PRo0dRVFQEjUYDg8Fg13xTrOVcunQprly5YrpuNBrx9OlTBAQE2CvejLq6uuDm5obFixebbvP29kZnZ6eIqabz8/OTbB/+rK+vTxLj0RqpzG1bYbGwg5aWFiQkJFi8LyUlBR0dHQgMDERNTQ0GBwdx8OBBnD59Gvn5+ZLIWFZWhtLSUqhUKvj7+9stkyXWcpaXl0OpVAIAJiYmkJ+fj48fP0piCUWv18PFxcXsNhcXF4yMjIiUSJjU+nDK5OQk0tPTJTEerRkaGpLE3LYVFgs7UCqVM25B3rhxAw4ODsjMzISTkxPkcjkOHDiAnJwcuw4oaxmvXr0KmUw244e0PVnLOaWvrw+pqano7++HRqPBkiVL7JRuZnK5fFphGB4exsKFC0VKZJ0U+3BKSUmJZMajNc7OzpKY27bCYiEyHx8fGI1GjI2NwcnJCcCPLTop0Wq1+PLlC4KCggDA9KHX3t6OmpoaMaNN09nZiX379mH16tUoLi7GggULxI4EAPD19cXAwAD6+/vh7u4OAHj37h18fHxETjadVPtwylwZj3Nhbv8WI4lqdHTUGBkZaUxNTTV+//7d+PnzZ2NcXJzx+PHjYkebUXZ2tjEzM1PsGNP09PQYQ0JCjCdOnBA7ikXbt283ZmZmGvV6vbGrq8u4YcMGY1VVldixzEi9Dy2R6nici3PbGv7rrMhkMhkqKiowPj6OyMhIbNu2DQqFYs5+CSamu3fvore3F5WVlVAoFKa/rVu3ih0NAHDp0iXodDps3LgRKpUK8fHxiI+PFzuWGan34VzyX5vbPFMeEREJ4p4FEREJYrEgIiJBLBZERCSIxYKIiASxWBARkSAWCyIiEsRiQWTB8uXLodVq7dbexMQEysrK7NYe0e/i7yyILOjp6YGrqyucnZ3t0l51dTXS09MldxRaoik8NhSRBR4eHnZtj9tsJHVchiKy4OdlKLVajezsbBQUFECpVGLdunU4cuQIdDodAKChoQH+/v6oq6tDREQEFAoFkpOT8enTJ9PrRUREoLi42KyNqdsaGhqQkZFhavfhw4emy5cvX7aY7/r16wgICEBHRweAH+f5iImJQXJysm07guh/WCyIZqG6uhoGgwF37txBbm4unjx5gvLyctP9BoMB58+fR0FBATQaDYaGhrBnz55ZHWVUoVAgLy8PAPDixQts2bLFdDkpKcnic5KSkrBq1Srk5OTAYDCgsLAQAwMDOHnypA3eLdF0XIYimgU3Nzfk5OTA0dER3t7eqK2tRUtLi9ljsrKyEBISAgA4c+YMNm3ahPr6eoSFhVl9bZlMZjqvxc/LX9aWwhwcHHDq1CnExsYiKysLNTU1KCkpMR3+nMjWuGdBNAteXl5wdHQ0XXd1dcX4+LjZY4KDg80e7+7ujrdv3/7VTGlpadBqtYiPj0d4ePhfa4uIxYJoFmQy2bTbfv1S+p9/zHfUJycn4eAw8xSzxYlw2tvb4ejoiKamJoyOjv7x6xHNhMWCyEbevHljuvz+/XsMDg5ixYoVAAAnJyfTF+IAoNPp0NfXZ7o+b968327v2bNn0Gq1KC0txdevX1FYWPgH6YmsY7EgspFjx47h9evXaGtrQ0ZGBlauXGlamlqzZg0ePXqE5uZmdHV1Qa1Wmy1rTZ26tK2tDXq9HsCP33pMXf5Vf38/8vLykJCQgNDQUOTm5uL27dtobGz8y++S/l+xWBDZSFxcHA4dOoTExER4eXnh2rVrpmWo1NRU+Pn5QaVSYffu3QgMDERgYKDpuUqlEsHBwdixYweqqqoAAOvXr8fNmzcttpWfnw+5XI6UlBQAQHR0NKKioqBWq832YIhshb/gJvpDDQ0NSEhIwPPnz+Hp6Sl2HKK/gnsWREQkiMWCiIgEcRmKiIgEcc+CiIgEsVgQEZEgFgsiIhLEYkFERIJYLIiISNC/uhwBR1jh2fkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x_N1, t_N, 'k.');\n",
    "\n",
    "ax_h = plt.gca()\n",
    "ax_h.set_xlim([-8, 8]); ax_h.set_xticks([-6, -4, -2, 0, 2, 4, 6]);\n",
    "ax_h.set_ylim([-.1, 1.1]);\n",
    "\n",
    "xgrid_G1 = np.linspace(-8, 8, 100)[:,np.newaxis]\n",
    "plt.plot(xgrid_G1, sigmoid(np.dot(calc_features(xgrid_G1), true_w_M)), 'c-', linewidth=3);\n",
    "\n",
    "plt.xlabel('input: x');\n",
    "plt.ylabel('output: t');\n",
    "plt.title(\"Toy Data\\n true_slope %.2f \\n true intercept %.2f\" % (\n",
    "    true_w_M[1], true_w_M[0]));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion 1b: What about this observed dataset of 10 points would prefer a *negative* slope vs. a positive slope?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO discuss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: MAP estimation : View as optimization problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is NO closed form for the posterior over weights $p( w | t)$. \n",
    "\n",
    "However, we can evaluate (and thus optimize) the MAP objective, since this doesn't require knowing the full posterior.\n",
    "\n",
    "Let's see how. Begin with the MAP optimization problem:\n",
    "\n",
    "\\begin{align}\n",
    "    w^* = \\arg \\max_{w \\in \\mathbb{R}^M} ~~ p( w | t_{1:N} )\n",
    "\\end{align}\n",
    "\n",
    "Rewriting using the log of the objective for tractability and simplifying via Bayes rule, we get the objective function to maximize is:\n",
    "\n",
    "\\begin{align}\n",
    "\\mathcal{M}(w) &= \\log p( w | t_{1:N})\n",
    "    \\\\\n",
    "    &= \\log p( w ) + \\log p( t_{1:N} | w ) - \\underbrace{\\log p(t_{1:N})}_{\\text{const wrt}~ w}\n",
    "\\end{align}\n",
    "\n",
    "Thus, we can simply ignore the constant term, and maximize the following alternative objective:\n",
    "\\begin{align}\n",
    "  \\mathcal{M}'(w)  &= \\log \\text{MVNormPDF}( w | 0, \\alpha^{-1} I_M ) + \\sum_{n=1}^N \\log \\text{BernPMF}( t_n | \\sigma(w^T \\phi(x_n) ) \n",
    "\\end{align}\n",
    "\n",
    "Finally, we can *standardize* our problem by transforming so we *minimize* rather than *maximize*, just by multiplying by -1. Now the *loss* function we wish to minimize is:\n",
    "\n",
    "\\begin{align}\n",
    "  \\mathcal{L}(w)  &= - \\log \\text{MVNormPDF}( w | 0, \\alpha^{-1} I_M ) - \\sum_{n=1}^N \\log \\text{BernPMF}( t_n | \\sigma(w^T \\phi(x_n) ) \n",
    "\\end{align}\n",
    "\n",
    "Thus, we can find our optimal weights $w^*$ via:\n",
    "\n",
    "\\begin{align}\n",
    "    w^* = \\arg \\min_{w \\in \\mathbb{R}^M} ~~ \\mathcal{L}(w)\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we compute each of these terms?\n",
    "\n",
    "* Use `scipy.stats.multivariate_normal.logpdf` to evaluate the log prior PDF $\\log \\text{MVNormPDF}(\\cdot)$\n",
    "* For the likelihood pdf, use this formula:\n",
    "\n",
    "$$\n",
    "\\sum_{n=1}^N \\log \\text{BernPMF}(t_n | p_n ) = \\sum_{n=1}^N t_n \\log p_n + (1-t_n) \\log (1 - p_n)\n",
    "$$\n",
    "\n",
    "This is translated into the code below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_sum_of_log_bern_pmf(t_N, p_N):\n",
    "    ''' Calculate the log of the bernoulli pmf for N observations\n",
    "    \n",
    "    Args\n",
    "    ----\n",
    "    t_N : 1D array, shape (N,)\n",
    "        Binary value (0 or 1) for each example n\n",
    "    p_N : 1D array, shape (N,)\n",
    "        Probability parameter of the Bernoulli for each example n\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    summed_logpmf : scalar float\n",
    "        Summed log PMF over all N examples given\n",
    "    '''\n",
    "    # Make sure provided probabilities are not hard 0 or hard 1\n",
    "    # so that the log values will not be numerically bad\n",
    "    safe_p_N = np.minimum(np.maximum(p_N, 1e-100), 1 - 1e-13)\n",
    "    return np.sum(np.log(safe_p_N)[t_N==1]) + np.sum(np.log(1-safe_p_N)[t_N==0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2a: Compute the objective of our minimization problem\n",
    "\n",
    "Translate the formula for $\\mathcal{L}(w)$ above into concrete NumPy expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_loss(wguess_M, phi_NM, t_N, alpha=0.1):\n",
    "    ''' Compute the MAP loss objective function.\n",
    "    \n",
    "    The loss is equal to the negative log prior plus negative log likelihood\n",
    "    \n",
    "    Args\n",
    "    ----\n",
    "    w_M : 1D array, shape (M,)\n",
    "        Weight parameter at which we want to evaluate the loss\n",
    "    phi_NM : 2D array, shape (N,M)\n",
    "        Observed input features\n",
    "        Each row is a feature vector for one example\n",
    "    t_N : 1D array, shape (N,)\n",
    "        Observed outputs\n",
    "        Each row is a output scalar value for one example\n",
    "    alpha : positive scalar\n",
    "        Prior precision\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    loss : scalar float\n",
    "        The value of the loss function at provided w value\n",
    "    '''\n",
    "    log_prior_pdf = 0.0 # TODO compute log prior pdf value\n",
    "    log_lik_pdf = 0.0 # TODO compute log likelihood pdf value\n",
    "    \n",
    "    return -1 * log_prior_pdf + -1 * log_lik_pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2b: Evaluate the MAP objective (aka MAP loss function) at possible w values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. , -5. ],\n",
       "       [ 1. , -0.8],\n",
       "       [ 1. , -0.7],\n",
       "       [ 1. , -0.6],\n",
       "       [ 1. , -0.4],\n",
       "       [ 1. ,  0.5],\n",
       "       [ 1. ,  0.8],\n",
       "       [ 1. ,  0.9],\n",
       "       [ 1. ,  4.3],\n",
       "       [ 1. ,  4.1]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phi_NM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 0., 0., 1., 0., 0., 1.])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.6931471805599453"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try with all zero weights\n",
    "w1_M = np.zeros(M)\n",
    "calc_loss(w1_M, phi_NM, t_N, true_alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try with all weights set to 10\n",
    "w2_M = 10 * np.ones(M)\n",
    "calc_loss(w2_M, phi_NM, t_N, true_alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try with all weights set to TRUE values\n",
    "\n",
    "# TODO write code using calc_loss(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion 2c: Which value of the weight vector out of the 3 tried had the \"best\" loss  value? Does that agree with what you expect?\n",
    "\n",
    "Use what you know about how this toy dataset was generated (hint: we know which weights were used to make the true observations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO discuss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo: Visualizing the MAP objective as a contour plot\n",
    "\n",
    "Step through the code below to see how we create a 2d contour plot visualization of our MAP optimization problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a 2-dim grid of possible w values\n",
    "\n",
    "G = 51 # G possible values for intercept\n",
    "w0_grid_G = np.linspace(-2, 2, G)\n",
    "\n",
    "H = 51 # H possible values for slope\n",
    "w1_grid_H = np.linspace(-2, 2, H)\n",
    "\n",
    "w0_GH, w1_GH = np.meshgrid(w0_grid_G, w1_grid_H,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute loss at each possible value in our grid\n",
    "loss_GH = np.zeros((G, H))\n",
    "for gg in range(G):\n",
    "    for hh in range(H):\n",
    "        cur_w_M = np.hstack([w0_GH[gg,hh], w1_GH[gg, hh]])        \n",
    "        loss_GH[gg, hh] = calc_loss(cur_w_M, phi_NM, t_N, true_alpha)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mhughes/miniconda3/envs/semimarkov_forecaster/lib/python3.7/site-packages/ipykernel_launcher.py:7: UserWarning: No contour levels were found within the data range.\n",
      "  import sys\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAAHNCAYAAABl+LNKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeVyVdf7//+dhRxAVNUypTKEsTaRQcgvX0SLTQE3NLc2lQlPTxq1lHM0lMyWXxCXFwTIlHcZKxdKmzD3UjxtFhLkrmCjIzvn9Mb+u7zCgnoNH4OjjfrtxG877evM6rzMzN3nyvq7rfZnMZrNZAAAAkhzKuwEAAFBxEAwAAICBYAAAAAwEAwAAYCAYAAAAA8EAAAAYCAYAAJSDgoIC9evXT+PHjzfGtm3bpmeeeUYBAQHq2rWr9u3bV+Z9EQwAACgH8+fPL/KLPyUlRaNGjdKbb76pffv2qVevXnrttdd07dq1Mu2LYAAAQBnbuXOnNm3apM6dOxtj69evV9OmTdWmTRs5Ozurd+/e8vHx0ZdfflmmvTmV6bsBAFCO8vPzVVBQYPO6165dU2ZmZrFxLy8veXl5FRlLS0vTpEmTNH/+fMXExBj9JCUlyd/fv8jc+vXrKzEx0eb93gjBAABwV8jPz9eef/8g1yqVbV47Ly9Pr776arFwEBERoREjRhivCwsLNW7cOA0cOFCPPvpokbmZmZlyc3MrMubm5qasrCyb93sjBAMAwF2hoKBArlUq6+shk5V5Ic1mdT3uqa6nl0zV+vXr5ejoWOTY/64WLF68WC4uLurfv3+xOu7u7srOzi4ylp2drXvuucdmvVqCYAAAuKtkXkhT5tmLNq9bq1Ytubq63nDOP//5T124cEFBQUGSZASBI0eOqF27dsVOGyQlJalFixY27/VGCAYAAJSRTZs2FXk9adIkFRQUaMaMGUpKStKKFSu0detWhYSEaN26dTp79qw6duxYpj1yVwIAABWAn5+fIiMjNW/ePDVr1kxr167V4sWLVbVq1TLtgxUDAADKybRp04q8DgkJUUhISDl18x+sGAAAAAPBAAAAGAgGAADAQDAAAAAGLj4EANxVvJwc5ORsu7+LKzndWX9j31mfBgAA3BKCAQAAMBAMAACAgWAAAAAMBAMAAGAgGAAAAAPBAAAAGNjHAABwV/F0cpCTk6PN6rmxjwEAALhTEQwAAICBYAAAAAwEAwAAYCAYAAAAA8EAAAAYCAYAAMBAMAAAAAY2OAIA3FUqOznI1dl2fxe7sMERAAC4UxEMAACAgWAAAAAMBAMAAGAgGAAAAAPBAAAAGAgGAADAUKGDwaFDh9S3b18FBQWpdevWmjp1qrKyskqcu27dOrVv315NmjRRnz59lJSUVMbdAgBg/ypsMLh69aqGDBmiDh06aNeuXVq7dq0SEhI0e/bsYnN3796tGTNmaM6cOdqzZ4+aNm2qESNGyGw2l0PnAICKrKqzo7xdbPdV1dmxvD+STVXYYHD69GkFBQVp4MCBcnJyUq1atdS1a1ft3bu32Ny1a9cqNDRUAQEBcnFx0ciRI5WWlqadO3eWQ+cAANivCrslcoMGDbRgwQLjtdlsVnx8vBo1alRsblJSkrp37268dnR0VN26dZWYmKgWLVrc9L0KCwuVmZkpZ2dnmUwm23wAAECpmM1m5eXlycPDQw4OFfbv1ztWhQ0G/y0/P1/vvvuuTp48WeKphMzMTLm5uRUZc3Nzu+71CCX9/M8//2yTXgEAtvHQQw+pcuXK5d3GXafCB4O0tDSNGTNGly5dUkxMjHx8fIrNcXd3V05OTpGx7OxseXp6WvQezs7OkqTh78fp4uXMW28aAFBqNat66ONxzxn/NqNsVehgkJiYqKFDhyogIEALFy6Uh4dHifP8/f2VnJxsvC4oKFBKSor8/f0tep8/Tx9cvJypc2kZt944AOCWcWq3fFTYkzepqakaNGiQOnfurMjIyOuGAkkKCwtTXFyc9u/fr9zcXEVGRsrLy0tBQUFl2DEAAPavwgaDzz77TKmpqVqzZo0CAwONr9DQUJ05c0aBgYHat2+fJKlly5YaP368Jk6cqODgYO3fv1+LFy9mGQoAACtV2FMJERERioiIuO7xhISEIq/Dw8MVHh5+u9sCANg5dzdHFbrb7tefgxv7GAAAgDsUwQAAABgIBgAAwEAwAAAABoIBAAAwEAwAAICBYAAAAAwEAwAAYKiwGxwBAHA7uHi6yJzjarN6Jk8Xm9WqCFgxAAAABoIBAAAwEAwAAICBYAAAAAwEAwAAYCAYAAAAA8EAAAAYCAYAAMDABkcAgLuKa2VXmfNsuMFRZdvVqghYMQAAAAaCAQAAMBAMAACAgWAAAAAMBAMAAGAgGAAAAAPBAAAAGNjHAABwV3HxdJbyXWxX0NPZdrUqAFYMAACAgWAAAAAMBAMAAGAgGAAAAAPBAAAAGAgGAADAQDAAAAAGggEAADCwwREA4K7iUtlFJrOrzeqZK7soz2bVyp9drBikpKSoWbNmOnXq1HXndOvWTY0bN1ZgYKDx9euvv5ZhlwAA2L8Kv2Kwbds2TZo0Senp6dedk5ubq6SkJMXFxalevXpl2B0AAHeWCr1isGjRIs2ePVujR4++4bxjx47J1dVVDz74YBl1BgDAnalCB4OwsDBt3LhRzZs3v+G8w4cPq1KlSurfv7+Cg4MVFhamb7/9toy6BADAclu3btVzzz2nwMBAtWnTRpGRkSosLJQkrVixQh07dtTjjz+uLl266Ouvvy7z/ir0qQQfHx+L5pnNZj322GMaO3asfH19FR8fr9dff12rVq1SkyZNbnOXAABY5sSJExo1apTmz5+vNm3a6MSJE3rxxRdVp04dOTk5aenSpVq6dKkefvhhbd++Xa+//rrq1Kmjxo0bl1mPFXrFwFJ9+/bVwoULVa9ePbm4uCg0NFQtWrTQ5s2by7s1AAAMDzzwgHbt2qU2bdrIbDbr0qVLysvLk7e3t9LS0vTKK6+oQYMGMplMatu2rerXr6+ffvqpTHus0CsGloqJiZGvr69CQkKMsZycHLm62u52FAAAbuTcuXNydHQsMubl5SUvL68iY56ensrPz1eTJk2Ul5enTp06qXXr1mrbtm2RecnJyfrll1/UqFGj2977f7sjVgwuXLigKVOmKCUlRfn5+YqNjVVCQoKef/758m4NAHCX6NOnj9q3b1/ka+XKlSXOdXR01E8//aTNmzfrt99+05QpU4ocP3nypIYNG6auXbsqKCioLNo32OWKwZkzZxQaGqolS5YoKChIERERys/PV//+/XXlyhXVr19fUVFReuCBB8q7VQBABePs4SoHs5vN6hV6uCpP0urVq0tcMSiJyWSSi4uL6tatq1dffVWTJ082wsH333+vsWPH6rnnntOECRNs1qel7CIY+Pr6KjEx0Xhdu3ZtJSQkGK+dnZ01btw4jRs3rjzaAwBAtWrVuukp7G+//VYLFixQbGysMZaTk6MqVapIkj755BPNmzdPb731lsLDw29rv9dzR5xKAADAHgQEBOjUqVNasmSJCgoKlJSUpEWLFqlXr1764osvNG/ePH3yySflFgokO1kxAADgTlC9enUtWbJE06dP18cff6zq1aurd+/eGjBggDp27Kjc3FwNGjSoyM8MGzZMw4cPL7MeCQYAAJShxo0b69NPPy02/s0335RDN8VxKgEAABhYMQAAwM7l5eXp4MGDOn36tDIyMlS1alXde++9CggIKHanxM0QDAAAsFPHjx/XsmXLtHXrVmVlZRU7XrlyZbVp00aDBw9WgwYNLKpJMAAAwM5cvnxZU6dO1ZYtW9SyZUu98847evTRR1W7dm25u7srPT1dFy5cUEJCgnbs2KEePXqoQ4cOevvtt1WtWrUb1iYYAADuKs6VK8nRId9m9Qo8KtmslqW6d++ubt26aceOHapcuXKx497e3vL29laDBg3Uu3dvXbp0SStXrlR4ePhNnz5MMAAAwM589tlnqlGjhsXzvb29NXr0aPXt2/emc7krAQAAO/PfoeCNN97Qnj17LPq5mjVr3nQOwQAAADu2fft2FRYW2qwewQAAADvWpk0brV69Wunp6TapxzUGAADYsezsbH333XeKj483Ljr8byaTSXFxcRbXIxgAAGDHvLy81KVLF5vVIxgAAGDHpk+fbtN6BAMAwF3F2cNNTjbcxyDf3c1mtUrdQ36+vvzyS+3atUsXL17U5MmTtX//fjVs2NDiHQ//xMWHAADYsT/++EM9e/bUxIkTdeTIEe3YsUOZmZmKj49X7969dfDgQavqEQwAALBj06dPV0ZGhrZs2aIvvvhCZrNZkhQZGanGjRtrzpw5VtUjGAAAYMe2bdumUaNGqU6dOjKZTMa4i4uLBg0apCNHjlhVj2AAAIAdKygokKura4nH8vPzjRUESxEMAACwY08++aQWLFhQZIMjk8mkvLw8RUdHq2nTplbV464EAADs2Pjx49W7d2917NhRTZo0kclk0rx585ScnKwrV65o9erVVtVjxQAAADt2//33Ky4uTi+88ILS09N1//33KzU1VW3bttWGDRtUv359q+qxYgAAgB07c+aMatasqTfeeKPYsZycHB04cEBNmjSxuB7BAABwV3Gt4iEnG+5J5OjqYbtipdC+fXutWbNGjRs3Lnbs4MGDGjJkiFV7GRAMAACwM2+//bYuXLggSTKbzZo5c6YqV65cbF5ycrKqVatmVW2uMQAAwM6EhIQoMzNTmZmZkqSsrCzj9Z9f2dnZatCggdUbHLFiAACAnWnfvr3at28vSerXr5/effddqy8yvB5WDAAAsGO+vr5ycXEp8VhycrJeeeUVq+qxYgAAgJ05evSosaPhhg0bFBwcrCtXrhSbt23bNv34449W1SYYAABgZ5YvX66NGzcaz0aYMGFCsTl/Bodu3bpZVZtgAACAnXnnnXfUs2dPmc1mDRgwQG+//bb8/PyKzHFwcJCXl5f8/f2tqk0wAADAzlSuXFnNmjWTJEVHR+vRRx+Vp6enTWoTDAAAdxVnD3c5l3ytXqmYnN1tV6wUmjVrpqtXr2rhwoXauXOnUlNTFRkZqW3btqlBgwZ66qmnrKpnF3clpKSkqFmzZjp16tR156xbt07t27dXkyZN1KdPHyUlJZVhhwAAlI9Tp06pS5cuWrFihTw9PZWSkqLc3FwlJibqlVde0XfffWdVvQofDLZt26Y+ffoUeZzk/9q9e7dmzJihOXPmaM+ePWratKlGjBhh9TOoAQCwN9OmTVPNmjW1bds2ffTRR8bvvg8++EAdOnTQwoULrapXoYPBokWLNHv2bI0ePfqG89auXavQ0FAFBATIxcVFI0eOVFpamnbu3FlGnQIAUD527dqlYcOGycPDw7hL4U+9evXSzz//bFW9Ch0MwsLCtHHjRjVv3vyG85KSkopcdeno6Ki6desqMTHxdrcIAEC5cnFxUU5OTonHLl++fN3Nj66nQgcDHx+fYumnJJmZmXJzK/qoLDc3N2VlZd2u1gAAqBBCQkI0d+5cpaSkGGMmk0mXL19WVFSUWrVqZVW9Ch0MLOXu7l4sLWVnZ9vs1g0AACqq8ePHy8XFRc8++6yee+45SdKkSZPUsWNHXb16VW+++aZV9e6I2xX9/f2VnJxsvC4oKFBKSorVmzoAAGBvvL299cUXX2j9+vXas2ePfHx85OnpqW7duik8PNzqP5LviGAQFhamUaNG6ZlnntFjjz2mBQsWyMvLS0FBQeXdGgCggnH08JBTvqPN6hU6ud180m3m6uqqXr16qVevXrdcyy6DwZkzZxQaGqolS5YoKChILVu21Pjx4zVx4kRduHBBDRs21OLFi+Xs7FzerQIAcNslJydr+fLlOnTokC5cuKCqVasqODhYL7/8su677z6ratlFMPD19S1yh0Ht2rWVkJBQZE54eLjCw8PLujUAAMrVjz/+qKFDh6pGjRpq1aqVvL29lZqaqm+++UZffvmlVq1apUceecTienYRDAAAQMlmz56tVq1a6aOPPiqyUp6dna1hw4Zp6tSpiomJsbjeHXFXAgAAd6ukpCS9+OKLxU6fu7m5afDgwTp8+LBV9QgGAADYMT8/Px09erTEY6dPn9b9999vVT1OJQAAYGeOHDlifP/CCy9o+vTpys3NVceOHVWjRg2lp6fr+++/V1RUlP72t79ZVZtgAACAnQkPDy+yM7DZbNaCBQuKPDDpz4cpjRw5UseOHbO4NsEAAAA7Ex0dfdtqEwwAAHcVB3cPORTabp8bBwfrHlJkC82aNbtttbn4EAAAGAgGAADAQDAAAAAGggEAADBw8SEAAHbuypUrysrKko+Pj/Lz87VixQqdPXtWHTt21JNPPmlVLVYMAACwY3v37lWbNm2MWxj//ve/64MPPtD27dv10ksv6auvvrKqHsEAAAA7NnfuXAUGBuqll15Senq6YmNjNWjQIH3zzTcaOHCgFi9ebFU9ggEAAHbsyJEjevnll1WjRg199913KigoUJcuXSRJbdu21W+//WZVPa4xAADcVUzulWUy59munsl2myWVhpubm3JzcyVJ3333nWrWrKkGDRpIks6ePasqVapYVY9gAACAHQsODta8efOUmJiozZs368UXX5Qkbd26VXPnzlWrVq2sqsepBAAA7Nhbb70lb29vLVq0SM2aNdNrr70mSZo2bZoeeOABjRs3zqp6rBgAAGDHatSooaVLlxYb/+KLL1StWjWr6xEMAACwM0eOHFH9+vXl5uamI0eOXHfemTNnJEkNGza0uDbBAAAAOxMeHq7PP/9cjRs3Vnh4uEwmU4nzzGazTCaTjh07ZnFtggEAAHYmOjpa9evXN763JYIBAAB2plmzZiV+bwsEAwDAXcWhkoccVGC7enK0Wa2KgNsVAQCAgWAAAAAMBAMAAOzYgQMHlJdnuy2eCQYAANixiIgIqx+tfCMEAwAA7JiLi4tcXFxsVo+7EgAAsGMDBw7UW2+9pf3796t+/fqqXr16sTl/+ctfLK5HMAAAwI699957kqR//OMfJR5n50MAAO4i33zzjU3rEQwAAHcVk7unHBzMtqtXaJJybFbOanXq1DG+P336tC5evKiHHnpIZrNZHh4eVtcjGAAAYOc2b96sDz74QL///rscHBy0du1azZ8/Xx4eHpo+fbqcnZ0trsVdCQAA2LGvvvpKo0aNUtOmTTV37lwVFhZKkjp27KitW7dqwYIFVtWr0CsGqampmjx5svbu3Ss3Nzf17t1bERERJc7t1q2bkpOT5ej4//asXrdunfH0KQAA7kQLFy5U//79NWHCBBUU/L9nQISFhenKlStatWqVRo0aZXG9Cr1iMHr0aFWrVk07duzQihUrFBsbq40bNxabl5ubq6SkJG3YsEEJCQnGF6EAAFDRHDp0SH379lVQUJBat26tqVOnKisrq8icn3/+WQEBAdq9e/dN6504cUIhISElHnvkkUd08eJFq/qrsMHgxIkT2rNnj9544w25ubnJ399fAwYMUExMTLG5x44dk6urqx588MFy6BQAAMtcvXpVQ4YMUYcOHbRr1y6tXbtWCQkJmj17tjEnKytLY8aMUXZ2tkU1a9eurf3795d47NChQ7r33nut6rHCnkr45ZdfVLVqVdWoUcMYq1evnhITE4vNPXz4sCpVqqT+/fvr559/Vp06dRQREaF27dqVZcsAANzQ6dOnFRQUpIEDB0qSatWqpa5du2rdunXGnL/97W9q3769kpOTLar54osvatasWTKbzQoJCZHJZNL58+d19OhRffzxx3r11Vet6rHCBoPMzEy5u7sXGXN3dy8xQZnNZj322GMaO3asfH19FR8fr9dff12rVq1SkyZNyqplAMBd7Ny5c0Wuc5MkLy8veXl5Ga8bNGhQ5GJAs9ms+Ph4NWrUSJK0YcMGpaSkaOrUqVqyZIlF79u/f39duXJFS5Ys0aJFi2Q2m/Xqq6/KyclJ/fr10+DBg636HBU2GFSqVKlYCMjKypKnp2exuX379lXfvn2N16GhoYqLi9PmzZsJBgCAMtGnTx+lpqYWGYuIiNCIESNKnJ+fn693331XJ0+e1OzZs/Xrr7/qww8/1OrVq+XkZN2v54iICA0YMEAJCQlKT09X5cqV1bhxY3l7e1v9OSpsMPD399cff/yhS5cuGR/s119/lZ+fX7G5MTEx8vX1LXLxRU5OjlxdXcusXwCAfTC5esjkePN5FtcrkJQjrV69usQVg5KkpaVpzJgxunTpkmJiYlS1alUNGTJEEyZMKLJhkSV69uypp556SiEhIXrqqadK+zEMFfbiw7p16yowMFCzZs3StWvXlJSUpOjoaD3//PPF5l64cEFTpkxRSkqK8vPzFRsbq4SEhBLnAgBwO9SqVUu+vr5FvkoKBomJiQoLC1OVKlX02WefqU6dOvq///s/paSkaPLkyQoKClJQUJAKCgo0fPhwRUVF3fB9GzVqpH/961/q0aOHWrRoob/+9a/68ssvlZ6eXqrPUWFXDCQpMjJSU6ZMUdu2beXs7Kw+ffqoR48eOnPmjEJDQ7VkyRIFBQUpIiJC+fn5xnmW+vXrKyoqSg888EB5fwQAAAypqakaNGiQnn32WU2YMMEYDwoK0qFDh4rMffTRR/Xxxx8rODj4hjXffvttSf+5sPGHH37Qjh07NGXKFGVkZKhx48Zq06aNhg0bZnGPJrPZbLsNo+1UTk6ODh8+rPBJn+pcWkZ5twMAd7Va1T0VO623GjVqZNNTwn/+W/+ol1muNjyVkFMgHb1isqjf+fPn66OPPpK7u7tMJpMxXrt2bX355ZdF5j766KP65JNPbhoM/tfly5e1e/duffLJJzpw4ABPVwQAoKKKiIi47g6+/+vo0aMWzTt9+rT27dun/fv3a//+/UpOTpabm5uaNGmiESNGqFmzZlb1SDAAAMCOtW/fXiaTSY888ojCw8P1+OOPq1GjRlbf2fAnggEAAHasX79+2rt3r44fP64//vhDR48eVdOmTRUUFFSqRwMQDAAAsGOTJk2SJF25ckV79uzRnj17tGbNGv3973+Xl5eXgoKCFBkZaXE9ggEAAHcALy8vdejQQf7+/qpfv762bdum7du369tvv7WqDsEAAHBXMbu4q9DJdtv4mPMLJVn2wKPb4cyZM9q1a5d2796t3bt36/z587rnnnvUqlUrzZ07Vy1btrSqHsEAAAA71q5dOzk5Oenxxx/Xiy++qKeeekoPP/xwqeuVOhjEx8fr7NmzatWqlerVq2eM/+Mf/yjy3AIAAHD7zJ8/X82bN5eHh4dN6pVqLWX27NmKjo7W77//rkGDBmnFihXGsdjYWJs0BgAAbq5Dhw66du2aZs6cqe7du6tz587q1auX3n//fZ0/f97qeqVaMfjuu++0fv16OTk56bXXXtPrr7+u8+fP669//avYSBEAgLJz4sQJ9enTR9nZ2WrevLmqV6+u1NRUffrpp/riiy/02WefWfWIgFIFg8LCQmPjhGrVqmnp0qUaO3asJk6cqMLCwtKUBAAApTBz5kxVr15dK1euVLVq1YzxS5cuafDgwZo9e7Y++ugji+uV6lRCzZo1deTIEeO1i4uL5s6dK5PJpF9++aU0JQEAQCns2rVLERERRUKBJHl7e2v48OHavXu3VfUsXjG4dOmSvL29JUkzZswo9sxpBwcHTZs2TeHh4VY1AAAASs/d3V0ODiX/ne/g4KD8/Hyr6lkcDJ566il17txZL774ogIDA6877/HHH7eqAQAAypLZ2U1mJ9s9XtFsKlB57mMQFBSkhQsXqmnTpqpSpYoxfvnyZS1cuPD2PUQpKipKn3/+ufr166f69eurb9++6tKli9zc3Kx6QwAAYDtvvvmmunfvrnbt2ik4OFg1atRQamqqdu/eLScnJ82ePduqehZfY9CiRQvNnTtX3333nUJDQxUVFaXWrVtr2rRp+u2336z+IAAA4NbVqVNHGzZsUI8ePXThwgXt2rVLFy9eVI8ePfTPf/7T6gcpWX1XQvXq1TV06FANHTpUO3bs0Jo1a9SlSxc1a9ZMy5cvt7YcAAC4RT4+Pho/frxNalkcDN5++22lp6fr6tWrunLlSpH/zM/P186dO23SEAAAuLFPPvnE4rkmk0kDBw60eL7FweDzzz+Xh4eHevbsqcaNG8vDw0Oenp5F/hMAANx+M2fOtHjubQsGsbGx+sc//qFPP/1Ux48fV9++fRUYGCiTyWTxmwEAgFt3/Pjx21bb4osPGzZsqOnTp2vbtm1q3ry5pk2bpvbt2ysqKkp//PHHbWsQAAAU9fvvv5fq506cOHHTOVbvfFitWjUNHTpUW7du1cSJE/Xjjz+qbdu2mjBhQqmaBAAA1hkyZIimTp1q8UOSfv/9d7377rsaOnToTedafCphwoQJyszMVGZmpq5du2Z8ZWZmKjc3Vxs2bND06dMtLQcAQLkwO7vL7FyqRwWVXM+UL+myzepZYsOGDfrggw/Uvn17NWnSRB07dlTDhg1Vq1Ytubu76+rVqzp37pwOHDigHTt2KCEhQeHh4RY9Adni/2bS09Pl5eUlHx8fVa5cWV5eXvL09JSXl5cqV66sypUr39KHBAAAlnF3d9fkyZM1YMAArVy5UlFRUUpLSyty3Z/ZbNY999yjDh066L333tN9991nUW2Lg8HChQut7xwAANw29913nyZPnqzJkycrKSlJp0+f1tWrV1W1alXde++9Vm9uJJXyscsAAKBi8fPzk5+f3y3XKdVjlwEAwJ2JYAAAAAwEAwAAYCAYAAAAAxcfAgBwBzhz5ox2796tixcv6vnnn9fZs2f18MMPy9XV1ao6BAMAwF3F7OQms7Oz7eqZ82xWqzQKCws1bdo0ffbZZyooKJDJZFLLli01d+5cnT59WtHR0fLx8bG4HqcSAACwY/Pnz9cXX3yh6dOn68cff5TZbJYkvfnmmyooKNAHH3xgVT2CAQAAdiw2NlZjxozRc889pypVqhjjDRo00Ouvv64dO3ZYVY9gAACAHbt8+bIefPDBEo95e3srIyPDqnoVOhikpqZq+PDheuKJJ9SyZUvNnz//unPXrVtnPEyiT58+SkpKKsNOAZMoNVsAACAASURBVAAoHw8//LDWr19f4rH4+Hg99NBDVtWr0Bcfjh49Wr6+vtqxY4dOnjypoUOHqm7dunr22WeLzNu9e7dmzJihZcuW6ZFHHtGCBQs0YsQIffXVV0UeKAEAwJ3m9ddf19ChQ3X+/HmFhITIZDLpm2++0SeffKKvvvrK6mcdVdgVgxMnTmjPnj1644035ObmJn9/fw0YMEAxMTHF5q5du1ahoaEKCAiQi4uLRo4cqbS0NO3cubMcOgcAoOy0bNlSS5cuVV5enj788EOZzWYtXLhQSUlJmj9/vtq0aWNVPatXDNLT07VmzRolJydr3Lhx2rt3r/z9/Uv1BKcb+eWXX1S1alXVqFHDGKtXr54SExOLzU1KSlL37t2N146Ojqpbt64SExPVokULm/YFAEBF07x5czVv3lzZ2dlKT0+Xp6enPDw8SlXLqhWD3377TU8//bRiY2O1ceNGXbt2TVu2bFH37t31008/laqB68nMzJS7u3uRMXd3d2VnZ5c4183NrciYm5ubsrKybNoTAMD+5ZsclWdystlXvsmxXD9Pu3btNGfOHCUmJsrNzU0+Pj6lDgWSlcFg+vTp6tSpkzZv3izn/39ziNmzZ6tz585W3yd5M5UqVSoWArKysuTp6Vlsrru7u3JycoqMZWdnlzgXAIA7SadOnbRx40Z169ZNoaGh+vjjj/X777+Xup5VweDgwYPq27dv0QIODho6dKiOHj1a6iZK4u/vrz/++EOXLl0yxn799dcSnzXt7++v5ORk43VBQYFSUlLk7+9v054AAKho/vrXv+rbb7/Vp59+qlatWmnNmjXq1KmTevbsqejoaF28eNGqelZffPi/f5lLUlpamlxcXKwtdUN169ZVYGCgZs2apWvXrikpKUnR0dF6/vnni80NCwtTXFyc9u/fr9zcXEVGRsrLy0tBQUE27QkAgIqqSZMmmjBhgrZt26ZPP/1UDz/8sGbOnKm2bdtaVceqYNCuXTvNnTtXmZmZxtjJkyf13nvvWX3VoyUiIyOVkZGhtm3bauDAgerRo4d69OihM2fOKDAwUPv27ZP0nysyx48fr4kTJyo4OFj79+/X4sWLjdMdAADcDS5evKiYmBjNmTNH69evV7Vq1dS7d2+rapjMf26qbIErV65oyJAhOnLkiPLz81W1alWlp6crICBACxculLe3t9UfoiLIycnR4cOHFT7pU51Ls26HKACAbdWq7qnYab3VqFEjq58MeCN//ltfr76fnG24yp2Xm6vkX5Ns3q+lLly4oC1btmjTpk366aef5OHhoQ4dOqhLly568skn5eBg3ckBq25X9PLy0po1a7Rz504dPXpUzs7O8vf3V/Pmza16UwAAYBshISFycXFRSEiI5s2bZ7wuLav3MTh16pR27dqlxMREOTg46MqVK/Lz81PNmjVL3QQAACid9957Tx07drTZnXhWBYN9+/Zp8ODBqlmzpho1aqTCwkJ98cUXWrlypVatWqUGDRrYpCkAAHB9R44cUf369eXm5qaHHnpIJ06cuOH8hg0bWlzbqmAwffp0de3aVe+++65xzqKgoEBvvfWWpk2bplWrVllTDgCAMpdTaFZhgcWX191UXqHtalkqPDxcn3/+uRo3bqzw8PDrPhfIbDbLZDLp2LFjFte2KhgkJSVp9uzZRS5kcHR01Msvv6ywsDBrSgEAgFKKjo42HkUQHR1t09pWBQM/Pz/t37+/2HOff/75Z9WtW9eWfQEAgOto1qyZ8f2ZM2cUEhKiatWqFZt38eJFxcXFFZl/M1YFg549e2rGjBlKTk5W06ZN5eTkpCNHjmj58uXq2bOn/vWvfxlzu3TpYk1pAABQChMmTNCaNWtKDAaHDh3S3LlzNXjwYIvrWRUM3nnnHUnS8uXLtXz58iLHli5danxvMpkIBgAA3CYDBgzQ//3f/0n6z3UEAwYMKPE6g+zsbKsuPJSsDAbHjx+3qjgAALC9t956S5s2bZLZbNaCBQsUGhqqWrVqFZnj4OAgLy8vPfPMM1bVtnofAwAAUL78/PwUEREh6T+r9D169JCPj49xvLCw0OodD/9k9U+dOXNG77//voYOHapXX31Vc+bM0alTp0r15gAA4NZEREQoLi5Or776qjG2b98+hYSEKCYmxup6VgWDY8eOqUuXLvrqq6/k7u4uR0dHxcXFqWvXrkpMTLT6zQEAwK1ZvHix5s2bZ9y+KEkPPPCAnn32Wc2aNUurV6+2qp5VpxJmzpypp556SrNmzTKeXJiXl6fx48dr1qxZWrZsmVVvDgBAWcvJN6vAZLtNifLzy36Do//2+eefa9y4cRowYIAx5uPjo3HjxqlGjRpauXKl+vTpY3E9q1YMDhw4oFdffbXI44ydnZ01bNgwJSQkWFMKAADYQFpamvz8/Eo89vDDD+vs2bNW1bMqGHh5eSkzM7PYeEZGhpycuI4RAICyVr9+fX311VclHtu0aVOxTQlvxqrf5m3atNGUKVM0d+5c3X///ZKklJQUTZs2TSEhIVa9MQAAuHXDhg3TyJEjdfbsWbVp00bVq1fXpUuXtH37dv3444+aO3euVfWsCgZjxozRSy+9pE6dOhk7LP3xxx8KCAjQhAkTrHpjAABw6/7yl79o3rx5+vjjj/Xee+8Z4w899JDmzp2rTp06WVXPqmBQtWpVxcbG6vvvv9cvv/wiV1dX+fn5qXnz5la9KQAAsJ1OnTqpU6dOysnJ0eXLl+Xp6SkPD49S1bppMPjv5x/8tz83UkhNTTXmsA0yAADlY9++fdq1a5cuXryoYcOGad++fXrkkUd0zz33WFXnpsFg3LhxFhXi+QgAAJS97OxsjRo1Stu3b5enp6cyMzPVs2dPrVixQomJiVq1alWRPQ5u5qbBgOcjAADuJLmFZhUUFNqsXkFh+e5jMHv2bB06dEgxMTEKCAhQo0aNJEnvv/++hgwZog8++EALFy60uJ7VWyL/85//1Llz5yRJy5Yt07PPPqtJkybp2rVr1pYCAAC36Msvv9TYsWP1xBNPFHnCYo0aNfTqq69q//79VtWzKhh89NFHevfdd3Xu3Dnt27dPs2fPVtOmTZWQkKD333/fqjcGAAC3LisrS9WrVy/xmKurq3Jzc62qZ1UwWL9+vd5//301adJEX3/9tQIDA/XOO+9o2rRpio+Pt+qNAQDArQsICFB0dLTy8/ONsT9XDmJjY/XYY49ZVc+q2xUvXrxonLv44Ycf1K1bN0lSzZo1lZGRYdUbAwCAWzdu3Dj169dPTz/9tFq2bCmTyaTVq1crOTlZhw8f1sqVK62qZ9WKwX333afDhw/r6NGjOnHihJ566ilJ0rZt23TfffdZ9cYAAODWNWrUSGvXrtVjjz2mrVu3ytHRUVu3blW1atX06aefKjAw0Kp6Vq0YvPzyyxo9erQcHBzUtGlTNWzYUAsXLtSCBQuK7LYEAADKjp+fn+bMmWOTWlYFg7CwMDVs2FAnT55U69atJUlNmjTRihUr1LRpU5s0BAAAbmzLli168skn5eXlpS1bttx0vru7u+677z7VrVv3pnOtfiTiww8/rIcffth43aJFC2tLAACAWzBy5Eh9/vnnaty4sUaOHGnRz5hMJg0ePFhjx4694TyelQwAuKtk55nlYLbdpkSF+WW/wdE333yjmjVrGt/fTE5OjqKjo7V27VqCAQAAd5o6deqU+H16erouXLigqlWrGsHhT08//bSSk5NvWptgAACAnYuNjdXixYt18uRJY+yBBx7QqFGj1LlzZ0lScHCwgoODb1qLYAAAgB1buXKlpk+frqeffloRERHy9vZWamqqNm/erNGjR6uwsFDPPPOMxfWsflYCAAC4dSkpKWrWrJlOnTpljO3bt0+9evVSUFCQOnTooLVr1960TnR0tAYNGqQPP/xQzz33nFq1aqVu3bpp0aJF6t27t+bNm2dVXxU2GKSmpmr48OF64okn1LJlS82fP/+G87t166bGjRsrMDDQ+Pr111/LqFsAACy3bds29enTR+np6cZYYmKiBg8erL59+2rv3r2aMWOG/v73v+vgwYM3rJWamqqWLVuWeKxjx47Ggw8tVWFPJYwePVq+vr7asWOHTp48qaFDh6pu3bp69tlni83Nzc1VUlKS4uLiVK9evXLoFgAAyyxatEgbN27U6NGjNXnyZGP8008/VadOnYzfc0FBQYqNjS12EeH/atWqlf71r3+VGA7+/e9/W73PUIUMBidOnNCePXv04Ycfys3NTf7+/howYIBiYmJKDAbHjh2Tq6urHnzwwXLoFgAA6dy5c3J0dCwy5uXlJS8vryJjYWFhGj58uE6fPl1k/PDhw2rRooVGjhyp3bt365577tFrr70mf3//Yu/1ySefGN/Xq1dPn3zyiU6fPq2OHTuqRo0aSk9P1/fff6+dO3dq1KhRVn2OcgsGOTk5unjxYonHfvnlF1WtWlU1atQwxurVq6fExMQS5x8+fFiVKlVS//799fPPP6tOnTqKiIhQu3btbkvvAAD8rz59+ig1NbXIWEREhEaMGFFkzMfHp8SfT09PV0xMjCIjIzVnzhx9//33GjlypHx8fIo972DmzJnFfn7v3r3au3dvsfEZM2ZowIABFn+OcgsGBw4cUP/+/Us8NmvWLLm7uxcZc3d3V3Z2donzzWazHnvsMY0dO1a+vr6Kj4/X66+/rlWrVqlJkyY27x0AYL+y8wsksw0vsSsokIOk1atXl7hiYClXV1f95S9/MU4JtG3bVq1bt9aWLVuKBYPjx4/fctvXU27BIDg4+LorAPHx8cVCQFZWljw9PUuc37dvX/Xt29d4HRoaqri4OG3evJlgAAAoE7Vq1ZKrq2upf75+/frKyckpMlZQUGDxz1++fFkHDx5URkaGqlWrpscee0yVK1e2uo8KeY2Bv7+//vjjD126dEne3t6SpF9//VV+fn4lzo+JiZGvr69CQkKMsZycnFv6HwgAgLL0wgsvaOjQofr666/VuXNnbd++XT/++KNFz0KYN2+eli1bptzcXGPMyclJgwYN0pgxY6zqo0Lerli3bl0FBgZq1qxZunbtmpKSkhQdHa3nn3++xPkXLlzQlClTlJKSovz8fMXGxiohIeG68wEAqGhatGihyMhIRUVF6YknntDMmTM1a9YsNWzY8IY/t3LlSkVFRWngwIHasGGDfvjhB23YsEEvvfSSli1bplWrVlnVR4VcMZCkyMhITZkyRW3btpWzs7P69OmjHj16SJLOnDmj0NBQLVmyREFBQYqIiFB+fr769++vK1euqH79+oqKitIDDzxQzp8CAICS+fr6Fjul3q5dO6svnF+9erVefvlljR492hirUaOGGjRoIEdHR61evVr9+vWzuF6FDQb33HPPdTc1ql27thISEozXzs7OGjdunMaNG1dW7QEAUCGcO3fuus9AaNasmZYvX25VvQp5KgEAAFjm/vvv1759+0o8tnfv3uveHnk9FXbFAAAA3Fy/fv30t7/9TQUFBercubOqV6+utLQ0bdq0ScuWLbOfDY4AAMCt69mzp06ePKnly5crKirKGHd0dFS/fv308ssvW1WPYAAAuKtk5BXIXGCyWT1TYYEs38bo9njjjTc0ePBgHTx4UOnp6apSpYoaN26satWqWV2LYAAAwB2gatWqRfbzKS0uPgQAAAaCAQAAMBAMAACAgWAAAAAMBAMAAOyY2WzW559/ru3bt0v6z0MHQ0NDFRgYqPHjxysrK8uqegQDAADs2KJFi/Tuu+/qt99+k/SfWxezsrI0ePBg/fDDD/rggw+sqsftigCAu8q13AIVONhuHwPHct7HYP369Ro1apReeukl/fLLLzp+/LimTZum8PBw+fr6avbs2Zo8ebLF9VgxAADAjp0/f16PP/64JGnbtm1ycHBQ27ZtJUn33nuvMjIyrKpHMAAAwI7VqlVLSUlJkqTNmzerUaNG8vb2liT9+OOPqlOnjlX1CAYAANixnj176r333tMzzzyjI0eOqE+fPpKkkSNHKioqynhtKa4xAADAjr388suqXbu29u3bp1deeUVdunSRJHl5eWnGjBnq2rWrVfUIBgAA2LlnnnlGzzzzTJGxqVOnlqoWpxIAALBj7GMAAAAMtt7HgGAAAIAdK2kfg9dee00REREaO3asNm3aZFU9rjEAANxVMnMLlG+7/Y3kZC6wXbFSYB8DAABgYB8DAABgYB8DAABgYB8DAABQhC33MSAYAABg53755Rd99NFH2rt3rzIyMlS1alU98cQTGj58uBo0aGBVLYIBAAB27PDhw+rbt6+8vb313HPPqXr16kpNTdXWrVv1wgsvKCYmRo0aNbK4HsEAAAA79v777ysgIEBLly6Vs7OzMT527FgNGTJEH374oZYtW2ZxPe5KAADAjh08eFAvvfRSkVAgSS4uLho4cKASEhKsqseKAQDgrpKZV6BcG9ZzUUG5/jatUqXKdTcxysjIkJOTdc2xYgAAgB1r3bq15s6dq+Tk5CLjycnJmjdvnlq3bm1VPVYMAACwY2+88YZ69eqlLl26yM/PTzVq1FBqaqqSkpJ077336s0337SqHsEAAAA7Vq1aNa1fv16xsbHat2+frly5ogcffFDdu3dXWFiYPDw8rKpHMAAAwM5VqlRJ/fr1U79+/W65FsEAAAA7Y+2uhpMnT7Z4boUPBikpKerZs6e++OIL+fr6XnfeunXrtGjRIqWlpenRRx/VlClT5OfnV4adAgBQNr799luL55pMpjsnGGzbtk2TJk1Senr6Deft3r1bM2bM0LJly/TII49owYIFGjFihL766iuZTDZ86DYAABWANcHAWhX2dsVFixZp9uzZGj169E3nrl27VqGhoQoICJCLi4tGjhyptLQ07dy5sww6BQDYk4ycfF3JzrPZV0ZOfnl/JJsqtxWDnJwcXbx4scRjNWvWVFhYmIYPH67Tp0/ftFZSUpK6d+9uvHZ0dFTdunWVmJioFi1a2KxnAADudOUWDA4cOKD+/fuXeCw6OlrBwcEW18rMzJSbm1uRMTc3N2VlZd1SjwAA3G3KLRgEBwcrMTHRJrXc3d2Vk5NTZCw7O1uenp42qQ8AwN2iwl5jYA1/f/8iW0EWFBQoJSVF/v7+5dgVAAD2544IBmFhYYqLi9P+/fuVm5uryMhIeXl5KSgoqLxbAwDgtti+fbuGDBmi0NBQDRs2zGZ3KthlMDhz5owCAwO1b98+SVLLli01fvx4TZw4UcHBwdq/f78WL15c7BGUAADcCb766isNHz5cBw8elIeHhw4fPqzXXntNS5cuveXaFXofA0ny9fUtdi1C7dq1iz1fOjw8XOHh4WXZGgAA5WLFihVq3bq1IiMj5e7urvz8fE2ePFlLlizRyy+/fEu17XLFAACAu1lSUpIGDBggd3d3SZKTk5NeeeUVpaen6+TJk7dUu8KvGAAAYEtXs/J0rdB2fxfnORRK1j3A8JZlZWWpcuXKRcbuvfdeSVJGRsYt1WbFAAAAO2M2m4tt+e/o6ChJKiwsvKXaBAMAAGDgVAIAAHZo+fLlqlGjhvHabDZLkpYtWyZvb+8ic++YpysCAIDiateurUOHDpU4fuDAgSJjd9RjlwEAQHF35WOXAQBA6eXm5iouLu66Dyy8HlYMAAC4gxw/flxr167Vxo0blZ6eripVqlj18wQDAADsXEZGhjZu3Ki1a9fq6NGjcnFxUZs2bdSlSxeFhIRYVYtgAAC4q1zJzldGvunmEy2U72S2WS1r7d+/X+vWrdOmTZuUk5OjJk2aSJI+/vhjNW/evFQ1CQYAANiZ5cuXa926dUpOTla9evU0fPhwde3aVZUqVVKzZs3k5FT6X+8EAwAA7MysWbPk7++vFStW6MknnzTGr169esu1uSsBAAA7069fP126dEmDBw9Wr169tHr1aqWnp9ukNsEAAAA7M2nSJP373//W/Pnz5ePjo5kzZ6pVq1YaM2aMTCbTLT0vgVMJAADYIUdHR7Vt21Zt27ZVRkaGvvzyS/3zn/+U2WzW8OHD9dRTT+npp59WSEiI8XhmSxAMAACwc56ennrhhRf0wgsv6OTJk9qwYYP+9a9/afPmzXJ3d1dCQoLFtTiVAADAHeS+++7TiBEjtGXLFv3jH/9Qly5drPp5VgwAAHeVy9dydSXPdvXynaWy/jv78uXLFs3z8/PTmDFjrKpNMAAAwM5Yu3nRsWPHLJ5LMAAAwM6Yzf/ZbbFBgwbq1KmTatWqZbPaBAMAAOzMt99+q82bN2vTpk366KOPFBAQoM6dO6tz587y8fG5pdpcfAgAgJ2pXbu2XnrpJa1Zs0bffPONOnXqpK+//lrt2rVTr169tHLlSp0/f75UtQkGAADYsXvvvVcDBw7UZ599pm+++UadO3fW5s2b1b59eyMkWINgAADAHaJWrVoaOHCg3n//fQ0aNEiHDx/WjBkzrKrBNQYAANwBfvvtN23ZskVbtmzR0aNHVbNmTfXs2VN/+ctfrKpDMAAAwE4dP35cmzdvVnx8vJKSklS7dm117NhREydO1OOPPy6TyWR1TYIBAOCuciUrT5dzbVjQRZJcbVjw5mbNmqX4+HidOnVK999/vzp27Kj33ntPjRs3vuXaBAMAAOzM8uXL5eDgoMcff1yPPPKIsrOzFRcXp7i4uBLnT5482eLaBAMAAOxM7dq1JUlnz57V2bNnbzjXZDIRDAAAuJN9++23t602tysCAFCGjhw5ohdffFFBQUFq1aqVpk2bptxcW170cGsIBgAAlBGz2axXXnlFnTp10p49e7Ru3Tp9//33Wrp0aXm3ZiAYAABQRjIyMnT+/HkVFhYaD0JycHCQm5tbOXf2/3CNAQAANnDu3Dk5OjoWGfPy8pKXl5fxunLlyho0aJBmzpypWbNmqaCgQB06dNDAgQPLuNvrq/ArBikpKWrWrJlOnTp1w3ndunVT48aNFRgYaHz9+uuvZdQlAOBu16dPH7Vv377I1/8+p6CwsFBOTk6aPHmyDhw4oI0bNyopKUmRkZHl1HVxFXrFYNu2bZo0aZLS09NvOC83N1dJSUmKi4tTvXr1yqg7AIA9ysrO17Ucs83quRaaJLlq9erVJa4Y/Lf4+HjFx8dr06ZNkiR/f3+99tprmjZtmkaNGmWznm5FhQ0GixYt0saNGzV69Oib3n957Ngxubq66sEHHyyj7gAAKKpWrVpydb3xDohnz55Vfn5+kTEnJyc5OzvfztasUm6nEnJycnTq1KkSv3JychQWFqaNGzeqefPmN611+PBhVapUSf3791dwcLDCwsJu6z2eAACURqtWrXT+/HlFRUWpoKBAJ0+e1KJFi9S1a9fybs1QbisGBw4cUP/+/Us8Fh0dreDgYItrmc1mPfbYYxo7dqx8fX0VHx+v119/XatWrVKTJk1s1TIAALfEz89Pixcv1ty5cxUVFSUvLy8999xzeu2118q7NUO5BYPg4GAlJibapFbfvn3Vt29f43VoaKji4uK0efNmggEAoEJp0aKFWrRoUd5tXFeFvyvBEjExMfruu++KjOXk5Nz0XA8AACjqjggGFy5c0JQpU5SSkqL8/HzFxsYqISFBzz//fHm3BgCAXamwdyXcyJkzZxQaGqolS5YoKChIERERys/PV//+/XXlyhXVr19fUVFReuCBB8q7VQAA7EqFDwa+vr7FrkWoXbu2EhISjNfOzs4aN26cxo0bV9btAQBwR6nwwQAAAFvKzS5QTnah7eqZ74iz8oY769MAAIBbQjAAAAAGggEAADAQDAAAgIFgAAAADAQDAABgIBgAAAAD+xgAAO4qeTl5ys2y3T4GeaY762/sO+vTAACAW0IwAAAABoIBAAAwEAwAAICBYAAAAAwEAwAAYCAYAAAAA8EAAAAY2OAIAHBXyc0uUE5Wge3qOTjarFZFwIoBAAAwEAwAAICBYAAAAAwEAwAAYCAYAAAAA8EAAAAYCAYAAMBAMAAAAAY2OAIA3FVys/Nsu8GRY6HNalUErBgAAAADwQAAABgIBgAAwEAwAAAABoIBAAAwEAwAAICBYAAAAAwVdh+DQ4cOadasWTp+/Ljc3d3VqVMnvfHGG3J3dy9x/rp167Ro0SKlpaXp0Ucf1ZQpU+Tn51fGXQMAKrrc7GzlZuXbrp5Thf1VWioVcsXg6tWrGjJkiDp06KBdu3Zp7dq1SkhI0OzZs0ucv3v3bs2YMUNz5szRnj171LRpU40YMUJms7mMOwcAwL5VyGBw+vRpBQUFaeDAgXJyclKtWrXUtWtX7d27t8T5a9euVWhoqAICAuTi4qKRI0cqLS1NO3fuLOPOAQCwb+W2/pGTk6OLFy+WeOzBBx/UggULjNdms1nx8fFq1KhRifOTkpLUvXt347Wjo6Pq1q2rxMREtWjR4qa9/LmyULOqhzUfAQBwG/z5bzGrvuWj3ILBgQMH1L9//xKPRUdHKzg4WJKUn5+vd999VydPnrzuqYTMzEy5ubkVGXNzc1NWVpZFveTl5UmSPh73nKXtAwBus7y8vGL/tuP2K7dgEBwcrMTExBvOSUtL05gxY3Tp0iXFxMTIx8enxHnu7u7KyckpMpadnS1PT0+LevHw8NBDDz0kZ2dnmUwmyz4AAOC2MJvNysvLk4cHq7jlocJeSpmYmKihQ4cqICBACxcuvOH/Qfz9/ZWcnGy8LigoUEpKivz9/S16LwcHB1WuXPmWewYA2AYrBeWnQl58mJqaqkGDBqlz586KjIy8aWoMCwtTXFyc9u/fr9zcXEVGRsrLy0tBQUFl1DEAAHeGChkMPvvsM6WmpmrNmjUKDAw0vkJDQyVJZ86cUWBgoPbt2ydJatmypcaPH6+JEycqODhY+/fv1+LFi+Xs7FyeHwMAALtjMnPZJwDgLpCTk6PDhw9r0KKfdCE95+Y/YKF7qrhq+SuPq1GjRnJ1dbVZ3fJSIVcMAABA+SAYAAAAmaGTcgAACzNJREFUA8EAAAAYCAYAAMBwVwaDQ4cOqW/fvgoKClLr1q01derUG+6SuG7dOrVv315NmjRRnz59lJSUVIbdFpWSkqJmzZrp1KlTN5zXrVs3NW7cuMhdHb/++v+1d/cxTZ1tGMAvx3cRLIvMZLrw2XUu2ECGM2jgzVxicC5b6AIqfmAmW7YGo4RlG5GokC3bGs3mxjJniDEwDKCN0xCXxUxmlJGICcsEFyadOl2ijFkQKEILff/w5bxUqD2n1vYpvX5Jk/b4+PTy9nC46Tmcx+yjlP8nN68INe7r68M777yDF154AStWrEB1dfVDx/ujxkoyilBTQFlmUfbbSXL2X1HqDMjLK0qNlRyHRapxMAi6xiCQV25saWlBYWEhBgYGHjpubGwMPT09+P7779HR0SE9UlJSfJT0Prl5RalxaWkp4uLi0NraisOHD8NkMqG5uXnGsf6qsdyMotRUSWZR9ttJcvZfkeosJ68oNVZyHBapxsEi6BqDQF258ZtvvsHevXtRWlrqduzvv/+OiIgIJCUl+SDZzJTkFaHG169fx4ULF1BWVobIyEhoNBoUFRWhvr5+xvH+qLGSjCLUVGlmEfbbSXL3X1HqLDevKDVWchwWpcbBZFY2BqOjo7h58+aMD09Wbpx6a+WpKzf6KvPo6Cj0ej2am5uRlZXldq7Ozk6oVCps3rwZy5Ytg16vx5kzZ4TNK0KNr1y5ArVajfnz50vjk5OTXWbwRY0fpCSjr2rqjpLM/qipK3L3X1HqLDevKDV+7rnnZB+HRalxMBF2rYRHIdLKjd7OLIfD4cCSJUvw3nvvYdGiRTh9+jS2b9+Ouro6pKenC5dXhBobjUZERUU5bYuKisK9e/dmHO+LGj9oeHhYdkZf1dQdJZn9UVNXXC3Y9iBR6iw3r0g1nuTuOPw4amyzDmJs2Hv/R7awKPeDAsisbAxEWrlRLjmZ5dq4cSM2btwovV6zZg1OnjyJH3/80Wtf/N7MK0KNT58+Pe2b1cjIiMsMvqjxg1QqleyMvqqpO0oy+6Omj0qUOsslWo3lHIcDrcazwaw8leBOd3c39Ho95s2bh4aGBixcuNDl2EddudEf6uvrcfbsWadto6Ojwt6qU4QaazQaWCwW3LlzR9pmNpuRmpo643h/1FhJRhFqOplDbuZA228Bceosl0g1lnscDrQazwZB1xgEw8qNvb29qKqqwrVr12C322EymdDR0YG8vDx/R5uRCDVOTExERkYGjEYjrFYrenp6UFtb67Jm/qixkowi1FRp5kDbbwFx6iyXKDVWchwOtBrPBkHXGMzGlRsfzFxSUoLc3Fxs3rwZmZmZOHLkCA4ePIiEhAQ/J71P1Bp/+eWXGBoawksvvYQtW7YgPz8f+fn5M2b2V41dZRS1pkoyi77fAuLuu66IWuOHHYcDrcazEVdXJCKioDC5uuKmfWdxu997Fx8uUEehruw/XF2RiIiIZh82BkRERCSZlb+uSERE5IptZBA2q9V780XYvTaXCPiJAREREUnYGBAREZGEjQERERFJ2BgQERGRhI0BERERSdgYELmh1Wpx4sQJWWN7enrw888/P95AXhAoOYnI99gYELlx/vx55ObmyhprMBhw6dKlx5zo0QVKTiLyPTYGRG7Ex8fLvs1poNxh3Bs53333XXzwwQfS666uLmi1WlRUVEjbfvnlF6SlpaGvr++R34+IfIONAZEbU08laLVaHDt2DBs2bIBOp0Nubi4aGxsBAJs2bcJff/2F6upqrFy5EgAwMDCA8vJyLFu2DC+++CLeeustpyVkJ+fcv38/cnJykJOTg3/++QdDQ0OorKzE8uXLkZGRga1bt0p/T+6cDQ0N0Ov10Ol00Ov1aG9vd5nTE7GxsRgeHpZef/fdd1CpVLh79660rbGxEatWrcL8+fM9fh8ib7ONDGJseMBrD9vIoL//SV7FxoBIob1792LDhg04fvw4MjMzsWfPHvz999/46quvsHDhQrz55ps4duwYHA4H3n77bfT29qKmpgZHjhzB008/jcLCQlgsFqc5jx49im+//RbV1dWIj4/Hjh070NbWhn379sFkMkGlUqG4uBg2m032nEajEevWrcPx48eRlpaGrVu34saNG9NyeiomJgbW/909zmKx4NSpU1i7dq3UGPz777/46aefUFhY6PF7EJHvsTEgUuiNN97AK6+8gpSUFLz//vuYmJjAb7/9BrVajZCQEKhUKjz55JNoa2vDpUuXsH//fixZsgSpqamorKzEvHnz0NTU5DRnXl4eFi9eDJ1Ohz///BPnzp1DZWUlsrKykJycjKqqKqxatQqnTp2SPWdBQQEKCgqQkpKC3bt3Iz4+Hk1NTdNyempqY3D06FGkp6dDp9NJjYHJZEJSUhIyMzMBALt370Z2dja0Wq3H70lEjx/XSiBSKDExUXoeGxsLALDZbNPGXb58GePj48jOznbaPjo6CrPZ7LTtmWeekZ7/8ccfAACdTidti4uLw4cffoiamhrZcy5dulR6HhISgrS0NGlub5g8lTA+Po6GhgaUl5cjPDwcg4ODcDgcaGpqQnFxsTT+1VdfxbZt27BixQqvZSAi72NjQKRQeHj4tG0zXcwXFhYGtVo97Sd5AFCpVE6vp17cGBrq+stSyZwPzjMxMYE5c+a4nFupuXPnwmq1oqWlBQ6HAytXrsSvv/6Ku3fvorW1FRaLBa+99po0fmqjQhTM+vr6UFFRgfb2dkRGRmL9+vUoKSnxdywJTyUQedHUb7wajQb9/f0AgISEBCQkJGDRokX44osvpAsBZ5KSkgIA6OzslLYNDQ0hKysLw8PDsufs6uqSntvtdnR1deH555+fltNTsbGxsFqtqKurw7p16xASEoLo6GgMDQ2hsbEReXl505oVIgJKS0sRFxeH1tZWHD58GCaTCc3Nzf6OJWFjQORF0dHRuHbtGm7fvo2srCykp6djx44duHjxIq5evYqKigq0tLTg2WefdTlHUlISXn75ZVRWVuLixYswm80oLy9HTEwMiouLZc956NAh/PDDDzCbzdi1axcGBgawdu3aaTk9FRMTg/7+fnR0dCA/Px/A/U8R7HY7zpw5g/Xr13s8N9Fsdf36dVy4cAFlZWWIjIyERqNBUVER6uvr/R1NwlMJRF60ZcsWfPTRRzh//jza2trw9ddf47PPPoPBYMDY2BgWL16MmpoapKamPnSeTz/9FJ988gkMBgPGx8exdOlS1NTUIDw8XPacBQUFOHDgAK5evQqdTofa2losWLBgxpxPPKH8Z4SYmBhMTExg9erV0kWM0dHRAIDMzEzpkw8i0Tyljn4s8926dQshISFOfxYbGytdiwQAV65cgVqtdvoV3uTkZHR3d3s106OY4wiUO7IQkWxarRZGoxGvv/66v6NMo9VqhToIUvCw2+3o7OzE+Pi41+e22WwwGAxO9/YAgJKSEmzbtk16feLECXz++edOtyRvb29HUVERLl++7PVcnuAnBkTkEzt37sS5c+cAADk5OcjOzsbHH3/s51QUTEJDQ5GWlvZYGgOr1YqTJ09O2z710wLg/kXC9+7dc9o2MjKCuXPnej2Tp9gYEJFPsAkgEYSGhj70N388FRERgbi4OLfjNBoNLBYL7ty5I52CM5vNbk8v+hIvPiSahbq7u4U8jUAU7BITE5GRkQGj0Qir1Yqenh7U1tYiLy/P39EkvMaAiIjIh3p7e1FVVYX29naEhYWhsLAQBoPB37EkbAyIiIhIwlMJREREJGFjQERERBI2BkRERCRhY0BEREQSNgZEREQkYWNAREREEjYGREREJGFjQERERBI2BkRERCT5LxTnIueOEtiCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a pretty contour plot over the grid of w[0], w[1], loss values\n",
    "\n",
    "levels = np.linspace(0, 40, 51) # 50 evenly spaced levels\n",
    "\n",
    "fig_handle, ax_handle = plt.subplots(nrows=1, ncols=1, figsize=(8,8));\n",
    "\n",
    "ax_handle.contour(w0_GH, w1_GH, loss_GH, levels=levels, linewidths=0, colors='k')\n",
    "cntrf_handle = ax_handle.contourf(w0_GH, w1_GH, loss_GH, levels=levels, cmap='RdBu_r', vmin=levels[0], vmax=levels[-1]);\n",
    "\n",
    "cbar = plt.colorbar(cntrf_handle, ax=ax_handle)\n",
    "cbar.set_label('MAP loss objective (lower is better)', fontsize=16);\n",
    "cbar.set_ticks(levels[::10]);\n",
    "plt.xlabel('intercept   $w_1$');\n",
    "plt.ylabel('slope   $w_2$');\n",
    "plt.gca().set_aspect('equal', 'box');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2d: Visually interpret the plot above. By inspection, which intercept and slope values are optimal? What is the loss at this optimal point?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO interpret the plot and discuss with your group"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2e: Numerically, search the grid of computed loss values `loss_GH` and determine the MAP value of weight vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO solve this cell\n",
    "\n",
    "# Hint: you might find it easier to flatten each array of shape (G,H) into shape (L,) where L=G*H\n",
    "loss_L = loss_GH.flatten()# new shape (G*H,)\n",
    "w0_L = w0_GH.flatten()  # new shape (G*H,)\n",
    "w1_L = w1_GH.flatten()  # new shape (G*H,)\n",
    "\n",
    "# TODO find values of w0 (intercept) and w1 (slope) that minimize the loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Gradients, Hessians, and Gradient Descent\n",
    "\n",
    "### Gradient and Hessian formulas\n",
    "\n",
    "We saw in lecture that we can compute the gradient and Hessian as:\n",
    "\n",
    "\\begin{align}\n",
    "\\nabla_w \\mathcal{L} &= \\Phi^T ( \\sigma(\\Phi w) - t ) + \\alpha w\n",
    "\\\\\n",
    "\\nabla_w \\nabla_w \\mathcal{L} &= \\Phi^T R(w) \\Phi  + \\alpha I_M\n",
    "\\end{align}\n",
    "\n",
    "where $R$ is a diagonal matrix given by \n",
    "\n",
    "$$\n",
    "R = \\text{diag}( \\sigma(\\Phi w) \\sigma(- \\Phi w ) )\n",
    "$$\n",
    "\n",
    "The functions below compute the gradient and Hessian. You don't need to do anything, just inspect them to gain understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_R(w_M, phi_NM):\n",
    "    s_N = np.dot(phi_NM, w_M)\n",
    "    R_NN = np.diag( sigmoid(s_N) * sigmoid(-s_N) )\n",
    "    return R_NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_gradient_of_map_loss(w_M, phi_NM, t_N, alpha):\n",
    "    ''' Calculate the gradient.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    g_M : 1D array, shape (M,)\n",
    "        Gradient vector evaluated at current weights w\n",
    "    '''\n",
    "    # Compute predicted probability of positive class\n",
    "    yproba_N = sigmoid( np.dot(phi_NM, w_M) )\n",
    "    return np.dot(phi_NM.T, (yproba_N - t_N)) + alpha * w_M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_hessian_of_map_loss(w_M, phi_NM, t_N, alpha):\n",
    "    ''' Calculate the Hessian.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    H_MM : 2D array, shape (M,M)\n",
    "        Hessian matrix evaluated at current weights w\n",
    "    '''\n",
    "    R_NN = calc_R(w_M, phi_NM)\n",
    "    return np.dot(phi_NM.T, np.dot(R_NN, phi_NM)) + alpha * np.eye(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First-order gradient descent\n",
    "\n",
    "The code below performs 1st-order GD. \n",
    "\n",
    "While not converged, we perform the updates:\n",
    "\n",
    "$$\n",
    "w_{t+1} \\gets w_t - \\epsilon g( w_t )\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step   0/100  loss     -0.0000 | gradient_norm   12.4385 | intercept     1.231 | slope    -0.719\n",
      "step   1/100  loss     -0.0000 | gradient_norm    2.8867 | intercept     1.098 | slope    -0.275\n",
      "step   2/100  loss     -0.0000 | gradient_norm    2.4277 | intercept     0.878 | slope    -0.516\n",
      "step   3/100  loss     -0.0000 | gradient_norm    1.7594 | intercept     0.815 | slope    -0.262\n",
      "step   4/100  loss     -0.0000 | gradient_norm    1.5637 | intercept     0.708 | slope    -0.423\n",
      "step   5/100  loss     -0.0000 | gradient_norm    1.0530 | intercept     0.683 | slope    -0.278\n",
      "step   6/100  loss     -0.0000 | gradient_norm    0.8692 | intercept     0.634 | slope    -0.363\n",
      "step   7/100  loss     -0.0000 | gradient_norm    0.5274 | intercept     0.620 | slope    -0.299\n",
      "step   8/100  loss     -0.0000 | gradient_norm    0.3724 | intercept     0.599 | slope    -0.330\n",
      "step   9/100  loss     -0.0000 | gradient_norm    0.2076 | intercept     0.590 | slope    -0.311\n",
      "step  10/100  loss     -0.0000 | gradient_norm    0.1210 | intercept     0.581 | slope    -0.317\n",
      "step  11/100  loss     -0.0000 | gradient_norm    0.0752 | intercept     0.575 | slope    -0.313\n",
      "step  12/100  loss     -0.0000 | gradient_norm    0.0413 | intercept     0.571 | slope    -0.313\n",
      "step  13/100  loss     -0.0000 | gradient_norm    0.0350 | intercept     0.568 | slope    -0.313\n",
      "step  14/100  loss     -0.0000 | gradient_norm    0.0247 | intercept     0.565 | slope    -0.312\n",
      "step  15/100  loss     -0.0000 | gradient_norm    0.0202 | intercept     0.563 | slope    -0.312\n",
      "step  16/100  loss     -0.0000 | gradient_norm    0.0161 | intercept     0.562 | slope    -0.312\n",
      "step  17/100  loss     -0.0000 | gradient_norm    0.0130 | intercept     0.561 | slope    -0.312\n",
      "step  18/100  loss     -0.0000 | gradient_norm    0.0107 | intercept     0.560 | slope    -0.312\n",
      "step  19/100  loss     -0.0000 | gradient_norm    0.0089 | intercept     0.560 | slope    -0.311\n",
      "step  20/100  loss     -0.0000 | gradient_norm    0.0074 | intercept     0.559 | slope    -0.311\n",
      "step  21/100  loss     -0.0000 | gradient_norm    0.0063 | intercept     0.559 | slope    -0.311\n",
      "step  22/100  loss     -0.0000 | gradient_norm    0.0053 | intercept     0.558 | slope    -0.311\n",
      "step  23/100  loss     -0.0000 | gradient_norm    0.0045 | intercept     0.558 | slope    -0.311\n",
      "step  24/100  loss     -0.0000 | gradient_norm    0.0039 | intercept     0.558 | slope    -0.311\n",
      "step  25/100  loss     -0.0000 | gradient_norm    0.0034 | intercept     0.558 | slope    -0.311\n",
      "step  26/100  loss     -0.0000 | gradient_norm    0.0030 | intercept     0.557 | slope    -0.311\n",
      "step  27/100  loss     -0.0000 | gradient_norm    0.0026 | intercept     0.557 | slope    -0.311\n",
      "step  28/100  loss     -0.0000 | gradient_norm    0.0023 | intercept     0.557 | slope    -0.311\n",
      "step  29/100  loss     -0.0000 | gradient_norm    0.0021 | intercept     0.557 | slope    -0.311\n",
      "step  30/100  loss     -0.0000 | gradient_norm    0.0019 | intercept     0.557 | slope    -0.311\n",
      "step  31/100  loss     -0.0000 | gradient_norm    0.0017 | intercept     0.557 | slope    -0.311\n",
      "step  32/100  loss     -0.0000 | gradient_norm    0.0015 | intercept     0.557 | slope    -0.311\n",
      "step  33/100  loss     -0.0000 | gradient_norm    0.0014 | intercept     0.557 | slope    -0.311\n",
      "step  34/100  loss     -0.0000 | gradient_norm    0.0013 | intercept     0.557 | slope    -0.311\n",
      "step  35/100  loss     -0.0000 | gradient_norm    0.0011 | intercept     0.557 | slope    -0.311\n",
      "step  36/100  loss     -0.0000 | gradient_norm    0.0011 | intercept     0.557 | slope    -0.311\n",
      "step  37/100  loss     -0.0000 | gradient_norm    0.0010 | intercept     0.557 | slope    -0.311\n",
      "step  38/100  loss     -0.0000 | gradient_norm    0.0009 | intercept     0.557 | slope    -0.311\n",
      "step  39/100  loss     -0.0000 | gradient_norm    0.0008 | intercept     0.557 | slope    -0.311\n",
      "step  40/100  loss     -0.0000 | gradient_norm    0.0008 | intercept     0.557 | slope    -0.311\n",
      "step  41/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.557 | slope    -0.311\n",
      "step  42/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.556 | slope    -0.311\n",
      "step  43/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.556 | slope    -0.311\n",
      "step  44/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311\n",
      "step  45/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311\n",
      "step  46/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311\n",
      "step  47/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311\n",
      "step  48/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311\n",
      "step  49/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311\n",
      "step  50/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311\n",
      "step  51/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  52/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  53/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  54/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  55/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  56/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  57/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  58/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311\n",
      "step  59/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  60/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  61/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  62/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  63/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  64/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  65/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  66/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  67/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  68/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  69/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  70/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  71/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  72/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  73/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  74/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  75/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  76/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311\n",
      "step  77/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  78/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  79/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  80/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  81/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  82/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  83/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  84/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  85/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  86/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  87/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  88/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  89/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  90/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  91/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  92/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  93/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  94/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  95/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  96/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  97/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  98/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n",
      "step  99/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311\n"
     ]
    }
   ],
   "source": [
    "max_n_steps = 100\n",
    "\n",
    "w_M = 1.5 * np.ones(M)\n",
    "\n",
    "step_size = 0.2 # Selected by starting at 1.0, and trying smaller values until first 5 steps made loss better\n",
    "\n",
    "GD1_history_of_w = [w_M]\n",
    "GD1_history_of_loss = [calc_loss(w_M, phi_NM, t_N, true_alpha)]\n",
    "\n",
    "for step in range(max_n_steps):\n",
    "    \n",
    "    # Compute gradient\n",
    "    g_M = calc_gradient_of_map_loss(w_M, phi_NM, t_N, true_alpha)\n",
    "\n",
    "    # Update the weights by taking a step downhill\n",
    "    w_M = w_M - step_size * g_M\n",
    "    \n",
    "    # Print out progress\n",
    "    cur_loss = calc_loss(w_M, phi_NM, t_N, true_alpha)\n",
    "    print(\"step %3d/%d  loss %11.4f | gradient_norm %9.4f | intercept %9.3f | slope %9.3f\" % (\n",
    "        step, max_n_steps, cur_loss, np.sum(np.abs(g_M)), w_M[0], w_M[1]))\n",
    "    GD1_history_of_loss.append(cur_loss)\n",
    "    GD1_history_of_w.append(w_M)\n",
    "    \n",
    "    if step % 10:\n",
    "        step_size = 0.95 * step_size  # slowly decay the step size\n",
    "        \n",
    "bestw_fromGD_M = w_M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion 3a: Compare the GD estimate of the best weights $w$ to those found via grid search\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal weights via grid search\n",
      "[-2. -2.]\n",
      "Optimal weights via 1st order gradient descent\n",
      "[ 0.556 -0.311]\n"
     ]
    }
   ],
   "source": [
    "print(\"Optimal weights via grid search\")\n",
    "bestid = np.argmin(loss_GH.flatten())\n",
    "bestw_fromgridsearch_M = np.asarray([w0_GH.flatten()[bestid], w1_GH.flatten()[bestid]])\n",
    "print(bestw_fromgridsearch_M)\n",
    "\n",
    "print(\"Optimal weights via 1st order gradient descent\")\n",
    "print(w_M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " calc_loss(bestw_fromgridsearch_M, phi_NM, t_N, true_alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " calc_loss(bestw_fromGD_M, phi_NM, t_N, true_alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO discuss: which is better? are they similar?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Second order gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step   0/100  loss     -0.0000 | gradient_norm   12.4385 | intercept    -0.466 | slope    -2.940\n",
      "step   1/100  loss     -0.0000 | gradient_norm    6.3231 | intercept     0.627 | slope    -0.103\n",
      "step   2/100  loss     -0.0000 | gradient_norm    3.1766 | intercept     0.613 | slope    -0.129\n",
      "step   3/100  loss     -0.0000 | gradient_norm    2.7249 | intercept     0.603 | slope    -0.151\n",
      "step   4/100  loss     -0.0000 | gradient_norm    2.3576 | intercept     0.595 | slope    -0.170\n",
      "step   5/100  loss     -0.0000 | gradient_norm    2.0560 | intercept     0.589 | slope    -0.186\n",
      "step   6/100  loss     -0.0000 | gradient_norm    1.8062 | intercept     0.584 | slope    -0.199\n",
      "step   7/100  loss     -0.0000 | gradient_norm    1.5977 | intercept     0.581 | slope    -0.210\n",
      "step   8/100  loss     -0.0000 | gradient_norm    1.4225 | intercept     0.578 | slope    -0.220\n",
      "step   9/100  loss     -0.0000 | gradient_norm    1.2743 | intercept     0.575 | slope    -0.228\n",
      "step  10/100  loss     -0.0000 | gradient_norm    1.1480 | intercept     0.573 | slope    -0.236\n",
      "step  11/100  loss     -0.0000 | gradient_norm    1.0400 | intercept     0.571 | slope    -0.242\n",
      "step  12/100  loss     -0.0000 | gradient_norm    0.9421 | intercept     0.570 | slope    -0.248\n",
      "step  13/100  loss     -0.0000 | gradient_norm    0.8578 | intercept     0.568 | slope    -0.253\n",
      "step  14/100  loss     -0.0000 | gradient_norm    0.7848 | intercept     0.567 | slope    -0.258\n",
      "step  15/100  loss     -0.0000 | gradient_norm    0.7214 | intercept     0.566 | slope    -0.261\n",
      "step  16/100  loss     -0.0000 | gradient_norm    0.6660 | intercept     0.565 | slope    -0.265\n",
      "step  17/100  loss     -0.0000 | gradient_norm    0.6174 | intercept     0.565 | slope    -0.268\n",
      "step  18/100  loss     -0.0000 | gradient_norm    0.5746 | intercept     0.564 | slope    -0.271\n",
      "step  19/100  loss     -0.0000 | gradient_norm    0.5367 | intercept     0.564 | slope    -0.273\n",
      "step  20/100  loss     -0.0000 | gradient_norm    0.5031 | intercept     0.563 | slope    -0.275\n",
      "step  21/100  loss     -0.0000 | gradient_norm    0.4732 | intercept     0.563 | slope    -0.277\n",
      "step  22/100  loss     -0.0000 | gradient_norm    0.4451 | intercept     0.562 | slope    -0.279\n",
      "step  23/100  loss     -0.0000 | gradient_norm    0.4199 | intercept     0.562 | slope    -0.281\n",
      "step  24/100  loss     -0.0000 | gradient_norm    0.3974 | intercept     0.562 | slope    -0.282\n",
      "step  25/100  loss     -0.0000 | gradient_norm    0.3771 | intercept     0.561 | slope    -0.284\n",
      "step  26/100  loss     -0.0000 | gradient_norm    0.3588 | intercept     0.561 | slope    -0.285\n",
      "step  27/100  loss     -0.0000 | gradient_norm    0.3423 | intercept     0.561 | slope    -0.286\n",
      "step  28/100  loss     -0.0000 | gradient_norm    0.3273 | intercept     0.561 | slope    -0.287\n",
      "step  29/100  loss     -0.0000 | gradient_norm    0.3137 | intercept     0.560 | slope    -0.288\n",
      "step  30/100  loss     -0.0000 | gradient_norm    0.3013 | intercept     0.560 | slope    -0.289\n",
      "step  31/100  loss     -0.0000 | gradient_norm    0.2900 | intercept     0.560 | slope    -0.290\n",
      "step  32/100  loss     -0.0000 | gradient_norm    0.2791 | intercept     0.560 | slope    -0.290\n",
      "step  33/100  loss     -0.0000 | gradient_norm    0.2692 | intercept     0.560 | slope    -0.291\n",
      "step  34/100  loss     -0.0000 | gradient_norm    0.2600 | intercept     0.560 | slope    -0.292\n",
      "step  35/100  loss     -0.0000 | gradient_norm    0.2517 | intercept     0.560 | slope    -0.292\n",
      "step  36/100  loss     -0.0000 | gradient_norm    0.2440 | intercept     0.560 | slope    -0.293\n",
      "step  37/100  loss     -0.0000 | gradient_norm    0.2369 | intercept     0.559 | slope    -0.293\n",
      "step  38/100  loss     -0.0000 | gradient_norm    0.2304 | intercept     0.559 | slope    -0.294\n",
      "step  39/100  loss     -0.0000 | gradient_norm    0.2243 | intercept     0.559 | slope    -0.294\n",
      "step  40/100  loss     -0.0000 | gradient_norm    0.2187 | intercept     0.559 | slope    -0.295\n",
      "step  41/100  loss     -0.0000 | gradient_norm    0.2136 | intercept     0.559 | slope    -0.295\n",
      "step  42/100  loss     -0.0000 | gradient_norm    0.2085 | intercept     0.559 | slope    -0.295\n",
      "step  43/100  loss     -0.0000 | gradient_norm    0.2038 | intercept     0.559 | slope    -0.296\n",
      "step  44/100  loss     -0.0000 | gradient_norm    0.1995 | intercept     0.559 | slope    -0.296\n",
      "step  45/100  loss     -0.0000 | gradient_norm    0.1954 | intercept     0.559 | slope    -0.296\n",
      "step  46/100  loss     -0.0000 | gradient_norm    0.1917 | intercept     0.559 | slope    -0.296\n",
      "step  47/100  loss     -0.0000 | gradient_norm    0.1881 | intercept     0.559 | slope    -0.297\n",
      "step  48/100  loss     -0.0000 | gradient_norm    0.1849 | intercept     0.559 | slope    -0.297\n",
      "step  49/100  loss     -0.0000 | gradient_norm    0.1818 | intercept     0.559 | slope    -0.297\n",
      "step  50/100  loss     -0.0000 | gradient_norm    0.1790 | intercept     0.559 | slope    -0.297\n",
      "step  51/100  loss     -0.0000 | gradient_norm    0.1763 | intercept     0.559 | slope    -0.298\n",
      "step  52/100  loss     -0.0000 | gradient_norm    0.1737 | intercept     0.559 | slope    -0.298\n",
      "step  53/100  loss     -0.0000 | gradient_norm    0.1712 | intercept     0.559 | slope    -0.298\n",
      "step  54/100  loss     -0.0000 | gradient_norm    0.1689 | intercept     0.558 | slope    -0.298\n",
      "step  55/100  loss     -0.0000 | gradient_norm    0.1667 | intercept     0.558 | slope    -0.298\n",
      "step  56/100  loss     -0.0000 | gradient_norm    0.1647 | intercept     0.558 | slope    -0.298\n",
      "step  57/100  loss     -0.0000 | gradient_norm    0.1628 | intercept     0.558 | slope    -0.299\n",
      "step  58/100  loss     -0.0000 | gradient_norm    0.1610 | intercept     0.558 | slope    -0.299\n",
      "step  59/100  loss     -0.0000 | gradient_norm    0.1593 | intercept     0.558 | slope    -0.299\n",
      "step  60/100  loss     -0.0000 | gradient_norm    0.1578 | intercept     0.558 | slope    -0.299\n",
      "step  61/100  loss     -0.0000 | gradient_norm    0.1563 | intercept     0.558 | slope    -0.299\n",
      "step  62/100  loss     -0.0000 | gradient_norm    0.1548 | intercept     0.558 | slope    -0.299\n",
      "step  63/100  loss     -0.0000 | gradient_norm    0.1534 | intercept     0.558 | slope    -0.299\n",
      "step  64/100  loss     -0.0000 | gradient_norm    0.1521 | intercept     0.558 | slope    -0.299\n",
      "step  65/100  loss     -0.0000 | gradient_norm    0.1509 | intercept     0.558 | slope    -0.299\n",
      "step  66/100  loss     -0.0000 | gradient_norm    0.1498 | intercept     0.558 | slope    -0.299\n",
      "step  67/100  loss     -0.0000 | gradient_norm    0.1487 | intercept     0.558 | slope    -0.300\n",
      "step  68/100  loss     -0.0000 | gradient_norm    0.1476 | intercept     0.558 | slope    -0.300\n",
      "step  69/100  loss     -0.0000 | gradient_norm    0.1467 | intercept     0.558 | slope    -0.300\n",
      "step  70/100  loss     -0.0000 | gradient_norm    0.1458 | intercept     0.558 | slope    -0.300\n",
      "step  71/100  loss     -0.0000 | gradient_norm    0.1449 | intercept     0.558 | slope    -0.300\n",
      "step  72/100  loss     -0.0000 | gradient_norm    0.1440 | intercept     0.558 | slope    -0.300\n",
      "step  73/100  loss     -0.0000 | gradient_norm    0.1432 | intercept     0.558 | slope    -0.300\n",
      "step  74/100  loss     -0.0000 | gradient_norm    0.1425 | intercept     0.558 | slope    -0.300\n",
      "step  75/100  loss     -0.0000 | gradient_norm    0.1417 | intercept     0.558 | slope    -0.300\n",
      "step  76/100  loss     -0.0000 | gradient_norm    0.1410 | intercept     0.558 | slope    -0.300\n",
      "step  77/100  loss     -0.0000 | gradient_norm    0.1404 | intercept     0.558 | slope    -0.300\n",
      "step  78/100  loss     -0.0000 | gradient_norm    0.1398 | intercept     0.558 | slope    -0.300\n",
      "step  79/100  loss     -0.0000 | gradient_norm    0.1392 | intercept     0.558 | slope    -0.300\n",
      "step  80/100  loss     -0.0000 | gradient_norm    0.1387 | intercept     0.558 | slope    -0.300\n",
      "step  81/100  loss     -0.0000 | gradient_norm    0.1381 | intercept     0.558 | slope    -0.300\n",
      "step  82/100  loss     -0.0000 | gradient_norm    0.1376 | intercept     0.558 | slope    -0.300\n",
      "step  83/100  loss     -0.0000 | gradient_norm    0.1371 | intercept     0.558 | slope    -0.300\n",
      "step  84/100  loss     -0.0000 | gradient_norm    0.1367 | intercept     0.558 | slope    -0.300\n",
      "step  85/100  loss     -0.0000 | gradient_norm    0.1362 | intercept     0.558 | slope    -0.300\n",
      "step  86/100  loss     -0.0000 | gradient_norm    0.1358 | intercept     0.558 | slope    -0.300\n",
      "step  87/100  loss     -0.0000 | gradient_norm    0.1354 | intercept     0.558 | slope    -0.300\n",
      "step  88/100  loss     -0.0000 | gradient_norm    0.1351 | intercept     0.558 | slope    -0.301\n",
      "step  89/100  loss     -0.0000 | gradient_norm    0.1347 | intercept     0.558 | slope    -0.301\n",
      "step  90/100  loss     -0.0000 | gradient_norm    0.1344 | intercept     0.558 | slope    -0.301\n",
      "step  91/100  loss     -0.0000 | gradient_norm    0.1341 | intercept     0.558 | slope    -0.301\n",
      "step  92/100  loss     -0.0000 | gradient_norm    0.1337 | intercept     0.558 | slope    -0.301\n",
      "step  93/100  loss     -0.0000 | gradient_norm    0.1334 | intercept     0.558 | slope    -0.301\n",
      "step  94/100  loss     -0.0000 | gradient_norm    0.1332 | intercept     0.558 | slope    -0.301\n",
      "step  95/100  loss     -0.0000 | gradient_norm    0.1329 | intercept     0.558 | slope    -0.301\n",
      "step  96/100  loss     -0.0000 | gradient_norm    0.1326 | intercept     0.558 | slope    -0.301\n",
      "step  97/100  loss     -0.0000 | gradient_norm    0.1324 | intercept     0.558 | slope    -0.301\n",
      "step  98/100  loss     -0.0000 | gradient_norm    0.1322 | intercept     0.558 | slope    -0.301\n",
      "step  99/100  loss     -0.0000 | gradient_norm    0.1320 | intercept     0.558 | slope    -0.301\n"
     ]
    }
   ],
   "source": [
    "max_n_steps = 100\n",
    "\n",
    "w_M = 1.5 * np.ones(M)\n",
    "\n",
    "step_size = 0.15 # Selected by starting at 1.0, and trying smaller values until first 5 steps made loss better\n",
    "\n",
    "GD2_history_of_w = [w_M]\n",
    "GD2_history_of_loss = [calc_loss(w_M, phi_NM, t_N, true_alpha)]\n",
    "\n",
    "for step in range(max_n_steps):\n",
    "    \n",
    "    g_M = calc_gradient_of_map_loss(w_M, phi_NM, t_N, true_alpha)\n",
    "    H_MM = calc_hessian_of_map_loss(w_M, phi_NM, t_N, true_alpha)\n",
    "    w_M = w_M - step_size * np.linalg.solve(H_MM, g_M) # compute H^1 times g\n",
    "    cur_loss = calc_loss(w_M, phi_NM, t_N, true_alpha)\n",
    "    \n",
    "    print(\"step %3d/%d  loss %11.4f | gradient_norm %9.4f | intercept %9.3f | slope %9.3f\" % (\n",
    "        step, max_n_steps, cur_loss, np.sum(np.abs(g_M)), w_M[0], w_M[1]))\n",
    "    GD2_history_of_loss.append(cur_loss)\n",
    "    GD2_history_of_w.append(w_M)\n",
    "    \n",
    "    if step % 10:\n",
    "        step_size = 0.95 * step_size # slowly decay step size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "whistory_GD1_T2 = np.vstack(GD1_history_of_w)\n",
    "whistory_GD2_T2 = np.vstack(GD2_history_of_w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mhughes/miniconda3/envs/semimarkov_forecaster/lib/python3.7/site-packages/ipykernel_launcher.py:7: UserWarning: No contour levels were found within the data range.\n",
      "  import sys\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAHxCAYAAAB9MoV9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3hT5RfA8W/SJN17skcZZYOUIoLs4WCjbGSpIAKioMhUGQIKygZBpixBGQr82EtBQBBkKXsW6KQTaJrm/v6oDS0dNKVtOs7neXwec+/NuSet9uR97ztUiqIoCCGEECLfUVs6ASGEEEJkjRRxIYQQIp+SIi6EEELkU1LEhRBCiHxKirgQQgiRT0kRF0IIIfKpPF3Ez5w5Q8+ePfH39+fll19m0qRJPHr0yNJpCSGEKGQSEhLo1asXn376qenY/v37ee2116hRowbt2rXjxIkTuZ5Xni3i0dHRvPPOOzRv3pyjR4+yYcMGTp06xfTp0y2dmhBCiEJm7ty5KYr0jRs3GDZsGJ988gknTpyga9euvP/++zx8+DBX88qzRTwwMBB/f3/69OmDRqPBx8eHdu3a8eeff1o6NSGEEIXIH3/8wY4dO3jllVdMxzZt2kSdOnVo3LgxWq2Wbt264e3tzbZt23I1N02u3s0Mfn5+zJs3z/RaURR2795N1apVn/leo9FIbGwsWq0WlUqVk2kKIUShpigK8fHx2Nvbo1Znf7vQYDCQkJCQrTEfPnxIbGxsquNOTk44OTmlOBYWFsaYMWOYO3cuq1evNuVy5coVypcvn+JaX19fLl68mK25PkueLeLJGQwGPv/8c27fvp2p7vTY2FguXbqUC5kJIYQAqFChAo6Ojtka02AwcPzQ71g7Z2/c+Ph4Bg0alKqQDx48mCFDhpheG41GPv74Y/r06UPlypVTXBsbG4uNjU2KYzY2Nrk+bivPF/GwsDA++ugjwsPDWb16Nd7e3s98j1arBWDg178QEpH625YQQojs4eliz8KP25r+7manhIQErJ0d+d87Y4kNDsuWmPZe7ry6eBKbNm3CysoqxbmnW+HfffcdOp2Ot956K1UcW1tbHj9+nOLY48eP8fLyypY8MytPF/GLFy/y7rvvUqNGDebPn4+9vX2m3pfUhR4SEcv9sJicTFEIIQTk6KPL2OAwYu+FZGtMHx8frK2tM7xmy5YtBAcH4+/vD2Aq2ufPn6dp06apus6vXLnCSy+9lK15PkueLeKhoaH069eP1q1bM2rUKEunI4QQopDZsWNHitdjxowhISGBqVOncuXKFZYvX86ePXto1KgRP/30E/fu3aNFixa5mmOeLeLr1q0jNDSUH3/8kfXr15uOFy1aNNdH/wkhhBDJlStXjtmzZzN9+nQ+/vhjypQpw3fffYeLi0uu5pFni/jgwYMZPHiwpdMQQgghAJg8eXKK140aNaJRo0YWyiZRnp0nLoQQQoiM5dmWeE4LqFycAe0DcHOyRS1zyYWwGKOiEB71iO82H+f4hTuWTkeIfKVQFvGAysUZ3bsJnl4+oLICKeJCWI6i4OWdwOje9ny5Yr8UciHMUCi70we0D0gs4GqNFHAhLE2lArUGTy8fBrQPsHQ2QuQrhbIl7uZkm9gCF0LkHSqrxP83hXiKk0aNRps9bU47TcFquxasT5NJapXKrBb4iiULOXHsiNn3OXHsCCuWLDT7fUIUSiqVjE8RwkyFsoibq0rV6nw6fLBZhfzEsSN8OnwwVapWz8HMhBBCFGZSxDPBv+5LTJ0xN9OFPKmAT50xF/+6ubsEX1qC7t+zdApCCCFygBTxTMpsIc+pAj5x7Ah+WrcyU9cuWTCLJQtmAXDqxDGmfP6p6dy0CaPZ/NPaTMXp3qElF879bX6yOWDh7K9ZOPtrS6chhBB5SqEc2JZVyQt5WkU6r7TA+7/3genfo6IiMRqNptcjx39piZSEEELkACniZkqvkOdmAT+4bxf7d/+PMr7lOHxoP1ZWVvgHvESfdwdjZWVlarE2bNqSpQtnYTAY6NutLbO/W8XMryZQqWp13uj6Fg/Cw/hh6UKuXv6XyIgIXN3d6dS5Jw0aN8/w/t9Om4CHpye9+r0HwOwZk7l57Soz5i015bf7f78y6es5Kd73xeiPqFy1Bm927206NmJwf15t25FmLV/n103r+f3AHsJCQ1Cr1bwQUI/+A4ei1epSxPlp3Ur+OXeGcZOe7C3fs9MrjP5iGpWr1uDhw1jWrlzCqRNHMRgM1KjlT6/+7+HgkL17EgshhKVJd3oWPN21bokW+KV/z+Ps7Mr8pev4ZOwkfj+0lxPHDqe4pnLVGvQb+AHuHl4sW/sLjk/tlbt4/rforK35evb3LF2zmeatWrNk4SweP2NTe/+6L3Hm1EkAjEYj58+cJjQ0mLDQxK0C/zr+B3VeTP1zaNL8FX4/tBdFUQC4culfwkKDqdegMUcPH2LH1k18+OlnfL96E59N+Za/jv/B4YP7zP7ZfDdnBsFB9/hyxny+mb8MRVGY+80Us+MIIUReJy3xLEoq5AP7dQdg4dI1udqFrtNZ0+6NbqhUKsr4lqdU6bIE3rllVoz+Az/A3t4BjVZLWGgItrZ2xMXFERMTjY1t+vN1a9UOYNHcGYSFBvMgPBw3dw/s7R04f+YULzZozJm/T9KlV79U76tbvyErlszn4j/n8Ktcjd/276buSw2xs7Onxgv+VPCbg5u7BzHRUcRER+Po5Ex4WKhZnyky4gF/Hv2d6XOX4OScuJtQz34DGdj7TYLu3cW7SFGz4gkhLM9Bo0ajyZ61PWwK2DxxKeL5lJOzC6pkc2qtrDQpnn1nRkjwfWavXMK9wNt4+RSlSNFiAChKxnEcHJ2oWKkqZ06d5MGDMKrVeAFbOzvOnT2No5MzHp7eFC1WItX7rK1teKlBE37bv4dy5f344/cDfDTq88R7GhXWr1nOqRPHcHBwpGTpssTHx2N8Ri6pP1MQAOM/GZriuFanIzjonhRxIUSBIkU8i5K60BcuXQOQJwa0mcNgMPD15HF07tGXlq+2RaVSce3qJQ4fylz3de2Al/j71J9ERUbQ/s3u2NnZs/t/v2Kts6ZOBj+Dxs1fYcoXo6hWszYOjk74Va4GwLpVSwgNDuLbBcuxs7MHYOQH76YZQ61Wk5CQYHodGxtj+gLj5u4BwIx5S3F2cQXAmJBA4J1bFClaPFOfTQgh8ouC1a+QS55+Bm7uPPLcpNVqiYt7nKLoARji49HHxWFtbYNKpSI0JJi1K75PPGdISCtUCv51X+Lc36e4ef0afpWqUbZcReLj4/n94B7qvNgg3ff5lq+Ih6cXa1YspnHzV0zHY2Ni0Gq1WFlZodfr2bblJ27fukFCGrkULVaCm9evcOPaFeLj9Wz8cZWpV8LN3YNqNWvzw9KFxMREY4iPZ/3aFUwYOwK9Xp+pn5kQQuQXUsTNlN4gtrxayCtVqY6TkzNv9+zAvcAnu0PZ2NoyYPBwNq1fRb9u7Zg8/hOq1ayNk5Mzd27feGZcTy9vPL28KeNbHp21NWq1mirVauLo5EIZ3/IZvrdxs1aEh4XwcrJR8J179CUuLo6BfToz9N2eXLn0L/UbNk0zF/+69Xnp5aZM+WIUHwx4C0cnZ7y8i5jOv//hp+h0OkZ+8C4D+rzJ5X8vMPrzqdjZ2z/7ByaEEPmISkkaKlyAxMXFce7cOTqNWcv9sJhU57dM64mHT+pnts+SmVHoeWWuuBD5Uej927QbucrSaQgz+Lg78PPkblStWhVra+tsjZ30t/xIlyE8vm/eINf02Ph48NKPc3IkX0uQlngmZbY459UWuRBCiIJHingmmNu6lkIuhBAiN0gRz4Tz586Y3T2eVMjPnzuTg5kJIYQozGSKWSb07j8wS+9LGrkuhBAi6xw1aqy12dPm1BWwxV4K1qcRQgghChEp4kIIIUQ+JUVcCCGEyKekiAshhBD5lBRxwcOHsURFRVo6DYLu37N0CrmqsH1eIUT2k9Hp+cTQd3sRGRGO2irldnw1atVh2CfjmDZhNBUrV6P9G93Mjv3RoL4MHTGGylVrZHjd1s0bOHXiGOMmTTcdS0hIYM2KRfx+cB+G+Hhq1alLvwFDTZuYZNaN61cZ/8lQVm7YZnb+Oemf82eYP3MaD2Nj6dl3AE1avJrmdZf+vcCkcR9nOv/dO37l2OFDjJ34dXamm8JP61byz7kzKX5fz+v2zev8unkDF86eJiY6GhsbG/yqVKNdp26m5XYP7tvForkz0P23GpZiVLCzt6dW7bp06dUPJyfnbMtHiMJOingWtGzon+Y+127uHuw6dCLH7ttv4Ac0atoyzXMjx3+Z5bhRkREZno+Le8xPa1eybctPVKpSPcW5TRtW88/5s0z9diFarZY530xhzYrFvP3eMLNyeBgbg8EQb3buOe33A3spXrI0I8dNTvO8oigc2r+bFYvnmZV/ZMQD8tuKx2f//osZX35Gy9faMn7yDDw8vYmOiuTIbweYMHY4E6bOokSpMgC4e3gxe9EPpvfevxfI9/NnMnncx0yaPhetVmepjyFEgSJFPAvSKuAZHc8NE8eOoFLV6rzR9S0mjh2Bh6cX58+exkqj4evZ37N103r27d5OXFwc3j5FeaNrL2rWDmD4+/0AmDZxDN169eeV1h1SxR790SDK+Jan+SutCbx9K8W5A3t20KPPu7i6uQPQrVd/Pvv0A3r2GYCNrW2Kax8/esTi+d9y9u+/sLKyonQZX3q/8z4qlZppE8cA0LdbWz6b/A0lS5dlx9ZN7NmxlaioSMqULUfvtwdRvGRpALp3aEmPPu+yc9sWYmOiqVK9Jv0HfoCzi2u69/EpUizVZzMajfy66Uf27f4fsdHRFC9Zmu6936aCXxXmzPiS43/8ZspryapNqXpCvp8/k9u3btCpay9WL1+U4tyd2zdZunAWt25cx87egarVa9L7nfc5dvgQW35eh9Fo5L2+XViw7Mc0f6d/HjvMxh9XEXz/Hm7uHrze/k0aN2tl+n0//Tu+ce0yK5csIPD2TUqV8cXbJ+Xe6X+fOsGGNcu5d/cOHp7evNHtLerUrZ9uPJ3uSaE1JiSwaO4MWrzahu693zEdd3Zx5dU2HdBoNcTGpt6nIIlPkWIM+2QcQ97pwaH9u2nW8vV0rxVCZJ4U8f9s3fIzv2xa/9xx3u3TJVPXte3QmdbtOj33/dLzz/kzTPp6LlqdjssXL7Bz+xa+nDEfdw9P9u7axrJFc/h2wQpmzFtK9w4tGTlucrrd6WMmfIWbuwc/rVuZoog/jI0lPCyU4iVKmY4VLV6S+Ph47t29k2o3s62bNxAVGcHsRT9gZWXFkgWz2Lh+NYM++ISR4yYzadzHLFv7CwA7t21mx9ZNjBg9gaLFS7Jv13YmfzaSGXOXmnYjO7R/N2MnfoWDgxPzvp3KrK8nMX7yjAzv87SN61fx2/7dDB89gaLFSnBgzw6mfDGKr2YtYsjw0SycndjdPXDox2n+bDp26Ym7hycXzv2d6tzShbOpWKkaYydOJyYmmsnjP+HIb/tp0vxVgoPvZ9jVff7saeZM/5KhI8bwgn9dLl28wIwpn2NjY8uL9Rum+h3HPX7MV5PG0a5TV15t05Grly/y1aQxlC5TDkh8XPHN1M8Z/OGn1K5Tj3//Occ3U7/A2dmVCn6VU8VLXsABLl38h7DQEJq3ap1mvi1eaZPm8eQcHJ2oWLka58+eliIuzOKitcKgs3r2hZmg0WZPnLxCing+suy7OfywZIHptdrKikUrf0rz2uq1/HFxdQMS9xR/+DCWg3t38kJAPZo0f5WmLV4z7cH9LG7uHmkef/ToIQA6axvTMZ1Oh0qlIi4uLtX1Wp2WwDu3OHJoP9Vq1ubdwcNRq9MeW7lnx1badepKydJlAWj5Wlv27PiVY38coknzxOfSnbv3MW1B2qPvu4wY3J+Q4Ptm3efg3p2069SNkv91Azd/pTW/H9zD4UP7MzW+wN3DM91zWp2O82dPUeZYeSpXq8GUbxakm0eqvPbtwv/FJyv++VWuRvNWrdm/e7upiCf/HR/atwsbGxtat38TlUpFxUpVaNysFTeuXQVg365t+Ae8ZNrrvXLVGtRv2JQ9O341FfHk8Z4WHhaS6vMe2LuTVUsXApBgTKBCxcqM+nxqhp/LwdHxmY9vChPDvdOo7T1RO6XuJcqIMSoQY2wImiI1cygzkV9IEf9P63adMt0y9q9SOt1zi5an3TWaHfoOGJLuM/Gnubg8+WNcwa8Kwz4ex87tm/ll44/Y2NrS8rW2tH+je6aLSlqsbRKLd7z+ScHW6/UoioKdnV2q69t06IJGo2XPzq0s+W42RYoUo3OPvgTUa5Dq2pDgIFYtX8TalUtMxxISDIQEB5le+xR98ofPw9MbgAfh4WbdJzIiAk8v7xTHPL18CAsJzuyPIV1DR4zhp7UrWLNiMSEhQVT0q0K/AUNMjwSS/H5wL0sWzjK97j/wAyIjHpha0Um8vH04fvQ30+vkv+Pw8DDc3D1TfDHz8ilqKuIhwUFcOHuat3s8eVxiNBopXbZcmvGellTcHzwIw9PLB0jcFz6pez9pEN2zREVG4OTs8szrCgu1vSfx1/agLds804XcGBVoeo8QUsQLqmSN7LDQYFzd3Bn9+TTi4/WcOXWS2dMnUaZseWr5183yLRwcHHF1c+du4G1TYQq8fRONRpvm8+c7t27gH/ASr7d7g5joKHbv+JU5Mybz3YrUvQmu7u507t6Heg0am47dDbydoqUYHhZKseIlAQgJug+Ah6dXhvdJ6opP4uHpRXDw/RTHgu7fo3qt2ln6mSRRFIWb16/yZrc+9H77fUKC77Nq2SIWzpnOpK/npri2QaNmNGjULMWxf86fISRVXndTFtpkv2M3dw9CQ4IwGo2mL2bhoU/GaLi5edCwaSv6DxxqOhYWGoyVVbI/ARl0zFTwq4Kziyv79+ygc/c+z/j0aYuJjuLyvxfo/fb7WXp/QaR2Koa2bPNMF/LkBdzc1rsomGSeeBak172c3nFLu3blElMnjObm9atotTpc3RMHoTk6OQGg0Wh5GBubpdgNm7Rk04Y1hIWGEBMdxbpVS3mxfkPT9KLk9u/ZwYLZXxEZ8QB7B0ccHByxtrFFo9Wi0WgBTHk0btaKzT+t5d7dQBRF4c9jhxk5bAA3r181xftp7UoiHoQTExPN6hWLqOVfFzd3jwzv87TGzVqxddN6bt28jsFgYM+Ordy8fiXFl4esUKlUrFyygPVrlqPX63F2ccPa2hpHx8SfedIjjvQ0atqKE8eOcOLYEYwJCfx74Rx7dm7j5SYt0ry+dkA9FEVh4/pVGAwGrl6+yP7d25/Ea9aSI7/t4/yZUyiKwp3bN/l81IfsS3ZNRjQaDQOHfsz/fvmZdauWEhaa2FMRHRXFzu1b2Llti2lwY1ru3L7JzK8mUqxEqVRfWAq75IXcGBWY7nVSwEVapCWeBTk5jSwn1HmxAXcDbzNjymdER0Xi7OLGW/3fo1yFSgA0afEqc7+ZQpuOnenUpZdZsTt16YnBEM+4T4ag1+upWbsOfQcMSfPaLj36smzRHD754F30+jiKFS/FiNFfoNPpKFm6DBX8KvN+/64MHz2B1u07A/DVxDFERjzAw9OL94Z+nGKKW+my5fh81IfEREfxQkA9er896Jn3edrr7d/EaDQyY8pnREVGUKJkaUaO/9LUwn8eQ4aPZvnieQzq1xVFMVK5Sg3eHvQhAC/4v8ieHVvp36M985euwzrZuAKACn6VGTJ8FD+vW8m8mVNxdXWnS48+pu7rp9nbO/Dp+C9ZtmgO2zZvwKdIMQJeepl7gXf+i1eFgUNGsHrFYoLu38XW1o6GTVvS/o3umf48NWr5M3nGfLZu3sAXo4cTHR2JVqOljG95+rzzfoovPmGhwfTt1hYAtUqNk7MLtQPq0bFLzzS/TBV2SYV83aW+uJH6sUY44XTS9JACLlJRKfltsmomxMXFce7cOTqNWcv9sNTTXrZM64mHTwkLZCayS/cOLRk78etnLlAj8pfQ+7dpN3KVpdOwmB0nuqZ7rmWFGXmygPu4O/Dz5G5UrVoV6zR64J5H0t/y230+whCcPVN4NV4elFj+TY7kawnSnS6EEPlAXizgwvKkO10IIUSeZmtjhdE2e8qV2kbmiQthcWs27bJ0CkIIYXHSnS6EEELkU1LEhRAij9CjT/N4OOEZTj8ThZd0pwshRB7gFRaNFVassd7GYsNsrGv1NZ2TOeIiPdISF0IICzNGBdLmhgcKsEW3G5XOIcX5zC4IIwofKeJCCGFBxqhAdFeP8JrqNQ653SbEcBuVzjHVdVLIRVqkiAshhIUkdZO3cRuCvdGajd4XUfQxYO2Q5vVSyMXTpIgLHj6MJSoq0tJpEHT/nqVTyFWF7fOK1IyxIViXaUH7yDqccQjmsvV9SNCn2RJPklTIjbEhuZipyKtkYFs+MfTdXkRGhKO2SrlQQY1adRj2yTimTRhNxcrVMrUH9tM+GtSXoSPGPHMJ062bN3DqxDHGTZpuOpaQkMCaFYv4/eA+DPHx1KpTl34DhmJnZ59BpNRuXL/K+E+GsnLDNrPzz0n/nD/D/JnTeBgbS8++A2jS4tUU5/8+dYL1q5dx/14g9vYONGzako6dez5zi9fdO37l2OFDjJ34dY7lnrQ9aPLf1/O6ffM6v27ewIWzp4mJjsbGxga/KtVo16kbZXzLA4l7oS+aO8O0CY5iVLCzt6dW7bp06dUPJyfnbMsnv9MUqUmD8OJ46+1ZWOKvxFY4pHom/jS1U7FCNcBN56BDicueJVJVDqn3UMjPpIhnQUzDmyhhxlTHVe5qHA6VyrH79hv4Qbr7iY8c/2WW40ZFRmR4Pi7uMT+tXcm2LT+l2IAEYNOG1fxz/ixTv12IVqtlzjdTWLNiMW+/N8ysHB7GxmAwxJude077/cBeipcszchxk1Odu38vkJnTvmDAkI8JqNeAe3fv8NXEMdja2vF6uzcyjBsZ8YD8tm3B2b//YsaXn9HytbaMnzwDD09voqMiOfLbASaMHc6EqbMoUaoMAO4eXsxe9IPpvffvBfL9/JlMHvcxk6bPRastWH9In0fHID/uWkdzzOUuSmQ0QIYtcSGSkyKeBWkV8IyO54aJY0dQqWp13uj6FhPHjsDD04vzZ09jpdHw9ezv2bppPft2bycuLg5vn6K80bUXNWsHMPz9fgBMmziGbr3680rrDqlij/5oEGV8y9P8ldYE3r6V4tyBPTvo0edd0zaU3Xr157NPP6BnnwHY2NqmuPbxo0csnv8tZ//+CysrK0qX8aX3O++jUqmZNnEMAH27teWzyd9QsnRZdmzdxJ4dW4mKiqRM2XL0fnuQad/y7h1a0qPPu+zctoXYmGiqVK9J/4Ef4Ozimu590trj3Gg08uumH9m3+3/ERkdTvGRpuvd+mwp+VZgz40uO//GbKa8lqzal6AkJCbpP/UbNeLF+QwCKFS+Jf936/Hv+LK+3e4M7t2+ydOEsbt24jp29A1Wr16T3O+9z7PAhtvy8DqPRyHt9u7Bg2Y9p/k7/PHaYjT+uIvj+PdzcPXi9/ZumXczS+h3fuHaZlUsWEHj7JqXK+OLtUzRFvL9PnWDDmuXcu3sHD09v3uj2FnXq1k83XvJd34wJCSyaO4MWr7ahe+93TMedXVx5tU0HNFoNsbGpNxtK4lOkGMM+GceQd3pwaP9umrV8Pd1rCxO/GHcqx3owv8RJjCoF4v5riafzTFyIp0kR/0/8lmjiN6X/RyizHvbJ3HNObQcHtO1y7tv2P+fPMOnruWh1Oi5fvMDO7Vv4csZ83D082btrG8sWzeHbBSuYMW8p3Tu0ZOS4yel2p4+Z8BVu7h78tG5liiL+MDaW8LBQipd40vtQtHhJ4uPjuXf3jql7NcnWzRuIioxg9qIfsLKyYsmCWWxcv5pBH3zCyHGTmTTuY5at/QWAnds2s2PrJkaMnkDR4iXZt2s7kz8byYy5S7GzT+yqP7R/N2MnfoWDgxPzvp3KrK8nMX7yjAzv87SN61fx2/7dDB89gaLFSnBgzw6mfDGKr2YtYsjw0SycndjdPXDox6neW61mbarVrG16HR+v5/Rfx6nfsCkASxfOpmKlaoydOJ2YmGgmj/+EI7/tp0nzVwkOvp9hV/f5s6eZM/1Lho4Ywwv+dbl08QIzpnyOjY2t6UtD8t9x3OPHfDVpHO06deXVNh25evkiX00aQ+ky5YDExxXfTP2cwR9+Su069fj3n3N8M/ULnJ1dqeBXOVW8p7dtvXTxH8JCQ2jeqnWa+bZ4pU2ax5NzcHSiYuVqnD97Wor4fzoEVSTGSs8uj+sAKPpoUFmBxvYZ7xQikRTxfGTZd3P4YckC02u1lRWLVv6U5rXVa/nj4pq4L7FWq+Xhw1gO7t3JCwH1aNL8VZq2eA2VSpWp+7q5e6R5/NGjhwDoku2FrdPpUKlUxMXFpbpeq9MSeOcWRw7tp1rN2rw7eHi6z4737NhKu05dKVm6LAAtX2vLnh2/cuyPQzRpnvhcunP3Pnh5FwGgR993GTG4PyHB9826z8G9O2nXqRsl/+sGbv5Ka34/uIfDh/abNb4gLu4xs7+ejE5nzWttOv33eXWcP3uKMsfKU7laDaZ8s+CZz8pNee3bhf+LL+Ff9yUA/CpXo3mr1uzfvd1UxJP/jg/t24WNjQ2t27+JSqWiYqUqNG7WihvXrgKwb9c2/ANeos6LDQCoXLUG9Rs2Zc+OX01FPHm8p4WHJQ6icvfwNB07sHcnq5YuBCDBmECFipUZ9fnUDD+Xg6PjMx/fFBaecXY0eFCcjd4XeWxlAEDRx6DSOWT6/00hpIj/R9vOMdMt4+gq19M9Z7e8SHallErfAUPSfSb+NBeXJ3+MK/hVYdjH49i5fTO/bPwRG1tbWr7WlvZvdM90UUmLtU1i8Y7XPynYer0eRVGws7NLdX2bDl3QaLTs2bmVJd/NpkiRYnTu0ZeAeg1SXRsSHMSq5YtYu3KJ6VhCgoGQ4CDTa5+iT7rHPTy9AXgQHm7WfSIjIvD08k5xzNPLh7CQ4Mz+GAi6d5eZX03Azt6BMRO+Mj1GGDpiDD+tXcGaFYsJCQmiol8V+g0YYmIpI+0AACAASURBVHokkOT3g3tZsnCW6XX/gR8QGfHA1IpO4uXtw/Gjv5leJ/8dh4eH4ebumeKPv5dPUVMRDwkO4sLZ07zd48njEqPRSOmy5dKM97Sk4v7gQRieXj4ANG7WytS9nzSI7lmiIiNwcnZ55nWFQbvgxJ6qLd6XTceUuGiwlufhIvOkiBdUyb7Ih4UG4+rmzujPpxEfr+fMqZPMnj6JMmXLU8u/bpZv4eDgiKubO3cDb5sKU+Dtm2g02jSfP9+5dQP/gJd4vd0bxERHsXvHr8yZMZnvVqTuTXB1d6dz9z7Ua9DYdOxu4O0ULcXwsFCKFS8JJD6fBvDw9MrwPkld8Uk8PL0IDr6f4ljQ/XtUr1WbzDh/9jTfTptA/YZN6dVvIBpN4v9SiqJw8/pV3uzWh95vv09I8H1WLVvEwjnTmfT13BQxGjRqRoNGzVIc++f8GUJS5XU3ZaFN9jt2c/cgNCQIo9Fo+mIWHhr65LybBw2btqL/wKGmY2GhwVhZJfsTkEHjr4JfFZxdXNm/Zwedu/fJ6EeSrpjoKC7/e4Heb7+fpfcXJLYJGl4N9eU319uE6h6ajiv6GNR27hbMTOQ3Mk88C1Tuaf/Y0jtuadeuXGLqhNHcvH4VrVaHq3viHwlHJycANBotD2NjsxS7YZOWbNqwhrDQEGKio1i3aikv1m9oml6U3P49O1gw+ysiIx5g7+CIg4Mj1ja2aLRaNBotgCmPxs1asfmntdy7G4iiKPx57DAjhw3g5vWrpng/rV1JxINwYmKiWb1iEbX86+Lm7pHhfZ7WuFkrtm5az62b1zEYDOzZsZWb16+k+PKQnju3bjB98jg6delF33cHmwo4gEqlYuWSBaxfsxy9Xo+zixvW1tY4Oib+zJMecaSnUdNWnDh2hBPHjmBMSODfC+fYs3MbLzdpkeb1tQPqoSgKG9evwmAwcPXyRfbv3v4kXrOWHPltH+fPnEJRFO7cvsnnoz5kX7JrMqLRaBg49GP+98vPrFu1lLDQxJ6K6Kgodm7fws5tW0yDG9P8Wd2+ycyvJlKsRKlUX1gKo5ahZbBP0LHR+6LpmGI0gOGRjEwXZpGWeBbk5DSynFDnxQbcDbzNjCmfER0VibOLG2/1f49yFSoB0KTFq8z9ZgptOnamU5deZsXu1KUnBkM84z4Zgl6vp2btOvQdMCTNa7v06MuyRXP45IN30evjKFa8FCNGf4FOp6Nk6TJU8KvM+/27Mnz0BFq37wzAVxPHEBnxAA9PL94b+nGKKW6ly5bj81EfEhMdxQsB9ej99qBn3udpr7d/E6PRyIwpnxEVGUGJkqUZOf5LUws/I9u2/ExcXBzr1yxj/ZplpuN+laoycvyXDBk+muWL5zGoX1cUxUjlKjV4e9CHALzg/yJ7dmylf4/2zF+6Dutk4woAKvhVZsjwUfy8biXzZk7F1dWdLj36mLqvn2Zv78Cn479k2aI5bNu8AZ8ixQh46WXuBd75L14VBg4ZweoViwm6fxdbWzsaNm1J+ze6P/NzJqlRy5/JM+azdfMGvhg9nOjoSLQaLWV8y9PnnfdTfPEJCw2mb7e2AKhVapycXagdUI+OXXqm+WWqMFErKtoFV+S8fQiXHMJNx01zxGVkujCDSslvk1UzIS4ujnPnztFpzFruh6Uecb5lWk88fEpYIDORXbp3aMnYiV8/c4Eakb+E3r9Nu5GrLJ1GjnrpQTHGX32ZSWV/53e3O6bjxsjbxF/ejrZiG9SORTOIkLf4uDvw8+RuVK1aFes0euCeR9Lf8rgvJqCEhz/7DZmgcnPD+rPxOZKvJeTN/l8hhCigOgRV5L4uhiOuKdc+f7Jam3Sni8yTIi6EELmkfKwr1WK82OJ1KXFxl2QUfTSgAp15SxaLwk2eiYt8ac2mXZZOQQizdQiqSKw6np2e11KdU+JiQGePSiVtK5F58l+LEELkAne9LQ0flGSn51Ue/re4S3KKPlq60oXZCmURNyoKFLzxfELkb4qS+P9mAdU2uDwqBbZ4XU7zvKKPkZHpwmyFsoiHRz0CJcHSaQghklMSEv/fLICsE6x4LcSXI66BBFmnXh9AUYygj5WWuDBboSzi320+nrgaltEgLXIhLE1RwGggJPg+320+bulsckSLsDI4Jliz0fvftC/QxwLKM/cRF+JphXJg2/ELd/hyxX4GtA/AzckWtWw2IITFGBWF8KhHfLf5OMcv3Hn2G/IZlQLtgyrwr30Y/9iHpXlN4sh0mV6WHp2DFgzZtAe9Q8FabKhQFnFILOQF8Q+GECJvCYgsSvE4J74sdjjd9emT5ogjz8SFmQpld7oQQuSWDkEVCdbF8rtr+o0GJS5poRcp4sI8UsSFECKHlH3oQs1ob37xupxqcZfkFH00aGxRqQtt56jIIiniQgiRQzoEVeSROp7/eVzN8DolLhqV7CMuskCKuBBC5AA3vQ2Nw0uyy+M6sZr4jC/Wx0hXusgSKeJCCJED2oSUx0pRs9nrUobXKYqSuNCLFHGRBVLEhRAim+mMVrwWUo4/XAK5Z5N6O+QUDImLT0l3usgKKeJCCJHNmoWVxtlgzSbvi8+8VolLnCOOtMRFFshQSCGEyEYqBToEVeCyXTjnHEKeeb3sI/5sOkcdKsU6W2IpjjqeMUIhX5GWuBBCZKPaUUUo+diZjd4X013cJbknq7VJS1yYT4q4EEJkow5BFQnVPuQ319uZul6JiwErHSpN9rQ0ReEiRVwIIbJJ6YfO1I7y4VevyxjUxsy9SR8trXCRZfmiiN+4cYOAgADu3JG1zoUQeVeH4Io8VhvY5pnx4i7JJU4vk+fhedWePXto27YttWrVonHjxsyePRujMfEL2vLly2nRogUvvPACbdq04X//+1+u55fnB7bt37+fMWPGEBkZaelUhBAiXS7x1jQJK8VOj2vEaPSZfp+ij0HlWCQHMxNZdfPmTYYNG8bcuXNp3LgxN2/epEePHhQrVgyNRsP333/P999/T8WKFTlw4AAffPABxYoVo3r16rmWY55uiS9YsIDp06fz4YcfWjoVIYTIUOvg8ugUKzZ7Z7y4S3KKIQ4S9NISz6NKlSrF0aNHady4MYqiEB4eTnx8PG5uboSFhfHee+/h5+eHSqWiSZMm+Pr68tdff+Vqjnm6Jd6xY0cGDhxIYGCgpVMRQoh0aY1qWoeU46hzIIE20Zl+n0wvs5z79+9jZWWV4piTkxNOTk4pjjk4OGAwGKhZsybx8fG0atWKl19+mSZNmqS47tq1a1y+fJmqVavmeO7J5emWuLe3NypVJuZoCCGEBTUNK42LwSZTi7skZ5peJvuI57ru3bvTrFmzFP+sWLEizWutrKz466+/2LlzJ9evX2fChAkpzt++fZsBAwbQrl07/P39cyN9kzzdEhdCiDzvv8Vdrto+4G/HYPPeGyct8czQ2lujVmyyJZbR3pp4YM2aNWm2xNOiUqnQ6XSULl2aQYMGMXbsWFMh/+233xgxYgRt27Zl1KhR2ZKjOaSICyHEc6gV5U3pxy5ML300U4u7JKfoo0GtAU32FCiReT4+PlhbZzw3f9++fcybN4+ff/7ZdCwuLg5nZ2cAli1bxqxZsxg3bhydOnXK0XzTk6e704UQIq/rGFSRcM0jDrrdMvu9SbuXyWPDvKlGjRrcuXOHxYsXk5CQwJUrV1iwYAFdu3Zl48aNzJo1i2XLllmsgIO0xIUQIstKPHKiTlRRVhQ9Q3xmF3dJRomLlo1P8jB3d3cWL17MlClTWLhwIe7u7nTr1o3evXvTokUL9Ho9/fr1S/GeAQMGMHDgwFzLMV8U8eLFi3PxonkDRoQQIqd1CKpAnMrANs8rWXq/oo9BbeeezVmJ7FS9enXWrl2b6vjevXstkE1q0p0uhBBZ4BSvo1lYafa63yRKm/nFXZIoRgMYHsmgNvFc8kVLXAgh8prXQ8phrWjYbOa0siSmOeIyvazQiY+P5++//yYwMJCYmBhcXFwoUqQINWrUSDVi/lmkiAshhJm0RjVtQsrzp9NdbtlGZS1IXNIWpNISLyz+/fdflixZwp49e3j06FGq846OjjRu3Jj+/fvj5+eXqZhSxIUQwkyNwkviFm/L9NLHshzjyWpt0hIv6CIiIpg0aRK7du2ifv36fPbZZ1SuXJmiRYtia2tLZGQkwcHBnDp1isOHD/Pmm2/SvHlzxo8fj6ura4axpYgLIYQ5lMQ9w2/YRPCX0/2sh4mLBlSgs8++3AooraMdVmpDtsRKsLfLljjmeOONN2jfvj2HDx/G0TF1z4ubmxtubm74+fnRrVs3wsPDWbFiBZ06dWLfvn0ZxpYiLoQQZqgR7YXvI1e+LXXc7MVdklP00aCzR6WS8cUF3bp16/Dw8Mj09W5ubnz44Yf07NnzmdfKfz1CCGGGDkEVidA8Zp/7jeeKI/uIFx7JC/jw4cM5fvx4pt7n6en5zGukiAshRCYVe+zIi5HF2Op5JUuLuySn6GNkZHohdODAAYzG5/tvJzkp4kIIkUntgyqgVyWw1evyc8VRjAmgj5WWeCHUuHFj1qxZQ2RkZLbEk2fiQgiRCQ4GHS3CyrDf/SYR2rjnCxb/EFBkZHoh9PjxYw4ePMju3btNA9qSU6lU/PLLL5mOJ0VcCCEy4bUQX2yMGjZ5Pf8S0KZ9xKUlXug4OTnRpk2bbIsnRVwIIZ5BY1TTNrg8fzne54bd83eDKv/tI448Ey90pkyZkq3xpIgLIcQzvPygBB7xdsws/We2xHvSEpcinhlaexs02TRP3GBr+b3bDQYD27Zt4+jRo4SEhDB27FhOnjxJlSpVMr1SWxIZ2CaEEBlREvcMv2UTyUmne9kTUh8DGltUamlHFTYPHjygc+fOjB49mvPnz3P48GFiY2PZvXs33bp14++//zYrnhRxIYTIQNUYT8o/dGOz9yWU51jcJTklLhqVtTwPL4ymTJlCTEwMu3btYuPGjSiKAsDs2bOpXr0633zzjVnxpIgLIUQGOgRVJFITx163G9kXVB8jXemF1P79+xk2bBjFihVDpXryrVCn09GvXz/Onz9vVjwp4kIIkY4ijx2oF1GM7Z5XiLNKyJaYiqLIam2FWEJCAtbW1mmeMxgMppZ5ZkkRF0KIdLQPrkCCysivns+3uEsKhkegJMhqbYXUiy++yLx581Is9qJSqYiPj2flypXUqVPHrHgyqkIIIdJgb9DSMrQMB9xuEa57nG1xlf/2EUda4oXSp59+Srdu3WjRogU1a9ZEpVIxa9Ysrl27RlRUFGvWrDErnrTEhRAiDa+G+mJr1LLJ+/kXd0lO9hEv3EqWLMkvv/xCly5diIyMpGTJkoSGhtKkSRM2b96Mr6+vWfGkJS6EEE9RKyraBpfntGMQ1+wisjW2aY64jE4vlO7evYunpyfDhw9PdS4uLo7Tp09Ts2bNTMeTIi6EEE9p8KA4Xnp75pU8me2xlbgYsNKhstJle+yCytrZHk02rdFiZW2fPYGyqFmzZvz4449Ur1491bm///6bd955x6y54lLEhRAiOQU6BvlxxzqK4853sz++PlpGphcy48ePJzg4GEicnTBt2jQcHVP/N3Dt2jVcXV3Nii3PxIUQIplKse74xbpn6+Iuyck+4oVPo0aNiI2NJTY2FoBHjx6ZXif98/jxY/z8/Mxe7EVa4kIIkUzHoIpEW8Wx2/16jsRX9DGoHYvmSGyRNzVr1oxmzZoB0KtXLz7//HOzB7ClR1riQgjxH+84e156UJztnlezbXGX5BRDHCToQUamF1rFixdHp0t7PMS1a9d47733zIonLXEhhPhPu+DyKCr41SsbF3dJ5sn0MnkmXphcuHDBtBLb5s2bqVu3LlFRUamu279/P0eOHDErthRxIYQA7BI0tArx5ZDrLUJ1j3LkHkkLvcgz8cJl6dKlbN261bRW+qhRo1Jdk1Tk27dvb1ZsKeJCCAG0CimLfQ4s7pLck33EpSVemHz22Wd07twZRVHo3bs348ePp1y5cimuUavVODk5Ub58ebNiSxEXQhR6akVFu+AKnHUI5rL9g5y7kT4G1BqybdKzyBccHR0JCAgAYOXKlVSuXBkHh+zpjZEiLoQo9OpFFMNH78CiEqdy9D6KPhqVziHFFpTi2bT2tmizaW0cldY2ewJlUUBAANHR0cyfP58//viD0NBQZs+ezf79+/Hz86Nhw4ZmxZPR6UKIQq/j/YrctY7hqEsOLO6SjBIXIyPTC7k7d+7Qpk0bli9fjoODAzdu3ECv13Px4kXee+89Dh48aFY8aYkLIQq1ijFuVIn1ZEGJkxhV5u3lbC5FH43azj1H7yHytsmTJ+Pp6cny5cuxtramatWqAMyYMQODwcD8+fNp1KhRpuNJS1wIUah1CKpIrJWeXR45s7hLEiUhHgyPZVBbIXf06FEGDBiAvb19qscqXbt25dKlS2bFkyIuhCi0POPsePlBCf7ncZVHVoYcvZdpjrhMLyvUdDodcXFxaZ6LiIhIdyGY9EgRF0IUWm2DE6fzbMmhxV1SkIVeBInrqM+cOZMbN26YjqlUKiIiIli0aBENGjQwK54UcSFEoWSToOHVUF9+d71DiPXDHL+f7CMuAD799FN0Oh2tW7embdu2AIwZM4YWLVoQHR3NJ598YlY8GdgmhCiUWoaWwSFBx0bvf3PlfkpcDKjUoLXLlfuJvMnNzY2NGzeyadMmjh8/jre3Nw4ODrRv355OnTqZPX9cirgQotBRKyraB1fggn0oFx3Cc+Weij4atPaoVNIBai4re3s0BqtsiWXMAwvtWFtb07VrV7p27frcsaSICyEKnboRRSka58iyYmdy7Z6KPgaVzBEXJO5WtnTpUs6cOUNwcDAuLi7UrVuXt99+mxIlSpgVS4q4EKLQ6RBUkSBdLIdd7+TaPWUfcQFw5MgR3n33XTw8PGjQoAFubm6Ehoayd+9etm3bxg8//EClSpUyHU+KuBCiUCkX60r1GC8WFT+V44u7JFGMCaCPlZa4YPr06TRo0IA5c+ag1WpNxx8/fsyAAQOYNGkSq1evznQ8eTgjhChUOgZV5KE6nh0e13LvpvEPAUWmlwmuXLlCjx49UhRwABsbG/r378+5c+fMiidFXAhRaLjrbWn4oCQ7PK7xUBOfa/d9Mr1MWuKFXbly5bhw4UKa5wIDAylZsqRZ8aQ7XQhRaLQJLo9KgS3e5i1t+byUuMSFXpCWeKF0/vx507936dKFKVOmoNfradGiBR4eHkRGRvLbb7+xaNEivvjiC7NiSxEXQhQK1glWvB7iyx8ugQRZx+bqvU0tcZ19rt5X5A2dOnVKsU66oijMmzeP+fPnpzgGMHToUP75559Mx5YiLoQoFJqHlcExwZqN3hdz/d6KPga0dqjU8ie3MFq5cmWOxZb/ooQQBZ5KgQ5BFbhoF8YFh9Bcv78SFy0j05+D2tYetVH77AszE0tt3gYj2SEgICDHYsvANiFEgVcnsijF45zY6HMRVM++PtvJQi8ih0gRF0IUeB2CKhKifcjvLrdz/d6Kovy3WpsMahPZT4q4EKJAK/vQhVrR3mzxukSCOncWd0kh/iEoCTK9TOQIKeJCiAKtQ1BFHqnj2eF51SL3V/QyvUzkHBnYJoQosNz0NjQOL8l2z6vE5OLiLsk9mV4mLXGRKCoqikePHuHt7Y3BYGD58uXcu3ePFi1a8OKLL5oVS1riQogCq3VIeawUNZu9cndxl+SSFnpRWUtLXMCff/5J48aNTdPOJk6cyIwZMzhw4AB9+/Zl+/btZsWTIi6EKJB0RiteC/HlqEsg92xiLJaHoo8GKx0qq9yf2iTynpkzZ1KrVi369u1LZGQkP//8M/369WPv3r306dOH7777zqx4UsSFEAVS07BSuBhs2GSBxV1SkJHpIpnz58/z9ttv4+HhwcGDB0lISKBNmzYANGnShOvXr5sVT56JCyEKHiVxQNsVu3DOOoRYNhV9jHSlPyeVrSMqJXvGNKhU2bNoTFbZ2Nig1+sBOHjwIJ6envj5+QFw7949nJ2dzYonRVwIUeDUjvKh1GNnvirzh2UWd/mPoigocdGoHYtaLgmRp9StW5dZs2Zx8eJFdu7cSY8ePQDYs2cPM2fOpEGDBmbFk+50IUSB0zGoImHaRxxyzf3FXVJI0IMxXqaXCZNx48bh5ubGggULCAgI4P333wdg8uTJlCpVio8//tiseNISF0IUKKUeOVM7qgjLip3BoDZaNBfZR1w8zcPDg++//z7V8Y0bN+Lq6mp2PCniQogCpUNQBR6rDWz3vGLpVJ5ML5OWeKF2/vx5fH19sbGxSbG3+NPu3r0LQJUqVTIdW4q4EKLAcI63pmlYaXZ5XCNao7d0OrLQiwAS9xNfv3491atXT7W3eHKKoqBSqWQ/cSFE4dQ6pBw6xYrN3pZb3CUFfQyoNaCxsXQmwoJWrlyJr6+v6d+zkxRxIUSBoDWqaR1cnmPOgdyxibZ0OkBiS1ylc0i35SUKh+T7iWf33uJSxIUQBUKT8FK4GmzYlFda4fz3TFy60p+b2s4eNQnZEwurbImTV8gUMyFE/vff4i7XbB9w2jHI0tmYJLbEZVCbyDlSxIUQ+V6taG/KPHJJbIXnkZ5rJSEeDI9lepnIUVLEhRD5Xof7FQnXPOKA201Lp2KStI+4tMRFcqdPnyY+Pvu2xZUiLoTI10o8ciIgqihbva4Qb+HFXVIwFXFpiYsnBg8ebPZ2oxmRIi6EyNfaB1dAr0pgWx5Y3CW5J6u1SUtcPKHT6dDpsm9bWhmdLoTIt5zidTQPLc1e9xtEauMsnU4KSlwMqNSgtbN0KiIP6dOnD+PGjePkyZP4+vri7u6e6pqWLVtmOp4UcSFEvvVaSDmsFQ2bLb1neBoUfTRo7VGppMNTPPHll18CsGrVqjTPy4ptQohCQWtU0zakPCec7nHTNsrS6aSi6GPkebhIZe/evdkaT4q4ECJfahheErd4W2aUPmbpVNKkxEWjdipm6TQKBJWtA2q1kj2xjCqw4JOXYsWe/DcRGBhISEgIFSpUQFEU7O3tzY4nRVwIkf/8t7jLTZtITjrdt3Q2qSjGBIh/KC1xkaadO3cyY8YMbt26hVqtZsOGDcydOxd7e3umTJmCVqvNdCx5WCOEyHeqR3tR7pErm7wv5pnFXVKIjwUUmSMuUtm+fTvDhg2jTp06zJw5E6MxcVpkixYt2LNnD/PmzTMrnrTEhRD5TsegikRoHrPPPe8s7pKcaR9xWa1NPGX+/Pm89dZbjBo1ioSEJ+vBd+zYkaioKH744QeGDRuW6Xh5uiUeGhrKwIEDqV27NvXr12fu3LmWTkkIYWHFHjsSEFmUrZ5X0KuzZ1OM7JY0Rxxpied7Z86coWfPnvj7+/Pyyy8zadIkHj16lOKaS5cuUaNGDY4de/b4jJs3b9KoUaM0z1WqVImQkBCz8svTRfzDDz/E1dWVw4cPs3z5cn7++We2bt1q6bSEEBbUPqgCBpWRrV6XLZ1KuhRZra1AiI6O5p133qF58+YcPXqUDRs2cOrUKaZPn2665tGjR3z00Uc8fvw4UzGLFi3KyZMn0zx35swZihQpYlaOebaI37x5k+PHjzN8+HBsbGwoX748vXv3ZvXq1ZZOTQhhIQ4GHS3CyrDf7SYReWxxl+QS54jboVIXrG0vC5vAwED8/f3p06cPGo0GHx8f2rVrx59//mm65osvvqBZs2ZYWWXud92jRw++++47Zs+ezblz51CpVAQFBbFhwwYWLlxI586dzcoxzz4Tv3z5Mi4uLnh4eJiOlS1blosX896iDkKI3PFqSFlsjJrEAW15mBInc8Tzuvv376cqvE5OTjg5OZle+/n5pRhopigKu3fvpmrVqgBs3ryZGzduMGnSJBYvXpyp+7711ltERUWxePFiFixYgKIoDBo0CI1GQ69evejfv79ZnyPPFvHY2FhsbW1THLO1tc10l4UQomCxMqpoG1yBU473uWEXael0MqaPRmXnaeksRAa6d+9OaGhoimODBw9myJAhaV5vMBj4/PPPuX37NtOnT+fq1at8++23rFmzBo3GvFI6ePBgevfuzalTp4iMjMTR0ZHq1avj5uZm9ufIs0Xczs4uVcF+9OgRDg7y7VaIwujlByXxjLdjdqk/n32xBSmKgqKPQe1axtKpFBgqa3tU2fRkQpUAxMGaNWvSbImnJSwsjI8++ojw8HBWr16Ni4sL77zzDqNGjUqxeEtmdO7cmYYNG9KoUSMaNmyY1Y9hkmefiZcvX54HDx4QHh5uOnb16lXKlStnwayEEBahQMegCty2ieKE8z1LZ5Ox+IegGGWOeB7n4+ND8eLFU/yTVhG/ePEiHTt2xNnZmXXr1lGsWDHOnj3LjRs3GDt2LP7+/vj7+5OQkMDAgQNZtGhRhvetWrUqv/76K2+++SYvvfQSI0eOZNu2bURGZq13Kc+2xEuXLk2tWrX46quvGD9+PHfv3mXlypUMGjTI0qkJIXJZlRhPKjx0Z3bJP1Hy4uIuySSNTEeeied7oaGh9OvXj9atWzNq1CjTcX9/f86cOZPi2sqVK7Nw4ULq1q2bYczx48cDiYPmfv/9dw4fPsyECROIiYmhevXqNG7cmAEDBmQ6xzxbxAFmz57NhAkTaNKkCVqtlu7du/Pmm29aOi0hRC7rGFSBKKs49rrfsHQqzyT7iBcc69atIzQ0lB9//JH169ebjhctWpRt27Y9V+xixYrRpUsXWrVqxbFjx1i2bBmnTp3i9OnTBaeIe3l5yQIvQhRyRR47UC+iOD/6XCDOKm8u7pKcabU2aYnne4MHD2bw4MGZuvbChQuZui4wMJATJ05w8uRJTp48ybVr17CxsaFmzZoMGTKEgIAAs3LM00VcCJG/rTndDjeDbarj4ZpHdK+5JVMx2gWXJ0Fl5Nc8vLhLcoo+GqysUVnpLJ2KyIOaNWuGSqWiUqVKdOrUiRdeeIGqVaua98PxRQAAIABJREFUPcI9iRRxIUSOSauAZ3T8afYGLS1Dy3LQ7RbhunwyvVT2ERcZ6NWrF3/++Sf//vsvDx484MKFC9SpUwd/f398fX3NjidFXAiRZ70SWhY7ozbPL+6SnKKPQWWd9lQlIcaMGQNAVFQUx48f5/jx4/z4449MnDgRJycn/P39mT17dqbjSREXQuSINafbPdf71Uri4i5nHIK5aheRTVnlLEVRUOKiUTsWtXQqIo9zcnKiefPmlC9fHl9fX/bv38+BAwfYt2+fWXGkiAshckRmu8zTU/9Bcbz19iwomfZmEXlSgh6M8bJ7WTZTdLYYNdmzrIliMAKWezRz9+5djh49yrFjxzh27BhBQUF4eXnRoEEDZs6cSf369c2KJ0VcCJHtnrcVDol7hgdaR3PM+W42ZJQ7nkwvk2fiIm1NmzZFo9Hwwgsv0KNHDxo2bEjFihWzHC/LRXz37t3cu3ePBg0aULZsWdPxVatW0bNnzywnJITI39IbkZ5cuOZRhucrxbhTKdaDeSVP5PnFXZJ7Mr1MWuIibXPnzqVevXrY29tnS7ws9U9Mnz6dlStXcuvWLfr168fy5ctN537++edsSUwIkT9lphv9WdPLOgRVJNpKz+58sLhLcqaWuIxOF+lo3rw5Dx8+ZNq0abzxxhu88sordO3ala+//pqgoCCz42WpJX7w4EE2bdqERqPh/fff54MPPiAoKIiRI0eiKEpWQgohBADecfbUf1Ccn30u8tjKYOl0zKOPAbUGNDaWzkTkUTdv3qR79+48fvyYevXq4e7uTmhoKGvXrmXjxo2sW7eOUqVKZTpeloq40Wg0TUx3dXXl+++/Z8SIEYwePRqj0ZiVkEKIAiAzz8Kf1ZXeNrg8igp+8bqUXWnlGiUuGpXOAZUqHz0DELlq2rRpuLu7s2LFClxdXU3Hw8PD6d+/P9OnT2fOnDmZjpel7nRPT0/Onz9veq3T6Zg5cyYqlYrLl/PHqkpCiOz3vF3pdgkaXgkty2+utwnVZVzs8yJFHy0bn4gMHT16lMGDB6co4ABubm4MHDiQY8eOmRUv0y3x8PBw04blU6dOTbUPq1qtZvLkyXTq1MmsBIQQhcezWuEtQ8tin6DLV4u7JKfoY1DbeVo6DZGH2draolan3X5Wq9UYDOY9Qsp0EW/YsCGvvPIKPXr0oFatWule98ILL5iVgBCi8MioFa5WVLQLqsA5hxAu2YfnYlbZQ0mIB8Nj2b0sByhaGxSN1bMvzEwsVQKWnCfu7+/P/PnzqVOnDs7OzqbjERERzJ8/P+c2QFm0aBHr16+nV69e+Pr60rNnT9q0aYONjQzgEEI8v3oRxSiid/g/e/cdHVW5/Q38Oy2990oKKSgIQSnSO4koXRERBAERFRWxXLmX+xN9RRQRARGVSxEUpEUQFemgKIJ0pBNqEtImfVKmnvePyEjMJJmTTDIzyfezlsvMmTPPbM6azM4553n2xvLwU9YOpU7u9BHnzHSqyRtvvIFHH30Uffv2RefOneHn5welUokjR45ALpdj/vz5osYzO4l37doVXbt2RW5uLpKTk7Fs2TLMmzcPw4YNw5gxYxAVFSX6H0NEzUdtl9JHZMUjw0GF373SjdvUp9YAOhOvkzvDMeEpS4dYP8blZTwTp+qFhoZi69atWLVqFY4dO4aUlBR4enrisccew4QJExAUFCRqPNGz0319fTFlyhRMmTIFv/32GzZs2IDBgwejU6dOWLlypdjhiKiZqOlSepzKB61V/vgs/AQMkruWqZpK4DVtrwNL/aFgPBNntTaqRWBgIN58802LjGV2Ev+///s/FBYWori4GEVFRZX+r9Pp8Pvvv1skICKyXwYIkKLq8ioD/k7MppLmUMyCCiXY5XetwWOswkJ/KAjqYkAiBRQuFgiKmpJVq1aZva9EIsGECRPM3t/sJL5x40a4urpi1KhRaNu2LVxdXeHm5lbp/0TUvC0LP4mpqffj2dY/4fKlTyonwmOmX+MPf/RGbyQjGWUy+60zIWhUgMIVEollGnVQ0/HBBx+YvW+DJfHk5GR8/fXX+Oabb3Dx4kWMHTsW7du3Z1EDIqogAInKaFx0zcVN50Kzz2SHYzgA4Ft8C2BYAwbYsCr6iPN+OFV18eLFBhvb7D8ZW7dujblz52L//v3o0qUL5syZg379+mHZsmXIz89vsACJyD7ElfogqsxL1CVxJzhhMAbjIA4iC+LrRtuSO9XaiP7p1q1bdXrdzZs3a91H9HUfb29vTJkyBXv27MG///1vHDp0CH369MHMmTPrFCQRNQ2JymiUS3XYlfYB1Me+MOs1SUiCG9ywERurPCdoaziTl9evV/kd+ryrFhlHMOgBbSmTOJn0zDPP4N133zW7wcmtW7cwe/ZsTJkypdZ9zb6cPnPmTJSUlKCkpASlpaXG/0pKSqDRaLB161bMnTvX3OGIqAlx1MvQO68FDnqnoiQ316zXSCHFSIzEOZzDBVyo8rwu9RAgkUJx70hInX2gu/Ub9MqLcEiYAIm0/oU/DEXp0F3fh4pzGRP34sX8oaAtASDwcnoDERTOEBR17pxdeSyJDkCBRcYy19atW/HRRx+hX79+SEhIwIABA9C6dWsEBQXB2dkZxcXFyMzMxKlTp/Dbb7/h5MmTGDlypFldQc0+KoWFhfDw8EBgYCDc3d3h4eEBNzc3eHh4wN3dHe7u/PASNVfdC8LhqneouJRuXg5HF3RBGMKwAisqNtyVNPX5N2DIS4EspAOkzhXlniXuIUD2WQglOZC4i1tL+0+GkhxoU3ZC4uQFRfwQSOSOAADNha2AoIfDveLKR//dR5xn4lSVs7MzZs2ahfHjx2P16tVYtmwZcnNzK80pEwQBAQEB6N+/P9577z2Eh4ebNbbZSXzp0qXiIyeiZiExJxrpjsX40y3HvBfInfGY4llklmXiUIwLHL2eNT4l6NTQ3ToIibMvZEEJxu1St4rEbSi+DWk9krihvBDaK9sBuSMUsYOMCRwApF4R0Kf/AUFTAomD+StuBBZ6ITOEh4dj1qxZmDVrFlJSUpCeno7i4mJ4eXkhODgYLVu2FD2mZa5PEFGzFVzuhraqAKwKPQ0TS8RNitWFoZ0uHEuxFPqyXMi9/j7r0KX+DmjLII9JqnTZXKJwhsTZBwZVRp1jFTQl0F7+EQCgiHu4SqK+k8QNhTch879XxLgVZ+LsYEbmiomJQUxMTL3H4YJGIqqXgcoo6GHAHt8bZr/mUTyKUqkW2+UHIJT9ff3dUJgKQ+4lyILaQepatRuYxD0YgiqzYiKZSIJOXXEGriuDIvYhSJ28qo7v5A04uMNQUPus4Epja4oBhYtF7tUTicEkTkR1JhUkGJAbhWOeGci90/+7lglhvvBFX/TFTr9rKHN1hlBW0bFM0GugvfkLJE5ekIU8YPr93EMAgw5CqZmX7f8iGHTQpuyEUF4ARctESF0DTO4nkUgg84qAoSi9oiuZueOrVbwfTlbBJE5EdfZAYRD8tC7Y5XfduM0x4Sk4dngWjh2eNfmaYRgGKaT4LvAyJM4+EMoLIBj00KUdATQqyCN7QSI1fadP6hYMADAUm39JXRAM0F3bC0GVAXlUH0g9w2rcX+oZAQh6GIrTa9yvEk0x74eTVTCJE1GdJSqjUSAvxxHP22bt7whHDMZg/IpfcfPCZ9BnngIEAzQnlsOQcx4AoE3ZVe3rJQpnSJy8YSg27/0EQYDu5kEYCm5A3qIbZD6134OUuAcDUoXZl9QFQfirWhvPxKnxcWIbEdWJp9YRDxaGYmvAZeik1dQ8lztXKr86EAPhCU9skm2rc+MRiXswDLmXIRj0td6D1qcfhUF5EbLg+yELaFPjvsbxpTJIPcNgKLwFQRBqLy2tLQUEA8/ESZTbt2/jyJEjyMnJwfDhw5GRkYH4+Hg4OjrW/uK7MIkTUZ30y42EXJDWWGb17laeEgEYdXYQLstyceWeDsDx43V6X6l7CAw55yGUKiFxC6x2P13Wn9BnnoTU7x7IQjqIew/PCBjyr1e8h4kJdnfjzPSGJ8idICgUlhlLMH+uQ0MwGAyYM2cO1q9fD71eD4lEgm7dumHhwoVIT0/HmjVrEBhY/ef6n3g5nYjE+6vZyQVXJW45F5n1ko6FwQhTe+DbwMtmL0UzReoeAqDm++L63CvQpx6C1CsK8ojuohs1ST1bVLxHwY1a9zWuEWe1NjLDkiVL8O2332Lu3Lk4dOgQBKGiTe8bb7wBvV6Pjz76SNR4TOJEJFqrEl9ElHtip4hmJ8Oz4pGjKMVB77o1g7ij4r64F4Rq7osbClOhu3EAEvcQyKP71qk1qEThDIlrIAyFtcfKam0kRnJyMmbMmIEhQ4bA09PTuL1Vq1Z4+eWX8dtvv4kaj0mciEQb+Fezk198zEvIUaVeaF8chG0Bl6GXCvV+f4l7CAyqTAhC5XvxBlUWtFd3QeLkDUXLgdXOcjeH1CsCQqkSgqakxv0ETTEgc4RE5lDn96Lmo6CgAFFRUSaf8/HxgUqlEjUekzgRieKol6FXXgv84n0LpTKdWa8ZnhWHcqkOP/nf1TWsuvXkZjQekboHAwYthFKlcZuhLB/alB2AwgWKuMrlVOtC6hVRMW5hLbPU2UecRIiPj8eWLVtMPrd7927ExcWJGo8T24hIlJ75LeBqUJh9Kd1b64TeeRHY4XcVKvnfk4runvQm1t/3xW9D6hoAQaOqqMYGCRxiB0GicKnz2HfcXb2tphKsgroYEifPap8nutvLL7+MKVOmICsrC7169YJEIsHevXuxatUqbN++XXSfEp6JE5EoA5VRSHMswjk3Ze07A3gkOwZyQYqtgZctFoPm3GYAgD7tCNTHvoDmzFpAo6pY6mWhhGpO9TbjGnHeDyczdevWDcuXL4dWq8XHH38MQRCwdOlSpKSkYMmSJejdu7eo8USfiRcWFmLDhg24du0aXn/9dRw9ehSxsbF16r5CRPYltNwd96kCsCL0lFkzzB0MMjycE4MjXum47STuXl+NqltLrldb7j3wV0OU7LMwFKdD5hVp+v0MWoBrxEmELl26oEuXLigvL0dhYSHc3Nzg6mp+17y7iToTv379Oh566CEkJyfjhx9+QGlpKXbt2oVHH30UJ06cqFMARGQ/7jQ72Wtms5O+uRHw0jlhS4DlzsIbk8St5uptd9aIs1pbw9JJZNBK5Bb5TyexbpOavn37YsGCBbh06RKcnJwQGBhY5wQOiEzic+fORWJiInbu3AnFXwvv58+fj6SkJNFr24jIvtxpdnLUMwN5DuW1v0CoWFaW4pyPM+7ZDR9gA6io3hZurN72T4KafcRJnMTERPzwww8YNmwYHn74YXz++ee4davuyy5FJfHTp09j7NixlQeQSjFlyhScP3++zkEQke3rWBgMH60zdpg5oe2BoiBElHtiS9ClehV3sTapZwSgLa00E/4O45k474mTmf71r39h3759+Oabb9C9e3ds2LABiYmJGDVqFNasWYOcHHEd+kRPbFOrq95zys3NhYMD10gSNWWJymjkyctw1MxmJ8Oz4pGrKMPP9SzuYm1Sz3AAEtPV2zTFgFQOyJ0aOyyycwkJCZg5cyb279+Pb775BvHx8fjggw/Qp08fUeOISuJ9+/bFwoULUVLyd/GD1NRUvPfee6Jn1BGR/fDWOqFzQQj2+t4wq1hLRJkHOhQF43v/K9U3R6mPeqwxF0uicIbEzXT1tjt9xMWWdSUCgJycHKxduxYLFizAli1b4O3tjSeeeELUGKJmp8+cORPPPPMMOnfuDJ1Oh8ceewyFhYVo164d/vWvf4l6YyKyH/1yIyGD1Oy14cOy4lEu1WG7f0qDxFOfNeZ1IfWMgD79SJXlZIKmmDPTSZTs7Gzs2rULO3bswIkTJ+Dq6or+/ftj6tSpePDBByGVirtALiqJe3h4YMOGDfj9999x/vx5KBQKxMbGokuXLqLelIjsiFAxK/2cWw7SnItr3d1T64h+uZHY7XcdRQpNIwTY8KReFUncUHALsoC/C78IGhWktXQ5I7pbr1694ODggF69emHRokXGx3Ulep14WloaDh8+jEuXLkEqlaKoqAgxMTHw9+cHmagpuqfEFy3KPbEg8ohZ+z+cEwMHQYYtAZcaOLLGI3HyAhw9YCi8aUzigl4L6Mo5M51Eee+99zBgwAC4uVlmMqSoJH7s2DFMmjQJ/v7+aNOmDQwGA7799lusXr0aX331FVq1amWRoIjIdiQpW6JMqsUv3qm17qswSDE4OxZ/eN4266zdXkgkEsg8W0CfcwGCXguJTMGZ6WS2c+fOoWXLlnByckJcXBxu3qy5Hn/r1q3NHltUEp87dy6GDh2K2bNnG6/b6/V6/Pe//8WcOXPw1VdfiRmOiGyck16Onnnh+MUnFeVmNDvpnRcBb50TtgQ2nbPwO4zV24rSIfOOrJiZDq4RbwxqgwCDvv7d7wBAa7DMOGKMHDkSGzduRNu2bTFy5MhqJ0IKggCJRIILFy6YPbaoJJ6SkoL58+dXuvEuk8kwefJkjBgxQsxQRGQHeuaHw9mgwA6/q7XvLAAjsuJx3bkAJ92zGj64RiZxCwZkDhWX1L0jWa2NzLZmzRpjafI1a9ZYdGxRSTwmJgbHjx+v0gv18uXLiIyMtGRcRGQDEnOiccupEBdcc2vdN6E4EFFlXvgo8ohdF3epjkQqg9QjzFi9TVAXAxIpYIGOadS0derUyfjz7du30atXL3h7e1fZLycnB9u2bau0f21EJfFRo0bh/fffx7Vr19CxY0fI5XKcO3cOK1euxKhRo/D9998b9x08eLCYoYnIxoSVuaN1iT+Wh5nX7GREVjzy5eU44FNL/207JvWKgCH/GoTSnIozcYUrJBI2gyTzzZw5Exs2bDCZxM+cOYOFCxdi0qRJZo8nKom/9dZbAICVK1di5cqVlZ5bvny58WeJRMIkTmTnEpXR0EkM2ON7vdZ9w8rc0akwBGtC/oS2IYq72AipZwtUVG+7WbFm3JH3w6l248ePx59//gmg4r73+PHjTd4XLy8vFzWpDRCZxC9evChqcCKyTzKDBP1zo3DE8zYKFLW39xyWHQeNRI8fG6i4i62QyJ0qqrcV3ISgK4fUI9TaIZEd+O9//4sdO3ZAEAR8+umnePjhhxEUFFRpH6lUCg8PDwwaNEjU2KLXiRNR09exMATeOifsMqNCm7vOAf1zo7DP9wYKzUj49u5O9TaAy8vIPDExMZg2bRqAiivVjz32GAIDA43PGwwG0ZXa7hCdxG/fvo21a9fiypUrkMvliImJwahRoxAWFlanAIjI9iQpo5GrKMNRz4xa9x2UEwMng7xJLiv7J/WpNYCuzPhYn3EC+owTgNy50UvBkn2aNm0a/ve//+HkyZNYunQpgIoaLK+//jqmTJmCJ598UtR4olL/hQsXMHjwYGzfvh3Ozs6QyWTYtm0bhg4dikuXmv4vMFFz4KNxQsfCYOzxvQ6DpOY1tXKDFEOyY3HcIwM3nYsaKUIruiuBm7Wd6B+++OILLFq0yLjkDAAiIiLwyCOPYN68eVi3bp2o8USdiX/wwQfo2bMn5s2bB4VCAQDQarV48803MW/ePKxYsULUmxOR7bnT7GSXX+0T2nrlt4Cv1tnskqxEdaHWCdDX8geluXS6xi/2creNGzfi9ddfx/jx443bAgMD8frrr8PPzw+rV6/GmDFjzB5P1Jn4qVOn8PzzzxsTOAAoFAo8++yzOHnypJihiMgWCRWz0v90y0a6Uy1lUwVgeFYcbjoV4rhHZuPER2TncnNzERMTY/K5+Ph4ZGTUfgvrbqKSuIeHR6Ve4neoVCrI5ZwjR2TvWqv8Eab2MKvlaNviAMSU+lTcC2+CxV2IGkLLli2xfft2k8/t2LGjSjG12ojKvL1798Y777yDhQsXokWLFgCAGzduYM6cOejVq5eoNyYi25OojEKJVIuDZjQ7GZ4dhwJ5Ofb5Nt3iLkSW9uyzz+Kll15CRkYGevfuDV9fX+Tl5eHAgQM4dOgQFi5cKGo8UUl8xowZePrpp5GYmGisNpOfn4927dph5syZot6YiGyLi16OnvktsM/nJtQyfY37hpS7oXNBKL4JPgeNtOZ9mxS5s+lJbHLnxo+F7NLAgQOxaNEifP7553jvvfeM2+Pi4rBw4UIkJiaKGk9UEvfy8kJycjIOHjyIK1euwNHRETExMejSpYuoNyUi29MzrwWcDHLsNKPZybCsOOgkBvwQ0LSLu/wTl5GRJSQmJiIxMRFqtRoFBQVwc3ODq6trncaqNYnfXQ/9bncWqiuVSuM+LLVKZL8SldG46VSIS655Ne7nplNgYG40fva5hXxFeSNFR9S0HDt2DIcPH0ZOTg6effZZHDt2DPfccw8CAgJEjVNrEn/99dfNGoj10onsV4syD9xT4odlYSdrnaSWlNMSTgY5vm0GxV2ILK28vBzTp0/HgQMH4ObmhpKSEowaNQpffvklLl26hK+++qrSGvLa1JrEWS+dqOm70+xkr++NGveTGSQYmh2Hk+5ZuO5S0DjBUbOnMQjQ6y3TWEdvsO468fnz5+PMmTNYu3Yt2rVrhzZt2gAAPvzwQzzzzDP46KOPjJXczCG6WOt3332HzMyKNaErVqzAI488gv/85z8oLS0VOxQR2QC5QYp+uZE47Jlea+3zHvnh8Ne6YEsg/7gnqosff/wRr732Gh544IFKncz8/Pzw/PPP4/jx46LGE5XEP/nkE8yePRuZmZk4duwY5s+fj44dO+LkyZP48MMPRb0xEdmGzoUh8NI51b42XKjoGZ7qVGRWTXUiqqqsrAy+vr4mn3N0dIRGoxE1nqgkvmXLFnz44YdISEjATz/9hPbt2+Ott97CnDlzsHv3blFvTES2YaAyCjmKUhz3rLnqWmuVH+JKfbE14DIEFnchqpN27dphzZo10Ol0xm13zsiTk5Nx3333iRpP1BKznJwc4/X7X3/9FcOGDQMA+Pv7Q6VSiXpjIrI+X40zOhQGY2PQhVqbnQzPikeRTI09vrXXVCci015//XWMGzcODz30ELp16waJRIJ169bh2rVrOHv2LFavXi1qPFFn4uHh4Th79izOnz+PmzdvomfPngCA/fv3Izw8XNQbE5H19c+NMqvZSZDaFV0LwrDd/2qthWCIqHpt2rTBpk2bcN9992HPnj2QyWTYs2cPvL298c0336B9+/aixhN1Jj558mS88sorkEql6NixI1q3bo2lS5fi008/rVR5hojsgFBRZvWMWzYynGq+kjY0Kw4GiYDvA640UnBETVdMTAwWLFhgkbFEJfERI0agdevWSE1NRY8ePQAACQkJ+PLLL9GxY0eLBEREjeM+lT9C1O5YG3Kuxv1cdAokKqPxs/ct5DqwbzaRWLt27cKDDz4IDw8P7Nq1q9b9nZ2dER4ejsjIyFr3Fd16LD4+HvHx8cbHXbt2FTsEEdmARGU0SmQa/OpVc7OTJGU0XAyKim5lRCTaSy+9hI0bN6Jt27Z46aWXzHqNRCLBpEmT8Nprr9W4H/uHEjVDLjoFuueHY6/vjRrvcUuFiuIuZ9yykeKa34gREv2tXCtAKlimSItB1/jFXvbu3Qt/f3/jz7VRq9VYs2YNNm3axCRORFX1zr/T7KTmteHd8sMQqHHFZ+EnGikyoqYnNDTU5M+FhYXIzs6Gl5eXMcnf8dBDD+HatVpqN4BJnKhZSsyJxnXnAlx2qbnZyYiseNx2LMYfXrcbKTKipi85ORlffPEFUlP/vpUVERGB6dOnIykpCQDQuXNndO7cudaxmMSJmpnIUk/El/ri8/ATNTY7aaXyxT0lflgafrzWNeREZJ7Vq1dj7ty5eOihhzBt2jT4+PhAqVRi586deOWVV2AwGDBo0CCzxxNdO72x3bhxA506dUJaWpq1QyFqEgYqo6GV6LHP50aN+w3PiodKpql1DTlRc2AqFx07dgyjR49Ghw4d0L9/f2zatKnWcdasWYOJEyfi448/xpAhQ9C9e3cMGzYMn332GZ544gksWrRIVFw2ncT379+PMWPGoLCw0NqhEDUJCoMU/fIi8btXOooU1ddoDlC7oHt+GH7yu4pyma7a/YiaA1O56NKlS5g0aRLGjh2Lo0eP4v3338f/+3//D6dPn65xLKVSiW7dupl8bsCAAcYGY+ay2ST+2WefYf78+XjllVesHQpRk9G5IASeOsdaJ7QNyY4DAHwXyOIu1LxVl4u++eYbJCYm4pFHHoFEIkGHDh2QnJyMiIiIGsfr3r07vv/+e5PP/fLLL6JrrljtnrharUZOTo7J5/z9/TFixAhMnToV6enpjRwZUdOVqIxGtkMJTnpkVbuPs16Oh5TROOiTCqUDWwxT05SZmQmZTFZpm4eHBzw8PCptqy4XnT17Fl27dsVLL72EI0eOICAgAC+88AJiY2OrvNeqVauMP0dHR2PVqlVIT0/HgAED4Ofnh8LCQhw8eBC///47pk+fLurfYbUkfurUKTz11FMmn1uzZo1Zs/KIyHx+Ghc8UBSMb4LP1ThRbaAyGq56B3wbwOIu1HSNGTMGSqWy0rZp06bhxRdfrLQtMDDQ5OsLCwuxdu1aLF68GAsWLMDBgwfx0ksvITAwsEr98w8++KDK648ePYqjR49W2f7+++9j/PjxZv87rJbEO3fujEuX+CVB1FgGKCMhhQS7a5ioJhUkGJYVh3NuObjsVvPyM6LGUq7TA4KF7v7q9ZACWLdunckzcXM5Ojpi4MCBxvvbffr0QY8ePbBr164qSfzixYv1Drs6XGJG1AxIhIoz7JPuWch0LKl2vy4FoQjWuGF5+KlGjI6o8QUFBcHR0bHOr2/ZsiXUanWlbXq9+R3+CgoKcPr0aahUKnh7e+O+++6Du7u76DiYxImagbbFAQjWuGF16J817jc8Kx6ZDir87sW5KEQ1efzxxzFlyhT89NNPSEpKwoEDB3Do0CGYm88eAAAgAElEQVSzaqMvWrQIK1asgEbz9woRuVyOiRMnYsaMGaLiYBInagYSldFQyTQ45F19vYW4Eh+0Ufnj8/ATLO5CVIuuXbti8eLF+OSTT/Cf//wHAQEBmDdvHlq3bl3j61avXo1ly5Zh0qRJGDRoEPz8/KBUKvHjjz9ixYoV8Pf3x7hx48yOw+aTeFhYGO+dE9WDq06Bbvlh2OV3HRpp9Zf7hmfFo0Sqxa5alp8RNUemclHfvn3Rt29fUeOsW7cOkydPrrRkzc/PD61atYJMJsO6detEJXGbXSdORJbRJy8CjkLNzU78NM7okR+Onf5XUcriLkQNJjMzs9rVV506dRK9rJpJnKiJG6iMxlXnfKS4VN9KdHB2LCQC8F0Ai7sQNaQWLVrg2LFjJp87evRotUvaqmPzl9OJqO6iSr0QV+qDpeHHq2124qSXY1BODA55pyGrhpnrRFR/48aNw9tvvw29Xo+kpCT4+voiNzcXO3bswIoVK+yn2AsRNbwkZTQ0Ej32+96sdp8BuZFw1zvg20DOPSFqaKNGjUJqaipWrlyJZcuWGbfLZDKMGzcOkydPFjUekzhRE6UwSNE3NwKHvNJQLDfd7EQiAMOy4nHBVYkLrrmNHCGReVRaPQR9DX1zRZAY9DC/pEvDePXVVzFp0iScPn0ahYWF8PT0RNu2beHt7S16LCZxoiaqS0Eo3PU1NzvpXBiCULU7VoeeqbG3OBFZlpeXF3r16lXvcTixjaiJSlRGI8uhBKdqaHYyPCseWQ4l+LWG9eNEZLuYxImaoAC1C9oXBWG373UI1Zxhtyz1QrviQGwLuMziLkR2ikmcqAkakBsFADUWbhmeFY8yqRY7WNyFyG4xiRM1MXeanZxyz0K2o+l+4D4aJ/TKa4GdftdQItc2coREZClM4kRNTEJxIAI1rtjpX/0Z9uDsWMgEKYu7EDUyQRCwceNGHDhwAABw9epVPPzww2jfvj3efPNNlJWViRqPSZyoiUlURqNYpsYhL9OT1Rz1MjycE4PfvdKR4aRq5OiImrfPPvsMs2fPxvXr1wFULDcrKyvDpEmT8Ouvv+Kjjz4SNR6XmBE1IW46B3TND8NP/lehlRpM7tMvLxIeekcWdyG7UarRQy+1zBpImZXXiW/ZsgXTp0/H008/jStXruDixYuYM2cORo4cibCwMMyfPx+zZs0yezyeiRM1IX3yIuAgyKpdGy4RKia0XXbJwzm3nEaOjoiysrJw//33AwD2798PqVSKPn36AACCg4OhUom7OsYkTtSEJOZE44pLHq65FJh8vkNhMMLLPbAl8BKLuxBZQVBQEFJSUgAAO3fuRJs2beDj4wMAOHToEEJDQ0WNxyRO1ES0LPVCTJl3rcvKchSlOOid2oiREdEdo0aNwnvvvYdBgwbh3LlzGDNmDADgpZdewrJly4yPzcV74kRNRGLOX81OfEw3O4kq9cT9xUFYGXoaumrulxNRw5o8eTJCQkJw7NgxPPfccxg8eDAAwMPDA++//z6GDh0qajwmcaImwMEgQ5+8SPzqnQpVNeu+h2XFo1yqw3b/q40cHRHdbdCgQRg0aFClbe+++26dxuLldKImoGt+KNz1Dtjld93k815aR/TJi8Bu3+tQVdPRjIgaHteJE1EVicpoZDiocNrddLOTR7Jj4SDIsDXwciNHRkR3s/Q6cSZxIjsXqHZF++Ig7PYz3exEYZDikZwYHPZMR7pTceMHSERGptaJv/DCC5g2bRpee+017NixQ9R4vCdOZOcGKKNggIDdvqYvpffNjYSXzonFXchulWj00FloSaRc0FtmoDriOnEiMpIKEgzMjcIJj0zkmGp2IgDDs+Jw1TkfZ9yzGz9AIqqE68SJyKh9USACNK7VVmi7vygIkeVeFWfhLO5CZHVcJ05ERgOV0SiUq3HYK93k8yOy4pGnKMPPPrcaOTIiMoXrxIkIAOCuc0CXglD86J9istlJizIPdCgKxuqQMyzuQmRDLLlOnEmcyE71zY2EgyCrtszqsKx4qCU6/Oif0siREVFNrly5gk8++QRHjx6FSqWCl5cXHnjgAUydOhWtWrUSNRaTOJE9EoAkZTQuu+Tiukthlac9tY7onxuJPb7XUaRgcRciW3H27FmMHTsWPj4+GDJkCHx9faFUKrFnzx48/vjjWLt2Ldq0aWP2eEziRHYottQbUWVeWNziqMnnH86JgYMgwxYWdyGyKR9++CHatWuH5cuXQ6FQGLe/9tpreOaZZ/Dxxx9jxYoVZo/H2elEdihRGQ21RGdywprCIMUj2TH4w+M2Up2LrBAdEVXn9OnTePrppyslcABwcHDAhAkTcPLkSVHj8UycyM446mXokxeBX73TUGKi2UnvvAj46JyxJfCwFaIjsrwSrR6WuinkAL1VM5+np2e1BV1UKhXkcnHB8UycyM50KwiDq97B9Nrwv4q7XHcuwEkP03XUich6evTogYULF+Latcq/v9euXcOiRYvQo0cPUePxTJzIzgxURuO2owp/mqjA1q44ANFl3lgQeYTFXYhs0KuvvorRo0dj8ODBiImJgZ+fH5RKJVJSUhAcHIw33nhD1HhM4kR2JLjcDQnFgfgy5IzJZicjsuKRLy/Hfp+bjR8cEdXK29sbW7ZsQXJyMo4dO4aioiJERUXh0UcfxYgRI+Dq6ipqPCZxIjsyMDcKehiwx0Tf8LByd3QuDMVXIX+aLP5CRLbBxcUF48aNw7hx4+o9FpM4kZ2QChL0V0bhuEcmlA5lVZ4fmhUHjUSPH1jchcimiK3GNmvWLLP3ZRInshP3FwXBX+uCz1ucqPKcu84BA3KjsN/3JgoVaitER0TV2bdvn9n7SiQSJnGipihRGY0CeTmOeN6u8tygnJZwMsjZM5zIBolJ4mIxiRPZAU+tIx4sCMH3AVeqNDORG6QYkh2H4x6ZuOlctQQrkb1TqXUoFywzz8NJYmhSmY/rxInsQN/cCCgEmcm14T3zw+GrdcYWnoUTNTtM4kS2Tqi4lH7BVYmb/yyjKgDDs+Jx06kQxz0yrBMfEVkNkziRjYsr8UFkuRd2mVhWdp/KH7GlPtgaeNnkunEiatqYxIlsXJIyGuVSHX42UcBleFY8CuVq7PW90fiBEZHVNaHb+0RNj6Nehl55ETjonYpSma7Sc8HlbniwIBTrg89DI9VbKUIiMseBAwewdu1a3L59G2FhYXj88cfRt2/feo/LM3EiG9Y9PxyuBoXJCW3DsuOglxjwg/8VK0RGRObavn07pk6ditOnT8PV1RVnz57FCy+8gOXLl9d7bCZxIhuWqIxGumMxzrrlVNruqlNgoDIKB3xuIc+h3ErREZE5vvzyS/To0QM///wzNm7ciJ9//hlDhw7F//73v3qPzSROZKNCyt3QVhWAXX7XqnQke0jZEs4GBZeVEdmBlJQUjB8/Hs7OzgAAuVyO5557DoWFhUhNTa3X2LwnTmSjBiqjoYcBu30rz0qXGSQYmhWHU+5ZuOZSYKXoiBpPcZkWpQbLnHNqpQZAXKOweisrK4O7u3ulbcHBwQAAlUpVr7F5Jk5kg6SCBANyo3DUM6PK5fLuBeHw17rwLJzITgiCAImk8uU0mUwGADAY6leJjkmcyAZ1KAyGr9a54lL63QRgRGY80hyL8IeJGupE1LzwcjqRDUpURiHfRLOTe1V+iC/1xZIWx1jchciOrFy5En5+fsbHgiAAAFasWAEfH59K+7KLGZEd89I6onNhKLYGXIJeKlR6bnh2PIpl6ir3yYnIdoWEhODMmTMmt586darSNrYiJbJz/XIjIRek2PmPMquBald0zQ/F5qCLUMtY3IXIXjRkK1LeEyeyJULFrPTzrkqk/qPZydCsOAgSYFsAi7sQNTUajQbbtm3DU089Jep1PBMnsiH3lPgiotwTH0f8UWm7i06BRGU0fvG+hVyHMitFR0SWdvHiRWzatAk//PADCgsL4enpKer1TOJENmSgMhplUi1+8blVaXuSMhquBgW+5bIyIrunUqnwww8/YNOmTTh//jwcHBzQu3dvDB48GL169RI1FpM4kY1w0svRK68FfvFORdldzU6kggRDs+Pwp1s2UlzzrRghkXUUleug0llmOYZOLtS+UwM5fvw4Nm/ejB07dkCtViMhIQEA8Pnnn6NLly51GpNJnMhG9MgPh4uJZidd80MRqHHF5+EnrBQZEdXHypUrsXnzZly7dg3R0dGYOnUqhg4dChcXF3Tq1Alyed1TMZM4kY1IVEYj1akI592UlbaPyGqF244qHPFicRciezRv3jzExsbiyy+/xIMPPmjcXlxcXO+xOTudyAaElbujjcofO30rNzuJV/ng3hI/fBdwCQaJ9S4DElHdjRs3Dnl5eZg0aRJGjx6NdevWobCw0CJjM4kT2YA7zU72+t2otH14VjxUMg12+bG4C5G9+s9//oNffvkFS5YsQWBgID744AN0794dM2bMgEQiqVf9dF5OJ7IymUGC/spI/OF1G/mKv5ud+Ktd0CM/HFsCL1Wa6EZE9kcmk6FPnz7o06cPVCoVfvzxR3z33XcQBAFTp05Fz5498dBDD6FXr17GlqXmYBInsrKORcHw0TlXmdA2NDsWAPAdi7sQNSlubm54/PHH8fjjjyM1NRVbt27F999/j507d8LZ2RknT540eyxeTieyskRlNPIUZfjDM8O4zVkvR5KyJQ56pyLHsdSK0RFRQwoPD8eLL76IXbt24euvv8bgwYNFvZ5n4kRW5K11QqeCECQHVZ64NlAZBTe9A3uGEwEoKNWgSGuZsXQKoLHPXwsKCszaLyYmBjNmzBA1NpM4kRX1U0ZCBmmlvuEVxV3icc41B5fc8qwYHRFZgthCLhcuXDB7XyZxImsRKsqpnnPLQZrT3+tFHywIQYjaDStDT9XwYiKyF3d6h7dq1QqJiYkICgqy2NhM4kRWcq/KD2FqD2wIPlJp+/CseGQ5lOCQd7qVIiMiS9q3bx927tyJHTt24JNPPkG7du2QlJSEpKQkBAYG1mtsTmwjspJEZTRKpVoc9E41bost8cZ9qgBsZXEXoiYjJCQETz/9NDZs2IC9e/ciMTERP/30E/r27YvRo0dj9erVyMrKqtPYTOJEVuCsl6Nnfjh+8bmF8rvWgA/PikeJVIud/tdqeDUR2avg4GBMmDAB69evx969e5GUlISdO3eiX79+xoQuBpM4kRX0zGsB5380O/HVOKNnfgvs9LuGUhZ3IWrygoKCMGHCBHz44YeYOHEizp49i/fff1/UGLwnTmQFicpo3HQqxAXXXOO2IdmxkAjAd4GXrRgZETWG69evY9euXdi1axfOnz8Pf39/jBo1CgMHDhQ1DpM4USMLL/PAvSV++F/YSWOzE0e9DINyWuKQdzqyHEusGyARNYiLFy9i586d2L17N1JSUhASEoIBAwbg3//+N+6//35IJOJ7pjOJEzWyRGUUdBID9vjeMG4bkBsFd70jtgSwuAvRPxWVaVGgsdBgDgDgaKHBzDNv3jzs3r0baWlpaNGiBQYMGID33nsPbdu2rffYTOJEjUhmkKBfbhSOeKajUKEGAEgEYFhWHC665lbpJU5E9m/lypWQSqW4//77cc8996C8vBzbtm3Dtm3bTO4/a9Yss8dmEidqRJ0LQ+Ctc8LOu1qLdioMQZjaA3NDD1XqJU5ETUNISAgAICMjAxkZGTXuK5FImMSJbNVAZTRyFWU4dlezk+FZ8chRlOJXr9QaXklE9mrfvn0NNrbNLjE7c+YMxo4diw4dOqBHjx549913UVZWZu2wiOrMR+OEjoXB2O173VjIJbrUCwnFgfgu8DL0UhZ3IbI1586dw5NPPokOHTqge/fumDNnDjQaS92grz+bTOLFxcV45pln0L9/fxw+fBibNm3CyZMnMX/+fGuHRlRnA3KjqjQ7GZ4VjzKpFj/5XbViZERkiiAIeO6555CYmIg//vgDmzdvxsGDB7F8+XJrh2Zkk0k8PT0dHTp0wIQJEyCXyxEUFIShQ4fi6NGj1g6NqG6EikvpZ9yycdtJBaDizLx3Xgvs8ruOErmF+iwSkcWoVCpkZWXBYDAYm5hIpVI4OTlZObK/We2euFqtRk5OjsnnoqKi8OmnnxofC4KA3bt3o02bNo0VHpFFtVH5I1TtjnXB54zbBufEQiZIsTWAxV2IGltmZiZkMlmlbR4eHvDw8DA+dnd3x8SJE/HBBx9g3rx50Ov16N+/PyZMmNDI0VbPakn81KlTeOqpp0w+t2bNGnTu3BkAoNPpMHv2bKSmpoq+nJ4854l6x0lkCWWzy6BL0+Htr/rjHWcJhHIBJQ+XQNZLhs0f8XNK1NjGjBkDpbLyks5p06bhxRdfND42GAyQy+WYNWsWHnvsMdy8eRPTpk3D4sWLMX369MYO2SSrJfHOnTvj0qWaC1vk5uZixowZyMvLw9q1a0W3bGvTpg0cHRt3UX9dHT9+HA888IC1wzCLPcUKWD9eXZEOh/YdQvDYYMR3jwcA3F52G5cLL6PNO23g9YCXcV9rxyqWPcVrT7EC9hOvWq3G2bNnG/Q9ysp1KFVbZuKno0ECwBHr1q0zeSZ+t927d2P37t3YsWMHACA2NhYvvPAC5syZwyRem0uXLmHKlClo164dli5dCldXV2uHRFQn2euzYSg1IHhSMABAMAhI+zgNbve7wbOHp5WjI2qegoKCaj3Jy8jIgE5XuRmRXC6HQqFoyNBEscmJbUqlEhMnTkRSUhIWL17MBE52LWNlBlxau8C9ozsAIG9nHkovliLslbA61UomosbRvXt3ZGVlYdmyZdDr9UhNTcVnn32GoUOHWjs0I5s8E1+/fj2USiU2bNiAjRs3GreHhITgxx9/tGJkROKUnCtB8ZFitFzQ0piw0z5Og0OIAwJGBVg5OiKqSUxMDL744gssXLgQy5Ytg4eHB4YMGYIXXnjB2qEZ2WQSnzZtGqZNm2btMIjqLWNFBiQKCQLHVsznUP2pQv7ufES9FwWpg01eCCOiu3Tt2hVdu3a1dhjV4rcIUQMxaAzI+ioLvkN84eDvAABIW5gGqbMUIc+GWDk6ImoKmMSJGkju97nQKrXGCW2aLA2y1mYhaEIQFD62MzGGiOwXkzhRA8lYkQGHUAf4DPQBAKR/lg5BLSDs5TArR0ZETQWTOFEDKE8rR97OPARNCIJEJoG+XI/bS2/D52EfuMS7WDs8ImoibHJiG5G9y1qdBRiA4KcrLqVnr8uGNkeL8BnhVo6MyP5oyvVQlxssM5bQtM5dm9a/hsgGCAYBGSsz4NXbC84tnSEIAlIXpMK1rSu8+njVPgARkZmYxIksrOCXApRfK0fQpCAAQP6efJSeK0X4jHAWdyEii2ISJ7KwzBWZkHnK4D/SHwCQtiANikAFAkazuAsRWRaTOJEFaQu0yNmcg8AnAiFzlqHkQgnyduQh9IVQSB3560ZElsVvFSILyl6fDUO5wXgpPW1hGqROUoRMZXEXIrI8JnEiC8pckQnXtq5wf8AdGqUGWWuyEDgu0FixjYjIkpjEiSxEdUaF4mPFCJ4YDIlEgtuf34ah3ICw6SzuQkQNg+vEiSwkY2UGJA4VzU4MagNuf3obPkk+cL2XrXSJ6kOr1kJTZpl14lpJ0zp3bVr/GiIrMagrmp34DfODwleB7PXZ0GRqEPYKz8KJqOEwiRNZgPI7JXR5OgRPDK4o7vJxKlxau8B7gLe1QyOiJoxJnMgCMlZmwDHcEd79vVFwoAAlp0sQ/gqLuxBRw2ISJ6qn8lvlyN+Vj6CnK5qdpC1Ig8JfgYAnWdyFiBoWkzhRPWWuzgQEIGhCEEovlyL3h1yEPBcCmZPM2qERURPHJE5UD4JBQObKTHj184JzlDPSFqVB4iBB6POh1g6NiJoBJnGieijYX4DyG+UInhQMbZ4WmV9mIvDJQDgEsrgLETU8JnGieshYmQG5lxx+w/xwe9ltGEoNXFZGRI2GxV6I6kibr0VOcg6CJwdDIpMg/ZN0ePf3htt9btYOjahJ0ZTroS7TW2YsadOaq8IzcaI6yl6XDUEtIHhiMHI25UBzm8VdiKhxMYkT1VHGygy4JbjBrb0bUhekwqWVC3ySfKwdFhE1I0ziRHVQfKoYqhMqBE0KQuHBQqhOqBA2PQwSKYu7EFHjYRInqoPMFZmQOEoQOCYQaR+nQe4jR+C4QGuHRUTNDJM4kUj6cj2y1mbBf7g/dPk6KL9TVhR3cWlaE2aIyPYxiROJpNyqhC5fh6BJQRXFXeQShL7A4i5E1PiYxIlEylyRCccIR7jf746MlRkIGB0Ax2BHa4dFRM0QkziRCGU3ypC/Nx/BTwcjY2UGDCUs7kJE1sNiL0QiZH6ZCQAIHBuIU31Owau3F9zbu1s5KqKmTVOutVyxF5nBIuPYCp6JE5lJ0AvIXJUJ7/7eKD5WDHWqmmfhRGRVTOJEZsrflw/1LTWCJwUjdUEqnGOc4fuIr7XDIqJmjEmcyEwZKzIg95FDEaBA8R/FLO5CRFbHJE5kBm2uFsotSgQ+GYjbS29D7iVH4HgWdyEi62ISJzJD1rosCBoB3v28kfNtDoKfDYbcjfNCici6mMSJaiEIAjJWZMDtATcU/FwAiVSC0Gks7kJE1sckTlQL1QkVSk6XwP9Rf2Qsz4D/KH84hTlZOywiIq4TJ6pNxsoMSJ2kMJQZoC/Wc1kZUSPTlJdDU6azzFjyppX2eCZOVAN9WUWzE79hfshakwXP7p7w6OBh7bCIiAAwiRPVSPmtEvpCPeQ+cpTfKEfYDJ6FE5HtYBInqkHGygw4RTlBdVIFp2gn+A3xs3ZIRERGTOJE1Si7VoaCfQVwbe2Kot+LEPZSGCQyFnchItvBJE5UjcwvMwEJoM3XQuYhQ9DEIGuHRERUCZM4kQl3mp243OOCosNFCH4mGHL3pjWrlYjsH5M4kQl5u/OgTlPDUGYABCDsRU5oIyLbwyROZELmykxIHCXQZGrg/6g/nCJY3IWIbA+vDxL9g0apgXKrEoJBgKAXWNyFyMq0JYXQFKstM5bU0SLj2AomcaJ/yPo6C4JWAAB4POgBzwc9rRwREZFpvJxOdBdBEJC5MtP4mMVdiMiWMYkT3aX4WDFK/iwBADhGOMJvOIu7EJHtYhInukvGigzjz2EvhUEq568IEdkufkMR/UVfqkf2N9kAAJmbDMGTgq0cERFRzZjEif6Sk5wDfZEeABA0KQhyT877JCLbxiRO9BfjpXRpxaV0IiJbxyROBKA0pRSFPxcCAPyG+cE52tnKERER1Y7XC4kAZK66a1kZi7sQ2RRtaTE0JWWWGUvRtP5A55k4NXsGnaGiYxkA947u8OzG4i5EZB+YxKnZy9+VD81tDYCKs3CJhD3Dicg+MIlTs3dnQptjmCP8H/W3cjREROZjEqdmTZOtQe62XABA6IuhkCr4K0FE9oPfWNSsZX2dBUEnQOoiRfAzLO5CRPaFSZyaLUEQjJfSg54OgsJbYeWIiIjEYRKnZqvoSBFKz5cCEiDsZS4rIyL7w3Xi1GzdaTnqO9gXLrEuVo6GiKqjLSuGtrTUMmM56iwyjq3gmTg1S/oSPbLXVzQ7CZ8RbuVoiIjqhkmcmqXsTdnQF+vh1t4Nnj1Z3IWI7BOTODVLdy6ls7gLEdkzJnFqdkovl6LwYCEcgh0Q8HiAtcMhIqozJnFqdu40OwmdFgqpA38FiMh+8RuMmhWDzoDM1ZmQOksR8myItcMhIqoXJnFqVvJ+yoMmQ4Og8UFQ+LK4CxHZNyZxalaME9qms7gLEdk/FnuhJk81UIUDeQcqbfuj1R9QBCrQLbObdYIiIrNpy4qhKVFZZiwnvUXGsRU8E6cmT8gTTG7XZmkbORIiIstiEiciIrJTTOJERER2ymaT+J49ezBkyBC0b98evXv3xuLFi2EwGKwdFhERNSNKpRJTp07FAw88gG7dumHJkiXWDqkSm5zYdvPmTUyfPh1LlixB7969cfPmTTz55JMIDQ3FyJEjrR0eERE1E6+88grCwsLw22+/ITU1FVOmTEFkZCQeeeQRa4cGwEbPxCMiInD48GH07t0bgiAgLy8PWq0WPj4+1g6N7JDEx3RtdEUg14kTUfVu3ryJP/74A6+++iqcnJwQGxuL8ePHY+3atdYOzchqZ+JqtRo5OTkmn/P394ebmxt0Oh0SEhKg1WqRmJiIHj16mDW2IFTMRtZoNBaLtzGo1Wprh2A2e4rVbZcb2rRpY/I5W/x32GJMNbGneO0pVsA+4r3zPXvne7chBHi5WnyszMxMyGSySs95eHjAw8PD+PjKlSvw8vKCn5+fcVt0dDQuXbpksXjqy2pJ/NSpU3jqqadMPrdmzRp07twZMpkMJ06cwO3bt/Hiiy/inXfewTvvvFPr2FptxdKhy5cvWzTmhnb27Flrh2A2e4oVsK947SlWwL7itadYAfuKV6vVwsnJyaJjymQyyGQyfPb6EIuOq9VqMXz4cJSUlFTaPm3aNLz44ovGxyUlJXB2dq60j7OzM8rLyy0aT31YLYl37ty51r9mJBIJHBwcEBkZieeffx6zZs0yK4m7uroiLi4OCoWCbSaJiBqQIAjQarVwdbXc2fIdcrkcbdq0gV5v2QItpaWl2LZtW5Xtd5+FA4CLi0uVhF1WVgY3NzeLxlMfNjmxbd++ffj000+RnJxs3KZWq+Hp6WnW66VSKdzd3RsqPCIiuoulz8DvJpfLIZdbNlU5OjrC29u71v1iY2ORn5+PvLw845ysq1evIiYmxqLx1IdNTmxr164d0tLS8L///Q96vR4pKSn47LPPMHr0aGuHRkREzURkZCTat2+PefPmobS0FCkpKVizZg2GDx9u7dCMJEJDzkaohzNnzmDu3Lm4fPkyfH198cQTT2D8+PGQSm3y7w4iImqCsrOz8c477+Do0aNQKBQYM2YMnn/+eWuHZWSzSZyIiIhqxtNaIiIiO+MkzeIAAAp9SURBVMUkTkREZKeYxImIiOwUkzgREZGdajJJ/MyZMxg7diw6dOiAHj164N1330VZWVm1+2/evBn9+vVDQkICxowZg5SUlEaMtsKNGzfQqVMnpKWl1bjfsGHD0LZtW7Rv397439WrVxspygrmxmrt4yq241BjH1sx8Vn7WALi4rWFzylg3mfVFo7tHebEa+1jK+b71ZaObbMgNAFFRUVCp06dhFWrVglarVbIyMgQRowYIbzzzjsm9z98+LDwwAMPCKdOnRLUarWwYMECISkpSTAYDI0W8759+4QuXboIcXFxQmpqarX7qdVqoXXr1sLVq1cbLbZ/MjdWWziuY8eOFd58802hrKxMuHz5stC7d2/h+++/N7mvNY6tufHZwrEUE68tfE4FwbzPqq0cW0EwL15rH1sx36+2dGybiyaRxC9cuCA8//zzlbatXr1aGDx4sMn9X331VeH//u//jI91Op3QsWNH4bfffmvQOO9YunSpMGjQIGHjxo21JsZTp04J999/v9V+CcTEau3jeuPGDSEuLk7Iyckxblu1apUwevRok/s39rEVE5+1j6UgiIvX2p9TQTD/s2oLx1YQzI/X2sdWzPerrRzb5sRuLqer1WqkpaWZ/C8qKgqffvqpcV9BELB79+5qO1elpKQgNjbW+FgmkyEyMtJinWlqilWtVmPEiBH44Ycf0KVLl1rHOnv2LFxcXPDUU0+hc+fOGDFiBPbt22eROC0da0Mf19riFdtxqKGP7T+Jia8xjmVtxMTb2MfSFHM/q7ZwbAHz47X2sW3VqpXZ36+2cmybE5usnW6KOV3PAECn02H27NlITU3F/PnzTe5fUlJSpdavk5NTjffQGyJWcwiCgPvuuw+vvfYawsLCsHv3brz88sv46quvkJCQYFOxNvRxBWqOd968eaI6DjX0sf0nMR2RGuNY1kZMvI19LE0JDAw0az9bOLaA+fHawrG9o7bvV1s5ts2J3SRxc7qe5ebmYsaMGcjLy8PatWur/SVxdnau0qe3vLzcYp1pzInVXGPHjsXYsWONjx9++GFs27YNO3futMgvsCVjbejjCtQc7+7du0V1HGroY/tPYjoiNcaxrI2YeBv7WNaHLRxbMWzl2Jrz/Wpvx7YpsJvL6bW5dOkSRowYAU9PT6xfvx6hoaHV7hsbG4tr164ZH+v1ety4caPSZSBbsXbtWvz888+VtqnVajg6OlopoupZ+7je3XHojpo6DjX2sRUTn7WP5Z0YzI2Xn9OGYwvH1tzvV3s7tk1Bk0jiSqUSEydORFJSEhYvXlxrX9sRI0Zg27ZtOH78ODQaDRYvXgwPDw906NChkSI2353i+zdu3IBOp0NycjJOnjxpU1107rD2cRXbcaixj62Y+Kx9LMXGy89pw7H2sRXz/Wpvx7ZJsOasOkv55JNPhLi4OKFdu3ZCQkKC8b9BgwYJgiAI6enpQkJCgnD06FHjazZv3iwMHDhQSEhIEJ588kkhJSWl0eNOTU2tMiv1n7FqNBph3rx5Qo8ePYR27doJI0aMEA4fPmyTsQqC9Y9rVlaW8MILLwidOnUSunXrJnz66afVxmuNY1tdfLZ4LMXEayufU0Go+lm11WN7R23xWvvY1vT9auvHtjlgFzMiIiI71SQupxMRETVHTOJERER2ikmciIjITjGJExER2SkmcSIiIjvFJE5ERGSnmMSJiIjsFJM4ERGRnWISpyYvPj4e3333nVn7pqSk4MCBAw0bkAXYS5xE1LCYxKnJ+/XXX5GUlGTWvs8//zz+/PPPBo6o/uwlTiJqWEzi1OT5+/ub3fHJXqoQWyLO5557Dv/617+Mj8+dO4f4+HjMmjXLuO3QoUNo06YNlEplvd+PiCyPSZyavLsvp8fHx2Pz5s148skn0bZtWyQlJWHDhg0AgHHjxuHWrVtYsmQJ+vbtCwAoLCzEzJkz0blzZ3Tq1AnPPPNMpVaLd8ZctGgRevbsiZ49eyInJwcqlQpvv/02unbtivbt22PSpEnG15k75vr16zFixAi0bdsWI0aMwNGjR6uNsy48PDxQUlJifPz111/DxcUFRUVFxm0bNmzAwIED4efnV+f3IaKGwyROzc78+fPx5JNPYsuWLejQoQNmz56N9PR0fPLJJwgNDcXEiROxefNmCIKAKVOmIDs7G8uXL8e6desQEhKCMWPGID8/v9KYmzZtwhdffIElS5bA398f06dPx++//46PPvoIycnJcHFxweTJk6HVas0ec968eRg9ejS2bNmCNm3aYNKkSUhNTa0SZ125u7ujtLQUAJCfn4/t27fj8ccfNybx3Nxc7N27F2PGjKnzexBRw2ISp2Zn5MiRGDRoEFq2bIk33ngDBoMBZ86cgZeXF2QyGVxcXODj44Pff/8df/75JxYtWoT77rsPMTExePvtt+Hp6YmNGzdWGnP48OG455570LZtW1y7dg0HDx7E22+/jS5duiA6OhrvvPMOBg4ciO3bt5s95qhRozBq1Ci0bNkSb731Fvz9/bFx48YqcdbV3Ul806ZNSEhIQNu2bY1JPDk5GVFRUcZe0G+99RZ69OiB+Pj4Or8nEVmW3NoBEDW2yMhI488eHh4AAK1WW2W/8+fPQ6/Xo0ePHpW2q9VqXL16tdK28PBw48+XL18GALRt29a4zdvbG2+++SaWL19u9pgdO3Y0/iyTydCmTRvj2JZw53K6Xq/H+vXrMXPmTDg4OKC4uBiCIGDjxo2YPHmycf9HHnkEL774Irp162axGIiofpjEqdlxcHCoss3URDGFQgEvL68qZ8gA4OLiUunx3RPn5PLqf63EjPnPcQwGAyQSSbVji+Xm5obS0lLs378fgiCgb9++OHXqFIqKivDbb78hPz8fQ4YMMe5/9x8VRGQbeDmd6C53J8nY2FgUFBQAACIiIhAREYGwsDAsXLjQOMnMlJYtWwIAzp49a9ymUqnQpUsXlJSUmD3muXPnjD/rdDqcO3cO9957b5U468rDwwOlpaX46quvMHr0aMhkMrj+/3bukFWRMIzi+NlyyzDZLAxOEgwWyxSTRZngiM1gMlllRDAIYhcMg8HmJ7ApmAfMhkGx2RRsyrhh4XJZ2UUM6x32//sAD+dNZ96HlzEMXS4Xzedzua778GEB4HuhxIEvDMPQfr/X8XhUoVBQLpdTu91WGIba7XbqdrtarVbKZDJ/nJFOp1UsFtXv9xWGoaIoUqfTkWmaajabT8+cTqdaLBaKoki9Xk/n81m1Wu0h56tM09TpdNJms1G1WpX063Z+u920XC5Vr9dfng3g36DEgS8ajYbW67XK5bLu97vG47Esy1Kr1ZLrutrv9wqCQJZl/XXOcDhUNptVq9WS53m6Xq8KgkAfHx9Pz/Q8T5PJRK7r6nA4aDabKZVKPeSM4/ils5qmqTiOVSqVPh/IGYYhScrn858bBQDf1497Uv5uAfxHbNvWaDRSpVJ5d5QHtm1ru92+OwYAcRMH8CTf9+U4jiTJcRz5vv/mRAB4nQ7gKYPB4N0RAPyGdToAAAnFOh0AgISixAEASChKHACAhKLEAQBIKEocAICEosQBAEgoShwAgISixAEASKifxUQnxQ4JHzgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a pretty contour plot over the grid of w[0], w[1], loss values\n",
    "\n",
    "levels = np.linspace(0, 40, 51) # 50 evenly spaced levels\n",
    "\n",
    "fig_handle, ax_handle = plt.subplots(nrows=1, ncols=1, figsize=(8,8));\n",
    "\n",
    "ax_handle.contour(w0_GH, w1_GH, loss_GH, levels=levels, linewidths=0, colors='k')\n",
    "cntrf_handle = ax_handle.contourf(w0_GH, w1_GH, loss_GH, levels=levels, cmap='RdBu_r', vmin=levels[0], vmax=levels[-1]);\n",
    "\n",
    "cbar = plt.colorbar(cntrf_handle, ax=ax_handle)\n",
    "cbar.set_label('MAP loss objective (lower is better)', fontsize=16);\n",
    "cbar.set_ticks(levels[::10]);\n",
    "\n",
    "# Show the first 10 iterates of GD\n",
    "ax_handle.plot(whistory_GD1_T2[:1,0], whistory_GD1_T2[:1,1], 'kx', markersize=15, label='Initial w value')\n",
    "ax_handle.plot(whistory_GD1_T2[:10,0], whistory_GD1_T2[:10,1], 'ks-', label='First 10 steps of 1st-order GD')\n",
    "ax_handle.plot(whistory_GD2_T2[:10,0], whistory_GD2_T2[:10,1], 'ms-', label='First 10 steps of 2st-order GD')\n",
    "\n",
    "plt.xlabel('intercept   $w_1$');\n",
    "plt.ylabel('slope   $w_2$');\n",
    "plt.gca().set_aspect('equal', 'box');\n",
    "plt.ylim([-3, 2]);\n",
    "plt.legend(loc='upper left');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion: Compare the *first step* that 1st-order GD took to the *first step* that 2nd-order GD took.\n",
    "\n",
    "* Which is a better *direction*?\n",
    "* Which ended up closer to the optimal value because of well-chosen step-length?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion: Compare the overall behavior of 1st and 2nd order GD.... do you see big qualitative differences?\n",
    "\n",
    "* Which one makes faster progress toward the minimum?\n",
    "* How are both sensitive to the step-size choice?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}