t81_558_class_04_4_backprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_4_backprop.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_1_feature_encode.ipynb)\n",
    "* Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_3_regression.ipynb)\n",
    "* **Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training** [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.4: Training Neural Networks\n",
    "\n",
    "Backpropagation [[Cite:rumelhart1986learning]](https://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf) is one of the most common methods for training a neural network. Rumelhart, Hinton, & Williams introduced backpropagation, and it remains popular today. Programmers frequently train deep neural networks with backpropagation because it scales really well when run on graphical processing units (GPUs). To understand this algorithm for neural networks, we must examine how to train it as well as how it processes a pattern.\n",
    "\n",
    "Researchers have extended classic backpropagation and modified to give rise to many different training algorithms. This section will discuss the most commonly used training algorithms for neural networks. We begin with classic backpropagation and end the chapter with stochastic gradient descent (SGD).\n",
    "\n",
    "Backpropagation is the primary means of determining a neural network's weights during training. Backpropagation works by calculating a weight change amount ($v_t$) for every weight($\\theta$, theta) in the neural network. This value is subtracted from every weight by the following equation: \n",
    "\n",
    "$$ \\theta_t = \\theta_{t-1} - v_t $$\n",
    "\n",
    "We repeat this process for every iteration($t$). The training algorithm determines how we calculate the weight change. Classic backpropagation calculates a gradient ($\\nabla$, nabla) for every weight in the neural network for the neural network's error function ($J$). We scale the gradient by a learning rate ($\\eta$, eta).\n",
    "\n",
    "$$ v_t = \\eta \\nabla_{\\theta_{t-1}} J(\\theta_{t-1}) $$\n",
    "\n",
    "The learning rate is an important concept for backpropagation training. Setting the learning rate can be complex:\n",
    "\n",
    "* Too low a learning rate will usually converge to a reasonable solution; however, the process will be prolonged.\n",
    "* Too high of a learning rate will either fail outright or converge to a higher error than a better learning rate.\n",
    "\n",
    "Common values for learning rate are: 0.1, 0.01, 0.001, etc.\n",
    "\n",
    "Backpropagation is a gradient descent type, and many texts will use these two terms interchangeably. Gradient descent refers to calculating a gradient on each weight in the neural network for each training element. Because the neural network will not output the expected value for a training element, the gradient of each weight will indicate how to modify each weight to achieve the expected output. If the neural network did output exactly what was expected, the gradient for each weight would be 0, indicating that no change to the weight is necessary.\n",
    "\n",
    "The gradient is the derivative of the error function at the weight's current value. The error function measures the distance of the neural network's output from the expected output. We can use gradient descent, a process in which each weight's gradient value can reach even lower values of the error function. \n",
    "  \n",
    "The gradient is the partial derivative of each weight in the neural network concerning the error function. Each weight has a gradient that is the slope of the error function. Weight is a connection between two neurons. Calculating the gradient of the error function allows the training method to determine whether it should increase or decrease the weight. In turn, this determination will decrease the error of the neural network. The error is the difference between the expected output and actual output of the neural network. Many different training methods called propagation-training algorithms utilize gradients. In all of them, the sign of the gradient tells the neural network the following information:\n",
    "\n",
    "* Zero gradient - The weight does not contribute to the neural network's error.\n",
    "* Negative gradient - The algorithm should increase the weight to lower error.\n",
    "* Positive gradient - The algorithm should decrease the weight to lower error.\n",
    "\n",
    "\n",
    "Because many algorithms depend on gradient calculation, we will begin with an analysis of this process. First of all, let's examine the gradient. Essentially, training is a search for the set of weights that will cause the neural network to have the lowest error for a training set. If we had infinite computation resources, we would try every possible combination of weights to determine the one that provided the lowest error during the training. \n",
    "\n",
    "Because we do not have unlimited computing resources, we have to use some shortcuts to prevent the need to examine every possible weight combination. These training methods utilize clever techniques to avoid performing a brute-force search of all weight values. This type of exhaustive search would be impossible because even small networks have an infinite number of weight combinations.\n",
    "\n",
    "Consider a chart that shows the error of a neural network for each possible weight. Figure 4.DRV is a graph that demonstrates the error for a single weight:\n",
    "\n",
    "**Figure 4.DRV: Derivative**\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "Looking at this chart, you can easily see that the optimal weight is where the line has the lowest y-value. The problem is that we see only the error for the current value of the weight; we do not see the entire graph because that process would require an exhaustive search. However, we can determine the slope of the error curve at a particular weight. In the above chart, we see the slope of the error curve at 1.5. The straight line barely touches the error curve at 1.5 gives the slope. In this case, the slope, or gradient, is -0.5622. The negative slope indicates that an increase in the weight will lower the error.\n",
    "The gradient is the instantaneous slope of the error function at the specified weight. The derivative of the error curve at that point gives the gradient. This line tells us the steepness of the error function at the given weight.  \n",
    "Derivatives are one of the most fundamental concepts in calculus. For this book, you need to understand that a derivative provides the slope of a function at a specific point. A training technique and this slope can give you the information to adjust the weight for a lower error. Using our working definition of the gradient, we will show how to calculate it.\n",
    "\n",
    "## Momentum Backpropagation\n",
    "\n",
    "Momentum adds another term to the calculation of $v_t$:\n",
    "\n",
    "$$ v_t = \\eta \\nabla_{\\theta_{t-1}} J(\\theta_{t-1}) + \\lambda v_{t-1} $$\n",
    "\n",
    "Like the learning rate, momentum adds another training parameter that scales the effect of momentum. Momentum backpropagation has two training parameters: learning rate ($\\eta$, eta) and momentum ($\\lambda$, lambda). Momentum adds the scaled value of the previous weight change amount ($v_{t-1}$) to the current weight change amount($v_t$).\n",
    "\n",
    "This technique has the effect of adding additional force behind the direction a weight is moving. Figure 4.MTM shows how this might allow the weight to escape local minima.\n",
    "\n",
    "**Figure 4.MTM: Momentum**\n",
    "![Momentum](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_5_momentum.png \"Momentum\")\n",
    "\n",
    "A typical value for momentum is 0.9.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch and Online Backpropagation\n",
    "\n",
    "How often should the weights of a neural network be updated?  We can calculate gradients for a training set element.  These gradients can also be summed together into batches, and the weights updated once per batch.\n",
    "\n",
    "* **Online Training** - Update the weights based on gradients calculated from a single training set element.\n",
    "* **Batch Training** - Update the weights based on the sum of the gradients over all training set elements.\n",
    "* **Batch Size** - Update the weights based on the sum of some batch size of training set elements.\n",
    "* **Mini-Batch Training** - The same as batch size, but with minimal batch size.  Mini-batches are very popular, often in the 32-64 element range.\n",
    "\n",
    "Because the batch size is smaller than the full training set size, it may take several batches to make it completely through the training set.  \n",
    "\n",
    "* **Step/Iteration** - The number of processed batches.\n",
    "* **Epoch** - The number of times the algorithm processed the complete training set.\n",
    "\n",
    "## Stochastic Gradient Descent\n",
    "\n",
    "Stochastic gradient descent (SGD) is currently one of the most popular neural network training algorithms.  It works very similarly to Batch/Mini-Batch training, except that the batches are made up of a random set of training elements.\n",
    "\n",
    "This technique leads to a very irregular convergence in error during training, as shown in Figure 4.SGD.\n",
    "\n",
    "**Figure 4.SGD: SGD Error**\n",
    "![SGD Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_5_sgd_error.png \"SGD Error\")\n",
    "[Image from Wikipedia](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)\n",
    "\n",
    "Because the neural network is trained on a random sample of the complete training set each time, the error does not make a smooth transition downward.  However, the error usually does go down.\n",
    "\n",
    "Advantages to SGD include:\n",
    "\n",
    "* Computationally efficient.  Each training step can be relatively fast, even with a huge training set.\n",
    "* Decreases overfitting by focusing on only a portion of the training set each step.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Techniques\n",
    "\n",
    "One problem with simple backpropagation training algorithms is that they are susceptible to learning rate and momentum. This technique is difficult because:\n",
    "\n",
    "* Learning rate must be adjusted to a small enough level to train an accurate neural network.\n",
    "* Momentum must be large enough to overcome local minima yet small enough not to destabilize the training.\n",
    "* A single learning rate/momentum is often not good enough for the entire training process. It is often helpful to automatically decrease the learning rate as the training progresses.\n",
    "* All weights share a single learning rate/momentum.\n",
    "\n",
    "Other training techniques:\n",
    "\n",
    "* **Resilient Propagation** - Use only the magnitude of the gradient and allow each neuron to learn at its rate. There is no need for learning rate/momentum; however, it only works in full batch mode.\n",
    "* **Nesterov accelerated gradient** - Helps mitigate the risk of choosing a bad mini-batch.\n",
    "* **Adagrad** - Allows an automatically decaying per-weight learning rate and momentum concept.\n",
    "* **Adadelta** - Extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate.\n",
    "* **Non-Gradient Methods** - Non-gradient methods can *sometimes* be useful, though rarely outperform gradient-based backpropagation methods.  These include: [simulated annealing](https://en.wikipedia.org/wiki/Simulated_annealing), [genetic algorithms](https://en.wikipedia.org/wiki/Genetic_algorithm), [particle swarm optimization](https://en.wikipedia.org/wiki/Particle_swarm_optimization), [Nelder Mead](https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method), and [many more](https://en.wikipedia.org/wiki/Category:Optimization_algorithms_and_methods)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ADAM Update\n",
    "\n",
    "ADAM is the first training algorithm you should try.  It is very effective.  Kingma and Ba (2014) introduced the Adam update rule that derives its name from the adaptive moment estimates. [[Cite:kingma2014adam]](https://arxiv.org/abs/1412.6980)  Adam estimates the first (mean) and second (variance) moments to determine the weight corrections.  Adam begins with an exponentially decaying average of past gradients (m):\n",
    "\n",
    "$$ m_t = \\beta_1 m_{t-1} + (1-\\beta_1) g_t $$\n",
    "\n",
    "This average accomplishes a similar goal as classic momentum update; however, its value is calculated automatically based on the current gradient ($g_t$).  The update rule then calculates the second moment ($v_t$):\n",
    "\n",
    "$$ v_t = \\beta_2 v_{t-1} + (1-\\beta_2) g_t^2 $$\n",
    "\n",
    "The values $m_t$ and $v_t$ are estimates of the gradients' first moment (the mean) and the second moment (the uncentered variance).  However, they will be strongly biased towards zero in the initial training cycles.  The first moment’s bias is corrected as follows.\n",
    "\n",
    "$$ \\hat{m}_t = \\frac{m_t}{1-\\beta^t_1} $$\n",
    "\n",
    "Similarly, the second moment is also corrected:\n",
    "\n",
    "$$ \\hat{v}_t = \\frac{v_t}{1-\\beta_2^t} $$\n",
    "\n",
    "These bias-corrected first and second moment estimates are applied to the ultimate Adam update rule, as follows:\n",
    "\n",
    "$$ \\theta_t = \\theta_{t-1} - \\frac{\\alpha \\cdot \\hat{m}_t}{\\sqrt{\\hat{v}_t}+\\eta} \\hat{m}_t $$\n",
    "\n",
    "Adam is very tolerant to initial learning rate (\\alpha) and other training parameters. Kingma and Ba (2014)  propose default values of 0.9 for $\\beta_1$, 0.999 for $\\beta_2$, and 10-8 for $\\eta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methods Compared\n",
    "\n",
    "The following image shows how each of these algorithms train. It is animated, so it is not displayed in the printed book, but can be accessed from here: [https://bit.ly/3kykkbn](https://bit.ly/3kykkbn).\n",
    "\n",
    "![Training Techniques](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/contours_evaluation_optimizers.gif \"Training Techniques\")\n",
    "Image credits: [Alec Radford](https://scholar.google.com/citations?user=dOad5HoAAAAJ&hl=en)\n",
    "\n",
    "## Specifying the Update Rule in Keras\n",
    "\n",
    "TensorFlow allows the update rule to be set to one of:\n",
    "\n",
    "* Adagrad\n",
    "* **Adam**\n",
    "* Ftrl\n",
    "* Momentum\n",
    "* RMSProp\n",
    "* **SGD**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Restoring model weights from the end of the best epoch.\n",
      "Epoch 00105: early stopping\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2deZgU1dX/P3f2GZiNYREYEFREtgFZRXDBBVyIWzSuUaO4Jv5M3tcoasxrjEvyxtfdBI1rohEV1xBMREDEDQVlR2QbYAAZYJhh9qX7/v64VV3VPd0zDUzP9HI+z9NPVd26VX1rlm+fPvfcc5TWGkEQBCFxSOroAQiCIAjtiwi/IAhCgiHCLwiCkGCI8AuCICQYIvyCIAgJRkpHDyAcunbtqvv169fRwxAEQYgpli5dukdr3S2wPSaEv1+/fixZsqSjhyEIghBTKKW2BGsXV48gCEKCIcIvCIKQYIjwC4IgJBgx4eMPRmNjIyUlJdTV1XX0UOKGjIwMCgsLSU1N7eihCIIQQWJW+EtKSsjOzqZfv34opTp6ODGP1pq9e/dSUlJC//79O3o4giBEkJh19dTV1VFQUCCi30YopSgoKJBvUIKQAMSs8AMi+m2M/DwFITGIaeEXBCGB8XqhvrKjR3FwrHgT6io67O1F+GOI4uJi/vGPfxzwdVdffTWzZs2KwIgEoQOZew88VAgNNR09kgNj1xp4exq89/MWu63fVcn873ZRXd/U5kMQ4Y8hDlb4BSEuWf6a2TbGmPA3VJnt/h0tdrtwxhdc89ISdla0/bybCP8h8MorrzB27FhGjBjBDTfcwJYtWxgwYAB79uzB6/Vywgkn8OGHH1JcXMwxxxzDVVddRVFRERdeeCE1NeaPdenSpZx00kmMGjWKKVOmsHPnTgA2bNjAaaedxvDhwxk5ciQbN25k+vTpLFq0iBEjRvDoo4/i8Xj49a9/zZgxYygqKuKZZ54BTITOL37xCwYPHszZZ59NaWlph/2MBCFi2NUDo7WKoNcDX/wZGmv927XXbFVyyEvrGj1U1DYyZUgPCvMz23xoMRvO6eZ3/1zNmh372/Seg3vl8D8/GhLy/Nq1a3n99df57LPPSE1N5eabb2bhwoXccccd3HjjjYwbN47BgwczefJkiouLWbduHc8//zwTJkzgmmuu4c9//jO33nort9xyC++99x7dunXj9ddf5+677+aFF17g8ssvZ/r06Zx//vnU1dXh9Xr5wx/+wMMPP8zs2bMBePbZZ8nNzeXrr7+mvr6eCRMmMHnyZL799lvWrVvHypUr2bVrF4MHD+aaa65p05+PIEQN3saOHkFwls+E/9wJdeUw6S6n3Wu5bpJCC395jXmmE4/uRkZq6H4HS1wIf0cwb948li5dypgxYwCora2le/fu3Hvvvbz55pvMmDGDZcuW+fr36dOHCRMmAHDFFVfwxBNPcMYZZ7Bq1SpOP/10ADweDz179qSyspLt27dz/vnnA2ZhVTA+/PBDVqxY4fPfV1RUsH79ej755BMuvfRSkpOT6dWrF6ecckrEfg6C0GHYUWieho4dRyiqrW/agRZ/o+W6UUnwj4uh5Gu4fZNfl/Ja80x5mWkRGVpcCH9Llnmk0Fpz1VVX8dBDD/m119TUUFJSAkBVVRXZ2dlA81BJpRRaa4YMGcIXX3zhd27//vC+vWitefLJJ5kyZYpf+5w5cyQ0U4h/bBePJ0otfnvSOTXLv72x2mxVEnz/76CX2hZ/flZkVtGLj/8gOfXUU5k1a5bPf15WVsaWLVu44447uPzyy7nvvvu47rrrfP23bt3qE/jXXnuNiRMnMnDgQHbv3u1rb2xsZPXq1eTk5FBYWMi7774LQH19PTU1NWRnZ1NZ6YSvTZkyhb/85S80Npo/ku+//57q6mpOPPFEZs6cicfjYefOnSxYsKBdfiaC0CFEq/Dbk85pAcJvfyC06OppIJ0G+hW/bsJW2xgR/oNk8ODB3H///UyePJmioiJOP/10iouL+frrr33in5aWxosvvgjAoEGDePnllykqKqKsrIybbrqJtLQ0Zs2axR133MHw4cMZMWIEn3/+OQB///vfeeKJJygqKuL444/nhx9+oKioiJSUFIYPH86jjz7KtGnTGDx4MCNHjmTo0KHccMMNNDU1cf755zNgwACGDRvGTTfdxEknndSRPypBiCzR6uqxhT8lI3i7csmvxz9ks7ymkQdTn6fnp3fDjm/afGhx4erpKC6++GIuvvhiv7Yvv/zSt//2228DJgwzKSmJGTNmNLvHiBEj+OSTT5q1DxgwgPnz5zdrnzdvnt/xgw8+yIMPPtis31NPPRXeQwhCrBOtFr9t2esAi90n/C6Lv34/ZHXxHZbXNHBt0uc0jbiS1MLRbT40EX5BEGKbaLf4m+p9TQ/OWcuRa9ZxMbB0WwWjrPZbXlzArpRevn5lZWXcqDzorkdEZGji6mkH+vXrx6pVqzp6GIIQn0RrOKcdzWN9MGmteemzYhpqzQKuVBz3ToG3jCv2P0eWriFJwcA8DwAqMy8iQxOLXxCE2CZaXT12HiHL4q9t9DDAu4mfet4BoKhAg7V49978f8OGjzhn3DFw4q9h12r4C5CRG5GhicUvCEKMYodzRqGrx9MIe9aZ/UUPQ8lS9tU0cm3KHKdP9R5nf/f3Zmv7/WvLzTYjMha/CL8gCLFNNFr8mz+B2n3O8XOnUF7TQL12xeVX73b2K7aarUqCvRvhh5XmWFw9giB0CCVLoHMPyOvT0SMJTlsKf30VpHcOfX75TEjrDIOmtnyfXc3n9CpqGuml9joNTUGSr9WWwZMjneNYtPiVUsVKqZVKqWVKqSVWWxel1Fyl1Hprmx/JMcQSnTubP7gdO3Zw4YUXttj3scce8yV6AzjrrLMoLy+P6PiEBOW5U+GxoR09iiC0ccqG1e/CQ73hw3vgydEmyVog79wAr1/e+r1qy43bJsmxrffVNHK42oU3JQNSXInXLnFl3C3f6n+fGPbxT9Jaj9Ba28Go04F5WusBwDzrOG7xeIL88bRCr169Ws2fHyj8c+bMIS8vMtaBIEQnlo+/raJ6Ni802y//DHvXw75i//NNAR8wdfvhudNg86Lm96rdB5n5TkI2oKK6mkK1m5qRN8KIS01jUgr0HQ+3b4a8w2FDwNqdGBb+QM4FXrb2XwbO64AxtAmh0i3369eP++67j4kTJ/Lmm2+yceNGzjjjDEaNGsUJJ5zAd999B8DmzZsZP348Y8aM4Z577vG779ChxsLyeDzcdtttDBs2jKKiIp588kmeeOIJduzYwaRJk5g0aRJgQkb37DGTRY888ghDhw5l6NChPPbYY757Dho0iOuuu44hQ4YwefJkamsDkkcJQixykK6eDbsq4d5c9v7jelavXcMPHpNXyxbrzWuWsGxbue+15rvVvmuXbStnw7KFJsHay1NZtnWfX999ZbupS83xe7/SretJUV7Sux8Fyemm8YiTzcKtrC6QfRjUV0B6DmR1NedbSOtwKETax6+BD5VSGnhGa/0s0ENrvRNAa71TKdX9kN/lg+nOZEhbcdgwOPMPrXYLlm4ZTEbNTz/9FDB5fWbMmMGAAQNYvHgxN998M/Pnz+fWW2/lpptu4sorr+Tpp58Oev9nn32WzZs38+2335KSkkJZWRldunThkUceYcGCBXTt2tWv/9KlS3nxxRdZvHgxWmvGjRvHSSedRH5+PuvXr+e1117jr3/9Kz/5yU946623uOKKKw7xByXENRHIE9PmHISrZ/2uSu55fAYz06Dg+9cp+P51FnmGcphLZ2f9+yOe9jj+/pOTlvGSlSzzgqcXcW3yHO625mrffOY+AF71nAbA31K3kK2SONZlWi9bvhTSILX7AFhrVvUz+Fynw7bFZnv+M9D/RP/J3zYm0sI/QWu9wxL3uUqp78K9UCl1PXA9QN++fSM1vkMmWLplwJfKoaqqis8//5yLLrrId019vYnr/eyzz3jrrbcA+OlPf8odd9zR7P4fffQRN954Iykp5lfVpUuXZn3cfPrpp5x//vl06tQJgAsuuIBFixZxzjnn0L9/f0aMGAHAqFGjKC4uPtjHFhKFYBOQ0YI++HDO+pXvMjPtfr+2E5L9J2SvPLKa0ceP4ahvfk+n/ZvovG8NWItw35lSz/CFjm/+gdQXADjtiumgFCM+hMb0XrBzg6/PXcelwzdAlyOgy5GwcT4c45oknnw/fPM3GHimSTnd0iTzIRJR4dda77C2pUqpd4CxwC6lVE/L2u8JBC0PZX07eBZg9OjRLZfYCcMyjxTB0i0DPuH1er3k5eX55eZv6fpAtNYHlGJZt1CNKD093befnJwsrh6hdaJZ+G08rdSk/ecvYf92uPxNX1OnbQtbvW2PumJ6HNMdZv6t2bnhO605uCHnw+p3fO2TCspg65dQsQ4GnQM7nWuOTv4B0rKhUzeY/Hs4/hd++Xk4/hbzagci5uNXSnVSSmXb+8BkYBXwPnCV1e0q4L1IjaE9CJZu2U1OTg79+/fnzTfNH53WmuXLlwMwYcIEZs6cCcCrr74a9P6TJ09mxowZNDWZP+6ysjKAZimabU488UTeffddampqqK6u5p133uGEE05ogyeNIu7Nhfn3t95POHQCi4hEE+Fa/EtfhPUf+jVl7N/c+v33roeaMv+2zoeZ7fcfQOEY+PHz/ufn/hZm/9KMKcPfx8/O5VBwhLHmUzMhv1/rY4gQkZzc7QF8qpRaDnwF/Etr/W/gD8DpSqn1wOnWccwSLN1yIK+++irPP/88w4cPZ8iQIbz3nvmse/zxx3n66acZM2YMFRUVQe8/bdo0+vbtS1FREcOHD/cVW7/++us588wzfZO7NiNHjuTqq69m7NixjBs3jmnTpnHssce28VN3ILbP+ZM/dew4EoVoEP5A8bWxo3ncwq81zLwcvvtX8/72B8W2rzhs31L/c90HN+/vaYA1Lrv04lfgEpeBVjjGTL527uG0uT9g7NW4NjtXGDdPFKBacg1EC6NHj9ZLlizxa1u7di2DBg3qoBEZiouLmTp1alwlYIuGn2uLNDXA/d3M/r3BPyyFg0Br+OetMPTHcISrfsMPK2GG9S22I37eZZvgiWPhrIeh/0km4qaHJdL3FZjj8b+AKQ+Ytt3fw9OmHCrn/hmOvdx8QwS4Y4tZCfvpo/DRvf7vM+GX8NljznFWAdS4FltdNx96jzIhnH+wFrKd9TCMvQ7+dRt8/VfoPRq2L4GcQsjpBSdPh1cu8H+fE26DU++hvVBKLXWF0vuQlbtCbBGtmRhjnbJN8M3LsG4O/NqZkPTVh20rvB6TliBw3qp0LTx7Mlw71xQu6Xa0aa/cZbaf/Anm3Gb2swpMNIwdI++2+De7fPfv3WyE36bkaxhwOtTtx0MyyVhrbCb9BkZd5S/8mfn+wp93uNm63Te5hWY7+X7zoTD4HJh3n4nLHxIiSj1KLH7J1XMISLrlDsDbykSecHAUW4uQcnoZq3bPenPc1IauHq3hvi7w7zudtrr9ZpXrqrfNRPIzJxiL3XbvNJgUxlTtcq6p2QtLXnCOm+rhgztg+1IzserGHeP/6oXmvvX7qUnq5LRP/CV07m6s/hNvN21ZBc7542+BTv5h04Aj/KkZZkFWWic484+hRR+g4MjQ59qRmBb+WHBTxRIx8fNsLYJDODi2W+X9MnLhb+fAU5Z3wO3jL98KXwRfbxKUHd8aN0uJ5U+v32+2i/9itmtnG7fJHw+HT/7X79IHZn7ErKemG7FuhVXffQeLZ9Dw3JnsWucv/Le/4h+988LLz/HVd8WUe10pE5KtYPzTf2dSIg+/FM51Pef4EJE2tvC3RLdjzNYusygW/6GRkZHB3r17Y0OsYgCtNXv37iUjI6P1zh2JuHoig71YqHqPEWwwpQPdwv/qRfCfuxz3i42nCbZ91fye31sTneusidYqK3I7yRLakq9DDufurddx4Z6/hDX0/NotAKTpeno0lvid213qP9Z+exfhrS2nLqkTG3v9CF++H5uUNDh/BnQd4LR16hbwhv3NNpwEatfOhV+tNvewQzmjgJj18RcWFlJSUsLu3ZFb3ZZoZGRkUFgYhhXTkfhcPeGvbRCCsG8LvPwjuHo25PV1csO7V4vW7PWP47eFe8H9cMYfIS3LHH/6CCx4AK79CPqMcfonWXalnezMvj7Vsraze7bJo/TWP/g35PQ2cfvAixcfBS8Al70Ja97jlO/+CV2PhuRe8LNXwnuDpAD7eNpHULmz+TxFMDJyzKtzd0CFd007ELPCn5qaSv/+/Tt6GEJ7E42512ONb/4G71vui11rjPDXWMLvntCs2evUjQXnZ//N34xoT7rLuoeVw6Zsk7/w20VFPA3GTWT76W3hd98bzATp9oAwy9YYMLlZjD4/+TvMvtVEJFVsM22ZeWZF7LJXzDeNgWe1fu9frWk+RjD+/mA+/5Y48faI5d05GGLW1SMkKLbFHyWWU0wy+7+cfbtYSLUl+NqVm6dmr39UT2N18+sAUrOat7lZ8QY8OcrJp2ULvz1xazPsIkKSEsIFOfzS5m3dj4GxN5j9nWaxJLmFRuwHTDbH6dmh38smt7e/y+dQGHwOHHN229yrDRDhF2ILcfUcOu55ktoyszaivsK4QNzUlPlH9bg/FGzrf88G+G622Q/MJW8Le80e/8VQSanw4W9g0f9RobP457An4ajTzSsUaSHy1hxxMpz3FxhznatvJ0fYty02vvjsnsZlc/QZpr1uf+j3SgBE+IXYwhacWLH4q0qNPz0Y9ZVOlFLlLnjnRmioDt43UtTuc9w7gREnNXtDj8fbaFZRPzXKidYpD3jO+oCUImUbzbahGj5/EgAPSVT0PgmumGVSGBx1WvD3S+sUvD0zH0ZcBmcFrOS24+23LYYeQ5y/lx5Dg481wRDhF2ILn7UaI8L/8AB4vCj4uYcK4V0rxcei/4Plr8GyfwTvGylq9zn+/WbCv8cp+h2IpwlmBrhZNi9yLOmdK5zJ3EBcHwgZNJKXZUX5JKfAFW8FvyaUa8YWdKVg9DUw6W6rv2uhVS9XyhJ71W/RxcHvlyDE7OSukKDYESKxYvGHwn6OlW/Aj//qCFsEc7ADNHm8fv/0NWU7qF/7CfnAvvRC3HVQq/aUoJpqCWZr11buI610GX7TlfUV8Ic+1BVOJKPk05Bj0A1Vvo/tDBrIz0oL3vHM/4UPrAVVbldPbh/jsrEid3xMfdTZt4U/qwBO/R9Xezbcs8evJGIiIha/EFv4XD0x+KfraYRHh5rJzkAXSqYVE26HVUaIO9/2L1iUtXEO+QtNdM6tc51cPPt0Z1av+pav1wbPYrlu43pUVSmPN5lcNI83ne8715Lo79dZKJy1N0lK07Vzun+nu3bCb0ph3A0mBv6/1/nnpr/oJTj7Ybj0tdAP2nUAnP1/cMtSE5vvJjk19g2HQyQG/3uEhCaaJ3cD3SLu6lVam9DCim0mL407THDlLOdadzhlSyz8k0lTcIAU76kKee6SKU5ytrrux1KUuYdhBcEXSBYlbyFJaY4fdSxvTl1Fr/N+3+L77u4ykq+LfsfGIc1XwR7dI2DiNi0LUqwPg9xCU5Lw9N+bJGjTt0Fhs5xjzVEKxkwzcwBCM0T4hdgiWlfu7lptUg+scIp9+IU3NtY6k7z1lf4W/1vXwqKHzX5VwKrYUCy4HxbPOOBhNtSEzrB51sRxvv2eQyaSWb+HAo/L9WT7z4EkbVxVY4YXcdHoPlw0uk+L79stpxNjLvglx44c2+xcWIWGegyG6+Y1z3EvHBQi/EJs4YnSOH574dGGuU5blWtFaX2lE0lSXxl8YRCEzj0fisD+u9Y4eee/fRXeuNLv9Pjq+aHvlezye9vRL5U7TXz9+F+YVyB21koIXj3KXsRlT+j2He+kbAAoOCr0eISIIcIvxBbR6uqpt1woKS5/tdt6r68MsPhDCH99C/HlFSXw5QznvQBK1zj76z6Av4yHVVZkzHs3m9j5pgbwNKG1ZnLTx+zKCmNRUr+JzgRobqHJd2+naMg7HG75xlSf6uJaPX/672HqY/73sROZ2TH9aZ1MARMwZQuvCVh1K7QLiT21LcQe3kOM4/d6TF6Zsdcb3/GhsvBPsHGeEzJoR+uAv6unfr/L4t8f2uIPjH1389LZsK/Y32+9d6NZOLWv2MkyuWs1DHNltXx0CBw2lJrz/8ZQtYk1Xa+gx9b1LT9XZp5JobBtMSS7JkenbzUWe1pW8xTDSkHRT4zIf/gb05bbxzy3+8PqiJNg6+fQ5zjoVIDQ/ojFL8QWnkO0+DfMMzHztjAdKgvuh61fONZ8pcu9414dWl/prGytr4T9O4Lfr6HKzAd88WcngmnHMvj6eSPuAPtckTYN1WYtwMI/OvMGgRFP1aWwcT41mxeTpjxUdg8yOTroR83bTrImj3N6OW0ZuY7lH4y0Tv4uH/vafq5a1P2tSWSZeO0wxOIXYotDzdVjx34nh4gdPxBK1zr7thi73Tt1rolUt6tn5zLjhgnFggfMyta0LBh1NTx7kv/53eucffck8YaPzDZEqGvq8leo0enU9T4e3JVMs3uZerIAv97k/GyPOhX+37eQ2zf0WEOhkkF7TNz8L5b4567vexxc8BwcE0aiNCEiiMUvxBaH6uqxF0gdaHbFAEr21cCfj3OGVfqduf2eEn7z7kp+8+5KPl7plDB877NvjeUdButXLgbg/W+38Jt3Vvidq0nOhtVv+44/Wb2ZnRmWy8VKgjZ/3W5+865/vD5A3oZ3mOsdRXZuvqk/e/Ucc0K73FOdCiCri3Pc5Qj/Sd9wsTNRJiWbmPpUV+ETpaDootBpGISIIxa/EFsc6uSu7W45lAI+ezbwz1WKm1xNSZiY/UxPBf9ZsR2vSmaY1wmFPLzEJDLbRRd6EDxyx4siCU1SpXED9Sz5kPtLHvHrU+rJph/OPEBS6VqS8c+KecquF/mmNPjz7c/oxYTunSEzHXKsfPjueYm2wv7WkeArZKMVsfiF2OJgSi821sJnj5tFUrbwH2wytJWz4KlRdN0eJCwyvx9JaL4e+g5Lh73DxUNzTATMyCsZgflG0GNA6MVHSVZhkiPTzdzAGJrXc+6X6h+HP5Fv6E4ZpOf6td+mXwr6Hj89uchZKWtXkMoMo5LUgWKHccbiCusEQH4rQmzhPYiUDe/cCHN/a8Id66wVsoG54G22f+P40L0eMxHsjpUvMc7xHmVLml/bbZDZrpgJy141JQwzcpwc8ODExwcj34qJbymk086JP+luKHCFZeYfHrx/IO5ygVldYMqDcPmbofsfLG5XjxB1iPALsUVLk7ueJrNy1p0qwet18sU31TmRNoEW/2uXwkN94K+T4OmxJqJm0wKYdx/8+06nn+UPT64PUnSk+yD/473rjdDmu2Lde40I/WxDfxz6HJjsk5e8BiMuh5Nu9/8ZBBP+mxc3b8vw/2bA+J9HpgC4/cGsRPijEXHACbFFS6UXF/0ffPygSco1+Fxjsd/nmqisq3Aibex4+c2LoPdIWDfH/16/72qiXcA/f47lD89uCDJR6xb+zj1MhE96jr8oFwRZPKWSTL9BP4I5twV/tjuKTZ+kZCcapqneOZ8XIPzXLQj+YRAJt04wxOKPasTiF2KLwMndf98F6600CXbahC2fw+tXNI+Vr9/vuFEaqo2//+WpMPtXwd/LmmT1WdbrPvDlru/ZuM3pN2Ay5BRCr5FO2+QHzDYjxz+XvHulqx0nP/FXMH2LWVD2sw9MSGUgmfnNRdTT4OwHRillFTjlCrPdcfjtJPxi8Uc1YvEL0cvSl+GYqf6rO33Cr431/+XT5nVvhSPUdvKywHJ9Vbuc6xuqoMqKulnxesvjaKqH8m3w2iW+pm7aSp98R7GzEMn9bWTYhWbFqp1J8tynTSy7O6zRdje5i4YcfnzLYwkcl01qQGhkVoH5wLryPeg6EB45xrQHunoihvVhKRZ/VCLCL0Qn+3fCP/+fsdDdK0FtcfU2mQRiFnXFX5G+f6d/kOdyk6+9acRPSdq8EL1vm69wiLe+ksaKnQRkgg/O5oU0ffzHZv8sFWk9yHWvPk1ONXlojp5iRPdEl9vm2Cuc/R89bj4s7AVXB5tx0m3xp2bCDYvgmRPMsR0jf8TJ/te0l6vHRqJ6ohL5rQjRie2SCXTX2Ba718Ocz5zImoyXTqepOsiEK3D24iF8t09RvOl7wBQZqa3Yw7LXW84h7yZl2d+bte3JDVJScdpHcOKvW77ZqKvNHIQ9z5AeRPgvesm/clQwmuqcfaWgZ5H/sRvb7RPsvSKJWPxRiVj8gsneuPINGH4ZJEWJLWCLYgvCX76z2O9Uqgq+EOnySSPJW11A98rV4AVPdi86VX3POPWdr8/u7EF0q1zb7NrteWPIqSshu875dlGaPZjulWs4bHiIwuDh0qmb2bpz4dgMsSpadR8cOqdNbqGTv8f+JnTrcseF5eaGT8zcR7sJsbWATHz8UYkIvwCfPgIfP2SsQndWx44klMXvcvWk1+wkJH3Hm+RpwJWnjoQ9PaDiGwC69hsGq773695t+JnwaXPh7z3keOjUHT60ipBc9BLd+58E5Vvo1LOF0MxwOO13xqffkl9/4Bmhz101G545EWrLnA/E/H7mFUi3gebV3sjK3agkSsw7oUOxM0rWBneVdAi2xV8ZIO62X1t7SK8PYtnaPuWew5225FR/P3pgvD1A16Od/ZRMJ81yTqH/uT7HmYVPvY499GIwaVmOZX8w5PWBC541++GUI+wIxNUTlcjHseDQVlWt/ny8KY6dkg5Hn2EWCQVSshTKi0MvWnILv9fruKDsSBjtJb3BisnvdaxZJQsmaqV2n3GjXPiiU6jE7S7pd4Kzn5YNDZVG3FOzTKTM7Rvhy7+Ye+b0cnLaQPSV/htwuilMnhLWNHX7YedCksndqESEX3AIlbjs21ehzzjoGmaZvNLVzv7mT4IL/3OnmG1rwu9tMhk1s3uYlbmuVMeZTRXsyDiSXtd/DOv+Dcv/Yaz1FTNh0DnQ7WjgAtO5/4lOmKfbgrct0uwecPsm0F4TEWNHv+T29o+DT20hF31HEW2i70Ys/qgk4sKvlErGZP/errWeqpR6CTgJsP+Dr7Alx5gAAB6JSURBVNZaL4v0OBIST6NTlalFWshUqbXJHZ+eA3duC92vJSpK4Ju/w2FDmxf8cFvz7v7/nu4c799uhPn+7n4phLO95TSmWhb4wDPMq6Ha5J8JrOx0pPVBM+IKf+v/mKmw7BXI7OIfY3/4RLMwq+vA5imFhfCRyd2opD0s/luBtYD7O/Kvtdaz2uG9E5eN8+Hv58O0+VA4Krxr3AuCbGzXSkuJw1pjwYMmaRnAz7/yn2SsK/fP/95UD3Nu97/e9vNr/6idPF2JJ63Qv29ap+B53lMz4fbNZhWtUnDBX60Jz0Fw8vTmVaW6HxOZ5GWJhlj8UUlEhV8pVQicDTwA/Fck30sIwE5jsPWL8IU/WKriA0xfXF7TQLMlQrboA/uev5CXRr2FnSThpblL2JfVz3f+rHV3MXDPXL/L53/1Lcu3DSAwsUK+qmTPgaQgcH/AFP3E2c/rE/49hANDLP6oJNIW/2PA7UB2QPsDSqnfAvOA6VrrZqamUup64HqAvn0PovRbouObXAvDNWFb+o3BhN9KXxzmP/CclT9wWQvn8+u28ty8FfzKWk/0ry9XUso27k55lVKdx8CUeX79G3Uya79fx+Nr1vuusclRtdTmtVOx7sveDF0gXQiNWPxRScSm3JVSU4FSrfXSgFN3AscAY4AuwB3BrtdaP6u1Hq21Ht2tW7dIDTOOsf32YQi/Le7BrHt7klV7YMsXrd5qX40rjUB2wMIkK23B6psdC/vNnw5g4fErmZy8lCsCRB+VRGqXPvy83w8UPzDFaU9xPgF6dDus1TG1CUdPhiHntc97xQUS1RPNRPK3MgE4RylVDMwETlFKvaK13qkN9cCLwNgIjiH+Kd9qKkwFoq2c9OH84zVYlmxQV4+rYMmLQRYTffIneOVCWPM+bFrI8HVPOOdyA/zvR0wy2xdchUl2r4M965vft+/xMH0bTPgllHwF3//bOecuFeieeBWiD1nAFZVE7Leitb4TY92jlDoZuE1rfYVSqqfWeqdSSgHnQZD6ckL4PDbMRKz89B1zvOY9I+AH4uqxBT+oxR+iUpXN/PvNdoPxy090n8stNBE5+7eb457DjcvIPUm74P7m9zzxdjjhv4yoDz4XZv8StnzmnPe6smCGqqQlRAfi6olKOuJ72KtKqZXASqArEOQ/XwgLu/7sRlf91zeuhHdvIixXj9amYpUdNbN9KXx0r1PBSmtY+lLo65f9o+Xx5faGW1yevs7dW08LPO5GOPlOx5LP6gK5fU2eGRu7EtVp98JxNwW7ixAtyORuVNIuwq+1/lhrPdXaP0VrPUxrPVRrfYXWWky2A6V2H6x6CzxBwi9tfK6eFoR/7fvw9jSTNx6MZf7po7DR8rVvmAfffxD6+ndbEd3cPkbApzxkFj4FZobscmTzawrHNI/r7zUcdrqWeiSnmSpUE38VOoGZEB2IxR+VyMxLLPLmz2DWNbB3Y+g+oVbhulnzXoj2d617BM92GTZ9xpnt+Jvh7p3NP4R+NgdOvsu/LT0wAAw4LmDlb+cehzYuof0Qiz8qEeGPRUqtLJItJVULx+Iv3xq83c6I6S70YWO7l1r7YLnktZYLi9+wyJQaDCwM0ilIBNfh4539cTfB8f+v5fcWoodoSfMt+CFT7rFIkxXFUx2QnbLRVZjD9vGHEOjl28rJKdlO/yDnVm/YzI/v+YBz1ef8MeD/duy977CfzqTTwPKWjLnUjODtmfkmjXDn7ubYzptz+n0m2VrvkcGvsxl5JSTLn23UoyUffzQjH8exiC3wVaX+7XXlzr79j+euA+ti9Y795OhK59Jkpz7twKQSfjUCTu3fPFRy+sBSlqdewy3DW/nTSQkRZnnZGzDpbsddc+QkuOZDGH+LSaQWCjssMFg6BiF6ER9/VCLCH4vYk7rVAcLvdv3Yq3F9xcn9Ka+pIw9nXj2jaz/ffopu5IaVlzD5SCt/zcCzfOcu2HAX6d4apvXc1PIYQ8XXdz0KTrrd3wXVd1zrLoGii8022ByAEL2IxR+ViPDHMoEl9tzCb6cXaKoz2S4DqKusIFm53ECBi63ArNpNTgu+CMeO758YmEHHoq2zWP7ocZPgzZ1vR4hirL8tWcAVlYjwxxruVbqBFn/NXlc/S/gXPACPDvHLYw9QX7XH/9rcIInKKkqMhW1PFLuxQ0A7t1PKhOTUjikdKBwaMrkblchvJdZwi/v6D/3P7Vrj7AeuuN27AX5wFkl7qgMigoJlqFw1C1DBhX+fLfzdm5+b8hAcVtS8XUg8xNUTlYjwxxpu4Q/ELj8IzfPn//UUmDHBd3jWnhf8z+f0Ntu0AB96zR7/3DijrjZb2+IP5iIaf7MULBEMMrkblYjwxxrVe4K3e70m5YKdnz7AteOjoQY8TYys/9ocX/MhXPq6s6o2t9D40t3YC7kue8P42guOctI8ZBXAb8ucvjd/eeDPJMQvYvFHJSL8MUSTx8vi1SaT5Y6C4/zOee7vAdWlbOhihUTawhzAv+bN450FJuHZwq6XmIiagWe4wiR1c/eNbzGY9U/snlRO6+xv1XUfdMDPJcQxYvFHJSL80cQ7N8LfQud8/6q4jA++MoXMp26/2u9csrcBj1Y8UNzyBOjZi6/gtEUmNLKul+vDwxZ+7TXfGk65B378PNzyjcmqCabuLZjMmYHXTZsPv1jS8vMJiYdY/FGJCH80sfw12LQAtn0V9PTeqga6qP1olcTcu85tdl5lFfDotDNbfZtsZSKDphznSqlgFzfR2vjnT7wNhl0IBUfCpN8YYT9smOkz8Zcw9VFznGrF+heOgq4Dwn9WITGQqJ6oRH4r0UjFtqDN5bWNdKESnZFPQU5Ws/NJmTnk5bSS9thNdk9n3zcZGyTFQ3JK87q9o6+BGz+Vf2yhZcTij0rkvzYa8TSaylaf/MmveX91DccnrYaCI4Jfl54TOkcOwPhfwDlPOsfuhGj5/aHfCXDu04cwcEGwsFOGiI8/KhHhjxbcydSa6k1lq/n+NWo6l37DEUk/kDTuRv9rbXdLeraJyjn/2eDvkZ5tBB6sFbmuf8qUNLh6NvQ9Lvi1gnBAyMrdaEaEvz2o3tt6GmP3ilx3OuQvHAs8ueoHs2P72m1Onm629j/ZsAuDv0d6NhSONkXPJexSaA/E1ROViPBHmtK18KcjWi5hCP61Y9158v9zl2+VbHKttXgrq6v/tXbiMttPH+rrdXq2SZ527tNm0lYQIo24eqKSsIRfKXVrOG1CEPZ8b7Z2OcNQ1Dspkn3X2Pz9fPjP3aTV78FLklNu0Lam7NW2yvXrvPYj6BQQjy+ZLYX2RlZwRyXhOuCuAh4PaLs6SJtgozX8Lg96DA2vv8vi3/f95/hVki3bCF88xYk6l8qUXHLtSJpbl5tEarX2ylnXP1mfMdDlCP9EblKyUBAEWhF+pdSlwGVAf6XU+65T2UALSWPiCK8XXrsYxt0IR50a/nW2Bb9rVcv9fP0d4c/HybOzJ7U32zIHcuz++XRTFdRkuxZo5fUxr+/mmONA6yo51WwvexPSO0Pf8QhCu5DT26QNUeJNjkZas/g/B3YCXYH/c7VXAisiNaioorHGZMHcvAh+80P417WUTC2Q8q1+k7iAqS2b1omuY6+ja/Zh8IfDoa6crPwgVrsvpULAP5k92Xv48Ub4BaG9uOJtKF4EGQewrkRoN1oUfq31FmALkLimYogKVq1SU9Z6H5sP74F1//Jvy+piKlXZdO5hSisGK0bebyIUDICT7/RvT0414i/lCoX2JqcnFP2ko0chhCAsH79SqhJnSWcakApUa61zIjWwqMEOrTzQSaraAxD+7UubtwWWLsy0sm52CRKNk5kHtwTJk5OUavLuyASbIAguwnLAaa2ztdY51isD+DHwVGSHFiW4Y+rDvqYJ5v8++LkN8/wjeKp2Q8U2KsbdxjUNt+FR1mdxSsAKXPuaA6lClZ4N2e1UIUsQhJjhoJbVaa3fVUpNb+vBRCV20fIDYeN82Lncv23tP2HV2zDrZyYc845i0/6l8e3v6XkS872VJNm571MDcvHY9XS7Hh3+OE67FxqqD3T0giDEOeG6ei5wHSYBowmazSsO8TQe+DV2NE0gs35mtrX7oKnBpElY/S4MmMLOToOBxSj7xxro6in6CXxmFUEJl5yerfcRBCHhCNfi/5FrvwkoBprnBY5HPAdu8evGGlrzqi/84jMqcwdyenU5O/PyWbAuoHB6oMV/6v/ACf8Nac2zcgqCIBwIYQm/1vpnkR5I1NJ04D7+ktIy+gDPNJ3NDSn/CtrnpHnn8Y33KNKT9vHB+mqeX7uZ1GTXx0WgxZ+ULKFxgiC0CeGmbDhCKfVPpdRupVSpUuo9pVSI3MBxxkFM7lZVmnq3w06+qMV+I5M2AHDR8YOZ+6sT+Wz6Kc7JQItfEAShjQh3Wd0/gDeAnkAv4E3gtUgNql3Z8FHLfvxwXD2NdVC22XdYX2smVA/v0zesIXQt6MqAHtl0z3ZF8gRa/IIgCG1EuMKvtNZ/11o3Wa9XiIfJ3c2L4JUfw8cPhe4TzuTu+7+AJ0aYDwCgsc4If+f87i1d5RAseZoIvyAIESJc4V+glJqulOqnlDpcKXU78C+lVBelVJdIDjCiVO82270bQ/cJJ5zTzpVTtQsAT53Ju9M5WHqFYAQVfnH1CIIQGcKN6rnY2t4Q0H4NxvKPcX9/C19ewvHx2zlxKn+A/MPxNNRQSzqZqWnhvX1GkAXQYvELghAhwhX+QVrrOneDUiojsC0YSqlkYAmwXWs9VSnVH5gJdAG+AX6qtT6I5bFtQDipDMISfisvfuVOALwNNdSRQdjSLa4eQRDakXBdPZ+H2RaMW4G1ruM/Ao9qrQcA+4Brw7xP5GipLOKBWvwADdU0JKUH7xtsAVZ6EIs/OcxvC4IgCAdIi8KvlDpMKTUKyFRKHauUGmm9TgZadUIrpQqBs4HnrGMFnALMsrq8DJx3COM/RGyLvwXhbyGOf9faz+HeXF+xk7kLPuKcpz6lprqSpqSM5hdMmweXvdG8PVj2TEmsJghChGjN1TMFU2mrEHjE1V4J3BXG/R8DbscUbgEoAMq11nau4xKgd7ALlVLXA9cD9O0bXljkAXOIrp6aL1/wOz69fi6fdrmAw7I0nYL57XuPCv6ema56W1fNhm1SCF0QhMjRosWvtX5Zaz0JuFprPcn1Okdr/XZL1yqlpgKlWmt3zuFgShvU3NZaP6u1Hq21Ht2tW5Ac9G1Ji64eV1TP2tnwr9t8h9XaZdUPMemMftfpLYbXfkVeqlUcJcXlqw8m+vdWQIrLLdT/BDjx1wcyekEQhAMi3MndoUqpIYGNWuv7WrhmAnCOUuosIAPIwXwDyFNKpVhWfyGw4wDH3IaEY/G74vhfv9xsR10NfzuHnFRXMNOIy6BkiVkQBlC6xmynbzUVsuwqWWDEvrHu4Iu8CIIgHALhTu5WAdXWywOcCfRr6QKt9Z1a60KtdT/gEmC+1vpyYAFwodXtKuC9Ax92G+EW41DYcfyeRuhs5bb/z11Qs5e+FV87/boNhF4jnOOBZ5ttShqkZjRPrpaaIeUQBUHoEMJN0uaut4tS6mHg/RDdW+MOYKZS6n7gW+D5g7zPoWNb3OFE9XgbTWHzqh9g88Lm/XIKIavA7KdkwMWvtO1YBUEQ2oiDKsSCiegJe9GW1vpj4GNrfxMw9iDft23xuVrCDOes2QvJ6cHz9yQlORZ8Zr45FgRBiELCzc65Uim1wnqtAtYBT0R2aO1Aaxa/pxGWvuQcV5Wagii3rvAVPS9OPQru3G7O2/H4EoMvCEIUE67FPxXIB04A8oA5AdE6sYlv4jaE8G/7yt/ib6hie30mW8s6c0zuYPKrF1Kd1tWx9H0rcGM/f50gCPFLuP6Ic4G/A12BVOBFpdQtERtVe9FaVE1dOQBbvU446avLK7j0r1/y0VYj7todr28Lf0tzBoIgCB1MuMI/DThOa/0/WuvfAuOB6yI3rHbCaxU2DyXUdfsBeC/DqTL544nDeO264xg3egwAAwcMdPr7hD+MaCFBEIQOIlxXj8KEcdp4CCsIPsrxhsi17/XCf+70uYLqsw+HvebUkX16c+SRBdDvLhh9NqndXMKfZrl8RPgFQYhiwhX+F4HFSql3rOPz6MgwzLYiVFRPdSksnuE7rOncxyf8ZPc02+QUKBzlf509uSvCLwhCFBNuHP8jSqmPgYkYS/9nWutvIzmwdsETIqrHVXylgRRUzmHOubwW8gaJj18QhBgg7Dh+rfU3mPz58YPb4t9XbHz+BUdCY62vS7XOILOTK4la9mGEJF1cPYIgRD+JvcrIFn5vE8z+L3j1QpNT/40rfV2yqCOvkysu3y66Egw7vXLXAREYrCAIQttwsCt34wN7ctfTBLW7oWwT/PUU2L/d1yVdNZGbmRre/TLz4dLXoXBMBAYrCILQNiS48FuBSt5Gk44B/ETfJj8rDa6e0zzRWjAGntGGAxQEQWh7Elv47ZW7HpfwB6GoMBdyJrTToARBECKL+PgBasuM1T/qZzDsJ77TmiQYfS3dc4KUURQEQYhRRPgBKneZbZ+xcNb/+k4vvWAhTH0kyIWCIAixiwg/QJMVvplV4Ky+BbI753bAoARBECKLCL+brAJIdiJ4crKlQpYgCPFHYgn/Z0/Avbmw6WNY/npz4S84yu8wNycbQRCEeCOxonq+fs5s/2Zl2xx0jnPuirchM8+ve2ZamPH7giAIMURiCX/gqluPk51zY8qReHZVAnC01aZU7CcgFQRBCCTBhN//ceur9pIOLPMeyXkzVvvaiyV6UxCEOCaxhF/5W/y6spRvvEfx8Qmv8XQPlz//7XYelyAIQjuSWMIfYPEn1+6hid5MHtyDob1doZuLR8EPq9p5cIIgCO1Dggm/v8Wf2lSFRyc3T8I2bV47DkoQBKF9SRzh3/AR7FzWrLmRZPKyAoRfJnUFQYhjEieO/5UfB21+yXsWndMT5/NPEAQhoRWvKiWfFSljJGxTEISEInGF//K3uO9zyNsri7QEQUgsEsfVE8B/f9uNhdshL9zqWoIgCHFCwgr/l5v2kpKUxKRjunf0UARBENqVhHX1fDb9lI4egiAIQoeQGBa/1v7HUx7qmHEIgiBEAYkh/I01vt1NSf1g/M0dNxZBEIQOJjGEv77Kt5uW5O3AgQiCIHQ8iSH8DY7wpypPBw5EEASh44mY8CulMpRSXymlliulViulfme1v6SU2qyUWma9RkRqDD7qK327aYjwC4KQ2EQyqqceOEVrXaWUSgU+VUp9YJ37tdZ6VgTf2x+XxZ8iwi8IQoITMYtfG2zFTbVeuoVLIkdNmW/3y4G3d8gQBEEQooWI+viVUslKqWVAKTBXa73YOvWAUmqFUupRpVR6iGuvV0otUUot2b1796ENpGoXAGPq/szYs352aPcSBEGIcSIq/Fprj9Z6BFAIjFVKDQXuBI4BxgBdgDtCXPus1nq01np0t27dDm0g1bvxoujbpw+5gSmYBUEQEox2ierRWpcDHwNnaK13Wm6geuBFYGzEB1C1i/1JuaSlpkX8rQRBEKKdSEb1dFNK5Vn7mcBpwHdKqZ5WmwLOAyJf47BqN/tUHhmpiRG9KgiC0BKRjOrpCbyslErGfMC8obWerZSar5TqBihgGXBjBMdgqNpFGXlkpCa33lcQBCHOiZjwa61XAMcGaW//7GjVu9nL4SL8giAIJMzK3Wr2ezNE+AVBEEgU4W+qo9qbJj5+QRAEEkH4tYbGGqq9KWLxC4IgkAjC72kE7aXKm0ZGigi/IAhC/Au/lYu/HnH1CIIgQCIIf1MdAHWkkZkmFr8gCEL8C79l8ddqcfUIgiBAQgi/Y/Gni6tHEAQhEYS/FoBa0iWqRxAEgUQQ/iYj/HWkifALgiCQCMJvWfx1Oo2MlPh/XEEQhNaIfyVsdCz+nEzJxS8IgpAwwl9LGvlZko9fEAQh/oW/yXH15En1LUEQhAQQfsvi96ZkyuSuIAgCiSD8JUsAyMjM6uCBCIIgRAfxLfzVe2D12+xO6UlWVueOHo0gCEJUEMnSix1P9R7QXl7tfBW5mTKxKwiCAPFu8ddVALCpMkUmdgVBECziWvgryvcCsK0mlUE9czp4NIIgCNFBXLt6mqr3AXDNaSP40alHd/BoBEEQooO4tvi9lqsntVNeB49EEAQheohr4dd1+81Ourh5BEEQbOJa+KmroEEnk5ImMfyCIAg28S389fupJItUWbErCILgI66FP6mugv06i9Rk1dFDEQRBiBriW/gbKqkki7TkuH5MQRCEAyKuFXHT4Jt5qOkyUkX4BUEQfMR1HP+evCK+8DaJ8AuCILiIa0Vs8GgA0lLExy8IgmAT18Lf2OQFEItfEATBRVwrYqPHCH+aFFkXBEHwEdeKaAu/WPyCIAgOEVNEpVSGUuorpdRypdRqpdTvrPb+SqnFSqn1SqnXlVIRS5Rv+/hF+AVBEBwiqYj1wCla6+HACOAMpdRxwB+BR7XWA4B9wLWRGoDP1SPCLwiC4CNiiqgNVdZhqvXSwCnALKv9ZeC8SI3BmdyVqB5BEASbiJrCSqlkpdQyoBSYC2wEyrXWTVaXEqB3iGuvV0otUUot2b1790G9f6PHi1KQnCTCLwiCYBNR4ddae7TWI4BCYCwwKFi3ENc+q7UerbUe3a1bt4N6/waPJjU5CaVE+AVBEGzaxfmttS4HPgaOA/KUUvaK4UJgR6Tet9HjFf++IAhCAJGM6ummlMqz9jOB04C1wALgQqvbVcB7kRpDo8crMfyCIAgBRDJXT0/gZaVUMuYD5g2t9Wyl1BpgplLqfuBb4PlIDaDR45WJXUEQhAAiJvxa6xXAsUHaN2H8/RGnoUlLDL8gCEIAca2K4uMXBEFoTlyronH1xPUjCoIgHDBxrYoNTV5SJSWzIAiCH3FdiGXk4flU1Te13lEQBCGBiGvh//mkozp6CIIgCFFHXLt6BEEQhOaI8AuCICQYIvyCIAgJhgi/IAhCgiHCLwiCkGCI8AuCICQYIvyCIAgJhgi/IAhCgqG0DloAK6pQSu0Gthzk5V2BPW04nFhAnjkxkGdODA7lmQ/XWjcrYRgTwn8oKKWWaK1Hd/Q42hN55sRAnjkxiMQzi6tHEAQhwRDhFwRBSDASQfif7egBdADyzImBPHNi0ObPHPc+fkEQBMGfRLD4BUEQBBci/IIgCAlGXAu/UuoMpdQ6pdQGpdT0jh5PW6GUekEpVaqUWuVq66KUmquUWm9t8612pZR6wvoZrFBKjey4kR8cSqk+SqkFSqm1SqnVSqlbrfZ4fuYMpdRXSqnl1jP/zmrvr5RabD3z60qpNKs93TreYJ3v15HjPxSUUslKqW+VUrOt47h+ZqVUsVJqpVJqmVJqidUW0b/tuBV+pVQy8DRwJjAYuFQpNbhjR9VmvAScEdA2HZintR4AzLOOwTz/AOt1PfCXdhpjW9IE/LfWehBwHPBz63cZz89cD5yitR4OjADOUEodB/wReNR65n3AtVb/a4F9WuujgEetfrHKrcBa13EiPPMkrfUIV7x+ZP+2tdZx+QLGA/9xHd8J3NnR42rD5+sHrHIdrwN6Wvs9gXXW/jPApcH6xeoLeA84PVGeGcgCvgHGYVZwpljtvr9x4D/AeGs/xeqnOnrsB/GshZbQnQLMBlQCPHMx0DWgLaJ/23Fr8QO9gW2u4xKrLV7pobXeCWBtu1vtcfVzsL7OHwssJs6f2XJ5LANKgbnARqBca91kdXE/l++ZrfMVQEH7jrhNeAy4HfBaxwXE/zNr4EOl1FKl1PVWW0T/tuO52LoK0paIsatx83NQSnUG3gJ+qbXer1SwRzNdg7TF3DNrrT3ACKVUHvAOMChYN2sb88+slJoKlGqtlyqlTrabg3SNm2e2mKC13qGU6g7MVUp910LfNnnmeLb4S4A+ruNCYEcHjaU92KWU6glgbUut9rj4OSilUjGi/6rW+m2rOa6f2UZrXQ58jJnfyFNK2Qab+7l8z2ydzwXK2nekh8wE4BylVDEwE+PueYz4fma01jusbSnmA34sEf7bjmfh/xoYYEUEpAGXAO938JgiyfvAVdb+VRg/uN1+pRUNcBxQYX+FjBWUMe2fB9ZqrR9xnYrnZ+5mWfoopTKB0zATnguAC61ugc9s/ywuBOZrywkcK2it79RaF2qt+2H+X+drrS8njp9ZKdVJKZVt7wOTgVVE+m+7oyc2IjxpchbwPcY3endHj6cNn+s1YCfQiLEArsX4NucB661tF6uvwkQ3bQRWAqM7evwH8bwTMV9nVwDLrNdZcf7MRcC31jOvAn5rtR8BfAVsAN4E0q32DOt4g3X+iI5+hkN8/pOB2fH+zNazLbdeq22divTftqRsEARBSDDi2dUjCIIgBEGEXxAEIcEQ4RcEQUgwRPgFQRASDBF+QRCEBEOEXxAEIcEQ4RcEQUgw/j+AU47ITQWW+QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values\n",
    "\n",
    "# Create train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam') # Modify here\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=0,epochs=1000)\n",
    "\n",
    "# Plot the chart\n",
    "pred = model.predict(x_test)\n",
    "chart_regression(pred.flatten(),y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}