t81_558_class_04_3_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_3_regression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_1_feature_encode.ipynb)\n",
    "* Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb)\n",
    "* **Part 4.3: Keras Regression for Deep Neural Networks with RMSE** [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.3: Keras Regression for Deep Neural Networks with RMSE\n",
    "\n",
    "We evaluate regression results differently than classification.  Consider the following code that trains a neural network for regression on the data set **jh-simple-dataset.csv**.  We begin by preparing the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values\n",
    "\n",
    "# Create train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create a neural network to fit the data we just loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 1s - loss: 1905.4454 - val_loss: 1628.1341\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1331.4213 - val_loss: 889.0575\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 554.8426 - val_loss: 303.7261\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 276.2087 - val_loss: 241.2495\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 232.2832 - val_loss: 208.2143\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 198.5331 - val_loss: 179.5262\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 169.0791 - val_loss: 154.5270\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 144.1286 - val_loss: 132.8691\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 122.9873 - val_loss: 115.0928\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 104.7249 - val_loss: 98.7375\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 89.8292 - val_loss: 86.2749\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 77.3071 - val_loss: 75.0022\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 67.0604 - val_loss: 66.1396\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 58.9584 - val_loss: 58.4367\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 51.2491 - val_loss: 52.7136\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 45.1765 - val_loss: 46.5179\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 39.8843 - val_loss: 41.3721\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 35.1468 - val_loss: 37.2132\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 31.1755 - val_loss: 33.0697\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 27.6307 - val_loss: 30.3131\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 24.8457 - val_loss: 26.9474\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 22.4056 - val_loss: 24.3656\n",
      "Epoch 23/1000\n",
      "1500/1500 - 0s - loss: 20.3071 - val_loss: 22.1642\n",
      "Epoch 24/1000\n",
      "1500/1500 - 0s - loss: 18.5446 - val_loss: 20.4782\n",
      "Epoch 25/1000\n",
      "1500/1500 - 0s - loss: 17.1571 - val_loss: 18.8670\n",
      "Epoch 26/1000\n",
      "1500/1500 - 0s - loss: 15.9407 - val_loss: 17.6862\n",
      "Epoch 27/1000\n",
      "1500/1500 - 0s - loss: 14.9866 - val_loss: 16.5275\n",
      "Epoch 28/1000\n",
      "1500/1500 - 0s - loss: 14.1251 - val_loss: 15.6342\n",
      "Epoch 29/1000\n",
      "1500/1500 - 0s - loss: 13.4655 - val_loss: 14.8625\n",
      "Epoch 30/1000\n",
      "1500/1500 - 0s - loss: 12.8994 - val_loss: 14.2826\n",
      "Epoch 31/1000\n",
      "1500/1500 - 0s - loss: 12.5566 - val_loss: 13.6121\n",
      "Epoch 32/1000\n",
      "1500/1500 - 0s - loss: 12.0077 - val_loss: 13.3087\n",
      "Epoch 33/1000\n",
      "1500/1500 - 0s - loss: 11.5357 - val_loss: 12.6593\n",
      "Epoch 34/1000\n",
      "1500/1500 - 0s - loss: 11.2365 - val_loss: 12.1849\n",
      "Epoch 35/1000\n",
      "1500/1500 - 0s - loss: 10.8074 - val_loss: 11.9388\n",
      "Epoch 36/1000\n",
      "1500/1500 - 0s - loss: 10.5593 - val_loss: 11.4006\n",
      "Epoch 37/1000\n",
      "1500/1500 - 0s - loss: 10.2093 - val_loss: 10.9751\n",
      "Epoch 38/1000\n",
      "1500/1500 - 0s - loss: 9.8386 - val_loss: 10.8651\n",
      "Epoch 39/1000\n",
      "1500/1500 - 0s - loss: 9.5938 - val_loss: 10.5728\n",
      "Epoch 40/1000\n",
      "1500/1500 - 0s - loss: 9.1488 - val_loss: 9.8661\n",
      "Epoch 41/1000\n",
      "1500/1500 - 0s - loss: 8.8920 - val_loss: 9.5228\n",
      "Epoch 42/1000\n",
      "1500/1500 - 0s - loss: 8.5156 - val_loss: 9.1506\n",
      "Epoch 43/1000\n",
      "1500/1500 - 0s - loss: 8.2628 - val_loss: 8.9486\n",
      "Epoch 44/1000\n",
      "1500/1500 - 0s - loss: 7.9219 - val_loss: 8.5034\n",
      "Epoch 45/1000\n",
      "1500/1500 - 0s - loss: 7.7077 - val_loss: 8.0760\n",
      "Epoch 46/1000\n",
      "1500/1500 - 0s - loss: 7.3165 - val_loss: 7.6620\n",
      "Epoch 47/1000\n",
      "1500/1500 - 0s - loss: 7.0259 - val_loss: 7.4933\n",
      "Epoch 48/1000\n",
      "1500/1500 - 0s - loss: 6.7422 - val_loss: 7.0583\n",
      "Epoch 49/1000\n",
      "1500/1500 - 0s - loss: 6.5163 - val_loss: 6.8024\n",
      "Epoch 50/1000\n",
      "1500/1500 - 0s - loss: 6.2633 - val_loss: 7.3045\n",
      "Epoch 51/1000\n",
      "1500/1500 - 0s - loss: 6.0029 - val_loss: 6.2712\n",
      "Epoch 52/1000\n",
      "1500/1500 - 0s - loss: 5.6791 - val_loss: 5.9342\n",
      "Epoch 53/1000\n",
      "1500/1500 - 0s - loss: 5.4798 - val_loss: 6.0110\n",
      "Epoch 54/1000\n",
      "1500/1500 - 0s - loss: 5.2115 - val_loss: 5.3928\n",
      "Epoch 55/1000\n",
      "1500/1500 - 0s - loss: 4.9592 - val_loss: 5.2215\n",
      "Epoch 56/1000\n",
      "1500/1500 - 0s - loss: 4.7189 - val_loss: 5.0103\n",
      "Epoch 57/1000\n",
      "1500/1500 - 0s - loss: 4.4683 - val_loss: 4.7098\n",
      "Epoch 58/1000\n",
      "1500/1500 - 0s - loss: 4.2650 - val_loss: 4.5259\n",
      "Epoch 59/1000\n",
      "1500/1500 - 0s - loss: 4.0953 - val_loss: 4.4263\n",
      "Epoch 60/1000\n",
      "1500/1500 - 0s - loss: 3.8027 - val_loss: 4.1103\n",
      "Epoch 61/1000\n",
      "1500/1500 - 0s - loss: 3.5759 - val_loss: 3.7770\n",
      "Epoch 62/1000\n",
      "1500/1500 - 0s - loss: 3.3755 - val_loss: 3.5737\n",
      "Epoch 63/1000\n",
      "1500/1500 - 0s - loss: 3.1781 - val_loss: 3.4833\n",
      "Epoch 64/1000\n",
      "1500/1500 - 0s - loss: 3.0001 - val_loss: 3.2246\n",
      "Epoch 65/1000\n",
      "1500/1500 - 0s - loss: 2.7691 - val_loss: 3.1021\n",
      "Epoch 66/1000\n",
      "1500/1500 - 0s - loss: 2.6227 - val_loss: 2.8215\n",
      "Epoch 67/1000\n",
      "1500/1500 - 0s - loss: 2.4682 - val_loss: 2.7528\n",
      "Epoch 68/1000\n",
      "1500/1500 - 0s - loss: 2.3243 - val_loss: 2.5394\n",
      "Epoch 69/1000\n",
      "1500/1500 - 0s - loss: 2.1664 - val_loss: 2.3886\n",
      "Epoch 70/1000\n",
      "1500/1500 - 0s - loss: 2.0377 - val_loss: 2.2536\n",
      "Epoch 71/1000\n",
      "1500/1500 - 0s - loss: 1.8845 - val_loss: 2.2354\n",
      "Epoch 72/1000\n",
      "1500/1500 - 0s - loss: 1.7931 - val_loss: 2.0831\n",
      "Epoch 73/1000\n",
      "1500/1500 - 0s - loss: 1.6889 - val_loss: 1.8866\n",
      "Epoch 74/1000\n",
      "1500/1500 - 0s - loss: 1.5820 - val_loss: 1.7964\n",
      "Epoch 75/1000\n",
      "1500/1500 - 0s - loss: 1.5085 - val_loss: 1.7138\n",
      "Epoch 76/1000\n",
      "1500/1500 - 0s - loss: 1.4159 - val_loss: 1.6468\n",
      "Epoch 77/1000\n",
      "1500/1500 - 0s - loss: 1.3606 - val_loss: 1.5906\n",
      "Epoch 78/1000\n",
      "1500/1500 - 0s - loss: 1.2652 - val_loss: 1.5063\n",
      "Epoch 79/1000\n",
      "1500/1500 - 0s - loss: 1.1937 - val_loss: 1.4506\n",
      "Epoch 80/1000\n",
      "1500/1500 - 0s - loss: 1.1180 - val_loss: 1.4817\n",
      "Epoch 81/1000\n",
      "1500/1500 - 0s - loss: 1.1412 - val_loss: 1.2800\n",
      "Epoch 82/1000\n",
      "1500/1500 - 0s - loss: 1.0385 - val_loss: 1.2412\n",
      "Epoch 83/1000\n",
      "1500/1500 - 0s - loss: 0.9846 - val_loss: 1.1891\n",
      "Epoch 84/1000\n",
      "1500/1500 - 0s - loss: 0.9937 - val_loss: 1.1322\n",
      "Epoch 85/1000\n",
      "1500/1500 - 0s - loss: 0.8915 - val_loss: 1.0847\n",
      "Epoch 86/1000\n",
      "1500/1500 - 0s - loss: 0.8562 - val_loss: 1.1110\n",
      "Epoch 87/1000\n",
      "1500/1500 - 0s - loss: 0.8468 - val_loss: 1.0686\n",
      "Epoch 88/1000\n",
      "1500/1500 - 0s - loss: 0.7947 - val_loss: 0.9805\n",
      "Epoch 89/1000\n",
      "1500/1500 - 0s - loss: 0.7807 - val_loss: 0.9463\n",
      "Epoch 90/1000\n",
      "1500/1500 - 0s - loss: 0.7502 - val_loss: 0.9965\n",
      "Epoch 91/1000\n",
      "1500/1500 - 0s - loss: 0.7529 - val_loss: 0.9532\n",
      "Epoch 92/1000\n",
      "1500/1500 - 0s - loss: 0.6857 - val_loss: 0.8712\n",
      "Epoch 93/1000\n",
      "1500/1500 - 0s - loss: 0.6717 - val_loss: 0.8498\n",
      "Epoch 94/1000\n",
      "1500/1500 - 0s - loss: 0.6869 - val_loss: 0.8518\n",
      "Epoch 95/1000\n",
      "1500/1500 - 0s - loss: 0.6626 - val_loss: 0.8275\n",
      "Epoch 96/1000\n",
      "1500/1500 - 0s - loss: 0.6308 - val_loss: 0.7850\n",
      "Epoch 97/1000\n",
      "1500/1500 - 0s - loss: 0.6056 - val_loss: 0.7708\n",
      "Epoch 98/1000\n",
      "1500/1500 - 0s - loss: 0.5991 - val_loss: 0.7643\n",
      "Epoch 99/1000\n",
      "1500/1500 - 0s - loss: 0.6102 - val_loss: 0.8104\n",
      "Epoch 100/1000\n",
      "1500/1500 - 0s - loss: 0.5647 - val_loss: 0.7227\n",
      "Epoch 101/1000\n",
      "1500/1500 - 0s - loss: 0.5474 - val_loss: 0.7107\n",
      "Epoch 102/1000\n",
      "1500/1500 - 0s - loss: 0.5395 - val_loss: 0.6847\n",
      "Epoch 103/1000\n",
      "1500/1500 - 0s - loss: 0.5350 - val_loss: 0.7383\n",
      "Epoch 104/1000\n",
      "1500/1500 - 0s - loss: 0.5551 - val_loss: 0.6698\n",
      "Epoch 105/1000\n",
      "1500/1500 - 0s - loss: 0.5032 - val_loss: 0.6520\n",
      "Epoch 106/1000\n",
      "1500/1500 - 0s - loss: 0.5418 - val_loss: 0.7518\n",
      "Epoch 107/1000\n",
      "1500/1500 - 0s - loss: 0.4949 - val_loss: 0.6307\n",
      "Epoch 108/1000\n",
      "1500/1500 - 0s - loss: 0.5166 - val_loss: 0.6741\n",
      "Epoch 109/1000\n",
      "1500/1500 - 0s - loss: 0.4992 - val_loss: 0.6195\n",
      "Epoch 110/1000\n",
      "1500/1500 - 0s - loss: 0.4610 - val_loss: 0.6268\n",
      "Epoch 111/1000\n",
      "1500/1500 - 0s - loss: 0.4554 - val_loss: 0.5956\n",
      "Epoch 112/1000\n",
      "1500/1500 - 0s - loss: 0.4704 - val_loss: 0.5977\n",
      "Epoch 113/1000\n",
      "1500/1500 - 0s - loss: 0.4687 - val_loss: 0.5736\n",
      "Epoch 114/1000\n",
      "1500/1500 - 0s - loss: 0.4497 - val_loss: 0.5817\n",
      "Epoch 115/1000\n",
      "1500/1500 - 0s - loss: 0.4326 - val_loss: 0.5833\n",
      "Epoch 116/1000\n",
      "1500/1500 - 0s - loss: 0.4181 - val_loss: 0.5738\n",
      "Epoch 117/1000\n",
      "1500/1500 - 0s - loss: 0.4252 - val_loss: 0.5688\n",
      "Epoch 118/1000\n",
      "1500/1500 - 0s - loss: 0.4675 - val_loss: 0.5680\n",
      "Epoch 119/1000\n",
      "1500/1500 - 0s - loss: 0.4328 - val_loss: 0.5463\n",
      "Epoch 120/1000\n",
      "1500/1500 - 0s - loss: 0.4091 - val_loss: 0.5912\n",
      "Epoch 121/1000\n",
      "1500/1500 - 0s - loss: 0.4047 - val_loss: 0.5459\n",
      "Epoch 122/1000\n",
      "1500/1500 - 0s - loss: 0.4456 - val_loss: 0.5509\n",
      "Epoch 123/1000\n",
      "1500/1500 - 0s - loss: 0.4081 - val_loss: 0.5540\n",
      "Epoch 124/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.4353 - val_loss: 0.5538\n",
      "Epoch 00124: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a40e6b0d0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "                        patience=5, verbose=1, mode='auto', \n",
    "                        restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Square Error\n",
    "\n",
    "The mean square error (MSE) is the sum of the squared differences between the prediction ($\\hat{y}$) and the expected ($y$).  MSE values are not of a particular unit. If an MSE value has decreased for a model, that is good. However, beyond this, there is not much more you can determine. We seek to achieve low MSE values. The following equation demonstrates how to calculate MSE.\n",
    "\n",
    "$$ \\mbox{MSE} = \\frac{1}{n} \\sum_{i=1}^n \\left(\\hat{y}_i - y_i\\right)^2 $$\n",
    "\n",
    "The following code calculates the MSE on the predictions from the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (MSE): 0.5463447829677607\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "# Measure MSE error.  \n",
    "score = metrics.mean_squared_error(pred,y_test)\n",
    "print(\"Final score (MSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Root Mean Square Error\n",
    "\n",
    "The root mean square (RMSE) is essentially the square root of the MSE. Because of this, the RMSE error is in the same units as the training data outcome. We desire Low RMSE values. The following equation calculates RMSE.\n",
    "\n",
    "$$ \\mbox{RMSE} = \\sqrt{\\frac{1}{n} \\sum_{i=1}^n \\left(\\hat{y}_i - y_i\\right)^2} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 0.7391513938076291\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lift Chart\n",
    "\n",
    "We often visualize the results of regression with a lift chart. To generate a lift chart, perform the following activities:\n",
    "\n",
    "* Sort the data by expected output and plot these values.\n",
    "* For every point on the x-axis, plot that same data point's predicted value in another color.\n",
    "* The x-axis is just 0 to 100% of the dataset. The expected always starts low and ends high.\n",
    "* The y-axis is ranged according to the values predicted.\n",
    "\n",
    "You can interpret the lift chart as follows:\n",
    "\n",
    "* The expected and predict lines should be close. Notice where one is above the other.\n",
    "* The below chart is the most accurate for lower ages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2deXgUVfa/35vOHsgGAUGWICKr7IuIyICKuMuICyMOjgui4/ZzVHCcGZfR0ZlxxMHx6zbuGwruqOMGbqgoCCKKCEiQnbAkIWTv3N8ftypVvSUNpJNO93mfp5+uunWr+lYHPnX63HPPUVprBEEQhPghobkHIAiCIDQtIvyCIAhxhgi/IAhCnCHCLwiCEGeI8AuCIMQZic09gHBo27atzs/Pb+5hCIIgtCiWLl26U2ud59/eIoQ/Pz+fJUuWNPcwBEEQWhRKqQ3B2sXVIwiCEGeI8AuCIMQZIvyCIAhxRovw8QejurqaTZs2UVFR0dxDiRlSU1Pp1KkTSUlJzT0UQRAiSIsV/k2bNtG6dWvy8/NRSjX3cFo8Wmt27drFpk2b6NatW3MPRxCECNJiXT0VFRW0adNGRL+RUErRpk0b+QUlCHFAixV+QES/kZHvUxDigxYt/IIgCLHK/BVbuPe91ewqrWz0a4vwtyAKCgp4/vnn9/u8Cy+8kHnz5kVgRIIgRAKtNVe/sIzZC9ayp6y60a8vwt+COFDhFwShZVFW5aVWw8yTenF4u1aNfn0R/oPg2WefZfjw4QwcOJDLLruMDRs20KNHD3bu3EltbS2jR4/mvffeo6CggF69ejF16lT69+/PpEmTKCsrA2Dp0qWMGTOGIUOGcOKJJ7J161YA1q5dy/HHH8+AAQMYPHgw69atY+bMmXz66acMHDiQWbNm4fV6ueGGGxg2bBj9+/fn4YcfBoy1cOWVV9KnTx9OOeUUduzY0WzfkSAI9VC4Gn75MqC5qLyaoxNWct6S80yfRqbFhnO6ue3N7/lhS0mjXrNPx0xuOa1vyOOrVq3ixRdfZNGiRSQlJXHFFVfw8ccfM2PGDKZPn86IESPo06cP48ePp6CggNWrV/PYY48xatQoLrroIv7v//6Pa665hquuuorXX3+dvLw8XnzxRW6++WYef/xxzj//fGbOnMnEiROpqKigtraWu+++m3vuuYf58+cD8Mgjj5CVlcXXX39NZWUlo0aNYvz48SxbtozVq1fz3XffsX37dvr06cNFF13UqN+PIAiNwAPDzfutxT7NRWVVDFRryd77E7Rq3+gfGxPC3xx8+OGHLF26lGHDhgFQXl5Ou3btuPXWW5k7dy4PPfQQy5cvr+vfuXNnRo0aBcCUKVOYPXs2EyZMYOXKlZxwwgkAeL1eOnTowN69e9m8eTMTJ04EzMKqYLz33nusWLGizn9fXFzMmjVr+OSTT5g8eTIej4eOHTsybty4iH0PgiA0AtXlkJRWt1tUVk3/hPWUt84nLS270T8uJoS/Pss8UmitmTp1KnfddZdPe1lZGZs2bQKgtLSU1q1bA4GhkkoptNb07duXL774wudYSUl4v1601tx///2ceOKJPu1vv/22hGYKQkti93po36dut6ismoEJP1PV/hjS6jntQBEf/wFy3HHHMW/evDr/+e7du9mwYQMzZszg/PPP5/bbb+fSSy+t6//LL7/UCfwLL7zAMcccQ8+ePSksLKxrr66u5vvvvyczM5NOnTrx2muvAVBZWUlZWRmtW7dm7969ddc88cQTefDBB6muNrP+P/30E/v27ePYY49lzpw5eL1etm7dysKFC5vkOxEEwcXONXBrFmxwGXa718OzZ0GFn3G3a63PblFZBR3YjafNYREZmgj/AdKnTx/uuOMOxo8fT//+/TnhhBMoKCjg66+/rhP/5ORknnjiCQB69+7NU089Rf/+/dm9ezeXX345ycnJzJs3jxkzZjBgwAAGDhzI559/DsAzzzzD7Nmz6d+/P0cffTTbtm2jf//+JCYmMmDAAGbNmsUll1xCnz59GDx4MP369eOyyy6jpqaGiRMn0qNHD4488kguv/xyxowZ05xflSDEJ2veN+8/vOa0rVsAaz+Awh/NvifZvK//BKqdVfNle4tIUJqU1rkRGZrSWkfkwo3J0KFDtX8hllWrVtG7d+9mGtH+UVBQwKmnnsrKlSubeygN0pK+V0GIaj65Bxb8FUZdCyfcBsA7s6ZxUvGLbKMtV3j+wkvea0mkFoA31Fhu9/wegOzKrXzguQrOeAAGTTngISillmqth/q3x4SPXxAEIeqosSx4a9JWa43eXQAeOISd/NdzF4neWta1GkL30qUcl7CMlUckM6VgJhVprWAvkNr4E7sgwt8k5OfntwhrXxCERqS63Lwnmqi80soaDlWFdYdzq8yane5jfwub+5LxzdP8cd8/oOwH5xoRiOgB8fELgiBEBtvityLsisqq6ayCLKZMzYKsLmZ7w2d+x0T4BUEQWgYlW+Dr/5rtGpNkbW/xHnJVaWDf1CzI7BD8OmLxC4IgNAPfvgg/fxzYXrUv9DnvzHC2Lcu/vHB98L5tekC2ZfGPvBLG3+kca4k+fqVUAWaKwgvUaK2HKqVygReBfKAAOEdrvSeS4xAEQThgXp1m3m8tBq1h3Yfmfc75cM1yyOwYeE6CS1qtME3v7gLfPqf/B7I6QXZn85q+CNr3hdXvOH2SMxr3XuzhReSqvozVWg90hRTNBD7UWvcAPrT2BaBVK5OFb8uWLUyaNKnevvfdd19dojeAk08+maKiooiOTxBiipItwdtrvfDxP01svT+r3jQLsF6/EryVsO0751h1uXkgAHhcdasti18VbQCg+JRH4DcvweALoPtYp98h/cx8QNahZj81u25+oLFpjqieM4BfWdtPAR8BM0J1bul4vV48Hs9+ndOxY8cG8+ffd999TJkyhfT0dMCkaRCEuKJsN2xeCj1O2O9T9YYvUE9MYPmIWWzrfLLPsS5rn6HP8jvYlTeCpcc8ynir/d1vf2H4J7PIASjdBsCqlcvYUDWAlPLtjH3rWFYOuYNN3c5m6LZNtEptR1JVEVt37mHld1vpunEhe3Qr0geeBUn1SG/7I+HMh6DnhP2+r3CJtMWvgfeUUkuVUtbvJdprrbcCWO/tgp2olJqmlFqilFpSWFgYrEuzEyrdcn5+PrfffjvHHHMMc+fOZd26dUyYMIEhQ4YwevRofvzRrNpbv349I0eOZNiwYfz5z3/2uW6/fv0A8+C4/vrrOfLII+nfvz/3338/s2fPZsuWLYwdO5axY43FkJ+fz86dOwG499576devH/369eO+++6ru2bv3r259NJL6du3L+PHj6e8vLwpvy5BaBwWPwwFn8GLU+C5SVDegKe4eBPsWufTtPXHxQAMXPz/+OCFe3n2+Se57tlF6Ben0Gf5HQD8tG0vf37Bsfo/felecnZ943Od3cve4P3nZ7HgtccB6LnkFu5/bh6F2zby6b7OFNS0IXf9W+TMPZPe+77mGc9EUuoTfYCEBBg4GdJywvk2DohIW/yjtNZblFLtgPeVUj+Ge6LW+hHgETArd+vt/M5M359cjcEhR8JJdzfYLVi6ZTAZNT/7zIRmHXfccTz00EP06NGDxYsXc8UVV7BgwQKuueYaLr/8cn7729/ywAMPBL3+I488wvr161m2bBmJiYns3r2b3Nxc7r33XhYuXEjbtm19+i9dupQnnniCxYsXo7VmxIgRjBkzhpycHNasWcMLL7zAo48+yjnnnMPLL7/MlCkHvipQEBqVki1QuRfyetbf750bzbuy7NbSwvpFcpaVxNGV+rh8n5Pz6p4kq45FQhKq1ql2NTi3kueP7wEvm/0/9N0HP/leepTne0Z5vqes/TDYDknKy1spNwNwSK9jSNu+l9Q9WxlhSd/U8ybXf29NREQtfq31Fut9B/AqMBzYrpTqAGC9t+gqIf7plm2xP/fccwGTofPzzz/n7LPPrivYYhdbWbRoEZMnm38IF1xwQdDrf/DBB0yfPp3ERPOMzs2tP3fHZ599xsSJE8nIyKBVq1b8+te/5tNPPwWgW7duDBw4EIAhQ4ZQUFBwEHcuCI3Mvb2d/PShqK11trW1vW//JUSVbAxsq/UtcZhSsZPuGU7+nJxdy6DTcMgIdFKkb/86oC0nqYZUXeHTltWxx36PNRJEzOJXSmUACVrrvdb2eOB24A1gKnC39f76QX9YGJZ5pAiWbhkgI8PMxtfW1pKdne2Tm7++8/3RWu9XiuX6ci+lpKTUbXs8HnH1CNGJ1qEnNav2BraVbt/v6ybv3RS8T6fhsOkrs11ZDE+f4RzbtQYGXQAVReZhk3kolGwOvMYZD8D272HAZHjqPd9jGXnhjTXCRNLibw98ppT6FvgKeEtr/T+M4J+glFoDnGDtt1iCpVt2k5mZSbdu3Zg7dy5ghPnbb78FYNSoUcyZMweA5557Luj1x48fz0MPPURNTQ1g0j8DASmabY499lhee+01ysrK2LdvH6+++iqjR49uhDsVhCai0i9l8Y5VsHWF2S4PErlW6jcH+N08kwHTz69PhePqSd9nLP7aQX6/tPN9//8G0H2c41Y6bGzwPl1HwYS7oEN/47pyEyV1MiIm/Frrn7XWA6xXX631nVb7Lq31cVrrHtb77kiNoSkIlm7Zn+eee47HHnuMAQMG0LdvX15/3fzI+fe//80DDzzAsGHDKC4uDjgP4JJLLqFLly7079+fAQMG1BVbnzZtGieddFLd5K7N4MGDufDCCxk+fDgjRozgkksuYdCgQY1814LQyLhEmX/1gpoqs717PfzfUfDwaNNWEeT/Sel2E3o5/zrYsgxevtiEXN4/2LffPhP8gLearIrNPMpEEs74j2+fPmeYiJpJjzttya1h0hMw9U3o92tnUZX9kOh3FvxxC0ydD71OdRZjAWiv1Xc0jAjUhuZC0jIfBC0p3XK4RMP3KsQh21bCQ6Oc/auXmWidR11lQy/50MTKP3Wq77m9TzPx9QCHnwBr33eO3VpsiqEA/O5/0HWk+SVw/2D+lnw1f/zjX+HuLs4DZcYGkyahcLUz33DZp8Z6t3llGqx40Yj9hi/MAyApeHnUus+eURDRKJ1QhErLLCkbBEFofop+8d0v3QELfcua8suXwS1+W/TBFDpx43VN2L44xSzO2rkGgD1pXU37H36CYVa1PDs3TlZn5zz/Yud5PaFtT7OqtsfxoUXfTYRSLxwoIvwHgaRbFoSDY/5XP8KtWfz0yl992me//ikfbHMEtSQhiwWLFvHIe75x9AVJ3X0vaLtWLKY/9pGzU7aTuf++juff/hCA0lb5pj0pFU65xyfck+R0Zzu9je9nHHMdXL6owXsDYMxMyD0sanz7Ni1a+FuCm6olId+nEFFsv72Lz742YZBHVP1AJSlcnfsgAGmVOyn1OmkPdqscWlXvJrHad+J3jccI/+4Ex43yTMbUum1VYSaDP075FQBnFz/JyH3vU6IyOW5wr/DG7fELflTKNyVDfYy9ybitoowWK/ypqans2rVLxKqR0Fqza9cuUlPD+NkqxB5am4WQGz4PPLZtJew1KQqoLIU3r214taw/W5bDHXmwbqFPc1W5k+EypW0+s688DxISuXRgOmf2shKUnXgX+V27MbxNORd18Y3gOeFYU086N39AXdsFk38LJ/0TgAdPM26aMadfCFNeAaBbzXoyD+3FpCGd6h9zl6MhudX+3WcLocVW4OrUqRObNm0iWtM5tERSU1Pp1KmB/wxCZNDarFy1E3Q1NTvXwOIHYflzMPUN408/6nLjE39oFGR3hWtXwDdPw9InIDUTTrg9/Ov/YC3XeesPJjrGuk9PhesBkt3ZpCvIaGcidapKoe0RMPIK2Locfl4IW7+FoRfBEivqxs6CmdcLfv7IbGd1ckI5i614/eTW0G0MKI9xB7UNYyHVhW+Ff38tjBYr/ElJSXTr1q25hyEIjcPy5+D138MlC6DTkMh+1o9vwcf/gEsXQIKVQHCd8XtTUwmP/MpsH3U57LDKAFqZJet86EHcNvWyxXJ37F4HL5wH081q8qSqIsfvkJNv3jPaQNkuM5aUTKvNWvjUtidM+Lsj/AN/Azu+hzEzYPFDVt92ziRtsbXAKqW1cdkkeMAbpvAntFiHSIO0WOEXhJhivRFCCldFXvhfmWas6YLPTD3YLiPg+9fMMW+l02/3ehOuCJCUDgWL4L0/mX3tSp3wxQNm0VJHkw6ErStMkRLthTeugtNmw3pXIRMrqqai2ku6d68j/HmWzz0l07iUasqdaBiv9aAZcB4kJsN1q0xoZ2omnH6/7/0lJJiqVgALTcI1WuX5XqfvxPC/rxhEhF8Q4g3bPfL06cb1cfUy2PglHHES/OQqAjJ7IHS1FilVl8GTrvTF2mv8/E+fYdwvANeuNJb2w9ZK8eRW5gHzzgzzoOgxHta8ZyZHa70Ul1eTo1wrW9seYd5TWsPerVBV5iyGsvPj2PH0wYqfuLGF3yY737yf+6x58Ni/LuIUEX5BiAaaKtzvw9tNrhkb7YWf3jXbx14PW77xzX3jX/zbpqLEVIqyRR/gvn7QZaSzn5JphH/H95DTzRQfWfokzL8WFtxBzoqXycEVjmln5UxpDT/9z2x3Pdq8j7rG/KI4/PjQ93bBa85kbNsjzIralVZqTdtt0/u00OfHEbHrxBKEaGT3z3Wl+HhgBCx9yvd4TWXgOY3Jp/8KbHvnBgA2eQ6luM/5YV2mvHg7e9cvCTzwi3ENaRQ1tU7E3b52g9m4p5zCBCuN+Gf3klyygd8kWguujr3BWSjljqTZazLZkpjccMGV7mOh8zCzneCBsx6DVoeYaws+iPALQlNRXQGzB8Frl5vtwh/hzautg5bFX1Uauc/3TxjmxzH//obBnwziLa9vauSt2qQCP6vylrq2dQUbWLfsIyp1oNPgh9quKDS1pU665Bu+O5TR/1jIRXMLAvoXjvgjjPuT86snpbVzcGB4D6KgKAXXrzbXFnwQV48gHAx7NhjXSOcG8siDE/v+41tQ7peb0I6WaUCcD4oGihXdc7aJhU/ffTUscgr0VHY+hnmD7mIywJu3AXB4agmJ3nLWd5lMStUuumx2Sn+mdR0CGzeQrLxUJLfh5/xzGXfENMYpRXpZnqm0bbGr3UjaTrjRdyB2JM/hx0PfMw/4doXQiPALwsEwe6CZuLw1eHZVnjsHynaa0Mk6sdcmXBHAY9VIqLIWMjWW8Bf+ZNwj7klM220CxgpOyXSqWZ35IJMGWms4theDKyNBfucu5NuLnZYMgK3fklpl7qVHv2EmP44rLX23I4+GjWaxVOqRp9PntLvoYx+synWE/7CxtJk8J3B+w7b4E2UxYaQQV48gHAx2WGOwFeS1XljzrikIDqY4uH2OLfy2uNk56CtLnXPD5dlJ8I/DTHWqlS/D2zfCA8Pg4WN9+1kPldk9Hodj/mBCNAFQJh7exn+1ao2ritRln8DFruyX7fpAbY1v/46uNOD2OoG6a6c7n9u+b/AEZynW53uSA48JjYJY/EJ8UeuFhXfC8GnQ+pDGu25liW8I4Y5Vvta7t8ax+HWtkxs+MQW+f9UpJlJZYuLlnzwZjr4ajroCVswxlZ8yXPWVN39jMk92GeGkIb7dL+2vXyZLXVGCAmqy8k2Ui52ITPnZf24fO5hc8m4OOdI8HDzJcEg/p//IK01sfXZXp6//tQGS0kx4aKhqVPbDVCz+iCHCL8QXBZ+ayJYdP8Lk5xvvuhu/MqtBc/Jh51pTPCTHtbL8rescS1jXmmIhYEr4zb3Q6Ve51xHyz2ebF8Cns+C8ZyEpA1a9Dov+bdpDuZjAZJWs2mdCKBfcQW3HwSitSG9lPaCSrFw4/uKcnOFs/3lnYEKypDS4erlZPJWYAif+zcTbj5lhLHx3Hh/lZ/GDcTGV7Qr94LUjmxJTgh8XDhoRfiG+qCoz73o/XCmhcEWt8Nwk835rseNL37PeOf7NU+FZsBVF1EX4uKkshqdOg7Rc34nhLcFrOdNxkImx//RfdSGcng2fUUI62emWCyWUxe8W3FBZKFu5rPX0XBj7R2ffLfbBLP7T/g3bVoSOqbcfkD3GBz8uHDQi/EJ8YfurG8N/fE+QfC9FGwMjdmy2hJGed/v3JitkKPyv/ciY4P3a9TWft/Ern+a9pJGdbol5cgiLH4wFP/i3DY83GG6/frB8N4eNMa9QdBoKM38JXH0rNBoyuSvEJhUlxt/sT50b4eD8x9uLy4IfuK8f7y8OIfCbjAgXZA5lYecr2NDatybsL60HgreKXcvfDHb2fvHBbqt4SMGnPu17tcviD+XqAbj2uwNf+ORj8Qdx9YSDiH5EEeEXYgdvDXw3z0S33N0ZHhwV2KfaCps8SP/xu4tXhDyWut4JVN+k8/iL95K6/Vu8F3Pi7uuZvv5Y3iv2TcH8bNGR7NKZtKnwK0MYhEk1d9Z7fM76tLrtcp1MoTZCWpmQzmFtLcG3XT3+kTcHS0IDrh6h2ZG/ihA7fP1fM2n67Qtmf/e6wD7lVp6ag504LCoIeWh0gvNQ6NTxUG7/q5Mm4bbb/8XqO05i9R0ncelx/X3O++N5x9Hm5PBWmc679dJ6j//31utN9A2Qlgh5Hc1E84DDO9Mu0/q14w7nbEzcVn5jP1SERkGEX4gdbP+3e1LVHzvEUWsz0XtbDqx4ab8/Krlko7Njpw6+5lsYcblvRzt2/8K34YrFvouV/PPypOXUn4TMTWIKnPNM6OOeJJj4iNn2VpkJWHBWxYITrz/0d+F9Zri471Es/qhEJneF2MGerNxXT1U2W/hrKk30ja6Ft6+H/uf49tO63oyZyWWuDJZDLjQrYT1JcNLdRrxLNsGb10Cx5bbJD+J2KvJz6aTlmsLcYKzm4dNMVSx/7IdDn9OdDJRtjzB571sf4oi7vWpXeZy2VJfwJ6XCzdsbf6GUj/CLxR+NyONYaHk882tYNd/ZL94Mewqc1bPFrvwBVft8Tq1LSVxT7qyW9VvoRNU++Gtb+OQeuDULfrTy0KxbUBcOmlBpndPtWBgx3Tfsscfx5mHQ8xQ44/9C30f/c33303KMaN60Gf60HUZfF3jO8bfBrx919u3c+iOmQ9eRkNvNVLAC48M/7i9mpa1dSeuwsb7XS0qNUKUpS/xjuIpVS0YsfqFlUVVmygSu+9BZvDTLygQz+nrzvqfA6f+3jjBzo/E1J2fAXstSr6l0/P3ga+F/84xJQ7Dgr2b/fzOhTXd4ZqJZQZuWzZn7XqLEk03m1HoicBpaINbjeHMPt1oRLHXuGMsF43bLDJ9mqkZ19Qv1tC3qUFb76D+Y97E3m9KEvU6tf0yNjbh6ohIRfqFlYee4SQiysKh0m3nftca3/e7OJqfM+XNh42LTVl3uFOIGkye/TXdYcCd88g/f8/dudfp+/xpUmVQMlR6/1AYHSpeRJo993WSrhXsC+uR/Bj/XnjxtaEFa3zObNtOlUtbDVFw90YgIv9Cy8E9u5qZgUWCbzY4fTI1ZtHGp2L8abFa/DUdfFSj6YCZHd/9stquc/DvVSY0k/OfPNb9E/OcUwqnKZbt69iepW1MiFn9UIn8VoWVRJ/zJvLx0EwNvddWI3bOen3SXkKfOfmUhAN+WtfVp36Lbwnt/4uJbglSnspj31tsBbYkqSEbOAyGlNbQ9/MDOdefViUYknDMqEeEXWhZ2eGRiKks27KZ1bYnP4dRW2exLzA566oTsX6jFQ0p2B5/2eUc+TGVCOv9MeSzkx47M2BrQlpPSSMJfH4OnwnG3hD5+7A3G/+9OqxwVWL9WxOKPSuSvIkSWNR+YilONRZ3Fn0JRWTW9WvvGwnfpeAgZ0z90UgkfOrQuCuaIqlUkZOTSq7NvOuCrJx1PypDzya2yooHa9nRSC1sx+ofWuOYD+pmEbEn45aGPBKfPDh7dY5OWbfz/SWmh+zQn4uOPSkT4hcjy3FkwpxGt0brJ3US67/6Ev5f7WcO22+TC+XDdj3Dph07ce/keEyufGEQku49ztqd/ZiZ6wUkd7PLtc5S1SKum6uDvJ1ZREs4ZzchfRWhZ2MJfXc7koofJrbWzVdqFul0hkJmWSyctxylxmN7GN1rmTGuBlB0mOWiKKVloPxzcOeOH/M5kjWxrZeU84sRGuaWYxF5TIa6eqET+KkLDlO4ITC8QDm6L+JN7jNvH5uN/wr96+5YsfP5ceHQclAT60wFY+yFssCJ3qkqptU+dPMdZ8ZrRNvA8pZyHQHquI/z9Jjm+8bRsuG6VWf1qnwPQ2jUf0KqdyRqZmgX/7weYcFe9ty8grp4oJeLCr5TyKKWWKaXmW/tPKqXWK6WWW6+BkR6DcJDc08O3SlR9lO02JQEB9m5x2hf81bh9ar3wzdOw8A5z/EvXytaf/mfq097bC54+w6RWdvPsr6HwR7NdvodDa7fxUcdLoedJTrm+9DbBx9W6o3XclRahz+m+fTI7uuLireu5LX73tbMODV2kRHAenGLxRyVN8Ve5Bljl13aD1nqg9QpRQkiICmyrfXVgOGMAWsM/usHrvzf77gVSNqvfgTeucvbf/aOpFOXPzx85RcpLtpgatn4kKE1ZzhFmx2uNMy03+Njs+rfZXWHYJfDnXdDnjHruxRL+ViGEXwgPCeeMSiIq/EqpTsApwH8j+TlCBKl25br57wn197WrW614kS/ffR7v0xN9Dpeld2TPe3fX7e/L6EKt8rDuw8d549st+FP4v7v56p1n4N7epoZtEGqzrbq2tisqlDgPuwja9zM5bZQCTwNrF+uEv53Tlh7ioSIEQSz+aCbSf5X7gBuBWr/2O5VSK5RSs5RSQROjK6WmKaWWKKWWFBbWk21RiCxVrkpTm74K3Q98Kl4d9cXleGqrmFtzLADveYdwU9GZ5Oz5rq7Pd3szWO7tRqc1z/LjS050zrveoQDkFX7J8MVX1vuRWR2tiVbbvRRKnIdeBJcvcvLgNIQt/Mmu/q3ah3eu4CA+/qgkYsKvlDoV2KG1Xup36CagFzAMyAVmBDtfa/2I1nqo1npoXl5esC5CU+Cf3bI+/EodVub2ZtDVL7D1xEfpPn0OV111o8/xI7u2p2d+F1JUDTcmmZz45R2Gc/j0+pOb1ZEORoIAAB2tSURBVGQ47pfRffPNhu3qaSx3jC38CQlw9lNmLUC7Po1z7XhAwjmjmkjm6hkFnK6UOhlIBTKVUs9qradYxyuVUk8A10dwDPHHqvlGrAec23DfcKhuQPh3rTOpjtv3CxD+lKx2HN6uFbQ7J+ipGekZTvk/i7Sex9H90Pot68QOR8Labb6N3gZcPfuLnftGeZo2uVmsIa6eqCRifxWt9U1a605a63zgPGCB1nqKUqoDgFJKAWcCKyM1hrjkxfPh1Wnh9S0vggeOgo1fh+5Tn8VfsgXuH2xCMO9oF1gApSGfeFKqU73KJpzcM4kp8NvX4XeuPD1T58PAKY2Xu8ZewJWW0zjXi1fE1ROVNEd2zueUUnmY2Z/lwPRmGIMAsGkJFK6CzUuMmGZ1MpE0hx/v/FR3+/jBRPkkWrnfX7vC99hev/j7hqy9xFQTE+/GPzVx0PNS4LBf+bbljwpe5epAOfFv0ONE6CjRxgeGTO5GM00i/Frrj4CPrO1x9XYWmo6tViTtgjtNSoKMdrBvB4y/E462JlX9XT2VJZBoLZLatsL3mL2q1kb7z+ljLEA7d3xiilk45SbZb/L16Kvh89m+bZ6DLJQeDklp0HNC5D8n1pFwzqhEHsfxjB0/b+eh2bfDvG/43Onj7+qxyxVqbRZYuVe2+rt6ggm/WwgS0wItftvn//uvYdrHvukV6s5r5BqxQuMjC7iiGvmrtERWvgJLn6y/T00VfPVo/QU6toZYO1fhKkno7+qxV9PWVEJtNbRx5ZH3F/6RQUIxE1w/MhOTA0sG2j76vCOMm2Xk76HzCN8+TWHxCwdHXa4esfijEanA1dKoKoN5vzPbh59gUgfYuEV+0X2w8E7jshg0BX82bd5Mp6Jf6va3eTpwiNf46Dds3sKND38BwBmlP+DOrXn7vC+BL2jn3cZ04P0dralb1rXkcafj+Dug8/DA8buFICEx8FeBf2WttBw473n4Z3fXNcKoTCVEBxLOGZWI8Lc03K6Xks2+wl9V6mzv2WDe/UIsbdau+JxO7v2knnXCn1FrrpOkK/nN3id8zkvT+7hhz+11+9s8HYOPM1hpRPAVAq2dvDk23iCpjtNyoNNwZwGZboICKMLBIa6eqEb+Ki2NapfrpWSz77FKV854213j70qxSNrzk8/+MWOcFMNtE8t58bKRPDs+0LK+4Whfn/wFpx4Po64N/IBQwu+TC19Dl6Pg8s/NIikwRVD8SfDAJe/DhLsDjwnRjbh6ohIR/paGj/D75bdxC3+5JfzBJkeBlJJfKNOuY+6wxapSkwKhdHvgif7J0lJaw+g/BPYLVRHqglcD29r3NYukbi120icLLRyx+KMZ+au0NMIV/oriwHM/mwXbfwAgo2wjm1V7mPqmKTCS18v0SbdCNX/+yLn+b+ZCSpaxxgs+871mSmbwRVOhLP72fWDcn8z2frts7F8g4uppMUg4Z1Qiwt/SqNpPV4+dMXPPBvjgVlPsBMiq2MT2xA7Q7Vg47T6zynbGBjMpC/DcJLMgK7k1HDEebvrFVKmy8+HbpGYG/8+dFEL4fdhPAbf9xuLjj37qfPwi/NGITO62NOzJ2qQMxyLftMT8pA7m6rHTFduVq8p2gta0qd7G0rTBvtf2X0xVtNHX9RIsSZm71KGbekMuDzYqR4S/xSCunqhE/irRxBtXw/t/qb+PvZK27eGO8P/3OHh0rK/w24uybIt/k5WPp9YL5XtI0ZWUpwbJepp/jLO96SvfBVrtgwl/6+DjTKjHpugyMvCzhNhEwjmjEvmrRBPfPAWL/l1vl6Ji47vfm5GPLtnC2m2OL79w166A/ruKSli7o5S9RTtNg7eSLSs/BqA6tV1Af7I7w/kvm+19hb6x+MEsfrv84KALYOzN0NkqmFJfrH3+KJi50eQE2i8kfr/lIJO70Yz8VSKF2/o+EBY/HPAQWLm5mGfeXgjAYz96UNrLlf92ctc//dF3+DPni584/t6PWbx6Y13bK2+YyJqEzBDpj91rA/q4UhKn50KHgdDl6MBzzvgPjLkRxw3TgEinhnAR1YedO2fw1P0/V2gexMcflYjwR4LNS+GuTrDqzQO/xjs3Brh9yn98n6sSXwNg9FHGXfK/lJl1x8/oHSimE3rmMHvyIAa0T2ZfuhH0cw8xuexPGTkg+GdnWUu7UjLhkH6+xy77GI63qmUFi9wZepF5b9M98NjBkt3FhHx26N/41xYaF1nAFdXIXyUSbLFy4Kz9oHGuV1EC+3aSWOjE0A8ZNCSg2+FZOkCMu+d4OL1PDnkpXjIO6QGeZPKKTFbNzLadAq4BGL/9lFfgmiBF0MHJoBlsYnfAeUacM9o2fF9C7GJHXkk4Z1Qiwh8J7NWydh3YcKhxpSpwn1dRDK9Mg392J7mkwGnPCDIxW7kXMg/1jahZ8jjceQjs3WYEO6sz1JSbydf6iowcflzoQir2xG2oiV1BsBFXT1Qiwh8J6oQ/SN6ZULjz7LgXX93XH34ylaY67vrCaQ9WYnDD56aY+LSFcNwtkOtyt5RsMgut8qyUCNldDzziIvcw6D7O1KEVhGDUuXpkQj4aEeGPBHaky/5Y/O7ka6U7nG1XiuT06t1Oe7BKVXu3mFw47fvC6OsC0yYkpUMHy6/f9ojwx+ZPYrJJvdAp0N0kCD6IqycqEeGPBPaE1n4Jv8viL94YtEtKrbV466grQltS+1wPDf88PUnpxtKH0Ll0BKFRkMndaEb+KpGgtsa8h+vqqSyFPQXOvp0nf/hlTpvlPipKyIEJd4W+livHfkDUTXI69DoFep4MxzWwUEwQGgPx8UclYQm/UuqacNoEC1vwvVVmpezCu6Bsd+j+j50AL5xXt7to6TIA3tjuTL5WYIS/0tNAMfIkV8K0YBZ/aiZMfgFyuzV8H4JwoEg4Z1QT7l8l2IqZCxtxHLGF7eKprYE178PHd8P/Zobuv+MHn92d242r59UNjsVe6jWWU4J/jVqAU/7lbF/8rrPtX90qWBZNQYgk4uOPSuoVfqXUZKXUm0A3pdQbrtdCIDA/QCzirYFbs+Djf+7HOZbFv2ERvGCyYdbVqg2DM/gEgCemnwDj74Sp82mbZQQ/r60rjLPXqebdLm6S1xva9XaOdxpu0il3tJKx1Zc/RxAigbh6opKGlOBzYCvQFnCZlewFVkRqUFFFpSXYn90LY24I75xgk7r+1rfNmnoWeeX1dFbO2mmO3bHz5zwD2gs/vG72PX5/znE3m9e7N8OWbw4+jYQghI2Ec0Yz9Qq/1noDsAEY2TTDiULsaJv9sVyCTepqr+9+RTGkZsFzZwW/xnF/ccJCwfHXu109CQlAghOhk+Dq78Y+R4RfaGpE+KOScCd39yqlSqxXhVLKq5QK33fRkrFdNPuz2CmYxV/rEv7V/4O7u8AvX4a+RqJfuKUdoRMsTYL9UPCEEP7h06DvRDjq8tCfJwiNieh9VBOWmmmtW2utM61XKnAW8J/IDi1KsK3k/YlOqG3A1VPwqXnf+JVJrzDySgD+U3OG08c/zj4xiKvHxvbdh7L407Lh7Cclf47Q9Ei1tKjkgGKttNavAeMaeSzRie3jP1hXz/qPneRttlBX7QNvJaRls/iCtdxTc47TP5TwB0tnbK8bkAgKIVq48C1j0NSXD0poNsIK81BK/dq1mwAMJV7q3x2Ixe+tNgnRzn0WnnHls39kjMlcabtkyq3Y/pQsiiq8+Pw+DhB+y50TzOL3WsIfytUjCE3NIUealxCVhBvfd5pruwYoAM4I3jXGqLP4wxf+feVl1FTB5CfW8LbfN/zbP/+T09TXnJ0Ab365ktMS4Mb563nF+41vx/3x8dshnAMmhz1GQRDil7CEX2v9u0gPJGqpm9wN341SUlpGovZwdM9DYZ3vsac9d9Rt983xQjGM6JVPbu5htGudAnZ0Z5JfuoX6XD05XeGWIomgEAQhLMKN6jlMKfWmUqpQKbVDKfW6UuqwSA8uKrBdPfVNUq35AJ4+E2rNBG51dRVVJPKbo3vUe+nD0isBOOvo3sw8qRcXHeNKo+CffbPO1ROiZKGIviAIYRKu/+J54CWgA9ARmAu8EKlBNSk/vu1bBMUf29VTvS/48fIimDMZfl5Y57P3VlVSrRNp3aqVb98zH/Td32pN9gYN0fSz+G2ffyjhFwRBCJNwhV9prZ/RWtdYr2eJhcnd9Z8Y0V54R+g+tsVftS/Q6v/hdfh7VyeKZ18hAN6aSmrwBAp/hwFw4t8CPyOY+ybU5O6BFCkXBEFwEa7wL1RKzVRK5SuluiqlbgTeUkrlKqVC1OdrAZRZ6YZ2rw/dp7rMvNfWBIZpbvjCd39fIax8mcMLPyBZeUlN9XPXpOU4+fDdBLPiA4Tftvil3KEgCAdHuFE9VqYxLvNrvwhj+Yf09yulPMASYLPW+lSlVDdgDpALfANcoLXejxqFjYgdqRMqjw5AdYWzXVnqm+rYP0Z54+K6vDld1PbA3DlpOYEZMrseEzzW2d/Vc+RZxtoX4RcE4SAJV/h7a60r3A1KqVT/thBcA6wCbLP278AsrfUcpdRDwMXAg6FOjih1wl+P16qm3NmuLIEMV63bFD9XzoI7qHetelKaie9387u3QvT1+7WQexiM8H/uCoIg7D/huno+D7PNB6VUJ+AU4L/WvsKs+J1ndXkKODP42U2AvRo3XIvfLoLurYZ3b8a76+cgJ4R4iEz72Ly7Lf7BwcocWPgXUREEQWgk6rX4lVKHAIcCaUqpQTjmbCbQQCkoAO4DbgRs/0QboEhrbS01ZZN1/WCfPQ2YBtClS5cwPuoAqLP4vaH71JQbH3xliVP4fNsK+OI/eFNyCDu6v+NA8+4W/tNn1zM2Cc8UBCEyNGTxnwjcA3QC7sXk5P8XcB3wx/pOVEqdCuzQWi91NwfpGtRE1lo/orUeqrUempeXF6zLwROuj7/1IWbbtvit+rjJlXsA2DT0JgpPfza8z/R39fgz7NLwriMIgnCANJSP/yngKaXUWVrrl/fz2qOA05VSJwOpmF8J9wHZSqlEy+rvBGw5gHE3LvUJf00F5OTDzp8ChB+gUGeRO/560pMT4ftxsG6BOeDvo7dpqPzhKfeYlyAIQoQId3K3n1Kqr3+j1vr2UCdorW8CbgJQSv0KuF5rfb5Sai4wCRPZMxV4fX8H3WjYWS3rtfjLHYu/3HL1uIS/nBTaJlkOnwzXL5Nrvwt+PfHdC4LQzIQr/KWu7VTgVEykzoEwA5ijlLoDWAY8doDXOXjCEf6aCiPoymN8/G9cDd88XXe4SqWibH+8HYKZ1cXJfd91lK+VL757QRCamXCTtLnr7aKUugd4I9wP0Vp/BHxkbf8MDA97hJHEFv7aBiz+pDRTvnDVm8bl4z7sccXb24uuEpOdtt+9Hfy6PU48gAELgiAcPOFa/P6kU8+irRaDXQ4xlMXvrTYRP4mW8PuJPoDX41pha1v8nuSAfj78edf+5fcXBEFoRMItxPIdTvRNAtAO+GukBtVkNOTqqbYWbyWlmvKFe1zH2vWBHT+QplxlFm3hbyiFs/+KXkEQhCYkXAU6FcgBRgPZwNt+YZotE7s2bijhrzGLtx7+fAsjKjUDXYfe2HsEp/MD7WtcQUl2Dv2Wn75OEIQYJlx/wxnAM0BbIAl4Qil1VcRG1VSEafH/UlJLVaJZg1ZDIiUJWSzKOhWAVt4ip7+dSK2+yWJBEIRmJlzhvwQ4Smt9i9b6L8BIoOWvNGrIx29Z/G1zshje20xpJPaaQOZffuHv0882fdxpF+osfhF+QRCil3BdPQpw5zXwqwzeQqmz+EOkbLAs/qSUDDO5C74plP+y29efb/v460sBIQiC0MyEK/xPAIuVUq9a+2fSnPH3jYXb1VNbC5/PhkFTnBh8S/iT09LN5C74hmr6T+ImisUvCEL0E24c/71KqY+AYzCW/u+01ssiObAmwWtP7mqTS/+DW2DLMpj4ENx5CPQ8GYDktNZgL7itL4Vzkvj4BUGIfsKOK9Raf4MpnBI7uH38Vr1cqkqd1AyrzeKrlPRMSLDz8tcj/GLxC4LQAojvVUS2q6eiBBbcabaT0nyLrwCprbKcVAv1Wfy28Ne3ElgQBKGZie+VRLbwl2wyL5tqX+FPb5UJSVa+nfqya0pUjyAILQARfn9WvUll95Nw59DMzc6B7hNh1xoYeWXo60kcvyAILQAR/iCkzL/CZ39I947gSYCx9daeEYtfEIQWQXz5+KvLTdTO5/ebylr+wt+6Q9DTEjxhfk12cjaJ4xcEIYqJL4v/zWtgxYtmu7LUV/iT0o0b572bD/z6tvDbhVsEQRCikPiy+Dd+5WxXFAda/EdfCVcuOfDrp+fCmQ/Bb+Ye+DUEQRAiTHxZ/O6yh189Ap2GOfveKvPuTslwIAycfHDnC4IgRJj4svg9Sc629sLGL5192/pPPUjhFwRBiHLiTPgbqIwFziIsQRCEGCW+XD0NCP/Nr34HwJ3uxoT4+ooEQYh94kvV3K4eixKdTqYqA+Dd77cBLuG/cimktG6iwQmCIDQNcSb8gRb/Lt26TviX/OkE03irdbDt4U0zLkEQhCYkvnz8CUEsfurJvSMIghCDxI/Fv3t9YOEUoAYPpaNm0KrnuGYYlCAIQtMTH8L/49swJ3h8fQ0eksbOgETXQ+GKL02qZkEQhBgkPoR/988hD2mVSEqi3y+Bdr0jPCBBEITmIz58/K3ahTyUmNDya8YLgiDsD/Eh/O5oHk8KTJ0PPU8BYFVK/2YalCAIQvMQH8LvSsb2YXVf8h8u4fgVY3jdezQLc89pxoEJgiA0PfHh4/dW121W6kSuHnc4qB6sYyxX9cxrxoEJgiA0PfEh/C6LX3uSuW58z2YcjCAIQvMSJ64ex+JX4SRqEwRBiGHiQ/i9jsWfGriGSxAEIa6ID+F3WfyZnopmHIggCELzEzHhV0qlKqW+Ukp9q5T6Xil1m9X+pFJqvVJqufUaGKkx1OGa3G2lRPgFQYhvIjm5WwmM01qXKqWSgM+UUu9Yx27QWs+L4Gf74rL4MxDhFwQhvomY8GutNVBq7SZZLx2pz6sXl48/TZc3yxAEQRCihYj6+JVSHqXUcmAH8L7WerF16E6l1Aql1CylVEqIc6cppZYopZYUFhYe3EBcFn9yYnxMawiCIIQioiqotfZqrQcCnYDhSql+wE1AL2AYkAvMCHHuI1rroVrroXl5B7nIyltNlUrmscTzyJz60sFdSxAEoYXTJOav1roI+AiYoLXeqg2VwBPA8IgPoLaGGhJ5u81UqaolCELcE8monjylVLa1nQYcD/yolOpgtSngTGBlpMZQR20NXjykJUkQvyAIQiSjejoATymlPJgHzEta6/lKqQVKqTxAAcuB6REcg8FbTTUeUpPEvy8IghDJqJ4VwKAg7U1f47C2mho8pIjFLwiCECcrd701VJNIqn+lLUEQhDgkPoS/tpoanSCuHkEQBOJF+L3VVGsPqeLqEQRBiBPhr62hSsvkriAIAsSJ8Nd6q0xUj/j4BUEQ4kT4a0xUT1qyCL8gCEJcCb+EcwqCIMSJ8Gt7clcStAmCIMSP8NeQKFE9giAIxI3w11gpG0T4BUEQ4kT4jY9fwjkFQRDiRPjtXD2SnVMQBCFOhF97q6kmkay0pOYeiiAIQrMTF8Kvasqp1ElkpYvwC4IgxL7w13pJrdzDTrLITktu7tEIgiA0O7Ev/GW7SMBLcUKuFFoXBEEgHoS/dDsA5Sltm3kggiAI0UHsC3/xJgAqUvOaeSCCIAjRQWwLf+VeeOE8ALzpIvyCIAgQ48KvS7Y6263aNeNIBEEQooeYFv7duwoBeJWxjDiiczOPRhAEITpIbO4BRBJvWREA2aMuZuKILs08GkEQhOggpi3+2nIj/Dols5lHIgiCED3EtPDrimKzkZrVvAMRBEGIImJa+KkT/uzmHYcgCEIUEfPCX6U9JKakN/dIBEEQooaYFn5VWUwJGSQlSjpmQRAEm5gW/oTKEkp0OsmJqrmHIgiCEDXEuPAXU0I6SZ6Yvk1BEIT9IqYVceXg27iu+goRfkEQBBcxrYglKR34WXcU4RcEQXAR04pYXVMLQLIIvyAIQh0xrYjVXiP8STK5KwiCUEd8CL9Y/IIgCHVETBGVUqlKqa+UUt8qpb5XSt1mtXdTSi1WSq1RSr2olIpYIdwqrwZE+AVBENxEUhErgXFa6wHAQGCCUuoo4O/ALK11D2APcHGkBmBb/OLjFwRBcIiYImpDqbWbZL00MA6YZ7U/BZwZqTHYk7tJHvHxC4Ig2ETUFFZKeZRSy4EdwPvAOqBIa11jddkEHBri3GlKqSVKqSWFhYUH9PnV3loSFCSKxS8IglBHRBVRa+3VWg8EOgHDgd7BuoU49xGt9VCt9dC8vAOrl1vl1eLfFwRB8KNJVFFrXQR8BBwFZCul7MpfnYAtkfrcam+t+PcFQRD8iGRUT55SKtvaTgOOB1YBC4FJVrepwOuRGkO1t5akRBF+QRAEN5GsudsBeEop5cE8YF7SWs9XSv0AzFFK3QEsAx6L1ACqvbUysSsIguBHxIRfa70CGBSk/WeMvz/iVNbUio9fEATBj5hWxWqvFh+/IAiCHzGtitVi8QuCIAQQ06poJnfFxy8IguAmkpO7zc7grjmUVtY03FEQBCGOiGnh//3Yw5t7CIIgCFFHTLt6BEEQhEBE+AVBEOIMEX5BEIQ4Q4RfEAQhzhDhFwRBiDNE+AVBEOIMEX5BEIQ4Q4RfEAQhzlBaBy2AFVUopQqBDQd4eltgZyMOpyUg9xwfyD3HBwdzz1211gElDFuE8B8MSqklWuuhzT2OpkTuOT6Qe44PInHP4uoRBEGIM0T4BUEQ4ox4EP5HmnsAzYDcc3wg9xwfNPo9x7yPXxAEQfAlHix+QRAEwYUIvyAIQpwR08KvlJqglFqtlFqrlJrZ3ONpLJRSjyuldiilVrracpVS7yul1ljvOVa7UkrNtr6DFUqpwc038gNDKdVZKbVQKbVKKfW9Uuoaqz2W7zlVKfWVUupb655vs9q7KaUWW/f8olIq2WpPsfbXWsfzm3P8B4NSyqOUWqaUmm/tx/Q9K6UKlFLfKaWWK6WWWG0R/bcds8KvlPIADwAnAX2AyUqpPs07qkbjSWCCX9tM4EOtdQ/gQ2sfzP33sF7TgAebaIyNSQ3wB611b+Ao4PfW3zKW77kSGKe1HgAMBCYopY4C/g7Msu55D3Cx1f9iYI/W+nBgltWvpXINsMq1Hw/3PFZrPdAVrx/Zf9ta65h8ASOBd137NwE3Nfe4GvH+8oGVrv3VQAdruwOw2tp+GJgcrF9LfQGvAyfEyz0D6cA3wAjMCs5Eq73u3zjwLjDS2k60+qnmHvsB3GsnS+jGAfMBFQf3XAC09WuL6L/tmLX4gUOBja79TVZbrNJea70VwHpvZ7XH1Pdg/ZwfBCwmxu/ZcnksB3YA7wPrgCKtdY3VxX1fdfdsHS8G2jTtiBuF+4AbgVprvw2xf88aeE8ptVQpNc1qi+i/7Vgutq6CtMVj7GrMfA9KqVbAy8C1WusSpYLdmukapK3F3bPW2gsMVEplA68CvYN1s95b/D0rpU4FdmitlyqlfmU3B+kaM/dsMUprvUUp1Q54Xyn1Yz19G+WeY9ni3wR0du13ArY001iagu1KqQ4A1vsOqz0mvgelVBJG9J/TWr9iNcf0PdtorYuAjzDzG9lKKdtgc99X3T1bx7OA3U070oNmFHC6UqoAmINx99xHbN8zWust1vsOzAN+OBH+tx3Lwv810MOKCEgGzgPeaOYxRZI3gKnW9lSMH9xu/60VDXAUUGz/hGwpKGPaPwas0lrf6zoUy/ecZ1n6KKXSgOMxE54LgUlWN/97tr+LScACbTmBWwpa65u01p201vmY/68LtNbnE8P3rJTKUEq1treB8cBKIv1vu7knNiI8aXIy8BPGN3pzc4+nEe/rBWArUI2xAC7G+DY/BNZY77lWX4WJbloHfAcMbe7xH8D9HoP5ObsCWG69To7xe+4PLLPueSXwF6v9MOArYC0wF0ix2lOt/bXW8cOa+x4O8v5/BcyP9Xu27u1b6/W9rVOR/rctKRsEQRDijFh29QiCIAhBEOEXBEGIM0T4BUEQ4gwRfkEQhDhDhF8QBCHOEOEXBEGIM0T4BUEQ4oz/D2IWkeFkUQ/9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "# Plot the chart\n",
    "chart_regression(pred.flatten(),y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}