t81_558_class_04_2_multi_class.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zt5vYf8K1AF"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-8F-M_WK1AI"
      },
      "source": [
        "# T81-558: Applications of Deep Neural Networks\n",
        "**Module 4: Training for Tabular Data**\n",
        "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
        "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYijxVF5K1AJ"
      },
      "source": [
        "# Module 4 Material\n",
        "\n",
        "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_1_feature_encode.ipynb)\n",
        "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb)\n",
        "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_3_regression.ipynb)\n",
        "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_4_backprop.ipynb)\n",
        "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_5_rmse_logloss.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "By-ggcQxK1AJ"
      },
      "source": [
        "# Google CoLab Instructions\n",
        "\n",
        "The following code ensures that Google CoLab is running the correct version of TensorFlow."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "19pvQwY3K1AK",
        "outputId": "335d3fbc-71d4-425a-e168-4c2d1efb117f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Note: using Google CoLab\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    %tensorflow_version 2.x\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZF1dep09K1AL"
      },
      "source": [
        "# Part 4.2: Multiclass Classification with ROC and AUC\n",
        "\n",
        "The output of modern neural networks can be of many different forms. However, classically, neural network output has typically been one of the following:\n",
        "\n",
        "* **Binary Classification** - Classification between two possibilities (positive and negative). Common in medical testing, does the person has the disease (positive) or not (negative).\n",
        "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
        "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
        "\n",
        "We will look at some visualizations for all three in this section.\n",
        "\n",
        "It is important to evaluate the false positives and negatives in the results produced by a neural network. We will now look at assessing error for both classification and regression neural networks.\n",
        "\n",
        "## Binary Classification and ROC Charts\n",
        "\n",
        "Binary classification occurs when a neural network must choose between two options: true/false, yes/no, correct/incorrect, or buy/sell. To see how to use binary classification, we will consider a classification system for a credit card company. This system will either \"issue a credit card\" or \"decline a credit card.\" This classification system must decide how to respond to a new potential customer.  \n",
        "\n",
        "When you have only two classes that you can consider, the objective function's score is the number of false-positive predictions versus the number of false negatives. False negatives and false positives are both types of errors, and it is essential to understand the difference. For the previous example, issuing a credit card would be positive. A false positive occurs when a model decides to issue a credit card to someone who will not make payments as agreed. A false negative happens when a model denies a credit card to someone who would have made payments as agreed.  \n",
        "\n",
        "Because only two options exist, we can choose the mistake that is the more serious type of error, a false positive or a false negative. For most banks issuing credit cards, a false positive is worse than a false negative. Declining a potentially good credit card holder is better than accepting a credit card holder who would cause the bank to undertake expensive collection activities.\n",
        "\n",
        "Consider the following program that uses the [wcbreast_wdbc dataset](https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "VmjrLkDwK1AM",
        "outputId": "650cc2a4-d295-459d-aa40-d9e2d2950c88"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "         id diagnosis  ...  worst_symmetry  worst_fractal_dimension\n",
              "0    842302         M  ...          0.4601                  0.11890\n",
              "1    842517         M  ...          0.2750                  0.08902\n",
              "..      ...       ...  ...             ...                      ...\n",
              "567  927241         M  ...          0.4087                  0.12400\n",
              "568   92751         B  ...          0.2871                  0.07039\n",
              "\n",
              "[569 rows x 32 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-88642839-c3c7-4244-8079-0889c84bb82e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>diagnosis</th>\n",
              "      <th>...</th>\n",
              "      <th>worst_symmetry</th>\n",
              "      <th>worst_fractal_dimension</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>842302</td>\n",
              "      <td>M</td>\n",
              "      <td>...</td>\n",
              "      <td>0.4601</td>\n",
              "      <td>0.11890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>842517</td>\n",
              "      <td>M</td>\n",
              "      <td>...</td>\n",
              "      <td>0.2750</td>\n",
              "      <td>0.08902</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>567</th>\n",
              "      <td>927241</td>\n",
              "      <td>M</td>\n",
              "      <td>...</td>\n",
              "      <td>0.4087</td>\n",
              "      <td>0.12400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>568</th>\n",
              "      <td>92751</td>\n",
              "      <td>B</td>\n",
              "      <td>...</td>\n",
              "      <td>0.2871</td>\n",
              "      <td>0.07039</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>569 rows × 32 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-88642839-c3c7-4244-8079-0889c84bb82e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-88642839-c3c7-4244-8079-0889c84bb82e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-88642839-c3c7-4244-8079-0889c84bb82e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\n",
        "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
        "    na_values=['NA','?'])\n",
        "\n",
        "pd.set_option('display.max_columns', 5)\n",
        "pd.set_option('display.max_rows', 5)\n",
        "\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ic_JG5KpK1AN"
      },
      "source": [
        "ROC curves can be a bit confusing. However, they are prevalent in analytics. It is essential to know how to read them. Even their name is confusing. Do not worry about their name; the receiver operating characteristic curve (ROC) comes from electrical engineering (EE).\n",
        "\n",
        "Binary classification is common in medical testing. Often you want to diagnose if someone has a disease. This diagnosis can lead to two types of errors, known as false positives and false negatives:\n",
        "\n",
        "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not.\n",
        "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
        "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
        "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
        "\n",
        "Figure 4.ETYP shows you these types of errors.\n",
        "\n",
        "**Figure 4.ETYP: Type of Error**\n",
        "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
        "\n",
        "Neural networks classify in terms of the probability of it being positive. However, at what possibility do you give a positive result? Is the cutoff 50%? 90%? Where you set, this cutoff is called the threshold. Anything above the cutoff is positive; anything below is negative. Setting this cutoff allows the model to be more sensitive or specific:\n",
        "\n",
        "More info on Sensitivity vs. Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "scrolled": false,
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "QfjMVDf2K1AO",
        "outputId": "9fefbe74-69f2-4c55-bad1-459a3c581c28"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUxdvG8e+kh5KQAIoSOogQSgApElCqgCIiLVEQUCmKBbu89vazFwREaSpIMAmggEqvFnoghKpI770FSD/vH7OEIiUJuztbns915cpm27n3ZPNkds6cGWVZFkIIIZzPx3QAIYTwVlKAhRDCECnAQghhiBRgIYQwRAqwEEIYIgVYCCEMkQLsBEqpV5VS65VSKUqpZKVUQzs+92Lb9/JKqQcvuP42pdQQe23HXpRS2bZ9sEYptUop1TgPj0l1RrbLbPctpdQLJrZ9NQ5+P01XShWzXX5aKbVRKRWnlOqglBpkr+0Izc90AE+nlLodaA/UtSwrXSlVAgiw1/NblnWugJUHHgQm2K5fCay013bs6KxlWVEASqk2wAfAnWYjuQ8nvJ/uvuDHAUAry7J2236eZq/tCE1awI53E3DYsqx0AMuyDluWtVcpVU8ptUgplaSUmqWUuglAKbVQKfWRUmq5UuofpVRT2/WRtuuSbS2fKrbrz7UOPwSa2m5/VinVTCn1q1LKRym1/VyrxvaYzUqpG5VSJZVSk5VSK2xf0U7dMxACHLNlKqKUmmdrFa9VSt136Z2vdB9b63+jUmqUrWU4WykVbLutslJq7gUt7kq261+0veYUpdTbF2zjVdt+/xOo6oydkE9Xej9tV0p9bNsvy5VSlQGu9Du27cvvbPdPUUp1tl2/XSlVQin1DVARmGF7P/VWSg2z3edGpdTPtn26Ji+fYsQVWJYlXw78AooAycA/wHB0a88fWAyUtN0nBvjWdnkh8Jnt8t3AXNvloUB32+UAINh2OdX2vRnw6wXbzf0Z+BJ42Ha54QXPOQFoYrtcFtjohP2Rbdsfm4ATQD3b9X5AiO1yCeBfQF3yGi97H3TrPwuIst2WCPSwXV4G3G+7HAQUAu4CRtoe6wP8CtwB1APW2u4TYnv+F0y/h671frJdvx141Xa55wW/+8v+joGPgMEXPG/YBc9T4jKXewPDbJcTgGdsl32BUNP7xV2/pAvCwSzLSlVK1QOaAs3Rb973gBrAHKUU6Dfxvgse9pPtexK6uAAsAV5VSkUAP1mWtTkfMRKAN4DvgFjbzwCtgOq2DAAhSqkilmU5ss/1wi6I24FxSqka6GL4vlLqDiAHKA3cCOy/4LFXug/ANsuykm2Xk4DySqmiQGnLsn4GsCwrzbbdu9BFeLXt/kWAKkBR4GfLss7Y7udyH7kv9366oG/2xwu+f2G7fNnfse362Aue91g+YrRAF3ksy8pG/yMVBSAF2Alsb9KFwEKl1FrgCWC9ZVm3X+Eh6bbv2dh+R5ZlTVBKLQPuAaYrpfpbljU/jxGWAJWVUiWBjuh/AKBbf43OFSZnsyxria0PsyS6tV8S3SLOVEptR7dYL9T9KvdJv+B+2UDwVTatgA8syxpx0ZVKPVPQ1+JMl3k/9Tp304V3s32/7O/4goIsDJI+YAdTSlU9119rEwVsBEraWoAopfyVUpHXeJ6KwFbLsoYAU4Fal9zlFLoF9x+W/qz4M/A5+iPoEdtNs4GnLthGVJ5fmB0opW5Ft/6PAKHAQVthbQ6Uu8xD8nKfXJZlnQJ2K6U62rYXqJQqBMwCHrG1BFFKlVZK3QD8DnRUSgXbWs/32ueV2s8V3k87bJdjLvi+xHb5Sr/jOeiGwLnrw/IRYx7wuO1xvkqp0Hw8VlxACrDjFQHGKqU2KKVSgOro7oAuwEdKqTXoPr1rHcjoBqxTSiWjuy/GXXJ7CpBtOyjy7GUenwD04Hz3A8DTwG22gzAbgMfy+doKIth2oDDZlqWXrUUXZ8uyFv3xdtNlHpuX+1zqIeBp275fDJSyLGs2um90ie25JgFFLctaZcu0BpgBrLieF+ogl3s/vWW7Lcx23UDg3HvgSr/j92z3X2d7DzbPR4aBQHPbvkuyZRAFcO4ghxDCjdm6Y26zLOuw6Swi76QFLIQQhkgLWAghDJEWsBBCGCIFWAghDJECLIQQhkgBFkIIQ6QACyGEIfk6FblEiRJW+fLlHRRF5Mfff/8NQNWqrjhhl3PJvjhP9oVrSkpKOmxZVslLr89XAS5fvjwrV7riFLPep1mzZgAsXLjQaA5XIPviPNkXrkkpteNy10sXhBBCGCIFWAghDJECLIQQhsh8wEKIXJmZmezevZu0NCNTRLu9oKAgIiIi8Pf3z9P9pQALIXLt3r2bokWLUr58eZm0PZ8sy+LIkSPs3r2bChUq5Okx0gUhhMiVlpZG8eLFpfgWgFKK4sWL5+vTgxRgIcRFpPgWXH73nRRgIYRLUUrx/PPP5/786aef8tZbb9l9O++///5FPzdufK1FaexPCrAQwqUEBgby008/cfiwYxf3uLQAL1682KHbuxwpwEIIl+Ln50e/fv344osv/nPboUOH6Ny5M/Xr16d+/fr89ddfude3bt2ayMhI+vTpQ7ly5XILeMeOHalXrx6RkZGMHDkSgEGDBnH27FmioqLo3r07AEWKFAEgNjaW3377LXebvXv3ZtKkSWRnZ/Piiy9Sv359atWqxYgRFy2qXSBSgIUQLueJJ54gLi6OEydOXHT9wIEDefbZZ1mxYgWTJ0+mT58+ALz99tu0aNGC9evX06VLF3bu3Jn7mG+//ZakpCRWrlzJkCFDOHLkCB9++CHBwcEkJycTFxd30TZiYmJITEwEICMjg3nz5nHPPfcwZswYQkNDWbFiBStWrGDUqFFs27btul6nDEMTQlxRs++b/ee6bpHdGFB/AGcyz3B33N3/ub13VG96R/Xm8JnDdEnsctFtC3svzNN2Q0JC6NmzJ0OGDCE4ODj3+rlz57Jhw4bcn0+ePElqaip//vknP//8MwBt27YlLCws9z5DhgzJvW3Xrl1s3ryZ4sWLX3Hb7dq1Y+DAgaSnpzNz5kzuuOMOgoODmT17NikpKUyaNAmAEydOsHnz5jwPObscKcBCCJf0zDPPULduXR5++OHc63Jycli6dClBQUF5eo6FCxcyd+5clixZQqFChWjWrNk1h4kFBQXRrFkzZs2aRUJCArGxsYAe5zt06FDatGlT8Bd1CSnAQogrulqLtZB/oaveXqJQiTy3eC8nPDycbt26MWbMGB555BEA7rrrLoYOHcqLL74IQHJyMlFRUURHR5OYmMjLL7/M7NmzOXbsGKBbqWFhYRQqVIhNmzaxdOnS3Of39/cnMzPzsmetxcTEMHr0aFauXMn3338PQJs2bfj6669p0aIF/v7+/PPPP5QuXZrChQsX+DVKH7AQwmU9//zzF42GGDJkCCtXrqRWrVpUr16db775BoA333yT2bNnU6NGDSZOnEipUqUoWrQobdu2JSsri2rVqjFo0CAaNWqU+1z9+vWjVq1auQfhLnTXXXexaNEiWrVqRUBAAAB9+vShevXq1K1blxo1atC/f3+ysrKu6/Xla1n62267zZL5gF2DzPt6nuyL8653X2zcuJFq1arZL5CTpKen4+vri5+fH0uWLOHxxx8nOTnZSJbL7UOlVJJlWbddel/pghBCuL2dO3fSrVs3cnJyCAgIYNSoUaYj5YkUYCGE26tSpQqrV682HSPfpA9YCCEMkQIshBCGSAEWQghDpAALIYQhUoCFEC7F19eXqKgoatSoQdeuXTlz5ky+Hr937166dNGnQCcnJzN9+vTc26ZNm8aHH35o17zXQwqwEMKlnJskZ926dQQEBOSebJFXN998c+58DZcW4A4dOjBo0CC75r0eUoCFEC6radOm/Pvvvxw9epSOHTtSq1YtGjVqREpKCgCLFi0iKiqKqKgo6tSpw6lTp9i+fTs1atQgIyODN954g4SEBKKiokhISOD777/nySef5MSJE5QrV46cnBwATp8+TZkyZcjMzGTLli20bduWevXq0bRpUzZt2uSw1ycFWAjhkrKyspgxYwY1a9bkzTffpE6dOqSkpPD+++/Ts2dPQK+W8dVXX5GcnMwff/xx0cxpAQEBvPPOO8TExJCcnExMTEzubaGhoURFRbFo0SIAfv31V9q0aYO/vz/9+vVj6NChJCUl8emnnzJgwACHvUY5EUMIcXnPPAP2Pp03KgoGD77qXc5NlA66Bfzoo4/SsGFDJk+eDECLFi04cuQIJ0+eJDo6mueee47u3bvTqVMnIiIi8hwlJiaGhIQEmjdvTnx8PAMGDCA1NZXFixfTtWvX3Pulp6cX4IXmjRRgIYRLOdcHnBeDBg3innvuYfr06URHRzNr1qw8T1XZoUMHXnnlFY4ePUpSUhItWrTg9OnTFCtWzGnzSEgBFkJc3jVaqs7UtGlT4uLieP3111m4cCElSpQgJCSELVu2ULNmTWrWrMmKFSvYtGlTbusZoGjRopw6deqyz1mkSBHq16/PwIEDad++Pb6+voSEhFChQgUmTpxI165dsSyLlJQUateu7ZDXJX3AQgiX99Zbb5GUlEStWrUYNGgQY8eOBWDw4MHUqFGDWrVq4e/vT7t27S56XPPmzdmwYUPuQbhLxcTEMH78+Iv6h+Pi4hgzZgy1a9cmMjKSqVOnOux1SQtYCOFSUlNT/3NdeHg4U6ZM+c/1Q4cO/c915cuXZ926dbmPW7FixUW39+7dO/dyly5duHRK3goVKjBz5syCRM83aQELIYQhUoCFEMIQKcBCCGGIFGAhxEXys0yZuFh+950UYCFErqCgII4cOSJFuAAsy+LIkSN5HocMMgpCCHGBiIgIdu/ezaFDh0xHcUtBQUH5OhtPCrAQIpe/vz8VKlQwHcNrSBeEEEIYIgVYCCEMkQIshBCGSAEWQghDpAALIYQhUoCFEMIQKcBCCGGIFGAhhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCEyG5qbOXb2GMv2LGP78e1k52Sz4/gOyhUrZzqWMO3wYVi6FLZtA8uCyZPh9tvh5ptNJxNXIQXYTew9tZfHf3ucaX9P01ecAIUiKycLgBV7VpCWlUbTck0NphRO9++/0L8/zJ9//jqloEsXfblePRg5EurWNZNPXJV0QbiJM5lnWLxrMa82fZV5PefRpEwTmpZtSsWwigC8sfAN7vz+Tp6e8TSnM04bTiscLjsbvvgCatWCpCR4911YtAiaNoUmTXRr+LPPYO9eaNAAXnsN0tNNpxaXkALswrJzspmwdgKWZVE5vDI7ntnBey3eo0WFFvj6+KKUQikFwMSuE3mywZMMXT6UuiPrsu/UPsPphcOkpcHdd8Nzz0HLlrB+vS6wd9wBPj76q2FDffv69dCjB/zvf7pL4tgx0+nFBaQAuyjLsuj/a3+6/9SdedvmAVDIv9AV718koAhD2g1hfs/57Dm5h7vG38Wxs/LH5nGysuDBB2H2bPj6a5g2DUqXvvL9w8Lg++9hyhRdjNu3hzNnnBZXXJ0UYBc1aO4gxqwew6tNX6VVxVZ5flzzCs2ZGjuVmjfUvGrBFm7IsuCxx+Dnn+HLL/Vl2yega7rvPpgwQXdNdOkCGRmOzSryRAqwC/piyRd8vPhjHqv3GO82fzffj29ZsSUTOk8g0C+QtKw0snOyHZBSON1rr8GYMfD66/D00/l/fOfO8M03MGMGPPKILujCKCnALubg6YO8Mv8V7r/1fobdPSy3j7cgjpw5Qt0Rdfly2Zd2TCiMWLAA3n8f+vSBt98u+PP07QvvvANxcTB+vP3yiQKRAuxibih8Awt6LWB0h9H4+vhe13OFB4dTObwyr8x7hY2HNtopoXC6kyfh4YehShXd9XAd/5QBeOUViI6Gp56C3bvtk1EUiBRgF7L9+HYAGkU0Ijw4/LqfTynFyHtHUjigML2m9ModMyzczAsvwK5d+mBaITv06/v66ufKzNQtaumKMEYKsItYtnsZVYZWYcLaCXZ93lJFSjH87uGs2LuCj//62K7PLZxg5kwYNUoX4caN7fe8lSvDxx/DrFkwerT9nlfkixRgF5Cdk03fX/pyU5GbuKfKPXZ//pgaMXSL7Ma0v6dJK9idnD0L/fpB9erX1+97JY8/rscRP/88HDhg/+cX1ySnIruAcWvGsfbgWiZ2nUhoUKhDtvFth28J9g/GR8n/XLcxdKjueliwAIKC7P/8Pj7w1VcQGanPpBs2zP7bEFclf42Gnc08y+sLXqdh6YZ0rtbZYdspHFAYH+XDsbPH2Hpsq8O2I+zkyBE96uGee6BZM8dtp2pV3coeMQI2b3bcdsRlSQE2bNW+VZxIP8HHrT++riFneZFj5XD7mNt5dNqjWHLgxbW9/z6cOgUffuj4bb35JgQG6tERwqmkABsWXTaaXc/u4o5ydzh8Wz7KhycbPMnC7QuZ8e8Mh29PFND27bo7oFcvqFHD8du78UZ9kG/SJFi2zPHbE7mkABu09sBaLMuiWFAxp22zX71+VA6vzMtzX5Yz5FzV66/r/tl33nHeNp9/Xhfil16SYWlOJAXYkAOpB6g/qj7v/f6eU7cb4BvAe83fY93BdUzZNMWp2xZ5sHmzPkvtqacgIsJ52y1aVBf+33/XX8IppAAbMnT5UDKyM+gW2c3p2+5SvQtVwquwZPcSp29bXMNnn0FAgJ5K0tkeeQRuuAE++sj52/ZSMgzNgFPpp/hqxVd0vLUjVUtUdfr2fX18SeqXRNHAok7ftriK/fv1GWq9ekGpUs7ffnCwnuTntdcgJUVP9i4cSlrABoxeNZrjacd5OfplYxnOFd8DqTIA32UMGaKniXzhBXMZBgyAwoX1WXLC4aQAGzBxw0TuLHcnDSMaGs0x7e9pRHwRQfL+ZKM5BHrCneHD9ZSRVaqYyxEWpscFx8fDjh3mcngJKcAGLOy9kPGdzE8FeEe5Owj2C5Y5IlzByJFw4gS8bO5TUa5nn9Uzrn3+uekkHk8KsBNZlkVWThYBvgFEhDjxCPcVFAsqRv96/UlYn8CO49LaMSYzEwYPhhYt4LbbTKeBMmWge3c9Sc/Ro6bTeDQpwE70+47fqfBlBdbsX2M6Sq6nGj4FwIikEYaTeLGpU2HPHnjmGdNJznv2Wb123NixppN4NCnATjR85XBOZ5ymSnGDfXyXKBtalva3tGfsmrFyYoYpX38N5crplY5dRe3aehXlr7+GnBzTaTyWFGAn2Z+6n582/kTvqN4ut1jmJ60/YXmf5de9AocogE2bYP586N9fT5TuSgYM0CeGzJ9vOonHkgLsJGNWjSErJ4vHbnvMdJT/uKX4LZQOucrS5sJxvvkG/P3h0UdNJ/mvLl2gRAk9OkM4hBRgJ8jOyWZE0ghaVWzFLcVvMR3nsv458g+tf2jN2gNrTUfxHqdP6xMvunTRZ6C5mqAgfXbctGmydpyDSAF2gnNrs73dzAGrGthJ8eDi/LHjD75e+bXpKN4jPl4PPXv8cdNJrqx/f90HPGqU6SQeSQqwE/goH9pWbkvjMnZc08vOihcqTkyNGH5I+YFT6adMx/EOw4fr6SabNDGd5MoqVoR27fQ45cxM02k8jhRgB9txfAeD5g5i36l9pqNc02P1HiM1I5WE9Qmmo3i+1ath1SrdwnTwRPzX7bHH9DwV06ebTuJxpAA72Ng1Y/nor49Iz043HeWaGkU04tYSt/Jd8nemo3i+777Ts549+KDpJNfWrp2eK/g7eV/Ym8yG5kA5Vg7fJ39PiwotKF+svOk416SU4v+a/B+HTh/CsiyHL5HktdLT9Zy/HTtCeLjpNNfm5wcPPaTP1jt40DUPGLopaQE70O87fmfb8W08HPWw6Sh51rN2T55v/LwUX0f65Rd9iu/D7vO+4OGHISsLxpufw8STSAF2oO+TvyckMIRO1TqZjpIvZzLPEL8uXs6Mc5TvvoPSpaF1a9NJ8q56dWjQQGeXJYvsRgqwA/n7+NO7tuud+XYtMzbP4IHJDzBn6xzTUTzP3r0wcyb07Ol6Z75dy8MPw7p1kJRkOonHkALsQKM6jOLLdl+ajpFv7W9pT3hwuByMc4QfftDjanv3Np0k/2Jj9ckZcjDObqQAO8jWY1tNRyiwQL9AetTswZRNUzh6VqYjtBvL0me+NWkCt7jmGZFXVawYdOoEEyZAWprpNB5BCrADbD22lUpDKjFm1RjTUQqsd1RvMrIzmLh+oukonmPlSj35Tq9eppMUXO/ecPw4/Pab6SQeQQqwA8SlxAHQupIbHWS5RFSpKKqXrM7i3YtNR/Ec48dDYKCe+8FdtWihFwyV0RB2IeOA7cyyLOLWxnFHuTsoG1rWdJwCU0rxx8N/EBYUZjqKZ8jK0nM/tG+vP8q7K19feOAB+OorPZTOHcYxuzBpAdtZ0r4k/j7yNz1q9jAd5bqFB4ejlMKSYUfXb+5cfRJDD/d/X9Cjh169edIk00ncnhRgO5uwdgIBvgF0qe7GHzMv8NGfH3H7mNulCF+vuDjd8m3XznSS61enDtx6q35N4rpIAbazN+98k98e/I2wYM/46B4aFMqyPctk6frrcfo0/PwzdO2q+4DdnVJ60c7ff4edO02ncWtSgO0sNCiUVhVbmY5hN12rd8Xfx5+4tdLaKbCpU3UR9oTuh3POTSI0YYLZHG5OCrAd/e/3/7n10LPLKV6oOO2qtGPC2glyanJBxcXppd5ded7f/KpYERo31qMhpHuqwKQA28mZzDN8+NeHLNm9xHQUu+tRswf7UvexcPtC01Hcz6FDMGuWbjH6eNifW48esH49pKSYTuK2POwdYc70zdNJzUjlwZpuML9rPrW/pT0v3P4C5YqVMx3F/UyeDNnZ7jHvb3516aKHpSXIBP4FJQXYTuLXxXNj4Ru5s9ydpqPYXbB/MJ/c9QmVwyubjuJ+4uP1TGI1a5pOYn8lS0KrVvo1SjdEgUgBtoOT6Sf5bfNvdIvshq+Pm81wlUc5Vg4Lti1gzf41pqO4jz179EiB2FjXX3aooGJjYds2WL7cdBK3JAXYDg6kHqBRRCMeqPGA6SgOk5WTRefEzny65FPTUdxHYqJuGcbEmE7iOB076qWV4uNNJ3FLUoDtoErxKizotYDby9xuOorDBPgG0LlaZ6ZsmsLZzLOm47iH+HioW9c9Zz7Lq2LF4O67dT9wtoySyS8pwNcpNSOVw2cOm47hFLE1YknNSGX6Zlkd95q2btUfy2NjTSdxvNhY2LcP/vzTdBK3IwX4Ov249kdKfVqKf4/+azqKwzUr34wbC9/Ij+t+NB3F9Z0bGdCtm9kcztC+PRQqJN0QBSAF+DrFr4+nUnglKoVVMh3F4Xx9fOkW2Y0Ve1eQlZNlOo5ri4/XJyqU84Khe4ULQ4cOenKezEzTadyKFODrsD91Pwu3LyQmMsZrVhF+t/m7bHl6C34+MpPpFW3cqE9O8Ibuh3NiY+HwYZg3z3QStyIF+DpM2jCJHCuHmEgPPsp9idCgUCm+15KQoIedufPE6/nVpg2EhOiRHyLPpABfh8T1iUSWjCTyhkjTUZzq139+peqwqhxPO246iuuxLF2E7rwTbrrJdBrnCQqC++7Ts75lZJhO4zakAF+HsR3HMqL9CNMxnK5koZL8c+QfpmyaYjqK61m3TndBePLY3yuJidHrxc2ZYzqJ25ACfB0qhFUgumy06RhO16B0A8oXK0/ievm4+R+JiXrSnU6dTCdxvtat9bhg6YbIMynABfR/c/+P2Vtmm45hhFKKbtW7MWfrHI6cOWI6juuwLN3/26IF3HCD6TTOFxAA998PU6bIsvV5JAW4AHae2MmHf33Iyr0rTUcxpltkN7JysqQb4kJr1sDmzd4x9vdKunWDkydhtnc2TvJLCnABTFw/EdBFyFvVvakuAxsOpFrJaqajuI6EBD09ozd2P5zTsqVeKVmmqMwTGU9UAIkbEql7U12vnp5RKcXgtoNNx3Ad50Y/tGoFxYubTmOOv7/+BxQfD2fPQnCw6UQuTVrA+bTt2DaW71lOt+re2/q90IZDG7y6KyZXUpKe/8Gbux/OiYmB1FSYMcN0EpcnBTif9pzaw60lbvXq7odzLMuiU0InXprzkuko5iUk6Nbf/febTmJes2Z6snbphrgmKcD51KRsEzY+sZEKYRVMRzFOKUW3yG4s2rGI/an7Tccx51z3w113QViY6TTm+flB587w6696NWhxRVKA8+FU+inSs9JNx3ApMZEx5Fg5TN4w2XQUc5Ytg507vfPkiyuJiYEzZ+C330wncWlSgPNh6PKhlPqsFCfSTpiO4jIib4ikesnqJG7w4sH3iYl6DGyHDqaTuI6mTaFUKTkp4xqkAOdDwvoEqpesTmhQqOkoLiUmMoYVe1Z459wQOTm6yLRrB6Hyvsjl66snI/rtNzh1ynQalyUFOI82Hd5EyoEUGf1wGU81eIr9L+ynWFAx01Gcb/FivfimjH74r5gYfUbcL7+YTuKypADnUeL6RBSKLtW9aIrBPAoLDiMkMMR0DDMSE/VMYPfeazqJ62ncGEqXlm6Iq5ACnEeJ6xNpUrYJpUNKm47ikpbtXkaj0Y3YeWKn6SjOk50NEyfqRSmLFjWdxvX4+EDXrno88Ak5bnI5UoDzwLIsRncYzfst3zcdxWUVL1ScZXuW5Z6m7RX++AP275fRD1cTE6PnB5461XQSlyQFOA+UUjSKaESTsk1MR3FZlcMrU++meiSs96LB9wkJej20e+4xncR1NWyo18WTkzIuSwrwNViWxavzXiV5f7LpKC4vtkYsK/auYMvRLaajOF5mpl6EskMHXYTF5SmlW8GzZ8MRmbr0UlKAr2HVvlW8/+f7JO1NMh3F5Z07PdsrWsHz5+tFKL1p4c2Cio2FrCz46SfTSVyOFOBriF8Xj7+PP/dXk3P8r6VsaFmebvA01Up4wRSV8fF63G+bNqaTuL6oKLjlFr3PxEVkOsqryLFySFifQJvKbQgPDjcdxy182e5L0xEcLz1dt+Y6d4bAQNNpXJ9SuhX87ruwb593LVZ6DdICvoolu5aw6+QuYiPlY2Z+HDx9kOV7lpuO4TgzZ+pVH6T7Ie9iYjUl3aoAABi8SURBVPSkRZMmmU7iUqQAX8Wuk7uICImgQ1U5xz8/uv/UnR4/9cCyLNNRHCM+HkqU0Gu/ibypXh1q1ZJuiEtIAb6K2Bqx7HhmB0UDZZB9fsRExrD56GbPHDly+jRMm6bnOfCTHrx8iY3Vp27v2GE6icuQAnwFqRmpWJaFj5JdlF+dqnXCz8ePCWsnmI5if7/8oqdZlO6H/Dt3woq0gnNJdbmCp2c8Td2RdT33Y7QDhQeH07ZyW+LXx5Nj5ZiOY19xcRARoadbFPlTsaI+MWOCB/5jLiApwJeRlpXG5I2TiSoVhVLKdBy31L1md3af3M3qfatNR7Gfw4f1AbgHHtDzHIj8694dUlJg3TrTSVyCvIsu49d/fuVk+km61+xuOorbuq/qfWwbuI16N9czHcV+Jk7UJxR0l/dFgXXrpucKllYwIAX4suLWxlGqSCmal29uOorbCvYPpnyx8qZj2NeECeeP5ouCufFGaNVK78scD+ueKgApwJc4dvYY0zdPJzYyFl8fX9Nx3NqB1AN0+LEDv/ztARNy79gBf/6pW7/SLXV9unfX+3PJEtNJjJNxNJcoHFCYhC4J3nE6rYOdm6Iy0C+Qe6u6+YTl5z4yP/CA2RyeoGNHCA7WBzSjo02nMUpawJcI8A2g460dqVqiqukobs/Px4+YyBh++fsX917I1LJ0sWjcGCpUMJ3G/RUtCvfdp1fKyMw0ncYoKcAX2HNyD28tfIv9qftNR/EYD9Z8kPTsdH7a6MYzYa1dC+vXw4MPmk7iOR58UE9POWuW6SRGSQG+wPiU8by96G1SM1JNR/EYDUs3pFJYJX5I+cF0lIIbN06f9SYLb9pPmzb6dO4f3Ph9YQfSB2xjWRZj14wlukw0lcMrm47jMZRSvNj4RQ6fOYxlWe43rjorC8aP16telCxpOo3nCAjQ/ekjR8KxYxAWZjqREdICtlm5dyUbD2+kZ+2epqN4nP639efVO151v+ILeiWHAwegVy/TSTxPz556ak8vXjVZCrDN2DVjCfQNzF3VQdhXWlYav/z9i/ud2j1uHISH65WPhX3Vq6fHVY8bZzqJMVKAbdKy0ugW2Y1iQcVMR/FICesS6BDfgaW7l5qOknfHj8OUKfqjsky8bn9K6U8WixfD5s2m0xghBdhmdIfRjO041nQMj9WpWieC/YIZu8aN9nFiov6ILN0PjtOjh55Xw0tbwVKA0Ss4AO7ZR+kmigYWpVO1TiSsTyAtK810nLwZNw6qVYPbbjOdxHPdfLM+NfmHH7zy1GSvL8CHzxwm4vMIhi4bajqKx+tVuxfH0467x6nJ//4Lf/2lW7/yj9mxevXSpyb//rvpJE7n9QV4fMp4MnMyaVa+mekoHq9FhRZEhEQwe8ts01Gu7bvv9EfjHj1MJ/F8HTtCSAh8+63pJE7n1QXYsixGrxpNg9INqHljTdNxPJ6vjy9LHl3CyHtHmo5ydVlZugC3awelS5tO4/kKFdJnxk2cqA98ehGvLsBLdy9l/aH19K3b13QUrxEREoFSyrWHo82YoZdP7yvvC6fp2xfS0vScG17Eqwvwt6u/pbB/YWIiY0xH8Spfr/iaqBFRZOdkm45yeaNHQ6lSMvbXmerWhTp1YNQoPfmRl/DqAvxR64+YGjtVVj12shKFSpByIIU5W+eYjvJfe/fCb79B797g7286jXfp2xfWrIFVq0wncRqvLsDhweG0rNjSdAyv06FqB0oUKsHoVaNNR/mv77+H7Gx49FHTSbzPAw/oeYJHjTKdxGm8tgD3ndaXKZummI7hlQL9AulVuxdT/56aOwbbJeTkwJgx0Lw5VJYJmZyuWDHo2lVPfn/6tOk0TuGVBXjN/jWMXj2anSd2mo7itR6t8yhZOVmMTXahM+MWLICtW6FPH9NJvFffvnDqlNdM0OOVBXj4iuEE+QXRo5aM8TSlWslqfNDyA9pUbmM6ynnDh0Px4nD//aaTeK/oaD1Bz/DhppM4hdcV4ONpxxm/djwP1niQ8OBw03G82qAmg6h1o4usMLxrF0ydqvt+g4NNp/FeSsGAAbByJSxfbjqNw3ldAR6bPJYzmWd4osETpqMIYNW+VXy2+DPTMWDECN0H/PjjppOInj31unHDhplO4nBeV4DLhpbl0TqPUvemuqajCOCXv3/hhTkv8O/Rf82FSE/XR97vvRfKlzeXQ2hFi+oinJAAhw6ZTuNQXleA7692P6M7uODwJy/Vr14//Hz8GL7CYJ/fpElw8CA8IZ+KXMYTT0BGhj4pxoN5VQGeumkqx9O861xzV3dT0ZvoXK0z3yV/x+kMQ0OPvvoKbrlFT4soXEO1atCiBXzzjZ6bw0N5TQHedmwb9yfcz+dLPjcdRVziyQZPcjztOBPWTnD+xletgiVLdIvLx2v+HNzDk0/Czp3w66+mkziM17zjhiwbgq+PL/3q9TMdRVwiukw0d5a7k7NZZ52/8S++gCJFZNULV3TvvVC2rP4deSivWJb+2NljjFo1itgasUSERJiOIy6hlGJBrwXOX5Fk1y6Ij4ennoLQUOduW1ybnx888ww895wektaggelEducVLeBvVn7D6czTvHD7C6ajiCs4N0Xlkl1LnLfRL7/UM28NHOi8bYr86dNH/3P89FPTSRzCKwrwukPraF2xNbVL1TYdRVzFmNVjaPxtY1buXen4jZ04ASNHQkwMlCvn+O2JgilaFB57DCZP1qeJexivKMBxneKYGjvVdAxxDd0iuxESGMKni53Q2hkxQs858IJ8KnJ5Tz8Nvr7wuecdQPfoApxj5bD31F4Agv3l9FJXFxIYQv96/Zm4YSLbjm1z3IYyMnT3Q8uWehJw4dpuvlmvzfftt3D4sOk0duXRBXj65umUH1yexbsWm44i8mhgw4H4Kl/HDheMi9MTr0vr1308/zycPavHbHsQjy3AlmXxzqJ3KB1Smvo31zcdR+RR6ZDS9Kzdk1lbZpGZnWn/DWRlwf/+B1FR0MaFZmITVxcZCR06wODBuv/eQ3hsAZ7x7wxW7F3Ba01fw99XlpZxJ5+0/oS1j691zO8tLg62bIG33tIzbwn38eabetXkIUNMJ7EbjyzAlmXx1sK3KF+sPD1r9zQdR+RTWHAYgX6BpGelcyLNjq2drCx4913d+u3QwX7PK5yjbl39e/v8c49pBXtkAV53cB2r9q2S1q8bS8tKo9pX1XhjwRv2e1Jp/bo/D2sFe2QBrnljTf556h9p/bqxIL8gmpdvzoikEbkjWa6LtH49g4e1gj2uAJ9MPwlAxbCK0vp1c6/e8SpZOVl88McH1/9kP/wgrV9Pca4V/OWXppNcN48qwFk5WTQe05iBM+TUUk9QMawij9R5hG+SvmHzkc0Ff6IzZ+CNN+C226T16wnq1tXr9n3yCRw4YDrNdfGoAvzd6u9Yf2g9d5a/03QUYSfvNH+HQN9Axq65jtWTBw+G3bv1fALS+vUMH34IaWn6E40b85gCnJqRyusLXie6TDT33yqr2nqKUkVKkdQviXebv1uwJzhwAD74AO67D+6Uf8we45Zb9Pp9o0bBhg2m0xSYxxTgT/76hAOnD/DZXZ85f1pD4VBVS1RFKcWh04ewLCt/D377bd1S+vhjx4QT5rzxBhQuDC+/bDpJgXlEAc7IzuDb5G+JiYyhYURD03GEA6QcSKHikIokrk/M+4M2btQznj32mG4xCc9SogS8+qpeMWP+fNNpCsQjCnCAbwApj6UwuO1g01GEg0SWjKRSWCVenPMiqRmp137AuXl+CxfWLSXhmZ5+Wk8nOnAgZDrg1HUHc/sCvOP4DrJzsgkLDqNUkVKm4wgH8fXx5au7v2LXyV28tfCtaz/gxx9hzhw970PJkg7PJwwJCtInZaxb55bTVbp1AT6beZaW41rS4+cepqMIJ4guG03fun0ZvHQwyfuTr3zHY8fg2Wehfn19oEZ4tg4doGNH3d+/zYHTmDqAWxfg//3xP7Yc20Lfun1NRxFO8mGrDylRqASzt8y+8p3+7//0vLEjRuiJvIXnGzJE/64HDNDdT27CbQvwhkMb+Pivj3mo1kO0qNDCdBzhJOHB4Wx8YiMvRb90+TssXqwL7zPPyGTr3qRMGXjvPZg5EyZONJ0mz9yyAGdmZ9JrSi+KBhbls7s+Mx1HOFlYcBgAy/csZ8vRLedvOH0aHn5Y/zG+/bahdMKYJ5+EevXgiSdg/37TafLELQvwvtR9nEo/xcj2IylZWA6weKMzmWdoP6E9PX7ugYXtI+dzz8HmzTB2LBQpYjagcD5fXxg3DlJT9T9iN+iKcMsCXDa0LCmPp9C5emfTUYQhhfwLMbTdUJbuXsqO4zt0n+/IkXqZoebNTccTplSvrueImDkThg83neaa3KoAn0g7wSvzXuF0xmkCfANMxxGGxdSIoUetHuw7sgPr7016qsl3C3jKsvAcTzwB7drpf8Yufpqy2xRgy7Lo80sfPv7rY9YfWm86jnARw1oPpvoRhZWdzYlvv4bAQNORhGlK6RWUixSBmBjdJeGi3KYAf/DnB0zaMIkPW31Ig9INTMcRLiL09fcIPWtxNKI4RaPkfSFsSpWCCRN0C7h3b5ftD3aLAvzrP7/y2vzX6F6zO8/f/rzpOMJVjBunp5osXZoSFWvgo3w4nnbcdCrhKlq31pMwTZ6sz4h0QS5fgDOyM3hy+pPUuakOo+4dJTOdCW3FCujXTx9wq1QJgL8P/80tQ2/hu9XfGQ4nXMZzz0GPHvD66zBtmuk0/+HyBTjAN4DZD83m55ifCfYPNh1HuIJ//oF77oGbboLExNxJ1iuFV6LWjbXo/2t/ZmyeYTikcAlK6dEx9erBAw/AkiWmE13EZQvwodOHGLZ8GJZlcUvxWygbWtZ0JOEKdu6EVq305Vmz9JSENn4+fkzuNpkaN9Sgc2Jn/tjxh6GQwqUEB+spK2++Ge6+G9asMZ0ol0sW4BNpJ2gb15YX57zI1mNbTccRruLAAd2vd/IkzJ592Tl+Q4NCmdljJmVDy9L+x/asPygjZgT6oNzcuXpkxF136RN2XIDLFeAjZ47QZnwbUg6k8FO3n6gUXsl0JOEKdu+GFi30999+02N+r+CGwjcw56E5xEbGUjGsohNDCpdWrpyeotSy9LEDFxgj7FIFeNeJXTT9rinJ+5OZ1HUS7aq0Mx1JuIJNmyA6Gnbt0sU3OvqaDykTWoYR944g2D+YI2eOsHD7QsfnFK7v1lth3jzIzoamTY33CbtUAU7en8z+1P3M6jGL+269z3Qc4QqWL4cmTfS6bosWQbNm+X6Kl+a8ROsfWjNh7QT75xPup2ZNPWteeDi0bAnTpxuL4hIFePMR3R9zb9V72TpwqywrL7QxY+COOyAkBP76q8DTS37e5nOiy0TT/afuvDTnJbJysuwcVLidChX0e6paNbj3Xnj/fcjJcXoMowU4MzuT52Y9R7WvqrF091IAigUVMxlJuIKzZ+HRR6FPH/0xcdkyqFy5wE8XGhTKrB6zePy2x/lk8Se0HNeSA6kH7BhYuKUbbtCfqmJi9OKe992nV1NxImMFeP3B9TT5rglfLP2CAfUHUPemuqaiCFeyYgU0aKDP5X/tNT2rlR3WdAv0C2T4PcP54f4fOHLmiEzmJLQiRSAuDoYN08Mao6L0CBsnMVKAP/rzI+qMqMOWo1tI6JLAkHZD5A/C2505o2evatQIjh7V/XLvvmv3JYV61OrBmsfWEBYcRnpWOs/Neo79qe4xebdwEKX0DGp//KHHDLdpo+cTPnrU4Zt2WgG2LAvLNiGGUoqukV3Z+MRGukV2c1YE4YpycmD8eD2P62efQd++enhQO8eNgPH10UV9+Z7lDF8xnFuH3cpHf37E2cyzDtumcAMNG0JyMrzyCvzwgx4xMWwYZGQ4bJMOL8CWZTF/23waf9uY+HXxALzY+EXiOsXJahbeLCdHn51Uty489JA+Ir1gAXzzDYSGOiVC03JNWfPYGpqWa8qgeYOoMrQKo5JGkWM5/2CMcBFBQXrinpUrITISnnpKH6iLi4Ms+x+8dVgBTs9KZ9yacdQdWZeW41qy68Qu/H39AWRCHW925oxeNDMyUh99PnUKfvxRv+ELMMTselUtUZVfHviFRb0XUSa0DGNWj8FH6T8LaRF7sagomD8fZsyAokX1hD4VK+rVNo7bb8Y9hxXg++Lvo9eUXmRkZzDq3lFsfmozXap3cdTmhCvLydH9a/36QenS8NhjULiw7nrYuBFiY8HH7IjIO8rdweJHFvPLA78AcPjMYW7+/Ga6JHZhyqYpZGQ77mOocFFKQdu2sGoVTJ2qZ9176SX9Hu7eXR8gvs5Wsd/1ZkzLSmPR9kVM3zydedvm8dcjfxEaFMozjZ7hudufo3XF1tLi9UanT+sWxPTp+uy1Xbt00e3U6fzwMhd7XyilcrvFsnKy6FmrJ/Hr45m8cTJhQWG0rdyWN+58g1tL3Go4qXAqHx/o0EF/rV6tP8ElJuoJ30uW1BP83H23nmOiWP6G0ea7AGflZOHn48ey3ct4cc6LLN+znPTsdIL8gmhRoQUHTh8gNCiUtpXb5vephbuyLNi+Xb85Fy/WA9yTkiAzUxfd1q3hgw+gY0f9sxsoVaQUX7b7ks/afMbcrXOJXxfPjH9n8HYzvdz91E1TmfnvTKLLRlOnVB2qlqiKn891t2eEq6tTRx+n+PJL3T2RmKjnGR47Vhfq2rX1qfLR0fq+lStfdSRPvt4xyfuTGZ8ynt5RvSkSUIT07HQG1B9Aq4qtaF6+uczX68mys2H/ft2S3bkTtmzRM0r98w+sXatnKAMICID69eHZZ3XhbdrUrddp8/Pxo23ltrSt3JYcKye3f3jLsS1MWDeBb5K+ASDQN5CaN9ZkyaNL8PPxY9W+VZzOOE1ESASlQ0rLMEtPExioGxQdO+puiGXL9PjhP//UY9iHDdP3K1QIatS44tPkqwCHBYdRJbwKAJE3RLKsz7KCvwBhH1lZuo81O/u/X1lZ+iszUw+lyczUcyqkp+vvZ8/qr9On9cKFp07pQnrsmP46ehQOHYKDB/X37OyLt12qFFSpog9Q1K59/isoyMy+cLBzxRfgudufY2DDgWw4tIHk/cmsObCGvaf25raC3170NtP+Pr8CQ/Hg4tQuVZt5PecBMHjpYPac3ENIYAhFA4tSNKAopUNK535yXL1vNZk5mQT6BhLgG4C/rz9FAopQqkgpQE/ZCnpInY/ywVf55g6vE07m53e+1Qv6b27tWj3v8Jo1kJJy5YfmZzvlQssRXfbaM1F5tIMHc5fAAfK22N+V7nPh9ecuX/j90ssXfp3j75/37Hnh7w9hYee/ypfXZ6bdcANERECZMvqrUiV9FpEX8/XxpeaNNal5Y00e4qGLbvuizRc8Uf8Jdp3Yxa6Tuzh4+iCF/c93v8zdOpf52+ZzNuv8SIuGpRvmFuBeU3qx9uDai56zZYWWzO05F4A6I+qw7fi2i27veGtHu74+UUB+frr74cK5S65wvENZ+VgtVCl1CNhxnfGuVwngsOEMrkL2xXmyL86TfXGeq+yLcpZl/efEh3wVYFeglFppWdZtpnO4AtkX58m+OE/2xXmuvi9cYjpKIYTwRlKAhRDCEHcswCNNB3Ahsi/Ok31xnuyL81x6X7hdH7AQQngKd2wBCyGER3DrAqyUel4pZSmlSpjOYopS6hOl1CalVIpS6mellNet6aSUaquU+lsp9a9SapDpPKYopcoopRYopTYopdYrpQaazmSSUspXKbVaKfWr6SxX4rYFWClVBrgL2Gk6i2FzgBqWZdUC/gH+z3Aep1JK+QJfAe2A6sADSqnqZlMZkwU8b1lWdaAR8IQX7wuAgcBG0yGuxm0LMPAF8BLg1Z3YlmXNtizr3Jx4S4EIk3kMaAD8a1nWVsuyMoB44D7DmYywLGufZVmrbJdPoYtPabOpzFBKRQD3AKNNZ7katyzASqn7gD2WZa0xncXFPALMMB3CyUoDuy74eTdeWnQupJQqD9QBvHXClsHoBppLL2/isvPnKaXmAqUuc9OrwCvo7gevcLV9YVnWVNt9XkV/BI1zZjbhepRSRYDJwDOWZZ00ncfZlFLtgYOWZSUppZqZznM1LluALctqdbnrlVI1gQrAGttE7xHAKqVUA8uyPHJ52yvti3OUUr2B9kBLy/vGFe4Bylzwc4TtOq+klPJHF984y7J+Mp3HkGigg1LqbiAICFFKjbcsq4fhXP/h9uOAlVLbgdssy3KFCTecTinVFvgcuNOyrEOm8zibUsoPffCxJbrwrgAetCxrvdFgBijdIhkLHLUs6xnTeVyBrQX8gmVZ7U1nuRy37AMWFxkGFAXmKKWSlVLfmA7kTLYDkE8Cs9AHnRK9sfjaRAMPAS1s74VkWytQuCi3bwELIYS7khawEEIYIgVYCCEMkQIshBCGSAEWQghDpAALIYQhUoCFEMIQKcBCCGGIFGAhhDDk/wGJbrX4hX+97gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import scipy.stats as stats\n",
        "import math\n",
        "\n",
        "mu1 = -2\n",
        "mu2 = 2\n",
        "variance = 1\n",
        "sigma = math.sqrt(variance)\n",
        "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
        "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
        "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\", \n",
        "         linestyle='dashed')\n",
        "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
        "plt.axvline(x=-2,color=\"black\")\n",
        "plt.axvline(x=0,color=\"black\")\n",
        "plt.axvline(x=+2,color=\"black\")\n",
        "plt.text(-2.7,0.55,\"Sensitive\")\n",
        "plt.text(-0.7,0.55,\"Balanced\")\n",
        "plt.text(1.7,0.55,\"Specific\")\n",
        "plt.ylim([0,0.53])\n",
        "plt.xlim([-5,5])\n",
        "plt.legend(['Negative','Positive'])\n",
        "plt.yticks([])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ew7B8lNK1AO"
      },
      "source": [
        "We will now train a neural network for the Wisconsin breast cancer dataset. We begin by preprocessing the data. Because we have all numeric data, we compute a z-score for each column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "m55_Ygs9K1AP"
      },
      "outputs": [],
      "source": [
        "from scipy.stats import zscore\n",
        "\n",
        "x_columns = df.columns.drop('diagnosis').drop('id')\n",
        "for col in x_columns:\n",
        "    df[col] = zscore(df[col])\n",
        "\n",
        "# Convert to numpy - Regression\n",
        "x = df[x_columns].values\n",
        "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, \n",
        "                                              # M is 1 and B is 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "as4rUCd1K1AP"
      },
      "source": [
        "We can now define two functions. The first function plots a confusion matrix. The second function plots a ROC chart."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "25ys9moDK1AP"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "\n",
        "# Plot a confusion matrix.\n",
        "# cm is the confusion matrix, names are the names of the classes.\n",
        "def plot_confusion_matrix(cm, names, title='Confusion matrix', \n",
        "                            cmap=plt.cm.Blues):\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(names))\n",
        "    plt.xticks(tick_marks, names, rotation=45)\n",
        "    plt.yticks(tick_marks, names)\n",
        "    plt.tight_layout()\n",
        "    plt.ylabel('True label')\n",
        "    plt.xlabel('Predicted label')\n",
        "    \n",
        "\n",
        "# Plot an ROC. pred - the predictions, y - the expected output.\n",
        "def plot_roc(pred,y):\n",
        "    fpr, tpr, _ = roc_curve(y, pred)\n",
        "    roc_auc = auc(fpr, tpr)\n",
        "\n",
        "    plt.figure()\n",
        "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
        "    plt.plot([0, 1], [0, 1], 'k--')\n",
        "    plt.xlim([0.0, 1.0])\n",
        "    plt.ylim([0.0, 1.05])\n",
        "    plt.xlabel('False Positive Rate')\n",
        "    plt.ylabel('True Positive Rate')\n",
        "    plt.title('Receiver Operating Characteristic (ROC)')\n",
        "    plt.legend(loc=\"lower right\")\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zotNGspK1AQ"
      },
      "source": [
        "### ROC Chart Example\n",
        "\n",
        "The following code demonstrates how to implement a ROC chart in Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tchQMWrIK1AQ",
        "outputId": "e4be0d2b-d974-4917-bc0f-908917773c99"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/1000\n",
            "14/14 - 2s - loss: 0.6850 - accuracy: 0.8427 - val_loss: 0.6673 - val_accuracy: 0.9441 - 2s/epoch - 164ms/step\n",
            "Epoch 2/1000\n",
            "14/14 - 0s - loss: 0.6261 - accuracy: 0.9319 - val_loss: 0.5393 - val_accuracy: 0.9720 - 130ms/epoch - 9ms/step\n",
            "Epoch 3/1000\n",
            "14/14 - 0s - loss: 0.4265 - accuracy: 0.9413 - val_loss: 0.2536 - val_accuracy: 0.9720 - 148ms/epoch - 11ms/step\n",
            "Epoch 4/1000\n",
            "14/14 - 0s - loss: 0.2112 - accuracy: 0.9437 - val_loss: 0.1067 - val_accuracy: 0.9720 - 145ms/epoch - 10ms/step\n",
            "Epoch 5/1000\n",
            "14/14 - 0s - loss: 0.1171 - accuracy: 0.9624 - val_loss: 0.0644 - val_accuracy: 0.9790 - 110ms/epoch - 8ms/step\n",
            "Epoch 6/1000\n",
            "14/14 - 0s - loss: 0.0852 - accuracy: 0.9789 - val_loss: 0.0552 - val_accuracy: 0.9860 - 114ms/epoch - 8ms/step\n",
            "Epoch 7/1000\n",
            "14/14 - 0s - loss: 0.0744 - accuracy: 0.9789 - val_loss: 0.0541 - val_accuracy: 0.9860 - 115ms/epoch - 8ms/step\n",
            "Epoch 8/1000\n",
            "14/14 - 0s - loss: 0.0662 - accuracy: 0.9812 - val_loss: 0.0473 - val_accuracy: 0.9930 - 154ms/epoch - 11ms/step\n",
            "Epoch 9/1000\n",
            "14/14 - 0s - loss: 0.0602 - accuracy: 0.9812 - val_loss: 0.0493 - val_accuracy: 0.9860 - 90ms/epoch - 6ms/step\n",
            "Epoch 10/1000\n",
            "14/14 - 0s - loss: 0.0548 - accuracy: 0.9859 - val_loss: 0.0468 - val_accuracy: 0.9860 - 225ms/epoch - 16ms/step\n",
            "Epoch 11/1000\n",
            "14/14 - 0s - loss: 0.0491 - accuracy: 0.9836 - val_loss: 0.0484 - val_accuracy: 0.9860 - 133ms/epoch - 10ms/step\n",
            "Epoch 12/1000\n",
            "14/14 - 0s - loss: 0.0458 - accuracy: 0.9836 - val_loss: 0.0486 - val_accuracy: 0.9860 - 119ms/epoch - 8ms/step\n",
            "Epoch 13/1000\n",
            "Restoring model weights from the end of the best epoch: 8.\n",
            "14/14 - 0s - loss: 0.0417 - accuracy: 0.9883 - val_loss: 0.0477 - val_accuracy: 0.9860 - 124ms/epoch - 9ms/step\n",
            "Epoch 13: early stopping\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f6a8aee46d0>"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "# Classification neural network\n",
        "import numpy as np\n",
        "import tensorflow.keras\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Split into train/test\n",
        "x_train, x_test, y_train, y_test = train_test_split(    \n",
        "    x, y, test_size=0.25, random_state=42)\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n",
        "                kernel_initializer='random_normal'))\n",
        "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
        "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
        "model.add(Dense(1,activation='sigmoid',kernel_initializer='random_normal'))\n",
        "model.compile(loss='binary_crossentropy', \n",
        "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
        "              metrics =['accuracy'])\n",
        "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
        "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
        "\n",
        "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
        "          callbacks=[monitor],verbose=2,epochs=1000)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "_95P1kTBK1AQ",
        "outputId": "35b353e5-0eaf-412f-83a3-1970e8e07bb1"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5wN9f/A8dcb65ZrRAqR/Kz7bVMuRRe5pJSUFEpKufflK0oXhZQoKUIqSkVSIhUpt0JyWddFvirWJZdYlrB2378/ZnYd2+7ZYzln9vJ+Ph7nsWdmPjPznjln533m85n5jKgqxhhjTGpyeB2AMcaYjM0ShTHGGL8sURhjjPHLEoUxxhi/LFEYY4zxyxKFMcYYvyxRZHEisklEmngdR0YhIs+IyCSP1j1ZRIZ6se6LTUQeFJH56Zw33d9JEflZRGqnZ970EpFeIvJqKNeZ0ViiCCER+UNE/hGRWBHZ5x44CgRznapaVVUXBXMdiUQkj4gMF5Gd7nb+JiL9RURCsf4U4mkiItG+41T1ZVV9NEjrExHpLSIbReS4iESLyAwRqR6M9aWXiAwWkakXsgxV/VhVbwtgXf9Kjun9TorIHcAxVV3rDg8WkTj3/+mIiCwTkfrJ5ikiIu+4/28nRGSDiHROYdkPiMgqd1l7ReRbEWnkTn4XeFBESpxvzFmFJYrQu0NVCwC1gNrA0x7Hc95EJFcqk2YAtwAtgYJAR6Ar8GYQYhARyWjf3zeBPkBv4FLg/4BZwO0Xe0V+PoOg83DdTwAfJRs33f1/Kg4sxPkOAiAiuYEFwFVAfaAw0B94RUT6+pTrC4wGXgZKAmWBcUBrAFU9CXwLdArKVmUGqmqvEL2AP4BbfYZHAHN9hq8HlgFHgHVAE59plwIfAHuAw8Asn2mtgEh3vmVAjeTrBK4A/gEu9ZlWGzgIhLnDjwBR7vLnAVf5lFWgB/Ab8HsK23YLcBIok2z8dUA8cI07vAgYDqwEjgJfJYvJ3z5YBAwDfna35RqgsxvzMWAH8Lhb9hK3TAIQ676uAAYDU90y5dztegjY6e6LQT7rywdMcfdHFPAUEJ3KZ1vR3c56fj7/ycBYYK4b7y9ABZ/pbwK73P2yGrjBZ9pg4HNgqjv9UaAesNzdV3uBt4HcPvNUBb4H/gb+Ap4BmgOngTh3n6xzyxYG3nOXsxsYCuR0pz3s7vM3gEPutIeBn9zp4k7b78a2AaiG8yMhzl1fLDAn+f8BkNON63/uPllNsu+QWy63+3mWTrZPpvoMV3E/z8vc4S5uTJckW1Y7N55C7nbHAvem8b/7ILDQ62OIVy/PA8hOr2T/IKXdf6g33eEr3X/Cljhnek3d4cQv/VxgOlAUCAMau+Nru/8M17n/dA+568mTwjp/BB7ziec1YLz7vjWwHagM5AKeBZb5lFX3oHMpkC+FbXsFWJzKdv/J2QP4IvdAVA3nYD6TswfutPbBIpwDelU3xjCcX+sVcA5WjYETQB23fBOSHdhJOVG8i5MUagKngMq+2+Tu89LA+uTL81nuE8CfaXz+k93tqefG/zEwzWd6B6CYO60fsA/I6xN3HHCXu2/yAXVxEmsud1uigCfd8gVxDvr9gLzu8HXJ94HPur8EJrifSQmcRJ74mT0MnAF6uevKx7mJohnOAb6I+zlUBkr5bPNQP/8H/XH+Dyq589YEiqWw76oCx/18lrndz+sgkMsdNw2YksKycrnb0wwncZ5JnMfPZ1cH+NvrY4hXr4x26p4dzBKRYzi/HPcDL7jjOwDfqOo3qpqgqt8Dq4CWIlIKaAE8oaqHVTVOVRe783UFJqjqL6oar6pTcA5216ew7k+A9uBU3QD3u+PAOdANV9UoVT2DcxpeS0Su8pl/uKr+rar/pLDs4jgHppTsdacn+khVN6rqceA54D4RyelvH/jMO1lVN6nqGXc/zFXV/6ljMTAfuCGVOFLzoqr+o6rrcM5iarrj7wNedvd5NDDGzzKK+dl+X1+q6kp3H3+MUwUJgKpOVdVD7raNAvLgHEATLVfVWe6++UdVV6vqCrf8HzgH+sZu2VbAPlUdpaonVfWYqv6SUkAiUhJnHz+pqsdVdT/OGcL9PsX2qOpb7rqSf/5xOIkoHBD3OxTIvgDnzOhZVd3qfobrVPVQCuWK4JxxJHefiBzBOdt4DGjr7ltI5TvpTj/oTi8GHPSZJzXHcM4+siVLFKF3l6oWxPm1G87ZA+hVwL1uo9wR98vfCCgFlMH5NXM4heVdBfRLNl8ZnGqW5GYC9d3EcyNOtcxSn+W86bOMv3F+4V3pM/8uP9t10I01JaXc6Skt50+cM4Pi+N8HKcYgIi1EZIWI/O2Wb8m5SSkQ+3zenwASLzC4Itn6/G3/IVLf/kDWhYj8V0SiRCTG3ZbCnLstybf9/0Tka7eh9ihOck8sXwanOicQV+F8Bnt99vsEnDOLFNftS1V/xKn2GgvsF5GJIlIowHUHGudhnGSU3GeqWgSnbWEjzllWohS/k24bS3F3+iGgeADtLgWBmADizJIsUXjE/fU7GRjpjtqF80u7iM/rElV9xZ12qYgUSWFRu4BhyebLr6qfprDOwzi/uNsBD+BUe6jPch5Ptpx8qrrMdxF+NmkBcJ2IlPEdKSLX4RwMfvQZ7VumLM4v0oNp7IN/xSAieXCS30igpHvA+AYnwaUVbyD24lQ5pRR3cj8ApUUkIj0rEpEbcNpA7gOKutsSw9ltgX9vzzvAFqCiqhbCqetPLL8LuDqV1SVfzi6cs9DiPvu9kKpW9TPPuQtUHaOqdXHaCf4Pp0opzfncdVdIoww41aIiIlemNFFVD+KcXQ92fwiB851sISKXJCt+D872rsBp4zmFU6XnT2Wcs81syRKFt0YDTUWkJk4j5R0i0kxEcopIXvfyztLuafy3wDgRKSoiYSJyo7uMd4EnROQ690qgS0TkdhFJ6dcXOFVNnYC2nK12AhgPPC0iVQFEpLCI3BvohqjqApyD5UwRqepuw/Xudr2jqr/5FO8gIlVEJD/wEvC5qsb72weprDY3TvXMAeCMiLQAfC/Z/AsoJiLprTL4DGefFHUPUD1TK+hu3zjgUzfm3G7894vIwADWVRCnrvwAkEtEnsdpbE1rnqNArIiEA918pn0NlBKRJ8W5bLmgm7TB2S/lEq8ac79f84FRIlJIRHKISAURaUwARORa9/sXBhzHuaghwWddqSUsgEnAEBGp6H5/a4hIseSFVPU0zoE/1ZhUdSvORRhPuaM+AqKBGSJSzv2/aYZThThYVWNUNQZ4HhgrIneJSH63XAsRGeGz+MY4/4PZkiUKD6nqAeBD4HlV3YXToPwMzsFiF86vssTPqCPOL+8tOG0bT7rLWIVTN/s2zun5dpyGxtTMxrlCZ59bJ58Yy5fAq8A0txpjI067yPm4B+cSxe9wriSZinMlTa9k5T7COZvah9PQ2tuNIa19cA5VPebO+xnOtj/gbl/i9C3Ap8AOt0olpeo4f17COdD8jnOQ+hzn12dqenO2CuYITpXK3cCcANY1D2e/bcOpjjuJ/6ougP/ibPMxnB8M0xMnuPumKXAHzn7+DbjJnZx4CekhEVnjvu+Ek3g34+zLzwmsKg2chPauO9+fONU5r7nT3gOquPt/Vgrzvo7z+c3HSXrv4TSWp2QCzv+BP68BXUWkhKqewrnibxfOFWZH3fUNUtXE+HDbg/riXMCR+L3riXNpMyKSF6dKc0oa686y5GzNgzHBJyKLcK5U8eTu6AshIt2A+1U1oF/a5uITkZ+BnuredBeidfbCuWT3qTQLZ1Ge3bRjTEbn1nVfjVOPXRHnUtO3PQ0qm1PVhh6s861QrzOjsURhTOpy41R3lMepSpqG0w5hTLZiVU/GGGP8ssZsY4wxfmW6qqfixYtruXLlvA7DGGMyldWrVx9U1cvSM2+mSxTlypVj1apVXodhjDGZioj8md55rerJGGOMX5YojDHG+GWJwhhjjF+WKIwxxvhlicIYY4xfliiMMcb4FbREISLvi8h+EdmYynQRkTEisl1E1otInWDFYowxJv2CeUYxGed5tKlpgdPRWkWcB468E8RYjDHGpFPQbrhT1SUiUs5PkdbAh+4T1laISBERKXUez9oNmk9+2clXkbu9DsMYYy6IqrI7cjG7Ixdf0HK8vDP7Ss59MEu0O+5fiUJEuuKcdVC2bNmgB/ZV5G427z1KlVKBPvbXGGMyluMH97Jm+ij2blhG4SuvuaBlZYouPFR1IjARICIiIiTd3VYpVYjpj9cPxaqMMeaiUlUiIiI4umMro0aNonfv3oSFhaV7eV4mit2c+7D60u44Y4wx6bBs2TKqV69OwYIFmTRpEsWLF6dMmTJpz5gGLxPFbKCniEwDrgNiAmmf2HHgOO0mLA9qYFbtZIzJTA4dOsTAgQOZNGkSL7zwAoMHD6Z27doXbflBSxQi8inQBCguItHAC0AYgKqOB77BeWD5duAE0DmQ5f4TFx+McM9RpVQhWte6MujrMcaYC6GqfPjhh/z3v//l8OHD9O/fn/79+1/09WS6J9xdelVl/fvPKK/DMMYYzz311FO89tprNGjQgPHjx1O9evVUy4rIalWNSM96MkVjtjHGGMc///zD8ePHKV68OF26dKFixYp06dKFHDmCd1ucdeFhjDGZxHfffUe1atV4/PHHAahUqRKPPfZYUJMEWKIwxpgMb8+ePdx33320aNGCsLAwevbsGdL1W9WTMcZkYD/88AN33303p0+fZsiQIfTv3588efKENAZLFMYYkwHFxcURFhZGzZo1admyJUOHDuWaay7sDuv0sqonY4zJQI4ePUqfPn244YYbiI+Pp3jx4kybNs2zJAGWKIwxJkNQVWbMmEF4eDhvvfUWERERnDp1yuuwAKt6MsYYzx04cICHHnqIb7/9ltq1a/PVV19x7bXXeh1WEjujMMYYjxUqVIiDBw8yevRoVq5cmaGSBFiiMMYYTyxZsoRmzZoRGxtLnjx5WLFiBX369CFXroxX0WOJwhhjQujgwYN07tyZxo0bs23bNv744w+AoN80dyEybmTGGJOFqCrvv/8+lSpVYurUqTz99NNs2rSJatWqeR1amjLeOY4xxmRRU6dOpUqVKowfP56qVat6HU7A7IzCGGOC5MSJEzz77LNER0cjIsycOZPFixdnqiQBliiMMSYovvnmG6pWrcqwYcOYM2cOAEWLFs3QbRGpyXwRG2NMBhYdHU3btm25/fbbyZcvH4sXL6Zbt25eh3VBLFEYY8xFNGzYMObOncvLL79MZGQkN954o9chXTB7wp0xxlyglStXki9fPqpXr86hQ4eIiYnh6quv9jqsc1zIE+7sjMIYY9IpJiaGHj16cP311zNo0CAAihUrluGSxIWyRGGMMedJVZk2bRrh4eGMHz+eXr16MXXqVK/DChq7j8IYY87T1KlT6dSpExEREXz99dfUrVvX65CCyhKFMcYE4NSpU+zYsYPKlStz3333cebMGTp16kTOnDm9Di3orOrJGGPSsHDhQmrWrEmzZs04deoUefLkoXPnztkiSYAlCmOMSdX+/fvp1KkTN998M3FxcUycODHkz6vOCKzqyRhjUrB9+3bq1atHbGwsgwYNYtCgQeTLl8/rsDxhicIYY3wcPXqUQoUKUaFCBbp06cIjjzxC5cqVvQ7LU1b1ZIwxwPHjxxkwYADlypVL6sTvtddey/ZJAuyMwhhjmDNnDj179mTnzp106dKF/Pnzex1ShmKJwhiTbZ05c4b77ruPL7/8kqpVq7J06VIaNWrkdVgZjlU9GWOyncQ+7nLlykWpUqV45ZVXWLNmjSWJVFiiMMZkKytWrCAiIoI1a9YAMHbsWAYMGEDu3Lk9jizjskRhjMkWDh8+TLdu3WjQoAF//fUXhw8f9jqkTCOoiUJEmovIVhHZLiIDU5heVkQWishaEVkvIi2DGY8xJnuaPn064eHhTJw4kSeffJKoqChuueUWr8PKNILWmC0iOYGxQFMgGvhVRGar6mafYs8Cn6nqOyJSBfgGKBesmIwx2dOWLVsoV64c3333HbVr1/Y6nEwnmGcU9YDtqrpDVU8D04DWycooUMh9XxjYE8R4jDHZxMmTJ3nxxReTnlX9zDPPsGzZMksS6RTMRHElsMtnONod52sw0EFEonHOJnqltCAR6Soiq0RkVVxcXDBiNcZkEQsWLKBGjRoMHjyYxYsXAxAWFpZtOvALBq8bs9sDk1W1NNAS+EhE/hWTqk5U1QhVjQgLCwt5kMaYjO+vv/7iwQcfpGnTpqgq8+fPZ+TIkV6HlSUEM1HsBsr4DJd2x/nqAnwGoKrLgbxA8SDGZIzJor7//ns+//xznn/+eTZs2EDTpk29DinLCOad2b8CFUWkPE6CuB94IFmZncAtwGQRqYyTKA4EMSZjTBaybt06fvvtN9q2bcuDDz5Iw4YNKV++vNdhZTlBO6NQ1TNAT2AeEIVzddMmEXlJRO50i/UDHhORdcCnwMOaeMukMcakIjY2ln79+lG3bl0GDhzImTNnEBFLEkEime24fOlVlfXvP6O8DsMY45FZs2bRq1cvoqOj6dq1K8OHD+fSSy/1OqwMT0RWq2pEeua1TgGNMZnGhg0buPvuu6levTrTp0+nQYMGXoeULXh91ZMxxvgVFxfHjz/+CED16tWZO3cuq1evtiQRQpYojDEZ1rJly6hbty5NmzZl+/btALRs2RK7TD60LFEYYzKcv//+m65du9KwYUOOHDnCF198wTXXXON1WNmWtVEYYzKUkydPUqtWLfbs2UO/fv0YPHgwBQoU8DqsbM0ShTEmQ4iOjqZ06dLkzZuXIUOGUKtWLWrWrOl1WAarejLGeOyff/7h+eefp0KFCkmd+D300EOWJDIQO6Mwxnhm/vz5dO/enf/973906NCBevXqeR2SSUHAZxQikj+YgRhjspdevXrRrFkzcuTIwYIFC/joo48oWbKk12GZFKR5RiEiDYBJQAGgrIjUBB5X1e7BDs4Yk7XEx8cDkDNnTq6//nqKFy/OgAEDyJs3r8eRGX8COaN4A2gGHAJQ1XXAjcEMyhiT9axZs4b69eszbtw4AB588EFeeOEFSxKZQEBVT6q6K9mo+CDEYozJgo4dO8Z//vMfrr32Wnbu3EmpUqW8Dsmcp0Aas3e51U8qImFAH5zeYI0xxq/58+fzyCOPsGfPHp544glefvllihQp4nVY5jwFkiieAN7EeYzpbmA+YO0Txpg05c6dmxIlSjBz5kyuu+46r8Mx6ZRmN+Mi0lBVf05rXKhYN+PGZFxxcXG8/vrrHD16lGHDhgGQkJBAjhx2y5bXLqSb8UA+vbcCHGeMycZ++uknateuzcCBA/ntt99ISEgAsCSRBaRa9SQi9YEGwGUi0tdnUiEgZ7ADM8ZkDocOHWLAgAG89957lC1bljlz5tCqVSuvwzIXkb9Unxvn3olcQEGf11GgbfBDM8ZkBocOHWLatGk89dRTbN682ZJEFhRIG8VVqvpniOJJk7VRGOO9qKgoPvvsM1544QXA6RbcHkeasQW7jeKEiLwmIt+IyI+Jr/SszBiTuZ04cYJBgwZRs2ZN3nzzTaKjowEsSWRxgSSKj4EtQHngReAP4NcgxmSMyYC+++47qlWrxssvv8wDDzzA1q1bKV26tNdhmRAI5D6KYqr6noj0UdXFwGIRsURhTDYSGxtLx44dKVasGAsXLqRJkyZeh2RCKJAzijj3714RuV1EagN2nmlMFhcfH8/UqVOJj4+nQIECLFiwgHXr1lmSyIYCOaMYKiKFgX44908UAp4MalTGGE+tXr2axx9/nNWrV5MvXz7uuecee5BQNpbmGYWqfq2qMaq6UVVvUtW6wN8hiM0YE2IxMTH07t2bevXqsXv3bqZNm0abNm28Dst4zN8NdzmB+3D6ePpOVTeKSCvgGSAfUDs0IRpjQuWee+7hxx9/pEePHgwdOpTChQt7HZLJAPxVPb0HlAFWAmNEZA8QAQxU1VmhCM4YE3w7duzgsssuo2DBggwbNowcOXJw7bXXeh2WyUD8VT1FAE1V9WmgJdAKaGhJwpis4fTp07z88stUrVqVoUOHAnDddddZkjD/4u+M4rSqJgCo6kkR2aGqh0IUlzEmiJYsWcITTzxBVFQUbdu2pXfv3l6HZDIwf4kiXETWu+8FqOAOC6CqWiPo0RljLro33niDvn37Uq5cOebOnUvLli29DslkcP4SReWQRWGMCaqEhASOHz9OwYIFuf322zlw4ADPPvss+fPn9zo0kwmk2SlgRmOdAhpzfjZt2sQTTzyR9KQ5kz0Fu1PAdBOR5iKyVUS2i8jAVMrcJyKbRWSTiHwSzHiMyU5OnDjB008/Ta1atYiKiqJVq1Zkth+GJmMI5M7sdHHvwxgLNAWigV9FZLaqbvYpUxF4GudqqsMiUiJY8RiTnaxdu5Y2bdrwxx9/0LlzZ0aMGEHx4sW9DstkUgElChHJB5RV1a3nsex6wHZV3eEuYxrQGtjsU+YxYKyqHgZQ1f3nsXxjTDKqiohQtmxZypYty5QpU7jxxhu9DstkcmlWPYnIHUAk8J07XEtEZgew7CuBXT7D0e44X/8H/J+I/CwiK0SkeWBhG2N8nTlzhtGjR3PLLbcQHx9PsWLFWLx4sSUJc1EE0kYxGOfs4AiAqkbiPJviYsgFVASaAO2Bd0WkSPJCItJVRFaJyKq4uLjkk43J1lauXEm9evX4z3/+Q968eTl69KjXIZksJqBuxlU1Jtm4QFrEduN0AZKotDvOVzQwW1XjVPV3YBtO4jh3ZaoTVTVCVSPCwsICWLUxWV9sbCw9evTg+uuv56+//mLGjBnMnTuXokWLeh2ayWICSRSbROQBIKeIVBSRt4BlAcz3K1BRRMqLSG7gfiB5ldUsnLMJRKQ4TlXUjkCDNyY7CwsLY9GiRfTq1SvpDmsR8ToskwUFkih6AVWBU8AnQAwBPI9CVc8APYF5QBTwmapuEpGXROROt9g84JCIbAYWAv2tmxBjUrd9+3Y6derEsWPHyJMnD6tXr+bNN9+kUKFCXodmsrA0b7gTkTqquiZE8aTJbrgz2dGpU6cYMWIEw4YNI3fu3MydO5cbbrjB67BMJhLsG+5GiUiUiAwRkWrpWYkxJv0WLlxIzZo1ef7557nrrrvYsmWLJQkTUmneR6GqN4nI5TgPMZogIoWA6ao6NOjRGZPNqSrDhg0jLi6O7777jmbNmnkdksmGzquvJxGpDjwFtFPV3EGLyg+rejJZXUJCAu+99x7NmzenTJky7N27lyJFipAvXz6vQzOZWFCrnkSksogMFpENQOIVT6XTszJjjH/r16+nUaNGdO3alUmTJgFQqlQpSxLGU4F04fE+MB1opqp7ghyPMdlSbGwsL774Im+88QZFixZl8uTJdOrUyeuwjAECa6OoH4pAjMnOBg8ezKhRo3j00Ud55ZVXKFasmNchGZMk1TYKEflMVe9zq5x8C3n6hDtrozBZxa5duzh+/Djh4eEcPHiQLVu20KhRI6/DMlnUhbRR+EsUpVR1r4hcldJ0Vf0zPSu8UJYoTGZ35swZxowZw/PPP0/dunVZvHix1yGZbCAojdmqutd9211V//R9Ad3TszJjsrsVK1YQERFBv379aNKkCVOmTPE6JGPSFMgNd01TGNfiYgdiTFY3d+5cGjRowMGDB/niiy+YM2cO5cqV8zosY9KUaqIQkW5u+0QlEVnv8/odWB+6EI3JvFSV3budTpNvvfVWXnrpJaKiorj77rutAz+TafhroygMFAWGA77Puz6mqn+HILYUWRuFySy2bdtG9+7d2bZtG5s3b6ZAgQJeh2SysWDdcKeq+gfQAzjm80JELk3PyozJDk6ePMngwYOpXr06q1at4umnn7Yb5kym5u8+ik+AVsBqnMtjfc+TFbg6iHEZkynt27ePG2+8kd9++4327dvz+uuvc/nll3sdljEXJNVEoaqt3L8X67GnxmRZcXFxhIWFUbJkSW688UbGjh1L06YpXQdiTOYTSF9PDUXkEvd9BxF5XUTKBj80YzK+hIQExo8fT4UKFYiOjkZEmDRpkiUJk6UEcnnsO8AJEakJ9AP+B3wU1KiMyQTWrVtHgwYN6NatGxUrViQuLs7rkIwJikASxRl1Lo1qDbytqmOBgsENy5iMS1X573//S926ddmxYwcfffQRCxYsoHx5q6U1WVMgieKYiDwNdATmikgOICy4YRmTcYkIhw8fpkuXLmzdupUOHTrYPREmSwskUbQDTgGPqOo+nGdRvBbUqIzJYP7880/uuusu1qxxHh//7rvvMmHCBIoWLepxZMYEX5qJwk0OHwOFRaQVcFJVPwx6ZMZkAHFxcYwYMYIqVarw/fffs3XrVgBy5AjkN5YxWUMgVz3dB6wE7sV5bvYvItI22IEZ47Vly5ZRp04dBgwYQNOmTYmKiqJ9+/Zeh2VMyAXyhLtBwLWquh9ARC4DFgCfBzMwY7y2YMECYmJimDVrFq1bt/Y6HGM8k2pfT0kFRDaoanWf4RzAOt9xoWR9PZlgUVU++ugjLrvsMlq0aMGpU6eIi4uzPppMlhCsvp4SfSci80TkYRF5GJgLfJOelRmTUW3ZsoWbb76Zhx56iA8++ACAPHnyWJIwhsAas/sDE4Aa7muiqg4IdmDGhMI///zDc889R40aNYiMjGTChAlMmzbN67CMyVBSbaMQkYrASKACsAH4r6ruDlVgxoTCnDlzGDp0KB06dGDkyJGULFnS65CMyXD8NWa/D3wILAHuAN4C2oQiKGOCad++fURGRtK8eXPuvfdeypUrR7169bwOy5gMy1+iKKiq77rvt4rImlAEZEywxMfHM2HCBJ5++mly587Nzp07yZcvnyUJY9Lgr40ir4jUFpE6IlIHyJds2JhMY82aNdSvX58ePXpQr149li1bZg8TMiZA/s4o9gKv+wzv8xlW4OZgBWXMxfT7779Tr149ihcvzieffML9999vfTMZcx78PbjoplAGYszFpKps2LCBGjVqUL58eT744APuuOMOihQp4nVoxmQ61mGNyXJ+//13WrVqRe3atVm/fj0AHTt2tCRhTDoFNVGISHMR2Soi2z7qvosAAByrSURBVEVkoJ9y94iIiki67ho0BuD06dO88sorVK1alcWLFzNy5EiqVKnidVjGZHqB9PWULiKSExgLNAWigV9FZLaqbk5WriDQB/glWLGYrC8+Pp4GDRqwevVq2rRpw+jRoylTpozXYRmTJQTSe6y4z8p+3h0uKyKBXE9YD9iuqjtU9TQwDecpeckNAV4FTp5H3MYAcPToUQBy5szJI488wpw5c5g5c6YlCWMuokCqnsYB9YHE/pWP4ZwppOVKYJfPcLQ7Lol7mW0ZVZ3rb0Ei0lVEVonIKnsusQGnsXry5MlcffXVfPXVVwB0796dVq1aeRyZMVlPIIniOlXtgfuLX1UPA7kvdMVuL7SvA/3SKquqE1U1QlUjwsLsKazZ3ebNm2nSpAmdO3cmPDycChUqeB2SMVlaIIkizm1vUEh6HkVCAPPtBnzP/0u74xIVBKoBi0TkD+B6YLY1aBt/RowYQc2aNdm4cSOTJk1iyZIlVKtWzeuwjMnSAkkUY4AvgRIiMgz4CXg5gPl+BSqKSHkRyQ3cD8xOnKiqMapaXFXLqWo5YAVwp6quOt+NMFlf4nNTLr/8ch588EG2bNlCly5d7JGkxoRAmg8uAhCRcOAWQIAfVDWgJweJSEtgNJATeF9Vh4nIS8AqVZ2drOwinB5q/SYKe3BR9rJnzx769OnDDTfcQO/evb0Ox5hM60IeXJTm5bEiUhY4AczxHaeqO9OaV1W/IdlDjlT1+VTKNklreSb7iI+PZ9y4cQwaNIi4uDgaNGjgdUjGZFuB3EcxF6d9QoC8QHlgK1A1iHGZbCwyMpJHH32U1atXc9tttzFu3DhrsDbGQ2kmiuTPxnYvae0etIhMthcTE8OePXuYPn069957r3XgZ4zHzvvObFVdIyLXBSMYkz2pKjNmzOC3335j0KBBNG7cmB07dpA3b16vQzPGEFgbRV+fwRxAHWBP0CIy2cr//vc/evbsyXfffce1117LU089RVhYmCUJYzKQQK4tLOjzyoPTZpFSVxzGBOzUqVMMGzaMatWq8fPPP/Pmm2+ybNky7IZKYzIev2cU7o12BVX1vyGKx2QTu3btYsiQIdxxxx2MHj2aK6+8Mu2ZjDGeSPWMQkRyqWo80DCE8Zgs7MCBA7z99tsAXHPNNWzevJkZM2ZYkjAmg/NX9bTS/RspIrNFpKOItEl8hSI4kzUkJCTw3nvvER4eTt++fdm6dSsAV199tceRGWMCEUgbRV7gEM4zslsBd7h/jUnTxo0bady4MY8++ihVq1YlMjKSSpUqeR2WMeY8+GujKOFe8bSRszfcJUq73w+T7Z0+fZrbbruN06dP8/777/Pwww/bPRHGZEL+EkVOoADnJohElihMqn788UcaN25M7ty5+eyzzwgPD6d48eJeh2WMSSd/iWKvqr4UskhMphcdHU2fPn344osveP/99+ncuTONGjXyOixjzAXy10ZhdQQmIGfOnGH06NFUrlyZb7/9luHDh/Pggw96HZYx5iLxd0ZxS8iiMJlax44dmTZtGi1atGDs2LGUL1/e65CMMRdRQM+jyEjseRQZw5EjR8iVKxcFChTgp59+Yt++fdxzzz3WWG1MBnUhz6Owx4OZ86KqTJs2jcqVK/Pcc88B0KhRI9q2bWtJwpgsyhKFCdj27dtp1qwZ7du3p3Tp0nTo0MHrkIwxIWCJwgTkk08+oVq1avzyyy+8/fbbrFixgrp163odljEmBM77eRQme4mLiyMsLIyIiAjatm3LiBEjuOKKK7wOyxgTQtaYbVK0f/9++vXrx/Hjx/niiy+8DscYc4GsMdtcNAkJCUycOJFKlSoxffp0qlatSnx8vNdhGWM8ZFVPJsmOHTvo0KEDy5cvp0mTJrzzzjuEh4d7HZYxxmOWKEySwoULc+TIEaZMmULHjh3tcldjDGBVT9ne7NmzadOmDfHx8RQrVoyNGzfSqVMnSxLGmCSWKLKpnTt3ctddd9G6dWu2bdvG3r17AciRw74Sxphz2VEhmzlz5gwjR46kcuXKzJ8/n1dffZW1a9dSunRpr0MzxmRQ1kaRzcTHxzNp0iRuvvlm3nrrLcqVK+d1SMaYDM7OKLKBw4cPM2DAAI4dO0aePHn4+eefmT17tiUJY0xALFFkYarKxx9/THh4OKNGjWLhwoUAFCtWzBqrjTEBs0SRRW3bto2mTZvSoUMHypUrx6pVq7jzzju9DssYkwlZG0UW9eSTT7Jq1SrGjRtH165dyZkzp9chGWMyKUsUWcj3339PeHg4ZcqU4Z133iFPnjxcfvnlXodljMnkglr1JCLNRWSriGwXkYEpTO8rIptFZL2I/CAiVwUznqxq3759PPDAA9x22228+uqrAFx11VWWJIwxF0XQEoWI5ATGAi2AKkB7EamSrNhaIEJVawCfAyOCFU9WlJCQwPjx4wkPD2fmzJm88MILjBw50uuwjDFZTDDPKOoB21V1h6qeBqYBrX0LqOpCVT3hDq4A7K6v8zB8+HC6detG3bp1Wb9+PYMHDyZv3rxeh2WMyWKC2UZxJbDLZzgauM5P+S7AtylNEJGuQFeAAqUqXKz4MqVjx45x8OBBypcvzxNPPEH58uVp3769Xe5qjAmaDHF5rIh0ACKA11KarqoTVTVCVSPCwsJCG1wGoap8+eWXVKlShXbt2qGqFCtWjAceeMCShDEmqIKZKHYDZXyGS7vjziEitwKDgDtV9VQQ48m0/vzzT+68807atGnDpZdeypgxYyw5GGNCJphVT78CFUWkPE6CuB94wLeAiNQGJgDNVXV/EGPJtJYvX86tt94KwMiRI+nTpw+5ctlVzcaY0AnaGYWqngF6AvOAKOAzVd0kIi+JSOItwq8BBYAZIhIpIrODFU9mc/ToUQDq1KnDI488QlRUFP369bMkYYwJOVFVr2M4L5deVVn//jPK6zCC5tChQwwcOJD58+ezadMmChQo4HVIxpgsQERWq2pEeubNEI3Zxmms/vDDDwkPD+eDDz6gXbt21g5hjMkQrB4jA4iJieGuu+5i0aJF1K9fn/Hjx1OjRg2vwzLGGMAShadUFRGhUKFCFC9enIkTJ9KlSxd7HKkxJkOxI5JH5s2bR506dYiOjkZEmDFjBo899pglCWNMhmNHpRDbu3cv999/P82bN+fEiRPs329XBRtjMjZLFCE0duxYwsPDmTVrFi+++CLr16+nTp06XodljDF+WRtFCK1evZrrrruOsWPHUrFiRa/DMcaYgNgZRRAdPXqUJ598ktWrVwMwbtw45s2bZ0nCGJOpWKIIAlXl888/p3LlyowZM4bFixcDkDdvXrs3whiT6ViiuMh+//13WrVqxb333kuJEiVYvnw5ffv29TosY4xJN0sUF9nHH3/MkiVLeOONN/j111+57jp/j+AwxpiMz/p6ugiWLl3KqVOnuPXWWzl16hQHDhygdGl7WJ8xJuOwvp48cvDgQR555BFuvPFGXnrpJQDy5MljScIYk6XY5bHpoKpMnjyZ/v37ExMTw4ABA3juuee8DstkMHFxcURHR3Py5EmvQzHZSN68eSldujQX82mglijS4ZtvvuGRRx6hYcOGjB8/nmrVqnkdksmAoqOjKViwIOXKlbOr3UxIqCqHDh0iOjqa8uXLX7TlWtVTgE6cOMHPP/8MQMuWLfnqq69YsmSJJQmTqpMnT1KsWDFLEiZkRIRixYpd9LNYSxQB+Pbbb6lWrRotWrTgyJEjiAh33nmndeBn0mRJwoRaML5zdqTzY/fu3dx77720bNmSPHnyMGfOHIoUKeJ1WMYYE1KWKFKxf/9+qlSpwtdff83QoUNZt24djRs39josY85Lzpw5qVWrFtWqVeOOO+7gyJEjSdM2bdrEzTffTKVKlahYsSJDhgzB93L5b7/9loiICKpUqULt2rXp16+fF5vg19q1a+nSpYvXYaRqyZIl1KlTh1y5cvH555+nWm716tVUr16da665ht69eyd9Dn///TdNmzalYsWKNG3alMOHDwPw9ddf8/zzz4dkGwCn8SMzvYqWDddgio6OTnr/5ptv6vbt24O6PpN1bd682esQ9JJLLkl636lTJx06dKiqqp44cUKvvvpqnTdvnqqqHj9+XJs3b65vv/22qqpu2LBBr776ao2KilJV1TNnzui4ceMuamxxcXEXvIy2bdtqZGRkSNd5Pn7//Xddt26dduzYUWfMmJFquWuvvVaXL1+uCQkJ2rx5c/3mm29UVbV///46fPhwVVUdPny4PvXUU6qqmpCQoLVq1dLjx4+nuLyUvnvAKk3ncdeuenLFxMTw7LPPMmHCBFasWEGdOnXo3bu312GZLOLFOZvYvOfoRV1mlSsK8cIdVQMuX79+fdavXw/AJ598QsOGDbntttsAyJ8/P2+//TZNmjShR48ejBgxgkGDBhEeHg44ZybdunX71zJjY2Pp1asXq1atQkR44YUXuOeeeyhQoACxsbEAfP7553z99ddMnjyZhx9+mLx587J27VoaNmzIF198QWRkZFKVbsWKFfnpp5/IkSMHTzzxBDt37gRg9OjRNGzY8Jx1Hzt2jPXr11OzZk0AVq5cSZ8+fTh58iT58uXjgw8+oFKlSkyePJkvvviC2NhY4uPj+eabb+jVqxcbN24kLi6OwYMH07p1a/744w86duzI8ePHAXj77bdp0KBBwPs3JeXKlQPw2565d+9ejh49yvXXXw9Ap06dmDVrFi1atOCrr75i0aJFADz00EM0adKEV199FRGhSZMmfP3119x3330XFGMgsn2iUFVmzJjBk08+yb59++jZsycVKlTwOixjLqr4+Hh++OGHpGqaTZs2Ubdu3XPKVKhQgdjYWI4ePcrGjRsDqmoaMmQIhQsXZsOGDQBJVSP+REdHs2zZMnLmzEl8fDxffvklnTt35pdffuGqq66iZMmSPPDAA/znP/+hUaNG7Ny5k2bNmhEVdW6PDKtWrTrnqsPw8HCWLl1Krly5WLBgAc888wwzZ84EYM2aNaxfv55LL72UZ555hptvvpn333+fI0eOUK9ePW699VZKlCjB999/T968efntt99o3749q1at+lf8N9xwA8eOHfvX+JEjR3Lrrbemuf3J7d69+5ybdEuXLs3u3bsB+OuvvyhVqhQAl19+OX/99VdSuYiICJYuXWqJIthUlTZt2jBr1izq1KnD7NmziYhI1x3uxvh1Pr/8L6Z//vmHWrVqsXv3bipXrkzTpk0v6vIXLFjAtGnTkoaLFi2a5jz33nsvOXPmBKBdu3a89NJLdO7cmWnTptGuXbuk5W7evDlpnqNHjxIbG0uBAgWSxu3du5fLLrssaTgmJoaHHnqI3377DREhLi4uaVrTpk259NJLAZg/fz6zZ89m5MiRgHMZ886dO7niiivo2bMnkZGR5MyZk23btqUY/9KlS9PcxmAQkXOuaCpRogR79uwJybqzZaKIi4sjLCwMEaFRo0bcfPPNdO/ePenLa0xWkS9fPiIjIzlx4gTNmjVj7Nix9O7dmypVqrBkyZJzyu7YsYMCBQpQqFAhqlatyurVq5Oqdc6X7wEt+TX9l1xySdL7+vXrs337dg4cOMCsWbN49tlnAUhISGDFihXkzZvX77b5Lvu5557jpptu4ssvv+SPP/6gSZMmKa5TVZk5cyaVKlU6Z3mDBw+mZMmSrFu3joSEhFTXfbHPKK688kqio6OThqOjo7nyyisBKFmyJHv37qVUqVLs3buXEiVKJJVLrGILhWx31dOiRYuoUaMGX331FQD9+vWjV69eliRMlpY/f37GjBnDqFGjOHPmDA8++CA//fQTCxYsAJwzj969e/PUU08B0L9/f15++eWkX9UJCQmMHz/+X8tt2rQpY8eOTRpOrHoqWbIkUVFRJCQk8OWXX6Yal4hw991307dvXypXrkyxYsUAuO2223jrrbeSykVGRv5r3sqVK7N9+/ak4ZiYmKQD7OTJk1NdZ7NmzXjrrbeSrixau3Zt0vylSpUiR44cfPTRR8THx6c4/9KlS4mMjPzXKz1JAqBUqVIUKlSIFStWoKp8+OGHtG7dGoA777yTKVOmADBlypSk8QDbtm0L2Q2/2SZRHDhwgIceeoibbrqJU6dOUbBgQa9DMiakateuTY0aNfj000/Jly8fX331FUOHDqVSpUpUr16da6+9lp49ewJQo0YNRo8eTfv27alcuTLVqlVjx44d/1rms88+y+HDh6lWrRo1a9Zk4cKFALzyyiu0atWKBg0aJNWxp6Zdu3ZMnTo1qdoJYMyYMaxatYoaNWpQpUqVFJNUeHg4MTExSb/un3rqKZ5++mlq167NmTNnUl3fc889R1xcHDVq1KBq1apJ/bR1796dKVOmULNmTbZs2XLOWUh6/frrr5QuXZoZM2bw+OOPU7Xq2SrIWrVqJb0fN24cjz76KNdccw0VKlSgRYsWAAwcOJDvv/+eihUrsmDBAgYOHJg0z8KFC7n99tsvOMZAZItuxj/99FN69OhBbGws/fv3Z9CgQeTPnz9IERrjiIqKonLlyl6HkaW98cYbFCxYkEcffdTrUELqr7/+4oEHHuCHH35IcXpK3z3rZjwNZ86coVq1akRGRjJs2DBLEsZkEd26dSNPnjxehxFyO3fuZNSoUSFbX5Y8ozh+/DhDhgyhbNmydO/ePaku0vrdMaFkZxTGK3ZGkYavv/6aqlWr8uqrryY1xCW/rMyYUMlsP8RM5heM71yWSRTR0dG0adOGO+64g0suuYQlS5YwevRor8My2VjevHk5dOiQJQsTMuo+j8LfZcXpkWXuo9ixYwfz5s1j+PDh9O3bl9y5c3sdksnmSpcuTXR0NAcOHPA6FJONJD7h7mLK1G0UK1euZPny5fTp0weAQ4cOJV2HbYwx5qwM20YhIs1FZKuIbBeRgSlMzyMi093pv4hIuUCWe+TIEbp3787111/P66+/ntSJlyUJY4y5+IKWKEQkJzAWaAFUAdqLSJVkxboAh1X1GuAN4NW0lnv6RAzh4eFMmDCB3r17s2HDhotyY4wxxpiUBfOMoh6wXVV3qOppYBrQOlmZ1sAU9/3nwC2SxuVJxw/uo0yZMvz666+MHj2aQoUKXfTAjTHGnBXMxuwrgV0+w9HAdamVUdUzIhIDFAMO+hYSka5AV3fw1KpVqzYm7yI5mypOsn2Vjdm+OMv2xVm2L86qlHaRlGWKq55UdSIwEUBEVqW3QSarsX1xlu2Ls2xfnGX74iwR+ffDNQIUzKqn3UAZn+HS7rgUy4hILqAwcCiIMRljjDlPwUwUvwIVRaS8iOQG7gdmJyszG3jIfd8W+FEz2/W6xhiTxQWt6sltc+gJzANyAu+r6iYReQnnId+zgfeAj0RkO/A3TjJJy8RgxZwJ2b44y/bFWbYvzrJ9cVa690Wmu+HOGGNMaGWZvp6MMcYEhyUKY4wxfmXYRBGs7j8yowD2RV8R2Swi60XkBxG5yos4QyGtfeFT7h4RURHJspdGBrIvROQ+97uxSUQ+CXWMoRLA/0hZEVkoImvd/5OWXsQZbCLyvojsF5GNqUwXERnj7qf1IlInoAWraoZ74TR+/w+4GsgNrAOqJCvTHRjvvr8fmO513B7ui5uA/O77btl5X7jlCgJLgBVAhNdxe/i9qAisBYq6wyW8jtvDfTER6Oa+rwL84XXcQdoXNwJ1gI2pTG8JfAsIcD3wSyDLzahnFEHp/iOTSnNfqOpCVT3hDq7AuWclKwrkewEwBKffsJOhDC7EAtkXjwFjVfUwgKruD3GMoRLIvlAgsb+fwsCeEMYXMqq6BOcK0tS0Bj5UxwqgiIiUSmu5GTVRpNT9x5WplVHVM0Bi9x9ZTSD7wlcXnF8MWVGa+8I9lS6jqnNDGZgHAvle/B/wfyLys4isEJHmIYsutALZF4OBDiISDXwD9ApNaBnO+R5PgEzShYcJjIh0ACKAxl7H4gURyQG8DjzscSgZRS6c6qcmOGeZS0Skuqoe8TQqb7QHJqvqKBGpj3P/VjVVTfA6sMwgo55RWPcfZwWyLxCRW4FBwJ2qeipEsYVaWvuiIFANWCQif+DUwc7Oog3agXwvooHZqhqnqr8D23ASR1YTyL7oAnwGoKrLgbw4HQZmNwEdT5LLqInCuv84K819ISK1gQk4SSKr1kNDGvtCVWNUtbiqllPVcjjtNXeqaro7Q8vAAvkfmYVzNoGIFMepitoRyiBDJJB9sRO4BUBEKuMkiuz4jNrZQCf36qfrgRhV3ZvWTBmy6kmD1/1HphPgvngNKADMcNvzd6rqnZ4FHSQB7otsIcB9MQ+4TUQ2A/FAf1XNcmfdAe6LfsC7IvIfnIbth7PiD0sR+RTnx0Fxtz3mBSAMQFXH47TPtAS2AyeAzgEtNwvuK2OMMRdRRq16MsYYk0FYojDGGOOXJQpjjDF+WaIwxhjjlyUKY4wxflmiMBmSiMSLSKTPq5yfsrEXYX2TReR3d11r3Lt3z3cZk0Skivv+mWTTll1ojO5yEvfLRhGZIyJF0ihfK6v2lGpCxy6PNRmSiMSqaoGLXdbPMiYDX6vq5yJyGzBSVWtcwPIuOKa0lisiU4BtqjrMT/mHcXrQ7XmxYzHZh51RmExBRAq4z9pYIyIbRORfvcaKSCkRWeLzi/sGd/xtIrLcnXeGiKR1AF8CXOPO29dd1kYRedIdd4mIzBWRde74du74RSISISKvAPncOD52p8W6f6eJyO0+MU8WkbYiklNEXhORX93nBDwewG5Zjtuhm4jUc7dxrYgsE5FK7l3KLwHt3FjaubG/LyIr3bIp9b5rzLm87j/dXvZK6YVzJ3Gk+/oSpxeBQu604jh3liaeEce6f/sBg9z3OXH6fiqOc+C/xB0/AHg+hfVNBtq67+8FfgHqAhuAS3DufN8E1AbuAd71mbew+3cR7vMvEmPyKZMY493AFPd9bpyePPMBXYFn3fF5gFVA+RTijPXZvhlAc3e4EJDLfX8rMNN9/zDwts/8LwMd3PdFcPp/usTrz9teGfuVIbvwMAb4R1VrJQ6ISBjwsojcCCTg/JIuCezzmedX4H237CxVjRSRxjgPqvnZ7d4kN84v8ZS8JiLP4vQB1AWnb6AvVfW4G8MXwA3Ad8AoEXkVp7pq6Xls17fAmyKSB2gOLFHVf9zqrhoi0tYtVxinA7/fk82fT0Qi3e2PAr73KT9FRCridFERlsr6bwPuFJH/usN5gbLusoxJkSUKk1k8CFwG1FXVOHF6h83rW0BVl7iJ5HZgsoi8DhwGvlfV9gGso7+qfp44ICK3pFRIVbeJ89yLlsBQEflBVV8KZCNU9aSILAKaAe1wHrIDzhPHeqnqvDQW8Y+q1hKR/Dh9G/UAxuA8rGmhqt7tNvwvSmV+Ae5R1a2BxGsMWBuFyTwKA/vdJHET8K/ngovzrPC/VPVdYBLOIyFXAA1FJLHN4RIR+b8A17kUuEtE8ovIJTjVRktF5ArghKpOxemQMaXnDse5ZzYpmY7TGVvi2Qk4B/1uifOIyP+560yROk807A30k7Pd7Cd2F/2wT9FjOFVwieYBvcQ9vRKn52Fj/LJEYTKLj4EIEdkAdAK2pFCmCbBORNbi/Fp/U1UP4Bw4PxWR9TjVTuGBrFBV1+C0XazEabOYpKprgerASrcK6AVgaAqzTwTWJzZmJzMf5+FSC9R5dCc4iW0zsEZENuJ0G+/3jN+NZT3OQ3lGAMPdbfedbyFQJbExG+fMI8yNbZM7bIxfdnmsMcYYv+yMwhhjjF+WKIwxxvhlicIYY4xfliiMMcb4ZYnCGGOMX5YojDHG+GWJwhhjjF//D4o6juCYbAT8AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "pred = model.predict(x_test)\n",
        "plot_roc(pred,y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yElFCdNQK1AR"
      },
      "source": [
        "### Multiclass Classification Error Metrics\n",
        "\n",
        "If you want to predict more than one outcome, you will need more than one output neuron. Because a single neuron can predict two results, a neural network with two output neurons is somewhat rare. If there are three or more outcomes, there will be three or more output neurons. The following sections will examine several metrics for evaluating classification error. We will assess the following classification neural network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "wD57aaWKK1AR"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from scipy.stats import zscore\n",
        "\n",
        "# Read the data set\n",
        "df = pd.read_csv(\n",
        "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
        "    na_values=['NA','?'])\n",
        "\n",
        "# Generate dummies for job\n",
        "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
        "df.drop('job', axis=1, inplace=True)\n",
        "\n",
        "# Generate dummies for area\n",
        "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
        "df.drop('area', axis=1, inplace=True)\n",
        "\n",
        "# Missing values for income\n",
        "med = df['income'].median()\n",
        "df['income'] = df['income'].fillna(med)\n",
        "\n",
        "# Standardize ranges\n",
        "df['income'] = zscore(df['income'])\n",
        "df['aspect'] = zscore(df['aspect'])\n",
        "df['save_rate'] = zscore(df['save_rate'])\n",
        "df['age'] = zscore(df['age'])\n",
        "df['subscriptions'] = zscore(df['subscriptions'])\n",
        "\n",
        "# Convert to numpy - Classification\n",
        "x_columns = df.columns.drop('product').drop('id')\n",
        "x = df[x_columns].values\n",
        "dummies = pd.get_dummies(df['product']) # Classification\n",
        "products = dummies.columns\n",
        "y = dummies.values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aggbZoEUK1AR",
        "outputId": "6893844e-1126-4e7d-b6cb-34b2ab04ab4f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/1000\n",
            "47/47 - 2s - loss: 1.4456 - accuracy: 0.4753 - val_loss: 1.1348 - val_accuracy: 0.4980 - 2s/epoch - 45ms/step\n",
            "Epoch 2/1000\n",
            "47/47 - 0s - loss: 1.1477 - accuracy: 0.4720 - val_loss: 1.0890 - val_accuracy: 0.4980 - 249ms/epoch - 5ms/step\n",
            "Epoch 3/1000\n",
            "47/47 - 0s - loss: 1.0847 - accuracy: 0.5040 - val_loss: 1.0205 - val_accuracy: 0.5380 - 482ms/epoch - 10ms/step\n",
            "Epoch 4/1000\n",
            "47/47 - 0s - loss: 0.9608 - accuracy: 0.5920 - val_loss: 0.9546 - val_accuracy: 0.5740 - 309ms/epoch - 7ms/step\n",
            "Epoch 5/1000\n",
            "47/47 - 0s - loss: 0.8508 - accuracy: 0.6480 - val_loss: 0.8616 - val_accuracy: 0.6600 - 290ms/epoch - 6ms/step\n",
            "Epoch 6/1000\n",
            "47/47 - 0s - loss: 0.7942 - accuracy: 0.6660 - val_loss: 0.8018 - val_accuracy: 0.6900 - 298ms/epoch - 6ms/step\n",
            "Epoch 7/1000\n",
            "47/47 - 0s - loss: 0.7581 - accuracy: 0.6927 - val_loss: 0.8044 - val_accuracy: 0.6740 - 271ms/epoch - 6ms/step\n",
            "Epoch 8/1000\n",
            "47/47 - 0s - loss: 0.7434 - accuracy: 0.6893 - val_loss: 0.7885 - val_accuracy: 0.6660 - 246ms/epoch - 5ms/step\n",
            "Epoch 9/1000\n",
            "47/47 - 0s - loss: 0.7522 - accuracy: 0.6867 - val_loss: 0.7835 - val_accuracy: 0.6720 - 281ms/epoch - 6ms/step\n",
            "Epoch 10/1000\n",
            "47/47 - 0s - loss: 0.7158 - accuracy: 0.6987 - val_loss: 0.7727 - val_accuracy: 0.6840 - 327ms/epoch - 7ms/step\n",
            "Epoch 11/1000\n",
            "47/47 - 0s - loss: 0.7129 - accuracy: 0.6887 - val_loss: 0.7966 - val_accuracy: 0.6820 - 231ms/epoch - 5ms/step\n",
            "Epoch 12/1000\n",
            "47/47 - 0s - loss: 0.7105 - accuracy: 0.6947 - val_loss: 0.7700 - val_accuracy: 0.6620 - 239ms/epoch - 5ms/step\n",
            "Epoch 13/1000\n",
            "47/47 - 0s - loss: 0.7119 - accuracy: 0.6940 - val_loss: 0.7680 - val_accuracy: 0.6700 - 254ms/epoch - 5ms/step\n",
            "Epoch 14/1000\n",
            "47/47 - 0s - loss: 0.6934 - accuracy: 0.7047 - val_loss: 0.7743 - val_accuracy: 0.6600 - 289ms/epoch - 6ms/step\n",
            "Epoch 15/1000\n",
            "47/47 - 0s - loss: 0.6904 - accuracy: 0.7093 - val_loss: 0.7564 - val_accuracy: 0.6860 - 266ms/epoch - 6ms/step\n",
            "Epoch 16/1000\n",
            "47/47 - 0s - loss: 0.6837 - accuracy: 0.7007 - val_loss: 0.7423 - val_accuracy: 0.7000 - 297ms/epoch - 6ms/step\n",
            "Epoch 17/1000\n",
            "47/47 - 0s - loss: 0.6783 - accuracy: 0.7120 - val_loss: 0.7519 - val_accuracy: 0.6840 - 258ms/epoch - 5ms/step\n",
            "Epoch 18/1000\n",
            "47/47 - 0s - loss: 0.6665 - accuracy: 0.7153 - val_loss: 0.7582 - val_accuracy: 0.6660 - 259ms/epoch - 6ms/step\n",
            "Epoch 19/1000\n",
            "47/47 - 0s - loss: 0.6702 - accuracy: 0.7000 - val_loss: 0.7504 - val_accuracy: 0.6880 - 271ms/epoch - 6ms/step\n",
            "Epoch 20/1000\n",
            "47/47 - 0s - loss: 0.6624 - accuracy: 0.7147 - val_loss: 0.7527 - val_accuracy: 0.6800 - 328ms/epoch - 7ms/step\n",
            "Epoch 21/1000\n",
            "Restoring model weights from the end of the best epoch: 16.\n",
            "47/47 - 1s - loss: 0.6558 - accuracy: 0.7160 - val_loss: 0.7653 - val_accuracy: 0.6720 - 527ms/epoch - 11ms/step\n",
            "Epoch 21: early stopping\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f6a8ad5db50>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "# Classification neural network\n",
        "import numpy as np\n",
        "import tensorflow.keras\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Split into train/test\n",
        "x_train, x_test, y_train, y_test = train_test_split(    \n",
        "    x, y, test_size=0.25, random_state=42)\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n",
        "                kernel_initializer='random_normal'))\n",
        "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
        "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
        "model.add(Dense(y.shape[1],activation='softmax',\n",
        "                kernel_initializer='random_normal'))\n",
        "model.compile(loss='categorical_crossentropy', \n",
        "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
        "              metrics =['accuracy'])\n",
        "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
        "                        verbose=1, mode='auto', restore_best_weights=True)\n",
        "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
        "          callbacks=[monitor],verbose=2,epochs=1000)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vr9U9rgvK1AR"
      },
      "source": [
        "### Calculate Classification Accuracy\n",
        " \n",
        "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
        "\n",
        "$$ accuracy = \\frac{c}{N} $$\n",
        "\n",
        "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
        "\n",
        "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "ldTptygpK1AR"
      },
      "outputs": [],
      "source": [
        "pred = model.predict(x_test)\n",
        "pred = np.argmax(pred,axis=1) \n",
        "# raw probabilities to chosen class (highest probability)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6_a5WcpJK1AS"
      },
      "source": [
        "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l7ZN4mO5K1AS",
        "outputId": "8a2bf888-d1e2-4317-bc55-855b78b5db99"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy score: 0.7\n"
          ]
        }
      ],
      "source": [
        "from sklearn import metrics\n",
        "\n",
        "y_compare = np.argmax(y_test,axis=1) \n",
        "score = metrics.accuracy_score(y_compare, pred)\n",
        "print(\"Accuracy score: {}\".format(score))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1zlG17MKK1AS"
      },
      "source": [
        "### Calculate Classification Log Loss\n",
        "\n",
        "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
        "\n",
        "The following code shows the output of predict_proba:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "id": "ucCNS9XAK1AS",
        "outputId": "76375262-9554-4105-9d2c-3b088732e2df"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Numpy array of predictions\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "array([[0.    , 0.1201, 0.7286, 0.1494, 0.0018, 0.    , 0.    ],\n",
              "       [0.    , 0.6962, 0.3016, 0.0001, 0.0022, 0.    , 0.    ],\n",
              "       [0.    , 0.7234, 0.2708, 0.0003, 0.0053, 0.0001, 0.    ],\n",
              "       [0.    , 0.3836, 0.6039, 0.0086, 0.0039, 0.    , 0.    ],\n",
              "       [0.    , 0.0609, 0.6303, 0.3079, 0.001 , 0.    , 0.    ]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "As percent probability\n",
            "[ 0.0001 12.0143 72.8578 14.9446  0.1823  0.0009  0.0001]\n",
            "Log loss score: 0.7423401429280638\n"
          ]
        }
      ],
      "source": [
        "from IPython.display import display\n",
        "\n",
        "# Don't display numpy in scientific notation\n",
        "np.set_printoptions(precision=4)\n",
        "np.set_printoptions(suppress=True)\n",
        "\n",
        "# Generate predictions\n",
        "pred = model.predict(x_test)\n",
        "\n",
        "print(\"Numpy array of predictions\")\n",
        "display(pred[0:5])\n",
        "\n",
        "print(\"As percent probability\")\n",
        "print(pred[0]*100)\n",
        "\n",
        "score = metrics.log_loss(y_test, pred)\n",
        "print(\"Log loss score: {}\".format(score))\n",
        "\n",
        "# raw probabilities to chosen class (highest probability)\n",
        "pred = np.argmax(pred,axis=1) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iU3NLdorK1AS"
      },
      "source": [
        "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
        "\n",
        "$$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $$\n",
        "\n",
        "\n",
        "You should use this equation only as an objective function for classifications that have two outcomes. The variable y-hat is the neural network’s prediction, and the variable y is the known correct answer.  In this case, y will always be 0 or 1.  The training data have no probabilities. The neural network classifies it either into one class (1) or the other (0).  \n",
        "\n",
        "The variable N represents the number of elements in the training set the number of questions in the test.  We divide by N because this process is customary for an average.  We also begin the equation with a negative because the log function is always negative over the domain 0 to 1.  This negation allows a positive score for the training to minimize.\n",
        "\n",
        "You will notice two terms are separated by the addition (+).  Each contains a log function.  Because y will be either 0 or 1, then one of these two terms will cancel out to 0.  If y is 0, then the first term will reduce to 0.  If y is 1, then the second term will be 0.  \n",
        "\n",
        "If your prediction for the first class of a two-class prediction is y-hat, then your prediction for the second class is 1 minus y-hat.  Essentially, if your prediction for class A is 70% (0.7), then your prediction for class B is 30% (0.3).  Your score will increase by the log of your prediction for the correct class.  If the neural network had predicted 1.0 for class A, and the correct answer was A, your score would increase by log (1), which is 0. For log loss, we seek a low score, so a correct answer results in 0.  Some of these log values for a neural network's probability estimate for the correct class:\n",
        "\n",
        "* -log(1.0) = 0\n",
        "* -log(0.95) = 0.02\n",
        "* -log(0.9) = 0.05\n",
        "* -log(0.8) = 0.1\n",
        "* -log(0.5) = 0.3\n",
        "* -log(0.1) = 1\n",
        "* -log(0.01) = 2\n",
        "* -log(1.0e-12) = 12\n",
        "* -log(0.0) = negative infinity\n",
        "\n",
        "As you can see, giving a low confidence to the correct answer affects the score the most.  Because log (0) is negative infinity, we typically impose a minimum value.  Of course, the above log values are for a single training set element.  We will average the log values for the entire training set.\n",
        "\n",
        "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 360
        },
        "id": "g5Zv2tgNK1AT",
        "outputId": "f0861083-7809-406f-9970-4c83505ca14a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
            "  if sys.path[0] == '':\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5icdX338c93z+dD9phzQkgIOQCGDSdRNhARWpF6fKgVlUdFsWppq9YWz61tpSfbq31auap9bKtGKh54FBFQArYQQhIOSQghJxKSbLKbTbLn4+z3+WNmw5Bssvfs7Ow9s/N+XddcM3Pfc/jON/fOfvLb3/zG3F0AAAAAJldO2AUAAAAA0xFBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoA0CaM7OXzWztJDzOm83sxwFvu9HMlif7nACQzQjaAJA9virpLwPe9q8lfSWFtQDAtEfQBoAsYGarJVW6+4aAd7lf0hoza0xhWQAwrRG0ASBDmFmhmX3dzA7HTl83s8K4/Z8xs5bYvg+ZmZvZ+bHdN0p6LO62V5nZMTObG7t+sZmdMLOlkuTu/ZI2S3rz1L1CAJheCNoAkDnuknSFpEskXSzpMkmfkyQzu0HSH0haK+l8Sc2n3XelpJ2jV9z9CUnfkPRtMyuW9J+SPu/uL8bdZ0fseQAAE0DQBoDM8TuSvuLure7eJunLkm6N7Xu3pH9z9+3u3ivpS6fdt0pS12nbviSpUtJGSYck/dNp+7ti9wMATABBGwAyxyxJ++Ou749tG933Sty++MuSdEJSefwGdx+S9H8lrZD0N+7up92nXNLJ5EoGgOxF0AaAzHFY0vy46/Ni2ySpRdKcuH1zT7vv85KWxG8ws9mSvijp3yT9Tfx875gLJT2XZM0AkLUI2gCQOb4n6XNmVmdmtZK+oOjcakm6V9JtZnahmZVI+vxp931A0jWjV8zMFB3N/qakDyoa1P80bn+RpEslPZyalwIA0x9BGwAyx59J2qTo6PRWSVti2+TuP5f0D5IelbRb0ugyfgOx/VskdZjZ5bHtn5RUr+gHIF3SbYoG9TfE9t8kab27j46YAwASZGdOyQMAZDozu1DSNkmF7j4c23a9pI+5+28FuP9Tkj7o7ttSWykATF8EbQCYJszsbYpOESmR9G1JI0FCNQAgNZg6AgDTx0cktUraIyki6Y5wywGA7MaINgAAAJACjGgDAAAAKUDQBgAAAFIgL+wCElFbW+sLFiwI7fl7enpUWloa2vNnOvo3cfQuOfQvOfRv4uhdcuhfcuhfcjZv3nzM3euSeYyMCtoLFizQpk2bQnv+9evXq7m5ObTnz3T0b+LoXXLoX3Lo38TRu+TQv+TQv+SY2f5kH4OpIwAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKhBq0zewGM9tpZrvN7LNh1gIAAABMptCCtpnlSvonSTdKWibpt81sWVj1AAAAAJMpzBHtyyTtdve97j4oaZ2km0OsBwBwmsHhEQ2NeNhlAEBGygvxuWdLeiXu+kFJl4dUCwBkhZER18m+IbV3D+hY96DaewbU3j0Yvd4TPW/vHlR77HJn/7Dee2GB3hR24QCQgcw9nJEKM3unpBvc/UOx67dKutzdP37a7W6XdLskNTQ0XLpu3bopr3VUd3e3ysrKQnv+TEf/Jo7eJWc698/d1R+RugZdnQOurqHoeedg9NQVO49ui95urHd9k1RWIFUUmCoKTOWxU0WBaVHpoFbMnJ79S7XpfOxNBfqXHPqXnDVr1mx296ZkHiPMEe1DkubGXZ8T2/Ya7n6PpHskqampyZubm6ekuLGsX79eYT5/pqN/E0fvkpNp/XN3dfYNq627X61dA2obPXW/OvocHXEe1LHuAQ0Mj4z5OGWFeaopK1RNaYFmNxSqtqxANaWFqikrUE1ZoWpLo+c1ZQWqLilQbo6N+TiZ1r90Qu+SQ/+SQ//CF2bQflrSYjNbqGjAvkXSe0KsBwBSqn8ociowt3UNvDZEx7Yfi10ejJwZnvNzTbWxYFxTWqjz68ui1+MC86nLpQUqys8N4VUCAEaFFrTdfdjMPi7pF5JyJX3L3beHVQ8ATMTIiOt476BaOwfU2tX/miAdH6DbugbU1T98xv3NpJrSAtWWFaquvFCL6kpVV16outj1uvJC1ZcXqq6sSBXFeTIbe9QZAJB+whzRlrs/IOmBMGsAgLG4u7oGhtXa2a8jHQM62tmvo139OtrRr6OdA6cut3YNaHiMVTnKCvNOBeYLGyv0xsWFZwTouvLoyHNeLt8dBgDTUahBGwDCMDqF40hnv4529utILDCffrl3MHLGfcuL8tRYUaSGiiJdsajm1OX68kLVV0RHnmvLC1RSwNsrAGQ7fhMAmFb6BiM63NGnlpP9OtzRpyMd/dq8Y0D/tndjNEh39utk79AZ9yvIy4mF5kItn1Wha5fWq6GiUA2xIN1YUaT6ikICNAAgMH5jAMgYA8MRHeno1+GT/Wrp6FNLR+z8ZL8Oxy6PFaIrCqT5dYOaU12ipgXVaigvUkPlqwG6oaJQlcX5zH8GAEwqgjaAtODuause0METfTp4ok+HT/ap5WSfDndEp3O0dPTpWPfgGferKsnXzMpizaos0qp5VZpVVayZlUXRbVXRML3hf36t5uarQ3hVAIBsRtAGMCVGRlytXQM6eKJXh072xQJ1rw6e6NOhE306eLJPg6etB11emKeZVdHQvHxWhWZWFmtmVZFmxc5nVhYxlQMAkLb4DQVgUkRGXEc6+6Oh+TUBOnq55WT/GWtD15QWaE51sZbOLNfaZQ2aXVWsOdXFmlNdoplVRaooyg/p1QAAkDyCNoDAegaGdeB4b/TU3qv9x3t04HifDrT36OCJvjOWuasrL9Sc6mKtnF2pG1fM1OzqaJCeW12sWVXFjEYDAKY1fssBOGV0nvSB9miY3h87H718rHvgNbevKMrT/JpSLZ9dqRtXztTc6pLYiHQ0SPPNhACAbEbQBrKMu6uta0B7j/VoX9zplVigjl872kyaVVmsuTOKdd3Ses2rKdG8GSWaX1Oi+TNKVVnC1A4AAM6GoA1MU539Q9rXFg3Rr4bqbu1r61FPXJguyMvR/Fh4vmpRrebHwvS8mujodGEeo9IAAEwEQRvIYEOREe1v79Hu1rggHQvV8Uvh5Zg0p7pEC2tL1TR/hs6rK9XC2uhpZmWxcnNYPxoAgMlG0AYywGDE9cLhTu1q7dKe1m7tip1ePtbzmg8g1pUXamFtqdZe2HAqSJ9XV6q5M0oYmQYAYIoRtIE00j0wfCpI727t1u7WLu1q7daB9l75w7+WFB2dXlBTqvPry3T9sgYtbijToroyLawtVTnL4QEAkDYI2kAIhiIj2tvWoxePdOrFI116saVTLx3t1qGTfaduk59rOq+2TCtmV+qSqiG96fIVWlxfrgW1jE4DAJAJCNpACrlHvw1xR0s0UO880qUdLZ3a09atoUh0ykd+rmlRXZlWL6jWexrm6fz6Mi2uL9O8GSXKy82RJK1fv17NF80K86UAAIAEEbSBSTI4PKJdrV3afqhTL7R0nhqtPtk7dOo2MyuLdEFjuZovqNeFM8t1QWO5zqstU0FeToiVAwCAVCBoAxPQPxTRziNd2na4Q9sOdWjboU7tPNJ16ivGSwpytaShXDeuaNTSxgpd0FiupY3lqiopCLlyAAAwVQjawDh6Boa1o6UzGqgPR893tXYrElvto7I4XytmV+i21y/Q8tmVWjGrQgtqSpXDknkAAGQ1gjYQJzLi2tXapWcPnNSzr0RPLx3t0ugKejWlBVoxu1LXXVivlbMrtXxWpeZUF8uMUA0AAF6LoI2sdrSzX8+cCtUntPVgx6lvTawsztfFc6t0/bIGrZxTpZWzK9VQUUioBgAAgRC0kTWGIyN6oaVTT798Qpv3H9czB06qpaNfUnTlj2UzK/SOS+fokrlVumRulRbWlhKqAQDAhBG0MW31DAzr2VdO6umXj2vTyye05cAJ9cZGq+dUF2v1ghnRUD2vSstmVqgon7WpAQDA5CFoY9o40TOop/a16+mXT2jTy8e17XCnIiMuM+nCxgq969I5Wr1whprmz1BjZVHY5QIAgGmOoI2M1dU/pI37juuJPe16Yk+7drR0SpIK83L0unlV+ljzIjUtmKHXzatSBV9NDgAAphhBGxmjbzCizftP6Ik9x/TEnnZtPdShyIirMC9HTQuq9anrl+jKRTVaObuKL4ABAAChI2gjbbm7dh7t0mM72/TYS23a9PIJDUZGlJdjunhudMT6ykU1WjWvmvnVAAAg7RC0kVY6+ob0P7uPnQrXRzqjq4IsbSzX+6+ar6vOr9XqBTNUVsihCwAA0htpBaFyd710tFuP7Diq9TtbteXASUVGXOWFebp6ca2aL6jTG5fUaWZlcdilAgAAJISgjSkXGXFt3n9CD20/ood3HNX+9l5J0vJZFfroNefpmiX1et28KuXnMs8aAABkLoI2pkTfYETPtA7rZ//1nH75YquO9wyqIDdHVy6q0YffcJ7etKxBDRUsuQcAAKYPgjZSpn8oovU72/TT5w/rlzta1TcUUXnREV27tF5vWtaga5bUqZxl9wAAwDRF0MakGhwe0X/vbtNPn2vRQy8cVffAsGaUFuhtq2ZrdqRVH/6tNSy9BwAAsgJBG0lzd23cd1w/euaQfr7tiDr6hlRRlKffWNmot1w0S1ctqlFebo7Wr19PyAYAAFmDoI0Je+V4r3645ZDu23JQB473qqQgV29e3qi3XDRTb1hcR6gGAABZjaCNhPQODuvnW4/oB5sP6sm97ZKkqxbV6M61i3XDikaVFHBIAQAASARtBLS7tUv/ueGA7tt8UF0Dw5o3o0R/8KYletvrZmvujJKwywMAAEg7BG2c1eDwiB564Yj+c8N+bdh7XAW5ObpxZaPec9k8XbZwhsws7BIBAADSFkEbZ2jvHtC/P7lf3914QG1dA5pTXaw/umGp3tU0R7VlhWGXBwAAkBEI2jhlT1u3vvnf+3Tf5oMaGB7RtUvrdesV8/XGJXXKzWH0GgAAIBEEbejpl4/rnsf36pEdR5Wfm6N3rJqjD169UOfXl4VdGgAAQMYiaGexJ/e06+uPvKSn9h1XdUm+PnHtYr3vyvlMDwEAAJgEBO0s4+56cm+7vv7ILm3cd1z15YX64k3LdMvqeSouyA27PAAAgGmDoJ1FNu8/rq89uPNUwP7STct0y2XzVJRPwAYAAJhsoQRtM/srSTdJGpS0R9Jt7n4yjFqywb5jPbr7wRf1821HVFdeqC+/dbn+1+q5BGwAAIAUCmtE+2FJf+zuw2b2NUl/LOmPQqpl2mrvHtA//HKXvvPUARXk5ej31y7Rh9+4kG9vBAAAmAKhJC53fyju6gZJ7wyjjukqMuL6zlP79Ve/2KnewYhuWT1Xd65dorpyPuQIAAAwVdJhaPN/S/p+2EVMF88cOKHP/2Sbth3q1OvPr9GX37pc59eXh10WAABA1jF3T80Dmz0iqXGMXXe5+09it7lLUpOkt/tZCjGz2yXdLkkNDQ2Xrlu3LiX1BtHd3a2ysvRcW7p3yHXvS4N67JVhVRaafntpgS5rzE2rr0lP5/6lO3qXHPqXHPo3cfQuOfQvOfQvOWvWrNns7k3JPEbKgva4T2z2AUkfkXSdu/cGuU9TU5Nv2rQppXWdy/r169Xc3Bza85/NYy+16bP3Pa+jnf267fULdefaxSovyg+7rDOka/8yAb1LDv1LDv2bOHqXHPqXHPqXHDNLOmiHterIDZI+I+maoCEbZ+rqH9JXf7ZD655+RefXl+mHH3u9LplbFXZZAAAAUHhztP9RUqGkh2NTGza4+0dDqiUjbd5/Qp/83jNq6ejTR69ZpDvXLma5PgAAgDQS1qoj54fxvNPByIjrG4/v1V8/tFOzqor0gzuu0qp51WGXBQAAgNOkw6ojCKi9e0C/f+9zevylNv3mRTP1F29fqYo0nIsNAAAAgnbG2H64Qx/+9iYd6xnUV9+2Qu+5bF5arSgCAACA1yJoZ4AHtrboD+99TlUl+brvo1dp5ZzKsEsCAADAOAjaaczd9fe/3KWvP7JLq+ZV6V9uvVT15UVhlwUAAIAACNppKjLi+tyPt+l7Gw/oHavm6M/fvkKFeawqAgAAkCkI2mmofyiiO9c9qwe3H9HvrlmkT11/AfOxAQAAMgxBO830DUb0wW8/rSf2tOvzb1mmD169MOySAAAAMAEE7TTSPxQN2Rv2tutv332x3r5qTtglAQAAYIII2mmifyiiD//7Jj25t11/8y5CNgAAQKbLCbsASMOREX38u1v0613H9LV3XETIBgAAmAYI2iFzd33h/u16ZEer/vTm5Xp309ywSwIAAMAkIGiH7J8f26PvPnVAdzQv0q1XLgi7HAAAAEwSgnaIHtjaorsf3KmbL5mlT19/QdjlAAAAYBIRtEOyu7VLn/qv57RqXpXufudFyslhnWwAAIDphKAdgq7+Id3+H5tVUpCnf37vpXzjIwAAwDTE8n5TzN312R9u1f72Xn3nQ5eroaIo7JIAAACQAoxoT7EfPXNIP3u+RX94/RJdcV5N2OUAAAAgRQjaU+jQyT598SfbtXpBtT7yxkVhlwMAAIAUImhPEXfXp+59TiPu+tt3X6JcPvwIAAAwrRG0p8h9Ww7pyb3tuus3l2nujJKwywEAAECKEbSnwMneQf35Azu0al6VblnNNz8CAABkA4L2FPjagzvV0Tekr75tJetlAwAAZAmCdortaOnUuqcP6P1XLtCFMyvCLgcAAABThKCdYnc/+KLKC/P0e9ctDrsUAAAATCGCdgpt2NuuR3e26WNrzldlSX7Y5QAAAGAKEbRTxN1194MvqrGiSB+4akHY5QAAAGCKjRu0zewTZlY9FcVMJxv2HteWAyf1u2sWqSg/N+xyAAAAMMWCjGg3SHrazO41sxvMjGUzAvjnx/aotqxA72piOT8AAIBsNG7QdvfPSVos6ZuSPiBpl5n9uZnxHeJnsf1whx5/qU23vX4ho9kAAABZKtAcbXd3SUdip2FJ1ZJ+YGZ3p7C2jPWvv96nssI8vfeK+WGXAgAAgJDkjXcDM/s9Se+TdEzSv0r6tLsPmVmOpF2SPpPaEjPLiZ5B/Wxri25ZPVeVxaw0AgAAkK3GDdqSZkh6u7vvj9/o7iNm9pbUlJW57ttyUIPDI3rP5fPCLgUAAAAhGjdou/sXz7Fvx+SWk9ncXd/deECr5lVpaSPfAgkAAJDNWEd7Em3af0J723r0nsuZmw0AAJDtCNqT6P5nD6soP0c3rmgMuxQAAACEjKA9SYYjI3pga4uuu7BBpYVBpr4DAABgOiNoT5In97arvWdQN100K+xSAAAAkAYI2pPkp8+1qKwwT80X1IVdCgAAANIAQXsSjIy4fvliq5ovqOObIAEAACCJoD0pth/u1LHuAV27tD7sUgAAAJAmCNqT4NGdrTKTrlnCtBEAAABEEbQnwaM7W3XxnCrVlBWGXQoAAADSBEE7SR29Q3r2lZOMZgMAAOA1Qg3aZvaHZuZmVhtmHcnYtP+43KUrF9WEXQoAAADSSGhB28zmSrpe0oGwapgMG/cdV0Fuji6ZWxV2KQAAAEgjYY5o/52kz0jyEGtI2saXj+uiOZUs6wcAAIDXCCVom9nNkg65+3NhPP9k6R0c1taDHbps4YywSwEAAECaMffUDCib2SOSGsfYdZekP5F0vbt3mNnLkprc/dhZHud2SbdLUkNDw6Xr1q1LSb1BdHd3q6ys7NT1He0Rfe3pfv3+pYW6uC4vtLoyxen9Q3D0Ljn0Lzn0b+LoXXLoX3LoX3LWrFmz2d2bknmMlKVDd1871nYzWylpoaTnzEyS5kjaYmaXufuRMR7nHkn3SFJTU5M3NzenquRxrV+/XvHPv+vxvZJ26L03voGl/QI4vX8Ijt4lh/4lh/5NHL1LDv1LDv0L35QPw7r7VkmnvkJxvBHtdLb1UIdmVRYRsgEAAHAG1tFOwrbDHVoxuzLsMgAAAJCGQg/a7r4gE0ezuweGte9YD0EbAAAAYwo9aGeqFw53yl1aSdAGAADAGAjaE7TzaJckaenM8pArAQAAQDoiaE/QntZulRbkqrGiKOxSAAAAkIYI2hO0p61bi+rLFFuiEAAAAHgNgvYE7Wnt1qI6FoEHAADA2AjaE9AzMKzDHf1aVFcadikAAABIUwTtCdh3rEeSGNEGAADAWRG0J2A0aC+oZUQbAAAAYyNoT8DBE32SpDnVxSFXAgAAgHRF0J6AQyd7VVmcr/Ki/LBLAQAAQJoiaE/AoRN9ml3FaDYAAADOjqA9AYdO9mk200YAAABwDgTtBLk7I9oAAAAYF0E7QR19Q+oZjPBBSAAAAJwTQTtBh0/2S5JmMaINAACAcyBoJ+hY94Akqa68MORKAAAAkM4I2glq64oG7doygjYAAADOjqCdoNER7dqygpArAQAAQDojaCfoWPeAivJzVFaYF3YpAAAASGME7QQd6x5UbVmhzCzsUgAAAJDGCNoJOtY9wPxsAAAAjIugnaC2LoI2AAAAxkfQTlB06ggfhAQAAMC5EbQT4O7q7BtSZUl+2KUAAAAgzRG0EzA0Ig1GRlRZTNAGAADAuRG0E9Az5JJE0AYAAMC4CNoJ6B2OnlcUEbQBAABwbgTtBPQyog0AAICACNoJGJ06UkHQBgAAwDgI2gkYnTrCiDYAAADGQ9BOwOjUkYqivJArAQAAQLojaCeAqSMAAAAIiqCdgL5hV3F+rvJzaRsAAADOjcSYgIGIVFKQG3YZAAAAyAAE7QQMRKRigjYAAAACIGgnYCDijGgDAAAgEIJ2AqIj2qw4AgAAgPERtBMwGHGV5DOiDQAAgPERtBPAHG0AAAAERdBOwEDECdoAAAAIhKCdgMGImDoCAACAQAjaCWDVEQAAAARF0E7AwDCrjgAAACCY0IK2mX3CzF40s+1mdndYdQQ1HBnRsEvFTB0BAABAAKEMz5rZGkk3S7rY3QfMrD6MOhLRNxSRJBUX8EcAAAAAjC+s1HiHpL909wFJcvfWkOoIbHB4RJJUmMeINgAAAMYXVtBeIukNZvaUmT1mZqtDqiOwoYhLkvJzGdEGAADA+MzdU/PAZo9Iahxj112SvirpUUmflLRa0vclnedjFGNmt0u6XZIaGhouXbduXUrqHU9b74g+/XifPrSyQFfPzg+lhkzX3d2tsrKysMvISPQuOfQvOfRv4uhdcuhfcuhfctasWbPZ3ZuSeYyUzdF297Vn22dmd0j6YSxYbzSzEUm1ktrGeJx7JN0jSU1NTd7c3Jyagsexu7VbevwxrVy+TM2XzA6lhky3fv16hfXvl+noXXLoX3Lo38TRu+TQv+TQv/CFNQ/ix5LWSJKZLZFUIOlYSLUEMhSJztEuYOoIAAAAAghrUehvSfqWmW2TNCjp/WNNG0knp4J2HkEbAAAA4wslaLv7oKT3hvHcEzW66ggfhgQAAEAQpMaABiMEbQAAAARHagxodHk/po4AAAAgCFJjQEPDfBgSAAAAwZEaAzo1dSTPQq4EAAAAmYCgHRDL+wEAACARpMaAWHUEAAAAiSA1BjTIOtoAAABIAKkxID4MCQAAgESQGgMaXd4vL5cPQwIAAGB8BO2AhkdiQTuHlgEAAGB8pMaARjwatMnZAAAACILYGFAkNqKda0wdAQAAwPgI2gGdCto5BG0AAACMj6Ad0Ii7TJIxog0AAIAACNoBRUZcDGYDAAAgKIJ2QBF3MZgNAACAoAjaAY0wog0AAIAEELQDiozQLAAAAARHdgxoxBnRBgAAQHAE7YD4MCQAAAASQdAOKMKINgAAABJA0A4o+mFIkjYAAACCIWgHxNQRAAAAJIKgHVAk9s2QAAAAQBAE7YBYRxsAAACJIGgHFHERtAEAABAYQTsgRrQBAACQCIJ2QHwYEgAAAIkgaAcUXUebpA0AAIBgCNoBMXUEAAAAiSBoBxRxp1kAAAAIjOwYUGTExcwRAAAABEXQDshZ3g8AAAAJIGgHNMI3QwIAACABBO2A3MXUEQAAAARG0A7I5WGXAAAAgAxC0E4AA9oAAAAIiqAdkDOgDQAAgAQQtAMiZwMAACARBO0E8GFIAAAABEXQDoohbQAAACSAoB2Qi3W0AQAAEBxBOyA+DAkAAIBEhBK0zewSM9tgZs+a2SYzuyyMOhLFHG0AAAAEFdaI9t2Svuzul0j6Qux6WmNAGwAAAIkIK2i7pIrY5UpJh0OqIzB35mgDAAAguLyQnvdOSb8ws79WNOxfFVIdAAAAQEqYp+hTfmb2iKTGMXbdJek6SY+5+31m9m5Jt7v72rM8zu2Sbo9dvUDSzlTUG1CtpGMhPn+mo38TR++SQ/+SQ/8mjt4lh/4lh/4l5wJ3L0/mAVIWtM/5pGYdkqrc3c3MJHW4e8V49wubmW1y96aw68hU9G/i6F1y6F9y6N/E0bvk0L/k0L/kTEb/wpqjfVjSNbHL10raFVIdAAAAQEqENUf7w5L+3szyJPXr1akhAAAAwLQQStB29/+WdGkYz52ke8IuIMPRv4mjd8mhf8mhfxNH75JD/5JD/5KTdP9CmaMNAAAATHd8BTsAAACQAgRtSWZ2g5ntNLPdZvbZMfYXmtn3Y/ufMrMFcfv+OLZ9p5m9eSrrThcB+vcHZvaCmT1vZr80s/lx+yJm9mzsdP/UVp4eAvTvA2bWFtenD8Xte7+Z7Yqd3j+1lYcvQO/+Lq5vL5nZybh9HHtm3zKzVjPbdpb9Zmb/EOvv82a2Km5fth974/Xud2I922pmT5jZxXH7XoQda5UAAAYsSURBVI5tf9bMNk1d1ekjQP+azawj7mf0C3H7zvlznw0C9O/Tcb3bFnu/mxHbl9XHn5nNNbNHY7lku5n93hi3mbz3PnfP6pOkXEl7JJ0nqUDSc5KWnXabj0n6l9jlWyR9P3Z5Wez2hZIWxh4nN+zXlIb9WyOpJHb5jtH+xa53h/0aMqB/H5D0j2Pcd4akvbHz6tjl6rBfUzr17rTbf0LSt+KuZ/WxF+vBGyWtkrTtLPt/Q9LPJZmkKyQ9Fdue1cdewN5dNdoTSTeO9i52/WVJtWG/hjTvX7Okn46xPaGf++l6Gq9/p932Jkm/irue1cefpJmSVsUul0t6aYzfu5P23seItnSZpN3uvtfdByWtk3Tzabe5WdK3Y5d/IOk6M7PY9nXuPuDu+yTtjj1eNhm3f+7+qLv3xq5ukDRnimtMZ0GOv7N5s6SH3f24u5+Q9LCkG1JUZzpKtHe/Lel7U1JZhnD3xyUdP8dNbpb07x61QVKVmc0Ux964vXP3J2K9kXjfO0OAY+9sknnPnDYS7B/vfXHcvcXdt8Qud0naIWn2aTebtPc+gna0ua/EXT+oMxt+6jbuPiypQ1JNwPtOd4n24IOK/i9xVJGZbTKzDWb2W6koMM0F7d87Yn+++oGZzU3wvtNV4Ncfm660UNKv4jZn+7EXxNl6nO3HXqJOf99zSQ+Z2WaLfvsxxnalmT1nZj83s+WxbRx7CTCzEkWD4H1xmzn+Yiw6Ffh1kp46bdekvfeFtY42spCZvVdSk179siJJmu/uh8zsPEm/MrOt7r4nnArT1v+T9D13HzCzjyj615VrQ64p09wi6QfuHonbxrGHlDOzNYoG7avjNl8dO/bqJT1sZi/GRijxqi2K/ox2m9lvSPqxpMUh15SJbpL0P+4eP/rN8SfJzMoU/Q/Ine7emarnYURbOiRpbtz1ObFtY97Gol+yUympPeB9p7tAPTCztZLukvRWdx8Y3e7uh2LneyWtV/R/ltlk3P65e3tcz/5Vr65Bn+3HXyKv/xad9qdTjr1AztbjbD/2AjGzixT9mb3Z3dtHt8cde62SfqTsm3I4LnfvdPfu2OUHJOWbWa049hJ1rve+rD3+zCxf0ZD9HXf/4Rg3mbT3PoK29LSkxWa20MwKFD0oT1+B4H5Jo58sfaeiHyrw2PZbLLoqyUJF/7e9cYrqThfj9s/MXifpG4qG7Na47dVmVhi7XCvp9ZJemLLK00OQ/s2Mu/pWReeTSdIvJF0f62O1pOtj27JFkJ9dmdlSRT+08mTcNo69YO6X9L7YJ/CvkNTh7i3i2BuXmc2T9ENJt7r7S3HbS82sfPSyor0bc+WIbGZmjbHPQsnMLlM0r7Qr4M89JDOrVPQvyD+J25b1x1/suPqmpB3u/rdnudmkvfdl/dQRdx82s48r2qhcRVcl2G5mX5G0yd3vV/Qf5D/MbLeiHz64JXbf7WZ2r6K/oIcl/e5pf5qe9gL2768klUn6r9j75gF3f6ukCyV9w8xGFH0T/Ut3z6qwE7B/nzSztyp6jB1XdBUSuftxM/tTRX/xSNJXTvvz4LQWsHdS9Od1Xew/x6Oy/tiTJDP7nqKrO9Sa2UFJX5SUL0nu/i+SHlD00/e7JfVKui22L6uPPSlQ776g6Gd5/k/sfW/Y3ZskNUj6UWxbnqTvuvuDU/4CQhagf++UdIeZDUvqk3RL7Gd4zJ/7EF5CqAL0T5LeJukhd++JuyvHX3Rg5VZJW83s2di2P5E0T5r89z6+GRIAAABIAaaOAAAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBYBoys9Vm9ryZFcW+DW67ma0Iuy4AyCZ8YQ0ATFNm9meSiiQVSzro7n8RckkAkFUI2gAwTZlZgaJfFdwv6Sp3j4RcEgBkFaaOAMD0VSOpTFK5oiPbAIApxIg2AExTZna/pHWSFkqa6e4fD7kkAMgqeWEXAACYfGb2PklD7v5dM8uV9ISZXevuvwq7NgDIFoxoAwAAACnAHG0AAAAgBQjaAAAAQAoQtAEAAIAUIGgDAAAAKUDQBgAAAFKAoA0AAACkAEEbAAAASAGCNgAAAJAC/x9OMXC1ykcagQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "%matplotlib inline\n",
        "from matplotlib.pyplot import figure, show\n",
        "from numpy import arange, sin, pi\n",
        "\n",
        "#t = arange(1e-5, 5.0, 0.00001)\n",
        "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
        "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
        "\n",
        "fig = figure(1,figsize=(12, 10))\n",
        "\n",
        "ax1 = fig.add_subplot(211)\n",
        "ax1.plot(t, np.log(t))\n",
        "ax1.grid(True)\n",
        "ax1.set_ylim((-8, 1.5))\n",
        "ax1.set_xlim((-0.1, 2))\n",
        "ax1.set_xlabel('x')\n",
        "ax1.set_ylabel('y')\n",
        "ax1.set_title('log(x)')\n",
        "\n",
        "show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZaRtddpcK1AT"
      },
      "source": [
        "### Confusion Matrix\n",
        "\n",
        "A confusion matrix shows which predicted classes are often confused for the other classes. The vertical axis (y) represents the true labels and the horizontal axis (x) represents the predicted labels. When the true label and predicted label are the same, the highest values occur down the diagonal extending from the upper left to the lower right. The other values, outside the diagonal, represent incorrect predictions. For example, in the confusion matrix below, the value in row 2, column 1 shows how often the predicted value A occurred when it should have been B."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 454
        },
        "id": "1pvqhwRcK1AT",
        "outputId": "89d39b8b-af2d-4861-81da-905acb8c5528"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Normalized confusion matrix\n",
            "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
            " [0.02 0.78 0.2  0.   0.   0.   0.  ]\n",
            " [0.   0.29 0.7  0.01 0.   0.   0.  ]\n",
            " [0.   0.   0.71 0.29 0.   0.   0.  ]\n",
            " [0.   1.   0.   0.   0.   0.   0.  ]\n",
            " [0.59 0.41 0.   0.   0.   0.   0.  ]\n",
            " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEmCAYAAAAA6gkZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debwcVZn/8c83CSHBLAIBhCxsBmRRECIIiEZABEYCiMomi4KBAcSBQWVGVIzjoIOOjgKDEZVFZJMtCgK/ATIIP4GEVRJkzCAhARQSwg5C4Jk/6jR0mt5u0tVVfe/3zateVFdVn/N035vnnlN16pQiAjMze6tBRQdgZlZWTpBmZg04QZqZNeAEaWbWgBOkmVkDTpBmZg04QfYASTMlHZHWD5J0fYfLX09SSBrSyXJb1ClJP5e0RNIdK1DOjpIe7GRsRZE0QdLzkgYXHYtlnCABSQ9LekLS26q2HSFpZoFh1RURF0TErkXH0QEfAD4CjIuIbZa3kIj4XURs3Lmw8pF+x3ZpdkxEPBIRIyLitW7FZc05Qb5pMPCFFS0ktYz8vba2LvBwRLxQdCBl0M3Wu7XP/5DfdBpwoqS319spaXtJsyQ9k/6/fdW+mZK+JelW4EVgg9RlPVrSnyQ9J+mbkjaU9P8lPSvpEklD0/tXlfQbSU+mLudvJI1rEMdhkm5J619KXbLK8qqkc9K+0ZJ+KulxSY9K+pdK103SYEnflbRI0kPA3zX7YiSNl3R5im+xpNPT9kGSTpY0P7XAz5M0Ou2rdNsPlfRIqusrad/hwNnAdinub1R/rqp6Q9I70/oekuam7/JRSSem7ZMlLax6zybp5/G0pDmSplTtO0fSGZKuTuXcLmnDBp+5Ev9nJC1IP5ejJL1P0n2p/NOrjt9Q0o3p+1kk6YLK75Kk84EJwK/T5/1SVfmHS3oEuLFq2xBJq0laKGnPVMYISfMkHdLsZ2UdFhEDfgEeBnYBLgf+JW07ApiZ1lcDlgAHA0OAA9Lr1dP+mcAjwGZp/0pAAFcBo9L2vwE3ABsAo4G5wKHp/asD+wKrACOBS4Erq+KbCRyR1g8DbqnzGcYDjwG7p9dXAD8G3gasCdwBHJn2HQX8Mb1nNeCmFO+QOuUOBu4Fvp/KGgZ8IO37LDAvfaYR6fs7P+1bL5X5E2A4sEX6Djap9znqfa70/nem9ceBHdP6qsBWaX0ysDCtr5Ti+WdgKLAT8Bywcdp/DrAY2Cb9nC4ALmrwO1GJ/6z0mXcFXgauTN/nWOAJ4EPp+HeSnTJYGVgDuBn4Qe3vWJ3yz0vf6/CqbUPSMbsCf0n1/QT4VdH/VgbaUngAZVh4M0FuDjyTfsGrE+TBwB017/k9cFhanwlMq9kfwA5Vr+8Evlz1+nvV/4Bq3rslsKTq9UyaJMj0j+uN8oG1UjIaXnXMAcBNaf1G4KiqfbvSOEFuBzzZYN8NwNFVrzcGXk3Jp/KPfVzV/juA/et9jgafqzpBPgIcCYyqOWYybybIHVNCGVS1/0LglLR+DnB21b49gD82+BlU4h9btW0xsF/V68uAf2jw/r2Bu2t/x+qUv0GdbUOqtv0I+APwKOkPspfuLe5iV4mI+4HfACfV7FoHmF+zbT5ZK6JiQZ0i/1q1/lKd1yMAJK0i6cepq/osWevj7Wr/auZPgQcj4jvp9bpkranHU1fwabLW5JpVn6c63trPVm08MD8iltbZV/u9zCdLjmtVbftL1fqLpM+8HPYlS2jzJf23pO0axLMgIl6vian659TXeNr9Ga4l6aLU/X8W+AUwpkXZUP/3ptp0sj/c50TE4jbKsw5ygnyrrwOfY9l/VI+RJZ1qE8j+qlesyLRI/0jW+to2IkYBH0zb1eqNkk4CNgIOr9q8gKwFOSYi3p6WURGxWdr/OFniq5jQpIoFwATVv4hQ+71MAJaybBJp1wtkpxgAkPSO6p0RMSsi9iJL8lcClzSIZ7yWvUhW+3PKy7+S/Q68O/0MP82yP79Gvx8Nf2/SH8jpZN3woyvnY617nCBrRMQ84GLguKrN1wAbSTownUDfD9iUrLXZCSPJWiNPS1qNLEm3JGn3FOc+EfFS1Wd4HLge+J6kUeliyoaSPpQOuQQ4TtI4Savy1hZztTvIEuq3Jb1N0jBJO6R9FwLHS1pf0giyJHFxg9ZmK/cCm0naUtIw4JSqzzlU2fjP0RHxKvAs8HqdMm4naxV+SdJKkiYDewIXLUc8fTUSeB54RtJY4Is1+/9Kdq62L/6ZLIF+luwi4nl96FVYBzhB1jeN7MQ5AKlr8zGylt5i4EvAxyJiUYfq+wHZecRFwG3AtW2+bz+y86UP6M0r2WelfYeQXaiYS3ZB6VfA2mnfT4DryJLSXWQXV+qKbEzenmQXIR4BFqZ6AX4GnE92SuDPZBcxPt9m7LX1/A/Z9/5fwJ+AW2oOORh4OHVfjwIOqlPGKynW3cm+yzOBQyLij8sTUx99A9iK7Bz21bz1Oz0VODmd8jixVWGStgZOIIv/NeA7ZMmy2R8z6zClE8FmZlbDLUgzswacIM2sX5D0s3TDwv0N9kvSD9OA+/skbdWqTCdIM+svzgF2a7J/d2BiWqYC/9mqQCdIM+sXIuJm4Kkmh+wFnBeZ28jGGq/d5HhKdYO8VloltPLoQmPYYqOxrQ/K2aCWox/Numv+/IdZtGhRR38zB49aN2LpS60PTOKlJ+eQjZSomB4R0/tQ5ViWHZi/MG17vNEbypUgVx7Nylse3vrAHN14/TcKrR9g+FAPdbNy2WHbSR0vM5a+xMobf6rt41++54yXI6LzgTRRqgRpZgOJoLszAz7KsneQjaPFXVY+B2lmxRAgtb+suBnAIelq9vuBZ9JdZw25BWlmxelgC1LShWSzO41Jc4R+nWzSFiLiLLJbhvcgmxLvReAzrcp0gjSzgggGde58e0Qc0GJ/AMf0pUwnSDMrTme6zrlxgjSzYohuX6TpMydIMytIxy6+5MYJ0syK4xakmVkDbkGamdXT9YHifeYEaWbFqAwULzEnSDMrjluQZmb1CAaXe2IWJ0gzK0YPjIPMNTpJV0q6U9IcSVPzrMvMelB3J6vos7xbkJ+NiKckDQdmSbosPUL1DSlxZslz5VE5h2Nm5eGr2MdJ2ietjyd7FsQyCTLNCDwdYNCItf0MWrOBZKBexZY0GdgF2C4iXpQ0ExiWV31m1oMGcAtyNLAkJcd3Ae/PsS4z6zUFnltsV54J8lrgKEkPAA8Ct+VYl5n1ooHagoyIv5E9h9bMrL4B3II0M2vCV7HNzOoTHX3kQh6cIM2sIG5Bmpk15nOQZmYNuAVpZtaAW5BmZnXI5yDNzBpzC9LMrD45QZqZvVX2SBonyLZtufFYbr5hWqExbHz8VYXWD3D1SbsUHQLvWmdk0SFYfyehQU6QZmZ1uQVpZtaAE6SZWQNOkGZm9SgtJeYEaWaFEHIL0sysESdIM7MGnCDNzBpwgjQzq8cXaczM6hNi0KByz+ZT7ujMrF+T1PbSRlm7SXpQ0jxJJ9XZP0HSTZLulnSfpD1alekEaWbFUR+WZsVIg4EzyB41vSlwgKRNaw47GbgkIt4L7A+c2So8J0gzK4Y62oLcBpgXEQ9FxCvARcBeNccEMCqtjwYea1VobglS0nqS7s+rfDPrfX1MkGMkza5aplYVNRZYUPV6YdpW7RTg05IWAtcAn28Vny/SmFlh+jjMZ1FETFqB6g4AzomI70naDjhf0uYR8XqjN+TdxR4i6QJJD0j6laRVcq7PzHpE5VbDDnWxHwXGV70el7ZVOxy4BCAifg8MA8Y0KzTvBLkxcGZEbAI8Cxxde4CkqZUm86Inn8w5HDMrlQ5dpAFmARMlrS9pKNlFmBk1xzwC7AwgaROyBNk06eSdIBdExK1p/RfAB2oPiIjpETEpIiaNWWONnMMxs9Lo4EWaiFgKHAtcBzxAdrV6jqRpkqakw/4R+Jyke4ELgcMiIpqVm/c5yNrKmwZjZgNLJ281jIhryC6+VG/7WtX6XGCHvpSZdwtyQjoZCnAgcEvO9ZlZD9Egtb0UIe8E+SBwjKQHgFWB/8y5PjPrIZ28kyYPuXWxI+Jh4F15lW9mva3IxNcuj4M0s8I4QZqZNeAEaWbWSLnzoxOkmRXHLUgzs3rkBGlmVpeAkudHJ0gzK4oYVNAA8HY5QZpZYdzFNjOrR+5im5nVJXAXuy8EDBlc7GNyfnrMW2Zk67pDzr696BC4/as7Fx1C6btftuLK/iMuVYI0s4Gl7H8EnSDNrBg+B2lmVl82DrLcGdIJ0swK4unOzMwaKnl+dII0s4LIw3zMzOryOUgzsyZKnh+dIM2sOG5Bmpk1UPL86ARpZgXxhLlmZvV5wlwzs4Y8UNzMrKGS50cnSDMrSA8MFM998kVJh0i6T9K9ks7Puz4z6w2VgeLtLkXItQUpaTPgZGD7iFgkabU6x0wFpgKMnzAhz3DMrGTKfg4y7xbkTsClEbEIICKeqj0gIqZHxKSImLTGmDVyDsfMykRqfymCz0GaWWEGegvyRuCTklYHqNfFNrMBqg+tx37ZgoyIOZK+Bfy3pNeAu4HD8qzTzHqDPA4SIuJc4Ny86zGz3lPy/OhzkGZWnEElz5DFPoTazAa0Tp6DlLSbpAclzZN0UoNjPiVprqQ5kn7Zqky3IM2sEBIM7tCdNJIGA2cAHwEWArMkzYiIuVXHTAT+CdghIpZIWrNVuW5BmllhOngnzTbAvIh4KCJeAS4C9qo55nPAGRGxBCAinmhVqBOkmRWmj13sMZJmVy1Tq4oaCyyoer0wbau2EbCRpFsl3SZpt1bxNexiS/oREI32R8RxrQo3M2tEZEN9+mBRRExagSqHABOBycA44GZJ746Ip5u9oZHZKxCImVlLHZzM51FgfNXrcWlbtYXA7RHxKvBnSf9DljBnNSq0YYJM4xffIGmViHixr1GbmdXV2Vl6ZgETJa1Plhj3Bw6sOeZK4ADg55LGkHW5H2pWaMtzkJK2kzQX+GN6vYWkM/sev5nZsjo1zCcilgLHAtcBDwCXpDv5pkmakg67Dlic8tlNwBcjYnGzctsZ5vMD4KPAjBTIvZI+2Mb7zMwaEp0dKB4R1wDX1Gz7WtV6ACekpS1tjYOMiAU1TeHX2q2g10zeuPgp11577fWiQ+CWeYuKDoEdJxb/s7B8lfxGmrYS5AJJ2wMhaSXgC2RNWDOzFdIfJqs4CvgPsjFFj5H144/JMygz6/86eSdNXlomyDQb+EFdiMXMBphyp8f2rmJvIOnXkp6U9ISkqyRt0I3gzKx/K/tDu9q51fCXwCXA2sA6wKXAhXkGZWb9X3YVu/2lCO0kyFUi4vyIWJqWXwDD8g7MzPq5PrQeS/fY16rnx/w2za12Edm92ftRM9bIzGx5lPwidtOLNHeSJcTKRziyal+QzatmZrbcenaYT0Ss381AzGxgqZyDLLO27qSRtDmwKVXnHiPivLyCMrOBoWdbkBWSvk42f9qmZOcedwduAZwgzWy5STC45AmynavYnwB2Bv4SEZ8BtgBG5xqVmQ0InXxoVx7a6WK/FBGvS1oqaRTwBMtOTGlmtlx6vosNzJb0duAnZFe2nwd+39eKJJ0CPB8R3+3re82sfyp5fmzrXuyj0+pZkq4FRkXEffmGZWb9nVBH54PMQ7OB4ls12xcRd7UqXNJXgEPJuuULyFqgZmZQ4LnFdjVrQX6vyb4AdmpWsKStyZ4LsWWq5y7qJMj06MapAOMnTGgRrpn1Jz17DjIiPryCZe8IXFF50JekGQ3qmQ5MB9h660kNHzNrZv1PO8NoitTWQHEzs04T5W9B5pnAbwb2ljRc0khgzxzrMrMeVPbpznJrQUbEXZIuBu4lu0jT8OHcZjbw9ItHLihrAx8EbBAR0yRNAN4REXe0em9EfAv41oqHaWb9UcnzY1td7DOB7YAD0uvngDNyi8jMBoz+cKvhthGxlaS7ASJiiaShOcdlZv1cNt1ZuZuQ7STIVyUNJhv7iKQ1gOKfbG9mPa/sw3zaie+HwBXAmpK+RTbV2b/mGpWZDQg938WOiAsk3Uk25ZmAvSPigdwjM7N+Terhe7Er0lXrF4FfV2+LiEfyDMzM+r+S58e2zkFezZsP7xoGrA88CGyWY1xmNgCUfZhPO13sd1e/TrP8HN3gcDOztoh+MFC8VrpDZts8gjGzAaTAWwjb1c45yBOqXg4CtgIeyy0iMxswRLkzZDstyJFV60vJzklelk84ZjZQ9PxzsdMA8ZERcWKX4incqu87tugQWDLr9KJDMOuKnk2QkoZExFJJO3QzIDMbOMo+H2SzFuQdZOcb70mzgV8KvFDZGRGX5xybmfVjvdDFbudWw2HAYrJn0HyMbOLbj+UZlJkNAH24zbCdhqak3SQ9KGmepJOaHLevpJA0qVWZzVqQa6Yr2Pfz5kDxCj87xsxWWKduNUzXS84APgIsBGZJmhERc2uOGwl8Abi9rfia7BsMjEjLyKr1ymJmttwqXewOPXJhG2BeRDwUEa8AFwF71Tnum8B3gJfbibFZC/LxiJjWTiFmZn0nBvetBTlG0uyq19PTU1EBxgILqvYtBJa5oSXdBTg+Iq6W9MV2KmyWIEt++tTMeln2VMM+vWVRRLQ8b1i3LmkQ8O/AYX15X7MEufPyBGJm1pbO3mr4KDC+6vW4tK1iJLA5MDMNLXoHMEPSlIiobpUuo2GCjIinVihcM7MWOjgf5CxgoqT1yRLj/sCBlZ0R8QwwpvJa0kzgxGbJEco/47mZ9VOVLnYnhvlExFLgWOA64AHgkoiYI2mapCnLG2Nuz8U2M2ulkzOKR8Q1wDU1277W4NjJ7ZTpBGlmhSn5nYZOkGZWDFH+c3xOkGZWDJV/sopcE7ikT0u6Q9I9kn6cbgcyMwPShZo2lyLkliAlbQLsB+wQEVsCrwEH1TluqqTZkmY/uejJvMIxs5IRMFhqeylCnl3snYGtyW4aBxgOPFF7ULpVaDrA1ltP8iQYZgNIyXvYuSZIAedGxD/lWIeZ9SwN6HOQNwCfkLQmgKTVJK2bY31m1kMqV7HbXYqQWwsyIuZKOhm4Pt0o/ipwDDA/rzrNrLeUvQWZ6zCfiLgYuDjPOsysd5U7PXocpJkVpQfGQTpBmlkhfCeNmVkTbkGamTVQ9se+OkGaWSGyLna5M6QTpJkVpuQ9bCdIMyuKkFuQZmb1uQVpZlaHz0GamTXSxsO4iuYEaWaFcYLsg/9d/AL7/XxWoTGc+h8nFFq/2UDiizRmZnUIDxQ3M2uok8/FzoMTpJkVxl1sM7M63MU2M2vId9KYmdXncZBmZo2VPD86QZpZMbJzkOVOkU6QZlaYcqdHJ0gzK1LJM6QTpJkVxl1sM7MGyp0eu/TURUnHSXpA0gXdqM/MeoT6sBSgWy3Io4FdImJhl+ozs5LL8l6525C5tyAlnQVsAPxW0vF512dmPSINFG93KULuLciIOErSbsCHI2JR3vWZWe8od/uxS+cgm5E0VdJsSbNfee7posMxs27q4DlISbtJelDSPEkn1dl/gqS5ku6TdIOkdVuVWXiCjIjpETEpIiYNHfn2osMxs65Rn/5rWpI0GDgD2B3YFDhA0qY1h90NTIqI9wC/Av6tVYSFJ0gzG7g6eA5yG2BeRDwUEa8AFwF7VR8QETdFxIvp5W3AuFaFOkGaWSH60rtO+XFM5XRcWqZWFTcWWFD1emHa1sjhwG9bxdiVYT4RsV436jGz3qK+XZ5eFBGTOlDnp4FJwIdaHes7acysMB0cvvMoML7q9bi0raY+7QJ8BfhQRPytVaHuYptZYTp4EXsWMFHS+pKGAvsDM5apS3ov8GNgSkQ80U58TpBmVozlOAnZSEQsBY4FrgMeAC6JiDmSpkmakg47DRgBXCrpHkkzGhT3BnexzawwnbzVMCKuAa6p2fa1qvVd+lqmE6SZFUL4mTRmZg2VPD86QZpZgUqeIZ0gzawwZZ/uzAnSzAozqNz50QnSzArkBGlm9la9MKN4qRLks08s5vozzy00hotnnV5o/WYDRoEzhberVAnSzAaWkudHJ0gzK1DJM6QTpJkVpPVM4UVzgjSzwvgcpJlZHW1OY1YoJ0gzK07JM6QTpJkVZlDJ+9hOkGZWmHKnRydIMyuKB4qbmTVT7gzpBGlmhfCM4mZmTZQ8PzpBmllx3II0M2tgQN9qKOmrwKeBJ4EFwJ0R8d086zSzHlLu/JhfgpT0PmBfYAtgJeAu4M686jOz3lPy/JhrC3IH4KqIeBl4WdKv6x0kaSowFYCVRuQYjpmViVT+O2kGFR1AREyPiEkRMUlDhhcdjpl1k/qwFCDPBHkrsKekYZJGAB/LsS4z60Elz4/5dbEjYpakGcB9wF+BPwDP5FWfmfWekvewc+9ifzciNgI+CqyLL9KY2RvUp/+KkPc4yOmSNgWGAedGxF0512dmPWLA32oYEQfmWb6ZWZ58J42ZFWZAtyDNzJoZ0Lcampk1kg0ULzqK5pwgzaw4TpBmZvW5i21m1kDZL9IUfi+2mQ1cnbzVUNJukh6UNE/SSXX2ryzp4rT/dknrtSrTCdLMitOhDClpMHAGsDuwKXBAukml2uHAkoh4J/B94DutwnOCNLPCdPBWw22AeRHxUES8AlwE7FVzzF7AuWn9V8DOUvNOfqnOQcZLTy56+Z4z5q9AEWOARSsSw/CVzliRt3ckhg5wDG8qQxz9IYZ1OxVIxd133XndKkM1pg9vGSZpdtXr6RExPa2PJXtqQcVCYNua979xTEQslfQMsDpNvpdyJciINVbk/ZJmR8SkTsXjGHo7hrLE4Rjqi4jdio6hFXexzaw/eBQYX/V6XNpW9xhJQ4DRwOJmhTpBmll/MAuYKGl9SUOB/YEZNcfMAA5N658AboyIaFZoqbrYHTC99SG5cwyZMsQA5YjDMeQsnVM8FrgOGAz8LCLmSJoGzI6IGcBPgfMlzQOeIkuiTalFAjUzG7DcxTYza8AJ0iwHklYpOgZbcU6Q1lGS1mo1+La/kzQF+FG6WGA9zAmyAyQNSv8f6IlhLHAy2W1ehX4XRdUvaXXgOLLb2MZJWq2IOKwz+k2ClDS8qLoj4vW0uq6kIUUmB0mbFFU38BjZkyvfC3y84CS5Drwx3q2bXgGWAl8H/h14vfnh+ZP0jqL/YPWqfpEg0+X9f5N0qqTRXax3e0n7p/XPA5cDPwNOqrQqu0nS3wOnSVqrgLqVxpQNIpss4MvAXkX8w0y/D2dJ+jZwtKSVu1V3RDwH3Eh23+99EfF0wX8w3wNMA/Z1kuy7nh8HKelo4JPAgcBdwFhJ34yIP3Wh+lWBUyVtBmyY4lgXmAx8W9JJVa3LXKXzXkcBUyLir92os1pEhKSDgM8DhwGfAT4MDJF0WasBuZ0iaW/gU8AU4ApgTkT8rRt1V7mYrCV9uqQlEfH9LtcPgKQ9gRPJ/p2vl7Z17WfRH/R0C1LSKGArsgGf+wJ3p10/lDQx7/oj4mpgKrBP9jL+F7gFOB9Yi6wl1S3rABdHxHxJK3Wx3mobA7+MiHuBLwHzgGOBT3ax9TIa+AGwN/AqcAKApI26VD8RMT8ibiD7o/33qUXbVakX8WXgyIjYgax3M5mCWvW9qqcTZEQ8CxwDrAnsk25+PxR4H3BwN64iRsT/A74C7CFpv4h4JbVeRwLdPB84H/igpI0j4lUASQenFlW33AXsIGmz9D38CBgGbA2M6FIMDwOnAYdHxK4R8Yqk44Ajuv2HIyLuJrul7RRJR3azbrJzoYPIZquB7C6SIcDxwEe7HEvP6vkudkT8TdKLZF25d5N1cW8Azk7zwnUjhqskHUzWct0EuAeYSJYwuuVWYHvgMEm3kiXo44ADuhjDTLI/TgdKuhEYDjwP/DCdm+uGO4GrgNclTQYmkP3RPLTyh6ObIuK+FMdLXa53iaTLgJ0kPRMR90u6gmwSh/0l3VTAqYee0y9uNUwn4f8B2IWsq/nJiJhbQBx7A5cBvwGOj4iHulz/2mQXB6YAzwCnRsR9XY5hHeDjaVkKnFhADGuTfQdTyGZrOS0i/tDNGMpA0jjgSGAS2R/rTwAHkw3F+mo6FWJN9IsECZC6T+8AXo+I2mmOuhnHh4D5EfFwgTEMBehWC7pBDG8j+/16vsAYVgIoouVYFuk8/fbAFsA1wCrAT4CPFHExr9f0mwRpZs1J+jBwKtmFG7ce2+AEaTZApFMPQyNiRR5rMqA4QZqZNdDTw3zMzPLkBGlm1oATpJlZA06Q/YSk1yTdI+l+SZeuyIStks6R9Im0frakhrdMSposafvlqONh6a3PRG60veaYPg0dknSKpBP7GqOZE2T/8VJEbBkRm5PdZnZU9c7lnfYrIo5oMeh+Mtk4O7N+xwmyf/od8M7UuvudpBnAXEmDJZ0maZak+yr3BytzuqQHJf0X2b3tpH0zJU1K67tJukvSvZJukLQeWSI+PrVed5S0hqTLUh2zJO2Q3ru6pOslzZF0NtBywgRJV0q6M71nas2+76ftN0haI23bUNK16T2/k/SuTnyZNnD1/L3YtqzUUtwduDZt2grYPCL+nJLMMxHxvnR75q2Srieb4HZjstmH1gLmks1rWV3uGmR3YHwwlbVaRDwl6Szg+Yj4bjrul8D3I+IWSRPIHsO5CdkEsrdExDRJfwcc3sbH+WyqYzgwK03VtRh4G9mjPI+X9LVU9rFkjzY9KiL+JGlb4Exgp+X4Gs0AJ8j+ZLike9L678hmb9keuCMi/py27wq8p3J+kWxqsInAB4ELI+I14LE00USt9wM3V8qKiKcaxLELsGnVjFqjJI1IdXw8vfdqSUva+EzHSdonrY9PsS4mm6X74rT9F8DlqY7tgUur6u7aRLnWPzlB9h8vRcSW1RtSonihehPw+Yi4rua4PToYxyDg/RHxcp1Y2pZmwNkF2C4iXpQ0k2zqtHoqM5k/XfsdmK0In4McWK4jm8B1JcgmkU2TStwM7JfOUa5NNhN4rdvI5ptcP7238jCq58imVqu4nmxWcdJxlYR1M9kEskjanWw29mZGA0tScnwXWQu2YhDZzDSkMm9Jc4P+WdInUx2StEWLOsyacoIcWM4mO794l6T7gR+T9SKuAP6U9p0H/L72jRHxJNns6fW9QYMAAAB4SURBVJdLupc3u7i/BvapXKQhm4NyUroINJc3r6Z/gyzBziHraj/SItZryeb4fAD4NlmCrngB2CZ9hp3InrkCcBBweIpvDtnUb2bLzfdim5k14BakmVkDTpBmZg04QZqZNeAEaWbWgBOkmVkDTpBmZg04QZqZNfB/0P4yWIFH4vcAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import numpy as np\n",
        "from sklearn import svm, datasets\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import confusion_matrix\n",
        "\n",
        "# Compute confusion matrix\n",
        "cm = confusion_matrix(y_compare, pred)\n",
        "np.set_printoptions(precision=2)\n",
        "\n",
        "# Normalize the confusion matrix by row (i.e by the number of samples\n",
        "# in each class)\n",
        "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "print('Normalized confusion matrix')\n",
        "print(cm_normalized)\n",
        "plt.figure()\n",
        "plot_confusion_matrix(cm_normalized, products, \n",
        "        title='Normalized confusion matrix')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "G0KvHKadK1AT"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3.9 (tensorflow)",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "name": "t81_558_class_04_2_multi_class.ipynb",
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}