cassava_disease_classification_inceptionv3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cassava_disease_classification_inceptionv3.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jzlNeBuClfh",
        "colab_type": "text"
      },
      "source": [
        "# Cassava Disease Classification\n",
        "\n",
        "Cassava is a plant that is an essential source of nutrition to many people in the world especially in West Africa and South America. Unfortunately, this plant is vulnarable against a number of viral infections. Thus it is important for us to be able to identify these diseases through images taken from the leaves of the plants. \n",
        "\n",
        "In this project we us Convolutional Neural Networks to achieve this task. In particular, we do so by transfer learning. We use the Inception V3 model to extract the features from the images and then we use two fully connected layers on top of the convolutional layers in order to classify the images into different types of diseases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xfK7DKQX_J3P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import zipfile\n",
        "import matplotlib\n",
        "from imutils import paths\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import cv2\n",
        "import sys\n",
        "import matplotlib.image as mpimg\n",
        "from google.colab.patches import cv2_imshow\n",
        "from glob import glob \n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.colors import LogNorm\n",
        "#Keras\n",
        "from keras.utils.np_utils import to_categorical # convert to one-hot-encoding\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "from keras import layers\n",
        "from keras import Model\n",
        "from keras.applications.inception_v3 import InceptionV3, preprocess_input\n",
        "from keras.optimizers import Adam\n",
        "from keras.callbacks import ReduceLROnPlateau, EarlyStopping, LearningRateScheduler\n",
        "from keras import backend as K\n",
        "#sklearn helpers\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "from sklearn import preprocessing"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bp3qC99sC1hb",
        "colab_type": "text"
      },
      "source": [
        "# Data Preperation\n",
        "\n",
        "Now we download the publicly available dataset containing the images of *Cassava leaves*. These images belong to five category: one healthy category, and 4 category corresponding to four different types of diseases. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KzLV3N-S_2ao",
        "colab_type": "code",
        "outputId": "db308509-04bc-431c-ee99-1937d9c2b696",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "#Download the file and save it in a temporary placeholder '/tmp'\n",
        "!wget --no-check-certificate \\\n",
        "  https://storage.googleapis.com/emcassavadata/cassavaleafdata.zip \\\n",
        "  -O /tmp/cassavaleafdata.zip\n",
        "\n",
        "#Unzip the file in '/tmp'\n",
        "local_zip = '/tmp/cassavaleafdata.zip'\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\n",
        "zip_ref.extractall('/tmp')\n",
        "zip_ref.close()\n",
        "\n",
        "#Split the data into train, valid, test folders\n",
        "base_dir = '/tmp/cassavaleafdata'\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "valid_dir = os.path.join(base_dir, 'validation')\n",
        "test_dir = os.path.join(base_dir, 'test')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-01-27 17:38:07--  https://storage.googleapis.com/emcassavadata/cassavaleafdata.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.195.128, 2607:f8b0:400e:c07::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.195.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1354096203 (1.3G) [application/zip]\n",
            "Saving to: ‘/tmp/cassavaleafdata.zip’\n",
            "\n",
            "/tmp/cassavaleafdat 100%[===================>]   1.26G  78.7MB/s    in 16s     \n",
            "\n",
            "2020-01-27 17:38:24 (81.3 MB/s) - ‘/tmp/cassavaleafdata.zip’ saved [1354096203/1354096203]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGOYIuc8LkUv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess(directory):\n",
        "  '''\n",
        "  Return the standardized format of all the \n",
        "  images in a given directory (ex: 'train', 'test', 'valid')\n",
        "  '''\n",
        "  image_files = list(paths.list_images(directory))\n",
        "  #data and label placeholders\n",
        "  arrays, labels = [], []\n",
        "  cnt = 0\n",
        "  for img in image_files:\n",
        "    #find the label of the file\n",
        "    label = img.split('/')[-2]\n",
        "    # standardize the images: convert to RGB & resize to (224, 224)\n",
        "    image = cv2.imread(img)\n",
        "    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
        "    image = cv2.resize(image, (224, 224))\n",
        "    #add the image and label to the place holders\n",
        "    arrays.append(image)\n",
        "    labels.append(label)\n",
        "    cnt += 1\n",
        "  #convert these to numpy arrays  \n",
        "  arrays = np.array(arrays, dtype=\"float32\")\n",
        "  labels = np.array(labels)\n",
        " \n",
        "  return arrays, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1-cFOjPLLkXS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#a placeholder for training data and the training labels\n",
        "train_data, train_labels = preprocess(train_dir)\n",
        "#a placeholder for validation (dev) data and the validation (dev) labels\n",
        "valid_data, valid_labels = preprocess(valid_dir)\n",
        "#a placeholder for test data and the test labels\n",
        "test_data, test_labels = preprocess(test_dir)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iz6A4sqaZo_h",
        "colab_type": "text"
      },
      "source": [
        "##One-hot encoding transformation of the labels:\n",
        "\n",
        "Now that we have a placeholder for the labels in the training, dev, and the test set, we perform one-hot encoding to transform the labels (strings) to one-hot vectors. The length of each vector will be equal to the number of classes in the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MrcqWIAFz3hH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Binarizer = LabelBinarizer()\n",
        "train_y = Binarizer.fit_transform(train_labels)\n",
        "valid_y = Binarizer.transform(valid_labels)\n",
        "test_y = Binarizer.transform(test_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOY1cyec6s8a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#shuffling the training data\n",
        "train_indx = np.arange(len(train_labels))\n",
        "np.random.shuffle(train_indx)\n",
        "train_data, train_y = train_data[train_indx], train_y[train_indx]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ayy5AoaywO5F",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#rescaling the images such that the pixel values are between one and zero\n",
        "train_data = train_data / 255.\n",
        "valid_data = valid_data / 255.\n",
        "test_data = test_data / 255."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "62y1RIWy7VWj",
        "colab_type": "text"
      },
      "source": [
        "Let's take a look at the distribution of classes. We note that the training, the dev, and the test sets are drawn from the same distribution which makes the model trained on training examples generalize better to the unseen examples in the dev and the test set. We also note that the data is very imbalanced. Therefore, we need to inspect the confusion matrix later on to see how successful the model is at distinguishing between the 5 classes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oLojBxfd2PHN",
        "colab_type": "code",
        "outputId": "5c964d2c-3fb0-4784-eff4-7cf5af84a8e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "dist1 = plt.hist(train_y.argmax(axis = 1), density = True)\n",
        "dist2 = plt.hist(valid_y.argmax(axis = 1), density = True)\n",
        "dist3 = plt.hist(test_y.argmax(axis = 1), density = True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAO40lEQVR4nO3df6zdd13H8eeLdhMTkCX0Gsna0RlL\nYqUg86bMLJEpkHTTtH9ATBsHGxk0UacYiDrUDBx/oQkadDgrLAyQjTkJuWKXhcgMibi5O36sa+fI\ndaDrJOll4JCgzOrbP86pHs7uved72+895+6z5yO5yfl+v5+cz2uf9vvq937Pj6WqkCQ98z1n1gEk\nSf2w0CWpERa6JDXCQpekRljoktSIrbOaeNu2bbVz585ZTS9Jz0gPPPDA16tqbqVjMyv0nTt3sri4\nOKvpJekZKck/r3bMWy6S1IiJhZ7kliSnkjy0yvFfSPJgkmNJPpfk5f3HlCRN0uUK/UPAvjWOfwV4\nVVXtAd4NHOkhlyRpnSbeQ6+qzybZucbxz41s3gtsP/dYkqT16vse+rXAXasdTHI4yWKSxeXl5Z6n\nlqRnt94KPclPMyj031xtTFUdqar5qpqfm1vxXTeSpLPUy9sWk7wM+ABwRVU90cdzSpLW55yv0JNc\nBHwCeENVffncI0mSzsbEK/QktwGXA9uSnATeCZwHUFU3AzcALwTenwTgdFXNb1RgSdLKurzL5dCE\n428G3txbIklTt+fWPTOb+9jVx2Y2d2v8pKgkNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w\n0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtd\nkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGTCz0JLckOZXkoVWOJ8n7kiwleTDJJf3HlCRN\n0uUK/UPAvjWOXwHsGv4cBv7k3GNJktZrYqFX1WeBb6wx5ADw4Rq4F7ggyYv6CihJ6qaPe+gXAo+N\nbJ8c7pMkTdFUXxRNcjjJYpLF5eXlaU4tSc3ro9AfB3aMbG8f7nuaqjpSVfNVNT83N9fD1JKkM/oo\n9AXgjcN3u1wKPFlVX+vheSVJ67B10oAktwGXA9uSnATeCZwHUFU3A0eBK4El4DvAmzYqrCRpdRML\nvaoOTThewC/3lkiSdFb8pKgkNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXC\nQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0\nSWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1IhOhZ5kX5JHkiwluX6F4xcluSfJF5I8mOTK/qNKktYy\nsdCTbAFuAq4AdgOHkuweG/Y7wB1V9QrgIPD+voNKktbW5Qp9L7BUVY9W1VPA7cCBsTEF/MDw8QuA\nf+0voiSpiy6FfiHw2Mj2yeG+Ue8CrkpyEjgK/MpKT5TkcJLFJIvLy8tnEVeStJq+XhQ9BHyoqrYD\nVwIfSfK0566qI1U1X1Xzc3NzPU0tSYJuhf44sGNke/tw36hrgTsAqurvgecC2/oIKEnqpkuh3w/s\nSnJxkvMZvOi5MDbmX4BXAyT5UQaF7j0VSZqiiYVeVaeB64C7gYcZvJvleJIbk+wfDns78JYkXwJu\nA66pqtqo0JKkp9vaZVBVHWXwYufovhtGHp8ALus3miRpPfykqCQ1wkKXpEZY6JLUCAtdkhphoUtS\nIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXC\nQpekRljoktQIC12SGmGhS1IjLHRJasTWWQdQN3tu3TOzuY9dfWxmc0vqzit0SWqEhS5JjbDQJakR\nnQo9yb4kjyRZSnL9KmN+PsmJJMeTfKzfmJKkSSa+KJpkC3AT8FrgJHB/koWqOjEyZhfwDuCyqvpm\nkh/cqMCSpJV1uULfCyxV1aNV9RRwO3BgbMxbgJuq6psAVXWq35iSpEm6FPqFwGMj2yeH+0a9BHhJ\nkr9Lcm+SfSs9UZLDSRaTLC4vL59dYknSivp6UXQrsAu4HDgE/FmSC8YHVdWRqpqvqvm5ubmeppYk\nQbdCfxzYMbK9fbhv1Elgoar+q6q+AnyZQcFLkqakS6HfD+xKcnGS84GDwMLYmE8yuDonyTYGt2Ae\n7TGnJGmCiYVeVaeB64C7gYeBO6rqeJIbk+wfDrsbeCLJCeAe4Ner6omNCi1JerpO3+VSVUeBo2P7\nbhh5XMDbhj+SpBnwk6KS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12S\nGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakR\nFrokNcJCl6RGWOiS1AgLXZIa0anQk+xL8kiSpSTXrzHudUkqyXx/ESVJXUws9CRbgJuAK4DdwKEk\nu1cY93zgrcB9fYeUJE3W5Qp9L7BUVY9W1VPA7cCBFca9G3gP8J895pMkddSl0C8EHhvZPjnc93+S\nXALsqKq/XuuJkhxOsphkcXl5ed1hJUmrO+cXRZM8B3gv8PZJY6vqSFXNV9X83NzcuU4tSRrRpdAf\nB3aMbG8f7jvj+cBLgb9N8lXgUmDBF0Ylabq6FPr9wK4kFyc5HzgILJw5WFVPVtW2qtpZVTuBe4H9\nVbW4IYklSSuaWOhVdRq4DrgbeBi4o6qOJ7kxyf6NDihJ6mZrl0FVdRQ4OrbvhlXGXn7usSRJ6+Un\nRSWpERa6JDXCQpekRljoktSITi+KSs82e27dM5N5j119bCbzqg1eoUtSIyx0SWqEhS5JjbDQJakR\nFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGh\nS1IjLHRJaoSFLkmNsNAlqREWuiQ1olOhJ9mX5JEkS0muX+H425KcSPJgkr9J8uL+o0qS1jKx0JNs\nAW4CrgB2A4eS7B4b9gVgvqpeBtwJ/F7fQSVJa9vaYcxeYKmqHgVIcjtwADhxZkBV3TMy/l7gqj5D\nPs27XrChT7/23E/Obm5JWkOXQr8QeGxk+yTwyjXGXwvcdS6hJtlz8UUb+fRrOjazmSVpbV0KvbMk\nVwHzwKtWOX4YOAxw0UWzK2VJalGXF0UfB3aMbG8f7vseSV4D/Dawv6q+u9ITVdWRqpqvqvm5ubmz\nyStJWkWXQr8f2JXk4iTnAweBhdEBSV4B/CmDMj/Vf0xJ0iQTC72qTgPXAXcDDwN3VNXxJDcm2T8c\n9vvA84C/SPLFJAurPJ0kaYN0uodeVUeBo2P7bhh5/Jqec0mS1slPikpSIyx0SWqEhS5JjbDQJakR\nFrokNaLXT4pK0jPFnlv3zGzuY1dvzJeIeIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1Ij\nLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJC\nl6RGWOiS1IhOhZ5kX5JHkiwluX6F49+X5OPD4/cl2dl3UEnS2iYWepItwE3AFcBu4FCS3WPDrgW+\nWVU/AvwB8J6+g0qS1tblCn0vsFRVj1bVU8DtwIGxMQeAW4eP7wRenST9xZQkTbK1w5gLgcdGtk8C\nr1xtTFWdTvIk8ELg66ODkhwGDg83v53kkbMJDWwbf+5pyTVr/js1s1wdnHW2Cf/N52qzrtlMcnVY\n6+bW69n49yvX5FxyvXi1A10KvTdVdQQ4cq7Pk2SxquZ7iNSrzZoLNm82c62Pudbn2Zaryy2Xx4Ed\nI9vbh/tWHJNkK/AC4Ik+AkqSuulS6PcDu5JcnOR84CCwMDZmAbh6+Pj1wGeqqvqLKUmaZOItl+E9\n8euAu4EtwC1VdTzJjcBiVS0AHwQ+kmQJ+AaD0t9I53zbZoNs1lywebOZa33MtT7PqlzxQlqS2uAn\nRSWpERa6JDViUxf6Zv3KgQ65rkmynOSLw583TynXLUlOJXloleNJ8r5h7geTXLJJcl2e5MmR9bph\nCpl2JLknyYkkx5O8dYUxU1+vjrmmvl7DeZ+b5B+SfGmY7XdXGDP1c7Jjrlmdk1uSfCHJp1Y41v9a\nVdWm/GHwAuw/AT8MnA98Cdg9NuaXgJuHjw8CH98kua4B/ngGa/ZTwCXAQ6scvxK4CwhwKXDfJsl1\nOfCpKa/Vi4BLho+fD3x5hT/Hqa9Xx1xTX6/hvAGeN3x8HnAfcOnYmFmck11yzeqcfBvwsZX+vDZi\nrTbzFfpm/cqBLrlmoqo+y+BdRqs5AHy4Bu4FLkjyok2Qa+qq6mtV9fnh438HHmbwiedRU1+vjrlm\nYrgO3x5unjf8GX9XxdTPyY65pi7JduBngQ+sMqT3tdrMhb7SVw6M/8X+nq8cAM585cCscwG8bvhr\n+p1JdqxwfBa6Zp+Fnxz+ynxXkh+b5sTDX3VfweDKbtRM12uNXDCj9RreQvgicAr4dFWtumZTPCe7\n5ILpn5N/CPwG8D+rHO99rTZzoT+T/RWws6peBnya//9XWCv7PPDiqno58EfAJ6c1cZLnAX8J/FpV\nfWta804yIdfM1quq/ruqfpzBJ8b3JnnptOZeS4dcUz0nk/wccKqqHtjIecZt5kLfrF85MDFXVT1R\nVd8dbn4A+IkNztRVlzWduqr61plfmavqKHBekm0bPW+S8xiU5p9X1SdWGDKT9ZqUa1brNZbh34B7\ngH1jh2b6NSCr5ZrBOXkZsD/JVxnclv2ZJB8dG9P7Wm3mQt+sXzkwMdfYfdb9DO6DbgYLwBuH7964\nFHiyqr4261BJfujMvcMkexn8vdzQEhjO90Hg4ap67yrDpr5eXXLNYr2Gc80luWD4+PuB1wL/ODZs\n6udkl1zTPier6h1Vtb2qdjLoiM9U1VVjw3pfq6l+2+J61Ob8yoGuuX41yX7g9DDXNRudCyDJbQze\nAbEtyUngnQxeIKKqbgaOMnjnxhLwHeBNmyTX64FfTHIa+A/g4BT+Yb4MeANwbHjvFeC3gItGcs1i\nvbrkmsV6weAdOLdm8D+9eQ5wR1V9atbnZMdcMzknx230WvnRf0lqxGa+5SJJWgcLXZIaYaFLUiMs\ndElqhIUuSY2w0CWpERa6JDXifwGUzSeWwwXhrAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3evSEcxsHPTc",
        "colab_type": "text"
      },
      "source": [
        "#Image augmentation\n",
        "\n",
        "For training our model, we have a dataset with a limited size. \n",
        "One way to alleviate this problem  is to use Image Augmentation.\n",
        "In image augmentation, a set of transformations are applied to the training examples *on the fly* during the training process. For instance, the images are \n",
        "*rotated*, *shifted*, *flipped*, *distorted* (with a prespecified range of a shear parameter). Note that these transformations are performed on the fly: these transformations are done on every example in a given batch during the gradient descent, and in the next batch iteration, a new set of transformations are done on a different set of examples in a different batch. This technique ensures that the network never looks at the same example in the original training data twice and therefore it helps avoid overfitting. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gyPDOdONLkcn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_augment = ImageDataGenerator(\n",
        "\trotation_range=30,\n",
        "\tzoom_range=0.15,\n",
        "\twidth_shift_range=0.2,\n",
        "\theight_shift_range=0.2,\n",
        "\tshear_range=0.15,\n",
        "\thorizontal_flip=True,\n",
        "\tfill_mode=\"nearest\")\n",
        "\n",
        "valid_augment = ImageDataGenerator()\n",
        "\n",
        "train_augment.fit(train_data)\n",
        "valid_augment.fit(valid_data)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AxCNd1TVkOb2",
        "colab_type": "text"
      },
      "source": [
        "#Inception V3 fine tuning\n",
        "\n",
        "We use transfer learning to the classify teh images in the data. \n",
        "In particular we use the Inception V3 model without the top dense layers. That is, we only keep the convolutional layers. We flatten the last convolutional activation layer of Inception V3 using the Global average pooling. The output is then feeded to a dense layer with 512 hidden units, followed by a dropout with dropout rate of 0.5, followed by a softmax activation layer for the purpose of multi-class classification.\n",
        "\n",
        "We train the model in two steps: \n",
        "\n",
        "1.   Step one: We keep the convolutional layers of the Inception model frozen, that is we don't let the inception weights vary. So the only trainable parameters are the parameters of the weight matrices of the top dense layer. \n",
        "\n",
        "\n",
        "2.  Once, the parameters of the top dense layers are tuned, we unfreeze the parameters of the convolutional layer and then we retrain all the parameters including the Inception weights and biases. In our experiments we have noticed that retraining the Inception V3 convolutional layers achieves better results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGgWH3jmKEaz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "baseModel = InceptionV3(weights=\"imagenet\", include_top=False, input_shape = (224, 224, 3))\n",
        "#Freeze the Inception V3 layers:\n",
        "for layer in baseModel.layers:\n",
        "\tlayer.trainable = False\n",
        "\n",
        "x = baseModel.output\n",
        "x = layers.GlobalMaxPooling2D()(x)\n",
        "x = layers.Dense(512, activation=\"relu\")(x)\n",
        "x = layers.Dropout(0.5)(x)\n",
        "x = layers.Dense(5, activation=\"softmax\")(x)\n",
        "\n",
        "# Configure and compile the model\n",
        "model = Model(baseModel.input, x)\n",
        "optimizer = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=True)\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer=optimizer,\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRTE3pSa0okQ",
        "colab_type": "code",
        "outputId": "45c8a1a4-5068-4a06-c4da-648ec7870612",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "BATCH_SIZE = 64\n",
        "NEPOCHS = 4\n",
        "history = model.fit_generator(train_augment.flow(train_data, train_y, batch_size = BATCH_SIZE),\n",
        "                              epochs = NEPOCHS, \n",
        "                              validation_data = valid_augment.flow(valid_data, valid_y),\n",
        "                              verbose = 1, \n",
        "                              steps_per_epoch=(train_data.shape[0] // BATCH_SIZE), \n",
        "                              validation_steps=(valid_data.shape[0] // BATCH_SIZE))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/4\n",
            "88/88 [==============================] - 63s 719ms/step - loss: 1.8919 - acc: 0.4125 - val_loss: 1.3826 - val_acc: 0.4623\n",
            "Epoch 2/4\n",
            "88/88 [==============================] - 61s 690ms/step - loss: 1.2887 - acc: 0.5131 - val_loss: 1.3330 - val_acc: 0.4828\n",
            "Epoch 3/4\n",
            "88/88 [==============================] - 60s 684ms/step - loss: 1.2267 - acc: 0.5345 - val_loss: 1.3158 - val_acc: 0.4649\n",
            "Epoch 4/4\n",
            "88/88 [==============================] - 60s 685ms/step - loss: 1.2036 - acc: 0.5476 - val_loss: 1.2863 - val_acc: 0.4903\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZsY6tZF5bkM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for layer in baseModel.layers:\n",
        "    layer.trainable = True\n",
        "\n",
        "optimizer = Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer=optimizer,\n",
        "              metrics=['acc'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O_Sevr5Q14vP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', patience=3, verbose=1, factor=0.5, \n",
        "                                            min_lr=0.000001, cooldown=2)\n",
        "early_stopping = EarlyStopping(\n",
        "    monitor='val_acc', \n",
        "    verbose=1,\n",
        "    patience=3,\n",
        "    mode='max',\n",
        "    restore_best_weights=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cn6Xjjlz2EgL",
        "colab_type": "code",
        "outputId": "bdc46449-509b-4752-b04b-a689de8bf39d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        }
      },
      "source": [
        "BATCH_SIZE = 64\n",
        "NEPOCHS = 30\n",
        "history = model.fit_generator(train_augment.flow(train_data, train_y, batch_size=BATCH_SIZE),\n",
        "                              epochs = NEPOCHS, \n",
        "                              validation_data = valid_augment.flow(valid_data, valid_y),\n",
        "                              verbose = 1, \n",
        "                              steps_per_epoch=(train_data.shape[0] // BATCH_SIZE), \n",
        "                              validation_steps=(valid_data.shape[0] // BATCH_SIZE), \n",
        "                              callbacks=[learning_rate_reduction, early_stopping])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/30\n",
            "88/88 [==============================] - 76s 860ms/step - loss: 1.0070 - acc: 0.6336 - val_loss: 0.7840 - val_acc: 0.7220\n",
            "Epoch 2/30\n",
            "88/88 [==============================] - 63s 716ms/step - loss: 0.7852 - acc: 0.7198 - val_loss: 0.6651 - val_acc: 0.7608\n",
            "Epoch 3/30\n",
            "88/88 [==============================] - 62s 705ms/step - loss: 0.6655 - acc: 0.7637 - val_loss: 0.6575 - val_acc: 0.7770\n",
            "Epoch 4/30\n",
            "88/88 [==============================] - 62s 700ms/step - loss: 0.5420 - acc: 0.8105 - val_loss: 0.5797 - val_acc: 0.8017\n",
            "Epoch 5/30\n",
            "88/88 [==============================] - 61s 697ms/step - loss: 0.4950 - acc: 0.8323 - val_loss: 0.5688 - val_acc: 0.7971\n",
            "Epoch 6/30\n",
            "88/88 [==============================] - 61s 698ms/step - loss: 0.4202 - acc: 0.8578 - val_loss: 0.5878 - val_acc: 0.8362\n",
            "Epoch 7/30\n",
            "88/88 [==============================] - 62s 702ms/step - loss: 0.3848 - acc: 0.8660 - val_loss: 0.5009 - val_acc: 0.8239\n",
            "Epoch 8/30\n",
            "88/88 [==============================] - 61s 694ms/step - loss: 0.3571 - acc: 0.8764 - val_loss: 0.5970 - val_acc: 0.8125\n",
            "Epoch 9/30\n",
            "88/88 [==============================] - 61s 699ms/step - loss: 0.3254 - acc: 0.8861 - val_loss: 0.5844 - val_acc: 0.8283\n",
            "\n",
            "Epoch 00009: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.\n",
            "Restoring model weights from the end of the best epoch\n",
            "Epoch 00009: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GgBbKfINPlHr",
        "colab_type": "text"
      },
      "source": [
        "Note that the learning rate is perhaps the most hyperparameter of our model. \n",
        "In this code block we perform a simple grid search to find an appropriate value of the learning rate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vcIsCtN9vKTE",
        "colab_type": "text"
      },
      "source": [
        "# Inspection of the learning curves:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L4PeN4ZvP5CQ",
        "colab_type": "code",
        "outputId": "2de38a0a-0c72-4e2a-a2f8-19bc3263563b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 351
        }
      },
      "source": [
        "fig, axs = plt.subplots(nrows = 1, ncols =2 , figsize = (10,5))\n",
        "axs[0].plot(range(len(history.history['acc'])), history.history['acc'], linewidth = 5, label = \"training\")\n",
        "axs[0].plot(range(len(history.history['acc'])), history.history['val_acc'], linewidth = 5, label = \"validation\")\n",
        "axs[1].plot(range(len(history.history['loss'])), history.history['loss'], linewidth = 5, label = \"training\")\n",
        "axs[1].plot(range(len(history.history['loss'])), history.history['val_loss'], linewidth = 5, label = \"validation\")\n",
        "axs[0].set_xlabel(\"epochs\")\n",
        "axs[0].set_xlabel(\"epochs\")\n",
        "axs[0].set_ylabel(\"accuracy\")\n",
        "axs[1].set_ylabel(\"loss\")\n",
        "plt.legend()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f7c37e94e80>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAE9CAYAAABOT8UdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdeVhV1frA8e9iFgEFUVDEMQcUBxSn\nJjXTzMopTS1v2aDlrew2W91flg3Xe5snLW22shxySE3L0sq0EsUB5yEHQBCcwIHhcNbvj40K7AMc\n4AwM7+d59iNn7b3PerWA9+y11ruU1hohhBBCCFE5eLg7ACGEEEIIcYkkZ0IIIYQQlYgkZ0IIIYQQ\nlYgkZ0IIIYQQlYgkZ0IIIYQQlYgkZ0IIIYQQlYiXuwNwlNDQUN2sWTN3hyGEcKGNGzema63ruzsO\nR5CfYULULCX9/Ko2yVmzZs2Ii4tzdxhCCBdSSh1ydwyOIj/DhKhZSvr5JcOaQgghhBCViCRnQggh\nhBCViCRnQgghhBCVSLWZcyaEEEKIisvNzSUxMZGsrCx3h1It+Pn50bhxY7y9ve2+R5IzIYQQQlyU\nmJhIYGAgzZo1Qynl7nCqNK01x48fJzExkebNm9t9nwxrCiGEEOKirKws6tWrJ4mZAyilqFevXpmf\nQkpyJoQQQohCJDFznPL8W0pyJoSoFCx5VtbtS+f/FiUQd/CEu8OptM5mW/h2UyLjP48jKzfP3eEI\n4XCnTp1i+vTpZb5v0KBBnDp1qsRrnn32WVatWlXe0FxG5pwJIdwm25LH7/vSWZGQwo87Ujl5LhcA\njSa2WYibo6tcft+Xzry4I6zcnsr5/KTs513HGNShoZsjE8KxLiRn//znPwu1WywWvLyKT1uWL19e\n6ntPnTq1wvG5gjw5E0K41LkcC99vO8pDX8cT+8Iq7vo0jrlxiRcTM4CV21OxWrUboyyeUupjpdQx\npVRCMeeVUuptpdQ+pdRWpVQXR/S7MD6JRZuTLyZmAN9uSnLEWwtRqUyePJn9+/fTuXNnunXrxlVX\nXcXgwYNp164dAEOHDqVr1660b9+emTNnXryvWbNmpKenc/DgQaKiohg/fjzt27dnwIABnD9/HoBx\n48Yxf/78i9dPmTKFLl260KFDB3bt2gVAWloa/fv3p3379txzzz00bdqU9PR0l/4byJMzIYTTZWTl\n8vPOY3yfcJRf9qSRlWst8fq0zGw2HT5ZWZ+efQq8C3xezPnrgVb5Rw9gRv6fFTIsJoL5GxMLta3Z\nfYwTZ3MIqe1T0bcXwqTZ5GVO7+PgtBtMbdOmTSMhIYHNmzezZs0abrjhBhISEi6udvz4448JCQnh\n/PnzdOvWjZtvvpl69eoVeo+9e/cyZ84cZs2axS233MKCBQsYO3asqa/Q0FA2bdrE9OnTefXVV/nw\nww95/vnnueaaa3jqqadYsWIFH330kXP+8iWQ5EwI4RTHz2Szamcq3yek8Pu+dHLzyvYkbEVCSqVM\nzrTWvyqlmpVwyRDgc621Bv5QStVVSjXUWh+tSL89W9QjLMiX1Izsi20Wq2bZtqP8o2fTiry1EJVa\n9+7dC5WhePvtt1m4cCEAR44cYe/evabkrHnz5nTu3BmArl27cvDgQZvvPXz48IvXfPvttwCsXbv2\n4vsPHDiQ4OBgh/597CHJmRDCYVJOZ7FyeworElL48+/jlHVk0s/bgz6tG3B9h3D6tm3gnCCdLwI4\nUuB1Yn6bKTlTSk0AJgA0adKkxDf19FAM6RzBzF8PFGpfFJ8kyZmo1mrXrn3x6zVr1rBq1SrWr1+P\nv78/ffr0sVmmwtfX9+LXnp6eF4c1i7vO09MTi8Xi4MjLT5IzIUSFHD5+jhXbj7IiIYVNh0teKWVL\noK8X/aIaMDA6nN6tG1DLx9MJUVZOWuuZwEyA2NjYUlPZoTaSs42HTnL4+Dma1PN3TpBCuFhgYCCZ\nmZk2z50+fZrg4GD8/f3ZtWsXf/zxh8P7v+KKK5g7dy5PPvkkP/zwAydPnnR4H6WR5EwIUWZ7UzNZ\nkZDC9wkp7DiaUeb7g/29GdAunIEdwrm8ZT18vapVQpYERBZ43Ti/rcKiGgbSJiyQ3amFf3Et2pzE\npH6tHNGFEG5Xr149rrjiCqKjo6lVqxZhYWEXzw0cOJD333+fqKgo2rRpQ8+ePR3e/5QpUxgzZgyz\nZ8+mV69ehIeHExgY6PB+SqKMaRFVX2xsrI6Li3N3GEJUS1prtidn8H2C8YRsf9rZMr9HWJAvA9uH\nc110ON2bheDlWfHF4kqpjVrr2Aq/Udn7bQYs1VpH2zh3A/AAMAhjIcDbWuvupb2nvT/DZqzZz39X\n7CrU1iK0Nj892lsKhwqH2LlzJ1FRUe4Ow22ys7Px9PTEy8uL9evXM3HiRDZv3lyh97T1b1rSzy95\nciaEsMlq1cQfOcn321JYsT2FxJO252yUJDKkFtdHN+S69uHERNbFw6PqJw9KqTlAHyBUKZUITAG8\nAbTW7wPLMRKzfcA54E5H9j+kcyP+t3IXBT9XH0g/y5bE03SOrOvIroSokQ4fPswtt9yC1WrFx8eH\nWbNmuTwGSc6EEIXsSM5g/sZElm5N5lhmduk3FNGqQQADo8MZGB1Ou4ZB1e5pjtZ6TCnnNXC/s/pv\nVLcWPZvXY/2B44XaF8UnSXImhAO0atWK+Ph4t8YgyZkQghNnc1i8OYn5GxPZnlz2OWTREUEXn5Bd\n1iDACRGKgobFRJiSs++2JPPMDVF4O2C4WAjhXpKcCVFDWfKs/LInjfkbE1m1M7VMdciUgq5NghkY\nHc517cOJDJGVgq40sEM4/16cQI7lUjHf42dzWLs3vSqXIBFC5JPkTIgaZm9qJvM2JvLtpiTSz9g/\nbOnpoejZIsSY1N8+nAZBfk6MUpQkyM+b/lFhLNtWuHTawvgkSc6EqAYkOROiBjh9LpclW5OZH3eE\nLYmn7b7Px9ODK1uFMjA6nGujwmSboEpkaEyEKTn7YUcKZ7ItBPjKj3YhqjKZnCBENZVn1fyyJ40H\nvtpEt5dX8X+LEuxOzDpH1uXFodFs+Pe1fDyuG7fERkpiVsn0bl2fuv7ehdqycq2sTEhxU0RCuEdA\ngDHPNTk5mREjRti8pk+fPpRWqubNN9/k3LlzF18PGjSIU6fKXljbEeTjlRDVzIG0M8zPH7ZMyTBv\na1Kc+oG+DI+JYETXxrQKc23BRVF2Pl4e3NixIV/8cbhQ+8L4JG7u2thNUQnhPo0aNWL+/Pnlvv/N\nN99k7Nix+Psbc2iXL1/uqNDKTJ6cCVENZGbl8vVfh7l5xjquee0Xpq/Zb1di5u2pGNQhnI/HxbJ+\n8jU8NShKErMqZFiMOQn7fX86qWVIyoWobCZPnsx777138fVzzz3Hiy++SL9+/ejSpQsdOnRg8eLF\npvsOHjxIdLRRF/r8+fOMHj2aqKgohg0bVmhvzYkTJxIbG0v79u2ZMmUKYGymnpycTN++fenbty8A\nzZo1Iz09HYDXX3+d6OhooqOjefPNNy/2FxUVxfjx42nfvj0DBgwodg/PspInZ0JUUVarZv2B48zf\nmMj3CUfJyrWWflO+6IggRnRpzJDOEQTLcGWV1aVJXZqE+HP4xKWhGK1hyeZkxl/dwo2RiWrhuTou\n6MM81WLUqFH861//4v77jXKBc+fOZeXKlUyaNImgoCDS09Pp2bMngwcPLraO4owZM/D392fnzp1s\n3bqVLl26XDz30ksvERISQl5eHv369WPr1q1MmjSJ119/ndWrVxMaGlrovTZu3Mgnn3zCn3/+idaa\nHj160Lt3b4KDg9m7dy9z5sxh1qxZ3HLLLSxYsICxY8dW+J9FkjMhqpjDx88xf1MiCzYmknTK/k9p\n9Wr7MDR/2DKqYZATIxSuopRiaEwEb/+0t1D7wvgkSc5ElRUTE8OxY8dITk4mLS2N4OBgwsPDefjh\nh/n111/x8PAgKSmJ1NRUwsPDbb7Hr7/+yqRJkwDo2LEjHTt2vHhu7ty5zJw5E4vFwtGjR9mxY0eh\n80WtXbuWYcOGUbt2bQCGDx/Ob7/9xuDBg2nevDmdO3cGoGvXrhw8eNAh/waSnAlRBZzNtrB821Hm\nb0zkz79P2H2fl4eib9sGjOjamL5tGuDjJTMZqpuhnRuZkrMdRzPYnZJJm3AZohZV08iRI5k/fz4p\nKSmMGjWKL7/8krS0NDZu3Ii3tzfNmjUjK6vsw/d///03r776Khs2bCA4OJhx48aV630u8PX1vfi1\np6enDGsKUZ1Z8qzsOJrBHweO88eBE/xx4DjncvLsvr9teCAjujZmaEwEoQG+pd8gqqwW9QPoFFmX\nLUcKrypbtDmJJwe2dVNUQlTMqFGjGD9+POnp6fzyyy/MnTuXBg0a4O3tzerVqzl06FCJ91999dV8\n9dVXXHPNNSQkJLB161YAMjIyqF27NnXq1CE1NZXvv/+ePn36ABAYGEhmZqZpWPOqq65i3LhxTJ48\nGa01CxcuZPbs2U75e18gyZkQlYAlz8r25AvJ2HE2HDzJmWxLmd6jrr83Qzo1YmRsJO0bVb89LUXx\nhnVuZErOFscn8fiANtVis3lR87Rv357MzEwiIiJo2LAht912GzfddBMdOnQgNjaWtm1L/uAxceJE\n7rzzTqKiooiKiqJr164AdOrUiZiYGNq2bUtkZCRXXHHFxXsmTJjAwIEDadSoEatXr77Y3qVLF8aN\nG0f37t0BuOeee4iJiXHYEKYtytijt+qLjY3VpdUwEaKysORZSchPxv4sZzIG4KGMelcjYyPpF9UA\nXy9PJ0RbeSmlNmqtY90dhyNU5GdY+plserz8E3nWwj/P54zvSa+W9RwRnqhBdu7cSVRUlLvDqFZs\n/ZuW9PNLnpwJ4QIFk7E/DhwnrpzJ2AUt69dmZGwkw2MiZBslQWiAL71b1+fnXccKtS+KT5LkTIgq\nSJIzIZwgN89KQtLpi/PF4g6e4GwZ5ozZEuTnxY2dGjGya2M6R9aVYUtRyNCYCFNytnzbUZ4f0h4/\n75r1RFWIqk6SMyEcIDfPyrak0/zpwGQs0NeL7s1D6NmiHj1b1KNdoyA8Zf6QKEb/qDBq+3gW+v8u\nM9vCz7uOMahDQzdGJoQoK0nOhCiHC8nYhdWUcQdPlGk1pS2SjImKqOXjycDohizYlFiofWF8kiRn\nosy01vJ03kHKM7dfkjMhyiAh6TRv/bSX3/elVzwZ8/OiR4FkLKqhJGOiYobFRJiSszW7j3HybI7s\nBCHs5ufnx/Hjx6lXr54kaBWkteb48eP4+ZVtbrAkZ0LYaenWZB75Zgs5efZvk1SQkYzVo2eLEEnG\nhFP0almPBoG+HMvMvtiWm6dZtu0oY3s2dWNkoipp3LgxiYmJpKWluTuUasHPz4/Gjc374JZEkjMh\n7PDx2r95YdkOyvJ0OsjPi+6SjAkX8vRQDOnciFm//V2ofWF8kiRnwm7e3t40b97c3WHUaJKcCVEC\nq1UzbcUuZv56oNRrg/y86JE/RNmzRQhtwyUZE643LKaxKTnbeOgkh4+fo0k9fzdFJYQoC0nOhChG\njsXKE/O3sGhzss3zkoyJyiiqYSBtwgLZnZpZqH3R5iQm9WvlpqiEEGUhyZkQNpzJtnDf7I2s3Zdu\n83z/dmG8MyZG6keJSkcpxdCYCP67Yleh9kXxSTx4zWUywVuIKsDD3QEIUdkcy8xi1Afri03Mbu3R\nhBm3dZHETFRaQzo3MrUdSD/L1sTTbohGCFFWTk3OlFIDlVK7lVL7lFKTbZxvopRarZSKV0ptVUoN\nym9vppQ6r5TanH+878w4hbjgQNoZhk9fx/bkDJvnH+nfmpeGRuPlKZ9rROXVqG4terYIMbUvjE9y\nQzRCiLJy2m8YpZQn8B5wPdAOGKOUalfksn8Dc7XWMcBoYHqBc/u11p3zj/ucFacQF8QfPsnNM9aR\nePK86Zynh+K/N3dgUr9WMiwkqoRhMRGmtu+2JJNbzlIwQgjXcebH/+7APq31Aa11DvA1MKTINRoI\nyv+6DmB75rUQTvbTzlTGzPqDk+dyTef8vD2YdXtXRnVr4obIhCifgdEN8fEq/CP++Nkc1u61PVwv\nhKg8nJmcRQBHCrxOzG8r6DlgrFIqEVgOPFjgXPP84c5flFJXOTFOUcN9s+EwE2ZvJCvX/EQh2N+b\nOeN7ck3bMDdEJiorO6ZsNFVK/ZQ/XWONUqpsFSgdoE4tb/pHmf+/laFNISo/d0+cGQN8qrVuDAwC\nZiulPICjQJP84c5HgK+UUkFFb1ZKTVBKxSml4qSSsSgrrTVv/7SXJxdsI89qri4bGVKLBRMvJ6ZJ\nsBuiE5WVnVM2XgU+11p3BKYC/3FtlIahNoY2f9iRwplsixuiEULYy5nJWRIQWeB14/y2gu4G5gJo\nrdcDfkCo1jpba308v30jsB9oXbQDrfVMrXWs1jq2fv36TvgriOrKkmflmUUJvP7jHpvn2zcKYsHE\ny2lRP8DFkYkqwJ4pG+2An/O/Xm3jvEv0bl2fuv7ehdqycq2sTEhxRzhCCDs5MznbALRSSjVXSvlg\nTPhfUuSaw0A/AKVUFEZylqaUqp//6RSlVAugFVB6iXYh7HA+J4/7vtjEV38etnn+qlahfHNvLxoE\nlm2jWlFj2DNlYwswPP/rYUCgUqqeC2IrxMfLgxs7NjS1L9osQ5tCVGZOS8601hbgAWAlsBNjVeZ2\npdRUpdTg/MseBcYrpbYAc4BxWmsNXA1sVUptBuYD92mtTzgrVlFznDybw20f/sGqnak2zw/t3IiP\n7uhGgK/UZxYV8hjQWykVD/TGGDXIK3qRK6Zm2Fq1+fu+dFIzspzSnxCi4pz6G0hrvRxjon/BtmcL\nfL0DuMLGfQuABc6MTdQ8iSfPccfHf7E/7azN8/de3YInB7bFQ7ZgEiUrdcqG1jqZ/CdnSqkA4Gat\n9amib6S1ngnMBIiNjTVPfHSALk2CiQypxZETl0rEWLVRVuOeq1o4o0shRAW5e0GAEC6xIzmD4dPX\n2UzMlIJnb2zHU4OiJDET9ih1yoZSKjR/cRPAU8DHLo6xYCwM62x+evbtJhnaFKKykuRMVHvr9qUz\n6oP1HMvMNp3z8fTgnTEx3HVlczdEJqoiO6ds9AF2K6X2AGHAS24JNp+tVZs7jmawOyXTxtVCCHeT\niTWiWluyJZlH524mN888YhTo68XM22Pp1dLl87RFFWfHlI35GPNlK4UW9QPoFFmXLUcKj6wu2pzE\nkwPbuikqIURx5MmZqLY+/O0Ak+bE20zMwoJ8mXtfL0nMRI0xzMZm6Ivjk7DaqPEnhHAvSc5EtWO1\nal5atoMXl+20ef6yBgF8+88riGpoqmssRLV1Y6dGeBaZU5l8Oou/DspCeCEqG0nORLWSY7Hy8NzN\nzPrtb5vnY5sGM/++XkTUrVW+DvJyYd8q2LkUzp+sQKRCuFZogC9Xtwo1tS+S7ZyEqHQkORPVRmZW\nLnd++heLNyfbPD+gXRhf3NODuv4+5esgNwu+uNk4vrkNXouCJQ/C0a0ViFoI17G1MGDZtqNk5ZpK\nsAkh3EiSM1EtHMvIYtQHf/D7vuM2z4/t2YQZY7vi5+1Z/k5WPgV//3LpteU8bPocPrgKProOts0H\nS075318IJxvQLpzaPoW/BzKzLPy865ibIhJC2CLJmajy9qedYfiMdew4mmHz/GMDWvPCkGjTfJsy\n2TYf4kooVXXkD1hwN7zRHn5+CTJsP70Twp1q+XhyXXS4qX2hDG0KUalIciaqtE2HTzJixjoST543\nnfP0UPxvREceuKYVSlUgMUvfB989ZN+1Z4/Br/+DN6Jh7u1wcC1oWQ0nKo/hMY1NbWt2H+PkWXnq\nK0RlIcmZqLKWbzvKrbP+4OS5XNO5Wt6efHh7LLfERtq4swxyz8O8cZBzpmz36TzYsRg+vQFmXA4b\nPoLsMr6HEE7Qq2U9GgT6FmrLzdMs23bUTREJIYqS5ExUOedyLDz17Vb++eUmsnKtpvMhtX2YM6En\nfds2qHhnKyZD6jZze7uhcNNbEBZd+nsc2wHLHoHXo+D7ycaTOCHcxNNDMcRGzTNZtSlE5SHJmahS\nEpJOc+M7a5nz1xGb55uE+LNg4uV0jqxb8c62zoONn5rbQ1rC4Heg6zi4by3c+T20Hw4epWy4kZ0B\nf86Ad7vC7GGwazlYZZVcIVmnjTIlwqlsrdqMO3SSw8fPuSEaIURRkpyJKsFq1cz69QDDpv/OARub\nlwNERwSxYOLlNA+tXfEO0/fC0n+Z2z19YeSn4JdfwFYpaHo5jPwE/pUAfZ6CgLDS33//z/D1GHir\nM6x9A87aXmVaY1iyYf17xr/H3H9A2h53R1SttWsYROuwAFP74s3y9EyIykCSM1HpHcvI4o5P/uKl\n5TttbsUEcF37ML6e0Iv6RebSlEvueZh7h+15ZtdPg4Ydbd8X1BD6TDaStBEfQ5PLS+/r9GFY9Zwx\n5LlwIiRtqlDoVY41DzbPgXdiYeXTcP4EaCv89Ly7I6vWlFI2n54tjE9CywIWIdxOkjNRqf20M5WB\nb/3Gb3vTbZ738/bg5WEdeH9sVwJ8SxlWtNf3T8Cx7eb26BHQ9c7S7/fygeib4a7vjWHPruPA27/k\ne/KyYctXMKsvzOoHW742niZVV1rDnh/g/atg0X1GklrQrqVw5C/3xFZDDOlsTs4OpJ9la+JpN0Qj\nhChIkjNRKWXl5vHs4gTu/iyOE8Us8W/XMIilD17JrT2aVKxURkFbvjEKyxZV7zK46U1jGLMswjsY\nCwce2QnX/QdCWpR+T1IcLLwXXm8Hq56HU7bn11VZiXHw6Y3w1UjbSfAFq19yXUw1UETdWvRsEWJq\nl5pnQrifJGei0tmVksHgd9fy+fpDxV5zz5XNWXj/5VzWINBxHaftgaUPm9u9/GDkZ+Bbgb5q1YVe\n/4QHNsJtC6D1QKCURO9cOqx9Hd7qCF/fBgfWVO2aael74Zux8GE/OLS2hAsVdL4NBr/rstBqqmE2\nhja/25JMbp55FbQQwnUcNA4kRMVprfl8/SFeWr6THIvtXw6hAb68dksnereu79jOc87BvDsg18Zi\ng+v/C+F2lMywh4cHtLrWOE78bew6sOlzyDpV/D3aagzz7VoKwc0hZix0vhWCzOUQKqWMo/DLNNg0\n26j/VpLWA6HfsxDW3jWx1XADoxvyf4u3F/p+O342h7X70unbxgGlaIQQ5SLJmagUjp/J5vH5W0vc\n469vm/q8MrIToQEOmPRf1PdPGPXIiuowErrc4fj+AEKaw4AXjBWeCQvgr5mQUsom6if/hp9fMIb8\nLusPXf5hJDSe3s6JsSKyTsPvb8H66cY+pCVp3B36P2+sfBUuU6eWN9dGNWD5tpRC7YvikyQ5E8KN\nJDkTbvfrnjQenbeFtEzbE+B9vDx4+vq23HF5M8fNLStoy9cQP9vcXq8V3FiOeWZl5eNvJFkxYyFx\nA/w1C7YvBKt554OLtBX2rjSO2vWh4yjocjvUb+PcWO2RmwUbPoTfXoXzJ0u+NrQ19JsCbW9w/r+z\nsGlo5whTcrZyewpnsi2OW2QjhCgT+c4TbpNtyeOVFbv5cO3fxV7TOiyAt8fE0DY8yDlBpO0uYZ7Z\np+BrrgXlNEpBZHfjuO4l2PiZMeyZWcom6mfTYP27xtG4u5HotR/u2tjBKIuxda7xVO90KYsYAhsa\nTww73wae8mPInfq0aUBdf29OFdgGLSvXysqEFG7uat6HUwjhfPJTUbjFvmNnmDQnnh1HM4q95vZe\nTXl6UBR+3p7OCSLnrFHPLNdGVfRBrzhunll5BDSA3o/DlQ/D7mUQ94mxIIBSFgQk/mUc30+G6GEQ\nc7uR7DnzqZTWsPcHo16braHhgnzrwFUPQ/d7jSeGwu18vDy4oUNDvvyzcDmTRZuTJDkTwk0kORMu\npbVmzl9HmLp0u819MQGC/b15ZUQnrm1nR6X9ilj+BKTtNLd3HAUx/3Bu3/by9IJ2Q4zj1GGI/xI2\nf1n6k6ncsxD/hXGEtjb+Pp3GQICDF1Ic2QCrpsCh30u+ztMXekyAKx8Bf3P5BuFew7tEmJKz3/el\nk5qRRViQn5uiEqLmkuRMuMzJszlM/nYrK7enFnvNlZeF8totnZz/C2HzV7D5C3N7aGu44fXKOf+p\nbhPo+xT0fsJ4ihY/G3YtgzzbdeAuSt8DP/6fUXW/9UBjblrLfhUbTkzbY7zfrlL2wVQe0OlWY+eE\nupHl7084VZcmwUSG1OLIiUsLN6zaKKtxz1V21OYTQjiUJGfCJdbtT+eRb7aQkpFl87y3p+KJ69py\n95XN8fBwcmJ0bCcsfcTc7lUrv56Zi+dqlZWHJ1zWzzjOnTDmecXPhtSEku+zWi6V5AhsaJTj6Hwb\n1Gtpf98ZybBmmvFErtSyGNfnl8VoZ//7C7dQSjGscwRv/7yvUPvC+CRJzoRwA0nOhFPl5ll5/cc9\nvP/L/mLrp7YIrc3bY2KIjqjj/IByzsK8cbZLOwx6peolEv4h0PM+6HEvJMcbSdq2+ZBd/Fw+ADKP\nwm+vGUfTK41FBFGDi58Hdv4U/P4m/PF+6WUxInvAtc9D017l+zsJtxgSY07OtidnsCc1k9ZhDiz2\nLIQolSRnwmkOpp/loa/j2VLCXn2ju0Xy7E3t8Pdx0f+Kyx6DtF3m9k5jjFIWVZVSENHFOAa8BDuX\nGEVfS6zEn+/QWuNY/gR0uNmYn9YoxnjP3CzYMAt+fbXkQrkAoW3g2inQZlDlHBYWJWpZP4BOjeuY\nvl8XxSfxxMC2bopKiJpJkjPhcFprFmxKYsriBM7m2B76qlPLm2nDO3B9h4auCyz+S2Nz8aJC28AN\nr1WfhMLHHzqNNo7j+40hyM1fwZmUku/LPm2U7oj7GMKiofV1xl6jGYkl3xfYyJgL1+lWKYtRxQ2N\niTAlZ4s3J/PYgDbOn24ghLhIfpIKhzp9PpdnFm5j6dajxV7To3kIb4zqTKO6tVwXWOoOWPaoud2r\nllHPzKe262JxpXotjadZfZ+BfauMYc89K4z5ZyVJTSh9DptfHWP1ZY97wduF/y2F09zUqREvLttJ\nnvXSHISkU+f56+AJerao58bIhKhZJDkTDrPh4An+9fVmkk7ZnpPk6aF4pH9r7uvdEk9XfgrPPmPs\nm2lrrtQNr1W9eWbl4ekFbZkKFH0AACAASURBVAYax5ljsGWOMex5fG853svXmOd25cNQK9jxsVYB\nSqmBwFuAJ/Ch1npakfNNgM+AuvnXTNZaL3d5oGUUGuDL1a1CWb07rVD7ovgkSc6EcCEPdwcgqocl\nW5IZ9cH6YhOzJiH+zL+vF/f3vcy1iZnWxhOz9D3mc51uhZjbXBdLZRHQAK54CB7YAHf9YMy187bj\nyaHyMK6dtAn6T63JiZkn8B5wPdAOGKOUKprh/xuYq7WOAUYD010bZfkNjYkwtS3bdpSs3FJW5woh\nHEaSM1FhiSfP8eT8rViLWY05PCaCZZOuJKaJG36Zx38BW782t9dvCze86vp4KhOloEkPGPIePLYb\nBr9jbP9kS5tBMHGdcW2dGl81vjuwT2t9QGudA3wNDClyjQYu7DlWByhlD67KY0C7cGr7FN6VIzPL\nwupdx9wUkRA1jwxrigrRWvPcku2ct/GpOtDXixeHRTOks/mTuEukboflj5nbvf2NembVdZ5ZefgG\nGsVpu9wOx3YZBXoP/wlBjaDnRGjS090RViYRQMEtGhKBHkWueQ74QSn1IFAbuNY1oVVcLR9ProsO\n59tNSYXaF8YnuXYBjxA1mCRnokJWbk9l1U7zJ+qYJnV5e3QMkSFu2j8x+0x+PTMbRW9veA0aSGmA\nYjVoCwNedHcUVd0Y4FOt9WtKqV7AbKVUtNa60J5lSqkJwASAJk2auCFM24bFRJiSs9W7j3HqXA51\n/X3cFJUQNYcMa4pyO5Nt4bkl203tDQJ9+eyu7u5LzLSGZY/YnmfWeaxRGV+I8ksCCu5F1Ti/raC7\ngbkAWuv1gB8QWvSNtNYztdaxWuvY+vUdvO9pBVzeMpT6gb6F2nLzdImrsIUQjiPJmSi3137YbXM7\npucGtyfIz9sNEeXb9Dls/cbcXj/K2AVAiIrZALRSSjVXSvlgTPhfUuSaw0A/AKVUFEZyloajHFoP\nJ/522NsV5emhGNKpkal9UXzRHFQI4QySnIlySUg6zWfrDpra+7apz/XR4a4P6IKUBPj+CXO7tz/c\n8lnx2xMJYSettQV4AFgJ7MRYlbldKTVVKTU4/7JHgfFKqS3AHGCc1sVtYFYGOedgxVPwyfWw+AGw\nWku/p5yGdTHPFY07dJLDx885rU8hhEGSM1FmeVbN0wu3mVZn+nl7MHVINMpdlfazM/PrmdmaZ/Y6\n1G/j+phEtaS1Xq61bq21bqm1fim/7Vmt9ZL8r3dora/QWnfSWnfWWv9Q4U4PrYf3r4Q/pgPa2HJr\nw4cVftvitGsYROuwAFP7J+uc98ROCGGQ5EyU2ez1B9laZIsXD6y8E32AyLj/wO9vwe7vja2D8kqp\nRO8oWsN3/4Lj+8znYsZC5zGuiUMIZ8jLhYX3won9hdtXTXHa8KZSymbNs0/XHWTjoRNO6VMIYXDq\nas2KVNFWSj2FMak2D5iktV7pzFiFfVJOZ/HqD4Un2gdyjo8DptNt5ybzDZ4+ENISQlsZT65CWxtf\n12sFvuZP5eW28VNImG9ub9AOrpd5ZqKK8/SGwW/D50XKqeWeM4Y37/gOPBz/WfvmLo15c9VeciyX\nhk+1hsfnb2X5pKvw8/Ys4W4hRHk5LTkrUEW7P0YdoA1KqSVa6x0FLrtQRXtGfoXt5UCz/K9HA+2B\nRsAqpVRrrbWUqHazqUu3cyb70tOwxuoYH3u/QmtLMROF83Igbadx7CxyLqixkahdSNguJG8BYWXb\nhDxlG3z/pLndu3Z+PTOZZyaqgRZ9IPYuY2P6gi4Mb/aY4PAuw4L8ePja1vx3xa5C7QfSzvLGj3t4\nalCUw/sUQjj3ydnFKtoASqkLVbQLJmfFVdEeAnyttc4G/lZK7ct/v/VOjFeUYvWuYyzflnLxdaza\nxQc+b1BPZZbvDTMSjePA6sLtvkH5SVubAslbawhpbjxBKCg7E+beAXnZ5ve/6U2o37p8sQlRGfWf\nCntXwenDhdtXTYFW/Y3vEQcbf1VzViQcZUuRqQyzfjvAddHhdHHHzh9CVHPOTM4qUkU7AvijyL1u\nKjMvAM7lWPj3ooSLr4d7/Mp/vD/EVzlhTll2BiRtNI6CPLwgpMWlJ22hrY25bUXn4YBR6b7jLY6P\nTQh38g2EIe+4dHjTy9ODV0Z24sa315KTd2l406rh8XlbWCbDm0I4nLsXBFyoot0YGIRRRdvumJRS\nE5RScUqpuLQ0x5UQEmZv/bSXpFPnUVh5wutrXvd533Zi5hMAnW+DJr3Av55jg7BajMKyu5bC2jdg\n0UTYWbS8FNCgPVz/P8f2LURl0aKPMbxZlBNXb7YOC+Sha1uZ2venneWNVTaKPQshKsSZT87sraI9\nEIwq2kqpC1W07bkXrfVMYCZAbGxsxWsICZt2pWTw0W9/U4ss3vCewUDPDbYvrBMJY76G8OhLbWeP\nGwlV0ePkIYxRbQfzrm3UM/Ou5fj3FqKycMPw5r1Xt2Dl9hTTSu1Zvx5gYPtwYmR4UwiHcWZydrGK\nNkZiNRooum/OhSranxapor0E+Eop9TrGgoBWwF9OjFUUw2rVPP3tNkKt6Xzo8xrRHgdtX9i4G4z+\nCgIaFG6vXQ9q94KmvQq352YZw5FpuyF9b37SthvS94HlfPkDvuktY8hTiOrMXcObIzpx0zs2hjfn\nb2Xpg1fK8KYQDuK05ExrbVFKXaii7Ql8fKGKNhCXX6zxUWCWUuphjMcoF6pob1dKzcVYPGAB7peV\nmu7x9YYj5B7ZxGLfVwlTp2xf1GEkDH4XvP3sf2NvPwhrbxwFWa3GIoH0PZB24UnbXiNxO1vK0HWX\nO6DjSPtjEKIqa9HH5as324Qbw5uvrNxdqH3fsTO89dNenhzY1uF9ClETKUfsKFIZxMbG6ri4OHeH\nUa2kZWYz7bVpvKjfpZbKsX1R33/D1Y+VrfRFeZ07YRSZTdtdIGnbA8oD2g2GPk+Dp1NL94lKRim1\nUWsd6+44HKFcP8OyM2H65ebhTW9/mLjOKcObljwrw6avY1tSkULUCr795xV0jqzr8D6FqI5K+vkl\nv8mEbVrz12eTeY2PwVbe5eUHQ2dA9HDXxeQfAv7dIbK76/oUojJz2+rNjtz0zlpy8y59uL+wevM7\nGd4UosLcvVpTVEa5WRz77HZuSP/Y9vmAcLhzuWsTMyGEbS36uHz1ZtvwIB7qZ57buffYGd7+aa9T\n+hSiJpHkTBR25hjWT2+kwUEbJSoAS4MOMP5niOjq4sCEEMXqPxXqNDG3O3HvzXt7tyQ6IsjU/v4v\n+9lypJj5qUIIu0hyJi5J3Q6zrsEjyXapjMSwa/C6ZyXUkXrAQlQqF4Y3i7owvGm1ms9VkHf+6k1v\nz8LzHozVm1vItsgaLiHKS5IzYdi9Aj4aAKeP2Dy9sPYtNJowH3xquzgwIYRdWvRx+fBmVMMgHrzG\nPLy5J1WGN4WoCEnOajqtYd27MGc05Jwxnc7RnjxuuY/2t7+Oh6dM8hWiUnPD8ObEPi1p38jW8OYB\ntibK8KYQ5SHJWU1myYHvJsEPz2CrWv9xHcitOc9Q/8o7aR0W6Pr4hBBl46bhzVdHdsLLo/DwZp5V\n8/i8rTK8KUQ5SHJWU507AV8Mh02f2zy9xxrB0JypHAvuYnPYQghRSbXoU2mGN3enZvLuz/uc0qcQ\n1ZkkZzVR+l74sB8c/M3m6TV5nbg553mO6DBeGBpNLR8ZzhSiSnHD8OY/+7akXUPz8Ob0NfvZVmQ/\nTiFEySQ5q2kOrDESsxMHbJ7+xHIdd+c+Rib+3NixIb1b13dtfEKIinPX6s2RHW0Obz42bws5Fsf3\nKUR1JclZTRL3McweDlnmT7EW7cEzuXfxvOUO8vAk0NeLZ29s54YghRAO0aKPy4c32zeqwwPXXGZq\nN4Y3ZfWmEPaS5KwmyLPA95Nh6cNgY//4DO3PHblP8mXetRfbnhjYhgZBZdjIXAhR+bhjeLPPZUTZ\nGN58b81+EpJkeFMIe0hyVt1lnTbKZPw5w+bpQzqMYTnP87u1w8W2TpF1ubVHU1dFKIRwFjcMb/p4\nefCqDG8KUSGSnFVnJw/CR9fBvh9tnt7h24kh2VPZry9V/Pf0ULw8LBpPD1u7nQshqpwWfdwyvPnP\nvubhzV0pmby7WlZvClEau5IzpdS3SqkblFKSzFUVh/+AWddA2k6bpw81HcGQ049yisL1y+68vBnt\nG9VxRYRCCFdxw/DmA30vo224uT7i9NX7ZHhTiFLYm2xNB24F9iqlpiml2jgxJlER504Y88s+vQHO\nHbdxgeJ836ncnDiaXLwKnWlUx4+H+7d2TZxCCNdx2/BmJ9NTeItV8/j8rTK8KUQJ7ErOtNartNa3\nAV2Ag8AqpdQ6pdSdSilvZwYo7JSbBb+/BW91NuaXWS3ma3wCYMwcXjp5Delnc0ynnxvcntq+Xub7\nhBBVX4s+Lh/ejI6ow/19Wpradx7NYPoaGd4Uojh2D1MqpeoB44B7gHjgLYxkzfaEJuEaVitsnQfv\ndoMfn4XsYoYL6kTCXSuJr9WTL/88bDo9oF0YA9qHOzlYIYRbuWN485pWNoc33/15H9uTZXhTCFvs\nnXO2EPgN8Adu0loP1lp/o7V+EAhwZoCiBAfXwofXwLf3wGlzwnVR424w/mcs9dvx9MIEdJFtNGv7\nePLc4PbOjVWIakQpNVAptVsptU8pNdnG+TeUUpvzjz1KqcqxA7ibhjdfGWF7ePOxeVvJzZPhTSGK\nsvfJ2dta63Za6/9orY8WPKG1jnVCXKIkaXtgzhhjXllyfPHXedeGPk/DuGUQ0IBPfj/IzqMZpsse\n7t+aRnVrOTFgIaoPpZQn8B5wPdAOGKOUKlSxWWv9sNa6s9a6M/AO8K3rIy1Giz4uH97s0LgOE3sX\nM7y5er9T+hSiKrM3OWunlKp74YVSKlgp9U8nxSSKcyYNlj4C03vC7uXFX6c8oOs4mBQPfZ4EL1+S\nTp3n9R/3mC5t1zCIcZc3c1rIQlRD3YF9WusDWusc4GtgSAnXjwHmuCQye7lhePPBfpfRJsw8vPnO\nz3vZkWz+0ChETWZvcjZea33xsbzW+iQw3jkhCZOcc/Drq/B2DMR9ZLPK/0WtroOJ6+CmtyAw7GLz\nlMXbOZ9b+D6l4OXhHfDylAopQpRBBHCkwOvE/DYTpVRToDnwczHnJyil4pRScWlpaQ4PtFhuGN70\n9fLklZEdi1m9uUWGN4UowN7fyp5KqYvfUfmP9X2cE5K4yJoHm7+Cd7rCzy9ATmbx14Z3hNuXwG1z\noUFUoVMrt6ewameq6ZZ/9GxK58i6pnYhhMOMBuZrbfsTldZ6ptY6VmsdW79+fddG1qKPy4c3Ozau\ny329W5jatydnMGONDG8KcYG9ydkK4BulVD+lVD+MR/QrnBeWYP9q+KA3LJoImcnFXxfUGIZ9ABN+\ngRa9TafPZFuYsni7qb1BoC+PXSfl6oQohyQgssDrxvlttoymsg1pFuSG4c1J/VrROsy8juydn/fa\nnBMrRE1kb3L2JLAamJh//AQ84aygarTUHfDFzTB7KKRuK/46n0DoNwUejINOo8HD9n/K13/YQ0pG\nlqn92ZvaEeQnJeqEKIcNQCulVHOllA9GArak6EVKqbZAMLDexfHZz03Dm7aK0+bmyfCmEBfYW4TW\nqrWeobUekX98UNxjelFOGUeNH4bvXwH7VhV/nYcXdJ8AD22Gqx4B7+JXWSYknebTdeZPv71b1+eG\nDg0dEbUQNY7W2gI8AKwEdgJztdbblVJTlVKDC1w6Gvha66LFayqZFn3cMrx579Xm4c2EpAw++EWG\nN4Wwqxy8UqoV8B+MZeN+F9q11ubvLlE22Wdg3Tuw7m3j02pJ2t4I1z4PoeYNhYvKs2qeXrgNa5Ff\nC75eHrwwJJoCUwiFEGWktV4OLC/S9myR18+5MqYK6T8V9q4y10tcNQVa9YeQ5g7v8qFrW/HjjlT2\nHjtTqP2tn/Zybbsw2oYHObxPIaoKe4c1PwFmABagL/A58IWzgqoR8iyw8VN4pwv8Mq3kxCyiK9z5\nPYz+0q7EDOCLPw6xNdFcffuha1vRpJ5/OYMWonpRSj2klApSho+UUpuUUgPcHZfLuW31ZieKjG6S\nm6d5bJ4Mb4qazd7krJbW+idAaa0P5X8ivMF5YVVjWsOeH+D9K+G7h+CMeRXlRXWbwIiP4Z6foOnl\ndneRmpHFKyt3m9pbhwUw/ip52ClEAXdprTOAARjzw/4BTHNvSG7Soo/Lhzc7R9ZlwtXm4rQJSRnM\n/PWAU/oUoiqwNznLVkp5AHuVUg8opYYh2zaV3dEt8PkQ+GokpO0s/jq/OjDgJXggDqJvNgqS2Ulr\nzbOLEziTbd74/OVhHfCWmmZCFHThm2sQMFtrvb1AW83jhtWb/7q2FZc1MP86eXPVHnanlFA+SIhq\nzN7f1A9h7Ks5CegKjAXucFZQ1U5GMiy8zyiN8fcvxV/n4Q0974dJm+HyB8DLt8xdLd6czMrt5qdx\no7tFEtsspMzvJ0Q1t1Ep9QNGcrZSKRUI1NzxNDcMb/p5e/LKiI42hzcfn78Fiwxvihqo1OQsv+Ds\nKK31Ga11otb6Tq31zVrrP1wQX9WXkgDvXwVb5gAlLNpqPwwe+AsGvgz+5UuiUjOyeHZxgqm9Xm0f\nJl/ftlzvKUQ1dzcwGeimtT4HeAN3ujckN2vRp/jhzV/+C8d2wrkTxhQNB4lpEsx4G6s3tyae5gMZ\n3hQ1UKmrNbXWeUqpK10RTLWTdRrm/gPOpRd/TWRPGPAiRHarUFdaayYv2EpGlnk4c+qQaOr6y4YO\nQtjQC9istT6rlBoLdAHecnNM7lfc6s1fphkHGE/6A8IgoAEEhht/BoQXeB126bwdowAPX9uaVTtS\n2Z92tlD7W6v2MqBdGK1s7MspRHVlVykNIF4ptQSYB1z8ztFaf+uUqKoDrWHJg3CimE99IS2MshhR\nN5VpTllx5sUlsnq3eW++Gzs25IaOUtNMiGLMADoppToBjwIfYqxGN2+3UZNcGN78vIT93K25kJFo\nHKWpFXwpUbOZwIXhFxjGKyM6MuL99YVKAOXkWXl64Ta+mdALj6Jjn0JUU/YmZ37AceCaAm0akOSs\nOH/NhB2Lze21QqDPZOh6J3g55mlW0qnzTF26w9QeGuDLC0OiHdKHENWURWutlVJDgHe11h8ppe52\nd1CVQos+xvBm3McVf6/zJ40jbVeJl3Xx9GVTYDB/ZwVwTNflb92QL/L6seEgzNlwmNt6NK14LEJU\nAXYlZ1rrmj0Ho6wSN8LKZ8ztPoFw94921yqzh9aaJ+dvtbk68z/DOxBcW4YzhShBplLqKYwSGlfl\nr0qXfc0u6D8VDq0veXW5I+VlUzcvhZgCs6Fv9fyJO3KeZNpyL66NCiMsyK/4+4WoJuzdIeATbMxm\n11rbmDVaw507AfPGGY/8ixr8tkMTM4Av/jzM2n3mOW3Du0TQv12YQ/sSohoaBdyKUe8sRSnVBHjF\nzTFVHr6BcOdy2PiJUQrozDHITDHqM5a2o4mDBKlzfO4zjXE5T/D8d6FMv62rS/oVwp3sHdZcWuBr\nP2AYkOz4cKo4qxUWTTRPogVjP8zo4Q7t7vDxc/xnufkTbXiQH1Nuau/QvoSojvITsi+BbkqpG4G/\ntNafuzuuSsU/BK561NyenVk4WTtzDM6kFGjLf302nRJXqtshUJ3nc59p3Jlg5ccdjeWDp6j27B3W\nXFDwtVJqDrDWKRFVZevfgT0rzO2NYowVmQ5ktRo1gM7lmPefn3ZzB+rUkpEZIUqjlLoF40nZGozi\ns+8opR7XWs93a2BVgW+gcdQzV/gvJM8CZ9OKJHCpkJma35Z66bXlfLFvU1tl86nP/3hioRe9Wj5A\ngK+9zxaEqHrK+393K6CBIwOp8g6th1XPm9v96sDIT8tVULYkn647yJ9/nzC1j+keSZ828p9GCDs9\ng1Hj7BiAUqo+sAqQ5MxRPL0gqKFxlETrAk/jkuH7J+FY4YVO/iqbV3NeZO68Otw+VqZCi+rLrh0C\nlFKZSqmMCwfwHfCkHfcNVErtVkrtU0pNtnH+DaXU5vxjj1LqVIFzeQXOLSnLX8rlzqbD/DtBm59i\nMfR9CG7m0O4OpJ3hfyvNq54i6tbi6UFRDu1LiGrO40Jilu849u+cIhxJKfALMublNr8a7vgOHWae\nnuGnchm193H2r1vohiCFcA17hzXLXP0vf2eB94D+QCKwQSm1RGt98aOQ1vrhAtc/CMQUeIvzWuvO\nZe3X5axW+HY8ZB41n7v8QWg7yKHd5Vk1j83bQlaueUuTV0Z0JNBPhjOFKIMVSqmVwJz816OA5W6M\nR1xQOxR1x1JyPrkJn7TCO5/4qlwif7gHS7APXlE3uClAIZzH3idnw5RSdQq8rquUGlrKbd2BfVrr\nA1rrHOBroISKhozh0g/IquO312D/z+b2yB7Qb4rDu5v12wE2HT5lar+9V1MuvyzU4f0JUZ1prR8H\nZgId84+ZWutSRwWEi/iH4HPXUo4FtjOd8sGCmns77Fxq40YhqjZ7H99P0VqfvvBCa30KKC3ziACO\nFHidmN9mopRqCjQHCmY5fkqpOKXUH3Ykgu5x4BdY87K5vVYIjPgEPB37FGtPaiav/7DH1N60nr/s\nnSlEOWmtF2itH8k/ZKyssqkVTMh9y9nl2cZ0ylNb0PPugO2L3BCYEM5jb3Jm6zpHLpUZDczXutCk\nraZa61iMGkRvKqVMS4KUUhPyE7i4tDTz1kVOlZkCC+4BXXR4UcHwWVDHZh5abrl5Vh6du4WcvML9\nKQWvjuyEv4+sXBLCXkXn0RY4MvPn1YpKxKt2MHm3fUuctbXpnLJa0PPvgoQFNu4UomqyNzmLU0q9\nrpRqmX+8Dmws5Z4kILLA68b5bbaMpsiQptY6Kf/PAxjL3GOK3qS1nqm1jtVax9avX9++v4kj5Flg\n/t1w9pj53NWPQatrHd7ljDX72ZZ02tR+9xXN6dYsxOH9CVGdaa0DtdZBNo5ArXWQu+MTZu1bNOan\nrjP402oeJVA6z/iwvOUbN0QmhOPZm5w9COQA32DMHcsC7i/lng1AK6VUc6WUD0YCZlp1qZRqCwQD\n6wu0BSulfPO/DgWuAMybR7rLmv/AIRtl3ppdBX2ecnh325NP8/ZPe03tLevX5rHrzI/6hRCiOnpg\nYGeerjWF3/NsFNnWVlh4L8R/6frAhHAwu5IzrfVZrfXk/KdU3bTWT2utz5ZyjwV4AFgJ7ATmaq23\nK6WmKqUGF7h0NPC11rpgCekojKd1W4DVwLSCqzzdau+P8Nur5vbaDeDmj8DD06Hd5ViM4UyLtXCF\nbY/84Uw/b8f2J4QQlVVtXy/+PSyWu3Mf49e8Djau0LD4ftj4mctjE8KR7N1b80dgZP5CAJRSwRgJ\n1XUl3ae1Xk6RZela62eLvH7Oxn3rAFvfee51OhG+nWBuVx4w4iMIdPyWIu/8vJddKZmm9vt6tySm\nSbDD+xNCiMqsb9sG9O/UnPFbHuUD3qCP55YiV2j4bhJYLdDtbrfEKERF2TusGXohMQPQWp+kpu0Q\nkJcL8+6E8+aq/PR92iia6GBbjpxi+pr9pva24YE8dG0rh/cnhBBVwbM3tsOvVm0m5D7CqjzTdGTD\nskfgz5muDUwIB7E3ObMqpZpceKGUakZFd7KtalY9B4l/mdtb9oMrbWwKXEFZuXk8Om8LeUWGM708\nFK+O7ISvlwxnCiFqpvqBvjwzKIocvJmY+zAr82JtX/j947D+PdcGJ4QD2JucPQOsVUrNVkp9AfwC\nOH7me2W1axmsf9fcHtgIhs8ED8fv9vLGj3vYd+yMqf2Bay4jOqKOjTuEEKLmGBnbmJ4tQsjFi/tz\nJ7Esr7vtC1c+DWvfdG1wQlSQvQsCVgCxwG6MkhePAuedGFflceJvWDjR3K48YeQnUNvxVfk3HjrB\nzN8OmNqjI4K4v+9lDu9PCCGqGqUULw/rgI+XBxa8mJT7IEvyetm+eNUU+NXGQi4hKil7t2+6B/gJ\nIyl7DJgNPOe8sCoJSzbMGwfZ5vpiXPscNOnp8C7P5+Tx2Lyt6CKDxt6eitdGdsbbU/ZkFkIIgBb1\nA3gw/wNrHp48nPtPFuZdYfvin1+ANf91YXRClJ+9v+kfAroBh7TWfTEKwpo3eKxuVj4DRzeb21tf\nb2xq7gT/XbGLv9PNVUr+dW1r2oSXef95IYQTKKUGKqV2K6X2KaUmF3PNLUqpHUqp7Uqpr1wdY01x\nb++WtGoQABgJ2qO5E5lnKWaB1pqX4ecXMX36FaKSsTc5y9JaZwEopXy11ruA6l39NGEBbJhlbq/b\nBIbNMPZNcrD1+4/z6bqDpvZOkXW59+oWDu9PCFF2SilP4D3geqAdMEYp1a7INa0w5uVeobVuD/zL\n5YHWED5eHky7ucPFH8lWPHjCMoE5lr62b/j1FWOBlyRoohKzd0PGRKVUXWAR8KNS6iRwyHlhuVn6\nPlgyydzu4Q0jP4Vajq8vdjbbwuPzi9brAV8vD14b2QkvGc4UorLoDuzL31oOpdTXwBAK72IyHngv\nv+wQWmsbe70JR+naNISxPZoy+w/j15LGg6ctd2PBk394rTLf8PubRh20AS865YO2cLPzpyDuYzh3\nHOo2hfqtIbQ1BDasMv+97UrOtNbD8r98Tim1GqgDrHBaVO6Uex7m3g455pWSXPcyRHR1SrcvL99J\n4knzGovHr2vDZfmP7IUQlUIEcKTA60SgR5FrWgMopX4HPIHn8hdWCSd5fGAbftiRQmpGNmAkaP9n\nuRNvbx9G6+XmG9a/ayRoA6dVmV/Ywg4ZyfDxdXDqsPmcTyCEtoL6bYw/Q9sYSVtIc/D0dn2sJbD3\nydlFWutfnBFIpbH8cTi23dzebih0H++ULn/bm8aXf5r/R+rWLJg7r2julD6FEE7lBbQC+gCNgV+V\nUh0KFvMGUEpNACYANGnSpOh7iDII8vPm+cHR3PfFxgKtisnnb6Nx4wCuTJ9rvunP940E7fpXnFIS\nSbhY7nn4+lbbiRlA6IHmTAAAIABJREFUTiYkbzKOgjy8IKRlgcQt/0lbaCvwdc9c7zInZ9Xa5q8g\nfra5PaQFDH7HKZ+uMrJyeWL+VlN7LW9PXh3ZCU8P+UQnRCWTBEQWeN04v62gROBPrXUu8LdSag9G\nsrah4EVa65nATIDY2FiZBFVBA6PDGdAujB92pBZoVYxNHMK6rnVptN3GjgEbPjR2gLnxzZqRoGUk\nw7kTENzUbYmHU2gNix+A5Piy32u1QPpu49i1tPC5oIgCT9kKJG8BYU594irJ2QWpO2DpI+Z2T1+4\n5XPwC3JKty98t4Ojp7NM7U8NakvTerWd0qcQokI2AK2UUs0xkrLRwK1FrlkEjAE+UUqFYgxzmosX\nCoebOiSadfuPcybbUqBVMfbQIH64IhCv318z37TpM7DmweC3waMa7b6iNRzfD4d+h8PrjT8vPFUK\nCDeKqLfo7d4YHWXt65Aw3/Hvm5FkHAfWFG73rWN7iDS4GXhWPLWS5Awg+wzMuwMsNurqDnoFwp2z\nB/tPO1OZtzHR1H55y3qM7dHUKX0KISpGa21RSj0ArMSYT/ax1nq7UmoqEKe1XpJ/boBSageQBzyu\ntT7uvqhrjvA6fjw5sA3/t7jw9JQD6ed4W4/ikT4+sOY/5hs3f2E8QRk6veomaNY8SN0Oh9bB4XVw\naD2cLWYtypkUmDMG7lgCjYvZ/qqq2LUMfppqbvf0hagb4fg+SN8Luecc12f2aUiKM46CPLyhXkuI\nGgzXPFPut5fkTGtY+jCk7zGf6zgautzulG5Pncth8rfbTO0Bvl78b0RHPGQ4U4hKS2u9HFhepO3Z\nAl9r4JH8Q7jYbT2asjA+iU2HC5fjnPHLfm6adD+tPDyNemdFbf3aSND6PQt1Glf+JM2SYwzjXXgy\ndvhP20XTi5N7Fr4cCXetMJ4AVUWp22FBMfPBB78DnUYZX1utxhOw9D2XjrT8P4tLYMvDmgtpuyCy\n6BqhspHkbNNnsM3GRNH6beHG1502pjxlyXbSMrNN7f++IYrGwf5O6VMIIWoCDw/Ff4Z35Ia3f8Ni\nvTSVLzdP89S325h772N4eHgZ9c6KSphvHB7exrys4ObGvOOQ/D+DmxvtXr6u+wtdkH0GEjfkPxlb\nb3xtMU+LKZPzJ2D2cLh7pZGQViVn02HOaCPJLOqKhy4lZmDMJ6wbaRyX9St87bkTxpO19D35c8/2\nQtpuOHUItLV8sYW2Lt99+Wp2cnZ0Kyx/wtzu7Q8jPwMf58z5WpFwlMWbk03tfdrUZ1S3SBt3CCGE\nKIs24YHc17sl767eV6g97tBJ5mw4zG1XPmwkYD8UM/RkzTWGw47vs3FSGYlMcDNz4hbS3HET7c+d\ngMN/XHoylrwZdJ5j3rugjMT/b+++46usz/+Pv64sQsKGsMIWFMLGgCwnoiIg4ATrrNbqr7Zaa4e2\n7vZbba2tba111NZWLShDBREXw83eYcgqITLCHiH78/vjPmiSc0IG5+Sc5Lyfj8d5mHzu+5xz3THc\nuc5nXB8vQfvuHEhqFvzXD4XCfK/sVaCVmd0uhhEPVf61kppBh7O8R0kFubB/c+letr0bvFqogaZB\nlXSKPZHRm5zlHvLmmRX5914x5k/QsntI3nbf0Tx+OWONX3ujxDgev7wPpno7IiJBcecFXXln9U6/\nLfEen72eC3u0otXQO70yCnN+XsVXdnAo03ts+8T/cHJKiWTNl7yd+DqpWfkjMoe/9nrFTvSM7ckI\nfF5VxNbz5pR1HAodhngJ5SuXw4Ftpc/buwFeuxpueCtkHRNB4xzMvtdLWstK6Q5XvBicIen4RGjV\n03uUVFzs/b/f+5UvWTuRvG3wCt+Ct0jgFERncuYcvP1D2B9g8dSAG0t3hQb1bR0PvLWGfcfy/Y49\nfFlPWjdODMn7iohEo8T4WH4zoRfXvrCwVPuRvEIembmWv33nTBh8u7e67p2fBO+Nj2V7j8yF/sfq\nNSqdrDVo6Y3ibP/cP2GqjoSGXg9Qx6HQYSikDvAfgr1+BvzjYv+5VjsWw+s3wqT/RlxR1lIWPe9N\nSSqrflOYNDlk1RW+ERPjG/LuCN0uLH3s2D4vWWt8anULozM5W/Q8ZLzl396qN4x6ImRvO3PVTmav\n3uXXPjKtFRP6p4bsfUVEotXQ01pwdXo7Xl9SemX87NW7+CBjNyPTWsHAW705QstfhX1feR/cjx8I\nTUB5h2HnSu8RDEktoOMQ6DjMS8ha9aq416hZF7huGvxrtBdPSZs+gLd+AOP/Hpl13zbPgzn3+bfH\nxHllr5qFuXB7cnNIHnLKLxN9ydmOpfBegDkGCQ3h6pchvn5I3nbPkVwefMt/OLNJUjy/mdBLw5ki\nIiFy/6U9+GjdHr9RiwfeXMPgLs1omBgPnc/xHiccPwD7t8KBrV6ytn+b998DW+HIzpq9gJIad/Al\nY76esRbdqrdwrU0fmPgavHKF//SeVVO8pO/i30TW1lb7NnvTkQLNuxv1ROn/f7VcdCVnOfvhjZu8\niZ5ljfuLV5skBJxz3D99NQdz/N/3sXG9aNlQw5kiIqHSJCmBB8emcdfkFaXadx3O5Q/vb+Thy3r6\nP6l+U0ht6g0LlpWf4w1BnkjW9m/99uuD26u/wi+QFmd82zPWYYi32jBYOp8NV/7Dm1hfNuYvn4EG\nKTD8x8F7v1Nx/CC8do03X7ysgbd6jzokepKz4mJ48w44FGBlx6DboOcE//YgmbYsiw/X+ddRGd27\nDWP7tg3Z+4qIiOeyvm2ZviyLBRuzS7W//MU2xvVrS/8OTSv/YglJ0CrNe5RVVOAlaKV63U4kb9sC\nL0I7wWK8oucnhig7DIHkFpWPqzp6jPUWwc38kf+xDx/2etAGXB/aGCpSXATTbvGGnMvqdLa3eX0d\nEz3J2ea5sHGOf3vb/nBRgGKEQbLrUC6PzPTfSL1FgwQeG98rZO8rIiLfMjN+Pb4XF/3xY44XfDss\n5hzcN301M384nPjYIMyxivVViA80ElNc7A2JftPjtgWOZkPD1l4i1n5Q6CezB3Lmjd4ChrmP+R+b\n+SNIag7dL635uE744EHY9KF/e9PO3jyzSF68UE0RONsvRLpd6E1wjC9R4DWxMVz1r5AWE/zb/E0c\nyS30a//NhN40S04I2fuKiEhp7Zslcc9I/+Kg63cd4YVPamDr05gYaJzqDScOuAEufBjGPwMjHvD+\nRoUjMTvh7J/AWbf7t7timHqzV94jHJa/Cl/81b89oaG3MrO21GWrouhJzgD6TYLvzfXG8MFL1pp2\nCtnbFRQVM2uV/8TRCf1Tubhn65C9r4iIBHbzsE70SvVPgp7+8Cu27Q1QaT5amMHFv4VeV/ofK8yF\n1ybCLv9FbSG1/UuYdXeAA+bNlQtRPdJIEF3JGUDLHl6CdsU/Qt5N+/nmfewvszooMT6Gh8YGmKcg\nIiIhFxcbw+OX96Hs9sV5hcXcP2M13raoUSomBsY/C6eN8D+Wdyhw8dpQOZgJU66DIv+6oIx8BE6/\nuGbiCJPoS84A6jWA3gE+HQTZrJX+WzSN6N6KJkkazhQRCZdeqY25Zbh/PazPN+9j2rKsMEQUQeIS\nvHlcqWf6Hzu6G/4zwZsnF0r5x+C/k7x5cGX1mQhDAyxeqGOiMzmrAXmFRcxZ619wdkyfNmGIRkRE\nSvrxyNNJbeJf1/LX72Sw7+hJVlRGg3oN4No3Am/evX8LvHoF5B72PxYMxcUw43bYvdr/WLuBMPbp\nyKq9FiJKzkLk4417/RYCJCfEcn73lmGKSERETkhKiOPXE/xXzB/MKeDuKSv4+mAFG1vXdcnN4brp\n0CjA7jU7V8KU70BhCJLYBU/Aurf92xu2hWte8fa7jAJKzkJk1ir/Ic2Raa1IjA/CZqwiInLKzj+j\nJZcFqDX5yVd7Of/J+Tz53gaO5vmvto8aTdp7CVpiE/9jWz+G6d/zapAFy9oZsCBAzbK4+jDpNa/k\nSJRQchYCx/OL+CBjt1+7Cs6KiESWB8em0bi+f52svMJi/jpvE+f9fj6vLdxOYVEQq/7XJi27w3fe\n8BKksjLegtn3esXiTtXXK2DGHYGPjf+bV5M0iig5C4F5G/aQk1/600SjxDjO7pYSpohERCSQFg3q\n8Uig7Zt89h7N4/4Zqxn950/5eGOIJ8JHqvaD4Jr/eJuLl7XkJZh/ihX6j+yGyddCYYCh5HN+Br0u\nP7XXr4WUnIXAzACrNC/p1ZqEOP24RUQizfj+qfzpmn4nLQy+YfcRbnhpETe+tIiNu4/UYHQRottI\nGPe3wMcWPA6LXqje6xbmeSUzDgdYJdt9DJx3X/Vet5ZTthBkR/MKmbvefx9NDWmKiESu8f1TmXfv\neXz/nC4knGQbpwUbs7nkTx9z/4zVZB+JslWdfa+Bi/8v8LHZP4U106v2es7BzLthxyL/Y616wYTn\nvNprUSg6rzqEPszYTV5h6bkJzZMTGNKleZgiEhGRymhcP577Lu3BRz8596Rlj4odvLZwO+c/OZ9n\n5m0ityCIk+Ij3ZAfwPAfBzjgYPptsHle5V/ri7/Cytf825NawKT/eiU9opSSsyALNKQ5qndr4oKx\noa6IiIRc+2ZJ/PXaAUy7Yyj9OwRYqehzNK+Q37+3gRF/WMBbK7IoLo6S3QVGPAT9r/NvLy7whiiz\nllX8Ghvfh/cf8G+PiYeJr0KTDqceZy2mjCGIDuUU8PFX/hNGx/bRkKaISG1zZsemTL9jKH+Z1J92\nTQOsVvTJOnicuyavYMLfPmPxtv01GGGYmMGYp+GMAFsg5h+FV6+EvZvKf372Bph2CxAgmR3zR+gw\nOGih1lZKzoLovbW7KCgq/cvWqlE9BnZqFqaIRETkVJgZY/u25cN7zuUXo7rTsF6AFYs+K3cc4qq/\nf8Edryzlf/vq+CbqsXFw5UvQYaj/sZx93jZPh3cGOLYfXrsG8gLsMDD4/8GA64Mfay2k5CyIZgYo\nPDu6d1tiyu6wKyIitUpifCy3n3sa8396HtcP7kjsSe7r767ZxYVPLeDXszI4lFNQg1HWsPj63tyw\nlgFKkRza7m2UfvzAt21FBfDGTXBgq//5p42AkY+FLNTaRslZkOw9msfnm/f5tY/tq700RUTqiuYN\n6vHY+F68d/fZXHCS7fgKihwvfrqVc5+cxz8/20pBXS1iW78JXD898ByxPRnw2kTIz/G+n3MfbF3g\nf17zbl4vXGz5vZLRJqTJmZldYmYbzGyTmf0iwPE/mtkK32OjmR0scexGM/vK97gxlHEGw7trdlFU\nZjJou6b16de+/MmkIiJSO3Vt2ZCXbhrIq7eeRY82jco972BOAY/MzOCiP37M+2t34YJRTT/SNGwN\n17/prbIsK/NLmHqzVwdtcYBaaImNYdJkL8mTb4QsOTOzWOAZYBSQBkwys7SS5zjnfuyc6+ec6wf8\nBZjue24z4CHgLGAQ8JCZNQ1VrMEQaJXmmD5tMdOQpkhdU4kPnjeZWXaJD5+3hiNOCb1hXVsw64fD\n+d0VfWjZsF65523de4zb/rOUic9/yZqsQzUYYQ1pfhpcNw0SGvof2zjH2+apLIuBK/8JLbqGPr5a\nJpQ9Z4OATc65Lc65fGAyMO4k508C/uv7+mLgA+fcfufcAeAD4JIQxnpKdh3KDbhC52R1ckSkdqrM\nB0+fKSc+fDrnXqzRIKVGxcYYVw9sz7x7z+NHI7qRGF/+n9aFW/cz9q+fcs/rK9h5KMB2RbVZ235e\nGYzY8ndaKOXi/4OuI0IbUy0VyuQsFcgs8f0OX5sfM+sIdAbmVvW5keCd1Tv99n3t0iKZnm3L7+oW\nkVqrqh88JUok14vjnpGnM//e87liQDvKGzhxDqYvy+L8J+fz1PsbOJZXWLOBhlKXc+HyF4AKRo0G\n3ABn3V4jIdVGkbIgYCIw1TlXpTLLZnabmS0xsyXZ2eHbkDbgkGZfDWmK1FGV/fB4hZmtMrOpZta+\nZkKTSNC6cSJ/uLovM+8czuAu5ZdSyi0o5s9zN3H+k/P5fPPeGowwxHqOh9F/KP94h6Fw6R8oN3uV\nkCZnWUDJG1I7X1sgE/l2SLPSz3XOPe+cS3fOpaekpJxiuNWTuT+HFZkH/drHakhTJJrNBDo55/rg\nTct4OdBJkfIBU0KjV2pj/vu9wbxwQzpdWiSXe96eI3nc9NJiPszYXYPRhdjAW+C8+/3bG3eAa/4D\ncZUc+oxSoUzOFgPdzKyzmSXgJWBvlz3JzLoDTYEvSjS/B1xkZk19CwEu8rVFnFmr/IvsdW/dkG6t\nAkyKFJG6oMIPj865fc65E7tivwicGeiFIuEDpoSWmTEyrRXv/fgcHh6bRpOk+IDn5RcVc/srSwOO\nxNRa5/4MzvkZWKz3fbPT4NopkBxgVaeUErKiIs65QjO7Ey+pigVecs6tNbNHgSXOuROJ2kRgsiux\nvtg5t9/MHsNL8AAedc5F5J4YswIUntVCAJE67ZsPnnhJ2UTg2pInmFkb59yJT26XAetqNkSJNPGx\nMdw0rDMT+rfjr/O+4l+fb/PbUaaw2HHX5OXkFhRxVXodGAk3gwt+Cf2u9XYNaJkGCUnhjqpWCGnF\nN+fcbGB2mbYHy3z/cDnPfQl4KWTBBcHm7KOs/dp/C4ox2ktTpM6q5AfPH5nZZUAhsB+4KWwBS0Rp\nnBTPL0encf3gTtw9ZTnLtpeeFlPs4KdTV5FbUMT1QzqFJ8hga9bZe0ilqRzvKZi10n9Is0+7xnQ6\nydwCEan9Kvrg6Zy7D7ivpuOS2qND8yT+c8tZfO/fSwLuLvPAW2vJyS/i++eeFoboJNwiZbVmreOc\nC7iXpoY0RUSkMpLrxfHSTQM5/4zA8w1/++56/vjBxrq5q4CclJKzatqw+wib9hz1ax+tIU0REamk\nxPhYnrs+nVG9Wgc8/vRHX/Hbd9crQYsySs6qKdCKmvSOTUltUj8M0YiISG2VEBfDXyb15/L+gWut\nP//xFh54aw3FxUrQooWSs2pwzgUsoaEhTRERqY642BievKov157VIeDxV77czk+nrqKwqLiGI5Nw\nUHJWDauzDvG/fTml2mIMLlVyJiIi1RQTY/xmfC9uGR54ZeO0ZTu4a/IK8guVoNV1Ss6qIVCv2Vmd\nm9OyYWIYohERkbrCzPjV6B786IKuAY+/s3ond7yylNyCKu12KLWMkrMqKi52zAow32xsXy0EEBGR\nU2dm3HPRGfz8ku4Bj3+0fg+3vryEnPw6tGG6lKLkrIqWbT/A14dyS7XFxRiXlLPSRkREpDruOO80\nHh6bFvDYp5v2csM/FnE4t6CGo5KaoOSsigINaQ7r2oJmydrEVUREguumYZ353RV9MPM/tuR/B7ju\nxYUcOJZf84FJSCk5q4KiYsc7q/2TMw1piohIqFw9sD1PT+xPbIx/hrZqxyEmPv8l2UfywhCZhIqS\nsypYuHWf3z+AhNgYLurZKkwRiYhINLisb1ue/c4AEmL9/2xv2H2Ea577gp2HjochMgkFJWdVMDPA\nXprnnpFCo8T4MEQjIiLR5KKerXnhxnQS4/3/dG/Ze4yr/v4F28uUeZLaSclZJRUUFfPuGg1piohI\n+Jx7egov3zyI5IRYv2M7Dhznquc+D7i1oNQuSs4q6bNNezmYU3pVTGJ8DCO6twxTRCIiEo3O6tKc\nV783mEaJcX7Hdh/O45rnvmDdzsNhiEyCRclZJQUa0hzRoxXJ9fz/cYiIiIRSv/ZNmHzbEJoHqBSw\n71g+E5//khWZB8MQmQSDkrNKyC0o4v21u/zax2q7JhERCZO0to2Y8v3BtGpUz+/YoeMFXPfiQhZu\n2ReGyORUKTmrhI83ZnMkr3Ql5gb14jjvDA1piohI+HRt2ZDXvz+E1Cb1/Y4dzSvkxn8u4uON2WGI\nTE6FkrNKmBmg8OxFaa1IjPefkCkiIlKTOjZP5o3bh9C5RbLfsdyCYm59eQkfZOwOQ2RSXUrOKpCT\nX8iHAX6px/TVkKaIiESGtk3qM+X7gzmjVUO/Y/lFxdz+ylLeDrAvtEQmJWcVmLt+D8cLikq1Na4f\nz/CuKWGKSERExF/LholMvm0wvVMb+x0rKnbcNXk5ry/JDENkUlVKziowM8AnjVG9WpMQpx+diIhE\nlqbJCbz6vbNI79jU75hz8LOpq/j3F9tqPC6pGmUYJ3Ekt4B5G/wnUo7po8KzIiISmRolxvPvWwYx\nrGvzgMcffGstz87fjHOuhiOTylJydhIfZOwmv7C4VFuLBgkM7tIsTBGJiIhULCkhjn/cOJALyimU\n/sSc9Vz8p495fXEmuWWm7kj4KTk7icBDmm2IC7DxrIiISCRJjI/l79edyejegRewbdx9lJ9NW8Xw\nJ+by54++Yt/RvBqOUMqjLKMcB3Py+eSrvX7t2ktTRERqi4S4GJ6e2I/LB6SWe87eo/k89cFGhj4+\nl/umr2bTniM1GKEEouSsHHPW7KKwuPR4fOtGiQEnWYqIiESquNgYnryyLzcM6XjS8/IKi/nvou1c\n+NTH3PzPRXy2aa/mpYWJNoYsx6wAhWdH92lDTIyFIRoREZHqi4kxHh3Xi/PPaMmzCzazaOv+k54/\nb0M28zZk06NNI24d3pmxfduqSkEN0k86gOwjeXy+WUOaIiJSt5zfvSWvf38Ib985jHH92hJbQYfD\nup2H+ckbKxn+xFyembeJA8fyayjS6KbkLIB31+ykzIgm7ZvVp287/8J+IhJ9zOwSM9tgZpvM7Bcn\nOe8KM3Nmll6T8YlUpE+7Jjw9sT+f/Ox8vn9OFxomnnwgbc+RPH7/3gaGPP4Rv3pzNVuyj9ZQpNFJ\nyVkAs1b6D2mO6dMWMw1pikQ7M4sFngFGAWnAJDNLC3BeQ+AuYGHNRihSeW2b1Oe+S3vwxX0jeGhs\nGu2b+W+gXlJuQTGvfLmdEU8t4NaXl/Dlln2alxYCSs7K2HnoOIu2+Y/Fj1XhWRHxDAI2Oee2OOfy\ngcnAuADnPQY8AeTWZHAi1dGgXhw3D+vM/HvP59nvDODMCha/OQcfrtvNxOe/ZOxfP+XN5VkUFBWf\n9DlSeUrOyngnwEKALinJ9Gjjv5msiESlVKDkBoU7fG3fMLMBQHvn3Ds1GZjIqYqNMUb1bsO0O4Yy\n/f8N9RbCVTBotCbrMHdPWcHZT8zj7ws2cyinoGaCrcO0WrOMmQGSs7Ea0hSRSjKzGOAp4KZKnHsb\ncBtAhw4dQhuYSBUN6NCUAdc2JXN/Dv/6fBtTFmdyNK+w3PN3Hc7l8XfX8+ePvuLq9PbcPKwTHZsn\n12DEdYd6zkrYvi+HlZkH/drH9g1cXVlEolIW0L7E9+18bSc0BHoB881sGzAYeDvQogDn3PPOuXTn\nXHpKSkoIQxapvvbNknhgTBqf33cBvxrdg9QmJ5+XlpNfxL8+38Z5T87n9v8sZcm2/ZqXVkXqOSth\n1mr/7Zq6t25I15Ya0hSRbywGuplZZ7ykbCJw7YmDzrlDQIsT35vZfOBe59ySGo5TJKgaJcZz69ld\nuGloJ95ds4sXP9nCyh2Hyj3fOZizdhdz1u6ib/sm3DC4I5f0ak1yPaUeFdFPqISZAVZpqraZiJTk\nnCs0szuB94BY4CXn3FozexRY4px7O7wRioRWXGwMY/u2ZUyfNiz93wFe/GQr72Xs4mSdYyszD/KT\nzIM88NYaLunZmgkDUhl6WosK66xFKyVnPpv2HGXdzsN+7VqlKSJlOedmA7PLtD1Yzrnn1URMIjXN\nzEjv1Iz0Ts34375j/POzbby+JJOc/KJyn5OTX8T05VlMX55Fq0b1GN8vlQkDUuneulENRh75NOfM\nZ9Yq/yHNvu0a06F5UhiiERERqT06Nk/m4ct68sUvRvCLUd1p3SixwufsPpzHcx9v4ZI/fcKlT3/C\ni59sYc8RVZ4B9ZwB4Jxj5kr/5ExDmiIiIpXXOCme2889jVuGd2b26p288MkW1mT5j0qVlbHzMBnv\nHOb/Zq/j7G4pXD4glYvSWlM/IbYGoo48IU3OzOwS4Gm8eRkvOuceD3DO1cDDgANWOueu9bUXAat9\np213zl0WqjjX7zrC5uxjfu2X9tYqTRERkaqKj41hXL9ULuvblqX/O8C0ZVm8s+prDueWX4oDoNjB\ngo3ZLNiYTXJCLKN6t+Hy/qkM7tKcmCianxay5KzEFicj8Yo0Ljazt51zGSXO6QbcBwxzzh0ws5Yl\nXuK4c65fqOIrKVCv2cBOTWlbwXJhERERKV/JeWkPjU1j3vo9TFuWxfwNeygsu4l1Gcfyi5i6dAdT\nl+6gbeNExvVP5fL+qXRrVfcrKISy5+ybLU4AzOzEFicZJc75HvCMc+4AgHNuTwjjCcg5x8wA883G\naCGAiIhI0CTGez1ho3q3Yd/RPGat2sn05VkB64uW9fWhXJ6dv5ln52+md2pjJvRP5bJ+bWnRoF4N\nRF7zQpmcBdri5Kwy55wOYGaf4Q19Puycm+M7lmhmS4BC4HHn3JuhCHLVjkNk7j9eqi3GYFTv1qF4\nOxERkajXvEE9bhzaiRuHdmJz9lFmLMtixvIssg4er/C5q7MOsTrrEL+ZvY5zT09hQv9URqa1IjG+\n7sxPC/eCgDigG3AeXpXtj82st3PuINDROZdlZl2AuWa22jm3ueSTg7H1SaAhzSGnNadlw4pXmoiI\niMipOS2lAfdefAb3jDydRdv2M2NZFrNX7+TISbaKAigqdsxdv4e56/fQsF4cl/Zuw+UDUhnYqVmt\nn58WyuSsoi1OwOtNW+icKwC2mtlGvGRtsXMuC8A5t8VXYbs/UCo5c849DzwPkJ6eXuW9IYqLHbMC\n7KWpIU0REZGaFRNjDO7SnMFdmvPIuJ58kLGbGcuzWLAxm6IK5qcdyStkypJMpizJJLVJfS4fkMqE\n/ql0SWlQQ9EHVyiTs5NuceLzJjAJ+KeZtcAb5txiZk2BHOdcnq99GPC7YAe4dPsBdh0uXVMlLsa4\npKeGNEVERMIlMT6WsX3bMrZvW7KP5DFz5dfMWJ7F6qzyt4s6Ievgcf4ydxN/mbuJAR2acHV6e8b0\nbUuDWrRtVMiFw70uAAALZElEQVQireQWJ+8BF5lZBlAE/NQ5t8/MhgLPmVkxXqHcx0uu8gyWQEOa\nZ3drQdPkhGC/lYiIiFRDSsN6fHd4Z747vDNf7T7C9OVZvLk8i52HKi5Yu2z7QZZtP8gjMzMY3acN\nV6e3Z2CnpphF9rCn1ZWd4tPT092SJZXfV7iwqJjBv/2IvUfzS7X/4aq+XHFmu2CHJyIhYGZLnXPp\n4Y4jGKp6DxOJZkXFjoVb9jFtWRZz1uzk2Em2jCqrc4tkrkpvx5UD2tGyEjsZhMrJ7l+1p48vyBZu\n3e+XmCXExTCyZ6swRSQiIiKVERtjDO3agqFdW/DYeG9+2rRlWXz6VTYVTE9j695j/G7OBv7w/kbO\nOz2Fq9LbM6JHS+JjI2dHy6hNzgINaZ53egqNEuPDEI2IiIhUR1JCHOP6pTKuXyp7Dufy9sqvmbp0\nB+t3HTnp84qKHR+t38NH6/fQokECE/qncnV6+4gochuVyVl+YTFz1u7ya9demiIiIrVXy0aJ3Hp2\nF24Z3pnVWYd4fUkmb634miMVbBu192g+L3yylRc+2Ur/E4sI+rShYZg6bKIyOfts014O5hSUaqsf\nH8uIHi3LeYaIiIjUFmZGn3ZN6NOuCb8ancacNbt4fUkmn2/eV+Fzl28/yPLtB3l0ZgaX9m7DNQNr\nfhFBVCZngYY0R/RoSVJCVP44RERE6qzE+FjG909lfP9UMvfn8MaSTKYu3cHXFaz2PF5QxLRlO5i2\nbAedmidxVXp7rhjQjtaNQ7+IIHJmv9WQ3IIi3s/Y7deuIU0REZG6rX2zJO656Aw++fkF/Pu7gxjd\npw0JlVgIsG1fDr9/bwNDH/+I7/5rMXPW7CS/sDhkcUZdV9H8DdkcLbMlRMN6cZx7ekqYIhIREZGa\nFBtjnHN6CuecnsKBY/m8uSKLKYszK1xEUOz4Zsuo5sm+RQQD23N6kBcRRF1yNmuV/5DmyJ51a8NU\nERERqZymyQncPKwzNw3txNqvDzNlcSZvrsiqcBHBvmP5vPjpVl78dCv92nuLCMb2Dc4igqhKznLy\nC/lo3R6/dg1pioiIRDczo1dqY3qlNuaXo3vw3lpvEcFnmypeRLAi8yArMg/y6Ky1XNrb24ngrM7N\nqr2IIKqSsw/X7eF4Qekqwk2S4hnetUWYIhIREZFIkxgf+03ttMz9ObyxdAdTl2RWuIggt6CY6cuy\n+DBjN4t+eWG1R+WiKjmbFWCV5qherSOqKrCIiIhEjvbNkrhn5OncNaIbn2/ey5TFmby/djf5ReUv\nCBjfP/WUpktFTXJ2OLeA+Ruy/drH9NGQpoiIiJxcbIxxdrcUzu6WwsGcfN5a8TVTFmeSsfOw37lX\np7c/pfeKmi6jBglxvHLrWdwwpCMtGiQA0KJBPQZ3aR7myERERKQ2aZKUwI1DOzH7rrOZ9cPh3DCk\nI40Svf6utDaN6JXa+JReP2p6zmJijEGdmzGoczMeHJPGl1v2s+9YHrExNVfxV0REROqWE4sI7r+0\nB+9n7CYx7tT7vaImOSspLjaG4d20CEBERESCIzE+lsuCVP0haoY1RURERGoDJWciIlVkZpeY2QYz\n22Rmvwhw/HYzW21mK8zsUzNLC0ecIlI7KTkTEakCM4sFngFGAWnApADJ12vOud7OuX7A74CnajhM\nEanFlJyJiFTNIGCTc26Lcy4fmAyMK3mCc67k2vpkwNVgfCJSy0XlggARkVOQCmSW+H4HcFbZk8zs\nB8A9QAJwQc2EJiJ1gXrORERCwDn3jHPuNODnwK8CnWNmt5nZEjNbkp3tXyRbRKKTkjMRkarJAkqW\n/27nayvPZGB8oAPOueedc+nOufSUlJQghigitZmSMxGRqlkMdDOzzmaWAEwE3i55gpl1K/HtaOCr\nGoxPRGo5c65uzFM1s2zgf1V4Sgtgb4jCiSTRcJ3RcI2g6wyko3OuxruczOxS4E9ALPCSc+43ZvYo\nsMQ597aZPQ1cCBQAB4A7nXNrK3jNqtzD9LtQt0TDdUbDNUKQ7l91JjmrKjNb4pxLD3ccoRYN1xkN\n1wi6TvlWtPyMdJ11RzRcIwTvOjWsKSIiIhJBlJyJiIiIRJBoTs6eD3cANSQarjMarhF0nfKtaPkZ\n6Trrjmi4RgjSdUbtnDMRERGRSBTNPWciIiIiESfqkjMzu8TMNpjZJjP7RbjjCQUza29m88wsw8zW\nmtld4Y4plMws1syWm9mscMcSCmbWxMymmtl6M1tnZkPCHVMomNmPfb+va8zsv2aWGO6YIpHuYXVL\nXb9/QXTcw4J9/4qq5MzMYoFngFFAGjDJzNLCG1VIFAI/cc6lAYOBH9TR6zzhLmBduIMIoaeBOc65\n7kBf6uC1mlkq8CMg3TnXC69+2MTwRhV5dA+rk+r6/Qvq+D0sFPevqErOgEHAJufcFudcPt62KuPC\nHFPQOed2OueW+b4+gvcPITW8UYWGmbXDq8D+YrhjCQUzawycA/wDwDmX75w7GN6oQiYOqG9mcUAS\n8HWY44lEuofVIXX9/gVRdQ8L6v0r2pKzVCCzxPc7qIP/4Esys05Af2BheCMJmT8BPwOKwx1IiHQG\nsoF/+oY+XjSz5HAHFWzOuSzgSWA7sBM45Jx7P7xRRSTdw+qWun7/gii4h4Xi/hVtyVlUMbMGwDTg\nbufc4XDHE2xmNgbY45xbGu5YQigOGAA865zrDxwD6tw8IzNritcD1BloCySb2XXhjUrCrS7fw6Lk\n/gVRcA8Lxf0r2pKzLKB9ie/b+drqHDOLx7upveqcmx7ueEJkGHCZmW3DG965wMxeCW9IQbcD2OGc\nO9FrMBXvRlfXXAhsdc5lO+cKgOnA0DDHFIl0D6s7ouH+BdFxDwv6/SvakrPFQDcz62xmCXgT9t4O\nc0xBZ2aGN76/zjn3VLjjCRXn3H3OuXbOuU54/y/nOufqVG+Lc24XkGlmZ/iaRgAZYQwpVLYDg80s\nyff7O4I6Nmk4SHQPqyOi4f4FUXMPC/r9Ky4oYdUSzrlCM7sTeA9vNcVLzrm1YQ4rFIYB1wOrzWyF\nr+1+59zsMMYk1fdD4FXfH+MtwM1hjifonHMLzWwqsAxvpd5yoqeieKXpHqZ7WC1Vp+9hobh/aYcA\nERERkQgSbcOaIiIiIhFNyZmIiIhIBFFyJiIiIhJBlJyJiIiIRBAlZyIiIiIRRMmZ1Flmdp6ZzQp3\nHCIiIlWh5ExEREQkgig5k7Azs+vMbJGZrTCz58ws1syOmtkfzWytmX1kZim+c/uZ2ZdmtsrMZvj2\nNMPMuprZh2a20syWmdlpvpdvYGZTzWy9mb3qq96MmT1uZhm+13kyTJcuIiLiR8mZhJWZ9QCuAYY5\n5/oBRcB3gGRgiXOuJ7AAeMj3lH8DP3fO9QFWl2h/FXjGOdcXb0+znb72/sDdQBrQBRhmZs2BCUBP\n3+v8OrRXKSIiUnlKziTcRgBnAot927SMwEuiioEpvnNeAYabWWOgiXNuga/9ZeAcM2sIpDrnZgA4\n53Kdczm+cxY553Y454qBFUAn4BCQC/zDzC4HTpwrIiISdkrOJNwMeNk518/3OMM593CA86q7z1he\nia+LgDjnXCEwCJgKjAHmVPO1RUREgk7JmYTbR8CVZtYSwMyamVlHvN/NK33nXAt86pw7BBwws7N9\n7dcDC5xzR4AdZjbe9xr1zCypvDc0swZAY98myj8G+obiwkRERKojLtwBSHRzzmWY2a+A980sBigA\nfgAcAwb5ju3Bm5cGcCPwd1/ytQW42dd+PfCcmT3qe42rTvK2DYG3zCwRr+funiBfloiISLWZc9Ud\nLRIJHTM76pxrEO44REREapqGNUVEREQiiHrORERERCKIes5EREREIoiSMxEREZEIouRMREREJIIo\nORMRERGJIErORERERCKIkjMRERGRCPL/AVhv31jZsF53AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2rrXKQ0wQ_jV",
        "colab_type": "text"
      },
      "source": [
        "#Prediction in the test set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-wI5i5NUP56-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_pred = model.predict(test_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fRJeXsV9P53D",
        "colab_type": "code",
        "outputId": "f2ccfd22-7178-4b9d-8d22-09fb298c99f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1885, 5)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XxYEUP6wP5_h",
        "colab_type": "code",
        "outputId": "41b865c2-887a-409e-f1a3-1998b79d9d10",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 371
        }
      },
      "source": [
        "from matplotlib.colors import LogNorm\n",
        "plt.hist2d(y_pred.argmax(axis = 1), test_y.argmax(axis =1), bins = 5, density=False, norm = LogNorm())\n",
        "plt.colorbar()\n",
        "plt.xticks([0,1,2,3,4])\n",
        "plt.yticks([0,1,2,3,4])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([<matplotlib.axis.YTick at 0x7f7c36dc42b0>,\n",
              "  <matplotlib.axis.YTick at 0x7f7c36dc4080>,\n",
              "  <matplotlib.axis.YTick at 0x7f7c36df6f98>,\n",
              "  <matplotlib.axis.YTick at 0x7f7c36d24e80>,\n",
              "  <matplotlib.axis.YTick at 0x7f7c36d24780>],\n",
              " <a list of 5 Text yticklabel objects>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAD8CAYAAAA7fRx2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAO5klEQVR4nO3df6jd9X3H8derN4lpWk23mj8kyRa3\niSMI0y5LO/zHhpVFrdU/BlVa/3JmHXWzrKMobBT/KeyfUgqycmmCGy1KacswYpcJtbhC628rxlQW\nXYtxhRBDtRbNj3tf++OcNLcxyfmce8/38z2fe58P+cA9J1+/583R++aT9+f9+XydRACA7r2n7wAA\nYKUg4QJAJSRcAKiEhAsAlZBwAaASEi4AVFKccG3P2H7W9kNdBgQAy9U4M9w7JR3oKhAAWO6KEq7t\nTZKul/T1bsMBgOVrVeF1X5H0BUkXnusC27sk7ZKkmfes+dN16y5eenRTZO6C5VfuXvXrk32H0In5\nNTN9hzBxJy5enjtCj73yf0eSbFjKPf7yo+/L60fniq59+vlj+5LsXMrnLcXIhGv745IOJ3na9jXn\nui7JrKRZSbrowo3Z/qHPTizIafDmlrV9hzBxH3zySN8hdOKdzev7DmHiXr3tRN8hdOLlT/7zz5d6\nj9ePzumJfb9XdO3MJf/T60ywZNp2taRP2P6ZpAck7bD9jU6jAoBCkTRf+I+k9bZnbd/QR6wjZ7hJ\n7pZ0tyQNZ7j/mOTTHccFAEWi6ETKSgqS3kiyq8t4zqe0hgsAU2s4e516YyXcJD+Q9INOIgGARYii\nuUaOmWWGC6B582oj4S6/XicAK0okzSlFQ9O+aAYA026MGS6LZgCwWJF0ghouAHQvp8sFU4+EC6Bt\nkebayLckXABtG+w0awNdCgAaZ80VDtGlAACLN1g0c+nldCkAwGIN+nCLE26vSLgAmjdfPsPtFQkX\nQNOY4QJAJZE118j6fxtRAsB5zMdFQ3QpAMDiRdbxFD/Hji4FAFiswcaHNv6yTsIF0DwWzQCggsSa\nCzNcAKhinhkuAHRvsGjWRiprI0oAOIeWFs3aiBIAzmMuLhqiDxcAFm/MnWb04QLAUszTpQAA3Rsc\nXkPCBYDORdaJ8q29vSLhAmhaIjY+AEAdZuMDANQQMcMFgGpYNAOACiLzTDMAqGHwmPTiVLbe9qyk\nvUn2dhfV2ZFwATTO45yHy04zAFisiJ1mAFANT3wAgAoSM8MFgBoGi2Zs7QWAClb6M83eelv+72c7\nuXVvtvx53xFM3M9v2tB3CJ144e//te8QJu6jt9/edwideHkC9xgsmlHDBYAq2GkGABWw0wwAKmrl\nIZIkXABNS6QT8yRcAOjcoKRAwgWAKthpBgAVtNQWNnIebnut7Sds/8T2ftv31AgMAMoMSgolQ8Pj\nGW3f0EekJTPcY5J2JHnL9mpJP7T9vSQ/7jg2ACgyxjPNpvt4xiSR9Nbw5erhSJdBAUCpQZdCG2cp\nFC3t2Z6x/Zykw5IeSfL4Wa7ZZfsp20+d0LFJxwkAZ3Vq40PJ6FtRwk0yl+RKSZskbbd9xVmumU2y\nLcm21bpg0nECwDnNDx+VPmr0bazmtSS/lPSopJ3dhAMA4znVpbAsZri2N9j+wPDn90r6mKSfdh0Y\nAJQao0uhVyVdCpdI+jfbMxok6G8leajbsACgTGKdnIJkWqKkS+F5SVdViAUAFmUaygUl2GkGoGkt\n7TQj4QJoHgkXACrgAHIAqGgaemxLkHABNC2RTnIAOQDUQUkBACqghgsAFYWECwB1sGgGABUk1HAB\noBJrji4FAKiDGi4AVMBZCgBQSwZ13D7YvknS9ZIukrQ7yX+d7/o2Ch8AcB6TfMSO7T22D9t+4Yz3\nd9p+yfZB23dJUpL/SHK7pM9I+uSoe5NwATQtw0WzklHoPp3xGLHhAxjulXStpK2SbrG9dcEl/zT8\n8/Mi4QJoXlI2JF186uniw7Hr3ffKY5KOnvH2dkkHk7yS5LikByTd6IF/kfS9JM+MipMaLoDmjdGl\ncCTJtkV8xEZJry54fUjShyX9naS/kLTe9h8l+dr5bkLCBdC0wey1ny6FJF+V9NXS60m4AJpXoS3s\nNUmbF7zeNHxvLNRwATRvjBruetuztm8Y8yOelHSZ7Uttr5F0s6QHx42TGS6ApkXWfHkHwhtJ3rVQ\ntpDt+yVdo8EC2yFJX0yy2/YdkvZJmpG0J8n+cWMl4QJo3iT3PSS55RzvPyzp4aXcm4QLoG09LpqN\nixougPalcCy+hjsRzHABNG+MGe7IGm6XSLgAmhZJ8/NtlBQ6Sbhee4FWXXpZF7fuzTu/28Z/0HH8\n2U0vjL6oQX/47b/pO4SJ++OXX+87hOkVSY3UcJnhAmheX8czjotFMwDtY9EMAGowi2YAUE0jJQUS\nLoC2RcpK7lIAgLraSLgsmgFoH4tmAFBJeQ2XRTMAWDQ2PgBAPa1sfCDhAmgfXQoAUIeZ4QJABac7\nEKYebWEAGufBolnJoC0MAJaItjAAqGS+7wDKkHABtK2hPtyRNVzbm20/avtF2/tt31kjMAAo5ZSN\nvpXMcE9K+nySZ2xfKOlp248kebHj2ACgzBQk0xIjZ7hJfpHkmeHPv5J0QNLGrgMDgOVmrBqu7S2S\nrpL0+Fn+bJekXZK0dtVFEwgNAMpMQ7mgRHEfru33S/qOpM8lefPMP08ym2Rbkm1rVq2bZIwAcG7R\nYGtvyWihD9f2ag2S7TeTfLfbkABgTMulD9e2Je2WdCDJl7sPCQDGs5xKCldLulXSDtvPDcd1HccF\nAOXKn/jQq5Ez3CQ/VCsPDAKwMk1BMi3BTjMATZuWTQ0lSLgA2scB5ABQBzNcAKiFhAsAFTRUw+WJ\nDwDaV94WNv07zQBgmrn8APJed5oxwwWASpjhAmhfIzVcEi6AtjW0aEbCBdA+Ei4AVELCBYDuWWN1\nKfSKhAugbdRwAaAiEi4AVLKSE+7x9at06LoNXdy6N/f87b/3HcLEfelLt/YdQjeubOS3bwzvbF7f\ndwjdODCZ21BSAIBaSLgAUEHoUgCAehqZ4XJ4DYDmnXqu2aghjmcEgCUqn+H2ejwjCRdA204fLj71\nSLgAmmbRFgYA1ZBwAaAWEi4AVELCBYAKOC0MACoi4QJAHWztBYBKKCkAQA1sfACAiki4ANA9dpoB\nQEWebyPjknABtI0aLgDUQ0kBAGoh4QJAHcxwAaAWEi4AVMBTewGgjpb6cHlqL4D2JWVjwmz/ge3d\ntr9dcj0JF0DzxnhM+uh72XtsH7b9whnv77T9ku2Dtu+SpCSvJLmtNE4SLoC2ZYxR5j5JOxe+YXtG\n0r2SrpW0VdIttreOG+rIhHuubA8A08LzZUPSxbafWjB2nXmvJI9JOnrG29slHRzOaI9LekDSjePG\nWTLDvU9nZHsAmCZjJNwjSbYtGLOFH7FR0qsLXh+StNH2B21/TdJVtu8edZORXQpJHrO9pTAoAKgr\n6mRBrOijk9clfab0+om1hQ2n5rskafWFvzOp2wLASBXawl6TtHnB603D98YysUWzJLOnpukz6943\nqdsCwGjli2brbc/avmHMT3hS0mW2L7W9RtLNkh4cN0w2PgBo2pgbH95I8q6Fst+6n32/pGs0WGA7\nJOmLSXbbvkPSPkkzkvYk2T9urCRcAG1LJnoAeZJbzvH+w5IeXsq9S9rC7pf0I0mX2z5ku7jJFwCq\nmGwfbmdKuhTOmu0BYFqMUVJYb3tW0t4ke7uL6OwoKQBoWySVlxRG1nC7RMIF0L4pKBeUIOECaF4r\nxzOScAE0r5XHpHNaGIC2jXda2GI3PkwEM1wATRtsfFjBi2bz6+b16w+93cWte/MP+z7VdwgT9/uH\n5/oOoRPH/nf5zSPe3rC67xCmG880A4A6xpjh9oqEC6BtU7KLrASLZgAaNzhLoWSIRTMAWKKVvGgG\nANXkN4/PmXokXADtY9EMACppI9+ScAG0z/Nt1BRIuADaFjWz8YG2MABNsyKnbIi2MABYItrCAKAS\nuhQAoIKGargkXADNo0sBAKoIJQUAqCIi4QJANW1UFOjDBdA++nABoBb6cAGggkSaa6OmQMIF0D4W\nzQCgEhIuAFQQSfMkXACoIFKo4QJA9yIWzQCgGmq4AFBJIwmXnWYAGjc8vKZksNMMAJYgksqPZ2Sn\nGQAsSSMlBRIugMaxtRcA6ogU+nABoBJ2mgFAJdRwAaCCZJwuhV6RcAG0jxkuANQQZW6u7yCKkHAB\ntI3jGQGgokbaworOUrC90/ZLtg/avqvroACgVCRlPkWjbyMTru0ZSfdKulbSVkm32N7adWAAUCTD\nA8hLRs9KSgrbJR1M8ook2X5A0o2SXuwyMAAo1cqimTOincL2X0nameSvh69vlfThJHeccd0uSadO\n4blC0guTD7dJF0s60ncQU4Dv4TS+i9MuT3LhUm5g+z81+E6LLpf0tKS9SfYu5XMXY2KLZklmJc1K\nku2nkmyb1L1bxncxwPdwGt/FabafWuo9kuycRCw1lCyavSZp84LXm4bvAQDGUJJwn5R0me1Lba+R\ndLOkB7sNCwCWn5ElhSQnbd8haZ+kGUl7kuwf8a/NTiK4ZYLvYoDv4TS+i9NW1HcxctEMADAZPEQS\nACoh4QJAJRNNuGwBHrC9x/Zh2yu+F9n2ZtuP2n7R9n7bd/YdU19sr7X9hO2fDL+Le/qOqW+2Z2w/\na/uhvmOpYWIJly3Av+U+Sc30BnbspKTPJ9kq6SOSPruC/784JmlHkj+RdKWknbY/0nNMfbtT0oG+\ng6hlkjPc32wBTnJc0qktwCtOksckHe07jmmQ5BdJnhn+/CsNfrk29htVPzLw1vDl6uFYsavWtjdJ\nul7S1/uOpZZJJtyNkl5d8PqQVugvFs7O9hZJV0l6vN9I+jP8K/Rzkg5LeiTJiv0uJH1F0hck9X+q\nTCUsmqEK2++X9B1Jn0vyZt/x9CXJXJIrNdixud32FX3H1AfbH5d0OMnTfcdS0yQTLluAcVa2V2uQ\nbL+Z5Lt9xzMNkvxS0qNaubX+qyV9wvbPNCg/7rD9jX5D6t4kEy5bgPEuti1pt6QDSb7cdzx9sr3B\n9geGP79X0sck/bTfqPqR5O4km5Js0SBXfD/Jp3sOq3MTS7hJTko6tQX4gKRvFWwBXpZs3y/pR5Iu\nt33I9m19x9SjqyXdqsEM5rnhuK7voHpyiaRHbT+vwQTlkSQroh0KA2ztBYBKWDQDgEpIuABQCQkX\nACoh4QJAJSRcAKiEhAsAlZBwAaCS/wcCaLOZ+xXX/wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dvv2FtmUP5zZ",
        "colab_type": "code",
        "outputId": "695ef846-068e-4d74-c02e-60cff2e47238",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "confusion_matrix(y_pred.argmax(axis = 1), test_y.argmax(axis =1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 70,  20,   4,   2,   6],\n",
              "       [ 47, 397,  18,  12,  12],\n",
              "       [  8,  13, 187,  23,   4],\n",
              "       [ 19,  44,  45, 841,   7],\n",
              "       [ 11,   7,   4,   8,  76]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zyPMMcrjP5vu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "unique_labels = Binarizer.inverse_transform(np.eye(5))\n",
        "report = classification_report(test_y.argmax(axis = 1), \n",
        "                      y_pred.argmax(axis = 1), \n",
        "                      labels = [0,1,2,3,4], \n",
        "                      target_names = unique_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ktHYnbTmP5r2",
        "colab_type": "code",
        "outputId": "915ac95d-6cc4-4d55-a4c2-5727fba9c8fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "print(report)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         cbb       0.69      0.45      0.54       155\n",
            "        cbsd       0.82      0.83      0.82       481\n",
            "         cgm       0.80      0.72      0.76       258\n",
            "         cmd       0.88      0.95      0.91       886\n",
            "     healthy       0.72      0.72      0.72       105\n",
            "\n",
            "    accuracy                           0.83      1885\n",
            "   macro avg       0.78      0.73      0.75      1885\n",
            "weighted avg       0.83      0.83      0.83      1885\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5hQjBhiQUbWN",
        "colab_type": "text"
      },
      "source": [
        "#Model Performance:\n",
        "\n",
        "Now we investigate the performance of the model at classifying examples that were not included in the training or in the dev set.\n",
        "\n",
        "We look at a few metrics: \n",
        "\n",
        "> ***Precision*** which is defined as `1 - false positive rate`. One can interpret precision as the purity or cleanness of the sample. \n",
        "\n",
        "> ***Recall*** which is defined as `true positive rate`. Recall assigned to each class can be interpreted as the completeness of the sample in that class. \n",
        "\n",
        "> ***$F_1$ score*** is the harmonic mean of the precision and recall: $F_1 = 2\\times\\frac{\\mathrm{Precision.Recall}}{\\mathrm{Precision + Recall}}$.\n",
        "\n",
        "\n",
        "By looking at the classification report above, we notice that our model is good at identifying the cmd ,cbsd, cgm, but less promissing at identifying the images in the healthy and the cbb class.\n",
        "\n",
        "Out of all the leaf images identified as cmd, 88\\% are in the true cmd class. Out of all the images in the cmd class 95\\% are correctly identified as cmd. 72\\% of the objects in the healthy class are true healthy object. Furthermore, out of all the true healthy objects, 72\\% are included in the healthy class. \n",
        "\n",
        "Note that the cmd class is dominant class in the sample, and the healthy class is the least common example in the data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lX7crn5UP5n3",
        "colab_type": "code",
        "outputId": "425b19af-5fea-4728-f63c-2dce2b9ae29a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "histogram = plt.hist(test_labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAPM0lEQVR4nO3df4xlZX3H8fdHVhSs8mOZEN3ddFC3\nUqwVdaMotTVgEwV1NykQDJXVYrYmWrXU1q020Sa1wfQHFm1IV1GwagWRBipGRRCtpqzOCroiVbYr\nwm4WGZUFLf5Cvv3jPui43d25s3Nn7/Ds+5XczDnPec45z3ly7uee+9x7z6SqkCT162HjboAkaWEZ\n9JLUOYNekjpn0EtS5wx6SercknE3AOCoo46qycnJcTdDkh5SNm3a9N2qmpit3qII+snJSaampsbd\nDEl6SEny7WHqOXQjSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdWxS/jJW0\neEyuv3ps+77tvFPHtu+eeUUvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0md\nM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SercUEGf5E+T3Jzka0n+LckjkxyTZGOSLUku\nTXJwq/uINr+lLZ9cyAOQJO3drEGfZBnwWmBVVf0WcBBwJvB24PyqeiJwN3BOW+Uc4O5Wfn6rJ0ka\nk2GHbpYAhyRZAhwK7ABOAi5vyy8B1rTp1W2etvzkJBlNcyVJczVr0FfVduDvgdsZBPw9wCZgZ1Xd\n36ptA5a16WXAHW3d+1v9pbtuN8m6JFNJpqanp+d7HJKkPRhm6OYIBlfpxwCPAx4FvGC+O66qDVW1\nqqpWTUxMzHdzkqQ9GGbo5vnAt6pquqp+BlwBnAgc3oZyAJYD29v0dmAFQFt+GPC9kbZakjS0YYL+\nduCEJIe2sfaTga8DnwFOa3XWAle26avaPG35dVVVo2uyJGkuhhmj38jgQ9UvA5vbOhuANwLnJtnC\nYAz+orbKRcDSVn4usH4B2i1JGtKS2atAVb0FeMsuxVuBZ+6m7o+B0+ffNEnSKPjLWEnqnEEvSZ0z\n6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNe\nkjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWp\ncwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4NFfRJDk9yeZL/\nTnJLkmcnOTLJNUlubX+PaHWT5IIkW5J8NcnTF/YQJEl7M+wV/T8Bn6iqY4GnArcA64Frq2olcG2b\nB3ghsLI91gEXjrTFkqQ5mTXokxwG/C5wEUBV/bSqdgKrgUtatUuANW16NfD+GrgBODzJY0fecknS\nUIa5oj8GmAbel+TGJO9J8ijg6Kra0ercCRzdppcBd8xYf1srkySNwTBBvwR4OnBhVT0N+F9+OUwD\nQFUVUHPZcZJ1SaaSTE1PT89lVUnSHAwT9NuAbVW1sc1fziD4v/PgkEz7e1dbvh1YMWP95a3sV1TV\nhqpaVVWrJiYm9rX9kqRZzBr0VXUncEeSJ7Wik4GvA1cBa1vZWuDKNn0VcHb79s0JwD0zhngkSfvZ\nkiHr/QnwwSQHA1uBVzB4kbgsyTnAt4EzWt2PA6cAW4D7Wl1J0pgMFfRVdROwajeLTt5N3QJePc92\nSZJGxF/GSlLnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPo\nJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzi0ZdwPma3L9\n1WPb923nnTq2fUvSsLyil6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0\nktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXNDB32Sg5LcmORjbf6YJBuTbElyaZKDW/kj2vyWtnxy\nYZouSRrGXK7oXwfcMmP+7cD5VfVE4G7gnFZ+DnB3Kz+/1ZMkjclQQZ9kOXAq8J42H+Ak4PJW5RJg\nTZte3eZpy09u9SVJYzDsFf07gL8AHmjzS4GdVXV/m98GLGvTy4A7ANrye1r9X5FkXZKpJFPT09P7\n2HxJ0mxmDfokLwLuqqpNo9xxVW2oqlVVtWpiYmKUm5YkzTDMPwc/EXhJklOARwKPAf4JODzJknbV\nvhzY3upvB1YA25IsAQ4DvjfylkuShjLrFX1V/WVVLa+qSeBM4LqqOgv4DHBaq7YWuLJNX9Xmacuv\nq6oaaaslSUObz/fo3wicm2QLgzH4i1r5RcDSVn4usH5+TZQkzccwQze/UFXXA9e36a3AM3dT58fA\n6SNomyRpBPxlrCR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS\n1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0md\nM+glqXMGvSR1zqCXpM4Z9JLUuSXjboDmbnL91WPb923nnTq2fUvaN17RS1LnDHpJ6pxBL0mdM+gl\nqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SercrEGfZEWSzyT5epKbk7yulR+Z5Jokt7a/R7Ty\nJLkgyZYkX03y9IU+CEnSng1zRX8/8GdVdRxwAvDqJMcB64Frq2olcG2bB3ghsLI91gEXjrzVkqSh\nzRr0VbWjqr7cpn8A3AIsA1YDl7RqlwBr2vRq4P01cANweJLHjrzlkqShzGmMPskk8DRgI3B0Ve1o\ni+4Ejm7Ty4A7Zqy2rZXtuq11SaaSTE1PT8+x2ZKkYQ0d9El+Dfgo8PqqunfmsqoqoOay46raUFWr\nqmrVxMTEXFaVJM3BUEGf5OEMQv6DVXVFK/7Og0My7e9drXw7sGLG6stbmSRpDIb51k2Ai4Bbquof\nZyy6CljbptcCV84oP7t9++YE4J4ZQzySpP1smP8wdSLwMmBzkpta2ZuA84DLkpwDfBs4oy37OHAK\nsAW4D3jFSFssSZqTWYO+qj4PZA+LT95N/QJePc92SZJGxF/GSlLnDHpJ6pxBL0mdM+glqXMGvSR1\nbpivV0oHrMn1V49t37edd+rY9q2+eEUvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ\n6pxBL0mdM+glqXPeAkHSAa/3W114RS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknq\nnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z\n9JLUuQUJ+iQvSPKNJFuSrF+IfUiShjPyoE9yEPDPwAuB44CXJjlu1PuRJA1nIa7onwlsqaqtVfVT\n4MPA6gXYjyRpCKmq0W4wOQ14QVW9ss2/DHhWVb1ml3rrgHVt9knAN/Zxl0cB393HdQ9E9tfc2F9z\nZ5/NzXz669eramK2Skv2cePzVlUbgA3z3U6SqapaNYImHRDsr7mxv+bOPpub/dFfCzF0sx1YMWN+\neSuTJI3BQgT9l4CVSY5JcjBwJnDVAuxHkjSEkQ/dVNX9SV4DfBI4CHhvVd086v3MMO/hnwOM/TU3\n9tfc2Wdzs+D9NfIPYyVJi4u/jJWkzhn0ktS5h0zQJ7m4fUd/X9Z9a5I3jLpN6tOezrUkL0/yrnG0\nqRc99WGSySRfG8F2ftEnSdbMvJNAkuuTzPurlw+ZoJekA8AaBreOGalFG/RJzk7y1SRfSfKvrfj5\nSaaSfDPJi1q9Jyf5YpKbWv2VrfzNrd7nGfzytnu79lmSJyS5IcnmJH+T5Iet3vOSfDbJlUm2Jjkv\nyVmtHzcnecK4j2V/GvZca1a0q6xbk7xlHO0dl92cXxcnubCdY1vbefXeJLckuXjGeq9o/fhF4MTx\nHcGCOCjJu5PcnORTSQ5pz7tPJNmU5D+THAuQ5MVJNia5Mcmnkxw9c0NJngO8BPi7lmcPPg9Pb8/N\nbyZ5bqv7uSTHz1j380meusdWVtWiewBPBr4JHNXmjwQuBj7B4MVpJbANeCTwTuCsVu9g4BDgGcBm\n4FDgMcAW4A3jPq4x9NnHgJe2+VcBP2zTzwN2Ao8FHsHgB21/3Za9DnjHuI9nkZ5rLwd2AEvbefY1\nYNW4j2HM/fRhIAzuZ3Uv8JTWb5uA49s5djsw0Z6fXwDeNe7jGVGfTAL3A8e3+cuAPwSuBVa2smcB\n17XpI/jlNx1fCfxDm375g33S+vS0Gfu4fka9U4BPt+m1Dz5Pgd8ApvbW1sV6RX8S8JGq+i5AVX2/\nlV9WVQ9U1a3AVuBY4L+ANyV5I4P7PvwIeC7w71V1X1Xdy4Hxg63d9dmzgY+05R/apf6XqmpHVf0E\n+B/gU618M4MT+EAxl3MN4Jqq+l47z64Afme/t3g89tRP/1GDtNkMfKeqNlfVA8DNDM6jZwHXV9V0\nDW5yeOn+b/qC+lZV3dSmNzE45ucAH0lyE/AvDF7sYHCXgE8m2Qz8OYMXz2Fcscv2YfC8flGShwN/\nxOAFYo8Wa9Dvya5f+q+q+hCDtzs/Aj6e5KT936yHpJ/MmH5gxvwDjPEeSIvI/zvXZik/UM08b3Y9\npw6E82jmMf+cwTudnVV1/IzHb7bl72Rw5f4U4I8ZvEucyz5+TuvTqroPuIbBO6kzgA/ubQOLNeiv\nYzAutRQgyZGt/PQkD2tjV48HvpHk8cDWqroAuBL4beBzwJo2XvZo4MX7/xD2u9312Q3AH7TlZ46r\nYYvc0OdaK//9JEcmOYTBB2df2O8tHo899dNsNgK/l2Rpu/o8faEauEjcC3wryekAGXhw7Pwwfnnf\nr7V7WP8HwKOH3Nd7gAsYvDu/e28VF+UrblXdnORtwGeT/By4sS26Hfgig3H3V1XVj5OcAbwsyc+A\nO4G/rarvJ7kU+ApwF4P773RtD332euADSd7MYMz5nnG2cTGa47lGK/sog7fhH6iqqTE0e7/bSz/N\ntt6OJG9lMMS6E7hp72t04SzgwiR/BTycwecYXwHeymBI524GL5zH7GbdDwPvTvJaYK9fJ6+qTUnu\nBd43W4O8BULHkhwK/KiqKsmZDD6Y9Z/ASB1I8jgGH9Ye2z4X2aNFeUWvkXkG8K4MLkV3MvjQRtJD\nXJKzgbcB584W8uAVvSR1b7F+GCtJGhGDXpI6Z9BLUucMeknqnEEvSZ37P4lZGebT6rQhAAAAAElF\nTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zx1xjf8LYGd9",
        "colab_type": "text"
      },
      "source": [
        "One way of handling this imbalanced nature of the data is assigning weights to the classes in claculation of the categorical cross entropy loss functions. Introducing such weights will make the model pay more attention to the less common classes in the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sjUXKuRBZGU0",
        "colab_type": "text"
      },
      "source": [
        "##How to find a good learning rate\n",
        "\n",
        "The most impactful hyperparameter of the model is probably the learning rate which we hard-coded in our model. \n",
        "\n",
        "\n",
        "One way to find an appropriate value of the learning rate is to make a learning rate scheduler and monitor the behaviour of the validation loss as a function of learning rate:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "learning_rate_schedule = LearningRateScheduler(lambda epoch: 1e-6 * 10**(epoch/4)) \n",
        "```\n",
        "\n",
        "This scheduler exponentially grows the learning rate starting at an initial value of $10^{-6}$. This can be passed to the fitting process through the `callbacks` argument\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "history = model.fit_generator(train_augment.flow(train_data, train_y, batch_size=BATCH_SIZE),\n",
        "                              epochs = NEPOCHS, \n",
        "                              validation_data = valid_augment.flow(valid_data, valid_y),\n",
        "                              verbose = 1, \n",
        "                              steps_per_epoch=(train_data.shape[0] // BATCH_SIZE), \n",
        "                              validation_steps=(valid_data.shape[0] // BATCH_SIZE), \n",
        "                              callbacks=[learning_rate_schedule])\n",
        "```\n",
        "Afterwards, one can choose a proper value of this hyperparameter by inspecting the value of the validation loss as a function of learning rate.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}