02-autoencoder-classifier-hybrid.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {
    "orig_nbformat": 4,
    "language_info": {
      "name": "python",
      "version": "3.7.9",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3.7.9 64-bit ('ml': conda)"
    },
    "interpreter": {
      "hash": "6b3ca784da768489958501c88883cb0bea2209d6cbfeece4e115b49d370e2f5b"
    },
    "colab": {
      "name": "02-autoencoder-classifier-hybrid.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Autoencoder Classifier Hybrid Model\n",
        "Autoencoders are artificial neural networks capable of learning dense representations of the input data, called **latent representation or codings,** without any supervision. These latent representation have much lower dimensionality and encode the different features of the input data. Thus, we can use this latent representation to do different tasks with much less computational load. In this notebook, let's take the encoder part of the Autoencoder and use it to predict the Sentiment of a IMDB Review."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "source": [
        "import os\r\n",
        "import re\r\n",
        "import itertools\r\n",
        "import numpy as np\r\n",
        "import pandas as pd\r\n",
        "import tensorflow as tf\r\n",
        "import tensorflow.keras as keras\r\n",
        "import spacy\r\n",
        "from utils.plotting import HistoryPlotter\r\n",
        "\r\n",
        "SEED = 42\r\n",
        "np.random.seed(SEED)"
      ],
      "outputs": [],
      "metadata": {
        "id": "3rZvcg02asS_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Preprocessing\n",
        "The dataset contains the review in the \"review\" column and the sentiment in the \"sentiment\" column. Let's load the dataset and convert the sentiment column into integers of 0 if the review is positive and 1 if the review is negative. Also let's split the dataset for Training, Validation and Testing. The HTML tags in the input data are also removed."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "source": [
        "dataset = pd.read_csv(os.path.expanduser(\"~\\Downloads\\datasets\\IMDB-Dataset.csv\"))\r\n",
        "dataset['sentiment'] = dataset['sentiment'].map(lambda label: 0 if label=='positive' else 1)\r\n",
        "# dataset = pd.read_csv(\"/content/drive/MyDrive/IMDB-Dataset.csv\")\r\n",
        "train_ae = dataset.iloc[:44000, 0]\r\n",
        "val_ae = dataset.iloc[44000:47000, 0]\r\n",
        "test_ae = dataset.iloc[47000:, 0]\r\n",
        "\r\n",
        "train_y = dataset.iloc[:44000, 1]\r\n",
        "val_y = dataset.iloc[44000:47000, 1]\r\n",
        "test_y = dataset.iloc[47000:, 1]\r\n",
        "\r\n",
        "print(train_ae.head())\r\n",
        "print(\"\\n\", train_y.head())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0    One of the other reviewers has mentioned that ...\n",
            "1    A wonderful little production. <br /><br />The...\n",
            "2    I thought this was a wonderful way to spend ti...\n",
            "3    Basically there's a family where a little boy ...\n",
            "4    Petter Mattei's \"Love in the Time of Money\" is...\n",
            "Name: review, dtype: object\n",
            "\n",
            " 0    0\n",
            "1    0\n",
            "2    0\n",
            "3    1\n",
            "4    0\n",
            "Name: sentiment, dtype: int64\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_xs_8L6AasTF",
        "outputId": "49593a8b-79c2-4952-a338-e05f08775046"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "source": [
        "print(train_ae[1][:190])\r\n",
        "html_re = re.compile(r'<[^>]+>')\r\n",
        "train_ae = train_ae.replace(html_re, \"\", regex=True)\r\n",
        "val_ae = val_ae.replace(html_re, \"\", regex=True)\r\n",
        "test_ae = test_ae.replace(html_re, \"\", regex=True)\r\n",
        "print(train_ae[1][:190])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "A wonderful little production. <br /><br />The filming technique is very unassuming- very old-time-BBC fashion and gives a comforting, and sometimes discomforting, sense of realism to the en\n",
            "A wonderful little production. The filming technique is very unassuming- very old-time-BBC fashion and gives a comforting, and sometimes discomforting, sense of realism to the entire piece. \n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L0yZjK3FasTJ",
        "outputId": "9d8f8bc5-0fd1-476b-d180-268e87e45069"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tokenization\r\n",
        "Tokenization is a way of separating a piece of text into smaller units called token, in this case into words. The purpose of tokenization is that we can use their corresponding token-id to represent the original text. We first create a vocabulary containing the counts of each word that occurs in the Training Set. Then we assign an index to the first 10000 most popular words, the rest of the words are given an index of 0, which is considered as an Out of Vocabulary word."
      ],
      "metadata": {
        "id": "l2bxq0KmasTI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "source": [
        "tokenizer = keras.preprocessing.text.Tokenizer(filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n')\r\n",
        "tokenizer.fit_on_texts(train_ae)\r\n",
        "print(\"Word Indices: \")\r\n",
        "print(dict(itertools.islice(tokenizer.index_word.items(), 20)), \"...\")\r\n",
        "print(f\"Word Count: {len(tokenizer.word_index)}\")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Word Indices: \n",
            "{1: 'the', 2: 'and', 3: 'a', 4: 'of', 5: 'to', 6: 'is', 7: 'in', 8: 'it', 9: 'i', 10: 'this', 11: 'that', 12: 'was', 13: 'as', 14: 'for', 15: 'with', 16: 'movie', 17: 'but', 18: 'film', 19: 'on', 20: 'not'} ...\n",
            "Word Count: 119224\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YhBsV23SasTK",
        "outputId": "68fc1b75-335d-4f49-ab7a-bc770839a3f3"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "source": [
        "MAX_NUM_TOKENS = 10000\r\n",
        "tokenizer.num_words = MAX_NUM_TOKENS\r\n",
        "train_ae = train_ae.apply(lambda x: tokenizer.texts_to_sequences([x])[0])\r\n",
        "val_ae = val_ae.apply(lambda x: tokenizer.texts_to_sequences([x])[0])\r\n",
        "test_ae = test_ae.apply(lambda x: tokenizer.texts_to_sequences([x])[0])\r\n",
        "\r\n",
        "train_ae.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    [26, 4, 1, 79, 2077, 44, 1059, 11, 98, 146, 38...\n",
              "1    [3, 393, 119, 353, 1, 1350, 2947, 6, 51, 51, 1...\n",
              "2    [9, 190, 10, 12, 3, 393, 94, 5, 1133, 54, 19, ...\n",
              "3    [689, 221, 3, 232, 116, 3, 119, 426, 3595, 128...\n",
              "4    [111, 7, 1, 54, 4, 287, 6, 3, 2074, 1438, 18, ...\n",
              "Name: review, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uqr3epOgasTL",
        "outputId": "b7e08bd2-4a6e-4bcb-96a2-cd9603ccfbf2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Padding\n",
        "Since each review has variable lengths, we can pad the shorter reviews to a particular length and truncate longer reviews to be the same length. The length is taken to be 300. Also, we pad/truncate the end of each review since in English Language most of the important information is present at the start, thus if we were to truncate the start of the sentence, we might lose more info."
      ],
      "metadata": {
        "id": "u6OQJHMjasTM"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "source": [
        "print('sequence_length mean: ', train_ae.apply(lambda x: len(x)).mean())\r\n",
        "print('sequence_length median: ', train_ae.apply(lambda x: len(x)).median())\r\n",
        "print('sequence_length std deviation: ', np.sqrt(train_ae.apply(lambda x: len(x)).var()))\r\n",
        "MAX_SEQUENCE_LENGTH = 300 # mean + std deviation*0.5"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "sequence_length mean:  217.23727272727274\n",
            "sequence_length median:  164.0\n",
            "sequence_length std deviation:  158.58784251928319\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ky6hzR38asTM",
        "outputId": "e15b0584-6c6a-4cec-ef7d-9db7b71bb3c8"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "source": [
        "train_ae = keras.preprocessing.sequence.pad_sequences(train_ae, maxlen=MAX_SEQUENCE_LENGTH, padding='post', truncating='post')\r\n",
        "val_ae = keras.preprocessing.sequence.pad_sequences(val_ae, maxlen=MAX_SEQUENCE_LENGTH, padding='post', truncating='post')\r\n",
        "test_ae = keras.preprocessing.sequence.pad_sequences(test_ae, maxlen=MAX_SEQUENCE_LENGTH, padding='post', truncating='post')\r\n",
        "print(train_ae)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[  26    4    1 ...    0    0    0]\n",
            " [   3  393  119 ...    0    0    0]\n",
            " [   9  190   10 ...    0    0    0]\n",
            " ...\n",
            " [   1 3047    5 ...  391    3  701]\n",
            " [ 528   10   26 ...    0    0    0]\n",
            " [   9  152   24 ...    0    0    0]]\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lPEZS3vVasTN",
        "outputId": "1f612f8f-2381-4427-9ed2-4f37d962d55e"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Word to Vector\n",
        "Now that each review is of the same length, the next step is to embed each of the index into unique vectors. We have 2 options in this step, using One Hot Encoders or using Word Embeddings. Since One Hot Encoders are really inefficient (size of vector = vocabulary size) and they don't have any relationship between the words, we will go with pretrained word embeddings. We will use Pretrained Models from the Spacy library for this."
      ],
      "metadata": {
        "id": "j4zFELEFasTO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "source": [
        "word2vec_model = spacy.load(\"en_core_web_sm\")\r\n",
        "\r\n",
        "df_index_word = pd.Series(tokenizer.index_word) # Get all the words in the vocabulary\r\n",
        "df_index_word = df_index_word[:MAX_NUM_TOKENS-1]\r\n",
        "df_index_word = pd.Series([\"temp_oov_word\"]).append(df_index_word) # Add a temporary row for oov words\r\n",
        "df_index_word = df_index_word.reset_index() # New column for token_id\r\n",
        "df_index_word.columns = ['token_id', 'token']\r\n",
        "\r\n",
        "df_index_word['word2vec'] = df_index_word.token.apply(lambda x: word2vec_model(x).vector) # Embedding for each word\r\n",
        "df_index_word.at[0, \"word2vec\"] = np.zeros_like(df_index_word.at[0, \"word2vec\"]) # Replace the embedding for \"temp_oov_word\" with 0s\r\n",
        "df_index_word.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>token_id</th>\n",
              "      <th>token</th>\n",
              "      <th>word2vec</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>temp_oov_word</td>\n",
              "      <td>[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>the</td>\n",
              "      <td>[-0.2032743, -0.4009009, -0.73730505, 0.536398...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>and</td>\n",
              "      <td>[-0.9291044, -0.13172856, -0.814826, 0.6040276...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>a</td>\n",
              "      <td>[0.02819018, 0.009864252, -0.095220566, 0.5522...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>of</td>\n",
              "      <td>[-1.1193825, 0.18575507, -0.59173, -0.4724092,...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   token_id          token                                           word2vec\n",
              "0         0  temp_oov_word  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n",
              "1         1            the  [-0.2032743, -0.4009009, -0.73730505, 0.536398...\n",
              "2         2            and  [-0.9291044, -0.13172856, -0.814826, 0.6040276...\n",
              "3         3              a  [0.02819018, 0.009864252, -0.095220566, 0.5522...\n",
              "4         4             of  [-1.1193825, 0.18575507, -0.59173, -0.4724092,..."
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4IQSBGYsasTP",
        "outputId": "1c74bf68-4879-4e0c-c409-f17b8fa8d5de"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "source": [
        "embedding_matrix = np.array([vec for vec in df_index_word.word2vec.values])\r\n",
        "print(embedding_matrix.shape) # MAX_NUM_TOKENS x vector dim \r\n",
        "embedding_matrix[0:3, :5]"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 96)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],\n",
              "       [-0.2032743 , -0.4009009 , -0.73730505,  0.5363982 ,  0.3764372 ],\n",
              "       [-0.9291044 , -0.13172856, -0.814826  ,  0.6040276 , -0.67340827]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GjIg_qvLasTP",
        "outputId": "48da28d9-1024-4b41-aa6f-f592a7bac5fb"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "source": [
        "# MAX_NUM_TOKENS = 20000\r\n",
        "# MAX_SEQUENCE_LENGTH = 300\r\n",
        "EMBEDDING_DIM = embedding_matrix.shape[1] # 96\r\n",
        "embedding_layer = keras.layers.Embedding(\r\n",
        "    input_dim=MAX_NUM_TOKENS,\r\n",
        "    output_dim=EMBEDDING_DIM,\r\n",
        "    embeddings_initializer=keras.initializers.Constant(embedding_matrix),\r\n",
        "    input_length=MAX_SEQUENCE_LENGTH,\r\n",
        "    mask_zero=True,\r\n",
        "    trainable=False)\r\n",
        "\r\n",
        "# test\r\n",
        "embedding_result = embedding_layer(inputs=train_ae[0])\r\n",
        "print(embedding_result.shape) # MAX_SEQUENCE_LENGTH x EMBEDDING_DIM\r\n",
        "print(\"Embeddings of the first 5 words (each vector is restricted to size 5)\")\r\n",
        "print(train_ae[0][:5])\r\n",
        "print(embedding_result[:5, :5]) \r\n",
        "\r\n",
        "train_ae = embedding_layer(inputs=train_ae)\r\n",
        "val_ae = embedding_layer(inputs=val_ae)\r\n",
        "test_ae = embedding_layer(inputs=test_ae)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(300, 96)\n",
            "Embeddings of the first 5 words (each vector is restricted to size 5)\n",
            "[  26    4    1   79 2077]\n",
            "tf.Tensor(\n",
            "[[ 0.49959356  0.704657    0.4497372   0.32913145  0.01701519]\n",
            " [-1.1193825   0.18575507 -0.59173    -0.4724092   0.04607756]\n",
            " [-0.2032743  -0.4009009  -0.73730505  0.5363982   0.3764372 ]\n",
            " [-0.33528453 -0.5446908   0.7285543  -0.51831293  0.10952553]\n",
            " [-0.39640257  0.09997782 -0.0515418   0.10987999  0.6886112 ]], shape=(5, 5), dtype=float32)\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iCnrIUeOasTQ",
        "outputId": "02685cab-4801-4816-d972-96601ec52ec8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Convolutional Autoencoder\n",
        "An Autoencoder is built using Conv1D Layers with MaxPooling in the Encoder and Conv1DTranspose and BatchNormalization Layers in the Decoder. Each sample of length $(300 \\times 96)$ is reduces to a latent representation of $(50 \\times 1)$ which hopefully captures all the meaningful data in the input. This representation is then used to reconstruct the input. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "source": [
        "conv_encoder = keras.models.Sequential([\r\n",
        "    keras.layers.Conv1D(filters=96, kernel_size=8, strides=2, input_shape=[MAX_SEQUENCE_LENGTH, EMBEDDING_DIM]),\r\n",
        "    keras.layers.MaxPool1D(pool_size=2),\r\n",
        "    keras.layers.Activation('relu'),\r\n",
        "    \r\n",
        "    keras.layers.Conv1D(filters=128, kernel_size=16, strides=1),\r\n",
        "    keras.layers.MaxPool1D(pool_size=4),\r\n",
        "    keras.layers.Activation('relu'),\r\n",
        "    \r\n",
        "    keras.layers.Conv1D(filters=50, kernel_size=14),\r\n",
        "    keras.layers.Reshape([50])\r\n",
        "])\r\n",
        "conv_encoder.summary()\r\n",
        "conv_decoder = keras.models.Sequential([\r\n",
        "    # kernel size and strides are chosen such that we get 300x96 = 28800 = 180x160 size.\r\n",
        "    keras.layers.Reshape([50,1], input_shape=[50]),\r\n",
        "    keras.layers.Conv1DTranspose(64, kernel_size=8, padding='valid', activation='linear', input_shape=[1,50]),\r\n",
        "    keras.layers.BatchNormalization(),\r\n",
        "    keras.layers.Conv1DTranspose(128, kernel_size=32, strides=2, padding='valid', activation='linear'),\r\n",
        "    keras.layers.BatchNormalization(),\r\n",
        "    keras.layers.Conv1DTranspose(160, kernel_size=37, strides=1, padding='valid', activation='linear'),\r\n",
        "    keras.layers.BatchNormalization(),\r\n",
        "    keras.layers.Reshape([MAX_SEQUENCE_LENGTH, EMBEDDING_DIM])\r\n",
        "])\r\n",
        "conv_decoder.summary()\r\n",
        "conv_ae = keras.models.Sequential([conv_encoder, conv_decoder])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv1d (Conv1D)              (None, 147, 96)           73824     \n",
            "_________________________________________________________________\n",
            "max_pooling1d (MaxPooling1D) (None, 73, 96)            0         \n",
            "_________________________________________________________________\n",
            "activation (Activation)      (None, 73, 96)            0         \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 58, 128)           196736    \n",
            "_________________________________________________________________\n",
            "max_pooling1d_1 (MaxPooling1 (None, 14, 128)           0         \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 14, 128)           0         \n",
            "_________________________________________________________________\n",
            "conv1d_2 (Conv1D)            (None, 1, 50)             89650     \n",
            "_________________________________________________________________\n",
            "reshape (Reshape)            (None, 50)                0         \n",
            "=================================================================\n",
            "Total params: 360,210\n",
            "Trainable params: 360,210\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "reshape_1 (Reshape)          (None, 50, 1)             0         \n",
            "_________________________________________________________________\n",
            "conv1d_transpose (Conv1DTran (None, 57, 64)            576       \n",
            "_________________________________________________________________\n",
            "batch_normalization (BatchNo (None, 57, 64)            256       \n",
            "_________________________________________________________________\n",
            "conv1d_transpose_1 (Conv1DTr (None, 144, 128)          262272    \n",
            "_________________________________________________________________\n",
            "batch_normalization_1 (Batch (None, 144, 128)          512       \n",
            "_________________________________________________________________\n",
            "conv1d_transpose_2 (Conv1DTr (None, 180, 160)          757920    \n",
            "_________________________________________________________________\n",
            "batch_normalization_2 (Batch (None, 180, 160)          640       \n",
            "_________________________________________________________________\n",
            "reshape_2 (Reshape)          (None, 300, 96)           0         \n",
            "=================================================================\n",
            "Total params: 1,022,176\n",
            "Trainable params: 1,021,472\n",
            "Non-trainable params: 704\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "source": [
        "conv_ae.compile(optimizer='adam', loss='mse')\r\n",
        "conv_ae_history = conv_ae.fit(x=train_ae, y=train_ae, epochs=20, batch_size=256, validation_data=(val_ae, val_ae))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "172/172 [==============================] - 40s 56ms/step - loss: 0.8150 - val_loss: 0.5757\n",
            "Epoch 2/20\n",
            "172/172 [==============================] - 9s 52ms/step - loss: 0.5847 - val_loss: 0.4714\n",
            "Epoch 3/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.4339 - val_loss: 0.3652\n",
            "Epoch 4/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.3342 - val_loss: 0.2872\n",
            "Epoch 5/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2720 - val_loss: 0.2407\n",
            "Epoch 6/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2358 - val_loss: 0.2217\n",
            "Epoch 7/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2164 - val_loss: 0.2092\n",
            "Epoch 8/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2068 - val_loss: 0.2040\n",
            "Epoch 9/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2025 - val_loss: 0.2015\n",
            "Epoch 10/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2007 - val_loss: 0.2007\n",
            "Epoch 11/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.2001 - val_loss: 0.2004\n",
            "Epoch 12/20\n",
            "172/172 [==============================] - 9s 52ms/step - loss: 0.1999 - val_loss: 0.2002\n",
            "Epoch 13/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1998 - val_loss: 0.2002\n",
            "Epoch 14/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1998 - val_loss: 0.2001\n",
            "Epoch 15/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n",
            "Epoch 16/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n",
            "Epoch 17/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n",
            "Epoch 18/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n",
            "Epoch 19/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n",
            "Epoch 20/20\n",
            "172/172 [==============================] - 9s 54ms/step - loss: 0.1997 - val_loss: 0.2001\n"
          ]
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "source": [
        "print(f\"Test Set metrics and Training History:\")\r\n",
        "conv_ae.evaluate(x=test_ae, y=test_ae)\r\n",
        "np.save(\"./models/conv_ae.npy\", conv_ae_history.history)\r\n",
        "# conv_ae_history = np.load(\"./models/conv_ae.npy\", allow_pickle='TRUE').item()\r\n",
        "conv_ae_plot = HistoryPlotter(history=conv_ae_history.history, accuracy_line=None, max_y_lim=[0.5, 1])\r\n",
        "conv_ae_plot.all(row_col=[1,0])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Set metrics and Training History:\n",
            "94/94 [==============================] - 0s 5ms/step - loss: 0.1990\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAs4UlEQVR4nO3de3xU9Z3/8dcnk0kChBBIuAQIhAoqIBAUUAFbra1FvLdW8dZau7psa6u23er+2u12u+223V62WrXWtm5vVKTWtrZitVoveAcElIsKIpJwvxgg3HL7/P44A8YwmUxIzswk834+HvPIzDnfM/lwjPOe7/ec8z3m7oiISPbKSXcBIiKSXgoCEZEspyAQEclyCgIRkSynIBARyXIKAhGRLKcgEGmDmVWYmZtZbhJtrzazZ1JRl0hnURBIt2Jm68yszsxKWyxfGvswr0hTae0KFJFUUhBId/QWcNmhF2Y2DuiRvnJEMpuCQLqj3wCfaPb6k8Cvmzcwsz5m9msz22Zmb5vZV80sJ7YuYmbfN7PtZrYWOCfOtr8ws01mtsHMvmlmkY4UbGaDzexBM9tpZmvM7Npm66aY2SIz221mW8zsh7HlBWb2WzPbYWY1ZrbQzAZ2pA7JTgoC6Y5eAIrMbHTsA/pS4Lct2vwY6AO8D/gAQXB8KrbuWuBcYCIwCbi4xba/AhqAkbE2ZwH/1MGa7wWqgcGx3/ffZnZmbN2twK3uXgQcA8yLLf9k7N9QDpQAs4H9HaxDspCCQLqrQ72CDwOvARsOrWgWDv/m7nvcfR3wA+CqWJNLgB+5e5W77wS+3WzbgcDZwI3uvtfdtwL/C8w62kLNrByYDtzs7gfcfSnw82b11AMjzazU3Wvd/YVmy0uAke7e6O6L3X330dYh2UtBIN3Vb4DLgatpMSwElAJ5wNvNlr0NDIk9HwxUtVh3yHAgCmyKDcfUAD8FBnSg1sHATnff00o9nwaOBV6LDf+cG1v+G+ARYK6ZbTSz/zGzaAfqkCylIJBuyd3fJjhoPBN4oMXq7QTfpoc3WzaMd3sNmwiGW5qvO6QKOAiUuntx7FHk7mM7UO5GoJ+Z9Y5Xj7uvdvfLCMLmu8D9ZtbL3evd/T/dfQwwlWA46xOItJOCQLqzTwMfdPe9zRe6eyPBOPu3zKy3mQ0HvsC7xxHmAZ83s6Fm1he4pdm2m4BHgR+YWZGZ5ZjZMWb2gXbUlR870FtgZgUEH/jPAd+OLRsfq30OgJldaWb93b0JqIm9R6OZnWFm42JDXbsJwq2xHXWIAAoC6cbc/U13X9TK6s8Be4G1wDPA74B7Yut+RjDksgx4mSN7FJ8gGFpaCbwD3A+UtaO0WoKDuoceHyQ43bWCoHfwR+A/3P3vsfYzgBVmVktw4HiWux8ABsV+925gFfAURx4UF2mT6cY0IiLZTT0CEZEspyAQEclyCgIRkSynIBARyXJdbhbE0tJSr6ioSHcZIiJdyuLFi7e7e/9467pcEFRUVLBoUWtnBIqISDxm9nZr6zQ0JCKS5RQEIiJZTkEgIpLlutwxAhGRo1FfX091dTUHDhxIdymhKigoYOjQoUSjyU9EqyAQkaxQXV1N7969qaiowMzSXU4o3J0dO3ZQXV3NiBEjkt5OQ0MikhUOHDhASUlJtw0BADOjpKSk3b0eBYGIZI3uHAKHHM2/MWuC4PXNe/jWQyvZV9eQ7lJERDJK1gRB9Tv7+NmCt1i+Qbd0FZHUq6mp4c4772z3djNnzqSmpqbzC2oma4KgsrwYgCXr30lvISKSlVoLgsbGxDeVmz9/PsXFxSFVFQg1CMxshpm9bmZrzOyWOOv7mNlfzGyZma0ws0+FVUtJYT7D+vVkaVVNWL9CRKRVt9xyC2+++SaVlZVMnjyZM844g8svv5xx48YBcOGFF3LSSScxduxY7r777sPbVVRUsH37dtatW8fo0aO59tprGTt2LGeddRb79+/vlNpCO300dh/VO4APA9XAQjN70N1XNmv2WWClu59nZv2B181sjrvXhVHTxGHFvLh2ZxhvLSJdyH/+ZQUrN3buMPGYwUX8x3ljW13/ne98h+XLl7N06VKefPJJzjnnHJYvX374NM977rmHfv36sX//fiZPnszHPvYxSkpK3vMeq1ev5t577+VnP/sZl1xyCX/4wx+48sorO1x7mD2CKcAad18b+2CfC1zQoo0DvS04zF0I7ARCO5pbWV7M5t0H2LSrc1JURORoTZky5T3n+t92221MmDCBU045haqqKlavXn3ENiNGjKCyshKAk046iXXr1nVKLWFeUDYEqGr2uho4uUWb24EHCW7Y3Ru41N2bWr6RmV0HXAcwbNiwoy7o0HGCpetrKBvX46jfR0S6tkTf3FOlV69eh58/+eSTPPbYYzz//PP07NmT008/Pe61APn5+YefRyKRThsaCrNHEO9kVm/x+iPAUmAwUAncbmZFR2zkfre7T3L3Sf37x51OOyljBheRF8nRcQIRSbnevXuzZ8+euOt27dpF37596dmzJ6+99hovvPBCSmsLs0dQDZQ3ez2U4Jt/c58CvuPuDqwxs7eA44GXwigoPzfCmMFFLFEQiEiKlZSUMG3aNE444QR69OjBwIEDD6+bMWMGd911F+PHj+e4447jlFNOSWltYQbBQmCUmY0ANgCzgMtbtFkPnAksMLOBwHHA2hBrYuKwYua+VEVDYxO5kaw5e1ZEMsDvfve7uMvz8/N5+OGH4647dBygtLSU5cuXH17+pS99qdPqCu2T0N0bgOuBR4BVwDx3X2Fms81sdqzZfwFTzexV4HHgZnffHlZNEBwn2F/fyOtb4nfRRESyTaizj7r7fGB+i2V3NXu+ETgrzBpaOnFYXwCWrK9h7OA+qfzVIiIZKevGRob27UFJrzwdMBYRicm6IDAzKsuLNdWEiEhM1gUBBAeM39y2l13769NdiohI2mVlEFSWB8cJXqmuSW8hIiIZICuDYHx5H8yCA8YiIpmosLAwZb8rK4OgqCDKyP6FOmAsIkIW37y+sryYx1Ztwd2z4vZ1IpJeN998M8OHD+czn/kMAF//+tcxM55++mneeecd6uvr+eY3v8kFF7ScmzN8WRsEE4f15feLq1m/cx/DS3q1vYGIdB8P3wKbX+3c9xw0Ds7+TqurZ82axY033ng4CObNm8ff/vY3brrpJoqKiti+fTunnHIK559/fsq/nGZtEByeibSqRkEgIqGbOHEiW7duZePGjWzbto2+fftSVlbGTTfdxNNPP01OTg4bNmxgy5YtDBo0KKW1ZW0QHDuwkJ55EZasr+GCyiHpLkdEUinBN/cwXXzxxdx///1s3ryZWbNmMWfOHLZt28bixYuJRqNUVFTEnX46bFkbBLmRHMYN6aOZSEUkZWbNmsW1117L9u3beeqpp5g3bx4DBgwgGo3yxBNP8Pbbb6elrqw8a+iQicP6snLjLg7UJ755tIhIZxg7dix79uxhyJAhlJWVccUVV7Bo0SImTZrEnDlzOP7449NSV9b2CCA4TlDf6KzctPvwZHQiImF69dV3D1KXlpby/PPPx21XW1ubqpKyvUdQDOjCMhHJblkdBAOLChjcp0AXlolIVsvqIACoHFbM0irNRCqSDYK74nZvR/NvzPogmFjel6qd+9leezDdpYhIiAoKCtixY0e3DgN3Z8eOHRQUFLRru6w+WAxBjwBg6foaPjRmYOLGItJlDR06lOrqarZt25buUkJVUFDA0KFD27VN1gfBCYP7EMkxllS9oyAQ6cai0SgjRoxIdxkZKeuHhnrkRRhd1lsHjEUka2V9EEBwPcGyql00NnXfsUMRkdYoCAgOGNcebODNbam7gENEJFMoCHjvAWMRkWwTahCY2Qwze93M1pjZLXHW/6uZLY09lptZo5n1C7OmeEaU9KJPjyhLdD2BiGSh0ILAzCLAHcDZwBjgMjMb07yNu3/P3SvdvRL4N+Apd98ZVk2tyckxJpQXa6oJEclKYfYIpgBr3H2tu9cBc4FE92C7DLg3xHoSqiwv5o0te9h7sCFdJYiIpEWYQTAEqGr2ujq27Ahm1hOYAfyhlfXXmdkiM1sU1sUgE4cV0+TwSvWuUN5fRCRThRkE8W662dr5mecBz7Y2LOTud7v7JHef1L9//04rsLnKocUAOk4gIlknzCCoBsqbvR4KbGyl7SzSOCwE0LdXHiNKe+nMIRHJOmEGwUJglJmNMLM8gg/7B1s2MrM+wAeAP4dYS1Iqy4tZUlXTrSelEhFpKbQgcPcG4HrgEWAVMM/dV5jZbDOb3azpRcCj7r43rFqSVVlezLY9B9m4K/U3jxYRSZdQJ51z9/nA/BbL7mrx+pfAL8OsI1kTm11YNqS4R3qLERFJEV1Z3Mzxg4rIy81hyXodMBaR7KEgaCYvN4dxQ/poJlIRySoKghYqy4t5dcMu6hub0l2KiEhKKAhamDismIMNTby2aU+6SxERSQkFQQuV5cUAuqG9iGQNBUELQ4p7UFqYrwnoRCRrKAhaMDMmDivWAWMRyRoKgjgqy4tZu30vNfvq0l2KiEjoFARxHL6wTL0CEckCCoI4xg8txkxBICLZQUEQR2F+LscO6K0DxiKSFRQErTh0wFgzkYpId6cgaEVleTG79tfz1va0T4oqIhIqBUErJg7rC+g4gYh0fwqCVowcUEivvIiOE4hIt6cgaEUkx5hQrgvLRKT7UxAkUFlezKpNuzlQ35juUkREQqMgSKCyvJiGJmf5hl3pLkVEJDQKggQqdYWxiGQBBUECA3oXMKS4hw4Yi0i3piBog2YiFZHuTkHQhsryYjbU7Gfr7gPpLkVEJBQKgjYcurBsiXoFItJNhRoEZjbDzF43szVmdksrbU43s6VmtsLMngqznqMxdnAR0YhpeEhEuq3csN7YzCLAHcCHgWpgoZk96O4rm7UpBu4EZrj7ejMbEFY9R6sgGmF0WRFL1usexiLSPYXZI5gCrHH3te5eB8wFLmjR5nLgAXdfD+DuW0Os56hNLC/mlepdNDZpJlIR6X7CDIIhQFWz19WxZc0dC/Q1syfNbLGZfSLeG5nZdWa2yMwWbdu2LaRyW1c5rJh9dY28sWVPyn+3iEjYwgwCi7Os5VfqXOAk4BzgI8C/m9mxR2zkfre7T3L3Sf379+/8StswsVwzkYpI9xVmEFQD5c1eDwU2xmnzN3ff6+7bgaeBCSHWdFSGl/Skb88oS3VhmYh0Q2EGwUJglJmNMLM8YBbwYIs2fwZOM7NcM+sJnAysCrGmo2IWzES6pEoHjEWk+wktCNy9AbgeeITgw32eu68ws9lmNjvWZhXwN+AV4CXg5+6+PKyaOmJieV9Wb61lz4H6dJciItKpQjt9FMDd5wPzWyy7q8Xr7wHfC7OOzjB5RF/c4dk1O5hxwqB0lyMi0ml0ZXGSplT0o2/PKA8v35TuUkREOlX2BMGGxTD3Cqjbd1Sb50Zy+MjYQTy+aqtuVCMi3Ur2BEFDHbz2V1jym6N+i5njyqg92MDTb6T+WgYRkbBkTxAMPxWGTYVnbw1C4SicekwJxT2jPLx8cycXJyKSPtkTBACnfRF2b4BX7juqzaORHM4aM5DHVm7hYIOGh0Ske8iuIBh5JpRNgGf+F5qO7oN85rgy9hxsYMEb2zu5OBGR9MiuIDALegU734SVfzqqt5g2spQ+PaLM19lDItJNZFcQABx/HpQeCwt+CN7+2USjkRw+PGYgf9fwkIh0E9kXBDk5MP0LsGU5vPHIUb3FOePK2HOggWfXaHhIRLq+7AsCgHEXQ/EwWPD9o+oVTBtZSu+CXOa/qrOHRKTry84giERh2g1QvRDWLWj35nm5wfDQoys2U9fQFEKBIiKpk51BAFB5JfQaAAt+cFSbnzOujN0HGnjuTQ0PiUjXlr1BEC2AqdfD2iehenG7N58+qpTe+bnMf1VnD4lI15a9QQAw6RooKD6qXkF+boQPjRnIoyu3UN+o4SER6bqyOwjye8PJs+H1h2DLynZvPnNcGTX76nn+zR0hFCcikhrZHQQAJ/8zRHsFVxu302mjSinU8JCIdHEKgp79YPI1sPx+2Lm2XZsWRCOcOXoAj6zYrOEhEemyFAQAp14POdFgZtJ2mjmujHf21fPi2p0hFCYiEr6kgsDMbjCzIgv8wsxeNrOzwi4uZXoPgolXwtLfwe6N7dr0A8f2p1dehIc0PCQiXVSyPYJr3H03cBbQH/gU8J3QqkqHaZ8PZiR97vZ2bVYQjfDB0QN5ZMVmGjQ8JCJdULJBYLGfM4H/c/dlzZZ1D30rYNzHYfH/wd72nQV0zrhB7Nxbx0tvaXhIRLqeZINgsZk9ShAEj5hZb6D7ff097QtQvx9e/Em7NvvAsQPoEdXwkIh0TckGwaeBW4DJ7r4PiBIMD3Uv/Y+D0efCi3fDgd1Jb9YjL8IHY2cPNTa1fxI7EZF0SjYITgVed/caM7sS+Cqwq62NzGyGmb1uZmvM7JY46083s11mtjT2+Fr7yg/BaV+Eg7tg4c/btdk548rYXqvhIRHpepINgp8A+8xsAvBl4G3g14k2MLMIcAdwNjAGuMzMxsRpusDdK2OPbyRfekgGT4RjzoTn74C6fUlvdvpx/SmI5ujiMhHpcpINggZ3d+AC4FZ3vxXo3cY2U4A17r7W3euAubHtM99pX4R922HJb5PepGdeLh88fgAPL9fwkIh0LckGwR4z+zfgKuCh2Lf9aBvbDAGqmr2uji1r6VQzW2ZmD5vZ2HhvZGbXmdkiM1u0bdu2JEvugIppMOzU4AKzhrqkN5s5rozttQdZtE7DQyLSdSQbBJcCBwmuJ9hM8IH+vTa2iXd6acuvyi8Dw919AvBj4E/x3sjd73b3Se4+qX///kmW3EGnfRF2V8Or85Le5IzjBpCfq+EhEelakgqC2If/HKCPmZ0LHHD3hMcICHoA5c1eDwXec9muu+9299rY8/lA1MxKky0+VCM/BIPGB5PRNSV3k/pe+bmccVwwPNSk4SER6SKSnWLiEuAl4OPAJcCLZnZxG5stBEaZ2QgzywNmAQ+2eN9BZmax51Ni9WTGnM5mQa9gxxpY+eekN5s5voytew6yeP07IRYnItJ5cpNs9xWCawi2AphZf+Ax4P7WNnD3BjO7HngEiAD3uPsKM5sdW38XcDHwL2bWAOwHZsUOSmeG0edByShY8EMYe1EQDm344PEDyMvN4aFXNjG5ol8KihQR6ZhkjxHkHAqBmB3JbOvu8939WHc/xt2/FVt2VywEcPfb3X2su09w91Pc/bl2/wvClBOB6TfBlldh9aNJbVKYn8vpx/bn4eWbNDwkIl1CskHwNzN7xMyuNrOrgYeA+eGVlUHGXwJ9yuHp70OSnZVzxpexZfdBllRpeEhEMl+yB4v/FbgbGA9MAO5295vDLCxjRKIw7QaofgnefjapTd4dHtoccnEiIh2X9I1p3P0P7v4Fd7/J3f8YZlEZZ+KV0GtA0CtIQu+CKO8fpeEhEekaEgaBme0xs91xHnvMLPlZ2bq6aA849bOw9gnYsDipTc4ZP4hNuw6wtLom3NpERDooYRC4e293L4rz6O3uRakqMiNMugby+8AzP0qq+ZmjB5IXyWH+K7q4TEQym+5ZnKyCIpjyT7DqL7B9dZvNiwqinDaqlIeXbyaTzogVEWlJQdAeJ/8L5OYnfZP7mePK2FCzn2XVbc7YLSKSNgqC9ijsDxOvgmVzk7rJ/YfGDCQaMc09JCIZTUHQXlOvB28K7lfQhj49okwfWcpDr2zS8JCIZCwFQXv1rYATPgaL/g/2tT3d9KHhoVc3aHhIRDKTguBoTL8R6vcmdTvLD48ZSG6O6cb2IpKxFARHY+BYOHYGvPATqNubsGlxzzymjSxl/qsaHhKRzKQgOFrTb4L9O5O6neXMcYOo2rmfFRuz5xo8Eek6FARHa9gpwe0sn/sxNNYnbHrWmEFENDwkIhlKQdAR02+CXVXwaqu3ZQCgb688ph5TouEhEclICoKOGHUWDBgLz/4ImpoSNp05roy3d+zj5fU1KSlNRCRZCoKOMAt6Bdtegzf+lrDpeRMGU9wzyu3/aHt6ChGRVFIQdNTYi6B4GDzzw4Q3rinMz+Xa097HE69vY2lVTerqExFpg4KgoyK5MPXzUL2wzRvXfHJqBcU9o9z62BspKk5EpG0Kgs4w8Uro1R+e+d+EzdQrEJFMpCDoDNEecMq/wJrHYNMrCZuqVyAimUZB0FkmfRryegdnECWgXoGIZBoFQWfpUQyTr4EVf4SdaxM2Va9ARDJJqEFgZjPM7HUzW2NmtyRoN9nMGs3s4jDrCd0pn4Gc3OBq4wTUKxCRTBJaEJhZBLgDOBsYA1xmZmNaafdd4JGwakmZ3oOg8nJYMgf2bEnYVL0CEckUYfYIpgBr3H2tu9cBc4EL4rT7HPAHYGuItaTO1M9DUz28cGfCZuoViEimCDMIhgBVzV5Xx5YdZmZDgIuAu0KsI7VKjoExF8Kie+BA4pvRqFcgIpkgzCCwOMtaXnr7I+Bmd29M+EZm15nZIjNbtG3bts6qLzzTb4SDu2HhLxI2U69ARDJBmEFQDZQ3ez0UaHnH90nAXDNbB1wM3GlmF7Z8I3e/290nufuk/v37h1RuJyqbAMecGdy4pn5/wqbqFYhIuoUZBAuBUWY2wszygFnAg80buPsId69w9wrgfuAz7v6nEGtKndO+AHu3wtI5CZupVyAi6RZaELh7A3A9wdlAq4B57r7CzGab2eywfm/GGD4Nhk6GZ2+DxoaETdUrEJF0CvU6Anef7+7Huvsx7v6t2LK73P2Ig8PufrW7J77DS1dyaIrqmrdh5Z8SNlWvQETSSVcWh+nYs6H0uGAyujbuTPbJqRX0Va9ARNJAQRCmnJzgDKIty4MJ6RIozM/l2verVyAiqacgCNu4j0PRUFjwwzabfuJU9QpEJPUUBGGLRGHq52D9c7D+hYRN1SsQkXRQEKTCiVdBj37wzI/abKpegYikmoIgFfJ6wcmz4Y2HYcPLCZuqVyAiqaYgSJVTZkPhIPjL56GxPmFT9QpEJJUUBKlS0AfO+T5sfhWevz1hU/UKRCSVFASpNPq84PHkd2DHmwmbqlcgIqmiIEi1s78HkXz4yw0JLzJTr0BEUkVBkGpFZXDWN2DdAljy24RN1SsQkVRQEKTDxE8Ek9I9+pWEt7RUr0BEUkFBkA45OXDerVB/AB7+csKm6hWISNgUBOlSOgo+8OVgZtLX5rfaTL0CEQmbgiCdpt0AA8bCQ1+EA7tbbaZegYiESUGQTpEonP9jqN0Mj/9nq82a9wqeeH1rCgsUkWygIEi3oScF008s/HnCSek+NXUEo8uKuOHeJazbvjeFBYpId6cgyARnfAX6DIMHPwcNB+M26ZEX4e6rTiInx7juN4vYezDx7S9FRJKlIMgE+YVw7v/C9jdgwQ9abVberye3X3Yia7bW8qXfL8PbuOuZiEgyFASZYtSHYPylwQ1stq5qtdn0UaX8v5mjeXj5Zu58MvE0FSIiyVAQZJKPfBsKioIhoqbGVpt9evoILqwczPcffZ0nXtPBYxHpGAVBJulVEoRB9UJY+ItWm5kZ3/7oeEYPKuLzc5fwlg4ei0gHKAgyzfhL4Jgzg9NJa6pabdYjL8JPrzqJ3Bzjul8volYHj0XkKCkIMo1ZcODYm4ILzRIcEC7v15M7Lj+Rtdv38sV5S2lq0sFjEWm/UIPAzGaY2etmtsbMbomz/gIze8XMlprZIjObHmY9XUbf4fDBf4fVj8CKBxI2nToyOHj8yIot3PHEmhQVKCLdSWhBYGYR4A7gbGAMcJmZjWnR7HFggrtXAtcAPw+rni7n5H+GwSfC/C/Dvp0Jm14zrYKLJg7hh4+9weOrWp/NVEQknjB7BFOANe6+1t3rgLnABc0buHutv3syfC9AYxuH5ESC6ScO1MCjX03YNDh4PI6xg4u4ce5S3txWm5oaRaRbCDMIhgDNj3ZWx5a9h5ldZGavAQ8R9AqOYGbXxYaOFm3bti2UYjPSoBOCiemWzoE3n0jYtCAa4adXTSIvN4frfr2IPQfqU1SkiHR1YQaBxVl2xDd+d/+jux8PXAj8V7w3cve73X2Su0/q379/51aZ6d7/ZSgZCX+9Eer2JWw6pLgHt19+Iut27OML85bp4LGIJCXMIKgGypu9HgpsbK2xuz8NHGNmpSHW1PVEC4Kb2LyzDp78dpvNTz2mhK+eM5q/r9zCbf9YHX59ItLlhRkEC4FRZjbCzPKAWcCDzRuY2Ugzs9jzE4E8YEeINXVNFdPhpKvh+dsT3sTmkKunVvCxE4fyo8dW8/eVOngsIomFFgTu3gBcDzwCrALmufsKM5ttZrNjzT4GLDezpQRnGF3qmkktvg9/AwZPhPuuhGX3JWxqZnzrohMYP7QPN923lDVbdfBYRFpnXe1zd9KkSb5o0aJ0l5EeB/fA3Mvhradh5vdhyrUJm2+s2c/5tz9DUUGUP10/jaKCaIoKFZFMY2aL3X1SvHW6srgrye8Nl/8ejjsH5n8Jnv5ewiuPBxf34I7LT2T9zn3cNFdXHotIfAqCriZaAJf8GsbPgn98M7jGIEEYnPy+Er523hgef20rP3pcB49F5Ei56S5AjkIkFy78STBl9fO3w4FdwZlFOZG4za86ZTivVu/itsdX4+7ccOYociP6DiAiAQVBV5WTA2f/DxQUw9P/Awd3w0d/Brn5RzQ1M7550QkA/Pgfa3h2zXZunTWR8n49U1y0iGQifS3syszgg1+Bj/w3rPwz3DsL6uLfmyA/N8L3Pj6B2y6byOottcy8bQF/WdbqZR0ikkUUBN3BqZ+F82+HtU/Cby6C/TWtNj1/wmDm33AaIwcU8rl7l/Dl+5exr073MhDJZgqC7uLEq+Djv4QNL8Mvz4Xa1m9hWd6vJ/P++VSuP2Mkv19czbk/foblG3alrlYRySgKgu5kzAVw+X2w80245yNQs77VptFIDl/6yHHM+aeT2XuwgY/e+Ry/eOYtutp1JSLScQqC7mbkmXDVn2DfDvjFR2Db6wmbTz2mlIdveD/vP7Y///XXlVzzy4Vsrz2YmlpFJCMoCLqjYSfD1fOhqQH+72zYuCRh83698vjZJ07ivy4Yy7Nv7mDGjxawYHUWTfctkuUUBN3VoBPgmr9BtBf88jxY92zC5mbGVadW8OD10+jbM8pVv3iJb89fRV1DU4oKFpF0URB0ZyXHBGFQVAa//Si88Uibmxw/qIgHr5/OFScP46dPr+Xiu55j3fb4p6SKSPegIOju+gyBTz0M/Y+Hey+D+64KAqGx9VNGe+RF+NZF47jryhN5e8c+zrltAQ+8XJ3CokUklTT7aLY4sBue+i4smwv7tkPhIJhwKVReCf2PbXWzjTX7uXHuUl5at5MLKwfzxbOO0xXJIl1QotlHFQTZpqEOVj8CS+bA6kfBG2HoZJh4JYz9aDB/UQuNTc7t/1jDbf9YTWOTM31kKZdMLuesMQMpiMaf30hEMouCQOLbswVeuQ+WzoFtr0FuDxhzPlReARWnBfMZNbOxZj/3L67mvoVVbKjZT3HPKBdNHMKlk8s5ftCRASIimUNBIIm5w4bFsOS3sPwBOLgLiofBhMuh8nLoO/w9zZuanGff3M7chVX8fcUW6hqbmFBezKzJ5Zw3YTCF+ZrLUCTTKAgkefX7YdVfYelvYe1TgMOI9wfHEkafB3nvPT6wc28df1yygfsWrueNLbX0zItw7vgyLp1czonD+hK7JbWIpJmCQI5OTRUsuzcYOnpnHeT1hoppwbBRxXQYNO7wPRDcnSVVNcxbWMWDyzayr66RkQMKmTW5nIsmDqGk8MjpsUUkdRQE0jFNTfD2s7D8D7BuAexYEyzP7wPDpwahMOI0GHgC5ESoPdjAQ69sZO7CKpasryEaMT48ZiAfn1TOlIp+9NLQkUjKKQikc+3eFATDugXw1oJgkjuAgj4wfFoQDBXTYeAJvLFtH/ctrOKBl6t5Z189ZjCitBdjB/dh7OAiToj97NsrL73/JpFuTkEg4dq9MZjCYt2C4LFzbbD8cDCcRl35VJ7ZM5BXNuxhxcbdrNy4mw01+w+/xeA+BYwZ3IcThhQdDomyPgU6xiDSSRQEklq7NrzbY1j3zLvBkN8nOBupqAx6l7G/YCDVjcWs3l/IK7t68uL2fJbuzME9+PDv1yuPsYOLGDM4CIcxZUE49MyLKCBE2iltQWBmM4BbgQjwc3f/Tov1VwA3x17WAv/i7ssSvaeCoAvaVR30GKpehN0bgh7Enk2w98gZTj2ST12PAezKLWFTUz/WHuzNa/sK2dTYly3elz30oD63J3k9+9CzsA9Fhb0p6Z1PSWE+Jb3yKC3Mp6Qwj5Je+ZQW5tGvVx65Ec2kIpKWIDCzCPAG8GGgGlgIXObuK5u1mQqscvd3zOxs4OvufnKi91UQdCMNdVC7JQiFQ+GwZ1NwDKL5svp9rb5FIznso4BaL6DWe7CXAvZ6AXspoJYe7PUCGnN74vmF5OQVkhPNIycnSk5uLpHcPCK5wfPc3Ci50TxyI7lEo3nkRqPBz7w8orlR8vKi5EXziEQiWE4OkZwccnIi5OTkkNt8WSQSW5dD5PDzCFhOcI9piP20Fj+Js6y1n7zbvrnm69RjkhYSBUGYp29MAda4+9pYEXOBC4DDQeDuzzVr/wIwNMR6JNPk5kFxefBojTsc3P1uOBzcA3V7oa4WDu4hUldL74O1FNbV0rB/D/X7d9O4fw9eV4vVbSZSv5do4z6iB+tA99uhqWV4tOBtru+Yzo6nlvUe+Tr5WqxF65av21NH68s6ZmXFJ6m8+ocdfJcjhRkEQ4CqZq+rgUTf9j8NPBxvhZldB1wHMGzYsM6qT7oCs+Cgc0EfGHB8682AaOwRV2N9EB5NjcHzpobYoxGaGmhsrKeuvo66unrq6w5SV1dHfX2wrKG+PvY4SKM73tREU1MT7k14UyPuHryOPfemJpq8CQ638eCnN8Y+CRy86fBzx5st9yN+unvwofSe3nvLjxRvtshbadbGvSXa/JTq+L0pOvRB2MbGbX+Qe5wPZ3vP2pbx4O9Zn9zvbW1Zh1MA6D1oSsffJI4wgyBe+MfdFWZ2BkEQTI+33t3vBu6GYGioswqULBKJQo++ra8GesQeItkmzCCoBpr3+YcCG1s2MrPxwM+Bs919R4j1iIhIHGGeTrEQGGVmI8wsD5gFPNi8gZkNAx4ArnL3N0KsRUREWhFaj8DdG8zseuARgp73Pe6+wsxmx9bfBXwNKAHujJ0X3tDaUW0REQmHLigTEckCiU4f1ZU2IiJZTkEgIpLlFAQiIllOQSAikuW63MFiM9sGvH2Um5cC2zuxnM6W6fVB5teo+jpG9XVMJtc33N37x1vR5YKgI8xsUSafnprp9UHm16j6Okb1dUym19caDQ2JiGQ5BYGISJbLtiC4O90FtCHT64PMr1H1dYzq65hMry+urDpGICIiR8q2HoGIiLSgIBARyXLdMgjMbIaZvW5ma8zsljjrzcxui61/xcxOTGFt5Wb2hJmtMrMVZnZDnDanm9kuM1sae3wtVfXFfv86M3s19ruPmOEvzfvvuGb7ZamZ7TazG1u0Sfn+M7N7zGyrmS1vtqyfmf3dzFbHfsa9M05bf68h1vc9M3st9t/wj2ZW3Mq2Cf8eQqzv62a2odl/x5mtbJuu/Xdfs9rWmdnSVrYNff91mMduhdddHgRTXr8JvA/IA5YBY1q0mUlwW0wDTgFeTGF9ZcCJsee9gTfi1Hc68Nc07sN1QGmC9Wnbf3H+W28muFAmrfsPeD9wIrC82bL/AW6JPb8F+G4r/4aEf68h1ncWkBt7/t149SXz9xBifV8HvpTE30Ba9l+L9T8Avpau/dfRR3fsEUwB1rj7WnevA+YCF7RocwHwaw+8ABSbWVkqinP3Te7+cuz5HmAVwf2du5K07b8WzgTedPejvdK807j708DOFosvAH4Ve/4r4MI4mybz9xpKfe7+qLs3xF6+QHAXwbRoZf8lI2377xALbqZyCXBvZ//eVOmOQTAEqGr2upojP2iTaRM6M6sAJgIvxll9qpktM7OHzWxsaivDgUfNbLGZXRdnfUbsP4K73rX2P186998hA919EwRfAIABcdpkyr68hqCXF09bfw9huj42dHVPK0NrmbD/TgO2uPvqVtanc/8lpTsGgcVZ1vIc2WTahMrMCoE/ADe6++4Wq18mGO6YAPwY+FMqawOmufuJwNnAZ83s/S3WZ8L+ywPOB34fZ3W69197ZMK+/ArQAMxppUlbfw9h+QlwDFAJbCIYfmkp7fsPuIzEvYF07b+kdccgqAbKm70eCmw8ijahMbMoQQjMcfcHWq53993uXht7Ph+Imllpqupz942xn1uBPxJ0v5tL6/6LORt42d23tFyR7v3XzJZDQ2axn1vjtEn33+IngXOBKzw2oN1SEn8PoXD3Le7e6O5NwM9a+b3p3n+5wEeB+1prk6791x7dMQgWAqPMbETsW+Ms4MEWbR4EPhE7++UUYNehLnzYYuOJvwBWufsPW2kzKNYOM5tC8N9pR4rq62VmvQ89JziguLxFs7Ttv2Za/RaWzv3XwoPAJ2PPPwn8OU6bZP5eQ2FmM4CbgfPdfV8rbZL5ewirvubHnS5q5fembf/FfAh4zd2r461M5/5rl3QfrQ7jQXBWyxsEZxN8JbZsNjA79tyAO2LrXwUmpbC26QRd11eApbHHzBb1XQ+sIDgD4gVgagrre1/s9y6L1ZBR+y/2+3sSfLD3abYsrfuPIJQ2AfUE31I/DZQAjwOrYz/7xdoOBuYn+ntNUX1rCMbXD/0d3tWyvtb+HlJU329if1+vEHy4l2XS/ost/+Whv7tmbVO+/zr60BQTIiJZrjsODYmISDsoCEREspyCQEQkyykIRESynIJARCTLKQhEUsiCmVH/mu46RJpTEIiIZDkFgUgcZnalmb0Um0P+p2YWMbNaM/uBmb1sZo+bWf9Y20oze6HZvP59Y8tHmtljscnvXjazY2JvX2hm91twL4A5h66CFkkXBYFIC2Y2GriUYLKwSqARuALoRTC/0YnAU8B/xDb5NXCzu48nuBL20PI5wB0eTH43leDKVAhmnL0RGENw5em0kP9JIgnlprsAkQx0JnASsDD2Zb0HwYRxTbw7udhvgQfMrA9Q7O5PxZb/Cvh9bH6ZIe7+RwB3PwAQe7+XPDY3TeyuVhXAM6H/q0RaoSAQOZIBv3L3f3vPQrN/b9Eu0fwsiYZ7DjZ73oj+P5Q009CQyJEeBy42swFw+N7Dwwn+f7k41uZy4Bl33wW8Y2anxZZfBTzlwT0mqs3swth75JtZz1T+I0SSpW8iIi24+0oz+yrBXaVyCGac/CywFxhrZouBXQTHESCYYvqu2Af9WuBTseVXAT81s2/E3uPjKfxniCRNs4+KJMnMat29MN11iHQ2DQ2JiGQ59QhERLKcegQiIllOQSAikuUUBCIiWU5BICKS5RQEIiJZ7v8D6wSmyXtK+W8AAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pre Trained Classifier\n",
        "The Encoder is fitted with a few dense layers that can be used for classification."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "source": [
        "encoder_classifier = keras.models.clone_model(conv_encoder)\r\n",
        "encoder_classifier.add(keras.layers.Dense(40, activation='selu', activity_regularizer='l2'))\r\n",
        "encoder_classifier.add(keras.layers.Dense(20, activation='selu', activity_regularizer='l2'))\r\n",
        "encoder_classifier.add(keras.layers.Dense(1, activation='sigmoid'))\r\n",
        "encoder_classifier.summary()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv1d (Conv1D)              (None, 147, 96)           73824     \n",
            "_________________________________________________________________\n",
            "max_pooling1d (MaxPooling1D) (None, 73, 96)            0         \n",
            "_________________________________________________________________\n",
            "activation (Activation)      (None, 73, 96)            0         \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 58, 128)           196736    \n",
            "_________________________________________________________________\n",
            "max_pooling1d_1 (MaxPooling1 (None, 14, 128)           0         \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 14, 128)           0         \n",
            "_________________________________________________________________\n",
            "conv1d_2 (Conv1D)            (None, 1, 50)             89650     \n",
            "_________________________________________________________________\n",
            "reshape (Reshape)            (None, 50)                0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 40)                2040      \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 20)                820       \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 1)                 21        \n",
            "=================================================================\n",
            "Total params: 363,091\n",
            "Trainable params: 363,091\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "source": [
        "early_stopping = keras.callbacks.EarlyStopping(monitor=\"val_accuracy\", patience=8, verbose=1, restore_best_weights=True)\r\n",
        "encoder_classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics='accuracy')\r\n",
        "encoder_classifier_history = encoder_classifier.fit(x=train_ae, y=tf.convert_to_tensor(train_y), epochs=30, batch_size=256, validation_data=(val_ae, tf.convert_to_tensor(val_y)), callbacks=[early_stopping])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/30\n",
            "172/172 [==============================] - 5s 21ms/step - loss: 0.7508 - accuracy: 0.5066 - val_loss: 0.6920 - val_accuracy: 0.5223\n",
            "Epoch 2/30\n",
            "172/172 [==============================] - 4s 21ms/step - loss: 0.6437 - accuracy: 0.6328 - val_loss: 0.6058 - val_accuracy: 0.6783\n",
            "Epoch 3/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.5539 - accuracy: 0.7282 - val_loss: 0.5724 - val_accuracy: 0.7140\n",
            "Epoch 4/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.4758 - accuracy: 0.7879 - val_loss: 0.5370 - val_accuracy: 0.7327\n",
            "Epoch 5/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.4065 - accuracy: 0.8294 - val_loss: 0.5942 - val_accuracy: 0.7107\n",
            "Epoch 6/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.3212 - accuracy: 0.8794 - val_loss: 0.6023 - val_accuracy: 0.7410\n",
            "Epoch 7/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.2365 - accuracy: 0.9231 - val_loss: 0.7110 - val_accuracy: 0.7203\n",
            "Epoch 8/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.1746 - accuracy: 0.9524 - val_loss: 0.7817 - val_accuracy: 0.7223\n",
            "Epoch 9/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.1196 - accuracy: 0.9775 - val_loss: 0.8289 - val_accuracy: 0.7180\n",
            "Epoch 10/30\n",
            "172/172 [==============================] - 4s 21ms/step - loss: 0.0847 - accuracy: 0.9916 - val_loss: 0.8871 - val_accuracy: 0.7173\n",
            "Epoch 11/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.0603 - accuracy: 0.9984 - val_loss: 0.8740 - val_accuracy: 0.7223\n",
            "Epoch 12/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.0442 - accuracy: 1.0000 - val_loss: 0.8351 - val_accuracy: 0.7217\n",
            "Epoch 13/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.0366 - accuracy: 1.0000 - val_loss: 0.8264 - val_accuracy: 0.7227\n",
            "Epoch 14/30\n",
            "172/172 [==============================] - 3s 20ms/step - loss: 0.0323 - accuracy: 1.0000 - val_loss: 0.8330 - val_accuracy: 0.7183\n",
            "Restoring model weights from the end of the best epoch.\n",
            "Epoch 00014: early stopping\n"
          ]
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "source": [
        "print(f\"Test Set metrics and Training History:\")\r\n",
        "encoder_classifier.evaluate(x=test_ae, y=tf.convert_to_tensor(test_y))\r\n",
        "np.save(\"./models/encoder_classifier.npy\", encoder_classifier_history.history)\r\n",
        "# encoder_classifier_history = np.load(\"./models/encoder_classifier.npy\", allow_pickle='TRUE').item()\r\n",
        "encoder_classifier_plot = HistoryPlotter(history=encoder_classifier_history.history, accuracy_line=None, max_y_lim=[0.5, 1])\r\n",
        "encoder_classifier_plot.all(row_col=[1,0])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Set metrics and Training History:\n",
            "94/94 [==============================] - 0s 3ms/step - loss: 0.5336 - accuracy: 0.7613\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAEWCAYAAABPDqCoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABdZUlEQVR4nO3dd3xUVfrH8c+TSQ8JCUnovfcaioCKCAoWUAGpihV7W9dV1/3p6q67rrp2FEFREWkiWABBYSlK7z30ktBSgJBC+vn9cQcMIYEAmdyZyfN+veY1M/femfkOCSfPnDn3HDHGoJRSSimllCoZH7sDKKWUUkop5Um0gFZKKaWUUuoSaAGtlFJKKaXUJdACWimllFJKqUugBbRSSimllFKXQAtopZRSSimlLoEW0MrriUhdETEi4luCY+8Rkd/LIpdSSnkzbXuVN9MCWrkVEdkvItkiElVo+wZnQ1zXpmiX9MdAKaU8iTu3vQWyhIhImojMsTuLUlpAK3e0Dxh65o6ItAKC7IujlFLlgru3vQOBLOAGEalWli+sHSeqMC2glTv6Gri7wP2RwISCB4hIRRGZICKJInJARP4mIj7OfQ4ReVtEkkRkL3BzEY/9XESOiMghEfmniDiuJLCIVBeRH0XkuIjsFpEHC+zrJCJrROSUiBwTkXec2wNFZKKIJIvISRFZLSJVriSHUkpdAXdve0cCY4BNwPBCz91dRJY529I4EbnHuT1IRP7rzJoiIr87t/UQkfhCz7FfRHo5b/9dRKY72+hTwD3Otny58zWOiMhHIuJf4PEtRORX59+BYyLyVxGpKiIZIhJZ4LgOzn8/v0t478rNaAGt3NEKIExEmjkb18HAxELHfAhUBOoD12I1+vc69z0I3AK0A2Kwei0K+grIBRo6j7kBeOAKM08G4oHqztf7l4hc79z3PvC+MSYMaABMc24f6XwPtYBI4GHg9BXmUEqpy+W2ba+I1AZ6AN84L3cX2vezM1s00BbY4Nz9NtAB6ApUAv4C5JfkNYH+wHQg3PmaecAzQBRwFXA98KgzQygwH5iL9XegIbDAGHMUWATcWeB5RwBTjDE5Jcyh3JAW0MpdnekJ6Q3EAofO7CjQsL9ojEk1xuwH/gvc5TzkTuA9Y0ycMeY48O8Cj60C9AWeNsakG2MSgHeBIZcbVERqAd2B540xmcaYDcBnBfLkAA1FJMoYk2aMWVFgeyTQ0BiTZ4xZa4w5dbk5lFKqFLhr23s3sMkYsw2rw6KFiLRz7hsOzDfGTDbG5Bhjko0xG5w94/cBTxljDjnb2WXGmKwSvuZyY8z3xph8Y8xpZxu9whiT63zvn2J9iADrg8NRY8x/nX8HUo0xK537vsIqms/8Gw7F+ndWHkzH9Ch39TWwBKhHoa8QsT79+wMHCmw7ANRw3q4OxBXad0YdwA84IiJntvkUOv5SVQeOG2NSC71mjPP2/cBrQKyI7ANeNcbMwnqPtYApIhKO1dPzkvZKKKVs5K5t793AOABjzGERWYz1Ld56rHZ0TxGPiQICi9lXEudkE5HGwDtYbXswVg211rm7uAwAPwBjRKQ+0BhIMcasusxMyk1oD7RyS8aYA1gntNwEzCi0Owmr97ZOgW21+aOn5AhWY1Zw3xlxWCehRBljwp2XMGNMiyuIexio5PwK77w8xphdxpihQGXgP8B0EQlx9pS8aoxpjvX14i2cO/5QKaXKlDu2vSLSFWgEvCgiR0XkKNAZGOo8uS8Oa3hcYUlAZjH70rGK4DOv4cAa/lGQKXT/E6xe+UbOIXl/Bc58GiguA8aYTKyhe8Oxeuu199kLaAGt3Nn9QE9jTHrBjcaYPKzG6HURCRWROsCf+GOs3jTgSRGpKSIRwAsFHnsE+AX4r4iEiYiPiDQQkWspuQDnCYCBIhKI9cdjGfBv57bWzuzfAIjICBGJNsbkAyedz5EnIteJSCtnw30K6w9T3iXkUEopV3C3tnck8CvQHGt8c1ugJVYB3Berre0lIneKiK+IRIpIW2ebOx54R6wTvR0icpWIBAA7gUARudl5Mt/fgICL5AjFaqvTRKQp8EiBfbOAqiLytIgEOP99OhfYPwG4B+jH+ePKlQfSAlq5LWPMHmPMmmJ2P4HVg7AX+B2YhNVQgvU13zxgI7CO83tR7sb6GnIbcALrJJFLmRIpDetkvzOXnlhj2upi9UbPBF4xxvzqPL4PsFVE0rBOKBzi7JGo6nztU8B2YDHasCqlbOZOba+zk+JO4ENjzNECl31YPbkjjTEHsXrMnwWOY51A2Mb5FH8GNgOrnfv+A/gYY1KwTgD8DKsTJB3rRPAL+TMwDEh1vtepZ3Y4h/D1Bm4FjgK7gOsK7F+KdfLiOuf4aeXhxJjC31AopZRSSqnSJCL/AyYZYz6zO4u6clpAK6WUUkq5kIh0xBqGUqvQCefKQ+kQDqWUUgCIyHgRSRCRLcXsFxH5QKzFgjaJSPuyzqiUpxGRr7DmiH5ai2fvoT3QSimlABCRa7DG+E8wxrQsYv9NWGNgb8KaBeF9Y0znwscppZS30x5opZRSABhjlmCdaFWc/ljFtXEuCBQuIpdyAq5SSnkFj1tIJSoqytStW9fuGEopdVnWrl2bZIwpPN+sp6jBuYtLxDu3HSl4kIiMAkYBhISEdGjatGmZBVRKQb4xZOXmk52bf/Y6Nz8fY6x9xV7bHdyFQgN8qRsVcsmPK67N9rgCum7duqxZU9zsOkop5d5E5MDFj3JbUsS28/7mGmPGAmMBYmJijLbZSpUuYwwnMnLYn5zOgeR0DiRncCA5g/3J6RxMzuB4evbZY32AWqEBVAkLINDXQaCfgwBfHwL8fAjwdRDovLa2Oa99fQoc5zj3foHbfg4ffHyKahYsxe8BudBOF/B3+BBZ4WJTfZ+vuDbb4wpopZRStonn3JXmamLNfa6UKmX5+YaE1KyzBfL+5HQOHM+w7idlkJqVe/ZYEaheMYg6kcHc0KIqdSODqRMZTJ3IEGpXCiYkQMu90qb/okoppUrqR+BxEZmCdRJhinOFOaXUFUpIzWTxjkQW70xk17E0DhxPJzMn/+x+Xx+hViWrMO5QO4I6kSHUjQqmdqUQalUKIsDXYWP68kcLaKWUUgCIyGSgBxAlIvHAK4AfgDFmDDAHawaO3UAGcK89SZXyfHn5hg1xJ1i0I5GFOxLYcugUAJVDA2hTK5yrG0VRJyrE6k2uFEL18EB8HTr3g7vwigI6JyeH+Ph4MjMz7Y7icoGBgdSsWRM/Pz+7oyilvIwxZuhF9hvgsTKKo5TXSUrLYsnORBbuSOS3XYmczMjB4SO0rx3Oczc2oUeTaJpXC0PKeoBwMbS+Kp5XFNDx8fGEhoZSt25dt/mlcwVjDMnJycTHx1OvXj274yillFLqAvLzDZsOpbAwNoFFOxLYdCgFYyCqQgDXN63CdU2jubphNBWD3bNTTOur4nlFAZ2Zmen1P1wAESEyMpLExES7oyillFKqCCfSs1myK5FFzvHMx9OzEYF2tcL5U6/G9GhSmRbVwy44e4W70PqqeF5RQANe/8M9o7y8T6WUUsoT5Ocbth4+xcIdVi/zhriT5BuoFOLPtY2j6dEkmmsaRRMR4m931MtSXuqOS32fXlNAK6WKcPokbJgE/iEQ2QAq1YfQamU/AadSSnmZ2KOn+Py3fSzckUhSWhYi0LpmOE9e34geTSrTukZFj+hlVpdHC+hScPLkSSZNmsSjjz56SY+76aabmDRpEuHh4a4JpsovY2D7jzDnL5B29Nx9fsEQUQ8i61sFdaX6UKlAce2jZ3krpVRxdiek8d78nczefIQQf196Nq3MdU2tXubLWahDXZi71lhaQJeCkydP8vHHH5/3w83Ly8PhKH5exjlz5rg6miqPUuJh9p9h589QtTUMmQQhkXB8LyTvgeP74PgeSNwBO+dB3h8rVuEbBJXqFSis6xfoua6uxbVSqtw6kJzO+wt28f36QwT6OXi0RwMevLo+4cGeOTTDU7hrjaUFdCl44YUX2LNnD23btsXPz48KFSpQrVo1NmzYwLZt27jtttuIi4sjMzOTp556ilGjRgF/LEuelpZG37596d69O8uWLaNGjRr88MMPBAUF2fzOlEfJz4NV4+B//wCTDzf8Ezo/Ag7nf/OIutCg5/mPSYm3iuvjzuI6eQ8k7YJdvxQqrgOtnutK9a3e63o9rOfTolop5cUOnTzNhwt28e3aeHx9hAeurs9D19TX3uYy4q41ltcV0K/+tJVth0+V6nM2rx7GK7e2KHb/G2+8wZYtW9iwYQOLFi3i5ptvZsuWLWenQhk/fjyVKlXi9OnTdOzYkQEDBhAZGXnOc+zatYvJkyczbtw47rzzTr777jtGjBhRqu9DebGjm+HHJ+HwOmjYC25+ByLqXPxxPg7ruIg60OC6c/fl58GpQwV6rvf+0Xu9ez4s+9Aqpjs+CG2HQVC4S96aUkrZ4dipTEYv3M2UVXEA3NWlDo/2aEDlsECbk9nDjvoK3LfG8roC2h106tTpnHkEP/jgA2bOnAlAXFwcu3btOu+HW69ePdq2bQtAhw4d2L9/f1nFVZ4sOwMW/8cqZoMrwYDPoeWA0jlJ0McB4bWtS/0e5+7LzbbGWK8aC/NetHq9Ww+GTg9ClQs3hkop5c6S0rL4ZNEeJq44QF6+YVBMLZ7o2ZDq4fqtsDtwlxrL6wroi32SKQshISFnby9atIj58+ezfPlygoOD6dGjR5Er+gQE/PFVkMPh4PTp02WSVXmw3Qtg1jNw8gC0uwt6v2YV0WXB1x9aDbQuhzfA6nGwcTKs/QLqdLcK6aY3g8M9FwdQSqnCTqRn8+mSvXy1bD9ZuXnc0b4mT/ZsRO3IYLujuQV3qK/AfWosryug7RAaGkpqamqR+1JSUoiIiCA4OJjY2FhWrFhRxumU10lPgnl/hU1TIbIh3DMb6na3L0/1ttB/NPT+B6z/GlZ/Bt+OtE46jLkPOoyECpXty6eUUheQcjqHz3/fx/jf95GencutravzVK9GNIiuYHc0hfvWWFpAl4LIyEi6detGy5YtCQoKokqVKmf39enThzFjxtC6dWuaNGlCly5dbEyqPJoxVi/vvL9CVhpc+zx0/xP4ucl4vOBK0O0puOpx6wTEVWNh4T+tISYtbodOo6BmjM5BrZRyC2lZuXy5dB9jl+zlVGYufVtW5elejWlSNdTuaKoAd62xxBhTZi9WGmJiYsyaNWvO2bZ9+3aaNWtmU6KyV97er8I6iW/W07BvCdTqAre+D5Wb2p3q4pJ2WT3S67+B7FSo1tYqpFveAX7lczyhiKw1xsTYnaOsFNVmK2Wn09l5fL1iP2MW7+V4eja9mlXm6V6NaVmjot3R3E55qzeKer/FtdnaA62UO8vNhmUfwOI3rWnkbnkX2t/jOVPHRTWCvv+Bnn+DjVOsafZ+eBR++Ru0vxs63m+dpKiUUi6WmZPH5FUH+XjRHhJTs7i6URR/6t2YdrUj7I6mPJAW0Eq5q7hV8NNTkLANmt9mFaKhVe1OdXkCQq0TCzs+YPWirxprfTBY9gE07mvtq99Dh3copUpdfr5h5vpDvP3LDo6kZNK5XiVGD2tPp3pldNK18kpaQCvlbjJTYMFrsPpzCKsBQ6dAk752pyodIlD/WutyMg7WjId1X8GO2RDVGFoOhIo1oEJV68TDClUgJMqaUs+d5OdZF19dgUwpd7bt8Cle/mELaw6coE2tcN4e1IauDSIR/bCurpAW0Eq5k+0/wZznIO0YdHkErnsJArz0TPDwWtDrFetkyK0zrV7pRf86/zjxgeAoCK1iFdQVqvxRXFcotC0g9NJ6sfPzIfMknD4Bp0/C6ePW7Qzn9ekTRW/LTIFef4fuT5fOv4VSqlSlnM7h3V93MmH5fsKD/XlzQGsGdqiJj48Wzqp0aAGtlDs4uNKasWLfEqjSCoZ8AzU62J2qbPgFQtuh1iU7w/rwkJbgvC58+xgkbLeu83OLeK7gAsW18zooArJSCxXBx/8omrnAidSBFa3HB1WyriMbOO9HQG2dUUcpd5Ofb5ix/hBv/Lyd4+nZDO9ch2dvaEx4sH5bpEqXFtBK2enQOlj4urU0dkg09HnDGidcXhcg8Q+GSvWsy4Wc6Tk+U1SnHju/4E7aDft/t4rkgDBrqfFgZyEcUeePojgo4o/tBbcFhbvf0BGlVLG2Hk7h5R+2svbACdrXDufLezvpzBrKZbSAtkGFChVIS0uzO4ay09EtsPBf1tjfoAjo9ap1Ip1/yMUfq6xZSIIrWZfKF5liyRg9OVEpL5ZyOod3ftnB1ysOEBHsz5sDWzOwvQ7XKK/KqsbSAlqpspQQC4v+Ddu+h4CK1hjnzg9DYJjdybyXFs9KeaX8fMP0dfH85+dYTmRkM6JLHZ7t3YSKweX0GzxVplxaQItIH+B9wAF8Zox5o9D+isBEoLYzy9vGmC9cmckVnn/+eerUqcOjjz4KwN///ndEhCVLlnDixAlycnL45z//Sf/+/W1OqmyTvAcWvQGbv7V6ma95Dq56zOp9VkopdUm2HErh5R+2sO7gSdrXDuer+3S4hrdy1xrLZQW0iDiA0UBvIB5YLSI/GmO2FTjsMWCbMeZWEYkGdojIN8aY7Mt+4Z9fgKObryT6+aq2gr5vFLt7yJAhPP3002d/uNOmTWPu3Lk888wzhIWFkZSURJcuXejXr59OnVPenDhgLYKycTI4/KHbk9D1KQiJtDuZUkp5nJSMHP776w4mOodrvDWwNQN0uEbZsKG+AvetsVzZA90J2G2M2QsgIlOA/kDBAtoAoWK94wrAcaCIU+vdW7t27UhISODw4cMkJiYSERFBtWrVeOaZZ1iyZAk+Pj4cOnSIY8eOUbWqhy6EoS5NyiH47W1YNwHEAZ0fgu7PWDNDKKWUuiT5+Ybpa+P5z1xruMZdXerwpxuaUDFIh2t4O3etsVxZQNcA4grcjwc6FzrmI+BH4DAQCgw2xuQXfiIRGQWMAqhd+yLL/l7kk4yrDBw4kOnTp3P06FGGDBnCN998Q2JiImvXrsXPz4+6deuSmZlpSzZVhlKPwe/vwJovwORD+5FwzZ8hrLrdyZRSyiNtOZTC//2whfUHTxJTJ4IJ/TvRoroO1yhzNtVX4J41lisL6KL60QtPuHojsAHoCTQAfhWR34wxp855kDFjgbEAMTExF5i01T5DhgzhwQcfJCkpicWLFzNt2jQqV66Mn58fCxcu5MCBA3ZHVK6UngRL34NVn0FeNrQdZo1zjqhjdzKllPJIKRk5vP3LDiauPEBkiD//HdSGO9rX0KGQ5ZA71liuLKDjgVoF7tfE6mku6F7gDWOMAXaLyD6gKbDKhblcokWLFqSmplKjRg2qVavG8OHDufXWW4mJiaFt27Y0bdrU7ojKFU6fgGUfwspPIScDWt0J1/7FWnBDKaXUJTszXOONubGczMhm5FV1eaZ3Yx2uUY65Y43lygJ6NdBIROoBh4AhwLBCxxwErgd+E5EqQBNgrwszudTmzX8Mro+KimL58uVFHqdzQHuBzFOw4hNYPhqyUqDFHdDjBYhuYncypZTyWCmnc3hk4lqW7UmmY90IXu3XmebVdZpP5X41lssKaGNMrog8DszDmsZuvDFmq4g87Nw/BvgH8KWIbMYa8vG8MSbJVZmUKhWH1sK0eyDlIDS9BXq8CFVb2p1KKaU82pGU09wzfjV7k9L49x2tGNKxlg7XUG7LpfNAG2PmAHMKbRtT4PZh4AZXZlCq1BgDaz6HuS9Chapw/69Qq5PdqZRSyuPtPJbKyPGrSM3M5ct7O9GtYZTdkZS6IK9ZidAYUy4+qVrDxVWZy06Hn56GzdOgYW+4Y6y1jLRSSqkrsmrfcR74ajUBfg6mPtRFZ9hwM1pfFc3HRTnKVGBgIMnJyV5fXBpjSE5OJjAw0O4o5UviThjX01pF8Lq/wbBpWjwrpVQp+HnzEUZ8vpKo0ABmPNJVi2c3o/VV8byiB7pmzZrEx8eTmJhodxSXCwwMpGbNmnbHKD+2zIAfnwDfQLhrJjS4zu5ESinlFb5atp+//7SVdrXC+XxkRyJC/O2OpArR+qp4XlFA+/n5Ua9ePbtjKG+Smw2//h+sHAO1OsPAL6BiDbtTKaWUxzPG8Oa8HXyyaA+9mlXhw6HtCPJ32B1LFUHrq+J5RQF9MVm5eWRk5emnW1UyKYfg23sgfhV0eRR6vwYOnX9UKaWuVHZuPi98t4kZ6w8xrHNtXuvXAl+HV4wmVeWM1//WGmN4cMJa7vtqNZk5eXbHUe5uz//g06shYRsM+hL6/FuLZ1VuiEgfEdkhIrtF5IUi9keIyEwR2SQiq0RE529UJZaWlcv9X61mxvpDPNu7Ma/f1lKLZ+WxvP43V0QY1qk2G+JO8uy0jeTne/dAeHWZ8vNh8Zvw9R0QUhlGLYIWt9udSqkyIyIOYDTQF2gODBWR5oUO+yuwwRjTGrgbeL9sUypPlZCayZCxy1m2J5k3B7TmiesblYuZHZT38voCGqBPy6r8tW8zZm8+wlu/7LA7jnI3Gcdh0iBY+Dq0vhMeXABRjexOpVRZ6wTsNsbsNcZkA1OA/oWOaQ4sADDGxAJ1navIKlWsvYlpDPhkGXsS0vns7hju7FjL7khKXbFyMQYa4IGr67E/OZ1PFu2hTqVghnSqbXck5Q4OrYVpIyHtGNz8DsTcB9orosqnGkBcgfvxQOdCx2wE7gB+F5FOQB2gJnCs4EEiMgoYBVC7tra15dn6gye4/6s1AEwe1YW2tcLtDaRUKSkXPdBgDeV4tV8Lrm0czUvfb+G3Xd4/JYu6AGNg9Wcwvg8gcN886Hi/Fs+qPCvql7/wmLc3gAgR2QA8AawHcs97kDFjjTExxpiY6OjoUg+qPMP/Yo8xbNxKKgT48t0jXbV4Vl6l3BTQAL4OHz4a1o5GlSvw6MR17DyWanckZYfsdJgxCmY/C/V7wEOLoUZ7u1MpZbd4oOB36zWBwwUPMMacMsbca4xpizUGOhrYV2YJlceYsuogD05YS8PKFfjuka7UiwqxO5JSpapcFdAAoYF+jL+nI0H+Du79YjUJqZl2R1Jl6cyqglumQ8+/wdCpuqqgUpbVQCMRqSci/sAQ4MeCB4hIuHMfwAPAEmPMqTLOqdyYMYb35+/ihRmb6dYwiimjuhAdGmB3LKVKXbkroAGqhwfx+ciOHE/P5sGv1nA6W6e3Kxe2zIBx10F6krWq4DXPgU+5/C+g1HmMMbnA48A8YDswzRizVUQeFpGHnYc1A7aKSCzWbB1P2ZNWuaPcvHz+OnML787fyR3ta/D5yBhCAsrNqVaqnCm3v9mtalbkg6HtGPX1Gp6eup5PhnfAx0fHv3olXVVQqRIxxswB5hTaNqbA7eWATlGjznM6O48nJq9j/vYEHruuAX++oYlOU6e8WrnufuvdvAr/d3Nz5m09xhtzY+2Oo1whKw2+utUqnrs8CvfM1uJZKaVK0fH0bIZ9toIFsQm81r8Fz93YVItn5fXKbQ/0Gfd2q8uB5HTGLtlL7UrBjOhSx+5IqrTk51knC8avggGfQ6uBdidSSimvEnc8g5HjVxF/8jSfDG9Pn5bV7I6kVJko9wW0iPB/tzQn7sRpXvlxKzUjgujRpLLdsVRpmP8K7JgNfd/U4lkppUrZ7oQ0ho1bQWZOHt880JmOdfWEbFV+lOshHGf4Onz4cGg7mlQJ5fFJ69l+RE8q93hrv4JlH0LHB6HzQ3anUUopr7IvKZ1h41aQbwzTH+mqxbMqd8pHAX1gOexZeMFDQgJ8GX9PRyoE+HLfl6s5dkqnt/NYexfD7D9Bw17Q5w270yillFc5mJzBsHEryM03fPNAFxpXCbU7klJlzvsLaGNgwWsw6U7Y/tMFD61aMZDP74kh5XQO93+1mozs8xbYUu4uaRdMuwsiG8HA8eAo96OUlFKq1MSfyGDouBVkZOcx8f7ONKmqxbMqn7y/gBaBoZOgWhuYdjdsmHTBw1tUr8hHw9qx7fApnpy8gbz8wivZKreVcRy+GQQ+fjBsCgRWtDuRUkp5jSMppxk2biWnMnOYeH9nmlcPszuSUrbx/gIaICgC7voe6l0D3z8CK8Zc8PCeTavw934tmL/9GK/P3l42GdWVyc2CqSPg1GEYOhki6tqdSCmlvMaxU5kMG7eS4+nZfH1/Z1rV1A4KVb6VjwIaIKACDJsGTW+Buc/D4jet4R3FuPuqutzXrR7jl+7jq2X7yy6nunTGwE9Pw4GlcNvHUKuT3YmUUsprJKZmMWzcChJOZfLVfR1pWyvc7khK2a78FNAAvgEw6CtoMwwWvg7zXrpgEf3Szc3o1awKr/60lf/FHivDoOqS/P4ubJwEPV7U6eqUUqoUJadZxfPhk5mMv6cjHerobBtKQXkroME6qaz/aOj8MKwYDT8+bi24UdShPsIHQ9vSvHoYj09az9bDKWUcVl3Uth9gwavQciBc+7zdaZRSymucSM9m+GcrOXg8g89HxtC5fqTdkZRyG+WvgAbw8bGmN7v2BVg/Eb69xxpDW4Rgf18+H9mR8CA/7vtyNUdSTpdtVlW8Q+tgxkNQs5P1oUiXjlVKqVKRkpHDiM9Xsjcpnc9GxtC1YZTdkZRyK+WzgAar2LruRbjx37D9R5g8BLLTizy0Slggn9/TkfSsPO7/cg1pWTq9ne1S4q2fWYVoGDIJ/ALtTqSUUl7hVGYOd49fya5jaXx6VweubhRtdySl3E75LaDPuOpRq/dy7yL4+nY4fbLIw5pVC2P08PbsOJbKE5PWkZuXX6YxVQFZac4PPBnWiaEVtHFXSqnSkJaVyz3jV7H18Ck+Ht6e65pUtjuSUm5JC2iAdiNg0JfWkICvboG0hCIPu7ZxNK/1b8HCHYm8Nmsb5gInICoXyc+D7x6AY1utn1nlZnYnUkopr5CRncu9X6xiY3wKHw1rR6/mVeyOpJTb0gL6jOb9rcU3kvfA+D5wMq7Iw4Z3rsOoa+ozYfkBvli6v2wzKvj1Zdj5M/R9Exr1sjuNUkp5hdPZedz35WrWHjjBe4Pb0qdlNbsjKeXWtIAuqGEva8GV9CSriE7aVeRhL/RpSp8WVfnH7G1MXX2wbDOWZ2u+gOUfQaeHoNODdqdRSimvkJmTx4MT1rBy33HeubMtt7apbnckpdyeFtCF1e4M98yCvCyriD6y8bxDfHyE94a05ZpG0Tz/3WYmrjhgQ9ByZs9CmP0sNOwNN/7L7jRKKeUVsnLzeOjrtSzdk8SbA1pzW7sadkdSyiNoAV2Uaq3h3rngGwhf3goHV5x3SKCfg7F3d6BXs8r87fstfLF0nw1By4nEnTBtJEQ3gYHjrbm8lVJKXZHs3HwenbiOxTsT+fftrRgUU8vuSEp5DC2gixPVEO6ba83wMOE22DX/vEMCfB18PLwDN7aowqs/bWPskj1ln9PbpSfDpEHg6w/DpkJgmN2JlFLK4+Xk5fPE5HUsiE3gH7e1ZEin2nZHUsqjaAF9IeG1rJ7oqIbWtGlbZ553iL+vDx8Na8/NravxrzmxjF6424agXio3C6YOh1NHYMhkCNcGXimlrlRuXj5PT9nAvK3HeOXW5tzVpY7dkZTyOPpd+MVUiIaRs2DSYJh+H2SlQvu7zznEz+HD+4Pb4u/w4a15O8jOzefpXo0QXRnv8hkDPz0FB5dbwzZqdbQ7kVJKeby8fMOz325k9uYjvHRTM+7tVs/uSEp5JC2gSyIoHO6aCVNHwI9PQOYp6Pr4OYf4Onx4e1AbfH2E9xfsIicvn+dubKJF9OX67b+wcTJc9xK0HGB3GqWU8nj5+Ya/TN/EDxsO89yNTXjwmvp2R1LKY7l0CIeI9BGRHSKyW0ReKOaYHiKyQUS2ishiV+a5Iv7BMHQKNL8NfnkJ/ve61UtagMNH+M+A1gztVJuPF+3hX3O262Irl2PrTPjfP6DVnXDNc3anUUopj5efb3hxxma+WxfPM70a89h1De2OpJRHc1kPtIg4gNFAbyAeWC0iPxpjthU4Jhz4GOhjjDkoIu69ZqivvzWc4KdQWPImpCdCjxcgtOrZQ3x8hH/d3hJ/hzDut33k5BleubW59kSXVPxamPkw1OoM/T4E/XdTSqkrkpmTx5+/3cisTUd4/LqGPHm9Fs9KXSlXDuHoBOw2xuwFEJEpQH9gW4FjhgEzjDEHAYwxRa+h7U58HFZhFxQOyz6EdV9B/R7QejA0vQUCKiAi/L1fC/wcPnz2+z6y8/L5Z/+W+PhoMQhAdob14ePMJS3hj9tbZkCFKjBkEvgF2p1UKaU8WkJqJqMmrGVj/Ele6NuUh66prx06SpUCVxbQNYCC62HHA50LHdMY8BORRUAo8L4xZoILM5UOEbjhn9Dubtg0FTZNg5kPgV8INLsFWg9G6vfgpZub4efrwyeL9pCTm88bA1rj8MYi2hg4faJQQZwE6c7COO1Msezcnp1W9PMEhEFEXbhjHIRElelbUEopbxN79BT3f7mG4+nZjBnRgRtbVL34g5RSJeLKArqoSrHwgGBfoANwPRAELBeRFcaYnec8kcgoYBRA7dpuNJVZdGO4/v+sE93iVsDGKbD1e6uorlAVaTWQv7S5E3+fhrz/v93k5OVbJxo6PHz2wNwsWPoBbP/xj6I5P/f848QHgiMhpLJVENfsCCHR1qVC5T9un7loj7NSSpWKhbEJPD5pHRUCffn24atoWaOi3ZGU8iquLKDjgYLLGtUEDhdxTJIxJh1IF5ElQBvgnALaGDMWGAsQExPjfmfl+fhAna7Wpe+bsHOuVUSvHIMs/4hnKjenU7PreXZDY57ON7w7uC1+nlpE71sCs/4EybugTndr1caQaGeRHG1N+3fmdnAla8iLUkqpMvPl0n28NmsbzaqF8fnIjlStqJ0TSpU2VxbQq4FGIlIPOAQMwRrzXNAPwEci4gv4Yw3xeNeFmVzPLxBa3GZd0pNh6wzYNJVu+z5keaCwdHtzvhlzE8PueQL/EA/qEUhPgl/+Zk0tF1EXRnwHDXvZnUoppZRTbl4+r83axoTlB+jdvArvDW5LSIDOVquUK7jsf5YxJldEHgfmAQ5gvDFmq4g87Nw/xhizXUTmApuAfOAzY8wWV2UqcyGR0OlB65K8B9k0ldarvqF74ltkvf0Bec1vwdF2KNS/Dhxu2sjl58OGifDry5CVBlc/a00t5xdkdzKllFJOpzJzeHzSepbsTOSha+rzfJ+meuK6Ui4knjZPcUxMjFmzZo3dMS6fMcyd+yNJS7/iNv9VVMhPtYY8tBoIre+Eam3dZ+q2hO0w6xlrNcDaXeGWd6FyU7tTKeXRRGStMSbG7hxlxePbbA8QdzyD+75czb6kdP55W0uGdHKjc4WU8nDFtdlu2u3pxUTo07c/U6Pa0WHGOh6uvocnItfiu2ocrPgYoptC22HQegiEVrEnY3YGLHkLln0AAaHQ7yNoO9wa662UUsptrD1wglET1pCTl8+E+zrRtaHOYKRUWdAC2iaDO9bG18eH56b7sdy/K+OffJ8Ku3+CDZOt4RLzX4XGfaD9XdCwd9kN8dg1H2b/CU4egDbD4IZ/6JRySinlhn7YcIjnpm+iWsVAxt/TkQbRFeyOpFS5oQW0jQZ0qImvQ/jTtI2MnLyLL++9i9CY+yBxJ6z/2poWb8dsa2GRNkOh3QiIauSaMKlHYe4L1jLaUY1h5Cyod7VrXksp5ZZEpA/wPtZ5K58ZY94otL8iMBGojfX3421jzBdlHrScM8bw/oJdvDd/F53qVeLTER2ICPG3O5ZS5YqOgXYDczYf4cnJ62lRoyIT7u1ExWA/a0deDuz6BdZPhJ3zwORB7ausQrr5bRBQCr0N+XmwZjwseM2a3/ma56Dbk+AbcOXPrZQ6j7uOgRYRB9YUor2xphhdDQw1xmwrcMxfgYrGmOdFJBrYAVQ1xmQX97ze2GbbKTMnj79M38SPGw8zsENN/nV7K/x9dXidUq6iY6Dd2E2tquHrIzw2aR3DPlvB1/d3plKIPzj8oOnN1iX1qNUjvf5r+OEx+Pl5aHE7tL/bWqDkck48PLIRfnoaDq+zZgK5+b8Q2aDU359SyiN0AnYbY/YCiMgUoD+wrcAxBggVay3oCsBxoIhVlJQrJKVlMWrCGtYdPMlf+jThkWsb6LLcStlEP7a6iRtaVGXsXTHsSkjjttFL2Z1QaLnr0KrQ/Wl4fA3cN8/qgd4yAz7vDaM7WysDpiWU7MWy0mDuX2FsD0iJhwGfw10ztXhWqnyrAcQVuB/v3FbQR0AzrEWxNgNPGWPyCz+RiIwSkTUisiYxMdFVecuVncdSuW30UrYdOcUnw9vzaI+GWjwrZSMtoN3IdU0rM/nBLqRn5XLHx0tZujvp/INEoHYXuG00/HkH9PsQgsLh1/+Dd5rBlOGwYy7kFdMptH0WjO5kzfjR4R54fLU1hZ42xEqVd0U1AoXH+N0IbACqA22xFsIKO+9Bxow1xsQYY2Kio6NLO2e5s3hnIgM+XkZWbj5TR11F31bV7I6kVLmnBbSb6VAngu8f60bVioGMHL+KyasOFn9wQKg1hOP+X+CxVdDlUYhbCZMHw7stYP7fIWm3dezJOJg8FKYOh6AIuP9Xa17noPCyeFtKqTIkIt+JyM0iciltfDxQq8D9mlg9zQXdC8wwlt3APkAnh3ehr5fv574vV1OzUjA/PNaNNrXC7Y6klEJPInRbBVeVGuVcVcpRklWlzpx4uO5r69rkWWOkj20DDPR4Ebo8Yo2vVkqVubI4iVBEemEVu12Ab4EvjTGxF3mML9ZJhNcDh7BOIhxmjNla4JhPgGPGmL+LSBVgHdDGGFPE12WW8tJml7bcvHz+OXs7Xy7bz/VNK/PB0Ha6LLdSNtCTCD1MWKAf40fG8OpP2xi7ZC/7ktJ5f0hbgv0v8iM778TDybB5OjTsCTf+C8J1hSqlvJ0xZj4w3znt3FDgVxGJA8YBE40xOUU8JldEHgfmYU1jN94Ys1VEHnbuHwP8A/hSRDZjDfl4/kLFs7o8qZk5PDF5PYt2JHJ/93r89aZmJetAUUqVGe2BdnPGGL5ctp9/zNpGs2phfD6yI1UrBtodSyl1mcpqGjsRiQRGAHdhDcX4BugOtDLG9HD1659R3trsK5Wdm8+Iz1ay9uAJXuvfguGd69gdSalyrbg2W8dAuzkR4d5u9fhsZAz7k9LpP/p3thxKsTuWUsqNicgM4DcgGLjVGNPPGDPVGPME1vRzyg0ZY3jlxy2s2n+cd+5so8WzUm5MC2gP0bNpFaY/0hWHCIPGLOeXrUftjqSUcl8fGWOaG2P+bYw5UnCHOy7ioixfrzjA5FVxPNqjAf3bFp5BUCnlTrSA9iDNqoXx/WPdaFylAg9NXMu4JXvxtCE4Sqky0UxEws/cEZEIEXnUxjzqIpbtTuLVn7bRq1ll/nxDE7vjKKUuQgtoD1M5LJApo66ib8uqvD5nO3+duZmcvPPWMVBKlW8PGmNOnrljjDkBPGhfHHUhB5LTeXTSOupHhfDu4Lb46AmDSrk9LaA9UJC/g4+GtufRHg2YvCqOe75YRUrGeSfVK6XKLx8psEydiDgAfxvzqGKkZubwwFdrMAY+GxlDaKBOMaqUJ9AC2kP5+Ah/6dOUtwa2ZtW+49zxyVIOJKfbHUsp5R7mAdNE5HoR6QlMBubanEkVkp9veGbqBvYmpfPx8PbUiQyxO5JSqoS0gPZwg2Jq8fX9nUlOz+a20UtZvf+43ZGUUvZ7Hvgf8AjwGLAA+IutidR5/vvrDuZvT+DlW5rTrWGU3XGUUpdAC2gv0KV+JDMf7UZ4sD/Dx61k5vp4uyMppWxkjMk3xnxijBlojBlgjPnUGJNndy71h582Hmb0wj0M7VSLu6/S6eqU8jRaQHuJelEhzHy0K+3rhPPM1I2888sOnaFDqXJKRBqJyHQR2SYie89c7M6lLJvjU3hu+kY61o3g1X4tKTBcXSnlIbSA9iLhwf5MuK8zd8bU5IP/7eaJyevJzNFOJ6XKoS+AT4Bc4DpgAvC1rYkUAAmpmYz6eg2RIQF8MqID/r76Z1gpT1Si/7ki8pSIhInlcxFZJyI3uDqcunT+vj78Z0Brnu/TlFmbjjB03AoSU7PsjqWUKltBxpgFgBhjDhhj/g70tDlTuZeVm8fDX6/lZEYOY+/uQFSFALsjKaUuU0k/+t5njDkF3ABEA/cCb7gslboiIsIjPRowZkR7th85xW2jlxJ79JTdsZRSZSdTRHyAXSLyuIjcDlS2O1R5ZozhpZlbWHfwJG8PakOL6hXtjqSUugIlLaDPDNC6CfjCGLOxwDblpvq0rMa0h64iJy+f20Yv5ds1cXZHUkqVjaeBYOBJoAMwAhhpZ6DybvzS/UxfG8+T1zfi5tbV7I6jlLpCJS2g14rIL1gF9DwRCQV0+TsP0LpmOLOe7E67WhE8N30Tf/52IxnZuXbHUkq5iHPRlDuNMWnGmHhjzL3OmThW2J2tvFqyM5HXZ2/jxhZVePr6RnbHUUqVgpIW0PcDLwAdjTEZgB/WMA7lASqHBjLxgc48eX0jvlsXT/+PlrLrWKrdsZRSLuCcrq6D6NQObmFfUjqPT1pH4yqhvHOnLtOtlLcoaQF9FbDDGHNSREYAfwNSXBdLlTaHj/Cn3o2ZcF8njqdn0++jpcxYp/NFK+Wl1gM/iMhdInLHmYvdocqbU5k5PPDVanwdPoy7O4aQAF+7IymlSklJC+hPgAwRaYO1mtUBrGmRlIe5ulE0c566mtY1K/KnaRv5y/SNnM7Wqe6U8jKVgGSsmTdudV5usTVROZOXb3hq8noOJGfw8fD21KoUbHckpVQpKunH4VxjjBGR/sD7xpjPRURPSPFQVcIC+eaBzrw3fxejF+1mY1wKo4e3p2HlCnZHU0qVAmOMDrGz2ZvzYlm4I5HXb29Jl/qRdsdRSpWykhbQqSLyInAXcLXzJBU/18VSrubr8OHPNzahY71KPDN1A/0++p1/39GK/m1r2B1NKXWFROQL4LylSI0x99kQp9yZuT6eTxfv5a4udRjeWZfpVsoblXQIx2AgC2s+6KNADeAtl6VSZebaxtHMefJqWlavyFNTNvDijE26eqFSnm8WMNt5WQCEAWm2JionNsSd5PnvNtOlfiVevrW53XFc69hWmDEKlrwFaYl2p1GqTJWoB9oYc1REvgE6isgtwCpjjI6B9hJVKwYy6cHO/PfXnXyyaA/rD57k4+HtqR+tQzqU8kTGmO8K3heRycB8m+KUG8dOZTJqwhoqhwbw8fAO+Dm8dJnu7HRY/B9YPhocAZCTDovfhBZ3QKdRULOD3QmVcrmSLuV9J7AKGATcCawUkYGuDKbKlq/Dh+f7NOWLezty7FQmt374Oz9uPGx3LKVU6WgE1LY7hDfLzMlj1NdrScvK5bORMVQK8bc7kmvsmAuju8DS96HNEHh6Mzy2GjrcC7Gz4bOeMPY62DAZcjLtTquUy5T04/FLWHNAjzTG3A10Av7PdbGUXa5rUpk5T11Ns2phPDl5PS/N3KxDOpTyMCKSKiKnzlyAn4Dn7c7lrYwx/HXGZjbGneTdwW1pWjXsyp4wNxuyM0onXGlJOQRTR8DkweAXBPfMgf6jISQSohvDTW/Cs9vhpretHurvH4Z3m8P8V+GkroKrvE9JTyL0McYkFLifTMmLb+VhqlUMYvKoLrz9yw4+Xbz37JCOulEhdkdTSpWAMSbU7gzlybjf9jJj/SGe7d2YG1tUvfwnOrIJ1k+EzdOs3tu2Q6HLYxDVsPTCXqq8XFg1Fha+Dvm5cP3LcNUT4FtED3tAKHR6EDo+APuWWI9b+p51aXKTNbyj3jWga/woL1DSAnquiMwDJjvvDwbmuCaScgd+Dh9e7NuMTnUr8ey3G7nlw9/5z4DW3Ny6mt3RlFIXISK3A/8zxqQ474cDPYwx39uZyxstjE3g3z/HcnOrajze8zIK3YzjsPlbq3A+ugkc/tD0FvAPtrat+QKa9IWrHoc6Xcu2+Dy0Fn562srVsJfVu1yp3sUfJwL1r7UuJw/CmvGw9iuInQXRTa0iu/UQCNDzbJTnEmPOm+mo6ANFBgDdAAGWGGNmujJYcWJiYsyaNWvseOly69DJ0zwxaR3rDp7k7qvq8NebmhHo57A7llIeSUTWGmNiXPwaG4wxbQttW2+MaefK1y2KN7fZpzJzuPbNhVSrGMT0R64i2L+EfVL5ebBnIWyYaI0bzsuGam2g7QhoNRCCK1nHpSXAqnGw+jM4fRyqt4euj0Oz/uBw4aqGmSmw4B/W61aoAn3fgOa3XVnxnpMJW2fAyk/hyAYICIO2w6ze6qhGpZVcqVJXXJtd4gLaXXhzY+zOcvLyeWveDsYu2UvLGmGMHtaeOpE6pEOpS1VGBfQmY0zrQts2G2NaufJ1i+LNbfZ783fy3vxdzHqiOy1rVLz4A5L3wIZvYOMUOHUIgipB6zuh7XCo1rr4x2VnwMbJ1qwXx/dAxdrQ5WFodxcEXuF464KMga0zYe6LkHbM6inu+TcILMF7u5TXOLTWGt6xZQbk50CDntbwjkY3gI92zij3clkFtIikUsRk/Fi90MYYc8H/uSLSB3gfcACfGWPeKOa4jsAKYLAxZvqFntObG2NPMH/bMZ79diP5+Ya3BrWmT0sd0qHUpSijAno8cBIYjdWGPwFEGGPuceXrFsVb2+yTGdlc/Z+FdG0Yyad3XeDHmZUG236wCucDS0F8rOEQbYdbQzN8A0r+ovn5sPNnWPYRHFxm9eJ2GAmdH4aKNa/sDR3fB3P+DLvnW73ht7wHNdpf2XNeTFoCrPsKVo+H1MMQXtvqkW531x+98AUZAzmnnZd053WG9QGj4LbsAvvO7s+wHh8SCSHREFIZKkT/cTs40rW9+mXFGDh9AlKPQuqRAtdHrN9FH4d1Eee1j2+B2wXv+4KPT6H7BR/rvC8+kJdjfRDKy7HGyedlF7hdYN/Z285jznlMjnNbrnUJCIMKlZ0/n+jzbwdHldnPq8x7oJ2rFe4EegPxwGpgqDFmWxHH/QpkAuO1gHZ/8ScyeHzSejbEneTVfi0Y2bWu3ZGU8hhlVECHYM2U1Mu56RfgdWNMuitftyje2ma/NS+Wjxft4eenrj5/1g1jIG6lNYZ560zIToNKDaDdcGgzFMKqX3mAQ2utQnrbD9bQiha3W+Okq7e9tOfJzYZlH1iLofj4Wj3OHR8s22IyL8cayrJqHBz4HXwDrbHSuZl/FL9nLpfK4W/NGuLn/MY0I8kq1M4jVtF+trB2FmzF3fYLvKK3fFmyUosojJ3Xp478cT8v6/zHBkVYJ3ka4yxS86xrk+e8XeC+yS+9zOJj/Qx8/KzfKR8/6/7Z237W753D/4/bPg5rGFFaIqQnXOTnVUyBfc6HpMpX9PMqrs125f+QTsBuY8xeZ4ApQH9gW6HjngC+Azq6MIsqRTUjgpkyqgtPTl7PKz9u5WRGDk9e3xDRM6uVcgvOQvkFu3N4q+S0LL5Yup9bWlc/t3g+dcQaarHhG0jebRVtLW6HdiOgdpfSPQGwRgcY9IV1kt6KMVZP7uZvoe7VViHd6AarB/FC9i+FWc9A0g5o1g/6/qd0ivtL5fCDFrdZl2NbrZMOTxywTqT0O3MJAv+QP4phv6AC+4va5tzu8Dv3tYyxirP0ROuSllD07UNrrevsYhbwDAhzFmlRzgLRceGeWh9fq5g85/4Fen5PnyxUJB+F7NTzc/iFQFg1CK0GtTpDaFXr9jnXVa1/i5IyplCBnftHkX3O/Vzr2ILFsMOvQGHsd/HfwZJkyTrlLKadBXVaAqQnWbfTE619h9db27JOFf08/qHWNz4Dxl1ZngJcWUDXAApO/hgPdC54gIjUAG4HenKBAlpERgGjAGrX1rUA3EGgn4OPh7fnhRmbeXf+Tk5kZPPyLc3x8dEiWim7icivwCBjzEnn/QhgijHmRluDeYkxi/eQmZPH070aWT24O3+G9d/A7l+t3rvaV0H3Z6wT71w900R4bejzL+jxvDXTxcox1lzNUY2hy6PWYieFi6f0ZPj1/6xCv2JtGDYNGrvJr0aVFnDzf133/CIQFG5dSnLyYnbGH0X12SLbWcClJTh7tHMhN6tAcZl/fvFp8i+t59cR8EcRXKUFNOpddHEc4IIZK0WsotgdhrSIWGPwAyuWbDrHnNMFflaJ5xbZEXVLNZor/3WKqqQKjxd5D3jeGJN3od5LY8xYYCxYXweWVkB1ZXwdPrw5oDXhQX589vs+Uk7n8ObA1t67fK1SniPqTPEMYIw5ISKVbczjNRJOZTJh+QFua1eDBhmbYeKDkBJnFTTdnrbGNtsxb3NgRej2JHR5BLZ+bw3LmPU0/O+f1rjijg9YvaXrJ1rFc1aqlffav1g9u6po/sHgXwci6rj+tQr2/PoG6HzZl8MvyPpQGe76zlZXFtDxQK0C92sChdeGjgGmOIvnKOAmEcnVuUo9h4+P8NLNzYgI8eeteTtIzczho2HtdZo7peyVLyK1jTEHAUSkLkWfEK4u0ceL9pCfn8fLoT/Bl+9AeB0YOtU6MdAdeuwcftB6kDUd3v7frHHSi9+A39+FyIaQsNX6qv+Wd62eTeU+3KnnV12UK39Kq4FGIlIPOAQMAYYVPMAYc3ZGdhH5EpilxbPnEREeu64hYUF+vPzDFu4ev4rPRsYQFuh38QcrpVzhJeB3EVnsvH8NzmFw6vIdOnmaBSvXMy9iLOErN0KrO63hBqU5lVxpEbFW/at3DSTusKbAi1sJt35gzXJxpWNTlSrnXFZAG2NyReRxYB7WNHbjjTFbReRh5/4xrnptZY+7utShYpAff5q6gaFjV/DVfZ2IqnAJUzQppUqFMWauiMRgFc0bgB+A07aG8gILZn7Bj76vUzE7H24bYy217Qmim0C/D+xOoZRXcen3BMaYORRa8ru4wtmO+UlV6evXpjphgb48PHEtd45ZzoT7O1EzItjuWEqVKyLyAPAU1tC5DUAXYDnWCdsXetwF5+4XkeeA4c67vkAzINoYc7w087udnExSf3qRuw+M53BwYyLun2zPOGellNvQ73BUqevRpDIT7+9MUloWg8YsZ3dCEVPvKKVc6SmsmY0OGGOuA9oBiRd6gHNO/tFAX6A5MFREmhc8xhjzljGmrXOZ8BeBxV5fPCfugHE9Cd00nvH5N+N4cL4Wz0opLaCVa8TUrcTUh64iJ88waMxyNsWftDuSUuVJpjEmE0BEAowxsUCTizzm7Nz9xphs4Mzc/cUZCkwulbTuyBhrWrhPryUv9Sj35TzH4U5/o0qlUlzWWinlsbSAVi7TrFoY3z1yFRUCfRk6dgXLdifZHUmp8iJeRMKB74FfReQHzp8FqbCi5u6vUdSBIhIM9MFaBKuo/aNEZI2IrElMvGDHt3s6fRKm3ws/PQm1OvFK9bGscMTwcI8GdidTSrkJLaCVS9WJDGH6w12pERHEPV+sZt7Wo3ZHUsrrGWNuN8acNMb8HWtJ78+B2y7ysJLM3X/GrcDS4oZvGGPGGmNijDEx0dHRJUztJuJWwadXw7Yf4fpX2HHD13yzLYuRXevqSdFKqbO0gFYuVyUskGkPXUWLGmE8MnEt09bEXfxBSqlSYYxZbIz50Tks40JKMnf/GUPwtuEb+Xmw5G0Y38e6f988uPpPvDt/NxX8fXnomvr25lNKuRUtoFWZCA/2Z+L9nenWMIq/TN/EZ7/ttTuSUupcZ+fuFxF/rCL5x8IHiUhF4FqsqfG8w6kj8PVt8L9/QPP+8PDvUKsjWw6lMHfrUe7rXo/wYH+7Uyql3IgW0KrMhAT48tnIGG5uVY1/zt7OW/NiMUYXR1PKHRhjcoEzc/dvB6admbv/zPz9TrcDvxhj0u3IWep2zoMx3SB+DfT7CAaOt5bFBt79dScVg/y4/+p6F3kSpVR5o+tFqjIV4Ovgg6HtCAvyZfTCPZzIyOEf/Vvi8Clq+KVSqiyVZO5+Y8yXwJdll8pFcrPg11dg5SdQpZVVOEc3Prt73cETLIhN4Lkbm+iqqkqp82gBrcqcw0f41+2tCA/255NFezh1Ood37myLv69+IaKUKgNJu61ZNo5ugk4PQe/XwC/wnEPe/XUnlUL8uadrXXsyKqXcmhbQyhYiwvN9mhIe5Me/f44lNTOXT0a0J9hffyWVUi5iDGyYBHOeA98AGDoFmvQ977CVe5P5bVcSL93UjJAAbZOUUufTLj9lq4eubcAbd7Tit12J3PX5KlIycuyOpJTyRnm5MGMU/PAo1GgPjywtsng2xvDfX3cSHRrAiC51bAiqlPIEWkAr2w3pVJvRw9qzOT6FwWOXk3Aq0+5ISilvs3MubJ4GV/8Z7v4BwqoXedjS3cms2necx3o0IMjfUcYhlVKeQgto5Rb6tqrG+Hs6cvB4BgPGLONAsnec4K+UchOxsyEwHHq8AD5FF8ZW7/MOqlcMZGjn2mWbTynlUbSAVm6je6MoJj3YhdTMXAZ8spxth0/ZHUkp5Q3ycmHnz9C4DziKn1Fj4Y4E1h88yeM9GxHgq73PSqniaQGt3ErbWuFMf/gq/BzC4LHLWbWvyJWClVKq5A4ug9MnoNktxR5ijOGdX3dSq1IQg2JqlmE4pZQn0gJauZ2GlUOZ/khXokMDuOvzlSzYfszuSEopTxY7G3wDoUHPYg+Zt/UYWw6d4smejfBz6J9GpdSFaSuh3FKN8CC+fegqGlcJZdTXa/lubbzdkZRSnsgYq4Bu0BP8Q4o8JD/f8O6vO6kfFcLt7WqUcUCllCfSAlq5rcgKAUwe1YXO9Srx7Lcb+fz3fXZHUkp5miMbISUOmhY/fGPW5iPsOJbKU70a4au9z0qpEtCWQrm1CgG+jL+nI31aVOUfs7bx9rwdGGPsjqWU8hSxs0F8rBMIi5Cbl89783fSuEoFbm1d9NR2SilVmBbQyu0F+jkYPbw9QzvV4qOFu3np+y3k5WsRrZQqgdjZULsrhEQWufuHDYfZm5jOM70a4+MjZRxOKeWpdI1S5REcPsK/bm9FRLA/Hy/aw8mMbN4d3FanmlJKFe/4XkjYCn3eKHJ3Tl4+7y/YRYvqYdzYomoZh1NKeTItoJXHEBH+0qcpEcH+vD5nO6dOr+HTuzoQEqC/xkqpIsTOtq6b3FTk7ulr4zl4PIPPR8Zo77NS6pLoEA7lcR68pj5vD2rD8r3JDBu3guPp2XZHUkq5o9jZULUVRNQ5b1dWbh4fLthF21rh9Gxa2YZwSilPpgW08kgDO9RkzIgObD+ayqAxyzh88rTdkZRS7iQtAQ6uKHb2jSmr4jicksmzNzRGRHuflVKXRgto5bF6N6/ChPs6kXAqi4GfLGN3QprdkZRS7mLHz4ApsoDOzMlj9MLddKpbie4No8o+m1LK42kBrTxal/qRTB7Vhey8fAaNWcbGuJN2R1JKuYPY2RBeB6q0OG/XxBUHSEjN4k/a+6yUukxaQCuP17JGRb59uCshAb4MG7eCpbuT7I6klLJTVirsXWj1PhcqkNOzcvlk0R66NYykS/2ip7ZTSqmL0QJaeYV6USF890hXakYEc+8Xq/l58xG7Iyml7LJ7PuRlQ7Pzh298uWw/yenZ/Kl3ExuCKaW8hRbQymtUCQtk6kNdaFkjjMcmrWPyqoN2R1JK2SF2NgRHQq3O52w+lZnD2CV7ua5JNB3qRNgUTinlDbSAVl4lPNifiQ905upG0bw4YzMfL9qtS38rVZ7kZsPOX6BJX/A5d6Gl8b/vI+V0jvY+K6WumBbQyusE+/sy7u4Y+rWpzptzd/D67O3k69LfSpUP+3+DrBRoeus5m1Mycvj8t33c0LwKrWpWtCmcUspb6BJuyiv5+/rw3uC2RAT78dnv+9iblM6/72hFlbBAu6MppVwpdjb4hUD9a8/ZPGN9PKlZuTx5fSObgimlvIn2QCuv5eMj/L1fC165tTlLdydxw7tL+H79IR3SoZS3ys+3CuiG14Nf0NnNxhimro6jZY0wWtbQ3mel1JXTAlp5NRHh3m71+Pmpq2kQHcLTUzfw0NdrSUzNsjuaUqq0HV4HaUeh2bnDNzYfSiH2aCqDO9a2KZhSyttoAa3KhfrRFfj24a789aamLNqZyA3vLmbWpsN2x1JKlabYWeDjC416n7N5yuo4Anx96Nemuk3BlFLeRgtoVW44fIRR1zRg9hPdqV0pmMcnreexb9aRnKa90Up5he2zoG53CPpjirrT2Xn8tOEwN7WqRsUgPxvDKaW8iRbQqtxpVCWU7x7pynM3NuGXbUe54d0lzN2iC68o5dESd0LyLmv1wQLmbD5CalYugzvWsimYUsobubSAFpE+IrJDRHaLyAtF7B8uIpucl2Ui0saVeZQ6w9fhw2PXNeSnJ7pTtWIgD09cx1NT1nMyI9vuaEqpyxE7y7puctM5m6euiaNuZDCd61WyIZRSylu5rIAWEQcwGugLNAeGikjzQoftA641xrQG/gGMdVUepYrStGoY3z/WjWd6NWb2piP0fncJ87cdszuWUupSxc6G6u2hYo2zm/YmprFq33EGxdRCRGwMp5TyNq7sge4E7DbG7DXGZANTgP4FDzDGLDPGnHDeXQHUdGEepYrk5/DhqV6N+P6xbkSG+PPAhDU8O20jKadz7I6mlCqJU4fh0BpoevM5m6eticfhIwzsoH9alFKly5UFdA0grsD9eOe24twP/FzUDhEZJSJrRGRNYmJiKUZU6g8ta1Tkx8e780TPhny/4RA3vruERTsS7I6llLqYHXOs6wLT1+Xk5fPduniuaxKtCygppUqdKwvoor4vK3IFCxG5DquAfr6o/caYscaYGGNMTHR0dClGVOpc/r4+PHtDE2Y+2pXQQF/u+WI1L3y3idRM7Y1Wym3FzobIhhDV+OymhbEJJKZmcWeMnjyolCp9riyg44GCLVdN4LyJd0WkNfAZ0N8Yk+zCPEqVWOua4fz0RHcevrYB09bEceO7S/h9V5LdsZRyqYud+O08poeIbBCRrSKyuKwznuf0Sdi3xBq+UWCc87Q1cUSHBnBd08r2ZVNKeS1XFtCrgUYiUk9E/IEhwI8FDxCR2sAM4C5jzE4XZlHqkgX6OXihb1OmP9KVQH8HIz5fyd++30x6Vq7d0ZQqdSU58VtEwoGPgX7GmBbAoLLOeZ5dv0J+LjT9Y/jGsVOZLNyRyID2NfFz6GytSqnS57KWxRiTCzwOzAO2A9OMMVtF5GERedh52MtAJPCxs0djjavyKHW52teOYM6TV/NA93p8s/IgN763hOV79MsS5XUueuI3MAyYYYw5CGCMsf8kgdhZUKEK1OhwdtP0tfHk5RvujNGTB5VSruHSj+bGmDnGmMbGmAbGmNed28YYY8Y4bz9gjIkwxrR1XmJcmUepyxXo5+BvtzRn2kNX4esjDB23gv/7fgspGTo2WnmNkpz43RiIEJFFIrJWRO4u6onK7MTvnEzYPd+a+9nH+nNmjOHbNXF0qleJ+tEVXPfaSqlyTb/bUuoSdKxbiTlPXc293eryzcoD9PzvIqatjiM/v8jzY5XyJCU58dsX6ADcDNwI/J+IND7vQWV14ve+xZCdBs3+WH1w5b7j7E/OYLCePKiUciEtoJW6RMH+vrxyawt+eqI7daNC+Mt3mxgwZhmb41PsjqbUlSjJid/xwFxjTLoxJglYAti3gmzsLAgIg7rXnN00bXUcoQG+3NSqmm2xlFLeTwtopS5Ti+oVmf7wVfx3UBvijp+m3+jfeWnmZl0OXHmqi574DfwAXC0iviISDHTGOsel7OXnQewcaNQbfP0BSDmdw+zNR+jXtjpB/g5bYimlygdfuwMo5clEhAEdatK7RRXe/XUnE5YfYM7mIzx3Y1MGd6yFw0eXD1aewRiTKyJnTvx2AOPPnPjt3D/GGLNdROYCm4B84DNjzBZbAsetgowkaPrH8I0fNx4mKzefwR11+IZSyrW0gFaqFIQF+vHKrS0Y3LEWL/+wlb/O3MyU1Qd5rX9L2tYKtzueUiVijJkDzCm0bUyh+28Bb5VlriLFzgKHPzTsdXbTtNVxNK0aSqsaFW0MppQqD3QIh1KlqGnVMKaO6sL7Q9pyNCWT2z9eygvfbSI5LcvuaEp5D2OsArretRAYBsDWwylsPpTCkI61ENFvfpRSrqUFtFKlTETo37YGC569lge612P62niue3sRE5bvJ09n61DqyiVsgxP7z5l9Y9rqOPx9fbitXeGZ95RSqvRpAa2Ui4QG+vHSzc35+amraVmjIi//sJVbP/ydtQeO2x1NKc8WOxsQaNwXgMycPGauP8SNLaoSHuxvbzalVLmgBbRSLtaoSijfPNCZj4a143h6NgM+Wc6z0zaSmKrDOpS6LLGzoFYnCK0CwLytRzmVmcsQPXlQKVVGtIBWqgyICLe0rs6CZ6/lkR4N+HHjIXq+vYjxv+8jNy/f7nhKeY6TB+HIRmh689lNU1fHUatSEFfVj7QxmFKqPNECWqkyFBLgy/N9mjL36WtoWzuc12Zt45YPf2fl3mS7oynlGWKdk4Q4p687mJzBsj3JDOpQCx+dNlIpVUa0gFbKBg2iKzDhvk6MGdGe1MxcBo9dwdNT1nPsVKbd0ZRyb7GzILoZRDYAYNqaOHwEBnaoaXMwpVR5ovNAK2UTEaFPy2pc27gyHy/azaeL9zJ361H6tqzGoJiadKkXqT1qShWUcRwOLIPuzwCQl2+YvjaeaxpHUz08yOZwSqnyRAtopWwW5O/g2RuaMKB9Tcb+tpefNh5m5vpD1IwIYmCHmgxoX5NalYLtjqmU/XbOBZN3dvq6JTsTOXoqk1dubW5zMKVUeaMFtFJuom5UCP+6vRUv39KceVuP8u2aeN5fsIv35u+ia4NI7oypxY0tqhLk77A7qlL2iJ0NYTWgWlvAOnkwMsSf65tVsTeXUqrc0QJaKTcT6Oegf9sa9G9bg/gTGXy39hDT18Xx9NQNhAb4ckub6gyKqUm7WuG64poqP7IzYPcCaH8XiJCYmsX87ce4t1td/H31dB6lVNnSAlopN1YzIpinejXiiZ4NWbnvON+ujWPm+ngmrzpIw8oVGNShJre3r0Hl0EC7oyrlWnv+B7mnz86+MXN9PLn5hsE697NSygZaQCvlAXx8hKsaRHJVg0he7deC2ZuO8O3aeP79cyxvzttBj8bRDIqpRc+mlbU3Tnmn2NkQGA51umKMYerqODrUiaBh5VC7kymlyiEtoJXyMKGBfgzpVJshnWqzJzGN6Wvj+W5tPAtiE6gU4s9tbWswKKYmzaqF2R1VqdKRlws7f4bGfcDhx9r9x9mTmM6bAxrYnUwpVU5pAa2UB2sQXYHn+zTl2d6N+W1XEt+ujePrFfsZv3QfrWpUZFBMTfq1qU54sL/dUZW6fAeXwekTZ2ffmLo6jhB/Bze3rmZzMKVUeaUFtFJewNfhw3VNK3Nd08ocT8/mhw2H+HZNPC//sJV/ztpOx3oRdG0QRfeGUbSsURGHzi+tPEnsbPANhAY9Sc3MYdamI/RvW52QAP0TppSyh7Y+SnmZSiH+3NutHvd2q8fWwyn8sOEwv+1K4q15O3hr3g7CAn3p2iCKbo2sgrpuZLDO5qHclzFWAd2gJ/iHMGvVQU7n5HGnnjyolLKRFtBKebEW1SvSonpFAJLSsli2J5mlu5L4fXcSc7ceBaB6xUC6NYyie6MoujaIIjo0wM7ISp3ryEZIiYMeLwLW8I1GlSvQrla4vbmUUuWaFtBKlRNRFQLo16Y6/dpUxxjDgeQMft+dxNLdSfyy7Rjfro0HoGnVUGu4R6NIOtWLpIJ+Ta7sFDsbxAca92HnsVQ2xJ3kbzc3029NlFK20r+MSpVDIkLdqBDqRoUwoksd8vINWw+nsHR3Mkt3JzFx5QHGL92Hr4/Qrna41UPdMIo2tcLxc+g0eaoMxc6C2l0hJJKpC7fh5xDuaF/T7lRKqXJOC2ilFA4foXXNcFrXDOeRHg3IzMlj7YETZ3uozywpHuLvoHP9SLo1jKJ97XAaVwnVE7mU6yTvgYRtcOO/ycrNY8a6eHo3r0KlEJ1VRillL/3Lp5Q6T6Cfg24No+jWMAqAkxnZLN+TzNI9SSzdncz/YhPOHlu7UjCNq4TStGooTapa1/WiQvDVnmp1pXbMsa6b3sz8bQmcyMhhcMfa9mZSSim0gFZKlUB4sD99W1Wjbytr3t1DJ0+z5VAKO46msuNoKrFHT7FwRwJ5+QYAf4cPDSpXoGnV0HOK62oVA3Xsqiq52NlQtRVE1GHqzFVUrxhId+eHOqWUspMW0EqpS1YjPIga4UHc2KLq2W2ZOXnsSUwrUFSnsmJvMjPXHzp7TFigL02cxXSTqmE0qWLdrhjkZ8fbUO4sLQEOroAeLxB/IoPfdiXyRM9GOoe5UsotaAGtlCoVgX6Oc6bNOyMlI4cdx1LZcfQUsUdT2XkslR82HCY18+DZY6pVDKRJ1VDqR1UgsoI/lUKsS+TZ6wDCgny197o82fEzYKDpLUx3zhAzqIOePKiUcg9aQCulXKpisB+d6lWiU71KZ7cZYziSknm2p3rnMet61b7jZGTnFfk8vj5CRIGi+o8CO4BKFQoW29Z1eLC/9lZ6stjZEF6HvOjmfLtmEd0bRlGrUrDdqZRSCtACWillAxGhengQ1cODuK5p5XP2ZebkkZyezfG0bJLTsziens3x9OwC27I5np7FlkMpJKdnk5qZW+Rr+Ig1djsi2I8KgX6E+DsI9vclJMB57e8gOKDQdcH9AQ5C/H0J9ncQEuBLgK+P9oCXlaxU2LsQOj7I0j3JHDp5mhf6NrU7lVJKnaUFtFLKrQT6Oc6OsS6J7Nx8TmRkk5x2ptA+t+g+kZ5NenYeGVm5nMg4TUZ2LulZeWRk5xbb210UH8EqqJ2F9f1X12N45zqX+zbVheyeD3nZ0OwWpi6NIzzYjxtaVLE7lVJKnaUFtFLKo/n7+lAlLJAqYYGX/Nj8fMPpnDzSs3PJyLKuT2fnnS2407Pzzim4z15n5xEZokueu0zGcYhsxPFK7fh16yKGd6lNgK/D7lRKKXWWFtBKqXLLx0cICfC1FoMJtTuNOqvj/RBzHzOX7ic7L5/BHWvZnUgppc6hKx0opZRyOwaYtjqONrXCaVo1zO44Sil1Di2glVJKuZ2N8SnsOJbK4BjtfVZKuR+XFtAi0kdEdojIbhF5oYj9IiIfOPdvEpH2rsyjlFLKM0xdHUeQn4Nb21SzO4pSSp3HZQW0iDiA0UBfoDkwVESaFzqsL9DIeRkFfOKqPEoppTxDRnYuP208zE2tqhEaqKtUKqXcjyt7oDsBu40xe40x2cAUoH+hY/oDE4xlBRAuItrdoJRS5djsTUdIy8plSCcdvqGUck+uLKBrAHEF7sc7t13qMYjIKBFZIyJrEhMTSz2oUkqpEg276yEiKSKywXl52RU5woL86NuyKjF1Ilzx9EopdcVcOY1dUUt2mcs4BmPMWGAsQExMzHn7lVJKXZkCw+56Y3VmrBaRH40x2wod+psx5hZXZrmxRVVubFHVlS+hlFJXxJU90PFAwe/fagKHL+MYpZRSrleSYXdKKaVwbQ/0aqCRiNQDDgFDgGGFjvkReFxEpgCdgRRjzJELPenatWuTROTAZeSJApIu43F28JSsnpITPCerp+QEz8nqbjnddf3voobUdS7iuKtEZCNWZ8efjTFbCx8gIqOwTgwHSBORHZeRx91+bhfiKVk9JSdoVlfwlJzgXlmLbLNdVkAbY3JF5HFgHuAAxhtjtorIw879Y4A5wE3AbiADuLcEzxt9OXlEZI0xJuZyHlvWPCWrp+QEz8nqKTnBc7J6Sk43UJIhdeuAOsaYNBG5Cfgeaxalcx9UYNjdZYfxoJ+bp2T1lJygWV3BU3KCZ2R16VLexpg5WEVywW1jCtw2wGOuzKCUUqpELjqkzhhzqsDtOSLysYhEGWPcpadIKaXKhK5EqJRSCgoMuxMRf6xhdz8WPEBEqoqIOG93wvobklzmSZVSymYu7YF2M1f0dWIZ85SsnpITPCerp+QEz8nqKTltVcJhdwOBR0QkFzgNDHF+k+gKnvRz85SsnpITNKsreEpO8ICs4rq2TymllFJKKe+jQziUUkoppZS6BFpAK6WUUkopdQm8voC+2NK07kJEaonIQhHZLiJbReQpuzNdiIg4RGS9iMyyO8uFiEi4iEwXkVjnv+1Vdmcqjog84/zZbxGRySISaHemM0RkvIgkiMiWAtsqicivIrLLeW37usvF5HzL+fPfJCIzRSTcxoiqBDyh3fa0Nhu03S5t2mZfOU9us726gC6wNG1foDkwVESa25uqWLnAs8aYZkAX4DE3zgrwFLDd7hAl8D4w1xjTFGiDm2YWkRrAk0CMMaYl1klcQ+xNdY4vgT6Ftr0ALDDGNAIWOO/b7UvOz/kr0NIY0xrYCbxY1qFUyXlQu+1pbTZou11qtM0uNV/ioW22VxfQeNDStMaYI8aYdc7bqVgNRg17UxVNRGoCNwOf2Z3lQkQkDLgG+BzAGJNtjDlpa6gL8wWCRMQXCMaNlrU3xiwBjhfa3B/4ynn7K+C2ssxUlKJyGmN+McbkOu+uwJrfWLkvj2i3PanNBm23XUTb7CvkyW22txfQRS1N67YN3BkiUhdoB6y0OUpx3gP+AuTbnONi6gOJwBfOry0/E5EQu0MVxRhzCHgbOAgcwVrW/hd7U11UFWPMEbCKCaCyzXlK4j7gZ7tDqAvyuHbbA9ps0Ha7VGmbXWbcts329gK6JEvTuhURqQB8BzxdcNUvdyEitwAJxpi1dmcpAV+gPfCJMaYdkI57fGV1HudYtP5APaA6ECIiI+xN5V1E5CWsr92/sTuLuiCParfdvc0GbbddQdts13P3NtvbC+iLLk3rTkTED6sh/sYYM8PuPMXoBvQTkf1YX632FJGJ9kYqVjwQb4w50ys0Hathdke9gH3GmERjTA4wA+hqc6aLOSYi1QCc1wk25ymWiIwEbgGGu3DhD1U6PKbd9pA2G7TddgVts13IE9psby+gL7o0rbsQEcEa87XdGPOO3XmKY4x50RhT0xhTF+vf83/GGLf81G2MOQrEiUgT56brgW02RrqQg0AXEQl2/i5cjxueOFPIj8BI5+2RwA82ZimWiPQBngf6GWMy7M6jLsoj2m1PabNB220X0TbbRTylzfbqAto5CP3M0rTbgWnGmK32pipWN+AurJ6BDc7LTXaH8gJPAN+IyCagLfAve+MUzdnbMh1YB2zG+r/pNkuZishkYDnQRETiReR+4A2gt4jsAno779uqmJwfAaHAr87/V2NsDakuyIPabW2zXcft221ts0uHJ7fZupS3UkoppZRSl8Cre6CVUkoppZQqbVpAK6WUUkopdQm0gFZKKaWUUuoSaAGtlFJKKaXUJdACWimllFJKqUugBbRSl0hEeojILLtzKKWUujhts5UraAGtlFJKKaXUJdACWnktERkhIqucE7F/KiIOEUkTkf+KyDoRWSAi0c5j24rIChHZJCIzRSTCub2hiMwXkY3OxzRwPn0FEZkuIrEi8o1zJSqllFKXSdts5Um0gFZeSUSaAYOBbsaYtkAeMBwIAdYZY9oDi4FXnA+ZADxvjGmNtarUme3fAKONMW2ArsAR5/Z2wNNAc6A+1qpkSimlLoO22crT+NodQCkXuR7oAKx2djQEAQlAPjDVecxEYIaIVATCjTGLndu/Ar4VkVCghjFmJoAxJhPA+XyrjDHxzvsbgLrA7y5/V0op5Z20zVYeRQto5a0E+MoY8+I5G0X+r9BxF1rL/kJf8WUVuJ2H/l9SSqkroW228ig6hEN5qwXAQBGpDCAilUSkDtbv/EDnMcOA340xKcAJEbnauf0uYLEx5hQQLyK3OZ8jQESCy/JNKKVUOaFttvIo+glMeSVjzDYR+Rvwi4j4ADnAY0A60EJE1gIpWGPuAEYCY5yN7V7gXuf2u4BPReQ153MMKsO3oZRS5YK22crTiDEX+jZEKe8iImnGmAp251BKKXVx2mYrd6VDOJRSSimllLoE2gOtlFJKKaXUJdAeaKWUUkoppS6BFtBKKaWUUkpdAi2glVJKKaWUugRaQCullFJKKXUJtIBWSimllFLqEvw/AFWMWAs5dFQAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 864x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusion\r\n",
        "The final Test Set Evaluation shows that the model is not particularly good at classifying whether a review is positive or negative, this is caused due to the **overfitting** of the `encoder_classifier` model on the Training Set. Also, the final result is achieved after experimenting with various hyperparameters and with various different Layer. Conv2D Layers were also used instead of Conv1D, but the latter had a much better Validation Accuracy. If I were to do this experiment over again, I would first try a baseline model by learning the Word Embeddings instead of using pretrained ones and also write a custom Metric to view the performance of the `conv_ae` model."
      ],
      "metadata": {}
    }
  ]
}