DNN-Training.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from functools import partial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full / 255.0\n",
    "X_test = X_test / 255.0\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
    "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
    "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
    "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
    "X_test_scaled = (X_test - pixel_means) / pixel_stds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problemi da affrontare per il training di un DNN - Deep Neural Network:\n",
    "\n",
    "- **vanishing** o **exploding gradient**. Il gradiente diminuisce o diventa sempre più grande man mano scendo verso i livelli più vicini all'input --> i livelli più vicini all'inout sono più difficili da addestrare\n",
    "- Non abbastanza dati per il training oppure troppo costoso etichettare i sample\n",
    "- Training molto lento\n",
    "- Rischio overfitting \n",
    "\n",
    "## Vanishing/exploding gradient\n",
    "\n",
    "Esiste il rischio che il gradiente diventi sempre più piccolo man mano il gradient descent si avvicina ai livelli più vicini all'input. In questi casi i pesi rimangono inalterati e non osservo alcun apprendimento = **vanishing gradient**. In altri casi il gradiente continua a crescere e l'algoritmo diverge = **exploding gradient**. In generale si osserva un problema di instabilità del gradiente che ha reso queste reti poco utilizzate  fino ai primi anni del 2000. \n",
    "\n",
    "Nel 2010, Glorot e Bengio mostrano che la funzione di attivazione sigmoide e il metodo di inizializzazione dei pesi determinano una crescita della varianza dopo ogni livello hidden, determinando una saturazione della funzione di attivazione nei livelli vicini all'output. Ciò è amplificato dalla funzione sigmoide che satura quando i valori di input crescono e per questi valori la derivata è prossima allo 0 => nel backpropagation viene diffuso un valore di gradiente già prossimo allo zero che a sua volta deve essere diviso tra i vari pesi. E' difficile che i livelli vicini ad input vengano aggiornati.\n",
    "\n",
    "![](dnn_1.png)\n",
    "\n",
    "### Weight Inizialization\n",
    "Glorot e Bengio osservano che per evitare i precedenti fenomeni la varianza dell'output di un layer deve essere simile alla varianza degli input dello stesso layer; lo stesso dicasi per il gradiente. Dal momento che il numero di input = **fan-in** e il numero di unità di output = **fan-out** sono generalmente diversi le due condizioni sono difficilmente verificabili contemporaneamente. Un compromesso è dato da **Xavier** o **Gorot inizialization**:\n",
    "\n",
    "![](dnn_2.png)\n",
    "\n",
    "Se si rimpiazza $fan_{avg}$ con $fan_{in}$, si ottiene la **LeCun inizialization**. Mediante la Glorot inizialization si rende il training più veloce + promosso il successo delle DNN.\n",
    "\n",
    "Nella seguente tabella vengono riassunte le inizializzazioni per diversi tipi di funzioni di attivazione:\n",
    "\n",
    "![](dnn_3.png)\n",
    "\n",
    "In Keras, l'inizializzazione di default è Glorot con distro uniforme, tuttavia mediante il parametro **kernel_initializer** è possibile modificare il metodo di inizializzazione.\n",
    "\n",
    "### Activation functions\n",
    "Il precedente paper ha evidenziato che anche la forma della funzione di attivazione influenza l'instabilità del gradiente. Anche se la sigmoide è utilizzata nei neuroni biologici, ReLU e funzioni affini si comportano meglio nelle DNN. \n",
    "La ReLU, tuttavia, determina il problema del **dying ReLUs**: molti neuroni della rete emettono 0 in output, poichè i pesi in ingresso sono negativi e la somma pesata è minore di 0. Per risolvere il problema si utilizza la funzione di attivazione **LeakyReLU**:\n",
    "$$LReLU_{\\alpha}(z) = max(\\alpha z, z)$$\n",
    "dove solitamente $\\alpha=0.01$\n",
    "\n",
    "![](dnn_4.png)\n",
    "\n",
    "In un articolo del 2015 sono state comparate diverse versioni di ReLU, e LReLU è risultata sempre migliore della ReLU.\n",
    "\n",
    "In Keras per utilizzare LReLU si crea un **LeakyReLU** layer e lo si aggiunge dopo il layer a cui si vuole applicare la funzione di attivazione."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 1.2819 - accuracy: 0.6229 - val_loss: 0.8886 - val_accuracy: 0.7160\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.7955 - accuracy: 0.7362 - val_loss: 0.7130 - val_accuracy: 0.7656\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.6816 - accuracy: 0.7721 - val_loss: 0.6427 - val_accuracy: 0.7898\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.6217 - accuracy: 0.7944 - val_loss: 0.5900 - val_accuracy: 0.8066\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5832 - accuracy: 0.8075 - val_loss: 0.5582 - val_accuracy: 0.8200\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5553 - accuracy: 0.8157 - val_loss: 0.5350 - val_accuracy: 0.8238\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5338 - accuracy: 0.8224 - val_loss: 0.5157 - val_accuracy: 0.8304\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5173 - accuracy: 0.8272 - val_loss: 0.5079 - val_accuracy: 0.8286\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5040 - accuracy: 0.8289 - val_loss: 0.4895 - val_accuracy: 0.8390\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4924 - accuracy: 0.8321 - val_loss: 0.4817 - val_accuracy: 0.8396\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nello stesso anno è stata proposta anche **ELU - exponential linear unit** che migliora le varie varianti della ReLU.\n",
    "\n",
    "![](dnn_5.png)\n",
    "\n",
    "- $\\alpha$ permette di settare a quale valore negativo la funzione deve tendere con valori negativi molto grandi.\n",
    "- a un gradiente diverso da 0 per valori negativi\n",
    "- se $\\alpha=1$ la derivata non ha salti\n",
    "- computazionalmente è più costosa della ReLU, ma con un convergence rate più veloce. Utile nel training, più lenta nella predizione.\n",
    "\n",
    "Nel 2017 è stata introdotta **Scaled ELU - SELU** che auto-normalizza la rete se tutti gli hidden layer utilizzano SELU e sono densi. Per auto-normalizzazione si intende che la media e la standard deviation dell'output di ogni livello sono 0 e 1 rispettivamente.  Alcune condizioni devono essere verificate:\n",
    "- input feature devono essere standardizzate\n",
    "- LeCun inizializer per ogni hidden layer\n",
    "- architettura sequenziale => no RNN o skip connections in CNN o Wide and Deep.\n",
    "\n",
    "Per utilizzare SELU basta definire il parametro **activation** nella definizione del layer **Dense** relativo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1719/1719 [==============================] - 19s 11ms/step - loss: 1.0076 - accuracy: 0.6162 - val_loss: 0.8929 - val_accuracy: 0.6836\n",
      "Epoch 2/5\n",
      "1719/1719 [==============================] - 19s 11ms/step - loss: 0.6358 - accuracy: 0.7773 - val_loss: 0.5437 - val_accuracy: 0.8146\n",
      "Epoch 3/5\n",
      "1719/1719 [==============================] - 18s 10ms/step - loss: 0.5336 - accuracy: 0.8146 - val_loss: 0.5033 - val_accuracy: 0.8214\n",
      "Epoch 4/5\n",
      "1719/1719 [==============================] - 18s 11ms/step - loss: 0.4889 - accuracy: 0.8297 - val_loss: 0.4757 - val_accuracy: 0.8342\n",
      "Epoch 5/5\n",
      "1719/1719 [==============================] - 19s 11ms/step - loss: 0.4487 - accuracy: 0.8438 - val_loss: 0.4298 - val_accuracy: 0.8504\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
    "                             kernel_initializer=\"lecun_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
    "                                 kernel_initializer=\"lecun_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch Normalization\n",
    "Per ridurre il problema del vanishing/exploding gradient, si può utilizzare la tecnica di **Batch Normalization** che consiste nel porre un'operazione prima o dopo la funzione di attivazione di ogni livello in modo da normalizzare l'input e in seguito applicare un rescaling e uno shifting. I parametri delle ultime due operazioni vengono appresi. \n",
    "\n",
    "![](dnn_6.png)\n",
    "\n",
    "Per normalizzare l'input, si utilizza il mini-batch corrente per stimare la media e la standard deviation. $\\mathbf{\\gamma}$ e $\\mathbf{\\beta}$ vengono appressi e influenzano lo scaling e lo shifting delle istanze/sample del mini-batch. Non variano al variare del mini-batch. Ogni mini-batch non ha una coppia specifica di parametri.\n",
    "\n",
    "Per la predizione di nuove istanze non possiamo calcolare la media e la standard deviation. In pratica le due grandezze vengono calcolate utilizzando un aggiornamento online  durante il training. Keras utilizza questo metodo. In sostanza oltre ai parametri di scaling e shifting vengono stimate anche la media e la deviazione standard.\n",
    "\n",
    "BN ha milgiorato le performance in molti task e con diverse architetture, riducendo il fenomeno del vanishing gradient a tal punto da permettere l'uso di sigmoidi e tanh come funzioni di attivazione.  Inoltre, gli autori di BN hanno potuto utilizzare learning rate più grandi velocizzando il processo di training. Infine BN può agire anche da regularization.\n",
    "\n",
    "Tuttavia BN aggiunge complessita' al modello e rende la predizine piu' lenta a causa delle computazioni extra, anche se e' possibile inglobare i fattori di scala e shifting direttamenta nella matrice dei pesi e nel vettore dei bias.\n",
    "\n",
    "In Keras si deve solo aggiungere il layer **Batch Normalization** prima o dopo ogni funzione di attivazione di un hidden layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_14\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_13 (Flatten)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_144 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_4 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_145 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_146 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ogni layer BN aggiunge quattro parametri (784*4 nel primo livello)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('batch_normalization/gamma:0', True),\n",
       " ('batch_normalization/beta:0', True),\n",
       " ('batch_normalization/moving_mean:0', False),\n",
       " ('batch_normalization/moving_variance:0', False)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1 = model.layers[1]\n",
    "[(var.name, var.trainable) for var in bn1.variables]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ma $\\mathbf{\\mu}$ e $\\mathbf{\\sigma}$ non sono addestrabili."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.8581 - accuracy: 0.7180 - val_loss: 0.5714 - val_accuracy: 0.8096\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5753 - accuracy: 0.8020 - val_loss: 0.4908 - val_accuracy: 0.8350\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5130 - accuracy: 0.8223 - val_loss: 0.4509 - val_accuracy: 0.8446\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4803 - accuracy: 0.8310 - val_loss: 0.4283 - val_accuracy: 0.8516\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4549 - accuracy: 0.8406 - val_loss: 0.4095 - val_accuracy: 0.8598\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4366 - accuracy: 0.8452 - val_loss: 0.3998 - val_accuracy: 0.8600\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4235 - accuracy: 0.8509 - val_loss: 0.3889 - val_accuracy: 0.8652\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4100 - accuracy: 0.8559 - val_loss: 0.3805 - val_accuracy: 0.8684\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4021 - accuracy: 0.8579 - val_loss: 0.3750 - val_accuracy: 0.8682\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3901 - accuracy: 0.8626 - val_loss: 0.3690 - val_accuracy: 0.8672\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gli autori di BN suggeriscono di inserire BN prima delle funzioni di attivazione, tuttavia il posizionamento e' dipendente dal task. Per aggiungere il layer BN prima della funzione di attivazione, si deve rimuovere la funzione di attivazione del livello denso e aggiungerla come ulteriore layer in seguito."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inoltre visto che viene introdotto un parametro di shifting/offset, e' possibile omettere il bias mediante il parametro **use_bias**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(100, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nel layer BN posso specificare il parametro **momentum** che agisce sulla stima della media e della standard deviation. Il metodo di aggiiornamento della media mobile diventa:\n",
    "$$\\mathbf{\\mu} \\leftarrow \\mathbf{\\mu} * momentum + \\mathbf{v} * (1-momentum)$$\n",
    "\n",
    "Un secondo parametro e' su quale dimensione calcolare la media e la standard deviation. E' specificabile mediante il parametro **axis**.\n",
    "\n",
    "BN e' molto utilizzato tanto da essere omesso nei diagrammi di rappresentazione delle DNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Clipping\n",
    "Un ulteriore metodo per ridurre l'instabilita' del gradiente e' **Gradient Clipping**. Questo metodo, utilizzato principalmente nelle RNN, forza il gradiente ad assumere valori compresi tra -1  e 1 (il valore assoluto del range e' un iperparametro - **clipvalue**). Tale metodo modifica la direzione del gradiente. Per evitare questo comportamento e' possibile normalizzare il gradiente utilizzando la norma $L_2$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Pretrained Layers\n",
    "In generale non si addestra una DNN da zero, ma si utilizza una rete gia' addestrata su un task simile sfruttano i livelli piu' vicini all'input = **Transfer Learning**. Abbiamo gia' visto questo metodo nella lezione precedente con CNN.\n",
    "\n",
    "I vantaggi sono:\n",
    "- training piu' veloce\n",
    "- training set richiesto meno grande\n",
    "\n",
    "![](dnn_7.png)\n",
    "\n",
    "Il layer di output viene generalmente rimpiazziato da un layer adatto al task da risolvere, cosi' come i layer piu' vicini al layer di output. Solitamente di procede in modo sequenziale, bloccando tutti i livelli e verificando le performance sbloccando prima quelli piu' alti. Il processo puo' essere velocizzato se la dimensione del training set e' elevata. Inoltre per non modificare drasticamente i pesi dei livelli non bloccati si utilizza un learning rate ridotto.\n",
    "\n",
    "Da un modello che addestriamo su 8 categorie di Fashion MNIST, sviluppiamo un modello che distingue tra le 2 categorie rimaste fuori dal primo modello: sandali e magliette.\n",
    "\n",
    "Prepariamo il dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y):\n",
    "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
    "    y_A = y[~y_5_or_6]\n",
    "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
    "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
    "    return ((X[~y_5_or_6], y_A),\n",
    "            (X[y_5_or_6], y_B))\n",
    "\n",
    "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
    "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
    "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
    "X_train_B = X_train_B[:200]\n",
    "y_train_B = y_train_B[:200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Addestriamo il primo  modello sulle 8 categorie:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.5940 - accuracy: 0.8072 - val_loss: 0.3940 - val_accuracy: 0.8702\n",
      "Epoch 2/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.3646 - accuracy: 0.8751 - val_loss: 0.3328 - val_accuracy: 0.8852\n",
      "Epoch 3/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.3239 - accuracy: 0.8871 - val_loss: 0.3092 - val_accuracy: 0.8954\n",
      "Epoch 4/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.3028 - accuracy: 0.8946 - val_loss: 0.2914 - val_accuracy: 0.9038\n",
      "Epoch 5/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2888 - accuracy: 0.8998 - val_loss: 0.2796 - val_accuracy: 0.9088\n",
      "Epoch 6/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2783 - accuracy: 0.9041 - val_loss: 0.2710 - val_accuracy: 0.9101\n",
      "Epoch 7/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2702 - accuracy: 0.9068 - val_loss: 0.2656 - val_accuracy: 0.9096\n",
      "Epoch 8/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2629 - accuracy: 0.9092 - val_loss: 0.2652 - val_accuracy: 0.9113\n",
      "Epoch 9/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2573 - accuracy: 0.9118 - val_loss: 0.2583 - val_accuracy: 0.9138\n",
      "Epoch 10/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2518 - accuracy: 0.9134 - val_loss: 0.2526 - val_accuracy: 0.9168\n",
      "Epoch 11/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2471 - accuracy: 0.9151 - val_loss: 0.2511 - val_accuracy: 0.9150\n",
      "Epoch 12/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2431 - accuracy: 0.9174 - val_loss: 0.2485 - val_accuracy: 0.9145\n",
      "Epoch 13/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2393 - accuracy: 0.9178 - val_loss: 0.2454 - val_accuracy: 0.9168\n",
      "Epoch 14/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2360 - accuracy: 0.9190 - val_loss: 0.2512 - val_accuracy: 0.9121\n",
      "Epoch 15/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2328 - accuracy: 0.9201 - val_loss: 0.2400 - val_accuracy: 0.9180\n",
      "Epoch 16/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2294 - accuracy: 0.9213 - val_loss: 0.2407 - val_accuracy: 0.9168\n",
      "Epoch 17/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2265 - accuracy: 0.9218 - val_loss: 0.2417 - val_accuracy: 0.9153\n",
      "Epoch 18/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2242 - accuracy: 0.9229 - val_loss: 0.2380 - val_accuracy: 0.9185\n",
      "Epoch 19/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2212 - accuracy: 0.9240 - val_loss: 0.2358 - val_accuracy: 0.9145\n",
      "Epoch 20/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.2191 - accuracy: 0.9240 - val_loss: 0.2415 - val_accuracy: 0.9170\n"
     ]
    }
   ],
   "source": [
    "model_A = keras.models.Sequential()\n",
    "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))\n",
    "\n",
    "model_A.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])\n",
    "\n",
    "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
    "                    validation_data=(X_valid_A, y_valid_A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.save(\"DNN_Fashion8_A.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ora definiamo la DNN che implementa la classificazione binaria senza Transfer Learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "7/7 [==============================] - 0s 44ms/step - loss: 0.6014 - accuracy: 0.6600 - val_loss: 0.5240 - val_accuracy: 0.7789\n",
      "Epoch 2/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.4455 - accuracy: 0.8400 - val_loss: 0.4110 - val_accuracy: 0.8611\n",
      "Epoch 3/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.3489 - accuracy: 0.9050 - val_loss: 0.3421 - val_accuracy: 0.9087\n",
      "Epoch 4/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.2823 - accuracy: 0.9450 - val_loss: 0.2924 - val_accuracy: 0.9270\n",
      "Epoch 5/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.2400 - accuracy: 0.9500 - val_loss: 0.2581 - val_accuracy: 0.9361\n",
      "Epoch 6/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.2072 - accuracy: 0.9800 - val_loss: 0.2304 - val_accuracy: 0.9452\n",
      "Epoch 7/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1810 - accuracy: 0.9800 - val_loss: 0.2087 - val_accuracy: 0.9533\n",
      "Epoch 8/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1608 - accuracy: 0.9850 - val_loss: 0.1915 - val_accuracy: 0.9564\n",
      "Epoch 9/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1450 - accuracy: 0.9850 - val_loss: 0.1783 - val_accuracy: 0.9625\n",
      "Epoch 10/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1321 - accuracy: 0.9850 - val_loss: 0.1667 - val_accuracy: 0.9665\n",
      "Epoch 11/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1217 - accuracy: 0.9900 - val_loss: 0.1568 - val_accuracy: 0.9686\n",
      "Epoch 12/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1126 - accuracy: 0.9900 - val_loss: 0.1486 - val_accuracy: 0.9696\n",
      "Epoch 13/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1043 - accuracy: 0.9900 - val_loss: 0.1414 - val_accuracy: 0.9706\n",
      "Epoch 14/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0972 - accuracy: 0.9900 - val_loss: 0.1351 - val_accuracy: 0.9716\n",
      "Epoch 15/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0910 - accuracy: 0.9950 - val_loss: 0.1293 - val_accuracy: 0.9716\n",
      "Epoch 16/20\n",
      "7/7 [==============================] - 0s 7ms/step - loss: 0.0856 - accuracy: 0.9950 - val_loss: 0.1240 - val_accuracy: 0.9726\n",
      "Epoch 17/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0809 - accuracy: 0.9950 - val_loss: 0.1196 - val_accuracy: 0.9726\n",
      "Epoch 18/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0768 - accuracy: 0.9950 - val_loss: 0.1156 - val_accuracy: 0.9726\n",
      "Epoch 19/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0732 - accuracy: 0.9950 - val_loss: 0.1117 - val_accuracy: 0.9757\n",
      "Epoch 20/20\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0697 - accuracy: 0.9950 - val_loss: 0.1083 - val_accuracy: 0.9746\n"
     ]
    }
   ],
   "source": [
    "model_B = keras.models.Sequential()\n",
    "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))    \n",
    "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))\n",
    "\n",
    "model_B.compile(loss=\"binary_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])\n",
    "\n",
    "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
    "                      validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ora addestriamo il nostro modello con utilizzando tutti i livelli tranne l'output del modello A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.load_model(\"DNN_Fashion8_A.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dal momento che devo copiare i layer in un altro modello. Il training del modello B modifica i pesi del modello originale A, quindi devo farne una copia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A_clone = keras.models.clone_model(model_A)\n",
    "model_A_clone.set_weights(model_A.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Per non modificare drasticamente i pesi dei livelli piu' bassi, nelle prime epoche vengono bloccati tutti i livelli tra quelli appena introdotti in modo da addestrare solo quelli nuovi e ridurre il gradiente da propagare nelle epoche successive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "7/7 [==============================] - 0s 25ms/step - loss: 0.8854 - accuracy: 0.5900 - val_loss: 0.8969 - val_accuracy: 0.6075\n",
      "Epoch 2/4\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.8128 - accuracy: 0.6150 - val_loss: 0.8312 - val_accuracy: 0.6146\n",
      "Epoch 3/4\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.7485 - accuracy: 0.6250 - val_loss: 0.7727 - val_accuracy: 0.6308\n",
      "Epoch 4/4\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.6914 - accuracy: 0.6300 - val_loss: 0.7201 - val_accuracy: 0.6430\n"
     ]
    }
   ],
   "source": [
    "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))\n",
    "\n",
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = False\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])\n",
    "\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ora possiamo modificare tutti i layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/16\n",
      "7/7 [==============================] - 0s 25ms/step - loss: 0.4002 - accuracy: 0.8400 - val_loss: 0.2537 - val_accuracy: 0.9341\n",
      "Epoch 2/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1765 - accuracy: 0.9750 - val_loss: 0.1890 - val_accuracy: 0.9625\n",
      "Epoch 3/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1345 - accuracy: 0.9900 - val_loss: 0.1625 - val_accuracy: 0.9686\n",
      "Epoch 4/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.1137 - accuracy: 0.9900 - val_loss: 0.1443 - val_accuracy: 0.9696\n",
      "Epoch 5/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0991 - accuracy: 0.9900 - val_loss: 0.1333 - val_accuracy: 0.9726\n",
      "Epoch 6/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0895 - accuracy: 0.9900 - val_loss: 0.1243 - val_accuracy: 0.9746\n",
      "Epoch 7/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0817 - accuracy: 0.9900 - val_loss: 0.1167 - val_accuracy: 0.9757\n",
      "Epoch 8/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0749 - accuracy: 0.9950 - val_loss: 0.1100 - val_accuracy: 0.9767\n",
      "Epoch 9/16\n",
      "7/7 [==============================] - 0s 7ms/step - loss: 0.0689 - accuracy: 0.9950 - val_loss: 0.1049 - val_accuracy: 0.9767\n",
      "Epoch 10/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0644 - accuracy: 0.9950 - val_loss: 0.1002 - val_accuracy: 0.9767\n",
      "Epoch 11/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0600 - accuracy: 0.9950 - val_loss: 0.0961 - val_accuracy: 0.9767\n",
      "Epoch 12/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0564 - accuracy: 0.9950 - val_loss: 0.0926 - val_accuracy: 0.9777\n",
      "Epoch 13/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0530 - accuracy: 0.9950 - val_loss: 0.0892 - val_accuracy: 0.9787\n",
      "Epoch 14/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0502 - accuracy: 0.9950 - val_loss: 0.0864 - val_accuracy: 0.9787\n",
      "Epoch 15/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0476 - accuracy: 0.9950 - val_loss: 0.0837 - val_accuracy: 0.9797\n",
      "Epoch 16/16\n",
      "7/7 [==============================] - 0s 6ms/step - loss: 0.0453 - accuracy: 0.9950 - val_loss: 0.0814 - val_accuracy: 0.9817\n"
     ]
    }
   ],
   "source": [
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = True\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 906us/step - loss: 0.1109 - accuracy: 0.9770\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.11086471378803253, 0.9769999980926514]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 1ms/step - loss: 0.0729 - accuracy: 0.9860\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.07285811752080917, 0.9860000014305115]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B_on_A.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Faster Optimizer\n",
    "Un aumento della performance computazionali nel training e' dato anche dagli algoritmi di ottimizzazione che estendono l'approccio Gradient Descent.\n",
    "\n",
    "### Momentum\n",
    "![](dnn_8.png)\n",
    "\n",
    "$\\mathbf{m}$ tiene traccia degli aggiornamenti precedenti, mentre $\\beta$ riduce l'effetto del vettore momentum. Solitamente momentum permette di uscire piu' facilmente da minimi locali o regioni molto \"piatte\" dal momento che e' come se avvesse un alto learning rate. Un buon valore di $\\beta$, solitamente 0.9, permette di ridurre i tempi di convergenza in presenza di minimo.\n",
    "\n",
    "In Keras posso utilizzare momentum specificando il parametro **momentum** quando creo un oggetto **SGD**. In questo caso definisco il parametro $\\beta$.\n",
    "\n",
    "### Nesterov Accelerated Gradient - NAG\n",
    "![](dnn_9.png)\n",
    "\n",
    "Misura il gradiente della cost function non in $\\theta$ ma nel punto $\\theta+\\beta m$. Dal momento che m punta nella direzione dell'ottimo, viene calcolato la direzione di movimento un \"passo\" avanti. Il metodo e' quindi piu' veloce di momentum.\n",
    "\n",
    "![](dnn_1-.png)\n",
    "\n",
    "Dove $\\nabla_1$ e' il gradiente della funzione in $\\theta$, mentre $\\nabla_2$ e' il gradiente della funzione in $\\theta +\\beta m$\n",
    "\n",
    "### AdaGrad\n",
    "\n",
    "Il metodo riscala il gradiente in modo da direzionarlo nella direzione dell'ottimo:\n",
    "\n",
    "![](dnn_11.png)\n",
    "\n",
    "$s$ accumula il quadrato delle derivate parziali della cost function. In questo modo amplifico le componenti che determinano maggiormente la direzione del gradiente. Nel secondo passo, il gradiente viene riscalato di un fattore $\\sqrt{s+\\epsilon}$. In questo modo il learning rate dipende dalle dimensioni che contribuiscono maggiormente alla discesa. In pratica AdaGrad viene spesso intrappolato in minimi locali e non e' adatto alle DNN.\n",
    "\n",
    "In Keras si ha la classe **Adagrad**.\n",
    "\n",
    "### RMSProp\n",
    "Si basa sulla stessa idea di AdaGrad ma accumula solo il gradiente delle ultime iterazioni utilizzando un decadimento esponenziale:\n",
    "\n",
    "![](dnn_12.png)\n",
    "\n",
    "Solitamente $\\beta=0.9$.\n",
    "\n",
    "In Keras RMSProp e' implemetato dalla classe **RMSProp** con parametri **lr** e **rho**.\n",
    "\n",
    "### Adam e Nadam\n",
    "Adam combina i concetti di momentum e RMSProp:\n",
    "\n",
    "![](dnn_13.png)\n",
    "\n",
    "Con $\\beta_1=0.9$ e $\\beta_2=0.999$.\n",
    "\n",
    "In Keras, Adam e' implementato dalla classe **Adam**.\n",
    "\n",
    "Esistono due varianti:\n",
    "1. **AdaMax**: viene utilizzata la regola di aggiornamento $s\\leftarrow max(\\beta_2s,\\nabla_{\\theta}J(\\theta))$. \n",
    "2. **NAdam**: viene implementato anche il metodo di Nestorov.\n",
    "\n",
    "In sintesi:\n",
    "\n",
    "![](dnn_14.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schedule Learning Rate\n",
    "\n",
    "Ci sono diverse strategie per ridurre il learning rate durante il training = **learning schedule**:\n",
    "1. **power scheduling**: $\\eta(t) = \\eta_0 / (1 + t/s )^c$. Solitamente $c=1$.\n",
    "2. **exponential scheduling**: $\\eta(t) = \\eta_0 0.1 ^ {t/s}$\n",
    "3. **piecewise constant scheduling**: definisco il valore di $eta$ a tratti in funzione dell'epoca\n",
    "4. **performance scaling**: misura il validation error e se l'error non si riduce, riduco il learning rate di un fattore $\\lambda$\n",
    "5. **lcycle scheduling**: inizia aumentando $\\eta_0$ linearmente fino a $\\eta_1$ durante la prima meta' del training, poi ritorna nella seconda meta' a $\\eta_0$. Le ultime epoche il learning rate viene ridotto sensibilmente di alcuni ordini di grandezza."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Power Scheduling\n",
    "optimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4910 - accuracy: 0.8289 - val_loss: 0.4135 - val_accuracy: 0.8626\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3810 - accuracy: 0.8647 - val_loss: 0.3789 - val_accuracy: 0.8674\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3479 - accuracy: 0.8766 - val_loss: 0.3614 - val_accuracy: 0.8732\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3258 - accuracy: 0.8851 - val_loss: 0.3508 - val_accuracy: 0.8792\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3092 - accuracy: 0.8902 - val_loss: 0.3577 - val_accuracy: 0.8702\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2964 - accuracy: 0.8947 - val_loss: 0.3389 - val_accuracy: 0.8810\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2859 - accuracy: 0.8989 - val_loss: 0.3432 - val_accuracy: 0.8820\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2767 - accuracy: 0.9019 - val_loss: 0.3329 - val_accuracy: 0.8816\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2690 - accuracy: 0.9043 - val_loss: 0.3305 - val_accuracy: 0.8848\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2624 - accuracy: 0.9065 - val_loss: 0.3317 - val_accuracy: 0.8842\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2558 - accuracy: 0.9090 - val_loss: 0.3271 - val_accuracy: 0.8840\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2504 - accuracy: 0.9115 - val_loss: 0.3252 - val_accuracy: 0.8866\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2451 - accuracy: 0.9136 - val_loss: 0.3236 - val_accuracy: 0.8876\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2403 - accuracy: 0.9150 - val_loss: 0.3254 - val_accuracy: 0.8866\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2362 - accuracy: 0.9171 - val_loss: 0.3213 - val_accuracy: 0.8880\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2322 - accuracy: 0.9186 - val_loss: 0.3215 - val_accuracy: 0.8868\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2283 - accuracy: 0.9199 - val_loss: 0.3209 - val_accuracy: 0.8870\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2249 - accuracy: 0.9208 - val_loss: 0.3258 - val_accuracy: 0.8842\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2215 - accuracy: 0.9220 - val_loss: 0.3221 - val_accuracy: 0.8858\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2185 - accuracy: 0.9231 - val_loss: 0.3213 - val_accuracy: 0.8866\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2154 - accuracy: 0.9245 - val_loss: 0.3199 - val_accuracy: 0.8850\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2130 - accuracy: 0.9251 - val_loss: 0.3230 - val_accuracy: 0.8850\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2099 - accuracy: 0.9261 - val_loss: 0.3207 - val_accuracy: 0.8854\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2072 - accuracy: 0.9277 - val_loss: 0.3186 - val_accuracy: 0.8892\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2052 - accuracy: 0.9288 - val_loss: 0.3219 - val_accuracy: 0.8860\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuHElEQVR4nO3dd3yV9d3/8dcngySshD3CVCAsUVAUFVtqpeLEn/W2WltX71pbbb2tWzu0t1YrHba3q9haR7XWVqs4ccaBgqCiLNmbsGcgQMbn98d1BY+Hk+QczMlJct7Px+M8cq7xvc7nXOSRD995mbsjIiISr4xUByAiIk2LEoeIiCREiUNERBKixCEiIglR4hARkYQocYiISEKUOEQaATO70MxKk3Tt2WZ2c4JllpnZ1TVtS3pT4pBGw8weMjMPX+VmtsTMfmtmrVIdW13MrK+Z/d3MVpnZHjNbY2YvmNnwVMdWT0YC96Y6CGkcslIdgEiU14DvAtnAccBfgFbAD1MZVDUzy3b38uh9wKvAYuBsYDVQCIwF2jd4kEng7htSHYM0HqpxSGOzx93XuvtKd38ceAw4A8DMcszsLjNbZ2a7zWyqmY2uLmhm08zsuojtx8LaS9dwu6WZ7TWzY8NtM7NrzWyxmZWZ2Swz+05E+T5h+XPN7A0zKwN+ECPmIcDBwGXu/p67Lw9/3uLur0dcr62Z3WdmJWH888zsW5EXMrOvh01LO83sTTPrG3X8NDP7MCy/1MxuM7MWEcc7m9mz4fdZbmYXRwcbfqezovbV2hQVo+nKzewSM/tXGOuSyHsXnnOUmX0UxvqxmZ0clhtT0+dI06DEIY1dGUHtA+BO4FvAxcBwYBbwspl1C48XA1+LKPtVYCMwJtw+FigHPgi3bwW+B1wGDAZuB/5sZqdExXA7QTPNYOCZGDFuAKqAb5pZzFq8mRnwUhjTReG1fgrsjTgtB7gh/H5HAwXA/RHXOJEgkd5NkKwuBs4Cfh1xjYeAfsAJBAn3fKBPrJjqwS+AZ4FDgX8CD5pZ7zDW1sDzwGfA4cC1wIQkxSENzd310qtRvAj+6D0fsX0kwR/+fxI0V+0Fzo84nknQPHRruH0SUErQBNsf2AHcBvw5PH4b8Gr4vhVBUjouKoa7gBfD930AB66KI/bLgJ3h578F/C8wJOL4WILkMqiG8heGn1UUse+88DtnhNtvAz+PKndG+JkGDAivcWzE8d5AJXBzxD4Hzoq6zjLg6gS2Hbg9YjsL2AV8J9z+AbAZyIs459thuTGp/l3T68u9VOOQxmacmZWa2W7gfYI/lj8maArKBqZUn+juleE5g8Nd7xD8r30kQS3jHYI+kzHh8TEEtRLCMrkENZbS6hdBX8rBUTHNqCtod78H6Erwx/FdYDww08y+G54yHChx93m1XGaPu8+P2F4TfueCcPtw4KaoeB8nSIJdgUEEyam6RoW7Lw+vkwyfRnxOBUHNq3O4ayAw293LIs6flqQ4pIGpc1wam7eBSwialNZ42BEd0RwVaznn4L/A7qVm9hFBc9UQ4E2CxNLbzPoTJJRrwzLV/2k6DVgRdb3yqO2d8QTu7juAScAkM/sZMJmg5vEoQY2gLhXRl4yKNQO4BfhXjLIb4vyM6utGn5sd68Q6RN8n5/NYjdj/VtIMKHFIY7PL3RfF2L+IoNlmNLAEwMwyCfoCHo84r5ggcQwC7nL33WY2DbiJL/ZvzAX2AL3d/Y36/hLu7mb2GTAi3PUR0M3MBtVR66jNR8DAGu4PZjaP4A/3SOC9cF8voHvUqRuAbhHlukRu15N5wPlmlhdR6ziynj9DUkSJQ5oEd99pZvcBd5jZRmApcCXQhS/OLygGriKoJXwUse8m4M3qGoy77zCz3wK/DTuu3wZaA6OAKnefGG9sZnYYQU3gUYKEtJegE/xi4B/haa8TNNU8ZWZXAgsIOrFbufszcX7Ur4DnzWw58CRBDWUocKS7X+vu883sZYIO/ksI+nB+H/6M9AZwmZm9R9D/8Wtgd7zfN06PEQw+eMDMfk2QvG4Mj6km0sSpj0OakusI/mD+DZgJDAPGuXtJxDnvEPxheifsA4GgySqTz/s3qv0cuBm4GphDMBfjmwRJKRGrCGpBvwCmhrFdBfyWoH8Gd68i6LyfAvyd4H/kfwRa7H+52Nx9MnAKQY3qg/B1PV9sarswjP8N4DmC2tiyqEtdFcZbDPybYK7M+njjiDPWUoJmwCHAxwQjqm4OD9d3kpIGZu5K/iKSfGY2HvgP0NndN6Y6HjlwaqoSkaQwswsIajYrCZrU7gKeU9Jo+pLaVGVm48xsvpktMrPrYxw3M/tTePxTMxsRcexBM1tvZrOjyrQ3s1fNbGH4s10yv4OIHLAuBP0+84F7CCZAfqfWEtIkJK2pKhzxsoBg4tMqYDpwrrvPjTjnZII24JOBo4A/uvtR4bGvEExsesTdh0aUuRPY7O53hMmonbvvW2ZCRESSK5k1jiOBRe6+xN33Ak8QTIqKNJ4gMbi7TwUKqsfru/vbBDNPo40HHg7fP0y4jpGIiDSMZPZxFBK0bVZbRVCrqOucQqCEmnWpHkXj7iVm1jnWSeFwxEsAMvLaHp6V//lpfdpqMBlAVVUVGRm6F5F0T2LTfYmtud+XBQsWbHT3TtH7k5k4Ys1ijW4Xi+ecAxKOw58IkNOtv3e74C4ACgvymHL98fXxEU1ecXExY8aMSXUYjYruSWy6L7E19/sSzhnaTzJT5SqgZ8R2D/ZfMyeec6Ktq27OCn/GPf48LzuTa04sivd0ERGJIZmJYzrQ34Ino7UAziFYxyfSJIJlCczMRgHboiZzxTIJuCB8fwHBss5xuXJsf84YXhjv6SIiEkPSEke4WublBAu9zQOedPc5ZnapmV0anvYiwTjvRcADwI+qy5vZPwgWqCuy4HGc3wsP3QGMNbOFBCO27qgrlp5tMmiRmcGqLdErL4iISKKSOgHQ3V8kSA6R++6PeO8EzzGIVfbcGvZvAr6eSByZBqcd1p1/zVjFVWOLyG95IAuBiogIpNFaVRcd24ey8kr+OSN6BW0REUlE2iSOId3zGXVQex5+bzkVlVWpDkdEpMlKm8QBcPGxfVm9tYxX5q5LdSgiIk1WWiWOrw/qQq/2LXnw3URXzRYRkWpplTgyM4wLj+nDjOVb+GTl1lSHIyLSJKVV4gD4ryN60Doni79NUa1DRORApF3iaJObzdlH9OT5T0tYt10PIhMRSVTaJQ6AC4/pQ6U7j74fcxkWERGpRVomjl4dWjJ2UBcem7ac3eWVdRcQEZF90jJxAFw8ui9bdpXzzMerUx2KiEiTkraJ46i+7RnSvS0PTllKsp6CKCLSHKVt4jAzLj62LwvWlTJl0aZUhyMi0mSkbeIAOPXQbnRsncODGporIhK3tE4cOVmZfHdUb974bD1LNpSmOhwRkSYhrRMHwHmjetEiM4OH3luW6lBERJqEtE8cHVvnMD58Vse2XeWpDkdEpNFL+8QBcNGxffWsDhGROClxAIO7t+XogzroWR0iInFQ4ghdPDp4VsfkOXpWh4hIbZQ4QscP7EzvDi01NFdEpA5KHKHqZ3V8uHwLM/WsDhGRGilxRPivI3rSRs/qEBGplRJHhNY5WZw9sicvfFrC2m16VoeISCxKHFEuPKYPFVXOCb9/i77Xv8Cxd7yhFXRFRCJkpTqAxubD5VvIMCjdUwHA6q1l3PD0LADOGF6YytBERBoF1TiiTJg8n6qoVdbLyiuZMHl+agISEWlklDiirNlaltB+EZF0o8QRpXtBXkL7RUTSjRJHlGtOLCIvO/ML+3KyMrjmxKIURSQi0rioczxKdQf4hMnz9zVP9e/cWh3jIiIhJY4YzhheuC9R3Fu8iDtfns9bCzbw1QGdUhyZiEjqqamqDt8b3Zc+HVpyy3Nz2FuhlXNFRJQ46pCTlckvThvMkg07eVhPCRQRUeKIx/EDu3D8wM788fWFrN+upUhEJL0pccTp56cOZm9FFb95WRMBRSS9JTVxmNk4M5tvZovM7PoYx83M/hQe/9TMRtRV1swOM7OpZjbTzGaY2ZHJ/A7V+nZsxfeO68tTH63iw+VbGuIjRUQapaQlDjPLBO4BTgIGA+ea2eCo004C+oevS4D74ih7J3CLux8G/CLcbhCXf60fXdrmcPOkOVRFr0siIpImklnjOBJY5O5L3H0v8AQwPuqc8cAjHpgKFJhZtzrKOtA2fJ8PrEnid/iCVjlZ3HjyIGat3saTM1Y21MeKiDQqyZzHUQhE/nVdBRwVxzmFdZT9H2Cymf2WIPEdE+vDzewSgloMnTp1ori4+EC+w37aujOgXQa3PTeLNtsW0yrb6uW6qVBaWlpv96W50D2JTfcltnS9L8lMHLH+oka379R0Tm1lfwhc6e5PmdnZwF+BE/Y72X0iMBGgqKjIx4wZE2fYdetctI3T/u9dppd15uaxQ+rtug2tuLiY+rwvzYHuSWy6L7Gl631JZlPVKqBnxHYP9m9Wqumc2speADwdvv8XQbNWgxrSPZ9zj+zFo1OXM3/tjob+eBGRlEpm4pgO9DezvmbWAjgHmBR1ziTg/HB01Shgm7uX1FF2DfDV8P3xwMIkfocaXf2NIlrnZHHzpDm4q6NcRNJH0hKHu1cAlwOTgXnAk+4+x8wuNbNLw9NeBJYAi4AHgB/VVjYs833gd2b2CfBrwn6MhtauVQuu/sYA3l+yiZdmr01FCCIiKZHURQ7d/UWC5BC57/6I9w5cFm/ZcP+7wOH1G+mB+fZRvXn8g5Xc9sI8vlbUmbwWmXUXEhFp4jRz/EvIzDBuPm0wq7eWcd9bi1MdjohIg1Di+JKOOqgDpx/anfvfWszKzbtSHY6ISNIpcdSDG04eSKYZt74wN9WhiIgknR7kVA+65edx+fH9mDB5Pof/76ts3rmX7gV5XHNikZ4cKCLNjhJHPenSJgcDNu3cC8DqrWXc8PQsACUPEWlW1FRVT/7w2sL9psWXlVcyYbKWYReR5kWJo56s2VqW0H4RkaZKiaOedC/IS2i/iEhTpcRRT645sYi87C9OAMwwuHrsgBRFJCKSHEoc9eSM4YXcfuYhFBbkYUB+XjZVDmv0jHIRaWY0qqoenTG8cN8IKnfniidm8rtX5jO8VwHHHNwxxdGJiNQP1TiSxMy4/cxD6NuxFT/5x0zWq+YhIs2EEkcStcrJ4t7zDqd0Tzk//sfHVFRWpTokEZEvTYkjyYq6tuG2Mw5h2tLN/OG1BakOR0TkS1PiaADfPLwH54zsyT1vLubNz9anOhwRkS9FiaOB3Hz6EAZ1a8uVT85ktSYFikgTpsTRQHKzM7n3vBFUVDqXPfYReyvU3yEiTZMSRwPq27EVd541jJkrt3L7S/NSHY6IyAFR4mhgJx/SjQuP6cPfpizjpVklqQ5HRCRhShwpcOPJgzisZwHX/vtTlm3cmepwREQSosSRAi2yMrjnvBFkZho/fOwjdpdXpjokEZG4acmRFCksyOP3Zx/KxQ/N4KK/fcCKzWWs2VqmJweKSKOnxJFCxw/swgmDOvPavM/ndujJgSLS2KmpKsXmlmzfb5+eHCgijZkSR4qVbI29+KGeHCgijZUSR4rpyYEi0tQocaRYrCcHGnDpmINSE5CISB3qTBxmNsDMXjez2eH2MDP7WfJDSw/RTw7s2LoFmRnw2NQVbN21N9XhiYjsJ55RVQ8A1wB/BnD3T83sceDWZAaWTiKfHAjw9oIN/PfDM7jgb9N57L+PonWOBr+JSOMRT1NVS3f/IGpfRTKCkcBXBnTi7m8PZ/bqbVz80HTK9mqCoIg0HvEkjo1mdjDgAGZ2FqBFlpLsG0O68odvHcb0ZZv5wd8/ZE+FkoeINA7xJI7LCJqpBprZauB/gEuTGZQETj+0O785cxhvL9jAjx//mHI9elZEGoF4Eoe7+wlAJ2Cgu4+Os5zUg7NH9uTm0wbzytx1XP2vT6is8lSHJCJpLp5e16eAEe4euYzrv4HDkxOSRLvw2L7sKq/kzpfnk5edye1nHoKZpTosEUlTNdYczGygmX0TyDezMyNeFwK58VzczMaZ2XwzW2Rm18c4bmb2p/D4p2Y2Ip6yZvbj8NgcM7sz7m/bhP1oTD8u/1o/npi+kl89Pxd31TxEJDVqq3EUAacCBcBpEft3AN+v68JmlgncA4wFVgHTzWySu8+NOO0koH/4Ogq4DziqtrJm9jVgPDDM3feYWee4vmkzcNU3BrBrbyUPTllKqxZZXH1iUapDEpE0VGPicPdngWfN7Gh3f/8Arn0ksMjdlwCY2RMEf/AjE8d44BEP/vs81cwKzKwb0KeWsj8E7nD3PWGc60kTZsbPTx1EWXkFd7+5iKUbS5m5cpuWYxeRBhVPH8fHZnYZMISIJip3v7iOcoXAyojtVQS1irrOKayj7ADgODO7DdgNXO3u06M/3MwuAS4B6NSpE8XFxXWE23SMbedMzzdemLV2377VW8u49l8zmTtvLsd0z47rOqWlpc3qvtQH3ZPYdF9iS9f7Ek/ieBT4DDgR+BVwHjAvjnKxem+jG+ZrOqe2sllAO2AUMBJ40swO8qhGf3efCEwEKCoq8jFjxsQRctPxs2mvE+TNz+2tghdWZHLjt8fEdY3i4mKa2335snRPYtN9iS1d70s8w2r7ufvPgZ3u/jBwCnBIHOVWAT0jtnsAa+I8p7ayq4CnPfABUAV0jCOeZkXLsYtIqsSTOMrDn1vNbCiQT9AHUZfpQH8z62tmLYBzgElR50wCzg9HV40Ctrl7SR1lnwGOh2ABRqAFsDGOeJqVmpZd75of14A3EZEDFk/imGhm7YCfEfzxngv8pq5C7l4BXA5MJmjaetLd55jZpWZWPfP8RWAJsIhgMcUf1VY2LPMgcFC4Wu8TwAXRzVTpINZy7BC08ZVsU61DRJKnzj4Od/9L+PZt4CAAM+sdz8Xd/UWC5BC57/6I906wpElcZcP9e4HvxPP5zVn16KkJk+fvG1V16qHdeGzqCsbfPYW/XjCSQ3rkpzhKEWmOak0cZnY0wQint919vZkNA64HjuOLfRCSAtHLsQOcObwHFz80nbP//D5/POcwvjGka4qiE5HmqraZ4xMImoW+CbxgZr8EXgWmEUzYk0aoqGsb/nPZMQzo2oYf/P1D/vLOEs0yF5F6VVuN4xRguLvvDvs41hDM1l7YMKHJgercJpcnvj+Kq/41k1tfmMeSjTu55fQhZGdqbUoR+fJq+0tS5u67Adx9CzBfSaPpyGuRyd3njuCHYw7m8WkruPih6WzfXV53QRGROtRW4zjYzCKHz/aJ3Hb305MXltSHjAzjunED6duhFTf+ZxZn3fcef71gJD3bt0x1aCLShNWWOMZHbf8umYFI8pw9sic92udx6aMf8v/uncJ3RvXmXzNWsXprGYVT39AaVyKSkNoWOXyrIQOR5Drm4I48/aNjOfvP73HXa5+3OK7eWsYNT88CUPIQkbiotzSN9Ovcmpys/ScNlpVXMmHy/BREJCJNkRJHmlm7TWtciciXo8SRZmpa4yo/L1vzPUQkLnUmDjN7zswmRb0eNbMrzEwr6jUxsda4yjDYWlbOpX//kC0796YoMhFpKuKpcSwBSgkWIXwA2A6sI3ig0gPJC02S4Yzhhdx+5iEUhjWPwoI8fnfWodx08iDe+Gw9J971Nu8s3JDiKEWkMYvnQU7D3f0rEdvPmdnb7v4VM5tTYylptKrXuIp+CM0x/TpwxRMz+e5fP+C/R/flmnFFMTvTRSS9xVPj6GRmvao3wvfVD05Su0YzMqR7Ps9dPprzj+7NX95dyvi7p7Bg3Y5UhyUijUw8ieMq4F0ze9PMioF3gGvMrBXwcDKDk4aX1yKTX40fyoMXHsHG0j2c9n/v8tCUpeo4F5F94nkex4tm1h8YSPCcoM+q17AC7kpibJJCxw/swktXfIVr//0JNz83l+IFGzh+YCf+/NbSfc//0IxzkfQUTx8HwOEEj4vNAoaZGe7+SNKikkahU5scHrxwJI9OXc4tk+ZQPP/zTnPNOBdJX/EMx30U+C0wGhgZvo5IclzSSJgZ5x/dhw6tc/Y7phnnIukpnhrHEcDgdHyut3xuw449MfdrxrlI+omnc3w2oOePprmaZpxnZhjTlmxq4GhEJJXiSRwdgblmNjly9niyA5PGJdaM8xaZRuucLL41cSpXPPEx67bHXgdLRJqXeJqqbk52ENL4VXeAT5g8/wujqk4c0pX7ihdx/1tLeG3uOq4cO4ALjumjx9SKNGPxDMfVczkE+HzGebSffqOIM0f04Jbn5nDrC/N4csZKbjl9KEcf3CEFUYpIstWYOMzsXXcfbWY7gMiOcQPc3dsmPTppMvp0bMWDF47ktXnrueW5OZz7wFROO7Q7N508iKlLNu1XU9EQXpGmq7YnAI4Of7ZpuHCkKTMzxg7uwnH9O3Jv8WLuf2sxk2eXUOVQURX830PzP0Savrgaos0s08y6m1mv6leyA5OmKzc7k5+OHcCrV36FDLN9SaOa5n+ING119nGY2Y+BXxIspV4V7nZgWBLjkmagd4dW7KmoinlM8z9Emq54RlVdARS5uwbrS8K6F+SxOkaSyM7M4L3FGznm4I4xSolIYxZPU9VKYFuyA5HmKdb8j+xMIzc7g28/MI1zJ05lxrLNKYpORA5EPDWOJUCxmb0A7Ft3wt1/n7SopNmoaf7HuKFdeXzaCu4tXsxZ97/PVwZ04qqxAzi0Z0FqAxaROsWTOFaErxbhSyQhNc3/uHh0X845siePvr+c+99azPh7pnDCoM5cOXYAQ7rn88zHqzWMV6QRqjVxmFkm0N/dv9NA8Uiaadkiix989WDOG9Wbh6YsZeLbSzjlT+9yaI+2fLa2dF/nuobxijQetfZxuHslwaNjVdOQpGqdk8Xlx/fnneuO5ydf78+nq7bvNyJLw3hFGod4mqqWAVPChQ13Vu9UH4ckQ35eNj8dO4D/e31hzOMaxiuSevGMqloDPB+e2ybiJZI0tS3j/vepyynbW9nAEYlItToTh7vfEusVz8XNbJyZzTezRWZ2fYzjZmZ/Co9/amYjEih7tZm5mWkiQDNU0zDebvm5/OyZ2Rx9x+tMmPyZlnIXSYF4Zo53Aq4FhgC51fvd/fg6ymUC9wBjgVXAdDOb5O5zI047Cegfvo4C7gOOqqusmfUMj62I83tKE1PTMN7xh3VnxvIt/PWdpdxbvJiJby/htGHduXh0X4YW5gNoNJZIksXTx/EY8E/gVOBS4AJgQxzljgQWufsSADN7AhgPRCaO8cAj4WNpp5pZgZl1A/rUUfYPBMns2TjikCaqpmG8I/u0Z2Sf9qzYtIu/vbeUJ6ev5OmPV3NU3/YMLczn8WnLKSvXaCyRZIkncXRw97+a2RXhszneMrN4ntFRSDDrvNoqglpFXecU1lbWzE4HVrv7J2ZW44eb2SXAJQCdOnWiuLg4jpDTS2lpaZO/L19tAyOPy+GtVRW8tnwL05buPwu9rLyS/332Ewq2xe5wj9Qc7kky6L7Elq73JZ7EUR7+LDGzUwg6y3vEUS7WX3WP85yY+82sJXAT8I26PtzdJwITAYqKinzMmDF1FUk7xcXFNJf7cjJQUVlFv5teinl8826P67s2p3tSn3RfYkvX+xLPqKpbzSwfuAq4GvgLcGUc5VYBPSO2exAknXjOqWn/wUBf4BMzWxbu/8jMusYRjzRzWZkZFNYwGisjw7i3eBHr1Zku8qXFM6rqeXff5u6z3f1r7n64u0+K49rTgf5m1jecQHgOEF1uEnB+OLpqFLDN3UtqKuvus9y9s7v3cfc+BAlmhLuvjf8rS3NW02is3u1bcufL8zn6jjf474en88qctZRXxl7yXURqF8+oqgEEo526uPtQMxsGnO7ut9ZWzt0rzOxyYDKQCTzo7nPM7NLw+P3AiwStDIuAXcBFtZU90C8p6aOm0VhnDC9k6cadPDljJU99uIrX5q2nY+scvjmikLNH9mTWqm1MmDyf1VvLKJz6hkZiidTCggFNtZwQdIRfA/zZ3YeH+2a7+9AGiK9eFBUV+fz5WqoiWrq2z1ZUVlE8fwP/nLGSNz5bT2WVk2EQ+aDCvOxMbj/zECWPULr+rtSlud8XM/vQ3Y+I3h9PH0dLd/8gal9F/YQl0vCyMjM4YXAXHjj/CN6/4Xja5mYR9XRbysorufPlz1IToEgjF0/i2GhmBxOOiDKzs4CSpEYl0kA6t8llx+7Y/w9as2031/77E95esIEK9YeI7BPPcNzLCIa1DjSz1cBS4LykRiXSgGp6vG1ediYvzlrLkzNW0b5VC04a2pVTh3XnyL7tycyoeQ6RSHNXZ+IIZ2+fYGatgAx332Fm/wPcleTYRBrENScWccPTsygr/3zhxOo+jnFDu/LWgg08/2kJT3+0msemraBzmxxOPqQbpx3ajRUbd/HbVxdoeRNJK/HUOABw950Rmz9FiUOaiciRWKu3llEYlQBOHNKVE4d0ZdfeCt74bD3Pf1LC4x+s4KH3lmF8PqtVy5tIuog7cURRPV2alep1sWobJdOyRRanDuvOqcO6s2N3OV+580227Cr/wjll5ZX87/NzGTe0K7lR80lEmosDTRy1j+EVaeba5GazNSppVNu0cy/Df/Uqx/XvyAmDuvC1gZ3p1CangSMUSZ4aE4eZ7SB2gjAg9roOImmkpk71Dq1acMqwbrw2dx2vzF2HGRzWs4ATBnXhhEFdGNClNc/OXKOl36XJqjFxuLue8idSi5o61X9+6mDOGF7ILacPYV7JDl6bt47X561jwuT5TJg8n/atstlWVkFlOHlEfSPS1BxoU5VI2qtteRMAM2Nw97YM7t6Wn3y9P+u27+aNz9Zzy6Q5+5JGtbLySn794jzGH9ad2h4XINIYKHGIfAk1PWwqli5tczn3yF7cGNYuoq3fsYfRv3mT0f06Mrp/R47t15H2rVrUZ7gi9UKJQ6SB1dQ3kp+XzSGF+bw4u4R/zliJGQztns/o/h05rl9HDu/TjpdmrVXfiKScEodIA6upb+SW04dwxvBCKiqr+HT1Nt5duJF3F27kgbeXcF/xYrIygoUYq1u51DciqaLEIdLA6uobycrMYESvdozo1Y6ffL0/pXsqmLp4E1f882N27qn8wrXKyiv55aTZHNSpFYO7tSUrM57l50S+HCUOkRRIpG+kdU4WJwzuwq6opFFtW1kFp989hdY5WRzeux1H9m3PqIPac0hhAS2ygkTyzMer1cQl9UaJQ6SJqKlvpGvbXG48ZRAfLN3EtCWbmTA5ePZMbnYGw3u2Iz8vizfmb2BvRbDCr5q45MtS4hBpImrqG7n+pIGcfmh3Tj+0OwCbSvcwfdkWPli6mWlLN/H+kk37XausvJI7XpqnxCEHRIlDpImoq2+kWofWOYwb2pVxQ7sC0Pf6F2IuAbF2+x5G/fp1hvcqYESvdgzvVcDQwvwvrLFV3cSlR+pKJCUOkSYkkb6RajUP/83iqIPa89GKLbw0ey0AWRnBpMURvdpRWVXFkzNWsUdNXBJFiUOkmat5+O/QfQlgw449zFy5lY9WbOHjFVt4csZKdu3dvzM+aOL6TIkjzSlxiDRz8TRxdWqTw9jBXRg7uAsAFZVV9L/ppRqauHZz5G2vcUhhPof0yA9+FubTuW3uvnM0iqt5U+IQSQOJNnFlZWbU2sR1bL+OzFq9jTfmr8fD7NK5TQ6HFOaTnWm88dkG9laqiau5UuIQkZjiaeLauaeCuSXbmbVqG7NXb+PT1dtYtL50v2uVlVdyy3NzGNStLQd1akV2DRMVVVNpGpQ4RCSmuh6pC9AqJ4uRfdozsk/7fftqGsW1ZVc5J971Ni0yM+jXuTWDurVlULc2DOrWloFd2/DOwo1fSFSqqTReShwiUqN4HqkbraYmrs5tcrjplEHMLdnOvJIdvLNwA099tGrf8Qz7fB2uamXllUyYPF+Jo5FR4hCRelVTE9eNJw9i/GGFjD/s8ySwqXQPn63dwbyS7dz6wryY11u9tYzvPzKD/p1bM6BLG/p3ac3BnVrHnG+iJq6GocQhIvUq3omKEExWPLZfDsf268jfpiyLWVPJzc5g2cadvPnZeirCKkmGQa/2LenfpQ3uzlsLNlBeqScqNhQlDhGpdwcyUbGmmsrtZx7CGcML2VtRxbJNO1m4rpQF63awcP0OFq4rZWENnfE3PTOLHbvLObhTaw7q1JoubXP2e7qiaioHRolDRBqFumoqLbIyGNClDQO6tOEUuu0rV1Nn/M49lfz82Tn7tlu1yOSgTq05qFMrDurYms079/DE9JWaGX8AlDhEpNGozyVVCgty+fcPj2HJhp0s2VDK4g07WbyhlBnLtvDszDUxr1VWXskvnp1NbnYmfTu2olf7luS1yNzvvHRfw0uJQ0SatJqauK45cSDd8vPolp/Hsf06fqFM2d5KBv/i5Zg1le27K7j07x/u2+7aNpfeHVrSt2MrendoxfrtZTz+QXrXVJQ4RKRJS6Qzvlpei8waayrd8nP583cPZ9mmXSzfuJOlm3ayfNMuXpu3jo2le2Ner6y8kp8/M5u9lVX0bNeSXh1a0rVtLpkZzbNPRYlDRJq8+uyMv27cQIb1KGBYj4L9yuzYXc6wm1+JWVPZsaeCa//96b7t7EyjsCCPnu1b0rN9S3aUlfPynLUHNPqrsSUcJQ4RSUsHUlNpk5tdY02le0Eu//j+KFZuLmPF5l2s3LKLFZt3sWrzLl6aVcKWXeX7lSkrr+S6pz5l+rLN9GjXksJ2eRQW5NGjXR6dWueQkWE88/HqRjejPqmJw8zGAX8EMoG/uPsdUcctPH4ysAu40N0/qq2smU0ATgP2AouBi9x9azK/h4g0T/VZU7n2xIH07hD0g8RS0+ivPRVVvDCrhK1RiaVFZgbdCnJZu233vv6UamXlldz5ct3L2yerppK0xGFmmcA9wFhgFTDdzCa5+9yI004C+oevo4D7gKPqKPsqcIO7V5jZb4AbgOuS9T1ERCLFs4ZXLDWP/spjyvXHU7qngjVby1i9pYxVW8tYtWUXq7eUsXzTrpjXW7NtN4f96hW65efRPT+XbgW5wfvw55zV25jwynx2lyfeiV+dcFp07Xd4rOPJrHEcCSxy9yUAZvYEMB6ITBzjgUfc3YGpZlZgZt2APjWVdfdXIspPBc5K4ncQEdnPgazhVfPoryIAWudk7ZunEunjFW/ETDhtc7M4dVg3SrbuZs223Xy4Yst+tZZo1cONyyur6JqfS9e2uXTJz6VNTta+yZHRTWOxJDNxFAIrI7ZXEdQq6jqnMM6yABcD/4z14WZ2CXAJQKdOnSguLk4g9PRQWlqq+xJF9yQ23ZfYErkvBcB3B2Xy1IIqNu12OuQa3xyQScG2hRQXL6yx3Cm9KnloO+yNaK1qkQHnDMjgmIJNwYWDveypyGbzbmfzbmfCjN0xr7d9dwXXRHTiA+RkQrsco12usXhbFTEe/vgFyUwcFmNfdBNfTefUWdbMbgIqgMdifbi7TwQmAhQVFXm8/ytIJ4n8byld6J7EpvsSW6L3ZQxwY4KfMQYYfAB9FY8vil1T6Z6fyxOXHM3a7btZu30367bt/sL7vZu31BlTMhPHKqBnxHYPIHq6Zk3ntKitrJldAJwKfD1s5hIRabbqtRN/3EB6dQjmmsRy7B2xE06k2I/hqh/Tgf5m1tfMWgDnAJOizpkEnG+BUcA2dy+prWw42uo64HR3j91rJCKS5s4YXsjtZx5CYUEeRtAJX71gZG2uObGIvOz9l1mJlLQaRzjq6XJgMsGQ2gfdfY6ZXRoevx94kWAo7iKC4bgX1VY2vPTdQA7watiZM9XdL03W9xARaaoOpKYSOWqspIZzkjqPw91fJEgOkfvuj3jvwGXxlg3396vnMEVEJEJ1wrEbFn0Y63gym6pERKQZUuIQEZGEKHGIiEhClDhERCQhShwiIpIQJQ4REUmIEoeIiCREiUNERBKixCEiIglR4hARkYQocYiISEKUOEREJCFKHCIikhAlDhERSYgSh4iIJESJQ0REEqLEISIiCVHiEBGRhChxiIhIQpQ4REQkIUocIiKSECUOERFJiBKHiIgkRIlDREQSosQhIiIJUeIQEZGEKHGIiEhClDhERCQhShwiIpIQJQ4REUmIEoeIiCREiUNERBKixCEiIglR4hARkYQocYiISEKSmjjMbJyZzTezRWZ2fYzjZmZ/Co9/amYj6iprZu3N7FUzWxj+bJfM7yAiIl+UtMRhZpnAPcBJwGDgXDMbHHXaSUD/8HUJcF8cZa8HXnf3/sDr4baIiDSQZNY4jgQWufsSd98LPAGMjzpnPPCIB6YCBWbWrY6y44GHw/cPA2ck8TuIiEiUrCReuxBYGbG9CjgqjnMK6yjbxd1LANy9xMw6x/pwM7uEoBYDsMfMZh/Il2jmOgIbUx1EI6N7EpvuS2zN/b70jrUzmYnDYuzzOM+Jp2yt3H0iMBHAzGa4+xGJlE8Hui/70z2JTfcltnS9L8lsqloF9IzY7gGsifOc2squC5uzCH+ur8eYRUSkDslMHNOB/mbW18xaAOcAk6LOmQScH46uGgVsC5uhais7CbggfH8B8GwSv4OIiERJWlOVu1eY2eXAZCATeNDd55jZpeHx+4EXgZOBRcAu4KLayoaXvgN40sy+B6wA/iuOcCbW3zdrVnRf9qd7EpvuS2xpeV/MPaGuAxERSXOaOS4iIglR4hARkYQ068RR15In6crMlpnZLDObaWYzUh1PqpjZg2a2PnKOj5a0qfG+3Gxmq8PfmZlmdnIqY2xoZtbTzN40s3lmNsfMrgj3p+XvS7NNHHEueZLOvubuh6XjGPQIDwHjovZpSZvY9wXgD+HvzGHu/mIDx5RqFcBV7j4IGAVcFv49Scvfl2abOIhvyRNJY+7+NrA5anfaL2lTw31Ja+5e4u4fhe93APMIVrhIy9+X5pw4alrORIJZ+K+Y2Yfh0izyuS8saQPEXNImTV0ermL9YLo0ycRiZn2A4cA00vT3pTknji+9bEkzdqy7jyBoxrvMzL6S6oCk0bsPOBg4DCgBfpfSaFLEzFoDTwH/4+7bUx1PqjTnxBHPkidpyd3XhD/XA/8haNaTgJa0icHd17l7pbtXAQ+Qhr8zZpZNkDQec/enw91p+fvSnBNHPEuepB0za2VmbarfA98AtHLw57SkTQzVfxxD/480+50xMwP+Csxz999HHErL35dmPXM8HDJ4F58vW3JbaiNKPTM7iKCWAcGSM4+n630xs38AYwiWxl4H/BJ4BngS6EW4pI27p1VHcQ33ZQxBM5UDy4AfVLftpwMzGw28A8wCqsLdNxL0c6Td70uzThwiIlL/mnNTlYiIJIESh4iIJESJQ0REEqLEISIiCVHiEBGRhChxiNQDM6uMWDl2Zn2uxmxmfSJXqhVJtaQ9OlYkzZS5+2GpDkKkIajGIZJE4bNPfmNmH4SvfuH+3mb2erho4Otm1ivc38XM/mNmn4SvY8JLZZrZA+GzIF4xs7yUfSlJe0ocIvUjL6qp6lsRx7a7+5HA3QQrGRC+f8TdhwGPAX8K9/8JeMvdDwVGAHPC/f2Be9x9CLAV+GZSv41ILTRzXKQemFmpu7eOsX8ZcLy7LwkXyVvr7h3MbCPQzd3Lw/0l7t7RzDYAPdx9T8Q1+gCvhg8LwsyuA7Ld/dYG+Goi+1GNQyT5vIb3NZ0Ty56I95Wof1JSSIlDJPm+FfHz/fD9ewQrNgOcB7wbvn8d+CEEjz82s7YNFaRIvPS/FpH6kWdmMyO2X3b36iG5OWY2jeA/aueG+34CPGhm1wAbgIvC/VcAE83sewQ1ix8SPDhJpNFQH4dIEoV9HEe4+8ZUxyJSX9RUJSIiCVGNQ0REEqIah4iIJESJQ0REEqLEISIiCVHiEBGRhChxiIhIQv4/eU80TOmhMlQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learning_rate = 0.01\n",
    "decay = 1e-4\n",
    "batch_size = 32\n",
    "n_steps_per_epoch = len(X_train) // batch_size\n",
    "epochs = np.arange(n_epochs)\n",
    "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
    "\n",
    "plt.plot(epochs, lrs,  \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Power Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Exponential Scheduling\n",
    "def exponential_decay(lr0, s):\n",
    "    def exponential_decay_fn(epoch):\n",
    "        return lr0 * 0.1**(epoch / s)\n",
    "    return exponential_decay_fn\n",
    "\n",
    "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.8295 - accuracy: 0.7602 - val_loss: 0.7660 - val_accuracy: 0.7976\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6672 - accuracy: 0.7981 - val_loss: 0.7441 - val_accuracy: 0.7976\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5934 - accuracy: 0.8173 - val_loss: 0.7156 - val_accuracy: 0.7826\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5269 - accuracy: 0.8380 - val_loss: 0.7140 - val_accuracy: 0.8080\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5063 - accuracy: 0.8409 - val_loss: 0.6247 - val_accuracy: 0.8298\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4468 - accuracy: 0.8561 - val_loss: 0.5791 - val_accuracy: 0.8246\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4211 - accuracy: 0.8633 - val_loss: 0.4660 - val_accuracy: 0.8566\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3790 - accuracy: 0.8761 - val_loss: 0.4765 - val_accuracy: 0.8616\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3620 - accuracy: 0.8802 - val_loss: 0.5108 - val_accuracy: 0.8540\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3362 - accuracy: 0.8881 - val_loss: 0.4643 - val_accuracy: 0.8660\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3102 - accuracy: 0.8933 - val_loss: 0.5241 - val_accuracy: 0.8594\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2893 - accuracy: 0.8992 - val_loss: 0.4695 - val_accuracy: 0.8690\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2767 - accuracy: 0.9045 - val_loss: 0.4659 - val_accuracy: 0.8762\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2542 - accuracy: 0.9093 - val_loss: 0.4357 - val_accuracy: 0.8820\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2386 - accuracy: 0.9150 - val_loss: 0.4391 - val_accuracy: 0.8792\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2255 - accuracy: 0.9182 - val_loss: 0.4622 - val_accuracy: 0.8792\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2074 - accuracy: 0.9248 - val_loss: 0.4323 - val_accuracy: 0.8840\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1964 - accuracy: 0.9278 - val_loss: 0.4909 - val_accuracy: 0.8752\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1828 - accuracy: 0.9318 - val_loss: 0.5291 - val_accuracy: 0.8800\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1726 - accuracy: 0.9369 - val_loss: 0.4970 - val_accuracy: 0.8824\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1647 - accuracy: 0.9390 - val_loss: 0.5245 - val_accuracy: 0.8814\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1550 - accuracy: 0.9435 - val_loss: 0.5119 - val_accuracy: 0.8822\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1451 - accuracy: 0.9476 - val_loss: 0.5290 - val_accuracy: 0.8872\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1365 - accuracy: 0.9501 - val_loss: 0.5481 - val_accuracy: 0.8886\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1296 - accuracy: 0.9541 - val_loss: 0.5882 - val_accuracy: 0.8884\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "\n",
    "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
    "\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwEElEQVR4nO3deXxU1f3/8dcngUDCkrCELYCsIosLguAubhXUirXU5dtv3dpaLVZbWy1urf6qVUtbrdaviK2tti51RVQUURpcERRklV1UAijIGghb+Pz+uDc4DjPJDGQySeb9fDzmMXPvPffO5x5CPrnn3HuOuTsiIiKJykp3ACIiUrcocYiISFKUOEREJClKHCIikhQlDhERSYoSh4iIJEWJQ2Q/mdnFZlaa5D7FZvbXVMUUfsdyM/tVCo47wsySuo8/uo72pc6k9lDikH1mZv80M4/xmpru2FIlPL8RUav/A3RLwXf9yMxmmlmpmW00s9lmdlt1f0+apKTOpGY0SHcAUue9Dvwgat2OdASSLu5eBpRV5zHN7FLgXuAXwBtADtAXOKo6vyddUlFnUnN0xSH7a7u7r456rQMwsxPMbKeZDakobGaXm9kmM+sWLheb2Rgz+4uZrQ9fo80sK2KfFmb2SLitzMxeN7O+EdsvDv8qP9nM5prZFjP7r5l1jQzUzL5tZh+a2TYz+8TMbjeznIjty83sJjN7MIxxhZldG7k9/Ph0eOWxPPL7I8p1N7MXzGx1GMsMMzszyXo9C3jO3R909yXuPt/dn3b3a6LO6Qwzez+sl6/M7EUzaxxRpHG88wn3zzezsWb2pZltNrMpZjYwqsyFZvapmW01s5eAtlHbbzGzuVHrKm2KilFnt4T/dueb2dIwlnFm1jqiTAMzuzvi5+RuM3vAzIqrrk6pTkockjLuPgUYDfzLzFqa2UHAn4CfufuyiKLfJ/hZPAr4CXAZ8POI7f8EBgPDgUHAVuBVM8uNKNMIuB64NDxOATCmYqOZnQY8BvyV4C/3S4ERwO+jwv4FMAc4HLgL+IOZVfyVf0T4/mOgfcRytKbAK8CpwKHAs8Bz4fknajUwqCLBxmJmQ4EXgEnAAOBEYArf/H8d93zMzICXgSLgTKA/8CYw2czah2UGE9T/WOAw4EXg/yVxHsnoApwHfAf4VhjP7RHbfwVcDPwIOJLgPP8nRbFIZdxdL7326UXwC2UXUBr1uiuiTENgOvAcMAP4T9QxioFFgEWsuwlYEX7uCThwfMT2fGAj8KNw+eKwTK+IMt8naDLLCpffBG6O+u6zw3gtXF4OPBFVZjFwU8SyAyOiylwMlFZRV1OjjlMM/LWS8u2B98LvWwz8G7gQaBhR5h3gyUqOUen5ACeF558bVeYj4Lrw8+PApKjtfwt+dexZvgWYW1mdJLB8C7ANyI9YdyOwJGJ5FTAqYtmABUBxuv8vZNpLVxyyv94k+Es08jW6YqO77yT4q/BMoA3BFUW0qR7+Jgi9BxSZWXOgN7A7XFdxzI0Ef0X3idhnu7svjFheSZC0CsLlAcCNYZNWadhM8jjQBGgXsd/sqNhWhnEnzMyamNkfzGx+2KRSCgwEOid6DHdf5e5HAQcD9xD8knwQmGZmeWGx/gT9H5Wp7HwGAHnAmqh66Qd0D8v0JqLuQ9HL1eXT8N92r1jNLJ/g32laxcbwZ2Z6imKRSqhzXPbXVndfUkWZimaFAqAQ2JDE8a2SbZHJZlecbVkR77cCT8c4zpqIzztjHCfZP7D+CAwlaFpZTNC09ihBB3dS3H0uMBe438yOBd4CziW42ktEZeeTBXwBHBdjv03he2X1X2F3jHINE4wvUiJ1r+G8awFdcUhKmVkXgn6FkQRt8Y+ZWfQfLIPD9vYKRwIr3X0TMJ+v+z8qjtmc4C/x+UmEMgM4yIOO5uhXdNKpzE4gu4oyxwKPuvuz7j4bWMHXf8Hvj4rzbRq+zwRO3o/jzSDo6N4do06+jPjOI6P2i15eA7SN+jc8bD/i2kt4JbKaoI8L2NNHE6+fSVJIVxyyvxqZWbuodeXuvsbMsgna5qe4+4Nm9gxBE9NvgZsjyncA7jGz/yNICNcCtwG4+2IzewF40MwuI7hauZ3gL+LHk4jz/wEvmdmnwFMEVyj9gEHufl0Sx1kOnGxmUwiax9bHKLMI+E4Y906C820co1xcZvYAQVPNZILE056g72cr8FpY7HbgRTNbQlAXRtCp/KC7b03ga14n6Cd5wcyuI+gvaEdwtfS6u79FcEvwu2Z2PfAMMISg8zpSMdASuMHMngzLRD/rUh3+AlxnZosIEtpPCOplVQq+SyqhKw7ZX6cQ/MeNfM0Mt90A9AB+CODuXwEXAaPCZpcKjxH8Ff8+8BDwd+DuiO2XELRtjw/f84ChHjwLkBB3nwicQXDn0bTwNQr4LPFTBeCX4TE+5+vzjHYN8CVBs9IrBB3jbyX5PZMI7iR7iiARPR+uP9XdFwG4+wSCX+LDwlimhLHtTuQLwj6C0wmS00PAwvD7ehEkLdx9KsG/3xUE/SXnEHRkRx7n43D7ZWGZU9n7brXq8EfgX8A/COoUgnrZloLvkkpU3E0ikhbhPfhz3f3KdMcidY+ZzQDecfefpTuWTKKmKhGpE8zsAOA0giurBgRXOIeG71KDlDhEpK7YTfAsy2iCZvb5wDB3/yCtUWUgNVWJiEhS1DkuIiJJyYimqoKCAu/Ro0e6w6h1tmzZQpMmTdIdRq2iOolN9RJbfa+XDz/8cK27F0avz4jE0bZtWz74QM2g0YqLixkyZEi6w6hVVCexqV5iq+/1Ej73tBc1VYmISFKUOEREJClKHCIikhQlDhERSYoSh4iIJEWJQ0REkqLEISIiSVHiEBGRpChxiIhIUpQ4REQkKUocIiKSFCUOERFJihKHiIgkRYlDRESSosQhIiJJSWniMLOhZrbQzJaY2agY283M7g23zzazwyO2PWxmX5rZ3Kh9WprZJDNbHL63qCqO5Zt2c8ydkxk3s6R6TkxEJIOlLHGYWTZwPzAM6ANcYGZ9oooNA3qGr8uAByK2/RMYGuPQo4A33L0n8Ea4XKWSDWVc/9wcJQ8Rkf2UyiuOQcASd1/m7juAJ4HhUWWGA496YCpQYGbtAdz9TWBdjOMOBx4JPz8CnJ1oQGU7yxk9cWFyZyEiIt+Qyqlji4DPI5ZXAIMTKFMErKrkuG3dfRWAu68yszaxCpnZZQRXMeS0+3q+8ZINZRQXFyd2BvVcaWmp6iKK6iQ21UtsmVovqUwcFmOd70OZfeLuY4GxAI3a99xzzKKC3Ho9R3Ay6vt8yftCdRKb6iW2TK2XVDZVrQA6RSx3BFbuQ5loX1Q0Z4XvXyYaUMNs49rTeiVaXEREYkhl4pgO9DSzrmaWA5wPjI8qMx64MLy76khgY0UzVCXGAxeFny8CXkgkmJzsLLIMjureKvEzEBGRvaQscbj7LuBKYCLwMfCUu88zs8vN7PKw2ARgGbAEeAj4acX+ZvYE8B7Qy8xWmNkPw013Aqea2WLg1HC5Ul2aZ/Hqz48DjJvHzcW9WlrDREQyUir7OHD3CQTJIXLdmIjPDoyMs+8FcdZ/BZycbCzdCpvyi1MP5M5XFvDK3NWcfnD7ZA8hIiJk2JPjPzq2K/2KmvObF+ayfsuOdIcjIlInZVTiaJCdxR++eygbtu7kdy/PT3c4IiJ1UkYlDoA+HZpzxZDuPDejhOKFCd+QJSIioYxLHABXntSD7oVNuPH5uZRu35XucERE6pSMTByNGmTzhxGHsHJjGaNfXZDucERE6pSMTBwAAw5oyUVHdeHRqZ8yfXmsIbFERCSWjE0cANee1osO+bn8+tnZbNtZnu5wRETqhIxOHE0aNeCOcw5m2Zot3PvG4nSHIyJSJ2R04gA4/sBCRgzoyINvLmNuycZ0hyMiUutlfOIAuOmM3rTIy+G6Z2azs3x3usMREanVlDiAgrwcfje8L/NXbeKht5alOxwRkVpNiSM07OD2DOvXjnteX8zSNaXpDkdEpNZS4ohw6/C+NG6QxahnZ7N7t0bQFRGJJaWj49Y1bZo15uYz+3DtM7Pp/7tJbCrbSYeCXK49rRdn9y9Kd3giIrWCEkeUBllGlsHGsp1AMEf59c/NAVDyEBFBTVV7+eNri4hupSrbWc7oiQvTE5CISC2jxBFl5YaypNaLiGQaJY4oHQpyk1ovIpJplDiiXHtaL3IbZn9jnQFXntQ9PQGJiNQyShxRzu5fxB3nHExRQS4GtG6aA8C7S9cRTJEuIpLZdFdVDGf3L/rGHVR/nbyYP762iON6tubcgZ3SGJmISPrpiiMBVwzpwZHdWvLbF+bpqXIRyXhKHAnIzjLuOa8/jRtm8bPHZ7J9l+buEJHMpcSRoHb5jRk94lDmr9rEna9oulkRyVxKHEk4pU9bLj66C/94ZzlvfPxFusMREUkLJY4kjRp2EL3bN+faZ2bzxaZt6Q5HRKTGKXEkqXHDbO67oD9lO8r5+ZMfUa5RdEUkwyhx7IMebZpy61l9eW/ZV4yZsjTd4YiI1Cgljn30vYEdOfOQ9vx50iI+/HR9usMREakxShz7yMz4/TkH0z6/MVc9MXPPMOwiIvWdEsd+aN64Ifde0J/Vm7Zxw/NzNCSJiGSElCYOMxtqZgvNbImZjYqx3czs3nD7bDM7vKp9zewwM5tqZh+Z2QdmNiiV51CVwzu34JffOpCXZ6/iqQ8+T2coIiI1ImVjVZlZNnA/cCqwAphuZuPdfX5EsWFAz/A1GHgAGFzFvn8AbnX3V8zs9HB5SKrOIxGXH9+dd5as5cbn5/Cn1xaxZvN2TTkrIvVWKq84BgFL3H2Zu+8AngSGR5UZDjzqgalAgZm1r2JfB5qHn/OBlSk8h4RkZRnf6tOWXbvhy83bcb6ecnbczJJ0hyciUq1SOTpuERDZdrOC4KqiqjJFVez7c2Cimf2RIPEdHevLzewy4DKAwsJCiouL9+UcEnZv8da91pXtLOd3L8yiYOPilH73viotLU15vdQ1qpPYVC+xZWq9pDJxWIx10b3H8cpUtu8VwC/c/VkzOxf4O3DKXoXdxwJjAXr16uVDhgxJMOx9s+7Vl2Ov3+ak+rv3VXFxca2NLV1UJ7GpXmLL1HpJZVPVCiBy8oqO7N2sFK9MZfteBDwXfn6aoFkr7TTlrIhkilQmjulATzPramY5wPnA+Kgy44ELw7urjgQ2uvuqKvZdCZwQfj4JqBXtQLGmnAW49NguNR+MiEgKpaypyt13mdmVwEQgG3jY3eeZ2eXh9jHABOB0YAmwFbiksn3DQ/8Y+IuZNQC2EfZjpFvF3VOjJy5k5YYyCps1YmPZDp7+YAUXDOpMXo4mWxSR+iGlv83cfQJBcohcNybiswMjE903XP82MKB6I60e0VPOTlm0hkv+MY1fPT2L+//ncMxidd2IiNQtenI8hU44sJDrh/VmwpzV3Dd5SbrDERGpFmo/SbEfHdeVj1dt4s+TFnFg22YM7dcu3SGJiOwXXXGkWMVgiId2KuCapz5iwepN6Q5JRGS/KHHUgMYNsxn7gwE0bdSAHz/6Aeu27Eh3SCIi+0yJo4a0bd6YB38wgC82bWfkYzPYWb473SGJiOwTJY4a1L9zC+4852DeW/YVt700v+odRERqIXWO17BzDu/Ix6s28dBbn9C7fXPOH9Q53SGJiCRFVxxpMGpYb44/sJCbX5jL9OXr0h2OiEhSlDjSIDvLuO/8/nRskccV//6Qkg1l6Q5JRCRhaqpKk/y8hjx04UC+c/87nDvmXdxh1cZtmgBKRGo9XXGkUY82TblgUCdKNmxj5cZtmgBKROqEKhOHmR1oZm+Y2dxw+RAzuyn1oWWGl+es3mtd2c5yRk9cmIZoRESqlsgVx0PA9cBOAHefTTDMuVSDlXH6N+KtFxFJt0QSR567T4tatysVwWQiTQAlInVNIoljrZl1J5y61cxGAKtSGlUGiTcB1FmHdkhDNCIiVUvkrqqRBHN3H2RmJcAnwPdTGlUGiZ4Aql1+Y3DnkfeWM7RfOw7tVJDeAEVEoiSSONzdTzGzJkCWu282s66pDiyTRE8A9eWmbXx3zLtc/I9pPH350fRo0zSN0YmIfFMiTVXPArj7FnffHK57JnUhSZvmjfnXpYPJzjIu/Pv7rNqojnIRqT3iJg4zO8jMvgvkm9k5Ea+LgcY1FmGG6tK6Cf+8ZBCbt+3iB3+fxnoNxS4itURlVxy9gDOBAuDbEa/DgR+nPDKhX1E+Yy8cyGfrtnLJP6ezdYduZhOR9Ivbx+HuLwAvmNlR7v5eDcYkEY7q3or7LujPFf/+kMv/PYO/XTiQnAZ64F9E0ieR30AzzWykmf2fmT1c8Up5ZLLHaX3bccc5B/PmojX86ulZ7N7t6Q5JRDJYIonjX0A74DRgCtAR2FzpHlLtzjuiM78eehDjZ63k1hfn4a7kISLpkcjtuD3c/XtmNtzdHzGzx4GJqQ5M9nb5Cd34qnQ7f3v7E1o1bcRVJ/dMd0gikoESSRw7w/cNZtYPWA10SVlEEpeZccPpvVm3dQd/nrSIz9Zt5b2lX7FyQ5mGYxeRGpNI4hhrZi2Am4DxQFPg5pRGJXFlZRl3ffcQFqzaxDMfrtizvmI4dkDJQ0RSqso+Dnf/m7uvd/c33b2bu7cBXq2B2CSOhtlZbNi6c6/1Go5dRGpCpYnDzI4ysxFm1iZcPiTs43i7RqKTuFZt3BZzvYZjF5FUq+zJ8dHAw8B3gZfN7LfAJOB9QL2yaabh2EUkXSrr4zgD6O/u28I+jpXAIe6+uGZCk8pce1ovrn9uDmU7y7+x/uTebdIUkYhkisqaqsrcfRuAu68HFiabNMxsqJktNLMlZjYqxnYzs3vD7bPN7PBE9jWzn4Xb5pnZH5KJqb44u38Rd5xzMEUFuRjQPr8x3Vo34d9TP+W5GSuq3F9EZF9VdsXR3czGRyx3iVx297MqO7CZZQP3A6cCK4DpZjbe3edHFBtG0OzVExgMPAAMrmxfMzsRGE5w9bO9ov8lE0UPx751xy5+9MgH/PLpWewqd849olMaoxOR+qqyxDE8avlPSR57ELDE3ZcBmNmT4TEjE8dw4FEPHoOeamYFZtae4DmRePteAdzp7tsB3P3LJOOqt/JyGvDwxUdw2b8+5LpnZ7O9fDc/OPKAdIclIvVMZYMcTtnPYxcBn0csryC4qqiqTFEV+x4IHGdmtwPbgF+5+/ToLzezy4DLAAoLCykuLt7nE6lr/vcAZ9P6bG4eN5f5CxZxWpeGMcuVlpZmVL0kQnUSm+oltkytl0QeANxXFmNd9ABL8cpUtm8DoAVwJHAE8JSZdfOowZvcfSzBlLf06tXLhwwZknjk9cCQE3Zz9ZMzeWLuajp36cYVQ7rvVaa4uJhMq5eqqE5iU73Elqn1ksrxuVcAkY3sHQnuzEqkTGX7rgCe88A0YDfQuhrjrhdyGmRx3wX9+fahHbjr1QX85fXFGhhRRKpFKhPHdKCnmXU1sxzgfIIhSyKNBy4M7646Etjo7quq2HcccBKAmR0I5ABrU3gedVaD7CzuOe8wvnt4R+5+fRF/fG2hkoeI7Lcqm6rM7EX2bmLaCHwAPFhxy240d99lZlcSjKSbDTzs7vPM7PJw+xhgAnA6sATYClxS2b7hoR8GHjazucAO4KLoZir5WnaWMXrEIeQ0MO7/71J27NrNDaf3xixWa6CISNUS6eNYBhQCT4TL5wFfEHRSPwT8IN6O7j6BIDlErhsT8dmBkYnuG67fAfxvAnFLKCvLuP3sg2mYncVDb33CglWbWLp2Cys3bKNo6mSNqisiSUkkcfR39+Mjll80szfd/Xgzmxd3L6lVsrKMW8/qy2dfbaF40dctexpVV0SSlUgfR6GZda5YCD9XdEbvSElUkhJmxuIvS/dar1F1RSQZiVxx/BJ428yWEtwm2xX4qZk1AR5JZXBS/VZu0Ki6IrJ/qkwc7j7BzHoCBxEkjgURHeL3pDA2SYEOBbmUxEgSbZo3SkM0IlIXJXo77gCgL3AIcK6ZXZi6kCSVrj2tF7kNs/dav7lsJx9+ui4NEYlIXVNl4jCzfwF/BI4leFL7CGBgiuOSFIkcVRegqCCXG0/vTdv8XC546H1emh39jKaIyDcl0scxEOijZyXqj4pRdSOHSxgxoCM/fvQDrnx8JivWl/GT47vpWQ8RiSmRpqq5QLtUByLp1aJJDv/+0WC+fWgH7nxlATeOm8uu8t3pDktEaqFErjhaA/PNbBqwvWJlVfNxSN3TuGE2fznvMDq1yOX/ipdSsr6M+79/OE0bpXIsTBGpaxL5jXBLqoOQ2iMry7hu6EF0apnHTePm8r0x7/GPi4+gXX7jdIcmIrVEIrfj7u+8HFIHXTCoMx0Kchn52AzOvv8dHr74CPp0aJ7usESkFoibOMzsbXc/1sw2881BDo1gmCn9FqnnTjiwkKd+chSX/nM63xvzLv971AG8NGsVKzeU0aEgV2NciWSouJ3j7n5s+N7M3ZtHvJopaWSOPh2aM27kMeTnNuTBKcso2VCG8/UYV+NmlqQ7RBGpYQk9AGhm2WbWwcw6V7xSHZjUHu3yG+81rj5ojCuRTJXIfBw/A35LMJR6xf2ZTvAUuWSI1Rs1xpWIBBK5q+pqoJe7f5XqYKT2ijfGVaumOWmIRkTSKZGmqs8JZvyTDBZrjCsD1pbu4P7/LmH3bg0sIJIpEp0BsNjMXuabDwD+OWVRSa1TcffU6IkL99xVddVJPXhn6VeMnriQGZ+u58/nHkZ+XsM0RyoiqZZI4vgsfOWEL8lQFWNcRTr3iE4MOKAFt708nzP/+hYPfH8A/Yry0xShiNSEShOHmWUDPd1dc3xLTGbGRUd3oV9RPiMfm8E5D7zLbcP7ce4RndIdmoikSKV9HO5eTjB1rK40pFIDDmjBy1cdy6AuLbnu2dlc98wstu0sT3dYIpICiTRVLQfeMbPxwJaKlerjkGitmjbikUsHcc/ri7hv8hLmlmxizP8OoHOrvHSHJiLVKJHEsTJ8ZQHNUhuO1HXZWcYvv9WL/p0L+MV/ZnHmfW9x7sBOvDJ3tYYqEaknEhnk8NaaCETql5MOastLPzuWC8a+x9/e/mTP+oqhSgAlD5E6KpEnxwuB6wjmHN8ztra7n5TCuKQe6NQyj1iPd1QMVaLEIVI3JfIA4GPAAqArcCtBn8f0FMYk9cgqDVUiUu8kkjhaufvfgZ3uPsXdLwWOTHFcUk90KMiNuT6nQVbc8a9EpHZLJHHsDN9XmdkZZtYf6JjCmKQeiTVUScNsY/du51t3T+GFjzQsu0hdk0jiuM3M8oFfAr8C/gb8IqVRSb1xdv8i7jjnYIoKcjGgqCCX0SMO5bVrTqB7m6Zc/eRHXPn4DNZv2ZHuUEUkQYncVfVS+HEjcGJqw5H6KNZQJQBP/+QoHnxzGfe8vohpn6zjrhGHcGKvNmmIUESSUeUVh5kdaGZvmNnccPkQM7sp9aFJfdcgO4uRJ/Zg3MhjaJGXwyX/mM4Nz89hy/Zd6Q5NRCqRSFPVQ8D1hH0d7j4bOD+Rg5vZUDNbaGZLzGxUjO1mZveG22eb2eFJ7PsrM3Mza51ILFJ79e2QzwtXHsNlx3fjiWmfMewvb/HB8nWMm1nCMXdOpuuolznmzsmaplaklkjkyfE8d59mZpHrqvyTMBwg8X7gVGAFMN3Mxrv7/Ihiw4Ce4Wsw8AAwuKp9zaxTuO2zBOKXOqBxw2xuOL03Jx/Uhl8+PYsRY96jQZaxK3wQRA8OitQeiVxxrDWz7gTTxWJmI4BVCew3CFji7svcfQfwJDA8qsxw4FEPTAUKzKx9AvveTfBQomYPqmcGd2vFqz8/nryc7D1Jo4LmOBepHRK54hgJjAUOMrMS4BPg+wnsV0Qwe2CFFQRXFVWVKapsXzM7Cyhx91lRV0HfYGaXAZcBFBYWUlxcnEDImaW0tLTW1svWHbFH1i3ZUJbSmGtznaST6iW2TK2XRO6qWgacYmZNgCx332xmPwfuqWLXWL/Vo68Q4pWJud7M8oAbgW9V8d24+1iChEevXr18yJAhVe2ScYqLi6mt9VI0dXLMOc5b5DXkhBNOoLI/GvZHba6TdFK9xJap9ZJIUxUA7r7F3TeHi9cksMsKIHI2n44Eo+wmUibe+u4EQ5/MMrPl4foZZtYuwdOQOiLmHOcG67fu5Pt/e5+la0rTFJmIJJw4oiTy5950oKeZdQ0ngjofGB9VZjxwYXh31ZHARndfFW9fd5/j7m3cvYu7dyFIMIe7++p9PA+ppWI9OPinEYfyu7P7MadkI8PueYs/vbZQk0WJpEEifRyxVNkp7e67zOxKYCKQDTzs7vPM7PJw+xhgAnA6sATYClxS2b77GKvUUfEeHBzatx2/n/Ax901ewgsfreTW4X314KBIDYqbOMxsM7EThAGxR66L4u4TCJJD5LoxEZ+doPM9oX1jlOmSSBxSvxQ2a8Td5x3G9wZ25OZxc7nkH9MZ1q8dv/l2H9rnJ/SjKSL7IW7icHfN9ie12tHdW/PK1cfz0FvLuPeNxby5aA2/OPVAWuY15E+TFmvGQZEU2demKpFaIadBMGzJWYd24Lfj53Hbyx9jfH2prAcHRarfvnaOi9QqnVrm8feLBtKySc5e7at6cFCkeilxSL1hZnGHZ9eMgyLVR4lD6pV4Mw4CPDhlqW7fFakGShxSr8R6cLBRgyx6t2/GHa8s4OQ/TeG5GSvYvVvDnInsKyUOqVdiPTh413cPYcLVx/P4jwfTskkO1zw1izPve5u3Fq9Jd7gidZLuqpJ6J96Dg0d3b80LI4/hpTmrGD1xAT/4+zSO69maUcMOYvEXpYyeuJCSDWUUTZ2sW3hFKqHEIRklK8s469AOnNa3Lf+e+hn3TV7MGfe+TbYZ5a65P0QSoaYqyUiNGmTzw2O7MuXaE2naqMGepFFBt/CKxKfEIRktP7dh3DnOdQuvSGxKHJLx4t3C68At4+cpgYhEUeKQjBfvFt7BXVvw76mfcsLo/3L9c3P4fN3WNEUoUruoc1wyXkUH+J67qiIGRlyxfitjpizlqekreOqDz/lO/yJ+OqQ73QqbpjlqkfRR4hDh61t4o6cC7dgij9vOPpgrT+zJ2DeX8fi0T3luxgrOPKQDV57Ug/krNzF64kKNxCsZRYlDJAHt8hvzm2/34Yoh3fnb28v413ufMn7WSrIMKh5C1228kinUxyGShMJmjbh+WG/e+fVJNGvUgOiRS3Qbr2QCJQ6RfdCiSQ6lcW7jLdlQpsEUpV5T4hDZR5WNxHvUHW8weuICVm/cVoMRidQMJQ6RfRTrNt7chln89MTuDOzSkv8rXsqxd03mqidmMvOz9WmKUqT6qXNcZB9F3sYb666qz77ayiPvLeep6Z8zftZKDutUwCXHdOH0g9vz8uxVuhtL6iwlDpH9EG8kXoDOrfK4+cw+/OLUA3n2wxX8893lXP3kR9w8bi5bd5Sza7cGVZS6SU1VIinWtFEDLjq6C29ccwIPXzyQ7bt270kaFXQ3ltQlShwiNSQryzjpoLbs2LU75vaSDWWsizNnukhtosQhUsMquxtr8O9f56ePfciURWso1/S2Ukupj0Okhl17Wi+uf24OZRHPeuQ2zOaqk3uwtnQHz81YwYQ5q+mQ35gRAzvxvQEd6dQyL40Ri3yTEodIDavqbqzrhvbi9flf8p8PPue+yYu5b/JijunemnOP6MT2neXc8/pi3Y0laaXEIZIGld2N1ahBNmcc0p4zDmlPyYYynvkgGJn3qidmfqOc7saSdFEfh0gtVlSQy9Wn9OSt606kVZOcvbaX7Sznjlc+TkNkksmUOETqgKwsi3vH1RebtnPmfW8x9s2llGi2QqkBKU0cZjbUzBaa2RIzGxVju5nZveH22WZ2eFX7mtloM1sQln/ezApSeQ4itUW8u7HycxuQnZXF7ycs4Jg7JzPigXd55N3lrNm8fU+ZcTNLOObOyXQd9TLH3DmZcTNLaipsqYdS1sdhZtnA/cCpwApgupmNd/f5EcWGAT3D12DgAWBwFftOAq53911mdhdwPfDrVJ2HSG0R726sW8/qx9n9i/j0qy28NHsVL85ayW/Hz+PWF+dxdPfWFLVozAsfrWTbzuD5EfWNyP5KZef4IGCJuy8DMLMngeFAZOIYDjzq7g5MNbMCM2sPdIm3r7u/FrH/VGBECs9BpNao6m6sA1o1YeSJPRh5Yg8WfbGZF2etZPyslby9ZO1ex6p4Ul2JQ/ZFKhNHEfB5xPIKgquKqsoUJbgvwKXAf2J9uZldBlwGUFhYSHFxcRKhZ4bS0lLVS5TaXicFwO1HZgFNghUbF1NcvDhm2QE5cPhAuGRi7GOVbCjj6QmTKcyrusW6ttdLumRqvaQycViMddGPwsYrU+W+ZnYjsAt4LNaXu/tYYCxAr169PHIeaQlEz68t9bNOit6fHLfT/No3yzioXTO+1actp/ZpR7+i5pjt/d+vPtZLdcjUekll4lgBdIpY7gisTLBMTmX7mtlFwJnAyWEzl4jEEa9v5JpTDwRg0vwv+Ot/l3Dv5CW0z2/MKb3bckqfthzVrRUT5gTDv5dsKKNo6mQ9cChAahPHdKCnmXUFSoDzgf+JKjMeuDLswxgMbHT3VWa2Jt6+ZjaUoDP8BHffmsL4ReqFqvpGfnx8N9Zt2cHkBV8yaf5qnvlwBf+a+imNso1du53y8E8zdapLhZQljvCupyuBiUA28LC7zzOzy8PtY4AJwOnAEmArcEll+4aH/ivQCJgUXlJPdffLU3UeIvVBZU+qA7RsksOIAR0ZMaAj23aW8+7StVz5+Ey2l39z7vSyneXc9vJ8zjykPQ2y9RhYpkrpkCPuPoEgOUSuGxPx2YGRie4bru9RzWGKSITGDbM56aC2lO0oj7l9bekO+v9uEkd3b8XxBxZyfM/CvQZhHDezRDMc1mMaq0pEYupQkBuzU71lXkNO69eONxetZeK8LwDo0iqP43oWclzP1qzbsp1bX/x4T5+KmrjqHyUOEYkpXqf6b77dl7P7F+HufLJ2C28tXsubi9bw7IygbyQWPTdSvyhxiEhMkZ3qJRvKKIpqcjIzuhU2pVthUy46ugs7du1mxmfrOX/s1JjHK9lQxvvLvuLQTgU0bphdY+ch1U+JQ0TiquhUT+R5hZwGWRzZrRVFcZq4AM4bO5WcBlkc1qmAI7u2ZHC3VvTvXEBeTvCrSH0jdYMSh4hUq3hNXDef2ZvCZo2Z9slXvP/Juj3PjjTIMg7pmE+LvBzeWryWHeUaU6u2U+IQkWpV1XMjp/ZpC8DmbTv58NP1vP/JOqZ9so43Fny517HKdpbz+wkfc9ahHcjKijWghKSDEoeIVLuqnhsBaNa4IUN6tWFIrzYAdB318l5jEgF8uXk7/X83icM6FXBYpwL6dw7eC/K+nthKTVw1S4lDRGqFeLf/FuQ2ZNjB7Zj52QbunbyYikGGuhU24bBOBWSZ8eKslWzfpSaumqLEISK1Qry+kVvO6rsnAZRu38XsFRuY+VnwenPRGtaW7j0zYkUT1xmHtKehnnCvdkocIlIrVNU3AtC0UQOO7t6ao7u3BsDd6Xb9hLhNXH1/M5Fe7ZrRr6g5fTvk07dDc3q3b77ndmA1ce0bJQ4RqTUS6RuJZGZxm7ha5DXk3IGdmLtyIxPmrOaJacEUP9lZRo/CpjRrnM2sFRvZGY7iqCauxClxiEidFq+J67ff/rqJy90p2VDG3JJNzFu5kbklG5myaA27oy5VynaWc9O4uex258C2zejRpmnMhxUrrlQydbh5JQ4RqdMSaeIyMzq2yKNjizyG9msHBHdxxVK6fRfXPDULgCyDLq2a0KtdMw5s24xe7Zrx+fqt3D1pUUbP4a7EISJ1XrJNXBD/Lq4OBY159NJBLFxdysIvNrNw9SYWrN7Mq/NWE2/auIrO+KH92lU6nEp96VNR4hCRjBSvieu60w6iR5tm9GjTjDNov2db2Y5ylq4p5cz73o55vC83b6f3b16lqCCXboVN6V7YJHhv3YTubZry7pK13PD83HoxarASh4hkpESauCLl5mTTryg/7lhcLfIacvHRXVm6ppRla0v5YPk6tkbMaWKw191fZTvLufPVBQw/rEPMud4r1LYrFSUOEclY+9LElUhnPAQd8qs3bWPZmi0sXVPKb16YF+twrN64jYNveY3OLfM4oFUeB7RqEry3zKNzqzymLVvHjeNq15WKEoeISBKqGm6+gpnRPj+X9vm5HNOjNQ9OWRbzSiU/twFnH1bEp+u2snD1Zl7/+Is9twjHU7aznNtf/phjerSmddOcuFcrqbpSUeIQEUlSMsPNV4h3pXLrWf2+8cu8fLezamMZn321leVfbeWG5+fEPN6a0u0ccfvrNGqQRVGLXIoKcunYIpeOLfIoKshl2dpSxk5ZxrZ9GIqlIuHktOsxINZ2JQ4RkRqQaJ9KdtbXtw4f3QPu/++S2FP4Nsnh6pN7smL9Vko2lFGyvozXVm7iqy17D8FSIXhOZQ5fbdlB+/zGtMtvTPv8xhQ2bUSDcGiWcTNL9kpw0ZQ4RERqSHX2qfzmzD4xj1W2o5ySDWWc8ucpMY9Xur2c3700/xvrsgzaNAsSyYLVm/Y8oxKPEoeISC22L3d/9WjTNO7dX0UFjXn5quNYtXEbqzduC9/LgvdN26pMGqDEISJS61Xnlcq1px1EQV4OBXk59G7ffK/9jrlzctypfytovGERkXro7P5F3HHOwRQV5GJAUUEud5xzcJUJ6NrTepFbydPvoCsOEZF6a1+uVCKbxlbFKaMrDhER+Yaz+xfxzqiT2LF6yYextitxiIhIUpQ4REQkKUocIiKSFCUOERFJihKHiIgkJaWJw8yGmtlCM1tiZqNibDczuzfcPtvMDq9qXzNraWaTzGxx+N4ilecgIiLflLLEYWbZwP3AMKAPcIGZ9YkqNgzoGb4uAx5IYN9RwBvu3hN4I1wWEZEaksorjkHAEndf5u47gCeB4VFlhgOPemAqUGBm7avYdzjwSPj5EeDsFJ6DiIhESeWT40XA5xHLK4DBCZQpqmLftu6+CsDdV5lZm1hfbmaXEVzFAGw3s7n7chL1XGtgbbqDqGVUJ7GpXmKr7/VyQKyVqUwcsaakip7WKl6ZRPatlLuPBcYCmNkH7j4wmf0zgeplb6qT2FQvsWVqvaSyqWoF0CliuSOwMsEyle37RdicRfj+ZTXGLCIiVUhl4pgO9DSzrmaWA5wPjI8qMx64MLy76khgY9gMVdm+44GLws8XAS+k8BxERCRKypqq3H2XmV0JTASygYfdfZ6ZXR5uHwNMAE4HlgBbgUsq2zc89J3AU2b2Q+Az4HsJhDO2+s6sXlG97E11EpvqJbaMrBdzT6rrQEREMpyeHBcRkaQocYiISFLqdeKoasiTTGVmy81sjpl9ZGYfpDuedDGzh83sy8hnfDSkTdx6ucXMSsKfmY/M7PR0xljTzKyTmf3XzD42s3lmdnW4PiN/Xupt4khwyJNMdqK7H5aJ96BH+CcwNGqdhrSJXS8Ad4c/M4e5+4QajinddgG/dPfewJHAyPD3SUb+vNTbxEFiQ55IBnP3N4F1UaszfkibOPWS0dx9lbvPCD9vBj4mGOEiI39e6nPiiDeciQRP4b9mZh+GQ7PI174xpA0Qc0ibDHVlOIr1w5nSJBOLmXUB+gPvk6E/L/U5cez3sCX12DHufjhBM95IMzs+3QFJrfcA0B04DFgF/Cmt0aSJmTUFngV+7u6b0h1PutTnxJHIkCcZyd1Xhu9fAs8TNOtJQEPaxODuX7h7ubvvBh4iA39mzKwhQdJ4zN2fC1dn5M9LfU4ciQx5knHMrImZNav4DHwL0MjBX9OQNjFU/HIMfYcM+5kxMwP+Dnzs7n+O2JSRPy/1+snx8JbBe/h62JLb0xtR+plZN4KrDAiGnHk8U+vFzJ4AhhAMjf0F8FtgHPAU0JlwSBt3z6iO4jj1MoSgmcqB5cBPKtr2M4GZHQu8BcwBdoerbyDo58i4n5d6nThERKT61eemKhERSQElDhERSYoSh4iIJEWJQ0REkqLEISIiSVHiEKkGZlYeMXLsR9U5GrOZdYkcqVYk3VI2daxIhilz98PSHYRITdAVh0gKhXOf3GVm08JXj3D9AWb2Rjho4Btm1jlc39bMnjezWeHr6PBQ2Wb2UDgXxGtmlpu2k5KMp8QhUj1yo5qqzovYtsndBwF/JRjJgPDzo+5+CPAYcG+4/l5girsfChwOzAvX9wTud/e+wAbguyk9G5FK6MlxkWpgZqXu3jTG+uXASe6+LBwkb7W7tzKztUB7d98Zrl/l7q3NbA3Q0d23RxyjCzApnCwIM/s10NDdb6uBUxPZi644RFLP43yOVyaW7RGfy1H/pKSREodI6p0X8f5e+PldghGbAb4PvB1+fgO4AoLpj82seU0FKZIo/dUiUj1yzeyjiOVX3b3iltxGZvY+wR9qF4TrrgIeNrNrgTXAJeH6q4GxZvZDgiuLKwgmThKpNdTHIZJCYR/HQHdfm+5YRKqLmqpERCQpuuIQEZGk6IpDRESSosQhIiJJUeIQEZGkKHGIiEhSlDhERCQp/x/I0bwtIb/YWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Piecewise constant scheduling\n",
    "def piecewise_constant(boundaries, values):\n",
    "    boundaries = np.array([0] + boundaries)\n",
    "    values = np.array(values)\n",
    "    def piecewise_constant_fn(epoch):\n",
    "        return values[np.argmax(boundaries > epoch) - 1]\n",
    "    return piecewise_constant_fn\n",
    "\n",
    "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.9139 - accuracy: 0.7435 - val_loss: 0.9956 - val_accuracy: 0.6576\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.9046 - accuracy: 0.7290 - val_loss: 0.9662 - val_accuracy: 0.7184\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.9898 - accuracy: 0.6946 - val_loss: 0.7441 - val_accuracy: 0.7690\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.9004 - accuracy: 0.7171 - val_loss: 1.0430 - val_accuracy: 0.5872\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.0519 - accuracy: 0.6448 - val_loss: 0.9499 - val_accuracy: 0.7070\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6929 - accuracy: 0.7753 - val_loss: 0.6733 - val_accuracy: 0.8114\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5903 - accuracy: 0.8015 - val_loss: 0.7792 - val_accuracy: 0.7792\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5679 - accuracy: 0.8171 - val_loss: 0.5871 - val_accuracy: 0.7708\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5568 - accuracy: 0.8230 - val_loss: 0.7414 - val_accuracy: 0.8180\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5723 - accuracy: 0.8250 - val_loss: 0.6358 - val_accuracy: 0.8294\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5044 - accuracy: 0.8438 - val_loss: 0.6550 - val_accuracy: 0.8362\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5097 - accuracy: 0.8478 - val_loss: 0.6355 - val_accuracy: 0.8284\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.5100 - accuracy: 0.8491 - val_loss: 0.6437 - val_accuracy: 0.8298\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4789 - accuracy: 0.8553 - val_loss: 0.6515 - val_accuracy: 0.8264\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4969 - accuracy: 0.8476 - val_loss: 0.6005 - val_accuracy: 0.8532\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3450 - accuracy: 0.8878 - val_loss: 0.4860 - val_accuracy: 0.8676\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3261 - accuracy: 0.8922 - val_loss: 0.4753 - val_accuracy: 0.8686\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3176 - accuracy: 0.8957 - val_loss: 0.5088 - val_accuracy: 0.8624\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3048 - accuracy: 0.9003 - val_loss: 0.5246 - val_accuracy: 0.8696\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2948 - accuracy: 0.9039 - val_loss: 0.5197 - val_accuracy: 0.8704\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2876 - accuracy: 0.9063 - val_loss: 0.5128 - val_accuracy: 0.8740\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2796 - accuracy: 0.9082 - val_loss: 0.5258 - val_accuracy: 0.8710\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2754 - accuracy: 0.9117 - val_loss: 0.5447 - val_accuracy: 0.8726\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2716 - accuracy: 0.9129 - val_loss: 0.5506 - val_accuracy: 0.8746\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2607 - accuracy: 0.9153 - val_loss: 0.5680 - val_accuracy: 0.8720\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnG0lEQVR4nO3de5xcdX3/8dd7N7fNkrC4Cdmwgd0AYUO4CIiAojZeuagNVVvhp3KzTbHQqlVaqFq1P/1J1baKRTBqfoA/vLVFiJqKiC6UFhAQ5B6JECQXEoIkJGRz//z+OGeTyWRm9swyJ5OdeT8fj3nsnMv3zHe+Ozuf/V7O96uIwMzMLKuWemfAzMxGFgcOMzOrigOHmZlVxYHDzMyq4sBhZmZVceAwM7OqOHA0GUnnSlpf73xUIikkvave+bBsJF0t6Uc5XHdS+lmYXUWa3jTN8aW2rTYcOBpM+kcc6WOLpCckfVFSe3rK94CD65nHDKYCP8zzBSRNkPS/JT0iaUDSSkn9ks6StEf+LvL8Uqvm2pL+QNItklZL2iDpt5KukzSx1vmqg6dJPk/31zkfDWVUvTNgufgZ8D5gNPBa4BtAO/CBiBgABuqYtyFFxDN5Xl9SB3A7sB/wceCXwGbgNcAngDuAJXnmYW8haRbwE+Aq4EPAi8ChwBnA2LplrEYiYhuQ6+epKUWEHw30AK4GflS07+vAivT5ucD6ouNvB+4FNgJPAp8FxhQcHwP8H+ApYBPwBPBXBcdnAT8G1gGrgO8AXemxw4Eo2B5P8iX9nwXp/wx4vGA7gHcVbP99wWs/A1xbcEzA3wC/JQmIDwLvHaKMvkryBTmtxLFxwLj0+X7ANcDz6bV/BhxRcO65wHrgjcBD6TV/AUwvOOdA4Ebg98AG4DHgzIL3WfjoT/e/EvgpsBp4gSTIvaoonwHMBf4tfd0nCt93uWuXeL8fApZm+FzNBBYAa9P3fAdwVOFnDvggsCwtr/8LjK/m95S+78HP4X3AW9O8z06Pz063JxWk6U33HZ9xe/AabwTuSn8n9wDHFeXlfOB36fEfAn8BRL3/vveWh5uqmsMASe1jN5JOAa4D/hU4guQP5l0kgWLQNcDZwF+TBIL3A2vS9FOB20i+OE8A3gTsAyyQ1BIRjwIrSf5gAU4m+fJ5jaTBGu9soL9M/t4JfJTkD3cG8DaSGsKgz6T5uZAkgH0O+Jqkt5a5XgtwJnBdRCwtPh4RGyNiY7p5NXAiMCd9bxuAn0hqK0gyFriUpNxeBXSQ/Pc+6KskwfL1JOX7IdKyS68JcCpJc8o70u0JwLdIaosnkDSzLJQ0qSi7f08SlF5O0gQ5X1LPENcu9gwwWdLryxxH0gEkwSuANwPHAVcArQWnvRY4kuT3/27gj0gCyaCKv6e0KfXHJAHweOAS4Ivl8lQDn0tf4zjgOeA6SUrz8iqSWvoVwDEkAfPTOeZl5Kl35PKjtg+KahwkXyCrge+l2+dSUOMg+dL/RNE1ziD5r1IkX9YBnFrm9f4BuKVo335pmhPS7e8BX0uffxa4kqQp6FXpvqXAewrS76hxkASrRcDoEq/dThIUX1u0/0vAwjL53T+9/oeHKMfB9/26gn37kgS9Py0oywD6Cs55D0mNqiXdfgD4ZJnX6KXgv+EKeRGwgt1rFJ8r2B5FEtjeW+W1W0lqB0ES4H+YlvnkgnM+S1LjG1PmGleT9CWMKtj3deBnWX9PJLWnNcA+BcffS341jlMKrnFyum9auv0d4CdFeZ2Haxw7Hq5xNKZTJa2XtJGkSeE24C/LnPsK4GPp+evTEVffJvlj7wKOBbaTNMGUS/+6ovRPp8cOSX/2s7PGMTu91q3AbEkzgG7K1DhImmLGAU9K+qakP5Y02PY+Kz32k6LX/0DBaxdTmf3FDid533cM7oiItSRNLLMKztsUEYsKtpeT1O460u0vAx+XdIekz0h6xVAvLGl/SV+T9BtJa0maAPcHDio69YGCvG0Fnk3PyywitkXEecA0kprd74CLgcckHZGedixwe0RsrnCpR9I8DFpekJcsv6fDgQcionDE3x3k54GC58vTn4P5ncmutVpImrUs5c7xxnQbyX9wW4DlEbGlwrktJNXwfytx7FmG/qJtIWli+GiJYyvTn/3AV9MgcXy63Q6cRVIbWhwRy0pdPCKeltRH0ib9JuCfgE9KOpGdowLfTvKFV6jce36WpA3+8CHeV6X3XTil9NYyx1oAIuKbkm4CTifJ//9I+lxEfKrC9a8BpgAfJqmZbQJuIelrKlT8HoNhjpRMy/9bwLckfRz4DUkAOZdswbZSXrL8nrK8xvYS55Zsgs2gML+7/M7S6wdWlmscjWlDRCyOiKeGCBoAvwJmpucXP7amx1tI2ujLpT8CeKpE+nUAsbOf42MkQWIVSa3jZJI28/5KGYyk3+HHEfFhkg7UI9K0j5B8qfaUeO2nylxrO0nT2XskTSs+LmmcpHHptVtI+i0Gj00EjkqPZRYRSyNiXkT8CUm/xNz00OB/8K1FSV4DfCV9zw+T1DimVvOaFa6dJb/PkzSN7ZPu+hVJn1Rx4Moqy+/pEeCogmHjACcVXefZ9GdhWRwzzDxV8ig7+4gGFW83NQcO+wfgf0n6B0lHSpop6V2SPg8QEY8D3we+IemdkqZLeq2k96XpryBp+/+epBMlHSzpTZLmSZpQ8Dq3krRZ/yK97hKSL4J3UCFwpDcs/qmkoyRNB84j+W/x8TQwfRH4oqTzJR0q6RhJF0iaW+6awN+R/Od7l6TzJB2Rpn0fyaiervR930jSgftaSUcB/49klNO3M5Ytkr4s6dS0XI4h6aweDDyrSNr+T5E0RdK+6f7fAO+VNEvSK4HvsjMQZFXu2sX5+3NJV0p6i6RD0rL4R5IAeUN62ldJgsj3Jb0yLauz0vczpIy/p2+T1N7mp3l4M8k/GoUWkzSDfkrSYZLeQjKcutYuB94i6WJJMyS9n6Sz3wbVu5PFj9o+KDEct+j4uew+HPctwH+RdK6+QDI88aKC42OBz5MMtdxEMqSy8PgM4N/ZOWx1EfAVdh3SewG7D7O9Ot3XXZSfws7xM0jauteQDDu9G3hbwbki6b8Z/K/2WeBm4M1DlNO+JJ2+j5EM/1xFEsDOZGfHdqbhuEXXnU1BB25aDo+nr/EsSRDoLjj/T0mC2DZ2Dsd9OUmb+kBa1u8jGbX2qVJlVLBvCfDRStcuUQ7Hpu9xcJjsc8CdwPuKzjsCWEgyaGId8D/AkeU+c8CngIeq+T2RjGD7VXr81yRNWzs6x9NzXk0yymwg/VwMDtmttnO8bAd7uu98kiA1QDJg4CPAQL3/vveWh9JCMjOzMiT9C/CmiDiq3nnZG7hz3MysiKSLSWpE60kGNVxA0sRp4BqHmVkxSd8jadbal2Q2ha8BXw5/YQIOHGZmViWPqjIzs6o0RR9HR0dHHHroofXOxl7nxRdfpL29fegTm4jLpDSXS2mNXi733nvv6oiYXLy/KQLHlClTuOeee+qdjb1Of38/s2fPrnc29iouk9JcLqU1erlIKnkjrZuqzMysKg4cZmZWFQcOMzOrigOHmZlVxYHDzMyq4sBhZmZVceAwM7OqOHCYmVlVHDjMzKwqDhxmZlYVBw4zM6uKA4eZmVXFgcPMzKriwGFmZlVx4DAzs6rkuh6HpFOBLwOtwDci4rKi40qPnw5sAM6NiF+lx+YDbwNWRcSRBWleBnwP6AWWAH8SEc9XyseSF7Zz8mU/5+JT+jjj2O4h833Dfcv4wk2LWL5mgAM62hounZnZS5FbjUNSK3AFcBowCzhL0qyi004DZqSPucCVBceuBk4tcelLgFsiYgZwS7o9pGVrBrj0+ge54b5lFc+74b5lXHr9gyxbM0A0YDozs5cqzxrHCcDiiHgCQNJ3gTnAIwXnzAGujYgA7pTUIWlqRKyIiNsk9Za47hxgdvr8GqAf+NssGRrYso2/+8GD3L54ddlzFj64goEt20Zsui/ctMi1DjPLVZ6Boxt4umB7KXBihnO6gRUVrjslIlYARMQKSfuXOknSXJJaDGO6dq43vmHzNn7xcPn/yjdsjjL7R0a6ZWsG6O/vL5uu0Pr16zOf2yxcJqW5XEpr1nLJM3CoxL7ib7ss5wxLRMwD5gGMnTpjxzW7O9r470veUDbdyZf9nGVrBnbbP5LSZV0DudHXSx4Ol0lpLpfSmrVc8hxVtRQ4sGB7GrB8GOcUWylpKkD6c1XWDLWNbuXiU/oqnnPxKX20jW5t2HRmZi9VnoHjbmCGpOmSxgBnAguKzlkAnK3EScDawWaoChYA56TPzwFuzJKZ7o42PveOo4Zs/z/j2G4+946j6O5oQyMg3T5jk0rjAR3jMqUzM3upcmuqioitki4CbiIZjjs/Ih6WdEF6/CpgIclQ3MUkw3HPG0wv6TskneCTJC0FPhkR3wQuA74v6f3A74A/HiovvRNbKjb7FDvj2O5hfQHXI93Alm1cev2DfP/PX8W0/cZXfQ0zs2rleh9HRCwkCQ6F+64qeB7AhWXSnlVm/3PAG2uYzRGtpzMJFk89t8GBw8z2CN85PsJNn9QOwJOrX6xzTsysWThwjHBTJoxj7KgWnnrOgcPM9gwHjhGupUX0dI5nyXMb6p0VM2sSDhwNoKez3TUOM9tjHDgaQG/neJ56bgPbt9fk3kkzs4ocOBpAT2c7m7ZuZ+W6jfXOipk1AQeOBtDbmYysWrLa/Rxmlj8HjgYweC/HEvdzmNke4MDRAA7oaGNMa4sDh5ntEQ4cDaC1RRz4sjaeclOVme0BDhwNorez3TUOM9sjHDgaRHIvxwaS6b/MzPLjwNEgeieNZ2DLNp5dt6neWTGzBufA0SB6BofkeuoRM8uZA0eD6PWQXDPbQxw4GkR3RxujWsQST69uZjlz4GgQo1pbOPBlyZxVZmZ5cuBoIMn06q5xmFm+HDgaSK+H5JrZHuDA0UB6OsezftNWnntxc72zYmYNzIGjgQzOkutFncwsTw4cDWTHLLmes8rMcuTA0UCm7TeeFvleDjPLlwNHAxkzqoVp+4333eNmlisHjgbT0znefRxmlisHjgbT29nOk6tf9JBcM8uNA0eD6ekcz7qNW1mzYUu9s2JmDcqBo8H07pgl181VZpYPB44G0zspGZLrOavMLC8OHA1m2n7jkYfkmlmOHDgazLjRrRywb5unVzez3OQaOCSdKmmRpMWSLilxXJIuT48/IOm4odJKOkbSnZLul3SPpBPyfA8jUe8k38thZvnJLXBIagWuAE4DZgFnSZpVdNppwIz0MRe4MkPazwOfjohjgL9Pt61AT2e77+Uws9zkWeM4AVgcEU9ExGbgu8CconPmANdG4k6gQ9LUIdIGMDF9vi+wPMf3MCL1do7n+Q1bWOshuWaWg1E5XrsbeLpgeylwYoZzuodI+yHgJklfJAl8ry714pLmktRimDx5Mv39/cN5DyPSupVbAbj+5tuYvm9r2fPWr1/fVOWShcukNJdLac1aLnkGDpXYV3w7c7lzKqX9APDhiPgPSX8CfBN4024nR8wD5gH09fXF7NmzM2Z75Jv6zDq+ct9tdPYezuyXH1D2vP7+fpqpXLJwmZTmcimtWcslz6aqpcCBBdvT2L1Zqdw5ldKeA1yfPv83kmYtK3DQy9J7OTyyysxykGfguBuYIWm6pDHAmcCConMWAGeno6tOAtZGxIoh0i4H/iB9/gbg8Rzfw4jUNqaVronjeNId5GaWg9yaqiJiq6SLgJuAVmB+RDws6YL0+FXAQuB0YDGwATivUtr00n8GfFnSKGAjaT+G7ap30njfPW5mucizj4OIWEgSHAr3XVXwPIALs6ZN998OvKK2OW08vZ3t/OzRlfXOhpk1IN853qB6OttZvX4z6zZ6SK6Z1ZYDR4Pq7fRkh2aWDweOBtWTTq/uwGFmtebA0aB60hqHZ8k1s1pz4GhQ7WNHMXnCWM9ZZWY158DRwKZ3trNktZuqzKy2HDgaWE/neDdVmVnNOXA0sN5J7axat4kNm7fWOytm1kAcOBpYj4fkmlkOHDgaWO+OIblurjKz2nHgaGAH7RiS6xqHmdXOkIFD0mGSbpH0ULp9tKSP5581e6kmjhtNZ/sY1zjMrKay1Di+DlwKbAGIiAdIpjm3EaCnczxPel0OM6uhLIFjfET8smifh+mMEL2T2t05bmY1lSVwrJZ0COnSrZLeBazINVdWM72d7axYu5GNW7bVOytm1iCyrMdxIcna3TMlLQOeBN6Ta66sZgaH5P7u9xs4bMqEOufGzBpBlhpHRMSbgMnAzIh4TcZ0thcYHJK7xP0cZlYjWQLAfwBExIsRsS7d9+/5ZclqqdfTq5tZjZVtqpI0EzgC2FfSOwoOTQTG5Z0xq419x4+mY/xoz1llZjVTqY+jD3gb0AG8vWD/OuDPcsyT1VhPp0dWmVntlA0cEXEjcKOkV0XEHXswT1Zj0zvHc/eS5+udDTNrEFlGVd0n6UKSZqsdTVQRcX5uubKa6uls58ZfL2fT1m2MHdVa7+yY2QiXpXP8W0AXcApwKzCNpLnKRojeSeOJgKd/P1DvrJhZA8gSOA6NiE8AL0bENcBbgaPyzZbVUo9nyTWzGsoSOLakP9dIOhLYF+jNLUdWczvu5XAHuZnVQJY+jnmS9gM+DiwA9gE+kWuurKb2Gz+aCeNGucZhZjUxZOCIiG+kT28DDgaQ1JNnpqy2JNHb2e4ah5nVRMWmKkmvkvQuSfun20dL+jZw+x7JndVMT+d4TztiZjVRNnBI+gIwH3gn8GNJnwRuBu4CZuyZ7FmtTJ/UztLnN7B56/Z6Z8XMRrhKTVVvBY6NiI1pH8dy4OiIeHzPZM1qqaezne0By9YMMH1Se72zY2YjWKWmqoGI2AgQEc8Di6oNGpJOlbRI0mJJl5Q4LkmXp8cfkHRclrSS/jI99rCkz1eTp2bVu2P9cTdXmdlLU6nGcYikBQXbvYXbEfGHlS4sqRW4AngzsBS4W9KCiHik4LTTSJq9ZgAnAlcCJ1ZKK+n1wByS2s+mwf4Xq2zHvRyrX0xmITMzG6ZKgWNO0fY/VXntE4DFEfEEgKTvptcsDBxzgGsjIoA7JXVImkpyn0i5tB8ALouITQARsarKfDWlSfuMoX1Mq0dWmdlLVmmSw1tf4rW7gacLtpeS1CqGOqd7iLSHAa+V9FlgI/DRiLi7+MUlzQXmAkyePJn+/v5hv5FG0Tk2+NXjT9Pf/ywA69evd7kUcZmU5nIprVnLJcsNgMOlEvsi4zmV0o4C9gNOAl4JfF/SwWmtZefJEfNIlrylr68vZs+enT3nDerIZffy2Ip1DJZFf38/LpdduUxKc7mU1qzlkucSsEuBAwu2p5GMzMpyTqW0S4HrI/FLYDswqYb5bli9ne387vcb2LrNQ3LNbPjyDBx3AzMkTZc0BjiTZMqSQguAs9PRVScBayNixRBpbwDeACDpMGAMsDrH99Ewejvb2bo9WL5mY72zYmYj2JBNVZJ+yO5NTGuBe4CvDQ7ZLRYRWyVdBNwEtALzI+JhSRekx68CFgKnA4uBDcB5ldKml54PzJf0ELAZOKe4mcpK6ykYkntQ+tzMrFpZ+jieACYD30m33w2sJOmk/jrwvnIJI2IhSXAo3HdVwfMALsyaNt2/GXhvhnxbkd5JhdOrT65vZsxsxMoSOI6NiNcVbP9Q0m0R8TpJD5dNZXud/SeMZdzoFg/JNbOXJEsfx2RJBw1upM8HO6M355Iry8XgLLmeXt3MXoosNY6PALdL+i3JMNnpwF9IageuyTNzVns9neP57bMOHGY2fFnW41goaQYwkyRwPFbQIf6lHPNmOejtbOcXjz3Ltu0eT2Bmw5P1BsBXkEwDMgo4WhIRcW1uubLc9E5qZ/O27axYO1DvrJjZCJVlOO63gEOA+4Ft6e4AHDhGoMEhuU+5g9zMhilLjeN4YJbvlWgMveksuUuee5HuOufFzEamLKOqHgK68s6I7RldE8cxZlSLaxxmNmxZahyTgEck/RLYNLhzqPU4bO/U0iJ6XpasP/7qg4Y+38ysWJbA8am8M2F7Vk9ne1LjcOAws2HIMhz3pa7LYXuZ3s7x3L74WbbHuHpnxcxGoLKBQ9LtEfEaSevYdZJDkUwzNTH33Fku1gxsYeOW7Zx/0wa67/o5F5/SxxnHDt1VfsN9y/jCTYtYvmaAAzraGirdYJplawbovtNlYlZJpRUAX5P+nLDnsmN5u+G+ZSy4f+eyKMvWDHDp9Q8CVPwiueG+ZVx6/YMMbNnWcOlGQh7rkc6sHGUZZSupFZhCQaCJiN/lmK+a6uvri0WLFtU7G3uFky/7OcvW7H7z3+hWMeuAfcume2T5WrZs2/2z0gjpRkIe80jX3dHGf1/yhrLpCjXrSndDafRykXRvRBxfvD/LDYB/CXySZCr1waXjAji6pjm0PWJ5iaABsGVb0NE2umy6Ul88jZJuJOQxj3TlPgtmQ8kyquqDQF9EPJd3Zix/B3S0laxxdHe0cc35J5RNV66m0gjpRkIe80h3QEdb2TRmlWS5AfBpkhX/rAFcfEofbaNbd9nXNrqVi0/pa9p0IyGP9UhnVk7WFQD7Jf2YXW8A/OfccmW5GewM3TGCKOMIm8J01YzMGQnpmqlMlq0ZoG10K597x1HuGLdhG7JzXNInS+2PiE/nkqMcuHO8tEbv2BuORi+Tc+b/klXrNvGfH3xtVekavVyGq9HLZVid4+loqhkR4TW+zRrAzK4J3PHb59iybTujW7O0VJvtruInJyK2kSwdO2YP5cfMctTXNYHN27azZLVXgbThy9LHsQT4b0kLgB2fNvdxmI08M7uSCR8ee2YdM6b43l4bnix11eXAj9JzJxQ8zGyEOWT/dlpbxKJn1tU7KzaCZZnkcMR0gptZZWNHtXLwpHYee+aFemfFRrAsd45PBv4GOALYMZ1qRGSbq8DM9ip9XRO4/+k19c6GjWBZmqquAx4DpgOfJunzuDvHPJlZjmZ2TWDp8wOs37S13lmxESpL4OiMiG8CWyLi1og4Hzgp53yZWU760g5y93PYcGUJHFvSnyskvVXSscC0HPNkZjma2ZWMbXHgsOHKMhz3M5L2BT4CfAWYCHw411yZWW66O9rYZ+woFrmD3IYpy6iqH6VP1wKvzzc7Zpa3lhZx2JR9eNQ1DhumIZuqJB0m6RZJD6XbR0v6eP5ZM7O89HVNZNEz68iykJtZsSx9HF8HLiXt64iIB4Azs1xc0qmSFklaLOmSEscl6fL0+AOSjqsi7UclhaRJWfJiZjvN7JrA2oEtrHxh09AnmxXJEjjGR8Qvi/YNOY4vnSDxCuA0YBZwlqRZRaedBsxIH3OBK7OklXQg8GZgxCxfa7Y36Us7yH0joA1HlsCxWtIhJMvFIuldwIoM6U4AFkfEExGxGfguMKfonDnAtZG4E+iQNDVD2n8huSnR9WyzYfDIKnspsoyquhCYB8yUtAx4EnhPhnTdJKsHDloKnJjhnO5KaSX9IbAsIn4tqeyLS5pLUoth8uTJ9Pf3Z8hyc1m/fr3LpUgzlcl+Y8Wtv15MXzw95LnNVC7VaNZyyTKq6gngTZLagZaIWCfpQ8CXhkha6lu9uIZQ7pyS+yWNBz4GvGWI1yYi5pEEPPr6+qKRF1sZrkZfhGY4mqlMjn4iWdRp9uyhF3VqpnKpRrOWS+aVXCLixYgYrNf+dYYkS4EDC7ankcy0m+WccvsPIZn65NeSlqT7fyWpK+PbMLPUzK4J/HbVerZs217vrNgIM9wlwMq3Ee10NzBD0vR0IagzgQVF5ywAzk5HV50ErI2IFeXSRsSDEbF/RPRGRC9JgDkuIp4Z5vswa1pe1MmGK0sfRylDdkpHxFZJFwE3Aa3A/Ih4WNIF6fGrgIXA6cBiYANwXqW0w8yrmZWwc2SVF3Wy6pQNHJLWUTpACGjLcvGIWEgSHAr3XVXwPEg63zOlLXFOb5Z8mNnuDt1/nx2LOr395fXOjY0kZQNHRPhfELMGtnNRJw/JteoMt4/DzBpAX9cEFq30TYBWHQcOsyY2s2sCT//eizpZdRw4zJqYF3Wy4XDgMGtinnrEhsOBw6yJdXe00T6m1Ys6WVUcOMyaWEuL6Oua4JFVVhUHDrMm19c1kUUrvaiTZefAYdbkZnZNYM0GL+pk2TlwmDU5L+pk1XLgMGtyHlll1XLgMGtyHePHMGXiWAcOy8yBw8yY2TXRI6ssMwcOM2Nm1wQWr1rPVi/qZBk4cJjZjkWdnvSiTpaBA4eZ7bKok9lQHDjMbJdFncyG4sBhZowd1cp0L+pkGTlwmBmQdJB7USfLwoHDzAAv6mTZOXCYGeBFnSw7Bw4zAzz1iGXnwGFmgBd1suwcOMwMSBZ1OsyLOlkGDhxmtsNML+pkGThwmNkOg4s6rVrnRZ2sPAcOM9thcOqRR1e4n8PKc+Awsx08ssqycOAwsx28qJNl4cBhZrvo86JONoRcA4ekUyUtkrRY0iUljkvS5enxByQdN1RaSV+Q9Fh6/g8kdeT5HsyazeFdE1j8rBd1svJyCxySWoErgNOAWcBZkmYVnXYaMCN9zAWuzJD2ZuDIiDga+A1waV7vwawZ9XVNYPPW7Sx5zos6WWl51jhOABZHxBMRsRn4LjCn6Jw5wLWRuBPokDS1UtqI+GlEDM7CdicwLcf3YNZ0do6scnOVlTYqx2t3A08XbC8FTsxwTnfGtADnA98r9eKS5pLUYpg8eTL9/f1VZL05rF+/3uVSxGUCW7YHLYKf3vUQE57/DeByKadZyyXPwKES+4pvRy13zpBpJX0M2ApcV+rFI2IeMA+gr68vZs+ePUR2m09/fz8ul125TBIH338rA2PbmT37eMDlUk6zlkuegWMpcGDB9jRgecZzxlRKK+kc4G3AG8NzI5jVXF/XBB5Yuqbe2bC9VJ59HHcDMyRNlzQGOBNYUHTOAuDsdHTVScDaiFhRKa2kU4G/Bf4wIjbkmH+zpjVzihd1svJyq3FExFZJFwE3Aa3A/Ih4WNIF6fGrgIXA6cBiYANwXqW06aX/FRgL3CwJ4M6IuCCv92HWjGZOTRZ1+s3KdRx30H51zo3tbfJsqiIiFpIEh8J9VxU8D+DCrGnT/YfWOJtmVmRw6pHHVjhw2O5857iZ7caLOlklDhxmthsv6mSVOHCYWUkzuyZ4UScryYHDzErqm+JFnaw0Bw4zK2lwZJWbq6yYA4eZlbRzUSd3kNuuHDjMrKTBRZ0e82SHVsSBw8zK8qJOVooDh5mVNTNd1Gnbdo+ssp0cOMysrL4pyaJOKzc4cNhODhxmVtbMqUkH+dJ1XkbWdsp1riozG9keXZ6MqPrqrzdx41M/5+JT+jjj2O4h091w3zK+cNMilq8Z4ICOtoZNt2zNAN13Nm65jOk69BWljjtwmFlJN9y3jE/c+PCO7WVrBrj0+gcBKn753HDfMi69/kEGtmxzugZIV4qaYTqBvr6+WLRoUb2zsddp1tXLKnGZ7HTyZT9n2ZqB3faPHdXCiQd3lk131xPPsWnr7k1bTjfy0q245kNsWvH4biuyusZhZiUtLxE0ADZt3c4LA1vKpiv1ZeV0Iz9dIQcOMyvpgI62kjWO7o42brjw5LLpytVUnG5kpyvkUVVmVtLFp/TRNrp1l31to1u5+JQ+p2uydMVc4zCzkgY7UHeMHso4KqcwXTWjeUZiukYvlxVlznHneBNzR/DuXCaluVxKa/RykXRvRBxfvN9NVWZmVhUHDjMzq4oDh5mZVcWBw8zMquLAYWZmVXHgMDOzqjhwmJlZVRw4zMysKg4cZmZWFQcOMzOrigOHmZlVxYHDzMyqkmvgkHSqpEWSFku6pMRxSbo8Pf6ApOOGSivpZZJulvR4+nO/PN+DmZntKrfAIakVuAI4DZgFnCVpVtFppwEz0sdc4MoMaS8BbomIGcAt6baZme0hedY4TgAWR8QTEbEZ+C4wp+icOcC1kbgT6JA0dYi0c4Br0ufXAGfk+B7MzKxIngs5dQNPF2wvBU7McE73EGmnRMQKgIhYIWn/Ui8uaS5JLQZgk6SHhvMmGtwkYHW9M7GXcZmU5nIprdHLpafUzjwDh0rsK141qtw5WdJWFBHzgHkAku4ptRhJs3O57M5lUprLpbRmLZc8m6qWAgcWbE8Dlmc8p1LalWlzFunPVTXMs5mZDSHPwHE3MEPSdEljgDOBBUXnLADOTkdXnQSsTZuhKqVdAJyTPj8HuDHH92BmZkVya6qKiK2SLgJuAlqB+RHxsKQL0uNXAQuB04HFwAbgvEpp00tfBnxf0vuB3wF/nCE782r3zhqKy2V3LpPSXC6lNWW5KKKqrgMzM2tyvnPczMyq4sBhZmZVaejAMdSUJ81K0hJJD0q6X9I99c5PvUiaL2lV4T0+ntKmbLl8StKy9DNzv6TT65nHPU3SgZJ+IelRSQ9L+mC6vyk/Lw0bODJOedLMXh8RxzTjGPQCVwOnFu3zlDalywXgX9LPzDERsXAP56netgIfiYjDgZOAC9Pvk6b8vDRs4CDblCfWxCLiNuD3RbubfkqbMuXS1CJiRUT8Kn2+DniUZIaLpvy8NHLgKDediSV34f9U0r3p1Cy20y5T2gAlp7RpUhels1jPb5YmmVIk9QLHAnfRpJ+XRg4cL3nakgZ2ckQcR9KMd6Gk19U7Q7bXuxI4BDgGWAH8U11zUyeS9gH+A/hQRLxQ7/zUSyMHjixTnjSliFie/lwF/ICkWc8SntKmhIhYGRHbImI78HWa8DMjaTRJ0LguIq5Pdzfl56WRA0eWKU+ajqR2SRMGnwNvATxz8E6e0qaEwS/H1B/RZJ8ZSQK+CTwaEf9ccKgpPy8Nfed4OmTwS+yctuSz9c1R/Uk6mKSWAcmUM99u1nKR9B1gNsnU2CuBTwI3AN8HDiKd0iYimqqjuEy5zCZppgpgCfDng237zUDSa4D/Ah4Etqe7/46kn6PpPi8NHTjMzKz2GrmpyszMcuDAYWZmVXHgMDOzqjhwmJlZVRw4zMysKg4cZjUgaVvBzLH313I2Zkm9hTPVmtVbbkvHmjWZgYg4pt6ZMNsTXOMwy1G69sk/Svpl+jg03d8j6ZZ00sBbJB2U7p8i6QeSfp0+Xp1eqlXS19O1IH4qqa1ub8qangOHWW20FTVVvbvg2AsRcQLwryQzGZA+vzYijgauAy5P918O3BoRLweOAx5O988AroiII4A1wDtzfTdmFfjOcbMakLQ+IvYpsX8J8IaIeCKdJO+ZiOiUtBqYGhFb0v0rImKSpGeBaRGxqeAavcDN6WJBSPpbYHREfGYPvDWz3bjGYZa/KPO83DmlbCp4vg33T1odOXCY5e/dBT/vSJ//D8mMzQDvAW5Pn98CfACS5Y8lTdxTmTTLyv+1mNVGm6T7C7Z/EhGDQ3LHSrqL5B+1s9J9fwXMl3Qx8CxwXrr/g8A8Se8nqVl8gGThJLO9hvs4zHKU9nEcHxGr650Xs1pxU5WZmVXFNQ4zM6uKaxxmZlYVBw4zM6uKA4eZmVXFgcPMzKriwGFmZlX5/yLeytPrfqz0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.5900 - accuracy: 0.8068 - val_loss: 0.4719 - val_accuracy: 0.8506\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4897 - accuracy: 0.8415 - val_loss: 0.6139 - val_accuracy: 0.8268\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5159 - accuracy: 0.8403 - val_loss: 0.5231 - val_accuracy: 0.8482\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5217 - accuracy: 0.8467 - val_loss: 0.4573 - val_accuracy: 0.8538\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5321 - accuracy: 0.8478 - val_loss: 0.4612 - val_accuracy: 0.8562\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4952 - accuracy: 0.8553 - val_loss: 0.5051 - val_accuracy: 0.8456\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4987 - accuracy: 0.8570 - val_loss: 0.5012 - val_accuracy: 0.8610\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5095 - accuracy: 0.8573 - val_loss: 0.6029 - val_accuracy: 0.8290\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4786 - accuracy: 0.8629 - val_loss: 0.5068 - val_accuracy: 0.8670\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2841 - accuracy: 0.9014 - val_loss: 0.4215 - val_accuracy: 0.8824\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2406 - accuracy: 0.9114 - val_loss: 0.4426 - val_accuracy: 0.8816\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2236 - accuracy: 0.9182 - val_loss: 0.4655 - val_accuracy: 0.8792\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2053 - accuracy: 0.9243 - val_loss: 0.4499 - val_accuracy: 0.8836\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1954 - accuracy: 0.9265 - val_loss: 0.4609 - val_accuracy: 0.8816\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1814 - accuracy: 0.9316 - val_loss: 0.4581 - val_accuracy: 0.8820\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.1293 - accuracy: 0.9494 - val_loss: 0.4363 - val_accuracy: 0.8970\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1164 - accuracy: 0.9545 - val_loss: 0.4604 - val_accuracy: 0.8944\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1083 - accuracy: 0.9579 - val_loss: 0.4803 - val_accuracy: 0.8886\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1016 - accuracy: 0.9600 - val_loss: 0.5021 - val_accuracy: 0.8948\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0947 - accuracy: 0.9629 - val_loss: 0.5158 - val_accuracy: 0.8946\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0762 - accuracy: 0.9715 - val_loss: 0.5115 - val_accuracy: 0.8960\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0711 - accuracy: 0.9732 - val_loss: 0.5313 - val_accuracy: 0.8982\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0668 - accuracy: 0.9752 - val_loss: 0.5358 - val_accuracy: 0.8996\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0644 - accuracy: 0.9759 - val_loss: 0.5494 - val_accuracy: 0.8968\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0608 - accuracy: 0.9775 - val_loss: 0.5631 - val_accuracy: 0.8968\n"
     ]
    }
   ],
   "source": [
    "## Performance Scaling\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.02, momentum=0.9)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\", color='b')\n",
    "plt.tick_params('y', colors='b')\n",
    "plt.gca().set_xlim(0, n_epochs - 1)\n",
    "plt.grid(True)\n",
    "\n",
    "ax2 = plt.gca().twinx()\n",
    "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
    "ax2.set_ylabel('Validation Loss', color='r')\n",
    "ax2.tick_params('y', colors='r')\n",
    "\n",
    "plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization in DNN\n",
    "\n",
    "### $l_1$ and $l_2$ Regularization\n",
    "Come per altri modelli posso vincolare i pesi delle connessioni tra layer utilizzando delle penalizzazioni basate su norme $l_1$ o $l_2$. In Keras introducono tale regolarizzazione mediante il parametro **kernel_regularizer**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 1.5956 - accuracy: 0.8124 - val_loss: 0.7169 - val_accuracy: 0.8340\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7197 - accuracy: 0.8274 - val_loss: 0.6850 - val_accuracy: 0.8376\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([    \n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(100, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropout\n",
    "Dropout e' un metodo di regolarizzazione che si e' dimostrato molto efficace. Ad ogni training step, ogni neurone ha una probabilita' $p$ di essere disattivato, cioe' viene ignorato durante lo step di training. Il parametro $p$ e' detto **dropout rate** ed e' settato tra $0.1$ e $0.5$ nelle feedforward, tra $0.2$ e $0.3$ nelle RNN e tra $0.4$ e $0.5$ nelle CNN.\n",
    "\n",
    "![](dnn_16.png)\n",
    "\n",
    "Dropout genera una sorta di ensemble di reti non indipendenti.\n",
    "\n",
    "Nel testing si deve moltiplicare ogni peso di input per la **keep probability** $1-p$.\n",
    "\n",
    "In Keras, Dropout viene implementato dal layer **Dropout**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max-Norm Regularization\n",
    "Per ogni neurone, si vincolano i pesi in ingresso  tale che $||w||_2\\leq r$. In questo modo non viene aggiunto un termine di regolarizzazione ma una rescaling dei pesi. Riducendo $r$ si aumenta la quantita' di regolarizzazione e riduce l'overfitting. \n",
    "\n",
    "In Keras la max-norm regularization viene inserita mediante il parametro **kernel_constraint**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.4751 - accuracy: 0.8321 - val_loss: 0.3832 - val_accuracy: 0.8660\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3534 - accuracy: 0.8721 - val_loss: 0.3857 - val_accuracy: 0.8632\n"
     ]
    }
   ],
   "source": [
    "MaxNormDense = partial(keras.layers.Dense,\n",
    "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    MaxNormDense(300),\n",
    "    MaxNormDense(100),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}