4.4-overfitting-and-underfitting.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.8'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting and underfitting\n",
    "\n",
    "This notebook contains the code samples found in Chapter 3, Section 6 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "\n",
    "In all the examples we saw in the previous chapter -- movie review sentiment prediction, topic classification, and house price regression -- \n",
    "we could notice that the performance of our model on the held-out validation data would always peak after a few epochs and would then start \n",
    "degrading, i.e. our model would quickly start to _overfit_ to the training data. Overfitting happens in every single machine learning \n",
    "problem. Learning how to deal with overfitting is essential to mastering machine learning.\n",
    "\n",
    "The fundamental issue in machine learning is the tension between optimization and generalization. \"Optimization\" refers to the process of \n",
    "adjusting a model to get the best performance possible on the training data (the \"learning\" in \"machine learning\"), while \"generalization\" \n",
    "refers to how well the trained model would perform on data it has never seen before. The goal of the game is to get good generalization, of \n",
    "course, but you do not control generalization; you can only adjust the model based on its training data.\n",
    "\n",
    "At the beginning of training, optimization and generalization are correlated: the lower your loss on training data, the lower your loss on \n",
    "test data. While this is happening, your model is said to be _under-fit_: there is still progress to be made; the network hasn't yet \n",
    "modeled all relevant patterns in the training data. But after a certain number of iterations on the training data, generalization stops \n",
    "improving, validation metrics stall then start degrading: the model is then starting to over-fit, i.e. is it starting to learn patterns \n",
    "that are specific to the training data but that are misleading or irrelevant when it comes to new data.\n",
    "\n",
    "To prevent a model from learning misleading or irrelevant patterns found in the training data, _the best solution is of course to get \n",
    "more training data_. A model trained on more data will naturally generalize better. When that is no longer possible, the next best solution \n",
    "is to modulate the quantity of information that your model is allowed to store, or to add constraints on what information it is allowed to \n",
    "store. If a network can only afford to memorize a small number of patterns, the optimization process will force it to focus on the most \n",
    "prominent patterns, which have a better chance of generalizing well.\n",
    "\n",
    "The processing of fighting overfitting in this way is called _regularization_. Let's review some of the most common regularization \n",
    "techniques, and let's apply them in practice to improve our movie classification model from  the previous chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: in this notebook we will be using the IMDB test set as our validation set. It doesn't matter in this context.\n",
    "\n",
    "Let's prepare the data using the code from Chapter 3, Section 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "import numpy as np\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # Create an all-zero matrix of shape (len(sequences), dimension)\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # set specific indices of results[i] to 1s\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data\n",
    "x_test = vectorize_sequences(test_data)\n",
    "# Our vectorized labels\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fighting overfitting\n",
    "\n",
    "## Reducing the network's size\n",
    "\n",
    "\n",
    "The simplest way to prevent overfitting is to reduce the size of the model, i.e. the number of learnable parameters in the model (which is \n",
    "determined by the number of layers and the number of units per layer). In deep learning, the number of learnable parameters in a model is \n",
    "often referred to as the model's \"capacity\". Intuitively, a model with more parameters will have more \"memorization capacity\" and therefore \n",
    "will be able to easily learn a perfect dictionary-like mapping between training samples and their targets, a mapping without any \n",
    "generalization power. For instance, a model with 500,000 binary parameters could easily be made to learn the class of every digits in the \n",
    "MNIST training set: we would only need 10 binary parameters for each of the 50,000 digits. Such a model would be useless for classifying \n",
    "new digit samples. Always keep this in mind: deep learning models tend to be good at fitting to the training data, but the real challenge \n",
    "is generalization, not fitting.\n",
    "\n",
    "On the other hand, if the network has limited memorization resources, it will not be able to learn this mapping as easily, and thus, in \n",
    "order to minimize its loss, it will have to resort to learning compressed representations that have predictive power regarding the targets \n",
    "-- precisely the type of representations that we are interested in. At the same time, keep in mind that you should be using models that have \n",
    "enough parameters that they won't be underfitting: your model shouldn't be starved for memorization resources. There is a compromise to be \n",
    "found between \"too much capacity\" and \"not enough capacity\".\n",
    "\n",
    "Unfortunately, there is no magical formula to determine what the right number of layers is, or what the right size for each layer is. You \n",
    "will have to evaluate an array of different architectures (on your validation set, not on your test set, of course) in order to find the \n",
    "right model size for your data. The general workflow to find an appropriate model size is to start with relatively few layers and \n",
    "parameters, and start increasing the size of the layers or adding new layers until you see diminishing returns with regard to the \n",
    "validation loss.\n",
    "\n",
    "Let's try this on our movie review classification network. Our original network was as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "original_model = models.Sequential()\n",
    "original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "original_model.add(layers.Dense(16, activation='relu'))\n",
    "original_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "original_model.compile(optimizer='rmsprop',\n",
    "                       loss='binary_crossentropy',\n",
    "                       metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to replace it with this smaller network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "smaller_model = models.Sequential()\n",
    "smaller_model.add(layers.Dense(4, activation='relu', input_shape=(10000,)))\n",
    "smaller_model.add(layers.Dense(4, activation='relu'))\n",
    "smaller_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "smaller_model.compile(optimizer='rmsprop',\n",
    "                      loss='binary_crossentropy',\n",
    "                      metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Here's a comparison of the validation losses of the original network and the smaller network. The dots are the validation loss values of \n",
    "the smaller network, and the crosses are the initial network (remember: a lower validation loss signals a better model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.4594 - acc: 0.8188 - val_loss: 0.3429 - val_acc: 0.8812\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2659 - acc: 0.9075 - val_loss: 0.2873 - val_acc: 0.8905\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2060 - acc: 0.9276 - val_loss: 0.2827 - val_acc: 0.8879\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1697 - acc: 0.9401 - val_loss: 0.2921 - val_acc: 0.8851\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1495 - acc: 0.9470 - val_loss: 0.3100 - val_acc: 0.8812\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1283 - acc: 0.9572 - val_loss: 0.3336 - val_acc: 0.8748\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1121 - acc: 0.9624 - val_loss: 0.3987 - val_acc: 0.8593\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0994 - acc: 0.9670 - val_loss: 0.3788 - val_acc: 0.8702\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0889 - acc: 0.9716 - val_loss: 0.4242 - val_acc: 0.8603\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0782 - acc: 0.9757 - val_loss: 0.4256 - val_acc: 0.8653\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0691 - acc: 0.9792 - val_loss: 0.4515 - val_acc: 0.8638\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0603 - acc: 0.9820 - val_loss: 0.5102 - val_acc: 0.8610\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0518 - acc: 0.9851 - val_loss: 0.5281 - val_acc: 0.8587\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0446 - acc: 0.9873 - val_loss: 0.5441 - val_acc: 0.8589\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0367 - acc: 0.9903 - val_loss: 0.5777 - val_acc: 0.8574\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0313 - acc: 0.9922 - val_loss: 0.6377 - val_acc: 0.8555\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0247 - acc: 0.9941 - val_loss: 0.7269 - val_acc: 0.8501\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0203 - acc: 0.9956 - val_loss: 0.6920 - val_acc: 0.8516\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0156 - acc: 0.9970 - val_loss: 0.7689 - val_acc: 0.8425\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0144 - acc: 0.9966 - val_loss: 0.7694 - val_acc: 0.8487\n"
     ]
    }
   ],
   "source": [
    "original_hist = original_model.fit(x_train, y_train,\n",
    "                                   epochs=20,\n",
    "                                   batch_size=512,\n",
    "                                   validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.5737 - acc: 0.8049 - val_loss: 0.4826 - val_acc: 0.8616\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.3973 - acc: 0.8866 - val_loss: 0.3699 - val_acc: 0.8776\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2985 - acc: 0.9054 - val_loss: 0.3140 - val_acc: 0.8860\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2428 - acc: 0.9189 - val_loss: 0.2913 - val_acc: 0.8870\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2085 - acc: 0.9290 - val_loss: 0.2809 - val_acc: 0.8897\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1849 - acc: 0.9360 - val_loss: 0.2772 - val_acc: 0.8899\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1666 - acc: 0.9430 - val_loss: 0.2835 - val_acc: 0.8863\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1515 - acc: 0.9487 - val_loss: 0.2909 - val_acc: 0.8850\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1388 - acc: 0.9526 - val_loss: 0.2984 - val_acc: 0.8842\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1285 - acc: 0.9569 - val_loss: 0.3102 - val_acc: 0.8818\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1194 - acc: 0.9599 - val_loss: 0.3219 - val_acc: 0.8794\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1105 - acc: 0.9648 - val_loss: 0.3379 - val_acc: 0.8774\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1035 - acc: 0.9674 - val_loss: 0.3532 - val_acc: 0.8730\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0963 - acc: 0.9688 - val_loss: 0.3651 - val_acc: 0.8731\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0895 - acc: 0.9724 - val_loss: 0.3858 - val_acc: 0.8703\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0838 - acc: 0.9734 - val_loss: 0.4157 - val_acc: 0.8654\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0785 - acc: 0.9764 - val_loss: 0.4214 - val_acc: 0.8677\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0733 - acc: 0.9784 - val_loss: 0.4390 - val_acc: 0.8644\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0685 - acc: 0.9796 - val_loss: 0.4539 - val_acc: 0.8638\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.0638 - acc: 0.9822 - val_loss: 0.4744 - val_acc: 0.8617\n"
     ]
    }
   ],
   "source": [
    "smaller_model_hist = smaller_model.fit(x_train, y_train,\n",
    "                                       epochs=20,\n",
    "                                       batch_size=512,\n",
    "                                       validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epochs = range(1, 21)\n",
    "original_val_loss = original_hist.history['val_loss']\n",
    "smaller_model_val_loss = smaller_model_hist.history['val_loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VPW97/H3N9xSEK+gRYEEEStgQCGgbGrFKmrdFby0\nXg7POYXWeoqi6O7xqNVnE7R2225b99Zqq7aKPSLWyxE5ra3WyqVasQkUUGBzkYJGqUYqYAQk4Pf8\nsVaGIUySlcxlzWQ+r+dZT2atWWvmm8Uw3/zu5u6IiIgAlMQdgIiI5A8lBRERSVBSEBGRBCUFERFJ\nUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRhM5xB9BWvXr18vLy8rjDEBEpKEuWLPnQ3Xu3dl7B\nJYXy8nJqamriDkNEpKCY2aYo56n6SEREEpQUREQkQUlBREQSCq5NIZWGhgZqa2vZtWtX3KFIRKWl\npfTt25cuXbrEHYqIJOkQSaG2tpaePXtSXl6OmcUdjrTC3dmyZQu1tbUMGDAg7nBEJEmHqD7atWsX\nRxxxhBJCgTAzjjjiCJXspOhUVcV7fRQdIikASggFRv9eUoxmzoz3+ig6TFIQEZH0KSlkSG1tLRMn\nTmTQoEEMHDiQ6dOns3v37pTnvvfee3zta19r9TXPO+88tm7d2q54qqqquOuuu9p1bVSzZs1i2rRp\naZ8j0pFVVYFZsMG+x1GrgtK9vq2KOilk6qa6OxdddBEXXHAB69atY+3atdTX13PLLbcccO6ePXs4\n+uijefrpp1t93eeff55DDz00M0GKSCyqqsA92GDf47YkhXSub6uiTgqZqp97+eWXKS0tZcqUKQB0\n6tSJu+++m4cffpgdO3Ywa9YsJkyYwJe//GXOPPNMNm7cyIknngjAjh07uOSSSxgyZAgXXnghp5xy\nSmIaj/Lycj788EM2btzI4MGD+fa3v83QoUM5++yz2blzJwAPPfQQo0aNYvjw4Vx88cXs2LGjxVgn\nT57M1KlTOfXUUzn22GNZsGAB3/zmNxk8eDCTJ09OnDdnzhwqKio48cQTufHGGxPHH3nkEY4//nhG\njx7Nq6++mjheV1fHxRdfzKhRoxg1atR+z4lI4SjqpJApK1euZOTIkfsdO/jgg+nfvz/r168HYOnS\npTz99NMsXLhwv/Puv/9+DjvsMFatWsXtt9/OkiVLUr7HunXruPrqq1m5ciWHHnoozzzzDAAXXXQR\n1dXVLF++nMGDB/PLX/6y1Xg/+ugjXnvtNe6++24mTJjA9ddfz8qVK3njjTdYtmwZ7733HjfeeCMv\nv/wyy5Yto7q6mrlz57J582ZmzJjBq6++yiuvvMKqVasSrzl9+nSuv/56qqureeaZZ7jiiivadA9F\nisGMGfFeH0WHGKfQFlVV+5cQGuvpZszIbnev8ePHc/jhhx9w/JVXXmH69OkAnHjiiQwbNizl9QMG\nDOCkk04CYOTIkWzcuBGAN998k1tvvZWtW7dSX1/POeec02os559/PmZGRUUFRx11FBUVFQAMHTqU\njRs3smnTJsaNG0fv3sGEipMmTWLRokUA+x2/9NJLWbt2LQAvvfTSfkli+/bt1NfXtxqLSDEphC6p\nRZkUGm+s2b56unQMGTLkgDaC7du38/bbb3PcccexdOlSevTokdZ7dOvWLfG4U6dOieqjyZMnM3fu\nXIYPH86sWbNYsGBB5NcqKSnZ73VLSkrYs2dPu0YZf/bZZyxevJjS0tI2Xysi+UPVRxlw5plnsmPH\nDn71q18BsHfvXr773e8yefJkunfv3uK1Y8eO5cknnwRg1apVvPHGG216748//pg+ffrQ0NDA7Nmz\n2/cLNDF69GgWLlzIhx9+yN69e5kzZw6nn346p5xyCgsXLmTLli00NDTw1FNPJa45++yzuffeexP7\ny5Yty0gsIpJbRZ0UMlU/Z2Y8++yzPPXUUwwaNIjjjz+e0tJSfvCDH7R67VVXXUVdXR1Dhgzh1ltv\nZejQoRxyyCGR3/v222/nlFNOYezYsZxwwgnp/BoJffr04c477+SMM85g+PDhjBw5kokTJ9KnTx+q\nqqoYM2YMY8eOZfDgwYlr7rnnHmpqahg2bBhDhgzh5z//eUZiEZHcMs9E/UkOVVZWetNFdlavXr3f\nF1Qh2bt3Lw0NDZSWlvLWW29x1llnsWbNGrp27Rp3aFlXyP9uIoXGzJa4e2Vr5xVdm0K+2bFjB2ec\ncQYNDQ24O/fff39RJAQRyU9KCjHr2bOnlhcVkbxR1G0KIiKyPyUFERFJUFIQEZEEJQURKRq5GBFc\n6JQUMuSOO+5g6NChDBs2jJNOOonXX389I6970EEHAew3iV4+GDduXKsN5FHOEcmlXCxSU+iKMinM\nng3l5VBSEvxMdyDwa6+9xm9+8xuWLl3KihUreOmll+jXr18mQm23PXv2xPr+IlKYii4pzJ4NV14J\nmzYF8x5t2hTsp5MYNm/eTK9evRLzCPXq1Yujjz4aCKa/vvnmmznppJOorKxk6dKlnHPOOQwcODAx\n6re+vp4zzzyTESNGUFFRwXPPPdfi++3du5cbbriBUaNGMWzYMB544AEAFixYwGmnncaECRMYMmTI\nAdcddNBB3HDDDQwdOpSzzjqLv/zlL4wbN45jjz2WefPmAcF611OmTKGiooKTTz6Z+fPnA7Bz504u\nu+wyBg8ezIUXXpiYewngxRdfZMyYMYwYMYKvf/3rmghP8kquF6kpeO5eUNvIkSO9qVWrVh1wrDll\nZY1LVOy/lZVFfokDfPzxxz58+HAfNGiQT5061RcsWJD0fmV+//33u7v7dddd5xUVFb59+3b/4IMP\n/Mgjj3R394aGBt+2bZu7u9fV1fnAgQP9s88+c3f3Hj16uLv73/72Nx86dKi7uz/wwAN+++23u7v7\nrl27fOTIkb5hwwafP3++d+/e3Tds2JAyTsCff/55d3e/4IILfPz48b57925ftmyZDx8+3N3d77rr\nLp8yZYq7u69evdr79evnO3fu9B//+MeJ48uXL/dOnTp5dXW119XV+Wmnneb19fXu7n7nnXf6zJkz\n3d399NNP9+rq6mbvW1v+3UQyAeKOID5AjUf4ji26wWtvv92241EcdNBBLFmyhD/96U/Mnz+fSy+9\nlDvvvDOxaM2ECRMAqKiooL6+np49e9KzZ0+6devG1q1b6dGjB9/73vdYtGgRJSUlvPvuu7z//vt8\n/vOfT/l+L774IitWrEjMzLpt2zbWrVtH165dGT16NAMGDEh5XdeuXTn33HMTsXTr1o0uXbpQUVGR\nmIr7lVde4ZprrgHghBNOoKysjLVr17Jo0SKuvfZaAIYNG5aY4nvx4sWsWrWKsWPHArB7927GjBnT\n/pspIrEquqTQv39QZZTqeDo6derEuHHjGDduHBUVFTz66KOJpNDaVNWzZ8+mrq6OJUuW0KVLF8rL\ny9m1a1ez7+Xu3HvvvQesnbBgwYIWp+ju0qULFpahk2NpjKM93J3x48czZ86cdl0vkku5WKSm0BVd\nm8Idd0DT2ay7dw+Ot9eaNWtYt25dYn/ZsmWUlZVFvn7btm0ceeSRdOnShfnz57MpVdZKcs455/Cz\nn/2MhoYGANauXcsnn3zSvuCbOO200xJTcK9du5a3336bL3zhC3zpS1/i8ccfB4KFfVasWAHAqaee\nyquvvppYYe6TTz5JLLwjkm/UjtC6oispTJoU/LzllqDKqH//ICE0Hm+P+vp6rrnmGrZu3Urnzp05\n7rjjePDBB9sQ0yTOP/98KioqqKysbHUK7CuuuIKNGzcyYsQI3J3evXszd+7c9v8CSa666iqmTp1K\nRUUFnTt3ZtasWXTr1o2pU6cyZcoUBg8ezODBgxPLj/bu3ZtZs2Zx+eWX8+mnnwLw/e9/n+OPPz4j\n8YhIbmnqbImN/t1Ecifq1NlZrT4ys3PNbI2ZrTezm1I8f7eZLQu3tWa2NZvxiIhIy7JWfWRmnYD7\ngPFALVBtZvPcPbG6u7tfn3T+NcDJ2YpHRERal82SwmhgvbtvcPfdwBPAxBbOvxxodxeWQqsGK3b6\n9xLJT9lMCscA7yTt14bHDmBmZcAA4OX2vFFpaSlbtmzRF02BcHe2bNlCaWlp3KGISBP50vvoMuBp\nd9+b6kkzuxK4EqB/igEFffv2pba2lrq6uqwGKZlTWlpK37594w5DpE2qqjp+t9as9T4yszFAlbuf\nE+7fDODu/5bi3L8CV7v7n1t73VS9j0REcsEsmBinEOVD76NqYJCZDTCzrgSlgXlNTzKzE4DDgNey\nGIuIiESQtaTg7nuAacALwGrgSXdfaWa3mdmEpFMvA55wNQiISB4qtllWO8TgNRGRXFD1kYiIFBUl\nBRGRiIphllUlBRGRiDpqO0IyJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQ\nUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBREpGMUwdXXclBREpGDMnBl3\nBB2fkoKIiCQoKYhIXquqArNgg32PVZWUHebuccfQJpWVlV5TUxN3GCISAzMosK+svGFmS9y9srXz\nVFIQEZEEJQURKRgzZsQdQcenpCAiBUPtCNmnpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoi\nIpLQalIwsx5mVhI+Pt7MJphZl+yHJiIiuRalpLAIKDWzY4AXgf8OzIry4mZ2rpmtMbP1ZnZTM+dc\nYmarzGylmT0eNXAREcm8zhHOMXffYWbfAu539x+Z2bJWLzLrBNwHjAdqgWozm+fuq5LOGQTcDIx1\n94/M7Mj2/RoiIpIJUUoKZmZjgEnAb8NjnSJcNxpY7+4b3H038AQwsck53wbuc/ePANz9g2hhi4hI\nNkRJCtcR/DX/rLuvNLNjgfkRrjsGeCdpvzY8lux44Hgze9XMFpvZualeyMyuNLMaM6upq6uL8NYi\nItIerVYfuftCYCFA2OD8obtfm8H3HwSMA/oCi8yswt23NonhQeBBCKbOztB7i4hIE1F6Hz1uZgeb\nWQ/gTWCVmd0Q4bXfBfol7fcNjyWrBea5e4O7/w1YS5AkREQkBlGqj4a4+3bgAuB3wACCHkitqQYG\nmdkAM+sKXAbMa3LOXIJSAmbWi6A6aUO00EVEJNOiJIUu4biECwj/qgdarcJx9z3ANOAFYDXwZNgm\ncZuZTQhPewHYYmarCNopbnD3Le35RUREJH1RuqQ+AGwElhPU+ZcB26O8uLs/Dzzf5Ni/Jj124F/C\nTUREYtZqScHd73H3Y9z9PA9sAs7IQWwi0sFokZz8F6Wh+RAz+0ljl1Az+zHQIwexiUgHM3Nm3BFI\na6K0KTwMfAxcEm7bgUeyGZSIiMQjSlIY6O4zwpHJG9x9JnBstgMTkY6hqgrMgg32PVZVUn6KkhR2\nmtkXG3fMbCywM3shiUhHUlUF7sEG+x4rKeSnKL2PpgKPmtkhgAH/ACZnMygREYlHlGkulgHDzezg\ncD9Sd1QRkaZmzIg7AmlNs0nBzFKOHbCwYtDdf5KlmESkg1KVUf5rqaTQM2dRiIhIXmg2KYS9jERE\npIhE6X0kIiJFQklBREQSlBRERCSh1S6pZtYNuBgoTz7f3W/LXlgiIhKHKCWF54CJwB7gk6RNRIqM\nupR2fObe8no5Zvamu5+Yo3haVVlZ6TU1NXGHIVKUzPZNVyGFxcyWuHtla+dFKSn82cwqMhCTiIjk\nuShJ4YvAEjNbY2YrzOwNM1uR7cBEJD9oltPiEqX6qCzV8XAFtpxT9ZFIfFR9VLgyVn0UfvkfCpwf\nbofGlRBERCS7oizHOR2YDRwZbo+Z2TXZDkxEMi/dKh/NctrxRak+WgGMcfdPwv0ewGvuPiwH8R1A\n1Uci7afqn+KVyd5HBuxN2t8bHhMRkQ4mSlJ4BHjdzKrMrApYDPwyq1GJSMao95C0RavVRwBmNoKg\nayrAn9z9r1mNqgWqPhJpP1UfFa+o1Uctrbx2sLtvN7PDgY3h1vjc4e7+j0wEKiIi+aOlCfEeB74K\nLAGS/7awcP/YLMYlIlmg3kPSmmbbFNz9q+HPAe5+bNI2wN2VEERikG47gNoRpDVRxin8McoxEcm+\nmVokV7KspTaFUqA70MvMDmNfN9SDgWNyEJuIiORYSyWF/0nQnnBC+LNxew74afZDExFQl1LJrSgj\nmq9x93vb9eJm5wL/CXQCfuHudzZ5fjLw78C74aGfuvsvWnpNdUmVYqYupdJeaXdJbeTu95rZicAQ\noDTp+K9aCaATcB8wHqgFqs1snruvanLqr919WmtxiIhI9kVZo3kGMI4gKTwPfAV4BWgxKQCjgfXu\nviF8nScIlvVsmhREJCJ1KZVsizLNxdeAM4G/u/sUYDhwSITrjgHeSdqvJXUD9cXh4j1Pm1m/VC9k\nZleaWY2Z1dTV1UV4a5GOSe0Ikm1RksJOd/8M2GNmBwMfACm/vNvh/wHl4YyrfwAeTXWSuz/o7pXu\nXtm7d+8MvbWIiDQVJSnUmNmhwEMEvY+WAq9FuO5d9k8efdnXoAyAu29x90/D3V8AIyO8roiIZEmU\nhuarwoc/N7PfAwe7e5Q1mquBQWY2gCAZXAb8t+QTzKyPu28OdycAqyNHLiIiGddsScHMRjTdgMOB\nzuHjFrn7HmAa8ALBl/2T7r7SzG4zswnhadea2UozWw5cC0xO9xdKZfZsKC+HkpLg5+zZ2XgXEZHC\n1+w4BTObHz4sBSqB5QSjmocBNe4+JicRNtHWcQqzZ8OVV8KOHfuOde8ODz4IkyZlIUARkTyU9spr\n7n6Gu58BbAZGhA29I4GTadI2kM9uuWX/hADB/i23xBOPiEg+i9LQ/AV3f6Nxx93fBAZnL6TMevvt\nth0XESlmUZLCCjP7hZmNC7eHgCgNzXmhf/+2HRcRyTe5bBeNkhSmACuB6eG2KjxWEO64I2hDSNa9\ne3BcRCTfNbaLbtoUzHu1aVOwn63EEGmN5nzSngnxZs8O2hDefjsoIdxxhxqZRaQwlJcHiaCpsjLY\nuDH660RtaG6p99GT7n6Jmb3B/stxAhCOQs45zZIqIsWkpCT1zLhm8Nln0V8nE7OkTg9/fjX624qI\nSCb175+6pJCtdtGWuqRuDn9uSrVlJxwREUmW63bRlpbj/JgU1UYEA9jc3Q/OTkgiItKosf0zV+2i\nzSYFd++ZnbcUEZG2mDQpd51jonRJBcDMjjSz/o1bNoMSEelICmn+tVaTgplNMLN1wN+AhcBG4HdZ\njkukw9ECOcUp1+MM0hWlpHA7cCqw1t0HEKzCtjirUYl0QDNnxh2BxKHQ5l+LkhQa3H0LUGJmJe4+\nn2DWVBERaUWhzb8WJSlsNbODgEXAbDP7T+CT7IYl0jFUVQWDjMyC/cbHqkoqHoU2/1qUpDAR2Alc\nD/weeAs4P5tBiXQUVVVBPXLjiNTGx0oKxaPQ5l9raeW1+8xsrLt/4u573X2Puz/q7veE1UkiIkUh\nnd5DkyYFi3qVlQWlxLKy/F7kq6VpLtYCd5lZH+BJYI67/zU3YYl0PDNmxB2BtEfT1Rsbew9B9C/2\nXI4zSFers6SaWRlwWbh9DphDkCDWZj+8A2lCPBHJpUzNUhq3tJfjbBTOdfRDdz8ZuBy4AFidgRhF\nRPJeofUeSleUwWudzex8M5tNMGhtDXBR1iMTEckDhdZ7KF0tNTSPN7OHgVrg28BvgYHufpm7P5er\nAEXyhXoMFadC6z2UrpZKCjcDfwYGu/sEd3/c3TU+QQpWul/qGpFcnAqt91C6imI5ThEI/kOn83FP\n93qJj5bkzWBDs0gx04jkwldoE9LFTUlBOrR0v9Q1Ijk/pDN4rNAmpIubqo+kaKj6qDA1HTwGQUNv\n1Hr9TC18X+hUfSSSYRqRHI90/9Ivti6l6VJSkKKR7pe6qozike7gsWLrUpouJQUpGvpSL0zp/qVf\nbF1K06WkICJ5LRN/6U+aFMxT9NlnwU8lhOZlNSmY2blmtsbM1pvZTS2cd7GZuZlpRTcR2Y/+0s+t\nrCUFM+sE3Ad8BRgCXG5mQ1Kc1xOYDryerVgkP6j6RtpLf+nnTjZLCqOB9e6+wd13A08QrOLW1O3A\nD4FdWYxF8oCmiRDJf9lMCscA7yTt14bHEsxsBNDP3X+bxThERCSi2BqazawE+Anw3QjnXmlmNWZW\nU1dXl/3gJGM0TYRIYcnaiGYzGwNUufs54f7NAO7+b+H+IcBbQH14yeeBfwAT3L3ZIcsa0Vy4NCJY\nJD75MKK5GhhkZgPMrCvBcp7zGp90923u3svdy929HFhMKwlBRApTOnMXSW51ztYLu/seM5sGvAB0\nAh5295VmdhtQ4+7zWn4F6Wg0TURxysTC95I7mhBPRLKqoyx8X+jyofpIJKPUOF2Yim3h+0KnpCAF\nQ+McCpNmKS0sSgoiklWapbSwKClIXtM4h8KnuYsKixqaI9Ci3/lB4xxE2i9qQ3PWuqR2FOpOJyLF\nRNVHrdCi3/lD4xzio8FnxUMlhVaoO13+UDtCPFRaLi4qKbRC3emk2Km0XFyUFFqh7nRS7FRaLi5K\nCq1QdzrpCNJpE1BpubgoKUSgpQClkDW2CWzaFHTpbWwTiJoYVFouLkoKIh1cum0CKi0XFw1eE+ng\nSkpSD/ozC0q/Uhw0S6qIAGoTkLYpqqSgfu5SjNQmIG1RVElBUy9LMVKbgLRFUSUFkUKV7jQT6kEn\nUXX4pKCplzNH9ywe6XYpFWmLoup9pKmX06P7Fw+tcSyZoN5HIh2EppmQXCqqpKCpl9tO1W/xU5dS\nyaWiqj6S9Kj6KB5Np66GoEupehBJW6j6SCRPZKLnkLqUSq5okR2JTNVvbZepBWomTVISkNxQ9ZFI\nFqnnkOQLVR/lEa1vW7zUc0gKjZJClmngUXFTzyEpNEoKWZZP69uqG2n7pFPS02R0UmiUFLIsn6oP\nNCFg26Vb0lPPISk0amjOsnxqaNQ4g7bLp38/kXTkRUOzmZ1rZmvMbL2Z3ZTi+e+Y2RtmtszMXjGz\nIdmMJw5xVx9oRHJ68qmkJ5ILWUsKZtYJuA/4CjAEuDzFl/7j7l7h7icBPwJ+kq144hJ39UFVVVA6\naCwhND4upqSQTpuAGoql2GSzpDAaWO/uG9x9N/AEMDH5BHffnrTbA+iQlRuayz4+6bYJxF3SE8m1\nbCaFY4B3kvZrw2P7MbOrzewtgpLCtVmMp+gV44jkdHt/xV3SE8m12Hsfuft97j4QuBG4NdU5Znal\nmdWYWU1dXV1uA8wDjdUfZukNfivUKqN0qn8y0Sagkp4Uk2wmhXeBfkn7fcNjzXkCuCDVE+7+oLtX\nuntl7969Mxhi/kuu/oDiG/yWbvWP2gRE2iabSaEaGGRmA8ysK3AZMC/5BDMblLT7z8C6LMaTtjj+\n0s6nwW9xSPf3V5uASNtkLSm4+x5gGvACsBp40t1XmtltZjYhPG2ama00s2XAvwDfyFY8mRDH4K9U\nfeRbOp6P4qz+UZuASBu5e0FtI0eO9LhAetfPmNH2a8rKGjuR7r+VlUV/jcceC843C34+9ljb42iv\nxx5z7959/9i7d48eQyZ+fxFxB2o8wnds7A3N+S6Tg7/aU9JIt/oj7gn5VP0jUmCiZI582gq5pNDe\n6xv/0m/8C7ktf+nHXdIwS/3+Zrl5fxEJELGkoLmP2qA9cwdVVaUuIcyYkZuG65KS1DGbBV0sW5Pu\n+sCaO0gkP+TF3EcdTXsGf8U9zUS6XTJV/SNSXJQU2qAQB3+l+6Ws3j8ixUVJIYfimGYi3S/lTAz+\n0ohgkcKhNgVpUbptCiKSH9SmIBmh6h+R4tI57gAk/02apCQgUixUUhARkQQlBRERSVBSEBGRBCUF\nERFJUFIQEZGEghunYGZ1QL6uJtAL+DDuIFqg+NKT7/FB/seo+NKTTnxl7t7q0pUFlxTymZnVRBkc\nEhfFl558jw/yP0bFl55cxKfqIxERSVBSEBGRBCWFzHow7gBaofjSk+/xQf7HqPjSk/X41KYgIiIJ\nKimIiEiCkkIbmVk/M5tvZqvMbKWZTU9xzjgz22Zmy8LtX3Mc40YzeyN87wPmGbfAPWa23sxWmNmI\nHMb2haT7sszMtpvZdU3Oyfn9M7OHzewDM3sz6djhZvYHM1sX/jysmWu/EZ6zzsy+kaPY/t3M/iv8\n93vWzA5t5toWPwtZjrHKzN5N+nc8r5lrzzWzNeHn8aYcxvfrpNg2mtmyZq7N6j1s7jslts9flIWc\nte3bgD7AiPBxT2AtMKTJOeOA38QY40agVwvPnwf8DjDgVOD1mOLsBPydoP90rPcP+BIwAngz6diP\ngJvCxzcBP0xx3eHAhvDnYeHjw3IQ29lA5/DxD1PFFuWzkOUYq4D/FeEz8BZwLNAVWN70/1O24mvy\n/I+Bf43jHjb3nRLX508lhTZy983uvjR8/DGwGjgm3qjabCLwKw8sBg41sz4xxHEm8Ja7xz4Y0d0X\nAf9ocngi8Gj4+FHgghSXngP8wd3/4e4fAX8Azs12bO7+orvvCXcXA30z+Z5t1cz9i2I0sN7dN7j7\nbuAJgvueUS3FZ2YGXALMyfT7RtHCd0osnz8lhTSYWTlwMvB6iqfHmNlyM/udmQ3NaWDgwItmtsTM\nrkzx/DHAO0n7tcST2C6j+f+Icd6/Rke5++bw8d+Bo1Kckw/38psEJb9UWvssZNu0sIrr4WaqP/Lh\n/p0GvO/u65p5Pmf3sMl3SiyfPyWFdjKzg4BngOvcfXuTp5cSVIkMB+4F5uY4vC+6+wjgK8DVZval\nHL9/q8ysKzABeCrF03HfvwN4UFbPu656ZnYLsAeY3cwpcX4WfgYMBE4CNhNU0eSjy2m5lJCTe9jS\nd0ouP39KCu1gZl0I/vFmu/v/bfq8u2939/rw8fNAFzPrlav43P3d8OcHwLMERfRk7wL9kvb7hsdy\n6SvAUnd/v+kTcd+/JO83VquFPz9IcU5s99LMJgNfBSaFXxoHiPBZyBp3f9/d97r7Z8BDzbx3rJ9F\nM+sMXAT8urlzcnEPm/lOieXzp6TQRmH94y+B1e7+k2bO+Xx4HmY2muA+b8lRfD3MrGfjY4IGyTeb\nnDYP+B/0glZdAAAC/UlEQVRhL6RTgW1JxdRcafavszjvXxPzgMbeHN8AnktxzgvA2WZ2WFg9cnZ4\nLKvM7FzgfwMT3H1HM+dE+SxkM8bkdqoLm3nvamCQmQ0IS4+XEdz3XDkL+C93r031ZC7uYQvfKfF8\n/rLVot5RN+CLBMW4FcCycDsP+A7wnfCcacBKgp4Ui4F/ymF8x4bvuzyM4ZbweHJ8BtxH0OvjDaAy\nx/ewB8GX/CFJx2K9fwQJajPQQFAv+y3gCOCPwDrgJeDw8NxK4BdJ134TWB9uU3IU23qCuuTGz+DP\nw3OPBp5v6bOQw/v3f8LP1wqCL7g+TWMM988j6HHzVrZiTBVfeHxW4+cu6dyc3sMWvlNi+fxpRLOI\niCSo+khERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBREQma21/afwTVjM3aaWXnyDJ0i+apz3AGI\n5JGd7n5S3EGIxEklBZFWhPPp/yicU/8vZnZceLzczF4OJ3z7o5n1D48fZcEaB8vD7Z/Cl+pkZg+F\nc+a/aGafC8+/NpxLf4WZPRHTrykCKCmIJPtck+qjS5Oe2+buFcBPgf8Ij90LPOruwwgmpLsnPH4P\nsNCDCf1GEIyEBRgE3OfuQ4GtwMXh8ZuAk8PX+U62fjmRKDSiWSRkZvXuflCK4xuBL7v7hnDisr+7\n+xFm9iHB1A0N4fHN7t7LzOqAvu7+adJrlBPMez8o3L8R6OLu3zez3wP1BLPBzvVwMkCROKikIBKN\nN/O4LT5NeryXfW16/0wwF9UIoDqcuVMkFkoKItFcmvTztfDxnwlm9QSYBPwpfPxHYCqAmXUys0Oa\ne1EzKwH6uft84EbgEOCA0opIrugvEpF9Pmf7L97+e3dv7JZ6mJmtIPhr//Lw2DXAI2Z2A1AHTAmP\nTwceNLNvEZQIphLM0JlKJ+CxMHEYcI+7b83YbyTSRmpTEGlF2KZQ6e4fxh2LSLap+khERBJUUhAR\nkQSVFEREJEFJQUREEpQUREQkQUlBREQSlBRERCRBSUFERBL+Py+ECKSeVveDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43c33d57f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# b+ is for \"blue cross\"\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "As you can see, the smaller network starts overfitting later than the reference one (after 6 epochs rather than 4) and its performance \n",
    "degrades much more slowly once it starts overfitting.\n",
    "\n",
    "Now, for kicks, let's add to this benchmark a network that has much more capacity, far more than the problem would warrant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bigger_model = models.Sequential()\n",
    "bigger_model.add(layers.Dense(512, activation='relu', input_shape=(10000,)))\n",
    "bigger_model.add(layers.Dense(512, activation='relu'))\n",
    "bigger_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "bigger_model.compile(optimizer='rmsprop',\n",
    "                     loss='binary_crossentropy',\n",
    "                     metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.4539 - acc: 0.8011 - val_loss: 0.4150 - val_acc: 0.8229\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.2148 - acc: 0.9151 - val_loss: 0.2742 - val_acc: 0.8901\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.1217 - acc: 0.9544 - val_loss: 0.5442 - val_acc: 0.7975\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0552 - acc: 0.9835 - val_loss: 0.4316 - val_acc: 0.8842\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0662 - acc: 0.9888 - val_loss: 0.5098 - val_acc: 0.8822\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0017 - acc: 0.9998 - val_loss: 0.6867 - val_acc: 0.8811\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.1019 - acc: 0.9882 - val_loss: 0.6737 - val_acc: 0.8800\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0735 - acc: 0.9896 - val_loss: 0.6185 - val_acc: 0.8772\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 3s - loss: 3.4759e-04 - acc: 1.0000 - val_loss: 0.7328 - val_acc: 0.8818\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0504 - acc: 0.9912 - val_loss: 0.7092 - val_acc: 0.8791\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0589 - acc: 0.9919 - val_loss: 0.6831 - val_acc: 0.8785\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 3s - loss: 1.9067e-04 - acc: 1.0000 - val_loss: 0.8005 - val_acc: 0.8784\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0623 - acc: 0.9916 - val_loss: 0.7540 - val_acc: 0.8740\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0274 - acc: 0.9954 - val_loss: 0.7806 - val_acc: 0.8670\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0011 - acc: 0.9998 - val_loss: 0.8107 - val_acc: 0.8783\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0445 - acc: 0.9943 - val_loss: 0.8394 - val_acc: 0.8702\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0268 - acc: 0.9959 - val_loss: 0.7708 - val_acc: 0.8745\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 3s - loss: 7.7057e-04 - acc: 1.0000 - val_loss: 0.8885 - val_acc: 0.8738\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0297 - acc: 0.9962 - val_loss: 0.8419 - val_acc: 0.8728\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.0018 - acc: 0.9998 - val_loss: 0.8896 - val_acc: 0.8682\n"
     ]
    }
   ],
   "source": [
    "bigger_model_hist = bigger_model.fit(x_train, y_train,\n",
    "                                     epochs=20,\n",
    "                                     batch_size=512,\n",
    "                                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how the bigger network fares compared to the reference one. The dots are the validation loss values of the bigger network, and the \n",
    "crosses are the initial network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYVNWZ7/HvS4siKAhKPCjSjQYDjYByU0ISbxGNUdpo\nLmA/RkwiiUqGODlzxMFHWj1MYkxi1OjkYETM2JHxkjicMyYao5Jg1DQ6KAIKqA02mtgQDToNQsN7\n/ti7yuqmuquqq3Zdf5/nqadrr1pV9VJd7LfXZa9l7o6IiAhAr0IHICIixUNJQURE4pQUREQkTklB\nRETilBRERCROSUFEROKUFEREJE5JQURE4pQUREQkbr9CB5Cpww47zGtqagodhohISXn++ee3uvvg\nVPVKLinU1NSwcuXKQochIlJSzGxTOvXUfSQiInGRJgUzO8vMXjWzjWY2L8nj1Wb2ezN7ycyeMrOh\nUcYjIiLdiywpmFkVcDvwOaAWmGlmtZ2q/RD4hbuPBa4HvhdVPCIiklqUYwqTgY3u/jqAmS0F6oC1\nCXVqgX8M7z8JPNyTN9q9ezctLS3s3Lkzi3Aln/r06cPQoUPp3bt3oUMRkQRRJoUjgTcTjluAEzvV\neRE4H7gF+AJwsJkd6u7bMnmjlpYWDj74YGpqajCzbGKWPHB3tm3bRktLC8OHDy90OCKSoNADzf8T\nONnM/gs4GdgC7Olcycxmm9lKM1vZ2tq6z4vs3LmTQw89VAmhRJgZhx56qFp2ImlqbISaGujVK/jZ\n2Bjde0WZFLYARyUcDw3L4tz9LXc/391PAOaHZe91fiF3X+TuE9194uDByafZKiGUFv2+pJTk86Sc\n7L1nz4ZNm8A9+Dl7dnQxRJkUmoARZjbczPYHZgDLEiuY2WFmFovhamBxhPGIiGQs3yflzubPh7a2\njmVtbUF5FCJLCu7eDswBHgXWAfe7+xozu97MpofVTgFeNbP1wOHAwqjiiVpLSwt1dXWMGDGCY445\nhrlz57Jr166kdd966y2++MUvpnzNs88+m/fe26fhlJaGhgZ++MMf9ui56VqyZAlz5szJuo5IMcv3\nSbmzzZszK89WpGMK7v6Iux/r7se4+8Kw7Fp3Xxbef9DdR4R1vuHuH0YZT2cNDbl5HXfn/PPP57zz\nzmPDhg2sX7+eDz74gPlJvjXt7e0cccQRPPjggylf95FHHuGQQw7JTZAi0iP5Pil3NmxYZuXZKvRA\nc0Fdd11uXueJJ56gT58+XHLJJQBUVVVx8803s3jxYtra2liyZAnTp0/ntNNO4/TTT6e5uZnjjjsO\ngLa2Nr785S9TW1vLF77wBU488cT4Mh41NTVs3bqV5uZmRo0axaWXXsro0aOZNm0aO3bsAODOO+9k\n0qRJjBs3jgsuuIC2zn/SdDJr1iwuu+wyTjrpJI4++mieeuopvva1rzFq1ChmzZoVr3ffffcxZswY\njjvuOK666qp4+d13382xxx7L5MmTefrpp+Plra2tXHDBBUyaNIlJkyZ1eEyklOX7pNzZwoXQt2/H\nsr59g/IoVHRSyJU1a9YwYcKEDmX9+/dn2LBhbNy4EYAXXniBBx98kOXLl3eod8cddzBw4EDWrl3L\nDTfcwPPPP5/0PTZs2MAVV1zBmjVrOOSQQ3jooYcAOP/882lqauLFF19k1KhR3HXXXSnjfffdd3nm\nmWe4+eabmT59OldeeSVr1qxh9erVrFq1irfeeourrrqKJ554glWrVtHU1MTDDz/M22+/zYIFC3j6\n6adZsWIFa9d+dMnJ3LlzufLKK2lqauKhhx7iG9/4RkafoUixyvdJubP6eli0CKqrwSz4uWhRUB6F\nklsQL1sNDR1bCLFJMAsW5K47KZkzzjiDQYMG7VO+YsUK5s6dC8Bxxx3H2LFjkz5/+PDhHH/88QBM\nmDCB5uZmAF5++WWuueYa3nvvPT744APOPPPMlLGce+65mBljxozh8MMPZ8yYMQCMHj2a5uZmNm3a\nxCmnnEJspld9fT1/+MMfADqUf+UrX2H9+vUAPP744x2SxPbt2/nggw9SxiJS7GIn3/nzgy6jYcOC\nhBDVSbmrGPL1fhWZFGInf7NgNkG2amtr9xkj2L59O5s3b+bjH/84L7zwAv369cvqPQ444ID4/aqq\nqnj30axZs3j44YcZN24cS5Ys4amnnkr7tXr16tXhdXv16kV7e3uPrjLeu3cvzz77LH369Mn4uSLF\nLp8n5UJT91EOnH766bS1tfGLX/wCgD179vDd736XWbNm0bdzu7OTqVOncv/99wOwdu1aVq9endF7\nv//++wwZMoTdu3fTmKM5cpMnT2b58uVs3bqVPXv2cN9993HyySdz4oknsnz5crZt28bu3bt54IEH\n4s+ZNm0at912W/x41apVOYlFRPKropPCggW5eR0z49e//jUPPPAAI0aM4Nhjj6VPnz78y7/8S8rn\nXn755bS2tlJbW8s111zD6NGjGTBgQNrvfcMNN3DiiScydepURo4cmc0/I27IkCF8//vf59RTT2Xc\nuHFMmDCBuro6hgwZQkNDA1OmTGHq1KmMGjUq/pxbb72VlStXMnbsWGpra/nZz36Wk1hEJL/Mc9F/\nkkcTJ070zpvsrFu3rsMJqpTs2bOH3bt306dPH1577TU++9nP8uqrr7L//vsXOrTIlfLvTaTUmNnz\n7j4xVb2KG1MoNm1tbZx66qns3r0bd+eOO+6oiIQgIsVJSaHADj74YG0vKiJFo6LHFEREpCMlBRER\niVNSEJGiV8ilq3OhlOLXmIKIFLXY0tWxZb1iS1dDaVxQVmrxq6WQI1VVVRx//PGMGzeO8ePH86c/\n/QlIf5nsYnbQQQflpI5ITxR66epslVr8FZkUomjKHXjggaxatYoXX3yR733ve1x99dUAaS+TnY32\n9vZIX1+kkAq9dHW2Si3+iksK+dhFafv27QwcOBAg7WWy77rrrviS1Jdeeml8Y5qulqRuaGjgoosu\nYurUqVx00UUd3v+pp57i5JNPpq6ujqOPPpp58+bR2NjI5MmTGTNmDK+99lo8ttNOO42xY8dy+umn\nszn8lr7xxhtMmTKFMWPGcM0113R47ZtuuolJkyYxduxYFuTqknCRbhR66epslVz87l5StwkTJnhn\na9eu3aesK9XV7kE66Hirrk77JZLq1auXjxs3zj/xiU94//79feXKle7u/sYbb/jo0aPd3f2mm27y\n2bNnu7v76tWrvaqqypuamnzLli1eXV3t27Zt8127dvmnPvUpv+KKK9zdfebMmf7HP/7R3d03bdrk\nI0eOdHf3BQsW+Pjx472trW2fWJ588kkfMGCAv/XWW75z504/4ogj/Nprr3V395/85Cc+d+5cd3c/\n55xzfMmSJe7uftddd3ldXZ27u5977rl+zz33uLv7T3/6U+/Xr5+7uz/66KN+6aWX+t69e33Pnj3+\n+c9/3pcvX+7uHq+TiUx+b1K57r3XvW/fjv9f+/YNyktBscQPrPQ0zrEV11KIqikX6z565ZVX+O1v\nf8tXv/pVvNMSIitWrGDGjBlAx2Wy//znP3PyySczaNAgevfuzZe+9KX4cx5//HHmzJnD8ccfz/Tp\n0zssST19+nQOPPDApPFMmjSJIUOGcMABB3DMMccwbdo0AMaMGRNfdvuZZ57hwgsvBOCiiy5ixYoV\nADz99NPMnDkzXh7z2GOP8dhjj3HCCScwfvx4XnnlFTZs2JDV5yaSSr73E8i1Uou/4mYfDRsWdBkl\nK8+VKVOmsHXrVlpbW7N+re6WpO5uOe7OS2InLpedzhiExTaaSODuXH311Xzzm99MJ3SRnCn1patL\nKf6KaynkYxelV155hT179nDooYd2KO9qmexJkyaxfPly3n33Xdrb2+O7qkG0S1J/8pOfZOnSpQA0\nNjby6U9/Oh5nYnnMmWeeyeLFi+MtlS1btvDOO+/kLB4RKbyKaylEtYvSjh074jujuTv33HMPVVVV\nHepcfvnlXHzxxdTW1jJy5Mj4MtlHHnkk//zP/8zkyZMZNGgQI0eOjC+ffeutt3LFFVcwduxY2tvb\n+cxnPpOzZalvu+02LrnkEm666SYGDx7M3XffDcAtt9zChRdeyI033khdXV28/rRp01i3bh1TpkwB\ngmmo9957Lx/72MdyEo+IFIF0Bh6K6ZbtQHMhtbe3+44dO9zdfePGjV5TU+Mffvihu7u///777u6+\ne/duP+ecc/xXv/pVweLMl1L5vUn27r03mMxhFvwslUHickKaA80V11IopO6WyW5oaODxxx9n586d\nTJs2jfPOO6/A0YrkRqld0VvpIt1kx8zOAm4BqoCfu/v3Oz0+DLgHOCSsM8/dH+nuNcttk51Kpt9b\nZaipST65o7oawolwkgfpbrIT2UCzmVUBtwOfA2qBmWZW26naNcD97n4CMAO4o6fvF2Vyk9zT7ysz\npbSgWmeldkVvpYty9tFkYKO7v+7uu4ClQF2nOg70D+8PAN7qyRv16dOHbdu26URTItydbdu2JZ1m\nK/vKx1X4USq5K3orXJRjCkcCbyYctwAndqrTADxmZt8G+gGf7ckbDR06lJaWlpxcFyD50adPH4YO\nHVroMEpCdwuqlUKf/MKFHccUIPfTwCV3Cj3QPBNY4u4/MrMpwL+Z2XHuvjexkpnNBmYDDEvy50Xv\n3r0ZPnx4PuIVybtS736Jahq4RCPK7qMtwFEJx0PDskRfB+4HcPdngD7AYZ1fyN0XuftEd584ePDg\niMIVKU7l0P1SXx8MKu/dG/xUQiheUSaFJmCEmQ03s/0JBpKXdaqzGTgdwMxGESQF9QGJJMjHVfgi\nMZElBXdvB+YAjwLrCGYZrTGz681seljtu8ClZvYicB8wyzVaLNJBMSyoVsqznyQzkV6nEIVk1ymI\nSHQ6X3wGQUulmFf6lH0V/DoFESkPpbadpGRHSUFEulXqs59A3V+ZUFIQkW6V+uynYrr4r6GhsM9P\nh8YURKRbpT6mUExrL5kFiakQz9eYgojkRDHMfspGOXR/5ZOSgoikVMoXnxW6+6uhIUimsR1uY/fT\n7QrK9vmZUveRiJS1Yur+UveRiBSFSp59U+rdX/lW6AXxRCRi2vks+HcWw791wYLCPj8d6j4SKXPF\nNPtGCkfdRyICaPaNZEZJQaTMFXr2jZQWJQWRMqeltyUTSgoiZU6zbyQTSgpS9ip5OmZMKV98Jvml\npCBlrZgWQ5PCy8eCcqVOU1KlrGk6piTK9oriUqYpqSJoOqZIppQUpKxpOqbke0G5UqekIGVN0zGl\noSHoMop1G8XuKykkp6QgZU3TMUUyo6QgZS/b6Zia0lo+8rGgXKnTKqki3dAKo+VFXUapRdpSMLOz\nzOxVM9toZvOSPH6zma0Kb+vN7L0o4xHJ1Pz5HTdngeB4/vzCxCMStchaCmZWBdwOnAG0AE1mtszd\n18bquPuVCfW/DZwQVTwiPaEprVJpomwpTAY2uvvr7r4LWArUdVN/JnBfhPGIZExTWqXSRJkUjgTe\nTDhuCcv2YWbVwHDgiQjjEcmYprRKpSmW2UczgAfdfU+yB81stpmtNLOVra2teQ5NKpmmtEqiShio\njjIpbAGOSjgeGpYlM4Nuuo7cfZG7T3T3iYMHD85hiJKOSp+SqRVGJea66wodQfSinJLaBIwws+EE\nyWAGcGHnSmY2EhgIPBNhLNJDmpIpUllSthTMrJ+Z9QrvH2tm082sd6rnuXs7MAd4FFgH3O/ua8zs\nejObnlB1BrDUS2251gqhKZnZq/SWVqmrtLWTUi6dbWbPA58m+Gv+aYIWwC53L8jfiVo6O7969Uq+\n1LBZ0J0i3evc0oJgoFrjEqWplJfezuXS2ebubcD5wB3u/iVgdLYBSmnQlMzsqKUlpSatpGBmU4B6\n4D/DsqroQpJikospmZXcfaKL38pLJaydlE5S+A5wNfDrcEzgaODJaMOSYpHtlMxK3w5TLa3yUq7j\nCIky2o4zHHA+yN23RxdS9zSmUFoqfTtMjSlIscjZmIKZ/dLM+ptZP+BlYK2Z/VMugpTyV+ndJ7r4\nTUpNOt1HtWHL4DzgNwTLUVwUaVRSNtR9oovfpLSkkxR6h9clnAcsc/fdQIlOypJ809pBIqUlnaTw\nf4BmoB/wh3DxuoKNKUhpUfeJSGnJaKA5/iSz/cIrlvNOA80iIpnL5UDzADP7cWyVUjP7EUGrQURE\nykw63UeLgfeBL4e37cDdUQYlIiKFkc4qqce4+wUJx9eZ2aqoAhIRkcJJp6Www8w+FTsws6nAjuhC\nEhFJrhKuKC60dJLCZcDtZtZsZpuAnwLfijYsEZF9VcImN4WWsvvI3VcB48ysf3is6agiImWqy6Rg\nZv/YRTkA7v7jiGISEYlraOjYQohtdrNggbqTotBdS+HgvEUhItKFhoaPTv6lvMlNqegyKbi7eu9E\nRCpMOgPNIgVVyZv0SEeVsMlNoaVznYJIwXTejyC2SQ9o/aRKpDGE6KmlIEVNexyL5FfKloKZHQBc\nANQk1nf366MLSyRQ6Zv0iORbOi2F/wDqgHbgvxNuIpHTJj0i+ZXOmMJQdz8r8khEkli4MPkex9qk\nRyQa6bQU/mRmY3ry4mZ2lpm9amYbzWxeF3W+bGZrzWyNmf2yJ+8j5Uub9IjkV8pNdsxsLfBx4A3g\nQ8AAd/exKZ5XBawHzgBagCZgpruvTagzArgfOM3d3zWzj7n7O929rjbZERHJXLqb7KTTffS5HsYw\nGdjo7q+HAS0lGJtYm1DnUuB2d38XIFVCEBGRaKXsPnL3TcAhwLnh7ZCwLJUjgTcTjlvCskTHAsea\n2dNm9qyZJR27MLPZsZ3fWltb03hrERHpiXS245wLNAIfC2/3mtm3c/T++wEjgFOAmcCdZnZI50ru\nvsjdJ7r7xMGDB+forUVEpLN0uo++Dpzo7v8NYGY3As8At6V43hbgqITjoWFZohbgOXffDbxhZusJ\nkkRTGnGJiEiOpTP7yIA9Ccd7wrJUmoARZjbczPYHZgDLOtV5mKCVgJkdRtCd9Hoary0iIhFIJync\nDTxnZg1m1gA8C9yV6knu3g7MAR4F1gH3u/saM7vezKaH1R4FtoUznJ4E/sndt/Xg3yEiJUBrFxW/\nlFNSAcxsPBDbp/mP7v5fkUbVDU1JFSld2g+hcLKekmpm/d19u5kNAprDW+yxQe7+t1wEKiIixaO7\n7qPY1cXPAysTbrFjEZGUGhqCFkJsG83YfXUlFae0uo+KibqPREqXuo8KJ93uo3SuU/h9OmUiIlL6\nuhtT6AP0BQ4zs4F8NA21P/temSwikpK20yx+3V289k3gO8ARBOMIsaSwHfhpxHGJSBnSOELx6zIp\nuPstwC1m9m13T3X1soiIlIGUy1y4+21mdhxQC/RJKP9FlIGJiEj+pbNH8wKCpShqgUcIltJeASgp\niIiUmXSWufgicDrwF3e/BBgHDIg0qjLT2Ag1NdCrV/CzsbHQEYn0jMYEyl86SWGHu+8F2s2sP/AO\nHVc/lW40NgZ7DG/aFMzP3rQpOFZikFJ03XWFjkCilk5SWBnucXAnwSykFwiWzpY0zJ/fcdN5CI7n\nzy9MPCIi3Uln57XL3f09d/8ZwX7LF4fdSJKGzZszKxcpNlqmorJ0ucxFuDJql9z9hUgiSqHUlrmo\nqQm6jDqrrobm5nxHI5IdLVNRunKxzMWPwtvtwHPAIoIupOfCMknDwoXQt2/Hsr59g3KRfNNf95JK\nl0nB3U9191OBt4Hx4R7JE4AT2HdbTelCfT0sWhS0DMyCn4sWBeUi+ZbtQLGWqSh/KVdJNbM17j46\nVVm+lFr3kUgxUfdP5crZKqnAS2b2czM7JbzdCbyUfYgikg8aKJZMpNNS6ANcBnwmLPoD8K/uvjPi\n2JJSS0Gk59RSqFxZb8cZE578bw5vIiJSxrrsPjKz+8Ofq83spc63/IUo2dIyG+Uj2y4fDRRLKt1d\npzDE3d82s+pkj7t7ktn30VP3UWZiy2wkXlXdt69mQJUqdf9IT6XbfaQ9msucLp4rL0oK0lNZzz4y\ns/fNbHuS2/tmtj3NIM4ys1fNbKOZzUvy+CwzazWzVeHtG+m8rqRPy2yUPs0eknyKrKVgZlXAeoL1\nklqAJmCmu69NqDMLmOjuc9J9XbUUMqOWQnlRS0F6KpfXKcRe8GNmNix2S+Mpk4GN7v66u+8ClgJ1\n6b6f5IaW2RCRTKRMCmY23cw2AG8Ay4Fm4DdpvPaRwJsJxy1hWWcXhDOaHjSzpPs0mNlsM1tpZitb\nW1vTeGuJ0TIb5UWzhyRq6bQUbgBOAta7+3CCXdiezdH7/1+gxt3HAr8D7klWyd0XhWsvTRw8eHCO\n3rpy1NcHXUV79wY/lRBKl8YRJGrpJIXd7r4N6GVmvdz9SSBlvxTBonmJf/kPpdNCeu6+zd0/DA9/\nDkxI43VFRCQiKa9oBt4zs4MIlrdoNLN3gP9O43lNwAgzG06QDGYAFyZWiF0LER5OB9alHbmIiORc\nOkmhDtgJXAnUAwOA61M9yd3bzWwO8ChQBSx29zVmdj2w0t2XAf9gZtOBduBvwKwe/StERCQnurui\n+Xbgl+7+dH5D6p6mpIqIZC4XU1LXAz80s2Yz+4GZnZC78EREpBh1t/PaLe4+BTgZ2AYsNrNXzGyB\nmR2btwhFRCRvUs4+cvdN7n6ju58AzATOQwPCIiJlKZ2L1/Yzs3PNrJHgorVXgfMjj0xERPKuy9lH\nZnYGQcvgbODPBMtUzHb3dKajiohICepuSurVwC+B77r7u3mKR0RECqjLpODup+UzEBERKby0V0kV\nkexo3SIpBUoKInly3XWFjkAkNSUFERGJU1IQiZC20pRSE9l2nFHR2kdSqrSVphRSzrfjFBGR8qek\nUAIaG6GmBnr1Cn42NhY6IukJbaUppSCd/RSkgBobYfZsaGsLjjdtCo5B22qWGo0jSClQS6HIzZ//\nUUKIaWsLyiW/dFKXSqCkUOQ2b86sXLqW7Uld1xlIJVBSKHLDhmVWLl3TSV0kNSWFIrdwIfTt27Gs\nb9+gXKKn6wyk0lREUijl2Tv19bBoEVRXByej6urgWIPM6cn2pN7QEFxbELu+IHZfSUHKVdlfvNZ5\n9g4Ef2nrxFp5sr14TBefSSnTxWshzd6RXNF1BlIJyj4paPaOxGR7UleXkVSCSJOCmZ1lZq+a2UYz\nm9dNvQvMzM0sZdMmU5q9IzE6qYukFllSMLMq4Hbgc0AtMNPMapPUOxiYCzwXRRyavVM8dFIWKX5R\nthQmAxvd/XV33wUsBeqS1LsBuBHYGUUQmr1TPHSdgEjxizIpHAm8mXDcEpbFmdl44Ch3/8/uXsjM\nZpvZSjNb2dramnEg9fXQ3Ax79wY/lRBERJIr2ECzmfUCfgx8N1Vdd1/k7hPdfeLgwYOjD05yRhd/\niZSWKJPCFuCohOOhYVnMwcBxwFNm1gycBCyLYrBZCkcXf4mUliiTQhMwwsyGm9n+wAxgWexBd/+7\nux/m7jXuXgM8C0x3d22rJiJSIJElBXdvB+YAjwLrgPvdfY2ZXW9m06N6Xyleuk5ApPiV/TIXUj60\nzIRIz2mZCxERyZiSghQ1zV4SyS91H0nJUPeRSM+p+0hERDKmpCAlQ0tXi0RPSUFKhsYRRKKnpCAi\nInFKCiIiEqekICIicUoKIiISV1FJQQOVIiLdq6ikoJ2/RES6V1FJQUREulf2SUFr5+SOPjOR8ldR\nax9p7Zzs6PMTKV1a+0hERDJWUUlBa+dkTt1vIpWlorqPJDvqPhIpXeo+EhGRjCkpSNrU/SZS/pQU\nKki24wAaRxApf0oKFURXdItIKkoKIiISF2lSMLOzzOxVM9toZvOSPP4tM1ttZqvMbIWZ1UYZTyXS\nlFIRyURkU1LNrApYD5wBtABNwEx3X5tQp7+7bw/vTwcud/ezuntdTUntOU0pFalcxTAldTKw0d1f\nd/ddwFKgLrFCLCGE+gE6ZYmIFFCUSeFI4M2E45awrAMzu8LMXgN+APxDhPGUvGy7fDSlVERSKfhA\ns7vf7u7HAFcB1ySrY2azzWylma1sbW3Nb4BFJNvZQxpHEJFUokwKW4CjEo6HhmVdWQqcl+wBd1/k\n7hPdfeLgwYNzGGJmdFIVkXIXZVJoAkaY2XAz2x+YASxLrGBmIxIOPw9siDCerBXiL3XNHhKRfIp0\nQTwzOxv4CVAFLHb3hWZ2PbDS3ZeZ2S3AZ4HdwLvAHHdf091rFnL2Ubazdwr9fBGpXMUw+wh3f8Td\nj3X3Y9x9YVh2rbsvC+/PdffR7n68u5+aKiEUgv5SF5FKUvCB5mLX0AD33gvV1cFxdXVwnG5SyGVS\n0ewhEYma9lNIobERZs+GtraPyvr2hUWLoL4+s9dS94+IFEpRdB+Vg/nzOyYECI7nzy9MPCIiUVJS\nSGHz5szKu6PuHxEpdkoKKQwblll5dzQ4LSLFTkkhhYULgzGERH37BuUiIuVGSSGF+vpgULm6Ohgo\nrq7u2SCziEgp2K/QAZSC+nolARGpDGopiIhInJKCiIjEKSmIiEickoKIiMQpKYiISFzJrX1kZq3A\npkLH0YXDgK2FDqIbii87xR4fFH+Mii872cRX7e4pdykruaRQzMxsZToLThWK4stOsccHxR+j4stO\nPuJT95GIiMQpKYiISJySQm4tKnQAKSi+7BR7fFD8MSq+7EQen8YUREQkTi0FERGJU1LIkJkdZWZP\nmtlaM1tjZnOT1DnFzP5uZqvC27V5jrHZzFaH773P3qUWuNXMNprZS2Y2Po+xfSLhc1llZtvN7Dud\n6uT98zOzxWb2jpm9nFA2yMx+Z2Ybwp8Du3juxWGdDWZ2cZ5iu8nMXgl/f782s0O6eG6334WIY2ww\nsy0Jv8ezu3juWWb2avh9nJfH+P49IbZmM1vVxXMj/Qy7OqcU7Pvn7rplcAOGAOPD+wcD64HaTnVO\nAf5fAWNsBg7r5vGzgd8ABpwEPFegOKuAvxDMny7o5wd8BhgPvJxQ9gNgXnh/HnBjkucNAl4Pfw4M\n7w/MQ2zTgP3C+zcmiy2d70LEMTYA/zON78BrwNHA/sCLnf8/RRVfp8d/BFxbiM+wq3NKob5/ailk\nyN3fdvfb1wHWAAAE8UlEQVQXwvvvA+uAIwsbVcbqgF944FngEDMbUoA4Tgdec/eCX4zo7n8A/tap\nuA64J7x/D3BekqeeCfzO3f/m7u8CvwPOijo2d3/M3dvDw2eBobl8z0x18fmlYzKw0d1fd/ddwFKC\nzz2nuovPzAz4MnBfrt83Hd2cUwry/VNSyIKZ1QAnAM8leXiKmb1oZr8xs9F5DQwceMzMnjez2Uke\nPxJ4M+G4hcIkthl0/R+xkJ9fzOHu/nZ4/y/A4UnqFMNn+TWCll8yqb4LUZsTdnEt7qL7oxg+v08D\nf3X3DV08nrfPsNM5pSDfPyWFHjKzg4CHgO+4+/ZOD79A0CUyDrgNeDjP4X3K3ccDnwOuMLPP5Pn9\nUzKz/YHpwANJHi7057cPD9rqRTdVz8zmA+1AYxdVCvld+FfgGOB44G2CLppiNJPuWwl5+Qy7O6fk\n8/unpNADZtab4JfX6O6/6vy4u2939w/C+48Avc3ssHzF5+5bwp/vAL8maKIn2gIclXA8NCzLp88B\nL7j7Xzs/UOjPL8FfY91q4c93ktQp2GdpZrOAc4D68KSxjzS+C5Fx97+6+x533wvc2cV7F/S7aGb7\nAecD/95VnXx8hl2cUwry/VNSyFDY/3gXsM7df9xFnf8R1sPMJhN8ztvyFF8/Mzs4dp9gQPLlTtWW\nAV8NZyGdBPw9oZmaL13+dVbIz6+TZUBsNsfFwH8kqfMoMM3MBobdI9PCskiZ2VnA/wKmu3tbF3XS\n+S5EGWPiONUXunjvJmCEmQ0PW48zCD73fPks8Iq7tyR7MB+fYTfnlMJ8/6IaUS/XG/ApgmbcS8Cq\n8HY28C3gW2GdOcAagpkUzwKfzGN8R4fv+2IYw/ywPDE+A24nmPWxGpiY58+wH8FJfkBCWUE/P4IE\n9Tawm6Bf9uvAocDvgQ3A48CgsO5E4OcJz/0asDG8XZKn2DYS9CXHvoM/C+seATzS3Xchj5/fv4Xf\nr5cITnBDOscYHp9NMOPmtahiTBZfWL4k9r1LqJvXz7Cbc0pBvn+6ollEROLUfSQiInFKCiIiEqek\nICIicUoKIiISp6QgIiJxSgoiITPbYx1XcM3Zip1mVpO4QqdIsdqv0AGIFJEd7n58oYMQKSS1FERS\nCNfT/0G4pv6fzezjYXmNmT0RLvj2ezMbFpYfbsEeBy+Gt0+GL1VlZneGa+Y/ZmYHhvX/IVxL/yUz\nW1qgf6YIoKQgkujATt1HX0l47O/uPgb4KfCTsOw24B53H0uwIN2tYfmtwHIPFvQbT3AlLMAI4HZ3\nHw28B1wQls8DTghf51tR/eNE0qErmkVCZvaBux+UpLwZOM3dXw8XLvuLux9qZlsJlm7YHZa/7e6H\nmVkrMNTdP0x4jRqCde9HhMdXAb3d/X+b2W+BDwhWg33Yw8UARQpBLQWR9HgX9zPxYcL9PXw0pvd5\ngrWoxgNN4cqdIgWhpCCSnq8k/HwmvP8nglU9AeqBP4b3fw9cBmBmVWY2oKsXNbNewFHu/iRwFTAA\n2Ke1IpIv+otE5CMHWsfN23/r7rFpqQPN7CWCv/ZnhmXfBu42s38CWoFLwvK5wCIz+zpBi+AyghU6\nk6kC7g0ThwG3uvt7OfsXiWRIYwoiKYRjChPdfWuhYxGJmrqPREQkTi0FERGJU0tBRETilBRERCRO\nSUFEROKUFEREJE5JQURE4pQUREQk7v8DxBgqjYrXWkoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43c0fbcba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bigger_model_val_loss = bigger_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The bigger network starts overfitting almost right away, after just one epoch, and overfits much more severely. Its validation loss is also \n",
    "more noisy.\n",
    "\n",
    "Meanwhile, here are the training losses for our two networks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucFPWZ7/HPw4giiorKy4MiM2gwAnIRBpCg8YLBSyJ4\nSxQ5RtRILnKWNdlEDK6MuuSmG+MFT5asqAkTTdQ18ZwYNd5IMBoZzSBCUEAHBE0EoiI7Erk8+0fV\nlD3jzHRNd1dX98z3/Xr1q7uqf9X9TE1PP/OrX/2eMndHREQEoFvaAYiISOlQUhARkYiSgoiIRJQU\nREQkoqQgIiIRJQUREYkoKYiISERJQUREIkoKIiIS2S3tADrqwAMP9KqqqrTDEBEpKy+88MImd++T\nrV3ZJYWqqirq6urSDkNEpKyY2do47XT4SEREIkoKIiISUVIQEZFI2Y0piEjxbN++nfXr17Nt27a0\nQ5GYevToQb9+/ejevXtO2yspiEib1q9fT69evaiqqsLM0g5HsnB3Nm/ezPr16xkwYEBOr9GlDh/V\n1KQdgUh52bZtGwcccIASQpkwMw444IC8enZdKilce23aEYiUHyWE8pLv76tLJQUREWlfp08KNTVg\nFtzgo8c6lCRSHtavX8/kyZMZOHAghx9+ODNnzuTDDz9ste2bb77Jueeem/U1Tz/9dN59992c4qmp\nqeHGG2/Madu47rrrLmbMmJF3m1x0+qQwcCD07Nl8Xc+ewXoRSUah/ulyd84++2zOPPNMVq1axauv\nvsrWrVuZPXv2x9ru2LGDgw8+mPvvvz/r6z788MPst99+hQmyk+n0SWH2bGhsbL6usTFYLyLJKNT4\n3ZNPPkmPHj24+OKLAaioqOCmm25iwYIFNDY2ctdddzFp0iROOukkJkyYQENDA0cddRQAjY2NfOEL\nX2Dw4MGcddZZjB07NiqRU1VVxaZNm2hoaGDQoEFcdtllDBkyhIkTJ/LBBx8A8JOf/ITRo0czfPhw\nzjnnHBpbfpG0MG3aNL761a9yzDHHcNhhh/H0009zySWXMGjQIKZNmxa1u+eeexg6dChHHXUUV155\nZbT+zjvv5IgjjmDMmDE888wz0fqNGzdyzjnnMHr0aEaPHt3suSR0+qSwbl3H1otI6Vi+fDmjRo1q\ntm6fffahf//+rF69GoAXX3yR+++/n0WLFjVrd/vtt9O7d29WrFjB9ddfzwsvvNDqe6xatYrLL7+c\n5cuXs99++/HAAw8AcPbZZ7NkyRKWLl3KoEGDuOOOO7LG+8477/Dss89y0003MWnSJK644gqWL1/O\nsmXLqK+v58033+TKK6/kySefpL6+niVLlvCrX/2Kt956izlz5vDMM8+wePFiVqxYEb3mzJkzueKK\nK1iyZAkPPPAAX/rSlzq0Dzuq089T6N8f1rZSBqp//+LHItKZ1dQ07yE0jePNmZPsGN5nPvMZ9t9/\n/4+tX7x4MTNnzgTgqKOOYtiwYa1uP2DAAEaMGAHAqFGjaGhoAODll1/m6quv5t1332Xr1q2ccsop\nWWM544wzMDOGDh3KQQcdxNChQwEYMmQIDQ0NrF27lhNOOIE+fYJipVOnTuX3v/89QLP15513Hq++\n+ioAjz/+eLMksWXLFrZu3Zo1llx1+p7C3LmtjynMnZtOPCKdVU0NuAc3+OhxPglh8ODBH/sPf8uW\nLaxbt45PfOITAOy11165vwGwxx57RI8rKirYsWMHEBwOuu2221i2bBlz5syJde5/02t169at2et2\n69Ytet2O2rVrF8899xz19fXU19ezYcMG9t5775xeK45OnxSmToX586GyMvjPpbIyWJ46Ne3IRCSb\nCRMm0NjYyE9/+lMAdu7cyTe+8Q2mTZtGz5b/7bUwfvx4fvnLXwKwYsUKli1b1qH3fv/99+nbty/b\nt2+ntrY2tx+ghTFjxrBo0SI2bdrEzp07ueeeezj++OMZO3YsixYtYvPmzWzfvp377rsv2mbixInc\neuut0XJ9fX1BYmlLp08KECSAhgbYtSu4V0IQSdacOYV5HTPjwQcf5L777mPgwIEcccQR9OjRg+98\n5ztZt/3a177Gxo0bGTx4MFdffTVDhgxh3333jf3e119/PWPHjmX8+PEceeSR+fwYkb59+/K9732P\nE088keHDhzNq1CgmT55M3759qampYdy4cYwfP55BgwZF29xyyy3U1dUxbNgwBg8ezI9//OOCxNIW\n86a+Xpmorq52XWRHpDj+8pe/NPuCKic7d+5k+/bt9OjRgzVr1nDyySfzyiuvsPvuu6cdWuJa+72Z\n2QvuXp1t204/0CwiXVNjYyMnnngi27dvx925/fbbu0RCyJeSgoh0Sr169dKle3PQJcYUREQkHiUF\nERGJKCmIiEhESUFERCJKCiJS0ioqKhgxYgTDhw9n5MiR/PGPfwTil8kuZXFmJic5e7k1SgoiUjC1\ntVBVBd26BfeFmAi85557Ul9fz9KlS/nud7/LVVddBRC7THY+ci1NUc6UFESkIGprYfr0oACle3A/\nfXphEkOTLVu20Lt3b4DYZbLvuOOOqCT1ZZddFl2Ypq2S1DU1NVx44YWMHz+eCy+8sNn7P/300xx/\n/PFMnjyZww47jFmzZlFbW8uYMWMYOnQoa9asiWI76aSTGDZsGBMmTGBdWJb59ddfZ9y4cQwdOpSr\nr7662WvfcMMNjB49mmHDhjGnUFPCc+HuZXUbNWqUi0hxrFixInbbysqmEnjNb5WV+cXQrVs3Hz58\nuH/yk5/0ffbZx+vq6tzd/fXXX/chQ4a4u/sNN9zg06dPd3f3ZcuWeUVFhS9ZssQ3bNjglZWVvnnz\nZv/www/92GOP9csvv9zd3adMmeJ/+MMf3N197dq1fuSRR7q7+5w5c3zkyJHe2Nj4sVieeuop33ff\nff3NN9/0bdu2+cEHH+zXXHONu7v/6Ec/8pkzZ7q7++c+9zm/66673N39jjvu8MmTJ7u7+xlnnOF3\n3323u7vfdtttvtdee7m7+6OPPuqXXXaZ79q1y3fu3Omf/exnfdGiRe7uUZuOaO33BtR5jO9Y9RRE\npCCSunZJ0+GjlStX8sgjj/DFL34Rb1GeZ/HixZx//vlA8zLZzz//PMcffzz7778/3bt35/Of/3y0\nzeOPP86MGTMYMWIEkyZNalaSetKkSey5556txjN69Gj69u3LHnvsweGHH87EiRMBGDp0aFR2+9ln\nn+WCCy4A4MILL2Tx4sUAPPPMM0yZMiVa3+Sxxx7jscce4+ijj2bkyJGsXLmSVatW5bXfcqUZzSJS\nEMW4dsm4cePYtGkTGzduzPu1mkpS9+jR42PPtVeOu2VJ7Mxy2XHGIKzpQhMZ3J2rrrqKL3/5y3FC\nT5R6CiJSEMW4dsnKlSvZuXMnBxxwQLP1bZXJHj16NIsWLeKdd95hx44d0VXVINmS1J/61Ke49957\nAaitreW4446L4sxc3+SUU05hwYIFUU9lw4YNvP322wWLpyMSTQpmdqqZvWJmq81sVjvtzjEzN7Os\nFfxEpDQlde2SDz74gBEjRjBixAjOO+887r77bioqKpq1aatM9iGHHMK3v/1txowZw/jx46mqqorK\nZydZkvrWW2/lzjvvZNiwYfzsZz/j5ptvBuDmm29m3rx5DB06lA0bNkTtJ06cyAUXXBANQp977rm8\n//77BYunIxIrnW1mFcCrwGeA9cASYIq7r2jRrhfwG2B3YIa7t1vBSqWzRYqnXEpnt1cme+vWrey9\n997s2LGDs846i0suuYSzzjor7ZATVaqls8cAq939tTCge4HJwIoW7a4Hvg98M8FYRKQTa69Mdk1N\nDY8//jjbtm1j4sSJnHnmmSlHW9qSTAqHAG9kLK8HxmY2MLORwKHu/hszU1IQkZy0Vyb7xhtvLHI0\n5S21gWYz6wb8EPhGjLbTzazOzOoKcdaBiMSX1CFmSUa+v68kk8IG4NCM5X7huia9gKOAp82sATgG\neKi1wWZ3n+/u1e5e3adPnwRDFpFMPXr0YPPmzUoMZcLd2bx5c6un2caV5OGjJcBAMxtAkAzOBy5o\netLd3wMObFo2s6eBf8k20CwixdOvXz/Wr19fkHkBUhw9evSgX79+OW+fWFJw9x1mNgN4FKgAFrj7\ncjO7jmC69UNJvbeIFEb37t0ZMGBA2mFIESU6o9ndHwYebrHumjbanpBkLCIikp1mNIuISERJQURE\nIkoKIiISUVIQEZGIkoKIiESUFEREJKKkICIiESUFERGJKCmIiEhESUFERCJKCiIiElFSEBGRiJKC\niIhElBRERCSipCAiIhElBRERiSgpiIhIRElBREQiSgoiIhJRUhARkYiSgoiIRJQUREQkoqQgIiIR\nJQUREYkoKYiISERJQUREIkoKIiISUVIQEZGIkoKIiESUFEREJKKkICIiESUFERGJKCmIiEhESUFE\nRCKJJgUzO9XMXjGz1WY2q5Xnv2Jmy8ys3swWm9ngJOMREZH2JZYUzKwCmAecBgwGprTypf9zdx/q\n7iOAHwA/TCoeERHJLsmewhhgtbu/5u4fAvcCkzMbuPuWjMW9AE8wHhERyWK3jjQ2MwN6uvt/x2h+\nCPBGxvJ6YGwrr3k58HVgd+CkjsQjIiKFlbWnYGY/NbN9zKwnsAxYbWZfL1QA7j7P3Q8HrgSubiOG\n6WZWZ2Z1GzduLNRbi4hIC3EOHw0LD/OcCfwOqASmxdhuA3BoxnK/cF1b7g3f42Pcfb67V7t7dZ8+\nfWK8tYiI5CJOUuhuZrsRjAf8Ohwf2BVjuyXAQDMbYGa7A+cDD2U2MLOBGYufBVbFC1tERJIQZ0zh\nP4F1wMvAIjPrD2zNtpG77zCzGcCjQAWwwN2Xm9l1QJ27PwTMMLOTge3AO8BFOf4cIiJSAObesRN+\nwsHm7mGPoeiqq6u9rq4ujbcWESlbZvaCu1dnaxdnoHmGme0TPv4P4E/AcfmHKCIipSbOmMJ0d99i\nZhOBg4DLCCaaiYhIJxMnKTQdXzod+Jm7L425nYiIlJk4X+5Lzexh4HPAb81sbzTzWESkU4pz9tHF\nwCiCkhWNZnYgcGmyYYmISBqyJgV33xkmgrODE49Y5O6/TTwyEREpujhnH80FvgW8Ft6+aWb/lnRg\nIiJSfHHGFM4ATg5LTcwHJgKTkg2rNNXUpB2BiEiy4p5F1KuNx13KtdemHYGISLLiDDT/AHjRzJ4A\nDDgB+NckgxIRkXRk7Sm4+0LgWOBh4DfAp93950kHVipqasAsuMFHj3UoSUQ6ozZrH5nZsPY2dPeX\nEokoizRrH5lBB0tFiYiUhLi1j9o7fDSvnecc+HSHoxIRkZLWZlJwdxW9a2HOnLQjEBFJlmoYdYDG\nEUSks1NSEBGRiJKCiIhEss5TaOMspPeAN9w9zrWaRUSkTMSZvHYHMAJYTjB5bRCwAuhlZtPd/YkE\n4xMRkSKKc/ioARjl7iPcfThBGe1XgVOAf08wNhERKbI4SWFQ5kQ1d18GDHb31cmFJSIiaYhz+Gil\nmd0K3Bsunxeu2wPYkVhkIiJSdHF6Cl8E1gOzwtubwEUECWFCcqGJiEixxbnyWiPw/fDW0nsFj0hE\nRFIT55TUY4A5QGVme3c/IsG4REQkBXHGFO4kuBznC8DOZMMREZE0xUkKW9z9/yUeiYiIpC5OUnjS\nzL4L/Bfwj6aVaV1PQUREkhMnKRzb4h50PQURkU4pzuU4j2vl1qUSQm0tVFVBt27BfW1t2hGJiCSj\nzZ6CmU1x93vM7J9ae97db0kurNJRWwvTp0NjY7C8dm2wDDB1anpxiYgkob2eQu/wvk8bty5h9uyP\nEkKTxsZgvYhIZ9Pe5ThvD+//tXjhlJ516zq2vj01Nbp6m4iUtqxjCmZ2oJl9y8xuN7P5TbdiBFcK\n+vfv2Pr2XHttfrGIiCQtTu2jXwMHAYuBJzJuWZnZqWb2ipmtNrNZrTz/dTNbYWYvmdkTZlbZkeCL\nYe5c6Nmz+bqePYP1IiKdTZyksJe7f8Pdf+7uv2i6ZdvIzCqAecBpwGBgipkNbtHsz0C1uw8D7gd+\n0MH4Ezd1KsyfD5WVYBbcz58ff5C5pibYzixYbnqsw0giUorM3dtvEExce8rdH+vQC5uNA2rc/ZRw\n+SoAd/9uG+2PBm5z9/HtvW51dbXX1dV1JJSSYQZZdreISCLM7AV3r87WLk5P4SvAI2a21cz+bmbv\nmNnfY2x3CPBGxvL6cF1bLgV+G+N1RUQkIXFmNB+YdBBm9r+BauD4Np6fDkwH6J/LCG+JmDMn7QhE\nRNrX3uS1ge6+ChjSRpNstY82AIdmLPcL17V8n5OB2cDx7v6Pls8DuPt8YD4Eh4+yvG/J0jiCiJS6\n9noKswgO6cxr5bk4tY+WAAPNbABBMjgfuCCzQTiO8B/Aqe7+dtygRUQkGe1NXrs0vD8ulxd29x1m\nNgN4FKgAFrj7cjO7Dqhz94eAG4C9gfssOD1nnbtPyuX9REQkf3HGFDCzIwlOK+3RtM7df55tO3d/\nGHi4xbprMh6fHDtSERFJXJzLcV4NTASOJPiv/xSCiWxZk4KIiJSXOKekngecCLzl7hcCw4G9Eo1K\nRERSEScpfODuO4EdZtYL+CtQcuUoREQkf3HGFP5sZvsBC4A6YAvwfKJRiYhIKtpNChacElTj7u8C\n88zsUWAfd3+xKNGJiEhRtXv4yIPCSL/LWF6thJAeTX4TkaTFGVOoDyeZScp0PQYRSVp7ZS52c/cd\nwNHAEjNbA/w3YASdiJFFilFERIqkvZ5C02DyJOCTwOnA54Fzw3spAl2PQUSKqc3rKZjZn9295A4b\nlfP1FPKl6zGISK7iXk+hvbOP+pjZ19t60t1/mFNkIiJSstpLChUExeqsSLFIFroeg4gkrb2k8Ja7\nX1e0SCQrjSOISNLaG2hWD0FEpItpLylMKFoUUhTqaYhINm0mBXf/ezEDkeRp8puIZBNnRrOIiHQR\nSgqdnCa/iUhHtDl5rVR15clr+dLkN5GuK+7kNfUUREQkoqTQhWjym4hko6TQhWgcQUSyUVIQEZGI\nkoLEpp6GSOenpCCxafKbSOenpCAiIhElhSKorYWqKujWLbivrU07ovg0+U2ka9HktYTV1sL06dDY\n+NG6nj1h/nyYOjW9uHKhyW8i5UuT10rE7NnNEwIEy7NnpxOPiEh7lBQStm5dx9aXMk1+E+n8lBQS\n1r9/x9aXsnzHETQOIVL6lBQSNnduMIaQqWfPYH1Xo1NaRUqfkkLCpk4NBpUrK4OB2srK8hxkFpGu\nQUmhCKZOhYYG2LUruO9KCUGntIqUl0STgpmdamavmNlqM5vVyvOfNrMXzWyHmZ2bZCySjpoaWLgw\n6CFBcL9woZKCSKlKLCmYWQUwDzgNGAxMMbPBLZqtA6YBP08qDklX0zyNtWuD5bVrg+VcJvApkYgk\nL8mewhhgtbu/5u4fAvcCkzMbuHuDu78E7EowDklRIedpaKBaJHlJJoVDgDcylteH66QL6UzzNES6\ngrIYaDaz6WZWZ2Z1GzduTDsc6YB852looFqkuJJMChuAQzOW+4XrOszd57t7tbtX9+nTpyDBSXHk\nO0+jpiaot9RUc6npsZKCSDKSTApLgIFmNsDMdgfOBx5K8P2kBJXSPA0lEpHsEq2SamanAz8CKoAF\n7j7XzK4D6tz9ITMbDTwI9Aa2AX919yHtvWa5VUmVwqmpye+LXVVepSuLWyVVpbOly1BSkK5MpbNF\n0EC1SEcpKUinVsiBaiUS6QqUFERi0uQ56QqUFKTkFeoa17pIkEh2SgpS0jJrJ7kXv3aSxiSkq9HZ\nR1LSqqo+KqaXqbIyKENeTPmevZTvKbUi+dDZR9IpdKbaSRqTkHKgpNAFFOqYfBpK6RrXGpOQrkBJ\noZMr5DH5NJTSNa7LeUyinP8xkCJz97K6jRo1yiW+ysqmM/Ob3yor044svoULg3jNgvuFC9OOKDeQ\n/2vMmdPxbRYudO/Zs/nvv2fP8t2PkhuC8kJZv2M10NzJdevW+uCoWXDNaElebW1wUaG1a4MB8rlz\ncy8ImMtgdykN1kt6NNDcieTT9S+lY/JdUSEvR5qrzjRYL8lTUihx+Y4JlNIx+a6oEJcjzXdcQv8Y\nSEcoKZS4fL9USul6BuUqn55aIf5Lz7d+k/4xkI7QmEKJ05hAupp6apmJuWfP+Im10Mfzc51AV8hx\nDSlPGlPoJNT1T1e+PbVC/5ee61yJqVM/SkINDUoI0jYlhRKnrn+68j38U+jDd2nPlUirTIfmWRSP\nDh+Vgaau/7p1QQ9BXf/i6Wync+ZbvymNq9flewhPArocp0gBdLYvpHJMCp0tMadFYwoiBdDZzt7K\nZUwi7cNPmmdRXOopiEhsafQ01FMoDPUURKRT0MkWxaWkICKxpXH4KfMQHpTnIbyyOnsqTtW8Urqp\nSqpI8RWqUm2+lWJz3T7NSrulUqUWVUkVkUIo5BlYaYxJpH0GWamMiWhMQUQKohBF/ZqkcfipkPHn\notzOnlJPQUTaVUr1t3LpKRQy/pqajp9Wq56CiHQq5V5/a599Ora+Pdde2/Ftyu3sKSUFEWlXKX2p\n5XL4ad681uOfN68wMWVTbmdPKSmISLtKaVZ3LjOi8/1SLsSM7kJVqS1GQUKNKYhIl5HLmECmtGtH\n5bO9xhRERFpIo/R3IWtHFYOSgohITLmeUpvP5VSLnVR0+EhEpEi6/OEjMzvVzF4xs9VmNquV5/cw\ns1+Ez//JzKqSjEdyU1Z1W0RKWK6XU236G4Tk/wYTSwpmVgHMA04DBgNTzGxwi2aXAu+4+yeAm4Dv\nJxWP5KapRMDatcF/KGvXBsvllBg6Q1Ir959B8QdyOeST+TcIRfgbjFMgKZcbMA54NGP5KuCqFm0e\nBcaFj3cDNhEe0mrrpoJ4xVVZ2byQV9OtsjLtyOIplWJk+Sj3n0Hx56dQf4OkXRDPzM4FTnX3L4XL\nFwJj3X1GRpuXwzbrw+U1YZtNbb2uxhSKq5RKHOSiVEoM5KPcfwbFn59C/Q2WxJhCoZjZdDOrM7O6\njRs3ph1Ol1LuJQ7KrRhZa8r9Z1D8+Sn232CSSWEDcGjGcr9wXattzGw3YF9gc8sXcvf57l7t7tV9\n+vRJKFxpTSmVOMhFuSc1KP+fQfHnp9h/g0kmhSXAQDMbYGa7A+cDD7Vo8xBwUfj4XOBJT+p4luSk\nlEoc5KLckxqU/8+g+PNT9L/BOAMPud6A04FXgTXA7HDddcCk8HEP4D5gNfA8cFi219RAs3RUmlfd\nKpRy/xkUf/pIe6A5KRpoFhHpuE410CwiIsWhpCAiIhElBRERiSgpiIhIRElBREQiZXf2kZltBFqZ\ndF4SDiSo31SqFF9+Sj0+KP0YFV9+8omv0t2zzv4tu6RQysysLs4pX2lRfPkp9fig9GNUfPkpRnw6\nfCQiIhElBRERiSgpFNb8tAPIQvHlp9Tjg9KPUfHlJ/H4NKYgIiIR9RRERCSipNBBZnaomT1lZivM\nbLmZzWylzQlm9p6Z1Ye3a4ocY4OZLQvf+2PVAy1wi5mtNrOXzGxkEWP7ZMZ+qTezLWb2zy3aFH3/\nmdkCM3s7vBpg07r9zex3ZrYqvO/dxrYXhW1WmdlFrbVJILYbzGxl+Pt70Mz2a2Pbdj8LCcdYY2Yb\nMn6Pp7ex7alm9kr4eZxVxPh+kRFbg5nVt7Ftovuwre+U1D5/cUqp6tasHHhfYGT4uBdBafDBLdqc\nAPz/FGNsAA5s5/nTgd8CBhwD/CmlOCuAvxKcP53q/gM+DYwEXs5Y9wNgVvh4FvD9VrbbH3gtvO8d\nPu5dhNgmAruFj7/fWmxxPgsJx1gD/EuMz8Aa4DBgd2Bpy7+npOJr8fy/A9eksQ/b+k5J6/OnnkIH\nuftb7v5i+Ph94C/AIelG1WGTgZ964DlgPzPrm0IcE4A17p76ZER3/z3w9xarJwN3h4/vBs5sZdNT\ngN+5+9/d/R3gd8CpScfm7o+5+45w8TmCKxumpo39F8cYYLW7v+buHwL3Euz3gmovPjMz4AvAPYV+\n3zja+U5J5fOnpJAHM6sCjgb+1MrT48xsqZn91syGFDUwcOAxM3vBzKa38vwhwBsZy+tJJ7GdT9t/\niGnuvyYHuftb4eO/Age10qYU9uUlBD2/1mT7LCRtRniIa0Ebhz9KYf8dB/zN3Ve18XzR9mGL75RU\nPn9KCjkys72BB4B/dvctLZ5+keCQyHDgVuBXRQ7vWHcfCZwGXG5mny7y+2cVXqJ1EsGV91pKe/99\njAd99ZI7Vc/MZgM7gNo2mqT5Wfi/wOHACOAtgkM0pWgK7fcSirIP2/tOKebnT0khB2bWneCXV+vu\n/9XyeXff4u5bw8cPA93N7MBixefuG8L7t4EHCbromTYAh2Ys9wvXFdNpwIvu/reWT6S9/zL8remw\nWnj/dittUtuXZjYN+BwwNfzS+JgYn4XEuPvf3H2nu+8CftLGe6f6WTSz3YCzgV+01aYY+7CN75RU\nPn9KCh0UHn+8A/iLu/+wjTb/K2yHmY0h2M+bixTfXmbWq+kxwYDkyy2aPQR8MTwL6RjgvYxuarG0\n+d9ZmvuvhYeAprM5LgJ+3UqbR4GJZtY7PDwyMVyXKDM7FfgWwfXOG9toE+ezkGSMmeNUZ7Xx3kuA\ngWY2IOw9nk+w34vlZGClu69v7cli7MN2vlPS+fwlNaLeWW/AsQTduJeA+vB2OvAV4CthmxnAcoIz\nKZ4DPlXE+A4L33dpGMPscH1mfAbMIzjrYxlQXeR9uBfBl/y+GetS3X8ECeotYDvBcdlLgQOAJ4BV\nwOPA/mHbauA/M7a9BFgd3i4uUmyrCY4lN30Gfxy2PRh4uL3PQhH338/Cz9dLBF9wfVvGGC6fTnDG\nzZqkYmwtvnD9XU2fu4y2Rd2H7XynpPL504xmERGJ6PCRiIhElBRERCSipCAiIhElBRERiSgpiIhI\nRElBJGRmO615BdeCVew0s6rMCp0ipWq3tAMQKSEfuPuItIMQSZN6CiJZhPX0fxDW1H/ezD4Rrq8y\nsyfDgm9PmFn/cP1BFlzjYGl4+1T4UhVm9pOwZv5jZrZn2P6fwlr6L5nZvSn9mCKAkoJIpj1bHD46\nL+O599zjMH0MAAABW0lEQVR9KHAb8KNw3a3A3e4+jKAg3S3h+luARR4U9BtJMBMWYCAwz92HAO8C\n54TrZwFHh6/zlaR+OJE4NKNZJGRmW91971bWNwAnuftrYeGyv7r7AWa2iaB0w/Zw/VvufqCZbQT6\nufs/Ml6jiqDu/cBw+Uqgu7v/m5k9AmwlqAb7Kw+LAYqkQT0FkXi8jccd8Y+Mxzv5aEzvswS1qEYC\nS8LKnSKpUFIQiee8jPtnw8d/JKjqCTAV+EP4+AngqwBmVmFm+7b1ombWDTjU3Z8CrgT2BT7WWxEp\nFv1HIvKRPa35xdsfcfem01J7m9lLBP/tTwnX/R/gTjP7JrARuDhcPxOYb2aXEvQIvkpQobM1FcDC\nMHEYcIu7v1uwn0ikgzSmIJJFOKZQ7e6b0o5FJGk6fCQiIhH1FEREJKKegoiIRJQUREQkoqQgIiIR\nJQUREYkoKYiISERJQUREIv8DVI4KezW+r2IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43befc12e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "original_train_loss = original_hist.history['loss']\n",
    "bigger_model_train_loss = bigger_model_hist.history['loss']\n",
    "\n",
    "plt.plot(epochs, original_train_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Training loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the bigger network gets its training loss near zero very quickly. The more capacity the network has, the quicker it will be \n",
    "able to model the training data (resulting in a low training loss), but the more susceptible it is to overfitting (resulting in a large \n",
    "difference between the training and validation loss)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding weight regularization\n",
    "\n",
    "\n",
    "You may be familiar with _Occam's Razor_ principle: given two explanations for something, the explanation most likely to be correct is the \n",
    "\"simplest\" one, the one that makes the least amount of assumptions. This also applies to the models learned by neural networks: given some \n",
    "training data and a network architecture, there are multiple sets of weights values (multiple _models_) that could explain the data, and \n",
    "simpler models are less likely to overfit than complex ones.\n",
    "\n",
    "A \"simple model\" in this context is a model where the distribution of parameter values has less entropy (or a model with fewer \n",
    "parameters altogether, as we saw in the section above). Thus a common way to mitigate overfitting is to put constraints on the complexity \n",
    "of a network by forcing its weights to only take small values, which makes the distribution of weight values more \"regular\". This is called \n",
    "\"weight regularization\", and it is done by adding to the loss function of the network a _cost_ associated with having large weights. This \n",
    "cost comes in two flavors:\n",
    "\n",
    "* L1 regularization, where the cost added is proportional to the _absolute value of the weights coefficients_ (i.e. to what is called the \n",
    "\"L1 norm\" of the weights).\n",
    "* L2 regularization, where the cost added is proportional to the _square of the value of the weights coefficients_ (i.e. to what is called \n",
    "the \"L2 norm\" of the weights). L2 regularization is also called _weight decay_ in the context of neural networks. Don't let the different \n",
    "name confuse you: weight decay is mathematically the exact same as L2 regularization.\n",
    "\n",
    "In Keras, weight regularization is added by passing _weight regularizer instances_ to layers as keyword arguments. Let's add L2 weight \n",
    "regularization to our movie review classification network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "l2_model = models.Sequential()\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu', input_shape=(10000,)))\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu'))\n",
    "l2_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l2_model.compile(optimizer='rmsprop',\n",
    "                 loss='binary_crossentropy',\n",
    "                 metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`l2(0.001)` means that every coefficient in the weight matrix of the layer will add `0.001 * weight_coefficient_value` to the total loss of \n",
    "the network. Note that because this penalty is _only added at training time_, the loss for this network will be much higher at training \n",
    "than at test time.\n",
    "\n",
    "Here's the impact of our L2 regularization penalty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.4880 - acc: 0.8218 - val_loss: 0.3820 - val_acc: 0.8798\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.3162 - acc: 0.9068 - val_loss: 0.3353 - val_acc: 0.8896\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2742 - acc: 0.9185 - val_loss: 0.3306 - val_acc: 0.8898\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2489 - acc: 0.9288 - val_loss: 0.3363 - val_acc: 0.8866\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2420 - acc: 0.9318 - val_loss: 0.3492 - val_acc: 0.8820\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2322 - acc: 0.9359 - val_loss: 0.3567 - val_acc: 0.8788\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2254 - acc: 0.9385 - val_loss: 0.3632 - val_acc: 0.8787\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2219 - acc: 0.9380 - val_loss: 0.3630 - val_acc: 0.8794\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2162 - acc: 0.9430 - val_loss: 0.3704 - val_acc: 0.8763\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2144 - acc: 0.9428 - val_loss: 0.3876 - val_acc: 0.8727\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2091 - acc: 0.9439 - val_loss: 0.3883 - val_acc: 0.8724\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2061 - acc: 0.9455 - val_loss: 0.3870 - val_acc: 0.8740\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2069 - acc: 0.9445 - val_loss: 0.4073 - val_acc: 0.8714\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2028 - acc: 0.9475 - val_loss: 0.3976 - val_acc: 0.8714\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1998 - acc: 0.9472 - val_loss: 0.4362 - val_acc: 0.8670\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2019 - acc: 0.9462 - val_loss: 0.4088 - val_acc: 0.8711\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1953 - acc: 0.9495 - val_loss: 0.4185 - val_acc: 0.8698\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1945 - acc: 0.9508 - val_loss: 0.4371 - val_acc: 0.8674\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1934 - acc: 0.9486 - val_loss: 0.4136 - val_acc: 0.8699\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1924 - acc: 0.9504 - val_loss: 0.4200 - val_acc: 0.8704\n"
     ]
    }
   ],
   "source": [
    "l2_model_hist = l2_model.fit(x_train, y_train,\n",
    "                             epochs=20,\n",
    "                             batch_size=512,\n",
    "                             validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYFPWd7/H3lxGcYPCOBkUcNKgMN5URNMb7BXUNiBqj\nh+esA4lGDYomx0sWN4xxzWM8iW50NQkqITniFaOSxCTGqLAazDKQAbmEiy6YMUQHIiIZWQf4nj+q\npm2Gmema6a6uvnxez9NPd1X/qus7RVPf/l3qV+buiIiIAPRIOgARESkcSgoiIpKipCAiIilKCiIi\nkqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIim7JR1AV+2///5eVVWVdBgiIkVl4cKFG9y9b6Zy\nRZcUqqqqqK+vTzoMEZGiYmbropRT85GIiKQoKYiISIqSgoiIpBRdn0J7WlpaaGxsZOvWrUmHIkWi\nsrKS/v3707Nnz6RDESkoJZEUGhsb6dOnD1VVVZhZ0uFIgXN3Nm7cSGNjIwMHDkw6HJGCUhLNR1u3\nbmW//fZTQpBIzIz99ttPNUvJu7q6ZLePoiSSAqCEIF2i74sk4bbbkt0+ipJJCiIikj0lhRxpbGxk\n3LhxDBo0iMMPP5wpU6bw8ccft1v2r3/9KxdffHHGzzzvvPPYtGlTt+Kpq6vje9/7Xre2jWrmzJlM\nnjw56zIipayuDsyCB3zyOmpTULbbd1VZJ4VcHVR358ILL+SCCy5g9erVrFq1ii1btjB16tRdym7b\nto2DDjqI2bNnZ/zc559/nr333js3QYpIIurqwD14wCevu5IUstm+q8o6KeSqfe6ll16isrKSiRMn\nAlBRUcE999zDjBkzaG5uZubMmYwdO5bTTz+dM844g7Vr1zJ06FAAmpubueSSS6iurmb8+PGMHj06\nNY1HVVUVGzZsYO3atQwePJgrrriCIUOGcPbZZ/PRRx8B8OCDD3LccccxYsQILrroIpqbmzuNtba2\nlquvvprjjz+eww47jFdeeYVJkyYxePBgamtrU+Uee+wxhg0bxtChQ7n55ptT63/yk59wxBFHMGrU\nKF577bXU+qamJi666CKOO+44jjvuuJ3eE5HiUdZJIVeWLVvGyJEjd1q35557MmDAANasWQPAokWL\nmD17NnPnzt2p3AMPPMA+++zD8uXLuf3221m4cGG7+1i9ejVf+9rXWLZsGXvvvTdPP/00ABdeeCEL\nFixg8eLFDB48mIcffjhjvO+//z7z58/nnnvuYezYsdxwww0sW7aMN954g4aGBv76179y880389JL\nL9HQ0MCCBQt49tlnWb9+PdOmTeO1117j1VdfZfny5anPnDJlCjfccAMLFizg6aef5itf+UqXjqFI\nOZg2LdntoyiJ6xS6oq5u5xpCazvdtGnxDvc666yz2HfffXdZ/+qrrzJlyhQAhg4dyvDhw9vdfuDA\ngRx99NEAjBw5krVr1wKwdOlSbr31VjZt2sSWLVsYM2ZMxli+8IUvYGYMGzaMAw88kGHDhgEwZMgQ\n1q5dy7p16zj11FPp2zeYUHHChAnMmzcPYKf1X/rSl1i1ahUAL7744k5JYvPmzWzZsiVjLCLlpBiG\npJZlUmg9sGaftNNlo7q6epc+gs2bN/P222/z2c9+lkWLFrHHHntktY/dd9899bqioiLVfFRbW8uz\nzz7LiBEjmDlzJq+88krkz+rRo8dOn9ujRw+2bdvWrat8d+zYweuvv05lZWWXtxWRwqHmoxw444wz\naG5u5mc/+xkA27dv5xvf+Aa1tbX07t27021PPPFEnnzySQCWL1/OG2+80aV9f/jhh/Tr14+WlhZm\nzZrVvT+gjVGjRjF37lw2bNjA9u3beeyxxzjllFMYPXo0c+fOZePGjbS0tPDUU0+ltjn77LO57777\nUssNDQ05iUVE8qusk0Ku2ufMjGeeeYannnqKQYMGccQRR1BZWcl3vvOdjNtec801NDU1UV1dza23\n3sqQIUPYa6+9Iu/79ttvZ/To0Zx44okcddRR2fwZKf369ePOO+/ktNNOY8SIEYwcOZJx48bRr18/\n6urqOOGEEzjxxBMZPHhwapt7772X+vp6hg8fTnV1NT/60Y9yEouI5Jd5LtpP8qimpsbb3mRnxYoV\nO52gisn27dtpaWmhsrKSN998kzPPPJOVK1fSq1evpEMrecX8vRHpKjNb6O41mcqVXZ9CoWlubua0\n006jpaUFd+eBBx5QQhCRxCgpJKxPnz66vaiIFIyy7lMQEZGdKSmIiEiKkoKIiKQoKYhI2cjHFcHF\nTkkhRz796U/vsu7uu++murqa4cOHc8YZZ7Bu3bq8x9WdKbTnzJnDnXfemfW+Tz311Ng70WtrazPO\nOBuljJSHfNykptiVZVKYNQuqqqBHj+A5RxcC7+KYY46hvr6eJUuWcPHFF3PTTTdl3Gbbtm3xBBPR\ntm3bGDt2LLfcckuicYhIMsouKcyaBVdeCevWBfMerVsXLMeRGE477bTUNBfHH388jY2N7Zarra3l\nqquuYvTo0dx000384x//YNKkSYwaNYpjjjmG5557Duh8mu30msrs2bN3mga7VUfTbLfdf/qNcY4+\n+ujU41Of+hRz587tML6PPvqISy+9lMGDBzN+/PjU/ExtVVVV8c1vfpOjjz6ampoaFi1axJgxYzj8\n8MNTV0K7OzfeeCNDhw5l2LBhPPHEE6n1kydP5sgjj+TMM8/kvffeS33uwoULOeWUUxg5ciRjxoxh\n/fr10f6hpKTl+yY1xa7srlOYOhXa3nKguTlYP2FCfPt9+OGHOffcczt8v7GxkT/84Q9UVFTwL//y\nL5x++unMmDGDTZs2MWrUKM4880x++MMfpqbZXrp0aWrW1KguvPBCrrjiCgBuvfVWHn74Ya699tpd\n9j9z5szUNq1zGP3iF7/grrvu4nOf+xzTpk1rN74f//jH9O7dmxUrVrBkyRKOPfbYDmMZMGAADQ0N\n3HDDDdTW1vLaa6+xdetWhg4dylVXXcXPf/5zGhoaWLx4MRs2bOC4447j5JNPZv78+axcuZLly5fz\n7rvvUl1dzaRJk2hpaeHaa6/lueeeo2/fvjzxxBNMnTqVGTNmdOkYSemJYxLMUlZ2SeHtt7u2Phce\neeQR6uvrd7mXQrovfvGLVFRUAPDCCy8wZ86cVF/A1q1befvttyNPs92RzqbZTt9/W6tXr+bGG2/k\n5ZdfpmfPnh3GN2/ePK677joAhg8f3ml8Y8eOBWDYsGFs2bKFPn360KdPH3bffXc2bdrEq6++ymWX\nXUZFRQUHHnggp5xyCgsWLGDevHmp9QcddBCnn346ACtXrmTp0qWcddZZQDB9SL9+/bp0fESkDJPC\ngAFBk1F76+Pw4osvcscddzB37tzUNNVTp07lV7/6FfDJL/H0qbXdnaeffpojjzwy8n6stW5McJJu\nT2fTbHc0tfeWLVu45JJLePDBB1Mn2e7E11am6bu7yt0ZMmQI8+fP73ZMUvrycZOaYld2fQp33AFt\nZ7Pu3TtYn2t/+tOf+OpXv8qcOXM44IAD0mK4g4aGhg6nlx4zZgz33XcfrZMV/ulPfwI6n2b7wAMP\nZMWKFezYsYNnnnmm3c/tzjTbkyZNYuLEiZx00kkZ4zv55JN59NFHgaBWsmTJkkj7aM9JJ53EE088\nwfbt22lqamLevHmMGjWKk08+ObV+/fr1vPzyywAceeSRNDU1pZJCS0sLy5Yt6/b+pTSpHyGzsqsp\ntPYbTJ0aNBkNGBAkhGz7E5qbm+nfv39q+etf/zrPP/88W7Zs4Ytf/CIQtKPPmTMn42f967/+K9df\nfz3Dhw9nx44dDBw4kF/+8pdcc801XH755VRXV3PUUUftNM32nXfeyfnnn0/fvn2pqalp965nrdNs\n9+3bl9GjR/Phhx92Gse6deuYPXs2q1atSrXNP/TQQx3Gd/XVVzNx4kQGDx7M4MGDd7lFaVeMHz+e\n+fPnM2LECMyMu+66i8985jOMHz+el156ierqagYMGMAJJ5wAQK9evZg9ezbXXXcdH3zwAdu2beP6\n669nyJAh3Y5BpBxp6uwiomm2c6tcvjcioKmzS5Km2RaRuMWaFMzsHOAHQAXwkLvf2eb9e4DTwsXe\nwAHuvnecMRUzTbMtInGLLSmYWQVwP3AW0AgsMLM57r68tYy735BW/lrgmO7uz913GoEj0pliazYV\nyZc4Rx+NAta4+1vu/jHwODCuk/KXAY91Z0eVlZVs3LhR/9ElEndn48aNVFZWJh2KSMGJs/noYOAv\nacuNwOj2CprZocBA4KUO3r8SuBKCETxt9e/fn8bGRpqamrIMWcpFZWXlTqPFRCRQKB3NlwKz3X17\ne2+6+3RgOgSjj9q+37NnTwYOHBhvhCJS9tKnzChVcTYfvQMckrbcP1zXnkvpZtORiEi+lMPU23Em\nhQXAIDMbaGa9CE78u1y5ZWZHAfsAmp9ARCRhsSUFd98GTAZ+C6wAnnT3ZWb2bTMbm1b0UuBxVy+x\niBSgcpt6uySuaBYRyYdinno76hXNZTchnoiIdExJQUQkonKYeltJQUQkolLtR0inpCAiIilKCiIi\nkqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIilKCiIikqKkICIiKUoKIiKSoqQgIiIpSgoiIpKi\npCAiIilKCiIikqKkICJFoxymrk6akoKIFI3bbks6gtKnpCAiIilKCiJS0OrqwCx4wCev1ZQUD3P3\npGPokpqaGq+vr086DBFJgBkU2SmrYJjZQnevyVRONQUREUlRUhCRojFtWtIRlD4lBREpGupHiJ+S\ngoiIpCgpiIhIipKCiIikKCmIiEiKkoKIiKRkTApmtoeZ9QhfH2FmY82sZ/yhiYhIvkWpKcwDKs3s\nYOAF4H8DM6N8uJmdY2YrzWyNmd3SQZlLzGy5mS0zs0ejBi4iIrm3W4Qy5u7NZvZl4AF3v8vMGjJu\nZFYB3A+cBTQCC8xsjrsvTyszCPgmcKK7v29mB3TvzxARkVyIUlMwMzsBmAD8KlxXEWG7UcAad3/L\n3T8GHgfGtSlzBXC/u78P4O7vRQtbRETiECUpXE/wa/4Zd19mZocBL0fY7mDgL2nLjeG6dEcAR5jZ\na2b2upmdEyVoERGJR8bmI3efC8wFCDucN7j7dTnc/yDgVKA/MM/Mhrn7pvRCZnYlcCXAgAEDcrRr\nERFpK8roo0fNbE8z2wNYCiw3sxsjfPY7wCFpy/3DdekagTnu3uLu/w2sIkgSO3H36e5e4+41ffv2\njbBrERHpjijNR9Xuvhm4APg1MJBgBFImC4BBZjbQzHoBlwJz2pR5lqCWgJntT9Cc9Fa00EVEJNei\nJIWe4XUJFxD+qgcy3ubC3bcBk4HfAiuAJ8M+iW+b2diw2G+BjWa2nKCf4kZ339idP0RERLIXZUjq\nj4G1wGKCNv9Dgc1RPtzdnweeb7PuW2mvHfh6+BARkYRF6Wi+F7g3bdU6MzstvpBERCQpUTqa9zKz\nu82sPnx8H9gjD7GJSInRTXIKX5Q+hRnAh8Al4WMz8JM4gxKR0nTbbUlHIJlE6VM43N0vSlu+Lco0\nFyIiUnyi1BQ+MrPPty6Y2YnAR/GFJCKlpK4OzIIHfPJaTUmFyYIBQJ0UMDsa+CmwF2DA34Fad18c\nf3i7qqmp8fr6+iR2LSJZMoMMpxyJiZktdPeaTOWijD5qAEaY2Z7hcqThqCIiUnw6TApm1u61AxbW\nAd397phiEpESNW1a0hFIJp3VFPrkLQoRKQvqRyh8HSYFd9fgMRGRMhNl9JGIiJQJJQUREUlRUhAR\nkZSMQ1LNbHfgIqAqvby7fzu+sEREJAlRprl4DvgAWAj8T7zhiIhIkqIkhf7ufk7skYhIwaur07DS\nUhelT+EPZjYs9khEpOBpltPSF6Wm8Hmg1sz+m6D5yAhumjY81shERCTvotQUzgUGAWcDXwDOD59F\npAxoltPyknGWVAAzGwGcFC7+Z1IzpIJmSRVJkmY5LV5RZ0mNcjvOKcAs4IDw8YiZXZt9iCIiUmii\nNB99GRjt7t9y928BxwNXxBuWiMQh2yYfzXJa+qIkBQO2py1vD9eJSJHJdvSQ+hFKX5TRRz8B/mhm\nz4TLFwAPxxeSiIgkJWNNIbyZzkSC23D+HZjo7v8ed2AikhsaPSRd0eHoIzPb0903m9m+7b3v7n+P\nNbIOaPSRSPdp9FD5ysU9mh8luCZhIZD+NbJw+bCsIhQRkYLT2Z3Xzg+fB+YvHBGJk0YPSSZRrlP4\nfZR1IlL41I8gmXSYFMysMuxP2N/M9jGzfcNHFXBwvgIUkU/opC5x66ym8FWC/oSjwufWx3PAf8Qf\nmoi0pVlKJW6d9Sn8APiBmV3r7vflMSYREUlIlOsU7jOzoWZ2iZn9c+sjyoeb2TlmttLM1pjZLe28\nX2tmTWbWED6+0p0/QqSU6ToDyaeMs6Sa2TTgVKAaeJ5gKu1X3f3iDNtVAKuAs4BGYAFwmbsvTytT\nC9S4++SoAes6BSlnus5Auitns6QCFwNnAH9z94nACGCvCNuNAta4+1vu/jHwODAuwnYiIpKQKEnh\nI3ffAWwzsz2B94BDImx3MPCXtOVG2h+1dJGZLTGz2WYW5XNFypauM5C4RUkK9Wa2N/AgweijRcD8\nHO3/F0BVeGvP3wE/ba+QmV1pZvVmVt/U1JSjXYsUH/UjSNwyzpLq7teEL39kZr8B9nT3JRE++x12\nrlH0D9elf/bGtMWHgLs6iGE6MB2CPoUI+xYRkW7oMCmY2bGdvefuizJ89gJgkJkNJEgGlwL/q83n\n9HP39eHiWGBFpKhFRCQWndUUvh8+VwI1wGKCyfCGA/XACZ19sLtvM7PJwG+BCmCGuy8zs28D9e4+\nB7jOzMYC2wim5a7N4m8REZEsRRmS+nNgmru/ES4PBeoyDUmNi4akioh0XS6HpB7ZmhAA3H0pMDib\n4EREpDBFuR3nEjN7CHgkXJ4AROloFhGRIhMlKUwErgamhMvzgB/GFpGIiCQmypDUrcA94UNEREpY\nZ0NSn3T3S8zsDXa+HScA4QVnIiJSQjqrKbQ2F52fj0BERCR5nd1PYX34vC5/4YiISJI6az76kHaa\njQguYHN33zO2qEREJBGd1RT65DMQERFJXpQhqQCY2QEEU14A4O5vxxKRiIgkJuMVzWY21sxWA/8N\nzAXWAr+OOS4REUlAlGkubgeOB1a5+0CCu7C9HmtUIiVI90KQYhAlKbSE9z3oYWY93P1lgllTRaQL\nbrst6QhEMovSp7DJzD5NML3FLDN7D/hHvGGJiEgSotQUxgEfATcAvwHeBL4QZ1AipaKuDsyCB3zy\nWk1JUqg6vJ+Cmd0PPOrur+U3pM7pfgpSrMwgw+1LRGKTi/sprAK+Z2ZrzewuMzsmd+GJiEgh6jAp\nuPsP3P0E4BRgIzDDzP5sZtPM7Ii8RShSIqZNSzoC6a5Zs6CqCnr0CJ5nzUo6ovhk7FNw93Xu/l13\nPwa4DLgAWBF7ZCIlppz7EYr5pDprFlx5JaxbFzT/rVsXLBfT39AVUS5e283MvmBmswguWlsJXBh7\nZCJSEor9pDp1KjQ377yuuTlYX4o662g+i6BmcB7wX8DjwHPunuhwVHU0ixSXqqogEbR16KGwdm2+\no+m6Hj3aHyBgBjt25D+e7spFR/M3gT8Ag919rLs/mnRCEJHi83YHs6R1tL7QDBjQtfXFrrOO5tPd\n/SF3fz+fAYnEJds2/XLuE8hGsZ9U77gDevfeeV3v3sH6UhTl4jWRkpDtNBOapqJ7iv2kOmECTJ8e\nNHeZBc/TpwfrS5GSgojEqhROqhMmBP0fO3YEz/mOPZ+jt5QUpKRlO82EpqnIjXI6qeZavkdvdTj6\nqFBp9JF0V7bTTGiaiuLUelJNH1bau3fx1FZyNXorF6OPRESKXiFcZ5BNTSXfo7eUFKRsZDvNhKap\nKE5JD4nNtvkn36O3lBSkbJTzkNRiblOH7OJPekhstjWVfI/eUlIQKXHFPs1EtvEnPSQ225pKvkdv\nqaNZpMQV+zQTuYh/1qzgl/nbbwc1hDvuyF8nc6Ec/4LoaDazc8xspZmtMbNbOil3kZm5menezyI5\nlnSberZyEX+SQ2KTrql0VWxJwcwqgPuBc4Fq4DIzq26nXB9gCvDHuGKRwlDMbfLFLBdt6kn2SSTd\nJ5CtYrt4L86awihgjbu/5e4fE8yyOq6dcrcD3wW2xhiLFABNE5GMbH+pJt0nUWy/tNuT9MV7XRFn\nUjgY+EvacmO4LsXMjgUOcfdfdfZBZnalmdWbWX1TU1PuIxUpYdn+Uk16nH+x/dIudomNPjKzHsDd\nwDcylXX36e5e4+41ffv2jT84yRlNE1EYsvmlWgh9EsX0S7vYxZkU3gEOSVvuH65r1QcYCrxiZmuB\n44E56mwuLXV1QZND6yC31tfFlBSybU8v9msEir1NX7omzqSwABhkZgPNrBdwKTCn9U13/8Dd93f3\nKnevAl4Hxrq7xptKwci2PT1X7fFJJpZSaNOXLnD32B4Et/JcBbwJTA3XfZvg5N+27CtATabPHDly\npEtxmjYt6Qi67tBDW+s2Oz8OPTQ/27u7P/KIe+/eO2/fu3ewPl8eeSSI2Sx4zue+JTeAeo9w3i6L\ni9eSvHBFilu29+fNxf19C+XiJyluBXHxWiFIejid5E4S/RDZtqfnoj2+EDp6pXyUfFJIejid5E4S\n1zlk256ei/Z4dfRKPpV8UtCvLMlGtmPkczHGXh29kk8lnxT0K6u45eI6h2xH7mQ7Rj4X2+viLcmX\nku9oLvZb8cknunM7TP37iwTU0RzSr6zypj4lka4p+aQAukQ+abm6IhgK//62IsVut6QDkNLWtvmm\ndUgwREvO2W4/YED7Y/zVpyTSvrKoKUhysm2+Kbb724oUOyUFySib5p9sm2+K7f62IsVOzUfSqaSb\nb3LR/DNhgpKASFSqKUinkm6+UfOPSH4pKZSBJJt/CuGKYBGJruQvXit32V68pRk6RUqDLl5rRzHd\n7StXkm7+EZHiUlZJIYlZNpOWdPOPiBQXjT4qcRq9IyJdUfI1hVzMspm0bDqK1fwjIl1RFkmh9c62\n8MnrfE69nI1s7xyXy+afYkqkItI9ZTX6KKmpl7O5R3Qhjf7pzvETkcKg0UftmDat69tkO3on21/6\nmuVTRPKprJJCd5o/sj0pZ5tUkr5zXCn0yYhIdGWVFLoj25Nytkkl6Y7iXPTJiEjxUFLIINuTcrZJ\nRdcJiEg+KSlkkO1JORe/9AvlznHd6ZMRkeJSVqOPkpLN6CMRkVyIOvpIVzTnga4IFpFioeYjERFJ\nUVIoIxoxJCKZKCmUkXKcJVZEukZJQUREUmJNCmZ2jpmtNLM1ZnZLO+9fZWZvmFmDmb1qZtVxxlOO\ndEWyiHRFbENSzawCWAWcBTQCC4DL3H15Wpk93X1z+HoscI27n9PZ5xbjkNRCoQntRMpXIUyINwpY\n4+5vufvHwOPAuPQCrQkhtAegU5aISILivE7hYOAvacuNwOi2hczsa8DXgV7A6THGU/Z0RbKIZJJ4\nR7O73+/uhwM3A7e2V8bMrjSzejOrb2pqym+ABSTbfgD1I4hIJnEmhXeAQ9KW+4frOvI4cEF7b7j7\ndHevcfeavn375jDE4qIhpSIStziTwgJgkJkNNLNewKXAnPQCZjYobfGfgNUxxpM1/dIWkVIXW1Jw\n923AZOC3wArgSXdfZmbfDkcaAUw2s2Vm1kDQr3B5XPHkQhK/1DWkVETySbOkdkG2Qzrr6rI7mWtI\nqYh0VyEMSS0Jufylrj4BESl0mjo7g/Rf90n/UteQUhGJm2oKMctlTUP9CCISN9UUuqA7v9QLqaYh\nIpKJagpdoF/qIlLqlBTySH0CIlLolBTySDUNESl0SgoiIpKipCAiIilKCiIikqKkICIiKUoKIiKS\nUnQT4plZE7Au6Tg6sD+wIekgOqH4slPo8UHhx6j4spNNfIe6e8Yb0hRdUihkZlYfZRbCpCi+7BR6\nfFD4MSq+7OQjPjUfiYhIipKCiIikKCnk1vSkA8hA8WWn0OODwo9R8WUn9vjUpyAiIimqKYiISIqS\nQheZ2SFm9rKZLTezZWY2pZ0yp5rZB2bWED6+lecY15rZG+G+d7mhtQXuNbM1ZrbEzI7NY2xHph2X\nBjPbbGbXtymT9+NnZjPM7D0zW5q2bl8z+52ZrQ6f9+lg28vDMqvN7PI8xfZ/zezP4b/fM2a2dwfb\ndvpdiDnGOjN7J+3f8bwOtj3HzFaG38db8hjfE2mxrTWzhg62jfUYdnROSez75+56dOEB9AOODV/3\nAVYB1W3KnAr8MsEY1wL7d/L+ecCvAQOOB/6YUJwVwN8Ixk8nevyAk4FjgaVp6+4Cbglf3wJ8t53t\n9gXeCp/3CV/vk4fYzgZ2C19/t73YonwXYo6xDvg/Eb4DbwKHAb2AxW3/P8UVX5v3vw98K4lj2NE5\nJanvn2oKXeTu6919Ufj6Q2AFcHCyUXXZOOBnHngd2NvM+iUQxxnAm+6e+MWI7j4P+Hub1eOAn4av\nfwpc0M6mY4Dfufvf3f194HfAOXHH5u4vuPu2cPF1oH8u99lVHRy/KEYBa9z9LXf/GHic4LjnVGfx\nmZkBlwCP5Xq/UXRyTknk+6ekkAUzqwKOAf7YztsnmNliM/u1mQ3Ja2DgwAtmttDMrmzn/YOBv6Qt\nN5JMYruUjv8jJnn8Wh3o7uvD138DDmynTCEcy0kENb/2ZPouxG1y2MQ1o4Pmj0I4ficB77r76g7e\nz9sxbHNOSeT7p6TQTWb2aeBp4Hp339zm7UUETSIjgPuAZ/Mc3ufd/VjgXOBrZnZynvefkZn1AsYC\nT7XzdtLHbxce1NULbqiemU0FtgGzOiiS5Hfhh8DhwNHAeoImmkJ0GZ3XEvJyDDs7p+Tz+6ek0A1m\n1pPgH2+Wu/+87fvuvtndt4Svnwd6mtn++YrP3d8Jn98DniGooqd7Bzgkbbl/uC6fzgUWufu7bd9I\n+vilebe1WS18fq+dMokdSzOrBc4HJoQnjV1E+C7Ext3fdfft7r4DeLCDfSf6XTSz3YALgSc6KpOP\nY9jBOSXUuapeAAADK0lEQVSR75+SQheF7Y8PAyvc/e4OynwmLIeZjSI4zhvzFN8eZtan9TVBh+TS\nNsXmAP8cjkI6HvggrZqaLx3+Okvy+LUxB2gdzXE58Fw7ZX4LnG1m+4TNI2eH62JlZucANwFj3b25\ngzJRvgtxxpjeTzW+g30vAAaZ2cCw9ngpwXHPlzOBP7t7Y3tv5uMYdnJOSeb7F1ePeqk+gM8TVOOW\nAA3h4zzgKuCqsMxkYBnBSIrXgc/lMb7Dwv0uDmOYGq5Pj8+A+wlGfbwB1OT5GO5BcJLfK21doseP\nIEGtB1oI2mW/DOwH/B5YDbwI7BuWrQEeStt2ErAmfEzMU2xrCNqSW7+DPwrLHgQ839l3IY/H7/+F\n368lBCe4fm1jDJfPIxhx82ZcMbYXX7h+Zuv3Lq1sXo9hJ+eURL5/uqJZRERS1HwkIiIpSgoiIpKi\npCAiIilKCiIikqKkICIiKUoKIiEz2247z+Casxk7zawqfYZOkUK1W9IBiBSQj9z96KSDEEmSagoi\nGYTz6d8Vzqn/X2b22XB9lZm9FE749nszGxCuP9CCexwsDh+fCz+qwsweDOfMf8HMPhWWvy6cS3+J\nmT2e0J8pAigpiKT7VJvmoy+lvfeBuw8D/gP493DdfcBP3X04wYR094br7wXmejCh37EEV8ICDALu\nd/chwCbgonD9LcAx4edcFdcfJxKFrmgWCZnZFnf/dDvr1wKnu/tb4cRlf3P3/cxsA8HUDS3h+vXu\nvr+ZNQH93f1/0j6jimDe+0Hh8s1AT3f/NzP7DbCFYDbYZz2cDFAkCaopiETjHbzuiv9Je72dT/r0\n/olgLqpjgQXhzJ0iiVBSEInmS2nP88PXfyCY1RNgAvCf4evfA1cDmFmFme3V0YeaWQ/gEHd/GbgZ\n2AvYpbYiki/6RSLyiU/Zzjdv/427tw5L3cfMlhD82r8sXHct8BMzuxFoAiaG66cA083sywQ1gqsJ\nZuhsTwXwSJg4DLjX3Tfl7C8S6SL1KYhkEPYp1Lj7hqRjEYmbmo9ERCRFNQUREUlRTUFERFKUFERE\nJEVJQUREUpQUREQkRUlBRERSlBRERCTl/wPKWI2iT5zC3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43bef16c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "l2_model_val_loss = l2_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "As you can see, the model with L2 regularization (dots) has become much more resistant to overfitting than the reference model (crosses), \n",
    "even though both models have the same number of parameters.\n",
    "\n",
    "As alternatives to L2 regularization, you could use one of the following Keras weight regularizers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "# L1 regularization\n",
    "regularizers.l1(0.001)\n",
    "\n",
    "# L1 and L2 regularization at the same time\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding dropout\n",
    "\n",
    "\n",
    "Dropout is one of the most effective and most commonly used regularization techniques for neural networks, developed by Hinton and his \n",
    "students at the University of Toronto. Dropout, applied to a layer, consists of randomly \"dropping out\" (i.e. setting to zero) a number of \n",
    "output features of the layer during training. Let's say a given layer would normally have returned a vector `[0.2, 0.5, 1.3, 0.8, 1.1]` for a \n",
    "given input sample during training; after applying dropout, this vector will have a few zero entries distributed at random, e.g. `[0, 0.5, \n",
    "1.3, 0, 1.1]`. The \"dropout rate\" is the fraction of the features that are being zeroed-out; it is usually set between 0.2 and 0.5. At test \n",
    "time, no units are dropped out, and instead the layer's output values are scaled down by a factor equal to the dropout rate, so as to \n",
    "balance for the fact that more units are active than at training time.\n",
    "\n",
    "Consider a Numpy matrix containing the output of a layer, `layer_output`, of shape `(batch_size, features)`. At training time, we would be \n",
    "zero-ing out at random a fraction of the values in the matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# At training time: we drop out 50% of the units in the output\n",
    "layer_output *= np.randint(0, high=2, size=layer_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "At test time, we would be scaling the output down by the dropout rate. Here we scale by 0.5 (because we were previous dropping half the \n",
    "units):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# At test time:\n",
    "layer_output *= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Note that this process can be implemented by doing both operations at training time and leaving the output unchanged at test time, which is \n",
    "often the way it is implemented in practice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# At training time:\n",
    "layer_output *= np.randint(0, high=2, size=layer_output.shape)\n",
    "# Note that we are scaling *up* rather scaling *down* in this case\n",
    "layer_output /= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This technique may seem strange and arbitrary. Why would this help reduce overfitting? Geoff Hinton has said that he was inspired, among \n",
    "other things, by a fraud prevention mechanism used by banks -- in his own words: _\"I went to my bank. The tellers kept changing and I asked \n",
    "one of them why. He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation \n",
    "between employees to successfully defraud the bank. This made me realize that randomly removing a different subset of neurons on each \n",
    "example would prevent conspiracies and thus reduce overfitting\"_.\n",
    "\n",
    "The core idea is that introducing noise in the output values of a layer can break up happenstance patterns that are not significant (what \n",
    "Hinton refers to as \"conspiracies\"), which the network would start memorizing if no noise was present. \n",
    "\n",
    "In Keras you can introduce dropout in a network via the `Dropout` layer, which gets applied to the output of layer right before it, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add(layers.Dropout(0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add two `Dropout` layers in our IMDB network to see how well they do at reducing overfitting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dpt_model = models.Sequential()\n",
    "dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(16, activation='relu'))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "dpt_model.compile(optimizer='rmsprop',\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s - loss: 0.6035 - acc: 0.6678 - val_loss: 0.4704 - val_acc: 0.8651\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.4622 - acc: 0.8002 - val_loss: 0.3612 - val_acc: 0.8724\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.3731 - acc: 0.8553 - val_loss: 0.2960 - val_acc: 0.8904\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.3162 - acc: 0.8855 - val_loss: 0.2772 - val_acc: 0.8917\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2762 - acc: 0.9033 - val_loss: 0.2803 - val_acc: 0.8889\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2454 - acc: 0.9172 - val_loss: 0.2823 - val_acc: 0.8892\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.2178 - acc: 0.9281 - val_loss: 0.2982 - val_acc: 0.8877\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1994 - acc: 0.9351 - val_loss: 0.3101 - val_acc: 0.8875\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1832 - acc: 0.9400 - val_loss: 0.3318 - val_acc: 0.8860\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1692 - acc: 0.9434 - val_loss: 0.3534 - val_acc: 0.8841\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1590 - acc: 0.9483 - val_loss: 0.3689 - val_acc: 0.8830\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1499 - acc: 0.9496 - val_loss: 0.4107 - val_acc: 0.8776\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1405 - acc: 0.9539 - val_loss: 0.4114 - val_acc: 0.8782\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1333 - acc: 0.9562 - val_loss: 0.4549 - val_acc: 0.8771\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1267 - acc: 0.9572 - val_loss: 0.4579 - val_acc: 0.8800\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1225 - acc: 0.9580 - val_loss: 0.4843 - val_acc: 0.8772\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1233 - acc: 0.9590 - val_loss: 0.4783 - val_acc: 0.8761\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1212 - acc: 0.9601 - val_loss: 0.5051 - val_acc: 0.8740\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1153 - acc: 0.9618 - val_loss: 0.5451 - val_acc: 0.8747\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s - loss: 0.1155 - acc: 0.9621 - val_loss: 0.5358 - val_acc: 0.8738\n"
     ]
    }
   ],
   "source": [
    "dpt_model_hist = dpt_model.fit(x_train, y_train,\n",
    "                               epochs=20,\n",
    "                               batch_size=512,\n",
    "                               validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXh4CmIgIqWipLgIJsgQhh8eKGe0VAtLVw\nuVb0qqWVSm3rT1t6S6jtvdZa/V3FDa2gLXUBK/Lr1aqtgsWlBbxxAZU10bhGFBCRGsjn98c5GYYw\nSSaZPfN+Ph7nMXPOnJn5ZBjOZ767uTsiIiIAbTIdgIiIZA8lBRERiVBSEBGRCCUFERGJUFIQEZEI\nJQUREYlQUhARkQglBRERiVBSEBGRiLaZDqC5Dj/8cC8qKsp0GCIiOWX16tUfuXuXps7LuaRQVFTE\nqlWrMh2GiEhOMbPKeM5T9ZGIiEQoKYiISISSgoiIRORcm0IsNTU1VFVVsWvXrkyHItKgwsJCunXr\nRrt27TIdikiDWkVSqKqqokOHDhQVFWFmmQ5HZD/uzpYtW6iqqqJXr16ZDkekQa2i+mjXrl0cdthh\nSgiStcyMww47TKXZPFdWltnnx6NVJAVACUGynr6jMmdOZp8fj1aTFEREJHFKCklSVVXFxIkT6du3\nL3369GHmzJl88cUXMc999913+frXv97ka5511lls3bq1RfGUlZVxww03tOi58VqwYAEzZsxI+ByR\n1qysDMyCDfbej7cqKNHnN1deJ4Vkfajuzrnnnss555zD+vXrWbduHTt27GDWrFn7nbt7926+8pWv\nsHjx4iZf97HHHqNTp07JCVJEMqKsDNyDDfbeb05SSOT5zZXXSSFZ9XNPP/00hYWFXHTRRQAUFBRw\n0003cc8997Bz504WLFjAhAkTOPnkkznllFOoqKhg8ODBAOzcuZPzzz+fgQMHMmnSJEaNGhWZxqOo\nqIiPPvqIiooKBgwYwKWXXsqgQYM4/fTT+fzzzwG46667GDFiBEOHDuW8885j586djcY6bdo0vvOd\n7zB69Gh69+7NsmXLuPjiixkwYADTpk2LnHf//fdTXFzM4MGDufrqqyPH58+fT79+/Rg5ciTPPfdc\n5Hh1dTXnnXceI0aMYMSIEfs8JiK5I6+TQrKsWbOG4cOH73PskEMOoUePHmzYsAGAl156icWLF7N8\n+fJ9zrvtttvo3Lkza9eu5dprr2X16tUx32P9+vVcfvnlrFmzhk6dOvHwww8DcO6557Jy5Upefvll\nBgwYwG9/+9sm4/3kk0944YUXuOmmm5gwYQJXXnkla9as4dVXX6W8vJx3332Xq6++mqeffpry8nJW\nrlzJkiVLeO+995g9ezbPPfccK1asYO3atZHXnDlzJldeeSUrV67k4Ycf5pJLLmnWZyiSD2bPzuzz\n49Eqxik0R1nZviWEunq62bNT293rtNNO49BDD93v+IoVK5g5cyYAgwcPZsiQITGf36tXL0pKSgAY\nPnw4FRUVALz22mv89Kc/ZevWrezYsYMzzjijyVjGjx+PmVFcXMyRRx5JcXExAIMGDaKiooLKykpO\nOukkunQJJlScOnUqzz77LMA+x7/5zW+ybt06AP7yl7/skyS2b9/Ojh07moxFJJ/kQpfUvEwKdR+s\n2d56ukQMHDhwvzaC7du389Zbb/HVr36Vl156ifbt2yf0HgceeGDkfkFBQaT6aNq0aSxZsoShQ4ey\nYMECli1bFvdrtWnTZp/XbdOmDbt3727RiNva2lpefPFFCgsLm/1cEckeqj5KglNOOYWdO3dy3333\nAbBnzx5++MMfMm3aNA466KBGnztmzBgeeughANauXcurr77arPf+9NNP6dq1KzU1NSxcuLBlf0A9\nI0eOZPny5Xz00Ufs2bOH+++/nxNPPJFRo0axfPlytmzZQk1NDYsWLYo85/TTT+eWW26J7JeXlycl\nFhFJr7xOCsmqnzMzHnnkERYtWkTfvn3p168fhYWF/Od//meTz/3ud79LdXU1AwcO5Kc//SmDBg2i\nY8eOcb/3tddey6hRoxgzZgz9+/dP5M+I6Nq1K9dddx1jx45l6NChDB8+nIkTJ9K1a1fKyso49thj\nGTNmDAMGDIg85+abb2bVqlUMGTKEgQMHcscddyQlFhFJL/Nk1J+kUWlpqddfZOf111/f5wKVS/bs\n2UNNTQ2FhYVs3LiRU089lTfffJMDDjgg06FJCuTyd1Vym5mtdvfSps7LuzaFbLNz507Gjh1LTU0N\n7s5tt92mhCAiGaOkkGEdOnTQ8qIikjXyuk1BRET2paQgIiIRSgoiIhKhpCAieSMdI4JznZJCkhQU\nFFBSUsKgQYMYOnQov/nNb6itrc1YPEuWLNln2ol0Ovjgg5v9nESmCa+zbNkyzj777IReoynRkxkm\nco5kRjoWqcl1eZkUFi6EoiJo0ya4TcZA4C996UuUl5ezZs0annrqKR5//HHmxPgG7t69O/E3i0O8\nScHdM5q86t5f04SLZIe8SwoLF8Jll0FlZTDvUWVlsJ+kGSIAOOKII5g3bx5z587F3febOtvdueqq\nqxg8eDDFxcU8+OCDQPBL94QTTmDcuHEcffTRTJ8+PXLBbmgq6+hf5YsXL2batGk8//zzLF26lKuu\nuoqSkhI2bty4T3wVFRUcffTRfOtb32Lw4MG8/fbbPPnkkxx77LEMGzaMb3zjG5HJ7B577DH69+/P\n8OHDueKKKyK/xOsv4jN48ODIJH11duzYwSmnnMKwYcMoLi7m0UcfbfD966YJv+OOOygpKaGkpIRe\nvXoxduxYgAbj+/Of/0z//v0ZNmwYf/zjH2P+eyxYsIBzzjmH0047jaKiIubOncuNN97IMcccw+jR\no/n444+BYGqO0aNHM2TIECZNmsQnn3wCwOrVqxk6dChDhw7l1ltvjbzunj17uOqqqxgxYgRDhgzh\nzjvvbPK7IemX7kVqcp6759Q2fPhwr2/t2rX7HWtIz551S1Tsu/XsGfdLxNS+ffv9jnXs2NHff/99\nnz9/vh911FG+ZcsWd3dfvHixn3rqqb57925///33vXv37v7uu+/6M8884wceeKBv3LjRd+/e7aee\neqovWrTI33nnHe/evbt/+OGHXlNT42PHjvVHHnlkv/ddtGiRX3jhhe7ufuGFF/qiRYtixrp582Y3\nM3/hhRfc3b26utqPP/5437Fjh7u7X3fddT5nzhz//PPPvVu3br5p0yZ3d588ebKPGzfO3d1nz57t\nv/71ryOvOWjQIN+8efM+MdXU1Pi2bdsi79GnTx+vra3d7/3d3Xv27OnV1dWR/S+++MKPO+44X7p0\naZPxrVu3zmtra/0b3/hGJL5o8+fP9z59+vj27dv9ww8/9EMOOcRvv/12d3f//ve/7zfddJO7uxcX\nF/uyZcvc3f0//uM/fObMmZHjy5cvd3f3H/3oRz5o0CB3d7/zzjv92muvdXf3Xbt2+fDhw33Tpk2+\nefPmyDn1Nee7KskHmY4gc4BVHsc1Nu9KCm+91bzjyRI9dfaKFSuYMmUKBQUFHHnkkZx44omsXLkS\nCCaj6927NwUFBUyZMoUVK1awcuXKyJTVbdu23Wcq65bq2bMno0ePBuDFF19k7dq1jBkzhpKSEu69\n914qKyt544036N27N7169QJgypQpzXoPd+cnP/kJQ4YM4dRTT+Wdd97hgw8+2O/9Y5k5cyYnn3wy\n48ePbzS+Xr160bdvX8yMf/u3f2vw9caOHUuHDh3o0qULHTt2ZPz48QAUFxdTUVHBtm3b2Lp1Kyee\neCIAF154Ic8++yxbt25l69atnHDCCQBccMEFkdd88sknue+++ygpKWHUqFFs2bKF9evXN+szEsk2\neTeiuUePoMoo1vFk2rRpEwUFBRxxxBEAcU+dbXVl3Ab2Gzt/165dMc95++23IxfB6dOnc+aZZ+4T\nj7tz2mmncf/99+/zvMZmOm3btu0+bRGx3nvhwoVUV1ezevVq2rVrR1FRUeS8xj6PBQsWUFlZydy5\nc1scX331pwiPnj68pe087s4tt9yy3xoW9avRJHukY5GaXJd3JYVf/hLqz2Z90EHB8WSprq5m+vTp\nzJgxI+ZF/fjjj+fBBx9kz549VFdX8+yzzzJy5EgA/vGPf7B582Zqa2t58MEHOe644xqcyhrgyCOP\n5PXXX6e2tpZHHnkk8h4dOnTg008/BaB79+6Ul5dTXl7O9OnT94tn9OjRPPfcc5FV4j777DPWrVvH\n0UcfzaZNmyIXubq2DwiWCn3ppZeAYFW5zZs37/e627Zt44gjjqBdu3Y888wzVMbKxvWsXr2aG264\ngd///ve0adOm0fj69+9PRUVFpM2kftJojo4dO9K5c2f+9re/AfC73/2OE088kU6dOtGpUydWrFgB\nsM/05GeccQa33347NTU1AKxbt47PPvusxTFI6qkdoWl5V1KYOjW4nTUrqDLq0SNICHXHW+rzzz+n\npKSEmpoa2rZtywUXXMAPfvCDmOdOmjSJF154gaFDh2JmXH/99Xz5y1/mjTfeYMSIEcyYMYMNGzYw\nduxYJk2aRJs2bSJTWbs748aNY+LEiQBcd911nH322XTp0oXS0tJIA+zkyZO59NJLufnmm1m8eDF9\n+vRpMPYuXbqwYMECpkyZwj//+U8AfvGLX9CvXz9uu+22SMlixIgRkeecd9553HfffQwaNIhRo0bR\nr1+//V536tSpjB8/nuLiYkpLS+Oa2nvu3Ll8/PHHkQbm0tJS7r777gbjmzdvHuPGjeOggw7i+OOP\njyTClrj33nuZPn06O3fupHfv3syfPx8I1qW++OKLMTNOP/30yPmXXHIJFRUVDBs2DHenS5cuLFmy\npMXvL5INNHV2Flm2bBk33HADf/rTnzIdSsSOHTs4+OCDcXcuv/xy+vbty5VXXpnpsHJWa/muSu6J\nd+rslFYfmdmZZvammW0ws2tiPH6TmZWH2zozS2z0kiTdXXfdFRmUt23bNr797W9nOiQRSaGUlRTM\nrABYB5wGVAErgSnuHnNElZl9DzjG3S9u7HVbc0lBWj99VyVTsqGkMBLY4O6b3P0L4AFgYiPnTwFa\n3FKYa9Vgkn/0HZVckMqkcBTwdtR+VXhsP2bWE+gFPN2SNyosLGTLli36TydZy93ZsmULhYWFmQ5F\npFHZ0vtoMrDY3ffEetDMLgMuA+gRY0BBt27dqKqqorq6OqVBiiSisLCQbt26ZToMSUBZWevv1prK\nNoVjgTJ3PyPc/zGAu/9XjHP/F7jc3Z9v6nVjtSmIiKSDWTAxTi7KhjaFlUBfM+tlZgcQlAaW1j/J\nzPoDnYEXUhiLiIjEIWVJwd13AzOAJ4DXgYfcfY2Z/dzMJkSdOhl4wNUgICJZKN9mWW0Vg9dERNJB\n1UciIpJXlBREROKUD7OsKimIiMSptbYjRFNSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRER\niVBSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiVBSEBGRCCUFERGJUFIQkZyRD1NXZ5qS\ngojkjDlzMh1B66ekICIiEUoKIpLVysrALNhg731VJaWGuXumY2iW0tJSX7VqVabDEJEMMIMcu2Rl\nDTNb7e6lTZ2nkoKIiEQoKYhIzpg9O9MRtH5KCiKSM9SOkHpKCiIiEqGkICIiEUoKIiISoaQgIiIR\nSgoiIhKhpCAiIhFNJgUza29mbcL7/cxsgpm1S31oIiKSbvGUFJ4FCs3sKOBJ4AJgQTwvbmZnmtmb\nZrbBzK5p4JzzzWytma0xsz/EG7iIiCRf2zjOMXffaWb/Dtzm7tebWXmTTzIrAG4FTgOqgJVmttTd\n10ad0xf4MTDG3T8xsyNa9meIiEgyxFNSMDM7FpgK/E94rCCO540ENrj7Jnf/AngAmFjvnEuBW939\nEwB3/zC+sEVEJBXiSQrfJ/g1/4i7rzGz3sAzcTzvKODtqP2q8Fi0fkA/M3vOzF40szNjvZCZXWZm\nq8xsVXV1dRxvLSIiLdFk9ZG7LweWA4QNzh+5+xVJfP++wElAN+BZMyt29631YpgHzINg6uwkvbeI\niNQTT++jP5jZIWbWHngNWGtmV8Xx2u8A3aP2u4XHolUBS929xt03A+sIkoSIiGRAPNVHA919O3AO\n8DjQi6AHUlNWAn3NrJeZHQBMBpbWO2cJQSkBMzucoDppU3yhi4hIssWTFNqF4xLOIfxVDzRZhePu\nu4EZwBPA68BDYZvEz81sQnjaE8AWM1tL0E5xlbtvackfIiIiiYunS+qdQAXwMkGdf09gezwv7u6P\nAY/VO/azqPsO/CDcREQkw5osKbj7ze5+lLuf5YFKYGwaYhORVkaL5GS/eBqaO5rZjXVdQs3sN0D7\nNMQmIq3MnDmZjkCaEk+bwj3Ap8D54bYdmJ/KoEREJDPiSQp93H12ODJ5k7vPAXqnOjARaR3KysAs\n2GDvfVUlZad4ksLnZnZc3Y6ZjQE+T11IItKalJWBe7DB3vtKCtkpnt5H3wHuNbOOgAEfA9NSGZSI\niGRGPNNclANDzeyQcD+u7qgiIvXNnp3pCKQpDSYFM4s5dsDCikF3vzFFMYlIK6Uqo+zXWEmhQ9qi\nEBGRrNBgUgh7GYmISB6Jp/eRiIjkCSUFERGJUFIQEZGIeOY+OtDM/tXMfmJmP6vb0hGciEhrsHAh\nFBVBmzbB7cKFmY6oYfGUFB4FJgK7gc+iNhHJM+pS2nwLF8Jll0FlZTCSu7Iy2M/WxGDuja+XY2av\nufvgNMXTpNLSUl+1alWmwxDJS2Z7p6uQ+BQVBYmgvp49oaIifXGY2Wp3L23qvHhKCs+bWXESYhIR\nyTtvvdW845kWT1I4DlhtZm+a2Stm9qqZvZLqwEQkO2iW08T06NG847Gks00inuqjnrGOhyuwpZ2q\nj0QyR9VHzVfXprBz595jBx0E8+bB1Kmpf36dpFUfhRf/TsD4cOuUqYQgIpJrpk4NLuA9ewZJtWfP\n5l3QZ83aNyFAsD9rVvJjhfi6pM4EFgJHhNvvzex7qQlHRFIp0SofzXLaMlOnBo3KtbXBbXN+4ae7\nTSKe6qNXgGPd/bNwvz3wgrsPSU1IjVP1kUjLqfon9ySr91Iyex8ZsCdqf094TEREUuyXvwzaEKId\ndFBwPBXiSQrzgb+bWZmZlQEvAr9NTTgikmzqPZTbEm2TaK4mq48AzGwYQddUgL+5+/+mJpymqfpI\npOVUfZS/4q0+amzltUPcfbuZHQpUhFvdY4e6+8fJCFRERLJHYyuv/QE4G1gNRP+2sHC/dwrjEpEU\nUO8haUqDbQrufnZ428vde0dtvdxdCUEkAxJtB1A7gjQlnnEKf43nmIik3hwtkisp1mBSMLPCsD3h\ncDPrbGaHhlsRcFS6AhQRybRcWg8hUY2VFL5N0J7QP7yt2x4F5qY+NBEBdSnNtFxbDyFR8Yxo/p67\n39KiFzc7E/hvoAC4292vq/f4NODXwDvhobnufndjr6kuqZLP1KU0/bJlPYREJdwltY6732Jmg4GB\nQGHU8fuaCKAAuBU4DagCVprZUndfW+/UB919RlNxiIhkQq6th5CoeBqaZwO3hNtY4HpgQhyvPRLY\n4O6b3P0L4AGCZT1FpIXUpTT9krEeQi6JZ5qLrwOnAO+7+0XAUKBjHM87Cng7ar+K2A3U54WL9yw2\ns+6xXsjMLjOzVWa2qrq6Oo63Fmmd1I6QfumeeyjT4kkKn7t7LbDbzA4BPgRiXrxb4P8BReGMq08B\n98Y6yd3nuXupu5d26dIlSW8tItK0dM89lGlNtikAq8ysE3AXQe+jHcALcTzvHfZNHt3Y26AMgLtv\nidq9m6BqSkQkq0yd2nqTQH3xNDR/N7x7h5n9GTjE3eNZo3kl0NfMehEkg8nAv0afYGZd3f29cHcC\n8HrckYuISNI1NnhtWP0NOBRoG95vlLvvBmYATxBc7B9y9zVm9nMzq2uovsLM1pjZy8AVwLRE/yAR\naX3yafBYpjU4TsHMngnvFgKlwMsEk+ENAVa5+7FpibAejVMQyS/JWrg+3yW88pq7j3X3scB7wLCw\noXc4cAz12gZERFIlGQvXq6QRv3gamo9291frdtz9NTMbkMKYREQiEh08Vr+kUTdNBaikEUs8XVJf\nMbO7zeykcLsLiKehWUQkYYkOHktGSSOfxJMULgLWADPDbW14LGeo6CiSuxIdPJZv01QkKp4uqbuA\nm8It56joKJLb6v6fzpoVXMh79AgSQrz/f3v0iD2hXWudpiJRjfU+esjdzzezV9l3OU4AwlHIadfc\n3ketZYZDEWkZ9V4KJGOW1Jnh7dnJCSkzVHQUyW+JljTyTYNJoW6ksbvH+J2dO1R0FJF8mqYiUY2N\naP7UzLbH2D41s+3pDDIR+TbDoUg2UmeP3NFYSaFDOgNJFRUdRTJLnT1ySzxdUgEwsyPMrEfdlsqg\nkm3q1KBRubY2uNUXUaR5Evmlr3ECuSWeldcmmNl6YDOwHKgAHk9xXCKtTq4ukJPowvXq7JFb4ikp\nXAuMBta5ey+CVdheTGlUIq3QnDmZjqBlEv2ln2/LWea6eJJCTbgYThsza+PuzxDMmioieSDRX/rq\n7JFb4kkKW83sYOBZYKGZ/TfwWWrDEmkdysqCJRzNgv26+7lUlZToL/18W84y1zU4ojlygll7YBfB\nWgpTgY7AwnpLaaaN1lOQXGUW1MnnGo0Ibh0SXk/BzG41szHu/pm773H33e5+r7vfnKmEICLpp1/6\n+aWxaS7WATeYWVfgIeB+d//f9IQl0vrMnp3pCFpOI4LzR2Mrr/13uOTmicAW4B4ze8PMZptZv7RF\nKNJK5FI7guSvJhua3b3S3X/l7scAU4BzgNdTHpmIiKRdPIPX2prZeDNbSDBo7U3g3JRHJiIiaddY\nQ/NpZnYPUAVcCvwP0MfdJ7v7o+kKUCRbqPpH8kFjJYUfA88DA9x9grv/wd01PkFyVqIX9VwdkSzS\nHE2OU8g2GqcgLZXoOIFcHWcgAkkYpyAirWNEskhzKClIq5boRb2sLCgd1JUQ6u4rKUhrpeojyRuq\nPpJ8puojkSTL5IjkRJez1HKYEq/GprkQaVUSvahnqsoo0eUstRymNIeqj0SyXFFRcCGvr2fPYHnZ\nVD9fWgdVH4m0EokucqPlMKU5UpoUzOxMM3vTzDaY2TWNnHeembmZaUU3kXoSXeRGy2FKc6QsKZhZ\nAXAr8DVgIDDFzAbGOK8DMBP4e6pikeygbpwtk+hylloOU5ojlSWFkcAGd9/k7l8ADwATY5x3LfAr\ngtXdpBXTNBEtk+giN1okR5ojlb2PjgLejtqvAkZFn2Bmw4Du7v4/ZnZVCmMRyWmJLnKjRXIkXhlr\naDazNsCNwA/jOPcyM1tlZquqq6tTH5wkjaaJEMktKeuSambHAmXufka4/2MAd/+vcL8jsBHYET7l\ny8DHwAR3b7DPqbqk5i6NCBbJnGzokroS6GtmvczsAGAysLTuQXff5u6Hu3uRuxcBL9JEQhARkdRK\nWVJw993ADOAJguU7H3L3NWb2czObkKr3leyVywvXi+QLjWgWEckD2VB9JJJUapwWST0lBckZGucg\nknpKCiIiEqGkIFlN4xxE0ktJQbJaa1gOUwvcSC7RIjsiKaQFbiTXqKQgOSMXxznMmrU3IdTZuTM4\nLpKNlBTioOJ/dsilKqM6WuBGco2SQhPqiv+VlUFddl3xX4lB4qEFbiTXKCk0QcV/SYQWuJFco6TQ\nBBX/JZHqQy1wI7lGvY+a0KNHUGUU67i0fsnoPaQFbiSXqKTQBBX/85uqDyXfKCk0QcX//KbqQ8k3\nqj6Kg4r/+UvVh5Jv8qqkkIv93CWzVH0o+SavkoKmXpbmUvWh5BtVH4k0QdWHkk9afUlBUy8njz4z\nkdYvL5JCrk+9nC1ytfpNc1eJxE/VR9KqaepqkeZp9SWFaLk49XKm5Xr1mwafiTSPeV29So4oLS31\nVatWZTqMvGS2txouV7RpEztmM6itTX88IpliZqvdvbSp8/KqpCD5R1NXizSPkoLELRer3zT4TKR5\nlBQkbrnSjhBNg89Emke9j6TV0+AzkfippCAiIhFKCnkkF6t/QIPPRNJJSSGP5OKI5LrBZ5WVQdfS\nusFnSgwiqaGkIFlNg89E0iulScHMzjSzN81sg5ldE+Px6Wb2qpmVm9kKMxuYynjyUa6PSNbKZyLp\nlbKkYGYFwK3A14CBwJQYF/0/uHuxu5cA1wM3piqefJUNEwIm0iagwWci6ZXKksJIYIO7b3L3L4AH\ngInRJ7j79qjd9kCOTaIQn3xuKE20TUCDz0TSK5VJ4Sjg7aj9qvDYPszscjPbSFBSuCKF8WRENjWU\nZmJEcqJtAhp8JpJeKZsQz8y+Dpzp7peE+xcAo9x9RgPn/ytwhrtfGOOxy4DLAHr06DG8MtZK6lmq\nqCj2wu89e0JFRfNeq6wsd9oC6mhCOpHskA0T4r0DdI/a7xYea8gDwDmxHnD3ee5e6u6lXbp0SWKI\nqZfMhtJc7FKqNgGR3JLKpLAS6GtmvczsAGAysDT6BDPrG7U7DlifwngS1pJf6fl+UVSbgEhuSVlS\ncPfdwAzgCeB14CF3X2NmPzezCeFpM8xsjZmVAz8A9qs6yiYt+aWe6EUx17uUqk1AJLdokZ1maOki\nMwsXBg2rlZXBRfGXv2zZRTFTi9zUxf/WW0EJp6Xxi0jmZEObQquQjF/qU6fubVSuqEj/BTWRLrHZ\n1HtKRFJPSaEJ2TT4C9J/Udc0EyL5RUkhxRItaURf1CH9F3VNMyGSX9Sm0AyJjhNoSZtAouMcEh0n\nkMxxFiKSOWpTSIFM9PhJ9Jd6ol1i1aVUJL8oKaRRS6aZyPRFXV1KRfKLkkIataSkkQ0X9breU7W1\nmek9JSLp0zbTAUjj6i7AiYwT0ML1IhIvJYUcoIu6iKSLqo9ERCRCSUFERCKUFEREJEJJQUREIpQU\nREQkIuemuTCzaiBb1+M8HPgo00E0QvElJtvjg+yPUfElJpH4erp7k0tX5lxSyGZmtiqeuUUyRfEl\nJtvjg+yPUfElJh3xqfpIREQilBRERCRCSSG55mU6gCYovsRke3yQ/TEqvsSkPD61KYiISIRKCiIi\nEqGk0Eyz/KDqAAAGF0lEQVRm1t3MnjGztWa2xsxmxjjnJDPbZmbl4fazNMdYYWavhu+93zJ1FrjZ\nzDaY2StmNiyNsR0d9bmUm9l2M/t+vXPS/vmZ2T1m9qGZvRZ17FAze8rM1oe3nRt47oXhOevN7MI0\nxfZrM3sj/Pd7xMw6NfDcRr8LKY6xzMzeifp3PKuB555pZm+G38dr0hjfg1GxVZhZeQPPTeln2NA1\nJWPfP3fX1owN6AoMC+93ANYBA+udcxLwpwzGWAEc3sjjZwGPAwaMBv6eoTgLgPcJ+k9n9PMDTgCG\nAa9FHbseuCa8fw3wqxjPOxTYFN52Du93TkNspwNtw/u/ihVbPN+FFMdYBvwoju/ARqA3cADwcv3/\nT6mKr97jvwF+lonPsKFrSqa+fyopNJO7v+fuL4X3PwVeB47KbFTNNhG4zwMvAp3MrGsG4jgF2Oju\nGR+M6O7PAh/XOzwRuDe8fy9wToynngE85e4fu/snwFPAmamOzd2fdPfd4e6LQLdkvmdzNfD5xWMk\nsMHdN7n7F8ADBJ97UjUWn5kZcD5wf7LfNx6NXFMy8v1TUkiAmRUBxwB/j/HwsWb2spk9bmaD0hoY\nOPCkma02s8tiPH4U8HbUfhWZSWyTafg/YiY/vzpHuvt74f33gSNjnJMNn+XFBCW/WJr6LqTajLCK\n654Gqj+y4fM7HvjA3dc38HjaPsN615SMfP+UFFrIzA4GHga+7+7b6z38EkGVyFDgFmBJmsM7zt2H\nAV8DLjezE9L8/k0yswOACcCiGA9n+vPbjwdl9azrqmdms4DdwMIGTsnkd+F2oA9QArxHUEWTjabQ\neCkhLZ9hY9eUdH7/lBRawMzaEfzjLXT3P9Z/3N23u/uO8P5jQDszOzxd8bn7O+Hth8AjBEX0aO8A\n3aP2u4XH0ulrwEvu/kH9BzL9+UX5oK5aLbz9MMY5GfsszWwacDYwNbxo7CeO70LKuPsH7r7H3WuB\nuxp474x+F82sLXAu8GBD56TjM2zgmpKR75+SQjOF9Y+/BV539xsbOOfL4XmY2UiCz3lLmuJrb2Yd\n6u4TNEi+Vu+0pcC3wl5Io4FtUcXUdGnw11kmP796lgJ1vTkuBB6Ncc4TwOlm1jmsHjk9PJZSZnYm\n8H+ACe6+s4Fz4vkupDLG6HaqSQ2890qgr5n1CkuPkwk+93Q5FXjD3atiPZiOz7CRa0pmvn+palFv\nrRtwHEEx7hWgPNzOAqYD08NzZgBrCHpSvAj8Sxrj6x2+78thDLPC49HxGXArQa+PV4HSNH+G7Qku\n8h2jjmX08yNIUO8BNQT1sv8OHAb8FVgP/AU4NDy3FLg76rkXAxvC7aI0xbaBoC657jt4R3juV4DH\nGvsupPHz+134/XqF4ALXtX6M4f5ZBD1uNqYqxljxhccX1H3vos5N62fYyDUlI98/jWgWEZEIVR+J\niEiEkoKIiEQoKYiISISSgoiIRCgpiIhIhJKCSMjM9ti+M7gmbcZOMyuKnqFTJFu1zXQAIlnkc3cv\nyXQQIpmkkoJIE8L59K8P59T/h5l9NTxeZGZPhxO+/dXMeoTHj7RgjYOXw+1fwpcqMLO7wjnznzSz\nL4XnXxHOpf+KmT2QoT9TBFBSEIn2pXrVR9+MemybuxcDc4H/Gx67BbjX3YcQTEh3c3j8ZmC5BxP6\nDSMYCQvQF7jV3QcBW4HzwuPXAMeErzM9VX+cSDw0olkkZGY73P3gGMcrgJPdfVM4cdn77n6YmX1E\nMHVDTXj8PXc/3MyqgW7u/s+o1ygimPe+b7h/NdDO3X9hZn8GdhDMBrvEw8kARTJBJQWR+HgD95vj\nn1H397C3TW8cwVxUw4CV4cydIhmhpCASn29G3b4Q3n+eYFZPgKnA38L7fwW+A2BmBWbWsaEXNbM2\nQHd3fwa4GugI7FdaEUkX/SIR2etLtu/i7X9297puqZ3N7BWCX/tTwmPfA+ab2VVANXBReHwmMM/M\n/p2gRPAdghk6YykAfh8mDgNudvetSfuLRJpJbQoiTQjbFErd/aNMxyKSaqo+EhGRCJUUREQkQiUF\nERGJUFIQEZEIJQUREYlQUhARkQglBRERiVBSEBGRiP8PEnLZyMua1xUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43baa91dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dpt_model_val_loss = dpt_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Again, a clear improvement over the reference network.\n",
    "\n",
    "To recap: here the most common ways to prevent overfitting in neural networks:\n",
    "\n",
    "* Getting more training data.\n",
    "* Reducing the capacity of the network.\n",
    "* Adding weight regularization.\n",
    "* Adding dropout."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}