LeNet-Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LeNet Lab\n",
    "![LeNet Architecture](lenet.png)\n",
    "Source: Yan LeCun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "Load the MNIST data, which comes pre-loaded with TensorFlow.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load pickled data\n",
    "import pickle\n",
    "\n",
    "# TODO: Fill this in based on where you saved the training and testing data\n",
    "\n",
    "training_file = 'train.p'\n",
    "testing_file = 'test.p'\n",
    "\n",
    "with open(training_file, mode='rb') as f:\n",
    "    train = pickle.load(f)\n",
    "with open(testing_file, mode='rb') as f:\n",
    "    test = pickle.load(f)\n",
    "    \n",
    "x_train, y_train           = train['features'], train['labels']\n",
    "x_test, y_test             = test['features'], test['labels']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data that TensorFlow pre-loads comes as 28x28x1 images.\n",
    "\n",
    "However, the LeNet architecture only accepts 32x32xC images, where C is the number of color channels.\n",
    "\n",
    "In order to reformat the MNIST data into a shape that LeNet will accept, we pad the data with two rows of zeros on the top and bottom, and two columns of zeros on the left and right (28+2+2 = 32).\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated Image Shape: (32, 32, 3)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_train, x_validation, y_train, y_validation = train_test_split(x_train, y_train, test_size=0.2, random_state=0)\n",
    "    \n",
    "print(\"Updated Image Shape: {}\".format(x_train[0].shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Data\n",
    "\n",
    "View a sample from the dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFsAAABZCAYAAABR/liSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEPRJREFUeJztnFuMXddZx3/fvp7bnLna41vixDXkAmkLhJTSB5BKBeKl8ACiDxVIoPSlEkg8UPHEYx+AV6REVOIBCSqBlCJFaSMaFNKEkLTkHjt2nNgeezy253bOnNu+LR6+bx87F3smHmfbis9fso5nX9Za+9vf+q/vtrY455igGni3egB3EibCrhATYVeIibArxETYFWIi7AoxEXaF2JWwReR3ROS4iJwUke/crEF9ViE36tSIiA+8A3wNWAJeAr7hnHvr5g3vs4VgF/c+Apx0zp0CEJF/Ab4OXFPYYRS6Wi3GOYcgelA+eI3v+UR2rFSEUVFwLZ0QwLNGRPQ3jGPm9ywAEAQ+AJcvrwFQ4FMU1piUY9AJXhQ5ZUdilxSuIM9TADw76LlCx1cU5C5nMBiRJOmHnuSj2I2wDwJnr/p7CfjShy8SkUeBRwHiOOLhhx8iyXJ8PwTAtwcobKhTzSnuClVALk8AONXtMUw/yHjlk4UUNHx9jDCMADhw+F6++a0/A2BhcQaAxx//VwCGrs3WSNsVT+/zanUAeoMeDAd6rND+hqMtut1LAMTeCIBm3gcgG/boDrd47oVXtxGVYjfC/rg3+RH9c849BjwG0G63nI+Hj8MVqh2p/RZ2fc8NuViLrAMVSv4BzTYtNtnnwKicCagGunhAL94CoOb0Qj/vArByeYOUKQBagfbTu7QCwKBwNKzhaTs3lyWsOW13kOf2TNphIYIXeOMZtR12I+wl4K6r/j4EnL/eDYLgewGe5OQm5JIqPHtPWZqxYeu275cvQq5qo5zm5YxwZF55TGfE8iqcWtd7p+ZUUKEM9Zo8YHHxXgAWmy0Ajp96CYDEBczPLgJweFZfyEy6yWsnLwKQu8zGoEL3/IKQcMfC3o018hLwcyJyr4hEwB8BP9hFe5953LBmO+cyEfk28EPAB77nnHvzuvegWipXlkc8T993ZAtZkTmcLUB+oMPLETybCRQ2lcsGxCMzLQ+MAtIiplPUtI2WamgcaftTfsFcXM6EHgCj0bo27dVIpQ3AalIuzgMS32afjSHwtfNaPWI4KvB2qLK7oRGcc08CT+6mjTsJuxL2jaAoCjzPH5tYiS06Rcm74ghsoYtELRY8Ga+8TkozT9UpB5JMNa5eK005x6icCPEcAFnU0N/8PN311/T/tvC1ax0AfNmis6r8vH5RRRM5h5cr30emwQcX1aycn/U5t/Qeof/pc/YEnxCVa7bD2RvWlZ1CtSvLrnD3TKx8WzO7uedycrEZYHc7d0WbPLOXXemcuJT+UG3hgagN3bzrbu3n/dOEsWrq7Izy+ZGZBwDoL53nhVNnANhIleNHEtAIjKNNgxuejv3IgTnuv/vzPPHUp29n3wAcgsMBcawPMzfXBGBkjsZgMIRI6SM3W9e5HD83Uy9ToUfmyAQOMvMIG57e1w49Fgcq7ENDFdDpgbb/y9MNGoFenw7V9l7QU+zdv498VSklMdE0mlNspaoQG4na7s2WUcx0wPRsHd+csO0woZEKUalmOwd5liPikxWqDamtZO2WangtDpmfmQYg8pVO8qygyM0cNDsrFr0/9H0Co5FaEAPQqM0x8/6qdnpcIwoPmim4QQPU68avaT/emqp20u/whUiP+b5eJOSMDhwB4O2+xleyurn7daHvOhTm5GyHiWZXiEo1WwDPOfAgCFTTMnNgtgaqSXum2jzSngXggK/a6+UOP1futbUKPy9tuwJniuWMW5PeOkVHA0rnjh8DIMyUn8PeOkmq9w4sulBG+AInFIlyfZFr8Cn3Aqa+oovrUkc5u/C0n/uCab58YJFmGO7o+Su3RnxPyFxOmupUzApd2YNQpThTb0OmU5nOaQCK80tQ6AN5scYz8lgpQwIPP9BzQaTH4oaPH+sbqMV6rhFZyDXaj1/TFy21xgfa9F3E8HV1gt879rzeN9ekkyp9XOpowKpvirF0ts4FIE0mNHLb4RbY2UJRFIwytXVLD7LmVAPXtvr8b6wxi4di1bzDqRBGagYe/NJvARDfr7axTPt4TYuD1PV6P/AJbGqH9huZzR6EMV5pqtkxfPtd77C2tQGAd1KpLJ6a49K6LraFjTUwG/+dE+dJekO6w3RHzz7R7ApRrekHGqETwfMsgmbB/TJVs9W7wElUs2cXDgOwv3WQ3FaxhSO/BsDMlzUpJPsA39rwrhejuN45i/B1z3L60tsArMfzAPRbd3Gu/642n+kM8G3svU3HSlyQZjvL4040u0JUztmIwyH4XumUWBTPNHyUDNna3ARgdVrNt8FCm9G6WgIXNs4BEA9U+xve9O5VZmTu+PMvcXZlGYDpL/wKAK+5glMblu0xrraEDf2NhHW3RZ4V7ASVCtsXYSaO2BwlYB5gPdQh9Cx2kRNQOmRnVi8AcPLBIxwJ1Dx7+9kntK1FDfIfPfAbeK36rsaVndd4yPJPXqUxUMFl/SUAlrpbbJXxGHup/ZGafsMsY3WwwSjNdtTPhEYqRLWa7QmtKKQ7THGm2WZNkZlHWDhHaBSz2lNv7sWVHkcPq6kXb6rT8c7T/wHA3F33MP/wfQBIsLMg/hVY1vy95wBIli8Rh5rDXjXHpdPtkhjFlctgYl5vP83o9zOyfLJA3nao3PTLfZ8cB5Y8tbwqrozqiaOwlJkzjUo3urxaV2fjaFtd+fnLmr5afuoFpvepmRbes+eTDchSYO89/zQAgyJhdd9BAF7pqLnXSTOcxc7Tcqx+WTeSk2QjdlrCV30OUsCRl7ImLGtEbI6J7zPKLV5Slntl65zeVL4JZ1QYC5EumOdP/pi5V/cBcGDPb2sbzW0mrDqvbLyoQarVY7oYzsxNcdLXRfl0Ty2iAh+MNsrYV2nP12qOvEjGPsN2mNBIhahcs8UVeJ43DmtGli3Px4UkEJbFjVaUJpKy1lcaudjS6N1wVjV7Y+UY86fU69u7+usAhPXr2d4ONjXW0XnrDT3UtTTXTJ3uqhV1WQ7TR67QGlaDaJo9PzvFgbkar0Tv7ujZJ5pdISpP+OZ5hu975BYLKZO1uZmCfihMt1Vry+RuEARg113aVE49P6/1enP7Fzjx0gsAzBx+CICj+756xQP5yBBS1s78DwDvH9PffkNjMO9GMRds4S0LP/04GnO2Py7qtLGKY3a2he/vTGe3vUpE7hKRZ0TkbRF5U0T+3I7PicjTInLCfmd31OMdjJ1odgb8pXPuZyIyBfxURJ4G/gT4T+fcd22Lx3eAv7peQ3kB/WEOLhg7CKmRt7NMTWu2xeJeNe8GXeXpdJjQQLMwna7GRJ5d1hjG147ez+GRxk3efPL7AMzuW2Tukc8DIOUTlsXty6ss/fgn1q62Ff+8Vk398OIZVhING/iRxsgb01P0+2qZBOa8eF5Z+5fhBSE7LGLdXtjOuWVg2f7fFZG30UL4rwO/aZf9E/BfbCNsQfAkIs+ScdJAzGZtTWn4shEJ5daD5rwmAy6f3SQ1u0vKMjW7f5iHpLbLoH/6FADHf/RTHpo5BMDUfVoMX9LQ1pun2XhLg1kNX1/g+Y0TAKyuXhgXSTatRnzvzBSJJv5Z29Q8ZmZJz/mZFvctLlDbYQ7yEy2QInIP8EvAi8CivYjyhey9xj2PisjLIvJyYnnHOxU7XiBFpAX8G/AXzrnOTgvAr955MDvddnHoMcxhlKnj4lnVUxTZVo0og8j2sFjdSBjVcImVfrX0+ta0/ubRGsuF1Ys01ZPsLv8fyy9rVLBhjg6xxjrWjz9D0tW+w0V1kNb7SkNhIOP4aWTXN5oZbavQcjbW/kivybKUC+dWSJObmBYTkRAV9D875/7dDq+IyH47vx+4uKMe72Bsq9miKvyPwNvOub+/6tQPgD8Gvmu/T2zXlgfEAuJycnNYMuPentVy1PC1tgRNzgLUmk1aoR7bO63FkPm0modrRZeBJWwfbOu5zpljLJ/Qew899IsAuMH7AJx8/Vm8fWo2XphV7V+6qOZk3KrhWySQQLV3VCtoWj3iQt1m00jHnCQJa8MR2U3cU/MV4JvA6yLyih37a1TI3xeRPwXOAH+wox7vYOzEGnmOa2dLv/pJOnNA7gryvLiyF88UKRuqZl++mJFsqKu8Z0bTYgcbMftn1TLZM6fm/InL6nKvDoaIr1p4z16NRc8W81w4pdsx1/77KQBGF/TvRtKldkjbfzfVOsAtu7851WbY1y0fQawMKwH4Zm0sLOhs6Vll1GgU49fr4+0o26HyEOtIBIIAPynrRWzX2EhDcaO0wPX14YeWZ2zde4D5BbWFz1xS2/vcuct6fVFQa+qLe7Wm9vCR1j4e6KuAzj+j4dNoqKmvdruBbznDfldNuYEtvsNhQmzUNTWt1BHhkXb1pTdb2ma5W6y7uUExcLhJWuz2Q+WanRaOosivxB5qFpj3bGdBIdStZq/dtjqNRoPlFcuuL+sU7isTEEcRU7a4rq0otaR+jUPz9wCQbGiBZG2oXmnz7l/ltNNj723q7Fi3EmKShP0HdQZJuQcnLbjYsQL5XMczSstZmJEPUwo3iWffdqg+no2HL/4429E35yZoqNZMNRssTKlZJ049znfPr9GwGEo2KBPD2t5ce56WlRaUHLzuneWthq68h+Y1VfbA51RTzyxk/OgNjVlvmQlXt52+jZmYQaZTxnX1/oYnbJhmb46sktaifqQBQfHB/T3XQ+X12QGCB4S2yVSsBrtu5bsz0232mA3d29KHzFxAzby41DYfYXGKy511embZdMyzK/wRS0N9eeEBDZ8efFC3SWe9FXKrwfYsrNtoat3J3EIbrPZ6ZB8MGCWj8YZV28nH0BZEKXzyXD66Yf8amNBIhbgFabGcoCgIjRZS04uyHC0OfFIrJ0aURpr1OmG5XW9ah9xNVOsHvT6JfR7DFJs8h5UtpZZsoAd/4ah6kvE5n6n4HQCcLXSFFeSLH1HqX2g05zdqDLpaDF9uASztag8YbmxxzY+hfAgTza4QlWq2J0I9CpgO6+PY8LqZTWXkrNft0rFFcO+CfYYijtgsTcW6moXTDdV0+j0G5ewo+yEgtI/H9DJLv02rSdfozdOyj7mILZBD4+fNzS6dddXitvH4nrkpvKY6OJl9YiMvK6RGI5q+XL9S+ern39llE9wMVMvZAn7kE+LGVkiels6McXiaj7/dFJiuNoKAkQVRthLV8LihTkqtKYjt2O1tKQc7ZKy9iVdyvPJyq1Ybc64f2xd7nPL76sVLJOYt+ZboaMcydq7K3Rwd6ycdjJiKo/FHC7ZDpcLOC0dnMGToOQZGH1kxLjMCwOUesZlyZUCq8IRpqxMZWMFj32JjYas1FlbPFlTnHFLmEi0ZMBzq32FYp7B537NzXlxS0oDA+pastOsTmnPatxso3Yi9kHRQsF7zx9872Q4TGqkQN/xdvxvqTOQS0AMuV9bpjWOBnY/zsHNu26rOSoUNICIvO+cerrTTG8CnMc4JjVSIibArxK0Q9mO3oM8bwU0fZ+WcfSdjQiMVojJh387f2r5Ope7fiMg5EXnF/v3urvqpgkZu929tW0XX/qsrdYHfA/4Q2HLO/e3N6KcqzR5/a9s5lwDlt7ZvCzjnlp1zP7P/d4GyUvemoiphf9y3tm/6w9wMfKhSF+DbIvKaiHxvtwX/VQl7R9/avtX4cKUu8A/A54AvojXqf7eb9qsS9if+1nbV+LhKXefcinMud/rZ48dROrxhVCXs2/pb29eq1C1Log2/D7yxm34qiWffyLe2K8a1KnW/ISJfRCnvfeBbu+lk4kFWiIkHWSEmwq4QE2FXiImwK8RE2BViIuwKMRF2hZgIu0L8P8Rl8UsN/7J7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1048764e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.gridspec as gridspec\n",
    "import os\n",
    "import cv2\n",
    "from math import floor, ceil, pi\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "index = random.randint(0, len(x_train))\n",
    "image = x_train[index].squeeze()\n",
    "\n",
    "plt.figure(figsize=(1,1))\n",
    "plt.imshow(image)\n",
    "print(y_train[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess Data\n",
    "\n",
    "Shuffle the training data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "x_train, y_train = shuffle(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup TensorFlow\n",
    "The `EPOCH` and `BATCH_SIZE` values affect the training speed and model accuracy.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from sklearn.utils import shuffle\n",
    "from skimage import exposure\n",
    "\n",
    "def preprocess_dataset(X, y = None):\n",
    "    #Convert to grayscale, e.g. single Y channel\n",
    "    X = 0.299 * X[:, :, :, 0] + 0.587 * X[:, :, :, 1] + 0.114 * X[:, :, :, 2]\n",
    "    #Scale features to be in [0, 1]\n",
    "    X = (X / 255.).astype(np.float32)\n",
    "      \n",
    "    # Apply localized histogram localization  \n",
    "    for i in range(X.shape[0]):\n",
    "        X[i] = exposure.equalize_adapthist(X[i])\n",
    "        \n",
    "    if y is not None:  \n",
    "        # Convert to one-hot encoding. Convert back with `y = y.nonzero()[1]`\n",
    "        y = np.eye(43)[y]\n",
    "        # Shuffle the data\n",
    "        X, y = shuffle(X, y)\n",
    "\n",
    "    # Add a single grayscale channel\n",
    "    X = X.reshape(X.shape + (1,)) \n",
    "    return X, y\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "def flip_extend(X, y):\n",
    "    # Classes of signs that, when flipped horizontally, should still be classified as the same class\n",
    "    self_flippable_horizontally = np.array([11, 12, 13, 15, 17, 18, 22, 26, 30, 35])\n",
    "    # Classes of signs that, when flipped vertically, should still be classified as the same class\n",
    "    self_flippable_vertically = np.array([1, 5, 12, 15, 17])\n",
    "    # Classes of signs that, when flipped horizontally and then vertically, should still be classified as the same class\n",
    "    self_flippable_both = np.array([32, 40])\n",
    "    # Classes of signs that, when flipped horizontally, would still be meaningful, but should be classified as some other class\n",
    "    cross_flippable = np.array([\n",
    "        [19, 20], \n",
    "        [33, 34], \n",
    "        [36, 37], \n",
    "        [38, 39],\n",
    "        [20, 19], \n",
    "        [34, 33], \n",
    "        [37, 36], \n",
    "        [39, 38],   \n",
    "    ])\n",
    "    num_classes = 43\n",
    "    \n",
    "    X_extended = np.empty([0, X.shape[1], X.shape[2], X.shape[3]], dtype = X.dtype)\n",
    "    y_extended = np.empty([0], dtype = y.dtype)\n",
    "    \n",
    "    for c in range(num_classes):\n",
    "        # First copy existing data for this class\n",
    "        X_extended = np.append(X_extended, X[y == c], axis = 0)\n",
    "        # If we can flip images of this class horizontally and they would still belong to said class...\n",
    "        if c in self_flippable_horizontally:\n",
    "            # ...Copy their flipped versions into extended array.\n",
    "            X_extended = np.append(X_extended, X[y == c][:, :, ::-1, :], axis = 0)\n",
    "        # If we can flip images of this class horizontally and they would belong to other class...\n",
    "        if c in cross_flippable[:, 0]:\n",
    "            # ...Copy flipped images of that other class to the extended array.\n",
    "            flip_class = cross_flippable[cross_flippable[:, 0] == c][0][1]\n",
    "            X_extended = np.append(X_extended, X[y == flip_class][:, :, ::-1, :], axis = 0)\n",
    "        # Fill labels for added images set to current class.\n",
    "        y_extended = np.append(y_extended, np.full((X_extended.shape[0] - y_extended.shape[0]), c, dtype = int))\n",
    "        \n",
    "        # If we can flip images of this class vertically and they would still belong to said class...\n",
    "        if c in self_flippable_vertically:\n",
    "            # ...Copy their flipped versions into extended array.\n",
    "            X_extended = np.append(X_extended, X_extended[y_extended == c][:, ::-1, :, :], axis = 0)\n",
    "        # Fill labels for added images set to current class.\n",
    "        y_extended = np.append(y_extended, np.full((X_extended.shape[0] - y_extended.shape[0]), c, dtype = int))\n",
    "        \n",
    "        # If we can flip images of this class horizontally AND vertically and they would still belong to said class...\n",
    "        if c in self_flippable_both:\n",
    "            # ...Copy their flipped versions into extended array.\n",
    "            X_extended = np.append(X_extended, X_extended[y_extended == c][:, ::-1, ::-1, :], axis = 0)\n",
    "        # Fill labels for added images set to current class.\n",
    "        y_extended = np.append(y_extended, np.full((X_extended.shape[0] - y_extended.shape[0]), c, dtype = int))\n",
    "    \n",
    "    return (X_extended, y_extended)\n",
    "\n",
    "\n",
    "from skimage.transform import rotate\n",
    "from skimage.transform import warp\n",
    "from skimage.transform import ProjectiveTransform\n",
    "\n",
    "def rotate(X, intensity):\n",
    "    for i in range(X.shape[0]):\n",
    "        delta = 30. * intensity # scale using augmentation intensity\n",
    "        X[i] = rotate(X[i], random.uniform(-delta, delta), mode = 'edge')\n",
    "    return X  \n",
    "\n",
    "def apply_projection_transform(X, intensity):\n",
    "    image_size = X.shape[1]\n",
    "    d = image_size * 0.3 * intensity\n",
    "    for i in range(X.shape[0]):\n",
    "        tl_top = random.uniform(-d, d)     # Top left corner, top margin\n",
    "        tl_left = random.uniform(-d, d)    # Top left corner, left margin\n",
    "        bl_bottom = random.uniform(-d, d)  # Bottom left corner, bottom margin\n",
    "        bl_left = random.uniform(-d, d)    # Bottom left corner, left margin\n",
    "        tr_top = random.uniform(-d, d)     # Top right corner, top margin\n",
    "        tr_right = random.uniform(-d, d)   # Top right corner, right margin\n",
    "        br_bottom = random.uniform(-d, d)  # Bottom right corner, bottom margin\n",
    "        br_right = random.uniform(-d, d)   # Bottom right corner, right margin\n",
    "\n",
    "        transform = ProjectiveTransform()\n",
    "        transform.estimate(np.array((\n",
    "                (tl_left, tl_top),\n",
    "                (bl_left, image_size - bl_bottom),\n",
    "                (image_size - br_right, image_size - br_bottom),\n",
    "                (image_size - tr_right, tr_top)\n",
    "            )), np.array((\n",
    "                (0, 0),\n",
    "                (0, image_size),\n",
    "                (image_size, image_size),\n",
    "                (image_size, 0)\n",
    "            )))\n",
    "        X[i] = warp(X[i], transform, output_shape=(image_size, image_size), order = 1, mode = 'edge')\n",
    "\n",
    "    return X\n",
    "\n",
    "\n",
    "EPOCHS = 30\n",
    "BATCH_SIZE = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Implement LeNet-5\n",
    "Implement the [LeNet-5](http://yann.lecun.com/exdb/lenet/) neural network architecture.\n",
    "\n",
    "This is the only cell you need to edit.\n",
    "### Input\n",
    "The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.\n",
    "\n",
    "### Architecture\n",
    "**Layer 1: Convolutional.** The output shape should be 28x28x6.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 14x14x6.\n",
    "\n",
    "**Layer 2: Convolutional.** The output shape should be 10x10x16.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 5x5x16.\n",
    "\n",
    "**Flatten.** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using `tf.contrib.layers.flatten`, which is already imported for you.\n",
    "\n",
    "**Layer 3: Fully Connected.** This should have 120 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 4: Fully Connected.** This should have 84 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 5: Fully Connected (Logits).** This should have 10 outputs.\n",
    "\n",
    "### Output\n",
    "Return the result of the 2nd fully connected layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.contrib.layers import flatten\n",
    "keep_prob = tf.placeholder_with_default(1.0, shape=())\n",
    "\n",
    "def LeNet(x):    \n",
    "    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer\n",
    "    mu = 0\n",
    "    sigma = 0.1\n",
    "    \n",
    "    # TODO: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
    "\n",
    "    # Layer 1 - 28*28*1 to 14*14*32\n",
    "    # FORMULA: new_height = (input_height - filter_height + 2 * P)/S + 1\n",
    "    # FORMULA: new_width = (input_width - filter_width + 2 * P)/S + 1\n",
    "    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 6), mean = mu, stddev = sigma))\n",
    "    conv1_b = tf.Variable(tf.zeros(6))\n",
    "    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "    \n",
    "    # TODO: Activation.\n",
    "    conv1 = tf.nn.relu(conv1)\n",
    "    conv1 = tf.nn.dropout(conv1, keep_prob)\n",
    "    \n",
    "    # TODO: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
    "    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "    \n",
    "    # TODO: Layer 2: Convolutional. Output = 10x10x16.\n",
    "    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))\n",
    "    conv2_b = tf.Variable(tf.zeros(16))\n",
    "    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    \n",
    "    # TODO: Activation.\n",
    "    conv2 = tf.nn.relu(conv2)\n",
    "    conv2 = tf.nn.dropout(conv2, keep_prob)\n",
    "    \n",
    "    # TODO: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
    "    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "    \n",
    "    # TODO: Flatten. Input = 5x5x16. Output = 400.\n",
    "    fc0   = flatten(conv2)\n",
    "        \n",
    "    # TODO: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
    "    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))\n",
    "    fc1_b = tf.Variable(tf.zeros(120))\n",
    "    fc1   = tf.matmul(fc0, fc1_W) + fc1_b\n",
    "    \n",
    "    # TODO: Activation.\n",
    "    fc1    = tf.nn.relu(fc1)\n",
    "    \n",
    "    # TODO: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
    "    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))\n",
    "    fc2_b  = tf.Variable(tf.zeros(84))\n",
    "    fc2    = tf.matmul(fc1, fc2_W) + fc2_b\n",
    "    \n",
    "    # TODO: Activation.\n",
    "    fc2    = tf.nn.relu(fc2)\n",
    "    \n",
    "    # TODO: Layer 5: Fully Connected. Input = 84. Output = 10.\n",
    "    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))\n",
    "    fc3_b  = tf.Variable(tf.zeros(43))\n",
    "    logits = tf.matmul(fc2, fc3_W) + fc3_b\n",
    "    \n",
    "    return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features and Labels\n",
    "Train LeNet to classify [MNIST](http://yann.lecun.com/exdb/mnist/) data.\n",
    "\n",
    "`x` is a placeholder for a batch of input images.\n",
    "`y` is a placeholder for a batch of output labels.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (None, 32, 32, 3))\n",
    "y = tf.placeholder(tf.int32, (None))\n",
    "one_hot_y = tf.one_hot(y, 43)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Pipeline\n",
    "Create a training pipeline that uses the model to classify MNIST data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "rate = 0.001\n",
    "\n",
    "logits = LeNet(x)\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)\n",
    "loss_operation = tf.reduce_mean(cross_entropy)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = rate)\n",
    "training_operation = optimizer.minimize(loss_operation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "Evaluate how well the loss and accuracy of the model for a given dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))\n",
    "accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "def evaluate(X_data, y_data):\n",
    "    num_examples = len(X_data)\n",
    "    total_accuracy = 0\n",
    "    sess = tf.get_default_session()\n",
    "    for offset in range(0, num_examples, BATCH_SIZE):\n",
    "        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]\n",
    "        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "        total_accuracy += (accuracy * len(batch_x))\n",
    "    return total_accuracy / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "Run the training data through the training pipeline to train the model.\n",
    "\n",
    "Before each epoch, shuffle the training set.\n",
    "\n",
    "After each epoch, measure the loss and accuracy of the validation set.\n",
    "\n",
    "Save the model after training.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./lenet\n",
      "Training...\n",
      "\n",
      "EPOCH 1 ...\n",
      "Validation Accuracy = 0.915\n",
      "\n",
      "EPOCH 2 ...\n",
      "Validation Accuracy = 0.933\n",
      "\n",
      "EPOCH 3 ...\n",
      "Validation Accuracy = 0.930\n",
      "\n",
      "EPOCH 4 ...\n",
      "Validation Accuracy = 0.937\n",
      "\n",
      "EPOCH 5 ...\n",
      "Validation Accuracy = 0.936\n",
      "\n",
      "EPOCH 6 ...\n",
      "Validation Accuracy = 0.940\n",
      "\n",
      "EPOCH 7 ...\n",
      "Validation Accuracy = 0.940\n",
      "\n",
      "EPOCH 8 ...\n",
      "Validation Accuracy = 0.939\n",
      "\n",
      "EPOCH 9 ...\n",
      "Validation Accuracy = 0.947\n",
      "\n",
      "EPOCH 10 ...\n",
      "Validation Accuracy = 0.924\n",
      "\n",
      "EPOCH 11 ...\n",
      "Validation Accuracy = 0.949\n",
      "\n",
      "EPOCH 12 ...\n",
      "Validation Accuracy = 0.949\n",
      "\n",
      "EPOCH 13 ...\n",
      "Validation Accuracy = 0.951\n",
      "\n",
      "EPOCH 14 ...\n",
      "Validation Accuracy = 0.955\n",
      "\n",
      "EPOCH 15 ...\n",
      "Validation Accuracy = 0.937\n",
      "\n",
      "EPOCH 16 ...\n",
      "Validation Accuracy = 0.942\n",
      "\n",
      "EPOCH 17 ...\n",
      "Validation Accuracy = 0.960\n",
      "\n",
      "EPOCH 18 ...\n",
      "Validation Accuracy = 0.961\n",
      "\n",
      "EPOCH 19 ...\n",
      "Validation Accuracy = 0.949\n",
      "\n",
      "EPOCH 20 ...\n",
      "Validation Accuracy = 0.962\n",
      "\n",
      "EPOCH 21 ...\n",
      "Validation Accuracy = 0.957\n",
      "\n",
      "EPOCH 22 ...\n",
      "Validation Accuracy = 0.957\n",
      "\n",
      "EPOCH 23 ...\n",
      "Validation Accuracy = 0.959\n",
      "\n",
      "EPOCH 24 ...\n",
      "Validation Accuracy = 0.967\n",
      "\n",
      "EPOCH 25 ...\n",
      "Validation Accuracy = 0.969\n",
      "\n",
      "EPOCH 26 ...\n",
      "Validation Accuracy = 0.964\n",
      "\n",
      "EPOCH 27 ...\n",
      "Validation Accuracy = 0.956\n",
      "\n",
      "EPOCH 28 ...\n",
      "Validation Accuracy = 0.954\n",
      "\n",
      "EPOCH 29 ...\n",
      "Validation Accuracy = 0.965\n",
      "\n",
      "EPOCH 30 ...\n",
      "Validation Accuracy = 0.964\n",
      "\n",
      "Model saved\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "    num_examples = len(x_train)\n",
    "    \n",
    "    print(\"Training...\")\n",
    "    print()\n",
    "    for i in range(EPOCHS):\n",
    "        x_train, y_train = shuffle(x_train, y_train)\n",
    "        for offset in range(0, num_examples, BATCH_SIZE):\n",
    "            end = offset + BATCH_SIZE\n",
    "            batch_x, batch_y = x_train[offset:end], y_train[offset:end]\n",
    "            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 0.5})\n",
    "            \n",
    "        validation_accuracy = evaluate(x_validation, y_validation)\n",
    "        print(\"EPOCH {} ...\".format(i+1))\n",
    "        print(\"Validation Accuracy = {:.3f}\".format(validation_accuracy))\n",
    "        print()\n",
    "        \n",
    "    saver.save(sess, './lenet')\n",
    "    print(\"Model saved\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model\n",
    "Once you are completely satisfied with your model, evaluate the performance of the model on the test set.\n",
    "\n",
    "Be sure to only do this once!\n",
    "\n",
    "If you were to measure the performance of your trained model on the test set, then improve your model, and then measure the performance of your model on the test set again, that would invalidate your test results. You wouldn't get a true measure of how well your model would perform against real data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./lenet\n",
      "Test Accuracy = 0.846\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "\n",
    "    test_accuracy = evaluate(x_test, y_test)\n",
    "    print(\"Test Accuracy = {:.3f}\".format(test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}