Convolutional_Neural_Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# What are Convolutional Neural Networks?\n",
    "\n",
    "Convolutional Neural Networks (ConvNets or CNNs) are a category of Neural Networks that have proven very effective in areas such as image recognition and classification. ConvNets have been successful in identifying faces, objects and traffic signs apart from powering vision in robots and self driving cars.\n",
    "\n",
    "\n",
    "A Convolutional Neural Network (CNN) is comprised of one or more convolutional layers (often with a subsampling step) and then followed by one or more fully connected layers as in a standard multilayer neural network. The architecture of a CNN is designed to take advantage of the 2D structure of an input image (or other 2D input such as a speech signal). This is achieved with local connections and tied weights followed by some form of pooling which results in translation invariant features. Another benefit of CNNs is that they are easier to train and have many fewer parameters than fully connected networks with the same number of hidden units. In this article we will discuss the architecture of a CNN and the back propagation algorithm to compute the gradient with respect to the parameters of the model in order to use gradient based optimization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intuition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's develop better intuition for how Convolutional Neural Networks (CNN) work. We'll examine how humans classify images, and then see how CNNs use similar approaches.\n",
    "\n",
    "Let’s say we wanted to classify the following image of a dog as a Golden Retriever:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377b77_dog-1210559-1280/dog-1210559-1280.jpg\" width=\"500\" height=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As humans, how do we do this?\n",
    "\n",
    "One thing we do is that we identify certain parts of the dog, such as the nose, the eyes, and the fur. We essentially break up the image into smaller pieces, recognize the smaller pieces, and then combine those pieces to get an idea of the overall dog.\n",
    "\n",
    "In this case, we might break down the image into a combination of the following:\n",
    "\n",
    "* A nose\n",
    "* Two eyes\n",
    "* Golden fur\n",
    "\n",
    "These pieces can be seen below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377bdb_screen-shot-2016-11-24-at-12.49.08-pm/screen-shot-2016-11-24-at-12.49.08-pm.png\" width=\"250\" height=\"250\">\n",
    "<center>The eye of the dog.</center>\n",
    "<br>\n",
    "\n",
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377bed_screen-shot-2016-11-24-at-12.49.43-pm/screen-shot-2016-11-24-at-12.49.43-pm.png\" width=\"250\" height=\"250\">\n",
    "<center>The nose of the dog.</center>\n",
    "<br>\n",
    "\n",
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377bff_screen-shot-2016-11-24-at-12.50.54-pm/screen-shot-2016-11-24-at-12.50.54-pm.png\" width=\"250\" height=\"250\">\n",
    "<center>The fur of the dog.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going One Step Further"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But let’s take this one step further. How do we determine what exactly a nose is? A Golden Retriever nose can be seen as an oval with two black holes inside it. Thus, one way of classifying a Retriever’s nose is to to break it up into smaller pieces and look for black holes (nostrils) and curves that define an oval as shown below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377c52_screen-shot-2016-11-24-at-12.51.47-pm/screen-shot-2016-11-24-at-12.51.47-pm.png\">\n",
    "<center>A curve that we can use to determine a nose</center>\n",
    "<br>\n",
    "\n",
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377c68_screen-shot-2016-11-24-at-12.51.51-pm/screen-shot-2016-11-24-at-12.51.51-pm.png\">\n",
    "<center>A nostril that we can use to classify the nose of the dog</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadly speaking, this is what a CNN learns to do. It learns to recognize basic lines and curves, then shapes and blobs, and then increasingly complex objects within the image. Finally, the CNN classifies the image by combining the larger, more complex objects.\n",
    "\n",
    "In our case, the levels in the hierarchy are:\n",
    "\n",
    "* Simple shapes, like ovals and dark circles\n",
    "* Complex objects (combinations of simple shapes), like eyes, nose, and fur\n",
    "* The dog as a whole (a combination of complex objects)\n",
    "\n",
    "With deep learning, we don't actually program the CNN to recognize these specific features. Rather, the CNN learns on its own to recognize such objects through forward propagation and backpropagation!\n",
    "\n",
    "It's amazing how well a CNN can learn to classify images, even though we never program the CNN with information about specific features to look for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583cb19d_heirarchy-diagram/heirarchy-diagram.jpg)\n",
    "<center>An example of what each layer in a CNN might recognize when classifying a picture of a dog</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A CNN might have several layers, and each layer might capture a different level in the hierarchy of objects. The first layer is the lowest level in the hierarchy, where the CNN generally classifies small parts of the image into simple shapes like horizontal and vertical lines and simple blobs of colors. The subsequent layers tend to be higher levels in the hierarchy and generally classify more complex ideas like shapes (combinations of lines), and eventually full objects like dogs.\n",
    "\n",
    "Once again, the CNN **learns all of this on its own**. We don't ever have to tell the CNN to go looking for lines or curves or noses or fur. The CNN just learns from the training set and discovers which characteristics of a Golden Retriever are worth looking for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Breaking up an Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step for a CNN is to break up the image into smaller pieces. We do this by selecting a width and height that defines a filter.\n",
    "\n",
    "The filter looks at small pieces, or patches, of the image. These patches are the same size as the filter. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377d67_vlcsnap-2016-11-24-15h52m47s438/vlcsnap-2016-11-24-15h52m47s438.png\" width=600 height=400>\n",
    "<center>A CNN uses filters to split an image into smaller patches. The size of these patches matches the filter size.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then simply slide this filter horizontally or vertically to focus on a different piece of the image.\n",
    "\n",
    "The amount by which the filter slides is referred to as the **'stride'**. The stride is a hyperparameter which the engineer can tune. Increasing the stride reduces the size of your model by reducing the number of total patches each layer observes. However, this usually comes with a reduction in accuracy.\n",
    "\n",
    "Let’s look at an example. In this zoomed in image of the dog, we first start with the patch outlined in red. The width and height of our filter define the size of this square."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/December/5840fdac_retriever-patch/retriever-patch.png\" width=500 height=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then move the square over to the right by a given stride (2 in this case) to get another patch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/December/5840fe04_retriever-patch-shifted/retriever-patch-shifted.png\" width=500 height=500>\n",
    "<center>We move our square to the right by two pixels to create another patch.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's important here is that we are **grouping together adjacent pixels** and treating them as a collective.\n",
    "\n",
    "In a normal, non-convolutional neural network, we would have ignored this adjacency. In a normal network, we would have connected every pixel in the input image to a neuron in the next layer. In doing so, we would not have taken advantage of the fact that pixels in an image are close together for a reason and have special meaning.\n",
    "\n",
    "By taking advantage of this local structure, our CNN learns to classify local patterns, like shapes and objects, in an image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter Depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common to have more than one filter. Different filters pick up different qualities of a patch. For example, one filter might look for a particular color, while another might look for a kind of object of a specific shape. The amount of filters in a convolutional layer is called the **filter depth**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377e4f_neilsen-pic/neilsen-pic.png)\n",
    "<center>In the above example, a patch is connected to a neuron in the next layer. Source: MIchael Neilsen.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many neurons does each patch connect to?\n",
    "\n",
    "That’s dependent on our filter depth. If we have a depth of `k`, we connect each patch of pixels to `k` neurons in the next layer. This gives us the height of `k` in the next layer, as shown below. In practice, `k` is a hyperparameter we tune, and most CNNs tend to pick the same starting values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/December/5840ffda_filter-depth/filter-depth.png\" width=\"300\" height=\"500\">\n",
    "<center>Choosing a filter depth of k connects each path to k neurons in the next layer</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But why connect a single patch to multiple neurons in the next layer? Isn’t one neuron good enough?\n",
    "\n",
    "Multiple neurons can be useful because a patch can have multiple interesting characteristics that we want to capture.\n",
    "\n",
    "For example, one patch might include some white teeth, some blonde whiskers, and part of a red tongue. In that case, we might want a filter depth of at least three - one for each of teeth, whiskers, and tongue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/December/584104c8_teeth-whiskers-tongue/teeth-whiskers-tongue.png\" width=\"350\" height=\"400\">\n",
    "<center>This patch of the dog has many interesting features we may want to capture. These include the presence of teeth, the presence of whiskers, and the pink color of the tongue.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having multiple neurons for a given patch ensures that our CNN can learn to capture whatever characteristics the CNN learns are important.\n",
    "\n",
    "Remember that the CNN isn't \"programmed\" to look for certain characteristics. Rather, it **learns on its own** which characteristics to notice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter Sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58377f77_vlcsnap-2016-11-24-16h01m35s262/vlcsnap-2016-11-24-16h01m35s262.png\" width=600 height=400>\n",
    "<center>The weights, w, are shared across patches for a given layer in a CNN to detect the cat above regardless of where in the image it is located.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are trying to classify a picture of a cat, we don’t care where in the image a cat is. If it’s in the top left or the bottom right, it’s still a cat in our eyes. We would like our CNNs to also possess this ability known as translation invariance. How can we achieve this?\n",
    "\n",
    "As we saw earlier, the classification of a given patch in an image is determined by the weights and biases corresponding to that patch.\n",
    "\n",
    "If we want a cat that’s in the top left patch to be classified in the same way as a cat in the bottom right patch, we need the weights and biases corresponding to those patches to be the same, so that they are classified the same way.\n",
    "\n",
    "This is exactly what we do in CNNs. The weights and biases we learn for a given output layer are shared across all patches in a given input layer. Note that as we increase the depth of our filter, the number of weights and biases we have to learn still increases, as the weights aren't shared across the output channels.\n",
    "\n",
    "There’s an additional benefit to sharing our parameters. If we did not reuse the same weights across all patches, we would have to learn new parameters for every single patch and hidden layer neuron pair. This does not scale well, especially for higher fidelity images. Thus, sharing parameters not only helps us with translation invariance, but also gives us a smaller, more scalable model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/5837d4d5_screen-shot-2016-11-24-at-10.05.37-pm/screen-shot-2016-11-24-at-10.05.37-pm.png)\n",
    "<center>A 5x5 grid with a 3x3 filter. Source: Andrej Karpathy.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we have a `5x5` grid (as shown above) and a filter of size `3x3` with a stride of `1`. What's the width and height of the next layer? We see that we can fit at most three patches in each direction, giving us a dimension of `3x3` in our next layer. As we can see, the width and height of each subsequent layer decreases in such a scheme.\n",
    "\n",
    "In an ideal world, we'd be able to maintain the same width and height across layers so that we can continue to add layers without worrying about the dimensionality shrinking and so that we have consistency. How might we achieve this? One way is to simple add a border of `0`s to our original `5x5` image. You can see what this looks like in the below image:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/5837d4ee_screen-shot-2016-11-24-at-10.05.46-pm/screen-shot-2016-11-24-at-10.05.46-pm.png)\n",
    "<center>The same grid with 0 padding. Source: Andrej Karpathy.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This would expand our original image to a `7x7`. With this, we now see how our next layer's size is again a `5x5`, keeping our dimensionality consistent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From what we've learned so far, how can we calculate the number of neurons of each layer in our CNN?\n",
    "\n",
    "Given our input layer has a volume of `W`, our filter has a volume `(height * width * depth)` of `F`, we have a stride of `S`, and a padding of `P`, the following formula gives us the volume of the next layer: `(W−F+2P)/S+1`.\n",
    "\n",
    "Knowing the dimensionality of each additional layer helps us understand how large our model is and how our decisions around filter size and stride affect the size of our network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing CNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s look at an example CNN to see how it works in action.\n",
    "\n",
    "The CNN we will look at is trained on ImageNet as described in [this paper](http://www.matthewzeiler.com/pubs/arxive2013/eccv2014.pdf) by Zeiler and Fergus. In the images below (from the same paper), we’ll see what each layer in this network detects and see *how* each layer detects more and more complex ideas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layer 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583cbd42_layer-1-grid/layer-1-grid.png)\n",
    "<center>Example patterns that cause activations in the first layer of the network. These range from simple diagonal lines (top left) to green blobs (bottom middle).</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The images above are from Matthew Zeiler and Rob Fergus' [deep visualization toolbox](https://www.youtube.com/watch?v=ghEmQSxT6tw), which lets us visualize what each layer in a CNN focuses on.\n",
    "\n",
    "Each image in the above grid represents a pattern that causes the neurons in the first layer to activate - in other words, they are patterns that the first layer recognizes. The top left image shows a -45 degree line, while the middle top square shows a +45 degree line. These squares are shown below again for reference:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583cbba2_diagonal-line-1/diagonal-line-1.png)\n",
    "<center>As visualized here, the first layer of the CNN can recognize -45 degree lines.</center>\n",
    "<br>\n",
    "\n",
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583cbc02_diagonal-line-2/diagonal-line-2.png)\n",
    "<center>The first layer of the CNN is also able to recognize +45 degree lines, like the one above.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see some example images that cause such activations. The below grid of images all activated the -45 degree line. Notice how they are all selected despite the fact that they have different colors, gradients, and patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583cbace_grid-layer-1/grid-layer-1.png)\n",
    "<center>Example patches that activate the -45 degree line detector in the first layer.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the first layer of our CNN clearly picks out very simple shapes and patterns like lines and blobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layer 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/583780f3_screen-shot-2016-11-24-at-12.09.02-pm/screen-shot-2016-11-24-at-12.09.02-pm.png\" width=700 height=700>\n",
    "<center>A visualization of the second layer in the CNN. Notice how we are picking up more complex ideas like circles and stripes. The gray grid on the left represents how this layer of the CNN activates (or \"what it sees\") based on the corresponding images from the grid on the right.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second layer of the CNN captures complex ideas.\n",
    "\n",
    "As you see in the image above, the second layer of the CNN recognizes circles (second row, second column), stripes (first row, second column), and rectangles (bottom right).\n",
    "\n",
    "**The CNN learns to do this on its own**. There is no special instruction for the CNN to focus on more complex objects in deeper layers. That's just how it normally works out when you feed training data into a CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layer 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/5837811f_screen-shot-2016-11-24-at-12.09.24-pm/screen-shot-2016-11-24-at-12.09.24-pm.png\" width=700 height=700>\n",
    "<center>A visualization of the third layer in the CNN. The gray grid on the left represents how this layer of the CNN activates (or \"what it sees\") based on the corresponding images from the grid on the right.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third layer picks out complex combinations of features from the second layer. These include things like grids, and honeycombs (top left), wheels (second row, second column), and even faces (third row, third column)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layer 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://d17h27t6h515a5.cloudfront.net/topher/2016/November/58378151_screen-shot-2016-11-24-at-12.08.11-pm/screen-shot-2016-11-24-at-12.08.11-pm.png\" width=500 height=500>\n",
    "<center>A visualization of the fifth and final layer of the CNN. The gray grid on the left represents how this layer of the CNN activates (or \"what it sees\") based on the corresponding images from the grid on the right.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll skip layer 4, which continues this progression, and jump right to the fifth and final layer of this CNN.\n",
    "\n",
    "The last layer picks out the highest order ideas that we care about for classification, like dog faces, bird faces, and bicycles. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Convolution Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine how to implement a CNN in TensorFlow.\n",
    "\n",
    "TensorFlow provides the `tf.nn.conv2d()` and `tf.nn.bias_add()` functions to create your own convolutional layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Output depth\n",
    "k_output = 64\n",
    "\n",
    "# Image Properties\n",
    "image_width = 10\n",
    "image_height = 10\n",
    "color_channels = 3\n",
    "\n",
    "# Convolution filter\n",
    "filter_size_width = 5\n",
    "filter_size_height = 5\n",
    "\n",
    "# Input/Image\n",
    "input = tf.placeholder(\n",
    "    tf.float32,\n",
    "    shape=[None, image_height, image_width, color_channels])\n",
    "\n",
    "# Weight and bias\n",
    "weight = tf.Variable(tf.truncated_normal(\n",
    "    [filter_size_height, filter_size_width, color_channels, k_output]))\n",
    "bias = tf.Variable(tf.zeros(k_output))\n",
    "\n",
    "# Apply Convolution\n",
    "conv_layer = tf.nn.conv2d(input, weight, strides=[1, 2, 2, 1], padding='SAME')\n",
    "# Add bias\n",
    "conv_layer = tf.nn.bias_add(conv_layer, bias)\n",
    "# Apply activation function\n",
    "conv_layer = tf.nn.relu(conv_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above uses the `tf.nn.conv2d()` function to compute the convolution with `weight` as the filter and `[1, 2, 2, 1]` for the strides. TensorFlow uses a stride for each `input` dimension, `[batch, input_height, input_width, input_channels]`. We are generally always going to set the stride for `batch` and `input_channels `(i.e. the first and fourth element in the strides array) to be `1`.\n",
    "\n",
    "You'll focus on changing `input_height` and `input_width` while setting batch and `input_channels` to `1`. The `input_height` and `input_width` strides are for striding the filter over `input`. This example code uses a stride of 2 with 5x5 filter over `input`.\n",
    "\n",
    "The `tf.nn.bias_add()` function adds a 1-d bias to the last dimension in a matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Covnet-ology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Max Pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/582aac09_max-pooling/max-pooling.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image above is an example of max pooling with a 2x2 filter and stride of 2. The four 2x2 colors represent each time the filter was applied to find the maximum value.\n",
    "\n",
    "For example, `[[1, 0], [4, 6]]` becomes `6`, because `6` is the maximum value in this set. Similarly, `[[2, 3], [6, 8]]` becomes `8`.\n",
    "\n",
    "Conceptually, the benefit of the max pooling operation is to reduce the size of the input, and allow the neural network to focus on only the most important elements. Max pooling does this by only retaining the maximum value for each filtered area, and removing the remaining values.\n",
    "\n",
    "TensorFlow provides the `tf.nn.max_pool()` function to apply max pooling to your convolutional layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "...\n",
    "conv_layer = tf.nn.conv2d(input, weight, strides=[1, 2, 2, 1], padding='SAME')\n",
    "conv_layer = tf.nn.bias_add(conv_layer, bias)\n",
    "conv_layer = tf.nn.relu(conv_layer)\n",
    "# Apply Max Pooling\n",
    "conv_layer = tf.nn.max_pool(\n",
    "    conv_layer,\n",
    "    ksize=[1, 2, 2, 1],\n",
    "    strides=[1, 2, 2, 1],\n",
    "    padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.nn.max_pool()` function performs max pooling with the `ksize` parameter as the size of the filter and the `strides` parameter as the length of the stride. 2x2 filters with a stride of 2x2 are common in practice.\n",
    "\n",
    "The `ksize` and `strides` parameters are structured as 4-element lists, with each element corresponding to a dimension of the input tensor (`[batch, height, width, channels]`). For both `ksize` and `strides`, the `batch` and `channel` dimensions are typically set to `1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1x1 Convolutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [1x1 Convolutions Video](https://www.youtube.com/watch?v=Zmzgerm6SjA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inception Module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Inception Module Video](https://www.youtube.com/watch?v=SlTm03bEOxA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Network in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's time to walk through an example Convolutional Neural Network (CNN) in TensorFlow.\n",
    "\n",
    "The structure of this network follows the classic structure of CNNs, which is a mix of convolutional layers and max pooling, followed by fully-connected layers.\n",
    "\n",
    "The code we'll be looking at is similar to what we saw in the segment on Deep Neural Network in TensorFlow, except we'll restructured the architecture of this network as a CNN.\n",
    "\n",
    "Just like in that segment, here we'll study the line-by-line breakdown of the code. [Link to download the code and run it.](https://d17h27t6h515a5.cloudfront.net/topher/2017/February/58a61ca1_cnn/cnn.zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen this section of code from previous lessons. Here we're importing the MNIST dataset and using a convenient TensorFlow function to batch, scale, and One-Hot encode the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./train-images-idx3-ubyte.gz\n",
      "Extracting ./train-labels-idx1-ubyte.gz\n",
      "Extracting ./t10k-images-idx3-ubyte.gz\n",
      "Extracting ./t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\".\", one_hot=True, reshape=False)\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Parameters\n",
    "learning_rate = 0.00001\n",
    "epochs = 1\n",
    "batch_size = 128\n",
    "\n",
    "# Number of samples to calculate validation and accuracy\n",
    "# Decrease this if you're running out of memory to calculate accuracy\n",
    "test_valid_size = 256\n",
    "\n",
    "# Network Parameters\n",
    "n_classes = 10  # MNIST total classes (0-9 digits)\n",
    "dropout = 0.75  # Dropout, probability to keep units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weights and Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Store layers weight & bias\n",
    "weights = {\n",
    "    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),\n",
    "    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),\n",
    "    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),\n",
    "    'out': tf.Variable(tf.random_normal([1024, n_classes]))}\n",
    "\n",
    "biases = {\n",
    "    'bc1': tf.Variable(tf.random_normal([32])),\n",
    "    'bc2': tf.Variable(tf.random_normal([64])),\n",
    "    'bd1': tf.Variable(tf.random_normal([1024])),\n",
    "    'out': tf.Variable(tf.random_normal([n_classes]))}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/581a58be_convolution-schematic/convolution-schematic.gif)\n",
    "<centre>Convolution with 3×3 Filter. Source: http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution</centre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above is an example of a convolution with a 3x3 filter and a stride of 1 being applied to data with a range of 0 to 1. The convolution for each 3x3 section is calculated against the weight, `[[1, 0, 1], [0, 1, 0], [1, 0, 1]]`, then a bias is added to create the convolved feature on the right. In this case, the bias is zero. In TensorFlow, this is all done using `tf.nn.conv2d()` and `tf.nn.bias_add()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv2d(x, W, b, strides=1):\n",
    "    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')\n",
    "    x = tf.nn.bias_add(x, b)\n",
    "    return tf.nn.relu(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.nn.conv2d()` function computes the convolution against weight `W` as shown above.\n",
    "\n",
    "In TensorFlow, stride is an array of 4 elements; the first element in the stride array indicates the stride for batch and last element indicates stride for features. It's good practice to remove the batches or features you want to skip from the data set rather than use stride to skip them. You can always set the first and last element to 1 in stride in order to use all batches and features.\n",
    "\n",
    "The middle two elements are the strides for height and width respectively. I've mentioned stride as one number because you usually have a square stride where `height = width`. When someone says they are using a stride of 3, they usually mean `tf.nn.conv2d(x, W, strides=[1, 3, 3, 1])`.\n",
    "\n",
    "To make life easier, the code is using `tf.nn.bias_add()` to add the bias. Using `tf.add()` doesn't work when the tensors aren't the same shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max Pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/581a57fe_maxpool/maxpool.jpeg)\n",
    "<centre>Max Pooling with 2x2 filter and stride of 2. Source: http://cs231n.github.io/convolutional-networks/</centre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above is an example of max pooling with a 2x2 filter and stride of 2. The left square is the input and the right square is the output. The four 2x2 colors in input represents each time the filter was applied to create the max on the right side. For example, `[[1, 1], [5, 6]]` becomes `6` and `[[3, 2], [1, 2]]` becomes `3`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def maxpool2d(x, k=2):\n",
    "    return tf.nn.max_pool(\n",
    "        x,\n",
    "        ksize=[1, k, k, 1],\n",
    "        strides=[1, k, k, 1],\n",
    "        padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.nn.max_pool()` function does exactly what you would expect, it performs max pooling with the `ksize` parameter as the size of the filter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2016/November/581a64b7_arch/arch.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code below, we're creating 3 layers alternating between convolutions and max pooling followed by a fully connected and output layer. The transformation of each layer to new dimensions are shown in the comments. For example, the first layer shapes the images from 28x28x1 to 28x28x32 in the convolution step. Then next step applies max pooling, turning each sample into 14x14x32. All the layers are applied from `conv1` to `output`, producing 10 class predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_net(x, weights, biases, dropout):\n",
    "    # Layer 1 - 28*28*1 to 14*14*32\n",
    "    conv1 = conv2d(x, weights['wc1'], biases['bc1'])\n",
    "    conv1 = maxpool2d(conv1, k=2)\n",
    "\n",
    "    # Layer 2 - 14*14*32 to 7*7*64\n",
    "    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])\n",
    "    conv2 = maxpool2d(conv2, k=2)\n",
    "\n",
    "    # Fully connected layer - 7*7*64 to 1024\n",
    "    fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])\n",
    "    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])\n",
    "    fc1 = tf.nn.relu(fc1)\n",
    "    fc1 = tf.nn.dropout(fc1, dropout)\n",
    "\n",
    "    # Output Layer - class prediction - 1024 to 10\n",
    "    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  1, Batch   1 -Loss: 111744.7109 Validation Accuracy: 0.093750\n",
      "Epoch  1, Batch   2 -Loss: 67202.3359 Validation Accuracy: 0.109375\n",
      "Epoch  1, Batch   3 -Loss: 47169.2031 Validation Accuracy: 0.128906\n",
      "Epoch  1, Batch   4 -Loss: 33749.8672 Validation Accuracy: 0.140625\n",
      "Epoch  1, Batch   5 -Loss: 31741.6973 Validation Accuracy: 0.140625\n",
      "Epoch  1, Batch   6 -Loss: 29342.8477 Validation Accuracy: 0.160156\n",
      "Epoch  1, Batch   7 -Loss: 28273.4082 Validation Accuracy: 0.152344\n",
      "Epoch  1, Batch   8 -Loss: 27469.5625 Validation Accuracy: 0.167969\n",
      "Epoch  1, Batch   9 -Loss: 25588.3848 Validation Accuracy: 0.167969\n",
      "Epoch  1, Batch  10 -Loss: 20927.6055 Validation Accuracy: 0.171875\n",
      "Epoch  1, Batch  11 -Loss: 21317.8477 Validation Accuracy: 0.179688\n",
      "Epoch  1, Batch  12 -Loss: 18316.5645 Validation Accuracy: 0.171875\n",
      "Epoch  1, Batch  13 -Loss: 18508.5449 Validation Accuracy: 0.183594\n",
      "Epoch  1, Batch  14 -Loss: 17032.8652 Validation Accuracy: 0.187500\n",
      "Epoch  1, Batch  15 -Loss: 16483.7871 Validation Accuracy: 0.187500\n",
      "Epoch  1, Batch  16 -Loss: 16698.2246 Validation Accuracy: 0.179688\n",
      "Epoch  1, Batch  17 -Loss: 17142.1406 Validation Accuracy: 0.203125\n",
      "Epoch  1, Batch  18 -Loss: 12621.2002 Validation Accuracy: 0.210938\n",
      "Epoch  1, Batch  19 -Loss: 14542.5488 Validation Accuracy: 0.207031\n",
      "Epoch  1, Batch  20 -Loss: 15313.0996 Validation Accuracy: 0.222656\n",
      "Epoch  1, Batch  21 -Loss: 16435.0586 Validation Accuracy: 0.230469\n",
      "Epoch  1, Batch  22 -Loss: 14525.4609 Validation Accuracy: 0.234375\n",
      "Epoch  1, Batch  23 -Loss: 14742.8789 Validation Accuracy: 0.242188\n",
      "Epoch  1, Batch  24 -Loss: 12611.0898 Validation Accuracy: 0.257812\n",
      "Epoch  1, Batch  25 -Loss: 15133.4121 Validation Accuracy: 0.289062\n",
      "Epoch  1, Batch  26 -Loss: 14653.9307 Validation Accuracy: 0.285156\n",
      "Epoch  1, Batch  27 -Loss: 12108.5723 Validation Accuracy: 0.300781\n",
      "Epoch  1, Batch  28 -Loss: 12403.2344 Validation Accuracy: 0.289062\n",
      "Epoch  1, Batch  29 -Loss: 12542.6035 Validation Accuracy: 0.277344\n",
      "Epoch  1, Batch  30 -Loss: 10421.4395 Validation Accuracy: 0.304688\n",
      "Epoch  1, Batch  31 -Loss: 10883.4648 Validation Accuracy: 0.335938\n",
      "Epoch  1, Batch  32 -Loss: 10846.6719 Validation Accuracy: 0.343750\n",
      "Epoch  1, Batch  33 -Loss: 10070.7754 Validation Accuracy: 0.335938\n",
      "Epoch  1, Batch  34 -Loss: 10742.5020 Validation Accuracy: 0.335938\n",
      "Epoch  1, Batch  35 -Loss: 10303.4941 Validation Accuracy: 0.332031\n",
      "Epoch  1, Batch  36 -Loss:  9846.2695 Validation Accuracy: 0.343750\n",
      "Epoch  1, Batch  37 -Loss:  9827.7451 Validation Accuracy: 0.355469\n",
      "Epoch  1, Batch  38 -Loss:  7552.3975 Validation Accuracy: 0.367188\n",
      "Epoch  1, Batch  39 -Loss:  8813.1729 Validation Accuracy: 0.367188\n",
      "Epoch  1, Batch  40 -Loss:  7697.9326 Validation Accuracy: 0.390625\n",
      "Epoch  1, Batch  41 -Loss:  8394.0107 Validation Accuracy: 0.402344\n",
      "Epoch  1, Batch  42 -Loss:  8569.7744 Validation Accuracy: 0.394531\n",
      "Epoch  1, Batch  43 -Loss:  8263.8701 Validation Accuracy: 0.390625\n",
      "Epoch  1, Batch  44 -Loss:  8054.9033 Validation Accuracy: 0.410156\n",
      "Epoch  1, Batch  45 -Loss:  7078.4082 Validation Accuracy: 0.417969\n",
      "Epoch  1, Batch  46 -Loss:  7593.7935 Validation Accuracy: 0.425781\n",
      "Epoch  1, Batch  47 -Loss:  8241.7217 Validation Accuracy: 0.417969\n",
      "Epoch  1, Batch  48 -Loss:  9209.3867 Validation Accuracy: 0.421875\n",
      "Epoch  1, Batch  49 -Loss:  5927.5537 Validation Accuracy: 0.429688\n",
      "Epoch  1, Batch  50 -Loss:  7306.0698 Validation Accuracy: 0.429688\n",
      "Epoch  1, Batch  51 -Loss:  8528.7891 Validation Accuracy: 0.437500\n",
      "Epoch  1, Batch  52 -Loss:  8665.8984 Validation Accuracy: 0.421875\n",
      "Epoch  1, Batch  53 -Loss:  6346.1621 Validation Accuracy: 0.429688\n",
      "Epoch  1, Batch  54 -Loss:  8154.9585 Validation Accuracy: 0.437500\n",
      "Epoch  1, Batch  55 -Loss:  6967.2222 Validation Accuracy: 0.445312\n",
      "Epoch  1, Batch  56 -Loss:  6008.4780 Validation Accuracy: 0.429688\n",
      "Epoch  1, Batch  57 -Loss:  7214.2856 Validation Accuracy: 0.449219\n",
      "Epoch  1, Batch  58 -Loss:  6797.4775 Validation Accuracy: 0.445312\n",
      "Epoch  1, Batch  59 -Loss:  6706.9790 Validation Accuracy: 0.453125\n",
      "Epoch  1, Batch  60 -Loss:  6783.0044 Validation Accuracy: 0.457031\n",
      "Epoch  1, Batch  61 -Loss:  6065.3525 Validation Accuracy: 0.464844\n",
      "Epoch  1, Batch  62 -Loss:  5509.5273 Validation Accuracy: 0.488281\n",
      "Epoch  1, Batch  63 -Loss:  8607.9326 Validation Accuracy: 0.500000\n",
      "Epoch  1, Batch  64 -Loss:  5156.8389 Validation Accuracy: 0.503906\n",
      "Epoch  1, Batch  65 -Loss:  6283.9658 Validation Accuracy: 0.503906\n",
      "Epoch  1, Batch  66 -Loss:  5851.2510 Validation Accuracy: 0.500000\n",
      "Epoch  1, Batch  67 -Loss:  6352.6045 Validation Accuracy: 0.503906\n",
      "Epoch  1, Batch  68 -Loss:  6058.0566 Validation Accuracy: 0.496094\n",
      "Epoch  1, Batch  69 -Loss:  6016.6387 Validation Accuracy: 0.496094\n",
      "Epoch  1, Batch  70 -Loss:  5970.3750 Validation Accuracy: 0.496094\n",
      "Epoch  1, Batch  71 -Loss:  5506.3516 Validation Accuracy: 0.492188\n",
      "Epoch  1, Batch  72 -Loss:  7253.0127 Validation Accuracy: 0.492188\n",
      "Epoch  1, Batch  73 -Loss:  5954.2720 Validation Accuracy: 0.496094\n",
      "Epoch  1, Batch  74 -Loss:  4770.0186 Validation Accuracy: 0.511719\n",
      "Epoch  1, Batch  75 -Loss:  5837.4155 Validation Accuracy: 0.519531\n",
      "Epoch  1, Batch  76 -Loss:  5699.6768 Validation Accuracy: 0.507812\n",
      "Epoch  1, Batch  77 -Loss:  4344.6846 Validation Accuracy: 0.527344\n",
      "Epoch  1, Batch  78 -Loss:  5331.6152 Validation Accuracy: 0.519531\n",
      "Epoch  1, Batch  79 -Loss:  4740.4526 Validation Accuracy: 0.527344\n",
      "Epoch  1, Batch  80 -Loss:  6168.3535 Validation Accuracy: 0.519531\n",
      "Epoch  1, Batch  81 -Loss:  5215.6665 Validation Accuracy: 0.523438\n",
      "Epoch  1, Batch  82 -Loss:  6182.4893 Validation Accuracy: 0.527344\n",
      "Epoch  1, Batch  83 -Loss:  5531.7524 Validation Accuracy: 0.531250\n",
      "Epoch  1, Batch  84 -Loss:  5291.4082 Validation Accuracy: 0.546875\n",
      "Epoch  1, Batch  85 -Loss:  5522.8535 Validation Accuracy: 0.546875\n",
      "Epoch  1, Batch  86 -Loss:  6164.5098 Validation Accuracy: 0.550781\n",
      "Epoch  1, Batch  87 -Loss:  5457.3521 Validation Accuracy: 0.535156\n",
      "Epoch  1, Batch  88 -Loss:  4696.8560 Validation Accuracy: 0.535156\n",
      "Epoch  1, Batch  89 -Loss:  4520.1230 Validation Accuracy: 0.542969\n",
      "Epoch  1, Batch  90 -Loss:  5144.9375 Validation Accuracy: 0.546875\n",
      "Epoch  1, Batch  91 -Loss:  3853.8267 Validation Accuracy: 0.542969\n",
      "Epoch  1, Batch  92 -Loss:  4553.4727 Validation Accuracy: 0.546875\n",
      "Epoch  1, Batch  93 -Loss:  5572.4375 Validation Accuracy: 0.550781\n",
      "Epoch  1, Batch  94 -Loss:  4486.1704 Validation Accuracy: 0.546875\n",
      "Epoch  1, Batch  95 -Loss:  4654.8896 Validation Accuracy: 0.554688\n",
      "Epoch  1, Batch  96 -Loss:  6267.2915 Validation Accuracy: 0.542969\n",
      "Epoch  1, Batch  97 -Loss:  4283.8423 Validation Accuracy: 0.550781\n",
      "Epoch  1, Batch  98 -Loss:  4492.2690 Validation Accuracy: 0.562500\n",
      "Epoch  1, Batch  99 -Loss:  4065.6846 Validation Accuracy: 0.542969\n",
      "Epoch  1, Batch 100 -Loss:  5342.8096 Validation Accuracy: 0.570312\n",
      "Epoch  1, Batch 101 -Loss:  4753.3921 Validation Accuracy: 0.566406\n",
      "Epoch  1, Batch 102 -Loss:  3920.9695 Validation Accuracy: 0.574219\n",
      "Epoch  1, Batch 103 -Loss:  3462.5432 Validation Accuracy: 0.578125\n",
      "Epoch  1, Batch 104 -Loss:  4011.7861 Validation Accuracy: 0.570312\n",
      "Epoch  1, Batch 105 -Loss:  3732.1162 Validation Accuracy: 0.578125\n",
      "Epoch  1, Batch 106 -Loss:  3755.5029 Validation Accuracy: 0.578125\n",
      "Epoch  1, Batch 107 -Loss:  4481.7939 Validation Accuracy: 0.578125\n",
      "Epoch  1, Batch 108 -Loss:  3982.7258 Validation Accuracy: 0.585938\n",
      "Epoch  1, Batch 109 -Loss:  4498.2085 Validation Accuracy: 0.589844\n",
      "Epoch  1, Batch 110 -Loss:  3410.3679 Validation Accuracy: 0.582031\n",
      "Epoch  1, Batch 111 -Loss:  4131.3232 Validation Accuracy: 0.582031\n",
      "Epoch  1, Batch 112 -Loss:  4601.0000 Validation Accuracy: 0.570312\n",
      "Epoch  1, Batch 113 -Loss:  5312.6240 Validation Accuracy: 0.574219\n",
      "Epoch  1, Batch 114 -Loss:  3379.0356 Validation Accuracy: 0.582031\n",
      "Epoch  1, Batch 115 -Loss:  3623.7559 Validation Accuracy: 0.574219\n",
      "Epoch  1, Batch 116 -Loss:  4046.1235 Validation Accuracy: 0.585938\n",
      "Epoch  1, Batch 117 -Loss:  4549.3262 Validation Accuracy: 0.582031\n",
      "Epoch  1, Batch 118 -Loss:  3727.3418 Validation Accuracy: 0.589844\n",
      "Epoch  1, Batch 119 -Loss:  4091.4507 Validation Accuracy: 0.593750\n",
      "Epoch  1, Batch 120 -Loss:  3936.9307 Validation Accuracy: 0.601562\n",
      "Epoch  1, Batch 121 -Loss:  3511.3735 Validation Accuracy: 0.597656\n",
      "Epoch  1, Batch 122 -Loss:  2545.4407 Validation Accuracy: 0.601562\n",
      "Epoch  1, Batch 123 -Loss:  3917.7234 Validation Accuracy: 0.593750\n",
      "Epoch  1, Batch 124 -Loss:  4647.7539 Validation Accuracy: 0.585938\n",
      "Epoch  1, Batch 125 -Loss:  3954.0461 Validation Accuracy: 0.574219\n",
      "Epoch  1, Batch 126 -Loss:  4322.2026 Validation Accuracy: 0.585938\n",
      "Epoch  1, Batch 127 -Loss:  3876.0728 Validation Accuracy: 0.585938\n",
      "Epoch  1, Batch 128 -Loss:  3227.1216 Validation Accuracy: 0.605469\n",
      "Epoch  1, Batch 129 -Loss:  3631.4617 Validation Accuracy: 0.605469\n",
      "Epoch  1, Batch 130 -Loss:  4003.9316 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 131 -Loss:  4554.0684 Validation Accuracy: 0.605469\n",
      "Epoch  1, Batch 132 -Loss:  3179.4692 Validation Accuracy: 0.601562\n",
      "Epoch  1, Batch 133 -Loss:  2350.5088 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 134 -Loss:  3303.8516 Validation Accuracy: 0.605469\n",
      "Epoch  1, Batch 135 -Loss:  2857.6790 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 136 -Loss:  3930.9756 Validation Accuracy: 0.605469\n",
      "Epoch  1, Batch 137 -Loss:  3497.2114 Validation Accuracy: 0.621094\n",
      "Epoch  1, Batch 138 -Loss:  2344.7463 Validation Accuracy: 0.625000\n",
      "Epoch  1, Batch 139 -Loss:  3969.8687 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 140 -Loss:  3011.2009 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 141 -Loss:  3581.0874 Validation Accuracy: 0.609375\n",
      "Epoch  1, Batch 142 -Loss:  2421.2861 Validation Accuracy: 0.613281\n",
      "Epoch  1, Batch 143 -Loss:  4006.6997 Validation Accuracy: 0.617188\n",
      "Epoch  1, Batch 144 -Loss:  3025.8408 Validation Accuracy: 0.625000\n",
      "Epoch  1, Batch 145 -Loss:  2965.5503 Validation Accuracy: 0.628906\n",
      "Epoch  1, Batch 146 -Loss:  3148.8352 Validation Accuracy: 0.628906\n",
      "Epoch  1, Batch 147 -Loss:  4084.6155 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 148 -Loss:  2075.7742 Validation Accuracy: 0.632812\n",
      "Epoch  1, Batch 149 -Loss:  3356.3647 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 150 -Loss:  3400.8853 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 151 -Loss:  2765.1655 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 152 -Loss:  3167.6770 Validation Accuracy: 0.640625\n",
      "Epoch  1, Batch 153 -Loss:  3522.9492 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 154 -Loss:  3264.6841 Validation Accuracy: 0.636719\n",
      "Epoch  1, Batch 155 -Loss:  3739.8657 Validation Accuracy: 0.644531\n",
      "Epoch  1, Batch 156 -Loss:  3450.6050 Validation Accuracy: 0.648438\n",
      "Epoch  1, Batch 157 -Loss:  3170.9375 Validation Accuracy: 0.648438\n",
      "Epoch  1, Batch 158 -Loss:  3129.5273 Validation Accuracy: 0.648438\n",
      "Epoch  1, Batch 159 -Loss:  3986.4336 Validation Accuracy: 0.648438\n",
      "Epoch  1, Batch 160 -Loss:  3462.6719 Validation Accuracy: 0.648438\n",
      "Epoch  1, Batch 161 -Loss:  3663.4919 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 162 -Loss:  2903.8901 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 163 -Loss:  3092.2178 Validation Accuracy: 0.664062\n",
      "Epoch  1, Batch 164 -Loss:  4317.8755 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 165 -Loss:  2384.0654 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 166 -Loss:  2597.6992 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 167 -Loss:  2517.1177 Validation Accuracy: 0.652344\n",
      "Epoch  1, Batch 168 -Loss:  2170.8257 Validation Accuracy: 0.660156\n",
      "Epoch  1, Batch 169 -Loss:  2890.2617 Validation Accuracy: 0.652344\n",
      "Epoch  1, Batch 170 -Loss:  3450.7715 Validation Accuracy: 0.667969\n",
      "Epoch  1, Batch 171 -Loss:  2507.4131 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 172 -Loss:  2795.4561 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 173 -Loss:  3060.7056 Validation Accuracy: 0.667969\n",
      "Epoch  1, Batch 174 -Loss:  3557.0090 Validation Accuracy: 0.664062\n",
      "Epoch  1, Batch 175 -Loss:  3521.1501 Validation Accuracy: 0.656250\n",
      "Epoch  1, Batch 176 -Loss:  2320.2678 Validation Accuracy: 0.660156\n",
      "Epoch  1, Batch 177 -Loss:  2546.6243 Validation Accuracy: 0.667969\n",
      "Epoch  1, Batch 178 -Loss:  2769.5322 Validation Accuracy: 0.671875\n",
      "Epoch  1, Batch 179 -Loss:  2366.5725 Validation Accuracy: 0.671875\n",
      "Epoch  1, Batch 180 -Loss:  3644.3638 Validation Accuracy: 0.671875\n",
      "Epoch  1, Batch 181 -Loss:  2477.2136 Validation Accuracy: 0.675781\n",
      "Epoch  1, Batch 182 -Loss:  3372.8584 Validation Accuracy: 0.671875\n",
      "Epoch  1, Batch 183 -Loss:  2246.2473 Validation Accuracy: 0.667969\n",
      "Epoch  1, Batch 184 -Loss:  3825.4521 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 185 -Loss:  3246.3657 Validation Accuracy: 0.671875\n",
      "Epoch  1, Batch 186 -Loss:  3077.3330 Validation Accuracy: 0.664062\n",
      "Epoch  1, Batch 187 -Loss:  2225.5498 Validation Accuracy: 0.675781\n",
      "Epoch  1, Batch 188 -Loss:  2428.6707 Validation Accuracy: 0.675781\n",
      "Epoch  1, Batch 189 -Loss:  2633.4573 Validation Accuracy: 0.687500\n",
      "Epoch  1, Batch 190 -Loss:  3423.2656 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 191 -Loss:  2055.9199 Validation Accuracy: 0.687500\n",
      "Epoch  1, Batch 192 -Loss:  3296.6182 Validation Accuracy: 0.683594\n",
      "Epoch  1, Batch 193 -Loss:  2551.9854 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 194 -Loss:  2467.2886 Validation Accuracy: 0.683594\n",
      "Epoch  1, Batch 195 -Loss:  2363.2493 Validation Accuracy: 0.695312\n",
      "Epoch  1, Batch 196 -Loss:  2913.3040 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 197 -Loss:  2931.2043 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 198 -Loss:  2596.8577 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 199 -Loss:  3046.2349 Validation Accuracy: 0.691406\n",
      "Epoch  1, Batch 200 -Loss:  2271.2607 Validation Accuracy: 0.683594\n",
      "Epoch  1, Batch 201 -Loss:  2326.6909 Validation Accuracy: 0.679688\n",
      "Epoch  1, Batch 202 -Loss:  2052.7712 Validation Accuracy: 0.687500\n",
      "Epoch  1, Batch 203 -Loss:  3105.4907 Validation Accuracy: 0.687500\n",
      "Epoch  1, Batch 204 -Loss:  2648.2207 Validation Accuracy: 0.691406\n",
      "Epoch  1, Batch 205 -Loss:  2365.4426 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 206 -Loss:  2419.6521 Validation Accuracy: 0.691406\n",
      "Epoch  1, Batch 207 -Loss:  2469.4626 Validation Accuracy: 0.691406\n",
      "Epoch  1, Batch 208 -Loss:  2387.4495 Validation Accuracy: 0.683594\n",
      "Epoch  1, Batch 209 -Loss:  2343.4761 Validation Accuracy: 0.691406\n",
      "Epoch  1, Batch 210 -Loss:  2638.1177 Validation Accuracy: 0.687500\n",
      "Epoch  1, Batch 211 -Loss:  2389.1179 Validation Accuracy: 0.695312\n",
      "Epoch  1, Batch 212 -Loss:  2120.8938 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 213 -Loss:  2862.5327 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 214 -Loss:  1971.4073 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 215 -Loss:  2869.1958 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 216 -Loss:  2011.7841 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 217 -Loss:  2684.0349 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 218 -Loss:  2372.3892 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 219 -Loss:  2219.0598 Validation Accuracy: 0.710938\n",
      "Epoch  1, Batch 220 -Loss:  2028.2227 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 221 -Loss:  2448.6396 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 222 -Loss:  2865.5503 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 223 -Loss:  2340.7607 Validation Accuracy: 0.695312\n",
      "Epoch  1, Batch 224 -Loss:  1898.6265 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 225 -Loss:  2275.1064 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 226 -Loss:  1755.2737 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 227 -Loss:  2442.2024 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 228 -Loss:  3107.4094 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 229 -Loss:  2436.4600 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 230 -Loss:  1751.3605 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 231 -Loss:  2374.6978 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 232 -Loss:  1920.6890 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 233 -Loss:  1964.6079 Validation Accuracy: 0.695312\n",
      "Epoch  1, Batch 234 -Loss:  3015.8105 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 235 -Loss:  3063.1387 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 236 -Loss:  3191.8787 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 237 -Loss:  2555.9143 Validation Accuracy: 0.710938\n",
      "Epoch  1, Batch 238 -Loss:  1976.9175 Validation Accuracy: 0.714844\n",
      "Epoch  1, Batch 239 -Loss:  3126.5034 Validation Accuracy: 0.710938\n",
      "Epoch  1, Batch 240 -Loss:  1934.2545 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 241 -Loss:  2906.7915 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 242 -Loss:  1968.8887 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 243 -Loss:  2371.5938 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 244 -Loss:  2357.6802 Validation Accuracy: 0.707031\n",
      "Epoch  1, Batch 245 -Loss:  1704.3728 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 246 -Loss:  2309.9863 Validation Accuracy: 0.699219\n",
      "Epoch  1, Batch 247 -Loss:  2239.4614 Validation Accuracy: 0.703125\n",
      "Epoch  1, Batch 248 -Loss:  2695.7424 Validation Accuracy: 0.718750\n",
      "Epoch  1, Batch 249 -Loss:  2667.8555 Validation Accuracy: 0.714844\n",
      "Epoch  1, Batch 250 -Loss:  1989.6545 Validation Accuracy: 0.726562\n",
      "Epoch  1, Batch 251 -Loss:  2691.2981 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 252 -Loss:  2130.4888 Validation Accuracy: 0.714844\n",
      "Epoch  1, Batch 253 -Loss:  2732.9822 Validation Accuracy: 0.722656\n",
      "Epoch  1, Batch 254 -Loss:  2299.6160 Validation Accuracy: 0.722656\n",
      "Epoch  1, Batch 255 -Loss:  2076.8735 Validation Accuracy: 0.722656\n",
      "Epoch  1, Batch 256 -Loss:  1897.2124 Validation Accuracy: 0.722656\n",
      "Epoch  1, Batch 257 -Loss:  2455.4082 Validation Accuracy: 0.718750\n",
      "Epoch  1, Batch 258 -Loss:  2163.2798 Validation Accuracy: 0.726562\n",
      "Epoch  1, Batch 259 -Loss:  2238.0469 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 260 -Loss:  1959.7179 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 261 -Loss:  1371.5947 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 262 -Loss:  1835.5634 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 263 -Loss:  1933.5645 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 264 -Loss:  1537.7841 Validation Accuracy: 0.726562\n",
      "Epoch  1, Batch 265 -Loss:  2466.4971 Validation Accuracy: 0.722656\n",
      "Epoch  1, Batch 266 -Loss:  2969.6335 Validation Accuracy: 0.726562\n",
      "Epoch  1, Batch 267 -Loss:  1984.5834 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 268 -Loss:  1946.6863 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 269 -Loss:  1797.5200 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 270 -Loss:  2466.8909 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 271 -Loss:  2068.2817 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 272 -Loss:  2329.9197 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 273 -Loss:  1240.5208 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 274 -Loss:  1671.7231 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 275 -Loss:  1633.1296 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 276 -Loss:  1790.0298 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 277 -Loss:  1827.2324 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 278 -Loss:  1361.1440 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 279 -Loss:  1807.4413 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 280 -Loss:  2543.8938 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 281 -Loss:  1085.2448 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 282 -Loss:  2060.8208 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 283 -Loss:  1953.9379 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 284 -Loss:  1718.4366 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 285 -Loss:  1754.0355 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 286 -Loss:  1702.5741 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 287 -Loss:  2019.0699 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 288 -Loss:  1639.8743 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 289 -Loss:  2144.3477 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 290 -Loss:  2093.0967 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 291 -Loss:  1572.3152 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 292 -Loss:  1320.0604 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 293 -Loss:  1316.5498 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 294 -Loss:  1979.2488 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 295 -Loss:  2194.4561 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 296 -Loss:  1747.8682 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 297 -Loss:  1953.3906 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 298 -Loss:  1876.6331 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 299 -Loss:  1550.2664 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 300 -Loss:  1602.9954 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 301 -Loss:  2400.2327 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 302 -Loss:  2185.1082 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 303 -Loss:  1831.4698 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 304 -Loss:  2554.6357 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 305 -Loss:  1497.9348 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 306 -Loss:  2664.4092 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 307 -Loss:  1463.0692 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 308 -Loss:  2593.5430 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 309 -Loss:  2038.2017 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 310 -Loss:  1578.7357 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 311 -Loss:  1510.6478 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 312 -Loss:  2346.0466 Validation Accuracy: 0.730469\n",
      "Epoch  1, Batch 313 -Loss:  1852.4004 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 314 -Loss:  1656.3496 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 315 -Loss:  1927.6550 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 316 -Loss:  1371.1362 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 317 -Loss:  1854.0973 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 318 -Loss:  1918.6969 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 319 -Loss:  2100.3115 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 320 -Loss:  2633.5591 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 321 -Loss:  2126.5969 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 322 -Loss:  1430.6205 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 323 -Loss:  2375.1980 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 324 -Loss:  1530.0378 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 325 -Loss:  1853.2107 Validation Accuracy: 0.734375\n",
      "Epoch  1, Batch 326 -Loss:  2338.9910 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 327 -Loss:  1890.5309 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 328 -Loss:  1755.6621 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 329 -Loss:  2388.6650 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 330 -Loss:  1700.1946 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 331 -Loss:  1790.5204 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 332 -Loss:  1871.0609 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 333 -Loss:  1732.3333 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 334 -Loss:  1141.6416 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 335 -Loss:  2121.4968 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 336 -Loss:  1427.9602 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 337 -Loss:  2349.7893 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 338 -Loss:  2183.6204 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 339 -Loss:  2431.6992 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 340 -Loss:  1816.7812 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 341 -Loss:  2086.8708 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 342 -Loss:  1775.0527 Validation Accuracy: 0.738281\n",
      "Epoch  1, Batch 343 -Loss:  1254.7478 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 344 -Loss:  1879.3735 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 345 -Loss:   991.2856 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 346 -Loss:  2367.9644 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 347 -Loss:  1715.3832 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 348 -Loss:  1802.4746 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 349 -Loss:  1356.4432 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 350 -Loss:  1972.1511 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 351 -Loss:  1082.9512 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 352 -Loss:  2017.9011 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 353 -Loss:  1981.4803 Validation Accuracy: 0.742188\n",
      "Epoch  1, Batch 354 -Loss:  1920.4313 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 355 -Loss:  1798.9478 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 356 -Loss:  2325.3191 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 357 -Loss:  1419.8755 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 358 -Loss:  1582.4097 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 359 -Loss:  1678.9819 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 360 -Loss:  1709.2119 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 361 -Loss:  1649.6176 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 362 -Loss:  1525.1434 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 363 -Loss:  1648.6536 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 364 -Loss:  1334.2378 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 365 -Loss:  2036.3528 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 366 -Loss:  1834.6067 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 367 -Loss:  2147.3821 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 368 -Loss:  1609.5696 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 369 -Loss:  1579.9126 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 370 -Loss:  1883.5566 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 371 -Loss:  2016.4192 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 372 -Loss:  1406.9568 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 373 -Loss:  1745.9078 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 374 -Loss:  2066.0266 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 375 -Loss:  1966.3066 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 376 -Loss:  1334.9584 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 377 -Loss:  2339.9888 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 378 -Loss:  1762.3768 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 379 -Loss:  1531.4026 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 380 -Loss:  1860.8330 Validation Accuracy: 0.750000\n",
      "Epoch  1, Batch 381 -Loss:  2203.4995 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 382 -Loss:  1582.5200 Validation Accuracy: 0.746094\n",
      "Epoch  1, Batch 383 -Loss:  1453.0111 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 384 -Loss:  2213.5571 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 385 -Loss:  2368.9810 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 386 -Loss:  1414.2412 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 387 -Loss:   903.1267 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 388 -Loss:  1782.3245 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 389 -Loss:   862.9780 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 390 -Loss:  1232.7622 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 391 -Loss:  1582.9609 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 392 -Loss:  1927.6155 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 393 -Loss:  2242.1313 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 394 -Loss:  1726.5844 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 395 -Loss:  1463.9338 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 396 -Loss:  1605.9963 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 397 -Loss:  1554.4803 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 398 -Loss:  1931.4158 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 399 -Loss:  1319.0159 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 400 -Loss:  1637.4636 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 401 -Loss:  1191.5931 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 402 -Loss:  1143.1002 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 403 -Loss:  1832.0996 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 404 -Loss:  1202.6232 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 405 -Loss:  2104.6499 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 406 -Loss:  1604.4165 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 407 -Loss:  1639.7313 Validation Accuracy: 0.757812\n",
      "Epoch  1, Batch 408 -Loss:  1886.4573 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 409 -Loss:  1717.2349 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 410 -Loss:  1130.0596 Validation Accuracy: 0.753906\n",
      "Epoch  1, Batch 411 -Loss:  1289.5361 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 412 -Loss:  1868.0343 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 413 -Loss:   875.2244 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 414 -Loss:  1481.8535 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 415 -Loss:  1163.5808 Validation Accuracy: 0.761719\n",
      "Epoch  1, Batch 416 -Loss:  1519.0931 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 417 -Loss:   968.8335 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 418 -Loss:  1696.7659 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 419 -Loss:  1761.3837 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 420 -Loss:  1152.8561 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 421 -Loss:  1362.4164 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 422 -Loss:  1291.4028 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 423 -Loss:  1775.1809 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 424 -Loss:  1433.0261 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 425 -Loss:  1582.6722 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 426 -Loss:  1431.9556 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 427 -Loss:  1013.2540 Validation Accuracy: 0.765625\n",
      "Epoch  1, Batch 428 -Loss:  2092.4258 Validation Accuracy: 0.769531\n",
      "Epoch  1, Batch 429 -Loss:  1135.9553 Validation Accuracy: 0.765625\n",
      "Testing Accuracy: 0.76953125\n"
     ]
    }
   ],
   "source": [
    "# tf Graph input\n",
    "x = tf.placeholder(tf.float32, [None, 28, 28, 1])\n",
    "y = tf.placeholder(tf.float32, [None, n_classes])\n",
    "keep_prob = tf.placeholder(tf.float32)\n",
    "\n",
    "# Model\n",
    "logits = conv_net(x, weights, biases, keep_prob)\n",
    "\n",
    "# Define loss and optimizer\n",
    "cost = tf.reduce_mean(\\\n",
    "    tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\\\n",
    "    .minimize(cost)\n",
    "\n",
    "# Accuracy\n",
    "correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "# Initializing the variables\n",
    "init = tf. global_variables_initializer()\n",
    "\n",
    "# Launch the graph\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        for batch in range(mnist.train.num_examples//batch_size):\n",
    "            batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "            sess.run(optimizer, feed_dict={\n",
    "                x: batch_x,\n",
    "                y: batch_y,\n",
    "                keep_prob: dropout})\n",
    "\n",
    "            # Calculate batch loss and accuracy\n",
    "            loss = sess.run(cost, feed_dict={\n",
    "                x: batch_x,\n",
    "                y: batch_y,\n",
    "                keep_prob: 1.})\n",
    "            valid_acc = sess.run(accuracy, feed_dict={\n",
    "                x: mnist.validation.images[:test_valid_size],\n",
    "                y: mnist.validation.labels[:test_valid_size],\n",
    "                keep_prob: 1.})\n",
    "\n",
    "            print('Epoch {:>2}, Batch {:>3} -'\n",
    "                  'Loss: {:>10.4f} Validation Accuracy: {:.6f}'.format(\n",
    "                epoch + 1,\n",
    "                batch + 1,\n",
    "                loss,\n",
    "                valid_acc))\n",
    "\n",
    "    # Calculate Test Accuracy\n",
    "    test_acc = sess.run(accuracy, feed_dict={\n",
    "        x: mnist.test.images[:test_valid_size],\n",
    "        y: mnist.test.labels[:test_valid_size],\n",
    "        keep_prob: 1.})\n",
    "    print('Testing Accuracy: {}'.format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We can further improve the accuracy by increasing the epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}