Deep_Learning_Project_On_Image_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep Learning Project On Image Classification.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "kb_CMQf9Mid-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "# Image Classification\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "SCiS_sXfMo8n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<p>In this project, I'll classify images from the CIFAR-10 dataset\n",
        "(https://www.cs.toronto.edu/~kriz/cifar.html). The dataset consists of airplanes, dogs, cats, and other\n",
        "objects. I'll preprocess the images, then train a convolutional neural network on all the samples.\n",
        "The images need to be normalized and the labels need to be one-hot encoded. I'll get to apply\n",
        "what I learned and build a convolutional, max pooling, dropout, and fully connected layers. At the\n",
        "end, I'll get to see your neural network's predictions on the sample images.</p>"
      ]
    },
    {
      "metadata": {
        "id": "dZeU8Oi9MzOi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Get the Data"
      ]
    },
    {
      "metadata": {
        "id": "UWATuXweNEl1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "<p>Download the CIFAR-10 dataset for python using this url: https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz </p>"
      ]
    },
    {
      "metadata": {
        "id": "DqYhH_h7Na6y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data\n",
        "<p>CIFAR-10 is an established computer-vision dataset used for object recognition. It is a subset of the\n",
        "80 million tiny images dataset and consists of 60,000 32x32 color images containing one of 10\n",
        "object classes, with 6000 images per class. It was collected by Alex Krizhevsky, Vinod Nair, and\n",
        "Geoffrey Hinton.</p>"
      ]
    },
    {
      "metadata": {
        "id": "U5_r16jcUfP3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ItE2f4DMUliV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7573c858-8164-4296-bdb3-8a788ba1a5d8"
      },
      "cell_type": "code",
      "source": [
        "tf.test.gpu_device_name()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/device:GPU:0'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "metadata": {
        "id": "ETlhsewvUrXB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e27be171-63c5-4eaf-852f-130e3f0c6eff"
      },
      "cell_type": "code",
      "source": [
        "!pip install tqdm"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (4.25.0)\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cK8c5KMVUu2J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from urllib.request import urlretrieve\n",
        "from os.path import isfile, isdir\n",
        "from tqdm import tqdm\n",
        "import tarfile"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "L7R5IejhUy-J",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "321ee1b7-7d7a-4370-bbd1-027e3d0401cb"
      },
      "cell_type": "code",
      "source": [
        "cifar10_dataset_folder_path = 'cifar-10-batches-py'\n",
        "\n",
        "class DLProgress(tqdm):\n",
        "    last_block = 0\n",
        "\n",
        "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
        "        self.total = total_size\n",
        "        self.update((block_num - self.last_block) * block_size)\n",
        "        self.last_block = block_num\n",
        "\n",
        "if not isfile('cifar-10-python.tar.gz'):\n",
        "    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:\n",
        "        urlretrieve(\n",
        "            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',\n",
        "            'cifar-10-python.tar.gz',\n",
        "            pbar.hook)\n",
        "\n",
        "if not isdir(cifar10_dataset_folder_path):\n",
        "    with tarfile.open('cifar-10-python.tar.gz') as tar:\n",
        "        tar.extractall()\n",
        "        tar.close()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CIFAR-10 Dataset: 171MB [00:47, 3.60MB/s]                           \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "7qZbUCTdU4XX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "\n",
        "\n",
        "def _load_label_names():\n",
        "    \"\"\"\n",
        "    Load the label names from file\n",
        "    \"\"\"\n",
        "    return ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
        "\n",
        "\n",
        "def load_cfar10_batch(cifar10_dataset_folder_path, batch_id):\n",
        "    \"\"\"\n",
        "    Load a batch of the dataset\n",
        "    \"\"\"\n",
        "    with open(cifar10_dataset_folder_path + '/data_batch_' + str(batch_id), mode='rb') as file:\n",
        "        batch = pickle.load(file, encoding='latin1')\n",
        "\n",
        "    features = batch['data'].reshape((len(batch['data']), 3, 32, 32)).transpose(0, 2, 3, 1)\n",
        "    labels = batch['labels']\n",
        "\n",
        "    return features, labels\n",
        "\n",
        "\n",
        "def display_stats(cifar10_dataset_folder_path, batch_id, sample_id):\n",
        "    \"\"\"\n",
        "    Display Stats of the the dataset\n",
        "    \"\"\"\n",
        "    batch_ids = list(range(1, 6))\n",
        "\n",
        "    if batch_id not in batch_ids:\n",
        "        print('Batch Id out of Range. Possible Batch Ids: {}'.format(batch_ids))\n",
        "        return None\n",
        "\n",
        "    features, labels = load_cfar10_batch(cifar10_dataset_folder_path, batch_id)\n",
        "\n",
        "    if not (0 <= sample_id < len(features)):\n",
        "        print('{} samples in batch {}.  {} is out of range.'.format(len(features), batch_id, sample_id))\n",
        "        return None\n",
        "\n",
        "    print('\\nStats of batch {}:'.format(batch_id))\n",
        "    print('Samples: {}'.format(len(features)))\n",
        "    print('Label Counts: {}'.format(dict(zip(*np.unique(labels, return_counts=True)))))\n",
        "    print('First 20 Labels: {}'.format(labels[:20]))\n",
        "\n",
        "    sample_image = features[sample_id]\n",
        "    sample_label = labels[sample_id]\n",
        "    label_names = _load_label_names()\n",
        "\n",
        "    print('\\nExample of Image {}:'.format(sample_id))\n",
        "    print('Image - Min Value: {} Max Value: {}'.format(sample_image.min(), sample_image.max()))\n",
        "    print('Image - Shape: {}'.format(sample_image.shape))\n",
        "    print('Label - Label Id: {} Name: {}'.format(sample_label, label_names[sample_label]))\n",
        "    plt.axis('off')\n",
        "    plt.imshow(sample_image)\n",
        "\n",
        "\n",
        "def _preprocess_and_save(normalize, one_hot_encode, features, labels, filename):\n",
        "    \"\"\"\n",
        "    Preprocess data and save it to file\n",
        "    \"\"\"\n",
        "    features = normalize(features)\n",
        "    labels = one_hot_encode(labels)\n",
        "\n",
        "    pickle.dump((features, labels), open(filename, 'wb'))\n",
        "\n",
        "\n",
        "def preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode):\n",
        "    \"\"\"\n",
        "    Preprocess Training and Validation Data\n",
        "    \"\"\"\n",
        "    n_batches = 5\n",
        "    valid_features = []\n",
        "    valid_labels = []\n",
        "\n",
        "    for batch_i in range(1, n_batches + 1):\n",
        "        features, labels = load_cfar10_batch(cifar10_dataset_folder_path, batch_i)\n",
        "        validation_count = int(len(features) * 0.1)\n",
        "\n",
        "        # Prprocess and save a batch of training data\n",
        "        _preprocess_and_save(\n",
        "            normalize,\n",
        "            one_hot_encode,\n",
        "            features[:-validation_count],\n",
        "            labels[:-validation_count],\n",
        "            'preprocess_batch_' + str(batch_i) + '.p')\n",
        "\n",
        "        # Use a portion of training batch for validation\n",
        "        valid_features.extend(features[-validation_count:])\n",
        "        valid_labels.extend(labels[-validation_count:])\n",
        "\n",
        "    # Preprocess and Save all validation data\n",
        "    _preprocess_and_save(\n",
        "        normalize,\n",
        "        one_hot_encode,\n",
        "        np.array(valid_features),\n",
        "        np.array(valid_labels),\n",
        "        'preprocess_validation.p')\n",
        "\n",
        "    with open(cifar10_dataset_folder_path + '/test_batch', mode='rb') as file:\n",
        "        batch = pickle.load(file, encoding='latin1')\n",
        "\n",
        "    # load the training data\n",
        "    test_features = batch['data'].reshape((len(batch['data']), 3, 32, 32)).transpose(0, 2, 3, 1)\n",
        "    test_labels = batch['labels']\n",
        "\n",
        "    # Preprocess and Save all training data\n",
        "    _preprocess_and_save(\n",
        "        normalize,\n",
        "        one_hot_encode,\n",
        "        np.array(test_features),\n",
        "        np.array(test_labels),\n",
        "        'preprocess_training.p')\n",
        "\n",
        "\n",
        "def batch_features_labels(features, labels, batch_size):\n",
        "    \"\"\"\n",
        "    Split features and labels into batches\n",
        "    \"\"\"\n",
        "    for start in range(0, len(features), batch_size):\n",
        "        end = min(start + batch_size, len(features))\n",
        "        yield features[start:end], labels[start:end]\n",
        "\n",
        "\n",
        "def load_preprocess_training_batch(batch_id, batch_size):\n",
        "    \"\"\"\n",
        "    Load the Preprocessed Training data and return them in batches of <batch_size> or less\n",
        "    \"\"\"\n",
        "    filename = 'preprocess_batch_' + str(batch_id) + '.p'\n",
        "    features, labels = pickle.load(open(filename, mode='rb'))\n",
        "\n",
        "    # Return the training data in batches of size <batch_size> or less\n",
        "    return batch_features_labels(features, labels, batch_size)\n",
        "\n",
        "\n",
        "def display_image_predictions(features, labels, predictions):\n",
        "    n_classes = 10\n",
        "    label_names = _load_label_names()\n",
        "    label_binarizer = LabelBinarizer()\n",
        "    label_binarizer.fit(range(n_classes))\n",
        "    label_ids = label_binarizer.inverse_transform(np.array(labels))\n",
        "\n",
        "    fig, axies = plt.subplots(nrows=4, ncols=2)\n",
        "    fig.tight_layout()\n",
        "    fig.suptitle('Softmax Predictions', fontsize=20, y=1.1)\n",
        "\n",
        "    n_predictions = 3\n",
        "    margin = 0.05\n",
        "    ind = np.arange(n_predictions)\n",
        "    width = (1. - 2. * margin) / n_predictions\n",
        "\n",
        "    for image_i, (feature, label_id, pred_indicies, pred_values) in enumerate(zip(features, label_ids, predictions.indices, predictions.values)):\n",
        "        pred_names = [label_names[pred_i] for pred_i in pred_indicies]\n",
        "        correct_name = label_names[label_id]\n",
        "\n",
        "        axies[image_i][0].imshow(feature*255)\n",
        "        axies[image_i][0].set_title(correct_name)\n",
        "        axies[image_i][0].set_axis_off()\n",
        "\n",
        "        axies[image_i][1].barh(ind + margin, pred_values[::-1], width)\n",
        "        axies[image_i][1].set_yticks(ind + margin)\n",
        "        axies[image_i][1].set_yticklabels(pred_names[::-1])\n",
        "        axies[image_i][1].set_xticks([0, 0.5, 1.0])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aoAyDQ-wYaDP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "03be58b2-d701-4980-bd84-f9745c719e56"
      },
      "cell_type": "code",
      "source": [
        "!ls"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cifar-10-batches-py  cifar-10-python.tar.gz  sample_data\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RmFku-00bmoi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "def normalize(x):\n",
        "  x_max = np.max(x)\n",
        "  x_min = np.min(x)\n",
        "  return (x - x_min) / (x_max - x_min)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7ylNfqvHgvGh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e87938bd-a52a-411d-8042-b45f5dc5538b"
      },
      "cell_type": "code",
      "source": [
        "from sklearn import preprocessing\n",
        "LB = preprocessing.LabelBinarizer()\n",
        "LB.fit(range(10))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LabelBinarizer(neg_label=0, pos_label=1, sparse_output=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "6iiX7ulHg18o",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def one_hot_encode(x):\n",
        "  return LB.transform(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eRZmJ0X9g6b-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WW6Ga_ckg9MV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "be08ca12-8398-4c94-b8f8-1730ebd7cfa0"
      },
      "cell_type": "code",
      "source": [
        "!ls"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cifar-10-batches-py\tpreprocess_batch_3.p   preprocess_validation.p\r\n",
            "cifar-10-python.tar.gz\tpreprocess_batch_4.p   sample_data\r\n",
            "preprocess_batch_1.p\tpreprocess_batch_5.p\r\n",
            "preprocess_batch_2.p\tpreprocess_training.p\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zgxQOLbplqLR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hbok8-4HhBkp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 178
        },
        "outputId": "ab61e579-6d07-481e-eee9-9b37956ef0f2"
      },
      "cell_type": "code",
      "source": [
        "tf.reset_default_graph()\n",
        "\n",
        "# Inputs\n",
        "x = tf.placeholder(tf.float32, shape=(None, 32, 32, 3), name='input_x')\n",
        "y =  tf.placeholder(tf.float32, shape=(None, 10), name='output_y')\n",
        "keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
        "\n",
        "def conv_net(x, keep_prob):\n",
        "    conv1_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 3, 64], mean=0, stddev=0.08))\n",
        "    conv2_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 64, 128], mean=0, stddev=0.08))\n",
        "    conv3_filter = tf.Variable(tf.truncated_normal(shape=[5, 5, 128, 256], mean=0, stddev=0.08))\n",
        "    conv4_filter = tf.Variable(tf.truncated_normal(shape=[5, 5, 256, 512], mean=0, stddev=0.08))\n",
        "\n",
        "    # 1, 2\n",
        "    conv1 = tf.nn.conv2d(x, conv1_filter, strides=[1,1,1,1], padding='SAME')\n",
        "    conv1 = tf.nn.relu(conv1)\n",
        "    conv1_pool = tf.nn.max_pool(conv1, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
        "    conv1_bn = tf.layers.batch_normalization(conv1_pool)\n",
        "\n",
        "    # 3, 4\n",
        "    conv2 = tf.nn.conv2d(conv1_bn, conv2_filter, strides=[1,1,1,1], padding='SAME')\n",
        "    conv2 = tf.nn.relu(conv2)\n",
        "    conv2_pool = tf.nn.max_pool(conv2, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')    \n",
        "    conv2_bn = tf.layers.batch_normalization(conv2_pool)\n",
        "  \n",
        "    # 5, 6\n",
        "    conv3 = tf.nn.conv2d(conv2_bn, conv3_filter, strides=[1,1,1,1], padding='SAME')\n",
        "    conv3 = tf.nn.relu(conv3)\n",
        "    conv3_pool = tf.nn.max_pool(conv3, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')  \n",
        "    conv3_bn = tf.layers.batch_normalization(conv3_pool)\n",
        "    \n",
        "    # 7, 8\n",
        "    conv4 = tf.nn.conv2d(conv3_bn, conv4_filter, strides=[1,1,1,1], padding='SAME')\n",
        "    conv4 = tf.nn.relu(conv4)\n",
        "    conv4_pool = tf.nn.max_pool(conv4, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
        "    conv4_bn = tf.layers.batch_normalization(conv4_pool)\n",
        "    \n",
        "    # 9\n",
        "    flat = tf.contrib.layers.flatten(conv4_bn)  \n",
        "\n",
        "    # 10\n",
        "    full1 = tf.contrib.layers.fully_connected(inputs=flat, num_outputs=128, activation_fn=tf.nn.relu)\n",
        "    full1 = tf.nn.dropout(full1, keep_prob)\n",
        "    full1 = tf.layers.batch_normalization(full1)\n",
        "    \n",
        "    # 11\n",
        "    full2 = tf.contrib.layers.fully_connected(inputs=full1, num_outputs=256, activation_fn=tf.nn.relu)\n",
        "    full2 = tf.nn.dropout(full2, keep_prob)\n",
        "    full2 = tf.layers.batch_normalization(full2)\n",
        "    \n",
        "    # 12\n",
        "    full3 = tf.contrib.layers.fully_connected(inputs=full2, num_outputs=512, activation_fn=tf.nn.relu)\n",
        "    full3 = tf.nn.dropout(full3, keep_prob)\n",
        "    full3 = tf.layers.batch_normalization(full3)    \n",
        "    \n",
        "    # 13\n",
        "    full4 = tf.contrib.layers.fully_connected(inputs=full3, num_outputs=1024, activation_fn=tf.nn.relu)\n",
        "    full4 = tf.nn.dropout(full4, keep_prob)\n",
        "    full4 = tf.layers.batch_normalization(full4)        \n",
        "    \n",
        "    # 14\n",
        "    out = tf.contrib.layers.fully_connected(inputs=full3, num_outputs=10, activation_fn=None)\n",
        "    return out\n",
        "\n",
        "epochs = 10\n",
        "batch_size = 128\n",
        "keep_probability = 0.7\n",
        "learning_rate = 0.001\n",
        "\n",
        "logits = conv_net(x, keep_prob)\n",
        "model = tf.identity(logits, name='logits')\n",
        "\n",
        "# Loss and Optimizer\n",
        "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
        "\n",
        "# Accuracy\n",
        "correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))\n",
        "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-14-4f20b172e36e>:74: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "\n",
            "Future major versions of TensorFlow will allow gradients to flow\n",
            "into the labels input on backprop by default.\n",
            "\n",
            "See @{tf.nn.softmax_cross_entropy_with_logits_v2}.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "00TEishXjdHO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):\n",
        "    session.run(optimizer, \n",
        "                feed_dict={\n",
        "                    x: feature_batch,\n",
        "                    y: label_batch,\n",
        "                    keep_prob: keep_probability\n",
        "                })"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GJclNtvHlNiO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def print_stats(session, feature_batch, label_batch, cost, accuracy):\n",
        "    loss = sess.run(cost, \n",
        "                    feed_dict={\n",
        "                        x: feature_batch,\n",
        "                        y: label_batch,\n",
        "                        keep_prob: 1.\n",
        "                    })\n",
        "    valid_acc = sess.run(accuracy, \n",
        "                         feed_dict={\n",
        "                             x: valid_features,\n",
        "                             y: valid_labels,\n",
        "                             keep_prob: 1.\n",
        "                         })\n",
        "    \n",
        "    print('Loss: {:>10.4f} Validation Accuracy: {:.6f}'.format(loss, valid_acc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZSOJ6U_olRz9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 905
        },
        "outputId": "30e4036b-2eff-48eb-f3d4-5b2c22eaf4b9"
      },
      "cell_type": "code",
      "source": [
        "save_model_path = './image_classification'\n",
        "with tf.Session() as sess:\n",
        "    # Initializing the variables\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    \n",
        "    # Training cycle\n",
        "    for epoch in range(epochs):\n",
        "        # Loop over all batches\n",
        "        n_batches = 5\n",
        "        for batch_i in range(1, n_batches + 1):\n",
        "            for batch_features, batch_labels in load_preprocess_training_batch(batch_i, batch_size):\n",
        "                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)\n",
        "                \n",
        "            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')\n",
        "            print_stats(sess, batch_features, batch_labels, cost, accuracy)\n",
        "    saver = tf.train.Saver()\n",
        "    save_path = saver.save(sess, save_model_path)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch  1, CIFAR-10 Batch 1:  Loss:     2.2212 Validation Accuracy: 0.223000\n",
            "Epoch  1, CIFAR-10 Batch 2:  Loss:     1.8779 Validation Accuracy: 0.263000\n",
            "Epoch  1, CIFAR-10 Batch 3:  Loss:     1.6098 Validation Accuracy: 0.307200\n",
            "Epoch  1, CIFAR-10 Batch 4:  Loss:     1.5532 Validation Accuracy: 0.353600\n",
            "Epoch  1, CIFAR-10 Batch 5:  Loss:     1.4270 Validation Accuracy: 0.369000\n",
            "Epoch  2, CIFAR-10 Batch 1:  Loss:     1.2796 Validation Accuracy: 0.491200\n",
            "Epoch  2, CIFAR-10 Batch 2:  Loss:     1.2753 Validation Accuracy: 0.497200\n",
            "Epoch  2, CIFAR-10 Batch 3:  Loss:     0.9487 Validation Accuracy: 0.577400\n",
            "Epoch  2, CIFAR-10 Batch 4:  Loss:     1.0787 Validation Accuracy: 0.595400\n",
            "Epoch  2, CIFAR-10 Batch 5:  Loss:     0.7656 Validation Accuracy: 0.608200\n",
            "Epoch  3, CIFAR-10 Batch 1:  Loss:     0.7313 Validation Accuracy: 0.639000\n",
            "Epoch  3, CIFAR-10 Batch 2:  Loss:     0.6231 Validation Accuracy: 0.633000\n",
            "Epoch  3, CIFAR-10 Batch 3:  Loss:     0.5073 Validation Accuracy: 0.669000\n",
            "Epoch  3, CIFAR-10 Batch 4:  Loss:     0.7080 Validation Accuracy: 0.660400\n",
            "Epoch  3, CIFAR-10 Batch 5:  Loss:     0.5399 Validation Accuracy: 0.673800\n",
            "Epoch  4, CIFAR-10 Batch 1:  Loss:     0.5155 Validation Accuracy: 0.686800\n",
            "Epoch  4, CIFAR-10 Batch 2:  Loss:     0.3037 Validation Accuracy: 0.703000\n",
            "Epoch  4, CIFAR-10 Batch 3:  Loss:     0.2483 Validation Accuracy: 0.703600\n",
            "Epoch  4, CIFAR-10 Batch 4:  Loss:     0.3564 Validation Accuracy: 0.721400\n",
            "Epoch  4, CIFAR-10 Batch 5:  Loss:     0.2547 Validation Accuracy: 0.723600\n",
            "Epoch  5, CIFAR-10 Batch 1:  Loss:     0.2437 Validation Accuracy: 0.700800\n",
            "Epoch  5, CIFAR-10 Batch 2:  Loss:     0.1516 Validation Accuracy: 0.717800\n",
            "Epoch  5, CIFAR-10 Batch 3:  Loss:     0.1896 Validation Accuracy: 0.662800\n",
            "Epoch  5, CIFAR-10 Batch 4:  Loss:     0.1647 Validation Accuracy: 0.722400\n",
            "Epoch  5, CIFAR-10 Batch 5:  Loss:     0.0898 Validation Accuracy: 0.729600\n",
            "Epoch  6, CIFAR-10 Batch 1:  Loss:     0.1140 Validation Accuracy: 0.719400\n",
            "Epoch  6, CIFAR-10 Batch 2:  Loss:     0.1198 Validation Accuracy: 0.734200\n",
            "Epoch  6, CIFAR-10 Batch 3:  Loss:     0.1376 Validation Accuracy: 0.723800\n",
            "Epoch  6, CIFAR-10 Batch 4:  Loss:     0.1084 Validation Accuracy: 0.731000\n",
            "Epoch  6, CIFAR-10 Batch 5:  Loss:     0.0462 Validation Accuracy: 0.736800\n",
            "Epoch  7, CIFAR-10 Batch 1:  Loss:     0.0413 Validation Accuracy: 0.744200\n",
            "Epoch  7, CIFAR-10 Batch 2:  Loss:     0.0778 Validation Accuracy: 0.753000\n",
            "Epoch  7, CIFAR-10 Batch 3:  Loss:     0.0364 Validation Accuracy: 0.732400\n",
            "Epoch  7, CIFAR-10 Batch 4:  Loss:     0.0376 Validation Accuracy: 0.731800\n",
            "Epoch  7, CIFAR-10 Batch 5:  Loss:     0.0304 Validation Accuracy: 0.740200\n",
            "Epoch  8, CIFAR-10 Batch 1:  Loss:     0.0349 Validation Accuracy: 0.723400\n",
            "Epoch  8, CIFAR-10 Batch 2:  Loss:     0.0297 Validation Accuracy: 0.750600\n",
            "Epoch  8, CIFAR-10 Batch 3:  Loss:     0.0214 Validation Accuracy: 0.742000\n",
            "Epoch  8, CIFAR-10 Batch 4:  Loss:     0.0327 Validation Accuracy: 0.741000\n",
            "Epoch  8, CIFAR-10 Batch 5:  Loss:     0.0173 Validation Accuracy: 0.742800\n",
            "Epoch  9, CIFAR-10 Batch 1:  Loss:     0.0233 Validation Accuracy: 0.749600\n",
            "Epoch  9, CIFAR-10 Batch 2:  Loss:     0.0146 Validation Accuracy: 0.744000\n",
            "Epoch  9, CIFAR-10 Batch 3:  Loss:     0.0215 Validation Accuracy: 0.752400\n",
            "Epoch  9, CIFAR-10 Batch 4:  Loss:     0.0414 Validation Accuracy: 0.710600\n",
            "Epoch  9, CIFAR-10 Batch 5:  Loss:     0.0076 Validation Accuracy: 0.733800\n",
            "Epoch 10, CIFAR-10 Batch 1:  Loss:     0.0182 Validation Accuracy: 0.745200\n",
            "Epoch 10, CIFAR-10 Batch 2:  Loss:     0.0052 Validation Accuracy: 0.753000\n",
            "Epoch 10, CIFAR-10 Batch 3:  Loss:     0.0039 Validation Accuracy: 0.753000\n",
            "Epoch 10, CIFAR-10 Batch 4:  Loss:     0.0091 Validation Accuracy: 0.754400\n",
            "Epoch 10, CIFAR-10 Batch 5:  Loss:     0.0056 Validation Accuracy: 0.756400\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VXOhBq7qmcOZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "7be1a87c-aa29-46fd-939e-4c32167b2047"
      },
      "cell_type": "code",
      "source": [
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "\n",
        "import tensorflow as tf\n",
        "import pickle\n",
        "import random\n",
        "\n",
        "save_model_path = './image_classification'\n",
        "batch_size = 64\n",
        "n_samples = 10\n",
        "top_n_predictions = 5\n",
        "\n",
        "def test_model():\n",
        "    test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))\n",
        "    loaded_graph = tf.Graph()\n",
        "\n",
        "    with tf.Session(graph=loaded_graph) as sess:\n",
        "        # Load model\n",
        "        loader = tf.train.import_meta_graph(save_model_path + '.meta')\n",
        "        loader.restore(sess, save_model_path)\n",
        "\n",
        "        # Get Tensors from loaded model\n",
        "        loaded_x = loaded_graph.get_tensor_by_name('input_x:0')\n",
        "        loaded_y = loaded_graph.get_tensor_by_name('output_y:0')\n",
        "        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
        "        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')\n",
        "        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')\n",
        "        \n",
        "        # Get accuracy in batches for memory limitations\n",
        "        test_batch_acc_total = 0\n",
        "        test_batch_count = 0\n",
        "        \n",
        "        for train_feature_batch, train_label_batch in batch_features_labels(test_features, test_labels, batch_size):\n",
        "            test_batch_acc_total += sess.run(\n",
        "                loaded_acc,\n",
        "                feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})\n",
        "            test_batch_count += 1\n",
        "\n",
        "        print('Testing Accuracy: {}\\n'.format(test_batch_acc_total/test_batch_count))\n",
        "\n",
        "test_model()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from ./image_classification\n",
            "Testing Accuracy: 0.7363654458598726\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "rLxHZi4xI_ge",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}