session-1/session-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 1 - Introduction to Tensorflow\n",
    "<p class=\"lead\">\n",
    "Assignment: Creating a Dataset/Computing with Tensorflow\n",
    "</p>\n",
    "\n",
    "<p class=\"lead\">\n",
    "Parag K. Mital<br />\n",
    "<a href=\"https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\">Creative Applications of Deep Learning w/ Tensorflow</a><br />\n",
    "<a href=\"https://www.kadenze.com/partners/kadenze-academy\">Kadenze Academy</a><br />\n",
    "<a href=\"https://twitter.com/hashtag/CADL\">#CADL</a>\n",
    "</p>\n",
    "\n",
    "This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Goals\n",
    "\n",
    "* Learn how to normalize a dataset by calculating the mean/std. deviation\n",
    "* Learn how to use convolution\n",
    "* Explore what representations exist in your dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outline\n",
    "\n",
    "<!-- MarkdownTOC autolink=true autoanchor=true bracket=round -->\n",
    "\n",
    "- [Assignment Synopsis](#assignment-synopsis)\n",
    "- [Part One - Create a Small Dataset](#part-one---create-a-small-dataset)\n",
    "  - [Instructions](#instructions)\n",
    "  - [Code](#code)\n",
    "- [Part Two - Compute the Mean](#part-two---compute-the-mean)\n",
    "  - [Instructions](#instructions-1)\n",
    "  - [Code](#code-1)\n",
    "- [Part Three - Compute the Standard Deviation](#part-three---compute-the-standard-deviation)\n",
    "  - [Instructions](#instructions-2)\n",
    "  - [Code](#code-2)\n",
    "- [Part Four - Normalize the Dataset](#part-four---normalize-the-dataset)\n",
    "  - [Instructions](#instructions-3)\n",
    "  - [Code](#code-3)\n",
    "- [Part Five - Convolve the Dataset](#part-five---convolve-the-dataset)\n",
    "  - [Instructions](#instructions-4)\n",
    "  - [Code](#code-4)\n",
    "- [Part Six - Sort the Dataset](#part-six---sort-the-dataset)\n",
    "  - [Instructions](#instructions-5)\n",
    "  - [Code](#code-5)\n",
    "- [Assignment Submission](#assignment-submission)\n",
    "\n",
    "<!-- /MarkdownTOC -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Notebook</h1>\n",
    "\n",
    "Everything you will need to do will be inside of this notebook, and I've marked which cells you will need to edit by saying <b><font color='red'>\"TODO! COMPLETE THIS SECTION!\"</font></b>.  For you to work with this notebook, you'll either download the zip file from the resources section on Kadenze or clone the github repo (whichever you are more comfortable with), and then run notebook inside the same directory as wherever this file is located using the command line \"jupyter notebook\" or \"ipython notebook\" (using Terminal on Unix/Linux/OSX, or Command Line/Shell/Powershell on Windows).  If you are unfamiliar with jupyter notebook, please look at [Installation Preliminaries](https://github.com/pkmital/CADL/blob/master/README.md#installation-preliminaries) and [Session 0](https://github.com/pkmital/CADL/blob/master/session-0/session-0.ipynb) before starting!\n",
    "\n",
    "Once you have launched notebook, this will launch a web browser with the contents of the zip files listed.  Click the file \"session-1.ipynb\" and this document will open in an interactive notebook, allowing you to \"run\" the cells, computing them using python, and edit the text inside the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment-synopsis\"></a>\n",
    "# Assignment Synopsis\n",
    "\n",
    "This first homework assignment will guide you through working with a small dataset of images.  For Part 1, you'll need to find 100 images and use the function I've provided to create a montage of your images, saving it to the file \"dataset.png\" (template code provided below).  You can load an existing dataset of images, find your own images, or perhaps create your own images using a creative process such as painting, photography, or something along those lines.  Each image will be reshaped to 100 x 100 pixels.  There needs to be at least 100 images.  For Parts 2 and 3, you'll then calculate the mean and deviation of it using a tensorflow session. In Part 4,  you'll normalize your dataset using the mean and deviation.  Then in Part 5, you will convolve your normalized dataset.  For Part 6, you'll need to sort the entire convolved dataset.  Finally, the last part will package everything for you in a zip file which you can upload to Kadenze to get assessed (only if you are a Kadenze Premium member, $10 p/m, free for the first month).  Remember to complete the additional excercises online, including the Gallery participation and the Forum post.  If you have any questions, be sure to enroll in the course and ask your peers in the \\#CADL community or me on the forums!\n",
    "\n",
    "https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\n",
    "\n",
    "The following assignment breakdown gives more detailed instructions and includes template code for you to fill out.  Good luck!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-one---create-a-small-dataset\"></a>\n",
    "# Part One - Create a Small Dataset\n",
    "\n",
    "<a name=\"instructions\"></a>\n",
    "## Instructions\n",
    "\n",
    "Use Python, Numpy, and Matplotlib to load a dataset of 100 images and create a montage of the dataset as a 10 x 10 image using the function below. You'll need to make sure you call the function using a 4-d array of `N x H x W x C` dimensions, meaning every image will need to be the same size! You can load an existing dataset of images, find your own images, or perhaps create your own images using a creative process such as painting, photography, or something along those lines.\n",
    "\n",
    "When you are creating your dataset, I want you to think about what representations might exist in the limited amount of data that you are organizing.  It is only 100 images after all, not a whole lot for a computer to reason about and learn something meaningful.  So <b>think about creating a *dataset* of images that could possibly reveal something fundamental about what is contained in the images</b>.  Try to think about creating a set of images that represents something.  For instance, this might be images of yourself over time.  Or it might be every picture you've ever taken of your cat.  Or perhaps the view from your room at different times of the day.  Consider making the changes within each image as significant as possible.  As \"representative\" of the thing you want to capture as possible.  Hopefully by the end of this lesson, you'll understand a little better the difference between what a computer thinks is significant and what you yourself thought was significant.\n",
    "\n",
    "The code below will show you how to resize and/or crop your images so that they are 100 pixels x 100 pixels in height and width.  Once you have 100 images loaded, we'll use a `montage` function to draw and save your dataset to the file <b>dataset.png</b>.\n",
    "\n",
    "<a name=\"code\"></a>\n",
    "## Code\n",
    "\n",
    "This next section will just make sure you have the right version of python and the libraries that we'll be using.  Don't change the code here but make sure you \"run\" it (use \"shift+enter\")!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First check the Python version\n",
    "import sys\n",
    "if sys.version_info < (3,4):\n",
    "    print('You are running an older version of Python!\\n\\n' \\\n",
    "          'You should consider updating to Python 3.4.0 or ' \\\n",
    "          'higher as the libraries built for this course ' \\\n",
    "          'have only been tested in Python 3.4 and higher.\\n')\n",
    "    print('Try installing the Python 3.5 version of anaconda '\n",
    "          'and then restart `jupyter notebook`:\\n' \\\n",
    "          'https://www.continuum.io/downloads\\n\\n')\n",
    "\n",
    "# Now get necessary libraries\n",
    "try:\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "except ImportError:\n",
    "    print('You are missing some packages! ' \\\n",
    "          'We will try installing them before continuing!')\n",
    "    !pip install \"numpy>=1.11.0\" \"matplotlib>=1.5.1\" \"scikit-image>=0.11.3\" \"scikit-learn>=0.17\"\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    print('Done!')\n",
    "\n",
    "# Import Tensorflow\n",
    "try:\n",
    "    import tensorflow as tf\n",
    "except ImportError:\n",
    "    print(\"You do not have tensorflow installed!\")\n",
    "    print(\"Follow the instructions on the following link\")\n",
    "    print(\"to install tensorflow before continuing:\")\n",
    "    print(\"\")\n",
    "    print(\"https://github.com/pkmital/CADL#installation-preliminaries\")\n",
    "\n",
    "# This cell includes the provided libraries from the zip file\n",
    "try:\n",
    "    from libs import utils\n",
    "except ImportError:\n",
    "    print(\"Make sure you have started notebook in the same directory\" +\n",
    "          \" as the provided zip file which includes the 'libs' folder\" +\n",
    "          \" and the file 'utils.py' inside of it.  You will NOT be able\"\n",
    "          \" to complete this assignment unless you restart jupyter\"\n",
    "          \" notebook inside the directory created by extracting\"\n",
    "          \" the zip file or cloning the github repo.\")\n",
    "\n",
    "# We'll tell matplotlib to inline any drawn figures like so:\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style> .rendered_html code { \n",
       "    padding: 2px 4px;\n",
       "    color: #c7254e;\n",
       "    background-color: #f9f2f4;\n",
       "    border-radius: 4px;\n",
       "} </style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bit of formatting because inline code is not styled very good by default:\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Places your images in a folder such as `dirname = '/Users/Someone/Desktop/ImagesFromTheInternet'`.  We'll then use the `os` package to load them and crop/resize them to a standard size of 100 x 100 pixels.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You need to find 100 images from the web/create them yourself\n",
    "# or find a dataset that interests you (e.g. I used celeb faces\n",
    "# in the course lecture...)\n",
    "# then store them all in a single directory.\n",
    "# With all the images in a single directory, you can then\n",
    "# perform the following steps to create a 4-d array of:\n",
    "# N x H x W x C dimensions as 100 x 100 x 100 x 3.\n",
    "\n",
    "dirname = ...\n",
    "\n",
    "# Load every image file in the provided directory\n",
    "filenames = [os.path.join(dirname, fname)\n",
    "             for fname in os.listdir(dirname)]\n",
    "\n",
    "# Make sure we have exactly 100 image files!\n",
    "filenames = filenames[:100]\n",
    "assert(len(filenames) == 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Read every filename as an RGB image\n",
    "imgs = [plt.imread(fname)[..., :3] for fname in filenames]\n",
    "\n",
    "# Crop every image to a square\n",
    "imgs = [utils.imcrop_tosquare(img_i) for img_i in imgs]\n",
    "\n",
    "# Then resize the square image to 100 x 100 pixels\n",
    "imgs = [resize(img_i, (100, 100)) for img_i in imgs]\n",
    "\n",
    "# Finally make our list of 3-D images a 4-D array with the first dimension the number of images:\n",
    "imgs = np.array(imgs).astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot the resulting dataset:\n",
    "# Make sure you \"run\" this cell after you create your `imgs` variable as a 4-D array!\n",
    "# Make sure we have a 100 x 100 x 100 x 3 dimension array\n",
    "assert(imgs.shape == (100, 100, 100, 3))\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(utils.montage(imgs, saveto='dataset.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-two---compute-the-mean\"></a>\n",
    "# Part Two - Compute the Mean\n",
    "\n",
    "<a name=\"instructions-1\"></a>\n",
    "## Instructions\n",
    "\n",
    "First use Tensorflow to define a session.  Then use Tensorflow to create an operation which takes your 4-d array and calculates the mean color image (100 x 100 x 3) using the function `tf.reduce_mean`.  Have a look at the documentation for this function to see how it works in order to get the mean of every pixel and get an image of (100 x 100 x 3) as a result.  You'll then calculate the mean image by running the operation you create with your session (e.g. <code>sess.run(...)</code>).  Finally, plot the mean image, save it, and then include this image in your zip file as <b>mean.png</b>.\n",
    "\n",
    "<a name=\"code-1\"></a>\n",
    "## Code\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First create a tensorflow session\n",
    "sess = ...\n",
    "\n",
    "# Now create an operation that will calculate the mean of your images\n",
    "mean_img_op = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# And then run that operation using your session\n",
    "mean_img = sess.run(mean_img_op)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then plot the resulting mean image:\n",
    "# Make sure the mean image is the right size!\n",
    "assert(mean_img.shape == (100, 100, 3))\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(mean_img)\n",
    "plt.imsave(arr=mean_img, fname='mean.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have seen the mean image of your dataset, how does it relate to your own expectations of the dataset?  Did you expect something different?  Was there something more \"regular\" or \"predictable\" about your dataset that the mean image did or did not reveal?  If your mean image looks a lot like something recognizable, it's a good sign that there is a lot of predictability in your dataset.  If your mean image looks like nothing at all, a gray blob where not much seems to stand out, then it's pretty likely that there isn't very much in common between your images.  Neither is a bad scenario.  Though, it is more likely that having some predictability in your mean image, e.g. something recognizable, that there are representations worth exploring with deeper networks capable of representing them.  However, we're only using 100 images so it's a *very* small dataset to begin with.\n",
    "\n",
    "<a name=\"part-three---compute-the-standard-deviation\"></a>\n",
    "# Part Three - Compute the Standard Deviation\n",
    "\n",
    "<a name=\"instructions-2\"></a>\n",
    "## Instructions\n",
    "\n",
    "Now use tensorflow to calculate the standard deviation and upload the standard deviation image averaged across color channels as a \"jet\" heatmap of the 100 images.  This will be a little more involved as there is no operation in tensorflow to do this for you.  However, you can do this by calculating the mean image of your dataset as a 4-D array.  To do this, you could write e.g.  `mean_img_4d = tf.reduce_mean(imgs, axis=0, keep_dims=True)` to give you a `1 x H x W x C` dimension array calculated on the `N x H x W x C` images variable.  The axis parameter is saying to calculate the mean over the 0th dimension, meaning for every possible `H`, `W`, `C`, or for every pixel, you will have a mean composed over the `N` possible values it could have had, or what that pixel was for every possible image.  This way, you can write `images - mean_img_4d` to give you a `N x H x W x C` dimension variable, with every image in your images array having been subtracted by the `mean_img_4d`.  If you calculate the square root of the expected squared differences of this resulting operation, you have your standard deviation!\n",
    "\n",
    "In summary, you'll need to write something like: `subtraction = imgs - tf.reduce_mean(imgs, axis=0, keep_dims=True)`, then reduce this operation using `tf.sqrt(tf.reduce_mean(subtraction * subtraction, axis=0))` to get your standard deviation then include this image in your zip file as <b>std.png</b>\n",
    "\n",
    "<a name=\"code-2\"></a>\n",
    "## Code\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a tensorflow operation to give you the standard deviation\n",
    "\n",
    "# First compute the difference of every image with a\n",
    "# 4 dimensional mean image shaped 1 x H x W x C\n",
    "mean_img_4d = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "subtraction = imgs - mean_img_4d\n",
    "\n",
    "# Now compute the standard deviation by calculating the\n",
    "# square root of the expected squared differences\n",
    "std_img_op = tf.sqrt(tf.reduce_mean(subtraction * subtraction, axis=0))\n",
    "\n",
    "# Now calculate the standard deviation using your session\n",
    "std_img = sess.run(std_img_op)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then plot the resulting standard deviation image:\n",
    "# Make sure the std image is the right size!\n",
    "assert(std_img.shape == (100, 100) or std_img.shape == (100, 100, 3))\n",
    "plt.figure(figsize=(10, 10))\n",
    "std_img_show = std_img / np.max(std_img)\n",
    "plt.imshow(std_img_show)\n",
    "plt.imsave(arr=std_img_show, fname='std.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have plotted your dataset's standard deviation per pixel, what does it reveal about your dataset?  Like with the mean image, you should consider what is predictable and not predictable about this image.\n",
    "\n",
    "<a name=\"part-four---normalize-the-dataset\"></a>\n",
    "# Part Four - Normalize the Dataset\n",
    "\n",
    "<a name=\"instructions-3\"></a>\n",
    "## Instructions\n",
    "Using tensorflow, we'll attempt to normalize your dataset using the mean and standard deviation. \n",
    "\n",
    "<a name=\"code-3\"></a>\n",
    "## Code\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "norm_imgs_op = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "norm_imgs = sess.run(norm_imgs_op)\n",
    "print(np.min(norm_imgs), np.max(norm_imgs))\n",
    "print(imgs.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then plot the resulting normalized dataset montage:\n",
    "# Make sure we have a 100 x 100 x 100 x 3 dimension array\n",
    "assert(norm_imgs.shape == (100, 100, 100, 3))\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(utils.montage(norm_imgs, 'normalized.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply another type of normalization to 0-1 just for the purposes of plotting the image.  If we didn't do this, the range of our values would be somewhere between -1 and 1, and matplotlib would not be able to interpret the entire range of values.  By rescaling our -1 to 1 valued images to 0-1, we can visualize it better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "norm_imgs_show = (norm_imgs - np.min(norm_imgs)) / (np.max(norm_imgs) - np.min(norm_imgs))\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(utils.montage(norm_imgs_show, 'normalized.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-five---convolve-the-dataset\"></a>\n",
    "# Part Five - Convolve the Dataset\n",
    "\n",
    "<a name=\"instructions-4\"></a>\n",
    "## Instructions\n",
    "Using tensorflow, we'll attempt to convolve your dataset with one of the kernels we created during the lesson, and then in the next part, we'll take the sum of the convolved output to use for sorting.  You should use the function `utils.gabor` to create an edge detector.  You can also explore with the `utils.gauss2d` kernel.  What you must figure out is how to reshape your kernel to be 4-dimensional: K_H, K_W, C_I, and C_O, corresponding to the kernel's height and width (e.g. 16), the number of input channels (RGB = 3 input channels), and the number of output channels, (1).\n",
    "\n",
    "<a name=\"code-4\"></a>\n",
    "## Code\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First build 3 kernels for each input color channel\n",
    "ksize = ...\n",
    "kernel = np.concatenate([utils.gabor(ksize)[:, :, np.newaxis] for i in range(3)], axis=2)\n",
    "                       \n",
    "# Now make the kernels into the shape: [ksize, ksize, 3, 1]:\n",
    "kernel_4d = ...\n",
    "assert(kernel_4d.shape == (ksize, ksize, 3, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll Perform the convolution with the 4d tensor in `kernel_4d`.  This is a `ksize` x `ksize` x 3 x 1 tensor, where each input color channel corresponds to one filter with 1 output.  Each filter looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5, 5))\n",
    "plt.imshow(kernel_4d[:, :, 0, 0], cmap='gray')\n",
    "plt.imsave(arr=kernel_4d[:, :, 0, 0], fname='kernel.png', cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform the convolution with the 4d tensors:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "convolved = utils.convolve(..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "convolved_show = (convolved - np.min(convolved)) / (np.max(convolved) - np.min(convolved))\n",
    "print(convolved_show.shape)\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(utils.montage(convolved_show[..., 0], 'convolved.png'), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we've just done is build a \"hand-crafted\" feature detector: the Gabor Kernel.  This kernel is built to respond to particular orientation: horizontal edges, and a particular scale.  It also responds equally to R, G, and B color channels, as that is how we have told the convolve operation to work: use the same kernel for every input color channel.  When we work with deep networks, we'll see how we can *learn* the convolution kernels for every color channel, and learn many more of them, in the order of 100s per color channel.  That is really where the power of deep networks will start to become obvious.  For now, we've seen just how difficult it is to get at any higher order features of the dataset.  We've really only picked out some edges!\n",
    "\n",
    "<a name=\"part-six---sort-the-dataset\"></a>\n",
    "# Part Six - Sort the Dataset\n",
    "\n",
    "<a name=\"instructions-5\"></a>\n",
    "## Instructions\n",
    "Using tensorflow, we'll attempt to organize your dataset. We'll try sorting based on the mean value of each convolved image's output to use for sorting.  To do this, we could calculate either the sum value (`tf.reduce_sum`) or the mean value (`tf.reduce_mean`) of each image in your dataset and then use those values, e.g. stored inside a variable `values` to sort your images using something like `tf.nn.top_k` and `sorted_imgs = np.array([imgs[idx_i] for idx_i in idxs])` prior to creating the montage image, `m = montage(sorted_imgs, \"sorted.png\")` and then include this image in your zip file as <b>sorted.png</b>\n",
    "\n",
    "<a name=\"code-5\"></a>\n",
    "## Code\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a set of operations using tensorflow which could\n",
    "# provide you for instance the sum or mean value of every\n",
    "# image in your dataset:\n",
    "\n",
    "# First flatten our convolved images so instead of many 3d images,\n",
    "# we have many 1d vectors.\n",
    "# This should convert our 4d representation of N x H x W x C to a\n",
    "# 2d representation of N x (H*W*C)\n",
    "flattened = tf.reshape(convolved...\n",
    "assert(flattened.get_shape().as_list() == [100, 10000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now calculate some statistics about each of our images\n",
    "values = tf.reduce_sum(flattened, axis=1)\n",
    "\n",
    "# Then create another operation which sorts those values\n",
    "# and then calculate the result:\n",
    "idxs_op = tf.nn.top_k(values, k=100)[1]\n",
    "idxs = sess.run(idxs_op)\n",
    "\n",
    "# Then finally use the sorted indices to sort your images:\n",
    "sorted_imgs = np.array([imgs[idx_i] for idx_i in idxs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then plot the resulting sorted dataset montage:\n",
    "# Make sure we have a 100 x 100 x 100 x 3 dimension array\n",
    "assert(sorted_imgs.shape == (100, 100, 100, 3))\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(utils.montage(sorted_imgs, 'sorted.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does your sorting reveal?  Could you imagine the same sorting over many more images reveal the thing your dataset sought to represent?  It is likely that the representations that you wanted to find hidden within \"higher layers\", i.e., \"deeper features\" of the image, and that these \"low level\" features, edges essentially, are not very good at describing the really interesting aspects of your dataset.  In later sessions, we'll see how we can combine the outputs of many more convolution kernels that have been assembled in a way that accentuate something very particular about each image, and build a sorting that is much more intelligent than this one!\n",
    "\n",
    "<a name=\"assignment-submission\"></a>\n",
    "# Assignment Submission\n",
    "\n",
    "Now that you've completed all 6 parts, we'll create a zip file of the current directory using the code below.  This code will make sure you have included this completed ipython notebook and the following files named exactly as:\n",
    "\n",
    "<pre>\n",
    "    session-1/\n",
    "      session-1.ipynb\n",
    "      dataset.png\n",
    "      mean.png\n",
    "      std.png\n",
    "      normalized.png\n",
    "      kernel.png\n",
    "      convolved.png\n",
    "      sorted.png\n",
    "      libs/\n",
    "        utils.py\n",
    "</pre>\n",
    "\n",
    "You'll then submit this zip file for your first assignment on Kadenze for \"Assignment 1: Datasets/Computing with Tensorflow\"!  If you have any questions, remember to reach out on the forums and connect with your peers or with me.\n",
    "\n",
    "<b>To get assessed, you'll need to be a premium student.</b>  If you aren't already enrolled as a student, register now at http://www.kadenze.com/ and join the #CADL community to see what your peers are doing! https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\n",
    "\n",
    "Then remember to complete the remaining parts of Assignemnt 1 on Kadenze!:\n",
    "* Comment on 1 student's open-ended arrangement (Part 6) in the course gallery titled \"Creating a Dataset/ Computing with Tensorflow\". Think about what images they've used in their dataset and how the arrangement reflects what could be represented by that data.\n",
    "* Finally make a forum post in the forum for this assignment \"Creating a Dataset/ Computing with Tensorflow\".\n",
    "    - Including a link to an artist making use of machine learning to organize data or finding representations within large datasets\n",
    "    - Tell a little about their work (min 20 words).\n",
    "    - Comment on at least 2 other student's forum posts (min 20 words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure your notebook is named \"session-1\" or else replace it with the correct name in the list of files below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "utils.build_submission('session-1.zip',\n",
    "                       ('dataset.png',\n",
    "                        'mean.png',\n",
    "                        'std.png',\n",
    "                        'normalized.png',\n",
    "                        'kernel.png',\n",
    "                        'convolved.png',\n",
    "                        'sorted.png',\n",
    "                        'session-1.ipynb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}