session-5/session-5-part-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 5: Generative Networks\n",
    "## Assignment: Generative Adversarial Networks and Recurrent Neural Networks\n",
    "\n",
    "<p class=\"lead\">\n",
    "<a href=\"https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\">Creative Applications of Deep Learning with Google's Tensorflow</a><br />\n",
    "<a href=\"http://pkmital.com\">Parag K. Mital</a><br />\n",
    "<a href=\"https://www.kadenze.com\">Kadenze, Inc.</a>\n",
    "</p>\n",
    "\n",
    "# Table of Contents\n",
    "\n",
    "<!-- MarkdownTOC autolink=\"true\" autoanchor=\"true\" bracket=\"round\" -->\n",
    "\n",
    "- [Overview](#overview)\n",
    "- [Learning Goals](#learning-goals)\n",
    "- [Part 1 - Generative Adversarial Networks \\(GAN\\) / Deep Convolutional GAN \\(DCGAN\\)](#part-1---generative-adversarial-networks-gan--deep-convolutional-gan-dcgan)\n",
    "  - [Introduction](#introduction)\n",
    "  - [Building the Encoder](#building-the-encoder)\n",
    "  - [Building the Discriminator for the Training Samples](#building-the-discriminator-for-the-training-samples)\n",
    "  - [Building the Decoder](#building-the-decoder)\n",
    "  - [Building the Generator](#building-the-generator)\n",
    "  - [Building the Discriminator for the Generated Samples](#building-the-discriminator-for-the-generated-samples)\n",
    "  - [GAN Loss Functions](#gan-loss-functions)\n",
    "  - [Building the Optimizers w/ Regularization](#building-the-optimizers-w-regularization)\n",
    "  - [Loading a Dataset](#loading-a-dataset)\n",
    "  - [Training](#training)\n",
    "  - [Equilibrium](#equilibrium)\n",
    "- [Part 2 - Variational Auto-Encoding Generative Adversarial Network \\(VAEGAN\\)](#part-2---variational-auto-encoding-generative-adversarial-network-vaegan)\n",
    "  - [Batch Normalization](#batch-normalization)\n",
    "  - [Building the Encoder](#building-the-encoder-1)\n",
    "  - [Building the Variational Layer](#building-the-variational-layer)\n",
    "  - [Building the Decoder](#building-the-decoder-1)\n",
    "  - [Building VAE/GAN Loss Functions](#building-vaegan-loss-functions)\n",
    "  - [Creating the Optimizers](#creating-the-optimizers)\n",
    "  - [Loading the Dataset](#loading-the-dataset)\n",
    "  - [Training](#training-1)\n",
    "- [Part 3 - Latent-Space Arithmetic](#part-3---latent-space-arithmetic)\n",
    "  - [Loading the Pre-Trained Model](#loading-the-pre-trained-model)\n",
    "  - [Exploring the Celeb Net Attributes](#exploring-the-celeb-net-attributes)\n",
    "  - [Find the Latent Encoding for an Attribute](#find-the-latent-encoding-for-an-attribute)\n",
    "  - [Latent Feature Arithmetic](#latent-feature-arithmetic)\n",
    "  - [Extensions](#extensions)\n",
    "- [Part 4 - Character-Level Language Model](session-5-part-2.ipynb#part-4---character-level-language-model)\n",
    "- [Part 5 - Pretrained Char-RNN of Donald Trump](session-5-part-2.ipynb#part-5---pretrained-char-rnn-of-donald-trump)\n",
    "    - [Getting the Trump Data](session-5-part-2.ipynb#getting-the-trump-data)\n",
    "    - [Basic Text Analysis](session-5-part-2.ipynb#basic-text-analysis)\n",
    "    - [Loading the Pre-trained Trump Model](session-5-part-2.ipynb#loading-the-pre-trained-trump-model)\n",
    "    - [Inference: Keeping Track of the State](session-5-part-2.ipynb#inference-keeping-track-of-the-state)\n",
    "    - [Probabilistic Sampling](session-5-part-2.ipynb#probabilistic-sampling)\n",
    "    - [Inference: Temperature](session-5-part-2.ipynb#inference-temperature)\n",
    "    - [Inference: Priming](session-5-part-2.ipynb#inference-priming)\n",
    "- [Assignment Submission](session-5-part-2.ipynb#assignment-submission)\n",
    "<!-- /MarkdownTOC -->\n",
    "\n",
    "\n",
    "<a name=\"overview\"></a>\n",
    "# Overview\n",
    "\n",
    "This is certainly the hardest session and will require a lot of time and patience to complete.  Also, many elements of this session may require further investigation, including reading of the original papers and additional resources in order to fully grasp their understanding.  The models we cover are state of the art and I've aimed to give you something between a practical and mathematical understanding of the material, though it is a tricky balance.  I hope for those interested, that you delve deeper into the papers for more understanding.  And for those of you seeking just a practical understanding, that these notebooks will suffice.\n",
    "\n",
    "This session covered two of the most advanced generative networks: generative adversarial networks and recurrent neural networks.  During the homework, we'll see how these work in more details and try building our own.  I am not asking you train anything in this session as both GANs and RNNs take many days to train.  However, I have provided pre-trained networks which we'll be exploring.   We'll also see how a Variational Autoencoder can be combined with a Generative Adversarial Network to allow you to also encode input data, and I've provided a pre-trained model of this type of model trained on the Celeb Faces dataset.  We'll see what this means in more details below.\n",
    "\n",
    "After this session, you are also required to submit your final project which can combine any of the materials you have learned so far to produce a short 1 minute clip demonstrating any aspect of the course you want to invesitgate further or combine with anything else you feel like doing.  This is completely open to you and to encourage your peers to share something that demonstrates creative thinking.  Be sure to keep the final project in mind while browsing through this notebook!\n",
    "\n",
    "<a name=\"learning-goals\"></a>\n",
    "# Learning Goals\n",
    "\n",
    "* Learn to build the components of a Generative Adversarial Network and how it is trained\n",
    "* Learn to combine the Variational Autoencoder with a Generative Adversarial Network\n",
    "* Learn to use latent space arithmetic with a pre-trained VAE/GAN network\n",
    "* Learn to build the components of a Character Recurrent Neural Network and how it is trained\n",
    "* Learn to sample from a pre-trained CharRNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First check the Python version\n",
    "import sys\n",
    "if sys.version_info < (3,4):\n",
    "    print('You are running an older version of Python!\\n\\n',\n",
    "          'You should consider updating to Python 3.4.0 or',\n",
    "          'higher as the libraries built for this course',\n",
    "          'have only been tested in Python 3.4 and higher.\\n')\n",
    "    print('Try installing the Python 3.5 version of anaconda'\n",
    "          'and then restart `jupyter notebook`:\\n',\n",
    "          'https://www.continuum.io/downloads\\n\\n')\n",
    "\n",
    "# Now get necessary libraries\n",
    "try:\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    from scipy.ndimage.filters import gaussian_filter\n",
    "    import IPython.display as ipyd\n",
    "    import tensorflow as tf\n",
    "    from libs import utils, gif, datasets, dataset_utils, nb_utils\n",
    "except ImportError as e:\n",
    "    print(\"Make sure you have started notebook in the same directory\",\n",
    "          \"as the provided zip file which includes the 'libs' folder\",\n",
    "          \"and the file 'utils.py' inside of it.  You will NOT be able\",\n",
    "          \"to complete this assignment unless you restart jupyter\",\n",
    "          \"notebook inside the directory created by extracting\",\n",
    "          \"the zip file or cloning the github repo.\")\n",
    "    print(e)\n",
    "\n",
    "# We'll tell matplotlib to inline any drawn figures like so:\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Bit of formatting because I don't like the default inline code style:\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-1---generative-adversarial-networks-gan--deep-convolutional-gan-dcgan\"></a>\n",
    "# Part 1 - Generative Adversarial Networks (GAN) / Deep Convolutional GAN (DCGAN)\n",
    "\n",
    "<a name=\"introduction\"></a>\n",
    "## Introduction\n",
    "\n",
    "Recall from the lecture that a Generative Adversarial Network is two networks, a generator and a discriminator.  The \"generator\" takes a feature vector and decodes this feature vector to become an image, exactly like the decoder we built in Session 3's Autoencoder.  The discriminator is exactly like the encoder of the Autoencoder, except it can only have 1 value in the final layer.  We use a sigmoid to squash this value between 0 and 1, and then interpret the meaning of it as: 1, the image you gave me was real, or 0, the image you gave me was generated by the generator, it's a FAKE!  So the discriminator is like an encoder which takes an image and then perfoms lie detection.  Are you feeding me lies?  Or is the image real?  \n",
    "\n",
    "Consider the AE and VAE we trained in Session 3.  The loss function operated partly on the input space.  It said, per pixel, what is the difference between my reconstruction and the input image?  The l2-loss per pixel.  Recall at that time we suggested that this wasn't the best idea because per-pixel differences aren't representative of our own perception of the image.  One way to consider this is if we had the same image, and translated it by a few pixels.  We would not be able to tell the difference, but the per-pixel difference between the two images could be enormously high.\n",
    "\n",
    "The GAN does not use per-pixel difference.  Instead, it trains a distance function: the discriminator.  The discriminator takes in two images, the real image and the generated one, and learns what a similar image should look like!  That is really the amazing part of this network and has opened up some very exciting potential future directions for unsupervised learning.  Another network that also learns a distance function is known as the siamese network.  We didn't get into this network in this course, but it is commonly used in facial verification, or asserting whether two faces are the same or not.\n",
    "\n",
    "The GAN network is notoriously a huge pain to train!  For that reason, we won't actually be training it.  Instead, we'll discuss an extension to this basic network called the VAEGAN which uses the VAE we created in Session 3 along with the GAN.  We'll then train that network in Part 2.  For now, let's stick with creating the GAN.\n",
    "\n",
    "Let's first create the two networks: the discriminator and the generator.  We'll first begin by building a general purpose encoder which we'll use for our discriminator.  Recall that we've already done this in Session 3.  What we want is for the input placeholder to be encoded using a list of dimensions for each of our encoder's layers.  In the case of a convolutional network, our list of dimensions should correspond to the number of output filters.  We also need to specify the kernel heights and widths for each layer's convolutional network.\n",
    "\n",
    "We'll first need a placeholder.  This will be the \"real\" image input to the discriminator and the discrimintator will encode this image into a single value, 0 or 1, saying, yes this is real, or no, this is not real.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll keep a variable for the size of our image.\n",
    "n_pixels = 32\n",
    "n_channels = 3\n",
    "input_shape = [None, n_pixels, n_pixels, n_channels]\n",
    "\n",
    "# And then create the input image placeholder\n",
    "X = tf.placeholder(name='X'..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-encoder\"></a>\n",
    "## Building the Encoder\n",
    "\n",
    "Let's build our encoder just like in Session 3.  We'll create a function which accepts the input placeholder, a list of dimensions describing the number of convolutional filters in each layer, and a list of filter sizes to use for the kernel sizes in each convolutional layer.  We'll also pass in a parameter for which activation function to apply.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encoder(x, channels, filter_sizes, activation=tf.nn.tanh, reuse=None):\n",
    "    # Set the input to a common variable name, h, for hidden layer\n",
    "    h = x\n",
    "\n",
    "    # Now we'll loop over the list of dimensions defining the number\n",
    "    # of output filters in each layer, and collect each hidden layer\n",
    "    hs = []\n",
    "    for layer_i in range(len(channels)):\n",
    "        \n",
    "        with tf.variable_scope('layer{}'.format(layer_i+1), reuse=reuse):\n",
    "            # Convolve using the utility convolution function\n",
    "            # This requirs the number of output filter,\n",
    "            # and the size of the kernel in `k_h` and `k_w`.\n",
    "            # By default, this will use a stride of 2, meaning\n",
    "            # each new layer will be downsampled by 2.\n",
    "            h, W = utils.conv2d(...\n",
    "\n",
    "            # Now apply the activation function\n",
    "            h = activation(h)\n",
    "                                            \n",
    "            # Store each hidden layer\n",
    "            hs.append(h)\n",
    "\n",
    "    # Finally, return the encoding.\n",
    "    return h, hs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-discriminator-for-the-training-samples\"></a>\n",
    "## Building the Discriminator for the Training Samples\n",
    "\n",
    "Finally, let's take the output of our encoder, and make sure it has just 1 value by using a fully connected layer.  We can use the `libs/utils` module's, `linear` layer to do this, which will also reshape our 4-dimensional tensor to a 2-dimensional one prior to using the fully connected layer.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discriminator(X,\n",
    "                  channels=[50, 50, 50, 50],\n",
    "                  filter_sizes=[4, 4, 4, 4],\n",
    "                  activation=utils.lrelu,\n",
    "                  reuse=None):\n",
    "\n",
    "    # We'll scope these variables to \"discriminator_real\"\n",
    "    with tf.variable_scope('discriminator', reuse=reuse):\n",
    "        # Encode X:\n",
    "        H, Hs = encoder(X, channels, filter_sizes, activation, reuse)\n",
    "        \n",
    "        # Now make one last layer with just 1 output.  We'll\n",
    "        # have to reshape to 2-d so that we can create a fully\n",
    "        # connected layer:\n",
    "        shape = H.get_shape().as_list()\n",
    "        H = tf.reshape(H, [-1, shape[1] * shape[2] * shape[3]])\n",
    "        \n",
    "        # Now we can connect our 2D layer to a single neuron output w/\n",
    "        # a sigmoid activation:\n",
    "        D, W = utils.linear(...\n",
    "    return D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the discriminator for the real training data coming from `X`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "D_real = discriminator(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can see what the network looks like now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "graph = tf.get_default_graph()\n",
    "nb_utils.show_graph(graph.as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-decoder\"></a>\n",
    "## Building the Decoder\n",
    "\n",
    "Now we're ready to build the Generator, or decoding network.  This network takes as input a vector of features and will try to produce an image that looks like our training data.  We'll send this synthesized image to our discriminator which we've just built above.\n",
    "\n",
    "Let's start by building the input to this network.  We'll need a placeholder for the input features to this network.  We have to be mindful of how many features we have.  The feature vector for the Generator will eventually need to form an image.  What we can do is create a 1-dimensional vector of values for each element in our batch, giving us `[None, n_features]`.  We can then reshape this to a 4-dimensional Tensor so that we can build a decoder network just like in Session 3.\n",
    "\n",
    "But how do we assign the values from our 1-d feature vector (or 2-d tensor with Batch number of them) to the 3-d shape of an image (or 4-d tensor with Batch number of them)?  We have to go from the number of features in our 1-d feature vector, let's say `n_latent` to `height x width x channels` through a series of convolutional transpose layers.  One way to approach this is think of the reverse process.  Starting from the final decoding of `height x width x channels`, I will use convolution with a stride of 2, so downsample by 2 with each new layer.  So the second to last decoder layer would be, `height // 2 x width // 2 x ?`.  If I look at it like this, I can use the variable `n_pixels` denoting the `height` and `width` to build my decoder, and set the channels to whatever I want.\n",
    "\n",
    "Let's start with just our 2-d placeholder which will have `None x n_features`, then convert it to a 4-d tensor ready for the decoder part of the network (a.k.a. the generator)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll need some variables first. This will be how many\n",
    "# channels our generator's feature vector has.  Experiment w/\n",
    "# this if you are training your own network.\n",
    "n_code = 16\n",
    "\n",
    "# And in total how many feature it has, including the spatial dimensions.\n",
    "n_latent = (n_pixels // 16) * (n_pixels // 16) * n_code\n",
    "\n",
    "# Let's build the 2-D placeholder, which is the 1-d feature vector for every\n",
    "# element in our batch.  We'll then reshape this to 4-D for the decoder.\n",
    "Z = tf.placeholder(name='Z', shape=[None, n_latent], dtype=tf.float32)\n",
    "\n",
    "# Now we can reshape it to input to the decoder.  Here we have to\n",
    "# be mindful of the height and width as described before.  We need\n",
    "# to make the height and width a factor of the final height and width\n",
    "# that we want.  Since we are using strided convolutions of 2, then\n",
    "# we can say with 4 layers, that first decoder's layer should be:\n",
    "# n_pixels / 2 / 2 / 2 / 2, or n_pixels / 16:\n",
    "Z_tensor = tf.reshape(Z, [-1, n_pixels // 16, n_pixels // 16, n_code])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll build the decoder in much the same way as we built our encoder.  And exactly as we've done in Session 3!  This requires one additional parameter \"channels\" which is how many output filters we want for each net layer.  We'll interpret the `dimensions` as the height and width of the tensor in each new layer, the `channels` is how many output filters we want for each net layer, and the `filter_sizes` is the size of the filters used for convolution.  We'll default to using a stride of two which will downsample each layer.  We're also going to collect each hidden layer `h` in a list.  We'll end up needing this for Part 2 when we combine the variational autoencoder w/ the generative adversarial network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decoder(z, dimensions, channels, filter_sizes,\n",
    "            activation=tf.nn.relu, reuse=None):\n",
    "    h = z\n",
    "    hs = []\n",
    "    for layer_i in range(len(dimensions)):\n",
    "        with tf.variable_scope('layer{}'.format(layer_i+1), reuse=reuse):\n",
    "            h, W = utils.deconv2d(x=h,\n",
    "                               n_output_h=dimensions[layer_i],\n",
    "                               n_output_w=dimensions[layer_i],\n",
    "                               n_output_ch=channels[layer_i],\n",
    "                               k_h=filter_sizes[layer_i],\n",
    "                               k_w=filter_sizes[layer_i],\n",
    "                               reuse=reuse)\n",
    "            h = activation(h)\n",
    "            hs.append(h)\n",
    "    return h, hs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-generator\"></a>\n",
    "## Building the Generator\n",
    "\n",
    "Now we're ready to use our decoder to take in a vector of features and generate something that looks like our training images. We have to ensure that the last layer produces the same output shape as the discriminator's input.  E.g. we used a `[None, 64, 64, 3]` input to the discriminator, so our generator needs to also output `[None, 64, 64, 3]` tensors.  In other words, we have to ensure the last element in our `dimensions` list is 64, and the last element in our `channels` list is 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Explore these parameters.\n",
    "def generator(Z,\n",
    "              dimensions=[n_pixels//8, n_pixels//4, n_pixels//2, n_pixels],\n",
    "              channels=[50, 50, 50, n_channels],\n",
    "              filter_sizes=[4, 4, 4, 4],\n",
    "              activation=utils.lrelu):\n",
    "\n",
    "    with tf.variable_scope('generator'):\n",
    "        G, Hs = decoder(Z_tensor, dimensions, channels, filter_sizes, activation)\n",
    "\n",
    "    return G"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's call the `generator` function with our input placeholder `Z`.  This will take our feature vector and generate something in the shape of an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "G = generator(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "graph = tf.get_default_graph()\n",
    "nb_utils.show_graph(graph.as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-discriminator-for-the-generated-samples\"></a>\n",
    "## Building the Discriminator for the Generated Samples\n",
    "\n",
    "Lastly, we need *another* discriminator which takes as input our generated images.  Recall the discriminator that we have made only takes as input our placeholder `X` which is for our actual training samples.  We'll use the same function for creating our discriminator and **reuse** the variables we already have.  This is the crucial part!  We aren't making *new* trainable variables, but reusing the ones we have.  We just create a new set of operations that takes as input our generated image.  So we'll have a whole new set of operations exactly like the ones we have created for our first discriminator.  But we are going to use the exact same variables as our first discriminator, so that we optimize the same values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "D_fake = discriminator(G, reuse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can look at the graph and see the new discriminator inside the node for the discriminator.  You should see the original discriminator and a new graph of a discriminator within it, but all the weights are shared with the original discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_utils.show_graph(graph.as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"gan-loss-functions\"></a>\n",
    "## GAN Loss Functions\n",
    "\n",
    "We now have all the components to our network.  We just have to train it.  This is the notoriously tricky bit.  We will have 3 different loss measures instead of our typical network with just a single loss.  We'll later connect each of these loss measures to two optimizers, one for the generator and another for the discriminator, and then pin them against each other and see which one wins!  Exciting times!\n",
    "\n",
    "Recall from Session 3's Supervised Network, we created a binary classification task: music or speech.  We again have a binary classification task: real or fake.  So our loss metric will again use the binary cross entropy to measure the loss of our three different modules: the generator, the discriminator for our real images, and the discriminator for our generated images.\n",
    "\n",
    "To find out the loss function for our generator network, answer the question, what makes the generator successful?  Successfully fooling the discriminator.  When does that happen?  When the discriminator for the fake samples produces all ones.  So our binary cross entropy measure will measure the cross entropy with our predicted distribution and the true distribution which has all ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('loss/generator'):\n",
    "    loss_G = tf.reduce_mean(utils.binary_cross_entropy(D_fake, tf.ones_like(D_fake)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we've just written is a loss function for our generator.  The generator is optimized when the discriminator for the generated samples produces all ones.  In contrast to the generator, the discriminator will have 2 measures to optimize.  One which is the opposite of what we have just written above, as well as 1 more measure for the real samples.  Try writing these two losses and we'll combine them using their average.  We want to optimize the Discriminator for the real samples producing all 1s, and the Discriminator for the fake samples producing all 0s:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('loss/discriminator/real'):\n",
    "    loss_D_real = utils.binary_cross_entropy(D_real, ...\n",
    "with tf.variable_scope('loss/discriminator/fake'):\n",
    "    loss_D_fake = utils.binary_cross_entropy(D_fake, ...\n",
    "with tf.variable_scope('loss/discriminator'):\n",
    "    loss_D = tf.reduce_mean((loss_D_real + loss_D_fake) / 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_utils.show_graph(graph.as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our loss functions, we can create an optimizer for the discriminator and generator:\n",
    "\n",
    "<a name=\"building-the-optimizers-w-regularization\"></a>\n",
    "## Building the Optimizers w/ Regularization\n",
    "\n",
    "We're almost ready to create our optimizers.  We just need to do one extra thing.  Recall that our loss for our generator has a flow from the generator through the discriminator.  If we are training both the generator and the discriminator, we have two measures which both try to optimize the discriminator, but in opposite ways: the generator's loss would try to optimize the discriminator to be bad at its job, and the discriminator's loss would try to optimize it to be good at its job.  This would be counter-productive, trying to optimize opposing losses.  What we want is for the generator to get better, and the discriminator to get better.  Not for the discriminator to get better, then get worse, then get better, etc... The way we do this is when we optimize our generator, we let the gradient flow through the discriminator, but we do not update the variables in the discriminator.  Let's try and grab just the discriminator variables and just the generator variables below:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Grab just the variables corresponding to the discriminator\n",
    "# and just the generator:\n",
    "vars_d = [v for v in tf.trainable_variables()\n",
    "          if ...]\n",
    "print('Training discriminator variables:')\n",
    "[print(v.name) for v in tf.trainable_variables()\n",
    " if v.name.startswith('discriminator')]\n",
    "\n",
    "vars_g = [v for v in tf.trainable_variables()\n",
    "          if ...]\n",
    "print('Training generator variables:')\n",
    "[print(v.name) for v in tf.trainable_variables()\n",
    " if v.name.startswith('generator')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply regularization to our network.  This will penalize weights in the network for growing too large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_reg = tf.contrib.layers.apply_regularization(\n",
    "    tf.contrib.layers.l2_regularizer(1e-6), vars_d)\n",
    "g_reg = tf.contrib.layers.apply_regularization(\n",
    "    tf.contrib.layers.l2_regularizer(1e-6), vars_g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing you may want to try is creating a separate learning rate for each of your generator and discriminator optimizers like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.0001\n",
    "\n",
    "lr_g = tf.placeholder(tf.float32, shape=[], name='learning_rate_g')\n",
    "lr_d = tf.placeholder(tf.float32, shape=[], name='learning_rate_d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can feed the placeholders to your optimizers.  If you run into errors creating these, then you likely have a problem with your graph's definition!  Be sure to go back and reset the default graph and check the sizes of your different operations/placeholders.\n",
    "\n",
    "With your optimizers, you can now train the network by \"running\" the optimizer variables with your session.  You'll need to set the `var_list` parameter of the `minimize` function to only train the variables for the discriminator and same for the generator's optimizer:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt_g = tf.train.AdamOptimizer(learning_rate=lr_g).minimize(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt_d = tf.train.AdamOptimizer(learning_rate=lr_d).minimize(loss_D + d_reg, var_list=vars_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"loading-a-dataset\"></a>\n",
    "## Loading a Dataset\n",
    "\n",
    "Let's use the Celeb Dataset just for demonstration purposes.  In Part 2, you can explore using your own dataset.  This code is exactly the same as we did in Session 3's homework with the VAE.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You'll want to change this to your own data if you end up training your own GAN.\n",
    "batch_size = 64\n",
    "n_epochs = 1\n",
    "crop_shape = [n_pixels, n_pixels, 3]\n",
    "crop_factor = 0.8\n",
    "input_shape = [218, 178, 3]\n",
    "\n",
    "files = datasets.CELEB()\n",
    "batch = dataset_utils.create_input_pipeline(\n",
    "        files=files,\n",
    "        batch_size=batch_size,\n",
    "        n_epochs=n_epochs,\n",
    "        crop_shape=crop_shape,\n",
    "        crop_factor=crop_factor,\n",
    "        shape=input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"training\"></a>\n",
    "## Training\n",
    "\n",
    "We'll now go through the setup of training the network. We won't actually spend the time to train the network but just see how it would be done.  This is because in Part 2, we'll see an extension to this network which makes it much easier to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ckpt_name = './gan.ckpt'\n",
    "\n",
    "sess = tf.Session()\n",
    "saver = tf.train.Saver()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "coord = tf.train.Coordinator()\n",
    "tf.get_default_graph().finalize()\n",
    "threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n",
    "\n",
    "if os.path.exists(ckpt_name + '.index') or os.path.exists(ckpt_name):\n",
    "    saver.restore(sess, ckpt_name)\n",
    "    print(\"VAE model restored.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_examples = 10\n",
    "\n",
    "zs = np.random.uniform(0.0, 1.0, [4, n_latent]).astype(np.float32)\n",
    "zs = utils.make_latent_manifold(zs, n_examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"equilibrium\"></a>\n",
    "## Equilibrium\n",
    "\n",
    " Equilibrium is at 0.693.  Why?  Consider what the cost is measuring, the binary cross entropy.  If we have random guesses, then we have as many 0s as we have 1s.  And on average, we'll be 50% correct.  The binary cross entropy is:\n",
    "\n",
    "\\begin{align}\n",
    "\\sum_i \\text{X}_i * \\text{log}(\\tilde{\\text{X}}_i) + (1 - \\text{X}_i) * \\text{log}(1 - \\tilde{\\text{X}}_i)\n",
    "\\end{align}\n",
    "\n",
    "Which is written out in tensorflow as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "(-(x * tf.log(z) + (1. - x) * tf.log(1. - z)))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where `x` is the discriminator's prediction of the true distribution, in the case of GANs, the input images, and `z` is the discriminator's prediction of the generated images corresponding to the mathematical notation of $\\tilde{\\text{X}}$.  We sum over all features, but in the case of the discriminator, we have just 1 feature, the guess of whether it is a true image or not.  If our discriminator guesses at chance, i.e. 0.5,  then we'd have something like:\n",
    "\n",
    "\\begin{align}\n",
    "0.5 * \\text{log}(0.5) + (1 - 0.5) * \\text{log}(1 - 0.5) = -0.693\n",
    "\\end{align}\n",
    "\n",
    "So this is what we'd expect at the start of learning and from a game theoretic point of view, where we want things to remain.  So unlike our previous networks, where our loss continues to drop closer and closer to 0, we want our loss to waver around this value as much as possible, and hope for the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "equilibrium = 0.693\n",
    "margin = 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we go to train the network, we switch back and forth between each optimizer, feeding in the appropriate values for each optimizer.  The `opt_g` optimizer only requires the `Z` and `lr_g` placeholders, while the `opt_d` optimizer requires the `X`, `Z`, and `lr_d` placeholders.\n",
    "\n",
    "Don't train this network for very long because GANs are a huge pain to train and require a lot of fiddling.  They very easily get stuck in their adversarial process, or get overtaken by one or the other, resulting in a useless model.  What you need to develop is a steady equilibrium that optimizes both.  That will likely take two weeks just trying to get the GAN to train and not have enough time for the rest of the assignment.  They require a lot of memory/cpu and can take many days to train once you have settled on an architecture/training process/dataset.  Just let it run for a short time and then interrupt the kernel (don't restart!), then continue to the next cell.\n",
    "\n",
    "From there, we'll go over an extension to the GAN which uses a VAE like we used in Session 3.  By using this extra network, we can actually train a better model in a fraction of the time and with much more ease!  But the network's definition is a bit more complicated.  Let's see how the GAN is trained first and then we'll train the VAE/GAN network instead.  While training, the \"real\" and \"fake\" cost will be printed out.  See how this cost wavers around the equilibrium and how we enforce it to try and stay around there by including a margin and some simple logic for updates.  This is highly experimental and the research does not have a good answer for the best practice on how to train a GAN.  I.e., some people will set the learning rate to some ratio of the performance between fake/real networks, others will have a fixed update schedule but train the generator twice and the discriminator only once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t_i = 0\n",
    "batch_i = 0\n",
    "epoch_i = 0\n",
    "n_files = len(files)\n",
    "\n",
    "if not os.path.exists('imgs'):\n",
    "    os.makedirs('imgs')\n",
    "    \n",
    "while epoch_i < n_epochs:\n",
    "\n",
    "    batch_i += 1\n",
    "    batch_xs = sess.run(batch) / 255.0\n",
    "    batch_zs = np.random.uniform(\n",
    "        0.0, 1.0, [batch_size, n_latent]).astype(np.float32)\n",
    "\n",
    "    real_cost, fake_cost = sess.run([\n",
    "        loss_D_real, loss_D_fake],\n",
    "        feed_dict={\n",
    "            X: batch_xs,\n",
    "            Z: batch_zs})\n",
    "    real_cost = np.mean(real_cost)\n",
    "    fake_cost = np.mean(fake_cost)\n",
    "    \n",
    "    if (batch_i % 20) == 0:\n",
    "        print(batch_i, 'real:', real_cost, '/ fake:', fake_cost)\n",
    "\n",
    "    gen_update = True\n",
    "    dis_update = True\n",
    "\n",
    "    if real_cost > (equilibrium + margin) or \\\n",
    "       fake_cost > (equilibrium + margin):\n",
    "        gen_update = False\n",
    "\n",
    "    if real_cost < (equilibrium - margin) or \\\n",
    "       fake_cost < (equilibrium - margin):\n",
    "        dis_update = False\n",
    "\n",
    "    if not (gen_update or dis_update):\n",
    "        gen_update = True\n",
    "        dis_update = True\n",
    "\n",
    "    if gen_update:\n",
    "        sess.run(opt_g,\n",
    "            feed_dict={\n",
    "                Z: batch_zs,\n",
    "                lr_g: learning_rate})\n",
    "    if dis_update:\n",
    "        sess.run(opt_d,\n",
    "            feed_dict={\n",
    "                X: batch_xs,\n",
    "                Z: batch_zs,\n",
    "                lr_d: learning_rate})\n",
    "\n",
    "    if batch_i % (n_files // batch_size) == 0:\n",
    "        batch_i = 0\n",
    "        epoch_i += 1\n",
    "        print('---------- EPOCH:', epoch_i)\n",
    "        \n",
    "        # Plot example reconstructions from latent layer\n",
    "        recon = sess.run(G, feed_dict={Z: zs})\n",
    "\n",
    "        recon = np.clip(recon, 0, 1)\n",
    "        m1 = utils.montage(recon.reshape([-1] + crop_shape),\n",
    "                'imgs/manifold_%08d.png' % t_i)\n",
    "\n",
    "        recon = sess.run(G, feed_dict={Z: batch_zs})\n",
    "\n",
    "        recon = np.clip(recon, 0, 1)\n",
    "        m2 = utils.montage(recon.reshape([-1] + crop_shape),\n",
    "                'imgs/reconstructions_%08d.png' % t_i)\n",
    "        \n",
    "        fig, axs = plt.subplots(1, 2, figsize=(15, 10))\n",
    "        axs[0].imshow(m1)\n",
    "        axs[1].imshow(m2)\n",
    "        plt.show()\n",
    "        t_i += 1\n",
    "\n",
    "        # Save the variables to disk.\n",
    "        save_path = saver.save(sess, \"./\" + ckpt_name,\n",
    "                               global_step=batch_i,\n",
    "                               write_meta_graph=False)\n",
    "        print(\"Model saved in file: %s\" % save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Tell all the threads to shutdown.\n",
    "coord.request_stop()\n",
    "\n",
    "# Wait until all threads have finished.\n",
    "coord.join(threads)\n",
    "\n",
    "# Clean up the session.\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-2---variational-auto-encoding-generative-adversarial-network-vaegan\"></a>\n",
    "# Part 2 - Variational Auto-Encoding Generative Adversarial Network (VAEGAN)\n",
    "\n",
    "In our definition of the generator, we started with a feature vector, `Z`.  This feature vector was not connected to anything before it.  Instead, we had to randomly create its values using a random number generator of its `n_latent` values from -1 to 1, and this range was chosen arbitrarily.  It could have been 0 to 1, or -3 to 3, or 0 to 100.  In any case, the network would have had to learn to transform those values into something that looked like an image.  There was no way for us to take an image, and find the feature vector that created it.  In other words, it was not possible for us to *encode* an image.\n",
    "\n",
    "The closest thing to an encoding we had was taking an image and feeding it to the discriminator, which would output a 0 or 1.  But what if we had another network that allowed us to encode an image, and then we used this network for both the discriminator and generative parts of the network?  That's the basic idea behind the VAEGAN: https://arxiv.org/abs/1512.09300.  It is just like the regular GAN, except we also use an encoder to create our feature vector `Z`.\n",
    "\n",
    "We then get the best of both worlds: a GAN that looks more or less the same, but uses the encoding from an encoder instead of an arbitrary feature vector; and an autoencoder that can model an input distribution using a trained distance function, the discriminator, leading to nicer encodings/decodings.\n",
    "\n",
    "Let's try to build it!  Refer to the paper for the intricacies and a great read.  Luckily, by building the `encoder` and `decoder` functions, we're almost there.  We just need a few more components and will change these slightly.\n",
    "\n",
    "Let's reset our graph and recompose our network as a VAEGAN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"batch-normalization\"></a>\n",
    "## Batch Normalization\n",
    "\n",
    "You may have noticed from the `VAE` code that I've used something called \"batch normalization\".  This is a pretty effective technique for regularizing the training of networks by \"reducing internal covariate shift\".  The basic idea is that given a minibatch, we optimize the gradient for this small sample of the greater population.  But this small sample may have different characteristics than the entire population's gradient.  Consider the most extreme case, a minibatch of 1.  In this case, we overfit our gradient to optimize the gradient of the single observation.  If our minibatch is too large, say the size of the entire population, we aren't able to manuvuer the loss manifold at all and the entire loss is averaged in a way that doesn't let us optimize anything.  What we want to do is find a happy medium between a too-smooth loss surface (i.e. every observation), and a very peaky loss surface (i.e. a single observation).  Up until now we only used mini-batches to help with this.  But we can also approach it by \"smoothing\" our updates between each mini-batch.  That would effectively smooth the manifold of the loss space.  Those of you familiar with signal processing will see this as a sort of low-pass filter on the gradient updates.\n",
    "\n",
    "In order for us to use batch normalization, we need another placeholder which is a simple boolean: True or False, denoting when we are training.  We'll use this placeholder to conditionally update batch normalization's statistics required for normalizing our minibatches.  Let's create the placeholder and then I'll get into how to use this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# placeholder for batch normalization\n",
    "is_training = tf.placeholder(tf.bool, name='istraining')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original paper that introduced the idea suggests to use batch normalization \"pre-activation\", meaning after the weight multipllication or convolution, and before the nonlinearity.  We can use the `tensorflow.contrib.layers.batch_norm` module to apply batch normalization to any input tensor give the tensor and the placeholder defining whether or not we are training.  Let's use this module and you can inspect the code inside the module in your own time if it interests you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function batch_norm in module tensorflow.contrib.layers.python.layers.layers:\n",
      "\n",
      "batch_norm(inputs, decay=0.999, center=True, scale=False, epsilon=0.001, activation_fn=None, param_initializers=None, updates_collections='update_ops', is_training=True, reuse=None, variables_collections=None, outputs_collections=None, trainable=True, batch_weights=None, fused=False, data_format='NHWC', scope=None)\n",
      "    Adds a Batch Normalization layer from http://arxiv.org/abs/1502.03167.\n",
      "    \n",
      "      \"Batch Normalization: Accelerating Deep Network Training by Reducing\n",
      "      Internal Covariate Shift\"\n",
      "    \n",
      "      Sergey Ioffe, Christian Szegedy\n",
      "    \n",
      "    Can be used as a normalizer function for conv2d and fully_connected.\n",
      "    \n",
      "    Note: When is_training is True the moving_mean and moving_variance need to be\n",
      "    updated, by default the update_ops are placed in `tf.GraphKeys.UPDATE_OPS` so\n",
      "    they need to be added as a dependency to the `train_op`, example:\n",
      "    \n",
      "      update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
      "      if update_ops:\n",
      "        updates = tf.group(*update_ops)\n",
      "        total_loss = control_flow_ops.with_dependencies([updates], total_loss)\n",
      "    \n",
      "    One can set updates_collections=None to force the updates in place, but that\n",
      "    can have speed penalty, specially in distributed settings.\n",
      "    \n",
      "    Args:\n",
      "      inputs: a tensor with 2 or more dimensions, where the first dimension has\n",
      "        `batch_size`. The normalization is over all but the last dimension if\n",
      "        `data_format` is `NHWC` and the second dimension if `data_format` is\n",
      "        `NCHW`.\n",
      "      decay: decay for the moving average. Reasonable values for `decay` are close \n",
      "        to 1.0, typically in the multiple-nines range: 0.999, 0.99, 0.9, etc. Lower \n",
      "        `decay` value (recommend trying `decay`=0.9) if model experiences reasonably \n",
      "        good training performance but poor validation and/or test performance.\n",
      "      center: If True, subtract `beta`. If False, `beta` is ignored.\n",
      "      scale: If True, multiply by `gamma`. If False, `gamma` is\n",
      "        not used. When the next layer is linear (also e.g. `nn.relu`), this can be\n",
      "        disabled since the scaling can be done by the next layer.\n",
      "      epsilon: small float added to variance to avoid dividing by zero.\n",
      "      activation_fn: activation function, default set to None to skip it and\n",
      "        maintain a linear activation.\n",
      "      param_initializers: optional initializers for beta, gamma, moving mean and\n",
      "        moving variance.\n",
      "      updates_collections: collections to collect the update ops for computation.\n",
      "        The updates_ops need to be executed with the train_op.\n",
      "        If None, a control dependency would be added to make sure the updates are\n",
      "        computed in place.\n",
      "      is_training: whether or not the layer is in training mode. In training mode\n",
      "        it would accumulate the statistics of the moments into `moving_mean` and\n",
      "        `moving_variance` using an exponential moving average with the given\n",
      "        `decay`. When it is not in training mode then it would use the values of\n",
      "        the `moving_mean` and the `moving_variance`.\n",
      "      reuse: whether or not the layer and its variables should be reused. To be\n",
      "        able to reuse the layer scope must be given.\n",
      "      variables_collections: optional collections for the variables.\n",
      "      outputs_collections: collections to add the outputs.\n",
      "      trainable: If `True` also add variables to the graph collection\n",
      "        `GraphKeys.TRAINABLE_VARIABLES` (see `tf.Variable`).\n",
      "      batch_weights: An optional tensor of shape `[batch_size]`,\n",
      "        containing a frequency weight for each batch item. If present,\n",
      "        then the batch normalization uses weighted mean and\n",
      "        variance. (This can be used to correct for bias in training\n",
      "        example selection.)\n",
      "      fused:  Use nn.fused_batch_norm if True, nn.batch_normalization otherwise.\n",
      "      data_format: A string. `NHWC` (default) and `NCHW` are supported.\n",
      "      scope: Optional scope for `variable_scope`.\n",
      "    \n",
      "    Returns:\n",
      "      A `Tensor` representing the output of the operation.\n",
      "    \n",
      "    Raises:\n",
      "      ValueError: if `batch_weights` is not None and `fused` is True.\n",
      "      ValueError: if `data_format` is neither `NHWC` nor `NCHW`.\n",
      "      ValueError: if the rank of `inputs` is undefined.\n",
      "      ValueError: if rank or channels dimension of `inputs` is undefined.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.contrib.layers import batch_norm\n",
    "help(batch_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-encoder-1\"></a>\n",
    "## Building the Encoder\n",
    "\n",
    "We can now change our encoder to accept the `is_training` placeholder and apply `batch_norm` just before the activation function is applied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encoder(x, is_training, channels, filter_sizes, activation=tf.nn.tanh, reuse=None):\n",
    "    # Set the input to a common variable name, h, for hidden layer\n",
    "    h = x\n",
    "\n",
    "    print('encoder/input:', h.get_shape().as_list())\n",
    "    # Now we'll loop over the list of dimensions defining the number\n",
    "    # of output filters in each layer, and collect each hidden layer\n",
    "    hs = []\n",
    "    for layer_i in range(len(channels)):\n",
    "        \n",
    "        with tf.variable_scope('layer{}'.format(layer_i+1), reuse=reuse):\n",
    "            # Convolve using the utility convolution function\n",
    "            # This requirs the number of output filter,\n",
    "            # and the size of the kernel in `k_h` and `k_w`.\n",
    "            # By default, this will use a stride of 2, meaning\n",
    "            # each new layer will be downsampled by 2.\n",
    "            h, W = utils.conv2d(h, channels[layer_i],\n",
    "                                k_h=filter_sizes[layer_i],\n",
    "                                k_w=filter_sizes[layer_i],\n",
    "                                d_h=2,\n",
    "                                d_w=2,\n",
    "                                reuse=reuse)\n",
    "            \n",
    "            h = batch_norm(h, is_training=is_training)\n",
    "\n",
    "            # Now apply the activation function\n",
    "            h = activation(h)\n",
    "            print('layer:', layer_i, ', shape:', h.get_shape().as_list())\n",
    "            \n",
    "            # Store each hidden layer\n",
    "            hs.append(h)\n",
    "\n",
    "    # Finally, return the encoding.\n",
    "    return h, hs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create the input to the network using a placeholder.  We can try a slightly larger image this time.  But be careful experimenting with much larger images as this is a big network.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_pixels = 64\n",
    "n_channels = 3\n",
    "input_shape = [None, n_pixels, n_pixels, n_channels]\n",
    "\n",
    "# placeholder for the input to the network\n",
    "X = tf.placeholder(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we'll connect the input to an encoder network.  We'll also use the `tf.nn.elu` activation instead.  Explore other activations but I've found this to make the training much faster (e.g. 10x faster at least!).  See the paper for more details: [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)\n",
    "](http://arxiv.org/abs/1511.07289)\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "channels = [64, 64, 64]\n",
    "filter_sizes = [5, 5, 5]\n",
    "activation = tf.nn.elu\n",
    "n_hidden = 128\n",
    "\n",
    "with tf.variable_scope('encoder'):\n",
    "    H, Hs = encoder(...\n",
    "    Z = utils.linear(H, n_hidden)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-variational-layer\"></a>\n",
    "## Building the Variational Layer\n",
    "\n",
    "In Session 3, we introduced the idea of Variational Bayes when we used the Variational Auto Encoder.  The variational bayesian approach requires a richer understanding of probabilistic graphical models and bayesian methods which we we're not able to go over in this course (it requires a few courses all by itself!).  For that reason, please treat this as a \"black box\" in this course.\n",
    "\n",
    "For those of you that are more familiar with graphical models, Variational Bayesian methods attempt to model an approximate joint distribution of $Q(Z)$ using some distance function to the true distribution $P(X)$.  Kingma and Welling show how this approach can be used in a graphical model resembling an autoencoder and can be trained using KL-Divergence, or $KL(Q(Z) || P(X))$.  The distribution Q(Z) is the variational distribution, and attempts to model the lower-bound of the true distribution $P(X)$ through the minimization of the KL-divergence.  Another way to look at this is the encoder of the network is trying to model the parameters of a known distribution, the Gaussian Distribution, through a minimization of this lower bound.  We assume that this distribution resembles the true distribution, but it is merely a simplification of the true distribution.  To learn more about this, I highly recommend picking up the book by Christopher Bishop called \"Pattern Recognition and Machine Learning\" and reading the original Kingma and Welling paper on Variational Bayes.\n",
    "\n",
    "Now back to coding, we'll create a general variational layer that does exactly the same thing as our VAE in session 3.  Treat this as a black box if you are unfamiliar with the math.  It takes an input encoding, `h`, and an integer, `n_code` defining how many latent Gaussians to use to model the latent distribution.  In return, we get the latent encoding from sampling the Gaussian layer, `z`, the mean and log standard deviation, as well as the prior loss, `loss_z`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def variational_bayes(h, n_code):\n",
    "    # Model mu and log(\\sigma)\n",
    "    z_mu = tf.nn.tanh(utils.linear(h, n_code, name='mu')[0])\n",
    "    z_log_sigma = 0.5 * tf.nn.tanh(utils.linear(h, n_code, name='log_sigma')[0])\n",
    "\n",
    "    # Sample from noise distribution p(eps) ~ N(0, 1)\n",
    "    epsilon = tf.random_normal(tf.stack([tf.shape(h)[0], n_code]))\n",
    "\n",
    "    # Sample from posterior\n",
    "    z = z_mu + tf.multiply(epsilon, tf.exp(z_log_sigma))\n",
    "\n",
    "    # Measure loss\n",
    "    loss_z = -0.5 * tf.reduce_sum(\n",
    "        1.0 + 2.0 * z_log_sigma - tf.square(z_mu) - tf.exp(2.0 * z_log_sigma),\n",
    "        1)\n",
    "\n",
    "    return z, z_mu, z_log_sigma, loss_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's connect this layer to our encoding, and keep all the variables it returns.  Treat this as a black box if you are unfamiliar with variational bayes!\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Experiment w/ values between 2 - 100\n",
    "# depending on how difficult the dataset is\n",
    "n_code = 32\n",
    "\n",
    "with tf.variable_scope('encoder/variational'):\n",
    "    Z, Z_mu, Z_log_sigma, loss_Z = variational_bayes(h=Z, n_code=n_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-the-decoder-1\"></a>\n",
    "## Building the Decoder\n",
    "\n",
    "In the GAN network, we built a decoder and called it the generator network.  Same idea here.  We can use these terms interchangeably.  Before we connect our latent encoding, `Z` to the decoder, we'll implement batch norm in our decoder just like we did with the encoder.  This is a simple fix: add a second argument for `is_training` and then apply batch normalization just after the `deconv2d` operation and just before the nonlinear `activation`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decoder(z, is_training, dimensions, channels, filter_sizes,\n",
    "            activation=tf.nn.elu, reuse=None):\n",
    "    h = z\n",
    "    for layer_i in range(len(dimensions)):\n",
    "        with tf.variable_scope('layer{}'.format(layer_i+1), reuse=reuse):\n",
    "            h, W = utils.deconv2d(x=h,\n",
    "                               n_output_h=dimensions[layer_i],\n",
    "                               n_output_w=dimensions[layer_i],\n",
    "                               n_output_ch=channels[layer_i],\n",
    "                               k_h=filter_sizes[layer_i],\n",
    "                               k_w=filter_sizes[layer_i],\n",
    "                               reuse=reuse)\n",
    "            h = batch_norm(h, is_training=is_training)\n",
    "            h = activation(h)\n",
    "    return h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll build a decoder just like in Session 3, and just like our Generator network in Part 1.  In Part 1, we created `Z` as a placeholder which we would have had to feed in as random values.  However, now we have an explicit coding of an input image in `X` stored in `Z` by having created the encoder network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dimensions = [n_pixels // 8, n_pixels // 4, n_pixels // 2, n_pixels]\n",
    "channels = [30, 30, 30, n_channels]\n",
    "filter_sizes = [4, 4, 4, 4]\n",
    "activation = tf.nn.elu\n",
    "n_latent = n_code * (n_pixels // 16)**2\n",
    "\n",
    "with tf.variable_scope('generator'):\n",
    "    Z_decode = utils.linear(\n",
    "        Z, n_output=n_latent, name='fc', activation=activation)[0]\n",
    "    Z_decode_tensor = tf.reshape(\n",
    "        Z_decode, [-1, n_pixels//16, n_pixels//16, n_code], name='reshape')\n",
    "    G = decoder(\n",
    "        Z_decode_tensor, is_training, dimensions,\n",
    "        channels, filter_sizes, activation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to build our discriminators.  We'll need to add a parameter for the `is_training` placeholder.  We're also going to keep track of every hidden layer in the discriminator.  Our encoder already returns the `Hs` of each layer.  Alternatively, we could poll the graph for each layer in the discriminator and ask for the correspond layer names.  We're going to need these layers when building our costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discriminator(X,\n",
    "                  is_training,\n",
    "                  channels=[50, 50, 50, 50],\n",
    "                  filter_sizes=[4, 4, 4, 4],\n",
    "                  activation=tf.nn.elu,\n",
    "                  reuse=None):\n",
    "\n",
    "    # We'll scope these variables to \"discriminator_real\"\n",
    "    with tf.variable_scope('discriminator', reuse=reuse):\n",
    "        H, Hs = encoder(\n",
    "            X, is_training, channels, filter_sizes, activation, reuse)\n",
    "        shape = H.get_shape().as_list()\n",
    "        H = tf.reshape(\n",
    "            H, [-1, shape[1] * shape[2] * shape[3]])\n",
    "        D, W = utils.linear(\n",
    "            x=H, n_output=1, activation=tf.nn.sigmoid, name='fc', reuse=reuse)\n",
    "    return D, Hs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the regular GAN and DCGAN required 2 discriminators: one for the generated samples in `Z`, and one for the input samples in `X`.  We'll do the same thing here.  One discriminator for the real input data, `X`, which the discriminator will try to predict as 1s, and another discriminator for the generated samples that go from `X` through the encoder to `Z`, and finally through the decoder to `G`.  The discriminator will be trained to try and predict these as 0s, whereas the generator will be trained to try and predict these as 1s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "D_real, Hs_real = discriminator(X, is_training)\n",
    "D_fake, Hs_fake = discriminator(G, is_training, reuse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"building-vaegan-loss-functions\"></a>\n",
    "## Building VAE/GAN Loss Functions\n",
    "\n",
    "Let's now see how we can compose our loss.  We have 3 losses for our discriminator.  Along with measuring the binary cross entropy between each of them, we're going to also measure each layer's loss from our two discriminators using an l2-loss, and this will form our loss for the log likelihood measure.  The details of how these are constructed are explained in more details in the paper: https://arxiv.org/abs/1512.09300 - please refer to this paper for more details that are way beyond the scope of this course!  One parameter within this to pay attention to is `gamma`, which the authors of the paper suggest control the weighting between content and style, just like in Session 4's Style Net implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('loss'):\n",
    "    # Loss functions\n",
    "    loss_D_llike = 0\n",
    "    for h_real, h_fake in zip(Hs_real, Hs_fake):\n",
    "        loss_D_llike += tf.reduce_sum(tf.squared_difference(\n",
    "            utils.flatten(h_fake), utils.flatten(h_real)), 1)\n",
    "\n",
    "    eps = 1e-12\n",
    "    loss_real = tf.log(D_real + eps)\n",
    "    loss_fake = tf.log(1 - D_fake + eps)\n",
    "    loss_GAN = tf.reduce_sum(loss_real + loss_fake, 1)\n",
    "    \n",
    "    gamma = 0.75\n",
    "    loss_enc = tf.reduce_mean(loss_Z + loss_D_llike)\n",
    "    loss_dec = tf.reduce_mean(gamma * loss_D_llike - loss_GAN)\n",
    "    loss_dis = -tf.reduce_mean(loss_GAN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_utils.show_graph(tf.get_default_graph().as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"creating-the-optimizers\"></a>\n",
    "## Creating the Optimizers\n",
    "\n",
    "We now have losses for our encoder, decoder, and discriminator networks.  We can connect each of these to their own optimizer and start training!  Just like with Part 1's GAN, we'll ensure each network's optimizer only trains its part of the network: the encoder's optimizer will only update the encoder variables, the generator's optimizer will only update the generator variables, and the discriminator's optimizer will only update the discriminator variables.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.0001\n",
    "\n",
    "opt_enc = tf.train.AdamOptimizer(\n",
    "    learning_rate=learning_rate).minimize(\n",
    "    loss_enc,\n",
    "    var_list=[var_i for var_i in tf.trainable_variables()\n",
    "              if ...])\n",
    "\n",
    "opt_gen = tf.train.AdamOptimizer(\n",
    "    learning_rate=learning_rate).minimize(\n",
    "    loss_dec,\n",
    "    var_list=[var_i for var_i in tf.trainable_variables()\n",
    "              if ...])\n",
    "\n",
    "opt_dis = tf.train.AdamOptimizer(\n",
    "    learning_rate=learning_rate).minimize(\n",
    "    loss_dis,\n",
    "    var_list=[var_i for var_i in tf.trainable_variables()\n",
    "              if var_i.name.startswith('discriminator')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"loading-the-dataset\"></a>\n",
    "## Loading the Dataset\n",
    "\n",
    "We'll now load our dataset just like in Part 1.  Here is where you should explore with your own data!\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from libs import datasets, dataset_utils\n",
    "\n",
    "batch_size = 64\n",
    "n_epochs = 100\n",
    "crop_shape = [n_pixels, n_pixels, n_channels]\n",
    "crop_factor = 0.8\n",
    "input_shape = [218, 178, 3]\n",
    "\n",
    "# Try w/ CELEB first to make sure it works, then explore w/ your own dataset.\n",
    "files = datasets.CELEB()\n",
    "batch = dataset_utils.create_input_pipeline(\n",
    "    files=files,\n",
    "    batch_size=batch_size,\n",
    "    n_epochs=n_epochs,\n",
    "    crop_shape=crop_shape,\n",
    "    crop_factor=crop_factor,\n",
    "    shape=input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also create a latent manifold just like we've done in Session 3 and Part 1.  This is a random sampling of 4 points in the latent space of `Z`.  We then interpolate between them to create a \"hyper-plane\" and show the decoding of 10 x 10 points on that hyperplane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_samples = 10\n",
    "zs = np.random.uniform(\n",
    "    -1.0, 1.0, [4, n_code]).astype(np.float32)\n",
    "zs = utils.make_latent_manifold(zs, n_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a session and create a coordinator to manage our queues for fetching data from the input pipeline and start our queue runners:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We create a session to use the graph\n",
    "sess = tf.Session()\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "saver = tf.train.Saver()\n",
    "coord = tf.train.Coordinator()\n",
    "threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n",
    "sess.run(init_op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load an existing checkpoint if it exists to continue training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if os.path.exists(\"vaegan.ckpt\"):\n",
    "    saver.restore(sess, \"vaegan.ckpt\")\n",
    "    print(\"GAN model restored.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also try resynthesizing a test set of images.  This will help us understand how well the encoder/decoder network is doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_files = len(files)\n",
    "test_xs = sess.run(batch) / 255.0\n",
    "if not os.path.exists('imgs'):\n",
    "    os.mkdir('imgs')\n",
    "m = utils.montage(test_xs, 'imgs/test_xs.png')\n",
    "plt.imshow(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"training-1\"></a>\n",
    "## Training\n",
    "\n",
    "Almost ready for training.  Let's get some variables which we'll need.  These are the same as Part 1's training process.  We'll keep track of `t_i` which we'll use to create images of the current manifold and reconstruction every so many iterations.  And we'll keep track of the current batch number within the epoch and the current epoch number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t_i = 0\n",
    "batch_i = 0\n",
    "epoch_i = 0\n",
    "ckpt_name = './vaegan.ckpt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like in Part 1, we'll train trying to maintain an equilibrium between our Generator and Discriminator networks.  You should experiment with the margin depending on how the training proceeds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "equilibrium = 0.693\n",
    "margin = 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll train!  Just like Part 1, we measure the `real_cost` and `fake_cost`.  But this time, we'll always update the encoder.  Based on the performance of the real/fake costs, then we'll update generator and discriminator networks.  This will take a long time to produce something nice, but not nearly as long as the regular GAN network despite the additional parameters of the encoder and variational networks.  Be sure to monitor the reconstructions to understand when your network has reached the capacity of its learning!  For reference, on Celeb Net, I would use about 5 layers in each of the Encoder, Generator, and Discriminator networks using as input a 100 x 100 image, and a minimum of 200 channels per layer.  This network would take about 1-2 days to train on an Nvidia TITAN X GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "while epoch_i < n_epochs:\n",
    "    if batch_i % (n_files // batch_size) == 0:\n",
    "        batch_i = 0\n",
    "        epoch_i += 1\n",
    "        print('---------- EPOCH:', epoch_i)\n",
    "\n",
    "    batch_i += 1\n",
    "    batch_xs = sess.run(batch) / 255.0\n",
    "    real_cost, fake_cost, _ = sess.run([\n",
    "        loss_real, loss_fake, opt_enc],\n",
    "        feed_dict={\n",
    "            X: batch_xs,\n",
    "            is_training: True})\n",
    "    real_cost = -np.mean(real_cost)\n",
    "    fake_cost = -np.mean(fake_cost)\n",
    "\n",
    "    gen_update = True\n",
    "    dis_update = True\n",
    "\n",
    "    if real_cost > (equilibrium + margin) or \\\n",
    "       fake_cost > (equilibrium + margin):\n",
    "        gen_update = False\n",
    "\n",
    "    if real_cost < (equilibrium - margin) or \\\n",
    "       fake_cost < (equilibrium - margin):\n",
    "        dis_update = False\n",
    "\n",
    "    if not (gen_update or dis_update):\n",
    "        gen_update = True\n",
    "        dis_update = True\n",
    "\n",
    "    if gen_update:\n",
    "        sess.run(opt_gen, feed_dict={\n",
    "            X: batch_xs,\n",
    "            is_training: True})\n",
    "    if dis_update:\n",
    "        sess.run(opt_dis, feed_dict={\n",
    "            X: batch_xs,\n",
    "            is_training: True})\n",
    "\n",
    "    if batch_i % 50 == 0:\n",
    "        print('real:', real_cost, '/ fake:', fake_cost)\n",
    "\n",
    "        # Plot example reconstructions from latent layer\n",
    "        recon = sess.run(G, feed_dict={\n",
    "                Z: zs,\n",
    "                is_training: False})\n",
    "\n",
    "        recon = np.clip(recon, 0, 1)\n",
    "        m1 = utils.montage(recon.reshape([-1] + crop_shape),\n",
    "                'imgs/manifold_%08d.png' % t_i)\n",
    "\n",
    "        # Plot example reconstructions\n",
    "        recon = sess.run(G, feed_dict={\n",
    "                X: test_xs,\n",
    "                is_training: False})\n",
    "        recon = np.clip(recon, 0, 1)\n",
    "        m2 = utils.montage(recon.reshape([-1] + crop_shape),\n",
    "                'imgs/reconstruction_%08d.png' % t_i)\n",
    "        \n",
    "        fig, axs = plt.subplots(1, 2, figsize=(15, 10))\n",
    "        axs[0].imshow(m1)\n",
    "        axs[1].imshow(m2)\n",
    "        plt.show()\n",
    "        \n",
    "        t_i += 1\n",
    "        \n",
    "\n",
    "    if batch_i % 200 == 0:\n",
    "        # Save the variables to disk.\n",
    "        save_path = saver.save(sess, \"./\" + ckpt_name,\n",
    "                               global_step=batch_i,\n",
    "                               write_meta_graph=False)\n",
    "        print(\"Model saved in file: %s\" % save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# One of the threads has issued an exception.  So let's tell all the\n",
    "# threads to shutdown.\n",
    "coord.request_stop()\n",
    "\n",
    "# Wait until all threads have finished.\n",
    "coord.join(threads)\n",
    "\n",
    "# Clean up the session.\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-3---latent-space-arithmetic\"></a>\n",
    "# Part 3 - Latent-Space Arithmetic\n",
    "\n",
    "<a name=\"loading-the-pre-trained-model\"></a>\n",
    "## Loading the Pre-Trained Model\n",
    "\n",
    "We're now going to work with a pre-trained VAEGAN model on the Celeb Net dataset.  Let's load this model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from libs import celeb_vaegan as CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = CV.get_celeb_vaegan_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll load the graph_def contained inside this dictionary.  It follows the same idea as the `inception`, `vgg16`, and `i2v` pretrained networks.  It is a dictionary with the key `graph_def` defined, with the graph's pretrained network.  It also includes `labels` and a `preprocess` key.  We'll have to do one additional thing which is to turn off the random sampling from variational layer.  This isn't really necessary but will ensure we get the same results each time we use the network.  We'll use the `input_map` argument to do this.  Don't worry if this doesn't make any sense, as we didn't cover the variational layer in any depth.  Just know that this is removing a random process from the network so that it is completely deterministic.  If we hadn't done this, we'd get slightly different results each time we used the network (which may even be desirable for your purposes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "g = tf.get_default_graph()\n",
    "tf.import_graph_def(net['graph_def'], name='net', input_map={\n",
    "        'encoder/variational/random_normal:0': np.zeros(512, dtype=np.float32)})\n",
    "names = [op.name for op in g.get_operations()]\n",
    "print(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the relevant parts of the network: `X`, the input image to the network, `Z`, the input image's encoding, and `G`, the decoded image.  In many ways, this is just like the Autoencoders we learned about in Session 3, except instead of `Y` being the output, we have `G` from our generator!  And the way we train it is very different: we use an adversarial process between the generator and discriminator, and use the discriminator's own distance measure to help train the network, rather than pixel-to-pixel differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = g.get_tensor_by_name('net/x:0')\n",
    "Z = g.get_tensor_by_name('net/encoder/variational/z:0')\n",
    "G = g.get_tensor_by_name('net/generator/x_tilde:0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get some data to play with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "files = datasets.CELEB()\n",
    "img_i = 50\n",
    "img = plt.imread(files[img_i])\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now preprocess the image, and see what the generated image looks like (i.e. the lossy version of the image through the network's encoding and decoding)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p = CV.preprocess(img)\n",
    "synth = sess.run(G, feed_dict={X: p[np.newaxis]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 2, figsize=(10, 5))\n",
    "axs[0].imshow(p)\n",
    "axs[1].imshow(synth[0] / synth.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we lost a lot of details but it seems to be able to express quite a bit about the image.  Our inner most layer, `Z`, is only 512 values yet our dataset was 200k images of 64 x 64 x 3 pixels (about 2.3 GB of information).  That means we're able to express our nearly 2.3 GB of information with only 512 values!  Having some loss of detail is certainly expected!\n",
    "\n",
    "<a name=\"exploring-the-celeb-net-attributes\"></a>\n",
    "## Exploring the Celeb Net Attributes\n",
    "\n",
    "Let's now try and explore the attributes of our dataset.  We didn't train the network with any supervised labels, but the Celeb Net dataset has 40 attributes for each of its 200k images.  These are already parsed and stored for you in the `net` dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(net['labels'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net['labels']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what attributes exist for one of the celeb images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.imshow(img)\n",
    "[net['labels'][i] for i, attr_i in enumerate(net['attributes'][img_i]) if attr_i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"find-the-latent-encoding-for-an-attribute\"></a>\n",
    "## Find the Latent Encoding for an Attribute\n",
    "\n",
    "The Celeb Dataset includes attributes for each of its 200k+ images.  This allows us to feed into the encoder some images that we know have a *specific* attribute, e.g. \"smiling\".  We store what their encoding is and retain this distribution of encoded values.  We can then look at any other image and see how it is encoded, and slightly change the encoding by adding the encoded of our smiling images to it!  The result should be our image but with more smiling.  That is just insane and we're going to see how to do it.  First lets inspect our latent space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Z.get_shape()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 512 features that we can encode any image with.  Assuming our network is doing an okay job, let's try to find the `Z` of the first 100 images with the 'Bald' attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_label = net['labels'].index('Bald')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get all the bald image indexes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_img_idxs = np.where(net['attributes'][:, bald_label])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_img_idxs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's just load 100 of their images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_imgs = [plt.imread(files[bald_img_i])[..., :3]\n",
    "             for bald_img_i in bald_img_idxs[:100]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if the mean image looks like a good bald person or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.imshow(np.mean(bald_imgs, 0).astype(np.uint8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes that is definitely a bald person.  Now we're going to try to find the encoding of a bald person.  One method is to try and find every other possible image and subtract the \"bald\" person's latent encoding.  Then we could add this encoding back to any new image and hopefully it makes the image look more bald.  Or we can find a bunch of bald people's encodings and then average their encodings together.  This should reduce the noise from having many different attributes, but keep the signal pertaining to the baldness.\n",
    "\n",
    "Let's first preprocess the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_p = np.array([CV.preprocess(bald_img_i) for bald_img_i in bald_imgs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can find the latent encoding of the images by calculating `Z` and feeding `X` with our `bald_p` images:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_zs = sess.run(Z, feed_dict=..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's calculate the mean encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_feature = np.mean(bald_zs, 0, keepdims=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_feature.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try and synthesize from the mean bald feature now and see how it looks:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bald_generated = sess.run(G, feed_dict=..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.imshow(bald_generated[0] / bald_generated.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"latent-feature-arithmetic\"></a>\n",
    "## Latent Feature Arithmetic\n",
    "\n",
    "Let's now try to write a general function for performing everything we've just done so that we can do this with many different features.  We'll then try to combine them and synthesize people with the features we want them to have..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_features_for(label='Bald', has_label=True, n_imgs=50):\n",
    "    label_i = net['labels'].index(label)\n",
    "    label_idxs = np.where(net['attributes'][:, label_i] == has_label)[0]\n",
    "    label_idxs = np.random.permutation(label_idxs)[:n_imgs]\n",
    "    imgs = [plt.imread(files[img_i])[..., :3]\n",
    "            for img_i in label_idxs]\n",
    "    preprocessed = np.array([CV.preprocess(img_i) for img_i in imgs])\n",
    "    zs = sess.run(Z, feed_dict={X: preprocessed})\n",
    "    return np.mean(zs, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try getting some attributes positive and negative features.  Be sure to explore different attributes!  Also try different values of `n_imgs`, e.g. 2, 3, 5, 10, 50, 100.  What happens with different values?\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Explore different attributes\n",
    "z1 = get_features_for('Male', True, n_imgs=10)\n",
    "z2 = get_features_for('Male', False, n_imgs=10)\n",
    "z3 = get_features_for('Smiling', True, n_imgs=10)\n",
    "z4 = get_features_for('Smiling', False, n_imgs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b1 = sess.run(G, feed_dict={Z: z1[np.newaxis]})\n",
    "b2 = sess.run(G, feed_dict={Z: z2[np.newaxis]})\n",
    "b3 = sess.run(G, feed_dict={Z: z3[np.newaxis]})\n",
    "b4 = sess.run(G, feed_dict={Z: z4[np.newaxis]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 4, figsize=(15, 6))\n",
    "axs[0].imshow(b1[0] / b1.max()), axs[0].set_title('Male'), axs[0].grid('off'), axs[0].axis('off')\n",
    "axs[1].imshow(b2[0] / b2.max()), axs[1].set_title('Not Male'), axs[1].grid('off'), axs[1].axis('off')\n",
    "axs[2].imshow(b3[0] / b3.max()), axs[2].set_title('Smiling'), axs[2].grid('off'), axs[2].axis('off')\n",
    "axs[3].imshow(b4[0] / b4.max()), axs[3].set_title('Not Smiling'), axs[3].grid('off'), axs[3].axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's interpolate between the \"Male\" and \"Not Male\" categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "notmale_vector = z2 - z1\n",
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([z1 + notmale_vector*amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same for smiling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "smiling_vector = z3 - z4\n",
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([z4 + smiling_vector*amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i] / g[i].max(), 0, 1))\n",
    "    ax_i.grid('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's also no reason why we have to be within the boundaries of 0-1.  We can extrapolate beyond, in, and around the space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(-1.5, 2.5, n_imgs)\n",
    "zs = np.array([z4 + smiling_vector*amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"extensions\"></a>\n",
    "## Extensions\n",
    "\n",
    "[Tom White](https://twitter.com/dribnet), Lecturer at Victoria University School of Design, also recently demonstrated an alternative way of interpolating using a sinusoidal interpolation.  He's created some of the most impressive generative images out there and luckily for us he has detailed his process in the arxiv preprint: https://arxiv.org/abs/1609.04468 - as well, be sure to check out his twitter bot, https://twitter.com/smilevector - which adds smiles to people :) - Note that the network we're using is only trained on aligned faces that are frontally facing, though this twitter bot is capable of adding smiles to any face.  I suspect that he is running a face detection algorithm such as AAM, CLM, or ASM, cropping the face, aligning it, and then running a similar algorithm to what we've done above.  Or else, perhaps he has trained a new model on faces that are not aligned.  In any case, it is well worth checking out!\n",
    "\n",
    "Let's now try and use sinusoidal interpolation using his implementation in [plat](https://github.com/dribnet/plat/blob/master/plat/interpolate.py#L16-L24) which I've copied below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def slerp(val, low, high):\n",
    "    \"\"\"Spherical interpolation. val has a range of 0 to 1.\"\"\"\n",
    "    if val <= 0:\n",
    "        return low\n",
    "    elif val >= 1:\n",
    "        return high\n",
    "    omega = np.arccos(np.dot(low/np.linalg.norm(low), high/np.linalg.norm(high)))\n",
    "    so = np.sin(omega)\n",
    "    return np.sin((1.0-val)*omega) / so * low + np.sin(val*omega)/so * high"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([slerp(amt_i, z1, z2) for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's certainly worth trying especially if you are looking to explore your own model's latent space in new and interesting ways.\n",
    "\n",
    "Let's try and load an image that we want to play with.  We need an image as similar to the Celeb Dataset as possible.  Unfortunately, we don't have access to the algorithm they used to \"align\" the faces, so we'll need to try and get as close as possible to an aligned face image.  One way you can do this is to load up one of the celeb images and try and align an image to it using e.g. Photoshop or another photo editing software that lets you blend and move the images around.  That's what I did for my own face..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img = plt.imread('parag.png')[..., :3]\n",
    "img = CV.preprocess(img, crop_factor=1.0)[np.newaxis]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the network encodes it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img_ = sess.run(G, feed_dict={X: img})\n",
    "fig, axs = plt.subplots(1, 2, figsize=(10, 5))\n",
    "axs[0].imshow(img[0]), axs[0].grid('off')\n",
    "axs[1].imshow(np.clip(img_[0] / np.max(img_), 0, 1)), axs[1].grid('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how blurry the image is.  Tom White's preprint suggests one way to sharpen the image is to find the \"Blurry\" attribute vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z1 = get_features_for('Blurry', True, n_imgs=25)\n",
    "z2 = get_features_for('Blurry', False, n_imgs=25)\n",
    "unblur_vector = z2 - z1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = sess.run(Z, feed_dict={X: img})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([z[0] + unblur_vector * amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i] / g[i].max(), 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the image also gets brighter and perhaps other features than simply the bluriness of the image changes.  Tom's preprint suggests that this is due to the correlation that blurred images have with other things such as the brightness of the image, possibly due biases in labeling or how photographs are taken.   He suggests that another way to unblur would be to synthetically blur a set of images and find the difference in the encoding between the real and blurred images.  We can try it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy.ndimage import gaussian_filter\n",
    "\n",
    "idxs = np.random.permutation(range(len(files)))\n",
    "imgs = [plt.imread(files[idx_i]) for idx_i in idxs[:100]]\n",
    "blurred = []\n",
    "for img_i in imgs:\n",
    "    img_copy = np.zeros_like(img_i)\n",
    "    for ch_i in range(3):\n",
    "        img_copy[..., ch_i] = gaussian_filter(img_i[..., ch_i], sigma=3.0)\n",
    "    blurred.append(img_copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now let's preprocess the original images and the blurred ones\n",
    "imgs_p = np.array([CV.preprocess(img_i) for img_i in imgs])\n",
    "blur_p = np.array([CV.preprocess(img_i) for img_i in blurred])\n",
    "\n",
    "# And then compute each of their latent features\n",
    "noblur = sess.run(Z, feed_dict={X: imgs_p})\n",
    "blur = sess.run(Z, feed_dict={X: blur_p})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "synthetic_unblur_vector = np.mean(noblur - blur, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([z[0] + synthetic_unblur_vector * amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some reason, it also doesn't like my glasses very much.  Let's try and add them back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z1 = get_features_for('Eyeglasses', True)\n",
    "z2 = get_features_for('Eyeglasses', False)\n",
    "glass_vector = z1 - z2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = sess.run(Z, feed_dict={X: img})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1, n_imgs)\n",
    "zs = np.array([z[0] + glass_vector * amt_i + unblur_vector * amt_i for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, more like sunglasses then.  Let's try adding everything in there now!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1.0, n_imgs)\n",
    "zs = np.array([z[0] + glass_vector * amt_i + unblur_vector * amt_i + amt_i * smiling_vector for amt_i in amt])\n",
    "g = sess.run(G, feed_dict={Z: zs})\n",
    "fig, axs = plt.subplots(1, n_imgs, figsize=(20, 4))\n",
    "for i, ax_i in enumerate(axs):\n",
    "    ax_i.imshow(np.clip(g[i], 0, 1))\n",
    "    ax_i.grid('off')\n",
    "    ax_i.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well it was worth a try anyway.  We can also try with a lot of images and create a gif montage of the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_imgs = 5\n",
    "amt = np.linspace(0, 1.5, n_imgs)\n",
    "z = sess.run(Z, feed_dict={X: imgs_p})\n",
    "imgs = []\n",
    "for amt_i in amt:\n",
    "    zs = z + synthetic_unblur_vector * amt_i + amt_i * smiling_vector\n",
    "    g = sess.run(G, feed_dict={Z: zs})\n",
    "    m = utils.montage(np.clip(g, 0, 1))\n",
    "    imgs.append(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gif.build_gif(imgs, saveto='celeb.gif')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ipyd.Image(url='celeb.gif?i={}'.format(\n",
    "        np.random.rand()), height=1000, width=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exploring multiple feature vectors and applying them to images from the celeb dataset to produce animations of a face, saving it as a GIF.  Recall you can store each image frame in a list and then use the `gif.build_gif` function to create a gif.  Explore your own syntheses and then include a gif of the different images you create as \"celeb.gif\" in the final submission.  Perhaps try finding unexpected synthetic latent attributes in the same way that we created a blur attribute.  You can check the documentation in scipy.ndimage for some other image processing techniques, for instance: http://www.scipy-lectures.org/advanced/image_processing/ - and see if you can find the encoding of another attribute that you then apply to your own images.  You can even try it with many images and use the `utils.montage` function to create a large grid of images that evolves over your attributes.  Or create a set of expressions perhaps.  Up to you just explore!\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "imgs = []\n",
    "\n",
    "... DO SOMETHING AWESOME ! ...\n",
    "\n",
    "gif.build_gif(imgs=imgs, saveto='vaegan.gif')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-4---character-level-recurrent-neural-network\"></a>\n",
    "# Part 4 - Character Level Recurrent Neural Network\n",
    "\n",
    "Please visit [session-5-part2.ipynb](session-5-part2.ipynb) for the rest of the homework!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}