session-5/session-5-part-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 5: Generative Networks\n",
    "## Assignment: Generative Adversarial Networks, Variational Autoencoders, and Recurrent Neural Networks\n",
    "<p class=\"lead\">\n",
    "<a href=\"https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\">Creative Applications of Deep Learning with Google's Tensorflow</a><br />\n",
    "<a href=\"http://pkmital.com\">Parag K. Mital</a><br />\n",
    "<a href=\"https://www.kadenze.com\">Kadenze, Inc.</a>\n",
    "</p>\n",
    "\n",
    "Continued from [session-5-part-1.ipynb](session-5-part-1.ipynb)...\n",
    "\n",
    "# Table of Contents\n",
    "\n",
    "<!-- MarkdownTOC autolink=\"true\" autoanchor=\"true\" bracket=\"round\" -->\n",
    "- [Overview](session-5-part-1.ipynb#overview)\n",
    "- [Learning Goals](session-5-part-1.ipynb#learning-goals)\n",
    "- [Part 1 - Generative Adversarial Networks \\(GAN\\) / Deep Convolutional GAN \\(DCGAN\\)](#part-1---generative-adversarial-networks-gan--deep-convolutional-gan-dcgan)\n",
    "  - [Introduction](session-5-part-1.ipynb#introduction)\n",
    "  - [Building the Encoder](session-5-part-1.ipynb#building-the-encoder)\n",
    "  - [Building the Discriminator for the Training Samples](session-5-part-1.ipynb#building-the-discriminator-for-the-training-samples)\n",
    "  - [Building the Decoder](session-5-part-1.ipynb#building-the-decoder)\n",
    "  - [Building the Generator](session-5-part-1.ipynb#building-the-generator)\n",
    "  - [Building the Discriminator for the Generated Samples](session-5-part-1.ipynb#building-the-discriminator-for-the-generated-samples)\n",
    "  - [GAN Loss Functions](session-5-part-1.ipynb#gan-loss-functions)\n",
    "  - [Building the Optimizers w/ Regularization](session-5-part-1.ipynb#building-the-optimizers-w-regularization)\n",
    "  - [Loading a Dataset](session-5-part-1.ipynb#loading-a-dataset)\n",
    "  - [Training](session-5-part-1.ipynb#training)\n",
    "  - [Equilibrium](session-5-part-1.ipynb#equilibrium)\n",
    "- [Part 2 - Variational Auto-Encoding Generative Adversarial Network \\(VAEGAN\\)](#part-2---variational-auto-encoding-generative-adversarial-network-vaegan)\n",
    "  - [Batch Normalization](session-5-part-1.ipynb#batch-normalization)\n",
    "  - [Building the Encoder](session-5-part-1.ipynb#building-the-encoder-1)\n",
    "  - [Building the Variational Layer](session-5-part-1.ipynb#building-the-variational-layer)\n",
    "  - [Building the Decoder](session-5-part-1.ipynb#building-the-decoder-1)\n",
    "  - [Building VAE/GAN Loss Functions](session-5-part-1.ipynb#building-vaegan-loss-functions)\n",
    "  - [Creating the Optimizers](session-5-part-1.ipynb#creating-the-optimizers)\n",
    "  - [Loading the Dataset](session-5-part-1.ipynb#loading-the-dataset)\n",
    "  - [Training](session-5-part-1.ipynb#training-1)\n",
    "- [Part 3 - Latent-Space Arithmetic](session-5-part-1.ipynb#part-3---latent-space-arithmetic)\n",
    "  - [Loading the Pre-Trained Model](session-5-part-1.ipynb#loading-the-pre-trained-model)\n",
    "  - [Exploring the Celeb Net Attributes](session-5-part-1.ipynb#exploring-the-celeb-net-attributes)\n",
    "  - [Find the Latent Encoding for an Attribute](session-5-part-1.ipynb#find-the-latent-encoding-for-an-attribute)\n",
    "  - [Latent Feature Arithmetic](session-5-part-1.ipynb#latent-feature-arithmetic)\n",
    "  - [Extensions](session-5-part-1.ipynb#extensions)\n",
    "- [Part 4 - Character-Level Language Model](session-5-part-2.ipynb#part-4---character-level-language-model)\n",
    "- [Part 5 - Pretrained Char-RNN of Donald Trump](session-5-part-2.ipynb#part-5---pretrained-char-rnn-of-donald-trump)\n",
    "    - [Getting the Trump Data](session-5-part-2.ipynb#getting-the-trump-data)\n",
    "    - [Basic Text Analysis](session-5-part-2.ipynb#basic-text-analysis)\n",
    "    - [Loading the Pre-trained Trump Model](session-5-part-2.ipynb#loading-the-pre-trained-trump-model)\n",
    "    - [Inference: Keeping Track of the State](session-5-part-2.ipynb#inference-keeping-track-of-the-state)\n",
    "    - [Probabilistic Sampling](session-5-part-2.ipynb#probabilistic-sampling)\n",
    "    - [Inference: Temperature](session-5-part-2.ipynb#inference-temperature)\n",
    "    - [Inference: Priming](session-5-part-2.ipynb#inference-priming)\n",
    "- [Assignment Submission](session-5-part-2.ipynb#assignment-submission)\n",
    "\n",
    "<!-- /MarkdownTOC -->\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First check the Python version\n",
    "import sys\n",
    "if sys.version_info < (3,4):\n",
    "    print('You are running an older version of Python!\\n\\n',\n",
    "          'You should consider updating to Python 3.4.0 or',\n",
    "          'higher as the libraries built for this course',\n",
    "          'have only been tested in Python 3.4 and higher.\\n')\n",
    "    print('Try installing the Python 3.5 version of anaconda'\n",
    "          'and then restart `jupyter notebook`:\\n',\n",
    "          'https://www.continuum.io/downloads\\n\\n')\n",
    "\n",
    "# Now get necessary libraries\n",
    "try:\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    from scipy.ndimage.filters import gaussian_filter\n",
    "    import IPython.display as ipyd\n",
    "    import tensorflow as tf\n",
    "    from libs import utils, gif, datasets, dataset_utils, nb_utils\n",
    "except ImportError as e:\n",
    "    print(\"Make sure you have started notebook in the same directory\",\n",
    "          \"as the provided zip file which includes the 'libs' folder\",\n",
    "          \"and the file 'utils.py' inside of it.  You will NOT be able\",\n",
    "          \"to complete this assignment unless you restart jupyter\",\n",
    "          \"notebook inside the directory created by extracting\",\n",
    "          \"the zip file or cloning the github repo.\")\n",
    "    print(e)\n",
    "\n",
    "# We'll tell matplotlib to inline any drawn figures like so:\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Bit of formatting because I don't like the default inline code style:\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\n",
    "\n",
    "\n",
    "\n",
    "<a name=\"part-4---character-level-language-model\"></a>\n",
    "# Part 4 - Character-Level Language Model\n",
    "\n",
    "We'll now continue onto the second half of the homework and explore recurrent neural networks.  We saw one potential application of a recurrent neural network which learns letter by letter the content of a text file.  We were then able to synthesize from the model to produce new phrases.  Let's try to build one.  Replace the code below with something that loads your own text file or one from the internet.  Be creative with this!\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from six.moves import urllib\n",
    "script = 'http://www.awesomefilm.com/script/biglebowski.txt'\n",
    "txts = []\n",
    "f, _ = urllib.request.urlretrieve(script, script.split('/')[-1])\n",
    "with open(f, 'r') as fp:\n",
    "    txt = fp.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the first part of this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "txt[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll just clean up the text a little.  This isn't necessary, but can help the training along a little.  In the example text I provided, there is a lot of white space (those \\t's are tabs).  I'll remove them.  There are also repetitions of \\n, new lines, which are not necessary.  The code below will remove the tabs, ending whitespace, and any repeating newlines.  Replace this with any preprocessing that makes sense for your dataset.  Try to boil it down to just the possible letters for what you want to learn/synthesize while retaining any meaningful patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "txt = \"\\n\".join([txt_i.strip()\n",
    "                 for txt_i in txt.replace('\\t', '').split('\\n')\n",
    "                 if len(txt_i)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see how much text we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, we'll want as much text as possible.  But I'm including this just as a minimal example so you can explore your own.  Try making a text file and seeing the size of it.  You'll want about 1 MB at least.\n",
    "\n",
    "Let's now take a look at the different characters we have in our file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocab = list(set(txt))\n",
    "vocab.sort()\n",
    "print(len(vocab))\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then create a mapping which can take us from the letter to an integer look up table of that letter (and vice-versa).  To do this, we'll use an `OrderedDict` from the `collections` library.  In Python 3.6, this is the default behavior of dict, but in earlier versions of Python, we'll need to be explicit by using OrderedDict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import OrderedDict\n",
    "\n",
    "encoder = OrderedDict(zip(vocab, range(len(vocab))))\n",
    "decoder = OrderedDict(zip(range(len(vocab)), vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll store a few variables that will determine the size of our network.  First, `batch_size` determines how many sequences at a time we'll train on.  The `seqence_length` parameter defines the maximum length to unroll our recurrent network for.  This is effectively the depth of our network during training to help guide gradients along.  Within each layer, we'll have `n_cell` LSTM units, and `n_layers` layers worth of LSTM units.  Finally, we'll store the total number of possible characters in our data, which will determine the size of our one hot encoding (like we had for MNIST in Session 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of sequences in a mini batch\n",
    "batch_size = 100\n",
    "\n",
    "# Number of characters in a sequence\n",
    "sequence_length = 50\n",
    "\n",
    "# Number of cells in our LSTM layer\n",
    "n_cells = 128\n",
    "\n",
    "# Number of LSTM layers\n",
    "n_layers = 3\n",
    "\n",
    "# Total number of characters in the one-hot encoding\n",
    "n_chars = len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create the input and output to our network.  We'll use placeholders and feed these in later.  The size of these need to be [`batch_size`, `sequence_length`].  We'll then see how to build the network in between.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = tf.placeholder(tf.int32, shape=..., name='X')\n",
    "\n",
    "# We'll have a placeholder for our true outputs\n",
    "Y = tf.placeholder(tf.int32, shape=..., name='Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we need to do is convert each of our `sequence_length` vectors in our batch to `n_cells` LSTM cells.  We use a lookup table to find the value in `X` and use this as the input to `n_cells` LSTM cells.  Our lookup table has `n_chars` possible elements and connects each character to `n_cells` cells.  We create our lookup table using `tf.get_variable` and then the function `tf.nn.embedding_lookup` to connect our `X` placeholder to `n_cells` number of neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we first create a variable to take us from our one-hot representation to our LSTM cells\n",
    "embedding = tf.get_variable(\"embedding\", [n_chars, n_cells])\n",
    "\n",
    "# And then use tensorflow's embedding lookup to look up the ids in X\n",
    "Xs = tf.nn.embedding_lookup(embedding, X)\n",
    "\n",
    "# The resulting lookups are concatenated into a dense tensor\n",
    "print(Xs.get_shape().as_list())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now recall from the lecture that recurrent neural networks share their weights across timesteps.  So we don't want to have one large matrix with every timestep, but instead separate them.  We'll use `tf.split` to split our `[batch_size, sequence_length, n_cells]` array in `Xs` into a list of `sequence_length` elements each composed of `[batch_size, n_cells]` arrays.  This gives us `sequence_length` number of arrays of `[batch_size, 1, n_cells]`.  We then use `tf.squeeze` to remove the 1st index corresponding to the singleton `sequence_length` index, resulting in simply `[batch_size, n_cells]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('reslice'):\n",
    "    Xs = [tf.squeeze(seq, [1])\n",
    "          for seq in tf.split(Xs, sequence_length, 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With each of our timesteps split up, we can now connect them to a set of LSTM recurrent cells.  We tell the `tf.contrib.rnn.BasicLSTMCell` method how many cells we want, i.e. how many neurons there are, and we also specify that our state will be stored as a tuple.  This state defines the internal state of the cells as well as the connection from the previous timestep.  We can also pass a value for the `forget_bias`.  Be sure to experiment with this parameter as it can significantly effect performance (e.g. Gers, Felix A, Schmidhuber, Jurgen, and Cummins, Fred. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cells = tf.contrib.rnn.BasicLSTMCell(num_units=n_cells, state_is_tuple=True, forget_bias=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the cell's state size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cells.state_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`c` defines the internal memory and `h` the output.  We'll have as part of our `cells`, both an `initial_state` and a `final_state`.  These will become important during inference and we'll see how these work more then.  For now, we'll set the `initial_state` to all zeros using the convenience function provided inside our `cells` object, `zero_state`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "initial_state = cells.zero_state(tf.shape(X)[0], tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at what this does, we can see that it creates a `tf.Tensor` of zeros for our `c` and `h` states for each of our `n_cells` and stores this as a tuple inside the `LSTMStateTuple` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have created a single layer of LSTM cells composed of `n_cells` number of cells.  If we want another layer, we can use the `tf.contrib.rnn.MultiRNNCell` method, giving it our current cells, and a bit of pythonery to multiply our cells by the number of layers we want.  We'll then update our `initial_state` variable to include the additional cells:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cells = tf.contrib.rnn.MultiRNNCell(\n",
    "    [cells] * n_layers, state_is_tuple=True)\n",
    "initial_state = cells.zero_state(tf.shape(X)[0], tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if we take a look at our `initial_state`, we should see one `LSTMStateTuple` for each of our layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we haven't connected our recurrent cells to anything.  Let's do this now using the `tf.contrib.rnn.static_rnn` method.  We also pass it our `initial_state` variables.  It gives us the `outputs` of the rnn, as well as their states after having been computed.  Contrast that with the `initial_state`, which set the LSTM cells to zeros.  After having computed something, the cells will all have a different value somehow reflecting the temporal dynamics and expectations of the next input.  These will be stored in the `state` tensors for each of our LSTM layers inside a `LSTMStateTuple` just like the `initial_state` variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "help(tf.contrib.rnn.static_rnn)\n",
    "\n",
    "Help on function static_rnn in module tensorflow.contrib.rnn.python.ops.core_rnn:\n",
    "\n",
    "static_rnn(cell, inputs, initial_state=None, dtype=None, sequence_length=None, scope=None)\n",
    "    Creates a recurrent neural network specified by RNNCell `cell`.\n",
    "    \n",
    "    The simplest form of RNN network generated is:\n",
    "    \n",
    "      state = cell.zero_state(...)\n",
    "      outputs = []\n",
    "      for input_ in inputs:\n",
    "        output, state = cell(input_, state)\n",
    "        outputs.append(output)\n",
    "      return (outputs, state)\n",
    "    \n",
    "    However, a few other options are available:\n",
    "    \n",
    "    An initial state can be provided.\n",
    "    If the sequence_length vector is provided, dynamic calculation is performed.\n",
    "    This method of calculation does not compute the RNN steps past the maximum\n",
    "    sequence length of the minibatch (thus saving computational time),\n",
    "    and properly propagates the state at an example's sequence length\n",
    "    to the final state output.\n",
    "    \n",
    "    The dynamic calculation performed is, at time t for batch row b,\n",
    "      (output, state)(b, t) =\n",
    "        (t >= sequence_length(b))\n",
    "          ? (zeros(cell.output_size), states(b, sequence_length(b) - 1))\n",
    "          : cell(input(b, t), state(b, t - 1))\n",
    "    \n",
    "    Args:\n",
    "      cell: An instance of RNNCell.\n",
    "      inputs: A length T list of inputs, each a `Tensor` of shape\n",
    "        `[batch_size, input_size]`, or a nested tuple of such elements.\n",
    "      initial_state: (optional) An initial state for the RNN.\n",
    "        If `cell.state_size` is an integer, this must be\n",
    "        a `Tensor` of appropriate type and shape `[batch_size, cell.state_size]`.\n",
    "        If `cell.state_size` is a tuple, this should be a tuple of\n",
    "        tensors having shapes `[batch_size, s] for s in cell.state_size`.\n",
    "      dtype: (optional) The data type for the initial state and expected output.\n",
    "        Required if initial_state is not provided or RNN state has a heterogeneous\n",
    "        dtype.\n",
    "      sequence_length: Specifies the length of each sequence in inputs.\n",
    "        An int32 or int64 vector (tensor) size `[batch_size]`, values in `[0, T)`.\n",
    "      scope: VariableScope for the created subgraph; defaults to \"RNN\".\n",
    "    \n",
    "    Returns:\n",
    "      A pair (outputs, state) where:\n",
    "        - outputs is a length T list of outputs (one for each input), or a nested\n",
    "          tuple of such elements.\n",
    "        - state is the final state\n",
    "    \n",
    "    Raises:\n",
    "      TypeError: If `cell` is not an instance of RNNCell.\n",
    "      ValueError: If `inputs` is `None` or an empty list, or if the input depth\n",
    "        (column size) cannot be inferred from inputs via shape inference.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the help on the function `tf.contrib.rnn.static_rnn` to create the `outputs` and `states` variable as below.  We've already created each of the variable you need to use:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs, state = tf.contrib.rnn.static_rnn(cell=..., inputs=..., initial_state=...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the state now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our outputs are returned as a list for each of our timesteps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now stack all our outputs for every timestep.  We can treat every observation at each timestep and for each batch using the same weight matrices going forward, since these should all have shared weights.  Each timstep for each batch is its own observation.  So we'll stack these in a 2d matrix so that we can create our softmax layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs_flat = tf.reshape(tf.concat(values=outputs, axis=1), [-1, n_cells])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our outputs are now concatenated so that we have [`batch_size * timesteps`, `n_cells`]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create a softmax layer just like we did in Session 3 and in Session 3's homework.  We multiply our final LSTM layer's `n_cells` outputs by a weight matrix to give us `n_chars` outputs.  We then scale this output using a `tf.nn.softmax` layer so that they become a probability by exponentially scaling its value and dividing by its sum.  We store the softmax probabilities in `probs` as well as keep track of the maximum index in `Y_pred`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('prediction'):\n",
    "    W = tf.get_variable(\n",
    "        \"W\",\n",
    "        shape=[n_cells, n_chars],\n",
    "        initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "    b = tf.get_variable(\n",
    "        \"b\",\n",
    "        shape=[n_chars],\n",
    "        initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "\n",
    "    # Find the output prediction of every single character in our minibatch\n",
    "    # we denote the pre-activation prediction, logits.\n",
    "    logits = tf.matmul(outputs_flat, W) + b\n",
    "\n",
    "    # We get the probabilistic version by calculating the softmax of this\n",
    "    probs = tf.nn.softmax(logits)\n",
    "\n",
    "    # And then we can find the index of maximum probability\n",
    "    Y_pred = tf.argmax(probs, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the network, we'll measure the loss between our predicted outputs and true outputs.  We could use the `probs` variable, but we can also make use of `tf.nn.softmax_cross_entropy_with_logits` which will compute the softmax for us.  We therefore need to pass in the variable just before the softmax layer, denoted as `logits` (unscaled values).  This takes our variable `logits`, the unscaled predicted outputs, as well as our true outputs, `Y`.  Before we give it `Y`, we'll need to reshape our true outputs in the same way, [`batch_size` x `timesteps`, `n_chars`].  Luckily, tensorflow provides a convenience for doing this, the `tf.nn.sparse_softmax_cross_entropy_with_logits` function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "help(tf.nn.sparse_softmax_cross_entropy_with_logits)\n",
    "\n",
    "Help on function sparse_softmax_cross_entropy_with_logits in module tensorflow.python.ops.nn_ops:\n",
    "\n",
    "sparse_softmax_cross_entropy_with_logits(logits, labels, name=None)\n",
    "    Computes sparse softmax cross entropy between `logits` and `labels`.\n",
    "\n",
    "    Measures the probability error in discrete classification tasks in which the\n",
    "    classes are mutually exclusive (each entry is in exactly one class).  For\n",
    "    example, each CIFAR-10 image is labeled with one and only one label: an image\n",
    "    can be a dog or a truck, but not both.\n",
    "\n",
    "    **NOTE:**  For this operation, the probability of a given label is considered\n",
    "    exclusive.  That is, soft classes are not allowed, and the `labels` vector\n",
    "    must provide a single specific index for the true class for each row of\n",
    "    `logits` (each minibatch entry).  For soft softmax classification with\n",
    "    a probability distribution for each entry, see\n",
    "    `softmax_cross_entropy_with_logits`.\n",
    "\n",
    "    **WARNING:** This op expects unscaled logits, since it performs a softmax\n",
    "    on `logits` internally for efficiency.  Do not call this op with the\n",
    "    output of `softmax`, as it will produce incorrect results.\n",
    "\n",
    "    A common use case is to have logits of shape `[batch_size, num_classes]` and\n",
    "    labels of shape `[batch_size]`. But higher dimensions are supported.\n",
    "\n",
    "    Args:\n",
    "      logits: Unscaled log probabilities of rank `r` and shape\n",
    "        `[d_0, d_1, ..., d_{r-2}, num_classes]` and dtype `float32` or `float64`.\n",
    "      labels: `Tensor` of shape `[d_0, d_1, ..., d_{r-2}]` and dtype `int32` or\n",
    "        `int64`. Each entry in `labels` must be an index in `[0, num_classes)`.\n",
    "        Other values will result in a loss of 0, but incorrect gradient\n",
    "        computations.\n",
    "      name: A name for the operation (optional).\n",
    "\n",
    "    Returns:\n",
    "      A `Tensor` of the same shape as `labels` and of the same type as `logits`\n",
    "      with the softmax cross entropy loss.\n",
    "\n",
    "    Raises:\n",
    "      ValueError: If logits are scalars (need to have rank >= 1) or if the rank\n",
    "        of the labels is not equal to the rank of the labels minus one.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('loss'):\n",
    "    # Compute mean cross entropy loss for each output.\n",
    "    Y_true_flat = tf.reshape(tf.concat(values=Y, axis=1), [-1])\n",
    "    # logits are [batch_size x timesteps, n_chars] and\n",
    "    # Y_true_flat are [batch_size x timesteps]\n",
    "    loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=Y_true_flat, logits=logits)\n",
    "    # Compute the mean over our `batch_size` x `timesteps` number of observations\n",
    "    mean_loss = tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can create an optimizer in much the same way as we've done with every other network.  Except, we will also \"clip\" the gradients of every trainable parameter.  This is a hacky way to ensure that the gradients do not grow too large (the literature calls this the \"exploding gradient problem\").  However, note that the LSTM is built to help ensure this does not happen by allowing the gradient to be \"gated\".  To learn more about this, please consider reading the following material:\n",
    "\n",
    "http://www.felixgers.de/papers/phd.pdf  \n",
    "https://colah.github.io/posts/2015-08-Understanding-LSTMs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('optimizer'):\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=0.001)\n",
    "    gradients = []\n",
    "    clip = tf.constant(5.0, name=\"clip\")\n",
    "    for grad, var in optimizer.compute_gradients(mean_loss):\n",
    "        gradients.append((tf.clip_by_value(grad, -clip, clip), var))\n",
    "    updates = optimizer.apply_gradients(gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_utils.show_graph(tf.get_default_graph().as_graph_def())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the rest of code we'll need to train the network.  I do not recommend running this inside Jupyter Notebook for the entire length of the training because the network can take 1-2 days at least to train, and your browser may very likely complain.  Instead, you should write a python script containing the necessary bits of code and run it using the Terminal.  We didn't go over how to do this, so I'll leave it for you as an exercise.  The next part of this notebook will have you load a pre-trained network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "\n",
    "    cursor = 0\n",
    "    it_i = 0\n",
    "    while it_i < 500:\n",
    "        Xs, Ys = [], []\n",
    "        for batch_i in range(batch_size):\n",
    "            if (cursor + sequence_length) >= len(txt) - sequence_length - 1:\n",
    "                cursor = 0\n",
    "            Xs.append([encoder[ch]\n",
    "                       for ch in txt[cursor:cursor + sequence_length]])\n",
    "            Ys.append([encoder[ch]\n",
    "                       for ch in txt[cursor + 1: cursor + sequence_length + 1]])\n",
    "\n",
    "            cursor = (cursor + sequence_length)\n",
    "        Xs = np.array(Xs).astype(np.int32)\n",
    "        Ys = np.array(Ys).astype(np.int32)\n",
    "\n",
    "        loss_val, _ = sess.run([mean_loss, updates],\n",
    "                               feed_dict={X: Xs, Y: Ys})\n",
    "        if it_i % 100 == 0:\n",
    "            print(it_i, loss_val)\n",
    "\n",
    "        if it_i % 500 == 0:\n",
    "            p = sess.run(probs, feed_dict={X: np.array(Xs[-1])[np.newaxis]})\n",
    "            ps = [np.random.choice(range(n_chars), p=p_i.ravel())\n",
    "                  for p_i in p]\n",
    "            p = [np.argmax(p_i) for p_i in p]\n",
    "            if isinstance(txt[0], str):\n",
    "                print('original:', \"\".join(\n",
    "                    [decoder[ch] for ch in Xs[-1]]))\n",
    "                print('synth(samp):', \"\".join(\n",
    "                    [decoder[ch] for ch in ps]))\n",
    "                print('synth(amax):', \"\".join(\n",
    "                    [decoder[ch] for ch in p]))\n",
    "            else:\n",
    "                print([decoder[ch] for ch in ps])\n",
    "\n",
    "        it_i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"part-5---pretrained-char-rnn-of-donald-trump\"></a>\n",
    "# Part 5 - Pretrained Char-RNN of Donald Trump\n",
    "\n",
    "Rather than stick around to let a model train, let's now explore one I've trained for you Donald Trump.  If you've trained your own model on your own text corpus then great!  You should be able to use that in place of the one I've provided and still continue with the rest of the notebook. \n",
    "\n",
    "For the Donald Trump corpus, there are a lot of video transcripts that you can find online.  I've searched for a few of these, put them in a giant text file, made everything lowercase, and removed any extraneous letters/symbols to help reduce the vocabulary (not that it's not very large to begin with, ha).\n",
    "\n",
    "I used the code exactly as above to train on the text I gathered and left it to train for about 2 days.  The only modification is that I also used \"dropout\" which you can see in the libs/charrnn.py file.  Let's explore it now and we'll see how we can play with \"sampling\" the model to generate new phrases, and how to \"prime\" the model (a psychological term referring to when someone is exposed to something shortly before another event).\n",
    "\n",
    "First, let's clean up any existing graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"getting-the-trump-data\"></a>\n",
    "## Getting the Trump Data\n",
    "\n",
    "Now let's load the text.  This is included in the repo or can be downloaded from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('trump.txt', 'r') as fp:\n",
    "    txt = fp.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at what's going on in here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "txt[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"basic-text-analysis\"></a>\n",
    "## Basic Text Analysis\n",
    "\n",
    "We can do some basic data analysis to get a sense of what kind of vocabulary we're working with.  It's really important to look at your data in as many ways as possible.  This helps ensure there isn't anything unexpected going on.  Let's find every unique word he uses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = set(txt.split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's count their occurrences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts = {word_i: 0 for word_i in words}\n",
    "for word_i in txt.split(' '):\n",
    "    counts[word_i] += 1\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort this like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[(word_i, counts[word_i]) for word_i in sorted(counts, key=counts.get, reverse=True)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we should expect, \"the\" is the most common word, as it is in the English language: https://en.wikipedia.org/wiki/Most_common_words_in_English\n",
    "\n",
    "<a name=\"loading-the-pre-trained-trump-model\"></a>\n",
    "## Loading the Pre-trained Trump Model\n",
    "\n",
    "Let's load the pretrained model.  Rather than provide a tfmodel export, I've provided the checkpoint so you can also experiment with training it more if you wish.  We'll rebuild the graph using the `charrnn` module in the `libs` directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from libs import charrnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the checkpoint and build the model then restore the variables from the checkpoint.  The only parameters of consequence are `n_layers` and `n_cells` which define the total size and layout of the model.  The rest are flexible.  We'll set the `batch_size` and `sequence_length` to 1, meaning we can feed in a single character at a time only, and get back 1 character denoting the very next character's prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ckpt_name = './trump.ckpt'\n",
    "g = tf.Graph()\n",
    "n_layers = 3\n",
    "n_cells = 512\n",
    "with tf.Session(graph=g) as sess:\n",
    "    model = charrnn.build_model(txt=txt,\n",
    "                                batch_size=1,\n",
    "                                sequence_length=1,\n",
    "                                n_layers=n_layers,\n",
    "                                n_cells=n_cells,\n",
    "                                gradient_clip=10.0)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    if os.path.exists(ckpt_name):\n",
    "        saver.restore(sess, ckpt_name)\n",
    "        print(\"Model restored.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now take a look at the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_utils.show_graph(g.as_graph_def())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_iterations = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"inference-keeping-track-of-the-state\"></a>\n",
    "## Inference: Keeping Track of the State\n",
    "\n",
    "Now recall from Part 4 when we created our LSTM network, we had an `initial_state` variable which would set the LSTM's `c` and `h` state vectors, as well as the final output state which was the output of the `c` and `h` state vectors after having passed through the network.  When we input to the network some letter, say 'n', we can set the `initial_state` to zeros, but then after having input the letter `n`, we'll have as output a new state vector for `c` and `h`.  On the next letter, we'll then want to set the `initial_state` to this new state, and set the input to the previous letter's output.  That is how we ensure the network keeps track of time and knows what has happened in the past, and let it continually generate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "curr_states = None\n",
    "g = tf.Graph()\n",
    "with tf.Session(graph=g) as sess:\n",
    "    model = charrnn.build_model(txt=txt,\n",
    "                                batch_size=1,\n",
    "                                sequence_length=1,\n",
    "                                n_layers=n_layers,\n",
    "                                n_cells=n_cells,\n",
    "                                gradient_clip=10.0)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    if os.path.exists(ckpt_name):\n",
    "        saver.restore(sess, ckpt_name)\n",
    "        print(\"Model restored.\")\n",
    "        \n",
    "    # Get every tf.Tensor for the initial state\n",
    "    init_states = []\n",
    "    for s_i in model['initial_state']:\n",
    "        init_states.append(s_i.c)\n",
    "        init_states.append(s_i.h)\n",
    "        \n",
    "    # Similarly, for every state after inference\n",
    "    final_states = []\n",
    "    for s_i in model['final_state']:\n",
    "        final_states.append(s_i.c)\n",
    "        final_states.append(s_i.h)\n",
    "\n",
    "    # Let's start with the letter 't' and see what comes out:\n",
    "    synth = [[encoder[' ']]]\n",
    "    for i in range(n_iterations):\n",
    "\n",
    "        # We'll create a feed_dict parameter which includes what to\n",
    "        # input to the network, model['X'], as well as setting\n",
    "        # dropout to 1.0, meaning no dropout.\n",
    "        feed_dict = {model['X']: [synth[-1]],\n",
    "                     model['keep_prob']: 1.0}\n",
    "        \n",
    "        # Now we'll check if we currently have a state as a result\n",
    "        # of a previous inference, and if so, add to our feed_dict\n",
    "        # parameter the mapping of the init_state to the previous\n",
    "        # output state stored in \"curr_states\".\n",
    "        if curr_states:\n",
    "            feed_dict.update(\n",
    "                {init_state_i: curr_state_i\n",
    "                 for (init_state_i, curr_state_i) in\n",
    "                     zip(init_states, curr_states)})\n",
    "            \n",
    "        # Now we can infer and see what letter we get\n",
    "        p = sess.run(model['probs'], feed_dict=feed_dict)[0]\n",
    "        \n",
    "        # And make sure we also keep track of the new state\n",
    "        curr_states = sess.run(final_states, feed_dict=feed_dict)\n",
    "        \n",
    "        # Find the most likely character\n",
    "        p = np.argmax(p)\n",
    "        \n",
    "        # Append to string\n",
    "        synth.append([p])\n",
    "        \n",
    "        # Print out the decoded letter\n",
    "        print(model['decoder'][p], end='')\n",
    "        sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"probabilistic-sampling\"></a>\n",
    "## Probabilistic Sampling\n",
    "\n",
    "Run the above cell a couple times.  What you should find is that it is deterministic.  We always pick *the* most likely character.  But we can do something else which will make things less deterministic and a bit more interesting: we can sample from our probabilistic measure from our softmax layer.  This means if we have the letter 'a' as 0.4, and the letter 'o' as 0.2, we'll have a 40% chance of picking the letter 'a', and 20% chance of picking the letter 'o', rather than simply always picking the letter 'a' since it is the most probable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "curr_states = None\n",
    "g = tf.Graph()\n",
    "with tf.Session(graph=g) as sess:\n",
    "    model = charrnn.build_model(txt=txt,\n",
    "                                batch_size=1,\n",
    "                                sequence_length=1,\n",
    "                                n_layers=n_layers,\n",
    "                                n_cells=n_cells,\n",
    "                                gradient_clip=10.0)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    if os.path.exists(ckpt_name):\n",
    "        saver.restore(sess, ckpt_name)\n",
    "        print(\"Model restored.\")\n",
    "        \n",
    "    # Get every tf.Tensor for the initial state\n",
    "    init_states = []\n",
    "    for s_i in model['initial_state']:\n",
    "        init_states.append(s_i.c)\n",
    "        init_states.append(s_i.h)\n",
    "        \n",
    "    # Similarly, for every state after inference\n",
    "    final_states = []\n",
    "    for s_i in model['final_state']:\n",
    "        final_states.append(s_i.c)\n",
    "        final_states.append(s_i.h)\n",
    "\n",
    "    # Let's start with the letter 't' and see what comes out:\n",
    "    synth = [[encoder[' ']]]\n",
    "    for i in range(n_iterations):\n",
    "\n",
    "        # We'll create a feed_dict parameter which includes what to\n",
    "        # input to the network, model['X'], as well as setting\n",
    "        # dropout to 1.0, meaning no dropout.\n",
    "        feed_dict = {model['X']: [synth[-1]],\n",
    "                     model['keep_prob']: 1.0}\n",
    "        \n",
    "        # Now we'll check if we currently have a state as a result\n",
    "        # of a previous inference, and if so, add to our feed_dict\n",
    "        # parameter the mapping of the init_state to the previous\n",
    "        # output state stored in \"curr_states\".\n",
    "        if curr_states:\n",
    "            feed_dict.update(\n",
    "                {init_state_i: curr_state_i\n",
    "                 for (init_state_i, curr_state_i) in\n",
    "                     zip(init_states, curr_states)})\n",
    "            \n",
    "        # Now we can infer and see what letter we get\n",
    "        p = sess.run(model['probs'], feed_dict=feed_dict)[0]\n",
    "        \n",
    "        # And make sure we also keep track of the new state\n",
    "        curr_states = sess.run(final_states, feed_dict=feed_dict)\n",
    "        \n",
    "        # Now instead of finding the most likely character,\n",
    "        # we'll sample with the probabilities of each letter\n",
    "        p = p.astype(np.float64)\n",
    "        p = np.random.multinomial(1, p.ravel() / p.sum())\n",
    "        p = np.argmax(p)\n",
    "        \n",
    "        # Append to string\n",
    "        synth.append([p])\n",
    "        \n",
    "        # Print out the decoded letter\n",
    "        print(model['decoder'][p], end='')\n",
    "        sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"inference-temperature\"></a>\n",
    "## Inference: Temperature\n",
    "\n",
    "When performing probabilistic sampling, we can also use a parameter known as temperature which comes from simulated annealing.  The basic idea is that as the temperature is high and very hot, we have a lot more free energy to use to jump around more, and as we cool down, we have less energy and then become more deterministic.  We can use temperature by scaling our log probabilities like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "temperature = 0.5\n",
    "curr_states = None\n",
    "g = tf.Graph()\n",
    "with tf.Session(graph=g) as sess:\n",
    "    model = charrnn.build_model(txt=txt,\n",
    "                                batch_size=1,\n",
    "                                sequence_length=1,\n",
    "                                n_layers=n_layers,\n",
    "                                n_cells=n_cells,\n",
    "                                gradient_clip=10.0)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    if os.path.exists(ckpt_name):\n",
    "        saver.restore(sess, ckpt_name)\n",
    "        print(\"Model restored.\")\n",
    "        \n",
    "    # Get every tf.Tensor for the initial state\n",
    "    init_states = []\n",
    "    for s_i in model['initial_state']:\n",
    "        init_states.append(s_i.c)\n",
    "        init_states.append(s_i.h)\n",
    "        \n",
    "    # Similarly, for every state after inference\n",
    "    final_states = []\n",
    "    for s_i in model['final_state']:\n",
    "        final_states.append(s_i.c)\n",
    "        final_states.append(s_i.h)\n",
    "\n",
    "    # Let's start with the letter 't' and see what comes out:\n",
    "    synth = [[encoder[' ']]]\n",
    "    for i in range(n_iterations):\n",
    "\n",
    "        # We'll create a feed_dict parameter which includes what to\n",
    "        # input to the network, model['X'], as well as setting\n",
    "        # dropout to 1.0, meaning no dropout.\n",
    "        feed_dict = {model['X']: [synth[-1]],\n",
    "                     model['keep_prob']: 1.0}\n",
    "        \n",
    "        # Now we'll check if we currently have a state as a result\n",
    "        # of a previous inference, and if so, add to our feed_dict\n",
    "        # parameter the mapping of the init_state to the previous\n",
    "        # output state stored in \"curr_states\".\n",
    "        if curr_states:\n",
    "            feed_dict.update(\n",
    "                {init_state_i: curr_state_i\n",
    "                 for (init_state_i, curr_state_i) in\n",
    "                     zip(init_states, curr_states)})\n",
    "            \n",
    "        # Now we can infer and see what letter we get\n",
    "        p = sess.run(model['probs'], feed_dict=feed_dict)[0]\n",
    "        \n",
    "        # And make sure we also keep track of the new state\n",
    "        curr_states = sess.run(final_states, feed_dict=feed_dict)\n",
    "        \n",
    "        # Now instead of finding the most likely character,\n",
    "        # we'll sample with the probabilities of each letter\n",
    "        p = p.astype(np.float64)\n",
    "        p = np.log(p) / temperature\n",
    "        p = np.exp(p) / np.sum(np.exp(p))\n",
    "        p = np.random.multinomial(1, p.ravel() / p.sum())\n",
    "        p = np.argmax(p)\n",
    "        \n",
    "        # Append to string\n",
    "        synth.append([p])\n",
    "        \n",
    "        # Print out the decoded letter\n",
    "        print(model['decoder'][p], end='')\n",
    "        sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"inference-priming\"></a>\n",
    "## Inference: Priming\n",
    "\n",
    "Let's now work on \"priming\" the model with some text, and see what kind of state it is in and leave it to synthesize from there.  We'll do more or less what we did before, but feed in our own text instead of the last letter of the synthesis from the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prime = \"obama\"\n",
    "temperature = 1.0\n",
    "curr_states = None\n",
    "n_iterations = 500\n",
    "g = tf.Graph()\n",
    "with tf.Session(graph=g) as sess:\n",
    "    model = charrnn.build_model(txt=txt,\n",
    "                                batch_size=1,\n",
    "                                sequence_length=1,\n",
    "                                n_layers=n_layers,\n",
    "                                n_cells=n_cells,\n",
    "                                gradient_clip=10.0)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    if os.path.exists(ckpt_name):\n",
    "        saver.restore(sess, ckpt_name)\n",
    "        print(\"Model restored.\")\n",
    "        \n",
    "    # Get every tf.Tensor for the initial state\n",
    "    init_states = []\n",
    "    for s_i in model['initial_state']:\n",
    "        init_states.append(s_i.c)\n",
    "        init_states.append(s_i.h)\n",
    "        \n",
    "    # Similarly, for every state after inference\n",
    "    final_states = []\n",
    "    for s_i in model['final_state']:\n",
    "        final_states.append(s_i.c)\n",
    "        final_states.append(s_i.h)\n",
    "\n",
    "    # Now we'll keep track of the state as we feed it one\n",
    "    # letter at a time.\n",
    "    curr_states = None\n",
    "    for ch in prime:\n",
    "        feed_dict = {model['X']: [[model['encoder'][ch]]],\n",
    "                     model['keep_prob']: 1.0}\n",
    "        if curr_states:\n",
    "            feed_dict.update(\n",
    "                {init_state_i: curr_state_i\n",
    "                 for (init_state_i, curr_state_i) in\n",
    "                     zip(init_states, curr_states)})\n",
    "        \n",
    "        # Now we can infer and see what letter we get\n",
    "        p = sess.run(model['probs'], feed_dict=feed_dict)[0]\n",
    "        p = p.astype(np.float64)\n",
    "        p = np.log(p) / temperature\n",
    "        p = np.exp(p) / np.sum(np.exp(p))\n",
    "        p = np.random.multinomial(1, p.ravel() / p.sum())\n",
    "        p = np.argmax(p)\n",
    "        \n",
    "        # And make sure we also keep track of the new state\n",
    "        curr_states = sess.run(final_states, feed_dict=feed_dict)\n",
    "        \n",
    "    # Now we're ready to do what we were doing before but with the\n",
    "    # last predicted output stored in `p`, and the current state of\n",
    "    # the model.\n",
    "    synth = [[p]]\n",
    "    print(prime + model['decoder'][p], end='')\n",
    "    for i in range(n_iterations):\n",
    "\n",
    "        # Input to the network\n",
    "        feed_dict = {model['X']: [synth[-1]],\n",
    "                     model['keep_prob']: 1.0}\n",
    "        \n",
    "        # Also feed our current state\n",
    "        feed_dict.update(\n",
    "            {init_state_i: curr_state_i\n",
    "             for (init_state_i, curr_state_i) in\n",
    "                 zip(init_states, curr_states)})\n",
    "            \n",
    "        # Inference\n",
    "        p = sess.run(model['probs'], feed_dict=feed_dict)[0]\n",
    "        \n",
    "        # Keep track of the new state\n",
    "        curr_states = sess.run(final_states, feed_dict=feed_dict)\n",
    "        \n",
    "        # Sample\n",
    "        p = p.astype(np.float64)\n",
    "        p = np.log(p) / temperature\n",
    "        p = np.exp(p) / np.sum(np.exp(p))\n",
    "        p = np.random.multinomial(1, p.ravel() / p.sum())\n",
    "        p = np.argmax(p)\n",
    "        \n",
    "        # Append to string\n",
    "        synth.append([p])\n",
    "        \n",
    "        # Print out the decoded letter\n",
    "        print(model['decoder'][p], end='')\n",
    "        sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment-submission\"></a>\n",
    "# Assignment Submission\n",
    "After you've completed both notebooks, create a zip file of the current directory using the code below. This code will make sure you have included this completed ipython notebook and the following files named exactly as:\n",
    "\n",
    "    session-5/  \n",
    "      session-5-part-1.ipynb  \n",
    "      session-5-part-2.ipynb  \n",
    "      vaegan.gif"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll then submit this zip file for your third assignment on Kadenze for \"Assignment 5: Generative Adversarial Networks and Recurrent Neural Networks\"!  If you have any questions, remember to reach out on the forums and connect with your peers or with me.\n",
    "\n",
    "To get assessed, you'll need to be a premium student! This will allow you to build an online portfolio of all of your work and receive grades. If you aren't already enrolled as a student, register now at http://www.kadenze.com/ and join the #CADL community to see what your peers are doing! https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\n",
    "\n",
    "Also, if you share any of the GIFs on Facebook/Twitter/Instagram/etc..., be sure to use the #CADL hashtag so that other students can find your work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "utils.build_submission('session-5.zip',\n",
    "                       ('vaegan.gif',\n",
    "                        'session-5-part-1.ipynb',\n",
    "                        'session-5-part-2.ipynb'))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}