Writing_like_Dante.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Writing like Dante.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1ZewaYqjGxiExEfvj6TWYJGigiR6uluMi",
      "authorship_tag": "ABX9TyNax8hMsJ7VaKSvb0aH1WHG",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/GiuliaLanzillotta/exercises/blob/master/Writing_like_Dante.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vKipj56slaV",
        "colab_type": "text"
      },
      "source": [
        "# Text character-level prediction\n",
        "The inspiration for this notebook is drawn from [this beautiful blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n",
        "\n",
        "\n",
        "> #### But what is our goal today? \n",
        "We’ll train **RNN character-level language models**. That is, we’ll give the RNN a huge chunk of text and ask it to model the probability distribution of the next character in the sequence given a sequence of previous characters. This will then allow us to generate new text one character at a time.\n",
        "\n",
        "\n",
        "Let's have some fun!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBFd4fjhs0MD",
        "colab_type": "text"
      },
      "source": [
        " \n",
        "We'll get the data from the Gutenberg dataset.\n",
        "> ### About the dataset: \n",
        "The **Gutemberg project** offers a large collection of free books that can be retrieved in plain text for a variety of languages.\n",
        "<br> I have picked the **Divina Commedia - Canto I** by *Dante ALighieri* (Italian version) for this notebook. The full text is available [here](http://www.gutenberg.org/cache/epub/1009/pg1009.txt).\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3KRE4suj2UQ9",
        "colab_type": "code",
        "outputId": "ce77cbd0-2bca-4318-b9f3-544f9a005b04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "!wget -nv 'http://www.gutenberg.org/cache/epub/1009/pg1009.txt' -O 'divina_commedia.txt'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-04-01 09:51:13 URL:http://www.gutenberg.org/cache/epub/1009/pg1009.txt [221280/221280] -> \"divina_commedia.txt\" [1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SmXJ8MeC2uli",
        "colab_type": "code",
        "outputId": "66aa41dd-a2d0-4e04-dea0-8fbbbd0a072d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        }
      },
      "source": [
        "!head -80 'divina_commedia.txt' | tail -20"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  Tant è amara che poco è più morte;\r\n",
            "  ma per trattar del ben chi vi trovai,\r\n",
            "  dirò de laltre cose chi vho scorte.\r\n",
            "\r\n",
            "  Io non so ben ridir com i vintrai,\r\n",
            "  tant era pien di sonno a quel punto\r\n",
            "  che la verace via abbandonai.\r\n",
            "\r\n",
            "  Ma poi chi fui al piè dun colle giunto,\r\n",
            "  là dove terminava quella valle\r\n",
            "  che mavea di paura il cor compunto,\r\n",
            "\r\n",
            "  guardai in alto e vidi le sue spalle\r\n",
            "  vestite già de raggi del pianeta\r\n",
            "  che mena dritto altrui per ogne calle.\r\n",
            "\r\n",
            "  Allor fu la paura un poco queta,\r\n",
            "  che nel lago del cor mera durata\r\n",
            "  la notte chi passai con tanta pieta.\r\n",
            "\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4JCoLKHuz7i",
        "colab_type": "text"
      },
      "source": [
        "## Preprocessing \n",
        "---\n",
        "We have now to pre-process this large .txt file to make it ready to be fed as input to the model. <br>\n",
        "\n",
        "Our steps will be : \n",
        "\n",
        "    1. Cut off the sections that do not belong to the original text .\n",
        "    2. Build a vocabulary for our inputs: <br> \n",
        "      Since the input are single characters, the vocabulary should contain the characters from the text language's alphabet.\n",
        "    3. Create the batches : the sequence of characters, which is the text, should be split into multiple batches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAiHqNw95d_U",
        "colab_type": "text"
      },
      "source": [
        "### 1. Cut off the added parts "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0dNi9HHH3cXM",
        "colab_type": "code",
        "outputId": "b309cbca-0ae2-425c-e251-5d0f6f33d9fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Let's cut the parts that were not included in the original text \n",
        "!wc -l 'divina_commedia.txt'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "6949 divina_commedia.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8eC5FumF4Rg2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!head -30 'divina_commedia.txt'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EhTcyMd84Snu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tail -410 'divina_commedia.txt'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jx1YFMXg40Ig",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# So we have to cut the first 30 lines and the last 410 \n",
        "# With a few calculations I came up with these numbers: \n",
        "!tail -6919 'divina_commedia.txt' | head -6505 > 'divina_commedia_cut.txt'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W5xrq5I65IiT",
        "colab_type": "code",
        "outputId": "bc67515f-50c1-4b68-def4-2b205e35cfb8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        }
      },
      "source": [
        "!head -10 'divina_commedia_cut.txt'\n",
        "!tail -10 'divina_commedia_cut.txt'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "  LA DIVINA COMMEDIA\r\n",
            "  di Dante Alighieri\r\n",
            "\n",
            "  e sanza cura aver dalcun riposo,\n",
            "\n",
            "  salimmo sù, el primo e io secondo,\n",
            "  tanto chi vidi de le cose belle\n",
            "  che porta l ciel, per un pertugio tondo.\n",
            "\n",
            "  E quindi uscimmo a riveder le stelle.\n",
            "\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WBEiR_9Y5l4s",
        "colab_type": "text"
      },
      "source": [
        "### 2. Build the vocabulary "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lq1CUbAu86WQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pickle\n",
        "import os\n",
        "import collections"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q7vIyqWuvKAh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Parameters :\n",
        "seq_length = 50\n",
        "batch_size = 128 \n",
        "encoding = 'utf-8'\n",
        "seed = np.random.RandomState(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vNc76XkQ0mOS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Files' locations :\n",
        "input_file = './divina_commedia_cut.txt'\n",
        "vocab_file = './vocab.pkl' # where we'll save the vocabulary \n",
        "tensor_file = '.data.npy' "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FRSbG2yt15Be",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# The vocabulary will contain all the characters that we can find in the text \n",
        "import codecs\n",
        "with codecs.open(input_file, \"r\", encoding=encoding) as f:\n",
        "    data = f.read()\n",
        "counter = collections.Counter(data)\n",
        "count_pairs = sorted(counter.items(), key=lambda x: -x[1])\n",
        "chars, _ = zip(*count_pairs)\n",
        "vocab_size = len(chars)\n",
        "vocab = dict(zip(chars, range(len(chars))))\n",
        "\n",
        "# save the vocabulary using pickle\n",
        "with open(vocab_file, 'wb') as f:\n",
        "    pickle.dump(chars, f)\n",
        "# save the text -as a sequence of characters, encoded with their \n",
        "# vocabulary index- in a numpy vector (tensor) using numpy\n",
        "tensor = np.array(list(map(vocab.get, data)))\n",
        "np.save(tensor_file, tensor)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9y9KlNu89yue",
        "colab_type": "code",
        "outputId": "332cc585-c1ec-4264-ea53-608433fc7d40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "print(\"loaded vocabulary with {} letters\".format(vocab_size))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loaded vocabulary with 73 letters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1vvsFmuxIWuR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Let's save a version of the vocabulary which cointains the characters as keys \n",
        "vocab_inv = {v: k for k, v in vocab.items()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkjfy3X574e6",
        "colab_type": "text"
      },
      "source": [
        "### 3. Create the batches "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xBf_5jq_7-FR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_batches = int(tensor.size/(batch_size*seq_length))\n",
        "tensor = tensor[:num_batches * batch_size * seq_length] # reshaping the tensor according to the batches defined"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8qp3MkH99SbW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# We now build input and output tensors: \n",
        "# the output tensor (the sequence of characters that have to be predicted)\n",
        "# will be equal to the input tensor- with each character shifted by 1 to the left\n",
        "# Example: \n",
        "#   input tensor = \"H e l l o\"\n",
        "#   output tensor = \" e l l o H\"\n",
        "xdata = tensor # the input tensor \n",
        "ydata = np.copy(tensor) # the output tensor \n",
        "ydata[:-1] = xdata[1:]\n",
        "ydata[-1] = xdata[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42423aY-9WJB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_batches = np.split(xdata.reshape(batch_size, -1),num_batches, 1)\n",
        "y_batches = np.split(ydata.reshape(batch_size, -1),num_batches, 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i90DN1Z3_4L5",
        "colab_type": "code",
        "outputId": "fc616342-b098-45f9-e509-e19fa1525847",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "# A quick look at the batches, to make sure everything matches our expectations \n",
        "b = x_batches[2]\n",
        "t = y_batches[2]\n",
        "print('total of {} batches of shape: {}'.format(len(x_batches), b.shape))\n",
        "print('content of batch 0, entry 0, time steps 0 to 10')\n",
        "print('input : {}'.format(b[0, :10]))\n",
        "print('target: {}'.format(t[0, :10]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total of 30 batches of shape: (128, 50)\n",
            "content of batch 0, entry 0, time steps 0 to 10\n",
            "input : [ 2  5  8  4  0 39 11 12 11 12]\n",
            "target: [ 5  8  4  0 39 11 12 11 12 11]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T98dqo28Iher",
        "colab_type": "code",
        "outputId": "cde662a0-8759-4db0-8b47-0751fc3e4510",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "print('input : {}'.format([vocab_inv[i] for i in b[0, :20]]))\n",
        "print('target: {}'.format([vocab_inv[i] for i in t[0, :20]]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "input : ['a', 'n', 't', 'o', ' ', 'I', '\\r', '\\n', '\\r', '\\n', '\\r', '\\n', ' ', ' ', 'N', 'e', 'l', ' ', 'm', 'e']\n",
            "target: ['n', 't', 'o', ' ', 'I', '\\r', '\\n', '\\r', '\\n', '\\r', '\\n', ' ', ' ', 'N', 'e', 'l', ' ', 'm', 'e', 'z']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6DQsVY1A-YK0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Helper function: this will be useful to scan the batches \n",
        "def next_batch(x_batches, y_batches, pointer):\n",
        "  \"\"\" \n",
        "  Scans the next batch. \n",
        "  Parameters: \n",
        "    - x_batches: numpy tensor\n",
        "    - y_batches: numpy tensor\n",
        "    - pointer: int \n",
        "        It represents the current index. \n",
        "  Returns: \n",
        "    - x: char\n",
        "    - y: char\n",
        "    - pointer : int\n",
        "        New position of the pointer \n",
        "  \"\"\"\n",
        "  x, y = x_batches[pointer], y_batches[pointer]\n",
        "  pointer += 1\n",
        "  return x, y, pointer\n",
        "pointer = 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sfoGUZSpAQeW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Helper function to reshuffle the batches. \n",
        "# To use when starting a new epoch\n",
        "def reshuffle(x_batches, y_batches):\n",
        "  \"\"\"\n",
        "  Permutes the order of the input and output batches.\n",
        "  \"\"\"\n",
        "  idx = seed.permutation(len(x_batches))\n",
        "  x_batches = [x_batches[i] for i in idx]\n",
        "  y_batches = [y_batches[i] for i in idx]\n",
        "  return x_batches, y_batches"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4R_ognb2Axtc",
        "colab_type": "text"
      },
      "source": [
        "## Model\n",
        "---\n",
        "In this section we will build and train an *LSTM* from scratch using Tensorlfow 1.x APIs. <br>\n",
        "The steps are the following: \n",
        "\n",
        "    1. Build the model. \n",
        "    2. Set the loss and the optimizer.\n",
        "    3. Train the model for a few steps.\n",
        "    4. Save the model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3OCSaolTWd1Q",
        "colab_type": "text"
      },
      "source": [
        "### Build the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B54t7EgOAy43",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Hyperparameters\n",
        "learning_rate = 1e-3\n",
        "hidden_size = 256 #Size of one LSTM hidden layer\n",
        "num_layers = 2 #How many LSTM layers to use\n",
        "print_every_steps = 20 #How often to print progress to the console\n",
        "log_dir= \"/tmp/tensorflow/divina_rnn/logs\" #Where to store summaries and checkpoints\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CfYTUFAtBV4M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.util import deprecation\n",
        "deprecation._PRINT_DEPRECATION_WARNINGS = False\n",
        "def rnn_lstm(inputs, seq_lengths, hidden_size=hidden_size, num_layers=num_layers):\n",
        "    \"\"\"\n",
        "    Builds an RNN with LSTM cells.\n",
        "    Parameters\n",
        "    -----------\n",
        "    - inputs: numpy tensor\n",
        "        The input tensor to the RNN in shape `[batch_size, seq_length]`.\n",
        "    - seq_lengths: \n",
        "        Tensor of shape `[batch_size]` specifying the total number \n",
        "        of time steps per sequence.\n",
        "    -hidden_size: int\n",
        "        The number of units for each LSTM cell.\n",
        "    -num_layers: int\n",
        "        The number of LSTM cells we want to use.\n",
        "    Returns \n",
        "    ------------\n",
        "    The initial state, final state, predicted logits and probabilities.\n",
        "    \"\"\"\n",
        "    # one-hot encoding of the inputs\n",
        "    # the resulting shape is `[batch_size, seq_length, vocab_size]`\n",
        "    input_one_hot = tf.one_hot(inputs, vocab_size, axis=-1)\n",
        "    \n",
        "    # create a list of all LSTM cells we want\n",
        "    cells = [tf.contrib.rnn.LSTMCell(hidden_size) for _ in range(num_layers)]\n",
        "    \n",
        "    # we stack the cells together and create one big RNN cell\n",
        "    cell = tf.contrib.rnn.MultiRNNCell(cells)\n",
        "    \n",
        "    # we need to set an initial state for the cells\n",
        "    batch_size = tf.shape(inputs)[0]\n",
        "    initial_state = cell.zero_state(batch_size, dtype=tf.float32)\n",
        "    \n",
        "    # now we are ready to unrol the graph\n",
        "    # outputs has shape [batch_size, seq_length, hidden_size]\n",
        "    outputs, final_state = tf.nn.dynamic_rnn(cell=cell,\n",
        "                                             initial_state=initial_state,\n",
        "                                             inputs=input_one_hot,\n",
        "                                             sequence_length=seq_lengths)\n",
        "\n",
        "\n",
        "    # Mapping the output to the vocabulary space with a dense layer \n",
        "    # FLATTENING: \n",
        "    max_seq_length = tf.shape(inputs)[1] \n",
        "    outputs_flat = tf.reshape(outputs, [-1, hidden_size]) # [batch_size*seq_length, hidden_size]\n",
        "    \n",
        "    # dense layer: hidden_size -> vocab_size \n",
        "    weights = tf.Variable(tf.truncated_normal([hidden_size, vocab_size], stddev=0.1))\n",
        "    bias = tf.Variable(tf.constant(0.1, shape=[vocab_size]))\n",
        "    logits_flat = tf.matmul(outputs_flat, weights) + bias\n",
        "    \n",
        "    # reshape back\n",
        "    logits = tf.reshape(logits_flat, [batch_size, max_seq_length, vocab_size])\n",
        "    \n",
        "    # activate to turn logits into probabilities\n",
        "    probs = tf.nn.softmax(logits)\n",
        "    \n",
        "    # we return the initial and final states because this will be useful later\n",
        "    return initial_state, final_state, logits, probs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MwMZTPmdDuIe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create input placeholders\n",
        "with tf.name_scope(\"input\"):\n",
        "    # shape is `[batch_size, seq_length]`, both are dynamic\n",
        "    text_input = tf.placeholder(tf.int32, [None, None], name='x-input')\n",
        "    # shape of target is same as shape of input\n",
        "    text_target = tf.placeholder(tf.int32, [None, None], name='y-input')\n",
        "    # sequence length placeholder\n",
        "    seq_lengths = tf.placeholder(tf.int32, [None], name='seq-lengths')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rAkUc1D1EOGr",
        "colab_type": "code",
        "outputId": "c2bdd15a-7b73-4a54-e97d-2c9f1320a5ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        }
      },
      "source": [
        "# build the model\n",
        "initial_state, final_state, logits, probs = rnn_lstm(text_input,seq_lengths)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1800C2PWpkj",
        "colab_type": "text"
      },
      "source": [
        "### Defining the loss and the optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-A43JIVKEhND",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# define the loss\n",
        "with tf.name_scope(\"cross-entropy\"):\n",
        "  # The loss operation : \n",
        "    cross_entropy_loss = tf.contrib.seq2seq.sequence_loss( # special loss for sequential output\n",
        "        logits, text_target,  \n",
        "        weights=tf.ones_like(text_input, dtype=tf.float32)) # weights is referring to the predictions weighting \n",
        "    tf.summary.scalar('cross_entropy_loss', cross_entropy_loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KhJp5eCzE-O-",
        "colab_type": "code",
        "outputId": "18bd906c-b63e-4980-a7ef-abbad68e8bca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# check number of trainable parameters\n",
        "def count_trainable_parameters():\n",
        "    \"\"\"Counts the number of trainable parameters in the current default graph.\"\"\"\n",
        "    tot_count = 0\n",
        "    for v in tf.trainable_variables():\n",
        "        v_count = 1\n",
        "        for d in v.get_shape():\n",
        "            v_count *= d.value\n",
        "        tot_count += v_count\n",
        "    return tot_count\n",
        "print(\"Number of trainable parameters: {}\".format(count_trainable_parameters()))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of trainable parameters: 881993\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5AdaiLtIFBpA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create the optimizer\n",
        "global_step = tf.Variable(1, name='global_step', trainable=False)\n",
        "with tf.name_scope(\"train\"):\n",
        "    optim = tf.train.AdamOptimizer(learning_rate)\n",
        "    params = tf.trainable_variables()\n",
        "    gradients = tf.gradients(cross_entropy_loss, params)\n",
        "    # We use gradient clipping to address the exploding gradients issue \n",
        "    clipped_gradients, _ = tf.clip_by_global_norm(gradients, 5)\n",
        "    # Finally the train step operation: \n",
        "    train_step = optim.apply_gradients(zip(clipped_gradients, params), global_step=global_step)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dO2423IXWwd7",
        "colab_type": "text"
      },
      "source": [
        "### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_MljJ2pgFPQM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Training step helper function \n",
        "def do_train_step(num_steps, summary_op, pointer=pointer, x_batches=x_batches, y_batches=y_batches):\n",
        "    \"\"\"Perform as many training steps as specified.\"\"\"\n",
        "    for i in range(num_steps):\n",
        "      step = tf.train.global_step(sess, global_step)\n",
        "\n",
        "      # Get the next batch of data \n",
        "      if pointer >= num_batches:\n",
        "        # Initialise the new epoch \n",
        "        pointer = 0\n",
        "        x_batches, y_batches = reshuffle(x_batches,y_batches)  \n",
        "      x, y, pointer = next_batch(x_batches,y_batches,pointer)\n",
        "      feed_dict = {text_input: x, \n",
        "                    text_target: y, \n",
        "                    seq_lengths: [x.shape[1]]*x.shape[0]}\n",
        "      \n",
        "      # Run the optimization over the data and evaluate the loss\n",
        "      summary, train_loss, _ = sess.run([summary_op, cross_entropy_loss, train_step],\n",
        "                                        feed_dict=feed_dict)\n",
        "      \n",
        "      writer_train.add_summary(summary, step)\n",
        "      if step % print_every_steps == 0:\n",
        "          print('[{}] Cross-Entropy Loss Training [{:.3f}]'.format(step, train_loss)) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-kDDabCXGJIF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create the session\n",
        "sess = tf.InteractiveSession()\n",
        "\n",
        "# Initialize all variables\n",
        "sess.run(tf.global_variables_initializer())\n",
        "\n",
        "summaries_merged = tf.summary.merge_all()\n",
        "writer_train = tf.summary.FileWriter(log_dir + '/train', sess.graph)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cDA_Xsa9GVwi",
        "colab_type": "code",
        "outputId": "0aa27bca-3cc7-4abb-da8a-2f7567ea12fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "pointer = 0\n",
        "do_train_step(5000, summaries_merged, pointer)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1020] Cross-Entropy Loss Training [1.682]\n",
            "[1040] Cross-Entropy Loss Training [1.640]\n",
            "[1060] Cross-Entropy Loss Training [1.699]\n",
            "[1080] Cross-Entropy Loss Training [1.658]\n",
            "[1100] Cross-Entropy Loss Training [1.652]\n",
            "[1120] Cross-Entropy Loss Training [1.664]\n",
            "[1140] Cross-Entropy Loss Training [1.652]\n",
            "[1160] Cross-Entropy Loss Training [1.644]\n",
            "[1180] Cross-Entropy Loss Training [1.598]\n",
            "[1200] Cross-Entropy Loss Training [1.640]\n",
            "[1220] Cross-Entropy Loss Training [1.626]\n",
            "[1240] Cross-Entropy Loss Training [1.605]\n",
            "[1260] Cross-Entropy Loss Training [1.586]\n",
            "[1280] Cross-Entropy Loss Training [1.615]\n",
            "[1300] Cross-Entropy Loss Training [1.596]\n",
            "[1320] Cross-Entropy Loss Training [1.596]\n",
            "[1340] Cross-Entropy Loss Training [1.563]\n",
            "[1360] Cross-Entropy Loss Training [1.608]\n",
            "[1380] Cross-Entropy Loss Training [1.571]\n",
            "[1400] Cross-Entropy Loss Training [1.581]\n",
            "[1420] Cross-Entropy Loss Training [1.542]\n",
            "[1440] Cross-Entropy Loss Training [1.540]\n",
            "[1460] Cross-Entropy Loss Training [1.552]\n",
            "[1480] Cross-Entropy Loss Training [1.540]\n",
            "[1500] Cross-Entropy Loss Training [1.564]\n",
            "[1520] Cross-Entropy Loss Training [1.552]\n",
            "[1540] Cross-Entropy Loss Training [1.560]\n",
            "[1560] Cross-Entropy Loss Training [1.538]\n",
            "[1580] Cross-Entropy Loss Training [1.521]\n",
            "[1600] Cross-Entropy Loss Training [1.504]\n",
            "[1620] Cross-Entropy Loss Training [1.521]\n",
            "[1640] Cross-Entropy Loss Training [1.464]\n",
            "[1660] Cross-Entropy Loss Training [1.523]\n",
            "[1680] Cross-Entropy Loss Training [1.516]\n",
            "[1700] Cross-Entropy Loss Training [1.504]\n",
            "[1720] Cross-Entropy Loss Training [1.509]\n",
            "[1740] Cross-Entropy Loss Training [1.506]\n",
            "[1760] Cross-Entropy Loss Training [1.447]\n",
            "[1780] Cross-Entropy Loss Training [1.470]\n",
            "[1800] Cross-Entropy Loss Training [1.484]\n",
            "[1820] Cross-Entropy Loss Training [1.450]\n",
            "[1840] Cross-Entropy Loss Training [1.483]\n",
            "[1860] Cross-Entropy Loss Training [1.458]\n",
            "[1880] Cross-Entropy Loss Training [1.444]\n",
            "[1900] Cross-Entropy Loss Training [1.443]\n",
            "[1920] Cross-Entropy Loss Training [1.411]\n",
            "[1940] Cross-Entropy Loss Training [1.439]\n",
            "[1960] Cross-Entropy Loss Training [1.404]\n",
            "[1980] Cross-Entropy Loss Training [1.411]\n",
            "[2000] Cross-Entropy Loss Training [1.404]\n",
            "[2020] Cross-Entropy Loss Training [1.420]\n",
            "[2040] Cross-Entropy Loss Training [1.420]\n",
            "[2060] Cross-Entropy Loss Training [1.392]\n",
            "[2080] Cross-Entropy Loss Training [1.395]\n",
            "[2100] Cross-Entropy Loss Training [1.392]\n",
            "[2120] Cross-Entropy Loss Training [1.372]\n",
            "[2140] Cross-Entropy Loss Training [1.352]\n",
            "[2160] Cross-Entropy Loss Training [1.386]\n",
            "[2180] Cross-Entropy Loss Training [1.340]\n",
            "[2200] Cross-Entropy Loss Training [1.358]\n",
            "[2220] Cross-Entropy Loss Training [1.350]\n",
            "[2240] Cross-Entropy Loss Training [1.334]\n",
            "[2260] Cross-Entropy Loss Training [1.335]\n",
            "[2280] Cross-Entropy Loss Training [1.323]\n",
            "[2300] Cross-Entropy Loss Training [1.312]\n",
            "[2320] Cross-Entropy Loss Training [1.303]\n",
            "[2340] Cross-Entropy Loss Training [1.300]\n",
            "[2360] Cross-Entropy Loss Training [1.262]\n",
            "[2380] Cross-Entropy Loss Training [1.310]\n",
            "[2400] Cross-Entropy Loss Training [1.262]\n",
            "[2420] Cross-Entropy Loss Training [1.276]\n",
            "[2440] Cross-Entropy Loss Training [1.279]\n",
            "[2460] Cross-Entropy Loss Training [1.237]\n",
            "[2480] Cross-Entropy Loss Training [1.233]\n",
            "[2500] Cross-Entropy Loss Training [1.277]\n",
            "[2520] Cross-Entropy Loss Training [1.236]\n",
            "[2540] Cross-Entropy Loss Training [1.228]\n",
            "[2560] Cross-Entropy Loss Training [1.217]\n",
            "[2580] Cross-Entropy Loss Training [1.209]\n",
            "[2600] Cross-Entropy Loss Training [1.192]\n",
            "[2620] Cross-Entropy Loss Training [1.235]\n",
            "[2640] Cross-Entropy Loss Training [1.197]\n",
            "[2660] Cross-Entropy Loss Training [1.177]\n",
            "[2680] Cross-Entropy Loss Training [1.189]\n",
            "[2700] Cross-Entropy Loss Training [1.177]\n",
            "[2720] Cross-Entropy Loss Training [1.158]\n",
            "[2740] Cross-Entropy Loss Training [1.160]\n",
            "[2760] Cross-Entropy Loss Training [1.113]\n",
            "[2780] Cross-Entropy Loss Training [1.124]\n",
            "[2800] Cross-Entropy Loss Training [1.114]\n",
            "[2820] Cross-Entropy Loss Training [1.124]\n",
            "[2840] Cross-Entropy Loss Training [1.109]\n",
            "[2860] Cross-Entropy Loss Training [1.098]\n",
            "[2880] Cross-Entropy Loss Training [1.088]\n",
            "[2900] Cross-Entropy Loss Training [1.084]\n",
            "[2920] Cross-Entropy Loss Training [1.080]\n",
            "[2940] Cross-Entropy Loss Training [1.047]\n",
            "[2960] Cross-Entropy Loss Training [1.043]\n",
            "[2980] Cross-Entropy Loss Training [1.037]\n",
            "[3000] Cross-Entropy Loss Training [1.036]\n",
            "[3020] Cross-Entropy Loss Training [1.018]\n",
            "[3040] Cross-Entropy Loss Training [1.044]\n",
            "[3060] Cross-Entropy Loss Training [1.006]\n",
            "[3080] Cross-Entropy Loss Training [0.995]\n",
            "[3100] Cross-Entropy Loss Training [1.009]\n",
            "[3120] Cross-Entropy Loss Training [0.992]\n",
            "[3140] Cross-Entropy Loss Training [0.970]\n",
            "[3160] Cross-Entropy Loss Training [0.990]\n",
            "[3180] Cross-Entropy Loss Training [0.989]\n",
            "[3200] Cross-Entropy Loss Training [0.941]\n",
            "[3220] Cross-Entropy Loss Training [0.977]\n",
            "[3240] Cross-Entropy Loss Training [0.919]\n",
            "[3260] Cross-Entropy Loss Training [0.919]\n",
            "[3280] Cross-Entropy Loss Training [0.918]\n",
            "[3300] Cross-Entropy Loss Training [0.935]\n",
            "[3320] Cross-Entropy Loss Training [0.868]\n",
            "[3340] Cross-Entropy Loss Training [0.909]\n",
            "[3360] Cross-Entropy Loss Training [0.888]\n",
            "[3380] Cross-Entropy Loss Training [0.849]\n",
            "[3400] Cross-Entropy Loss Training [0.882]\n",
            "[3420] Cross-Entropy Loss Training [0.902]\n",
            "[3440] Cross-Entropy Loss Training [0.841]\n",
            "[3460] Cross-Entropy Loss Training [0.885]\n",
            "[3480] Cross-Entropy Loss Training [0.816]\n",
            "[3500] Cross-Entropy Loss Training [0.818]\n",
            "[3520] Cross-Entropy Loss Training [0.841]\n",
            "[3540] Cross-Entropy Loss Training [0.830]\n",
            "[3560] Cross-Entropy Loss Training [0.800]\n",
            "[3580] Cross-Entropy Loss Training [0.828]\n",
            "[3600] Cross-Entropy Loss Training [0.774]\n",
            "[3620] Cross-Entropy Loss Training [0.765]\n",
            "[3640] Cross-Entropy Loss Training [0.793]\n",
            "[3660] Cross-Entropy Loss Training [0.757]\n",
            "[3680] Cross-Entropy Loss Training [0.750]\n",
            "[3700] Cross-Entropy Loss Training [0.758]\n",
            "[3720] Cross-Entropy Loss Training [0.712]\n",
            "[3740] Cross-Entropy Loss Training [0.718]\n",
            "[3760] Cross-Entropy Loss Training [0.744]\n",
            "[3780] Cross-Entropy Loss Training [0.736]\n",
            "[3800] Cross-Entropy Loss Training [0.715]\n",
            "[3820] Cross-Entropy Loss Training [0.727]\n",
            "[3840] Cross-Entropy Loss Training [0.710]\n",
            "[3860] Cross-Entropy Loss Training [0.683]\n",
            "[3880] Cross-Entropy Loss Training [0.698]\n",
            "[3900] Cross-Entropy Loss Training [0.688]\n",
            "[3920] Cross-Entropy Loss Training [0.698]\n",
            "[3940] Cross-Entropy Loss Training [0.699]\n",
            "[3960] Cross-Entropy Loss Training [0.692]\n",
            "[3980] Cross-Entropy Loss Training [0.645]\n",
            "[4000] Cross-Entropy Loss Training [0.665]\n",
            "[4020] Cross-Entropy Loss Training [0.616]\n",
            "[4040] Cross-Entropy Loss Training [0.617]\n",
            "[4060] Cross-Entropy Loss Training [0.645]\n",
            "[4080] Cross-Entropy Loss Training [0.609]\n",
            "[4100] Cross-Entropy Loss Training [0.605]\n",
            "[4120] Cross-Entropy Loss Training [0.591]\n",
            "[4140] Cross-Entropy Loss Training [0.608]\n",
            "[4160] Cross-Entropy Loss Training [0.588]\n",
            "[4180] Cross-Entropy Loss Training [0.605]\n",
            "[4200] Cross-Entropy Loss Training [0.562]\n",
            "[4220] Cross-Entropy Loss Training [0.576]\n",
            "[4240] Cross-Entropy Loss Training [0.566]\n",
            "[4260] Cross-Entropy Loss Training [0.556]\n",
            "[4280] Cross-Entropy Loss Training [0.543]\n",
            "[4300] Cross-Entropy Loss Training [0.556]\n",
            "[4320] Cross-Entropy Loss Training [0.530]\n",
            "[4340] Cross-Entropy Loss Training [0.520]\n",
            "[4360] Cross-Entropy Loss Training [0.558]\n",
            "[4380] Cross-Entropy Loss Training [0.513]\n",
            "[4400] Cross-Entropy Loss Training [0.491]\n",
            "[4420] Cross-Entropy Loss Training [0.500]\n",
            "[4440] Cross-Entropy Loss Training [0.517]\n",
            "[4460] Cross-Entropy Loss Training [0.482]\n",
            "[4480] Cross-Entropy Loss Training [0.520]\n",
            "[4500] Cross-Entropy Loss Training [0.486]\n",
            "[4520] Cross-Entropy Loss Training [0.465]\n",
            "[4540] Cross-Entropy Loss Training [0.507]\n",
            "[4560] Cross-Entropy Loss Training [0.464]\n",
            "[4580] Cross-Entropy Loss Training [0.456]\n",
            "[4600] Cross-Entropy Loss Training [0.460]\n",
            "[4620] Cross-Entropy Loss Training [0.447]\n",
            "[4640] Cross-Entropy Loss Training [0.449]\n",
            "[4660] Cross-Entropy Loss Training [0.452]\n",
            "[4680] Cross-Entropy Loss Training [0.439]\n",
            "[4700] Cross-Entropy Loss Training [0.427]\n",
            "[4720] Cross-Entropy Loss Training [0.454]\n",
            "[4740] Cross-Entropy Loss Training [0.418]\n",
            "[4760] Cross-Entropy Loss Training [0.394]\n",
            "[4780] Cross-Entropy Loss Training [0.418]\n",
            "[4800] Cross-Entropy Loss Training [0.398]\n",
            "[4820] Cross-Entropy Loss Training [0.384]\n",
            "[4840] Cross-Entropy Loss Training [0.422]\n",
            "[4860] Cross-Entropy Loss Training [0.402]\n",
            "[4880] Cross-Entropy Loss Training [0.376]\n",
            "[4900] Cross-Entropy Loss Training [0.385]\n",
            "[4920] Cross-Entropy Loss Training [0.369]\n",
            "[4940] Cross-Entropy Loss Training [0.359]\n",
            "[4960] Cross-Entropy Loss Training [0.370]\n",
            "[4980] Cross-Entropy Loss Training [0.358]\n",
            "[5000] Cross-Entropy Loss Training [0.367]\n",
            "[5020] Cross-Entropy Loss Training [0.379]\n",
            "[5040] Cross-Entropy Loss Training [0.356]\n",
            "[5060] Cross-Entropy Loss Training [0.343]\n",
            "[5080] Cross-Entropy Loss Training [0.371]\n",
            "[5100] Cross-Entropy Loss Training [0.356]\n",
            "[5120] Cross-Entropy Loss Training [0.348]\n",
            "[5140] Cross-Entropy Loss Training [0.359]\n",
            "[5160] Cross-Entropy Loss Training [0.332]\n",
            "[5180] Cross-Entropy Loss Training [0.328]\n",
            "[5200] Cross-Entropy Loss Training [0.350]\n",
            "[5220] Cross-Entropy Loss Training [0.320]\n",
            "[5240] Cross-Entropy Loss Training [0.313]\n",
            "[5260] Cross-Entropy Loss Training [0.320]\n",
            "[5280] Cross-Entropy Loss Training [0.304]\n",
            "[5300] Cross-Entropy Loss Training [0.302]\n",
            "[5320] Cross-Entropy Loss Training [0.320]\n",
            "[5340] Cross-Entropy Loss Training [0.311]\n",
            "[5360] Cross-Entropy Loss Training [0.303]\n",
            "[5380] Cross-Entropy Loss Training [0.291]\n",
            "[5400] Cross-Entropy Loss Training [0.297]\n",
            "[5420] Cross-Entropy Loss Training [0.278]\n",
            "[5440] Cross-Entropy Loss Training [0.277]\n",
            "[5460] Cross-Entropy Loss Training [0.266]\n",
            "[5480] Cross-Entropy Loss Training [0.264]\n",
            "[5500] Cross-Entropy Loss Training [0.268]\n",
            "[5520] Cross-Entropy Loss Training [0.256]\n",
            "[5540] Cross-Entropy Loss Training [0.247]\n",
            "[5560] Cross-Entropy Loss Training [0.264]\n",
            "[5580] Cross-Entropy Loss Training [0.255]\n",
            "[5600] Cross-Entropy Loss Training [0.264]\n",
            "[5620] Cross-Entropy Loss Training [0.269]\n",
            "[5640] Cross-Entropy Loss Training [0.261]\n",
            "[5660] Cross-Entropy Loss Training [0.243]\n",
            "[5680] Cross-Entropy Loss Training [0.257]\n",
            "[5700] Cross-Entropy Loss Training [0.268]\n",
            "[5720] Cross-Entropy Loss Training [0.243]\n",
            "[5740] Cross-Entropy Loss Training [0.260]\n",
            "[5760] Cross-Entropy Loss Training [0.252]\n",
            "[5780] Cross-Entropy Loss Training [0.247]\n",
            "[5800] Cross-Entropy Loss Training [0.258]\n",
            "[5820] Cross-Entropy Loss Training [0.261]\n",
            "[5840] Cross-Entropy Loss Training [0.275]\n",
            "[5860] Cross-Entropy Loss Training [0.279]\n",
            "[5880] Cross-Entropy Loss Training [0.293]\n",
            "[5900] Cross-Entropy Loss Training [0.288]\n",
            "[5920] Cross-Entropy Loss Training [0.274]\n",
            "[5940] Cross-Entropy Loss Training [0.261]\n",
            "[5960] Cross-Entropy Loss Training [0.245]\n",
            "[5980] Cross-Entropy Loss Training [0.256]\n",
            "[6000] Cross-Entropy Loss Training [0.224]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lcc5tNEaHEmx",
        "colab_type": "text"
      },
      "source": [
        "### Saving (and loading) the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4eLWTYalHG3c",
        "colab_type": "code",
        "outputId": "88b7c3bf-4977-40fc-e132-61d59674a69b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "saver = tf.train.Saver(var_list=tf.trainable_variables(), max_to_keep=2)\n",
        "saver.save(sess, os.path.join(log_dir, 'checkpoints', 'model_name'), global_step)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/tmp/tensorflow/divina_rnn/logs/checkpoints/model_name-6002'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xwhmYBYKHNsj",
        "colab_type": "code",
        "outputId": "7e380d66-361c-4330-c9b7-6ed2f9caaf86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# loading the model\n",
        "ckpt_path = tf.train.latest_checkpoint(os.path.join(log_dir, 'checkpoints'))\n",
        "saver.restore(sess, ckpt_path)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/tensorflow/divina_rnn/logs/checkpoints/model_name-6002\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rGssSAPCHaKo",
        "colab_type": "text"
      },
      "source": [
        "## Generating text \n",
        "We will use the model we have just trained to generate new text. \n",
        "<br> *How to do this?*\n",
        "<br> We generate text character-by-character and feed the output of each time step back as input to the model. In other words, we get the output character for a given sequence, append that character to the sequence and repeat the whole process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DzMaR2KtHYAr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sample(prime_text, num_steps, vocab = vocab):\n",
        "    \"\"\"\n",
        "    Sample `num_steps` characters from the model and initialize it with `prime_text`.\n",
        "    Parameters: \n",
        "    ---------------\n",
        "    - prime_text: str.\n",
        "        A string that we want to initialize the RNN with.\n",
        "    - num_steps: int.\n",
        "        Integer specifying how many characters we want to predict after `prime_text`.\n",
        "    Returns:\n",
        "    ---------------\n",
        "        str\n",
        "        The `prime_text` plus prediction.\n",
        "    \"\"\"\n",
        "    \n",
        "    input_prime = [vocab[c] for c in prime_text]\n",
        "    \n",
        "    # Feed the prime sequence into the model. \n",
        "    feed_dict = {text_input: [input_prime],\n",
        "                 seq_lengths: [len(input_prime)]}\n",
        "    state, out_probs = sess.run([final_state, probs], feed_dict=feed_dict)\n",
        "    \n",
        "    next_char_probs = out_probs[0, -1] # the output of the model is a probability distr.\n",
        "    # over all the characters in the vocabulary. We sample from this categorical:\n",
        "    def weighted_pick(p_dist):\n",
        "        cs = np.cumsum(p_dist)\n",
        "        idx = int(np.sum(cs < np.random.rand()))\n",
        "        return idx\n",
        "    next_char = weighted_pick(next_char_probs)\n",
        "    predicted_text = vocab_inv[next_char]\n",
        "    \n",
        "    # now we can sample for `num_steps`\n",
        "    for _ in range(num_steps):\n",
        "        feed_dict = {text_input: [[next_char]],\n",
        "                     seq_lengths: [1],\n",
        "                     initial_state: state}\n",
        "        \n",
        "        state, out_probs = sess.run([final_state, probs], feed_dict=feed_dict)\n",
        "        next_char = weighted_pick(out_probs[0, -1])\n",
        "        predicted_text += vocab_inv[next_char]   \n",
        "    \n",
        "    return prime_text + predicted_text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z9fFMUtTH0nO",
        "colab_type": "code",
        "outputId": "3f04899b-f9a7-4002-fa59-e89060fd6916",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 589
        }
      },
      "source": [
        "print(sample('Il ', 1000))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Il chi move?».\r\n",
            "\r\n",
            "  E io a lui: «Dor, fosse, tra miggia,\r\n",
            "  non li oche giaco, se Cercoresto daicra\r\n",
            "  feruto alcundon par de Fuorchi sonno,\r\n",
            "  me steso le sue cadëa in avro si tecco.\r\n",
            "\r\n",
            "  Tattir si poccia a corpo par savilsa;\r\n",
            "  e chio forse di quanto a terra soscruto,\r\n",
            "  mi fu maca scricciatà il focose\r\n",
            "  al figuboro e deggea feder puoso.\r\n",
            "\r\n",
            "  Per convien la bercine dol posova\r\n",
            "  de lun cheggio, e a le piaggi traddume.\r\n",
            "\r\n",
            "  Per li vede; anoma sonesta,\r\n",
            "  chi conaviesti mitri con umanto,\r\n",
            "  di cotando vi là dener savermi?\r\n",
            "\r\n",
            "  Quand ionime nuver vosta dieto,\r\n",
            "  di tembili a de loro, ov io viva sicisu;\r\n",
            "  perch io non li confeani a De chosì forta.\r\n",
            "\r\n",
            "  Onde vi rispavar per laere partia\r\n",
            "  con grando amenta, quanda gnata fava:\r\n",
            "  volsi per adirgorer, Tu per vinte;\r\n",
            "  ma quell altra è malerento mungo,\r\n",
            "  piangendo così da ciascuna folse,\r\n",
            "  e volse più chïoltro e disse anosa;\r\n",
            "  nellon che tra la vertù prea suppe,\r\n",
            "\r\n",
            "  e gridò edi che piangea con cigimutor;\r\n",
            "  a quei dise:: «Per\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EAzEI1vyH4VJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# cleanup\n",
        "sess.close()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}