CV19Net.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "colab": {
      "name": "CV19Net.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ammarisme/covid-19/blob/master/CV19Net.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C83XgaWprQnn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "!pip install torch-geometric \\\n",
        "  torch-sparse==latest+cu101 \\\n",
        "  torch-scatter==latest+cu101 \\\n",
        "  torch-cluster==latest+cu101 \\\n",
        "  -f https://pytorch-geometric.com/whl/torch-1.5.0.html"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gRUdaEpW2GsF",
        "colab_type": "text"
      },
      "source": [
        "!pip install torch-geometric \\\n",
        "  torch-sparse==latest+cu101 \\\n",
        "  torch-scatter==latest+cu101 \\\n",
        "  torch-cluster==latest+cu101 \\\n",
        "  -f https://pytorch-geometric.com/whl/torch-1.5.0.html\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "anKz91FjrQn3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import unicode_literals, print_function, division\n",
        "from io import open\n",
        "import unicodedata\n",
        "import string\n",
        "import re\n",
        "import random\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "from datetime import datetime\n",
        "\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print('running on '+ str(device))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bFsXKb1CzwSY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "import os.path\n",
        "from os import path\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "PATH = '/content/drive/My Drive/covid'\n",
        "MODEL_DIR =  PATH +'/models'\n",
        "INPUT_DIR = PATH + '/input'\n",
        "VALIDATION_DATA_DIR = '/input/validation'\n",
        "TRAINING_DATA_DIR = '/input/training'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lX8YjbZkzjeA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torch_geometric.data import Data, DataLoader, InMemoryDataset\n",
        "\n",
        "class CovidDataSet(InMemoryDataset):\n",
        "    def __init__(self, root, input_sequence, output_sequence, transform=None, pre_transform=None):\n",
        "        super(CovidDataSet, self).__init__(root, transform, pre_transform)\n",
        "        self.data, self.slices = torch.load(self.processed_paths[0])\n",
        "    \n",
        "    @property\n",
        "    def raw_file_names(self):\n",
        "      return ['']#left this to be empty to avoid accidential processing\n",
        "\n",
        "    @property\n",
        "    def processed_file_names(self):\n",
        "      print(self.root)\n",
        "      return ['processed.dt']\n",
        "\n",
        "    def download(self):\n",
        "      pass\n",
        "    \n",
        "    def process(self):\n",
        "      pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bba9dPo0rQop",
        "colab_type": "text"
      },
      "source": [
        "The Seq2Seq Model\n",
        "=================\n",
        "\n",
        "A Recurrent Neural Network, or RNN, is a network that operates on a\n",
        "sequence and uses its own output as input for subsequent steps.\n",
        "\n",
        "A `Sequence to Sequence network <https://arxiv.org/abs/1409.3215>`__, or\n",
        "seq2seq network, or `Encoder Decoder\n",
        "network <https://arxiv.org/pdf/1406.1078v3.pdf>`__, is a model\n",
        "consisting of two RNNs called the encoder and decoder. The encoder reads\n",
        "an input sequence and outputs a single vector, and the decoder reads\n",
        "that vector to produce an output sequence.\n",
        "\n",
        ".. figure:: /_static/img/seq-seq-images/seq2seq.png\n",
        "   :alt:\n",
        "\n",
        "Unlike sequence prediction with a single RNN, where every input\n",
        "corresponds to an output, the seq2seq model frees us from sequence\n",
        "length and order, which makes it ideal for translation between two\n",
        "languages.\n",
        "\n",
        "Consider the sentence \"Je ne suis pas le chat noir\" → \"I am not the\n",
        "black cat\". Most of the words in the input sentence have a direct\n",
        "translation in the output sentence, but are in slightly different\n",
        "orders, e.g. \"chat noir\" and \"black cat\". Because of the \"ne/pas\"\n",
        "construction there is also one more word in the input sentence. It would\n",
        "be difficult to produce a correct translation directly from the sequence\n",
        "of input words.\n",
        "\n",
        "With a seq2seq model the encoder creates a single vector which, in the\n",
        "ideal case, encodes the \"meaning\" of the input sequence into a single\n",
        "vector — a single point in some N dimensional space of sentences.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50K2H_e0rQor",
        "colab_type": "text"
      },
      "source": [
        "The Encoder\n",
        "-----------\n",
        "\n",
        "The encoder of a seq2seq network is a RNN that outputs some value for\n",
        "every word from the input sentence. For every input word the encoder\n",
        "outputs a vector and a hidden state, and uses the hidden state for the\n",
        "next input word.\n",
        "\n",
        ".. figure:: /_static/img/seq-seq-images/encoder-network.png\n",
        "   :alt:\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Rv5DjHyrQor",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class EncoderRNN(nn.Module):\n",
        "    def __init__(self, input_size, parameter_sizes, repeats ,output_size):\n",
        "        super(EncoderRNN, self).__init__()\n",
        "        self.input_size = input_size\n",
        "        self.repeater_input_size = parameter_sizes[0]\n",
        "        self.hidden_size = parameter_sizes[1]\n",
        "        self.repeats = repeats\n",
        "        self.output_size = output_size\n",
        "\n",
        "        self.fc1 = nn.Linear(input_size, self.repeater_input_size)\n",
        "\n",
        "        self.layers = dict()\n",
        "\n",
        "        \n",
        "        k = 0\n",
        "        for i in range(repeats):\n",
        "          i = i+k\n",
        "          self.layers['fc_'+str(i)] = nn.Linear(self.repeater_input_size, self.hidden_size)\n",
        "          self.layers['gru_'+str(i+1)] = nn.GRU(self.hidden_size, self.hidden_size)\n",
        "          self.layers['fc_'+str(i+2)] = nn.Linear(self.hidden_size, self.repeater_input_size)\n",
        "          k+=2\n",
        "\n",
        "        self.module_list = nn.ModuleDict(self.layers)\n",
        "\n",
        "        self.fc2 = nn.Linear(self.repeater_input_size, output_size)\n",
        "        \n",
        "    def forward(self, input, hidden):\n",
        "      output = self.fc1(input)\n",
        "\n",
        "      k = 0\n",
        "      for i in range(self.repeats):\n",
        "        i = i+k\n",
        "        output = self.layers['fc_'+str(i)](output)\n",
        "        output, hidden[i-k] = self.layers['gru_'+str(i+1)](output, hidden[i-k])#should be different. check the nlp page\n",
        "        output = self.layers['fc_'+str(i+2)](output)\n",
        "        k +=2\n",
        "\n",
        "      output = self.fc2(output)\n",
        "\n",
        "      return output, hidden\n",
        "\n",
        "    def initHidden(self, batch_size):\n",
        "        return torch.zeros(1, batch_size, self.hidden_size, device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8kKB2k0rQov",
        "colab_type": "text"
      },
      "source": [
        "The Decoder\n",
        "-----------\n",
        "\n",
        "The decoder is another RNN that takes the encoder output vector(s) and\n",
        "outputs a sequence of words to create the translation.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FsqjeIHErQow",
        "colab_type": "text"
      },
      "source": [
        "Simple Decoder\n",
        "^^^^^^^^^^^^^^\n",
        "\n",
        "In the simplest seq2seq decoder we use only last output of the encoder.\n",
        "This last output is sometimes called the *context vector* as it encodes\n",
        "context from the entire sequence. This context vector is used as the\n",
        "initial hidden state of the decoder.\n",
        "\n",
        "At every step of decoding, the decoder is given an input token and\n",
        "hidden state. The initial input token is the start-of-string ``<SOS>``\n",
        "token, and the first hidden state is the context vector (the encoder's\n",
        "last hidden state).\n",
        "\n",
        ".. figure:: /_static/img/seq-seq-images/decoder-network.png\n",
        "   :alt:\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dqCaWmkMrQpM",
        "colab_type": "text"
      },
      "source": [
        "Attention Decoder\n",
        "^^^^^^^^^^^^^^^^^\n",
        "\n",
        "If only the context vector is passed betweeen the encoder and decoder,\n",
        "that single vector carries the burden of encoding the entire sentence.\n",
        "\n",
        "Attention allows the decoder network to \"focus\" on a different part of\n",
        "the encoder's outputs for every step of the decoder's own outputs. First\n",
        "we calculate a set of *attention weights*. These will be multiplied by\n",
        "the encoder output vectors to create a weighted combination. The result\n",
        "(called ``attn_applied`` in the code) should contain information about\n",
        "that specific part of the input sequence, and thus help the decoder\n",
        "choose the right output words.\n",
        "\n",
        ".. figure:: https://i.imgur.com/1152PYf.png\n",
        "   :alt:\n",
        "\n",
        "Calculating the attention weights is done with another feed-forward\n",
        "layer ``attn``, using the decoder's input and hidden state as inputs.\n",
        "Because there are sentences of all sizes in the training data, to\n",
        "actually create and train this layer we have to choose a maximum\n",
        "sentence length (input length, for encoder outputs) that it can apply\n",
        "to. Sentences of the maximum length will use all the attention weights,\n",
        "while shorter sentences will only use the first few.\n",
        "\n",
        ".. figure:: /_static/img/seq-seq-images/attention-decoder-network.png\n",
        "   :alt:\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Jdq_KmirQpW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "MAX_LENGTH = 20\n",
        "class AttnDecoderRNN(nn.Module):\n",
        "  def __init__(self, input_size, parameter_sizes,repeats ,output_size, dropout_p=0.1, max_length=MAX_LENGTH):\n",
        "    super(AttnDecoderRNN, self).__init__()\n",
        "    \n",
        "    self.input_size = input_size\n",
        "    self.repeater_input_size = parameter_sizes[0]\n",
        "    self.hidden_size = parameter_sizes[1]\n",
        "    self.repeats = repeats\n",
        "    self.output_size = output_size\n",
        "    self.dropout_p = dropout_p\n",
        "    self.max_length = max_length\n",
        "\n",
        "    self.fc1 = nn.Linear(self.input_size, self.hidden_size)\n",
        "    self.attn = nn.Linear(self.hidden_size * 2, self.max_length)\n",
        "    self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)\n",
        "    self.dropout = nn.Dropout(self.dropout_p)\n",
        "\n",
        "    self.layers = dict()\n",
        "    k = 0\n",
        "    for i in range(repeats):\n",
        "      i = i+k\n",
        "      self.layers['fc_'+str(i)] = nn.Linear(self.repeater_input_size, self.hidden_size)\n",
        "      self.layers['gru_'+str(i+1)] = nn.GRU(self.hidden_size, self.hidden_size)\n",
        "      self.layers['fc_'+str(i+2)] = nn.Linear(self.hidden_size, self.repeater_input_size)\n",
        "      k+=2\n",
        "\n",
        "    self.module_list = nn.ModuleDict(self.layers)\n",
        "\n",
        "    self.out =  nn.Linear(self.repeater_input_size, output_size)\n",
        "    #nn.Linear(self.hidden_size, self.output_size)\n",
        "\n",
        "  def forward(self, input,  hidden, encoder_outputs):\n",
        "    output = self.fc1(input)\n",
        "    embedded = self.dropout(output)\n",
        "    \n",
        "    hidden[0] = hidden[0].squeeze()\n",
        "    attn_weights = F.softmax(\n",
        "        self.attn(torch.cat((embedded, hidden[0]), 1)), dim=1) #embedded[0]\n",
        "    attn_applied = torch.bmm(attn_weights.unsqueeze(0),\n",
        "                              encoder_outputs.unsqueeze(0))\n",
        "    \n",
        "    output = torch.cat((embedded, attn_applied[0]), 1)\n",
        "    output = self.attn_combine(output).unsqueeze(0)\n",
        "    hidden[0] = hidden[0].unsqueeze(0)\n",
        "\n",
        "    k = 0\n",
        "    for i in range(self.repeats):\n",
        "      i = i+k\n",
        "      output = self.layers['fc_'+str(i)](output)\n",
        "      output, hidden[i-k] = self.layers['gru_'+str(i+1)](output, hidden[i-k])#should be different. check the nlp page\n",
        "      output = self.layers['fc_'+str(i+2)](output)\n",
        "      k +=2\n",
        "\n",
        "    output = self.out(output[0])\n",
        "    output = F.log_softmax(output, dim=1)\n",
        "\n",
        "    return output, hidden, attn_weights\n",
        "\n",
        "  def initHidden(self, batch_size):\n",
        "    return torch.zeros(1, batch_size, self.hidden_size, device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OXw5KYrFsE3N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_n_params(model):\n",
        "    pp=0\n",
        "    for p in list(model.parameters()):\n",
        "        nn=1\n",
        "        for s in list(p.size()):\n",
        "            nn = nn*s\n",
        "        pp += nn\n",
        "    return pp\n",
        "  \n",
        "\n",
        "def save_model_parameters(model_parameters_dict, models = None, fileName=None):\n",
        "  if fileName == None:\n",
        "    fileName = 'model_parameters_'+str(np.round(np.random.randn(1000,200)[0][0]*1000))\n",
        "\n",
        "  model_summary = pd.DataFrame(columns =[])\n",
        "  for i in sorted(model_parameters_dict.keys()):\n",
        "    param_size = 0\n",
        "    if models != None:\n",
        "      param_size = get_n_params(models[i][0])\n",
        "      param_size += get_n_params(models[i][1])\n",
        "    row = pd.Series({\n",
        "                        'model_id':i,\n",
        "                        'encoder repeats' : model_parameters_dict[i][0]['repeats'],\n",
        "                        'decoder repeats' : model_parameters_dict[i][1]['repeats'],\n",
        "                        'parameters' :param_size\n",
        "    })\n",
        "    row_df = pd.DataFrame([row], index = [i])\n",
        "    model_summary = pd.concat([model_summary, row_df])\n",
        "  model_summary.to_csv(fileName)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mMsOerOWgJ5d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Generate the list of repeat count parameters list\n",
        "def generate_repeat_counts(alphas , training_dataset_length, enc_repeated_nn_parameters = 132096, dec_repeated_nn_parameters = 132096):\n",
        "  repeats = []\n",
        "  for alpha in alphas:\n",
        "    preferred_parameters  = (1/alpha)* (training_dataset_length / (feature_len+yhat_size))\n",
        "    for i in range(1,100):\n",
        "      dec_repeats = i\n",
        "      enc_repeats = int((preferred_parameters - \n",
        "                  (preferred_parameters%enc_repeated_nn_parameters) -\n",
        "                    (dec_repeated_nn_parameters*dec_repeats)\n",
        "                  )/ enc_repeated_nn_parameters)\n",
        "      model_id = 0\n",
        "      for j in range(1, enc_repeats+1):\n",
        "        repeats.append((i, j))\n",
        "  return repeats"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IHFFiWrWhsQx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def generate_model_parameters(repeat_counts, feature_len, output_len):\n",
        "  model_dict = dict()\n",
        "  model_id = 0\n",
        "  for dec_repeat, enc_repeat in repeat_counts:\n",
        "    model_dict[model_id] = [{\n",
        "         'input_size': feature_len,\n",
        "         'parameter_sizes': [128, 128] ,\n",
        "         'repeats' : enc_repeat ,\n",
        "         'output_size' : 128\n",
        "        },\n",
        "        {\n",
        "            'input_size' : 1, \n",
        "            'parameter_sizes' : [128, 128],\n",
        "            'repeats' : dec_repeat ,\n",
        "            'output_size': output_len,\n",
        "            'dropout_p' : 0.1, \n",
        "        }]\n",
        "    model_id+=1\n",
        "  return model_dict\n",
        "\n",
        "#generate_model_parameters(generate_repeat_counts())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZBmfYEtiWem",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion,batch_size, max_length=MAX_LENGTH):\n",
        "  input_tensor = input_tensor.type(torch.FloatTensor)\n",
        "  target_tensor= target_tensor.type(torch.FloatTensor)\n",
        "\n",
        "  encoder_hidden = []\n",
        "  for i in range(encoder.repeats):\n",
        "    encoder_hidden.append(encoder.initHidden(batch_size))\n",
        "\n",
        "  encoder_optimizer.zero_grad()\n",
        "  decoder_optimizer.zero_grad()\n",
        "  \n",
        "  input_length = input_tensor.size(1)\n",
        "  target_length = target_tensor.size(1)\n",
        "\n",
        "  encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)\n",
        "\n",
        "  loss = 0\n",
        "  \n",
        "  for ei in range(input_sequence_len):\n",
        "    input_tensor_seq = input_tensor.view(input_sequence_len,batch_size, feature_len)[ei]\n",
        "    input_tensor_seq = input_tensor_seq.view(1, batch_size, feature_len).to(device)\n",
        "\n",
        "    encoder_output, encoder_hidden = encoder(\n",
        "        input_tensor_seq, encoder_hidden)\n",
        "    encoder_outputs[ei] = encoder_output[0, 0]\n",
        "\n",
        "  decoder_input = torch.ones((batch_size,1), device=device).type(torch.FloatTensor)\n",
        "\n",
        "  decoder_hidden = encoder_hidden  \n",
        "  for i in range(decoder.repeats):\n",
        "    decoder_hidden.append(decoder.initHidden(batch_size))\n",
        "\n",
        "  for di in range(target_length):\n",
        "    target_tensor_seq = target_tensor.view(output_sequence_len, batch_size, yhat_size)\n",
        "    decoder_input = decoder_input.type(torch.FloatTensor).to(device)\n",
        "    decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "        decoder_input, decoder_hidden, encoder_outputs)\n",
        "    topv, topi = decoder_output.topk(1)\n",
        "    decoder_input = topi.detach()  # detach from history as input\n",
        "    loss += criterion(decoder_output, target_tensor_seq[di].to(device))\n",
        "\n",
        "  loss.backward()\n",
        "  encoder_optimizer.step()\n",
        "  decoder_optimizer.step()\n",
        "\n",
        "  return loss.item() / target_length"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Bb9yefOKZrz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def evaluate(input_tensor, target_tensor, encoder, decoder, criterion,batch_size, max_length=MAX_LENGTH):\n",
        "  with torch.no_grad():\n",
        "    input_tensor = input_tensor.type(torch.FloatTensor)\n",
        "    target_tensor= target_tensor.type(torch.FloatTensor)\n",
        "\n",
        "    encoder_hidden = []\n",
        "    for i in range(encoder.repeats):\n",
        "      encoder_hidden.append(encoder.initHidden(batch_size))\n",
        "\n",
        "    input_length = input_tensor.size(1)\n",
        "    target_length = target_tensor.size(1)\n",
        "\n",
        "    encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)\n",
        "\n",
        "    validation_loss = 0\n",
        "    \n",
        "    for ei in range(input_sequence_len):\n",
        "      input_tensor_seq = input_tensor.view(input_sequence_len,batch_size, feature_len)[ei]\n",
        "      input_tensor_seq = input_tensor_seq.view(1, batch_size, feature_len).to(device)\n",
        "\n",
        "      encoder_output, encoder_hidden = encoder(\n",
        "          input_tensor_seq, encoder_hidden)\n",
        "      encoder_outputs[ei] = encoder_output[0, 0]\n",
        "\n",
        "    decoder_input = torch.ones((batch_size,1), device=device).type(torch.FloatTensor)\n",
        "\n",
        "    decoder_hidden = encoder_hidden  \n",
        "    for i in range(decoder.repeats):\n",
        "      decoder_hidden.append(decoder.initHidden(batch_size))\n",
        "\n",
        "\n",
        "    for di in range(target_length):\n",
        "      target_tensor_seq = target_tensor.view(output_sequence_len, batch_size, yhat_size)\n",
        "      decoder_input = decoder_input.type(torch.FloatTensor).to(device)\n",
        "      decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "          decoder_input, decoder_hidden, encoder_outputs)\n",
        "      topv, topi = decoder_output.topk(1)\n",
        "      decoder_input = topi.detach()  # detach from history as input\n",
        "      validation_loss += criterion(decoder_output, target_tensor_seq[di].to(device))\n",
        "\n",
        "  return validation_loss.item() / target_length"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Ni9AejIrQpi",
        "colab_type": "text"
      },
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>There are other forms of attention that work around the length\n",
        "  limitation by using a relative position approach. Read about \"local\n",
        "  attention\" in `Effective Approaches to Attention-based Neural Machine\n",
        "  Translation <https://arxiv.org/abs/1508.04025>`__.</p></div>\n",
        "\n",
        "Training\n",
        "========\n",
        "\n",
        "Preparing Training Data\n",
        "-----------------------\n",
        "\n",
        "To train, for each pair we will need an input tensor (indexes of the\n",
        "words in the input sentence) and target tensor (indexes of the words in\n",
        "the target sentence). While creating these vectors we will append the\n",
        "EOS token to both sequences.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dgj6u5-ZrQpo",
        "colab_type": "text"
      },
      "source": [
        "Training the Model\n",
        "------------------\n",
        "\n",
        "To train we run the input sentence through the encoder, and keep track\n",
        "of every output and the latest hidden state. Then the decoder is given\n",
        "the ``<SOS>`` token as its first input, and the last hidden state of the\n",
        "encoder as its first hidden state.\n",
        "\n",
        "\"Teacher forcing\" is the concept of using the real target outputs as\n",
        "each next input, instead of using the decoder's guess as the next input.\n",
        "Using teacher forcing causes it to converge faster but `when the trained\n",
        "network is exploited, it may exhibit\n",
        "instability <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.378.4095&rep=rep1&type=pdf>`__.\n",
        "\n",
        "You can observe outputs of teacher-forced networks that read with\n",
        "coherent grammar but wander far from the correct translation -\n",
        "intuitively it has learned to represent the output grammar and can \"pick\n",
        "up\" the meaning once the teacher tells it the first few words, but it\n",
        "has not properly learned how to create the sentence from the translation\n",
        "in the first place.\n",
        "\n",
        "Because of the freedom PyTorch's autograd gives us, we can randomly\n",
        "choose to use teacher forcing or not with a simple if statement. Turn\n",
        "``teacher_forcing_ratio`` up to use more of it.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mG_XR_uufapa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_batch(dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion):\n",
        "  loss = 0\n",
        "  for batch in dataloader:\n",
        "    batch_size = int(len(batch.x)/input_sequence_len)\n",
        "    loss += (train(batch.x.view(batch_size, input_sequence_len , feature_len), batch.y, encoder,\n",
        "                     decoder, encoder_optimizer, decoder_optimizer, criterion, batch_size))/ batch_size\n",
        "  return loss / len(dataloader)\n",
        "\n",
        "def batch_evaluate(dataloader, encoder, decoder, criterion, output_sequence_len):\n",
        "  loss = 0\n",
        "  for batch in dataloader:\n",
        "    batch_size = int(len(batch.x)/input_sequence_len)\n",
        "    loss += (evaluate(batch.x.view(batch_size, input_sequence_len , feature_len), batch.y, encoder,\n",
        "                     decoder, criterion, batch_size))/ batch_size\n",
        "  return loss / len(dataloader)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "658hctperQpu",
        "colab_type": "text"
      },
      "source": [
        "This is a helper function to print time elapsed and estimated time\n",
        "remaining given the current time and progress %.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWbvNuNfrQp1",
        "colab_type": "text"
      },
      "source": [
        "The whole training process looks like this:\n",
        "\n",
        "-  Start a timer\n",
        "-  Initialize optimizers and criterion\n",
        "-  Create set of training pairs\n",
        "-  Start empty losses array for plotting\n",
        "\n",
        "Then we call ``train`` many times and occasionally print the progress (%\n",
        "of examples, time so far, estimated time) and average loss.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOclGefDrQp2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def trainValidationIters(train_dataloader,validation_dataloader, encoder, decoder,  encoder_optimizer, \n",
        "                         decoder_optimizer, criterion ,n_iters, output_sequence_len,input_sequence_len,\n",
        "                         current_model_dir, model_id,session_id,print_every=1000, plot_every=100,save_every = 100):\n",
        "  #yhat_size = 2\n",
        "  #input_sequence_len = 10\n",
        "\n",
        "  start = time.time()\n",
        "  plot_losses = []\n",
        "  \n",
        "  # Reset every print_every # Reset every plot_every\n",
        "  print_training_loss_total = 0\n",
        "  plot_training_loss_total = 0\n",
        "  print_validation_loss_total = 0\n",
        "  plot_validation_loss_total = 0\n",
        "  save_validation_loss_total = 0\n",
        "  save_training_loss_total = 0\n",
        "      \n",
        "  losses = []\n",
        "\n",
        "  for iter in range(1, n_iters + 1):\n",
        "      training_loss = train_batch(train_dataloader, encoder,\n",
        "                    decoder, encoder_optimizer, decoder_optimizer, criterion )\n",
        "\n",
        "      validation_loss = batch_evaluate(validation_dataloader, encoder,\n",
        "                    decoder, criterion,output_sequence_len )\n",
        "      \n",
        "      print_training_loss_total += training_loss\n",
        "      plot_training_loss_total += training_loss\n",
        "      print_validation_loss_total += validation_loss\n",
        "      plot_validation_loss_total += validation_loss\n",
        "      \n",
        "      save_training_loss_total += training_loss\n",
        "      save_validation_loss_total += validation_loss\n",
        "      \n",
        "      if iter % print_every == 0:\n",
        "          print_training_loss_avg = print_training_loss_total / print_every\n",
        "          print_training_loss_total = 0\n",
        "          print_validation_loss_avg = print_validation_loss_total / print_every\n",
        "          print_validation_loss_total = 0\n",
        "          print('%s (%d %d%%) training:%.4f validation:%.4f' % (timeSince(start, iter / n_iters),\n",
        "                                        iter, iter / n_iters * 100, print_training_loss_avg,print_validation_loss_avg, ))\n",
        "          print('train/validation difference : ',print_validation_loss_avg-print_training_loss_avg)\n",
        "\n",
        "      #if iter % plot_every == 0:\n",
        "      #    plot_training_loss_avg = plot_training_loss_total / plot_every\n",
        "      #    plot_validation_loss_avg = plot_validation_loss_total / plot_every\n",
        "      #    plot_training_loss.append(plot_training_loss_avg)\n",
        "      #    plot_validation_loss.append(plot_validation_loss_avg)\n",
        "      #    showPlot(plot_training_losses,plot_validation_loss)\n",
        "      #    plot_training_loss_total = 0\n",
        "      #    plot_validation_loss_total\n",
        "        \n",
        "      if iter % save_every == 0:\n",
        "        save_training_loss_avg = save_training_loss_total / save_every\n",
        "        save_validation_loss_avg = plot_validation_loss_total / save_every\n",
        "        \n",
        "        #'arch': args.arch,\n",
        "        save_checkpoint({\n",
        "            'epoch': iter + 1,\n",
        "            'encoder_state_dict': encoder.state_dict(),\n",
        "            'decoder_state_dict': decoder.state_dict(),\n",
        "            'training_loss': save_training_loss_avg,\n",
        "            'validation_loss': save_validation_loss_avg,\n",
        "            'encoder_optimizer' : encoder_optimizer.state_dict(),\n",
        "            'decoder_optimizer' : decoder_optimizer.state_dict(),\n",
        "        }, False, filename=current_model_dir+'/'+str(iter)+'_'+str(session_id)+'.pt')\n",
        "      losses.append([training_loss, validation_loss])\n",
        "\n",
        "  save_dict({\n",
        "      'losses' : losses\n",
        "  }, filename=current_model_dir+'/losses_'+session_id)\n",
        "\n",
        "  return losses"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h3sGikLKufvN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "input_sequence_len = 10\n",
        "output_sequence_len = 20\n",
        "training_batch_size = 8192 # for production 131072\n",
        "validation_batch_size = 2048\n",
        "yhat_size = 2\n",
        "feature_len = 8\n",
        "\n",
        "training_dataset = CovidDataSet(PATH+TRAINING_DATA_DIR, input_sequence_len, output_sequence_len)\n",
        "training_dataset = training_dataset.shuffle()\n",
        "training_dataset = training_dataset[:131072]\n",
        "training_dataloader = DataLoader(training_dataset,batch_size=training_batch_size)\n",
        "\n",
        "validation_dataset = CovidDataSet(PATH+VALIDATION_DATA_DIR, input_sequence_len, output_sequence_len)\n",
        "validation_dataset = validation_dataset.shuffle()\n",
        "validation_dataloader = DataLoader(validation_dataset,batch_size=validation_batch_size)\n",
        "\n",
        "print('validation :', len(validation_dataloader), 'training : ', len(training_dataloader))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OU2b74ly9SFs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5Ts09ldvpdE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "session_id =str(1001)+ '_'+str(datetime.now(tz=None).microsecond)\n",
        "repeat_counts = generate_repeat_counts([1,2,3], 3*len(training_dataset))\n",
        "\n",
        "model_configurations = generate_model_parameters(repeat_counts, 8, 2)\n",
        "save_model_parameters(model_configurations, fileName= PATH + '/model_configurations_'+session_id+'.csv')\n",
        "models = dict()\n",
        "len(model_configurations)\n",
        "len(repeat_counts)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W34bK4bcqCMi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Run all the candidate models\n",
        "learning_rate = 0.1\n",
        "for i in model_configurations:\n",
        "  if i < 10:\n",
        "    continue\n",
        "  print('training :',i)\n",
        "  #instantiate the models\n",
        "  current_model_dir =  MODEL_DIR +'/'+str(i)\n",
        "  if os.path.exists(current_model_dir) == False:\n",
        "    os.mkdir(current_model_dir)\n",
        "  \n",
        "  encoder = EncoderRNN(**model_configurations[i][0]).to(device)\n",
        "  decoder = AttnDecoderRNN(**model_configurations[i][1]).to(device)\n",
        "  encoder_optimizer= optim.Adam(encoder.parameters(), lr=learning_rate)\n",
        "  decoder_optimizer= optim.Adam(decoder.parameters(), lr=learning_rate)\n",
        "  criterion = nn.MSELoss()\n",
        "\n",
        "  losses = trainValidationIters(training_dataloader,validation_dataloader,\n",
        "                    encoder, decoder,\n",
        "                    encoder_optimizer, decoder_optimizer,\n",
        "                    criterion, n_iters = 100,output_sequence_len=output_sequence_len,\n",
        "                    input_sequence_len=input_sequence_len,\n",
        "                    current_model_dir=current_model_dir,  model_id = i,session_id = session_id,\n",
        "                     print_every=10, save_every = 10)\n",
        "  models[i] = {\n",
        "      'encoder' : encoder.state_dict(),\n",
        "      'decoder' : decoder.state_dict(),\n",
        "      'encoder_optimizer' : encoder_optimizer.state_dict(),\n",
        "      'decoder_optimizer' : decoder_optimizer.state_dict(),\n",
        "      'losses' : losses\n",
        "  }\n",
        "  print('Model ',str(i), ': trained and evaluated')\n",
        "\n",
        "save_dict(models, MODEL_DIR+'/models_'+session_id+'.pt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4s3JT5u06qL",
        "colab_type": "text"
      },
      "source": [
        "training : 3\n",
        "3m 33s (- 32m 1s) (10 10%) training:173.4740 validation:2190.2324\n",
        "train/validation difference :  2016.7583596612699\n",
        "7m 6s (- 28m 26s) (20 20%) training:170.9638 validation:2177.7728\n",
        "train/validation difference :  2006.8089751082975\n",
        "10m 41s (- 24m 57s) (30 30%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090948874492\n",
        "14m 15s (- 21m 23s) (40 40%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091080481547\n",
        "17m 49s (- 17m 49s) (50 50%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090967471142\n",
        "21m 23s (- 14m 15s) (60 60%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091005141275\n",
        "24m 57s (- 10m 41s) (70 70%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090756232277\n",
        "28m 32s (- 7m 8s) (80 80%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090975577372\n",
        "32m 6s (- 3m 34s) (90 90%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090999896065\n",
        "35m 40s (- 0m 0s) (100 100%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091089064615\n",
        "Model  3 : trained and evaluated\n",
        "training : 4\n",
        "3m 45s (- 33m 48s) (10 10%) training:173.5196 validation:2202.3471\n",
        "train/validation difference :  2028.8275353356416\n",
        "7m 30s (- 30m 2s) (20 20%) training:170.9638 validation:2177.7730\n",
        "train/validation difference :  2006.809155901524\n",
        "11m 14s (- 26m 14s) (30 30%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.809106283857\n",
        "14m 59s (- 22m 29s) (40 40%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091004187602\n",
        "18m 45s (- 18m 45s) (50 50%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091198737163\n",
        "22m 30s (- 15m 0s) (60 60%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091135317818\n",
        "26m 15s (- 11m 15s) (70 70%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090731507014\n",
        "30m 1s (- 7m 30s) (80 80%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091096240596\n",
        "33m 46s (- 3m 45s) (90 90%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.809140234663\n",
        "37m 31s (- 0m 0s) (100 100%) training:170.9638 validation:2177.7730\n",
        "train/validation difference :  2006.8091474825876\n",
        "Model  4 : trained and evaluated\n",
        "training : 5\n",
        "3m 56s (- 35m 29s) (10 10%) training:172.8378 validation:2179.0659\n",
        "train/validation difference :  2006.228061978189\n",
        "7m 52s (- 31m 31s) (20 20%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.809092407896\n",
        "11m 49s (- 27m 35s) (30 30%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090995127695\n",
        "15m 47s (- 23m 41s) (40 40%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8090824466838\n",
        "19m 45s (- 19m 45s) (50 50%) training:170.9638 validation:2177.7729\n",
        "train/validation difference :  2006.8091159159676"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Db2gIwi407ni",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I7fGhjUMfOdI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j0uf8RzfJZ7M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'):\n",
        "    torch.save(state, filename)\n",
        "    if is_best:\n",
        "        shutil.copyfile(filename, 'model_best.pth.tar')\n",
        "\n",
        "def save_dict(dictionary, filename):\n",
        "  np.save(filename, dictionary) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D4-qjbMUrQp7",
        "colab_type": "text"
      },
      "source": [
        "Plotting results\n",
        "----------------\n",
        "\n",
        "Plotting is done with matplotlib, using the array of loss values\n",
        "``plot_losses`` saved while training.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S4rOMRwdrQp8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.switch_backend('agg')\n",
        "import matplotlib.ticker as ticker\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "def showPlot(points, points2):\n",
        "    plt.figure()\n",
        "    fig, ax = plt.subplots()\n",
        "    # this locator puts ticks at regular intervals\n",
        "    loc = ticker.MultipleLocator(base=0.2)\n",
        "    ax.yaxis.set_major_locator(loc)\n",
        "    plt.xlabel('Iteration')\n",
        "    plt.ylabel('Loss')\n",
        "    plt.plot(points)\n",
        "    plt.plot(points2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YfBXtfHGrQqB",
        "colab_type": "text"
      },
      "source": [
        "Evaluation\n",
        "==========\n",
        "\n",
        "Evaluation is mostly the same as training, but there are no targets so\n",
        "we simply feed the decoder's predictions back to itself for each step.\n",
        "Every time it predicts a word we add it to the output string, and if it\n",
        "predicts the EOS token we stop there. We also store the decoder's\n",
        "attention outputs for display later.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PMW-TZtQKVw4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "import math\n",
        "\n",
        "def asMinutes(s):\n",
        "    m = math.floor(s / 60)\n",
        "    s -= m * 60\n",
        "    return '%dm %ds' % (m, s)\n",
        "\n",
        "def timeSince(since, percent):\n",
        "    now = time.time()\n",
        "    s = now - since\n",
        "    es = s / (percent)\n",
        "    rs = es - s\n",
        "    return '%s (- %s)' % (asMinutes(s), asMinutes(rs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TFR4mWBnrQqI",
        "colab_type": "text"
      },
      "source": [
        "We can evaluate random sentences from the training set and print out the\n",
        "input, target, and output to make some subjective quality judgements:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2x8hi9MrQqJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def evaluateRandomly(encoder, decoder, n=10):\n",
        "    for i in range(n):\n",
        "        pair = random.choice(pairs)\n",
        "        print('>', pair[0])\n",
        "        print('=', pair[1])\n",
        "        output_words, attentions = evaluate(encoder, decoder, pair[0])\n",
        "        output_sentence = ' '.join(output_words)\n",
        "        print('<', output_sentence)\n",
        "        print('')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCMyyjhgrQqU",
        "colab_type": "text"
      },
      "source": [
        "Training and Evaluating\n",
        "=======================\n",
        "\n",
        "With all these helper functions in place (it looks like extra work, but\n",
        "it makes it easier to run multiple experiments) we can actually\n",
        "initialize a network and start training.\n",
        "\n",
        "Remember that the input sentences were heavily filtered. For this small\n",
        "dataset we can use relatively small networks of 256 hidden nodes and a\n",
        "single GRU layer. After about 40 minutes on a MacBook CPU we'll get some\n",
        "reasonable results.\n",
        "\n",
        ".. Note::\n",
        "   If you run this notebook you can train, interrupt the kernel,\n",
        "   evaluate, and continue training later. Comment out the lines where the\n",
        "   encoder and decoder are initialized and run ``trainIters`` again.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_3B2yQGDrQqZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "evaluateRandomly(encoder1, attn_decoder1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjGtOB9GrQqd",
        "colab_type": "text"
      },
      "source": [
        "Visualizing Attention\n",
        "---------------------\n",
        "\n",
        "A useful property of the attention mechanism is its highly interpretable\n",
        "outputs. Because it is used to weight specific encoder outputs of the\n",
        "input sequence, we can imagine looking where the network is focused most\n",
        "at each time step.\n",
        "\n",
        "You could simply run ``plt.matshow(attentions)`` to see attention output\n",
        "displayed as a matrix, with the columns being input steps and rows being\n",
        "output steps:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oWf3VERirQqe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\"output_words, attentions = evaluate(\n",
        "    encoder1, attn_decoder1, \"je suis trop froid .\")\n",
        "plt.matshow(attentions.numpy())\n",
        "\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9v3Slwa9rQqi",
        "colab_type": "text"
      },
      "source": [
        "For a better viewing experience we will do the extra work of adding axes\n",
        "and labels:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RzokwvozrQqj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def showAttention(input_sentence, output_words, attentions):\n",
        "    # Set up figure with colorbar\n",
        "    fig = plt.figure()\n",
        "    ax = fig.add_subplot(111)\n",
        "    cax = ax.matshow(attentions.numpy(), cmap='bone')\n",
        "    fig.colorbar(cax)\n",
        "\n",
        "    # Set up axes\n",
        "    ax.set_xticklabels([''] + input_sentence.split(' ') +\n",
        "                       ['<EOS>'], rotation=90)\n",
        "    ax.set_yticklabels([''] + output_words)\n",
        "\n",
        "    # Show label at every tick\n",
        "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "def evaluateAndShowAttention(input_sentence):\n",
        "    output_words, attentions = evaluate(\n",
        "        encoder1, attn_decoder1, input_sentence)\n",
        "    print('input =', input_sentence)\n",
        "    print('output =', ' '.join(output_words))\n",
        "    showAttention(input_sentence, output_words, attentions)\n",
        "\n",
        "\n",
        "evaluateAndShowAttention(\"elle a cinq ans de moins que moi .\")\n",
        "\n",
        "evaluateAndShowAttention(\"elle est trop petit .\")\n",
        "\n",
        "evaluateAndShowAttention(\"je ne crains pas de mourir .\")\n",
        "\n",
        "evaluateAndShowAttention(\"c est un jeune directeur plein de talent .\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7qvAWCvTrQqq",
        "colab_type": "text"
      },
      "source": [
        "Exercises\n",
        "=========\n",
        "\n",
        "-  Try with a different dataset\n",
        "\n",
        "   -  Another language pair\n",
        "   -  Human → Machine (e.g. IOT commands)\n",
        "   -  Chat → Response\n",
        "   -  Question → Answer\n",
        "\n",
        "-  Replace the embeddings with pre-trained word embeddings such as word2vec or\n",
        "   GloVe\n",
        "-  Try with more layers, more hidden units, and more sentences. Compare\n",
        "   the training time and results.\n",
        "-  If you use a translation file where pairs have two of the same phrase\n",
        "   (``I am test \\t I am test``), you can use this as an autoencoder. Try\n",
        "   this:\n",
        "\n",
        "   -  Train as an autoencoder\n",
        "   -  Save only the Encoder network\n",
        "   -  Train a new Decoder for translation from there\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LiuSerpCa7au",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}