visual_turing_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Turing Test - Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[ Scalable Learning and Perception Group](http://scalable.mpi-inf.mpg.de), authored by [Dr. Mario Fritz](http://scalable.mpi-inf.mpg.de/publications/) and [Mateusz Malinowski](https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/people/mateusz-malinowski/). We are also curious about your opinion on the tutorial; every feedback is welcomed, please write to mmalinow@mpi-inf.mpg.de."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is based on our ICCV'15 paper [\"Ask Your Neurons: A Neural-based Approach to Answering Questions about Images\"](https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf), and, more broadly, our project on [Visual Turing Test](https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/research/vision-and-language/visual-turing-challenge/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since visual features are large, you should download the features separately, and put them to data/visual_features/visual_features or data/vqa/visual_features directory.\n",
    " * daquar (it is recommended to download all them)\n",
    "  * [residual net [23 MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/daquar/fb_resnet.zip)\n",
    "   -- place under: data/daquar/visual_features/fb_resnet/blobs.*.npy\n",
    "  * [googlenet [17 MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/daquar/googlenet.zip)\n",
    "   -- place under: data/daquar/visual_features/googlenet/blobs.*.npy\n",
    "  * [NYU-Depth images [430 MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16language_vision_tutorial/nyu_depth-images.zip)\n",
    "   -- place under: data/daquar/images/*.png\n",
    " * vqa\n",
    "  * [residual_net train [1.2GB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/vqa/train2014/fb_resnet.zip) \n",
    "   -- recommended, place under: data/vqa/visual_features/train2014/fb_resnet/blobs.*.npy\n",
    "  * [residual_net val [573MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/vqa/val2014/fb_resnet.zip) \n",
    "   -- recommended, place under: data/vqa/visual_features/val2014/fb_resnet/blobs.*.npy\n",
    "  * [vgg_net train [2.2GB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/vqa/train2014/vgg_net.zip)\n",
    "   -- a few different visual features, place under: data/vqa/visual_features/train2014/vgg_net/blobs.*.npy\n",
    "  * [vgg_net val [1.3GB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16visual_features/vqa/val2014/vgg_net.zip)\n",
    "   -- a few different visual features, place under: data/vqa/visual_features/val2014/vgg_net/blobs.*.npy\n",
    " * vqa - question answer pairs\n",
    "  * [Questions [134MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16language_vision_tutorial/Questions.zip) -- place under: data/vqa/Questions/\n",
    "  * [Annotations [56MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16language_vision_tutorial/Annotations.zip) -- place under: data/vqa/Annotations\n",
    " * nltk data [recommended]\n",
    "  * [nltk_data [20MB]](http://datasets.d2.mpi-inf.mpg.de/mateusz16language_vision_tutorial/nltk_data.zip)\n",
    "   -- place under: data/nltk_data\n",
    " * tutorial\n",
    "  * this version of tutorial can be found [here](http://datasets.d2.mpi-inf.mpg.de/mateusz16language_vision_tutorial/visual_turing_test-tutorial.zip)\n",
    "  * [github version](https://github.com/mateuszmalinowski/visual_turing_test-tutorial) -- this version will be further updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use this tutorial or its parts or Kraino in your project, please consider citing at least our 'Ask Your Neurons' paper (you can find the bibtex below).\n",
    "\n",
    "Bibtex:\n",
    "\n",
    "```\n",
    "@article{malinowski2016ask\n",
    "  title={Ask your neurons: A Deep Learning Approach to Visual Question Answering},\n",
    "  author={Malinowski, Mateusz and Rohrbach, Marcus and Fritz, Mario},\n",
    "  booktitle={arXiv preprint arXiv:1605.02697},\n",
    "  year={2016}\n",
    "}\n",
    "or\n",
    "@inproceedings{malinowski2015ask,\n",
    "  title={Ask your neurons: A neural-based approach to answering questions about images},\n",
    "  author={Malinowski, Mateusz and Rohrbach, Marcus and Fritz, Mario},\n",
    "  booktitle={Proceedings of the IEEE International Conference on Computer Vision},\n",
    "  pages={1--9},\n",
    "  year={2015}\n",
    "}\n",
    "or\n",
    "@article{malinowski2016tutorial,\n",
    "  title={Tutorial on Answering Questions about Images with Deep Learning},\n",
    "  author={Malinowski, Mateusz and Fritz, Mario},\n",
    "  journal={arXiv preprint arXiv:1610.01076},\n",
    "  year={2016}\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting the tutorial make sure that you have the following hierarchy of the folders:\n",
    "boring_function.py\n",
    "neural_solver.py\n",
    "visual_turing_test.ipynb\n",
    "data/*\n",
    "kraino/*\n",
    "local/*\n",
    "fig/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use this notebook?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before focusing on the actual task, let's briefly see what we can do in the Jupyter Notebook. We will also introduce notation that we use in this tutorial.\n",
    "\n",
    "Shortcuts:\n",
    " * Shift + Enter - runs the cell, and step inside the next cell\n",
    " * Ctrl + Enter - runs the cell (stay in the same cell)\n",
    " * Esc + x - deletes the cell (be careful)\n",
    " * Esc + b - creates a cell bellow the current cell\n",
    "\n",
    "\n",
    "The following represents an exercise that doesn't need programming. Its role is to practice some newly introduced concepts.\n",
    "```\n",
    "Exercise\n",
    "```\n",
    "\n",
    "The following is a python script. \n",
    "```python\n",
    "print(\"Hello world\")\n",
    "\n",
    "# Comment\n",
    "# Now we write a loop printing numbers from 0 to 9\n",
    "for k in xrange(10):\n",
    "    # remember that the python's syntax is driven by indentation\n",
    "    print k\n",
    "```\n",
    "\n",
    "Some exercises need some programming, or at least executing the code.\n",
    "However, in this tutorial, we try to keep the programming part rather minimal, and focus on the Visual Turing Test.\n",
    "The following cell is a small programming exercise. You can edit it by double clicking the cell, and execute it by running the cell (Shift + Enter). We use #TODO: to give some hints or more detailed explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def print_n_numbers(n):\n",
    "    #TODO: write a loop that prints numbers from 0 to n (excluding n)\n",
    "    for i in xrange(n):\n",
    "        print(i)\n",
    "\n",
    "# now we execute the function\n",
    "print_n_numbers(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function below print each element in the list in a new line. We will use this function later, so please run the interpreter over the following cell (Shift+Enter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def print_list(ll):\n",
    "    # Prints the list\n",
    "    print('\\n'.join(ll))\n",
    "    \n",
    "print_list(['Visual Turing Test', 'Summer School', 'Dr. Mario Fritz', 'Mateusz Malinowski'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook can also interface with the command line. Try the following line (again Shift+Enter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's execute python's 'boring_function' with an argument. It prints the available GPU, the argument, as well as versions of Theano, and Keras (we will talk about both frameworks later in the tutorial). Since the boring_function imports Theano, its execution may take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't import dot_parser, loading of dot files will not be possible.\n",
      "Using gpu device 0: Tesla K40m (CNMeM is disabled, cuDNN 5004)\n",
      "hello world\n",
      "0.9.0dev0.dev-beefa9396a0e089f6b35b43f71c05557b1083515\n",
      "0.3.3\n"
     ]
    }
   ],
   "source": [
    "! python boring_function.py 'hello world'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the content of the function, run the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import sys\r\n",
      "import theano\r\n",
      "import keras\r\n",
      "\r\n",
      "if __name__ == '__main__':\r\n",
      "    print sys.argv[1]\r\n",
      "    print theano.__version__\r\n",
      "    print keras.__version__\r\n"
     ]
    }
   ],
   "source": [
    "! tail boring_function.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command below checks the available GPU machines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jan 30 00:54:14 2017       \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |\r\n",
      "|-------------------------------+----------------------+----------------------+\r\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
      "|===============================+======================+======================|\r\n",
      "|   0  Tesla K40m          Off  | 0000:05:00.0     Off |                    0 |\r\n",
      "| N/A   39C    P0    61W / 235W |      0MiB / 11439MiB |      0%      Default |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "|   1  Tesla K40m          Off  | 0000:42:00.0     Off |                    0 |\r\n",
      "| N/A   29C    P0    61W / 235W |      0MiB / 11439MiB |     63%      Default |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "                                                                               \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| Processes:                                                       GPU Memory |\r\n",
      "|  GPU       PID  Type  Process name                               Usage      |\r\n",
      "|=============================================================================|\r\n",
      "|  No running processes found                                                 |\r\n",
      "+-----------------------------------------------------------------------------+\r\n"
     ]
    }
   ],
   "source": [
    "! nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During this tutorial, we will look at very recent research thread that interlinks language and vision together -- a Visual Turing Test -- that is about answering on natural language questions about images by machines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![challenges](fig/challenges.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Roadmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Datasets](#Datasets)\n",
    "1. [Textual Features](#Textual-Features)\n",
    "1. [Language Only](#Language-Only)\n",
    "1. [Evaluation Measures](#Evaluation-Measures)\n",
    "1. [New Predictions](#New-Predictions)\n",
    "1. [Visual Features](#Visual-Features)\n",
    "1. [Vision+Language](#Vision+Language)\n",
    "1. [New Predictions with Vision+Language](#New Predictions-with-Vision+Language)\n",
    "1. [VQA](#VQA)\n",
    "1. [Kraino](#Kraino)\n",
    "1. [Further Experiments](#Further-Experiments)\n",
    "1. [New Research Opportunities](#New-Research-Opportunities)\n",
    "1. [External Links](#External-Links)\n",
    "1. [Logs](#Logs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will get familiar with both datasets, accuracy measures, and features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DAQUAR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first look into the folder *data/daquar* to make the problem a bit more tangible for us. Here, we have training and test data in *qa.894.raw.train.format_triple* and *qa.894.raw.test.format_triple*.\n",
    "\n",
    "```\n",
    "Execute the cell below to see how input data look like.\n",
    "Please use Shift+Enter on the cell below.\n",
    "Make sure you understand the format.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "what is on the right side of the black telephone and on the left side of the red chair ?\r\n",
      "desk\r\n",
      "image3\r\n",
      "what is in front of the white door on the left side of the desk ?\r\n",
      "telephone\r\n",
      "image3\r\n",
      "what is on the desk ?\r\n",
      "book, scissor, papers, tape_dispenser\r\n",
      "image3\r\n",
      "what is the largest brown objects ?\r\n",
      "carton\r\n",
      "image3\r\n",
      "what color is the chair in front of the white wall ?\r\n",
      "red\r\n",
      "image3\r\n"
     ]
    }
   ],
   "source": [
    "! head -15 data/daquar/qa.894.raw.train.format_triple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Let's have a look at the figure in [Introduction->Challenge](#Challenge). The figure lists images with associated question-answer pairs. It also comments on challenges associated with every question-answer-triplet. We see that to answer properly on the questions, the answerer needs to understand the scene visually, understand the question, but also, arguably, has to resort to common sense knowledge, or even know the preferences of the person asking a question ('What is behind the table?' - what 'behind' means?).\n",
    "\n",
    "```\n",
    "Can you see anything particularly interesting in the first column of the figure in [Introduction->Challenge]?\n",
    "Think about a spatial relationship between an observer, object of interest, and the world. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'perception': <function kraino.utils.data_provider.<lambda>>,\n",
       " 'save_predictions': <function kraino.utils.data_provider.daquar_save_results>,\n",
       " 'text': <function kraino.utils.data_provider.daquar_qa_triples>}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#TODO: Execute the following procedure (Shift+Enter)\n",
    "from kraino.utils import data_provider\n",
    "\n",
    "dp = data_provider.select['daquar-triples']\n",
    "dp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above returns a dictionary of three representations of the DAQUAR dataset. For now, we will look only into the 'text' representation. dp['text'] returns a function from dataset split into the dataset's textual representation. It will be more clear after executing the following instruction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['answer_words_delimiter',\n",
       " 'end_of_answer',\n",
       " 'img_name',\n",
       " 'y',\n",
       " 'x',\n",
       " 'end_of_question',\n",
       " 'img_ind',\n",
       " 'question_id']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the keys of the representation of DAQUAR train\n",
    "train_text_representation = dp['text'](train_or_test='train')\n",
    "train_text_representation.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This representation specifies how questions are ended ('?'), answers are ended ('.'), answer words are delimited (DAQUAR sometimes has a set of answer words as an answer, for instance 'knife, fork' may be a valid answer), but most important, it has questions (key 'x'), answers (key 'y'), and names of the corresponding images (key 'img_name')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Questions:\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'print_list' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-7-a25d18255c1b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mn_elements\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'== Questions:'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mprint_list\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtrain_text_representation\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'x'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mn_elements\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;32mprint\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;32mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'== Answers:'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'print_list' is not defined"
     ]
    }
   ],
   "source": [
    "# let's check some entries of the text's representation\n",
    "n_elements = 10\n",
    "print('== Questions:')\n",
    "print_list(train_text_representation['x'][:n_elements])\n",
    "print\n",
    "print('== Answers:')\n",
    "print_list(train_text_representation['y'][:n_elements])\n",
    "print\n",
    "print('== Image Names:')\n",
    "print_list(train_text_representation['img_name'][:n_elements])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Summary__\n",
    "\n",
    "DAQUAR consists of question, answer, image triplets. Pairs question, answer for different folds are accessible from\n",
    "```python\n",
    "data_provider.select['text']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Textual Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. We have a text. But unfortunately neural networks expect numerical input, so we cannot really work with the raw text. We need to transform an raw input into some numerical value or a vector of values. One particularly successful representation is called one-hot vector and it is a binary vector with exactly one non-zero entry. This entry points to the corresponding word in the vocabulary. See the illustration below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![one hot](fig/one_hot.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "* How does a vector computed after we sum up one-hot representations of 'what is behind the table' (see the illustration above) look like?\n",
    "* What if we sum up 'What table is behind the table'? Can you interpret the resulting vector?\n",
    "* Can you guess why it's nice to work with one-hot vector representation of the text?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see from the illustrative example above, we first need to build a suitable vocabulary from our raw textual training data, and next transform them into one-hot representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 9847), ('?', 6795), ('what', 5847), ('is', 5368), ('on', 2909)]\n",
      "[('all', 1), ('surrounded', 1), ('four', 1), ('displaying', 1), ('children', 1)]\n"
     ]
    }
   ],
   "source": [
    "from toolz import frequencies\n",
    "train_raw_x = train_text_representation['x']\n",
    "# we start from building the frequencies table\n",
    "wordcount_x = frequencies(' '.join(train_raw_x).split(' '))\n",
    "# print the most and least frequent words\n",
    "n_show = 5\n",
    "print(sorted(wordcount_x.items(), key=lambda x: x[1], reverse=True)[:n_show])\n",
    "print(sorted(wordcount_x.items(), key=lambda x: x[1])[:n_show])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3': 506,\n",
       " u'<eoa>': 2,\n",
       " u'<eoq>': 3,\n",
       " u'<pad>': 0,\n",
       " u'<unk>': 1,\n",
       " '?': 52,\n",
       " 'a': 205,\n",
       " 'above': 80,\n",
       " 'ac': 817,\n",
       " 'across': 512,\n",
       " 'against': 533,\n",
       " 'air': 588,\n",
       " 'airconditionerg': 790,\n",
       " 'alarm': 424,\n",
       " 'all': 4,\n",
       " 'along': 92,\n",
       " 'amidst': 783,\n",
       " 'and': 382,\n",
       " 'any': 390,\n",
       " 'apart': 688,\n",
       " 'apples': 513,\n",
       " 'appliance': 510,\n",
       " 'appliances': 112,\n",
       " 'are': 488,\n",
       " 'arm': 494,\n",
       " 'armchair': 546,\n",
       " 'armchairs': 320,\n",
       " 'around': 765,\n",
       " 'at': 828,\n",
       " 'attached': 807,\n",
       " 'audio': 647,\n",
       " 'available': 514,\n",
       " 'away': 501,\n",
       " 'baby': 137,\n",
       " 'back': 708,\n",
       " 'backpack': 541,\n",
       " 'bag': 471,\n",
       " 'bags': 607,\n",
       " 'ball': 623,\n",
       " 'bananas': 651,\n",
       " 'bars': 477,\n",
       " 'base': 303,\n",
       " 'basin': 566,\n",
       " 'basins': 545,\n",
       " 'basket': 215,\n",
       " 'baskets': 110,\n",
       " 'bath': 257,\n",
       " 'bathroom': 558,\n",
       " 'bathtub': 592,\n",
       " 'bean': 463,\n",
       " 'bear': 462,\n",
       " 'bed': 374,\n",
       " 'bedding': 720,\n",
       " 'beds': 873,\n",
       " 'bedside': 51,\n",
       " 'been': 563,\n",
       " 'before': 234,\n",
       " 'behind': 660,\n",
       " 'beige': 757,\n",
       " 'below': 811,\n",
       " 'belt': 556,\n",
       " 'bench': 667,\n",
       " 'beneath': 269,\n",
       " 'benhind': 95,\n",
       " 'between': 509,\n",
       " 'bicycle': 335,\n",
       " 'big': 679,\n",
       " 'bigger': 712,\n",
       " 'biggest': 787,\n",
       " 'billing': 559,\n",
       " 'bills': 270,\n",
       " 'bin': 595,\n",
       " 'bins': 219,\n",
       " 'black': 124,\n",
       " 'blackboard': 411,\n",
       " 'blackboards': 265,\n",
       " 'blades': 404,\n",
       " 'blanket': 33,\n",
       " 'blind': 323,\n",
       " 'blinds': 350,\n",
       " 'blocks': 282,\n",
       " 'blue': 70,\n",
       " 'board': 569,\n",
       " 'boards': 746,\n",
       " 'body': 86,\n",
       " 'book': 677,\n",
       " 'bookcase': 625,\n",
       " 'books': 812,\n",
       " 'bookshelf': 797,\n",
       " 'border': 389,\n",
       " 'both': 571,\n",
       " 'bottle': 645,\n",
       " 'bottles': 200,\n",
       " 'bottom': 718,\n",
       " 'bowl': 614,\n",
       " 'bowls': 278,\n",
       " 'box': 96,\n",
       " 'boxes': 640,\n",
       " 'bricks': 260,\n",
       " 'briefcase': 285,\n",
       " 'bright': 438,\n",
       " 'broom': 238,\n",
       " 'brown': 28,\n",
       " 'brush': 772,\n",
       " 'brushes': 361,\n",
       " 'bucket': 480,\n",
       " 'built': 395,\n",
       " 'bunk': 177,\n",
       " 'burner': 502,\n",
       " 'burners': 436,\n",
       " 'business': 733,\n",
       " 'but': 792,\n",
       " 'by': 741,\n",
       " 'cabin': 378,\n",
       " 'cabinet': 793,\n",
       " 'cabinets': 835,\n",
       " 'cable': 43,\n",
       " 'cables': 832,\n",
       " 'cage': 258,\n",
       " 'cair': 310,\n",
       " 'calendar': 492,\n",
       " 'camel': 652,\n",
       " 'cameras': 293,\n",
       " 'can': 152,\n",
       " 'candelabra': 148,\n",
       " 'candle': 711,\n",
       " 'candles': 208,\n",
       " 'candlestick': 150,\n",
       " 'candlesticks': 261,\n",
       " 'cap': 149,\n",
       " 'card': 308,\n",
       " 'cardboard': 93,\n",
       " 'carpet': 82,\n",
       " 'carpets': 107,\n",
       " 'carrying': 136,\n",
       " 'carton': 582,\n",
       " 'cartons': 551,\n",
       " 'case': 505,\n",
       " 'cash': 587,\n",
       " 'casing': 576,\n",
       " 'cd': 549,\n",
       " 'ceiling': 356,\n",
       " 'center': 368,\n",
       " 'centerpiece': 865,\n",
       " 'central': 745,\n",
       " 'centre': 721,\n",
       " 'certificates': 81,\n",
       " 'chair': 15,\n",
       " 'chairs': 696,\n",
       " 'chandelier': 439,\n",
       " 'chart': 413,\n",
       " 'chiar': 312,\n",
       " 'children': 17,\n",
       " 'circle': 454,\n",
       " 'circles': 405,\n",
       " 'circular': 189,\n",
       " 'cistern': 386,\n",
       " 'cleaner': 383,\n",
       " 'clock': 73,\n",
       " 'clockes': 83,\n",
       " 'close': 491,\n",
       " 'closed': 508,\n",
       " 'closer': 495,\n",
       " 'closest': 719,\n",
       " 'closet': 498,\n",
       " 'cloth': 794,\n",
       " 'clothes': 56,\n",
       " 'clothing': 176,\n",
       " 'cobalt': 10,\n",
       " 'coffee': 525,\n",
       " 'coil': 419,\n",
       " 'color': 687,\n",
       " 'colored': 98,\n",
       " 'colorful': 459,\n",
       " 'colors': 47,\n",
       " 'colour': 638,\n",
       " 'coloured': 202,\n",
       " 'colours': 504,\n",
       " 'column': 749,\n",
       " 'comforter': 334,\n",
       " 'comforters': 246,\n",
       " 'computer': 487,\n",
       " 'computers': 313,\n",
       " 'conditioner': 603,\n",
       " 'conditioning': 304,\n",
       " 'conference': 557,\n",
       " 'connected': 416,\n",
       " 'connecting': 726,\n",
       " 'container': 287,\n",
       " 'containers': 568,\n",
       " 'containing': 608,\n",
       " 'content': 295,\n",
       " 'control': 108,\n",
       " 'controls': 450,\n",
       " 'cooker': 806,\n",
       " 'cooking': 75,\n",
       " 'cooler': 830,\n",
       " 'corck': 372,\n",
       " 'cord': 145,\n",
       " 'cork': 736,\n",
       " 'corner': 67,\n",
       " 'corridor': 249,\n",
       " 'cot': 530,\n",
       " 'couch': 661,\n",
       " 'couches': 42,\n",
       " 'counter': 210,\n",
       " 'counters': 85,\n",
       " 'countries': 468,\n",
       " 'course': 226,\n",
       " 'cover': 426,\n",
       " 'covered': 244,\n",
       " 'covering': 263,\n",
       " 'covers': 823,\n",
       " 'cpu': 872,\n",
       " 'cream': 620,\n",
       " 'crib': 328,\n",
       " 'cup': 532,\n",
       " 'cupbaord': 739,\n",
       " 'cupboard': 521,\n",
       " 'cupboards': 18,\n",
       " 'cups': 336,\n",
       " 'curtain': 565,\n",
       " 'curtains': 355,\n",
       " 'cushion': 465,\n",
       " 'cushions': 358,\n",
       " 'cutting': 254,\n",
       " 'cylinder': 109,\n",
       " 'dark': 681,\n",
       " 'decoration': 599,\n",
       " 'decorations': 351,\n",
       " 'decorative': 156,\n",
       " 'deoderant': 777,\n",
       " 'design': 65,\n",
       " 'desk': 655,\n",
       " 'desks': 306,\n",
       " 'details': 846,\n",
       " 'detergent': 723,\n",
       " 'diagonally': 232,\n",
       " 'dining': 665,\n",
       " 'dinner': 756,\n",
       " 'dinning': 666,\n",
       " 'dish': 14,\n",
       " 'dishes': 396,\n",
       " 'dishwasher': 309,\n",
       " 'dispenser': 381,\n",
       " 'dispensers': 101,\n",
       " 'display': 815,\n",
       " 'displayed': 224,\n",
       " 'displaying': 16,\n",
       " 'divider': 188,\n",
       " 'do': 517,\n",
       " 'does': 725,\n",
       " 'dog': 427,\n",
       " 'doll': 782,\n",
       " 'dolls': 247,\n",
       " 'domes': 314,\n",
       " 'dominant': 458,\n",
       " 'dominantly': 102,\n",
       " 'dominating': 331,\n",
       " 'door': 130,\n",
       " 'doors': 678,\n",
       " 'doorway': 534,\n",
       " 'down': 768,\n",
       " 'draw': 853,\n",
       " 'drawer': 166,\n",
       " 'drawers': 104,\n",
       " 'drawing': 648,\n",
       " 'draws': 497,\n",
       " 'dress': 170,\n",
       " 'dresser': 851,\n",
       " 'dressers': 876,\n",
       " 'drier': 58,\n",
       " 'dryer': 683,\n",
       " 'drying': 178,\n",
       " 'dustbin': 479,\n",
       " 'dvd': 869,\n",
       " 'each': 267,\n",
       " 'edge': 643,\n",
       " 'edible': 144,\n",
       " 'eggplant': 624,\n",
       " 'eimage999': 266,\n",
       " 'either': 729,\n",
       " 'electric': 408,\n",
       " 'electrical': 845,\n",
       " 'electronic': 337,\n",
       " 'end': 174,\n",
       " 'equipment': 400,\n",
       " 'equipments': 868,\n",
       " 'every': 168,\n",
       " 'exercise': 473,\n",
       " 'exhauster': 833,\n",
       " 'ext': 430,\n",
       " 'extinguisher': 173,\n",
       " 'facing': 695,\n",
       " 'fan': 36,\n",
       " 'fans': 538,\n",
       " 'far': 32,\n",
       " 'faucet': 132,\n",
       " 'fax': 29,\n",
       " 'few': 121,\n",
       " 'file': 829,\n",
       " 'files': 789,\n",
       " 'filing': 432,\n",
       " 'filled': 84,\n",
       " 'fire': 572,\n",
       " 'fireplace': 38,\n",
       " 'first': 322,\n",
       " 'fishes': 860,\n",
       " 'fitted': 433,\n",
       " 'fixed': 474,\n",
       " 'flags': 644,\n",
       " 'flat': 129,\n",
       " 'flipboard': 415,\n",
       " 'floor': 639,\n",
       " 'flour': 251,\n",
       " 'flower': 286,\n",
       " 'flowers': 366,\n",
       " 'flush': 822,\n",
       " 'folder': 535,\n",
       " 'food': 250,\n",
       " 'foosball': 633,\n",
       " 'foot': 423,\n",
       " 'for': 717,\n",
       " 'fornt': 50,\n",
       " 'found': 268,\n",
       " 'four': 11,\n",
       " 'fourth': 186,\n",
       " 'frame': 481,\n",
       " 'framed': 767,\n",
       " 'frames': 773,\n",
       " 'frequent': 321,\n",
       " 'from': 117,\n",
       " 'front': 255,\n",
       " 'fruit': 630,\n",
       " 'fruits': 520,\n",
       " 'furnitures': 540,\n",
       " 'game': 682,\n",
       " 'garbage': 764,\n",
       " 'gate': 187,\n",
       " 'gery': 125,\n",
       " 'girls': 345,\n",
       " 'glass': 135,\n",
       " 'globe': 277,\n",
       " 'gloves': 472,\n",
       " 'glue': 444,\n",
       " 'glued': 54,\n",
       " 'gray': 467,\n",
       " 'green': 212,\n",
       " 'grinder': 609,\n",
       " 'ground': 443,\n",
       " 'guitar': 528,\n",
       " 'gym': 360,\n",
       " 'hair': 709,\n",
       " 'hand': 629,\n",
       " 'handle': 748,\n",
       " 'handles': 122,\n",
       " 'hang': 628,\n",
       " 'hanged': 280,\n",
       " 'hanger': 273,\n",
       " 'hangers': 297,\n",
       " 'hanging': 13,\n",
       " 'has': 673,\n",
       " 'hat': 585,\n",
       " 'have': 387,\n",
       " 'heat': 795,\n",
       " 'heater': 760,\n",
       " 'held': 827,\n",
       " 'help': 703,\n",
       " 'her': 779,\n",
       " 'high': 165,\n",
       " 'hight': 196,\n",
       " 'hing': 671,\n",
       " 'hold': 139,\n",
       " 'holder': 448,\n",
       " 'holders': 539,\n",
       " 'holds': 222,\n",
       " 'hole': 138,\n",
       " 'hood': 339,\n",
       " 'hooks': 7,\n",
       " 'horse': 831,\n",
       " 'hot': 240,\n",
       " 'house': 276,\n",
       " 'how': 182,\n",
       " 'hung': 700,\n",
       " 'id': 605,\n",
       " 'if': 606,\n",
       " 'image10': 394,\n",
       " 'image1007': 292,\n",
       " 'image1008': 864,\n",
       " 'image1035': 375,\n",
       " 'image1043': 694,\n",
       " 'image1045': 185,\n",
       " 'image1072': 672,\n",
       " 'image108': 420,\n",
       " 'image114': 659,\n",
       " 'image116': 657,\n",
       " 'image12': 392,\n",
       " 'image135': 275,\n",
       " 'image139': 274,\n",
       " 'image379': 675,\n",
       " 'image486': 175,\n",
       " 'image654': 550,\n",
       " 'image888': 264,\n",
       " 'image912': 114,\n",
       " 'image929': 445,\n",
       " 'image95': 133,\n",
       " 'image982': 537,\n",
       " 'immediately': 685,\n",
       " 'in': 602,\n",
       " 'inbetweeng': 674,\n",
       " 'inner': 785,\n",
       " 'inside': 805,\n",
       " 'instrument': 669,\n",
       " 'iron': 379,\n",
       " 'ironing': 162,\n",
       " 'is': 597,\n",
       " 'island': 750,\n",
       " 'it': 598,\n",
       " 'item': 49,\n",
       " 'items': 100,\n",
       " 'its': 233,\n",
       " 'jackets': 705,\n",
       " 'jeans': 548,\n",
       " 'jersey': 542,\n",
       " 'juice': 385,\n",
       " 'juicers': 713,\n",
       " 'kept': 365,\n",
       " 'kettle': 821,\n",
       " 'keyboard': 230,\n",
       " 'keys': 626,\n",
       " 'kid': 635,\n",
       " 'kids': 97,\n",
       " 'kind': 370,\n",
       " 'kinds': 854,\n",
       " 'kitchen': 527,\n",
       " 'knife': 728,\n",
       " 'knifes': 650,\n",
       " 'knives': 377,\n",
       " 'knob': 676,\n",
       " 'knobs': 402,\n",
       " 'ladder': 194,\n",
       " 'ladders': 40,\n",
       " 'lamp': 183,\n",
       " 'lamps': 738,\n",
       " 'lampshade': 78,\n",
       " 'laptop': 654,\n",
       " 'laptops': 574,\n",
       " 'larg': 26,\n",
       " 'large': 48,\n",
       " 'largest': 618,\n",
       " 'last': 529,\n",
       " 'laundry': 192,\n",
       " 'leaf': 716,\n",
       " 'leaning': 591,\n",
       " 'leather': 469,\n",
       " 'left': 649,\n",
       " 'leg': 88,\n",
       " 'legs': 353,\n",
       " 'letter': 763,\n",
       " 'letters': 762,\n",
       " 'light': 380,\n",
       " 'lights': 79,\n",
       " 'liquid': 460,\n",
       " 'liquod': 493,\n",
       " 'lists': 362,\n",
       " 'little': 340,\n",
       " 'locker': 151,\n",
       " 'long': 332,\n",
       " 'lying': 877,\n",
       " 'machine': 181,\n",
       " 'machines': 69,\n",
       " 'made': 804,\n",
       " 'magazines': 771,\n",
       " 'main': 245,\n",
       " 'maker': 641,\n",
       " 'man': 204,\n",
       " 'mantel': 867,\n",
       " 'manu': 580,\n",
       " 'many': 531,\n",
       " 'map': 190,\n",
       " 'markers': 722,\n",
       " 'maroon': 577,\n",
       " 'mat': 193,\n",
       " 'mats': 288,\n",
       " 'mattress': 241,\n",
       " 'mauve': 707,\n",
       " 'metallic': 798,\n",
       " 'microwave': 12,\n",
       " 'middle': 589,\n",
       " 'mirror': 710,\n",
       " 'mirrors': 159,\n",
       " 'mixer': 753,\n",
       " 'modem': 422,\n",
       " 'monitor': 236,\n",
       " 'monitors': 284,\n",
       " 'monkey': 343,\n",
       " 'more': 128,\n",
       " 'most': 414,\n",
       " 'mounted': 20,\n",
       " 'mouse': 604,\n",
       " 'mouses': 155,\n",
       " 'mug': 307,\n",
       " 'mugs': 446,\n",
       " 'music': 123,\n",
       " 'name': 262,\n",
       " 'napkin': 818,\n",
       " 'near': 77,\n",
       " 'nect': 217,\n",
       " 'nest': 627,\n",
       " 'newspapers': 751,\n",
       " 'next': 120,\n",
       " 'night': 699,\n",
       " 'not': 837,\n",
       " 'notebook': 197,\n",
       " 'notebooks': 239,\n",
       " 'notes': 46,\n",
       " 'notice': 515,\n",
       " 'number': 327,\n",
       " 'nyu': 180,\n",
       " 'obejct': 483,\n",
       " 'object': 412,\n",
       " 'objects': 90,\n",
       " 'of': 160,\n",
       " 'off': 636,\n",
       " 'oil': 279,\n",
       " 'on': 744,\n",
       " 'one': 329,\n",
       " 'onto': 319,\n",
       " 'open': 291,\n",
       " 'opened': 391,\n",
       " 'opposite': 409,\n",
       " 'or': 759,\n",
       " 'orange': 243,\n",
       " 'ornamental': 57,\n",
       " 'other': 844,\n",
       " 'ottoman': 441,\n",
       " 'outlet': 464,\n",
       " 'outlets': 858,\n",
       " 'outside': 622,\n",
       " 'oven': 229,\n",
       " 'ovens': 730,\n",
       " 'packet': 484,\n",
       " 'pad': 161,\n",
       " 'painted': 842,\n",
       " 'painting': 544,\n",
       " 'pairs': 774,\n",
       " 'pan': 621,\n",
       " 'pane': 478,\n",
       " 'paper': 227,\n",
       " 'papers': 847,\n",
       " 'part': 363,\n",
       " 'partition': 425,\n",
       " 'patterns': 299,\n",
       " 'peices': 131,\n",
       " 'pen': 724,\n",
       " 'pencil': 820,\n",
       " 'pens': 235,\n",
       " 'people': 706,\n",
       " 'perfume': 470,\n",
       " 'person': 642,\n",
       " 'pf': 564,\n",
       " 'photo': 653,\n",
       " 'photocopying': 143,\n",
       " 'photographs': 691,\n",
       " 'photos': 575,\n",
       " 'piano': 141,\n",
       " 'picture': 852,\n",
       " 'pictures': 500,\n",
       " 'piece': 814,\n",
       " 'pieces': 164,\n",
       " 'pilar': 870,\n",
       " 'pile': 834,\n",
       " 'piled': 755,\n",
       " 'pillar': 693,\n",
       " 'pillars': 668,\n",
       " 'pillow': 106,\n",
       " 'pillows': 855,\n",
       " 'ping': 179,\n",
       " 'pink': 172,\n",
       " 'pinned': 348,\n",
       " 'pipe': 429,\n",
       " 'place': 318,\n",
       " 'placed': 499,\n",
       " 'plant': 191,\n",
       " 'plants': 581,\n",
       " 'plastic': 347,\n",
       " 'plate': 457,\n",
       " 'plates': 619,\n",
       " 'play': 406,\n",
       " 'player': 146,\n",
       " 'playing': 218,\n",
       " 'plug': 743,\n",
       " 'plugged': 841,\n",
       " 'plum': 154,\n",
       " 'pockets': 758,\n",
       " 'pong': 171,\n",
       " 'pool': 863,\n",
       " 'portion': 875,\n",
       " 'post': 740,\n",
       " 'poster': 791,\n",
       " 'posters': 861,\n",
       " 'pot': 547,\n",
       " 'pots': 326,\n",
       " 'pouch': 140,\n",
       " 'predominantly': 399,\n",
       " 'present': 579,\n",
       " 'printed': 417,\n",
       " 'printer': 407,\n",
       " 'printers': 453,\n",
       " 'projector': 118,\n",
       " 'pull': 803,\n",
       " 'puncher': 596,\n",
       " 'purple': 874,\n",
       " 'put': 305,\n",
       " 'rack': 316,\n",
       " 'racket': 490,\n",
       " 'racks': 203,\n",
       " 'railing': 61,\n",
       " 'reception': 878,\n",
       " 'rectangle': 455,\n",
       " 'rectangular': 68,\n",
       " 'red': 298,\n",
       " 'reflected': 252,\n",
       " 'reflection': 637,\n",
       " 'refridgerator': 184,\n",
       " 'refrigerator': 421,\n",
       " 'remote': 866,\n",
       " 'rest': 656,\n",
       " 'rested': 859,\n",
       " 'rests': 99,\n",
       " 'righ': 302,\n",
       " 'right': 704,\n",
       " 'rocking': 578,\n",
       " 'rod': 8,\n",
       " 'rods': 169,\n",
       " 'roll': 850,\n",
       " 'rolled': 518,\n",
       " 'rolling': 769,\n",
       " 'rolls': 776,\n",
       " 'roof': 536,\n",
       " 'room': 39,\n",
       " 'rotates': 802,\n",
       " 'round': 55,\n",
       " 'row': 19,\n",
       " 'rows': 778,\n",
       " 'rug': 735,\n",
       " 'rulers': 600,\n",
       " 'runner': 384,\n",
       " 'runners': 754,\n",
       " 's': 615,\n",
       " 'saffron': 840,\n",
       " 'salt': 431,\n",
       " 'sandal': 664,\n",
       " 'saucer': 714,\n",
       " 'says': 59,\n",
       " 'scanner': 819,\n",
       " 'scattered': 437,\n",
       " 'scissor': 397,\n",
       " 'scissors': 281,\n",
       " 'screen': 519,\n",
       " 'sculptures': 207,\n",
       " 'seat': 485,\n",
       " 'seater': 523,\n",
       " 'seats': 482,\n",
       " 'second': 64,\n",
       " 'section': 74,\n",
       " 'sections': 788,\n",
       " 'see': 486,\n",
       " 'seen': 388,\n",
       " 'self': 398,\n",
       " 'shade': 214,\n",
       " 'shades': 103,\n",
       " 'shadow': 686,\n",
       " 'shaker': 731,\n",
       " 'shape': 658,\n",
       " 'shaped': 690,\n",
       " 'shaver': 511,\n",
       " 'sheet': 342,\n",
       " 'sheets': 418,\n",
       " 'shelf': 359,\n",
       " 'shelve': 153,\n",
       " 'shelves': 311,\n",
       " 'shirt': 610,\n",
       " 'shirts': 66,\n",
       " 'shoe': 440,\n",
       " 'shoes': 223,\n",
       " 'show': 434,\n",
       " 'shower': 376,\n",
       " 'shutter': 715,\n",
       " 'side': 272,\n",
       " 'sides': 862,\n",
       " 'sidetable': 289,\n",
       " 'silver': 259,\n",
       " 'sink': 91,\n",
       " 'sinks': 87,\n",
       " 'sitting': 31,\n",
       " 'slippers': 593,\n",
       " 'small': 53,\n",
       " 'smallest': 507,\n",
       " 'soap': 403,\n",
       " 'soaps': 338,\n",
       " 'socks': 766,\n",
       " 'sofa': 692,\n",
       " 'sofas': 611,\n",
       " 'soft': 701,\n",
       " 'solor': 237,\n",
       " 'spatula': 838,\n",
       " 'speaker': 616,\n",
       " 'speakers': 393,\n",
       " 'spice': 801,\n",
       " 'splits': 584,\n",
       " 'spoon': 35,\n",
       " 'spot': 195,\n",
       " 'spray': 524,\n",
       " 'spread': 670,\n",
       " 'sprinklers': 702,\n",
       " 'square': 770,\n",
       " 'squer': 296,\n",
       " 'squere': 594,\n",
       " 'sre': 62,\n",
       " 'stack': 634,\n",
       " 'stacked': 37,\n",
       " 'stacks': 742,\n",
       " 'stair': 447,\n",
       " 'stairs': 367,\n",
       " 'stamp': 561,\n",
       " 'stand': 799,\n",
       " 'standing': 116,\n",
       " 'stands': 72,\n",
       " 'stapler': 41,\n",
       " 'staplers': 826,\n",
       " 'statue': 632,\n",
       " 'steel': 369,\n",
       " 'step': 737,\n",
       " 'steps': 697,\n",
       " 'sticker': 127,\n",
       " 'sticks': 775,\n",
       " 'sticky': 824,\n",
       " 'stnad': 503,\n",
       " 'stool': 76,\n",
       " 'stools': 680,\n",
       " 'storage': 839,\n",
       " 'store': 354,\n",
       " 'stove': 27,\n",
       " 'striped': 466,\n",
       " 'stuck': 784,\n",
       " 'student': 325,\n",
       " 'students': 856,\n",
       " 'suitcase': 813,\n",
       " 'surround': 167,\n",
       " 'surrounded': 5,\n",
       " 'surrounding': 63,\n",
       " 'suspended': 333,\n",
       " 'sweatshirts': 410,\n",
       " 'switch': 211,\n",
       " 'switchboard': 796,\n",
       " 'switchboards': 586,\n",
       " 'switched': 6,\n",
       " 'switches': 221,\n",
       " 'swivel': 163,\n",
       " 'symbols': 225,\n",
       " 'system': 346,\n",
       " 't': 698,\n",
       " 'table': 553,\n",
       " 'tablecloth': 201,\n",
       " 'tables': 663,\n",
       " 'tall': 216,\n",
       " 'tallest': 567,\n",
       " 'tank': 583,\n",
       " 'tap': 158,\n",
       " 'tape': 452,\n",
       " 'tea': 60,\n",
       " 'teacher': 94,\n",
       " 'teddy': 560,\n",
       " 'telelvision': 290,\n",
       " 'telephon': 761,\n",
       " 'telephone': 134,\n",
       " 'telephones': 213,\n",
       " 'television': 461,\n",
       " 'televisions': 157,\n",
       " 'text': 435,\n",
       " 'th': 24,\n",
       " 'that': 357,\n",
       " 'the': 646,\n",
       " 'their': 349,\n",
       " 'them': 248,\n",
       " 'there': 324,\n",
       " 'thing': 317,\n",
       " 'things': 612,\n",
       " 'third': 371,\n",
       " 'this': 147,\n",
       " 'through': 228,\n",
       " 'tier': 199,\n",
       " 'tin': 843,\n",
       " 'tissue': 601,\n",
       " 'to': 23,\n",
       " 'toaster': 613,\n",
       " 'together': 871,\n",
       " 'toilet': 816,\n",
       " 'toiletries': 209,\n",
       " 'tooth': 428,\n",
       " 'toothbrush': 522,\n",
       " 'toothbrushes': 105,\n",
       " 'toothpaste': 562,\n",
       " 'top': 344,\n",
       " 'topmost': 442,\n",
       " 'towel': 689,\n",
       " 'towels': 552,\n",
       " 'toy': 341,\n",
       " 'toys': 113,\n",
       " 'trampoline': 734,\n",
       " 'trash': 849,\n",
       " 'tray': 271,\n",
       " 'trays': 555,\n",
       " 'treadmill': 727,\n",
       " 'tree': 373,\n",
       " 'tricycle': 44,\n",
       " 'trollies': 115,\n",
       " 'tshirtsg': 570,\n",
       " 'tub': 526,\n",
       " 'tube': 879,\n",
       " 'tumblers': 780,\n",
       " 'tupperware': 857,\n",
       " 'tv': 22,\n",
       " 'two': 119,\n",
       " 'type': 126,\n",
       " 'umbrella': 300,\n",
       " 'under': 25,\n",
       " 'up': 808,\n",
       " 'us': 809,\n",
       " 'used': 315,\n",
       " 'utensil': 301,\n",
       " 'utensils': 825,\n",
       " 'vacuum': 684,\n",
       " 'varieties': 206,\n",
       " 'variety': 836,\n",
       " 'vase': 34,\n",
       " 'vases': 198,\n",
       " 'vegetables': 781,\n",
       " 'vent': 543,\n",
       " 'view': 476,\n",
       " 'viisble': 294,\n",
       " 'violet': 489,\n",
       " 'visible': 111,\n",
       " 'waht': 662,\n",
       " 'wall': 617,\n",
       " 'walls': 253,\n",
       " 'wardrobe': 364,\n",
       " 'wash': 330,\n",
       " 'washing': 256,\n",
       " 'water': 89,\n",
       " 'way': 786,\n",
       " 'wearing': 456,\n",
       " 'weighing': 516,\n",
       " 'what': 71,\n",
       " 'whats': 231,\n",
       " 'where': 475,\n",
       " 'which': 401,\n",
       " 'while': 590,\n",
       " 'whit': 283,\n",
       " 'white': 352,\n",
       " 'whiteboard': 752,\n",
       " 'who': 732,\n",
       " 'window': 242,\n",
       " 'windows': 810,\n",
       " 'windowsill': 21,\n",
       " 'wine': 220,\n",
       " 'wire': 496,\n",
       " 'wired': 631,\n",
       " 'wires': 573,\n",
       " 'with': 800,\n",
       " 'woman': 30,\n",
       " 'women': 142,\n",
       " 'wood': 451,\n",
       " 'wooden': 45,\n",
       " 'world': 747,\n",
       " 'writing': 554,\n",
       " 'written': 449,\n",
       " 'yellow': 9,\n",
       " 'you': 848}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Kraino is a framework that helps in fast prototyping Visual Turing Test models\n",
    "from kraino.utils.input_output_space import build_vocabulary\n",
    "\n",
    "# This function takes wordcounts and returns word2index - mapping from words into indices, \n",
    "# and index2word - mapping from indices to words.\n",
    "word2index_x, index2word_x = build_vocabulary(\n",
    "    this_wordcount=wordcount_x,\n",
    "    truncate_to_most_frequent=0)\n",
    "word2index_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, we are using a few special symbols that don't occur in the training dataset.\n",
    "Most important are $<pad>$ and $<unk>$. We will use the former to pad sequences in order to have the same \n",
    "number of temporal elements; we will use the latter for words (at test time) that don't exist in training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Armed with vocabulary, we can build one-hot representation of the training data. However, this is not neccessary and maybe even wasteful. Our one-hot representation of the input text doesn't explicitely build long vectors, but instead it operates on indices. The example above would be encoded as [0,1,4,2,7,3]. \n",
    "```\n",
    "Can you prove the equivalence in the claim?\n",
    "```\n",
    "__claim__:\n",
    "\n",
    "Let $x$ be a binary vector with exactly one value $1$ at position $index$, that is $x[index]=1$. Then $$W[:,index] = Wx$$ where $W[:,b]$ denotes a vector built from a column $b$ of $W$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['what is on the right side of the black telephone and on the left side of the red chair ?', 'what is in front of the white door on the left side of the desk ?', 'what is on the desk ?']\n",
      "[[71, 597, 744, 646, 704, 272, 160, 646, 124, 134, 382, 744, 646, 649, 272, 160, 646, 298, 15, 52, 3], [71, 597, 602, 255, 160, 646, 352, 130, 744, 646, 649, 272, 160, 646, 655, 52, 3], [71, 597, 744, 646, 655, 52, 3]]\n"
     ]
    }
   ],
   "source": [
    "from kraino.utils.input_output_space import encode_questions_index\n",
    "one_hot_x = encode_questions_index(train_raw_x, word2index_x)\n",
    "print(train_raw_x[:3])\n",
    "print(one_hot_x[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the sequences have different elements. We will pad the sequences to have the same length $MAXLEN$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0,   0,   0,   0,  71, 597, 744, 646,\n",
       "        704, 272, 160, 646, 124, 134, 382, 744, 646, 649, 272, 160, 646,\n",
       "        298,  15,  52,   3],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         71, 597, 602, 255, 160, 646, 352, 130, 744, 646, 649, 272, 160,\n",
       "        646, 655,  52,   3],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  71, 597, 744,\n",
       "        646, 655,  52,   3]], dtype=int32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We use another framework that is useful to build deep learning models - Keras\n",
    "from keras.preprocessing import sequence\n",
    "MAXLEN=30\n",
    "train_x = sequence.pad_sequences(one_hot_x, maxlen=MAXLEN)\n",
    "train_x[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And do the same with the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6795, 30)\n",
      "(6795, 972)\n"
     ]
    }
   ],
   "source": [
    "# for simplicity, we consider only first answer words; that is, if answer is 'knife,fork' we encode only 'knife'\n",
    "# note, however, that the tutorial supports multiple word answers (is_only_first_answer_word=False)\n",
    "# with MAX_ANSWER_TIME_STEPS defining number of answer words (ignored if is_only_first_answer_word=True)\n",
    "MAX_ANSWER_TIME_STEPS=10\n",
    "\n",
    "from kraino.utils.input_output_space import encode_answers_one_hot\n",
    "train_raw_y = train_text_representation['y']\n",
    "wordcount_y = frequencies(' '.join(train_raw_y).replace(', ',',').split(' '))\n",
    "word2index_y, index2word_y = build_vocabulary(this_wordcount=wordcount_y)\n",
    "train_y, _ = encode_answers_one_hot(\n",
    "    train_raw_y, \n",
    "    word2index_y, \n",
    "    answer_words_delimiter=train_text_representation['answer_words_delimiter'],\n",
    "    is_only_first_answer_word=True,\n",
    "    max_answer_time_steps=MAX_ANSWER_TIME_STEPS)\n",
    "print(train_x.shape)\n",
    "print(train_y.shape)\n",
    "# word2index_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can also encode test questions. We need it later to see how well our models generalise to new question,answer,image triplets. Remember however that we should use vocabulary we generated from training samples.\n",
    "\n",
    "```\n",
    "Why should we use the training vocabulary to encode test questions?\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_list' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-15-f9c5c8c52ceb>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mtest_one_hot_x\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mencode_questions_index\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_raw_x\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mword2index_x\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mtest_x\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msequence\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpad_sequences\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_one_hot_x\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmaxlen\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mMAXLEN\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprint_list\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_raw_x\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[0mtest_x\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'print_list' is not defined"
     ]
    }
   ],
   "source": [
    "test_text_representation = dp['text'](train_or_test='test')\n",
    "test_raw_x = test_text_representation['x']\n",
    "test_one_hot_x = encode_questions_index(test_raw_x, word2index_x)\n",
    "test_x = sequence.pad_sequences(test_one_hot_x, maxlen=MAXLEN)\n",
    "print_list(test_raw_x[:3])\n",
    "test_x[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With encoded question, answer pairs we finish the first section. But before delving into building and training new models, let's have a look at summary to see bigger picture.\n",
    "\n",
    "__Summary__\n",
    "\n",
    "We started from raw questions from the training set. Use them to build a vocabulary. Next, we encode questions into sequences of one-hot vectors based on the vocabulary. Finally, we use the same vocabulary to encode questions from test set, if a word is absent we use extra token $<unk>$ to encode this fact (we encode the $<unk>$ token, not the word)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Language Only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training - overall picture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. We have textual features already built. Let's create some models that we will use for training and later for answering on questions about images.\n",
    "\n",
    "As you may already know, we train models by weights updates. Let $x$ and $y$ be training samples (input, output), and $\\ell(x,y)$ is an objective function. The formula for weights updates is:\n",
    "$$w := w - \\alpha \\nabla \\ell(x,y; w)$$\n",
    "with $\\alpha$ that we call the learning rate, and $\\nabla$ is  a gradient wrt. weights $w$. This is a hyper-parameter that must be set in advance. The rule shown above is called SGD update, but other its variants are also possible. In fact, we will use a variant called [ADAM](http://arxiv.org/pdf/1412.6980v8.pdf).\n",
    "\n",
    "We cast the question answering problem into a classification framework, so that we classify input $x$ into some class that represents an answer word. Therefore, we use, popular in classification, logistic regression as the objective:\n",
    "$$\\ell(x,y;w):=\\sum_{y' \\in \\mathcal{C}} \\mathbb{1}\\{y'=y\\}\\log p(y'\\;|\\;x,w)$$\n",
    "where $\\mathcal{C}$ is a set of all classes, and for $p(y\\;|\\;x,w)$ we will use softmax: $e^{w^y\\phi(x)} / \\sum_{z}e^{w^z\\phi(x)}$. Here $\\phi(x)$ denotes an output of a model (more precisely, it's often a neural network's response to the input, just before softmax).\n",
    "\n",
    "The training can be formalised (and automatised) so that you need to execute a procedure that looks something like that:\n",
    "```python\n",
    "training(gradient_of_the_model, optimizer='Adam')\n",
    "```\n",
    "\n",
    "__Summary__\n",
    "Given a model, and an optimization procedure (SGD, Adam, etc.) all we need is to compute gradient of the model $\\nabla \\ell(x,y;w)$ wrt. to its parameters $w$, and next plug it to the optimisation procedure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theano"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since computing gradients $\\nabla \\ell(x,y;w)$ may become quickly tedious, especially for more complex models, we search for tools that could automitise it as well. Imagine that you build some model $M$ and you get its gradient $\\nabla M$ by just executing the tool\n",
    "```python\n",
    "nabla_M = compute_gradient_symbolically(M,x,y)\n",
    "```\n",
    "This would definitely speed up prototyping.\n",
    "\n",
    "[Theano](http://deeplearning.net/software/theano/) is such a tool that is specifically tailored to work with deep learning models. For broader understanding Theano, you can check [this nice tutorial](http://deeplearning.net/tutorial/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The programmming example shown cell below defines ReLU, a popular activation function, as well as shows its derivative using Theano.\n",
    "However, with this example, we only scratch the surface.\n",
    "\n",
    "Assume ReLU is defined as follows $ReLU(x) = \\max(x,0)$.\n",
    "```\n",
    "What is the gradient of ReLU? Consider two cases. \n",
    "```\n",
    "Btw. ReLU is a nondifferentiable function, so technically we are computing its [subgradient](https://en.wikipedia.org/wiki/Subderivative) - it is still fine for Theano."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't import dot_parser, loading of dot files will not be possible.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 0: Tesla K40m (CNMeM is disabled, cuDNN 5004)\n",
      "WARNING (theano.sandbox.cuda.opt): Optimization Warning: Got the following error, but you can ignore it. This could cause less GpuElemwise fused together.\n",
      "CudaNdarray_ptr_int_size: error when calling the gpu code. (invalid device function)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<TensorType(float32, scalar)>\n",
      "Elemwise{maximum,no_inplace}.0\n",
      "Elemwise{mul}.0\n",
      "3.0\n",
      "0.0\n",
      "1.0\n",
      "0.0\n"
     ]
    }
   ],
   "source": [
    "import theano\n",
    "import theano.tensor as T\n",
    "\n",
    "# Theano is using symbolic calculations, so we need to first create symbolic variables\n",
    "theano_x = T.scalar()\n",
    "# we define a relationship between a symbolic input and a symbolic output\n",
    "theano_y = T.maximum(0,theano_x)\n",
    "# now it's time for a symbolic gradient wrt. to symbolic variable x\n",
    "theano_nabla_y = T.grad(theano_y, theano_x)\n",
    "\n",
    "# we can see that both variables are symbolic, they don't have any numerical values\n",
    "print(theano_x)\n",
    "print(theano_y)\n",
    "print(theano_nabla_y)\n",
    "\n",
    "# theano.function compiles the symbolic representation of the network\n",
    "theano_f_x = theano.function([theano_x], theano_y)\n",
    "print(theano_f_x(3))\n",
    "print(theano_f_x(-3))\n",
    "# and now for gradients\n",
    "\n",
    "nabla_f_x = theano.function([theano_x], theano_nabla_y)\n",
    "print(nabla_f_x(3))\n",
    "print(nabla_f_x(-3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Summary__\n",
    "\n",
    "To compute gradient symbolically, we can use [Theano](http://deeplearning.net/software/theano/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "[Keras](http://keras.io) builds on top of Theano, and significantly simplifies creating new models as well as training such models, effectively speeding up the prototyping even further. Keras also abstracts away from some technical burden such as symbolic variable creation. Metaphorically, while Theano can be seen as a deep learning equivalent of assembler, Keras is more like Java :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "10000/10000 [==============================] - 0s - loss: 1430.5141     \n",
      "Epoch 2/100\n",
      "10000/10000 [==============================] - 0s - loss: 1145.2346     \n",
      "Epoch 3/100\n",
      "10000/10000 [==============================] - 0s - loss: 850.6267     \n",
      "Epoch 4/100\n",
      "10000/10000 [==============================] - 0s - loss: 565.8305     \n",
      "Epoch 5/100\n",
      "10000/10000 [==============================] - 0s - loss: 346.8505     \n",
      "Epoch 6/100\n",
      "10000/10000 [==============================] - 0s - loss: 217.7273     \n",
      "Epoch 7/100\n",
      "10000/10000 [==============================] - 0s - loss: 163.6030     \n",
      "Epoch 8/100\n",
      "10000/10000 [==============================] - 0s - loss: 150.3268     \n",
      "Epoch 9/100\n",
      "10000/10000 [==============================] - 0s - loss: 140.7599     \n",
      "Epoch 10/100\n",
      "10000/10000 [==============================] - 0s - loss: 137.4868     \n",
      "Epoch 11/100\n",
      "10000/10000 [==============================] - 0s - loss: 130.3813     \n",
      "Epoch 12/100\n",
      "10000/10000 [==============================] - 0s - loss: 127.8312     \n",
      "Epoch 13/100\n",
      "10000/10000 [==============================] - 0s - loss: 125.1653     \n",
      "Epoch 14/100\n",
      "10000/10000 [==============================] - 0s - loss: 120.6361     \n",
      "Epoch 15/100\n",
      "10000/10000 [==============================] - 0s - loss: 113.2545     \n",
      "Epoch 16/100\n",
      "10000/10000 [==============================] - 0s - loss: 108.8070     \n",
      "Epoch 17/100\n",
      "10000/10000 [==============================] - 0s - loss: 107.1277     \n",
      "Epoch 18/100\n",
      "10000/10000 [==============================] - 0s - loss: 103.7616     \n",
      "Epoch 19/100\n",
      "10000/10000 [==============================] - 0s - loss: 99.4801     \n",
      "Epoch 20/100\n",
      "10000/10000 [==============================] - 0s - loss: 98.4273     \n",
      "Epoch 21/100\n",
      "10000/10000 [==============================] - 0s - loss: 95.5986     \n",
      "Epoch 22/100\n",
      "10000/10000 [==============================] - 0s - loss: 89.4088     \n",
      "Epoch 23/100\n",
      "10000/10000 [==============================] - 0s - loss: 89.0551     \n",
      "Epoch 24/100\n",
      "10000/10000 [==============================] - 0s - loss: 84.7646     \n",
      "Epoch 25/100\n",
      "10000/10000 [==============================] - 0s - loss: 82.3652     \n",
      "Epoch 26/100\n",
      "10000/10000 [==============================] - 0s - loss: 80.0207     \n",
      "Epoch 27/100\n",
      "10000/10000 [==============================] - 0s - loss: 78.4248     \n",
      "Epoch 28/100\n",
      "10000/10000 [==============================] - 0s - loss: 74.7715     \n",
      "Epoch 29/100\n",
      "10000/10000 [==============================] - 0s - loss: 73.6129     \n",
      "Epoch 30/100\n",
      "10000/10000 [==============================] - 0s - loss: 71.8952     \n",
      "Epoch 31/100\n",
      "10000/10000 [==============================] - 0s - loss: 70.8732     \n",
      "Epoch 32/100\n",
      "10000/10000 [==============================] - 0s - loss: 69.9281     \n",
      "Epoch 33/100\n",
      "10000/10000 [==============================] - 0s - loss: 66.7601     \n",
      "Epoch 34/100\n",
      "10000/10000 [==============================] - 0s - loss: 68.5129     \n",
      "Epoch 35/100\n",
      "10000/10000 [==============================] - 0s - loss: 64.2157     \n",
      "Epoch 36/100\n",
      "10000/10000 [==============================] - 0s - loss: 64.0088     \n",
      "Epoch 37/100\n",
      "10000/10000 [==============================] - 0s - loss: 62.1084     \n",
      "Epoch 38/100\n",
      "10000/10000 [==============================] - 0s - loss: 60.4780     \n",
      "Epoch 39/100\n",
      "10000/10000 [==============================] - 0s - loss: 58.8717     \n",
      "Epoch 40/100\n",
      "10000/10000 [==============================] - 0s - loss: 59.4814     \n",
      "Epoch 41/100\n",
      "10000/10000 [==============================] - 0s - loss: 57.8463     \n",
      "Epoch 42/100\n",
      "10000/10000 [==============================] - 0s - loss: 56.6349     \n",
      "Epoch 43/100\n",
      "10000/10000 [==============================] - 0s - loss: 53.3060     \n",
      "Epoch 44/100\n",
      "10000/10000 [==============================] - 0s - loss: 53.8152     \n",
      "Epoch 45/100\n",
      "10000/10000 [==============================] - 0s - loss: 53.2868     \n",
      "Epoch 46/100\n",
      "10000/10000 [==============================] - 0s - loss: 51.8249     \n",
      "Epoch 47/100\n",
      "10000/10000 [==============================] - 0s - loss: 51.4696     \n",
      "Epoch 48/100\n",
      "10000/10000 [==============================] - 0s - loss: 49.4345     \n",
      "Epoch 49/100\n",
      "10000/10000 [==============================] - 0s - loss: 49.5804     \n",
      "Epoch 50/100\n",
      "10000/10000 [==============================] - 0s - loss: 46.1419     \n",
      "Epoch 51/100\n",
      "10000/10000 [==============================] - 0s - loss: 46.0174     \n",
      "Epoch 52/100\n",
      "10000/10000 [==============================] - 0s - loss: 45.5138     \n",
      "Epoch 53/100\n",
      "10000/10000 [==============================] - 0s - loss: 43.0275     \n",
      "Epoch 54/100\n",
      "10000/10000 [==============================] - 0s - loss: 44.4757     \n",
      "Epoch 55/100\n",
      "10000/10000 [==============================] - 0s - loss: 43.1267     \n",
      "Epoch 56/100\n",
      "10000/10000 [==============================] - 0s - loss: 43.1435     \n",
      "Epoch 57/100\n",
      "10000/10000 [==============================] - 0s - loss: 42.6600     \n",
      "Epoch 58/100\n",
      "10000/10000 [==============================] - 0s - loss: 40.1952     \n",
      "Epoch 59/100\n",
      "10000/10000 [==============================] - 0s - loss: 39.9561     \n",
      "Epoch 60/100\n",
      "10000/10000 [==============================] - 0s - loss: 37.6587     \n",
      "Epoch 61/100\n",
      "10000/10000 [==============================] - 0s - loss: 38.5544     \n",
      "Epoch 62/100\n",
      "10000/10000 [==============================] - 0s - loss: 38.8892     \n",
      "Epoch 63/100\n",
      "10000/10000 [==============================] - 0s - loss: 37.9775     \n",
      "Epoch 64/100\n",
      "10000/10000 [==============================] - 0s - loss: 36.9681     \n",
      "Epoch 65/100\n",
      "10000/10000 [==============================] - 0s - loss: 37.5887     \n",
      "Epoch 66/100\n",
      "10000/10000 [==============================] - 0s - loss: 35.8916     \n",
      "Epoch 67/100\n",
      "10000/10000 [==============================] - 0s - loss: 34.5924     \n",
      "Epoch 68/100\n",
      "10000/10000 [==============================] - 0s - loss: 35.3941     \n",
      "Epoch 69/100\n",
      "10000/10000 [==============================] - 0s - loss: 34.5986     \n",
      "Epoch 70/100\n",
      "10000/10000 [==============================] - 0s - loss: 34.1270     \n",
      "Epoch 71/100\n",
      "10000/10000 [==============================] - 0s - loss: 33.3335     \n",
      "Epoch 72/100\n",
      "10000/10000 [==============================] - 0s - loss: 34.9812     \n",
      "Epoch 73/100\n",
      "10000/10000 [==============================] - 0s - loss: 33.6671     \n",
      "Epoch 74/100\n",
      "10000/10000 [==============================] - 0s - loss: 32.1354     \n",
      "Epoch 75/100\n",
      "10000/10000 [==============================] - 0s - loss: 32.4637     \n",
      "Epoch 76/100\n",
      "10000/10000 [==============================] - 0s - loss: 32.2159     \n",
      "Epoch 77/100\n",
      "10000/10000 [==============================] - 0s - loss: 31.7502     \n",
      "Epoch 78/100\n",
      "10000/10000 [==============================] - 0s - loss: 31.5369     \n",
      "Epoch 79/100\n",
      "10000/10000 [==============================] - 0s - loss: 31.7230     \n",
      "Epoch 80/100\n",
      "10000/10000 [==============================] - 0s - loss: 30.7868     \n",
      "Epoch 81/100\n",
      "10000/10000 [==============================] - 0s - loss: 31.0717     \n",
      "Epoch 82/100\n",
      "10000/10000 [==============================] - 0s - loss: 30.1934     \n",
      "Epoch 83/100\n",
      "10000/10000 [==============================] - 0s - loss: 30.4590     \n",
      "Epoch 84/100\n",
      "10000/10000 [==============================] - 0s - loss: 31.4997     \n",
      "Epoch 85/100\n",
      "10000/10000 [==============================] - 0s - loss: 28.4967     \n",
      "Epoch 86/100\n",
      "10000/10000 [==============================] - 0s - loss: 30.0666     \n",
      "Epoch 87/100\n",
      "10000/10000 [==============================] - 0s - loss: 29.2898     \n",
      "Epoch 88/100\n",
      "10000/10000 [==============================] - 0s - loss: 29.7264     \n",
      "Epoch 89/100\n",
      "10000/10000 [==============================] - 0s - loss: 29.2317     \n",
      "Epoch 90/100\n",
      "10000/10000 [==============================] - 0s - loss: 28.1746     \n",
      "Epoch 91/100\n",
      "10000/10000 [==============================] - 0s - loss: 29.5667     \n",
      "Epoch 92/100\n",
      "10000/10000 [==============================] - 0s - loss: 29.0125     \n",
      "Epoch 93/100\n",
      "10000/10000 [==============================] - 0s - loss: 28.0620     \n",
      "Epoch 94/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.6503     \n",
      "Epoch 95/100\n",
      "10000/10000 [==============================] - 0s - loss: 28.5997     \n",
      "Epoch 96/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.9306     \n",
      "Epoch 97/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.1531     \n",
      "Epoch 98/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.0039     \n",
      "Epoch 99/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.0244     \n",
      "Epoch 100/100\n",
      "10000/10000 [==============================] - 0s - loss: 27.4526     \n",
      "         X         Y        GT\n",
      "         1      0.76         1\n",
      "         3      8.03         9\n",
      "         6     37.11        36\n",
      "        12    116.20       144\n",
      "        20    221.47       400\n"
     ]
    }
   ],
   "source": [
    "# we sample from noisy x^2 function\n",
    "from numpy import asarray\n",
    "from numpy import random\n",
    "def myfun(x):\n",
    "    return x*x\n",
    "\n",
    "NUM_SAMPLES = 10000\n",
    "HIGH_VALUE=10\n",
    "keras_x = asarray(random.randint(low=0, high=HIGH_VALUE, size=NUM_SAMPLES))\n",
    "keras_noise = random.normal(loc=0.0, scale=0.1, size=NUM_SAMPLES)\n",
    "keras_noise = asarray([max(x,0) for x in keras_noise])\n",
    "keras_y = asarray([myfun(x) + n for x,n in zip(keras_x, keras_noise)])\n",
    "# print('X:')\n",
    "# print(keras_x)\n",
    "# print('Noise')\n",
    "# print(keras_noise)\n",
    "# print('Noisy X^2:')\n",
    "# print(keras_y)\n",
    "\n",
    "keras_x = keras_x.reshape(keras_x.shape[0],1)\n",
    "keras_y = keras_y.reshape(keras_y.shape[0],1)\n",
    "\n",
    "# import keras packages\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout\n",
    "\n",
    "# build a regression network\n",
    "KERAS_NUM_HIDDEN = 150\n",
    "KERAS_NUM_HIDDEN_SECOND = 150\n",
    "KERAS_NUM_HIDDEN_THIRD = 150\n",
    "KERAS_DROPOUT_FRACTION = 0.5\n",
    "m = Sequential()\n",
    "m.add(Dense(KERAS_NUM_HIDDEN, input_dim=1))\n",
    "m.add(Activation('relu'))\n",
    "m.add(Dropout(KERAS_DROPOUT_FRACTION))\n",
    "#TODO: add one more layer\n",
    "# m.add(Dense(KERAS_NUM_HIDDEN_SECOND))\n",
    "# m.add(Activation('relu'))\n",
    "# m.add(Dropout(KERAS_DROPOUT_FRACTION))\n",
    "#TODO: add one more layer\n",
    "# m.add(Dense(KERAS_NUM_HIDDEN_THIRD))\n",
    "# m.add(Activation('relu'))\n",
    "# m.add(Dropout(KERAS_DROPOUT_FRACTION))\n",
    "m.add(Dense(1))\n",
    "\n",
    "# compile and fit\n",
    "m.compile(loss='mse', optimizer='adam')\n",
    "m.fit(keras_x, keras_y, nb_epoch=100, batch_size=250)\n",
    "\n",
    "keras_x_predict = asarray([1,3,6,12,HIGH_VALUE+10])\n",
    "keras_x_predict = keras_x_predict.reshape(keras_x_predict.shape[0],1)\n",
    "keras_predictions = m.predict(keras_x_predict)\n",
    "print(\"{0:>10}{1:>10}{2:>10}\".format('X', 'Y', 'GT'))\n",
    "for x,y in zip(keras_x_predict, keras_predictions):\n",
    "    print(\"{0:>10}{1:>10.2f}{2:>10}\".format(x[0], y[0], myfun(x[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can play with the example above.\n",
    "\n",
    " * What happens if you add one more layer? Or two more layers?\n",
    " * What happens if you change hidden size?\n",
    " * What happens if you use more/less samples?\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For the purpose of Visual Turing Test, and this tutorial, we have compiled a light framework that builds on top of Keras, and simplify building and training question answering machines. With the tradition of using fancy Greek names, we call it Kraino. Note that some parts of the Kraino, such as data provider, you have already seen. \n",
    "\n",
    "In the following, we will go through BOW and LSTM approaches to answer questions about images, but, surprisingly, without the images. It turns out that a substantial fraction of questions can be answered without an access to an image, but rather by resorting to common sense (or statistics of the dataset). For instance, what can be placed at the table? How many eyes this human have?. Answers like 'chair' and '2' are quite likely to be good answers.\n",
    "\n",
    "Please make sure that all the cells from [Datasets](#Datasets) and [Textual Features](#Textual-Features) have been executed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BOW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure below illustrates BOW (Bag Of Words) approach. As we have already seen in [Textual Features](#Textual-Features), we first encode the input sentence into one-hot vector representations. Such (very) sparse representation is next embedded into a denser space by a matrix $W_e$. Next, the denser representations are summed up and classified via 'Softmax'. Notice that, if $W_e$ were an identity matrix, we would obtain a histogram of the word's occurrences. \n",
    "\n",
    "\n",
    "```\n",
    "What is your biggest complain about such BOW representation? What happens if instead of \n",
    "'What is behind the table' we would have 'is What the behind table'? How does the BOW representation change? \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![bow](fig/BOW_model.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model definition\n",
    "\n",
    "# First we define a model using keras/kraino\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.core import Dropout\n",
    "from keras.layers.core import TimeDistributedMerge\n",
    "from keras.layers.embeddings import Embedding\n",
    "\n",
    "from kraino.core.model_zoo import AbstractSequentialModel\n",
    "from kraino.core.model_zoo import AbstractSingleAnswer\n",
    "from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer\n",
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.keras_extensions import DropMask\n",
    "from kraino.core.keras_extensions import LambdaWithMask\n",
    "from kraino.core.keras_extensions import time_distributed_masked_ave\n",
    "\n",
    "# This model inherits from AbstractSingleAnswer, and so it produces single answer words\n",
    "# To use multiple answer words, you need to inherit from AbstractSequentialMultiplewordAnswer\n",
    "class BlindBOW(AbstractSequentialModel, AbstractSingleAnswer):\n",
    "    \"\"\"\n",
    "    BOW Language only model that produces single word answers.\n",
    "    \"\"\"\n",
    "    def create(self):\n",
    "        self.add(Embedding(\n",
    "                self._config.input_dim, \n",
    "                self._config.textual_embedding_dim, \n",
    "                mask_zero=True))\n",
    "        self.add(LambdaWithMask(time_distributed_masked_ave, output_shape=[self.output_shape[2]]))\n",
    "        self.add(DropMask())\n",
    "        self.add(Dropout(0.5))\n",
    "        self.add(Dense(self._config.output_dim))\n",
    "        self.add(Activation('softmax'))\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_config = Config(\n",
    "    textual_embedding_dim=500,\n",
    "    input_dim=len(word2index_x.keys()),\n",
    "    output_dim=len(word2index_y.keys()))\n",
    "model = BlindBOW(model_config)\n",
    "model.create()\n",
    "\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')\n",
    "text_bow_model = model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 6115 samples, validate on 680 samples\n",
      "Epoch 1/40\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Cuda error: CudaNdarray_TakeFrom: invalid device function.\n\nApply node that caused the error: GpuAdvancedSubtensor1(GpuElemwise{mul,no_inplace}.0, Elemwise{Cast{int64}}.0)\nToposort index: 40\nInputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]\nInputs shapes: [(880, 500), (15360,)]\nInputs strides: [(500, 1), (8,)]\nInputs values: ['not shown', 'not shown']\nOutputs clients: [[GpuReshape{3}(GpuAdvancedSubtensor1.0, MakeVector{dtype='int64'}.0)]]\n\nHINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-23-52e919d865ac>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      6\u001b[0m     \u001b[0mnb_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m40\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m     \u001b[0mvalidation_split\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0.1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m     show_accuracy=True)\n\u001b[0m",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/models.pyc\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, show_accuracy, class_weight, sample_weight)\u001b[0m\n\u001b[0;32m    699\u001b[0m                          \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    700\u001b[0m                          \u001b[0mval_f\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mval_f\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mval_ins\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mval_ins\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 701\u001b[1;33m                          shuffle=shuffle, metrics=metrics)\n\u001b[0m\u001b[0;32m    702\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    703\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m128\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/models.pyc\u001b[0m in \u001b[0;36m_fit\u001b[1;34m(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, metrics)\u001b[0m\n\u001b[0;32m    315\u001b[0m                 \u001b[0mbatch_logs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'size'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch_ids\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    316\u001b[0m                 \u001b[0mcallbacks\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_batch_begin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch_index\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_logs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 317\u001b[1;33m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    318\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    319\u001b[0m                     \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/backend/theano_backend.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m    444\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    445\u001b[0m         \u001b[1;32massert\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32min\u001b[0m \u001b[1;33m{\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 446\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfunction\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    447\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    448\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    906\u001b[0m                     \u001b[0mnode\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnodes\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mposition_of_error\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    907\u001b[0m                     \u001b[0mthunk\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mthunk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 908\u001b[1;33m                     storage_map=getattr(self.fn, 'storage_map', None))\n\u001b[0m\u001b[0;32m    909\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    910\u001b[0m                 \u001b[1;31m# old-style linkers raise their own exceptions\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/gof/link.pyc\u001b[0m in \u001b[0;36mraise_with_op\u001b[1;34m(node, thunk, exc_info, storage_map)\u001b[0m\n\u001b[0;32m    312\u001b[0m         \u001b[1;31m# extra long error message in that case.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    313\u001b[0m         \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 314\u001b[1;33m     \u001b[0mreraise\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexc_type\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexc_value\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexc_trace\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    315\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    316\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    893\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    894\u001b[0m             \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 895\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0moutput_subset\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[1;32melse\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    896\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    897\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: Cuda error: CudaNdarray_TakeFrom: invalid device function.\n\nApply node that caused the error: GpuAdvancedSubtensor1(GpuElemwise{mul,no_inplace}.0, Elemwise{Cast{int64}}.0)\nToposort index: 40\nInputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]\nInputs shapes: [(880, 500), (15360,)]\nInputs strides: [(500, 1), (8,)]\nInputs values: ['not shown', 'not shown']\nOutputs clients: [[GpuReshape{3}(GpuAdvancedSubtensor1.0, MakeVector{dtype='int64'}.0)]]\n\nHINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node."
     ]
    }
   ],
   "source": [
    "#== Model training\n",
    "text_bow_model.fit(\n",
    "    train_x, \n",
    "    train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurrent Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although BOW is working pretty well, there is still something very disturbing about this approach.\n",
    "Consider the following question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_raw_x[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we swap 'chair' with 'telephone' in the question, we would get a different meaning, wouldn't we? Recurrent Neural Networks (RNNs) have been developed to mitigate this issue by directly processing the time series. As the figure below illustrates, the (temporarily) first word's embedding is given to an RNN unit. The RNN unit next 'processes' such embedding and outputs to the second RNN unit. This unit takes both the output of the first RNN unit and the 2nd word's embedding as inputs, and outputs some algebraic combination of both inputs. And so on. The last recurrent unit builds the representation of the whole sequence. Its output is next given to Softmax for the classification. One of the challenged that such approaches have to deal with are keeping long-term dependencies. Roughly speaking, as new inputs are coming it's getting easier to 'forget' information from the beginning. [LSTM](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) and [GRU](http://www.aclweb.org/anthology/W14-4012) are two particularly successful Recurrent Neural Networks that can preserve such longer dependencies to [some degree](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n",
    "\n",
    "__Note__: If the code below is not compiling, please restart the notebook, and run only [Datasets](#Datasets) and [Textual Features](#Textual-Features). In particular, don't run BOW."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![LSTM](fig/LSTM_model.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model definition\n",
    "\n",
    "# First we define a model using keras/kraino\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.core import Dropout\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.recurrent import GRU\n",
    "from keras.layers.recurrent import LSTM\n",
    "\n",
    "from kraino.core.model_zoo import AbstractSequentialModel\n",
    "from kraino.core.model_zoo import AbstractSingleAnswer\n",
    "from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer\n",
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.keras_extensions import DropMask\n",
    "from kraino.core.keras_extensions import LambdaWithMask\n",
    "from kraino.core.keras_extensions import time_distributed_masked_ave\n",
    "\n",
    "# This model inherits from AbstractSingleAnswer, and so it produces single answer words\n",
    "# To use multiple answer words, you need to inherit from AbstractSequentialMultiplewordAnswer\n",
    "class BlindRNN(AbstractSequentialModel, AbstractSingleAnswer):\n",
    "    \"\"\"\n",
    "    RNN Language only model that produces single word answers.\n",
    "    \"\"\"\n",
    "    def create(self):\n",
    "        self.add(Embedding(\n",
    "                self._config.input_dim, \n",
    "                self._config.textual_embedding_dim, \n",
    "                mask_zero=True))\n",
    "        #TODO: Replace averaging with RNN (you can choose between LSTM and GRU)\n",
    "#         self.add(LambdaWithMask(time_distributed_masked_ave, output_shape=[self.output_shape[2]]))\n",
    "        self.add(GRU(self._config.hidden_state_dim, \n",
    "                      return_sequences=False))\n",
    "        self.add(Dropout(0.5))\n",
    "        self.add(Dense(self._config.output_dim))\n",
    "        self.add(Activation('softmax'))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_config = Config(\n",
    "    textual_embedding_dim=500,\n",
    "    hidden_state_dim=500,\n",
    "    input_dim=len(word2index_x.keys()),\n",
    "    output_dim=len(word2index_y.keys()))\n",
    "model = BlindRNN(model_config)\n",
    "model.create()\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')\n",
    "text_rnn_model = model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 6115 samples, validate on 680 samples\n",
      "Epoch 1/40\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Cuda error: CudaNdarray_TakeFrom: invalid device function.\n\nApply node that caused the error: GpuAdvancedSubtensor1(GpuElemwise{mul,no_inplace}.0, Elemwise{Cast{int64}}.0)\nToposort index: 68\nInputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]\nInputs shapes: [(880, 500), (15360,)]\nInputs strides: [(500, 1), (8,)]\nInputs values: ['not shown', 'not shown']\nOutputs clients: [[GpuReshape{2}(GpuAdvancedSubtensor1.0, TensorConstant{[ -1 500]})]]\n\nHINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-26-c93cbd64b6a3>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      6\u001b[0m     \u001b[0mnb_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m40\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m     \u001b[0mvalidation_split\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0.1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m     show_accuracy=True)\n\u001b[0m",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/models.pyc\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, show_accuracy, class_weight, sample_weight)\u001b[0m\n\u001b[0;32m    699\u001b[0m                          \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    700\u001b[0m                          \u001b[0mval_f\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mval_f\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mval_ins\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mval_ins\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 701\u001b[1;33m                          shuffle=shuffle, metrics=metrics)\n\u001b[0m\u001b[0;32m    702\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    703\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m128\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/models.pyc\u001b[0m in \u001b[0;36m_fit\u001b[1;34m(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, metrics)\u001b[0m\n\u001b[0;32m    315\u001b[0m                 \u001b[0mbatch_logs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'size'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch_ids\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    316\u001b[0m                 \u001b[0mcallbacks\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_batch_begin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch_index\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_logs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 317\u001b[1;33m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    318\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    319\u001b[0m                     \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Keras-0.3.3-py2.7.egg/keras/backend/theano_backend.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m    444\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    445\u001b[0m         \u001b[1;32massert\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32min\u001b[0m \u001b[1;33m{\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 446\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfunction\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    447\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    448\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    906\u001b[0m                     \u001b[0mnode\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnodes\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mposition_of_error\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    907\u001b[0m                     \u001b[0mthunk\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mthunk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 908\u001b[1;33m                     storage_map=getattr(self.fn, 'storage_map', None))\n\u001b[0m\u001b[0;32m    909\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    910\u001b[0m                 \u001b[1;31m# old-style linkers raise their own exceptions\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/gof/link.pyc\u001b[0m in \u001b[0;36mraise_with_op\u001b[1;34m(node, thunk, exc_info, storage_map)\u001b[0m\n\u001b[0;32m    312\u001b[0m         \u001b[1;31m# extra long error message in that case.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    313\u001b[0m         \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 314\u001b[1;33m     \u001b[0mreraise\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexc_type\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexc_value\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexc_trace\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    315\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    316\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/BS/mmalinow-projects/work/gpuEnv/local/lib/python2.7/site-packages/Theano-0.9.0.dev0-py2.7.egg/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    893\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    894\u001b[0m             \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 895\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0moutput_subset\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[1;32melse\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    896\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    897\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: Cuda error: CudaNdarray_TakeFrom: invalid device function.\n\nApply node that caused the error: GpuAdvancedSubtensor1(GpuElemwise{mul,no_inplace}.0, Elemwise{Cast{int64}}.0)\nToposort index: 68\nInputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]\nInputs shapes: [(880, 500), (15360,)]\nInputs strides: [(500, 1), (8,)]\nInputs values: ['not shown', 'not shown']\nOutputs clients: [[GpuReshape{2}(GpuAdvancedSubtensor1.0, TensorConstant{[ -1 500]})]]\n\nHINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node."
     ]
    }
   ],
   "source": [
    "#== Model training\n",
    "text_rnn_model.fit(\n",
    "    train_x, \n",
    "    train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this Tutorial, you are free to experiment with two examples above.\n",
    "* You can change the size of embedding.\n",
    "* You can change the number of hidden state of RNN.\n",
    "* You can change number of epochs to train.\n",
    "* You can experiment with different batch sizes.\n",
    "* You can modify the models (many RNN layers, deeper classifiers). Use [Keras](http://keras.io) documentation if you are in needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Summary__\n",
    "\n",
    "RNN models, as opposite to BOW, consider order of the words in the question. Moreover, apparently, a substantial number of questions can be answered without any access to image. This can be explained as models learn some specific dataset biases, some of them can be interpreted as common sense knowledge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, please run the cell below to set up a link to the NLTK data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%env NLTK_DATA=/home/ubuntu/data/visual_turing_test/nltk_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ambiguities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to monitor progress on any task, we need to find ways to evaluate the task. Otherwise, we wouldn't know how to compare two architectures, or even worse, we wouldn't even know what our goal is. Moreover, we should also aim at automatic evaluation measures, otherwise reproducibility is questionable, and the costs are high (speed and money; just imagine that you want to evaluate 100 different architectures of yours)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other hand, it's difficult to automatically evaluate holistic tasks such as question answering about images, because of, in just one word, ambiguities. We have ambiguities in naming objects, sometimes due to synonyms, but sometimes due to fuzziness. For instance is 'chair' == 'armchair' or 'chair' != 'armchair' or something in between? Such semantic boundaries become even more fuzzy when we increase the number of categories. We could easily find a mutually exclusive set of 10 different categories, but what if there are 1000 categories, or 10000 categories? Arguably, we cannot think in terms of equivalence class anymore, but rather in terms of similarities. That is 'chair' is semantically more similar to 'armchair', than to 'horse'. This simple example shows the main drawback of a traditional binary evaluation measure Accuracy, which scores 1 if the names are the same and 0 otherwise. So that Acc('chair', 'armchair') == Acc('chair', 'horse'). We use WUPS to handle such ambiguities.\n",
    "\n",
    "We call these ambiguities, word-level ambiguities, but there are other ambiguities that are arguably more difficult to handle. For instance, the same question can be phrased in multiple other ways. The language of spatial relations is also ambiguous (you may be surprised that what you think is on the left, for others may be on the right). Language tends to be also rather vague - we sometimes skip details and resort to common sense. Some such ambiguities are rooted in a culture. A couple of such question-level ambiguities, we handle with Consensus Measure.\n",
    "\n",
    "From an another side, arguably, it's easier to evaluate architectures on DAQUAR than on the Image Captioning tasks. The former restricts the output space to $N$ categories, while it still requires holistic (visual and linguistic) comprehension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wu-Palmer Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given an ontology a Wu-Palmer Similarity between two words (or broader concepts) is a soft measure defined as\n",
    "$$WuP(a,b) := \\frac{lca(a,b)}{depth(a) + depth(b)}$$\n",
    "where $lca(a,b)$ is the least common ancestor of $a$ and $b$, and $depth(a)$ is depth of $a$ in the ontology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![small taxonomy](fig/small_taxonomy.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "What is WuP(Dog, Horse) and WuP(Dog, Dalmatian) according to the ontology above? \n",
    "Can you also calculate Acc(Dog, Horse) and Acc(Dog, Dalmatian)?\n",
    "What are your conclusions?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WUPS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wu-Palmer Similarity depends on a ontology. One popular, large ontology is [WordNet](https://wordnet.princeton.edu). Although Wu-Palmer Similarity may work on shallow ontologies, we are rather interested in ontologies with hundreds or even thousands categories. In indoor scenerio, it turns out that many indoor things share similar levels in the taxomy, and hence Wu-Palmer Similarities are very small between each other.\n",
    "\n",
    "The code below exemplifies the issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import wordnet as wn\n",
    "armchair_synset = wn.synset('armchair.n.01')\n",
    "chair_synset = wn.synset('chair.n.01')\n",
    "wardrobe_synset = wn.synset('wardrobe.n.01')\n",
    "\n",
    "print(armchair_synset.wup_similarity(armchair_synset))\n",
    "print(armchair_synset.wup_similarity(chair_synset))\n",
    "print(armchair_synset.wup_similarity(wardrobe_synset))\n",
    "wn.synset('chair.n.01').wup_similarity(wn.synset('person.n.01'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the code we see that 'armchair' and 'wardrobe' are surprisingly close to each other. It is because, for large ontologies like [WordNet](https://wordnet.princeton.edu), all the indoor things are essentially 'indoor things'.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This issue has motivated us to define thresholded Wu-Palmer Similarity Score, defined as follows\n",
    "$$\n",
    "\\begin{array}{rl}\n",
    "WuP(a,b) & \\text{if}\\; WuP(a,b) \\ge \\tau \\\\\n",
    "0.1 \\cdot WuP(a,b) & \\text{otherwise}\n",
    "\\end{array}\n",
    "$$\n",
    "where $\\tau$ is a hand-chosen threshold. Empirically, we found that $\\tau=0.9$ works fine on DAQUAR.\n",
    "\n",
    "Moreover, since DAQUAR has answers as set answer words, so that 'knife,fork' == 'fork,knife', we have extended the above measure to work with sets. We call it Wu-Palmer Set score, or shortly WUPS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A detailed exposition of WUPS is beyond this tutorial, but a curious reader is encoraged to read the 'Performance Measure' paragraph in [our paper](http://arxiv.org/pdf/1410.0210v4.pdf). Note that the measure in [the paper](http://arxiv.org/pdf/1410.0210v4.pdf) is defined broader, and essentially it abstracts from any particular similarities such as Wu-Palmer Similarity. WUPS at 0.9 is WUPS with threshold $\\tau=0.9$.\n",
    "\n",
    "Although the WUPS is conceptually as we described here, technically, it's slightly different as it also needs to deal with synsets. Thus it's recommended to download the script from [here](http://datasets.d2.mpi-inf.mpg.de/mateusz14visual-turing/calculate_wups.py), or re-implement with caution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Consensus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we won't cover the consensus measure.\n",
    "The curious reader is encouraged to read the 'Human Consensus' in the [Ask Your Neurons paper](https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A few caveats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few caveats with WUPS, especially useful if you want to use the measure to your own dataset.\n",
    "\n",
    "__Lack of coverage__\n",
    "Since WUPS is based on an ontology, not always it recognises words. For instance 'garbage bin' is missing, but 'garbage can' is perfectly fine. You can check it by yourself, either with the source code provided above, or by using [this online script](http://wordnetweb.princeton.edu/perl/webwn).\n",
    "\n",
    "__Synsets__\n",
    "If you execute \n",
    "```python\n",
    "wn.synsets('chair')\n",
    "```\n",
    "you will notice a list with many elements, these elements are [semantically equivalent](https://en.wikipedia.org/wiki/Synonym_ring). You can check their definitions, for instance\n",
    "```python\n",
    "wn.synset('chair.n.03').definition()\n",
    "```\n",
    "indicates a person. Indeed, the following has quite high value\n",
    "```python\n",
    "wn.synset('chair.n.03').wup_similarity(wn.synset('person.n.01'))\n",
    "```\n",
    "but this one has a more preffered low value\n",
    "```python\n",
    "wn.synset('chair.n.01').wup_similarity(wn.synset('person.n.01'))\n",
    "```\n",
    "How to deal with it? In DAQUAR we take an optimistic perspective and always consider the highest similarity score. This works with WUPS 0.9 and a restricted indoor domain with a vocabulary only from the trainin set. This may not be true in other domains though.\n",
    "\n",
    "__Ontology__\n",
    "Since WUPS is based on an ontology, specifically on WordNet, it may give different scores on different ontologies, or even on different versions of the same ontology.\n",
    "\n",
    "__Threshold__\n",
    "A good threshold $\\tau$ is dataset dependent. In our case $\\tau = 0.9$ seems to work well, while $\\tau = 0.0$ is too forgivable and is rather reported due to the 'historical' reasons. However, following our papers, you should still consider to report plain set-based accuracy scores (so that Acc('knife,'fork','fork,knife')==1; it can be computed with WUPS -1 using [our script](http://datasets.d2.mpi-inf.mpg.de/mateusz14visual-turing/calculate_wups.py)) as this metric is widely recognised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WUPS is an evaluation measure that works with sets and word-level ambiguities. Arguably, WUPS at 0.9 is the most practical measure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions - BOW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With more and more iterations we can increase training accuracy, however our goal is to see how well the models generalise. For that, we take a test, previously unknown, set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_text_representation = dp['text'](train_or_test='test')\n",
    "test_raw_x = test_text_representation['x']\n",
    "test_one_hot_x = encode_questions_index(test_raw_x, word2index_x)\n",
    "test_x = sequence.pad_sequences(test_one_hot_x, maxlen=MAXLEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given encoded test questions, we use the maximum likelihood principle to withdraw answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from numpy import argmax\n",
    "# predict the probabilities for every word\n",
    "predictions_scores = text_bow_model.predict([test_x])\n",
    "print(predictions_scores.shape)\n",
    "# follow the maximum likelihood principle, and get the best indices to vocabulary\n",
    "predictions_best = argmax(predictions_scores, axis=-1)\n",
    "print(predictions_best.shape)\n",
    "# decode the predicted indices into word answers\n",
    "predictions_answers = [index2word_y[x] for x in predictions_best]\n",
    "print(len(predictions_answers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate the answers using [WUPS scores](#Evaluation-Measures). For this tutorial, we care only about Accuracy, and WUPS at 0.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.utils import print_metrics\n",
    "test_raw_y = test_text_representation['y']\n",
    "_ = print_metrics.select['wups'](\n",
    "        gt_list=test_raw_y,\n",
    "        pred_list=predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also see the predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from numpy import random\n",
    "test_image_name_list = test_text_representation['img_name']\n",
    "indices_to_see = random.randint(low=0, high=len(test_image_name_list), size=5)\n",
    "for index_now in indices_to_see:\n",
    "    print(test_raw_x[index_now], predictions_answers[index_now])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Do you agree with the answers given above? What are your guesses?\n",
    "Of course,neither you nor the model have seen any images so far.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, what if you actually see the images? \n",
    "```\n",
    "Execute the code below.\n",
    "Do your answers change after seeing the images?\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from matplotlib.pyplot import axis\n",
    "from matplotlib.pyplot import figure\n",
    "from matplotlib.pyplot import imshow\n",
    "\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "\n",
    "%matplotlib inline\n",
    "for index_now in indices_to_see:\n",
    "    image_name_now = test_image_name_list[index_now]\n",
    "    pil_im = Image.open('data/daquar/images/{0}.png'.format(image_name_now), 'r')\n",
    "    fig = figure()\n",
    "    fig.text(.2,.05,test_raw_x[index_now], fontsize=14)\n",
    "    axis('off')\n",
    "    imshow(np.asarray(pil_im))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's also see the ground truths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('question, prediction, ground truth answer')\n",
    "for index_now in indices_to_see:\n",
    "    print(test_raw_x[index_now], predictions_answers[index_now], test_raw_y[index_now])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code above, we have randomly taken questions, so for different executations we may get different answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions - RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Curious how predictions with blind RNN went? \n",
    "\n",
    "This time, we will use the help of Kraino, to make the predictions shorter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.core.model_zoo import word_generator\n",
    "# we first need to add word_generator to _config (we could have done this before, in the Config constructor)\n",
    "# we use maximum likelihood as a word generator\n",
    "text_rnn_model._config.word_generator = word_generator['max_likelihood']\n",
    "predictions_answers = text_rnn_model.decode_predictions(\n",
    "    X=test_x,\n",
    "    temperature=None,\n",
    "    index2word=index2word_y,\n",
    "    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = print_metrics.select['wups'](\n",
    "        gt_list=test_raw_y,\n",
    "        pred_list=predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Visualise question, predicted answers, ground truth answers as before.\n",
    "Check also images.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't go very far using only textual features. Hence, it's now time to consider its visual counterpart.\n",
    "\n",
    "As shown in the figure below, a quite common procedure works as follows:\n",
    "* Use a CNN already pre-trained on some large-scale classification task, most often it is [ImageNet](http://image-net.org) with $1000$ for recognition.\n",
    "* 'Chop off' CNN after some layer. We will use responses of that layer as visual features.\n",
    "\n",
    "In this tutorial, we will use features extracted from the second last $4096$ dimensional layer of [VGG NET-19](http://arxiv.org/pdf/1409.1556.pdf). We have already extracted features in advance using [Caffe](http://caffe.berkeleyvision.org) - another excellent framework for deep learning, particularly good for CNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![features extractor](fig/features_extractor.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please run the cell below in order to get visual features aligned with textual featurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this contains a list of the image names of our interest; \n",
    "# it also makes sure that visual and textual features are aligned correspondingly\n",
    "train_image_names = train_text_representation['img_name']\n",
    "# the name for visual features that we use\n",
    "# CNN_NAME='vgg_net'\n",
    "# CNN_NAME='googlenet'\n",
    "CNN_NAME='fb_resnet'\n",
    "# the layer in CNN that is used to extract features\n",
    "# PERCEPTION_LAYER='fc7'\n",
    "# PERCEPTION_LAYER='pool5-7x7_s1'\n",
    "# PERCEPTION_LAYER='res5c-152'\n",
    "PERCEPTION_LAYER='l2_res5c-152' # l2 prefix since there are l2-normalized visual features\n",
    "\n",
    "train_visual_features = dp['perception'](\n",
    "    train_or_test='train',\n",
    "    names_list=train_image_names,\n",
    "    parts_extractor=None,\n",
    "    max_parts=None,\n",
    "    perception=CNN_NAME,\n",
    "    layer=PERCEPTION_LAYER,\n",
    "    second_layer=None\n",
    "    )\n",
    "train_visual_features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vision+Language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are talking about answering on questions about images, we likely need images too :)\n",
    "\n",
    "Take a look at the figure below one more time. How far can you go by blind guesses? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![challenges](fig/challenges.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's creat an input as a pair of textual and visual features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_input = [train_x, train_visual_features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BOW + Vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with Language Only model, we start from a simpler BOW model that we will combine with [visual features](fig/#Visual Features). Here, we will explore two ways of combining both modalities (circle with 'C' in the figure below): concatenation, and piece-wise multiplication. We will use CNN features extracted from the image, but for the sake of simplicity we won't backprop to fine tune the visual representation (dot line symbolizes the barrier that blocks back-prop in the figure below). Although in our [Ask Your Neurons](http://arxiv.org/abs/1505.01121) fine-tuning the last layer was actually important, benefits of end-to-end training on DAQUAR or larger [VQA](http://visualqa.org) datasets remain an open question. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BOW_vision](fig/BOW_vision_model.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model definition\n",
    "\n",
    "# First we define a model using keras/kraino\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.core import Dropout\n",
    "from keras.layers.core import Layer\n",
    "from keras.layers.core import Merge\n",
    "from keras.layers.core import TimeDistributedMerge\n",
    "from keras.layers.embeddings import Embedding\n",
    "\n",
    "from kraino.core.model_zoo import AbstractSequentialModel\n",
    "from kraino.core.model_zoo import AbstractSingleAnswer\n",
    "from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer\n",
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.keras_extensions import DropMask\n",
    "from kraino.core.keras_extensions import LambdaWithMask\n",
    "from kraino.core.keras_extensions import time_distributed_masked_ave\n",
    "\n",
    "# This model inherits from AbstractSingleAnswer, and so it produces single answer words\n",
    "# To use multiple answer words, you need to inherit from AbstractSequentialMultiplewordAnswer\n",
    "class VisionLanguageBOW(AbstractSequentialModel, AbstractSingleAnswer):\n",
    "    \"\"\"\n",
    "    BOW Language only model that produces single word answers.\n",
    "    \"\"\"\n",
    "    def create(self):\n",
    "        language_model = Sequential()\n",
    "        language_model.add(Embedding(\n",
    "                self._config.input_dim, \n",
    "                self._config.textual_embedding_dim, \n",
    "                mask_zero=True))\n",
    "        language_model.add(LambdaWithMask(\n",
    "                time_distributed_masked_ave, \n",
    "                output_shape=[language_model.output_shape[2]]))\n",
    "        language_model.add(DropMask())\n",
    "        visual_model = Sequential()\n",
    "        if self._config.visual_embedding_dim > 0:\n",
    "            visual_model.add(Dense(\n",
    "                    self._config.visual_embedding_dim,\n",
    "                    input_shape=(self._config.visual_dim,)))\n",
    "        else:\n",
    "            visual_model.add(Layer(input_shape=(self._config.visual_dim,)))\n",
    "        self.add(Merge([language_model, visual_model], mode=self._config.multimodal_merge_mode))\n",
    "        self.add(Dropout(0.5))\n",
    "        self.add(Dense(self._config.output_dim))\n",
    "        self.add(Activation('softmax'))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dimensionality of embeddings\n",
    "EMBEDDING_DIM = 500\n",
    "# kind of multimodal fusion (ave, concat, mul, sum)\n",
    "MULTIMODAL_MERGE_MODE = 'concat'\n",
    "\n",
    "model_config = Config(\n",
    "    textual_embedding_dim=EMBEDDING_DIM,\n",
    "    visual_embedding_dim=0,\n",
    "    multimodal_merge_mode=MULTIMODAL_MERGE_MODE,\n",
    "    input_dim=len(word2index_x.keys()),\n",
    "    output_dim=len(word2index_y.keys()),\n",
    "    visual_dim=train_visual_features.shape[1])\n",
    "model = VisionLanguageBOW(model_config)\n",
    "model.create()\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "model.fit(\n",
    "    train_input, \n",
    "    train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, if we use a piece-wise multiplication to merge both modalities together, we will get better results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#== Model definition\n",
    "\n",
    "# First we define a model using keras/kraino\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.core import Dropout\n",
    "from keras.layers.core import Layer\n",
    "from keras.layers.core import Merge\n",
    "from keras.layers.core import TimeDistributedMerge\n",
    "from keras.layers.embeddings import Embedding\n",
    "\n",
    "from kraino.core.model_zoo import AbstractSequentialModel\n",
    "from kraino.core.model_zoo import AbstractSingleAnswer\n",
    "from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer\n",
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.keras_extensions import DropMask\n",
    "from kraino.core.keras_extensions import LambdaWithMask\n",
    "from kraino.core.keras_extensions import time_distributed_masked_ave\n",
    "\n",
    "# This model inherits from AbstractSingleAnswer, and so it produces single answer words\n",
    "# To use multiple answer words, you need to inherit from AbstractSequentialMultiplewordAnswer\n",
    "class VisionLanguageBOW(AbstractSequentialModel, AbstractSingleAnswer):\n",
    "    \"\"\"\n",
    "    BOW Language only model that produces single word answers.\n",
    "    \"\"\"\n",
    "    def create(self):\n",
    "        language_model = Sequential()\n",
    "        language_model.add(Embedding(\n",
    "                self._config.input_dim, \n",
    "                self._config.textual_embedding_dim, \n",
    "                mask_zero=True))\n",
    "        language_model.add(LambdaWithMask(\n",
    "                time_distributed_masked_ave, \n",
    "                output_shape=[language_model.output_shape[2]]))\n",
    "        language_model.add(DropMask())\n",
    "        visual_model = Sequential()\n",
    "        if self._config.visual_embedding_dim > 0:\n",
    "            visual_model.add(Dense(\n",
    "                    self._config.visual_embedding_dim,\n",
    "                    input_shape=(self._config.visual_dim,)))\n",
    "        else:\n",
    "            visual_model.add(Layer(input_shape=(self._config.visual_dim,)))\n",
    "        self.add(Merge([language_model, visual_model], mode=self._config.multimodal_merge_mode))\n",
    "        self.add(Dropout(0.5))\n",
    "        self.add(Dense(self._config.output_dim))\n",
    "        self.add(Activation('softmax'))\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dimensionality of embeddings\n",
    "EMBEDDING_DIM = 500\n",
    "# kind of multimodal fusion (ave, concat, mul, sum)\n",
    "MULTIMODAL_MERGE_MODE = 'mul'\n",
    "\n",
    "model_config = Config(\n",
    "    textual_embedding_dim=EMBEDDING_DIM,\n",
    "    visual_embedding_dim=EMBEDDING_DIM,\n",
    "    multimodal_merge_mode=MULTIMODAL_MERGE_MODE,\n",
    "    input_dim=len(word2index_x.keys()),\n",
    "    output_dim=len(word2index_y.keys()),\n",
    "    visual_dim=train_visual_features.shape[1])\n",
    "model = VisionLanguageBOW(model_config)\n",
    "model.create()\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')\n",
    "text_image_bow_model = model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "If we merge language and visual features with 'mul', do we need to set both embeddings to have the same number  of dimensions (textual_embedding_dim == visual_embedding_dim)?\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "text_image_bow_model.fit(\n",
    "    train_input, \n",
    "    train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN + Vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will repeat the BOW experiments but with RNN.\n",
    "\n",
    "![LSTM_vision](fig/LSTM_vision_model.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model definition\n",
    "\n",
    "# First we define a model using keras/kraino\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.core import Dropout\n",
    "from keras.layers.core import Layer\n",
    "from keras.layers.core import Merge\n",
    "from keras.layers.core import TimeDistributedMerge\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.recurrent import GRU\n",
    "from keras.layers.recurrent import LSTM\n",
    "\n",
    "from kraino.core.model_zoo import AbstractSequentialModel\n",
    "from kraino.core.model_zoo import AbstractSingleAnswer\n",
    "from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer\n",
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.keras_extensions import DropMask\n",
    "from kraino.core.keras_extensions import LambdaWithMask\n",
    "from kraino.core.keras_extensions import time_distributed_masked_ave\n",
    "\n",
    "# This model inherits from AbstractSingleAnswer, and so it produces single answer words\n",
    "# To use multiple answer words, you need to inherit from AbstractSequentialMultiplewordAnswer\n",
    "class VisionLanguageLSTM(AbstractSequentialModel, AbstractSingleAnswer):\n",
    "    \"\"\"\n",
    "    BOW Language only model that produces single word answers.\n",
    "    \"\"\"\n",
    "    def create(self):\n",
    "        language_model = Sequential()\n",
    "        language_model.add(Embedding(\n",
    "                self._config.input_dim, \n",
    "                self._config.textual_embedding_dim, \n",
    "                mask_zero=True))\n",
    "        #TODO: Replace averaging with RNN (you can choose between LSTM and GRU)\n",
    "#         language_model.add(LambdaWithMask(time_distributed_masked_ave, output_shape=[self.output_shape[2]]))\n",
    "        language_model.add(LSTM(self._config.hidden_state_dim, \n",
    "                      return_sequences=False))\n",
    "\n",
    "        visual_model = Sequential()\n",
    "        if self._config.visual_embedding_dim > 0:\n",
    "            visual_model.add(Dense(\n",
    "                    self._config.visual_embedding_dim,\n",
    "                    input_shape=(self._config.visual_dim,)))\n",
    "        else:\n",
    "            visual_model.add(Layer(input_shape=(self._config.visual_dim,)))\n",
    "        self.add(Merge([language_model, visual_model], mode=self._config.multimodal_merge_mode))\n",
    "        self.add(Dropout(0.5))\n",
    "        self.add(Dense(self._config.output_dim))\n",
    "        self.add(Activation('softmax'))\n",
    "        \n",
    "        \n",
    "# dimensionality of embeddings\n",
    "EMBEDDING_DIM = 500\n",
    "# kind of multimodal fusion (ave, concat, mul, sum)\n",
    "MULTIMODAL_MERGE_MODE = 'sum'\n",
    "\n",
    "model_config = Config(\n",
    "    textual_embedding_dim=EMBEDDING_DIM,\n",
    "    visual_embedding_dim=EMBEDDING_DIM,\n",
    "    hidden_state_dim=EMBEDDING_DIM,\n",
    "    multimodal_merge_mode=MULTIMODAL_MERGE_MODE,\n",
    "    input_dim=len(word2index_x.keys()),\n",
    "    output_dim=len(word2index_y.keys()),\n",
    "    visual_dim=train_visual_features.shape[1])\n",
    "model = VisionLanguageLSTM(model_config)\n",
    "model.create()\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')\n",
    "text_image_rnn_model = model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch Size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, again, let's train the model (if the following cell crashes, please move to the next cell)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "text_image_rnn_model.fit(\n",
    "    train_input, \n",
    "    train_y,\n",
    "    batch_size=5500,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ooops, apparently we run out of memory in our GPUs. Note, how large our batches are! \n",
    "Let's make them much smaller (argument batch_size)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "text_image_rnn_model.fit(\n",
    "    train_input, \n",
    "    train_y,\n",
    "    batch_size=1,\n",
    "    nb_epoch=1,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Please, stop it! Batch size 1 is not good neither. Training is very slow.\n",
    "Let's use standard batch size 512. Please re-run the cell with the model definition, and next run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "text_image_rnn_model.fit(\n",
    "    train_input, \n",
    "    train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=40,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Can you explain both issues regarding the batch size? Why training is impossible in the first case, and very tedious in the second case?\n",
    "\n",
    "When do you get the best performance, with multiplication, concatenation, or summation?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Summary__\n",
    "\n",
    "As previously, using RNN makes the sequence processing order-aware. This time, however, we combine two modalities so that the whole model 'sees' the image. Finally, it's important how both modalities are combined, we have found that piece-wise multiplication outperforms traditional concatenation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Predictions with Vision+Language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions (Features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_image_names = test_text_representation['img_name']\n",
    "test_visual_features = dp['perception'](\n",
    "    train_or_test='test',\n",
    "    names_list=test_image_names,\n",
    "    parts_extractor=None,\n",
    "    max_parts=None,\n",
    "    perception=CNN_NAME,\n",
    "    layer=PERCEPTION_LAYER,\n",
    "    second_layer=None\n",
    "    )\n",
    "test_visual_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_input = [test_x, test_visual_features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions (Bow with Vision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate the Vision+Language architectures as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.core.model_zoo import word_generator\n",
    "# we first need to add word_generator to _config (we could have done this before, in the Config constructor)\n",
    "# we use maximum likelihood as a word generator\n",
    "text_image_bow_model._config.word_generator = word_generator['max_likelihood']\n",
    "predictions_answers = text_image_bow_model.decode_predictions(\n",
    "    X=test_input,\n",
    "    temperature=None,\n",
    "    index2word=index2word_y,\n",
    "    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = print_metrics.select['wups'](\n",
    "        gt_list=test_raw_y,\n",
    "        pred_list=predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions (RNN with Vision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.core.model_zoo import word_generator\n",
    "# we first need to add word_generator to _config (we could have done this before, in the Config constructor)\n",
    "# we use maximum likelihood as a word generator\n",
    "text_image_rnn_model._config.word_generator = word_generator['max_likelihood']\n",
    "predictions_answers = text_image_rnn_model.decode_predictions(\n",
    "    X=test_input,\n",
    "    temperature=None,\n",
    "    index2word=index2word_y,\n",
    "    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = print_metrics.select['wups'](\n",
    "        gt_list=test_raw_y,\n",
    "        pred_list=predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VQA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The models that we have built so far can be transferred to other dataset.\n",
    "Let's consider recently introduced large-scale [VQA](visual question answering) built on top of [COCO](http://mscoco.org). In this section, we will train and evaluate VQA models. Since we are using all pieces introduced before, we will just quickly go into coding. For the sake of simplicity, we will use only BOW architectures, but you are free to experiment with RNN. Please also to pay attention to the comments.\n",
    "\n",
    "Since VQA hides the test data for the purpose of challenge, we will use the publically validation set to evaluate the architectures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VQA Language Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#TODO: Execute the following procedure (Shift+Enter)\n",
    "from kraino.utils import data_provider\n",
    "\n",
    "vqa_dp = data_provider.select['vqa-real_images-open_ended']\n",
    "# VQA has a few answers associated with one question. \n",
    "# We take the most frequently occuring answers (single_frequent).\n",
    "# Formal argument 'keep_top_qa_pairs' allows to filter out rare answers with the associated questions.\n",
    "# We use 0 as we want to keep all question answer pairs, but you can change into 1000 and see how the results differ\n",
    "vqa_train_text_representation = vqa_dp['text'](\n",
    "    train_or_test='train',\n",
    "    answer_mode='single_frequent',\n",
    "    keep_top_qa_pairs=1000)\n",
    "vqa_val_text_representation = vqa_dp['text'](\n",
    "    train_or_test='val',\n",
    "    answer_mode='single_frequent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from toolz import frequencies\n",
    "vqa_train_raw_x = vqa_train_text_representation['x']\n",
    "vqa_train_raw_y = vqa_train_text_representation['y']\n",
    "vqa_val_raw_x = vqa_val_text_representation['x']\n",
    "vqa_val_raw_y = vqa_val_text_representation['y']\n",
    "# we start from building the frequencies table\n",
    "vqa_wordcount_x = frequencies(' '.join(vqa_train_raw_x).split(' '))\n",
    "# we can keep all answer words in the answer as a class\n",
    "# therefore we use an artificial split symbol '{' to not split the answer into words\n",
    "# you can see the difference if you replace '{' with ' ' and print vqa_wordcount_y\n",
    "vqa_wordcount_y = frequencies('{'.join(vqa_train_raw_y).split('{'))\n",
    "vqa_wordcount_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language-Only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing import sequence\n",
    "from kraino.utils.input_output_space import build_vocabulary\n",
    "from kraino.utils.input_output_space import encode_questions_index\n",
    "from kraino.utils.input_output_space import encode_answers_one_hot\n",
    "MAXLEN=30\n",
    "vqa_word2index_x, vqa_index2word_x = build_vocabulary(this_wordcount = vqa_wordcount_x)\n",
    "vqa_word2index_y, vqa_index2word_y = build_vocabulary(this_wordcount = vqa_wordcount_y)\n",
    "vqa_train_x = sequence.pad_sequences(encode_questions_index(vqa_train_raw_x, vqa_word2index_x), maxlen=MAXLEN)\n",
    "vqa_val_x = sequence.pad_sequences(encode_questions_index(vqa_val_raw_x, vqa_word2index_x), maxlen=MAXLEN)\n",
    "vqa_train_y, _ = encode_answers_one_hot(\n",
    "    vqa_train_raw_y, \n",
    "    vqa_word2index_y, \n",
    "    answer_words_delimiter=vqa_train_text_representation['answer_words_delimiter'],\n",
    "    is_only_first_answer_word=True,\n",
    "    max_answer_time_steps=1)\n",
    "vqa_val_y, _ = encode_answers_one_hot(\n",
    "    vqa_val_raw_y, \n",
    "    vqa_word2index_y, \n",
    "    answer_words_delimiter=vqa_train_text_representation['answer_words_delimiter'],\n",
    "    is_only_first_answer_word=True,\n",
    "    max_answer_time_steps=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.model_zoo import word_generator\n",
    "# We are re-using the BlindBOW mode\n",
    "# Please make sure you have run the cell with the class definition\n",
    "# VQA is larger, so we can increase the dimensionality of the embedding\n",
    "vqa_model_config = Config(\n",
    "    textual_embedding_dim=1000,\n",
    "    input_dim=len(vqa_word2index_x.keys()),\n",
    "    output_dim=len(vqa_word2index_y.keys()),\n",
    "    word_generator = word_generator['max_likelihood'])\n",
    "vqa_text_bow_model = BlindBOW(vqa_model_config)\n",
    "vqa_text_bow_model.create()\n",
    "vqa_text_bow_model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vqa_text_bow_model.fit(\n",
    "    vqa_train_x, \n",
    "    vqa_train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=10,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we first need to add word_generator to _config (we could have done this before, in the Config constructor)\n",
    "# we use maximum likelihood as a word generator\n",
    "vqa_predictions_answers = vqa_text_bow_model.decode_predictions(\n",
    "    X=vqa_val_x,\n",
    "    temperature=None,\n",
    "    index2word=vqa_index2word_y,\n",
    "    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Using VQA is unfortunately not that transparent\n",
    "# we need extra VQA object.\n",
    "vqa_vars = {\n",
    "    'question_id':vqa_val_text_representation['question_id'],\n",
    "    'vqa_object':vqa_val_text_representation['vqa_object'],\n",
    "    'resfun': \n",
    "        lambda x: \\\n",
    "            vqa_val_text_representation['vqa_object'].loadRes(x, vqa_val_text_representation['questions_path'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.utils import print_metrics\n",
    "\n",
    "\n",
    "_ = print_metrics.select['vqa'](\n",
    "        gt_list=vqa_val_raw_y,\n",
    "        pred_list=vqa_predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=vqa_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VQA Language+Vision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the name for visual features that we use\n",
    "VQA_CNN_NAME='vgg_net'\n",
    "# VQA_CNN_NAME='googlenet'\n",
    "# the layer in CNN that is used to extract features\n",
    "VQA_PERCEPTION_LAYER='fc7'\n",
    "# PERCEPTION_LAYER='pool5-7x7_s1'\n",
    "\n",
    "vqa_train_visual_features = vqa_dp['perception'](\n",
    "    train_or_test='train',\n",
    "    names_list=vqa_train_text_representation['img_name'],\n",
    "    parts_extractor=None,\n",
    "    max_parts=None,\n",
    "    perception=VQA_CNN_NAME,\n",
    "    layer=VQA_PERCEPTION_LAYER,\n",
    "    second_layer=None\n",
    "    )\n",
    "vqa_train_visual_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vqa_val_visual_features = vqa_dp['perception'](\n",
    "    train_or_test='val',\n",
    "    names_list=vqa_val_text_representation['img_name'],\n",
    "    parts_extractor=None,\n",
    "    max_parts=None,\n",
    "    perception=VQA_CNN_NAME,\n",
    "    layer=VQA_PERCEPTION_LAYER,\n",
    "    second_layer=None\n",
    "    )\n",
    "vqa_val_visual_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.core.model_zoo import Config\n",
    "from kraino.core.model_zoo import word_generator\n",
    "\n",
    "# dimensionality of embeddings\n",
    "VQA_EMBEDDING_DIM = 1000\n",
    "# kind of multimodal fusion (ave, concat, mul, sum)\n",
    "VQA_MULTIMODAL_MERGE_MODE = 'mul'\n",
    "\n",
    "vqa_model_config = Config(\n",
    "    textual_embedding_dim=VQA_EMBEDDING_DIM,\n",
    "    visual_embedding_dim=VQA_EMBEDDING_DIM,\n",
    "    multimodal_merge_mode=VQA_MULTIMODAL_MERGE_MODE,\n",
    "    input_dim=len(vqa_word2index_x.keys()),\n",
    "    output_dim=len(vqa_word2index_y.keys()),\n",
    "    visual_dim=vqa_train_visual_features.shape[1],\n",
    "    word_generator=word_generator['max_likelihood'])\n",
    "vqa_text_image_bow_model = VisionLanguageBOW(vqa_model_config)\n",
    "vqa_text_image_bow_model.create()\n",
    "vqa_text_image_bow_model.compile(\n",
    "    loss='categorical_crossentropy', \n",
    "    optimizer='adam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vqa_train_input = [vqa_train_x, vqa_train_visual_features]\n",
    "vqa_val_input = [vqa_val_x, vqa_val_visual_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#== Model training\n",
    "vqa_text_image_bow_model.fit(\n",
    "    vqa_train_input, \n",
    "    vqa_train_y,\n",
    "    batch_size=512,\n",
    "    nb_epoch=10,\n",
    "    validation_split=0.1,\n",
    "    show_accuracy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we first need to add word_generator to _config (we could have done this before, in the Config constructor)\n",
    "# we use maximum likelihood as a word generator\n",
    "vqa_predictions_answers = vqa_text_image_bow_model.decode_predictions(\n",
    "    X=vqa_val_input,\n",
    "    temperature=None,\n",
    "    index2word=vqa_index2word_y,\n",
    "    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Using VQA is unfortunately not that transparent\n",
    "# we need extra VQA object.\n",
    "vqa_vars = {\n",
    "    'question_id':vqa_val_text_representation['question_id'],\n",
    "    'vqa_object':vqa_val_text_representation['vqa_object'],\n",
    "    'resfun': \n",
    "        lambda x: \\\n",
    "            vqa_val_text_representation['vqa_object'].loadRes(x, vqa_val_text_representation['questions_path'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from kraino.utils import print_metrics\n",
    "\n",
    "\n",
    "_ = print_metrics.select['vqa'](\n",
    "        gt_list=vqa_val_raw_y,\n",
    "        pred_list=vqa_predictions_answers,\n",
    "        verbose=1,\n",
    "        extra_vars=vqa_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kraino"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purpose of fast experimentations on Visual Turing Test, we have prepared Kraino that builds on top of Keras.\n",
    "In this short section, you will see how to use it from a command line, example by example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kraino on DAQUAR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a blind model with a temporal fusion of the question (equivalent of BOW).\n",
    "\n",
    "One era consists of max_epoch epochs, at the end of era we gather some statistics such as the model performance, or we dump the model weights. Since calculating wups scores is slow, we use 5 epoch before we output such information. In the example below we also use 5 eras, so in total we perform 25 epochs.\n",
    "\n",
    "In the example below we monitor wups scores on test set (--metric=wups, --verbosity=monitor_test_metric), but we also use 10% of training data for validation (--validation_split=0.1).\n",
    "__Please remember to pick up the model based on validation set, NOT the test set!___\n",
    "\n",
    "We use one_hot vector representation. As an alternative we could use --word_representation=dense with a pre-trained embedding such as Word2Vec or Glove. You need to download both pre-trained embeddings.\n",
    "\n",
    "The code below may be slow due to WUPS calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --dataset=daquar-triples --model=sequential-blind-temporal_fusion-single_answer --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=2 --verbosity=monitor_test_metric --word_representation=one_hot "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe we should use smaller embedding layer with --embedding_size=500 (500 dimensions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --textual_embedding_size=500 --dataset=daquar-triples --model=sequential-blind-temporal_fusion-single_answer --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=2 --verbosity=monitor_test_metric --word_representation=one_hot "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we replace the temporal by the recurrent fussion (LSTM) with --model=sequential-blind-__reccurent_fusion__-single_answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --dataset=daquar-triples --model=sequential-blind-recurrent_fusion-single_answer --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=1 --verbosity=monitor_test_metric --word_representation=one_hot "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily replace LSTM by GRU as a question encoder with --text_encoder=gru."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --text_encoder=gru --dataset=daquar-triples --model=sequential-blind-recurrent_fusion-single_answer --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=1 --verbosity=monitor_test_metric --word_representation=one_hot  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use 1 dimensional CNN to represent questions with --model=sequential-blind-cnn_fusion-single_answer_with_temporal_fusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --text_encoder=gru --dataset=daquar-triples --model=sequential-blind-cnn_fusion-single_answer_with_temporal_fusion --temporal_fusion=sum --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=1 --verbosity=monitor_test_metric --word_representation=one_hot  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can combine GRU with visual CNN.\n",
    "\n",
    "--model=sequential-multimodal-recurrent_fusion-at_last_timestep_multimodal_fusion-single_answer \n",
    "\n",
    "--multimodal_fusion=mul"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --text_encoder=gru --dataset=daquar-triples --model=sequential-multimodal-recurrent_fusion-at_last_timestep_multimodal_fusion-single_answer --temporal_fusion=sum --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=1 --verbosity=monitor_test_metric --word_representation=one_hot --multimodal_fusion=mul "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or use the above with Resnet (by default it's GoogLeNet) with piece-wise summation. \n",
    "We use parameters: --perception=fb_resnet --perception_layer=l2_res5c-152."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --text_encoder=gru --dataset=daquar-triples --model=sequential-multimodal-recurrent_fusion-at_last_timestep_multimodal_fusion-single_answer --temporal_fusion=sum --validation_split=0.1 --metric=wups --max_epoch=20 --max_era=1 --verbosity=monitor_test_metric --word_representation=one_hot --multimodal_fusion=sum --perception=fb_resnet --perception_layer=l2_res5c-152 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But there are more possibilities.\n",
    "Unfortunately the documentation is not ready yet, but you are welcome to experiment with different settings.\n",
    "Interestingly, some subsets of the parameters don't go well with each other.\n",
    "\n",
    "To see available models, check kraino/core/model_zoo.py.\n",
    "\n",
    "To see available command-line parameters together with default values, check kraino/utils/parsers.py."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kraino on VQA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can, however, also work with VQA.\n",
    "\n",
    "We need to switch dataset (--dataset=vqa-real_images-open_ended), metric (--metric=vqa), and add the answer mode (--vqa_answer_mode=single_frequent).\n",
    "Moreover, we also truncate question answer pairs according to 2000 most frequent answers, and --use_whole_answer_as_answer_word as we want to treat answer 'yellow cab' as the answer, and won't split into words.\n",
    "\n",
    "If the following code is too slow, try using BOW (--model=sequential-blind-temporal_fusion-single_answer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --dataset=vqa-real_images-open_ended --model=sequential-blind-recurrent_fusion-single_answer --vqa_answer_mode=single_frequent --metric=vqa --max_epoch=10 --max_era=1 --verbosity=monitor_val_metric --word_representation=one_hot --number_most_frequent_qa_pairs=2000 --use_whole_answer_as_answer_word "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use Vision + Language model. If there are memory problems try either smaller batches (--batch_size=...), smaller model (--hidden_state_size or --textual_embedding_size), or use BOW model (--model=equential-multimodal-temporal_fusion-single_answer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! python neural_solver.py --dataset=vqa-real_images-open_ended --model=sequential-multimodal-recurrent_fusion-at_last_timestep_multimodal_fusion-single_answer --vqa_answer_mode=single_frequent --metric=vqa --max_epoch=10 --max_era=1 --verbosity=monitor_val_metric --word_representation=one_hot --number_most_frequent_qa_pairs=2000 --use_whole_answer_as_answer_word --perception=fb_resnet --perception_layer=l2_res5c-152 --multimodal_fusion=mul "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Data and Task Understanding.\n",
    " * Try to find by yourself how difficult/easy is to answer questions without looking into images (do this on DAQUAR or VQA). If you are better than our models, that's great. If you are worse, that's fine, our models, even the blind ones, were trained to answer on this particular dataset.\n",
    " * Take a look at more images, and questions. Can you recognise other challenges that machines can potentially face off? Can you classify the new challenges? \n",
    " * Experiment with evaluation measures. For instance Wu-Palmer Similarities with other categories, or look into [WUPS source code](http://datasets.d2.mpi-inf.mpg.de/mateusz14visual-turing/calculate_wups.py), or check [Consensus](http://datasets.d2.mpi-inf.mpg.de/mateusz15neural_qa/compute_consensus.py). You are also encouraged to check 'Performance Measure' in the [Multiworld paper](http://arxiv.org/pdf/1410.0210v4.pdf), 'Quantifying the Performance of Holistic Architectures in the [Towards Visual Turing Challenge](http://arxiv.org/pdf/1410.8027v3.pdf), and 'Human Consensus' in the [Ask Your Neurons paper](https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf).\n",
    " * You can experiment with New Predictions sections. Particularly, [the last predictions section](#New-Predictions-with-Vision+Language) was quite short. For instance, you can visualise the predictions, similarly to what you did [here](#New-Predictions). Maybe, you can dump a file with predictions, and come up with new conclusions by inspecting it.\n",
    " * Look into [VQA](http://visualqa.org). [Check images, and question answer pairs](http://visualqa.org/browser/). What are the differences between VQA and DAQUAR?\n",
    "* Experiment with the provided code.\n",
    " * More RNN layers, deeper classifiers.\n",
    " * Different RNN models (GRU, LSTM, your own?).\n",
    " * Using the best model found on a validation set. For this you may be willing to pass the [checkpoint callback](http://keras.io/callbacks/#modelcheckpoint) to the fit function as [this example suggests](http://keras.io/callbacks/#example-model-checkpoints).\n",
    " * Recognise and change hyperparameters such as dimensionality of embeddings or the number of hidden units.\n",
    " * Different ways to fuse two modalities (concat, ave, ...). If you use many RNN layers, you can fuse the modalities at different levels.\n",
    " * Investigate different visual features\n",
    "   * vgg_net: fc6, fc7, fc8\n",
    "   * googlenet: loss3-classifier, pool5-7x7_s1\n",
    " * If in needs, please consult with the [official documentation](keras.io).\n",
    "* Experiments with Keras.\n",
    "* Experiments with [Kraino](#Kraino). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Research Opportunities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* __Global Representation__ So far we have been using so called global representations of the images. Such representations may destroy too much information, and so we should consider a fine-grained alternative. Maybe we should use detections, or attention models. The latter becomes recently quite successful in answering questions about images. However, there is still a hope for global representations if they are trained end-to-end for the task. Recall that our global representation is extracted from CNN trained on different dataset (ImageNet), and for different task (object classification). \n",
    "* __3D Scene Representation__ Most of current approaches, and all neural approaches, are trained on 2D images. However, it seems that some spatial relations such as 'behind' may need 3d representation of the scene. Luckily, DAQUAR is built on [NYU-Depth dataset](http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) that provides both modes (2D images, and 3D depth). The question if such extra information helps remains open.\n",
    "* __Recurrent Neural Networks__ There is disturbingly small gap between BOW and RNN models. As we have seen before, some questions clearly require an order, but such questions at the same time become longer, semantically more difficult, and require better visual understanding of the world. To handle them we may need other RNNs architectures, or better ways of fusing two modalities, or better __Global Representation__.\n",
    "* __Logical Reasoning__ There are few questions that require a bit more sophisticated logical reasoning such as negation. Can Reccurent Neural Networks learn such logical operators? What about compositionality of the language?\n",
    "* __Language + Vision__ There is too small gap between Language Only and Vision + Language models. But clearly, we need pictures to answer questions about images. So what is missing here? Is it due to __Global Representation__, __3D Scene Representation__ or there is something missing in fusing two modalities?\n",
    "* __Learning from Few Examples__ In the Visual Turint Test, many questions are quite unique. But then how the models can generalise to new questions? What if a question is completely new, but its parts have been already observed (compositionality)? Can models guess the meaning of a new word from its context?\n",
    "* __Ambiguities__ How to deal with ambiguities? They are all inherent in the task, so cannot be just ignored, and should be incorporated into question answering methods as well as evaluation metrics. \n",
    "* __Evaluation Measures__ Although we have WUPS and Consensus, both are far from being perfect. Consensus has higher annotation cost for ambiguous tasks, and is unclear how to formally define good consensus measure. WUPS is an ontology dependent, but can we build one complete ontology that covers all cases?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# External Links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * Visual Turing Test - project webpage: [https://www.d2.mpi-inf.mpg.de/visual-turing-challenge](https://www.d2.mpi-inf.mpg.de/visual-turing-challenge)\n",
    " * Ask Your Neurons - the main inspiration for this tutorial. [https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf](https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf)\n",
    " * Multiworld Approach - our first paper on Visual Turing Test and DAQUAR. It also describes a symbolic approach to handle the challenge. [http://arxiv.org/pdf/1410.0210v4.pdf](http://arxiv.org/pdf/1410.0210v4.pdf)\n",
    " * Towards a Visual Turing Challenge - we hope that it's a nice and accessible introduction to the task. [http://arxiv.org/abs/1410.8027](http://arxiv.org/abs/1410.8027). Its more compact version: [http://arxiv.org/abs/1501.03302](http://arxiv.org/abs/1501.03302)\n",
    " * My [talk from the ICCV'15 conference](http://videolectures.net/iccv2015_malinowski_your_neurons/) together with the [slides](http://datasets.d2.mpi-inf.mpg.de/mateusz15neural_qa/ask_your_neurons-slides-low_res.pdf)\n",
    " * VQA - a complementary, large-scale dataset for Visual Question Answering. Project webpage: [http://visualqa.org](http://visualqa.org)\n",
    " * Image Question Answering - another large-scale dataset for Image Question Answering. Project webpage [http://www.cs.toronto.edu/~mren/imageqa/](http://www.cs.toronto.edu/~mren/imageqa/)\n",
    " * Learning to Answer Questions about Images - similar to Ask Your Neurons model. [http://www.hangli-hl.com/uploads/3/4/4/6/34465961/ma_et_al_aaai_2016.pdf](http://www.hangli-hl.com/uploads/3/4/4/6/34465961/ma_et_al_aaai_2016.pdf)\n",
    " * A nice introduction to VQA with working source codes, also in Keras. [https://avisingh599.github.io/deeplearning/visual-qa/](https://avisingh599.github.io/deeplearning/visual-qa/)\n",
    " * In the Tutorial we use the [facebook residual net features](http://torch.ch/blog/2016/02/04/resnets.html)\n",
    " * DAQUAR uses NYU-Depth images, for more information please take a look [here](http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 25.04.2016: Added residual net features, more on Kraino, made it publicly available\n",
    "1. 20.03.2016: 2nd Summer School on Integrating Vision & Language: Deep Learning"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}