5_word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec: Obtain word embeddings\n",
    "\n",
    "## 0. Introduction\n",
    "\n",
    "**Word2vec** is the tool for generating the distributed representation of words, which is proposed by Mikolov et al[[1]](#1). When the tool assigns a real-valued vector to each word, the closer the meanings of the words, the greater similarity the vectors will indicate.\n",
    "\n",
    "**Distributed representation** means assigning a real-valued vector for each word and representing the word by the vector. When representing a word by distributed representation, we call the vector **word embeddings**. In this notebook, we aim at explaining how to get the word embeddings from Penn Tree Bank dataset.\n",
    "\n",
    "Let's think about what the meaning of word is. Since we are human, so we can understand that the words \"animal\" and \"dog\" are deeply related each other. But what information will Word2vec use to learn the vectors for words? The words \"animal\" and \"dog\" should have similar vectors, but the words \"food\" and \"dog\" should be far from each other. How to know the features of those words automatically?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Basic Idea\n",
    "\n",
    "Word2vec learns the similarity of word meanings from simple information. It learns the representation of words from sentences. The core idea is based on the assumption that the meaning of a word is affected by the words around it. This idea follows **distributional hypothesis**[[2]](#2).\n",
    "\n",
    "The word we focus on to learn its representation is called **\"center word\"**, and the words around it are called **\"context words\"**. Depending on the window size `C` determines the number of context words which is considered.\n",
    "\n",
    "Here, let's see the algorithm by using an example sentence: \"**The cute cat jumps over the lazy dog.**\"\n",
    "\n",
    "- All of the following figures consider \"cat\" as the center word.\n",
    "- According to the window size `C`, you can see that the number of context words is changed.\n",
    "\n",
    "![](center_context_word.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Main Algorithm\n",
    "\n",
    "Word2vec, the tool for creating the word embeddings, is actually built with two models, which are called **Skip-gram** and **CBoW**.\n",
    "\n",
    "To explain the models with the figures below, we will use the following symbols.\n",
    "\n",
    "| Symbol    | Definition                                               |\n",
    "| --------: | :------------------------------------------------------- |\n",
    "| $|\\mathcal{V}|$       | The size of vocabulary                                   |\n",
    "| $D$       | The size of embedding vector                             |\n",
    "| ${\\bf v}_t$     | A one-hot center word vector                             |\n",
    "| $V_{\\pm C}$ | A set of $C$ context vectors around ${\\bf v}_t$, namely, $\\{{\\bf v}_{t+c}\\}_{c=-C}^C \\backslash {\\bf v}_t$        |\n",
    "| ${\\bf l}_H$     | An embedding vector of an input word vector              |\n",
    "| ${\\bf l}_O$     | An output vector of the network                          |\n",
    "| ${\\bf W}_H$     | The embedding matrix for inputs                          |\n",
    "| ${\\bf W}_O$     | The embedding matrix for outputs                         |\n",
    "\n",
    "**Note**\n",
    "\n",
    "Using **negative sampling** or **hierarchical softmax** for the loss function is very common, however, in this notebook, we will use the **softmax over all words** and skip the other variants for the sake of simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Skip-gram\n",
    "\n",
    "This model learns to predict context words $V_{t \\pm C}$ when a center word ${\\bf v}_t$ is given. In the model, each row of the embedding matrix for input $W_H$ becomes a word embedding of each word.\n",
    "\n",
    "When you input a center word ${\\bf v}_t$ into the network, you can predict one of context words $\\hat{\\bf v}_{t+i} \\in V_{t \\pm C}$ as follows:\n",
    "\n",
    "1. Calculate an embedding vector of the input center word vector: ${\\bf l}_H = {\\bf W}_H {\\bf v}_t$\n",
    "2. Calculate an output vector of the embedding vector: ${\\bf l}_O = {\\bf W}_O {\\bf l}_H$\n",
    "3. Calculate a probability vector of a context word: $\\hat{\\bf v}_{t+i} = \\text{softmax}({\\bf l}_O)$\n",
    "\n",
    "Each element of the $|\\mathcal{V}|$-dimensional vector $\\hat{\\bf v}_{t+i}$ is a probability that a word in the vocabulary turns out to be a context word at position $i$. So, the probability $p({\\bf v}_{t+i} \\mid {\\bf v}_t)$ can be estimated by a dot product of the one-hot vector ${\\bf v}_{t+i}$ which represents the actual word at the position $i$ and the output vector $\\hat{\\bf v}_{t+i}$.\n",
    "\n",
    "$p({\\bf v}_{t+i} \\mid {\\bf v}_t) = {\\bf v}_{t+i}^T \\hat{\\bf v}_{t+i}$\n",
    "\n",
    "The loss function for all the context words $V_{t \\pm C}$ given a center word ${\\bf v}_t$ is defined as following:\n",
    "\n",
    "$\n",
    "\\begin{eqnarray}\n",
    "L(V_{t \\pm C} | {\\bf v}_t; {\\bf W}_H, {\\bf W}_O)\n",
    "&=& \\sum_{V_{t \\pm C}} -\\log\\left(p({\\bf v}_{t+i} \\mid {\\bf v}_t)\\right) \\\\\n",
    "&=& \\sum_{V_{t \\pm C}} -\\log({\\bf v}_{t+i}^T \\hat{\\bf v}_{t+i})\n",
    "\\end{eqnarray}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Continuous Bag of Words (CBoW)\n",
    "\n",
    "This model learns to predict the center word ${\\bf v}_t$ when context words $V_{t \\pm C}$ is given.\n",
    "\n",
    "When you give a set of context words $V_{t \\pm C}$ to the network, you can estimate the probability of the center word $\\hat{v}_t$ as follows:\n",
    "\n",
    "1. Calculate a mean embedding vector over all context words: ${\\bf l}_H = \\frac{1}{2C} \\sum_{V_{t \\pm C}} {\\bf W}_H {\\bf v}_{t+i}$\n",
    "2. Calculate an output vector: ${\\bf l}_O = {\\bf W}_O {\\bf l}_H$\n",
    "3. Calculate an probability vector: $\\hat{\\bf v}_t = \\text{softmax}({\\bf l}_O)$\n",
    "\n",
    "Each element of $\\hat{\\bf v}_t$ is a probability that a word in the vocabulary is considered as the center word. So, the prediction $p({\\bf v}_t \\mid V_{t \\pm C})$ can be calculated by ${\\bf v}_t^T \\hat{\\bf v}_t$, where ${\\bf v}_t$ denots the one-hot vector of the actual center word vector in the sentence from the dataset.\n",
    "\n",
    "The loss function for the center word prediction is defined as follows:\n",
    "\n",
    "$\n",
    "\\begin{eqnarray}\n",
    "L({\\bf v}_t|V_{t \\pm C}; W_H, W_O)\n",
    "&=& -\\log(p({\\bf v}_t|V_{t \\pm C})) \\\\\n",
    "&=& -\\log({\\bf v}_t^T \\hat{\\bf v}_t)\n",
    "\\end{eqnarray}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Details of skip-gram\n",
    "\n",
    "In this notebook, we mainly explain skip-gram model because\n",
    "\n",
    "1. It is easier to understand the algorithm than CBoW.\n",
    "2. Even if the number of words increases, the accuracy is largely maintained. So, it is more scalable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's think about a concrete example of calculating skip-gram under this setup:\n",
    "\n",
    "* The size of vocabulary $|\\mathcal{V}|$ is 10.\n",
    "* The size of embedding vector $D$ is 2.\n",
    "* Center word is \"dog\".\n",
    "* Context word is \"animal\".\n",
    "\n",
    "Since there should be more than one context words, repeat the following process for each context word.\n",
    "\n",
    "1. The one-hot vector of \"dog\" is `[0 0 1 0 0 0 0 0 0 0]` and you input it as the center word.\n",
    "2. The third row of embedding matrix ${\\bf W}_H$ is used for the word embedding of \"dog\" ${\\bf l}_H$.\n",
    "3. Then multiply ${\\bf W}_O$ with ${\\bf l}_H$ to obtain the output vector ${\\bf l}_O$\n",
    "4. Give ${\\bf l}_O$ to the softmax function to make it a predicted probability vector $\\hat{\\bf v}_{t+c}$ for a context word at the position $c$.\n",
    "5. Calculate the error between $\\hat{\\bf v}_{t+c}$ and the one-hot vector of \"animal\"; `[1 0 0 0 0 0 0 0 0 0 0]`.\n",
    "6. Propagate the error back to the network to update the parameters.\n",
    "\n",
    "![](skipgram_detail.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Implementation of skip-gram in Chainer\n",
    "\n",
    "There is an example of Word2vec in the official repository of Chainer, so we will explain how to implement skip-gram based on this: [chainer/examples/word2vec](https://github.com/chainer/chainer/tree/master/examples/word2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Preparation\n",
    "\n",
    "First, let's import necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import argparse\n",
    "import collections\n",
    "\n",
    "import numpy as np\n",
    "import six\n",
    "\n",
    "import chainer\n",
    "from chainer import cuda\n",
    "import chainer.functions as F\n",
    "import chainer.initializers as I\n",
    "import chainer.links as L\n",
    "import chainer.optimizers as O\n",
    "from chainer import reporter\n",
    "from chainer import training\n",
    "from chainer.training import extensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Define a skip-gram model\n",
    "\n",
    "Next, let's define a network for skip-gram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SkipGram(chainer.Chain):\n",
    "\n",
    "    def __init__(self, n_vocab, n_units):\n",
    "        super().__init__()\n",
    "        with self.init_scope():\n",
    "            self.embed = L.EmbedID(\n",
    "                n_vocab, n_units, initialW=I.Uniform(1. / n_units))\n",
    "            self.out = L.Linear(n_units, n_vocab, initialW=0)\n",
    "\n",
    "    def __call__(self, x, context):\n",
    "        e = self.embed(context)\n",
    "        shape = e.shape\n",
    "        x = F.broadcast_to(x[:, None], (shape[0], shape[1]))\n",
    "        e = F.reshape(e, (shape[0] * shape[1], shape[2]))\n",
    "        x = F.reshape(x, (shape[0] * shape[1],))\n",
    "        center_predictions = self.out(e)\n",
    "        loss = F.softmax_cross_entropy(center_predictions, x)\n",
    "        reporter.report({'loss': loss}, self)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "- The weight matrix `self.embed.W` is the embbeding matrix for input vector `x`.\n",
    "- `__call__` takes the word ID of a center word `x` and word IDs of context words `contexts` as inputs, and outputs the error calculated by the loss function `softmax_cross_entropy`.\n",
    "- Note that the initial shape of `x` and `contexts` are `(batch_size,)` and `(batch_size, n_context)`, respectively.\n",
    "- The `batch_size` means the size of mini-batch, and `n_context` means the number of context words.\n",
    "\n",
    "First, we obtain the embedding vectors of `contexts` by `e = self.embed(contexts)`. \n",
    "\n",
    "Then `F.broadcast_to(x[:, None], (shape[0], shape[1]))` performs broadcasting of `x` (`(batch_size,)`) to `(batch_size, n_context)` by copying the same value `n_context` time to fill the second axis, and then the broadcasted `x` is reshaped into 1-D vector `(batchsize * n_context,)` while `e` is reshaped to `(batch_size * n_context, n_units)`.\n",
    "\n",
    "In skip-gram model, predicting a context word from the center word is the same as predicting the center word from a context word because the center word is always a context word when considering the context word as a center word. So, we create `batch_size * n_context` center word predictions by applying `self.out` linear layer to the embedding vectors of context words. Then, calculate softmax cross entropy between the broadcasted center word ID `x` and the predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Prepare dataset and iterator\n",
    "\n",
    "Let's retrieve the Penn Tree Bank (PTB) dataset by using Chainer's dataset utility `get_ptb_words()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train, val, _ = chainer.datasets.get_ptb_words()\n",
    "n_vocab = max(train) + 1  # The minimum word ID is 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then define an iterator to make mini-batches that contain a set of center words with their context words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class WindowIterator(chainer.dataset.Iterator):\n",
    "\n",
    "    def __init__(self, dataset, window, batch_size, repeat=True):\n",
    "        self.dataset = np.array(dataset, np.int32)\n",
    "        self.window = window\n",
    "        self.batch_size = batch_size\n",
    "        self._repeat = repeat\n",
    "\n",
    "        self.order = np.random.permutation(\n",
    "            len(dataset) - window * 2).astype(np.int32)\n",
    "        self.order += window\n",
    "        self.current_position = 0\n",
    "        self.epoch = 0\n",
    "        self.is_new_epoch = False\n",
    "\n",
    "    def __next__(self):\n",
    "        if not self._repeat and self.epoch > 0:\n",
    "            raise StopIteration\n",
    "\n",
    "        i = self.current_position\n",
    "        i_end = i + self.batch_size\n",
    "        position = self.order[i: i_end]\n",
    "        w = np.random.randint(self.window - 1) + 1\n",
    "        offset = np.concatenate([np.arange(-w, 0), np.arange(1, w + 1)])\n",
    "        pos = position[:, None] + offset[None, :]\n",
    "        context = self.dataset.take(pos)\n",
    "        center = self.dataset.take(position)\n",
    "\n",
    "        if i_end >= len(self.order):\n",
    "            np.random.shuffle(self.order)\n",
    "            self.epoch += 1\n",
    "            self.is_new_epoch = True\n",
    "            self.current_position = 0\n",
    "        else:\n",
    "            self.is_new_epoch = False\n",
    "            self.current_position = i_end\n",
    "\n",
    "        return center, context\n",
    "\n",
    "    @property\n",
    "    def epoch_detail(self):\n",
    "        return self.epoch + float(self.current_position) / len(self.order)\n",
    "\n",
    "    def serialize(self, serializer):\n",
    "        self.current_position = serializer('current_position',\n",
    "                                           self.current_position)\n",
    "        self.epoch = serializer('epoch', self.epoch)\n",
    "        self.is_new_epoch = serializer('is_new_epoch', self.is_new_epoch)\n",
    "        if self._order is not None:\n",
    "            serializer('_order', self._order)\n",
    "\n",
    "def convert(batch, device):\n",
    "    center, context = batch\n",
    "    if device >= 0:\n",
    "        center = cuda.to_gpu(center)\n",
    "        context = cuda.to_gpu(context)\n",
    "    return center, context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In the constructor, we create an array `self.order` which denotes shuffled indices of `[window, window + 1, ..., len(dataset) - window - 1]` in order to choose a center word randomly from `dataset` in a mini-batch.\n",
    "- The iterator definition `__next__` returns `batch_size` sets of  center word and context words.\n",
    "- The code `self.order[i:i_end]` returns the indices for a set of center words from the random-ordered array `self.order`. The center word IDs `center` at the random indices are retrieved by `self.dataset.take`.\n",
    "- `np.concatenate([np.arange(-w, 0), np.arange(1, w + 1)])` creates a set of offsets to retrieve context words from the dataset.\n",
    "- The code `position[:, None] + offset[None, :]` generates the indices of context words for each center word index in `position`. The context word IDs `context` are retrieved by `self.dataset.take`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Prepare model, optimizer, and updater"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unit = 100  # number of hidden units\n",
    "window = 5\n",
    "batchsize = 1000\n",
    "gpu = 0\n",
    "\n",
    "# Instantiate model\n",
    "model = SkipGram(n_vocab, unit)\n",
    "\n",
    "if gpu >= 0:\n",
    "    model.to_gpu(gpu)\n",
    "\n",
    "# Create optimizer\n",
    "optimizer = O.Adam()\n",
    "optimizer.setup(model)\n",
    "\n",
    "# Create iterators for both train and val datasets\n",
    "train_iter = WindowIterator(train, window, batchsize)\n",
    "val_iter = WindowIterator(val, window, batchsize, repeat=False)\n",
    "\n",
    "# Create updater\n",
    "updater = training.StandardUpdater(\n",
    "    train_iter, optimizer, converter=convert, device=gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Start training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch       main/loss   validation/main/loss\n",
      "\u001b[J1           6.87469     6.49239               \n",
      "\u001b[J2           6.43766     6.42476               \n",
      "\u001b[J3           6.34942     6.36353               \n",
      "\u001b[J4           6.28435     6.31737               \n",
      "\u001b[J5           6.23287     6.283                 \n",
      "\u001b[J6           6.20336     6.24662               \n",
      "\u001b[J7           6.16982     6.26347               \n",
      "\u001b[J8           6.14064     6.21212               \n",
      "\u001b[J9           6.11639     6.17406               \n",
      "\u001b[J10          6.0834      6.23088               \n",
      "\u001b[J11          6.04547     6.22154               \n",
      "\u001b[J12          6.04376     6.22592               \n",
      "\u001b[J13          6.03126     6.17224               \n",
      "\u001b[J14          6.00218     6.21196               \n",
      "\u001b[J15          6.01313     6.15059               \n",
      "\u001b[J16          6.00215     6.21771               \n",
      "\u001b[J17          5.97584     6.18996               \n",
      "\u001b[J18          5.96465     6.21517               \n",
      "\u001b[J19          5.95188     6.19383               \n",
      "\u001b[J20          5.9473      6.13596               \n",
      "\u001b[J21          5.92897     6.17627               \n",
      "\u001b[J22          5.92385     6.17938               \n",
      "\u001b[J23          5.9195      6.18834               \n",
      "\u001b[J24          5.90871     6.16518               \n",
      "\u001b[J25          5.91193     6.22971               \n",
      "\u001b[J26          5.91069     6.16714               \n",
      "\u001b[J27          5.8846      6.19366               \n",
      "\u001b[J28          5.90398     6.21329               \n",
      "\u001b[J29          5.90064     6.21574               \n",
      "\u001b[J30          5.88258     6.18508               \n",
      "\u001b[J31          5.8702      6.19865               \n",
      "\u001b[J32          5.86399     6.20083               \n",
      "\u001b[J33          5.869       6.16945               \n",
      "\u001b[J34          5.85692     6.21721               \n",
      "\u001b[J35          5.85175     6.24535               \n",
      "\u001b[J36          5.8673      6.22052               \n",
      "\u001b[J37          5.85207     6.22343               \n",
      "\u001b[J38          5.83775     6.19369               \n",
      "\u001b[J39          5.85815     6.28352               \n",
      "\u001b[J40          5.85035     6.2091                \n",
      "\u001b[J41          5.8423      6.28353               \n",
      "\u001b[J42          5.83698     6.22302               \n",
      "\u001b[J43          5.84559     6.28724               \n",
      "\u001b[J44          5.82942     6.24757               \n",
      "\u001b[J45          5.81868     6.2275                \n",
      "\u001b[J46          5.84483     6.31015               \n",
      "\u001b[J47          5.82578     6.2163                \n",
      "\u001b[J48          5.81876     6.27374               \n",
      "\u001b[J49          5.80941     6.27729               \n",
      "\u001b[J50          5.82054     6.30504               \n",
      "\u001b[J51          5.80205     6.25924               \n",
      "\u001b[J52          5.82125     6.24315               \n",
      "\u001b[J53          5.8075      6.27823               \n",
      "\u001b[J54          5.81458     6.29426               \n",
      "\u001b[J55          5.80863     6.2662                \n",
      "\u001b[J56          5.81012     6.27777               \n",
      "\u001b[J57          5.80712     6.21873               \n",
      "\u001b[J58          5.82886     6.26017               \n",
      "\u001b[J59          5.80509     6.31339               \n",
      "\u001b[J60          5.80734     6.2874                \n",
      "\u001b[J61          5.80876     6.28746               \n",
      "\u001b[J62          5.82057     6.26831               \n",
      "\u001b[J63          5.80071     6.27469               \n",
      "\u001b[J64          5.82219     6.33319               \n",
      "\u001b[J65          5.8104      6.27122               \n",
      "\u001b[J66          5.80218     6.34508               \n",
      "\u001b[J67          5.81691     6.30597               \n",
      "\u001b[J68          5.80138     6.29948               \n",
      "\u001b[J69          5.80251     6.31008               \n",
      "\u001b[J70          5.79986     6.31869               \n",
      "\u001b[J71          5.8007      6.33197               \n",
      "\u001b[J72          5.7917      6.34002               \n",
      "\u001b[J73          5.80931     6.34416               \n",
      "\u001b[J74          5.80188     6.31156               \n",
      "\u001b[J75          5.80004     6.3059                \n",
      "\u001b[J76          5.78182     6.34326               \n",
      "\u001b[J77          5.80898     6.3267                \n",
      "\u001b[J78          5.80455     6.30006               \n",
      "\u001b[J79          5.79808     6.35709               \n",
      "\u001b[J80          5.80054     6.36302               \n",
      "\u001b[J81          5.8025      6.31012               \n",
      "\u001b[J82          5.78773     6.3225                \n",
      "\u001b[J83          5.7871      6.33635               \n",
      "\u001b[J84          5.80807     6.32977               \n",
      "\u001b[J85          5.78313     6.3488                \n",
      "\u001b[J86          5.79799     6.32949               \n",
      "\u001b[J87          5.79194     6.36984               \n",
      "\u001b[J88          5.78191     6.28538               \n",
      "\u001b[J89          5.80466     6.33613               \n",
      "\u001b[J90          5.79181     6.30647               \n",
      "\u001b[J91          5.81321     6.34016               \n",
      "\u001b[J92          5.80324     6.4427                \n",
      "\u001b[J93          5.7999      6.3277                \n",
      "\u001b[J94          5.78875     6.31837               \n",
      "\u001b[J95          5.7871      6.36152               \n",
      "\u001b[J96          5.78073     6.37049               \n",
      "\u001b[J97          5.80366     6.35217               \n",
      "\u001b[J98          5.78997     6.42968               \n",
      "\u001b[J99          5.78576     6.32253               \n",
      "\u001b[J100         5.78377     6.3595                \n"
     ]
    }
   ],
   "source": [
    "epoch = 100\n",
    "\n",
    "trainer = training.Trainer(updater, (epoch, 'epoch'), out='word2vec_result')\n",
    "trainer.extend(extensions.Evaluator(val_iter, model, converter=convert, device=gpu))\n",
    "trainer.extend(extensions.LogReport())\n",
    "trainer.extend(extensions.PrintReport(['epoch', 'main/loss', 'validation/main/loss']))\n",
    "trainer.run()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}