chainer-beginners-handson-en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "- This tutorial does not explain Neural Network itself.\n",
    "- This is created with Jupyter Notebook, so you can run all the code in this tutorial from top to below\n",
    "- You can find the original Jupyter Notebook [here](https://github.com/mitmul/chainer-handson/blob/master/Chainer%20Beginer's%20Hands-on.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to install Chainer\n",
    "\n",
    "The installation of Chainer is very easy. Chainer is written in Python only, so you can easily install the package of Chainer using `pip`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: chainer in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages\n",
      "Requirement already satisfied: six>=1.9.0 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from chainer)\n",
      "Requirement already satisfied: filelock in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from chainer)\n",
      "Requirement already satisfied: numpy>=1.9.0 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from chainer)\n",
      "Requirement already satisfied: protobuf>=3.0.0 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from chainer)\n",
      "Requirement already satisfied: setuptools in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from protobuf>=3.0.0->chainer)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 9.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pip install chainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to enable GPU\n",
    "\n",
    "All the GPU implementation in Chainer is based on [CuPy](https://cupy.chainer.org/), so you need to install CuPy to use GPUs in Chainer.\n",
    "\n",
    "CuPy is also distributed on PyPI, and the pre-built binaries for CUDA 8.0/9.0/9.1/9.2 environments. So, **you do not have to compile CuPy from source basically.** And, those pre-built binaries are built with cuDNN and NCCL2, so **you do not even need to care about cuDNN. It's automatically installed along with CuPy wheels!**\n",
    "\n",
    "For example, if your environment has CUDA 9.0, please just run this command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cupy-cuda90 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages\n",
      "Requirement already satisfied: numpy>=1.9.0 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from cupy-cuda90)\n",
      "Requirement already satisfied: fastrlock>=0.3 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from cupy-cuda90)\n",
      "Requirement already satisfied: six>=1.9.0 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from cupy-cuda90)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 9.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pip install cupy-cuda90"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using another version of CUDA, please just replace `cupy-cuda90` with `cupy-cuda80`, `cupy-cuda91`, or `cupy-cuda92`. In summary, the following pre-built packages are provided through PyPI:\n",
    "\n",
    "- `cupy-cuda80` (for CUDA 8.0 environment)\n",
    "- `cupy-cuda90` (for CUDA 9.0 environment)\n",
    "- `cupy-cuda91` (for CUDA 9.1 environment)\n",
    "- `cupy-cuda92` (for CUDA 9.2 environment)\n",
    "\n",
    "FYI, you can check your CUDA version by executing this command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nvcc: NVIDIA (R) Cuda compiler driver\n",
      "Copyright (c) 2005-2017 NVIDIA Corporation\n",
      "Built on Fri_Sep__1_21:08:03_CDT_2017\n",
      "Cuda compilation tools, release 9.0, V9.0.176\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "nvcc -V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check your Chainer installation\n",
    "\n",
    "You can check all the environment setups for Chainer easily by calling a function provided in Chainer like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chainer: 4.4.0\n",
      "NumPy: 1.14.0\n",
      "CuPy:\n",
      "  CuPy Version          : 4.4.0\n",
      "  CUDA Root             : /usr/local/cuda\n",
      "  CUDA Build Version    : 9000\n",
      "  CUDA Driver Version   : 9000\n",
      "  CUDA Runtime Version  : 9000\n",
      "  cuDNN Build Version   : 7201\n",
      "  cuDNN Version         : 7005\n",
      "  NCCL Build Version    : 2213\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -c 'import chainer; chainer.print_runtime_info()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add some useful packages for visualization\n",
    "\n",
    "In this tutorial, we use `matplotlib` for plotting and `graphviz` for graph visualization, so let's install them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: matplotlib in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages\n",
      "Requirement already satisfied: seaborn in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages\n",
      "Requirement already satisfied: numpy>=1.7.1 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: six>=1.10 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: pytz in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from matplotlib)\n",
      "Requirement already satisfied: scipy in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from seaborn)\n",
      "Requirement already satisfied: pandas in /home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages (from seaborn)\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 9.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pip install matplotlib seaborn\n",
    "\n",
    "# If you can use \"sudo\", just execute\n",
    "# sudo apt-get install graphviz\n",
    "\n",
    "# If you cannot use sudo, use conda:\n",
    "conda install graphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write a training loop\n",
    "\n",
    "Let's start with hand-written digit images recognition task using MNIST dataset. This is a simple image classification task.\n",
    "\n",
    "## 1. Prepare the dataset\n",
    "\n",
    "In supervised learning, a dataset for Chainer should be an object that returns tuples of input data and the corresponding labels. Chainer has some useful functions that prepare some famous datasets such as MNIST, Fashion MNIST, CIFAR10, CIFAR100, SVHN, and PTB, etc. You can create a dataset object by just calling one of those functions provided by Chainer and start using it for training of neural nets. Those functions automatically retrieve the data itself from servers if they have not been downloaded.\n",
    "\n",
    "Let's create an MNIST dataset object using `get_mnist` function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shunta/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "from chainer.datasets import mnist\n",
    "\n",
    "train_val, test = mnist.get_mnist(withlabel=True, ndim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset preparation has been done! Using this object, you can access the i-th datum like `(data, label)` by just using `[]` accessor like this: `train_val[i]`. Actually, **you can use a normal Python list as a dataset object for Chainer.** In such cases, the list will be like this: `[(x1, t1), (x2, t2), ..., (xn, tn)]`, where `x1, x2, ..., xn` mean data, while `t1, t2, ..., tn` mean labels.\n",
    "\n",
    "To make sure that the dataset objects have been correctly created, try visualizing a MNIST image using matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAABrBJREFUeJzt3blrFX0fxuH3vIqFooY0CoKIFhEVSaOCCCISRNAiaiNYKVYGrNLYWUQElyJokUqwEUuXRgu3QggElyZgr6TTuC/EnOcvON/oyWru62rvjDOFH6b4ObHRbDb/B+T5/3w/ADA/xA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hls7lzRqNhn9OCLOs2Ww2/uTnvPkhlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPgh1NL5fgBm15IlS8p99erVs3r/vr6+ltvy5cvLa7u6usr9zJkz5X758uWW2/Hjx8trf/z4Ue4XL14s9/Pnz5f7QuDND6HED6HED6HED6HED6HED6HED6Gc88+B9evXl/uyZcvKfffu3eW+Z8+elltHR0d57dGjR8t9Pr19+7bcBwcHy723t7fl9vnz5/La169fl/vTp0/L/V/gzQ+hxA+hxA+hxA+hxA+hxA+hGs1mc+5u1mjM3c3mUHd3d7k/evSo3Gf7s9qFanJystxPnjxZ7l++fGn73mNjY+X+4cOHcn/z5k3b955tzWaz8Sc/580PocQPocQPocQPocQPocQPocQPoZzzz4DOzs5yHx4eLveNGzfO5OPMqKmefXx8vNz37dvXcvv161d5beq/f5gu5/xASfwQSvwQSvwQSvwQSvwQSvwQyq/ungHv378v9/7+/nI/dOhQub98+bLcp/oV1pVXr16Ve09PT7l//fq13Ldu3dpyO3v2bHkts8ubH0KJH0KJH0KJH0KJH0KJH0KJH0L5nn8BWLVqVblP9d9JDw0NtdxOnTpVXnvixIlyv3XrVrmz8PieHyiJH0KJH0KJH0KJH0KJH0KJH0L5nn8B+PTp07Su//jxY9vXnj59utxv375d7pOTk23fm/nlzQ+hxA+hxA+hxA+hxA+hxA+hfNK7CKxYsaLldu/evfLavXv3lvvBgwfL/eHDh+XO3PNJL1ASP4QSP4QSP4QSP4QSP4QSP4Ryzr/Ibdq0qdxfvHhR7uPj4+X++PHjch8ZGWm5Xb9+vbx2Lv9uLibO+YGS+CGU+CGU+CGU+CGU+CGU+CGUc/5wvb295X7jxo1yX7lyZdv3PnfuXLnfvHmz3MfGxtq+92LmnB8oiR9CiR9CiR9CiR9CiR9CiR9COeentG3btnK/evVque/fv7/tew8NDZX7wMBAub97967te//LnPMDJfFDKPFDKPFDKPFDKPFDKPFDKOf8TEtHR0e5Hz58uOU21e8KaDTq4+pHjx6Ve09PT7kvVs75gZL4IZT4IZT4IZT4IZT4IZSjPubNz58/y33p0qXlPjExUe4HDhxouT158qS89l/mqA8oiR9CiR9CiR9CiR9CiR9CiR9C1QepxNu+fXu5Hzt2rNx37NjRcpvqHH8qo6Oj5f7s2bNp/fmLnTc/hBI/hBI/hBI/hBI/hBI/hBI/hHLOv8h1dXWVe19fX7kfOXKk3NeuXfvXz/Snfv/+Xe5jY2PlPjk5OZOPs+h480Mo8UMo8UMo8UMo8UMo8UMo8UMo5/z/gKnO0o8fP95ym+ocf8OGDe080owYGRkp94GBgXK/e/fuTD5OHG9+CCV+CCV+CCV+CCV+CCV+COWobw6sWbOm3Lds2VLu165dK/fNmzf/9TPNlOHh4XK/dOlSy+3OnTvltT7JnV3e/BBK/BBK/BBK/BBK/BBK/BBK/BDKOf8f6uzsbLkNDQ2V13Z3d5f7xo0b23qmmfD8+fNyv3LlSrk/ePCg3L9///7Xz8Tc8OaHUOKHUOKHUOKHUOKHUOKHUOKHUDHn/Lt27Sr3/v7+ct+5c2fLbd26dW0900z59u1by21wcLC89sKFC+X+9evXtp6Jhc+bH0KJH0KJH0KJH0KJH0KJH0KJH0LFnPP39vZOa5+O0dHRcr9//365T0xMlHv1zf34+Hh5Lbm8+SGU+CGU+CGU+CGU+CGU+CGU+CFUo9lszt3NGo25uxmEajabjT/5OW9+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CCV+CDWnv7obWDi8+SGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CHUf+FsNTkv2hLSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2b8e280c87f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label: 5\n"
     ]
    }
   ],
   "source": [
    "# Run a magic to show images inline in the notebook\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Show a datum!\n",
    "x, t = train_val[0]  # Retrieve 0-th tuple of (data, label)\n",
    "plt.imshow(x.reshape(28, 28), cmap='gray')\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print('label:', t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Split a dataset into two sub datasets\n",
    "\n",
    "\n",
    "To create a validation dataset, you need to split the `train_val` dataset object into two different dataset objects. Chainer has some useful functions to split dataset objects in some manner. If you want to split the dataset into a `train` dataset which has 50000 data out of 60000 and a `valid` dataset which has remaining 10000 data **randomly**, you can use `split_dataset_random` function as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from chainer.datasets import split_dataset_random\n",
    "\n",
    "train, valid = split_dataset_random(train_val, 50000, seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reproducibility of this splitting, specifying a certain `seed` to the argument of `split_dataset_random` is recommended.\n",
    "\n",
    "How many data are there in those datasets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset size: 50000\n",
      "Validation dataset size: 10000\n"
     ]
    }
   ],
   "source": [
    "print('Training dataset size:', len(train))\n",
    "print('Validation dataset size:', len(valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an `Iterator`\n",
    "\n",
    "We have finished preparing the dataset objects, but those are not so useful to use directly for training. Because neural nets usually are trained with the sotchastic gradient descent (SGD) which takes a set of data called **minibatch** to calculate the objective function to be minimized. Each datum in a minibatch is usually sampled from the training dataset randomly without replacement during an epoch.\n",
    "\n",
    "To make a minibatch easily from the dataset object, Chainer provides some `Iterator` classes. An `Iterator` object is initialized with a dataset object, and you can retrieve the next minibatch by calling `next()` method of it. How many times all the data are retrieved, namely, epoch is stored in the `Iterator` object, so it's useful to write a training loop by yourself.\n",
    "\n",
    "How to create an `Iterator` object is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from chainer import iterators\n",
    "\n",
    "batchsize = 128\n",
    "\n",
    "train_iter = iterators.SerialIterator(train, batchsize)\n",
    "valid_iter = iterators.SerialIterator(\n",
    "    valid, batchsize, repeat=False, shuffle=False)\n",
    "test_iter = iterators.SerialIterator(\n",
    "    test, batchsize, repeat=False, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, three `Iterator`s for training, validation, and testing are created. `batchsize` is set to 128 in the beginning of this cell, so these `Iterator`s return a minibatch which has 128 data when you call `next()` method like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num of samples in the minibatch: 128\n",
      "\n",
      "Shape of the image: (784,)\n",
      "Label of the first data in the minibatch: 1\n",
      "\n",
      "Shape of the image: (784,)\n",
      "Label of the first data in the minibatch: 1\n"
     ]
    }
   ],
   "source": [
    "minibatch = train_iter.next()\n",
    "\n",
    "print('Num of samples in the minibatch:', len(minibatch))\n",
    "\n",
    "# Extract the first (image, label) pair in the minibatch\n",
    "image, label = minibatch[0]\n",
    "\n",
    "print('\\nShape of the image:', image.shape)\n",
    "print('Label of the first data in the minibatch:', label)\n",
    "\n",
    "# Retrieve the next minibatch\n",
    "minibatch = train_iter.next()\n",
    "\n",
    "# Extract the first (image, label) pair in the minibatch\n",
    "image, label = minibatch[0]\n",
    "\n",
    "print('\\nShape of the image:', image.shape)\n",
    "print('Label of the first data in the minibatch:', label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE: What's `SerialIterator`**\n",
    "\n",
    "`SerialIterator` is one of the `Iterator`s which Chainer provides, and it is the simplest one among all. It creates minibatches by taking data sequentially from the given dataset object. There are several options you can specify when making an `SerialIterator` object:\n",
    "\n",
    "- `repeat`: If `True`, it infinitely loops over the dataset. Otherwise, it stops iteration at the end of the first epoch.\n",
    "- `shuffle`: If `True`, the order of examples is shuffled at the beginning of each epoch. Otherwise, examples are extracted in the order of indexes.\n",
    "\n",
    "You can also use `MultiprocesIterator` to eploit multi-core CPUs to make minibatches, and `MultithreadIterator` that uses threads instead of processes. Please take a look into the documentation page to know more about Chainer's `Iterator`s:\n",
    "\n",
    "- [A list of `Iterator`s in Chainer](https://docs.chainer.org/en/stable/reference/iterators.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Write a neural network\n",
    "\n",
    "Now, it's time to design a neural network with Chainer! In this section, let's see how to write a multi-layer perceptron which have 3 fully-connected layers. To train the network which clasifies images into a class out of 10 classes, the number of output units should be 10. We set the number of units in intermediate layers to 100 this time. In Chainer, a network is written using some classes like `Link`, `Function`, and `Chain` etc.:\n",
    "\n",
    "### `Link` and `Function`\n",
    "\n",
    "In Chainer, a layer in a neural network is either `Link` or `Function`.\n",
    "\n",
    "- `Link` is parametric function which has learnable parameters\n",
    "- `Function` is non-parametric function such as activations, four arithmetic operations, etc.\n",
    "\n",
    "Networks are written in Chainer by combining `Link`s and `Function`s. You can find many parametric functions here: [`chainer.links` module](https://docs.chainer.org/en/stable/reference/links.html), and many non-parametric functions here [`chainer.functions` module](https://docs.chainer.org/en/stable/reference/functions.html). There is a common convention to access those modules easily by making shortcuts when importing them:\n",
    "\n",
    "```\n",
    "import chainer.links as L\n",
    "import chainer.functions as F\n",
    "```\n",
    "\n",
    "It enables to use `Link`s and `Function`s like `L.Convolution2D(...)` and `F.relu(...)`, but this is not necessarily to be so.\n",
    "\n",
    "### `Chain`\n",
    "\n",
    "`Chain` is a class to bundle some learnable layers (`Link`s) together. `Link`s are learnable layers, so that they have learnable parameters. This means that the optimizer needs to find all of them to update their values during training (you can let a `Link` have \"fixed parameters\", though). To make finding all parameters to be updated in the network easily, `Chain` provides some functionalities to manage all learnable parameters in the network. For example, to collect all learnable parameters from `Link`s registered to the `Chain`, there is the `Chain.params()` method that returns all `Parameter` objects found under the `Chain`.\n",
    "\n",
    "### Define your own network by inheriting `Chain`\n",
    "\n",
    "The most common way to define a neural network in Chainer is to inherit `Chain` class and write down a new class for the network. In the constructor of the class, `self.init_scope()` is used to create a context where you can register child `Link`s (learnable layers) to the class (which is a `Chain` because it inherits `Chain` class). Then the optimizer can easily find all the learnable parameters in your network to train.\n",
    "\n",
    "The forward pass computation of your network is written in `forward()` method of the class. If you write the forward computation in the method, that can be called through the `()` accessor, e.g., `output = net(data)`.\n",
    "\n",
    "### How to run the network on GPU\n",
    "\n",
    "`Chain` class provides `to_gpu()` method, so that you can send all the parameters to a GPU by calling it. `to_gpu()` takes a GPU ID where the network is sent to. Once you move the network to GPU memory, all the calculations defined in the `forward()` method is done on the GPU. If you specify -1 as the GPU ID, it means CPU.\n",
    "\n",
    "### To ensure the reproducibility\n",
    "\n",
    "There are some tips to ensure the reproducibility of network training. First, fix the random seeds for three modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy\n",
    "import chainer\n",
    "\n",
    "random.seed(0)\n",
    "numpy.random.seed(0)\n",
    "if chainer.cuda.available:\n",
    "    chainer.cuda.cupy.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, if it is acceptable to slow down the calculation with cuDNN a bit, you can turn on a configuration `chainer.config.cudnn_deterministic` to use cuDNN without non-deterministic atomic operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer perceptron\n",
    "\n",
    "Let's write down a multi-layer perceptron in Chainer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import chainer\n",
    "import chainer.links as L\n",
    "import chainer.functions as F\n",
    "\n",
    "class MLP(chainer.Chain):\n",
    "\n",
    "    def __init__(self, n_mid_units=100, n_out=10):\n",
    "        super(MLP, self).__init__()\n",
    "        \n",
    "        # Register learnable layers\n",
    "        with self.init_scope():\n",
    "            self.l1 = L.Linear(None, n_mid_units)\n",
    "            self.l2 = L.Linear(n_mid_units, n_mid_units)\n",
    "            self.l3 = L.Linear(n_mid_units, n_out)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Write the forward pass computation here\n",
    "        h1 = F.relu(self.l1(x))\n",
    "        h2 = F.relu(self.l2(h1))\n",
    "        return self.l3(h2)\n",
    "\n",
    "gpu_id = 0  # If you use CPU, specify -1 for gpu_id\n",
    "\n",
    "net = MLP()\n",
    "\n",
    "if gpu_id >= 0:\n",
    "    net.to_gpu(gpu_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it. Do you have any questions? If you want to customize this network using varied `Function`s and `Link`s, please check these documents to know what type of layers are provided in Chainer:\n",
    "\n",
    "- [A list of `Function`s](https://docs.chainer.org/en/stable/reference/functions.html)\n",
    "- [A list of learnable layers (`Link`s)](https://docs.chainer.org/en/stable/reference/links.html)\n",
    "\n",
    "In the above document pages, many learnable/unlearnable functions which are commonly used in various types of neural networks, e.g., fully-connected layers, convolutional layers, LSTM, ReLU, and also many loss functions are also provided. All are differentiable because they are provided to be a part of your neural network!\n",
    "\n",
    "Of course, you can define **your own original learnable layers and functions with NumPy (for CPU) and CuPy (for GPU)** very easily thanks to its \"Define-by-Run\" approach. Please check this document to see how to define your own functions in Chainer: [Define your own function](https://docs.chainer.org/en/stable/guides/functions.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**\n",
    "\n",
    "In the above network definition, the first linear layer (fully-connected layer) takes `None` as the first argument when being registered in the constructor. It means **the number of input units are determined at the first forward computation (=when an actual data is given to the network for the first time)**. Until then, the parameter is not allocated and not initialized. So, **you can use this network definition for arbitorary size of inputs without any change to the code**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Choose an optimizer\n",
    "\n",
    "To train the network defined above, it is necessary to setup an optimizer. Chainer provides various implementation of SGD such as Adam, RMSProp, MomentumSGD, AdaGrad, AdaDelta, ... Please check all the optimizers you can immediately use with Chainer here:\n",
    "\n",
    "- [List of `Optimizer`s](https://docs.chainer.org/en/stable/reference/optimizers.html)\n",
    "\n",
    "Here, we choose the simplest optimizer `optimizers.SGD` to traine the network on MNIST dataset. Once you choose an optimizer, it is required to give the network object to the optimizer class to initialize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from chainer import optimizers\n",
    "\n",
    "optimizer = optimizers.SGD(lr=0.01).setup(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Start training\n",
    "\n",
    "In multi-class classification problem, the softmax cross entropy is oftenly used as a loss function to train a neural net. It's calculated from a pair of a predicted label which is the output of the network and the correct label. The loss should be a scalar value in this case. Then, `F.softmax_cross_entropy()` function takes two inputs (the network output and the label) and returns a scalar value as a `Variable` object. This `Variable` object stores history of all the computations applied to the network input as a computational graph, so we can trace back the network architecture along with the actual calculation history to reconstruct the network actually used to make the output from the input. This way to determine the network structure is what we call \"Define-by-Run\". It enables us to write dynamic network architecture very easily compared to \"Define-and-Run\" style frameworks.\n",
    "\n",
    "Once you get the loss `Variable`, the backward computation to calculate the gradients of all the parameters in the network can be done by just calling `Variable.backward()` method from that loss `Variable`. It starts gradient calculation by the chain rule for all the learnable parameters. The parameters are stored in the `Link`s as `Parameter` objects, but `Parameter` is a child class of `Variable`. And all the `Parameter`s will have `grad_var` property for the gradient after calling `backward()` method. Now the optimizer can update the parameter using the gradient with the update rule corresponding to the optimization method.\n",
    "\n",
    "Repeating this flow is so-called \"training loop\". In summary, the following steps are performed in a single training iteration:\n",
    "\n",
    "1. Give an input data `x` to the network and get the output `y`\n",
    "2. Calculate the loss `Variable` from `y` and the correct labels `t`\n",
    "3. Call the `backward()` method of the loss `Variable` to compute the gradients for all the learnable parameters in the network\n",
    "4. Call `update()` method of the optimizer to update the parameters using gradients calculated in the step 3.\n",
    "\n",
    "**NOTE: Various loss functions**\n",
    "\n",
    "Chainer provides many loss functions for varios tasks. For example, if you want to train a network to solve a regression problem, you may want to calculate `F.mean_squared_error()` or `F.mean_absolute_error()` etc. Please check this document to see what types of loss functions are provided in Chainer:\n",
    "\n",
    "- [Loss functions available in Chainer](https://docs.chainer.org/en/stable/reference/functions.html#loss-functions)\n",
    "\n",
    "### The training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:01 train_loss:0.9192 val_loss:0.9769 val_accuracy:0.8034\n",
      "epoch:02 train_loss:0.5859 val_loss:0.5341 val_accuracy:0.8652\n",
      "epoch:03 train_loss:0.5736 val_loss:0.4262 val_accuracy:0.8821\n",
      "epoch:04 train_loss:0.4714 val_loss:0.3747 val_accuracy:0.8935\n",
      "epoch:05 train_loss:0.3406 val_loss:0.3462 val_accuracy:0.8997\n",
      "epoch:06 train_loss:0.3153 val_loss:0.3255 val_accuracy:0.9089\n",
      "epoch:07 train_loss:0.3463 val_loss:0.3098 val_accuracy:0.9124\n",
      "epoch:08 train_loss:0.3130 val_loss:0.2983 val_accuracy:0.9152\n",
      "epoch:09 train_loss:0.2293 val_loss:0.2896 val_accuracy:0.9177\n",
      "epoch:10 train_loss:0.3513 val_loss:0.2788 val_accuracy:0.9212\n",
      "test_accuracy:0.9375\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from chainer.dataset import concat_examples\n",
    "from chainer.cuda import to_cpu\n",
    "\n",
    "max_epoch = 10\n",
    "\n",
    "while train_iter.epoch < max_epoch:\n",
    "    \n",
    "    # ---------- 1 iteration of training ----------\n",
    "    train_batch = train_iter.next()\n",
    "    x, t = concat_examples(train_batch, gpu_id)\n",
    "    \n",
    "    # Create a prediction\n",
    "    y = net(x)\n",
    "\n",
    "    # Calculate the loss\n",
    "    loss = F.softmax_cross_entropy(y, t)\n",
    "\n",
    "    # Calculate the gradients\n",
    "    net.cleargrads()\n",
    "    loss.backward()\n",
    "\n",
    "    # Update all the learnable parameters\n",
    "    optimizer.update()\n",
    "    # ----------------- to here -------------------\n",
    "\n",
    "    # Every 1 epoch, check the generalization ability of the\n",
    "    # network by measuring the accuracy on the validation data\n",
    "    if train_iter.is_new_epoch:  # when this iteration was the final iteration in this epoch\n",
    "\n",
    "        # Show the loss value\n",
    "        print('epoch:{:02d} train_loss:{:.04f} '.format(\n",
    "            train_iter.epoch, float(to_cpu(loss.data))), end='')\n",
    "\n",
    "        valid_losses = []\n",
    "        valid_accuracies = []\n",
    "        while True:\n",
    "            valid_batch = valid_iter.next()\n",
    "            x_valid, t_valid = concat_examples(valid_batch, gpu_id)\n",
    "\n",
    "            # Create a prediction on the validation data\n",
    "            with chainer.using_config('train', False), \\\n",
    "                    chainer.using_config('enable_backprop', False):\n",
    "                y_valid = net(x_valid)\n",
    "\n",
    "            # Calculate the loss\n",
    "            loss_valid = F.softmax_cross_entropy(y_valid, t_valid)\n",
    "            valid_losses.append(to_cpu(loss_valid.array))\n",
    "\n",
    "            # Calculate the accuracy\n",
    "            accuracy = F.accuracy(y_valid, t_valid)\n",
    "            accuracy.to_cpu()\n",
    "            valid_accuracies.append(accuracy.array)\n",
    "                        \n",
    "            if valid_iter.is_new_epoch:\n",
    "                valid_iter.reset()\n",
    "                break\n",
    "\n",
    "        print('val_loss:{:.04f} val_accuracy:{:.04f}'.format(\n",
    "            np.mean(valid_losses), np.mean(valid_accuracies)))\n",
    "        \n",
    "# Evaluation of the network on the test dataset\n",
    "test_accuracies = []\n",
    "while True:\n",
    "    test_batch = test_iter.next()\n",
    "    x_test, t_test = concat_examples(test_batch, gpu_id)\n",
    "\n",
    "    # Create a prediction on the test data\n",
    "    with chainer.using_config('train', False), \\\n",
    "            chainer.using_config('enable_backprop', False):\n",
    "        y_test = net(x_test)\n",
    "\n",
    "    # Calculate the accuracy\n",
    "    accuracy = F.accuracy(y_valid, t_valid)\n",
    "    accuracy.to_cpu()\n",
    "    test_accuracies.append(accuracy.array)\n",
    "\n",
    "    if test_iter.is_new_epoch:\n",
    "        test_iter.reset()\n",
    "        break\n",
    "\n",
    "print('test_accuracy:{:.04f}'.format(np.mean(test_accuracies)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}