06 Deep Residual Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tutorial: Classifying tiny images with a Convolutional Neural Network\n",
    "======================================\n",
    "\n",
    "Outline\n",
    "------------------------\n",
    "This interactive notebook shows how to do image classification with a Convnet. You can edit code in the code cells, and run it with `Shift+Return`. The notebook is read-only, so feel free to hack the code, and reload the page if something breaks. The tutorial covers how to:\n",
    "* Build a small convNet in neon.\n",
    "* Train it on the [Cifar10](https://www.kaggle.com/c/cifar-10) dataset. \n",
    "* Upload a new image, and classify it into one of the 10 categories.\n",
    "\n",
    "\n",
    "<img src=\"https://kaggle2.blob.core.windows.net/competitions/kaggle/3649/media/cifar-10.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up a model\n",
    "==================\n",
    "The pieces we need to set up a model are described in the [neon user guide](http://neon.nervanasys.com/docs/latest/index.html):\n",
    "* The CIFAR10 dataset.\n",
    "* layer configuration and a  [model](http://neon.nervanasys.com/docs/latest/models.html).\n",
    "* a compute [backend](http://neon.nervanasys.com/docs/latest/backends.html).\n",
    "* an [optimizer](http://neon.nervanasys.com/docs/latest/optimizers.html) to train the model.\n",
    "* [callbacks](http://neon.nervanasys.com/docs/latest/callbacks.html) to keep us updated about the progress of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Start by generating the backend:\n",
    "from neon.backends import gen_backend\n",
    "be = gen_backend(backend='gpu',             \n",
    "                 batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading a dataset\n",
    "-----------------\n",
    "We use the [aeon](aeon.nervanasys.com) dataloader to present the data to the model. \n",
    "\n",
    "**Note: This assumes the data has already been downloaded and ingested. If that is not the case, follow the instructions in the 02 VGG Fine-tuning notebook to process the CIFAR-10 dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.data.aeon_shim import AeonDataLoader\n",
    "from neon.data.dataloader_transformers import OneHot, TypeCast, BGRMeanSubtract\n",
    "import numpy as np\n",
    "\n",
    "# define configuration file for CIFAR-10 dataset\n",
    "config = {\n",
    "    'manifest_filename': 'data/cifar10/train-index.csv',  # CSV manifest of data\n",
    "    'manifest_root': 'data/cifar10',  # root data directory\n",
    "    'image': {'height': 32, 'width': 32,  # output image size \n",
    "              'scale': [0.8, 0.8],  # random scaling of image before cropping\n",
    "              'flip_enable': True},  # randomly flip image\n",
    "    'type': 'image,label',  # type of data\n",
    "    'minibatch_size': be.bsz  # batch size\n",
    "}\n",
    "\n",
    "from neon.data.aeon_shim import AeonDataLoader\n",
    "\n",
    "# build train_set\n",
    "train_set = AeonDataLoader(config, be)\n",
    "train_set = OneHot(train_set, index=1, nclasses=10)  # perform onehot on the labels\n",
    "train_set = TypeCast(train_set, index=0, dtype=np.float32)  # cast the image to float32\n",
    "train_set = BGRMeanSubtract(train_set, index=0)  # subtract image color means (based on default values)\n",
    "\n",
    "# build test set\n",
    "config['manifest_filename'] = 'data/cifar10/val-index.csv'\n",
    "test_set = AeonDataLoader(config, be)\n",
    "test_set = OneHot(test_set, index=1, nclasses=10)  # perform onehot on the labels\n",
    "test_set = TypeCast(test_set, index=0, dtype=np.float32)  # cast the image to float32\n",
    "test_set = BGRMeanSubtract(test_set, index=0)  # subtract image color means (based on default values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating layers\n",
    "The core of the model is the layers. This can be as simple as a list, but merging and branching makes it easy to specify complex topologies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Uniform\n",
    "from neon.transforms import Rectlin, Softmax\n",
    "from neon.layers import Activation, Conv, Pooling, Affine, MergeSum\n",
    "\n",
    "# This is a simple convnet with a one conv layer,\n",
    "# max-pooling, and a fully connected layer. \n",
    "#\n",
    "# input - Conv - ReLu - Pooling - Affine - ReLu - Affine - Softmax\n",
    "#\n",
    "layers = [Conv((5, 5, 16), init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Pooling((2, 2)),\n",
    "          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]\n",
    "\n",
    "# We can use a MergeSum layer to combine differnt layers in parallel\n",
    "#\n",
    "#             - Conv3 - ReLu - \n",
    "#           /                  \\\n",
    "# input   -                     Sum - ReLu - ...\n",
    "#           \\                  /\n",
    "#             - Conv5 - ReLu - \n",
    "#\n",
    "conv3 = Conv((3, 3, 16), init=Uniform(-0.1, 0.1), activation=Rectlin())\n",
    "conv5 = Conv((5, 5, 16), padding=1, init=Uniform(-0.1, 0.1), activation=Rectlin())\n",
    "\n",
    "\n",
    "layers = [MergeSum([conv3, conv5]), Activation(Rectlin()),\n",
    "          Pooling((2, 2)),\n",
    "          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Deep Residual Network\n",
    "A resnet module is a MergeSum layer containing a main path with conv layers, and a side path with a SkipNode() configured as the identity function. This allows earlier layer activations to bypass a series of layers.\n",
    "\n",
    "We use some helper functions to succinclty define the deep network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Kaiming, IdentityInit\n",
    "from neon.layers import SkipNode\n",
    "from neon.models import Model\n",
    "\n",
    "# helper functions simplify init params for conv and identity layers\n",
    "def conv_params(fsize, nfm, stride=1, relu=True, batch_norm=True):\n",
    "    return dict(fshape=(fsize, fsize, nfm), \n",
    "                strides=stride, \n",
    "                padding=(1 if fsize > 1 else 0),\n",
    "                activation=(Rectlin() if relu else None),\n",
    "                init=Kaiming(local=True),\n",
    "                batch_norm=batch_norm)\n",
    "\n",
    "def id_params(nfm):\n",
    "    return dict(fshape=(1, 1, nfm), \n",
    "                strides=2, \n",
    "                padding=0, \n",
    "                activation=None, \n",
    "                init=IdentityInit())\n",
    "\n",
    "# A resnet module\n",
    "#\n",
    "#             - Conv - Conv - \n",
    "#           /                \\\n",
    "# input   -                   Sum - Relu - output\n",
    "#           \\               /\n",
    "#            -  Identity - \n",
    "#\n",
    "def module_factory(nfm, stride=1):\n",
    "    mainpath = [Conv(**conv_params(3, nfm, stride=stride)),\n",
    "                Conv(**conv_params(3, nfm, relu=False))]\n",
    "    sidepath = [SkipNode() if stride == 1 else Conv(**id_params(nfm))]\n",
    "\n",
    "    module = [MergeSum([mainpath, sidepath]),\n",
    "              Activation(Rectlin())]\n",
    "    return module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is a collection of resnet modules between an input conv and output pooling and affine layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set depth = 3 for quick results \n",
    "# or depth = 9 to reach 6.7% top1 error in 150 epochs\n",
    "depth = 3\n",
    "nfms = [2**(stage + 4) for stage in sorted(range(3) * depth)]\n",
    "strides = [1] + [1 if cur == prev else 2 for cur, prev in zip(nfms[1:], nfms[:-1])]\n",
    "\n",
    "layers = [Conv(**conv_params(3, 16))]\n",
    "for nfm, stride in zip(nfms, strides):\n",
    "    layers.append(module_factory(nfm, stride))\n",
    "layers.append(Pooling('all', op='avg'))\n",
    "layers.append(Affine(10, init=Kaiming(local=False), \n",
    "                     batch_norm=True, activation=Softmax()))\n",
    "model = Model(layers=layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost function\n",
    "--------------\n",
    "The cost function compares network outputs with ground truth labels, and produces and error that we can backpropagate through the layers of the network.\n",
    "\n",
    "For our binary classification task, we use a cross entropy cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.transforms import CrossEntropyMulti\n",
    "from neon.layers import GeneralizedCost\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimizer\n",
    "---------\n",
    "We now have a cost function to minimize by gradient descent. We do this\n",
    "iteratively over small batches of the data set, making it stochastic gradient \n",
    "decesent (SGD). There are other [optimizers](http://neon.nervanasys.com/docs/latest/optimizers.html) such as RMSProp and AdaDelta that are supported in neon, but often simple gradient descent works well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.optimizers import GradientDescentMomentum, Schedule\n",
    "\n",
    "opt = GradientDescentMomentum(0.1, 0.9, wdecay=0.0001, \n",
    "                              schedule=Schedule([90, 135], 0.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks\n",
    "---------\n",
    "To provide feedback while the model is training, neon lets the user specify a set of callbacks that get evaluated at the end of every iteration (minibatch) or pass through the dataset (epoch). Callbacks include evaluating the model on a validation set or computing missclassification percentage. There are also callbacks for saving to disk and for generating visualizations. Here we will set up a progress bar to monitor training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set up callbacks. By default sets up a progress bar\n",
    "from neon.transforms import Misclassification\n",
    "from neon.callbacks.callbacks import Callbacks\n",
    "\n",
    "valmetric = Misclassification()\n",
    "callbacks = Callbacks(model, eval_set=test_set, metric=valmetric)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model\n",
    "------------------\n",
    "Now all the pieces are in place to run the network. We use the fit function and pass it a dataset, cost, optmizer, and the callbacks we set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# And  run the model\n",
    "epochs = 10\n",
    "model.fit(train_set, optimizer=opt, num_epochs=epochs, \n",
    "          cost=cost, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! If you made it this far you have trained a convolutional network in neon.\n",
    "\n",
    "Evaluating the model\n",
    "--------------------\n",
    "We can now compute the misclassification on the test set to see how well we did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check the performance on the supplied test set\n",
    "from neon.transforms import Misclassification\n",
    "\n",
    "error_pct = 100 * model.eval(test_set, metric=Misclassification())\n",
    "print 'Misclassification error = %.1f%%' % error_pct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By increasing the depth of the network and the number of epochs, we can improve the performance to match state of the art.\n",
    "\n",
    "This was quite a lot of code! Generally, to set up a new model from scratch it is best to follow one of the examples from the neon/examples directory. It's easy to mix and match parts!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference\n",
    "=========\n",
    "Now we want to grab a few new images from the internet and classify them through our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import urllib\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "# download images from the web\n",
    "imgs = {\n",
    "       'frog': \"https://upload.wikimedia.org/wikipedia/commons/thumb/5/55/Atelopus_zeteki1.jpg/440px-Atelopus_zeteki1.jpg\",\n",
    "       'airplane': \"https://img0.etsystatic.com/016/0/5185796/il_570xN.433414910_p5n3.jpg\",\n",
    "       'cat': \"https://s-media-cache-ak0.pinimg.com/236x/8e/d7/41/8ed7410285f101ba5892ff723c91fa75.jpg\",\n",
    "       'car': \"http://static01.nyt.com/images/2012/09/09/automobiles/09REFI2/09REFI2-articleLarge.jpg\",\n",
    "       }\n",
    "\n",
    "# empty buffer to use for inference dataset\n",
    "# dims [minibatch, imgsize]\n",
    "x_new = np.zeros((128, 32*32*3), dtype=np.float32)\n",
    "\n",
    "# crop/resize images and assign them to slots in x_new\n",
    "# also display with true labels\n",
    "plt.figure(1)\n",
    "for i, name in enumerate(imgs):\n",
    "    imgs[name] = urllib.urlretrieve(imgs[name], filename=\"data/{}.jpg\".format(name))\n",
    "    plt.subplot(100 + (10 * len(imgs)) + 1 + i)\n",
    "\n",
    "    img = Image.open(\"data/{}.jpg\".format(name))\n",
    "    crop = img.crop((0,0,min(img.size),min(img.size)))\n",
    "    crop.thumbnail((32, 32))\n",
    "\n",
    "    plt.imshow(crop, interpolation=\"nearest\")\n",
    "    plt.title(name)\n",
    "    plt.axis('off')\n",
    "\n",
    "    x_new[i,:] = np.asarray(crop, dtype=np.float32)[:,:,(2,0,1)].transpose(2,0,1).reshape(1,3072) -127\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataset with this image for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.data import ArrayIterator\n",
    "\n",
    "# create a minibatch with the new image \n",
    "inference_set = ArrayIterator(x_new, None, nclass=10, \n",
    "                             lshape=(3, 32, 32))\n",
    "# inference_set = ArrayIterator(x_train, None, nclass=10, \n",
    "#                              lshape=(3, 32, 32))\n",
    "classes =[\"airplane\", \"auto\", \"bird\", \"cat\", \"deer\", \n",
    "          \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
    "out = model.get_outputs(inference_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get model outputs on the inference data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(2)\n",
    "for i, name in enumerate(imgs):\n",
    "    plt.subplot(100 + (10 * len(imgs)) + 1 + i)\n",
    "\n",
    "    img = Image.open(\"data/{}.jpg\".format(name))\n",
    "    crop = img.crop((0,0,min(img.size),min(img.size)))\n",
    "    crop.thumbnail((32, 32))\n",
    "\n",
    "    title = \"{} ({:.2})\".format(classes[out[i].argmax()], out[i].max())\n",
    "        \n",
    "    plt.imshow(crop, interpolation=\"nearest\")\n",
    "    plt.title(title)\n",
    "    plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}