autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Architectures\n",
    "\n",
    "First, we'll discuss Torch Autograd and its significance, along with the implications for how we build neural networks. Then, we'll take a high-level look at the following sample NN architectures: \n",
    "+ Dense\n",
    "+ CNN\n",
    "+ RNN\n",
    "+ [VGG](https://arxiv.org/pdf/1409.1556.pdf)\n",
    "+ [ResNet](https://arxiv.org/pdf/1512.03385.pdf)\n",
    "+ V/AE\n",
    "\n",
    "Finally, we'll open up discussion to focus on how to learn more on your own, be it by participating in labs or dissecting research papers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Torch Autograd\n",
    "\n",
    "PyTorch comes with a package known as `autograd` which serves as the basis for its flexibility and functionality. Autograd overloads many operations on tensors and dynamically builds a computational graph with the initial tensors as leaf nodes all the way until it reaches a terminal value (e.g. loss) which then is back-differentiated through leveraging the built up computational graph (see: Workshop 4).\n",
    "\n",
    "We begin by creating a torch tensor, and in particular we ensure that the `requires_grad` flag is set to `True`. This prompts autograd to keep track of all the operations which occur on this tensor, and subsequent tensors which rely on it. One point of potential trouble is that none of the operations can be in place. That is to say, you can't write `x = x*3`. Thinking back to the computational graphs we defined before, this would end up representing a self loop, making our graph ill-defined. Instead, we keep track of the operation and add the output as a *new* variable/node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Create a differentiable tensor\n",
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "print(x)\n",
    "\n",
    "# Multiply x by 3 and store result as a *new* node y\n",
    "y = x * 3\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can chain together as many operations as we want (physical memory permitting) in order to build even the most complex machine learning architectures. For now, we'll stick to a couple simple operations. Remember, we'll be recording these as new variables each time. In particular, we'll culminate our operations in our final variable that we'll call `out`. This is exactly what we'll apply a backward pass through in order to get our partial derivatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>)\n",
      "tensor(27., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to note that in order to actually call `tensor.backward()`, one must ensure that the tensor that is calling the function is a tensor containing only a scalar. That is to say, it shouldn't be a vector, matrix or general k-th order tensor. Here, out is a tensor containing a single scalar, and hence we can call `.backward()` on it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now the backward pass has completed and the computational graph has its partial derivatives! We've done this for each individual element of x, and so that means we simultaneously computed $$\\frac{\\partial (out)}{\\partial x_{i,j}}\\text{ for } 1\\leq i,j\\leq 2$$. If we had other variables which were used to compute `out` then `out.backward()` will also compute their gradients. In this case, we only used `x`. To access the gradient for `x`, we simply access `x.grad`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Aside:** As we mentioned above, in-place operations are not handled by PyTorch, which means that one must instead *clone* the starting tensor into a new one using `.clone()`, then apply in-place operations to the new tensor. An example is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "tensor([[1., 1.],\n",
      "        [3., 1.]], grad_fn=<CopySlices>)\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "leaf variable has been moved into the graph interior",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-48-4956f375b199>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[1;31m# Backward pass\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mout\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\torch\\tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[1;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[0;32m    105\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[1;33m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    106\u001b[0m         \"\"\"\n\u001b[1;32m--> 107\u001b[1;33m         \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    108\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    109\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\torch\\autograd\\__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[1;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[0;32m     91\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[0;32m     92\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 93\u001b[1;33m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[0;32m     94\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     95\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: leaf variable has been moved into the graph interior"
     ]
    }
   ],
   "source": [
    "# Starting tensor\n",
    "a = torch.ones(2, 2, requires_grad=True)\n",
    "print(a)\n",
    "\n",
    "# In-place operation on the starting tensor\n",
    "a[1,0]=a[1,0]*3\n",
    "print(a)\n",
    "\n",
    "# Scalar tensor for gradient calculation\n",
    "out = a.mean()\n",
    "\n",
    "# Backward pass\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "tensor([[1., 1.],\n",
      "        [1., 1.]], grad_fn=<CloneBackward>)\n",
      "tensor([[1., 1.],\n",
      "        [3., 1.]], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "# Starting tensor\n",
    "a = torch.ones(2, 2, requires_grad=True)\n",
    "print(a)\n",
    "\n",
    "# New tensor using .clone()\n",
    "A=a.clone()\n",
    "print(A)\n",
    "\n",
    "# In-place operation on the cloned tensor\n",
    "A[1,0]=A[1,0]*3\n",
    "print(A)\n",
    "\n",
    "# Scalar tensor for gradient calculation\n",
    "out = A.mean()\n",
    "\n",
    "# Backward pass\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2500, 0.2500],\n",
       "        [0.7500, 0.2500]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.grad"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}