PyTorch 1.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is PyTorch?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"download.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch is a Python-based scientific computing package that uses the power of graphics processing units(GPU). It is also one of the preferred deep learning research platforms built to provide maximum flexibility and speed. It is known for providing two of the most high-level features; namely, tensor computations with strong GPU acceleration support and building deep neural networks on a tape-based autograd systems.\n",
    "\n",
    "There are many existing Python libraries which have the potential to change how deep learning and artificial intelligence are performed, and this is one such library. One of the key reasons behind PyTorch’s success is it is completely Pythonic and one can build neural network models effortlessly. It is still a young player when compared to its other competitors, however, it is gaining momentum fast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief History about PyTorch\n",
    "\n",
    "Since its release in January 2016, many researchers have continued to increasingly adopt PyTorch. It has quickly become a go-to library because of its ease in building extremely complex neural networks. It is giving a tough competition to TensorFlow especially when used for research work. However, there is still some time before it is adopted by the masses due to its still “new” and “under construction” tags.\n",
    "\n",
    "PyTorch creators envisioned this library to be highly imperative which can allow them to run all the numerical computations quickly. This is an ideal methodology which fits perfectly with the Python programming style. It has allowed deep learning scientists, machine learning developers, and neural network debuggers to run and test part of the code in real time. Thus they don’t have to wait for the entire code to be executed to check whether it works or not.\n",
    "You can always use your favorite Python packages such as NumPy, SciPy, and Cython to extend PyTorch functionalities and services when required. Now you might ask, why PyTorch? What’ so special in using it to build deep learning models?\n",
    "\n",
    "The answer is quite simple, PyTorch is a dynamic library (very flexible and you can use as per your requirements and changes) which is currently adopted by many of the researchers, students, and artificial intelligence developers. In the recent Kaggle competition, PyTorch library was used by nearly all of the top 10 finishers.\n",
    "\n",
    "Some of the key highlights of PyTorch includes:\n",
    "\n",
    "__Simple Interface:__ It offers easy to use API, thus it is very simple to operate and run like Python.\n",
    "\n",
    "__Pythonic in nature:__ This library, being Pythonic, smoothly integrates with the Python data science stack. Thus it can leverage all the services and functionalities offered by the Python environment.\n",
    "\n",
    "__Computational graphs:__ In addition to this, PyTorch provides an excellent platform which offers dynamic computational graphs, thus you can change them during runtime. This is highly useful when you have no idea how much memory will be required for creating a neural network model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why we use PyTorch in research field?\n",
    "\n",
    "Anyone who is working in the field of deep learning and artificial intelligence has likely worked with TensorFlow before, Google’s most popular open source library. However, the latest deep learning framework – PyTorch solves major problems in terms of research work. Arguably PyTorch is TensorFlow’s biggest competitor to date, and it is currently a much favored deep learning and artificial intelligence library in the research community.\n",
    "\n",
    "\n",
    "You might be thinking why we use PyTorch? I list down the three factors for that\n",
    "\n",
    "- Using API is Easy: It is as simple as Python.\n",
    "- Pyt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CPU v/s GPU\n",
    "  - CPU have fewer but more powerful compute cores, and whereas GPUs have a large number of lower-performant cores.\n",
    "  - CPUs are more suited to sequential tasks and GPUs are suitable for tasks with significant parallelization.\n",
    "\n",
    "<img src=\"Cpu vs Gpu.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install\n",
    "\n",
    "<mark style=\"background-color: Blue\">__In CPU__</mark>\n",
    "\n",
    "__For Windows__\n",
    "\n",
    "* Install PyTorch using conda\n",
    "      conda install pytorch torchvision cpuonly -c pytorch\n",
    "\n",
    "* Using pip\n",
    "      pip3 install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html\n",
    "        \n",
    "__For Mac__\n",
    "\n",
    "* Using conda\n",
    "      conda install pytorch torchvision -c pytorch\n",
    "    \n",
    "* Using pip\n",
    "      pip3 install torch torchvision\n",
    "    \n",
    "__For Linux__\n",
    "\n",
    "* Using conda\n",
    "      conda install pytorch torchvision cpuonly -c pytorch\n",
    "    \n",
    "* Using pip\n",
    "      pip3 install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html\n",
    "      \n",
    "      \n",
    "      \n",
    "<mark style=\"background-color: Green\">__In GPU__</mark>\n",
    "\n",
    "__For Windows__\n",
    "\n",
    "* Install PyTorch using conda cuda=9.2 and Python=3.6\n",
    "      conda install pytorch torchvision cudatoolkit=9.2 -c pytorch\n",
    "     \n",
    "* Using conda cuda=10.1 and Python=3.6 \n",
    "      conda install pytorch torchvision cudatoolkit=10.1 -c pytorch\n",
    "     \n",
    "* Install Pytorch using pip cuda=9.2 and Python=3.6\n",
    "      pip3 install torch==1.4.0+cu92 torchvision==0.5.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html\n",
    "     \n",
    "* Using pip cuda=10.1 and Python=3.6\n",
    "      pip3 install torch torchvision\n",
    "     \n",
    "__For Linux__\n",
    "\n",
    "* Install PyTorch using conda cuda=9.2 and Python=3.6\n",
    "      conda install pytorch torchvision cudatoolkit=9.2 -c pytorch\n",
    "     \n",
    "* Using conda cuda=10.1 and Python=3.6\n",
    "      conda install pytorch torchvision cudatoolkit=10.1 -c pytorch\n",
    "     \n",
    "* Install Pytorch using pip cuda=9.2 and Python=3.6\n",
    "      pip3 install torch==1.4.0+cu92 torchvision==0.5.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html\n",
    "     \n",
    "* Using pip cuda=10.1 and Python=3.6\n",
    "      pip3 install torch torchvision\n",
    "     \n",
    "__For Mac__\n",
    "\n",
    "* Install PyTorch using conda  for cuda=9.2 and 10.1 we can use same command and Python=3.6\n",
    "\n",
    "      conda install pytorch torchvision -c pytorch\n",
    "         # MacOS Binaries dont support CUDA, install from source if CUDA is needed\n",
    "       \n",
    "* Install Pytorch using pip for cuda=9.2 and 10.1 we can use same command and Python=3.6\n",
    "\n",
    "      pip3 install torch torchvision\n",
    "         # MacOS Binaries dont support CUDA, install from source if CUDA is needed\n",
    "      \n",
    "You have to run all these commands in __Anaconda Prompt__ , if you want to install in a notebook just put \" ! \" mark before the command like: !pip3 install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about Installation you can go through this site : \"https://pytorch.org/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensors\n",
    "\n",
    "Tensor is similar to Numpy's ndarray, the additional point for Tensors in we can use it in GPUs to accelerate computing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conda install pytorch torchvision cudatoolkit=9.2 -c pytorch\n",
    "conda install pytorch torchvision cudatoolkit=10.1 -c pytorch\n",
    "pip3 install torch==1.4.0+cu92 torchvision==0.5.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html\n",
    "pip3 install torch torchvision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note:__ Uninitialized matrix is declared, but doesn't contain definite known values before it is used. When we created an Unintialized matrix, whatever values were allocated inside the memory will apear as the initial values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct 6x3 matrix, uninitialized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[8.9082e-39, 7.5306e-39, 8.9082e-39],\n",
      "        [9.6429e-39, 9.4592e-39, 7.3470e-39],\n",
      "        [1.0653e-38, 1.0194e-38, 8.4490e-39],\n",
      "        [1.1112e-38, 9.5511e-39, 1.0102e-38],\n",
      "        [5.0510e-39, 1.0561e-38, 1.0653e-38],\n",
      "        [4.1327e-39, 8.9082e-39, 9.8265e-39]])\n"
     ]
    }
   ],
   "source": [
    "a = torch.empty(6,3)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a randomly initialized matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2941, 0.5160, 0.1714],\n",
      "        [0.9668, 0.3726, 0.3770],\n",
      "        [0.8461, 0.5253, 0.8368],\n",
      "        [0.0520, 0.3275, 0.9602]])\n"
     ]
    }
   ],
   "source": [
    "a = torch.rand(4,3)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a matrix filled zeros and of dtype long:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0]])\n"
     ]
    }
   ],
   "source": [
    "a = torch.zeros(4,3, dtype=torch.long)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a tensor with data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Tensor"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.tensor([7.8, 5])\n",
    "type(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or we can create new tensor with existing tensor.These methods we reuse its properties of input tensor, e.g. dtype, unless new values are provided by us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.]], dtype=torch.float64)\n",
      "tensor([[ 0.4651, -0.4985,  1.0767, -0.0487, -2.8595],\n",
      "        [ 0.0768, -2.2247,  0.0974, -0.7699, -0.2998],\n",
      "        [ 0.2819, -1.8610, -0.2074,  2.0058, -0.1106],\n",
      "        [-0.2680, -0.2563,  2.2448, -0.3995, -1.0533],\n",
      "        [ 1.0814,  0.1571, -0.2700,  1.1200,  1.3445],\n",
      "        [ 1.2481,  1.9628,  0.0129,  1.7214,  0.1227]])\n"
     ]
    }
   ],
   "source": [
    "a = a.new_ones(6,5, dtype=torch.double)    # new methods take in sizes\n",
    "print(a)\n",
    "\n",
    "a = torch.randn_like(a, dtype=torch.float)  # override dtype\n",
    "print(a)                                    # result will be the same size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 5])\n"
     ]
    }
   ],
   "source": [
    "print(a.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: <mark style=\"background-color: Yellow\">torch_size</mark> is actually a tuple, so it supports all tuple operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operations\n",
    "\n",
    "There are multiple syntaxes for operations. In the following examples, we used addition operation,\n",
    "\n",
    "Addition: syntax1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.4417, -0.3508,  1.8380,  0.0328, -2.3075],\n",
      "        [ 0.5720, -1.8070,  0.5284, -0.1215, -0.2671],\n",
      "        [ 0.4630, -0.9210,  0.5204,  2.9264,  0.7492],\n",
      "        [ 0.4013,  0.2857,  2.6331,  0.4339, -0.4086],\n",
      "        [ 1.6345,  1.0105, -0.1928,  1.2427,  2.2360],\n",
      "        [ 2.1688,  2.2751,  0.8861,  2.7092,  0.5242]])\n"
     ]
    }
   ],
   "source": [
    "b = torch.rand(6,5)\n",
    "print(a + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Addition: syntax2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.4417, -0.3508,  1.8380,  0.0328, -2.3075],\n",
      "        [ 0.5720, -1.8070,  0.5284, -0.1215, -0.2671],\n",
      "        [ 0.4630, -0.9210,  0.5204,  2.9264,  0.7492],\n",
      "        [ 0.4013,  0.2857,  2.6331,  0.4339, -0.4086],\n",
      "        [ 1.6345,  1.0105, -0.1928,  1.2427,  2.2360],\n",
      "        [ 2.1688,  2.2751,  0.8861,  2.7092,  0.5242]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.add(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Addition: providing an output as an argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.4417, -0.3508,  1.8380,  0.0328, -2.3075],\n",
      "        [ 0.5720, -1.8070,  0.5284, -0.1215, -0.2671],\n",
      "        [ 0.4630, -0.9210,  0.5204,  2.9264,  0.7492],\n",
      "        [ 0.4013,  0.2857,  2.6331,  0.4339, -0.4086],\n",
      "        [ 1.6345,  1.0105, -0.1928,  1.2427,  2.2360],\n",
      "        [ 2.1688,  2.2751,  0.8861,  2.7092,  0.5242]])\n"
     ]
    }
   ],
   "source": [
    "result = torch.empty(6, 5)\n",
    "torch.add(a, b, out=result)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Addition: in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.4417, -0.3508,  1.8380,  0.0328, -2.3075],\n",
      "        [ 0.5720, -1.8070,  0.5284, -0.1215, -0.2671],\n",
      "        [ 0.4630, -0.9210,  0.5204,  2.9264,  0.7492],\n",
      "        [ 0.4013,  0.2857,  2.6331,  0.4339, -0.4086],\n",
      "        [ 1.6345,  1.0105, -0.1928,  1.2427,  2.2360],\n",
      "        [ 2.1688,  2.2751,  0.8861,  2.7092,  0.5242]])\n"
     ]
    }
   ],
   "source": [
    "# adds a to b\n",
    "b.add_(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note:__ Any operation that mutates a tensor in-place is post-fixed with an <mark style=\"background-color: red\">_.</mark> For example: a.copy_(b), a.b_(), will change a."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use standard NumPy-like indexing with all bells and whistles!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 1.0767,  0.0974, -0.2074,  2.2448, -0.2700,  0.0129])\n"
     ]
    }
   ],
   "source": [
    "print(a[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resizing: We can resize or reshape tensor, use <mark style=\"background-color: Yellow\">tensor.view</mark> for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3]) torch.Size([9]) torch.Size([1, 9])\n"
     ]
    }
   ],
   "source": [
    "a = torch.randn(3, 3)\n",
    "b = a.view(9)\n",
    "c = a.view(-1, 9)  # the size -1 is inferred from other dimensions\n",
    "print(a.size(), b.size(), c.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have one value tensor, use <mark style=\"background-color: Yellow\">.item()</mark> to get the value of the Python number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.0956])\n",
      "-1.0956109762191772\n"
     ]
    }
   ],
   "source": [
    "a = torch.randn(1)\n",
    "print(a)\n",
    "print(a.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy Bridge\n",
    "\n",
    "Converting a Torch Tensor to NumPy array and vice versa is breeze.\n",
    "\n",
    "The Torch Tensor and NumPy array will share their underlying memory locations (if the Torch Tensor is on CPU), and changing one will change the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Converting a Torch tensor to NumPy array__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(4)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1. 1. 1.]\n"
     ]
    }
   ],
   "source": [
    "y = x.numpy()\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how numpy array changed in value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 2., 2., 2.])\n",
      "[2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "x.add_(1)\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Converting NumPy array to Torch tensor__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets see how changing the numpy array changed the Torch Tensor automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2. 2. 2. 2.]\n",
      "tensor([2., 2., 2., 2.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "f = np.ones(4)\n",
    "g = torch.from_numpy(f)\n",
    "np.add(f, 1, out=f)\n",
    "print(f)\n",
    "print(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the Tensors on the CPU except a CharTensor support converting to NumPy and back.\n",
    "\n",
    "__CUDA Tensors__\n",
    "\n",
    "Tensors can be moved onto any device using the <mark style=\"background-color: Yellow\">.to</mark> method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "# We will use ``torch.device`` objects to move tensors in and out of GPU\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")          # a CUDA device object\n",
    "    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU\n",
    "    x = x.to(device)                       # or just use strings ``.to(\"cuda\")``\n",
    "    z = x + y\n",
    "    print(z)\n",
    "    print(z.to(\"cpu\", torch.double))       # ``.to`` can also change dtype together!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AUTOGRAD : Automatic Differentiaition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definition - \n",
    "\n",
    "  - This class is an engine to calculate derivatives. It records the graph of all the operations performed on a gradient      enabled tensor and creates a acyclic graph called the dynamic computational graph(DCG).The leaves of this graph are input tensors and the roots are output tensors. Gradients are calculated by tracing the graph from the root to the leaf and multiplying every gradient in the way using the chain rule."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see this in some more easy terms with some examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor\n",
    "\n",
    "<mark style=\"background-color: Yellow\">torch.Tensor</mark> is the central class of the package.If we set its attribute <mark style=\"background-color: Yellow\">.requires_grad</mark> as <mark style=\"background-color: dark grey\">True</mark>, it starts to track all operations on it. When you finish your computation you can call <mark style=\"background-color: Yellow\">.backward()</mark> and have all the gradients computed automatically. The gradient of this tensor will be accumulated into <mark style=\"background-color: Yellow\">.grad</mark> attribute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To stop a tensor from tracking history, you can call <mark style=\"background-color: Yellow\">.detach()</mark> to detach it from the computation history, and to prevent future computation from being tracked.\n",
    "\n",
    "\n",
    "To prevent tracking history(and using memory), you can also wrap the code block in with <mark style=\"background-color: Yellow\">torch.no_grad():</mark>. This can be particularly helpful when evaluating a model because the model may have trainable parameters with <mark style=\"background-color: Yellow\">requires_grad=True</mark>, but for which we don’t need the gradients.\n",
    "\n",
    "There's one more class which is very important in autograd implementation - a <mark style=\"background-color: Yellow\">Function</mark>\n",
    "\n",
    "<mark style=\"background-color: Yellow\">Tensor</mark> and <mark style=\"background-color: Yellow\">Function</mark> are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each tensor has a <mark style=\"background-color: Yellow\">.grad_fn</mark> attribute that references a Function that has created the Tensor (except for Tensors created by the user - their <mark style=\"background-color: Yellow\">grad_fn is None</mark>).\n",
    "\n",
    "If you want to compute the derivatives, you can call <mark style=\"background-color: Yellow\">.backward()</mark> on a <mark style=\"background-color: Yellow\">Tensor</mark>. If <mark style=\"background-color: Yellow\">Tensor</mark> is a scalar (i.e. it holds a one element data), you don’t need to specify any arguments to <mark style=\"background-color: Yellow\">backward()</mark>, however if it has more elements, you need to specify a <mark style=\"background-color: Yellow\">gradient</mark> argument that is a tensor of matching shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a tensor and set requires_grad=True to track computation with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(2, 2, requires_grad=True)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do a tensor operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "b = a + 2\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b was created as a result of an operatio, so it has a <mark style=\"background-color: Yellow\">grad_fn</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<AddBackward0 object at 0x000002A19C7DB668>\n"
     ]
    }
   ],
   "source": [
    "print(b.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do more operation on b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "c = b * b * 3\n",
    "out = c.mean()\n",
    "\n",
    "print(c, out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark style=\"background-color: Yellow\">.requires_grad_( ... )</mark>  changes an existing Tensor’s <mark style=\"background-color: Yellow\">requires_grad</mark> flag in-place. The input flag defaults to False if not given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n",
      "<SumBackward0 object at 0x000002A19C7E04E0>\n"
     ]
    }
   ],
   "source": [
    "p = torch.randn(3, 3)\n",
    "p = ((p * 3) / (p - 1))\n",
    "print(p.requires_grad)\n",
    "p.requires_grad_(True)\n",
    "print(p.requires_grad)\n",
    "q = (p * p).sum()\n",
    "print(q.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradients\n",
    "\n",
    "Let's backdrop now. Because <mark style=\"background-color: magenta\">out</mark> contains a single scalar, <mark style=\"background-color: Yellow\">out.backword</mark> is equivalent to  <mark style=\"background-color: Yellow\">out.backward(torch.tensor(1.))</mark>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print gradients d(out)/dx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "source": [
    "print(a.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have got a matrix of 4.5. Let’s call the out Tensor “o”. We have that \n",
    "\n",
    "$o= \\frac{1}{4} \\sum c_i $ $c_i= 3(a_i+2)^2$  and $c_i|_{a_i=1}= 27$ . \n",
    "\n",
    "Therefore, $\\frac{\\partial_0}{\\partial a_i}=\\frac{3(x_i+2)}{2}$, \n",
    "\n",
    "hence\n",
    "$\\frac{\\partial_0}{\\partial a_i}|_{a_i=1} = \\frac{9}{2} = 4.5$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mathematically - Jacobians and vectors\n",
    "\n",
    "Mathematically, the autograd class is just a Jacobian-vector product computing engine. A Jacobian matrix in very simple words is a matrix representing all the possible partial derivatives of two vectors. It’s the gradient of a vector with respect to another vector.\n",
    "\n",
    "If a vector X = [x1, x2,….xn] is used to calculate some other vector f(X) = [f1, f2, …. fn] through a function f then the Jacobian matrix (J) simply contains all the partial derivative combinations as follows:\n",
    "\n",
    "<img src=\"jacobian-vector.png\">\n",
    "\n",
    "Above matrix represents the gradient of f(X)with respect to X\n",
    "Suppose a PyTorch gradient enabled tensors X as:\n",
    "X = [x1, x2, ….. xn] (Let this be the weights of some machine learning model)\n",
    "X undergoes some operations to form a vector Y\n",
    "Y = f(X) = [y1, y2, …. ym]\n",
    "Y is then used to calculate a scalar loss l. Suppose a vector v happens to be the gradient of the scalar loss l with respect the vector Y as follows\n",
    "\n",
    "<img src=\"jacob1.png\">       *The vector v is called the grad_tensor and passed to the backward() function as an argument*\n",
    "\n",
    "To get the gradient of the loss l with respect to the weights X the Jacobian matrix J is vector-multiplied with the vector v.\n",
    "\n",
    "<img src=\"jac1.png\"> \n",
    "\n",
    "This method of calculating the Jacobian matrix and multiplying it with a vector v enables the possibility for PyTorch to feed external gradients with ease for even the non-scalar outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's have a look at an example of vector-Jacobian product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([  622.1337,  1041.7932, -1389.5249], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now in this case y is no longer a scalar.<mark style=\"background-color: Yellow\">torch.autograd(torch.tensor(1.))</mark> could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to backward as argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2.0480e+02, 2.0480e+03, 2.0480e-01])\n"
     ]
    }
   ],
   "source": [
    "v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)\n",
    "y.backward(v)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also stop autograd from tracking history on Tensors with <mark style=\"background-color: Yellow\">.requires_grad=True</mark> either by wrapping the code block in with <mark style=\"background-color: Yellow\">torch.no_grad()</mark>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "\n",
    "with torch.no_grad():\n",
    "    print((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by using .detach() to get a new Tensor with the same content but that does not require gradients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n",
      "tensor(True)\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "y = x.detach()\n",
    "print(y.requires_grad)\n",
    "print(x.eq(y).all())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "We can construct neural networks using the <mark style=\"background-color: Yellow\">torch.nn</mark> package.\n",
    "\n",
    "Now that you had a glimpse of autograd, nn depends on autograd to define models and differentiate them. An nn.Module contains layers, and a method forward(input)that returns the output.\n",
    "\n",
    "For example, look at this network that classifies digit images:\n",
    "<img src=\"neural.png\"> \n",
    "\n",
    "\n",
    "### convnet\n",
    "\n",
    "It is a simple feed-forward network. It takes the input,feeds it through several layers one after the other, and then finally gives the output.\n",
    "\n",
    "A typical training procedure for a neural network is as follows:\n",
    "- Define the neural network that has some learnable parameters (or weights)\n",
    "- Iterate over a dataset of inputs\n",
    "- Process input through the network\n",
    "- Compute the loss (how far is the output from being correct)\n",
    "- Propagate gradients back into the network’s parameters\n",
    "- Update the weights of the network, typically using a simple update rule: <mark style=\"background-color: light-blue\">weight = weight - learning_rate * gradient</mark>\n",
    "        \n",
    "### Define the network\n",
    "\n",
    "Let's define the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "network(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=576, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class network(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(network, self).__init__()\n",
    "        # 1 input image channel, 6 output channels, 3x3 square convolution\n",
    "        # kernel\n",
    "        self.conv1 = nn.Conv2d(1, 6, 3)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 3)\n",
    "        # an affine operation: y = mx + b\n",
    "        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Max pooling over a (2, 2) window\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        # If the size is a square you can only specify a single number\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, self.num_flat_features(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "    def num_flat_features(self, x):\n",
    "        size = x.size()[1:]  # all dimensions except the batch dimension\n",
    "        num_features = 1\n",
    "        for s in size:\n",
    "            num_features *= s\n",
    "        return num_features\n",
    "\n",
    "\n",
    "net = network()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You just have to define the <mark style=\"background-color: yellow\">forward</mark>,  and the <mark style=\"background-color: yellow\">backward</mark> function (where gradients are computed) is automatically defined for you using autograd. You can use any of the Tensor operations in the <mark style=\"background-color: yellow\">forward</mark> function.\n",
    "\n",
    "The learnable parameters of a model are returned by <mark style=\"background-color: yellow\">net.parameters()</mark>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "torch.Size([6, 1, 3, 3])\n"
     ]
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))\n",
    "print(params[0].size())  # conv1's .weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s try a random 32x32 input. Note: expected input size of this net (LeNet) is 32x32. To use this net on the MNIST dataset, please resize the images from the dataset to 32x32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0746, -0.0389, -0.0148, -0.0516, -0.0599,  0.1114, -0.0071, -0.0573,\n",
      "         -0.0589, -0.1054]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "inp = torch.randn(1, 1, 32, 32)\n",
    "out = net(inp)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zero the gradient buffers of all parameters and backprops with random gradients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad()\n",
    "out.backward(torch.randn(1, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note:__\n",
    "<mark style=\"background-color: yellow\">torch.nn</mark> only supports mini-batches. The entire <mark style=\"background-color: yellow\">torch.nn</mark> package only supports inputs that are a mini-batch of samples, and not a single sample.\n",
    "For example, <mark style=\"background-color: yellow\">nn.Conv2d</mark> will take in a 4D Tensor of <mark style=\"background-color: yellow\">nSamples x nChannels x Height x Width</mark>.\n",
    "If you have a single sample, just use <mark style=\"background-color: yellow\">input.unsqueeze(0)</mark> to add a fake batch dimension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__At this point, we covered:__\n",
    "- Defined neutral network\n",
    "- Processing input and calling backward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function\n",
    "\n",
    "A loss function takes the (output, target) pair of inputs, and computes a value that estimates how far away the output is from the target.\n",
    "There are several different loss functions under the nn package . A simple loss is: <mark style=\"background-color: yellow\">nn.MSELoss</mark> which computes the mean-squared error between the input and the target.\n",
    "    \n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.8201, grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "output = net(inp)\n",
    "target = torch.randn(10)  # a dummy target, for example\n",
    "target = target.view(1, -1)  # make it the same shape as output\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "loss = criterion(output, target)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if you follow loss in the backward direction, using its <mark style=\"background-color: yellow\">.grad_fn</mark>attribute, you will see a graph of computations that looks like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "input  ->   conv2d  ->   relu  ->   maxpool2d  ->   conv2d ->   relu  ->   maxpool2d\n",
    "      \n",
    "      -> view -> linear -> relu -> linear -> relu -> linear\n",
    "      \n",
    "      -> MSELoss\n",
    "      \n",
    "      -> loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, when we call <mark style=\"background-color: yellow\">loss.backward()</mark>, the whole graph is differentiated w.r.t. the loss, and all Tensors in the graph that has requires_grad=True will have their .grad Tensor accumulated with the gradient.\n",
    "\n",
    "\n",
    "For illustration, let us follow a few steps <mark style=\"background-color: yellow\">backward</mark>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MseLossBackward object at 0x000001552B599108>\n",
      "<AddmmBackward object at 0x000001552B5990C8>\n",
      "<AccumulateGrad object at 0x000001552B595308>\n"
     ]
    }
   ],
   "source": [
    "print(loss.grad_fn)  # MSELoss\n",
    "print(loss.grad_fn.next_functions[0][0])  # Linear\n",
    "print(loss.grad_fn.next_functions[0][0].next_functions[0][0])  # ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backprop\n",
    "\n",
    "To backpropagate the error all we have to do is to <mark style=\"background-color: yellow\">loss.backward()</mark>. You need to clear the existing gradients though, else gradients will be accumulated to existing gradients.\n",
    "\n",
    "\n",
    "Now we shall call <mark style=\"background-color: yellow\">loss.backward()</mark>, and have a look at conv1’s bias gradients before and after the backward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.bias.grad before backward\n",
      "tensor([0., 0., 0., 0., 0., 0.])\n",
      "conv1.bias.grad after backward\n",
      "tensor([ 0.0048,  0.0182, -0.0052,  0.0043,  0.0079, -0.0025])\n"
     ]
    }
   ],
   "source": [
    "net.zero_grad()     # zeroes the gradient buffers of all parameters\n",
    "\n",
    "print('conv1.bias.grad before backward')\n",
    "print(net.conv1.bias.grad)\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "print('conv1.bias.grad after backward')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update the weights\n",
    "\n",
    "The simplest update rule used in practice is the Stochastic Gradient Descent (SGD):\n",
    "    \n",
    "   <mark style=\"background-color: yellow\"> weight = weight - learning_rate * gradient </mark>\n",
    "   \n",
    "We can implement this using simple Python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "for f in net.parameters():\n",
    "    f.data.sub_(f.grad.data * learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, as you use neural networks, you want to use various different update rules such as SGD, Nesterov-SGD, Adam, RMSProp, etc. To enable this, we built a small package: torch.optim that implements all these methods. Using it is very simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# create your optimizer\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "# in your training loop:\n",
    "optimizer.zero_grad()   # zero the gradient buffers\n",
    "output = net(inp)\n",
    "loss = criterion(output, target)\n",
    "loss.backward()\n",
    "optimizer.step()    # Does the update"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how gradient buffers had to be manually set to zero using <mark style=\"background-color:yellow\">optimizer.zero_grad()</mark>. This is because gradients are accumulated as explained in the <mark style=\"font-color:red\">Backprop</mark> section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}