6_7_Part_3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "colab": {
      "name": "6_7_Part_3.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/GeoLabUniLaSalle/Python/blob/main/6_7_Part_3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 0,
        "id": "9QKPyH8N2eh0"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "# **3. Automatic Differentiation**\n",
        "\n",
        "\n",
        "Differentiation is a crucial step in nearly all deep learning optimization algorithms.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3pXiMPwgbn2"
      },
      "source": [
        "# **3.1 Gradients**\n",
        "\n",
        "\n",
        "We can concatenate partial derivatives of a multivariate function with respect to all its variables to obtain the *gradient* vector of the function.\n",
        "Suppose that the input of function $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is an $n$-dimensional vector $\\mathbf{x} = [x_1, x_2, \\ldots, x_n]^\\top$ and the output is a scalar. The gradient of the function $f(\\mathbf{x})$ with respect to $\\mathbf{x}$ is a vector of $n$ partial derivatives:\n",
        "\n",
        "$$\\nabla_{\\mathbf{x}} f(\\mathbf{x}) = \\bigg[\\frac{\\partial f(\\mathbf{x})}{\\partial x_1}, \\frac{\\partial f(\\mathbf{x})}{\\partial x_2}, \\ldots, \\frac{\\partial f(\\mathbf{x})}{\\partial x_n}\\bigg]^\\top,$$\n",
        "\n",
        "where $\\nabla_{\\mathbf{x}} f(\\mathbf{x})$ is often replaced by $\\nabla f(\\mathbf{x})$ when there is no ambiguity.\n",
        "\n",
        "Let $\\mathbf{x}$ be an $n$-dimensional vector, the following rules are often used when differentiating multivariate functions:\n",
        "\n",
        "* For all $\\mathbf{A} \\in \\mathbb{R}^{m \\times n}$, $\\nabla_{\\mathbf{x}} \\mathbf{A} \\mathbf{x} = \\mathbf{A}^\\top$,\n",
        "* For all  $\\mathbf{A} \\in \\mathbb{R}^{n \\times m}$, $\\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{A}  = \\mathbf{A}$,\n",
        "* For all  $\\mathbf{A} \\in \\mathbb{R}^{n \\times n}$, $\\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{A} \\mathbf{x}  = (\\mathbf{A} + \\mathbf{A}^\\top)\\mathbf{x}$,\n",
        "* $\\nabla_{\\mathbf{x}} \\|\\mathbf{x} \\|^2 = \\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{x} = 2\\mathbf{x}$.\n",
        "\n",
        "Similarly, for any matrix $\\mathbf{X}$, we have $\\nabla_{\\mathbf{X}} \\|\\mathbf{X} \\|_F^2 = 2\\mathbf{X}$. As we will see later, gradients are useful for designing optimization algorithms in deep learning.\n",
        "\n",
        "Any Machine learning model will learn to predict better using an algorithm knows as Backpropagation. During backpropagation you need gradients to updates with new weights. This is done with the .backward() method.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5HcwIA0od57"
      },
      "source": [
        "# **Example**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IUnTlRNfouLj",
        "outputId": "64edf741-6dc7-42cc-a2fd-0fbebf2f113c"
      },
      "source": [
        "import torch\n",
        "x = torch.tensor(1.0, requires_grad = True) # if you set requires_grad to True to any tensor, then PyTorch will automatically track and calculate gradients for that tensor  \n",
        "z = x ** 3 # z=x^3\n",
        "z.backward() #Computes the gradient \n",
        "print(x.grad.data) # this is dz/dx"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(3.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c80jS2pXgKN_"
      },
      "source": [
        "# **3.2 Chain Rule**\n",
        "\n",
        "However, such gradients can be hard to find.\n",
        "This is because multivariate functions in deep learning are often *composite*,\n",
        "so we may not apply any of the aforementioned rules to differentiate these functions.\n",
        "Fortunately, the *chain rule* enables us to differentiate composite functions.\n",
        "\n",
        "Let us first consider functions of a single variable.\n",
        "Suppose that functions $y=f(u)$ and $u=g(x)$ are both differentiable, then the chain rule states that\n",
        "\n",
        "$$\\frac{dy}{dx} = \\frac{dy}{du} \\frac{du}{dx}.$$\n",
        "\n",
        "Now let us turn our attention to a more general scenario\n",
        "where functions have an arbitrary number of variables.\n",
        "Suppose that the differentiable function $y$ has variables\n",
        "$u_1, u_2, \\ldots, u_m$, where each differentiable function $u_i$\n",
        "has variables $x_1, x_2, \\ldots, x_n$.\n",
        "Note that $y$ is a function of $x_1, x_2, \\ldots, x_n$.\n",
        "Then the chain rule gives\n",
        "\n",
        "$$\\frac{dy}{dx_i} = \\frac{dy}{du_1} \\frac{du_1}{dx_i} + \\frac{dy}{du_2} \\frac{du_2}{dx_i} + \\cdots + \\frac{dy}{du_m} \\frac{du_m}{dx_i}$$\n",
        "\n",
        "for any $i = 1, 2, \\ldots, n$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_WkaXvSfq7Y"
      },
      "source": [
        "# **3.3. A Simple Example**\n",
        "\n",
        "As a toy example, say that we are interested\n",
        "in differentiating the function\n",
        "$y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
        "with respect to the column vector $\\mathbf{x}$.\n",
        "To start, let us create the variable `x` and assign it an initial value.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 2,
        "tab": [
          "pytorch"
        ],
        "id": "_-NF_6Dj2eh4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a1a2ffa2-4027-46c9-ebda-3bc8cd260b93"
      },
      "source": [
        "import torch\n",
        "\n",
        "x = torch.arange(4.0) \n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0., 1., 2., 3.])"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 4,
        "id": "P7fNqESF2eh5"
      },
      "source": [
        "Before we even calculate the gradient\n",
        "of $y$ with respect to $\\mathbf{x}$,\n",
        "we will need a place to store it.\n",
        "It is important that we do not allocate new memory\n",
        "every time we take a derivative with respect to a parameter\n",
        "because we will often update the same parameters\n",
        "thousands or millions of times\n",
        "and could quickly run out of memory. \n",
        "Note that a gradient of a scalar-valued function\n",
        "with respect to a vector $\\mathbf{x}$\n",
        "is itself vector-valued and has the same shape as $\\mathbf{x}$.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 6,
        "tab": [
          "pytorch"
        ],
        "id": "RCS3LN732eh6"
      },
      "source": [
        "x.requires_grad_(True)  # Same as `x = torch.arange(4.0, requires_grad=True)`\n",
        "x.grad  # The default value is None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 8,
        "id": "TfysLK9H2eh7"
      },
      "source": [
        "Now let us calculate $y$.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 10,
        "tab": [
          "pytorch"
        ],
        "id": "sXUcLh6B2eh7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e04b14ad-6cca-413d-f366-c77402fedfe7"
      },
      "source": [
        "y = 2 * torch.dot(x, x) # y = 2*(x1*x1 + x2*x2 + x3*x3 + x4*x4)\n",
        "y"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(28., grad_fn=<MulBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 12,
        "id": "xBCuUUJg2eh7"
      },
      "source": [
        "Since `x` is a vector of length 4,\n",
        "an dot product of `x` and `x` is performed,\n",
        "yielding the scalar output that we assign to `y`.\n",
        "Next, we can automatically calculate the gradient of `y`\n",
        "with respect to each component of `x`\n",
        "by calling the function for backpropagation and printing the gradient.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 14,
        "tab": [
          "pytorch"
        ],
        "id": "-0rrifzA2eh8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5f3ef59a-9959-4652-822c-f02ba1f345a0"
      },
      "source": [
        "y.backward()\n",
        "x.grad"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 0.,  4.,  8., 12.])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 16,
        "id": "NvGHjPdQ2eh9"
      },
      "source": [
        "The gradient of the function $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
        "with respect to $\\mathbf{x}$ should be $4\\mathbf{x}$.\n",
        "Let us quickly verify that our desired gradient was calculated correctly.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 18,
        "tab": [
          "pytorch"
        ],
        "id": "cKHS9iEY2eh9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fae71f70-0335-4e43-ad71-32469a61c781"
      },
      "source": [
        "x.grad == 4 * x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([True, True, True, True])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 20,
        "id": "2FyUX37p2eh-"
      },
      "source": [
        "Now let us calculate another function of `x`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gFVuUASARt3c",
        "outputId": "1e5a619b-767c-4e26-e922-293ab263e03f"
      },
      "source": [
        "x.grad.zero_()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0., 0., 0., 0.])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 22,
        "tab": [
          "pytorch"
        ],
        "id": "hxM6refP2eh-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ac6284fa-abd7-4460-b608-6fbfc3de03f6"
      },
      "source": [
        "# PyTorch accumulates the gradient in default, we need to clear the previous\n",
        "# values\n",
        "x.grad.zero_()\n",
        "y = x.sum() # y = x1 + x2 + x3 + x4\n",
        "y.backward()\n",
        "x.grad"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1., 1., 1., 1.])"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 24,
        "id": "tz_KkGL62eh-"
      },
      "source": [
        "# **3.4. Backward for Non-Scalar Variables**\n",
        "\n",
        "Technically, when `y` is not a scalar,\n",
        "the most natural interpretation of the differentiation of a vector `y`\n",
        "with respect to a vector `x` is a matrix.\n",
        "For higher-order and higher-dimensional `y` and `x`,\n",
        "the differentiation result could be a high-order tensor.\n",
        "\n",
        "However, while these more exotic objects do show up\n",
        "in advanced machine learning including **in deep learning**,\n",
        "more often **when we are calling backward on a vector,**\n",
        "we are trying to calculate the derivatives of the loss functions\n",
        "for each constituent of a *batch* of training examples.\n",
        "Here, our intent is not to calculate the differentiation matrix\n",
        "but rather **the sum of the partial derivatives\n",
        "computed individually for each example** in the batch.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 26,
        "tab": [
          "pytorch"
        ],
        "id": "62QnrSJ52eh_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "efa50910-9205-4795-ea48-75c6d5205c5f"
      },
      "source": [
        "# Invoking `backward` on a non-scalar requires passing in a `gradient` argument\n",
        "# which specifies the gradient of the differentiated function w.r.t `self`.\n",
        "# In our case, we simply want to sum the partial derivatives, so passing\n",
        "# in a gradient of ones is appropriate\n",
        "\n",
        "x.grad.zero_()  \n",
        "y = x * x  # y is a vector y = [x1*x1 , x2*x2, x3*x3, x4*x4]\n",
        "# y.backward(torch.ones(len(x))) equivalent to the below\n",
        "y.sum().backward()  # we derive the sum of components of vector y (needs explanations) \n",
        "x.grad"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0., 2., 4., 6.])"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kDVhdcnFDigq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fe007ba8-6e73-4db5-f228-8fa72df7a03d"
      },
      "source": [
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0., 1., 2., 3.], requires_grad=True)"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 28,
        "id": "uVbHZ_-E2eh_"
      },
      "source": [
        "# **3.5. Detaching Computation**\n",
        "\n",
        "Sometimes, we wish to move some calculations\n",
        "outside of the recorded computational graph.\n",
        "For example, say that `y` was calculated as a function of `x`,\n",
        "and that subsequently `z` was calculated as a function of both `y` and `x`.\n",
        "Now, imagine that we wanted to calculate\n",
        "the gradient of `z` with respect to `x`,\n",
        "but wanted for some reason to treat `y` as a constant,\n",
        "and only take into account the role\n",
        "that `x` played after `y` was calculated.\n",
        "\n",
        "Here, we can detach `y` to return a new variable `u`\n",
        "that has the same value as `y` but discards any information\n",
        "about how `y` was computed in the computational graph.\n",
        "In other words, the gradient will not flow backwards through `u` to `x`.\n",
        "Thus, the following backpropagation function computes\n",
        "the partial derivative of `z = u * x` with respect to `x` while treating `u` as a constant,\n",
        "instead of the partial derivative of `z = x * x * x` with respect to `x`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 30,
        "tab": [
          "pytorch"
        ],
        "id": "RY7qr2xq2eiA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "23243ca8-a93b-4761-91f5-378577ac5b19"
      },
      "source": [
        "x.grad.zero_()\n",
        "y = x * x # y is a vector y = [x1*x1 , x2*x2, x3*x3, x4*x4]\n",
        "u = y.detach()\n",
        "z = u * x # z is a vector z = [u1*x1 , u2*x2, u3*x3, u4*x4]\n",
        "\n",
        "z.sum().backward() # gradient of u1*x1 + u2*x2 +  u3*x3 + u4*x4 with respect to x = [x1 , x2, x3, x4]\n",
        "x.grad == u"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([True, True, True, True])"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 32,
        "id": "H4jTUOjk2eiA"
      },
      "source": [
        "Since the computation of `y` was recorded,\n",
        "we can subsequently invoke backpropagation on `y` to get the derivative of `y = x * x` with respect to `x`, which is `2 * x`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 34,
        "tab": [
          "pytorch"
        ],
        "id": "Mm7Ubc1T2eiA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "17b9972c-df35-460a-a966-992b13b6b504"
      },
      "source": [
        "x.grad.zero_()\n",
        "y.sum().backward() # gradient of x1*x1 + x2*x2 +  x3*x3 + x4*x4 with respect to x = [x1 , x2, x3, x4]\n",
        "x.grad == 2 * x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([True, True, True, True])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 36,
        "id": "YOGKpj7R2eiB"
      },
      "source": [
        "# **3.6. Computing the Gradient of Python Control Flow**\n",
        "\n",
        "One benefit of using automatic differentiation\n",
        "is that **even if** building the computational graph of a function\n",
        "required passing through a maze of Python control flow\n",
        "(e.g., conditionals, loops, and arbitrary function calls),\n",
        "we can still calculate the gradient of the resulting variable.\n",
        "In the following snippet, note that\n",
        "the number of iterations of the `while` loop\n",
        "and the evaluation of the `if` statement\n",
        "both depend on the value of the input `a`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 38,
        "tab": [
          "pytorch"
        ],
        "id": "pfeCuYaM2eiB"
      },
      "source": [
        "def f(a):\n",
        "    b = a * 2\n",
        "    while b.norm() < 1000:\n",
        "        b = b * 2\n",
        "    if b.sum() > 0:\n",
        "        c = b\n",
        "    else:\n",
        "        c = 100 * b\n",
        "    return c"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 40,
        "id": "exapl4Dv2eiB"
      },
      "source": [
        "Let us compute the gradient.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 42,
        "tab": [
          "pytorch"
        ],
        "id": "vG1eJg7r2eiB"
      },
      "source": [
        "a = torch.randn(size=(), requires_grad=True)\n",
        "d = f(a)\n",
        "d.backward() # gradient of the function d with respect to vector a = [a1 , a2, a3, ...]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 44,
        "id": "SvXqHyVh2eiB"
      },
      "source": [
        "We can now analyze the `f` function defined above.\n",
        "Note that it is piecewise linear in its input `a`.\n",
        "In other words, for any `a` there exists some constant scalar `k`\n",
        "such that `f(a) = k * a`, where the value of `k` depends on the input `a`.\n",
        "Consequently `d / a` allows us to verify that the gradient is correct.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "origin_pos": 46,
        "tab": [
          "pytorch"
        ],
        "id": "onbmpOKQ2eiC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08046493-eda0-4b7e-f9cb-eafd93ed007a"
      },
      "source": [
        "a.grad == d / a"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(True)"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 48,
        "id": "X3k4NhZ42eiC"
      },
      "source": [
        "# **Summary**\n",
        "\n",
        "* Machine/Deep learning frameworks can automate the calculation of derivatives. To use it, we first attach gradients to those variables with respect to which we desire partial derivatives. We then record the computation of our target value, execute its function for backpropagation, and access the resulting gradient.\n",
        "\n",
        "\n"
      ]
    }
  ]
}