Data_Manipulation_Deep_Learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 0,
        "id": "kVnVrPpqljvF"
      },
      "source": [
        "# Data Manipulation\n",
        "\n",
        "In order to get anything done, we need some way to store and manipulate data.\n",
        "Generally, there are two important things we need to do with data: (i) acquire\n",
        "them; and (ii) process them once they are inside the computer.  There is no\n",
        "point in acquiring data without some way to store it, so let us get our hands\n",
        "dirty first by playing with synthetic data.  To start, we introduce the\n",
        "$n$-dimensional array, which is also called the *tensor*.\n",
        "\n",
        "If you have worked with NumPy, the most widely-used\n",
        "scientific computing package in Python,\n",
        "then you will find this section familiar.\n",
        "No matter which framework you use,\n",
        "its *tensor class* (`ndarray` in MXNet,\n",
        "`Tensor` in both PyTorch and TensorFlow) is similar to NumPy's `ndarray` with\n",
        "a few killer features.\n",
        "First, GPU is well-supported to accelerate the computation\n",
        "whereas NumPy only supports CPU computation.\n",
        "Second, the tensor class\n",
        "supports automatic differentiation.\n",
        "These properties make the tensor class suitable for deep learning.\n",
        "Throughout the class note, when we say tensors,\n",
        "we are referring to instances of the tensor class unless otherwise stated.\n",
        "\n",
        "## Getting Started\n",
        "\n",
        "In this section, we aim to get you up and running,\n",
        "equipping you with the basic math and numerical computing tools\n",
        "that you will build on as you progress through the book.\n",
        "Do not worry if you struggle to grok some of\n",
        "the mathematical concepts or library functions.\n",
        "The following sections will revisit this material\n",
        "in the context of practical examples and it will sink.\n",
        "On the other hand, if you already have some background\n",
        "and want to go deeper into the mathematical content, just skip this section.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 1,
        "tab": [
          "mxnet"
        ],
        "id": "ru4omVONljvL"
      },
      "source": [
        "To start, we import the `np` (`numpy`) and\n",
        "`npx` (`numpy_extension`) modules from MXNet.\n",
        "Here, the `np` module includes functions supported by NumPy,\n",
        "while the `npx` module contains a set of extensions\n",
        "developed to empower deep learning within a NumPy-like environment.\n",
        "When using tensors, we almost always invoke the `set_np` function:\n",
        "this is for compatibility of tensor processing by other components of MXNet.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "origin_pos": 4,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gs2sUl9sljvM",
        "outputId": "d7d471a6-f0d6-4330-8fbb-0ed27b74ce65"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting mxnet\n",
            "  Downloading mxnet-1.9.0-py3-none-manylinux2014_x86_64.whl (47.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 47.3 MB 1.2 MB/s \n",
            "\u001b[?25hCollecting graphviz<0.9.0,>=0.8.1\n",
            "  Downloading graphviz-0.8.4-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.7/dist-packages (from mxnet) (2.23.0)\n",
            "Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.7/dist-packages (from mxnet) (1.19.5)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (2021.10.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (3.0.4)\n",
            "Installing collected packages: graphviz, mxnet\n",
            "  Attempting uninstall: graphviz\n",
            "    Found existing installation: graphviz 0.10.1\n",
            "    Uninstalling graphviz-0.10.1:\n",
            "      Successfully uninstalled graphviz-0.10.1\n",
            "Successfully installed graphviz-0.8.4 mxnet-1.9.0\n"
          ]
        }
      ],
      "source": [
        "!pip install mxnet\n",
        "from mxnet import np, npx\n",
        "\n",
        "npx.set_np()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 7,
        "id": "pSRrta8-ljvN"
      },
      "source": [
        "[**A tensor represents a (possibly multi-dimensional) array of numerical values.**]\n",
        "With one axis, a tensor corresponds (in math) to a *vector*.\n",
        "With two axes, a tensor corresponds to a *matrix*.\n",
        "Tensors with more than two axes do not have special\n",
        "mathematical names.\n",
        "\n",
        "To start, we can use `arange` to create a row vector `x`\n",
        "containing the first 12 integers starting with 0,\n",
        "though they are created as floats by default.\n",
        "Each of the values in a tensor is called an *element* of the tensor.\n",
        "For instance, there are 12 elements in the tensor `x`.\n",
        "Unless otherwise specified, a new tensor\n",
        "will be stored in main memory and designated for CPU-based computation.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "origin_pos": 8,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dwDADgSCljvO",
        "outputId": "2e6cb21c-0476-48bf-8afb-fab797d4c567"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.])"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "x = np.arange(12)\n",
        "x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 11,
        "id": "I8J-ba0VljvP"
      },
      "source": [
        "**We can access a tensor's *shape*** (the length along each axis)\n",
        "by inspecting its `shape` property.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "origin_pos": 12,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tS8BQJGHljvQ",
        "outputId": "fb7b1a5c-9ae0-4897-ff74-7101ecb8bcd8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(12,)"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "x.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 13,
        "id": "PsU4Od-1ljvQ"
      },
      "source": [
        "If we just want to know the total number of elements in a tensor,\n",
        "i.e., the product of all of the shape elements,\n",
        "we can inspect its size.\n",
        "Because we are dealing with a vector here,\n",
        "the single element of its `shape` is identical to its size.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "origin_pos": 14,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OtI2wnviljvR",
        "outputId": "d0a71594-a068-4b60-fdec-96161861783a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "12"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "x.size"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 17,
        "id": "kcLDxN1mljvS"
      },
      "source": [
        "To [**change the shape of a tensor without altering\n",
        "either the number of elements or their values**],\n",
        "we can invoke the `reshape` function.\n",
        "For example, we can transform our tensor, `x`,\n",
        "from a row vector with shape (12,) to a matrix with shape (3, 4).\n",
        "This new tensor contains the exact same values,\n",
        "but views them as a matrix organized as 3 rows and 4 columns.\n",
        "To reiterate, although the shape has changed,\n",
        "the elements have not.\n",
        "Note that the size is unaltered by reshaping.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "origin_pos": 18,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Yz8bBO8dljvS",
        "outputId": "eda7daa9-0c31-4294-c016-58415a5818f4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.,  1.,  2.,  3.],\n",
              "       [ 4.,  5.,  6.,  7.],\n",
              "       [ 8.,  9., 10., 11.]])"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "X = x.reshape(3, 4)\n",
        "X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 20,
        "id": "UvrSU40RljvT"
      },
      "source": [
        "Reshaping by manually specifying every dimension is unnecessary.\n",
        "If our target shape is a matrix with shape (height, width),\n",
        "then after we know the width, the height is given implicitly.\n",
        "Why should we have to perform the division ourselves?\n",
        "In the example above, to get a matrix with 3 rows,\n",
        "we specified both that it should have 3 rows and 4 columns.\n",
        "Fortunately, tensors can automatically work out one dimension given the rest.\n",
        "We invoke this capability by placing `-1` for the dimension\n",
        "that we would like tensors to automatically infer.\n",
        "In our case, instead of calling `x.reshape(3, 4)`,\n",
        "we could have equivalently called `x.reshape(-1, 4)` or `x.reshape(3, -1)`.\n",
        "\n",
        "Typically, we will want our matrices initialized\n",
        "either with zeros, ones, some other constants,\n",
        "or numbers randomly sampled from a specific distribution.\n",
        "[**We can create a tensor representing a tensor with all elements\n",
        "set to 0**]\n",
        "and a shape of (2, 3, 4) as follows:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "origin_pos": 21,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2-KCuLqfljvT",
        "outputId": "1a4a7a21-3804-4101-a11e-b72bfe1aa365"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[[0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.]],\n",
              "\n",
              "       [[0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.]]])"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "np.zeros((2, 3, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 24,
        "id": "n7JFbxSzljvU"
      },
      "source": [
        "Similarly, we can create tensors with each element set to 1 as follows:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "origin_pos": 25,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qAzT17KwljvU",
        "outputId": "c0f87744-329f-4683-bc5b-26f9d04306de"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[[1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.]],\n",
              "\n",
              "       [[1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.]]])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "np.ones((2, 3, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 28,
        "id": "hlKC3XdFljvV"
      },
      "source": [
        "Often, we want to [**randomly sample the values\n",
        "for each element in a tensor**]\n",
        "from some probability distribution.\n",
        "For example, when we construct arrays to serve\n",
        "as parameters in a neural network, we will\n",
        "typically initialize their values randomly.\n",
        "The following snippet creates a tensor with shape (3, 4).\n",
        "Each of its elements is randomly sampled\n",
        "from a standard Gaussian (normal) distribution\n",
        "with a mean of 0 and a standard deviation of 1.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "origin_pos": 29,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CFZaaPJ6ljvV",
        "outputId": "a468c35c-d7b3-49ff-a894-295d9d5614da"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 2.2122064 ,  1.1630787 ,  0.7740038 ,  0.4838046 ],\n",
              "       [ 1.0434403 ,  0.29956347,  1.1839255 ,  0.15302546],\n",
              "       [ 1.8917114 , -1.1688148 , -1.2347414 ,  1.5580711 ]])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "np.random.normal(0, 1, size=(3, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 32,
        "id": "Prb46x3gljvV"
      },
      "source": [
        "We can also [**specify the exact values for each element**] in the desired tensor\n",
        "by supplying a Python list (or list of lists) containing the numerical values.\n",
        "Here, the outermost list corresponds to axis 0, and the inner list to axis 1.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "origin_pos": 33,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zayuqbyJljvW",
        "outputId": "60c999db-e860-4bf8-f77f-ed01683e3d81"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[2., 1., 4., 3.],\n",
              "       [1., 2., 3., 4.],\n",
              "       [4., 3., 2., 1.]])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "np.array([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 36,
        "id": "sw03KeQ0ljvW"
      },
      "source": [
        "## Operations\n",
        "\n",
        "This class note is not about software engineering.\n",
        "Our interests are not limited to simply\n",
        "reading and writing data from/to arrays.\n",
        "We want to perform mathematical operations on those arrays.\n",
        "Some of the simplest and most useful operations\n",
        "are the *elementwise* operations.\n",
        "These apply a standard scalar operation\n",
        "to each element of an array.\n",
        "For functions that take two arrays as inputs,\n",
        "elementwise operations apply some standard binary operator\n",
        "on each pair of corresponding elements from the two arrays.\n",
        "We can create an elementwise function from any function\n",
        "that maps from a scalar to a scalar.\n",
        "\n",
        "In mathematical notation, we would denote such\n",
        "a *unary* scalar operator (taking one input)\n",
        "by the signature $f: \\mathbb{R} \\rightarrow \\mathbb{R}$.\n",
        "This just means that the function is mapping\n",
        "from any real number ($\\mathbb{R}$) onto another.\n",
        "Likewise, we denote a *binary* scalar operator\n",
        "(taking two real inputs, and yielding one output)\n",
        "by the signature $f: \\mathbb{R}, \\mathbb{R} \\rightarrow \\mathbb{R}$.\n",
        "Given any two vectors $\\mathbf{u}$ and $\\mathbf{v}$ *of the same shape*,\n",
        "and a binary operator $f$, we can produce a vector\n",
        "$\\mathbf{c} = F(\\mathbf{u},\\mathbf{v})$\n",
        "by setting $c_i \\gets f(u_i, v_i)$ for all $i$,\n",
        "where $c_i, u_i$, and $v_i$ are the $i^\\mathrm{th}$ elements\n",
        "of vectors $\\mathbf{c}, \\mathbf{u}$, and $\\mathbf{v}$.\n",
        "Here, we produced the vector-valued\n",
        "$F: \\mathbb{R}^d, \\mathbb{R}^d \\rightarrow \\mathbb{R}^d$\n",
        "by *lifting* the scalar function to an elementwise vector operation.\n",
        "\n",
        "The common standard arithmetic operators\n",
        "(`+`, `-`, `*`, `/`, and `**`)\n",
        "have all been *lifted* to elementwise operations\n",
        "for any identically-shaped tensors of arbitrary shape.\n",
        "We can call elementwise operations on any two tensors of the same shape.\n",
        "In the following example, we use commas to formulate a 5-element tuple,\n",
        "where each element is the result of an elementwise operation.\n",
        "\n",
        "### Operations\n",
        "\n",
        "[**The common standard arithmetic operators\n",
        "(`+`, `-`, `*`, `/`, and ****)\n",
        "have all been *lifted* to elementwise operations.]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "origin_pos": 37,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8DGfdE0BljvW",
        "outputId": "201337ed-44a8-403f-ffec-cac371621af2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([ 3.,  4.,  6., 10.]),\n",
              " array([-1.,  0.,  2.,  6.]),\n",
              " array([ 2.,  4.,  8., 16.]),\n",
              " array([0.5, 1. , 2. , 4. ]),\n",
              " array([ 1.,  4., 16., 64.]))"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "x = np.array([1, 2, 4, 8])\n",
        "y = np.array([2, 2, 2, 2])\n",
        "x + y, x - y, x * y, x / y, x**y  # The ** operator is exponentiation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 40,
        "id": "6tQUyJgOljvX"
      },
      "source": [
        "Many (**more operations can be applied elementwise**),\n",
        "including unary operators like exponentiation.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "origin_pos": 41,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DvCDOusSljvX",
        "outputId": "7c2ebbec-7d92-4f59-d7dc-9ed69d968260"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2.7182817e+00, 7.3890562e+00, 5.4598148e+01, 2.9809580e+03])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "np.exp(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 44,
        "id": "1ZnXoPbvljvY"
      },
      "source": [
        "In addition to elementwise computations,\n",
        "we can also perform linear algebra operations,\n",
        "including vector dot products and matrix multiplication.\n",
        "We will explain the crucial bits of linear algebra\n",
        "(with no assumed prior knowledge).\n",
        "\n",
        "We can also [***concatenate* multiple tensors together,**]\n",
        "stacking them end-to-end to form a larger tensor.\n",
        "We just need to provide a list of tensors\n",
        "and tell the system along which axis to concatenate.\n",
        "The example below shows what happens when we concatenate\n",
        "two matrices along rows (axis 0, the first element of the shape)\n",
        "vs. columns (axis 1, the second element of the shape).\n",
        "We can see that the first output tensor's axis-0 length ($6$)\n",
        "is the sum of the two input tensors' axis-0 lengths ($3 + 3$);\n",
        "while the second output tensor's axis-1 length ($8$)\n",
        "is the sum of the two input tensors' axis-1 lengths ($4 + 4$).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "origin_pos": 45,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Was_sF3yljvY",
        "outputId": "215ff48f-7f25-4c76-b2c7-6ff193225891"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[ 0.,  1.,  2.,  3.],\n",
              "        [ 4.,  5.,  6.,  7.],\n",
              "        [ 8.,  9., 10., 11.],\n",
              "        [ 2.,  1.,  4.,  3.],\n",
              "        [ 1.,  2.,  3.,  4.],\n",
              "        [ 4.,  3.,  2.,  1.]]),\n",
              " array([[ 0.,  1.,  2.,  3.,  2.,  1.,  4.,  3.],\n",
              "        [ 4.,  5.,  6.,  7.,  1.,  2.,  3.,  4.],\n",
              "        [ 8.,  9., 10., 11.,  4.,  3.,  2.,  1.]]))"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "X = np.arange(12).reshape(3, 4)\n",
        "Y = np.array([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])\n",
        "np.concatenate([X, Y], axis=0), np.concatenate([X, Y], axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 48,
        "id": "blZo70PNljvY"
      },
      "source": [
        "Sometimes, we want to [**construct a binary tensor via *logical statements*.**]\n",
        "Take `X == Y` as an example.\n",
        "For each position, if `X` and `Y` are equal at that position,\n",
        "the corresponding entry in the new tensor takes a value of 1,\n",
        "meaning that the logical statement `X == Y` is true at that position;\n",
        "otherwise that position takes 0.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "origin_pos": 49,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xX1baSbiljvY",
        "outputId": "a5f9ebca-e235-443b-886d-ddfcd5e4e32f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[False,  True, False,  True],\n",
              "       [False, False, False, False],\n",
              "       [False, False, False, False]])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "X == Y"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 50,
        "id": "2T-2Jut5ljvY"
      },
      "source": [
        "[**Summing all the elements in the tensor**] yields a tensor with only one element.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "origin_pos": 51,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AAQ0IsQ3ljvZ",
        "outputId": "f3555618-6613-41c2-c9ca-2cf9d38f6bd5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(66.)"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "X.sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 53,
        "id": "vv2D21HbljvZ"
      },
      "source": [
        "## Broadcasting Mechanism\n",
        "\n",
        "In the above section, we saw how to perform elementwise operations\n",
        "on two tensors of the same shape. Under certain conditions,\n",
        "even when shapes differ, we can still [**perform elementwise operations\n",
        "by invoking the *broadcasting mechanism*.**]\n",
        "This mechanism works in the following way:\n",
        "First, expand one or both arrays\n",
        "by copying elements appropriately\n",
        "so that after this transformation,\n",
        "the two tensors have the same shape.\n",
        "Second, carry out the elementwise operations\n",
        "on the resulting arrays.\n",
        "\n",
        "In most cases, we broadcast along an axis where an array\n",
        "initially only has length 1, such as in the following example:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "origin_pos": 54,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Oz2bMT2iljvZ",
        "outputId": "e32477f8-342e-4398-9c7e-044d21eceb3f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[0.],\n",
              "        [1.],\n",
              "        [2.]]), array([[0., 1.]]))"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "a = np.arange(3).reshape(3, 1)\n",
        "b = np.arange(2).reshape(1, 2)\n",
        "a, b"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 57,
        "id": "5rMYbaDCljvZ"
      },
      "source": [
        "Since `a` and `b` are $3\\times1$ and $1\\times2$ matrices respectively,\n",
        "their shapes do not match up if we want to add them.\n",
        "We *broadcast* the entries of both matrices into a larger $3\\times2$ matrix as follows:\n",
        "for matrix `a` it replicates the columns\n",
        "and for matrix `b` it replicates the rows\n",
        "before adding up both elementwise.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "origin_pos": 58,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LRP6GnQJljva",
        "outputId": "3c98bec3-f5e7-48bd-817b-70e6b59cb8ab"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 1.],\n",
              "       [1., 2.],\n",
              "       [2., 3.]])"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "a + b"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 59,
        "id": "wmjcmOBWljva"
      },
      "source": [
        "## Indexing and Slicing\n",
        "\n",
        "Just as in any other Python array, elements in a tensor can be accessed by index.\n",
        "As in any Python array, the first element has index 0\n",
        "and ranges are specified to include the first but *before* the last element.\n",
        "As in standard Python lists, we can access elements\n",
        "according to their relative position to the end of the list\n",
        "by using negative indices.\n",
        "\n",
        "Thus, [**`[-1]` selects the last element and `[1:3]`\n",
        "selects the second and the third elements**] as follows:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "origin_pos": 60,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iIRxc_B3ljva",
        "outputId": "e4647e4f-007b-44fa-f67a-f6420a4df7ac"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([20., 21., 22., 23.]), array([[ 9., 12., 15., 18.],\n",
              "        [20., 21., 22., 23.]]))"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "X[-1], X[1:3]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 61,
        "tab": [
          "mxnet"
        ],
        "id": "lEALD61hljva"
      },
      "source": [
        "Beyond reading, (**we can also write elements of a matrix by specifying indices.**)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "origin_pos": 63,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lV1p88Uuljvb",
        "outputId": "b82a9153-8cb4-4400-e12e-e0c64d5543c9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 2.,  3.,  8.,  9.],\n",
              "       [ 9., 12.,  9., 18.],\n",
              "       [20., 21., 22., 23.]])"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "X[1, 2] = 9\n",
        "X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 65,
        "id": "HxLichHiljvb"
      },
      "source": [
        "If we want [**to assign multiple elements the same value,\n",
        "we simply index all of them and then assign them the value.**]\n",
        "For instance, `[0:2, :]` accesses the first and second rows,\n",
        "where `:` takes all the elements along axis 1 (column).\n",
        "While we discussed indexing for matrices,\n",
        "this obviously also works for vectors\n",
        "and for tensors of more than 2 dimensions.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "origin_pos": 66,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HPlfEgMZljvb",
        "outputId": "6e4b811c-778d-48f9-e4de-3475e88aa5a1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[12., 12., 12., 12.],\n",
              "       [12., 12., 12., 12.],\n",
              "       [20., 21., 22., 23.]])"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "X[0:2, :] = 12\n",
        "X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 68,
        "id": "uOr5PH8Tljvb"
      },
      "source": [
        "## Saving Memory\n",
        "\n",
        "[**Running operations can cause new memory to be\n",
        "allocated to host results.**]\n",
        "For example, if we write `Y = X + Y`,\n",
        "we will dereference the tensor that `Y` used to point to\n",
        "and instead point `Y` at the newly allocated memory.\n",
        "In the following example, we demonstrate this with Python's `id()` function,\n",
        "which gives us the exact address of the referenced object in memory.\n",
        "After running `Y = Y + X`, we will find that `id(Y)` points to a different location.\n",
        "That is because Python first evaluates `Y + X`,\n",
        "allocating new memory for the result and then makes `Y`\n",
        "point to this new location in memory.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "origin_pos": 69,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CLjwISc_ljvb",
        "outputId": "5df79ced-e5af-45b4-824b-f928c4b17370"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "before = id(Y)\n",
        "Y = Y + X\n",
        "id(Y) == before"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 70,
        "id": "69bocO0Yljvc"
      },
      "source": [
        "This might be undesirable for two reasons.\n",
        "First, we do not want to run around\n",
        "allocating memory unnecessarily all the time.\n",
        "In machine learning, we might have\n",
        "hundreds of megabytes of parameters\n",
        "and update all of them multiple times per second.\n",
        "Typically, we will want to perform these updates *in place*.\n",
        "Second, we might point at the same parameters from multiple variables.\n",
        "If we do not update in place, other references will still point to\n",
        "the old memory location, making it possible for parts of our code\n",
        "to inadvertently reference state parameters.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 71,
        "tab": [
          "mxnet"
        ],
        "id": "PeUHJNF0ljvc"
      },
      "source": [
        "Fortunately, (**performing in-place operations**) is easy.\n",
        "We can assign the result of an operation\n",
        "to a previously allocated array with slice notation,\n",
        "e.g., `Y[:] = <expression>`.\n",
        "To illustrate this concept, we first create a new matrix `Z`\n",
        "with the same shape as another `Y`,\n",
        "using `zeros_like` to allocate a block of $0$ entries.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "origin_pos": 73,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alzWkuHaljvc",
        "outputId": "21bbf9ad-2d6d-42f0-e8f1-302622a1c229"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "id(Z): 140632629410304\n",
            "id(Z): 140632629410304\n"
          ]
        }
      ],
      "source": [
        "Z = np.zeros_like(Y)\n",
        "print('id(Z):', id(Z))\n",
        "Z[:] = X + Y\n",
        "print('id(Z):', id(Z))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 76,
        "tab": [
          "mxnet"
        ],
        "id": "gvGiMne1ljvc"
      },
      "source": [
        "[**If the value of `X` is not reused in subsequent computations,\n",
        "we can also use `X[:] = X + Y` or `X += Y`\n",
        "to reduce the memory overhead of the operation.**]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "origin_pos": 78,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KaUJvlbOljvc",
        "outputId": "48b23a7d-ac95-4b21-f276-53b930fcbd54"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ],
      "source": [
        "before = id(X)\n",
        "X += Y\n",
        "id(X) == before"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 80,
        "id": "UYfcupERljvc"
      },
      "source": [
        "## Conversion to Other Python Objects\n",
        "\n",
        "[**Converting to a NumPy tensor**], or vice versa, is easy.\n",
        "The converted result does not share memory.\n",
        "This minor inconvenience is actually quite important:\n",
        "when you perform operations on the CPU or on GPUs,\n",
        "you do not want to halt computation, waiting to see\n",
        "whether the NumPy package of Python might want to be doing something else\n",
        "with the same chunk of memory.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "origin_pos": 81,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oJun4wN9ljvd",
        "outputId": "5928bdd7-cc4e-4fdf-8afb-ada63911997f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(numpy.ndarray, mxnet.numpy.ndarray)"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "A = X.asnumpy()\n",
        "B = np.array(A)\n",
        "type(A), type(B)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 84,
        "id": "DCv4Hng8ljvd"
      },
      "source": [
        "To (**convert a size-1 tensor to a Python scalar**),\n",
        "we can invoke the `item` function or Python's built-in functions.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "origin_pos": 85,
        "tab": [
          "mxnet"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w3A6-Gu4ljvd",
        "outputId": "8ce08a7a-85a4-4261-d885-27ca879746bc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([3.5]), 3.5, 3.5, 3)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "a = np.array([3.5])\n",
        "a, a.item(), float(a), int(a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "origin_pos": 88,
        "id": "8ZxSHHxQljvd"
      },
      "source": [
        "## Summary\n",
        "\n",
        "* The main interface to store and manipulate data for deep learning is the tensor ($n$-dimensional array). It provides a variety of functionalities including basic mathematics operations, broadcasting, indexing, slicing, memory saving, and conversion to other Python objects.\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python"
    },
    "colab": {
      "name": "Data Manipulation_Deep Learning.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}