01_intro/seminar_pytorch_vs_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H_dXubpsEN0p"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/karfly/learning-deep-learning/blob/master/01_intro/seminar_pytorch_vs_numpy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kN12HfjKEN0r"
   },
   "source": [
    "<img src=\"https://github.com/karfly/learning-deep-learning/raw/master/01_intro/static/pytorch-vs-numpy-logo.png\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o5UspNoIEN0t"
   },
   "source": [
    "References:\n",
    "- [PyTorch tutorial by Jeremy Howard](https://pytorch.org/tutorials/beginner/nn_tutorial.html) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zDUvmqlzEN0v"
   },
   "source": [
    "# PyTorch vs. NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tQv2qhSmEN0y"
   },
   "source": [
    "`NumPy` is the fundamental package for **scientific computing** in Python. It provides a **multidimensional array** object (and other) for fast operations on arrays, including mathematical, logical, shape manipulation, etc. We hope you're familiar with NumPy. If you feel some gaps in knowledge check out [this tutorial](https://docs.scipy.org/doc/numpy-1.15.0/user/quickstart.html).\n",
    "\n",
    "\n",
    "Today we'll explore **PyTorch** library. PyTorch is also a scientific computing package (like NumPy), but it has two awesome features:\n",
    "- Automatic differentiation (very useful for deep learning)\n",
    "- GPU support for calculations\n",
    "\n",
    "\n",
    "Actually PyTorch has great features. Later on we'll use them to build neural networks, but for now let's consider it just like a scientific computing package. Let's start with installation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fC2iYMtQEN00"
   },
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tfm_LyKyEN01"
   },
   "source": [
    "### PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eZOzj3ItEN02"
   },
   "source": [
    "PyTorch is pretty easy to install:\n",
    "1. Go to [pytorch.org](https://pytorch.org) and scroll down to **\"Quick start locally\"** section.\n",
    "2. Choose options suitable for you. E.g.:\n",
    "    - PyTorch Build: **Stable (1.4)**\n",
    "    - Your OS: **Mac**\n",
    "    - Package: **Pip**\n",
    "    - Language: **Python 3.7** (check *your* python version via terminal `$ python --version`)\n",
    "    - CUDA: **None** (if you don't have GPU)\n",
    "3. Just run given command from termial. E.g.: `$ pip3 install torch torchvision`.\n",
    "4. PyTorch is quite big (especially CUDA versions), so you can have a cup of coffee while it's downloading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M4mxVrpDEN03"
   },
   "source": [
    "**Lifehack**: you can run termial commands directly from notebooks providing `!` in the beginning of the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gzhuMb93EN05"
   },
   "outputs": [],
   "source": [
    "# !pip install -U torch torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cKJqWEQ7EN09"
   },
   "source": [
    "Verify proper installation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cmvSp9N2EN0-"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "\n",
    "print(\"torch version:\", torch.__version__)\n",
    "print(\"torchvision version:\", torchvision.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kB4XdaQmEN1D"
   },
   "source": [
    "### NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Uh_GiVREN1E"
   },
   "source": [
    "NumPy installation is much easier:\n",
    "1. Run `$ pip install numpy`\n",
    "2. That's it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XFFfMVWUEN1F"
   },
   "outputs": [],
   "source": [
    "# !pip install numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3gQbc--TEN1J"
   },
   "source": [
    "Verify proper installation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C_6ZU3iXEN1J"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(\"numpy version:\", np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kzWtJHl-EN1P"
   },
   "source": [
    "## Learning PyTorch like robots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F4t4mbvZEN1R"
   },
   "source": [
    "We all love **machine learning**, so let's try to study PyTorch in a *supervised learning* manner. Here're some pairs of examples for NumPy and corresponding PyTorch:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LYOMuUo1EN1S"
   },
   "source": [
    "### Creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_EuABh-dEN1S"
   },
   "source": [
    "NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uYcsH_B0EN1U",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "x_numpy = np.array([\n",
    "    [1.0, 2.0, 3.0],\n",
    "    [4.0, 5.0, 6.0],\n",
    "    [7.0, 8.0, 9.0]\n",
    "])\n",
    "\n",
    "print(\"x_numpy = \\n{}\\n\".format(x_numpy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "keDdQ9uuEN1a"
   },
   "source": [
    "PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VwT8bOqhEN1b"
   },
   "outputs": [],
   "source": [
    "x_torch = torch.tensor([\n",
    "    [1.0, 2.0, 3.0],\n",
    "    [4.0, 5.0, 6.0],\n",
    "    [7.0, 8.0, 9.0]\n",
    "])\n",
    "\n",
    "print(\"x_torch = \\n{}\\n\".format(x_torch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0uweboz3EN1e"
   },
   "source": [
    "### Indexing:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8mbQQ3Y8EN1f"
   },
   "source": [
    "NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_us7LQadEN1g"
   },
   "outputs": [],
   "source": [
    "print(\"x_numpy[0, 0] = \\n{}\\n\".format(x_numpy[0, 0]))\n",
    "print(\"x_numpy[:, 0] = \\n{}\\n\".format(x_numpy[:, 0]))\n",
    "print(\"x_numpy[:2, 1:3] = \\n{}\\n\".format(x_numpy[:2, 1:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EGD_eVTlEN1i"
   },
   "source": [
    "PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Nrf5UxdDEN1j"
   },
   "outputs": [],
   "source": [
    "print(\"x_torch[0, 0] = \\n{}\\n\".format(x_torch[0, 0]))\n",
    "print(\"x_torch[:, 0] = \\n{}\\n\".format(x_torch[:, 0]))\n",
    "print(\"x_torch[:2, 1:3] = \\n{}\\n\".format(x_torch[:2, 1:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Wwoj2bfEN1n"
   },
   "source": [
    "### Operations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VZAoYz9QEN1n"
   },
   "source": [
    "NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n7Tqkx0yEN1o"
   },
   "outputs": [],
   "source": [
    "print(\"x_numpy ** 2 = \\n{}\\n\".format(x_numpy ** 2))\n",
    "print(\"np.cos(x_numpy) = \\n{}\\n\".format(np.cos(x_numpy)))\n",
    "print(\"x_numpy.mean(axis=0) = \\n{}\\n\".format(x_numpy.mean(axis=0)))\n",
    "print(\"x_numpy.T = \\n{}\\n\".format(x_numpy.T))\n",
    "print(\"x_numpy.reshape(1, -1) = \\n{}\\n\".format(x_numpy.reshape(1, -1)))\n",
    "print(\"x_numpy.flatten() = \\n{}\\n\".format(x_numpy.flatten()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L3GiVyzqEN1r"
   },
   "source": [
    "PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CCZVGSkGEN1s"
   },
   "outputs": [],
   "source": [
    "print(\"x_torch ** 2 = \\n{}\\n\".format(x_torch ** 2))\n",
    "print(\"np.cos(x_torch) = \\n{}\\n\".format(torch.cos(x_torch)))\n",
    "print(\"x_torch.mean(dim=0) = \\n{}\\n\".format(x_torch.mean(dim=0)))\n",
    "print(\"x_torch.t() = \\n{}\\n\".format(x_torch.t()))\n",
    "print(\"x_torch.reshape(1, -1) = \\n{}\\n\".format(x_torch.reshape(1, -1)))\n",
    "print(\"x_torch.flatten() = \\n{}\\n\".format(x_torch.flatten()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uQQP4MHXEN1v"
   },
   "source": [
    "### Construction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DPJIJe70EN1v"
   },
   "source": [
    "NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pgAqkIwAEN1w"
   },
   "outputs": [],
   "source": [
    "print(\"np.arange(3) = \\n{}\\n\".format(np.arange(3)))\n",
    "print(\"np.linspace(0.0, 1.0, num=9).reshape(3, 3) = \\n{}\\n\".format(np.linspace(0.0, 1.0, num=9).reshape(3, 3)))\n",
    "print(\"np.ones() = \\n{}\\n\".format(np.ones((2, 5))))\n",
    "print(\"np.random.rand(3, 2) = \\n{}\\n\".format(np.random.rand(3, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PZIqJqoKEN1y"
   },
   "source": [
    "PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OBTpQ2wtEN1z"
   },
   "outputs": [],
   "source": [
    "print(\"torch.arange(3) = \\n{}\\n\".format(torch.arange(3)))\n",
    "print(\"torch.linspace(0.0, 1.0, steps=9).reshape(3, 3) = \\n{}\\n\".format(torch.linspace(0.0, 1.0, steps=9).reshape(3, 3)))\n",
    "print(\"torch.ones() = \\n{}\\n\".format(torch.ones((2, 5))))\n",
    "print(\"torch.random.rand(3, 2) = \\n{}\\n\".format(torch.rand(3, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fY4gNOx3EN13"
   },
   "source": [
    "### 2 arrays/tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GK6EUMZtEN13"
   },
   "source": [
    "NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cYMnjrpIEN14"
   },
   "outputs": [],
   "source": [
    "x_numpy = np.arange(0, 9).reshape(3, 3)\n",
    "y_numpy = np.linspace(0.0, 1.0, num=9).reshape(3, 3)\n",
    "\n",
    "print(\"x = \\n{}\\n\".format(x_numpy))\n",
    "print(\"y = \\n{}\\n\".format(y_numpy))\n",
    "print(\"x * y = \\n{}\\n\".format(x_numpy * y_numpy))\n",
    "print(\"x.dot(y) = \\n{}\\n\".format(x_numpy.dot(y_numpy)))\n",
    "print(\"np.concatenate([x, y], axis=1) = \\n{}\\n\".format(np.concatenate([x_numpy, y_numpy], axis=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "amWUZPdwEN16"
   },
   "source": [
    "PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YRsm1knzEN17"
   },
   "outputs": [],
   "source": [
    "x_torch = torch.arange(0, 9).reshape(3, 3).type(torch.float)\n",
    "y_torch = torch.linspace(0.0, 1.0, steps=9).reshape(3, 3)\n",
    "\n",
    "print(\"x = \\n{}\\n\".format(x_torch))\n",
    "print(\"y = \\n{}\\n\".format(y_torch))\n",
    "print(\"x * y = \\n{}\\n\".format(x_torch * y_torch))\n",
    "print(\"x.mm(y) = \\n{}\\n\".format(x_torch.mm(y_torch)))\n",
    "print(\"torch.cat([x, y], axis=1) = \\n{}\\n\".format(torch.cat([x_torch, y_torch], dim=1)))\n",
    "# print(\" = \\n{}\\n\".format())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Um7HM4rSEN19"
   },
   "source": [
    "## What are the differences?\n",
    "<img src=\"https://github.com/karfly/learning-deep-learning/raw/master/01_intro/static/spiderman-meme.jpg\" width=500px align=\"center\"/>\n",
    "\n",
    "On the first sight PyTorch is about replacing `np.` with `torch.`. In most cases it's really true, but there are some more *key* differences:\n",
    "\n",
    "- by default NumPy creates float arrays with float64 dtype; PyTorch's default dtype is float32\n",
    "- `axis` in NumPy [`x.mean(axis=0)`] vs. `dim` in PyTorch [`x.mean(dim=0)`]\n",
    "- some functions' naming [e.g. `np.concatenate` vs. `torch.cat`]\n",
    "\n",
    "All these differences can be easily googled or found in awesome [PyTorch docs](https://pytorch.org/docs/stable/index.html). Checkout [this repository](https://github.com/wkentaro/pytorch-for-numpy-users) where all differences are listed in a table. If you still have questions, you can ask them on [PyTorch forum](https://discuss.pytorch.org) (it's alive and questions are frequently answered).\n",
    "\n",
    "So now you can ask: **why the hell should I use PyTorch instead of NumPy?** Actually PyTorch has some wonderful features, but we'll discuss them a bit later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tREZcXqsEN1-"
   },
   "source": [
    "## A bit more examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ccoS9gOeEN1_"
   },
   "source": [
    "NumPy -> PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6g0bQXy0EN1_"
   },
   "outputs": [],
   "source": [
    "x_numpy = np.arange(9).reshape(3, 3)\n",
    "x_torch = torch.from_numpy(x_numpy)\n",
    "\n",
    "print(\"x_torch = \\n{}\\n\".format(x_torch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gWoj9g95EN2C"
   },
   "source": [
    "PyTorch -> NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yRcWb2zFEN2C"
   },
   "outputs": [],
   "source": [
    "x_torch = torch.arange(9).reshape(3, 3)\n",
    "x_numpy = x_torch.numpy()\n",
    "\n",
    "print(\"x_numpy = \\n{}\\n\".format(x_numpy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HNzWxR3gEN2E"
   },
   "source": [
    "Inplace operations. In PyTorch you can peform inplace operations (no copying). Many tensor methods have their inplace twins with `_` symbol in the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yQRzJP3SEN2E"
   },
   "outputs": [],
   "source": [
    "x = torch.arange(5).type(torch.float)\n",
    "print(\"[before] x = {}\".format(x))\n",
    "\n",
    "x.sqrt_()\n",
    "print(\"[after]  x = {}\".format(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uh85O9IWEN2I"
   },
   "outputs": [],
   "source": [
    "x = torch.arange(5).type(torch.float)\n",
    "print(\"[before] x = {}\".format(x))\n",
    "\n",
    "x.zero_()\n",
    "print(\"[after]  x = {}\".format(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P1an_NwGEN2K"
   },
   "source": [
    "## **Task 1 (1 point).** Drawing with PyTorch\n",
    "\n",
    "Implement this function and plot it using matplotlib:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "x = 16 \\sin^3(t) \\\\\n",
    "y = 13\\cos(t) - 5\\cos(2t) - 2\\cos(3t) - \\cos(4t) \\\\\n",
    "\\end{cases}\n",
    ",~ t \\in [0, 2\\pi]\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qtpm-NHhEN2L"
   },
   "outputs": [],
   "source": [
    "from matplotlib import pylab as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TD2DA1p2EN2M"
   },
   "outputs": [],
   "source": [
    "t = torch.linspace(0.0, 2 * np.pi, 100)\n",
    "\n",
    "x = # your code here\n",
    "y = # your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZoYtHv1UEN2O"
   },
   "outputs": [],
   "source": [
    "plt.plot(x.numpy(), y.numpy(), c='red');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "djnViH0QEN2R"
   },
   "source": [
    "## Automatic differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Heo06YY7EN2R"
   },
   "source": [
    "The most important feature of PyTorch is that it can **differentiate (almost) any expression written** in PyTorch.\n",
    "\n",
    "For example you have function $f(x) = x^2$. You want to calculate partial detivative $\\frac{\\partial f}{\\partial x}$. PyTorch allows you to do it in 3 lines! Look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f74iMZmwEN2R"
   },
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)  # tells PyTorch that we'll need gradient of this tensor\n",
    "f_x = x ** 2  # run our function\n",
    "f_x.backward()  # calculate gradient\n",
    "\n",
    "print(\"df/dx = {}\".format(x.grad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tGiS_KvzEN2T"
   },
   "source": [
    "The salt is that $f(x)$ can be any (almost) any function you want (e.g. your neural network). It totally looks like magic. More on PyTorch automatic differentiation read [here](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EYgCpK_BEN2T"
   },
   "source": [
    "## Simple linear regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xKiz71jOEN2T"
   },
   "source": [
    "Load data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PEdYJTCrEN2U"
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "\n",
    "plt.scatter(boston.data[:, -1], boston.target)\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OVrvakLdEN2W"
   },
   "source": [
    "Convert data to torch tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BR4PhsKxEN2W"
   },
   "outputs": [],
   "source": [
    "x = torch.from_numpy(boston.data[:, -1]).type(torch.float)\n",
    "x = (x - x.mean()) / x.std()  # normalization\n",
    "\n",
    "y = torch.from_numpy(boston.target).type(torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xFjYW3l7EN2Y"
   },
   "source": [
    "1-dimensional linear regression is formulated like this:\n",
    "$$\\normalsize y^{pred} = wx + b$$\n",
    "where $x$ - is a feature, $w, b$ - model parameters (weight and bias), and $y^{pred}$ - model's prediction.\n",
    "\n",
    "As a loss function we'll use **Mean Square Error** (MSE):\n",
    "$$MSE(y, y^{pred}) = \\frac{1}{N}\\sum_{i=0}^{N-1}(y_{i} - y_{i}^{pred}) ^ 2$$\n",
    "where $N$ - is a length of training set.\n",
    "\n",
    "To train our model, we'll use **Gradient Descent** (GD):\n",
    "$$\\normalsize w_{i} = w_{i - 1} - \\eta \\frac{\\partial loss}{\\partial w_{i - 1}}$$\n",
    "\n",
    "$$\\normalsize b_{i} = b_{i - 1} - \\eta \\frac{\\partial loss}{\\partial b_{i - 1}}$$\n",
    "where $\\eta$ - is a learning rate.\n",
    "\n",
    "But we're not going to calculate this partial derivative by hand. PyTorch will do it for us!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UCdA7jAjEN2Y"
   },
   "source": [
    "Declare model's parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CX8MKX2FEN2Y"
   },
   "outputs": [],
   "source": [
    "w = torch.tensor(1.0, requires_grad=True)\n",
    "b = torch.tensor(0.0, requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nbLG5n-2EN2a"
   },
   "source": [
    "Train loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lUe0S6PnEN2a"
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "lr = 0.01\n",
    "n_iters = 200\n",
    "\n",
    "losses = []\n",
    "\n",
    "for i in range(n_iters):\n",
    "    # forward pass\n",
    "    y_pred = w * x + b\n",
    "    \n",
    "    # calculate loss\n",
    "    loss = torch.mean((y - y_pred) ** 2)\n",
    "    losses.append(loss.item())\n",
    "    \n",
    "    # calculate gradients\n",
    "    loss.backward()\n",
    "    \n",
    "    # make gradient step\n",
    "    w.data = w.data - lr * w.grad.detach()  # detaching tensor from computational graph\n",
    "    b.data = b.data - lr * b.grad.detach()  #\n",
    "    \n",
    "    # zero gradients (otherwise they'll accumulate)\n",
    "    w.grad.zero_()\n",
    "    b.grad.zero_()\n",
    "\n",
    "    # visualization\n",
    "    if i % 10 == 0:\n",
    "        clear_output(True)\n",
    "        \n",
    "        print(\"loss: {:.04}\".format(loss.item()))\n",
    "        \n",
    "        # training set\n",
    "        plt.scatter(x.numpy(), y.numpy())\n",
    "        \n",
    "        # our predictions\n",
    "        plt.scatter(x.numpy(), y_pred.detach().numpy(), color='red')\n",
    "        \n",
    "        plt.xlabel(\"x\")\n",
    "        plt.ylabel(\"y\")\n",
    "        plt.show()\n",
    "    \n",
    "        \n",
    "plt.plot(losses)\n",
    "plt.xlabel(\"iters\")\n",
    "plt.ylabel(\"loss\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FJvPSu0_EN2c"
   },
   "source": [
    "## Task 2 (1 point). Linear regression with polynomial features\n",
    "\n",
    "As you can see above simple linear regression doesn't fit data well. Let's add polynomial features:\n",
    "$$\\normalsize y^{pred} = \\sum_{i=0}^{D-1}w_{i}x^{i}$$\n",
    "\n",
    "To get total score for this task:\n",
    " - choose any $D$ you want and make loss **< 30.0**.\n",
    " - answer the question: \"why don't we have bias (b) here?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GnzZep0AEN2c"
   },
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S9YGy2pbEN2g"
   },
   "source": [
    "## Accelerating PyTorch with GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3SdJSwEzEN2g"
   },
   "source": [
    "A **Graphics Processing Unit (GPU)** is a specialized electronic circuit designed to rapidly perform single-type matematical calculations. And for playing computer games...\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/karfly/learning-deep-learning/raw/master/01_intro/static/gpu.jpg\" align=\"center\"/>\n",
    "\n",
    "GPUs are built for doing same operation on multiple data efficiently. Machine learning algorithms (e.g. neural networks) usually consist of doing same operation (e.g. matrix multiplication) again and again, that's why GPUs are successfully applied for boosting algorithms' performance.\n",
    "\n",
    "If you're lucky enough to have access to a CUDA-capable GPU (you can rent one for about about $0.50/hour from most cloud providers or utilize free Google Colab) you can use it to speed up your code. First check that your GPU is working in Pytorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XvyK8BXVEN2h"
   },
   "outputs": [],
   "source": [
    "print(torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Qc_OnXaLEN2j"
   },
   "source": [
    "And then create a device object for it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2Hak5WpBEN2j"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qfU9dB8ZEN2l"
   },
   "source": [
    "Now we can create tensor and transfer it to GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XpRI7FHTEN2m"
   },
   "outputs": [],
   "source": [
    "x = torch.arange(0, 9).reshape(3, 3).type(torch.float)\n",
    "x = x.to(device)\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iEKnKb9bEN2o"
   },
   "source": [
    "If you have a GPU than you'll see that now `x` is a `cuda`-tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oplW5EhtEN2o"
   },
   "source": [
    "Now let's multiply some large matricies on CPU and GPU to compare the speed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EXAll2b8EN2p"
   },
   "outputs": [],
   "source": [
    "size = 1000\n",
    "\n",
    "x = torch.arange(0, size ** 2).reshape(size, size).type(torch.float)\n",
    "x_gpu = x.clone().to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OeSolHCkEN2q"
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "\n",
    "x.mm(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KVo8FPk5EN2s"
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "\n",
    "# your code here (multiply matricies on GPU) \n",
    "\n",
    "if device == 'cuda':\n",
    "    torch.cuda.synchronize()  # always add synchronize() when you measure GPU runtime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7zrF6g_DEN2t"
   },
   "source": [
    "As you can see, we can get dramatic boost of speed using GPU!\n",
    "\n",
    "Note, we can't compute operations with tensors located on different devices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e0dLcFqaEN2u"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    x.mm(x_gpu)\n",
    "except RuntimeError as e:\n",
    "    print(\"Tensors are located on different devices!\")\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mTef-MJUEN2w"
   },
   "source": [
    "## Task 3 (2 points). Earthquake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IaN57-sOEN2w"
   },
   "source": [
    "Big earthquake happened! Seismic sensors fixed heatmap of underground activity. You can find this map on the path `./data/earthquake-heatmap.npy` or download it from GitHub:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uFSQEyC3EN2w"
   },
   "outputs": [],
   "source": [
    "# url = \"https://github.com/karfly/learning-deep-learning/raw/master/01_intro/data/earthquake-heatmap.npy\"\n",
    "# !mkdir data\n",
    "# !wget $url -O ./data/earthquake-heatmap.npy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9e1QxmxkEN2y"
   },
   "source": [
    "Read earthquake heatmap and draw it with matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V45Y-gndEN2y"
   },
   "outputs": [],
   "source": [
    "earthquake_heatmap = # your code here (use np.load)\n",
    "\n",
    "shape = earthquake_heatmap.shape\n",
    "print(\"shape: {}\".format(shape))\n",
    "\n",
    "plt.imshow(earthquake_heatmap)\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Brp--RGUEN20"
   },
   "outputs": [],
   "source": [
    "earthquake_heatmap = np.load(\"./data/earthquake-heatmap.npy\")\n",
    "\n",
    "shape = earthquake_heatmap.shape\n",
    "print(\"shape: {}\".format(shape))\n",
    "\n",
    "plt.imshow(earthquake_heatmap)\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U-8p4hLnEN21"
   },
   "source": [
    "Your task is to find coordinates of **earthquake epicenter** and **radius of seismic activity**.\n",
    "\n",
    "Possibly, your first thoughts will be: \"Hmm, epicenter I'll find by applying argmax for both dimensions. For radius I can write for-loop and... blablabla\". Such methods are okay, but we'll apply more **SCIENTIFIC** (actually not) method here. Scheme of the proposed method is below.\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/karfly/learning-deep-learning/raw/master/01_intro/static/earthquake-method.png\" align=\"center\"/>\n",
    "\n",
    "We'll fit a 2D gaussian to our earthquake heatmap.\n",
    "\n",
    "Overall algorithm is like this:\n",
    "1. generate 2D gaussian heatmap for current $\\mu$ and $\\sigma$ \n",
    "2. calculate per-pixel MSE of generated heatmap and earthquake heatmap\n",
    "3. calculate partial derivatives $\\frac{\\partial{loss}}{\\partial{\\mu}}$, $\\frac{\\partial{loss}}{\\partial{\\sigma}}$ and make gradient descent step to minimize loss\n",
    "4. go to step (1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9VG-bXD6EN22"
   },
   "source": [
    "To generate heatmap we'll need a Probability Dense Function (PDF) of independent 2D normal distribution:\n",
    "\n",
    "$$p(x_1, x_2) = \\frac{1}{2\\pi \\sigma_1 \\sigma_2}\\exp\\left[{-\\frac{(x_1 - \\mu_1)^2}{2\\sigma_1^2} - \\frac{(x_2 - \\mu_2)^2}{2\\sigma_2^2}}\\right]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QSiWYOGJEN22"
   },
   "source": [
    "Implement missing parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VIjjfDCREN22"
   },
   "outputs": [],
   "source": [
    "def gaussian_2d_pdf(coords, mu, sigma):\n",
    "    \"\"\"Independent 2D normal distribution PDF\n",
    "    \n",
    "    Args:\n",
    "        coords (tensor of shape (N, 2)): coordinates, where to calculate function\n",
    "        mu (tensor of shape (2,)): mu values\n",
    "        sigma (tensor of shape (2,)): sigma values\n",
    "        \n",
    "    \"\"\"\n",
    "    normalization = # your code here\n",
    "    exp = # your code here\n",
    "    return exp / normalization\n",
    "\n",
    "\n",
    "def draw_heatmap(mu, sigma, shape):\n",
    "    xx, yy = torch.meshgrid(torch.arange(shape[0]), torch.arange(shape[1]))\n",
    "    grid = torch.stack([xx, yy], dim=-1).type(torch.float32)\n",
    "    grid = grid.reshape((-1, 2))\n",
    "    \n",
    "    heatmap = gaussian_2d_pdf(grid, mu, sigma)\n",
    "    heatmap = heatmap.reshape(*shape)\n",
    "    \n",
    "    heatmap = (heatmap - heatmap.mean()) / heatmap.std()\n",
    "\n",
    "    return heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "asIEhpBrEN23"
   },
   "source": [
    "Generate heatmap with some $\\mu$ and $\\sigma$; shape same as earthquake heatmap: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2FpsmVDHEN24"
   },
   "outputs": [],
   "source": [
    "mu = # your code here\n",
    "sigma = # your code here\n",
    "\n",
    "heatmap = # your code here\n",
    "\n",
    "plt.imshow(heatmap.numpy())\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JKIu0NUgEN25"
   },
   "source": [
    "Define parameters of your model (initialize them with reasonable values):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bhl9G7n8EN25"
   },
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lz-mgTfxEN27"
   },
   "source": [
    "Build optimization loop and fit `mu` and `sigma`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OpudRszhEN27"
   },
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t3Dyc5QTEN28"
   },
   "source": [
    "### Tips:\n",
    "- Plot your map and loss curve every n-th iteration\n",
    "- Tune learning rate\n",
    "- Try different initializations of $\\mu$ and $\\sigma$\n",
    "- Play with different number of iterations\n",
    "- Try Mean Absolute Error (MAE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tD7zFxOKEN28"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "seminar_pytorch_vs_numpy.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}