02-linear-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression and Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will see how we can perform linear regression in three different ways: \n",
    "1. pure numpy\n",
    "2. numpy + pytorch's autograd \n",
    "3. pure pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "torch.manual_seed(0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we want to predict a real-valued quantity $y \\in \\mathbb{R}$ for a given input $\\mathbf{x} \\in \\mathbb{R}^d$. This is known as **regression**. \n",
    "\n",
    "The most common loss function for regression is the **quadractic loss** or **$\\ell_2$ loss**:\n",
    "\n",
    "$$\n",
    "\\ell_2(y, \\hat{y}) = (y - \\hat{y})^2\n",
    "$$\n",
    "\n",
    "The empirical risk becomes the **mean squared error (MSE)**:\n",
    "\n",
    "$$\n",
    "MSE(\\theta) = \\frac{1}{N} \\sum\\limits_{n=1}^{N} (y_n - f(\\mathbf{x}_n; \\theta))^2\n",
    "$$\n",
    "\n",
    "The model $f(\\mathbf{x}_n; \\theta)$ can be parameterized in many ways. In this lecture we will focus on a linear parameterization, leading to the well-known **Linear Regression** formulation:\n",
    "\n",
    "$$\n",
    "f(\\mathbf{x}; \\theta) = \\mathbf{w}^\\top \\mathbf{x} + b = w_1 x_1 + w_2 x_2 + \\cdots + w_D x_D + b\n",
    "$$\n",
    "\n",
    "where $\\theta = (b, \\mathbf{w})$ are the parameters of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "Let's create a synthetic regression dataset using `sklearn`'s `make_regression` function. For better visualization, we will use only a single feature.$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "\n",
    "\n",
    "n_features = 1\n",
    "n_samples = 100\n",
    "\n",
    "X, y = make_regression(\n",
    "    n_samples=n_samples,\n",
    "    n_features=n_features,\n",
    "    noise=20,\n",
    "    random_state=42,\n",
    ")\n",
    "\n",
    "fix, ax = plt.subplots()\n",
    "ax.plot(X, y, \".\")\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, by looking at the plot above, let's say that $w \\approx 40$ and $b \\approx 2$. Then, we would arrive at the following predictions (with vertical bars indicating the errors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our estimate\n",
    "w = 40.0\n",
    "b = 2.0\n",
    "y_pred = w*X + b\n",
    "\n",
    "# subplots\n",
    "fig, axs = plt.subplots(1, 2, figsize=(16, 4))\n",
    "\n",
    "# left plot\n",
    "axs[0].plot(X, y, 'o')\n",
    "axs[0].plot(X, y_pred, '-')\n",
    "\n",
    "# right plot\n",
    "axs[1].vlines(X, y, y_pred, color='black')\n",
    "axs[1].plot(X, y, 'o')\n",
    "axs[1].plot(X, y_pred, '-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By adjusting our parameters $\\theta=(w, b)$, we can minimize the sum of squared errors to find the **least squares solution**\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{\\theta} &= \\arg\\min_\\theta MSE(\\theta) \\\\\n",
    "&= \\arg\\min_\\theta \\frac{1}{N} \\sum\\limits_{n=1}^{N} (y_n - f(\\mathbf{x}_n; \\theta))^2 \\\\\n",
    "&= \\arg\\min_{w,b} \\frac{1}{N} \\sum\\limits_{n=1}^{N} (y_n - (w \\cdot x_n + b))^2\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Which can be found by taking the gradient of the loss function w.r.t. $\\theta$. \n",
    "\n",
    "<!-- \n",
    "<table>\n",
    "    <tr>\n",
    "        <td style=\"vertical-align: top; text-align: left; padding-right: 20px; background:#fff\">\n",
    "            <center><b>Solving for $b$:</b></center><br>\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{b} &= \\text{solve}\\bigg[ \\frac{\\partial MSE(\\theta)}{\\partial b} \\bigg] \\\\\n",
    "&= \\text{solve}\\bigg[ \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n - b) \\cdot 1 \\bigg] \\\\\n",
    "&= \\text{solve}\\bigg[ \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n - b) \\bigg]\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n - b) = 0 \\\\\n",
    "\\sum\\limits_{n=1}^{N} b = \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n) \\\\\n",
    "N b = \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n) \\\\\n",
    "b = \\frac{1}{N}\\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n) \\\\\n",
    "b = \\bar{y} - w \\bar{x} \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "        </td>\n",
    "        <td style=\"vertical-align: top; text-align: left; background:#fff\">\n",
    "            <center><b>Solving for $w$:</b></center><br>\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{w} &= \\text{solve}\\bigg[ \\frac{\\partial MSE(\\theta)}{\\partial w} \\bigg] \\\\\n",
    "&= \\text{solve}\\bigg[ \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - w \\cdot x_n - b) \\cdot (x_n) \\bigg] \\\\\n",
    "&= \\text{solve}\\bigg[ \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n \\cdot x_n - w \\cdot x_n^2 - b \\cdot x_n) \\bigg]\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n \\cdot x_n - w \\cdot x_n^2 - b \\cdot x_n) = 0 \\\\\n",
    "\\sum\\limits_{n=1}^{N} (y_n \\cdot x_n - w \\cdot x_n^2 - (\\bar{y} - w \\bar{x}) \\cdot x_n) = 0 \\\\\n",
    "\\sum\\limits_{n=1}^{N} (w \\cdot x_n^2 - w \\cdot \\bar{x} \\cdot x_n) = \\sum\\limits_{n=1}^{N} (y_n \\cdot x_n - \\bar{y}) \\\\\n",
    "w  = \\sum\\limits_{n=1}^{N} (y_n \\cdot x_n - \\bar{y}) \\bigg/ \\sum\\limits_{n=1}^{N} (x_n^2 - \\bar{x} \\cdot x_n) \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    " -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, for inputs with higher dimensionality $d$, we have $\\mathbf{w} \\in \\mathbb{R}^d$, and thus we have the following gradient (assuming that $b$ is absorbed by $w$):\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\nabla_\\mathbf{w} MSE(\\theta) &= \\nabla_\\mathbf{w} \\frac{1}{N} \\sum\\limits_{n=1}^{N} (y_n - f(\\mathbf{x}_n; \\theta))^2 \\\\\n",
    "&= \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - f(\\mathbf{x}_n; \\theta)) \\cdot \\nabla_\\mathbf{w} f(\\mathbf{x}_n; \\theta) \\\\\n",
    "&= \\frac{-2}{N} \\sum\\limits_{n=1}^{N} (y_n - (\\mathbf{w}^\\top \\mathbf{x}_n + b)) \\cdot \\mathbf{x}_n\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Now, we just have follow the gradient descent rule to update $\\mathbf{w}$: \n",
    "\n",
    "$$\n",
    "\\mathbf{w}_{t+1} = \\mathbf{w}_{t} - \\alpha \\nabla_{\\mathbf{w}} MSE(\\theta)\n",
    "$$\n",
    "\n",
    "Where $\\alpha$ represents the learning rate. So, let's implement this in numpy to see what happens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numpy Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearRegression(object):\n",
    "    def __init__(self, n_features, n_targets=1, lr=0.1):\n",
    "        self.W = np.zeros((n_targets, n_features))\n",
    "        self.lr = lr\n",
    "\n",
    "    def update_weight(self, X, y, y_hat):\n",
    "        N = X.shape[0]\n",
    "        W_grad = - 2 * np.dot(X.T,  y - y_hat) / N\n",
    "        self.W = self.W - self.lr * W_grad\n",
    "\n",
    "    def loss(self, y_hat, y):\n",
    "        return np.mean(np.power(y - y_hat, 2))\n",
    "\n",
    "    def predict(self, X):\n",
    "        return np.dot(X, self.W.T).squeeze(-1)\n",
    "\n",
    "    def train(self, X, y, epochs=50):\n",
    "        \"\"\"\n",
    "        X (n_examples x n_features): input matrix\n",
    "        y (n_examples): gold labels\n",
    "        \"\"\"\n",
    "        loss_history = []\n",
    "        for _ in range(epochs):\n",
    "            # get prediction for computing the loss\n",
    "            y_hat = self.predict(X)\n",
    "            loss = self.loss(y_hat, y)\n",
    "\n",
    "            # update weights\n",
    "            self.update_weight(X, y, y_hat)\n",
    "            # (thought exercise): what happens if we do this instead?\n",
    "            # for x_i, y_i in zip(X, y):\n",
    "            #        self.update_weight(x_i, y_i)\n",
    "\n",
    "            # save loss value\n",
    "            loss_history.append(loss)\n",
    "        return loss_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trick for handling the bias term:\n",
    "# concat a columns of 1s to the original input matrix X\n",
    "use_bias = True\n",
    "if use_bias:\n",
    "    X_np = np.hstack([np.ones((n_samples,1)), X])\n",
    "    n_features += 1\n",
    "else:\n",
    "    X_np = X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LinearRegression(n_features=n_features, n_targets=1, lr=0.1)\n",
    "loss_history = model.train(X_np, y, epochs=50)\n",
    "y_hat = model.predict(X_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('b:', model.W[0,0])\n",
    "print('W:', model.W[0,1])\n",
    "plt.plot(loss_history)\n",
    "plt.title('Loss per epoch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vis\n",
    "fig, axs = plt.subplots(1, 2, figsize=(16, 4))\n",
    "axs[0].plot(X, y, \"o\", label=\"data\")\n",
    "axs[0].plot(X, 40*X + 2, \"-\", label=\"pred\")\n",
    "axs[0].set_title(\"Guess\")\n",
    "axs[0].legend();\n",
    "\n",
    "axs[1].plot(X, y, \"o\", label=\"data\")\n",
    "axs[1].plot(X, y_hat, \"-\", label=\"pred\")\n",
    "axs[1].set_title(\"Numpy solution\")\n",
    "axs[1].legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numpy + Autograd Solution\n",
    "\n",
    "In the previous implementation, we had to derive the gradient $\\frac{\\partial MSE(\\theta)}{\\partial \\theta}$ manually. If the model $f(\\cdot;\\theta)$ is more complex, this might be a cumbersome and error-prone task. To avoid this, we will use PyTorch `autograd` to automatically compute gradients.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixedLinearRegression(object):\n",
    "    def __init__(self, n_features, n_targets=1, lr=0.01):\n",
    "        # note requires_grad=True!\n",
    "        self.W = torch.zeros(n_targets, n_features, requires_grad=True)\n",
    "        self.lr = lr\n",
    "    \n",
    "    def update_weight(self):\n",
    "        # Gradients are given to us by autograd!\n",
    "        self.W.data = self.W.data - self.lr * self.W.grad.data\n",
    "\n",
    "    def loss(self, y_hat, y):\n",
    "        return torch.mean(torch.pow(y - y_hat, 2))\n",
    "\n",
    "    def predict(self, X):\n",
    "        return torch.matmul(X, self.W.t()).squeeze(-1)\n",
    "\n",
    "    def train(self, X, y, epochs=50):\n",
    "        \"\"\"\n",
    "        X (n_examples x n_features): input matrix\n",
    "        y (n_examples): gold labels\n",
    "        \"\"\"\n",
    "        loss_history = []\n",
    "        for _ in range(epochs):\n",
    "            # Our neural net is a Line function!\n",
    "            y_hat = self.predict(X)\n",
    "            \n",
    "            # Compute the loss using torch operations so they are saved in the gradient history.\n",
    "            loss = self.loss(y_hat, y)\n",
    "            \n",
    "            # Computes the gradient of loss with respect to all Variables with requires_grad=True.\n",
    "            # where Variables are tensors with requires_grad=True\n",
    "            loss.backward()\n",
    "            loss_history.append(loss.item())\n",
    "\n",
    "            # Update weights using gradient descent; W.data is a Tensor.\n",
    "            self.update_weight()\n",
    "\n",
    "            # Reset the accumulated gradients\n",
    "            self.W.grad.data.zero_()\n",
    "            \n",
    "        return loss_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_pt = torch.from_numpy(X_np).float()\n",
    "y_pt = torch.from_numpy(y).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MixedLinearRegression(n_features=n_features, n_targets=1, lr=0.1)\n",
    "loss_history = model.train(X_pt, y_pt, epochs=50)\n",
    "with torch.no_grad():\n",
    "    y_hat = model.predict(X_pt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('b:', model.W[0,0].item())\n",
    "print('W:', model.W[0,1].item())\n",
    "plt.plot(loss_history)\n",
    "plt.title('Loss per epoch');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vis\n",
    "fig, axs = plt.subplots(1, 3, figsize=(16, 4))\n",
    "axs[0].plot(X, y, \"o\", label=\"data\")\n",
    "axs[0].plot(X, 40*X + 2, \"-\", label=\"pred\")\n",
    "axs[0].set_title(\"Guess\")\n",
    "axs[0].legend();\n",
    "\n",
    "axs[1].plot(X, y, \"o\", label=\"data\")\n",
    "axs[1].plot(X, 47.12483907744531*X + 2.3264433961431727, \"-\", label=\"pred\")\n",
    "axs[1].set_title(\"Numpy solution\")\n",
    "axs[1].legend();\n",
    "\n",
    "axs[2].plot(X, y, \"o\", label=\"data\")\n",
    "axs[2].plot(X, y_hat, \"-\", label=\"pred\")\n",
    "axs[2].set_title(\"Mixed solution\")\n",
    "axs[2].legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Solution\n",
    "\n",
    "Mixing PyTorch and Numpy is no fun. PyTorch is actually very powerful and provides most of the things we need to apply gradient descent for any model $f$, as long all operations applied over the inputs are Torch operations (so gradients can be tracked). \n",
    "\n",
    "To this end, we will use the submodule `torch.nn`, which provides us a way for encapsulating our model into a `nn.Module`. With this, all we need to do is define the our parameters in the `__init__` method and then the _forward_ pass of our model in the `forward` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "from torch import optim\n",
    "\n",
    "# See the inheritance from nn.Module\n",
    "class TorchLinearRegression(nn.Module):\n",
    "    \n",
    "    def __init__(self, n_features, n_targets=1):\n",
    "        super().__init__()  # this is mandatory!\n",
    "        \n",
    "        # encapsulate our weights into a nn.Parameter object\n",
    "        self.W = torch.nn.Parameter(torch.zeros(n_targets, n_features))\n",
    "\n",
    "    def forward(self, X):\n",
    "        \"\"\"\n",
    "        X (n_examples x n_features): input matrix\n",
    "        \"\"\"\n",
    "        #if self.training:\n",
    "        #    X = X ** 2\n",
    "        #else:\n",
    "        #    X = X ** 3\n",
    "        #    import ipdb; ipdb. set_trace()\n",
    "        return X @ self.W.t()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define model, loss function and optmizer\n",
    "model = TorchLinearRegression(n_features)\n",
    "loss_fn = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)\n",
    "\n",
    "# move to CUDA if available\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "model = model.to(device)\n",
    "X = X_pt.to(device)\n",
    "y = y_pt.to(device).unsqueeze(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All done! Now we just have to write a training loop, which is more or less a standard set of steps for training all models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, X, y, epochs=50):\n",
    "    # inform PyTorch that we are in \"training\" mode\n",
    "    model.train()\n",
    "    \n",
    "    loss_history = []\n",
    "    for _ in range(epochs):\n",
    "        # reset gradients before learning\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # get predictions and and the final score from the loss function \n",
    "        y_hat = model(X)\n",
    "        loss = loss_fn(y_hat, y)\n",
    "        loss_history.append(loss.item())\n",
    "        \n",
    "        # compute gradients of the loss wrt parameters\n",
    "        loss.backward()\n",
    "        \n",
    "        # perform gradient step to update the parameters\n",
    "        optimizer.step()\n",
    "\n",
    "    return loss_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, X):\n",
    "    # inform PyTorch that we are in \"evaluation\" mode\n",
    "    model.eval()\n",
    "    \n",
    "    # disable gradient tracking\n",
    "    with torch.no_grad():\n",
    "        # get prediction\n",
    "        y_hat = model(X)\n",
    "    \n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_history = train(model, X, y, epochs=50)\n",
    "y_hat = evaluate(model, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('b:', model.W[0,0].item())\n",
    "print('W:', model.W[0,1].item())\n",
    "plt.plot(loss_history)\n",
    "plt.title('Loss per epoch');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vis\n",
    "X = X_pt[:, 1:].numpy()\n",
    "y = y_pt.squeeze(-1).numpy()\n",
    "\n",
    "fig, axs = plt.subplots(1, 4, figsize=(16, 4))\n",
    "axs[0].plot(X, y, \"o\", label=\"data\")\n",
    "axs[0].plot(X, 40*X + 2, \"-\", label=\"pred\")\n",
    "axs[0].set_title(\"Guess\")\n",
    "axs[0].legend();\n",
    "\n",
    "axs[1].plot(X, y, \"o\", label=\"data\")\n",
    "axs[1].plot(X, 47.12483907744531*X + 2.3264433961431727, \"-\", label=\"pred\")\n",
    "axs[1].set_title(\"Numpy solution\")\n",
    "axs[1].legend();\n",
    "\n",
    "axs[2].plot(X, y, \"o\", label=\"data\")\n",
    "axs[2].plot(X, 47.12483596801758*X + 2.3264429569244385, \"-\", label=\"pred\")\n",
    "axs[2].set_title(\"Mixed solution\")\n",
    "axs[2].legend();\n",
    "\n",
    "axs[3].plot(X, y, \"o\", label=\"data\")\n",
    "axs[3].plot(X, y_hat, \"-\", label=\"pred\")\n",
    "axs[3].set_title(\"PyTorch solution\")\n",
    "axs[3].legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** I did gradient descent with the entire dataset rather than splitting the data into `train` and `valid` subsets, which should be done in practice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Write a proper training loop for PyTorch:\n",
    "  - add support for batches\n",
    "  - add a stop criterion for the convergence of the model\n",
    "  \n",
    "- Add L2 regularization"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}