Simple Neural Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised learning paradigm\n",
    "\n",
    "In this tutorial we will be looking at a method in supervised learning paradigm called neural networks. What does supervised learning mean? I suggest you to find this out [here](https://en.wikipedia.org/wiki/Supervised_learning). _TLDR?_ In Supervised Learning we **train** a **model** (typically using some kind of a machine learning algorithm) on a **labeled dataset** to predict labels on a new previously unseen data.\n",
    "\n",
    "## Loss function\n",
    "First of all we need to measure how good our model's predictions are. For this sake let's define a loss function which will take a model prediction (possibly incorrect), a true prediction from labeled dataset (people also frequently call it the train dataset). One of the popular ones is Mean Squared Error, which is defined as\n",
    "$$MSE(y, \\hat{y}) = \\frac12 (y - \\hat{y})^2$$\n",
    "\n",
    "Where $y$ is a vector that contains true values that we need to predict or generalize on (for example, from the test set) and $\\hat{y}$ is a vector that contains model predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def mse(target_y, model_y):\n",
    "    return 0.5 * np.square(target_y - model_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us plot two linear functions\n",
    "$$ y = y_{true}(x) = 2 * x + 3$$ \n",
    "$$\\hat{y} = y_{model}(x) = 3 * x + 3$$ \n",
    "\n",
    "and their elementwise $MSE(y, \\hat{y})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 520,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f20c90736a0>"
      ]
     },
     "execution_count": 520,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEPCAYAAABIut/fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdYU2f7B/DvSSDMQADBAVoUF6RuqDhQUBx1VFoqraAi\nWrWOVmltHW3FWesWR12t4qqv8FZx/Nq+XaB1teCqgli1anFUWQkzAZLz+wMJRFaAkJNxf66LS5Kc\ncecRbu485znPw7Asy4IQQojR4XEdACGEkKZBCZ4QQowUJXhCCDFSlOAJIcRIUYInhBAjRQmeEEKM\nFCV4YrLatm2Lzz//nOswTNrp06fB4/Hw+PFjrkMxSpTgdWjSpEng8Xh48803q7x24sQJ8Hg8CAQC\nDiKr3j///IPw8HC0adMGlpaWaNmyJYYMGYJffvmF69B0ojz5vPjF5/Nx9OhRrsMzGgzDcB2C0TLj\nOgBTwjAM2rRpg1OnTiEjIwPOzs6q13bu3Al3d3c8fPiQwwgrlJaWYvDgwXjppZfwn//8B61bt8bT\np0+RmJiIrKwsrsMDAJSUlMDc3LxJz8EwDK5cuYIWLVqoPe/g4FDjPqWlpTAzU//VYlkWLMuCx6t/\nTaWL90mME1XwOtaxY0f4+voiJiZG9Vx6ejp++uknREREVNn+0qVLGDZsGIRCIVxcXBAcHIx//vlH\n9fr9+/cRHBwMV1dX2NjYoGvXrjh48KDaMQICAjB16lSsWLECLVu2hJOTE8LDw1FYWFhjnCkpKbh7\n9y42b96Mvn37onXr1vD29sa8efMQEhKi2i4nJwdvvfUWbG1t0bJlS3z22WeYNGkShgwZonb+adOm\nqR1/5cqVaNu2rerxlStXMGLECDRv3hxCoRCvvPIK/ve//6nt07ZtW3z22WeYNWsWmjVrhgEDBgAA\nCgoKMGfOHLi5ucHGxga9evXCsWPH1Pa9du0a+vbtC0tLS3Tq1AlxcXE1vvcXNWvWDC4uLmpf5Ql3\n3759MDc3R2JiInr27AlLS0v88ssvWLp0KTp06IDY2Fh4enrCwsICt2/fBgCsW7cOHh4esLCwQPv2\n7REdHa3R+6xOXT8fo0aNwiuvvAKFQgGg7A9NYGAg/P39UX4T++bNm9GjRw8IhUK0bNkS48aNw7//\n/qs6Rvknme+//x59+/aFtbU1vL29kZqaitTUVPj5+cHGxga9e/fGzZs3VfuVt80vv/yCl19+GVZW\nVvD19cW1a9dqbe+7d+/izTffhIODAxwdHTFs2DDcuHFDk/8q8gJK8ByYNm0adu/erXr81VdfITAw\nEG3atFHbLjU1Ff7+/ujXrx8uX76MhIQEmJmZYciQISguLgYA5OfnY/Dgwfjf//6HGzduYPr06Zg8\neTJOnz6tdqxvv/0WOTk5OH36NI4cOYJTp05h9erVNcbo4uICPp+PuLg4lJSU1Ljd5MmTceXKFfzf\n//0ffv31V9y/fx/Hjh3T6GN35W1yc3Px9ttv4/Tp07hy5QqGDx+OMWPG4M6dO2r7bNmyBc2bN8fF\nixexd+9eAGVJ7Pr164iLi0NKSgpmzJiBcePGISEhAQAgk8kwcuRIODk5ITk5Gfv378fatWuRkZFR\nZ4yaUCqVWLBgATZu3Ii0tDR4e3sDAB4/fozt27dj//79SE1NhZubG7Zt24bFixdj0aJFSE1Nxccf\nf4wFCxao3ktt7/NFmvx8xMTE4PHjx1i4cCEA4PPPP8e1a9fwzTffqNqfYRisX78eN27cQHx8PNLT\n0zFu3Lgq5/v000+xatUqXL58GQKBAOPGjcPMmTOxfPly1XOTJ0+u0jbz58/Hjh07kJSUBGdnZ4wa\nNQpyubza9/Ts2TP0798fLVq0wLlz5/D777+jc+fOCAgI0JtPjgaFJTozadIkdsiQIaxMJmOdnJzY\nxMREVqFQsG5ubuyxY8fYmJgY1tzcXG37cePGqR1DJpOx1tbW7PHjx2s8z5gxY9hp06apHvv7+7Pd\nu3dX22bGjBls3759a413586drFAoZK2srNh+/fqx8+fPZ5OSklSv37lzh2UYhv3ll19UzxUXF7Ou\nrq7skCFD1M4/depUtWOvWLGCbdu2ba3n79atG/v555+rHru7u7OBgYFq2yQkJLBWVlZsbm6u2vOT\nJ09mX3/9dZZlWXb37t2sUChkpVKp6vUbN26wDMOwK1eurPH8iYmJLMMwrK2trdqXUChknzx5wrIs\ny8bExLA8Ho89d+6c2r5Llixh+Xw++/DhQ7XnW7duzS5YsEDtucjISNbDw6PW91kdTX8+EhISWHNz\nc3bp0qWsQCBgT5w4UetxL1++zPJ4PPbx48dq7VB5v7i4OJZhGPbYsWOq544dO8byeDy2oKCAZdmK\ntklISFBtk5OTw9ra2rJ79uxRHZvH47GPHj1iWbas3fr06aMWj1KpZD08PNjo6Og624Sooz54DlhY\nWGDChAnYtWsXcnNzoVAoMHr06CpdK0lJSbh79y6EQqHa83K5XPVxv6ioCEuXLsWpU6fw5MkTFBcX\no7i4GAEBAWr7dOvWTe1xq1at8OOPP9Ya57Rp0zB+/HicOXMGf/zxB3744QesXbsWX3zxBT766COk\npqaCYRj06dNHtY+5uTl8fHxQUFBQrzbJzMzE4sWLkZCQgH///RelpaWQy+V48OCB2navvPKK2uPk\n5GTI5XK0atVK7fmSkhJ07NgRAHDz5k14enrCzs5O9bpYLIa9vX2dcTEMgx9//BHNmzdXe/7Fx+VV\n+4vbuLq6qh7n5eXh4cOH8PPzU9tu4MCB2Lx5M2QyGSwtLat9n9XR5OcDAPz9/fHBBx9gyZIlmDFj\nBkaPHq22fWJiIr744gukpqZCIpFAqVQCAB48eICWLVuq2qFr166qfVq0aAGGYdClSxe154CyKtzd\n3V31vK+vr+p7kUgET09PpKSk1PiekpOTq7wnmUym9p6IZijBc2TatGno2bMn0tPTERERAT6fX2Ub\npVKJCRMmYOHChar+0nJOTk4AgHnz5uHkyZPYuHEjOnbsCBsbG3zwwQfIzc1V2/7F0TkMw6h+kWtj\nbW2N4cOHY/jw4Vi8eDGmTp2KxYsXIzIyUuP3yuPxqsT/YrdPeHg4Hj58iHXr1sHd3R1WVlZ46623\nVF0N5WxsbNQeK5VKiEQiJCcnVzmHtkYkvfTSS1X+gFTG5/OrPdeLsdaHJvtq8vNRvt3Zs2dhZmaG\nu3fvqm2Xnp6OkSNHIjw8HFFRUWjWrBnS09MRGBhYpe0rX+gt796p7jlNfq5qe0+BgYHYtm1blfek\nyR9koo4SPEc8PT3h4+OD8+fPY9++fdVu4+3tjT///FPtYuSLfvvtN4SFhSE4OBhA2UW0v/76q8qo\nD23p3LkziouLIZVK4eXlBZZlcf78eQwePBhAWeJOSkqCl5eXah8XF5cq45wvXbpU5X2sXbsWI0eO\nBFB24fTvv/9WqxCr4+3tDYlEgqKiIrVzVubl5YXdu3cjNzdXVcWnpKRAKpXW7803klAohJubG86c\nOYMRI0aonk9MTETbtm1V1bumNPn5AICoqCj8/fffOHfuHIYOHYq1a9fio48+AlBWMctkMmzcuBEW\nFhaq57Q5dPHixYvw9/cHAEgkEty8eRMzZsyo8T3t27cPrq6uejVk2FDRRVYO/fjjj8jMzKzxF3TR\nokW4efMmxo8fj6SkJNy/fx8JCQmYO3cu7t+/DwDo1KkTjh8/jqSkJKSmpmLatGlauWnk6tWreO21\n11QXLu/du4cjR45g7dq16N+/P5ycnODh4YHXXnsNs2bNQmJiIlJTU/HOO+8gPz9f7ViBgYH4+eef\n8d///hd3797F6tWrcfbsWbVtOnXqhEOHDuHGjRu4evUqQkNDNaoEBw0ahMDAQLzxxhs4fvw47t27\nh8uXL2Pr1q34+uuvAQChoaGwtbVFWFgY/vzzT1y8eBFTpkyBtbV1ncdnWRbPnj3D06dP1b6Kiorq\n0ZoVFi5ciC1btuCrr77CnTt3sHPnTuzcuROffPJJvY+lyc/H6dOnsXr1auzfvx8+Pj7YtWsXPv30\nUyQlJQEAOnToAIZhsG7dOty/fx/x8fFYvnx5te3Q0Oc+/vhj/Pbbb7h+/TomTpwIOzs7tYu4lfeZ\nPXs2FAoFXnvtNZw9exYPHjzA2bNn8emnn+LixYv1biNTRwmeQ5aWlhCJRDW+3rlzZ5w/fx4FBQUY\nPnw4xGIxpk+fDplMptpv48aNeOmllzBo0CAMGTIEbm5uGDt2rNpxGlKNtW7dGh06dMCqVavQv39/\ndO3aFVFRUYiIiMDx48dV2+3duxfdu3fH6NGjERAQADc3NwQFBakdKzw8HLNmzcLs2bPh4+ODhw8f\nYs6cOWrbxMTEQKlUonfv3njjjTfw6quvwsfHR6P3ceLECbzxxhv44IMP4OnpiVGjRuG7776Dh4cH\nAMDKygrff/89srOz0bt3b0yYMAEffPABXFxc6mwHhmHQq1cvtGrVSu1r+/btGrXji2bMmIFly5Zh\n1apVEIvFWLt2LVavXo1JkybV+T5fVNfPR05ODiZMmIDIyEgEBgYCAMaOHYuIiAiEhoaioKAAXbp0\nwZYtW7Br1y6IxWJs2LChyrDNmmLS5Dk+n4/PP/8c06dPxyuvvIKMjAx89913ap9WKu/j4uKCCxcu\nwNnZGcHBwejcuTMmTJiAf/75R3U9gGiOYav7k/uCWbNmwdraGgzDgM/nY9WqVcjPz8emTZuQkZEB\nFxcXREZGalQREeMXERGBR48e1XkRlxi3ffv2YerUqVX68onuaFTBMwyDqKgorFmzBqtWrQIAxMfH\no0uXLoiOjoZYLK5yY0lNarp6boqoLSpQW1SgtqhAbVGhIW2hUYJnn99mXVlycjIGDhwIoGwYVnmf\nXl3oP6wCtUUFaosK1BYVqC0qNKQtNBpFwzAMVqxYAR6Ph8DAQAwePBhSqVTVDywSiXQ+IoHor5ru\nvCSmJTw8HOHh4VyHYdI0SvDLly+Hg4MDcnNzsWLFimrHBNOMcIQQol80ushaWVxcHCwtLfHrr78i\nKioKIpEIEokES5cuxcaNG6tsn5KSovbRovJEVYQQQjQXGxur+l4sFkMsFte6fZ0JXi6Xg2VZWFpa\nQiaTYeXKlXjzzTdx/fp12NraIigoCPHx8SgoKEBYWJhGQdLk/mWEQiHy8vK4DkMvUFtUoLaowFVb\n8J4+hXNgIJQtWyLj5Eng+U1gXKrtbuqa1NlFI5VKsXbtWjAMA4VCAT8/P3Tr1g0eHh7YuHEjEhIS\n4OzsXK9b1wkhRG8plRBFRoIpLETOtm16kdwbqt5dNNpAFXwZqtQqUFtUoLaowEVb2OzaBfulSyFZ\ntQqFEyfq9Ny1aUgFT3eyEkLIc2Y3bsBu1SoUDRuGwgkTuA6n0SjBE0IIAKaoCA6zZ0Pp4ADpunWA\nEYwMpNkkCSGcsbW1rXWINZ/PrzI3fFPhZ2RAuWUL5F5esKlljqimxrJslQn7GooSPCGEMwzD6M/1\nBmfnsi8A4DAmbf5Boy4aQggxUpTgCSHESFGCJ4QQI0UJnhBCjBQleEIIMVKU4AkhxEhRgieEkBps\n27YN/fr1Q6dOnTBo0CD88MMPXIdULzQOnhCitxYvtkNqqnmjj+PlVYJly3LrvZ+7uzvi4+Ph7OyM\nkydP4r333sP58+fhXD5eXs9RBU8IITUYOXKkKpmPHj0abdu2xZUrVziOSnNUwRNC9FZDqm5tiouL\nw+7du/Hw4UMAQGFhIXJycjiNqT4owRNCSDUePXqE+fPnIzY2Ft7e3gCAoUOHgoMZ1huMEjwhhFSj\nsLAQDMPA0dERSqUScXFxuHXrFtdh1Qv1wRNCSDU6dOiA6dOnY/To0ejevTtu3boFHx8frsOqF1rR\niUO0ck8FaosKptQWpvReNVVTm9CKToQQQlQowRNCiJGiBE8IIUaKEjwhhBgpSvCEEGKkKMETQoiR\nogRPCCFGihI8IYQYKUrwhBBSA19fX5w9e5brMBqMEjwhhBgpSvCEEGKkKMETQkgtrl69ioCAAIjF\nYnz44YcoLi7mOiSN0XTBhBC9tfjCYqRmpTb6OF5OXljWZ1mD9o2Pj8fhw4dhZWWF8PBwREdH46OP\nPmp0TLpAFTwhhNQiIiICLVq0gL29Pd5//33Ex8dzHZLGqIInhOithlbd2tSyZUvV925ubnj69CmH\n0dQPVfCEEFKLyutXPHz4EM2bN+cwmvqhBE8IIbWIiYnBkydPkJOTgy1btmDMmDFch6QxSvCEEFID\nhmHw+uuvIzQ0FP3790fbtm3x/vvvcx2WxjTug1cqlVi4cCEcHR0xf/585OfnY9OmTcjIyICLiwsi\nIyNhbW3dlLESQohOXbhwAQAwa9YsjiNpGI0r+O+++w6urq6qx/Hx8ejSpQuio6MhFotx7NixJgmQ\nEEJIw2iU4LOysnDlyhUMHjxY9VxycjIGDhwIAPD390dSUlLTREgIIaRBNErw+/btw4QJE8AwjOo5\nqVQKkUgEABCJRJBKpU0TISGEkAapsw/+8uXLsLe3h7u7O1JSUmrcrnLyrywlJUVtv5CQEAiFwgaE\nanwEAgG1xXPUFhVMqS34fD7XIegdPp9f4/9/bGys6nuxWAyxWFzrsepM8GlpaUhOTsaVK1dQXFyM\noqIibNmyBSKRCBKJRPWvvb19tftXF0ReXl5dpzUJQqGQ2uI5aosKptQWpvKHrD4UCkW1//9CoRAh\nISH1OladCT40NBShoaEAgNTUVJw8eRLvvfceDh48iMTERAQFBSExMRHe3t71OjEhhJCm1eBx8EFB\nQbh+/TrmzJmDGzduICgoSJtxEUIIaSSGZVlW1yetfOuvKTOlj+J1obaoYEptYUrvVVM1tUmrVq3q\nfSy6k5UQQowUJXhCCKlB79690a5dO+Tk5Kg9P3ToULi5ueHRo0d48uQJpk6dii5dusDLywuBgYGI\ni4sDUDY5mZubGzp16oROnTqhY8eO6NSpE06ePKmT+Gm6YEIIqQHDMGjdujWOHz+OSZMmASgbWSiT\nyVRDw99//328/PLLSEpKgkAgwM2bN5GRkaF2jLS0tBqHkjclquAJIaQWwcHBqoocAOLi4jB27FgA\nAMuyuHbtGsaOHQtLS0vweDyIxWL4+/urHYODS50AqIInhHCptLTWl+0WL4Z5auOX7Cvx8kLusoYt\nHtKzZ098++23uHPnDtq1a4cTJ04gPj4ea9asAcMw6NWrFxYtWoSIiAh4e3urzdlVjqsETxU8IYQb\nSiXMbt/mOgqNlFfxZ86cQYcOHdCiRQtV0t65cyd69+6NzZs3o2/fvhg2bBiuXbum2pdlWXTt2hVi\nsRheXl4Qi8W4c+eOTuKmCp4Qwgnh2rXgjRgBuLnVuE1Dq25tCw4OxhtvvIH09HS8+eabaq/Z2dlh\n4cKFWLhwIXJycrBs2TJMmTIFycnJAMr64G/cuEF98IQQ02B58iSEmzdDYSDL37m6uqJ169ZISEjA\nq6++WuN2Dg4OePfdd/H06VNIJBLV89RFQwgxCWYpKRBFRqK4Vy8o2rblOhyNbdiwAbGxsbCyslJ7\n/vPPP8etW7egUCiQn5+Pffv2wd3dXTXbLsuydJGVEGL8eNnZcJwyBay9PbJ374YNT79rzMrdKm3a\ntKn2taKiIkyZMgUZGRmwtLREjx49sHfvXrXtvLy8AJQle4ZhMG/ePEydOrXp46epCrhDt2lXoLao\nYLRtUVICp9BQCC5dQua336KkRw/jfa+NoM2pCqiCJ4TohN3y5bA4fx45mzahpEcPrsMxCfr9+YgQ\nYhSsjhyB7ddfI/+dd1D0/CYh0vQowRNCmpT5pUsQLVgAef/+yP3sM67DMSmU4AkhTYb3779wnDoV\nihYtkL19O2BGvcK6RK1NCGkaMhkc33kHTF4esg4dAuvoyHVEJocSPCFE+1gWokWLILhyBdm7dqHU\n05PriEwSJXhCiNbZ7N0L6yNHkDd3LmQjR9a4HcuytS68zefzoVAomiJEvaXNkeuU4AkhWiU4dw52\nS5agaOhQ5H34Ya3b5ufn1/o6jZNvHLrISgjRGv4//8Bh+nSUtmsHyebNgJ7fqWrsqPUJIVrBFBbC\ncfJkMEolsvfsAVtL1wvRDeqiIYQ0HstCFBkJs1u3kL1/PxTt2nEdEQFV8IQQLbDdsgVWp04hd9Ei\nyAMCuA6HPEcJnhDSKBY//wzhmjUoDApCwbvvch0OqYQSPCGkwczu3IHD7NkoEYshXbcO4GDVIlIz\nSvCEkAZhpFI4RkSAFQiQs2cP2BcWwiDawbIsLj291KB96SIrIaT+FAo4zJ4N/j//ICs2FgpXV64j\nMjpSuRRH7xzFobRDuJl9E2yP+t8ARQmeEFJvwjVrYPnrr5CsWoXi3r25DsdosCyLS88u4VDaIZy4\newIyhQzdmnXDGr81DToeJXhCSL1YHj8O4datKAgLQ+HEiVyHYxRerNZtzG3wZoc3Md5zPLo069Lg\n41KCJ4RozOzGDYg++AByHx9IV6zgOhyDVlu1HuQRBBtzm0afgxI8IUQjvKwsOE6eDNbBATm7dwMC\nAdchGaSmqtarQwmeEFK3khI4TJ8OflYWMo8ehdLZmeuIDIouqvXqUIInhNTJbulSWFy4gJzNm1HS\nrRvX4RgMXVbr1aEETwiplfXhw7Dduxf506ejKDiY63D0HlfVenUowRNCamSenAz7hQshGzAAuYsW\ncR2OXuO6Wq9OnQm+pKQEUVFRKC0thUKhgK+vL8aOHYv8/Hxs2rQJGRkZcHFxQWRkJKytrXURMyFE\nB3hPnpQtmO3qipwvv6QFs6uhT9V6dRhWg/Wh5HI5LCwsoFQq8dlnnyEiIgIXL16EUCjEmDFjEB8f\nj4KCAoSFhWl00sePHzc6cGNAq9VUoLaooBdtIZOhWXAwzG7fRubJkyjt1ImTMPSiLapRXbX+usfr\nTVqtt2rVqt77aDQXjYWFBYCyar58fcTk5GQMHDgQAODv74+kpKR6n5wQoodYFqIFCyC4ehWSzZs5\nS+76hmVZJD9NRuTpSPQ81BOfnv8UAp4Aa/zW4ErYFaz2W81ZV0xNNPrMpVQqsWDBAjx9+hTDhg1D\n+/btIZVKIRKJAAAikQhSqbRJAyWE6Ibttm2wjotD7ocfQjZ8ONfhcE4f+9Y1pVGC5/F4WLNmDQoL\nC7Fu3Tqkp6dX2YahaUIJMXjWBw7AbtUqFI4Zg/y5c7kOhzP63reuqXpdNbG2toaXlxeuXr0KkUgE\niUSi+tfe3r7afVJSUpCSkqJ6HBISAiGt1QgAEAgE1BbPUVtU4KotzP77X1guXIjSYcOg2LMHQnNz\nncfwIl23hUQmwZGbRxBzPQYpmSmwNbdFqDgUk7pMQvfm3XUWR01iY2NV34vFYojF4lq3r/Mia25u\nLszMzGBtbY3i4mKsXLkSY8aMQWpqKmxtbREUFEQXWRtIXy8gcYHaogIXbWHxyy9wnDwZxd7eyDp4\nENCTud110Rbl1frBmwdx8u+Tqmo9zDNMr6r1hlxkrbOCl0gk2LZtG5RKJViWRd++fdGzZ0907NgR\nGzduREJCApydnREZGdmgoAkh3BL8/jscp01DiacnsmNi9Ca5NzWpXIpvb3+LQ2mHkJaTZlB965rS\naJiktlEFX4aq1grUFhV02Rbm16/DaexYKJo3R9bRo1A6OenkvJrSdlsYSrVenSap4Akhxsnszh04\nhoZCaWeHrG++0bvkrk2mUK1XhxI8ISaI//AhnN5+G+DzkfWf/0BphEvu1VStG9pImMagBE+IieFl\nZMDp7bfBFBYiMy4OinbtuA5Jq0y1Wq8OJXhCTAgjlcIpNBS8f/9F1uHDKK1jmJ2hoGq9epTgCTER\nTGEhnCZOhNnt28iOiUGJjw/XITUaVeu1owRPiCkoLobD1Kkwv3wZOdu3Q+7vz3VEDUbVuuYowRNi\n7BQKOLz3HiwTE5Gzfj1ko0ZxHVGDULVef5TgCTFmLAv7+fNhdeoUpIsXo+jtt7mOqF7KZ3Csrlof\n024MbAW2XIeo1yjBE2KsWBZ2y5fD5vBh5M2Zg4Lp07mOSGPl1frhvw4jNSuVqvUGogRPiJGy3bwZ\ntjt3Ij8iAnkffcR1OHWqrm+9R/MeVK03AiV4QoyQdUwM7NasQWFwMHKXLQP0eDrv2vrW+7btS1NY\nNAIleEKMjNXRoxB98gmKhg6FZP16gKfRwm06VdtIGKrWtYcSPCFGxOLHHyGaOxfyvn2Rs307oAdz\nuldGI2F0ixI8IUZCcO4cHN99FyVduiB7717A0pLrkABQtc4lSvCEGAHzq1fhGBGBUnd3ZB04ANaW\n+6RJ1Tr3KMETYuDM/voLTmFhUDo5Ieubb8A6OnIWC1Xr+oUSPCEGjP/PP3AaNw6sQICsw4ehbNGC\nkzikcimO3jmKQ2mHcDP7JlXreoISPCEGivf0adm0vzIZMr/9Fgp3d52ev7xaP5R2CCfunqBqXQ9R\ngifEADE5OWXT/mZkIOvIEZR27qyzc1O1bjgowRNiYJiCAjhNmACzv/9G1v79KOnZs8nPSdW6YaIE\nT4ghkcngOHkyzP/8Ezm7dqHYz69JT0fVumGjBE+IoSgthcOsWbA4exY5mzZBNnx4k5yGqnXjQQme\nEEOgVEI0bx6sfvgB0uXLUTR2rNZPQdW68aEET4i+Y1nYLVkC67g45M6bh4LJk7V4aKrWjRkleEL0\nnO3GjbD9+mvkv/MO8ufO1coxqVo3DZTgCdFjNl99Bbv161H41lvIjYpq1LS/VK2bHkrwhOgpqyNH\nYB8VhaIRIyBZs6bB0/5StW64JBIGFy5Y4LffLBATU//9KcEToocsv/8eonnzIBswADlbtwJm9ftV\nra5a79qsK1Xrek4uBy5dEuDMGQucPWuBa9fMoVQysLZWUoInxBjwExJgO3MmSrp3R85XXwEWFhrv\nS9W6YVEqgZs3zfDbb2VV+sWLAshkPPD5LHr2LMbcufnw85OjR49iAK3qfXxK8IToEfNLl2AVGopS\nD4+yaX9tbOrch/rWDcujR3z89ltFlZ6VxQcAdOhQgrCwQvTvL0efPsUQCtlGn4sSPCF6QnDxIhwj\nIsA2b15+UufFAAAaN0lEQVQ27a9IVOv2VK0bhvJ+9DNnyqr0e/fK0q6LiwIDB8oxYIAc/fvL0bKl\nUuvnpgRPiB6w/P57OMyahdLWrSE/fhzKGpI7Vev6r7Z+9D59ijFpUgH8/OTo2LG0yddCpwRPCMes\nDxyA/aJFKOneHVn79sG2dWsgL09tG6rW9Vdt/eg9epSo+tG7dy+GQKDb2CjBE8IVloXtpk2wW7cO\nssGDkbNjB1hr60ovU7Wur8r70cuTeuV+9NDQQvj5aa8fvTEowRPCBYUC9p98ApsDB1A4diwka9cC\n5uYAAIlMgv0p+6la1yNSKYPz58uS+Zkzuu1HbwxK8ITomkwGh/feg9V33yFv9mzkLVgAFsClp8k4\nlHYIJ/8+iaLSIqrWOVTej15eob/Yjx4eXoABA3TTj94YdSb4rKwsbN26FVKpFAzDYPDgwRgxYgTy\n8/OxadMmZGRkwMXFBZGRkbCu9PGSEFIVk5sLx8mTYXHhAqRLluDxxBAcTY1Rq9bHeY1DiEcIVes6\nVFc/+pw5+RgwgJt+9MZgWJattZNIIpFAIpHA3d0dMpkM8+fPx8cff4yEhAQIhUKMGTMG8fHxKCgo\nQFhYmEYnffz4sVaCN3RCoRB5L1xMM1Wm0Ba8p0/hNH48zG7fxuUl72FN24dqfethnmEY024MWjq1\nNPq20FRT/lzUNB69ffsSDBgg15t+9HKtWjXBjU4ikQii50O2LC0t4erqiqysLCQnJ2PJkiUAAH9/\nfyxZskTjBE+IqeHfvQuH0HFgMzMw9Z0W+LpkA2zuUd+6LtU1Ht3Pr+xL3/rRG6NeffDPnj3DgwcP\n0LFjR0ilUlXiF4lEkEqlTRIgIYaMZVncTfgPus/8BBKlHCMmAKViR6zxfI/61ptYXePRDaUfvTE0\nTvAymQwbNmzApEmTYGlpWeV1xlhbiJAGKB+3fu/YTmz6Kh0Ztgy2LX4NKwfNpGq9iRhrP3pjaJTg\nFQoF1q9fjwEDBsDHxwdAWdUukUhU/9rb21e7b0pKClJSUlSPQ0JCIBQKtRC64RMIBNQWzxlDW7As\niz+e/IGY6zE4eusogi4XYV88IGnnCqv4k4hq016j4xhDW2hLXW2Rns4gMZGPX381w+nTfGRmlk2p\n3KmTAuHhpRg0qBT9+ilgZ1e+h8XzL8MUGxur+l4sFkMsFte6fZ0XWQFg69atEAqFCA8PVz138OBB\n2NraIigoiC6yNpApXFjUlCG3RXV3me5I64TxBy5D3qcPsvfsAVuRYepkyG2hbS+2RW396P376+94\ndG1oyEXWOhN8WloaoqKi0KZNGzAMA4ZhMG7cOLRv3x4bN25EZmYmnJ2dERkZCRsNZr4DKMGXo1/k\nCobWFjXdZRrWORRT4m7DcedXKBo5EjmbNwPVdGnWxtDaoikJBEIkJMjx22/V96OXj3Yx5n70ck2S\n4JsCJfgy9ItcwVDaorpq/XWP18tGwth3huijj2AdF4eC8HBIly8H+Px6n8NQ2qIpKJVAaqoZzp4t\nq9B//90CRUWMqh+9PKGbUj96uSYZJkmIqdNkThimsBAOkyfD8tdfkTtvXtni2MZeUmrJo0f8510u\ngirzo0+cWAJf3zy9Go9uSCjBE1IDTWdwZLKz4TRxIsyvXYNk9WoUjh/PYdT6TyKpmNelcj968+YK\n+PuXVejl/ehln2bkHEdsuCjBE1JJfWdw5D96BMfQUJilpyNn927Ihg/nKHL9JZcDycmCKv3oNjZK\n+Prqdn50U0MJnhA0bL51s7Q0OIWFgSksRNY336DY11fHUesnTedH79GjuHwCTdJEKMETk9WY+dYF\nf/wBx0mTwFpaIvPoUZR6euowcv1T2zqj+jQ/uqmhBE9MTmNXR7L48Uc4zpgBRatWyPrmGyhat9ZB\n1PqFy3VGieYowROToK3Vkay/+Qb28+ejpFs3ZO/fD6WjYxNHrh/0aZ1RojlK8MSoaW0tU5aF7ebN\nsFuzBrKAAOTs2qW2vJ6x0ed1RonmKMETo6P1tUwVCthFRcF2714UBgdDsn49jPHqIPWjGx9K8MRo\naK1ar0wuh8P778Pq1Cnkv/sucj/5BODxtBs4R+paZ9QY50c3NZTgiUHTerVeCZOXB8cpU2Bx7hyk\nn32Ggnff1WLkumcs64wSzVGCJwapSar1SnjPnpUtr3frFnI2b0ZRcLAWotatyv3oZ8+W9aMXFZn2\n/OimhhI8MRg1Veur+69GkEeQ1lZH4t+7B6ewMPAyMpAdEwN5QIBWjqsL5f3o5VV65X70t98uxIAB\n1I9uSijBE73X1NV6ZebXr8Nx/HhAoUBWbCxKevTQ6vG1zRTXGSWaowRP9FJT9q3XRHDmDBzfeQdK\nBwdkHToERXvNVmDSJVpnlNQHJXiiV3RZrauwbNkNTJ98gtL27ZF18CCULVo0zbnqidYZJY1BCZ5w\njmVZ/P74d+y6tAsn/z6pk2q9HJOXB/v582F9/DhkAwYgZ8cOsDWsL6wr6ekMfvjBisajk0ajBE84\nI5VL8e3tb3Eo7RDSctJ0U61XYv7nn3CYMQP89HTkLliA/FmzOBnjTvO6kKZCCZ7oVHnf+sGbB9Wq\n9c2BmzHMdViTVuuVgoDNnj2wW74cymbNkPXttyj28Wn68z5XVz/69OmleOWVXOpHJ41GCZ7oRF3V\nuq7WIWVyciD68ENY/e9/kA0ZgpwNG8A28YRh9Z3XpawtSps0JmIaKMGTJlNTta6LvvXqmCclwWHm\nTPAzMiBdsgQF77zTZOum1rbOKPWjE12hBE+0juu+9SqUSth++SWEa9ZA4eaGzOPHUdKtm1ZPUdM6\no9SPTrhECZ5ohb5V6+V4GRkQzZkDy9OnUTR6NCRr1oC1s2v0cWk8OjEElOBJo+hdtV6J4OxZOLz3\nHni5uZCsXo3CsLAGd8nQ/OjEEFGCJ/Wmr9W6SmkphBs3wjY6GqUeHsj65psGrZlK86MTQ0cJnmhM\nn6v1crwnT+AwezYsLl5EYUgIpCtXarzyEo1HJ8aGEjypld5X65VY/PwzRHPngpHLkRMdjaI336x1\ne1pnlBg7SvCkWtVV68EdgjHBc4LeVOsqxcWwW70atjt2oMTTE9k7dlQ7URj1oxNTQwmeqBhStV6O\n/88/cJg5E4IrV1AQHg7p4sWApaXq9drmR6d+dGLsKMETg+hbr47l//0fRPPmASyL7J07IRs1qmyd\n0YSa1xmlfnRiSijBmyhDrNZVZDLYL1sGm337IOvWA79O3Y3/pXTEbzuoH52QyijBmxhDrdbL8e7c\nhU3EDNj8nYLY1nMw9dYXyJ1tSf3ohFSDErwJMOhqHRX96LxD3+Kdy3MhgyXexincthyKN0Pl8PPL\non50QqpBCd6IGWq1/uJ49Kf35NiK2YhADFKd+uHCezuwbFQztGyZwXWohOg1SvBGhou1TBtLLgeu\nXuXjhx+EVcajh3a5jM/zx6NZ5l/IfX8uRB9E4lUzMwB0kZSQulCCNxKcrGXaQBqNR+8vQ/+be+G0\nPApKOztkHT6MYj8/rkMnxKDUmeC3b9+Oy5cvw97eHuvWrQMA5OfnY9OmTcjIyICLiwsiIyNhreHt\n4ER7DKlar2tel2HDeOjWTQqhkAWTlwfRxx/D6sQJyAYMgGTzZiidnTl+B4QYnjoTfEBAAF599VVs\n3bpV9Vx8fDy6dOmCMWPGID4+HseOHUNYWFiTBkoqGEK1Xt/50ctWMWJhfu1a2TqpDx8id+FC5M+c\nyck6qYQYgzoTfOfOnZGRoX4xKzk5GUuWLAEA+Pv7Y8mSJZTgm5i+V+tyOZCcXHbHaOV+dBsbDcej\nsyxsvvoKditWQOHsrPN1UgkxRg3qg5dKpRCJRAAAkUgEqVSq1aBIBX2t1pVKIDXVDGfPVu1H79mz\nWDUevUePYpib134sJicHllOnwvy771A0dCgkGzaAdXDQzRshxIhp5SIrU8stgikpKUhJSVE9DgkJ\ngVAo1MZpDZ5AIKi2LViWxR9P/kDM9RgcvXUURaVF6NG8BzYHbkZw52AIBdy0X3o6g4QEMyQk8HH6\nNB+ZmWVdJ506KTBpUikCAkrRr58CFQsmWTz/qoFSCbPDh2GxdCmYrCzIvvgCpTNmwNbEbzmt6efC\nFFFbqIuNjVV9LxaLIRaLa92+QQleJBJBIpGo/rW3t69x2+qCyMvLa8hpjU5Zv3NFW1RXrQe3D1av\n1uVAnlw37VdTP3rz5goMHCiDn1/187po8t9rnpwM+6goCK5eRXGPHiiJjYXUwwPIz2+Kt2JQXvy5\nMGXUFhWEQiFCQkLqtY9GCZ5lWbBsxV2CvXr1QmJiIoKCgpCYmAhvb+/6RUpUaupbX91/NYI8gnTa\nt15XP7o21hnlPX4Mu1WrYH30KBQtWpTN2/7GGxDa22v2l4EQojGGrZy5qxEdHY3U1FTk5eXB3t4e\nISEh8PHxwcaNG5GZmQlnZ2dERkbCxsZG45M+fvy40YEbOqlciv9L/z/subZHVa2/7vG6TvvW6+pH\n9/Mr1rgfvU5FRbDdsQO227aBUSqRP3068mfPBvv854YqtQrUFhWoLSq0atWq3vvUmeCbgqkm+Jqq\n9TDPMAR5BMHGXPM/kg316BH/+RQAgirj0cuHLmp1XheWheWJE7BbuRJmjx6haORI5H76KRRt2qht\nRr/IFagtKlBbVGhIgqc7WXWgppEw03pNg4e1R5Oeu7Z+dH9/eY396Npgfv067BYvhsUff6DEywuZ\n0dEo7tNH6+chhFSPEnwTqW3cenm13hTViT6sM8rLyIBw9WpY/+c/UDo6QrJmDQrffhvg85vmhISQ\nalGC1zJdj1vXq3VG5XLY7NkD4aZNYGQyFEybhry5c8FWjJskhOgQJXgt0KRa16a65nXR+TqjLAuL\nn36C/dKlMLt/H7LAQEgXL4bCo2m7nwghtaME3wi6qtZfnB9dn9YZNbt1C3ZLlsDyzBmUdOiArEOH\nIPf313kchJCqKMHXky6qdX3oR68Lk50Nu/XrYX3gAFhbW0iXLUPBxIlo/HhKQoi2UILXUFNW63X1\no8+Zk48BA/RkndGSEtgcOADh+vVgcnNROGEC8ubNg9LRkePACCEvogRfi6as1h894iMpyQw//ijS\nj350DVicOQO7qCiY//UX5P37Q7pkCUo9PbkOixBSA0rw1WiKal2f+9Hrwv/7b9gvWwbLn35C6Usv\nIXvPHsiGDgVn/UOEEI1Qgn9O29W6Jv3ow4ebwdVVqrd5ksnNhTA6GjZffw1WIEDuJ58gf8oUwKKW\nGSIJIXrD5BO8tqr1hoxHL7vRqYneWGMoFLA+cgTC1avBy8pC4VtvIW/+fChdXLiOjBBSDyaZ4LVV\nrdc2r4u+9qPXRXDxIuyioiC4cQNyHx/kHjiAkq5duQ6LENIAJpXgG1ut13edUUPCf/gQdsuXw+rU\nKZS2aoXsL7+E7LXXqJ+dEANm9Am+MdV6TfOj69N49MZiCgthu20bbHfsAAsg98MPUTBjBlgrK65D\nI4Q0ktEm+IZU65quM6oX49EbicnOhnVsLGx37wb/339RGBSE3EWLoHR15To0QoiWGFWCb0i1Xls/\nelhYofbnR+cSy8L8yhXY7NsHq5MnwcjlkL/yCnJ27ECxjw/X0RFCtMwoEnx9qnUu50fnClNYCKv4\neFjv2wfBjRtQ2tigMCQEBRMnotTLi+vwCCFNxGATvKbVel3rjE6aVID+/eXo1Mlw+9FrYnb7Nqz3\n74f1f/8LXm4uSjp3hmTlShQFB4OlleoJMXoGl+DrqtY17UfXyjqj+qi4GJY//ACb/fthceECWHNz\nFI0cicLw8LJuGGP7K0YIqZFBJPi6qnXJMzuc+ckC202hH70GvEePYHPoEKwPHwb/2TOUurkhd+FC\nFL79NpTNmnEdHiGEA3qd4Guq1se4jYfkpg9+22eB7UY2Hr1elEpYnDkD6/37YfnTTwDLQj5oECQT\nJ0IeEEBL5BFi4vQuwVdXrXdx6oZpLTeAufE2/vivE74xwvHo9VE+xNHmwAGY3b8PhZMT8mfORGFY\nGBRt2nAdHiFET+hNgn+xWrfi2cCrdByYy1OR8nNvXOdqnVF9wbIwv3wZNvv3qw1xzJs3D0UjRtAE\nYISQKjhN8JWr9eN3TkKuLIKooCdsft+BgothuFxsa9DzumgDU1gIq2PHYLNvH8xTUsqGOL71VtkQ\nR5qLnRBSC04SvFQuxcHrR7Hv+jd4VJoKpsQW7LWJwKVpECi6YVB/OfxWy+Hn96/x96PXwOyvvyqG\nOOblocTTE5JVq1D0xhtgbW25Do8QYgA4SfDiPb3AmhUBj7whuL4DfYTBCOhnjgEfyNGx41OT6Eev\nVnExLL//HjYHDpQNcRQIUDRqFAonTkSxtzcNcSSE1AsnCb7ZozAMcZyIsSM80X1RMQSCYgDFXISi\nF/iPHsH64MGyIY4ZGSht3Rq5ixaVDXF0cuI6PEKIgeIkwV9dsfT5d6ab1KFUgv/TT3DYuROWP/9c\nNsRx8OCyIY7+/jTEkRDSaHozisbosSzM7t6F4Px5CC5ehMXFi+A/fVoxxHH8eChat+Y6SkKIEaEE\n31RYFmZ//QXBhQuwuHABgt9/Bz8jAwCgaN4ccl9fMK+9hpyAABriSAhpEpTgtUWphNnNm7C4eBGC\n51/87GwAgKJlS8j9/FDs6wt5nz5QtG0LMAyEQiH0c1FWQogxoATfUAoFzFNTIbhwoazL5fffwZNI\nAAClbm6QDxoEed++KPb1Lbu7lEbAEEJ0jBK8pkpLYX7jRlkyv3ABgj/+AC83t+yll15C0fDhKPb1\nRXGfPlC4uXEcLCGEUIKvWUkJzP/8s6zL5cIFCJKSwMvPBwCUtmuHotGjy7pcfH2hbNWK42AJIaQq\nSvDlioshuHZNNcpFkJwMXmEhAKCkQwcUvf465H36oNjXF8rmzTkOlhBC6taoBH/16lXExMSAZVkE\nBAQgKChIW3E1PZkMgitXVF0u5pcugSeTAQBKOndGUUgI5L6+ZQnd2ZnjYAkhpP4anOCVSiW+/vpr\nLF68GA4ODli4cCF8fHzg6uqqzfjqj2XB5OWBl5kJXlYW+NnZ4GVlqX3xHz2C4OpVMHI5WIZBqacn\nCsPCUNynD4p794bS0ZHb90AIIVrQ4AR/584dtGzZEs7Pq9t+/fohKSlJ+wleqQQjlYL/QpLmZWWB\n9zx58194zJSUVH8oKysonZygdHFBwcSJZaNcfHzAOjhoN2ZCCNEDDU7w2dnZcKo0T4qjoyPu3Lmj\n0b7l1fWLyZpfTfLmZWeDUSiqPY7S1rYsYTs5QeHqiuKuXcseOzqqnq/8xVpZNfTtEkKIweHkImuL\nbt2qfV4pEkHp6AiFkxNK3d2h7NWrSrJWVErgdAcoIYTUrMEJ3tHREZmZmarH2dnZcKym7zolJQUp\nKSmqxyEhIQBb/aIdvOdfpjS0RygUch2C3qC2qEBtUYHaokJsbKzqe7FYDLFYXPsObAMpFAp29uzZ\n7LNnz9iSkhJ23rx5bHp6ep37HTlypKGnNDrUFhWoLSpQW1SgtqjQkLZocLHM4/EwZcoUrFixAizL\nYtCgQXCjOzgJIURvNKo3pHv37oiOjtZWLIQQQrSIp+sT1tlnZEKoLSpQW1SgtqhAbVGhIW3BsGwN\nVzwJIYQYNJ1X8IQQQnSDEjwhhBgpnQ05N+iJybQoKysLW7duhVQqBcMwGDx4MEaMGMF1WJxSKpVY\nuHAhHB0dMX/+fK7D4UxhYSF27NiB9PR0MAyDGTNmoEOHDlyHxYlTp04hISEBDMOgTZs2mDlzJszM\nTOcOme3bt+Py5cuwt7fHunXrAAD5+fnYtGkTMjIy4OLigsjISFhbW9d+IC0P1axWdWPmHz58qItT\n652cnBz23r17LMuybFFREfv++++bbFuUO3nyJBsdHc1+8cUXXIfCqa1bt7K//vory7IsW1payhYU\nFHAcETeysrLYWbNmsSUlJSzLsuyGDRvYxMREjqPSrZs3b7L37t1jP/zwQ9VzBw4cYOPj41mWZdlj\nx46xBw8erPM4OumiqTwxmZmZmWpiMlMkEong7u4OALC0tISrqyuyn6/daoqysrJw5coVDB48mOtQ\nOFVYWIi0tDQEBAQAAPh8ft3VmRFTKpWQyWRQKBSQy+VwMLEJATt37gwbGxu155KTkzFw4EAAgL+/\nv0Y5VCefeRozMZkxe/bsGR48eGCyH8MBYN++fZgwYQIKny+uYqqePXsGoVCIL7/8Eg8ePEC7du0Q\nEREBgUDAdWg65+joiFGjRmHmzJmwsLBA165d0bVrV67D4pxUKoVIJAJQVihKpdI696GLrByRyWTY\nsGEDJk2aBEtLS67D4UR5H6O7uztYlgVrwiN2lUol7t27h2HDhmH16tWwsLBAfHw812FxoqCgAMnJ\nyfjyyy+xc+dOyGQynD17luuw9A7DMHVuo5MEr+nEZKZCoVBg/fr1GDBgAHx8fLgOhzNpaWlITk7G\n7NmzER0djZSUFGzdupXrsDjh6OgIJycneHh4AAB8fX3x999/cxwVN65fvw4XFxfY2tqCx+Ohd+/e\nuHXrFtdhcU4kEkEikQAAJBIJ7O3t69xHJwm+ffv2+Pfff5GRkYHS0lKcO3cO3t7euji1Xtq+fTvc\n3NxMfvRMaGgotm/fjq1bt2Lu3Ll4+eWXMXv2bK7D4oRIJIKTkxMeP34MoCzJmercTs2aNcPt27dR\nXFwMlmVx/fp17leK48CLn2p79eqFxMREAEBiYqJGOVRnd7JevXoVe/fuVU1MZqrDJNPS0hAVFYU2\nbdqAYRgwDINx48ahe/fuXIfGqdTUVJw8edKkh0nev38fO3fuRGlpKZo3b46ZM2ea7IXWuLg4nD9/\nHnw+H+7u7nj33XdNaphkdHQ0UlNTkZeXB3t7e4SEhMDHxwcbN25EZmYmnJ2dERkZWeVC7ItoqgJC\nCDFSdJGVEEKMFCV4QggxUpTgCSHESFGCJ4QQI0UJnhBCjBQleEIIMVKU4AkhxEhRgieEECP1/6Nf\nDV1IO+CGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20c90d48d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(0, 10, 10)\n",
    "y_true = np.multiply(x, 2) + 3\n",
    "y_model = np.multiply(x, 3) + 3\n",
    "mse_error = mse(y_true, y_model)\n",
    "\n",
    "plt.style.use('ggplot')\n",
    "plt.title(\"Mean Squared Error example\")\n",
    "yt_plot, = plt.plot(x, y_true, linewidth=1.5, color='blue', label='y_true')\n",
    "ym_plot, = plt.plot(x, y_model, linewidth=1.5, color='green', label='y_model')\n",
    "mse_plot, = plt.plot(mse_error, linewidth=1.5, color='red', label='MSE')\n",
    "plt.legend([yt_plot, ym_plot, mse_plot], ['a', 'b', 'MSE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see the more our funtions are different the larger their MSE gets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi Layer Perceptron\n",
    "Multi Layer Perceptron is a conventional neural network architecture we will be looking at during this tutorial. Here I will give the basics but I strongly suggest that you read more comprehensive material like those [excellent tutorials here](neuralnetworksanddeeplearning.com/chap1.html) and [here](neuralnetworksanddeeplearning.com/chap2.html). \n",
    "\n",
    "### Neurons\n",
    "Each neural network model consists of several collections of units called neurons (this concept is loosely based on the structure of our brain). In its most simple form (there are more advanced ones like LSTM which we will not cover here) each unit accepts a single real-valued input $x \\in \\mathbb{R}$ and evaluates on it a special function called activation function.\n",
    "\n",
    "#### Activation function\n",
    "Now lets construct our activation function. For this example we will use so called Rectified Linear Unit or ReLU which is defined as \n",
    "$$ReLU(x) = max(x, 0)$$\n",
    "\n",
    "This kind of activation function turns out to be useful for its properties:\n",
    "1. It is partially differentiable and differentiability is of paramount importance for neural networks as you will see\n",
    "1. It's derivative does not saturate anywhere. Actually, it is just a constant anywhere ReLU is geather that 1. That helps to avoid a common problem in neural networks called gradient vanishing problem\n",
    "\n",
    "$$ \\frac{dReLU(x)}{dx} = I(x>0) $$\n",
    "where $I$ is an [indicator function](https://en.wikipedia.org/wiki/Indicator_function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f20caf8b588>]"
      ]
     },
     "execution_count": 288,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAEPCAYAAAC0r/QVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt0k/X9B/D3k7Rp0vRpQqFIWy61ICI5A12ZgIoTUJke\nceM3FqbnMEQYWoFBKaVULsWWArXUcpHBQVxhOMSCh0484vwNUJgwmAWEE0QGMkDKpRfSpre0SZ7f\nH4z+RKBJ2yd5cnm/ztlZ0+by9kv58O27z/NEkCRJAhERBSSV0gGIiKj9OMSJiAIYhzgRUQDjECci\nCmAc4kREAYxDnIgogHGIExEFMA5xCigTJ06ESqWCSqVCWFgYevTogQkTJqCsrKxNz3PvvfdiyZIl\nd/za+fPnoVKpcODAgdu+9sYbb+C+++5rV3Yib+AQp4Dz+OOP4+rVq7h48SLef/99HD16FGazWdbX\nEAShXV8j8jUOcQo4Go0GsbGxiIuLw2OPPYYpU6bg4MGDqK2tBQA4HA4sWrQISUlJ0Ol0+MlPfoL1\n69e36TV4IjMFijClAxB1RFlZGbZt2wa1Wg21Wg0AmDx5Mo4dO4Z33nkHffr0weHDh/HKK68gPDwc\nEydOVDgxkbw4xCng7N27F6IowuVyoaGhAYIgYPbs2dDpdDh37hw2b96Mb775Bn379gUA9OrVC6dO\nncKqVas4xCnocIhTwBkyZAj+/Oc/o6GhAcXFxfj73/+OnJwcAEBpaSkkScKgQYNuqUQcDgfCw8OV\nikzkNRziFHB0Oh3uvfdeAMCiRYtw5swZTJs2DevXr4fL5YIgCDh48CB0Ot0tj/P0F5IGgwEAUF1d\nfdvXrFYrtFptB/8LiOTDIU4Bb9GiRXjggQfw6quvIjk5GcCNwwSfffbZdj2f0WhE165dcfjwYTzz\nzDO3fO3w4cPo169fhzMTyYVDnAJenz59MHr0aLz++uv49NNPMXHiRPz+979HXl4ehg4dirq6OpSW\nlqK8vBxz5sxpedyVK1fw9ddf3/JcXbp0QUJCAtLT05GdnY34+Hg88cQTaGxsxMaNG3H48GF88cUX\nvv5PJLo7iSiAvPTSS9JTTz112+cPHDggqVQq6YsvvpBcLpeUn58vPfDAA1JERIQUGxsrPfHEE9L2\n7dtb7p+YmCipVKrb/peSkiJJkiS5XC5pw4YNUnJysmQwGKR77rlHeuqpp6QDBw747L+VyBOCJLk/\nIPaTTz7B7t27AQAjR45s94+pREQkL7cn+1y8eBF79uzBsmXLkJ+fjyNHjuDq1atun9hiscgSkLiW\ncuN6yovrKZ/2rKXbIX7p0iX06dMH4eHhUKlUeOCBB3Do0CGvhKE741rKi+spL66nfLwyxHv06IFT\np06htrYWdrsdR48eRWVlZbsCEhGRvNwenZKQkIBf/vKXWLx4MbRaLRITE6FS8ZIrRET+wKNfbP7Q\n+++/j86dO+Ppp5++5fMWi+WWHwXkvqocEVEoKC4ubvnYZDLBZDK1en+PhnhNTQ2io6NRUVGB3Nxc\n5ObmIjIy0m2Ytl7jme5MFEXYbDalYwQNrqe8Qn09Fy2KxjvvRGHr1goMG9bUoeeKj49v82M8Otmn\noKAAtbW1UKvVmDx5skcDnIgo2B06pMGGDXpMmFDX4QHeXm2uU9qCO3F5hPpOR25cT3mF6nrW1Ql4\n6qlYAMD//m859PqOj1Kv7cSJiOhWubnRuHBBjQ8/rJRlgLcXDzMhImqj/fs12LRJj8mT6zB4sDI1\nyk0c4kREbWCzCUhLMyIpyYGMjBql47BOISJqi+zsaFy+rEZJSQV+dMl6RXAnTkTkoT17IrBlix4p\nKbVITm5WOg4ADnEiIo9YrQLS0424//5mpKX5z9E4rFOIiDyQlWVAebkKRUVViIhQOs3/406ciMiN\nzz6LwPbtkZg+vRYDBvhHjXIThzgRUSuqqgTMmWOEydSMGTP8p0a5iXUKEVEr5s0zwmpVYcuWSmg0\nSqe5HXfiRER3sXOnFh99pENqqg39+zuUjnNHHOJERHdQXq5CZqYBAwc2YerUWqXj3BWHOBHRj0gS\nkJlpQH29CitWWBHmx8UzhzgR0Y/s2KHDrl06pKfXoG9f/6xRbuIQJyL6gStXVJg/34BBg5owZUqd\n0nHc4hAnIvovSQLS042w24HCwutQq5VO5J5HTc/HH3+MvXv3QhAE9OzZE6+99hrC/LkkIiJqh+Ji\nHfbs0SI7uxpJSU6l43jE7U68qqoKn376KfLy8rB8+XI4nU58+eWXvshGROQzly6pkJVlwNChdkyc\n6P81yk0e1SkulwuNjY1wOp2w2+3o1KmTt3MREfmMJAGzZxvhdAIFBVaoAqhodtuJxMTE4LnnnsNr\nr72GiIgIDBgwAAMGDPBFNiIin9i8ORL79mmxdKkVvXoFRo1yk9s3Sq6rq0NBQQFmzZqFyMhIFBQU\nYOjQoXjsscduuZ/FYoHFYmm5bTabQ/LNU71Bo9GgqUnZt4AKJlxPeQX6ep47J+CRR/R4+GEnSkoa\nIAjKZRFFEcXFxS23TSYTTCZTq49xuxM/ceIEunbtiqioKADA4MGD8e233942xO/0Yhzi8gjVdxP3\nFq6nvAJ5PV0u4NVXO0OlkpCXV4HaWpeieURRhNlsbtNj3DY/Xbp0wb///W80NTVBkiScOHECCQkJ\n7Q5JROQvior0OHgwAosWVSMhQdkB3l5ud+J9+vTBkCFDkJGRAbVajcTERDz55JO+yEZE5DVnz6qx\nZImIESMaMW5cg9Jx2s1tJ94RZWVl3nrqkBLIP676I66nvAJxPZ1OYMyYLjh7Ngy7d19Dt27+sQuP\nj49v82N4xg4RhZz16/UoLdVg9errfjPA2yuAjoYkIuq406fDkJ8fjV/8ogFjxgRujXIThzgRhQyH\nA0hNNSIy0oVly6oVPZxQLqxTiChkrFkThWPHNFi3rgqxsYFdo9zEnTgRhQSLJQyFhSKef74Bo0c3\nKh1HNhziRBT0mpqA1NROMBpdyM21Kh1HVqxTiCjorVolwmIJx5/+VIWYGK8dVa0I7sSJKKgdPx6O\nVaui8Otf12PUqOCpUW7iECeioGW3AzNnGhEb60J2drXScbyCdQoRBa2CAhHffhuOzZsrYTQGV41y\nE3fiRBSUSkvDsXZtFF54oQ4jRtiVjuM1HOJEFHQaGm6c1BMX50RWVo3ScbyKdQoRBZ28vGicPRuO\nrVsrIIrBWaPcxJ04EQWVQ4c02LBBjwkT6jBsWOC+45CnOMSJKGjU1wtITTWiZ08n5s0L7hrlJtYp\nRBQ0cnOjceGCGtu3V0KvD+4a5Sa3Q7ysrAwrVqyAIAiQJAlXr17FuHHj8Oyzz/oiHxGRR/bv12Dj\nRj0mT67FkCHBX6Pc5HaIx8fH48033wQAuFwupKSk4OGHH/Z6MCIiT9lsAtLSjEhKcmDu3NCoUW5q\nU51y4sQJ3HPPPejSpYu38hARtVl2djQuX1ajpKQCOp3SaXyrTb/YPHDgAB599FFvZSEiarO9eyOw\nZYser75ai+TkZqXj+JzHb5TscDjwyiuvoLCwENHR0bd93WKxwGKxtNw2m80B9+ap/kqj0aCpKXQ6\nPm/jespLyfW0WoEhQ/SIjpawb189tFpFYshGFEUUFxe33DaZTDCZTK0+xuM65dixY0hKSrrjAL/b\ni3GIyyMQ303cn3E95aXkes6aZcTVqwLefbcSzc3NaA7wjbgoijCbzW16jMd1yj/+8Q9WKUTkNz77\nLALbt0di+vRaDBgQ4NO7Azwa4na7HSdOnMDgwYO9nYeIyK2qKgFz5hhhMjVjxozQ/qnKozolIiIC\n7777rrezEBF5ZP58A6xWFf7yl0poNEqnURZPuyeigPLxx1r89a+RmDnTBpPJoXQcxXGIE1HAqKhQ\nITPTgIEDmzBtWq3ScfwChzgRBQRJAubONaCuToUVK6wI45WfAHCIE1GA2LFDh127dEhPr0HfvqxR\nbuIQJyK/d+WKCgsWGDBoUBOmTKlTOo5f4RAnIr8mScCcOUY0NgJvvXUdarXSifwLhzgR+bXiYh12\n79YiM9OG3r2dSsfxOxziROS3Ll1SISvLgKFD7Xj5ZdYod8IhTkR+SZKA2bONcDqBggIrVJxWd8SD\ndIjIL23eHIl9+7RYutSKXr1Yo9wN/20jIr9z4YIaOTnRePzxRowfX690HL/GIU5EfsXlunGJWbUa\nWL7cCkFQOpF/Y51CRH6lqEiPgwcj8NZb15GQ4FI6jt/jTpyI/MbZs2osWSJixIhGmM0NSscJCBzi\nROQXnE4gNbUTtFogP581iqdYpxCRX3jnHT1KSzVYvfo6unVjjeIpj4Z4fX091q1bh4sXL0IQBKSk\npOC+++7zdjYiChGnT4fhzTej8cwzDRgzhjVKW3g0xIuKivDQQw9h1qxZcDqdsNvt3s5FRCHC4QBS\nU43Q611YtqyaNUobue3E6+vrcerUKQwfPhwAoFarERkZ6fVgRBQa1qyJwrFjGixZUo0uXVijtJXb\nnfi1a9cgiiL++Mc/4vz580hKSsLEiROhCfU3tiOiDjt5MgyFhSKef74Bo0c3Kh0nILkd4i6XC+fO\nncOkSZPQu3dvbNy4ESUlJTCbzbfcz2KxwGKxtNw2m80QRVH+xCFIo9FwLWXE9ZRXe9ezqQmYNSsS\nnTpJWLnSyT+T/youLm752GQywWQytXp/t0M8JiYGnTt3Ru/evQEAQ4YMQUlJyW33u9OL2Ww2j0JT\n60RR5FrKiOspr/au5/LlIk6cUKOoqBIajR38I7mxlj/eILvjthM3Go3o3LkzysrKAAAnTpxA9+7d\n25eQiAjA8ePhWLUqCmPH1uPpp3mgREd4dHTKxIkTsXr1ajgcDtxzzz147bXXvJ2LiIKU3Q7MnGlE\nbKwL2dnVSscJeB4N8cTERCxdutTbWYgoBBQUiPj223C8914lDAZJ6TgBj6fdE5HPlJaGY+3aKLz4\nYh2GD2eNIgcOcSLyiYYGYObMToiLc2Lhwhql4wQNXjuFiHwiLy8a330Xhq1bKyCKrFHkwp04EXnd\noUMabNigx4QJdRg2rEnpOEGFQ5yIvKquTkBqqhE9ezoxbx5rFLmxTiEir8rNjcaFC2p8+GEl9HrW\nKHLjTpyIvGb/fg02bdJj8uQ6DB7MGsUbOMSJyCtsNgFpaUYkJTmQkcEaxVtYpxCRV2RnR+PyZTVK\nSiqg0ymdJnhxJ05EstuzJwJbtuiRklKL5ORmpeMENQ5xIpKV1SogPd2I++9vRloaL03obaxTiEhW\nWVkGlJerUFRUhYgIpdMEP+7EiUg2n30Wge3bIzF9ei0GDGCN4gsc4kQki6oqAXPmGGEyNWPGDNYo\nvsI6hYhkMW+eEVarClu2VIJvwes7Hg3xqVOnIjIyEoIgQK1W89riRHSLnTu1+OgjHebMqUH//g6l\n44QUj4a4IAjIyspCVFSUt/MQUYC5dk1AZqYBAwc2YerUWqXjhByPOnFJkiBJvOYBEd1KkoDU1AjU\n16uwYoUVYSxofc7jnfjixYuhUqkwcuRIPPnkk97ORUQBYMcOHXbuDMf8+dXo25c1ihI8GuI5OTno\n1KkTampqkJOTg+7du6Nfv37ezkZEfuzKFRUWLDBg8GAnpkypUzpOyBKkNvYk27Ztg06nw3PPPXfL\n5y0WCywWS8tts9kMm42HGclBo9GgqYlXgJML17PjJAn4zW902L9fjcOHm9GrF98vUw6iKKK4uLjl\ntslkgslkavUxbnfidrsdkiRBq9WisbERx48fx9ixY2+7351ejENcHqIoci1lxPXsuA8+0OGzz8KQ\nnV2NXr1UXE+ZiKIIs9ncpse4HeLV1dXIz8+HIAhwOp0YNmwYBg4c2O6QRBTYLl1SISvLgKFD7Zg4\nsQ6AqHSkkOZ2iHft2hX5+fm+yEJEfk6SgNmzjXA6gYICK1Q851txPCCIiDy2eXMk9u3TYulSK3r1\nciodh8BrpxCRh86fVyMnJxqPP96I8ePrlY5D/8UhTkRuuVxAWpoRajWwfLkVgqB0IrqJdQoRuVVU\npMfBgxEoKLiOhASX0nHoB7gTJ6JWnT2rxpIlIkaMaMS4cQ1Kx6Ef4RAnortyOoHU1E7QaoH8fNYo\n/oh1ChHd1fr1epSWarB69XV068YaxR9xJ05Ed3T6dBjy86PxzDMNGDOGNYq/4hAnots4HMDMmUbo\n9S4sW1bNGsWPsU4hotusWROFr7/WYN26KnTpwhrFn3EnTkS3sFjCUFgo4vnnGzB6dKPSccgNDnEi\natHUBMyc2QlGowu5uVal45AHWKcQUYuVK0WcPBmOoqJKxMTwLRkDAXfiRAQA+PrrcKxeHYWxY+vx\n9NN8k4dAwSFORLDbbxyNEhvrwhtvVCsdh9qAdQoRoaBAxOnT4XjvvUoYjaxRAonHO3GXy4WMjAzk\n5eV5Mw8R+VhpaTjWro3Ciy/WYfhw1iiBxuMh/sknnyAhIcGbWYjIxxoaBMyc2QlxcU4sXFijdBxq\nB4+GeGVlJY4ePYqRI0d6Ow8R+dCyZSK++y4MBQVWiCJrlEDk0RDftGkTxo8fD4Hn3hIFjX/+U4N3\n39VjwoQ6DBvWpHQcaie3Q/zIkSMwGAxITEyEJEmQJP5rTRTo6uoEzJplRM+eTsybxxolkLk9OuXU\nqVP46quvcPToUTQ1NaGhoQFvv/02pk2bdsv9LBYLLBZLy22z2QxRFOVPHII0Gg3XUkZcTyArKwIX\nLqixa1cDunWL6tBzcT3lVVxc3PKxyWSCyWRq9f6C1Iat9cmTJ7Fz505kZGR4dP+ysjJPn5paIYoi\nbDab0jGCRqiv5/79Gvz2t13w+9/XYtGiju/CQ3095RQfH9/mx/BkH6IQYrMJSEszIinJgYwM1ijB\noE0n+/Tv3x/9+/f3VhYi8rLs7GhcvqxGSUkFdDql05AcuBMnChF79kRgyxY9UlJqkZzcrHQckgmH\nOFEIsFoFpKcbcf/9zUhLY38dTHjtFKIQsHChAeXlKhQVVSEiQuk0JCfuxImC3N/+psWHH0biD3+o\nxYABrFGCDYc4URCrqhKQkWGAydSMP/yBNUowYp1CFMTmzTPCalVhy5ZKaDRKpyFv4E6cKEjt3KnF\nRx/pkJpqQ//+DqXjkJdwiBMFofJyFTIzDRg4sAlTp9YqHYe8iEOcKMhIEpCZaUB9vQorVlgRxtI0\nqHGIEwWZHTt02LVLh/T0GvTtyxol2HGIEwWRK1dUmD/fgEGDmjBlSp3SccgHOMSJgoQkAenpRtjt\nQGHhdajVSiciX+AQJwoSxcU67Nmjxeuv25CU5FQ6DvkIhzhRELh0SYWsLAOGDrVj4kTWKKGEQ5wo\nwEkSMHu2EU4nUFBghYp/q0MKDz4iCnCbN0di3z4tli61olcv1iihxu0Qb25uRlZWFhwOB5xOJ4YM\nGYLf/OY3vshGRG6cP69GTk40hg2zY/z4eqXjkALcDvHw8HBkZWUhIiICLpcLCxYswEMPPYQ+ffr4\nIh8R3YXLBaSlGaFW36hRBEHpRKQEj+qUiP9egLi5uRlOJ39cI/IHRUV6HDwYgYKC60hI4N/LUOXR\nEHe5XJg7dy6uXr2KUaNGcRdOpLDvvlNjyRIRI0Y0Yty4BqXjkIIESZIkT+9cX1+P/Px8TJo0Cd27\nd7/laxaLBRaLpeW22WyGzcbrF8tBo9GgqalJ6RhBI9DX0+kERo2KxOnTKhw6VIe4OI//CntFoK+n\nPxFFEcXFxS23TSYTTCZTq49p0xAHgO3bt0Or1eK5555ze9+ysrK2PDXdhSiK/AdRRoG+nmvX6rF4\nsQGrV1/H//yP8rvwQF9PfxIfH9/mx7g9orSmpgb19Td+693U1IQTJ06064WIqONOnw5Dfn40fvGL\nBowZo/wAJ+W57cStVivWrFkDl8sFSZLwyCOP4Kc//akvshHRDzgcQGqqEZGRLixbVs2jUQiAB0O8\nZ8+eyMvL80UWImrFmjVROHZMg3XrqhAb61I6DvkJnqBLFAAsljAUFop4/vkGjB7dqHQc8iMc4kR+\nrqkJmDmzEwwGF3Jzq5WOQ36G104h8nOrVok4eTIcf/pTFWJiWKPQrbgTJ/Jjx4+HY9WqKPz61/UY\nNYo1Ct2OQ5zIT9ntwMyZRsTGupCdzRqF7ox1CpGfKigQ8e234di8uRJGo7JnZZL/4k6cyA+VloZj\n7doovPBCHUaMsCsdh/wYhziRn2louHE0SlycE1lZNUrHIT/HOoXIz+TlReO778KwdWsFRJE1CrWO\nO3EiP3LokAYbNugxYUIdhg3jlQHJPQ5xIj9RVycgNdWInj2dmDePNQp5hnUKkZ/IzY3GhQtqbN9e\nCb2eNQp5hjtxIj+wf78GmzbpMWlSHYYMYY1CnuMQJ1KYzSYgLc2IpCQH5s5ljUJtwzqFSGHZ2dG4\nfFmNkpIK6HRKp6FAw504kYL27InAli16vPpqLZKTm5WOQwHI7U68srISb7/9NqqrqyEIAkaOHIln\nn33WF9mIgprVKiA93Yi+fZuRlsb3qKT2cTvE1Wo1JkyYgMTERDQ2NiIjIwMDBw5EQkKCL/IRBa2F\nCw0oL1ehqKgKWq3SaShQua1TjEYjEhMTAQBarRYJCQmoqqrydi6ioPa3v2nx4YeRmD69FgMGsEah\n9mtTJ37t2jWcP38e9913n7fyEAW9qioBGRkG9O/fjBkzWKNQx3h8dEpjYyPeeustvPTSS9De4Wc/\ni8UCi8XScttsNkMURXlShjiNRsO1lJHS6zl9uhZWqwolJY3o3Dnw/1yVXs9gU1xc3PKxyWSCyWRq\n9f6CJEluTw1zOp1YtmwZHnrooTb9UrOsrMzj+9LdiaIIm407NrkouZ47d2rx6qsxSE+vwcyZtYpk\nkBu/P+UTHx/f5sd4VKesXbsW3bt351EpRB1QXq5CZqYBAwc2Ydq04BjgpDy3dcqpU6ewf/9+9OzZ\nE3PmzIEgCHjhhRfw4IMP+iIfUVCQJCAz04D6ehVWrKhEGE+zI5m4/Vbq168fPvjgA19kIQpaO3bo\nsGuXDvPnV6NvX4fScSiI8IxNIi+7ckWF+fMNSE5uwpQpdUrHoSDDIU7kRZIEpKcbYbcDhYXXoVYr\nnYiCDYc4kRd98IEOe/ZokZlpQ+/eTqXjUBDiECfykkuX1Fi0yIChQ+14+WXWKOQdHOJEXiBJwOzZ\nBjidQEGBFSr+TSMv4YFORF6weXMk9u3TYulSK3r1Yo1C3sP9AZHMzp9XIycnGsOG2TF+fL3ScSjI\ncYgTycjlAtLSjFCrb9QogqB0Igp2rFOIZFRUpMfBgxEoKLiOhATWKOR93IkTyeTsWTWWLBExYkQj\nxo1rUDoOhQgOcSIZOJ1AamonaLVAfj5rFPId1ilEMli/Xo/SUg1Wr76Obt1cSsehEMKdOFEHnT4d\nhvz8aDzzTAPGjGGNQr7FIU7UAQ4HkJpqRGSkC0uXVrNGIZ9jnULUAWvWROHYMQ3WratCbCxrFPI9\n7sSJ2sliCUNhoYjnn2/A6NGNSsehEOV2J7527VocOXIEBoMBy5cv90UmIr/X1HTjaBSj0YXcXKvS\ncSiEud2JDx8+HPPmzfNFFqKAsWqVCIslHG++aUVMjNv3GifyGrdDvF+/ftDr9b7IQhQQjh8Px6pV\nURg7th5PP21XOg6FOHbiRG1gtwMzZxoRG+vCG29UKx2HSL6jUywWCywWS8tts9kMURTlevqQptFo\nuJYy6sh6Ll+uwbffhmP79nr06BElc7LAxO9PeRUXF7d8bDKZYDKZWr2/bEP8Ti9ms9nkevqQJooi\n11JG7V3P0tJwrFwZhRdfrMPQodXgH8kN/P6UjyiKMJvNbXqMR3WKJEmQJP7yhkJXQ8ONk3ri4pxY\nuLBG6ThELdzuxFeuXImTJ0/CZrMhJSUFZrMZw4cP90U2Ir+RlxeNs2fDsXVrBUSRGxryH26H+IwZ\nM3yRg8hvHTqkwYYNekyYUIdhw5qUjkN0Cx6dQtSK+noBqalG9OzpxLx5rFHI//DaKUStyM2NxoUL\nanz4YSX0etYo5H+4Eye6i/37Ndi4UY/Jk+sweDBrFPJPHOJEd2CzCUhLMyIpyYGMDNYo5L9YpxDd\nQXZ2NC5fVqOkpAI6ndJpiO6OO3GiH9m7NwJbtuiRklKL5ORmpeMQtYpDnOgHqqsFzJ5txP33NyMt\njWchkv9jnUL0AwsXGlBerkJRURUiIpROQ+Qed+JE//XZZxHYvj0S06fXYsAA1igUGDjEiQBUVQmY\nM8cIk6kZM2awRqHAwTqFCMD8+QZYrSps2VIJjUbpNESe406cQt7HH2vx179GIjXVhv79HUrHIWoT\nDnEKaRUVKmRmGjBwYBOmTq1VOg5Rm3GIU8iSJGDuXAPq6lRYscKKMJaLFIA4xClk7dihw65dOqSn\n16BvX9YoFJg82nscO3YMGzduhCRJGD58OH71q195OxeRV125osKCBQYkJzdhypQ6peMQtZvbnbjL\n5cK7776LefPmoaCgAF9++SUuXbrki2xEXiFJwJw5RjQ2AoWF16FWK52IqP3cDvEzZ84gLi4OsbGx\nCAsLw6OPPop//etfvshG5BV/+UsYdu/WIjPTht69nUrHIeoQt0O8qqoKnTt3brkdExODqqoqr4Yi\n8pZLl1SYO1eLoUPtePll1igU+Lz6+/i+fbt58+lDhiAIkCS90jGCQnOzgPBwoKDAChV/rU9BwO0Q\nj4mJQUVFRcvtqqoqxMTE3HY/i8UCi8XScttsNqO2ln9L5CMoHSDI3KN0gKAiiqLSEYJGcXFxy8cm\nkwkmk6n1B0huOJ1Oadq0adK1a9ek5uZmafbs2dLFixfdPUz64IMP3N6HPMO1lBfXU15cT/m0Zy3d\n7sRVKhUmTZqExYsXQ5IkjBgxAt27d+/wvzZERNRxHnXiDz74IFauXOntLERE1EZeK63d9jjkMa6l\nvLie8uJ6yqc9aylIkiR5IQsREfkADx8hIgpgHOJERAFM1pN9/vnPf2Lbtm34/vvvsXTpUiQlJbV8\nbceOHdi7dy/UajVeeuklDBw4UM6XDnrbtm3D7t27YTAYAAAvvPACHnzwQYVTBRZeyE1eU6dORWRk\nJARBgFpJOtFQAAADDklEQVStxtKlS5WOFFDWrl2LI0eOwGAwYPny5QCA2tparFixAuXl5ejatStS\nU1MRGRnZ+hPJeYzjpUuXpLKyMmnRokXS2bNnWz5/8eJFKT09XXI4HNLVq1eladOmSS6XS86XDnrF\nxcXSzp07lY4RsO50vsP333+vdKyANnXqVMlmsykdI2B988030rlz56S0tLSWz23evFkqKSmRJEmS\nduzYIb333ntun0fWOiU+Ph5xcXG3ff6rr77CI488ArVaja5duyIuLg5nzpyR86VDgsTfQbcbL+Qm\nP0mS+D3ZAf369YNef+vlNL766iv8/Oc/BwA88cQTHn2P+uS9TKqqqtC3b9+W27yIVvt8+umn2Ldv\nH3r37o3f/e537n/MohZ3upAbNxIdIwgCFi9eDJVKhZEjR+LJJ59UOlLAq66uhtFoBAAYjUZUV1e7\nfUybh3hOTs4tTyxJEgRBwG9/+1sMGjSorU9HP9Da2o4aNQpjx46FIAjYunUrNm3ahJSUFAXTUqjL\nyclBp06dUFNTg5ycHHTv3h39+vVTOlZQEQT310xq8xBfsGBBm4P8+CJalZWVd7yIVqjzdG1HjhyJ\nvLw8L6cJLp5eyI0816lTJwBAdHQ0Hn74YZw5c4ZDvIOMRiOsVmvL/988kKE1PjnEcNCgQThw4AAc\nDgeuXbuGK1euoE+fPr546aBhtVpbPj506BB69OihYJrA06dPH1y5cgXl5eVwOBz48ssv+ZNjB9jt\ndjQ2NgIAGhsbcfz4cX5PtsOPf6+QnJyMzz//HADw+eefe/Q9KusZm4cPH0ZRURFqamqg1+uRmJiI\n119/HcCNQwz37NmDsLAwHmLYDm+//Tb+85//QBAExMbGYsqUKS3dGXnm2LFjKCoqarmQGw8xbL9r\n164hPz8fgiDA6XRi2LBhXM82WrlyJU6ePAmbzQaDwQCz2Yyf/exnKCwsREVFBWJjY5GamnrbLz9/\njKfdExEFMJ6xSUQUwDjEiYgCGIc4EVEA4xAnIgpgHOJERAGMQ5yIKIBxiBMRBTAOcSKiAPZ/hpdv\nv14bAGQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20cb22bd30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def relu(val):\n",
    "    return np.maximum(val, 0)\n",
    "\n",
    "x = np.arange(-10, 10)\n",
    "y = relu(x)\n",
    "\n",
    "plt.style.use('ggplot')\n",
    "plt.title(\"ReLU\")\n",
    "plt.plot(x, y, linewidth=1.5, color='blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layers\n",
    "Each neural network consists of groups of neurons called layers. Layers are connected between each other with weights.\n",
    "\n",
    "Multi Layer Perceptron consists of an input layer $I$, any number of hidden layers $H_i$ and an output layer $O$. All variables I mentioned up there are matrices and their size is depicted below each matrix name. Hidden layers also are attached to a special function $f$ called activation function.\n",
    "\n",
    "The forward pass to the network goes like this:\n",
    "$$f_o(f_h(I \\times H) \\times O)$$\n",
    "where $f_h$ is a hidden layer activation function applied to each neuron inside and $f_o$ is an output layer activation functioon.\n",
    "\n",
    "### Backpropogation and stochastic gradient descent (SGD)\n",
    "For now for the explanation of those concepts I will refer to the tutorial links I've left before.\n",
    "\n",
    "## MLP Implementation\n",
    "Below goes simplistic implementation of Multi Layer Perceptron which is trained using SGD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 527,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1   2]\n",
      " [  3   4]\n",
      " [  5   6]\n",
      " [  7   8]\n",
      " [  9  10]\n",
      " [ 11  12]\n",
      " [ 13  14]\n",
      " [ 15  16]\n",
      " [ 17  18]\n",
      " [ 19  20]\n",
      " [ 21  22]\n",
      " [ 23  24]\n",
      " [ 25  26]\n",
      " [ 27  28]\n",
      " [ 29  30]\n",
      " [ 31  32]\n",
      " [ 33  34]\n",
      " [ 35  36]\n",
      " [ 37  38]\n",
      " [ 39  40]\n",
      " [ 41  42]\n",
      " [ 43  44]\n",
      " [ 45  46]\n",
      " [ 47  48]\n",
      " [ 49  50]\n",
      " [ 51  52]\n",
      " [ 53  54]\n",
      " [ 55  56]\n",
      " [ 57  58]\n",
      " [ 59  60]\n",
      " [ 61  62]\n",
      " [ 63  64]\n",
      " [ 65  66]\n",
      " [ 67  68]\n",
      " [ 69  70]\n",
      " [ 71  72]\n",
      " [ 73  74]\n",
      " [ 75  76]\n",
      " [ 77  78]\n",
      " [ 79  80]\n",
      " [ 81  82]\n",
      " [ 83  84]\n",
      " [ 85  86]\n",
      " [ 87  88]\n",
      " [ 89  90]\n",
      " [ 91  92]\n",
      " [ 93  94]\n",
      " [ 95  96]\n",
      " [ 97  98]\n",
      " [ 99 100]]\n",
      "[[-0.02116747]\n",
      " [ 0.5330343 ]\n",
      " [ 1.00442367]\n",
      " [ 0.39888346]\n",
      " [-0.02040472]\n",
      " [ 0.7365517 ]\n",
      " [ 1.04541355]\n",
      " [ 0.13780551]\n",
      " [ 0.20741879]\n",
      " [ 0.93780908]\n",
      " [ 0.70351133]\n",
      " [-0.00249494]\n",
      " [ 0.46394943]\n",
      " [ 0.98345534]\n",
      " [ 0.42731812]\n",
      " [ 0.00975616]\n",
      " [ 0.66355492]\n",
      " [ 0.9617698 ]\n",
      " [ 0.18474016]\n",
      " [ 0.16200475]\n",
      " [ 0.95006402]\n",
      " [ 0.73679223]\n",
      " [ 0.02572038]\n",
      " [ 0.44562919]\n",
      " [ 0.99991868]\n",
      " [ 0.35509903]\n",
      " [ 0.04720004]\n",
      " [ 0.7442825 ]\n",
      " [ 0.84450407]\n",
      " [ 0.08299746]\n",
      " [ 0.19274719]\n",
      " [ 0.8948872 ]\n",
      " [ 0.59889561]\n",
      " [ 0.0681247 ]\n",
      " [ 0.39999143]\n",
      " [ 0.97651522]\n",
      " [ 0.32914158]\n",
      " [-0.01171004]\n",
      " [ 0.8325338 ]\n",
      " [ 0.95214759]\n",
      " [ 0.18167159]\n",
      " [ 0.29791868]\n",
      " [ 0.86907727]\n",
      " [ 0.66559844]\n",
      " [ 0.03026829]\n",
      " [ 0.49876363]\n",
      " [ 0.93886304]\n",
      " [ 0.44643203]\n",
      " [ 0.00927138]\n",
      " [ 0.75530891]]\n",
      "Out delta - [[ 4.74886357]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.58621009]]\n",
      "Total error: [[ 11.2758526]]\n",
      "Prediction: [[ 4.95628236]]\n",
      "Out delta - [[-143.84047785]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 2650.18189182]]\n",
      "Total error: [[ 10345.0415335]]\n",
      "Prediction: [[-143.24158224]]\n",
      "Out delta - [[ 14.87889314]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 110.69073057]]\n",
      "Prediction: [[ 14.90916143]]\n",
      "Out delta - [[ 12.95732268]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 83.94610558]]\n",
      "Prediction: [[ 13.42127211]]\n",
      "Out delta - [[ 11.17547582]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 62.44562993]]\n",
      "Prediction: [[ 12.12553984]]\n",
      "Out delta - [[ 10.54404283]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 55.58841964]]\n",
      "Prediction: [[ 11.00799226]]\n",
      "Out delta - [[ 9.12105418]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 41.59681466]]\n",
      "Prediction: [[ 9.95358798]]\n",
      "Out delta - [[ 8.07971276]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 32.64087915]]\n",
      "Prediction: [[ 9.04148256]]\n",
      "Out delta - [[ 7.78788209]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 30.32555376]]\n",
      "Prediction: [[ 8.23351129]]\n",
      "Out delta - [[ 7.05583962]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 24.89243636]]\n",
      "Prediction: [[ 7.45472308]]\n",
      "Out delta - [[ 6.25037548]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 19.53359682]]\n",
      "Prediction: [[ 6.74913911]]\n",
      "Out delta - [[ 6.1143454]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 18.69260986]]\n",
      "Prediction: [[ 6.12410157]]\n",
      "Out delta - [[ 4.53615181]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 10.28833662]]\n",
      "Prediction: [[ 5.51266703]]\n",
      "Out delta - [[ 4.97605438]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 12.38055862]]\n",
      "Prediction: [[ 5.05905184]]\n",
      "Out delta - [[ 4.4236409]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 9.78429939]]\n",
      "Prediction: [[ 4.56144641]]\n",
      "Out delta - [[ 3.27457825]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 5.36143135]]\n",
      "Prediction: [[ 4.11908232]]\n",
      "Out delta - [[ 2.79170581]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 3.89681066]]\n",
      "Prediction: [[ 3.79162449]]\n",
      "Out delta - [[ 3.53362138]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 6.24324005]]\n",
      "Prediction: [[ 3.51245391]]\n",
      "Out delta - [[ 2.49553685]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 3.1138521]]\n",
      "Prediction: [[ 3.15909177]]\n",
      "Out delta - [[ 2.7278665]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 3.72062781]]\n",
      "Prediction: [[ 2.90953809]]\n",
      "Out delta - [[ 1.90019974]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 1.80537953]]\n",
      "Prediction: [[ 2.63675144]]\n",
      "Out delta - [[ 1.84783586]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 1.70724868]]\n",
      "Prediction: [[ 2.44673146]]\n",
      "Out delta - [[ 1.51766537]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 1.1516541]]\n",
      "Prediction: [[ 2.26194788]]\n",
      "Out delta - [[ 2.04205664]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 2.08499767]]\n",
      "Prediction: [[ 2.11018134]]\n",
      "Out delta - [[ 1.9263804]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 1.85547071]]\n",
      "Prediction: [[ 1.90597568]]\n",
      "Out delta - [[ 0.66792408]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.22306129]]\n",
      "Prediction: [[ 1.71333764]]\n",
      "Out delta - [[ 0.70873615]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.25115347]]\n",
      "Prediction: [[ 1.64654523]]\n",
      "Out delta - [[ 1.41366686]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.99922699]]\n",
      "Prediction: [[ 1.57567161]]\n",
      "Out delta - [[ 1.44601497]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 1.04547965]]\n",
      "Prediction: [[ 1.43430493]]\n",
      "Out delta - [[ 0.9346044]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.43674269]]\n",
      "Prediction: [[ 1.28970343]]\n",
      "Out delta - [[ 0.79735953]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.31789111]]\n",
      "Prediction: [[ 1.19624299]]\n",
      "Out delta - [[ 0.16435944]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.01350701]]\n",
      "Prediction: [[ 1.11650704]]\n",
      "Out delta - [[ 0.20518389]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.02105021]]\n",
      "Prediction: [[ 1.10007109]]\n",
      "Out delta - [[ 0.63392351]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.20092951]]\n",
      "Prediction: [[ 1.0795527]]\n",
      "Out delta - [[ 0.96896031]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.46944204]]\n",
      "Prediction: [[ 1.01616035]]\n",
      "Out delta - [[-0.08515935]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.00362606]]\n",
      "Prediction: [[ 0.91926432]]\n",
      "Out delta - [[ 0.22426892]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.02514828]]\n",
      "Prediction: [[ 0.92778026]]\n",
      "Out delta - [[ 0.20184203]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.0203701]]\n",
      "Prediction: [[ 0.90535336]]\n",
      "Out delta - [[ 0.8021717]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.32173972]]\n",
      "Prediction: [[ 0.88516916]]\n",
      "Out delta - [[ 0.81666203]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.33346844]]\n",
      "Prediction: [[ 0.80495199]]\n",
      "Out delta - [[ 0.05973087]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.00178389]]\n",
      "Prediction: [[ 0.72328579]]\n",
      "Out delta - [[ 0.53564111]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.1434557]]\n",
      "Prediction: [[ 0.7173127]]\n",
      "Out delta - [[ 0.26375716]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.03478392]]\n",
      "Prediction: [[ 0.66374859]]\n",
      "Out delta - [[ 0.61165249]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.18705939]]\n",
      "Prediction: [[ 0.63737287]]\n",
      "Out delta - [[-0.16034407]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.01285511]]\n",
      "Prediction: [[ 0.57620762]]\n",
      "Out delta - [[-0.30264517]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.04579705]]\n",
      "Prediction: [[ 0.59224203]]\n",
      "Out delta - [[ 0.46050179]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.10603095]]\n",
      "Prediction: [[ 0.62250655]]\n",
      "Out delta - [[ 0.52925632]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.14005613]]\n",
      "Prediction: [[ 0.57645637]]\n",
      "Out delta - [[-0.30900306]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.04774145]]\n",
      "Prediction: [[ 0.52353074]]\n",
      "Out delta - [[ 0.05566741]]\n",
      "((1, 2) x (1, 1)) x (2, 1)\n",
      "Hidden delta - [[ 0.]]\n",
      "Total error: [[ 0.00154943]]\n",
      "Prediction: [[ 0.55443104]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f20c9030320>]"
      ]
     },
     "execution_count": 527,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEPCAYAAACzwehFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtYVOeBP/DvmRkuDo6Mg5CAaFTQWMcbEVq1CaDGdjXp\n6i+2PGn6aLD6s2rSJJO4m5DrmprH2KiIisYtTbAb93mW3QZqstls8otMTJS0UCU1oyYiasQrMIVA\nBGaYeX9/DJxxuM016hy+n+eZwJw5Z+Y9L8g373kvRxJCCBAREQ1AdbMLQEREtz6GBRERecWwICIi\nrxgWRETkFcOCiIi8YlgQEZFXDAv6TuzYsQOjRo2CRqPByy+/fLOLE5A5c+Zg1apVN7sYWL58OX70\nox/5dczevXsRGRn5HZXo1qVSqfDv//7vN7sYisSwUIjc3FyoVCr89Kc/7fXa/v37oVKpev3x2L9/\nP+655x7ExcVh6NChGD9+PJYuXYrW1lYAwLlz56BSqXo91Go1tm7d2m9ZLl26BJPJhOeeew4XL17E\nunXrQnuyt7iIiAj84Q9/CNn7bd++Hf/5n//p1zEPPvggLly4ELIyEGludgEoNCRJwujRo/Huu++i\nvr4e8fHx8mt79uzBmDFjUFdXJ287cOAAlixZgpdeegm/+93vEBUVhZqaGpSVlaGjowNDhw6V33f/\n/v3IyMjw+Lxhw4b1W5bTp09DCIGf/OQnSEhICPic7HY7IiIiAj7+VtfZ2QmNxvs/QZ1O5/d7R0VF\nefwOEAWLLQsFmTBhAmbOnIni4mJ52/nz5/Hhhx9i+fLlHvu+8847SEtLw/PPP4+JEydi7NixmD9/\nPgoLCxEXFyfvJ4TA8OHDkZCQ4PGIjo7uswzr169HZmYmAGDUqFFQq9X4+uuvAbgujRiNRkRFRWHU\nqFF44YUX4HA45GPnzJmDlStX4sUXX0RSUhLuuOOOfs/19OnT+OlPf4rhw4fDYDDgxz/+Mb744gv5\n9aamJixduhR33HEHtFotJk6c2Gdr6D/+4z+Qnp6OIUOGYMSIEbjvvvvQ3Nzssc+GDRuQmJiIuLg4\nPPzww7h27Vq/5Ro7diycTieWL18ut8IAoLi4GBERETCbzbjrrrsQHR2Njz76CGfPnsWSJUswcuRI\nxMTEYOrUqXjrrbc83rPnZajly5dj/vz5+N3vfocxY8YgNjYWixYtQn19vbxP9+d127t3LyIiInD4\n8GHMmDEDMTExSE9PR1VVlcdnffTRR5g6dSqGDBmCu+66C59++qlPl3Y+/PBD3H333dBqtUhOTsYv\nf/lLWK3WXmXetm0bkpOTERMTg5ycHPz973/3eJ/NmzcjJSUFUVFRSE1NRUFBgcfrDocD69evR2pq\nKqKjozFq1Cg8/vjjHvs0Nzdj2bJlGDZsGEaNGoVXX311wLKTjwQpQm5urpg/f77Yt2+fGD9+vLz9\nxRdfFAsWLBDFxcUiIiJC3r5p0yYxfPhw8Ze//KXf9zx79qyQJEkcOnTI53J8++234u233xYqlUp8\n/vnn4sqVK8LpdIp3331XqNVqsWnTJnHq1ClRUlIihg8fLl588UX52OzsbDFs2DCxZs0aceLECfHF\nF1/0+RlXrlwRt99+u3jkkUeExWIRX331lXjsscfEiBEjRENDgxBCiMuXL4tNmzaJ6upqcfbsWbFv\n3z6h0+lEcXGx/D5vvPGGiIiIEK+88oo4ceKEsFgsYufOnaKxsVEuj16vF08++aT48ssvxYcffigM\nBoNHmXuqr68XGo1G7NixQ1y5ckVcuXJFCCFEcXGxUKlU4gc/+IEwm83izJkzoqGhQRw7dkwUFhaK\nY8eOidraWrFz504REREhzGaz/J7dP9vrn8fGxoqHHnpIWCwW8dlnn4mxY8eKZcuWyfv0/Hl3f35W\nVpY4dOiQ+PLLL8WCBQvEuHHjhMPhEEIIceHCBaHVasWqVavEiRMnxIEDB8SMGTOESqUS+/bt6/ec\nP/roI6HVakVhYaE4ffq0qKqqEnPnzhVZWVkeZR42bJhYtGiRsFgs4uOPPxbjx48XDzzwgLzPzp07\nhVarFUVFRaKmpkbs2bNHREdHizfeeEPeZ9myZeK2224T+/btE7W1taKqqkoUFBTIr0uSJG6//XZR\nVFQkamtrRWFhoZAkSRw4cKDf8pNvGBYK0f0Hpb29XcTFxQmz2SwcDodITk4WpaWlvf54XLt2TSxa\ntEioVCqRmJgoFi1aJAoKCuQ/lEK4wyImJkYMHTpUfuh0OvHZZ5/1Wxaz2SxUKpW4cOGCvO2ee+4R\nDz74oMd+BQUFQqvVCrvdLoRw/XG+8847vZ7rv/zLv4hZs2Z5bHM6nSIlJcXjD0dPjz/+uPjRj34k\nPx89erR47LHH+t0/OztbTJ8+3WPbmjVrxOzZswcsn0ajEXv37vXY1v3H2pfgXbRokVi1apX8vK+w\nuO222+R6E8IV/klJSR6f11dYVFdXy9v+/Oc/C5VKJb766ishhBDPPvusGDt2rHA6nfI+77//vpAk\nacCwyM7OFnl5eR7bzp07JyRJEp9//rlcZp1OJ1paWuR9PvjgAyFJkjh9+rQQQohRo0aJZ555xuN9\nTCaTSElJEUIIcerUKSFJknj77bf7LYskSeKJJ57w2Pa9731PPPvss/0eQ77hZSiFiYqKwtKlS/Gv\n//qv+O///m84HA785Cc/6bXfkCFDUFZWhjNnzuDVV19FcnIyNm7ciDvvvBNffvmlx77FxcX4/PPP\n5Ud1dTXS0tL8KpfFYsE999zjsS0rKwvt7e04ffq0vG3GjBle36uyshJVVVXQ6XTyY9iwYTh37hxO\nnToFwHX57NVXX0VaWhri4+Oh0+nw+uuv49y5cwCA+vp6nD9/HvPnzx/ws6ZNm+bxPCkpCVeuXPHp\nnPuSnp7u8bytrQ3PPPMMJk+ejLi4OOh0OvzP//yPXM7+TJw40aO/w5dySZKEqVOnehwjhJCPO3Hi\nBDIyMiBJkrzPrFmzvJ5TZWUltm3b5vHzMBqNkCRJ/nkAwKRJk+S+MAD44Q9/CAA4fvw4WlpaUFdX\n1+fvyNmzZ9He3o4jR45AkqQb/jMjF3ZwK9CqVatw11134fz581i+fLl83bwvo0ePxrJly7Bs2TK8\n8sorGD9+PH7729/i97//vbxPUlISxo0b952UVfRY9DgmJsbrMU6nE/feey8KCwt7HR8bGwvAde17\n06ZN2LZtG6ZPnw6dToetW7fivffe86t8PUeQSZIEp9Pp13t0U6vVvd5v3bp1eOedd5Cfn48JEyYg\nJiYGTz75JL755hu/y9WzLnpSqVQeQdD9/fXnc/3rvnI6nXj66aexdOnSXq/dfvvtfr9ff3wtWyh/\nZuTGloUCfe9730NGRgYOHz6MlStX+nxcbGwsbr/9dly9ejXkZTIajTh48KDHNrPZDK1Wi5SUFL/e\nKz09HRaLBSNHjsS4ceM8Ht2d85988gn+4R/+AQ8//DCmTZuGcePG4auvvpLfIz4+HsnJyfjggw+C\nP7keIiMjPTruB/LJJ5/gF7/4BZYsWYIpU6Zg7NixHuW8kSZNmoTKykqP0KmoqPB6XPfPo+fPYty4\ncdBqtfJ+J06ckIdlA8ChQ4cgSRKMRiN0Oh2Sk5P7/B0ZO3YsoqOjcdddd0EI8Z38zMg7hoVCffDB\nB2hoaMDYsWP7fH39+vX4p3/6J5jNZpw9exZffPEF1q1bB4vFggceeMBjX6vViitXrng8rv9H35ee\n/5ebl5eHP/7xj9i0aRNOnTqFkpISrF+/HuvWrfNp+Oj1Hn30UTgcDvzjP/4jPv30U5w7dw6ffvop\nnn/+eXz22WcAgDvvvBNmsxlmsxmnTp3CCy+8gL/85S8e7/PSSy9hz5492LBhA06ePAmLxYLCwkKP\nUTyBGDt2LMrLy3Hp0iU0NjYOuO+dd96JP/3pT6isrMTx48exatUqXLx4MajPD9TatWtx5coVrF69\nGidPnkR5eTmef/55SJI04P/Vv/zyy/jTn/6Ep556Cp9//jlqa2vx/vvvY+XKlejo6JD3kyQJy5Yt\ng8ViwcGDB/Hoo49i0aJF8u9oXl4eduzYgaKiItTU1GDPnj3Ys2cPnnvuOQBASkoKfvGLX2Dt2rXY\nt28famtrUVlZie3bt3+3FUMAGBaKFR0dDb1e3+/rWVlZqKurwy9/+UtMmjQJc+fOxZ///Gfs27fP\nY5itJElYtGgRkpKSPB7PPvvsgJ/f84/LggUL8MYbb+APf/gDpkyZgqeeegqPPvooXnzxxX6P6U9C\nQgIqKioQHx+PJUuWYOLEiVi6dCm+/vprJCYmAgBeeOEFZGVlYfHixZg9ezaampp6DbFcsWIFiouL\n8cc//hFpaWnIzs7G+++/L4dXIJdkAGDLli3461//ijFjxnidZ5Kfn4877rgDc+fOxfz585GcnIyf\n/exnAX1uIK4/x6SkJOzfvx8VFRVIS0uDyWTChg0bIITod6g0AGRnZ+PAgQM4duwYMjMzMW3aNDz1\n1FMYNmyYx/Dd73//+7j77rsxf/58LFy4ENOmTfO43LlmzRq8/PLL2LhxI4xGI1577TVs2rQJubm5\n8j7FxcX41a9+hRdeeAGTJk3CkiVLcPbs2T7Ph0JLEl4udO7evRtHjhxBbGwsNm/eDAB466238Ne/\n/hUajQa33XYb1q5dKzc3S0tLUV5eDrVajdzcXLmzqba2Frt27YLdbkdaWpr8C9DZ2YmdO3eitrYW\nOp0OJpMJI0aM+A5PmYh8dfDgQcyZMwd/+9vfYDQaA36f5cuX48KFC7yEFMa8tizmzJkjNwO7TZ06\nFVu2bMFrr72GxMRElJWVAQDq6upQUVGB/Px85OXloaioSL4cUVRUhNWrV6OgoACXLl1CdXU1ANdM\n4qFDh2L79u247777ek1IGojFYvF5X6VjXbixLtz8rYvXX38dFRUVOHfuHN577z2sWrUKM2fODCoo\nbhX8vXALpC68hsXEiRN7jVCZOnUqVCrXoePHj5evy1ZVVWH27NlQq9VISEhAYmIiampq0NTUhLa2\nNqSmpgIAMjMzUVlZCcA17C4rKwsAMHPmTBw7dsznwvOH78a6cGNduPlbF+fOncODDz6IiRMn4pFH\nHkFWVhbefffd76h0NxZ/L9wCqYugh86Wl5fL46WtVismTJggv2YwGGC1WqFWqz2WkIiLi5M7Ea1W\nq/yaSqVCTEwMWltbPcZjE9GNsXHjRmzcuDHk7/vmm2+G/D3pxgqqg/vtt9+GWq3G3XffHaryeB0r\nTkREN17ALQuz2YyjR496jGYxGAxoaGiQnzc2NsJgMMBgMHgMIeze3n1M93On04m2trZ+WxUWi8Wj\n+ZSTkxNo8RWHdeHGunBjXbixLtxycnJQUlIiPzcajV77pXwKC+FaQ0p+Xl1djf3792P9+vUeQ+PS\n09Oxfft23H///bBarbh8+TJSU1MhSRK0Wi1qamqQkpKCgwcPYsGCBfIxH3/8McaPH4+KigpMnjy5\n33L0dUI3a0z6rUan06GlpeVmF+OWwLpwY124sS7ckpKS/A5Pr0NnCwoK5LVbYmNjkZOTg9LSUnR2\ndsrr7I8fP16eKVxaWooDBw5Ao9H0GjpbWFgoD53tHstvt9uxY8cOnD17FjqdDo8//rhf90BgWLjw\nH4Ib68KNdeHGunBLSkry+xivYXGrY1i48B+CG+vCjXXhxrpwCyQsOIObiIi8YlgQEZFXDAsiIvKK\nYUFERF4xLIiIyCuGBRERecWwICIirxgWRETkFcOCiIi8YlgQEZFXDAsiIvKKYUFERF4xLIiIyCuG\nBRERecWwICIirxgWRETkFcOCiIi8YlgQEZFXDAsiIvKKYUFERF4xLIiIyCuGBRERecWwICIirxgW\nRETkFcOCiIi8UmRYCLsdQoibXQwiIsVQZFg4X38VOHX8ZheDiEgxFBkW+LYFuNZys0tBRKQYGm87\n7N69G0eOHEFsbCw2b94MAGhtbcW2bdtQX1+PhIQEmEwmaLVaAEBpaSnKy8uhVquRm5uLadOmAQBq\na2uxa9cu2O12pKWlITc3FwDQ2dmJnTt3ora2FjqdDiaTCSNGjAjurGwdEDYbpODehYiIunhtWcyZ\nMwfPPfecx7aysjJMmTIFBQUFMBqNKC0tBQDU1dWhoqIC+fn5yMvLQ1FRkdx3UFRUhNWrV6OgoACX\nLl1CdXU1AODAgQMYOnQotm/fjvvuuw9vvfVW8Gdlt7keREQUEl7DYuLEiYiJifHYVlVVhaysLABA\ndnY2Kisr5e2zZ8+GWq1GQkICEhMTUVNTg6amJrS1tSE1NRUAkJmZKR9TWVkpv9fMmTNx7Nix4M/K\nZnM9iIgoJALqs2huboZerwcA6PV6NDc3AwCsVqvHJSSDwQCr1Qqr1Yq4uDh5e1xcHKxWq3xM92sq\nlQoxMTFobW0N7Gy62W2AvSO49yAiIllIOrglKXS9AyEZ8sqWBRFRSHnt4O6LXq9HU1OT/DU2NhaA\nqyXR0NAg79fY2AiDwQCDwYDGxsZe27uP6X7udDrR1taGoUOH9vm5FosFFotFfp6TkwOdTtdrvyZ7\nByIlYEgfrylVZGRkn3UxGLEu3FgXbqwLTyUlJfL3RqMRRqNxwP19CgshhMf/8c+YMQNmsxmLFy+G\n2WxGeno6ACA9PR3bt2/H/fffD6vVisuXLyM1NRWSJEGr1aKmpgYpKSk4ePAgFixYIB/z8ccfY/z4\n8aioqMDkyZP7LUdfJ9TS4jlEVnR2Ak4nbK2t6GwZPMNndTpdr7oYrFgXbqwLN9aFm06nQ05Ojl/H\neA2LgoICHD9+HC0tLVizZg1ycnKwePFi5Ofno7y8HPHx8TCZTACA5ORkzJo1CyaTCRqNBitXrpQv\nUa1YsQKFhYXy0Nnp06cDAObOnYsdO3bgscceg06nw+OPP+7veXvq7Lr8xD4LIqKQkUSYr4tx8eJF\nj+fimyY4n1oGadZcqH75xE0q1Y3H/2tyY124sS7cWBduSUlJfh+jvBnc3fMrOM+CiChklBcWXaOg\nBMOCiChklBcW3X0VNvZZEBGFivLCwsbLUEREoaa8sLDbgOghnJRHRBRCygsLmw2I0bFlQUQUQsoL\nC3sHEDOUfRZERCGkuLAQbFkQEYWc4sICdhsk7VCGBRFRCCkyLBCjYwc3EVEIKTMshmgBpwPC6bjZ\npSEiUgTlhYXNBkRGAhGRbF0QEYWI8sLC3gFERLkCg/0WREQhobywkFsWUWxZEBGFiPLCwm5zXYKK\niOQ9LYiIQkR5YWHrCotItiyIiEJFcWEh7B2QIiPZZ0FEFEKKCwuPy1Bc8oOIKCSUFxbXX4Ziy4KI\nKCSUFxZ2mysoIngZiogoVJQXFrYOICISUmSka1FBIiIKmvLCwm53z+Dm0FkiopBQYFjYumZwc+gs\nEVGoKDAsOq5rWTAsiIhCQXlhYbt+6CzDgogoFBQVFkKIrtFQ3ZPy2GdBRBQKigoLdHYCKjUklZoL\nCRIRhZCywsLeAUREuL7nch9ERCGjrLDo7q8A2GdBRBRCmmAOfvfdd1FeXg5JkjB69GisXbsW7e3t\n2LZtG+rr65GQkACTyQStVgsAKC0tRXl5OdRqNXJzczFt2jQAQG1tLXbt2gW73Y60tDTk5uYGVqDu\n2dsApMhIONlnQUQUEgG3LKxWK95//31s2rQJmzdvhsPhwKeffoqysjJMmTIFBQUFMBqNKC0tBQDU\n1dWhoqIC+fn5yMvLQ1FRkatDGkBRURFWr16NgoICXLp0CdXV1YEVyqNlwbWhiIhCJajLUE6nE+3t\n7XA4HLDZbDAYDKiqqkJWVhYAIDs7G5WVlQCAqqoqzJ49G2q1GgkJCUhMTERNTQ2amprQ1taG1NRU\nAEBmZqZ8jN+651gArq+8DEVEFBIBX4YyGAy4//77sXbtWkRFRWHq1KmYOnUqmpubodfrAQB6vR7N\nzc0AXC2RCRMmeBxvtVqhVqsRFxcnb4+Li4PVag2sUD37LNiyICIKiYDD4ttvv0VVVRV27doFrVaL\nrVu34pNPPum1nyRJQRXwehaLBRaLRX6ek5MDnU4nP7dHaNAxRIuhOh0c+uH41tHp8bqSRUZGDppz\n9YZ14ca6cGNdeCopKZG/NxqNMBqNA+4fcFgcO3YMCQkJGDp0KADg+9//Pr788kvo9Xo0NTXJX2Nj\nYwG4WhINDQ3y8Y2NjTAYDDAYDGhsbOy1vS99nVBLS4v8vWhuglOlRktLC4S9E872No/XlUyn0w2a\nc/WGdeHGunBjXbjpdDrk5OT4dUzAfRYjRozAqVOnYLPZIITAsWPHkJycjBkzZsBsNgMAzGYz0tPT\nAQDp6ek4fPgwOjs7cfXqVVy+fBmpqanQ6/XQarWoqamBEAIHDx5ERkZGQGUSdhskDp0lIgq5gFsW\nqampmDlzJp5++mmo1WqMGTMG9957L9rb25Gfn4/y8nLEx8fDZDIBAJKTkzFr1iyYTCZoNBqsXLlS\nvkS1YsUKFBYWykNnp0+fHlihbDbPDm72WRARhYQkusevhqmLFy/K3zvN7wHnz0K1dC1ERwecT/4C\n6sL/uomlu3HYxHZjXbixLtxYF25JSUl+H6O8GdzdLYuICMBuR5hnIRHRLUFZYWG3yWtDSSoVoNYA\nnfabXCgiovCnrLCwdd0lrxsn5hERhYSywuL6GdxA15IfXB+KiChYCguL62ZwA2xZEBGFiLLCwtYj\nLLjkBxFRSCgrLK5bohwAJ+YREYWIosLCYwY3wPtwExGFiKLCwmOeBcD7cBMRhYiywsLe0XvoLPss\niIiCpqyw6NGykCIiIWy8DEVEFCxlhUVfQ2fZsiAiCprywqLXpDyGBRFRsJQVFjYboOGkPCKiUFNW\nWPRa7oNDZ4mIQkFZYdFzIUFOyiMiCgnFhIVwOl3LkXctUQ7ANZubfRZEREFTTFjAbgc0Ea77WHRj\ny4KIKCQUFBY9+iuArg5u9lkQEQVLQWFh9+yvAICIKAh2cBMRBU1BYdG7ZSFx6CwRUUgoJyx63ssC\n4P0siIhCRDlh0XOpD4DLfRARhYhywqLn8uRA1xLl7LMgIgqWcsKi5/LkAFsWREQhopywsNk8J+QB\nnGdBRBQiigkLYbdBimTLgojou6CYsICto4/RULytKhFRKGiCOfjatWt4/fXXcf78eUiShDVr1iAx\nMRHbtm1DfX09EhISYDKZoNVqAQClpaUoLy+HWq1Gbm4upk2bBgCora3Frl27YLfbkZaWhtzcXP8L\n0/NeFgBXnSUiCpGgWhZvvvkm0tLSkJ+fj9deew0jR45EWVkZpkyZgoKCAhiNRpSWlgIA6urqUFFR\ngfz8fOTl5aGoqAhCCABAUVERVq9ejYKCAly6dAnV1dX+F6aPobOSxpWForMzmNMkIhr0Ag6La9eu\n4eTJk5gzZw4AQK1WQ6vVoqqqCllZWQCA7OxsVFZWAgCqqqowe/ZsqNVqJCQkIDExETU1NWhqakJb\nWxtSU1MBAJmZmfIxfum5PHk33i2PiChoAV+Gunr1KnQ6HXbt2oVz585h3LhxyM3NRXNzM/R6PQBA\nr9ejubkZAGC1WjFhwgT5eIPBAKvVCrVajbi4OHl7XFwcrFar/wWy24Co6N7bI7suRQ3R+v+eREQE\nIIiwcDqdOHPmDFasWIGUlBQUFxejrKys136SJAVVwOtZLBZYLBb5eU5ODnQ6HQCgDYCk0yG663m3\n5qhoxEREQt1ju9JERkbKdTHYsS7cWBdurAtPJSUl8vdGoxFGo3HA/QMOC4PBgLi4OKSkpAAAZs6c\nibKyMuj1ejQ1NclfY2Nj5f0bGhrk4xsbG2EwGGAwGNDY2Nhre1/6OqGWlhYAgPPbVsAwAvau592E\nRoNvm6yQhsQEeqphQafTyXUx2LEu3FgXbqwLN51Oh5ycHL+OCbjPQq/XIy4uDhcvXgQAHDt2DMnJ\nyZgxYwbMZjMAwGw2Iz09HQCQnp6Ow4cPo7OzE1evXsXly5eRmpoKvV4PrVaLmpoaCCFw8OBBZGRk\n+F+gvmZwA5yYR0QUAkENnV2+fDl27NiBzs5O3HbbbVi7di2cTify8/NRXl6O+Ph4mEwmAEBycjJm\nzZoFk8kEjUaDlStXypeoVqxYgcLCQnno7PTp0/0vTF9rQwFdt1bl8FkiomBIonv8apjqbtk4dvwG\nqnt+BGn6Dzxed2x5HqoFP4U0KYAACiNsYruxLtxYF26sC7ekpCS/j1HODO6+ligHeE8LIqIQUE5Y\n9LXcBwBERkKwz4KIKCjKCYu+lvsAIEWwz4KIKFjKCYv+ZnDzPtxEREFTTlj007JgnwURUfCUExYD\n9Fnw1qpERMFRTlh02vtpWXAhQSKiYCknLAbqs2BYEBEFRRFhIZwOwOkANH1MSOdyH0REQVNEWLha\nFZF9r3DLu+UREQVNGWHR30gowLU2FFsWRERBUUZY9NdfAUCKiIRgnwURUVCUERb2DiAiou/XOCmP\niChoyggLWz+LCAJdQ2fZZ0FEFAxlhIXd5uqb6AtbFkREQVNGWPQ3exvgch9ERCGgjLAYcDQUw4KI\nKFgKCQv7wH0WvAxFRBQURYSFsHe47lvRl0hOyiMiCpYiwgK2AS5DcbkPIqKgKSMs+rv/NuDa7uiE\ncDpvbJmIiBREGWExQMtCkiRAE+Hq1yAiooAoIyzsHf0u9wGAiwkSEQVJGWExUJ8FwIl5RERBUkZY\n2G39rw0FcGIeEVGQFBQWA1yGiuT6UEREwVBGWAy03AfA4bNEREFSRFgIuw2Stz4LXoYiIgpYHzet\n9o/T6UReXh4MBgOefvpptLa2Ytu2baivr0dCQgJMJhO0Wi0AoLS0FOXl5VCr1cjNzcW0adMAALW1\ntdi1axfsdjvS0tKQm5vrXyEGuPkRALYsiIiCFHTL4r333sPIkSPl52VlZZgyZQoKCgpgNBpRWloK\nAKirq0NFRQXy8/ORl5eHoqIiCCEAAEVFRVi9ejUKCgpw6dIlVFdX+1eIgRYSBHhPCyKiIAUVFo2N\njTh69CjmzZsnb6uqqkJWVhYAIDs7G5WVlfL22bNnQ61WIyEhAYmJiaipqUFTUxPa2tqQmpoKAMjM\nzJSP8dmMmXkgAAAOxklEQVRAM7gBSJGREGxZEBEFLKiw2Lt3L5YuXeqaJd2lubkZer0eAKDX69Hc\n3AwAsFqtGDFihLyfwWCA1WqF1WpFXFycvD0uLg5Wq9W/gnibZ8Ghs0REQQm4z+LIkSOIjY3FmDFj\nYLFY+t3v+iAJlsVi8fisnJwc6HQ6fOPoRIzeALVO1+dx12KGQq2SENXP60oQGRkJnYLPzx+sCzfW\nhRvrwlNJSYn8vdFohNFoHHD/gMPi5MmTqKqqwtGjR2Gz2dDW1oYdO3ZAr9ejqalJ/hobGwvA1ZJo\naGiQj29sbITBYIDBYEBjY2Ov7X3p64RaWlrg7GjDt3Y7pJaWPo9zQoK95RvY+nldCXQ6HVoUfH7+\nYF24sS7cWBduOp0OOTk5fh0T8GWohx56CLt378bOnTvxxBNPYPLkyfj1r3+NGTNmwGw2AwDMZjPS\n09MBAOnp6Th8+DA6Oztx9epVXL58GampqdDr9dBqtaipqYEQAgcPHkRGRoZ/hbEN3GfB5T6IiIIT\n9NDZnhYvXoz8/HyUl5cjPj4eJpMJAJCcnIxZs2bBZDJBo9Fg5cqV8iWqFStWoLCwUB46O336dP8+\n1OtoqEig7Vqgp0RENOhJonv8api6cOECnL/6P1Dt+i9Imr6zz/n//gQ0XIXqwf97g0t347CJ7ca6\ncGNduLEu3JKSkvw+JvxncDs6AQn9BgWArvtwc54FEVGgwj8svM3eBjh0logoSOEfFt76K8BJeURE\nwQr/sPC24izQtdwHw4KIKFDhHxaddq8tC9fQWfZZEBEFKvzDwtscC4B9FkREQQr/sLB3uO6ENxBO\nyiMiCkr4h4VPLQv2WRARBSP8w8LL8uQAuu6Uxz4LIqJAhX1YCG/LkwO8Ux4RUZDCPixg74DkbVIe\n78FNRBSU8A8Lmw2IiBh4n4gowGZDmC+DRUR004R/WNhtXkdDSWo1oJJc60gREZHfwj8sfJnBDbDf\ngogoCOEfFr6MhgI4MY+IKAjKCAtvo6EA16UqLvlBRBQQZYSFt9FQAFsWRERBCP+w8GWeBcDhs0RE\nQQj/sPCnz4Id3EREAQn7sBA2GyRf+yy45AcRUUDCPixg7/C9z4ItCyKigCggLHzrs5AiIiHYZ0FE\nFJDwDwtfligHeE8LIqIghH9Y+NzBzT4LIqJAhX9Y+LrcB1sWREQBC/+w8HUGNyflEREFLPzDwubj\nDG62LIiIAhb+YeFzy4J9FkREgdIEemBjYyN27tyJ5uZmSJKEefPmYeHChWhtbcW2bdtQX1+PhIQE\nmEwmaLVaAEBpaSnKy8uhVquRm5uLadOmAQBqa2uxa9cu2O12pKWlITc31/eCdNp977PgZSgiooAE\n3LJQq9V4+OGHsXXrVrzyyiv43//9X1y4cAFlZWWYMmUKCgoKYDQaUVpaCgCoq6tDRUUF8vPzkZeX\nh6KiIvnOdUVFRVi9ejUKCgpw6dIlVFdX+1MQSCofToOT8oiIAhZwWOj1eowZMwYAEB0djZEjR6Kx\nsRFVVVXIysoCAGRnZ6OyshIAUFVVhdmzZ0OtViMhIQGJiYmoqalBU1MT2trakJqaCgDIzMyUj/GJ\nL/0VADu4iYiCEJI+i6tXr+LcuXOYMGECmpubodfrAbgCpbm5GQBgtVoxYsQI+RiDwQCr1Qqr1Yq4\nuDh5e1xcHKxWq+8f7kt/BQApMgqC97MgIgpI0GHR3t6OrVu3Ijc3F9HR0b1elyQp2I8YmC/9Fd37\nsWVBRBSQgDu4AcDhcGDLli3IzMxERkYGAFdroqmpSf4aGxsLwNWSaGhokI9tbGyEwWCAwWBAY2Nj\nr+19sVgssFgs8vOcnByooqOh0+m8lrVTr0ebw+HTvuEoMjJSsefmL9aFG+vCjXXhqaSkRP7eaDTC\naDQOuH9QYbF7924kJydj4cKF8rYZM2bAbDZj8eLFMJvNSE9PBwCkp6dj+/btuP/++2G1WnH58mWk\npqZCkiRotVrU1NQgJSUFBw8exIIFC/r8vL5OyKmOQEtLi9eyCrsDzvY2n/YNRzqdTrHn5i/WhRvr\nwo114abT6ZCTk+PXMQGHxcmTJ/HJJ59g9OjR+Od//mdIkoSf//znWLx4MfLz81FeXo74+HiYTCYA\nQHJyMmbNmgWTyQSNRoOVK1fKl6hWrFiBwsJCeejs9OnTfS9IRIRv+0VG8h7cREQBkkT3+NUwdX7d\nCqif/I3X/cTVS3Dmvwj1xt/dgFLdePy/JjfWhRvrwo114ZaUlOT3MeE/gzvSx6GznJRHRBSwsA8L\nyefRUFGclEdEFKCwDwufh85GRnJtKCKiAIV/WPg4KQ+aCMDhgHA6vtvyEBEpUPiHhY/LfUiS5Bo5\nZbd/xwUiIlKe8A8LX1sWgKsznP0WRER+C/+w8LXPAuA9LYiIAhT+YeFPy4LLlBMRBST8w8LXJcoB\nzrUgIgpQ+IeF3y0LXoYiIvJX+IeFxse1oQBXBzdbFkREfgv7sJB8Xe4D4D0tiIgCFPZh4ddoqEh2\ncBMRBSL8w8KPPgspIhKCQ2eJiPwW/mHh12goTsojIgpE+IeFv6Oh2GdBROS38A8Lv2Zws8+CiCgQ\n4R8Wfq0NxWXKiYgCEf5h4U+fBW+AREQUkPAPC79bFgwLIiJ/hX9Y+DODmx3cREQBCfuwkCTJ950j\nuTYUEVEgwj4s/CFFREGwZUFE5LdBFRZc7oOIKDCDKyzYZ0FEFJDBFxbssyAi8tvgCgvez4KIKCCD\nKyy43AcRUUA0N7sA3aqrq1FcXAwhBObMmYPFixeH/kM4KY+IKCC3RMvC6XTi97//PZ577jls2bIF\nhw4dwoULF0L/QRFRXBuKiCgAt0TLoqamBomJiYiPjwcA/PCHP0RlZSVGjhwZ2g+KjALarsFR+Aqk\nGB0wVAfEDAOG6lzPY3TAEC2gjQGGxABDhkBSqUNbBiKiMHRLhIXVakVcXJz83GAwoKamJuSfI0VF\nQfXMbwFrA0TrN8C3LUDrN8CVOjhbW4BvW4G2b4G2a65HR5srYIbEuPo71GpAren62vXQRAARka57\ngfd6RLr212hc+2k0gFoDKSICUKkASQVIkvsBCVBJgErteqjVrv1UakCt6rG9e5tru1MCxLVW13sA\nXe/X9VTd9bmqW6IhSURh6JYIixtJGjMeGDMeviwSIpxOoL3NFRx2G+BwAI7OHl/tgN0OYetwDcvt\nfnR0uMLH0Ql0dnZ9tQOdnXDa7YBwAkJ0Pa7/XgBOp+u9nQ7XV9H13OFwvebsfq37eye+6X4P4Lqv\nXf/pLq8kdQVWd3CpXftc//kQgLPreJXkCjSgK9wkd8Cp3EHl/l4FQLouqHp8f/05enym5P6s7veW\nrvtsiOt+KNd93x2wEjyCt1WjgcPh8PEXosdvgujns7wd3zP0vS1DI793j5+ZLzzeW+pjm1urRg1H\nZz910V8Z/SmLr/xZluc7KkOrRgNHZ2fI37dPt0DdSuPuhOq+nJB9zC0RFgaDAQ0NDfJzq9UKg8HQ\naz+LxQKLxSI/z8nJQVJS0g0pIxGRkpSUlMjfG41GGI3GAfe/Ja5LpKam4vLly6ivr0dnZycOHTqE\n9PT0XvsZjUbk5OTIj+tPdrBjXbixLtxYF26sC7eSkhKPv6XeggK4RVoWKpUKK1aswIYNGyCEwNy5\nc5GcnHyzi0VERF1uibAAgOnTp6OgoOBmF4OIiPpwS1yGCpQvTafBgnXhxrpwY124sS7cAqkLSYjv\nonueiIiUJKxbFkREdGMwLIiIyKtbpoPbXzdk4cFb1O7du3HkyBHExsZi8+bNAIDW1lZs27YN9fX1\nSEhIgMlkglarvckl/W41NjZi586daG5uhiRJmDdvHhYuXDgo68Jut+Oll15CZ2cnHA4HZs6ciZ/9\n7GeDsi66OZ1O5OXlwWAw4Omnnx60dfHII49Aq9VCkiSo1Wps3LgxsLoQYcjhcIhHH31UXL16Vdjt\ndrFu3TpRV1d3s4t1w5w4cUKcOXNGPPXUU/K2f/u3fxNlZWVCCCFKS0vFW2+9dbOKd8P8/e9/F2fO\nnBFCCNHW1iYee+wxUVdXNyjrQggh2tvbhRCufx/PPvusOHXq1KCtCyGEeOedd0RBQYF49dVXhRCD\n89+IEEI88sgjoqWlxWNbIHURlpehrl94UKPRyAsPDhYTJ05ETEyMx7aqqipkZWUBALKzswdFfej1\neowZMwYAEB0djZEjR6KxsXFQ1gUAREVFAXC1MrqXOxmsddHY2IijR49i3rx58rbBWhdCCIge45gC\nqYuwvAx1oxYeDCfNzc3Q6/UAXH9Em5ubb3KJbqyrV6/i3LlzmDBhwqCtC6fTiWeeeQZXrlzBj3/8\nY6Smpg7auti7dy+WLl2Ka9euydsGa11IkoQNGzZApVLh3nvvxbx58wKqi7AMC/JO8nfhtjDW3t6O\nrVu3Ijc3F9HR0b1eHyx1oVKp8Nvf/hbXrl3D5s2bcf78+V77DIa66O7PGzNmjMdacj0NhroAgN/8\n5jcYPnw4vvnmG2zYsKHP9fR8qYuwDAtfFx4cTPR6PZqamuSvsbGxN7tIN4TD4cCWLVuQmZmJjIwM\nAIO3LrpptVpMmjQJ1dXVg7IuTp48iaqqKhw9ehQ2mw1tbW3YsWPHoKwLABg+fDgAYNiwYcjIyEBN\nTU1AdRGWfRa+LjyoZD2vQ86YMQNmsxkAYDabB0197N69G8nJyVi4cKG8bTDWxTfffCNfcrHZbDh2\n7BhGjhw5KOvioYcewu7du7Fz50488cQTmDx5Mn79618Pyrro6OhAe3s7AFcL/G9/+xtGjx4dUF2E\n7Qzu6upqvPnmm/LCg4Np6GxBQQGOHz+OlpYWxMbGIicnBxkZGcjPz0dDQwPi4+NhMpl6dYIrzcmT\nJ/HSSy9h9OjRkCQJkiTh5z//OVJTUwddXXz99dcoLCyE0+mEEAKzZ8/GAw88gNbW1kFXF9c7fvw4\n3nnnHXno7GCri6tXr+K1116DJElwOBy45557sHjx4oDqImzDgoiIbpywvAxFREQ3FsOCiIi8YlgQ\nEZFXDAsiIvKKYUFERF4xLIiIyCuGBRERecWwICIir/4/UQZtchLj83wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20c90302e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def relu(val):\n",
    "    return np.maximum(val, 0)\n",
    "\n",
    "\n",
    "def relu_der(val):\n",
    "    return (val > 0).astype(float)\n",
    "\n",
    "\n",
    "def mse(target_y, model_y):\n",
    "    return 0.5 * np.square(target_y - model_y)\n",
    "\n",
    "def mse_der(target_y, model_y):\n",
    "    return target_y - model_y\n",
    "\n",
    "\n",
    "def delta(output, weights_prev, delta_prev):\n",
    "    print(\"(%s x %s) x %s\" % (weights_prev.shape, delta_prev.shape, output.shape))\n",
    "    dweights = np.dot(weights_prev.transpose(), delta_prev)\n",
    "    dact = relu_der(output)\n",
    "    return dweights.T * dact\n",
    "\n",
    "n_input = 2\n",
    "n_hidden = 2\n",
    "n_output = 1\n",
    "\n",
    "hidden_weights = np.random.standard_normal((n_hidden, n_input))\n",
    "hidden_bias = np.ones((n_hidden, 1))\n",
    "out_weights = np.random.standard_normal((n_output, n_hidden))\n",
    "out_bias = np.ones((n_output, 1))\n",
    "\n",
    "train_x = np.mat(np.arange(1, 101).reshape((-1, 2)))\n",
    "train_y = np.apply_along_axis(lambda x: np.sin(x[0] + x[1]) ** 2 + np.random.normal() * 0.05, 1, train_x).reshape(-1, 1)\n",
    "\n",
    "print(train_x)\n",
    "print(train_y)\n",
    "\n",
    "eta = 0.1\n",
    "\n",
    "errors = []\n",
    "\n",
    "for e in range(0, 50):\n",
    "    # forward pass\n",
    "    # (2x2) * (2x1) = (2x1)\n",
    "    # first layer activation\n",
    "    idx = np.random.randint(train_x.shape[0])\n",
    "    x = train_x[idx].reshape(-1, 1)\n",
    "    y = train_y[idx]\n",
    "    hidden_out = np.dot(hidden_weights, x) + hidden_bias\n",
    "    hidden_activated = relu(hidden_out)\n",
    "\n",
    "    # (1x2) * (2x1) = (1x1)\n",
    "    out_y = np.dot(out_weights, hidden_activated) + out_bias\n",
    "\n",
    "    #backprop\n",
    "    # (1x1) - (1x1) = (1x1)\n",
    "    dout = mse_der(out_y, y) # delta rule for linear unit\n",
    "    print(\"Out delta - %s\" % dout)\n",
    "\n",
    "    dhidden = delta(hidden_activated, out_weights, dout) # delta rule for hidden layer\n",
    "    print(\"Hidden delta - %s\" % dhidden)\n",
    "\n",
    "    # gradient descent\n",
    "    out_weights = out_weights - eta * np.dot(dout, hidden_activated.T)\n",
    "    hidden_weights = hidden_weights - eta * np.dot(dhidden, x.T)\n",
    "    out_bias = out_bias - eta * dout\n",
    "    hidden_bias = hidden_bias - eta * dhidden\n",
    "\n",
    "    errors.append(mse(y, out_y))\n",
    "    print(\"Total error: %s\" % mse(y, out_y))\n",
    "    print(\"Prediction: %s\" % out_y)\n",
    "    \n",
    "errors = np.array(errors).ravel()\n",
    "\n",
    "plt.title(\"MSE for each training epoch\")\n",
    "plt.plot(range(0, errors.shape[0]), errors)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}