Backprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation\n",
    "\n",
    "In this notebook, we will implement the backpropagation procedure for a two-node network. We'll start by implementing each step of the backpropagation procedure, and then combine these steps together to create a complete backpropagation algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Preliminaries\n",
    "Text preceded by a `#` indicates a 'comment'.  I will use comments to explain what we're doing and to ask you questions.  Also, comments are useful in your own code to note what you've done (so it makes sense when you return to the code in the future).  It's a good habit to *always* comment your code.  I'll try to set a good example, but won't always . . . \n",
    "\n",
    "Before beginning, let's load in the Python packages we'll need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pylab import *\n",
    "%matplotlib inline\n",
    "rcParams['figure.figsize']=(12,3)                # Change the default figure size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Steps to backpropagation\n",
    "\n",
    "We outlined 4 steps to perform backpropagation,\n",
    "\n",
    "   1. Choose random initial weights.\n",
    "   2. Fix input at desired value, and calculate output.\n",
    "   3. Update the weights.\n",
    "   4. Repeat steps 2 & 3 many times.\n",
    "\n",
    "Let's now implement these steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Choose random initial weights.\n",
    "  This step is easy. We need to choose the three initial weights in our\n",
    "  2-node model.  We can choose these weights to be anything we like,\n",
    "  within reason.  Let's set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "w0 = 2\n",
    "w1 = 1\n",
    "w2 = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"question\">\n",
    "    \n",
    "**Q:**  Why choose these weights?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, in backpropagation, we'll update these weights so that our neural\n",
    "network - when given a specific input - produces a desired output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2.  Fix input at desired value, and calculate output. \n",
    "  Our next step is to choose an input, send this input to our neural\n",
    "  network, and compute the output.  This is called the *feedforward\n",
    "  solution*; we're \"feeding\" our neural network the input, sending this\n",
    "  input \"forward\" through the network, and returning the output.\n",
    "\n",
    "  Let's start by choosing an input,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "s0 = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  Then, we need to send this input through our network. We'll do this in\n",
    "  multiple steps.  First, let's compute the activity of the first neuron:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = w0*s0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  Then, to determine the output of the first neuron, we need to send this\n",
    "  activity through a sigmoid function.  To do that, we first need to\n",
    "  *define* the sigmoid function ... Let's do so here in Python,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1/(1+exp(-x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  This function takes one input (`x`), and returns one output equal to the numerical\n",
    "  value of the sigmoid function at that value of `x`. For example, to\n",
    "  compute the value of the sigmoid function at `x=0.5`, and save this output\n",
    "  in a variable `s`, we execute the following command,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6224593312018546\n"
     ]
    }
   ],
   "source": [
    "s = sigmoid(0.5)\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're able to compute the output of our first neuron,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1 = sigmoid(x1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can ontinue our feedforward computation to determine the output of our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.36375380675182073\n"
     ]
    }
   ],
   "source": [
    "x2 = w1*s1\n",
    "s2 = sigmoid(x2)\n",
    "out = w2*s2\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3.  Update the weights.\n",
    " To update the weights, we'll use the expressions we derived in class \n",
    " for the updated values of `w0` and `w1`.  Let's write those here ...\n",
    "\n",
    " But first, remember that to compute the weight update, we need to define\n",
    " a learning rate,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha  = 0.1    #Here we'll set the learning rate to 0.1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we also need to define a target value we would like our neural network to produce as output,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's define two weight updates,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "w2 = w2 - alpha*(out-target)*s2\n",
    "w1 = w1 - alpha*(out-target)*(w2*s2*(1-s2)*s1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"question\">\n",
    "\n",
    "**Q:**  Do these two expressions match the equations for the weight updates we derived in class?  HINT: They should!\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need one more expression, to update the last weight, w0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "w0 = '????????'  # <-- Fill in the proper expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"question\">\n",
    "\n",
    "**Q:** What is the expression for `w0` in the equation above?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4.  Repeat steps 2 & 3 many times.\n",
    " Now, we'd like to repeat Steps 2 & 3 many times. In doing so, we'll\n",
    " perform gradient descent, and move (slowly) down our cost function,\n",
    " until we reach a minimum.\n",
    "\n",
    " We could try to compute these repetitions by hand, for example by repeating\n",
    " the cells above. To do so, we'd need to take the new weights we just\n",
    " found in Step 3, compute the feedforward solution to our neural network,\n",
    " and then update the neural network's weights (which depends on how close\n",
    " our neural network's output is to the target value).\n",
    "\n",
    " But, we want to evaluate this procedure 1000 times. We could do this by\n",
    " hand, but it'd be a total pain, and highly error prone. Instead, let's\n",
    " ask a computer to do the boring work of multiple repetitions. To do so,\n",
    " let's collect the code above, and repeat it 1000 times. We'll wrap our\n",
    " code above inside a `for-loop` to make this efficient,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAADTCAYAAABz5445AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU9b3v/9ckkwsXuZkU5KJi5VREQSQmFKpQ90Zwi/XXx9HPUfvwsssudR/pPsftPt3WY6VSbfXstmprrWC1Vn9by6dWW3aP4qZb3YBWiqhcxEvxgkAgcichJGQy6/wxK+kwTCAhM5mQeT8fj3Fmfdd31voknwx+8s13fVckCAJEREREROQvCnIdgIiIiIhId6MiWUREREQkhYpkEREREZEUKpJFRERERFKoSBYRERERSaEiWUREREQkRTTXAbRB69KJiIiISFeIpGvsrkUy1dXVXX7OsrIyduzY0eXnla6lPOcH5Tk/KM89n3KcH3KV56FDh7a5T9MtRERERERSqEgWEREREUlx1OkWZjYCeBwYAsSBBe5+f0qfCHA/8DdAPXC9u78R7rsOuC3seqe7/zJz4YuIiIiIZF575iTHgJvd/Q0zOwFYZWZL3H19Up+LgVHhowr4GVBlZoOAuUAFiYvxVpnZInffndGvQkRERETSCoKAhoYG4vE4kUjaa9RyrqamhsbGxqwcOwgCCgoKKC0t7dDXf9Qi2d23AlvD17Vm9g4wDEguki8DHnf3AHjNzAaY2UnAVGCJu+8CMLMlwAzgqXZH2EV21DSxd2ct+/YdJAgCgoCURwBp2g7ZjictyxEcvkBHmqZjam85iZYAOTalJTU0ZOmDKN2H8pwflOeeTznuvBMGNFE+uJRotLgbFw9RevcuzNrRY7EYDQ0N9OrVqwMRdYCZnQqMB1ak7BoGbEra3hy2tdWe7tizgdkA7k5ZWVlHQuu0lcu3sG1LTYffF4m0PCKtrwl/S4m0/iep/2EvWo4TaWe/NAEc3k2O6ECuA5AuoTznB+W551OOO+uE/kAQJdaU60jaVlQEJSXZW3QtGo0SiUQ6VF+2Oxoz6wv8Bvif7r4vZXe6Gi04Qvth3H0BsKClT1cvAzJmfBGTppazZ8/uw4re5EKY8HVBhPC1ytPjjZYTyg/Kc35Qnns+5bjz6uvrszpKmwnRaCGxWCyr52hsbDzsZ+lIS8C1q0g2syISBfK/uvszabpsBkYkbQ8HqsP2qSntL7fnnF2td58C+g8opinWvX+IRERERHqKAwcOMHv2bD755BMKCgqYNm0at956a67DAtqxBFy4csUjwDvu/qM2ui0CrjWziJlNBPaGc5lfAC4ys4FmNhC4KGwTEREREeGGG27glVde4YUXXmDlypW8+OKLuQ4JaN9I8mTgGmCtmb0Vtt0KnAzg7g8Bz5FY/m0DiSXg/jbct8vMvgusDN83r+UiPhERERHp+R588EFKSkqYNWsWc+fOZf369fz6179m2bJluDs/+clPACguLubss89m69atOY44oT2rWyznKNeFhata3NjGvkeBR48pOhERERHJmPivHibY9FFGjxkZMZKCK7/W5v6qqirmz5/PrFmzWLNmDQcPHqSpqYmVK1dSWVnZ2m/v3r0sWbKEWbNmZTS+Y6U77omIiIhI1owdO5a1a9dSV1dHcXExEyZMYPXq1axYsYKqqiogsUTbjTfeyFe/+lVOOeWUHEeckL21NkRERESkWznSiG+2FBUVMXz4cBYuXEhFRQWjR4/m1VdfZePGjYwaNQqAm2++mZEjR/K1r3V9fG3RSLKIiIiIZNXEiRN56KGHqKqqoqqqiieeeIIxY8YQiUS45557qK2t5Y477sh1mIdQkSwiIiIiWVVZWcmnn35KRUUF5eXllJSUUFlZSXV1NT/+8Y95//33mT59OtOmTePJJ5/MdbiApluIiIiISJadf/75bNy4sXV7+fLlra+3bNlCNBrN+s1EOkojySIiIiIiKVQki4iIiIikUJEsIiIiIpJCRbKIiIiISAoVySIiIiIiKVQki4iIiIikUJEsIiIiIjlz9913M378+Na773UXKpJFREREJGemTZvG4sWLcx3GYVQki4iIiEjWPPjggzzyyCMAzJ07lyuuuAKAZcuW8Y1vfIMJEyYwePDgXIaY1lHvuGdmjwIzgU/d/aw0+/8X8JWk440Gyt19l5l9DNQCzUDM3SsyFbiIiIiIdMzPX6/ho90NGT3myIGl/F1F20VuVVUV8+fPZ9asWaxZs4aDBw/S1NTEypUrqayszGgsmdSe21I/BjwAPJ5up7v/C/AvAGZ2KXCTu+9K6vJFd9/RyThFRERE5Dg0duxY1q5dS11dHcXFxZx99tmsXr2aFStW8N3vfjfX4bXpqEWyuy81s1PbebyrgKc6FZGIiIiIZMWRRnyzpaioiOHDh7Nw4UIqKioYPXo0r776Khs3bux2F+sla89IcruYWW9gBjAnqTkA/t3MAmC+uy84wvtnA7MB3J2ysrJMhdZu0Wg0J+eVrqU85wflOT8ozz2fctx5NTU1RKMZK/mOyaRJk5g/fz733nsvo0ePZt68eYwdO5aioqJD+mUzzpKSkg79LGUykkuBV1KmWkx292oz+wywxMzedfel6d4cFtAtRXSwY0fXz9AoKysjF+eVrqU85wflOT8ozz2fctx5jY2NFBYW5jSGiooK7rvvPsaPH0/v3r0pLi7mvPPOIxaLceedd/Lb3/6WAwcOMG7cOK6++mpuvvnmjMfQ2Nh42M/S0KFD2+yfySL5SlKmWrh7dfj8qZk9C1QCaYtkEREREemZzj//fDZu3Ni6vXz58tbXt912G9/5zneIxWK5CK1NGVkCzsz6A1OA3yW19TGzE1peAxcB6zJxPhERERGRbGrPEnBPAVOBMjPbDMwFigDc/aGw25eBf3f3/UlvHQw8a2Yt53nS3bvfStEiIiIiIikiQRDkOoZ0gurq6i4/qeY95QflOT8oz/lBee75lOPOq6+vp3fv3rkO44ii0WjWp1uk+z6Ec5Ij6frrjnsiIiIiIilUJIuIiIiIpFCRLCIiIiKSQkWyiIiIiOTMmjVrmDJlCpMnT+bb3/423eV6ORXJIiIiIpIz3/rWt/jBD37A8uXL+eijj3jppZdyHRKgIllEREREsujBBx/kkUceAWDu3LlcccUVACxbtgwzo7a2lvPOO49IJMLll1/O4sXdY8Xg3N7IW0RERES6zLo36tm3pzmjx+w3oJCzzm17ibmqqirmz5/PrFmzWLNmDQcPHqSpqYmVK1cyZcoUli79y82YTzrpJLZt25bR+I6VRpJFREREJGvGjh3L2rVrqauro7i4mAkTJrB69WpWrFjBueeee1j/SCTtssVdTiPJIiIiInniSCO+2VJUVMTw4cNZuHAhFRUVjB49mldffZWNGzdy2mmnsXXr1ta+W7duZfDgwV0eYzoaSRYRERGRrJo4cSIPPfQQVVVVVFVV8cQTTzBmzBgGDx5M3759ef311wmCgKeffprp06fnOlxARbKIiIiIZFllZSWffvopFRUVlJeXU1JSQmVlJQDf//73+cd//EcmT57MKaecwoUXXpjjaBM03UJEREREsur8889n48aNrdvLly9vfT1u3DiWLl1KLBbLRWht0kiyiIiIiEiKo44km9mjwEzgU3c/K83+qcDvgI/CpmfcfV64bwZwP1AI/Nzd785Q3CIiIiIiWdOe6RaPAQ8Ajx+hzzJ3n5ncYGaFwE+BacBmYKWZLXL39ccYq4iIiIhIlzjqdAt3XwrsOoZjVwIb3P1Ddz8I/Aq47BiOIyIiIiLSpTJ14d7nzWw1UA38k7u/DQwDNiX12QxUZeh8IiIiIiJZk4ki+Q3gFHevM7O/AX4LjALS3S4laOsgZjYbmA3g7pSVlWUgtI6JRqM5Oa90LeU5PyjP+UF57vmU486rqakhGu3+C5plO8aSkpIO/Sx1Ohp335f0+jkze9DMykiMHI9I6jqcxEhzW8dZACwIN4MdO3Z0NrQOKysrIxfnla6lPOcH5Tk/KM89n3LceY2NjRQWFuY6jCOKRqOtS8AtXLiQKVOmMGTIkIyeo7Gx8bCfpaFDh7YdU2dPaGZDgBp3D8ysksQ8553AHmCUmY0EtgBXAld39nwiIiIi0nP9+te/5owzzsh4kdxR7VkC7ilgKlBmZpuBuUARgLs/BFwO/L2ZxYADwJXuHgAxM5sDvEBiCbhHw7nKIiIiIpJH5s+fz8KFCwG46qqrmDFjBtdddx0vvvgiAA8++CC1tbV87nOfY/Xq1cyZM4fS0lIWLVpEr169chLzUYtkd7/qKPsfILFEXLp9zwHPHVtoIiIiIpJJS5cuZfv27Rk9Znl5ORdccEGb+9esWYO78/vf/54gCJg5cyaf//zn0/adOXMmjz32GN/+9rcZN25cRuPsqO4/i1tEREREjlt/+tOfmDFjBr179wbg4osvZsWKFTmO6uhUJIuIiIjkiSON+GZLEBy+uNm+ffuIx+Ot2w0NDV0ZUrsc9WYiIiIiIiLHauLEibzwwgscOHCA+vp6Fi9ezBe/+EV27NjBrl27aGxsZMmSJa39+/TpQ11dXQ4jTtBIsoiIiIhkzdlnn80VV1zBJZdcAiQu3DvnnHO46aabuPTSSxkxYgSjRo1q7W9m3HLLLTm/cC+Sbgi8Gwiqq9tcUjlrtBZjflCe84PynB+U555POe68+vr61vnA3VXyOsnZku77EK6TnO4GeJpuISIiIiKSSkWyiIiIiEgKFckiIiIiIilUJIuIiIiIpFCRLCIiIiKSQkWyiIiIiEgKFckiIiIikjV79+7lsccey/p5Xn31VVauXJmx46lIFhEREZGs2bdvH48//ni7+wdBcMgtq9vrj3/8I6tWrerw+9qiO+6JiIiISNZ873vfY+PGjUybNo1JkybxzjvvsHfvXmKxGN/85jeZPn06n3zyCVdddRWTJk1i1apVPProoyxbtoyf/vSnDBkyhJEjR1JcXMxdd93Fzp07ueWWW9iyZQsAd9xxB0OGDOGJJ56gsLCQ3/zmN9x5551UVVV1Ku6jFslm9igwE/jU3c9Ks/8rwD+Hm3XA37v76nDfx0At0AzE3L2iU9GKiIiIyDG7/fbbWb9+fUaPeeaZZzJv3rw2999666289957LFmyhFgsxoEDBzjhhBPYtWsXl156KRdddBEAH3zwAT/60Y/4/ve/z7Zt27jvvvtYvHgxffv2xcw488wzW7+Gr33ta1RWVrJlyxauvvpq/vM//5NrrrmGPn36cMMNN2Tk62rPSPJjwANAW+PkHwFT3H23mV0MLACSS/cvurvuJykiIiKS54Ig4O6772bFihVEIhG2bdvG9u3bARg+fDgTJkwA4K233mLixIkMHDgQgJkzZ/Lhhx8CsGzZMt5///3WY9bV1VFXV5fxWI9aJLv7UjM79Qj7X03afA0YnoG4RERERCTDjjTi2xWeeeYZdu7cyfPPP09RURFVVVU0NjZSWFhI7969W/sFQdDmMeLxOIsWLaJXr15ZjTXTc5JnAc8nbQfAv5tZAMx39wVtvdHMZgOzAdydsrKyDId2dNFoNCfnla6lPOcH5Tk/KM89n3LceTU1NUSjubsMrX///uzfv59oNMr+/fspLy+nV69eLF++nM2bN1NYWNjatyXOiooK7rjjDurq6ujbty/PP/88o0ePJhqNMnXqVB5//HFuvPFGANatW8dZZ51Fv379qK2tbfNrLSkp6dDPUsa+Y2b2RRJF8heSmie7e7WZfQZYYmbvuvvSdO8PC+iWIjrYsaPrZ2iUlZWRi/NK11Ke84PynB+U555POe68lpHaXOnXrx8VFRVccMEFjBs3jg0bNjBt2jTGjBnD6aefTnNzc2t8sVgMgPLycubMmcOMGTMYMmQIp59+On369CEWizFv3jxuvfVWpk6dSiwWo6qqinvuuYcLL7yQr3/96zz//PNpL9xrbGw87Gdp6NChbcYdOdJwdotwusXv0124F+4fCzwLXOzu77fR5ztAnbv/4KgnhKC6urod3TJLH8T8oDznB+U5PyjPPZ9y3Hn19fWHTGXojqLRaGuB3GL//v2thfGsWbO48sorufjii4/5HOm+D2GRHEkb0zGfKWRmJwPPANckF8hm1gcocPfa8PVFQG4nwoiIiIjIceGHP/why5Yto7GxkSlTpjBjxowuPX97loB7CpgKlJnZZmAuUATg7g8BtwMnAg+aGfxlqbfBwLNhWxR40t0XZ+FrEBEREZEe5vbbb8/p+ds13SIHNN1CskZ5zg/Kc35Qnns+5bjzjtfpFpnW0ekWui21iIiISA/WTQdEu1xHvw8qkkVERER6sIKCgqyP0nZ3sViMgoKOlb25WzRPRERERLKutLSUhoYGGhsbiUTSzizIuZKSEhobG7Ny7CAIKCgooLS0tEPvU5EsIiIi0oNFIpGs352us7rj3HNNtxARERERSaEiWUREREQkhYpkEREREZEUKpJFRERERFLowr1uoqk5TmNzQKw5oCkeEIsHNIWvE89xmuMQDwLiQeI5CEi8JiAeh4A29rexLmBqa1vLBwZJPY+0xGDyru68JGOfPo3s378/12FIlinP+UF57vmU4/xQdXoxQ4pyHcWhVCRnQRAE7DoQo6auiV0HYuxpiLH7QDN7GmLsbYix/2CcA7E49U1xDjQlnmPxblxVioiIiGRRQXEpl362e90VUEVyJ7QUwx/sauDDXY18uLuBbbVNbKs7SGPzoUVvQQT6lxTSvzRK3+ICynoX0buogN5FBfQKHyWFBRQVRigqiLQ+R5O2o5EIBQURCiIQIfFcEEks7ZJ4hsJIhAhQEIkQSdqfvCriISskpiyX2Ha/SPr2IxzvkGN1o2UZTxx0Ijt37cx1GJJlynN+UJ57PuU4PwwpL2ffnl25DuMQKpI7aPv+JlZv28+abfWsraln14HEHWwiwNB+xQzrV8y4k3pz0gnFDOlbxIm9ixhQWsgJJYUUdKdKMY/1KYlyoKgw12FIlinP+UF57vmU4/xQHO1+l8mpSG6HrbUHeeWTWl79pJYPdjUAMKC0kLMH92Z0eW9OG1TCyIGllHbDBIuIiIhIx7WrSDazR4GZwKfuflaa/RHgfuBvgHrgend/I9x3HXBb2PVOd/9lJgLPtqbmOH/cVMcLG/awrqYegFEnlnLdOeVUDOvLiP7F3fbWjiIiIiLSOe0dSX4MeAB4vI39FwOjwkcV8DOgyswGAXOBChKLH6wys0XuvrszQWdT3cFmnn9/N//27m72NjYzuG8R14wrZ8rIfpT36WaXXYqIiIhIVrSrSHb3pWZ26hG6XAY87u4B8JqZDTCzk4CpwBJ33wVgZkuAGcBTnYo6C/YciPHrVz7mmdXV1DfFmTC0D5eeMYhxQ3prLrGIiIhInsnUnORhwKak7c1hW1vt3c68lzfz4a4GJp18ApePOZHTBpXmOiQRERERyZFMFcnphlqDI7QfxsxmA7MB3J2ysrIMhdY+//RXxZT1LWVov+IuPa90vWg02uU/X9L1lOf8oDz3fMpxfuiOec5UkbwZGJG0PRyoDtunprS/nO4A7r4AWBBuBjt27MhQaO0ztBjK+vWjq88rXa+srEx5zgPKc35Qnns+5Tg/5CrPQ4cObXNfporkRcAcM/sViQv39rr7VjN7AfiemQ0M+10EfCtD5xQRERERyYr2LgH3FIkR4TIz20xixYoiAHd/CHiOxPJvG0gsAfe34b5dZvZdYGV4qHktF/GJiIiIiHRXkSBIO0U414Lq6uouP+nx8iedIB6Hgw3Q0ACNDdBwABoPwMFGaG6GWIygOQaxGDSHj1gs3NcEQTwxMzyIQxAkPZLb27H/6JG28wvK4M9gO45VUlpK44EDmTundEslpaU0NjTkOgzJMuW551OO80P/KRdRO/KMLj9vON0i7TJmuuNeNxLEYrBrO+zYRrC9BnbWwL49BPv2Qm34qKtNFMSZEokc+iACBeFzpCDpdZpHe4/fvo4Z6dKejgcLCwia4x2ITY5HBwsKEr9QSo+mPPd8ynF+aP7cGMhBkXwkKpJzJGhqgo/eJ9i4AT75gGDjB1CzBZL/ISiMQr8BcEI/OKE/kSHDoW8/6NULSnpBSSmU9iJSWprYLiqGoiIoLITCIohGE6+j0cR2YWHimIUFECnI2zsGHi9/MZDOUZ7zg/Lc8ynH+aF3WRn13SzPKpK7ULB7J8EbrxK8/Sa8tzYxPQJgwCA4+bNExn8ePnMSkfLBUDYYBgwiUlCY26BFRERE8pCK5CwLYk0Er79CsHwJvL8uMWd2yDAik/+KyOhz4LTPEek/8OgHEhEREZEuoyI5S4KDjQQvPUfwh0WwZyd8ZiiRS68iUnkBkcFtr8knIiIiIrmnIjnDgiAgWLmM4JnHYeencMZYCq69EcacS6SgINfhiYiIiEg7qEjOoGD3TuKP/RjWvwkjRlJw/Z1Ezhib67BEREREpINUJGdIsG4V8Yd/CLEmIld/nciUGbroTkREROQ4pSI5A+J/+B2B/wKGnULB179JZMiwXIckIiIiIp2gIrkTgiAgWPQkwe8XwviJFHz1JiKlvXIdloiIiIh0korkTgh+968E/9eJfGEakWtu1IV5IiIiIj2EqrpjFH/5ORXIIiIiIj2UKrtjELz9JsFTC+DsCiLX/HcVyCIiIiI9jKq7Dgp27yT+8A/gpBEUzP4nrWAhIiIi0gO1a06ymc0A7gcKgZ+7+90p++8Fvhhu9gY+4+4Dwn3NwNpw3yfu/qVMBJ4LQTxO/NF7oekgBTf8M5HS3rkOSURERESy4KhFspkVAj8FpgGbgZVmtsjd17f0cfebkvp/AxifdIgD7n5O5kLOneAPi+DdNUSunUNkyPBchyMiIiIiWdKe6RaVwAZ3/9DdDwK/Ai47Qv+rgKcyEVx3EuzaTrDoSRh7HpEvTMt1OCIiIiKSRe2ZbjEM2JS0vRmoStfRzE4BRgIvJjWXmtnrQAy4291/28Z7ZwOzAdydsrKydoSWWdFotM3z7vnFfTQGccpuvIXC8vIujkwy6Uh5lp5Dec4PynPPpxznh+6Y5/YUyZE0bUEbfa8Ennb35qS2k9292sxOA140s7Xu/kHqG919AbCg5fg7duxoR2iZVVZWRrrzBu+sJv7qi0Qu+wq7C4ogB7FJ5rSVZ+lZlOf8oDz3fMpxfshVnocOHdrmvvZMt9gMjEjaHg5Ut9H3SlKmWrh7dfj8IfAyh85X7vaCICD+7BMwqJzI9C/nOhwRERER6QLtGUleCYwys5HAFhKF8NWpnczsc8BA4I9JbQOBendvNLMyYDLwfzIReJdZtwo+ej9xsV5Rca6jEREREZEucNQi2d1jZjYHeIHEEnCPuvvbZjYPeN3dF4VdrwJ+5e7JUzFGA/PNLE5i1Pru5FUxupPbb7+dP//5zzQ1NbW2BUEAn3wAzc2waz6R+xcc4QjZEwQB8XiceDxOEASt2y2vkx+tcYfPqW2pr1PP05627qgjcUYikePm65JjpzznB+W551OO88OECRO49957cx3GISLd9AcvqK5ua0ZHdqQtkutqYcvHMGQYkf6DsnLeIAhobm4mFosRi8VaXzc3N7cWxvF4vEPHjEQihz23vE7Xrz3tbfU9HhUUFHT4eyrHH+U5PyjPPZ9ynB8qKir44Q9/2OXnDeckpy1yVCQnSZ003nz/d2DTxxTc/TCRaFFGzlFXV8emTZuoqalh+/bt7Nix45DCvLCwkH79+tGnTx969ep1yKOkpISioqLWR3FxMUVFRUSjUQoLCyksLKSgoKBHFbTZoItA8oPynB+U555POc4POb5wL23h1K477uWjYNtmWPcGkS9d3akCOQgCampqeP/999m4cSO7d+8GoKioiPLycs4880zKy8sZMGBAa3GsIldEREQkt1QktyF46TkojBKZMv2Y3t/Q0MC6detYt24d+/bto6CggOHDhzNmzBhGjBhBWVmZimERERGRbkpFchpBQz3BK/9B5LwvEOk3sEPvra+vZ+XKlbz99tvEYjGGDx9OZWUln/3sZykpKclSxCIiIiKSSSqS0wjeeA0aDxC5YEa739Pc3Mybb77JypUricVijB49mnPOOafb3T1GRERERI5ORXIawWsvQdlgOH10u/pv376dJUuWsGPHDkaOHMnkyZMZNCg7q2GIiIiISPapSE4R7N4J764hcom1a87wunXrePnllyktLeXSSy9l5MiRXRCliIiIiGSTiuQUwZ+WQhAQmfjFI/aLx+MsXbqUNWvWcPLJJzN9+nR69erVRVGKiIiISDapSE4RvPYyjPwvRAYPbbNPPB5nyZIlvPfee4wfP57JkydTUFDQdUGKiIiISFapsksS27YFNn9E5Lzz2+wTj8dZvHgx7733HpMmTeL8889XgSwiIiLSw2gkOUnjyuUARM6pSrs/CAKWLVvGhg0b+MIXvsC5557bleGJiIiISBfREGiSxj8tg2GnECkfknb/mjVrWL16Neecc44KZBEREZEeTEVyKNhfS9P61UTGpR9F3rZtG8uWLWPkyJGcf37b0zFERERE5PinIjkUrH0d4s1Ezqk8bF9jYyOLFy+mT58+TJs2TbeTFhEREenh2jUn2cxmAPcDhcDP3f3ulP3XA/8CbAmbHnD3n4f7rgNuC9vvdPdfZiDujAveWkHBwDI45fTD9i1dupTa2louv/xySktLcxCdiIiIiHSloxbJZlYI/BSYBmwGVprZIndfn9J1obvPSXnvIGAuUAEEwKrwvbszEn0GRU4fTe8x4zmQslLFpk2beOedd6ioqOCkk07KUXQiIiIi0pXaM92iEtjg7h+6+0HgV8Bl7Tz+dGCJu+8KC+MlwIxjCzW7Cv76Mvp8+SuHtMViMV566SX69+9PZeXh0zBEREREpGdqz3SLYcCmpO3NQLqr2/6rmV0AvA/c5O6b2njvsHQnMbPZwGwAd6esrKwdoWVWNBo95LzLli1jz549XHvttQwZkn7FCzn+pOZZeiblOT8ozz2fcpwfumOe21Mkp7tKLUjZ/jfgKXdvNLMbgF8CF7bzvQC4+wJgQUufHTt2tCO0zCorKySG5UMAAAsNSURBVKPlvA0NDSxdupRTTz2VAQMGkIt4JDuS8yw9l/KcH5Tnnk85zg+5yvPQoW3fYbk9RfJmYETS9nCgOrmDu+9M2nwYuCfpvVNT3vtyO86Zc6tWraKxsZFJkyblOhQRERER6WLtKZJXAqPMbCSJ1SuuBK5O7mBmJ7n71nDzS8A74esXgO+Z2cBw+yLgW52OOsv279/PW2+9xRlnnNHthv5FREREJPuOeuGeu8eAOSQK3ncSTf62mc0zsy+F3f7BzN42s9XAPwDXh+/dBXyXRKG9EpgXtnVrq1evprm5WRfriYiIiOSpSBCknSKca0F1dfXRe2VYWVkZ1dXV/OIXv2D48OFccsklXR6DZJ/mt+UH5Tk/KM89n3KcH3I8JzntXeJ0x70U69evp7GxkXPPPTfXoYiIiIhIjqhIThKPx3nzzTcZOnSobhwiIiIiksdUJCf54IMPqK2tZezYsbkORURERERySEVykjfeeIPS0lJOO+20XIciIiIiIjmkIjlUX1/Pu+++yxlnnEE02p6V8URERESkp1KRHHr33Xdpbm5mzJgxuQ5FRERERHJMRXLoz3/+MyNGjODEE0/MdSgiIiIikmOaVxD68pe/TElJCd103WgRERER6UIaSQ4VFxdrFFlEREREABXJIiIiIiKHUZEsIiIiIpJCRbKIiIiISAoVySIiIiIiKVQki4iIiIikiHTTJc+6ZVAiIiIi0uNE0jV215HkSC4eZrYqV+fWQ3nWQ3nWQ3nWQznO10eO85xWdy2SRURERERyRkWyiIiIiEgKFcmHWpDrAKRLKM/5QXnOD8pzz6cc54dul+fueuGeiIiIiEjOaCRZRERERCRFNNcBdBdmNgO4HygEfu7ud+c4JDkGZjYCeBwYAsSBBe5+v5kNAhYCpwIfA+buu80sQiLvfwPUA9e7+xu5iF06zswKgdeBLe4+08xGAr8CBgFvANe4+0EzKyHxczEB2An8N3f/OEdhSweY2QDg58BZJJYH/SrwHvo89yhmdhPwdyRyvBb4W+Ak9Hk+rpnZo8BM4FN3Pyts6/D/j83sOuC28LB3uvsvuyJ+jSTT+j/anwIXA2cCV5nZmbmNSo5RDLjZ3UcDE4Ebw1zeAvyHu48C/iPchkTOR4WP2cDPuj5k6YT/AbyTtH0PcG+Y593ArLB9FrDb3U8H7g37yfHhfmCxu58BjCORb32eexAzGwb8A1ARFlKFwJXo89wTPAbMSGnr0Oc3LKrnAlVAJTDXzAZmPXJUJLeoBDa4+4fufpDEb66X5TgmOQbuvrXlN093ryXxP9RhJPLZ8pvnL4H/L3x9GfC4uwfu/howwMxO6uKw5RiY2XDgEhKjjISjEBcCT4ddUvPckv+ngb8K+0s3Zmb9gAuARwDc/aC770Gf554oCvQysyjQG9iKPs/HPXdfCuxKae7o53c6sMTdd7n7bmAJhxfeWaEiOWEYsClpe3PYJscxMzsVGA+sAAa7+1ZIFNLAZ8Juyv3x6z7gmySm1QCcCOxx91i4nZzL1jyH+/eG/aV7Ow3YDvzCzN40s5+bWR/0ee5R3H0L8APgExLF8V5gFfo891Qd/fzm7HOtIjkh3W+gWvbjOGZmfYHfAP/T3fcdoatyfxwys5Y5bquSmo+US+X5+BQFzgV+5u7jgf385U+z6SjPx6HwT+eXASOBoUAfEn96T6XPc8/WVl5zlm8VyQmbgRFJ28OB6hzFIp1kZkUkCuR/dfdnwuaalj+7hs+fhu3K/fFpMvAlM/uYxPSoC0mMLA8I/1wLh+ayNc/h/v4c/idA6X42A5vdfUW4/TSJolmf557lr4GP3H27uzcBzwCT0Oe5p+ro5zdnn2sVyQkrgVFmNtLMiklcMLAoxzHJMQjnpT0CvOPuP0ratQi4Lnx9HfC7pPZrzSxiZhOBvS1/BpLuy92/5e7D3f1UEp/XF939K8BLwOVht9Q8t+T/8rC/Rp66OXffBmwys8+FTX8FrEef557mE2CimfUO/w1vybM+zz1TRz+/LwAXmdnA8K8OF4VtWacl4EjMaTKzOSS+6YXAo+7+do7DkmMzGbgGWGtmb4VttwJ3A25ms0j8g3xFuO85EsvNbCCx5Mzfdm24kmH/DPzKzO4E3iS84Ct8fsLMNpAYcboyR/FJx30D+NdwAONDEp/RAvR57jHcfYWZPU1imbcYic/uAuD/os/zcc3MngKmAmVmtpnEKhUd+v+xu+8ys++SGNAEmOfuXfKXA91xT0REREQkhaZbiIiIiIikUJEsIiIiIpJCRbKIiIiISAoVySIiIiIiKVQki4iIiIikUJEsIpJjZlZnZqflOo7OMLOPzeyvcx2HiEimqEgWkbyWXNyZ2fVmtjzL53vZzP4uuc3d+7r7h9k8r4iIdIyKZBGRDEm6ha4cI30PRaS70M1ERCSvmdnHwN8BW0jc1asIOADE3H2AmZUAdwEGlADPAje5+wEzmwr8/8BPgJuAJcA/AE8AVSTuavoKcIO7bzazu4BbgCYSdxZ7zN3nmFkAjHL3DWbWPzzexSTuOvUw8D13j5vZ9WGsrwGzgD3Af3f354/wtT0AXAucAiwGrnP3hpZjufsXkvonx/FYeP6RwPnAauC/hvFfB9QAV7n7m0nnmk/ijpcnAb8F/t7dG8L9M4E7gVNJ3HL4Bndfk/TenwFfAT4H9HH3WNtZExHJPo0ki4gA7v4OcAPwx3D6w4Bw1z3AfwHOAU4HhgG3J711CDCIRBE6m8S/q78It08mUXA/EJ7jfwPLgDnhOeakCeUnQH/gNGAKiQI3+fbKVcB7QBnwf4BHzCxyhC/NgBkkit2xwPVH+Vakvve28FyNwB9J3Dq4DHga+FFK/68A04HPkvie3QZgZucCjwJfB04kUUwvCn8BaXEVcAkwQAWyiHQH+rOWiEgbwuLza8BYd98Vtn0PeBL4VtgtDsx198Zw+wDwm6Rj3AW81M7zFQL/DRjv7rVArZn9kMTo7CNht43u/nDY/5fAg8BgYFsbh/2xu1eH/f+NRLHfXs+6+6rwvc+SGLV+PNxeCKQW+Q+4+6Zw/10kCv7bSHwP57v7irDfL83sVmAi8J9JcW7qQGwiIlmlIllEpG3lQG9glZm1tEWAwqQ+21umFACYWW/gXhKjtwPD5hPMrNDdm49yvjKgGNiY1LaRxOh1i9Zi2N3rw7j6HuGYycVzPTD0KDEkq0l6fSDNdup5k4vcjUnnOgW4zsy+kbS/OCUWFcgi0q2oSBYR+YvUizR2kCgGx7j7lna+52YS82qr3H2bmZ1DYq5zpI3+qedrIlFUrg/bTiYxXzrT9pP4BQAAMxuSgWOOSHp9MlAdvt4E3OXudx3hvbpARkS6FRXJIiJ/UQMMN7Nidz8YXiz3MHCvmc1x90/NbBhwlru/0MYxTiBRWO8xs0HA3DTnSLsmsrs3m5kDd5nZtSTmOv8j8IMMfG2pVgNjwiL+XeA7GTjmjWb2exIj1rcCC8P2h4FnzewPwJ9IFOdTgaXhtBIRkW5HF+6JiPzFi8DbwDYz2xG2/TOwAXjNzPYBfyAxUtyW+4BeJEaFXyOxokSy+4HLzWy3mf04zfu/QWKU90NgOYn5z48e25fTNnd/H5hH4uv5c3iuznoS+HcSsX9IYjUL3P11EvOSHwB2k/h+Xp+B84mIZI2WgBMRERERSaGRZBERERGRFCqSRURERERSqEgWEREREUmhIllEREREJIWKZBERERGRFCqSRURERERSqEgWEREREUmhIllEREREJIWKZBERERGRFP8PTqVwv4H2XG8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "s0 = 2                          # Define the input,\n",
    "target = 0.7                    # ... and the target output.\n",
    "\n",
    "w0 = 2                          # Choose initial values for the weights.\n",
    "w1 = 1\n",
    "w2 = 0.5\n",
    "\n",
    "alpha = 0.1                     # Set the learning constant.\n",
    "\n",
    "def sigmoid(x):\n",
    "    return 1/(1+exp(-x))     # Define the sigmoid anonymous function.\n",
    "\n",
    "results = zeros([1000,4])    # Define a variable to hold the results of each iteration.    \n",
    "for k in range(1000):           # For 1000 iteractions,\n",
    "    \n",
    "    #Step 2. Calculate feedforward solution to get output.\n",
    "    x1 = w0*s0                  # ... activity of first neuron,\n",
    "    s1 = sigmoid(x1)            # ... output of first neuron,\n",
    "    x2 = w1*s1                  # ... activity of second neuron,\n",
    "    s2 = sigmoid(x2)            # ... output of second neuron,\n",
    "    out= w2*s2                  # Output of neural network.\n",
    "    \n",
    "    #Step 3. Update the weights.\n",
    "    w2 = w2 - alpha*(out-target)*s2\n",
    "    w1 = w1 - alpha*(out-target)*(w2*s2*(1-s2)*s1)\n",
    "    w0 = w0 - alpha*(out-target)*(w2*s2*(1-s2)*w1)*(s1*(1-s1)*s0)\n",
    "    \n",
    "    # Save the results of this step! --------------------------------------\n",
    "    results[k,:] = [w2,w1,w0,  out]\n",
    "    # Here we save the 3 weights, the neural network output.\n",
    "\n",
    "clf()\n",
    "plot(results[:,0], label='w2')\n",
    "plot(results[:,1], label='w1')\n",
    "plot(results[:,2], label='w0')\n",
    "plot(results[:,3], label='out')\n",
    "plot(arange(1000), 1000*[target], 'k', label='target')  #... and plot the *target*.\n",
    "legend() #Iclude a legend,\n",
    "xlabel('Iteration number');         #... and axis label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"donate\"></a>\n",
    "## Donate\n",
    "If you enjoy Case-Studies-Python, and would like to share your enjoyment with us, sponsor our coffee consuption <a href=\"https://www.paypal.com/donate/?hosted_button_id=DL8P5ZGS9962U\">here</a>."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md:myst"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}