04.gradient_descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Learning: Gradient Descent\n",
    "\n",
    "In this Chapter, we will:\n",
    "\n",
    "- Check if neural network can make accurate predictions.\n",
    "- Investigate the importance of measuring error.\n",
    "- Define **Hot** and **Cold** learning.\n",
    "- Calculate the following \"learning\" measures: \"direction\" & \"emount\", from the error. \n",
    "- Examine **Gradient Descent**.\n",
    "- Learn about derivatives and how to use them to learn.\n",
    "- Examine divergance and the learning rate (alpha).\n",
    "\n",
    "> [Milton Friedman] \"The only relevant test of the validity of a hypothesis is comparison of prediction with experience.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict, Compare, & Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We surely came out of the last chapter asking the following question: **How do we set weight values so the network predicts accurately?**.\n",
    "\n",
    "Answering this question is the main focus of this chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have made a prediction, the next step is to evaluate how well we did. Coming up with a good way to measure error is one of the most **important** and **complicated** subjects in the fild of representation learning and machine learning in general.\n",
    "\n",
    "We make one assumption, **error is always positive**, a negative error would not make sense because we assume that a prediction can not be any more \"truer\" than the given label or numerical target. An error measure gives an estimate of how much a prediction \"missed\" by. \n",
    "\n",
    "The output of the \"Compare\" step is a \"Hot or Cold\" type Signal. Given some prediction, we will calculate an error measure that says either \"A Lot\" or \"A Little\". It will more or less say \"Big miss\", \"Little miss\", or \"Perfect prediction\".     \n",
    "\n",
    "In this chapter, we evaluate only one simple way of measuring error: `mean squared error`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning, which comes next, tells each weight how it can change to reduce the error signal. So learning is all about **error attribution** (or how each weight played its part in creating error).\n",
    "\n",
    "The goal of the `learn` step is to compute an \"update\" value for each weight."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare: Does our Network make good predictions?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's measure the error and find out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.30250000000000005\n"
     ]
    }
   ],
   "source": [
    "knob_weight = .5\n",
    "x0 = .5\n",
    "\n",
    "# A number you recorded in the real world somewhere.\n",
    "goal_prediction = .8\n",
    "\n",
    "pred = knob_weight * x0\n",
    "\n",
    "# mean squared error over one data point.\n",
    "error = (pred - goal_prediction) ** 2  # ^2 will ignore very small errors and will amplify big errors [negative or positive]\n",
    "\n",
    "print(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We square the errors not only to force the output to be positive, but also to amplify big errors and reduce small ones which proved to be just \"OK\". We'd have the network pay attention to the big errors and not worry so much about the small ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Measure Error ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measuring error simplifies the problem, changing `knob_weight` to make the network correctly predict `goal_prediction` is slightly more complicated than changing `knob_weight` to make `error == 0` (which can be categorized as a **minimization** problem).\n",
    "\n",
    "We also stress that different ways of measuring error prioritize error differently. As mentioned in the previous example, while formulating the loss/error function, we will prioritize big error by amplifying them and ignoring small errors by diminishing them. If we took the `absolute value` instead of squaring the error, we wouldn't have this type of prioritization.\n",
    "\n",
    "On the other hand, by forcing errors to be positive, we try to take the average error down to zero. If errors are allowed to be negative then the average of `err1 = -E` and `err2 = +E` is zero. We would fool ourserves into thinking that we've predicted perfectly, when in fact we had missed by an average of `E`. In summary, **we want errors to be positive so that they do not cancel each other out**.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's the Simplest form of Neural Learning ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest form of neural learning is using the hot and cold method. We know whether to turn the knob up or down by trying both and seeing which one reduces the error. \n",
    "\n",
    "Hot & Cold learning wiggles the weights to see which direction reduces the error the most, moving the weights in that direction, and repeating until the error gets to **0**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. An Empty Network.\n",
    "weight = .1\n",
    "lr = .01\n",
    "\n",
    "def neural_network(X, W):\n",
    "    prediction = X * W\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.022499999999999975\n"
     ]
    }
   ],
   "source": [
    "# 1. PREDICT: Making a Prediction & Evaluating Error.\n",
    "number_of_toes = [8.5]\n",
    "win_or_loss_binary = [1]\n",
    "\n",
    "input = number_of_toes[0]\n",
    "true = win_or_loss_binary[0]\n",
    "\n",
    "pred = neural_network(input, weight)\n",
    "\n",
    "error = (pred - true) ** 2  # ^2 forces the error to be positive.\n",
    "print(error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.004224999999999993\n"
     ]
    }
   ],
   "source": [
    "# 2. COMPARE: Making a Prediction with a higher weight and evaluating error.\n",
    "p_up = neural_network(input, weight + lr)\n",
    "\n",
    "e_up = (p_up - true) ** 2 \n",
    "\n",
    "print(e_up)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.05522499999999994\n"
     ]
    }
   ],
   "source": [
    "# 3. COMPARE: Making a Prediction with a lower weight and evaluating error.\n",
    "p_dn = neural_network(input, weight - lr)\n",
    "\n",
    "e_dn = (p_dn - true) ** 2\n",
    "\n",
    "print(e_dn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. COMPARE ERROR & SET NEW WEIGHT\n",
    "if error > e_up or error > e_down:\n",
    "    if e_up < e_dn:\n",
    "        weight += lr\n",
    "    else:\n",
    "        weight =- lr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These last five steps represent only one iteration of hot and cold learning. Some people have to train their networks for **weeks or months** before they find a good enough **weight configuration**.\n",
    "\n",
    "This reveals that learning in neural networks really is a **Search Problem**. We are searching for the best possible weight configuration that results in the lowest error possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hot & Cold Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement iterative hot & cold learning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.302 | Prediction: 0.251\n",
      "Error: 0.2911 | Prediction: 0.261\n",
      "Error: 0.2804 | Prediction: 0.271\n",
      "Error: 0.2699 | Prediction: 0.281\n",
      "Error: 0.2596 | Prediction: 0.291\n",
      "Error: 0.2495 | Prediction: 0.301\n",
      "Error: 0.2396 | Prediction: 0.311\n",
      "Error: 0.2299 | Prediction: 0.321\n",
      "Error: 0.2204 | Prediction: 0.331\n",
      "Error: 0.2111 | Prediction: 0.341\n",
      "Error: 0.2021 | Prediction: 0.351\n",
      "Error: 0.1932 | Prediction: 0.361\n",
      "Error: 0.1845 | Prediction: 0.371\n",
      "Error: 0.176 | Prediction: 0.381\n",
      "Error: 0.1677 | Prediction: 0.391\n",
      "Error: 0.1596 | Prediction: 0.401\n",
      "Error: 0.1517 | Prediction: 0.411\n",
      "Error: 0.144 | Prediction: 0.421\n",
      "Error: 0.1365 | Prediction: 0.431\n",
      "Error: 0.1292 | Prediction: 0.441\n",
      "Error: 0.1222 | Prediction: 0.451\n",
      "Error: 0.1153 | Prediction: 0.461\n",
      "Error: 0.1086 | Prediction: 0.471\n",
      "Error: 0.1021 | Prediction: 0.481\n",
      "Error: 0.0958 | Prediction: 0.491\n",
      "Error: 0.0897 | Prediction: 0.501\n",
      "Error: 0.0838 | Prediction: 0.51\n",
      "Error: 0.0781 | Prediction: 0.52\n",
      "Error: 0.0726 | Prediction: 0.53\n",
      "Error: 0.0673 | Prediction: 0.54\n",
      "Error: 0.0623 | Prediction: 0.55\n",
      "Error: 0.0574 | Prediction: 0.56\n",
      "Error: 0.0527 | Prediction: 0.57\n",
      "Error: 0.0482 | Prediction: 0.58\n",
      "Error: 0.0439 | Prediction: 0.59\n",
      "Error: 0.0398 | Prediction: 0.6\n",
      "Error: 0.0359 | Prediction: 0.61\n",
      "Error: 0.0322 | Prediction: 0.62\n",
      "Error: 0.0287 | Prediction: 0.63\n",
      "Error: 0.0254 | Prediction: 0.64\n",
      "Error: 0.0224 | Prediction: 0.65\n",
      "Error: 0.0195 | Prediction: 0.66\n",
      "Error: 0.0168 | Prediction: 0.67\n",
      "Error: 0.0143 | Prediction: 0.68\n",
      "Error: 0.012 | Prediction: 0.69\n",
      "Error: 0.0099 | Prediction: 0.7\n",
      "Error: 0.008 | Prediction: 0.71\n",
      "Error: 0.0063 | Prediction: 0.72\n",
      "Error: 0.0048 | Prediction: 0.73\n",
      "Error: 0.0035 | Prediction: 0.74\n",
      "Error: 0.0025 | Prediction: 0.75\n",
      "Error: 0.0016 | Prediction: 0.76\n",
      "Error: 0.0009 | Prediction: 0.77\n",
      "Error: 0.0004 | Prediction: 0.78\n",
      "Error: 0.0001 | Prediction: 0.79\n"
     ]
    }
   ],
   "source": [
    "weight = .5\n",
    "input = .5\n",
    "goal_prediction = .8\n",
    "\n",
    "step_amount = .001\n",
    "\n",
    "for iteration in range(10000):\n",
    "    prediction = input * weight\n",
    "    \n",
    "    # calculate error.\n",
    "    error = (prediction - goal_prediction) ** 2\n",
    "    \n",
    "    # try knob up.\n",
    "    p_up = input * (weight + step_amount)\n",
    "    error_up = (p_up - goal_prediction) ** 2\n",
    "    \n",
    "    # try knob down.\n",
    "    p_dn = input * (weight - step_amount)\n",
    "    error_dn = (p_dn - goal_prediction) ** 2\n",
    "    \n",
    "    if error > error_up or error > error_dn:\n",
    "        if error_up < error_dn:\n",
    "            weight += step_amount\n",
    "            if iteration % 20 == 0:\n",
    "                print(f\"Error: {round(error_up, 4)} | Prediction: {round(input * weight, 3)}\")\n",
    "        else:\n",
    "            weight -= step_amount\n",
    "            if iteration % 20 == 0:\n",
    "                print(f\"Error: {round(error_dn, 4)} | Prediction: {round(input * weight, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Characteristics of Hot & Cold Learning\n",
    "### It's Simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We list the following characteristics of Hot & Cold learning:\n",
    "\n",
    "- It is simple: we only have to change the weight and compare.\n",
    "- It is inefficient: we have to predict multiple times to make a single knob update, this seems very inefficient.\n",
    "- We have to guess the step size: even though we know the correct direction to move, we won't know the correct update value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating Both Direction & Amount from Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the **gradient** as a measure of direction and amount, we know the following:\n",
    "\n",
    "$$\n",
    "f(W) = (WX - \\hat{y})^2 \\\\\n",
    "\\frac{\\partial f}{\\partial W} = 2(y - \\hat{y})X\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.3025 | Gradient: -0.275 | Prediction: 0.388\n",
      "Error: 0.170156 | Gradient: -0.206 | Prediction: 0.491\n",
      "Error: 0.095713 | Gradient: -0.155 | Prediction: 0.568\n",
      "Error: 0.053839 | Gradient: -0.116 | Prediction: 0.626\n",
      "Error: 0.030284 | Gradient: -0.087 | Prediction: 0.669\n",
      "Error: 0.017035 | Gradient: -0.065 | Prediction: 0.702\n",
      "Error: 0.009582 | Gradient: -0.049 | Prediction: 0.727\n",
      "Error: 0.00539 | Gradient: -0.037 | Prediction: 0.745\n",
      "Error: 0.003032 | Gradient: -0.028 | Prediction: 0.759\n",
      "Error: 0.001705 | Gradient: -0.021 | Prediction: 0.769\n",
      "Error: 0.000959 | Gradient: -0.015 | Prediction: 0.777\n",
      "Error: 0.00054 | Gradient: -0.012 | Prediction: 0.783\n",
      "Error: 0.000304 | Gradient: -0.009 | Prediction: 0.787\n",
      "Error: 0.000171 | Gradient: -0.007 | Prediction: 0.79\n",
      "Error: 9.6e-05 | Gradient: -0.005 | Prediction: 0.793\n",
      "Error: 5.4e-05 | Gradient: -0.004 | Prediction: 0.794\n",
      "Error: 3e-05 | Gradient: -0.003 | Prediction: 0.796\n",
      "Error: 1.7e-05 | Gradient: -0.002 | Prediction: 0.797\n",
      "Error: 1e-05 | Gradient: -0.002 | Prediction: 0.798\n",
      "Error: 5e-06 | Gradient: -0.001 | Prediction: 0.798\n"
     ]
    }
   ],
   "source": [
    "weight = .5\n",
    "goal_pred = .8\n",
    "input = .5\n",
    "\n",
    "for iteration in range(20):\n",
    "    pred = input * weight\n",
    "    error = (pred - goal_pred) ** 2\n",
    "    direction_and_amount = (pred - goal_pred) * input  # pure error * scale.\n",
    "    weight -= direction_and_amount\n",
    "    print(f\"Error: {round(error, 6)} | Gradient: {round(direction_and_amount, 3)} | Prediction: {round(input * weight, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we see here is a superior form of learning known as **gradient descent**. GD allows us to calculate both the direction and amount we should change the weight by to reduce error.\n",
    "\n",
    "We use scaling, negative reversal, & stopping to translate the pure error into the absolute amount we want to change the weight. The stopping criteria, stops learning if `error` is close enough to zero, making the gradient also close to zero. Scaling, on the other hand, would change the updates to grow as bit as the inputs or/and errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One Iteration of Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will perform a weight update on a single training example (`Input -> true`) pair:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. an empty network.\n",
    "weight = .1\n",
    "alpha = .01\n",
    "\n",
    "def neural_network(X, W):\n",
    "    prediction = X * W\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.022499999999999975\n"
     ]
    }
   ],
   "source": [
    "# 2. PREDICT: Making a Prediction & Evaluating Error.\n",
    "number_of_toes = [8.5]\n",
    "win_or_lose_binary = [1]\n",
    "input = number_of_toes[0]\n",
    "goal_prediction = win_or_lose_binary[0]\n",
    "\n",
    "pred = neural_network(input, weight)\n",
    "\n",
    "error = (pred - goal_prediction) ** 2\n",
    "\n",
    "print(error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.1499999999999999\n"
     ]
    }
   ],
   "source": [
    "# 3. COMPARE: Calculating the Node `Delta` and putting it on the output node.\n",
    "delta = pred - goal_prediction  # pure error.\n",
    "print(delta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delta is a measure of how much the node has missed. In the previous example, the true prediction was `1.0` & the network's prediction was `.85`, so the network was too low by `0.15`. This is why delta is `-0.15`.\n",
    "\n",
    "The primary difference between gradient descent & this implementation is the new variable *delta*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. LEARN\n",
    "number_of_toes = [8.5]\n",
    "win_or_lose_binary = [1]\n",
    "\n",
    "input = number_of_toes[0]\n",
    "goal_pred = win_or_lose_binary[0]\n",
    "\n",
    "pred = neural_network(input, weight)\n",
    "error = (pred - goal_pred) ** 2\n",
    "delta = pred - goal_pred\n",
    "\n",
    "weight_delta = input * delta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. LEARN: Updating the weight\n",
    "alpha = .01  # learning step.\n",
    "weight -= alpha * weight_delta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Alpha` lets us control how fast the network learns. If it learns too fast, it can update weights too aggressively & overshoot. If it learns too slow, it will spend a lot of time & computing resources to converge to the optimal weight configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning is Just Reducing Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can modify weights to reduce error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.64 | Prediction: 0.0\n",
      "Error: 0.36 | Prediction: 0.2\n",
      "Error: 0.2025 | Prediction: 0.35\n",
      "Error: 0.11391 | Prediction: 0.463\n",
      "Error: 0.06407 | Prediction: 0.547\n"
     ]
    }
   ],
   "source": [
    "# putting it all together.\n",
    "weight, goal_prediction, input = (0, .8, .5)\n",
    "\n",
    "for iteration in range(5):\n",
    "    pred = input * weight\n",
    "    error = (pred - goal_prediction) ** 2\n",
    "    delta = pred - goal_prediction\n",
    "    gradient = delta * input\n",
    "    weight -= gradient\n",
    "    print(f\"Error: {round(error, 5)} | Prediction: {round(pred, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we are trying to do is figure out the **right direction and amount** to modify *weight* so that error goes down. For any `input` and `goal_predict`, we can find an exact mapping between network **weights** and the resulting **error**. We call this mapping: the **Loss Function**:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:400px\" src=\"static/imgs/04/Loss_function.png\" /></div>\n",
    "\n",
    "The slope (which is the gradient) points to the bottom of the bowel (lowest error) no matter where we are in the bowel. We can use this slope to help the neural network reduce the error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's Watch several steps of Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to reach the bottom of the bowel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.64 | Prediction: 0.0 | Weight: 0.88 | Gradient: -0.88\n",
      "Error: 0.02822 | Prediction: 0.968 | Weight: 0.6952 | Gradient: 0.1848\n",
      "Error: 0.00124 | Prediction: 0.76472 | Weight: 0.73401 | Gradient: -0.03881\n",
      "Error: 5e-05 | Prediction: 0.80741 | Weight: 0.72586 | Gradient: 0.00815\n"
     ]
    }
   ],
   "source": [
    "weight, goal_prediction, input = 0, .8, 1.1\n",
    "\n",
    "for iteration in range(4):\n",
    "    prediction = input * weight\n",
    "    error = (prediction - goal_prediction) ** 2\n",
    "    delta = prediction - goal_prediction\n",
    "    gradient = input * delta\n",
    "    weight -= gradient\n",
    "    print(f\"Error: {round(error, 5)} | Prediction: {round(prediction, 5)} | Weight: {round(weight, 5)} | Gradient: {round(gradient, 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the above steps:\n",
    "\n",
    "<div style=\"text-align:center;\">\n",
    "    <img style=\"width:200px\" src=\"static/imgs/04/step-0.png\" />\n",
    "    <img style=\"width:200px\" src=\"static/imgs/04/step-1.png\" />\n",
    "    <img style=\"width:200px\" src=\"static/imgs/04/step-2.png\" />\n",
    "    <img style=\"width:200px\" src=\"static/imgs/04/step-3.png\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Does this Work?\n",
    "### Let's back up & talk about functions. What is a Function & How do you understand One? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's back up & talk about functions. What is a function and how do we understand it?\n",
    "\n",
    "We start by analyzing the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_function(x):\n",
    "    return x * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`my_function` defines a relationship between the input number `x` and the output `my_function(x)`. Functions are powerful because they can take numbers (say pixels) and convert them into other numbers (the probability that the image contains a cat).\n",
    "\n",
    "Each function has what we call \"moving parts\". Moving parts are the pieces we can tweak so that the function generates different outputs. The moving parts of a loss function $J_{W}(\\hat{y}, y)$ are its general formula and the weights. We can say the same about neural networks.\n",
    "\n",
    "Now let's study what we can change to drive the overall error down to zero:\n",
    "- Changing the input: this would be the same as seeing the world as you want to see it instead of as it actually is, which is a bad idea.\n",
    "- Changing the formula of $J(w)$: the error calculation is meaningless if it doesn't actually give a good measure of how much we missed. So a fake formula that always gives zero won't solve our problem.\n",
    "- Changing the weights\n",
    "    - Doesn't change our perception of the world.\n",
    "    - Doesn't change our goal.\n",
    "    - Doesn't change our error measure.\n",
    "    \n",
    "Changing the weights means that the function conforms to the patterns of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tunnel Vision on one Concept"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We consider learning as the process of adjusting the weight to reduce the error down to **0**. For us to be able to do that, **we need to understand how changing the weights changes the error**. Specifically, we want to know the direction and amount that error changes when weight changes.\n",
    "\n",
    "Let's go back to the loss function formulation:\n",
    "\n",
    "$$J(w)=(w*x - y)^2$$\n",
    "\n",
    "We can solve this problem since this relationship is deterministic, computable, and continuous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Box with Rods poking out of it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing a derivative can be as simple as answering the following question: \"When I tug this part, how much this other part move?\".\n",
    "\n",
    "We always have the derivative between two variables. When the derivative is positive, it means that when we change one variable, the other will move in the same direction. If the derivative is negative, then when we change one variable, the other will move in the opposite direction.\n",
    "\n",
    "The derivative represents the direction and the amount that one variable changes if we change the other variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What You Really Need to Know\n",
    "### With derivatives, you can take any two variables in any formula, and know how they interact"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With derivatives, we can take any two variables in any function, and know how they interract. A neural network is just a function that consists of a bunch of weights we use to compute an output that gets used by the error or loss function.\n",
    "\n",
    "So, for any error function, we can compute the relationship between any weight and the final error of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you Don't Really Need to Know: Calculus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Us, practitioners, don't really need to know about calculus. Calculus is just about memorizing and practicing every possible derivative rule for every possible function. We just need to remember this: \"the derivative between two variables represent the sensitivity between them\", meaning:\n",
    "- In which direction one variable moves when we move the other variable.\n",
    "- How much one variable changes when we change the other variable.\n",
    "\n",
    "In the context of our error function, reaching zero sensitivity points us to a local minimum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use a derivative to learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The slope of a line or curve always points in the opposite direction of the curve's lowest point. To minimize error, we need to move in the opposite direction of the slope.\n",
    "\n",
    "We can take each weight value, calcuate its derivative with respect to the loss function and then change the weight in the opposite direction of that slope. This method of learning (finding error minimums) is called **Gradient descent**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Breaking Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement the gradient descent algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# weight, goal_prediction, input, lr = .5, .8, .5, .5\n",
    "weight, goal_prediction, input, lr = .5, .8, 2, .5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: 0.04 | GRADIENT: 0.8 | PREDICTION: 0.2\n",
      "ERROR: 0.36 | GRADIENT: -2.4 | PREDICTION: 2.6\n",
      "ERROR: 3.24 | GRADIENT: 7.2 | PREDICTION: -4.6\n",
      "ERROR: 29.16 | GRADIENT: -21.6 | PREDICTION: 17.0\n",
      "ERROR: 262.44 | GRADIENT: 64.8 | PREDICTION: -47.8\n",
      "ERROR: 2361.96 | GRADIENT: -194.4 | PREDICTION: 146.6\n",
      "ERROR: 21257.64 | GRADIENT: 583.2 | PREDICTION: -436.6\n",
      "ERROR: 191318.76 | GRADIENT: -1749.6 | PREDICTION: 1313.0\n",
      "ERROR: 1721868.84 | GRADIENT: 5248.8 | PREDICTION: -3935.8\n",
      "ERROR: 15496819.56 | GRADIENT: -15746.4 | PREDICTION: 11810.6\n",
      "ERROR: 139471376.04 | GRADIENT: 47239.2 | PREDICTION: -35428.6\n",
      "ERROR: 1255242384.36 | GRADIENT: -141717.6 | PREDICTION: 106289.0\n",
      "ERROR: 11297181459.24 | GRADIENT: 425152.8 | PREDICTION: -318863.8\n",
      "ERROR: 101674633133.15994 | GRADIENT: -1275458.4 | PREDICTION: 956594.6\n",
      "ERROR: 915071698198.4395 | GRADIENT: 3826375.2 | PREDICTION: -2869780.6\n"
     ]
    }
   ],
   "source": [
    "for iteration in range(15):\n",
    "    error = ((input * weight) - goal_prediction) ** 2\n",
    "    gradient = 2 * (input * weight - goal_prediction) * input\n",
    "    weight -= gradient * lr\n",
    "    print(f\"ERROR: {round(error, 5)} | GRADIENT: {round(gradient, 5)} | PREDICTION: {round(input * weight, 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we set the input to 2, learning overshoots because the input is part of the gradient, with a large input, the gradient will be big and the new weight will overshoot. All in all, the predictions exploded and got further away from the true answer at every step.\n",
    "\n",
    "A simple solution to this problem is to introduce a learning rate and find the optimal value to scale the effects of the gradient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Divergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center;\"><img style=\"width:400px\" src=\"static/imgs/04/divergence.png\" /></div>\n",
    "\n",
    "The explosion in the error was caused by the fact that we have made the norm of the weight larger, which is a direct result of having large weight updates and small errors.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducting $\\alpha$: the Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learning rate is the simplest way to prevent overcorrecting weight updates.\n",
    "\n",
    "Divergance comes when the new derivative is even larger in magnitude than when we started. The solution to this is to multiply the weight update by a fraction to make it smaller. This envolves multipliying the weight update by a single real-valued number between $0$ and $1$. Finding the appropriate $\\alpha$, even for state-of-the-art neural networks, is often done by guessing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha in Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add a learning rate to the gradient descent algorithm: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: 0.04 | PREDICTION: 1.0\n",
      "ERROR: 0.0016 | PREDICTION: 0.84\n",
      "ERROR: 6e-05 | PREDICTION: 0.808\n",
      "ERROR: 0.0 | PREDICTION: 0.8016\n",
      "ERROR: 0.0 | PREDICTION: 0.80032\n",
      "ERROR: 0.0 | PREDICTION: 0.80006\n",
      "ERROR: 0.0 | PREDICTION: 0.80001\n"
     ]
    }
   ],
   "source": [
    "# implementation.\n",
    "weight, goal_pred, input, alpha = .5, .8, 2, .1\n",
    "\n",
    "for iteration in range(7):\n",
    "    prediction = input * weight\n",
    "    error = (prediction - goal_pred) ** 2\n",
    "    gradient = input * 2 * (prediction - goal_pred)\n",
    "    weight = weight - alpha * gradient\n",
    "    print(f\"ERROR: {round(error, 5)} | PREDICTION: {round(prediction, 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Coming up with a value for the `learning rate` is a matter of **guessing**. Despite all the crazy advancements of deep learning in the past few years, most people just try several orders of magnitudes of alpha (10, 1, .1, .01, .001, ..) and tweak the value from there to see what works best. Generally speaking, neural network design is more art than science. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memorizing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's re-code the algorithm on our own:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, w, y_hat, lr = .1, .3, 1, 10  # tested .1, .3, 1, & 10 worked as a learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: 0.04138 | WEIGHT: 8.37261 | PREDICTION: 0.79658\n",
      "ERROR: 0.02648 | WEIGHT: 8.69809 | PREDICTION: 0.83726\n",
      "ERROR: 0.01695 | WEIGHT: 8.95847 | PREDICTION: 0.86981\n",
      "ERROR: 0.01085 | WEIGHT: 9.16678 | PREDICTION: 0.89585\n",
      "ERROR: 0.00694 | WEIGHT: 9.33342 | PREDICTION: 0.91668\n",
      "ERROR: 0.00444 | WEIGHT: 9.46674 | PREDICTION: 0.93334\n",
      "ERROR: 0.00284 | WEIGHT: 9.57339 | PREDICTION: 0.94667\n"
     ]
    }
   ],
   "source": [
    "for iteration in range(7):\n",
    "    pred = x * w\n",
    "    error = (pred - y_hat) ** 2\n",
    "    gradient = 2 * x * (pred - y_hat)\n",
    "    w = w - lr * gradient\n",
    "    print(f\"ERROR: {round(error, 5)} | WEIGHT: {round(w, 5)} | PREDICTION: {round(pred, 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sketches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center;\">\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/Loss_function.jpg\" />\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/derivatives.jpg\" /><br>\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/derivatives_2.jpg\" />\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/dependence.jpg\" /><br>\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/model_and_loss.jpg\" />\n",
    "    <img style=\"width:333px\" src=\"static/imgs/04/optimization.jpg\" />\n",
    "</div>\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}