001-neural-nets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single-layered neural net is simply a set of weights "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://raw.githubusercontent.com/arokem/conv-nets/master/img/nn-1.png' width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement this in code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x11 = 1 \n",
    "x12 = 2\n",
    "w_2_11 = -2\n",
    "w_2_21 = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x21 = w_2_11 * x11 + w_2_21 * x12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x21)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x21 = np.dot([w_2_11, w_2_21], [x11, x12])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x21)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A multi-layered network will add to that another set of weights: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://raw.githubusercontent.com/arokem/conv-nets/master/img/nn-2.png' width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x11 = 1 \n",
    "x12 = 2\n",
    "\n",
    "w_2_11 = -2\n",
    "w_2_21 = 3\n",
    "w_2_12 = 2\n",
    "w_2_22 = -3\n",
    "\n",
    "w_3_11 = 3\n",
    "w_3_21 = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x21 = np.dot([w_2_11, w_2_21], [x11, x12])\n",
    "x22 = np.dot([w_2_12, w_2_22], [x11, x12])\n",
    "\n",
    "x31 = np.dot([w_3_11, w_3_21], [x21, x22])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things get more interesting when an activation function is added to each unit:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://raw.githubusercontent.com/arokem/conv-nets/master/img/nn-3.png' width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different functions that are used include the hyperbolic tangent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(-np.pi, np.pi, 0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x, np.tanh(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another function that has been used a lot, for both convenience, and because it induces sparsity, is a rectified linear unit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x, np.max([x, np.zeros(x.shape[0])], axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x21 = np.max([np.dot([w_2_11, w_2_21], [x11, x12]), 0])\n",
    "x22 = np.max([np.dot([w_2_12, w_2_22], [x11, x12]), 0])\n",
    "\n",
    "x31 = np.max([np.dot([w_3_11, w_3_21], [x21, x22]), 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Networks are trained through gradient descent: gradual changes to the values of the weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gradients are calculate through **backpropagation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Error is propagated back through the network to calculate a gradient (derivative) for each weight by multiplying:\n",
    "\n",
    "- The gradient of the loss function with respect to the node a weight feeds into\n",
    "- The value of the node feeding into the weight\n",
    "- Ths slope of the activation function of the node it feeds into\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, for the network we had above, let's assume the desired output was 10, instead of 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We take the simplest possible error, the absolute difference:\n",
    "e31 = x31 - 10\n",
    "\n",
    "# We'll use this helper function to derive ReLU functions:\n",
    "def d_relu(x):\n",
    "    if x > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "e_3_11 = e31 * x21 * d_relu(x31)\n",
    "e_3_21 = e31 * x22 * d_relu(x31)\n",
    "\n",
    "e_2_11 = e_3_11 * x11 * d_relu(x21)\n",
    "e_2_21 = e_3_11 * x12 * d_relu(x21)\n",
    "\n",
    "e_2_12 = e_3_21 * x11 * d_relu(x22)\n",
    "e_2_22 = e_3_21 * x12 * d_relu(x22)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_3_11 = w_3_11 - e_3_11 * lr \n",
    "w_3_21 = w_3_11 - e_3_21 * lr \n",
    "\n",
    "w_2_11 = w_2_11 - e_2_11 * lr\n",
    "w_2_12 = w_2_12 - e_2_12 * lr\n",
    "\n",
    "w_2_21 = w_2_21 - e_2_21 * lr\n",
    "w_2_22 = w_2_22 - e_2_22 * lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x21 = np.max([np.dot([w_2_11, w_2_21], [x11, x12]), 0])\n",
    "x22 = np.max([np.dot([w_2_12, w_2_22], [x11, x12]), 0])\n",
    "\n",
    "x31 = np.max([np.dot([w_3_11, w_3_21], [x21, x22]), 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we'll see next, depending on the error function that is used, neural nets can be used for other tasks as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}