t81_558_class_03_1_neural_net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* **Part 3.1: Deep Learning and Neural Network Introduction** [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.1: Deep Learning and Neural Network Introduction\n",
    "\n",
    "Neural networks were one of the first machine learning models. Their popularity has fallen twice and is now on its third rise. Deep learning implies the use of neural networks. The \"deep\" in deep learning refers to a neural network with many hidden layers. Because neural networks have been around for so long, they have quite a bit of baggage. Researchers have created many different training algorithms, activation/transfer functions, and structures. This course is only concerned with the latest, most current state-of-the-art techniques for deep neural networks. I will not spend much time discussing the history of neural networks. \n",
    "\n",
    "Neural networks accept input and produce output. The input to a neural network is called the feature vector. The size of this vector is always a fixed length. Changing the size of the feature vector usually means recreating the entire neural network. Though the feature vector is called a \"vector,\" this is not always the case. A vector implies a 1D array. Later we will learn about convolutional neural networks (CNNs), which can allow the input size to change without retraining the neural network. Historically the input to a neural network was always 1D. However, with modern neural networks, you might see input data, such as:\n",
    "\n",
    "* **1D vector** - Classic input to a neural network, similar to rows in a spreadsheet. Common in predictive modeling.\n",
    "* **2D Matrix** - Grayscale image input to a CNN.\n",
    "* **3D Matrix** - Color image input to a CNN.\n",
    "* **nD Matrix** - Higher-order input to a CNN.\n",
    "\n",
    "Before CNNs, programs either encoded images to an intermediate form or sent the image input to a neural network by merely squashing the image matrix into a long array by placing the image's rows side-by-side. CNNs are different as the matrix passes through the neural network layers.\n",
    "\n",
    "Initially, this book will focus on 1D input to neural networks. However, later modules will focus more heavily on higher dimension input.\n",
    "\n",
    "The term dimension can be confusing in neural networks. In the sense of a 1D input vector, dimension refers to how many elements are in that 1D array. For example, a neural network with ten input neurons has ten dimensions. However, now that we have CNNs, the input has dimensions. The input to the neural network will *usually* have 1, 2, or 3 dimensions. Four or more dimensions are unusual. You might have a 2D input to a neural network with 64x64 pixels. This configuration would result in 4,096 input neurons. This network is either 2D or 4,096D, depending on which dimensions you reference.\n",
    "\n",
    "## Classification or Regression\n",
    "\n",
    "Like many models, neural networks can function in classification or regression:\n",
    "\n",
    "* **Regression** - You expect a number as your neural network's prediction.\n",
    "* **Classification** - You expect a class/category as your neural network's prediction.\n",
    "\n",
    "A classification and regression neural network is shown by Figure 3.CLS-REG.\n",
    "\n",
    "**Figure 3.CLS-REG: Neural Network Classification and Regression**\n",
    "![Neural Network Classification and Regression](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann_class_reg.png \"Neural Network Classification and Regression\")\n",
    "\n",
    "Notice that the output of the regression neural network is numeric, and the classification output is a class. Regression, or two-class classification, networks always have a single output. Classification neural networks have an output neuron for each category. \n",
    "\n",
    "## Neurons and Layers\n",
    "\n",
    "Most neural network structures use some type of neuron. Many different neural networks exist, and programmers introduce experimental neural network structures. Consequently, it is not possible to cover every neural network architecture. However, there are some commonalities among neural network implementations. A neural network algorithm would typically be composed of individual, interconnected units, even though these units may or may not be called neurons. The name for a neural network processing unit varies among the literature sources. It could be called a node, neuron, or unit.\n",
    "\n",
    "A diagram shows the abstract structure of a single artificial neuron in Figure 3.ANN.\n",
    "\n",
    "**Figure 3.ANN: An Artificial Neuron**\n",
    "![An Artificial Neuron](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_abstract_nn.png \"An Artificial Neuron\")\n",
    " \n",
    "The artificial neuron receives input from one or more sources that may be other neurons or data fed into the network from a computer program. This input is usually floating-point or binary. Often binary input is encoded to floating-point by representing true or false as 1 or 0. Sometimes the program also depicts the binary information using a bipolar system with true as one and false as -1.\n",
    "\n",
    "An artificial neuron multiplies each of these inputs by a weight. Then it adds these multiplications and passes this sum to an activation function. Some neural networks do not use an activation function. The following equation summarizes the calculated output of a neuron:\n",
    "\n",
    "$$ f(x,w) = \\phi(\\sum_i(\\theta_i \\cdot x_i)) $$\n",
    "\n",
    "In the above equation, the variables $x$ and $\\theta$ represent the input and weights of the neuron. The variable $i$ corresponds to the number of weights and inputs. You must always have the same number of weights as inputs. The neural network multiplies each weight by its respective input and feeds the products of these multiplications into an activation function, denoted by the Greek letter $\\phi$ (phi). This process results in a single output from the neuron.  \n",
    "\n",
    "The above neuron has two inputs plus the bias as a third. This neuron might accept the following input feature vector:\n",
    "\n",
    "$$ [1,2] $$\n",
    "\n",
    "Because a bias neuron is present, the program should append the value of one as follows:\n",
    "\n",
    "$$ [1,2,1] $$\n",
    "\n",
    "The weights for a 3-input layer (2 real inputs + bias) will always have additional weight for the bias. A weight vector might be:\n",
    "\n",
    "$$ [ 0.1, 0.2, 0.3] $$\n",
    "\n",
    "To calculate the summation, perform the following:\n",
    "\n",
    "$$ 0.1*1 + 0.2*2 + 0.3*1 = 0.8 $$\n",
    "\n",
    "The program passes a value of 0.8 to the $\\phi$ (phi) function, representing the activation function.\n",
    "\n",
    "The above figure shows the structure with just one building block. You can chain together many artificial neurons to build an artificial neural network (ANN). Think of the artificial neurons as building blocks for which the input and output circles are the connectors. Figure 3.ANN-3 shows an artificial neural network composed of three neurons:\n",
    "\n",
    "**Figure 3.ANN-3: Three Neuron Neural Network**\n",
    "![Three Neuron Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ann-simple.png \"Three Neuron Neural Network\")\n",
    "\n",
    "The above diagram shows three interconnected neurons. This representation is essentially this figure, minus a few inputs, repeated three times and then connected. It also has a total of four inputs and a single output. The output of neurons **N1** and **N2** feed **N3** to produce the output **O**.  To calculate the output for this network, we perform the previous equation three times. The first two times calculate **N1** and **N2**, and the third calculation uses the output of **N1** and **N2** to calculate **N3**.\n",
    "   \n",
    "Neural network diagrams do not typically show the detail seen in the previous figure. We can omit the activation functions and intermediate outputs to simplify the chart, resulting in Figure 3.SANN-3. \n",
    "\n",
    "**Figure 3.SANN-3: Three Neuron Neural Network**\n",
    "![Three Neuron Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/typical-ann.png \"Three Neuron Neural Network\")\n",
    " \n",
    "Looking at the previous figure, you can see two additional components of neural networks. First, consider the graph represents the inputs and outputs as abstract dotted line circles. The input and output could be parts of a more extensive neural network. However, the input and output are often a particular type of neuron that accepts data from the computer program using the neural network. The output neurons return a result to the program. This type of neuron is called an input neuron. We will discuss these neurons in the next section. This figure shows the neurons arranged in layers. The input neurons are the first layer, the **N1** and **N2** neurons create the second layer, the third layer contains **N3**, and the fourth layer has **O**.  Most neural networks arrange neurons into layers. \n",
    "\n",
    "The neurons that form a layer share several characteristics. First, every neuron in a layer has the same activation function. However, the activation functions employed by each layer may be different. Each of the layers fully connects to the next layer. In other words, every neuron in one layer has a connection to neurons in the previous layer. The former figure is not fully connected. Several layers are missing connections. For example, **I1** and **N2** do not connect. The next neural network in Figure 3.F-ANN is fully connected and has an additional layer.\n",
    "\n",
    "**Figure 3.F-ANN: Fully Connected Neural Network Diagram**\n",
    "![Fully Connected Neural Network Diagram](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ann-dense.png \"Fully Connected Neural Network Diagram\")\n",
    "\n",
    "In this figure, you see a fully connected, multilayered neural network. Networks such as this one will always have an input and output layer. The hidden layer structure determines the name of the network architecture. The network in this figure is a two-hidden-layer network. Most networks will have between zero and two hidden layers. Without implementing deep learning strategies, networks with more than two hidden layers are rare.    \n",
    "\n",
    "You might also notice that the arrows always point downward or forward from the input to the output. Later in this course, we will see recurrent neural networks that form inverted loops among the neurons. This type of neural network is called a feedforward neural network.\n",
    "\n",
    "## Types of Neurons\n",
    "\n",
    "In the last section, we briefly introduced the idea that different types of neurons exist. Not every neural network will use every kind of neuron. It is also possible for a single neuron to fill the role of several different neuron types. Now we will explain all the neuron types described in the course.\n",
    "\n",
    "There are usually four types of neurons in a neural network:\n",
    "\n",
    "* **Input Neurons** - We map each input neuron to one element in the feature vector.\n",
    "* **Hidden Neurons** - Hidden neurons allow the neural network to be abstract and process the input into the output.\n",
    "* **Output Neurons** - Each output neuron calculates one part of the output.\n",
    "* **Bias Neurons** - Work similar to the y-intercept of a linear equation.  \n",
    "\n",
    "We place each neuron into a layer:\n",
    "\n",
    "* **Input Layer** - The input layer accepts feature vectors from the dataset. Input layers usually have a bias neuron.\n",
    "* **Output Layer** - The output from the neural network. The output layer does not have a bias neuron.\n",
    "* **Hidden Layers** - Layers between the input and output layers. Each hidden layer will usually have a bias neuron.\n",
    "\n",
    "\n",
    "## Input and Output Neurons\n",
    "\n",
    "Nearly every neural network has input and output neurons. The input neurons accept data from the program for the network. The output neuron provides processed data from the network back to the program. The program will group these input and output neurons into separate layers called the input and output layers. The program normally represents the input to a neural network as an array or vector. The number of elements contained in the vector must equal the number of input neurons. For example, a neural network with three input neurons might accept the following input vector:\n",
    "\n",
    "$$ [0.5, 0.75, 0.2] $$\n",
    "\n",
    "Neural networks typically accept floating-point vectors as their input. To be consistent, we will represent the output of a single output neuron network as a single-element vector. Likewise, neural networks will output a vector with a length equal to the number of output neurons. The output will often be a single value from a single output neuron.\n",
    "\n",
    "## Hidden Neurons\n",
    "\n",
    "Hidden neurons have two essential characteristics. First, hidden neurons only receive input from other neurons, such as input or other hidden neurons. Second, hidden neurons only output to other neurons, such as output or other hidden neurons. Hidden neurons help the neural network understand the input and form the output. Programmers often group hidden neurons into fully connected hidden layers. However, these hidden layers do not directly process the incoming data or the eventual output.\n",
    "\n",
    "A common question for programmers concerns the number of hidden neurons in a network. Since the answer to this question is complex, more than one section of the course will include a relevant discussion of the number of hidden neurons. Before deep learning, researchers generally suggested that anything more than a single hidden layer is excessive. [[Cite:hornik1989multilayer]](https://www.sciencedirect.com/science/article/abs/pii/0893608089900208) Researchers have proven that a single-hidden-layer neural network can function as a universal approximator. In other words, this network should be able to learn to produce (or approximate) any output from any input as long as it has enough hidden neurons in a single layer. \n",
    "\n",
    "Training refers to the process that determines good weight values. Before the advent of deep learning, researchers feared additional layers would lengthen training time or encourage overfitting. Both concerns are true; however, increased hardware speeds and clever techniques can mitigate these concerns. Before researchers introduced deep learning techniques, we did not have an efficient way to train a deep network, which is a neural network with many hidden layers. Although a single-hidden-layer neural network can theoretically learn anything, deep learning facilitates a more complex representation of patterns in the data.  \n",
    "\n",
    "## Bias Neurons\n",
    "Programmers add bias neurons to neural networks to help them learn patterns. Bias neurons function like an input neuron that always produces a value of 1. Because the bias neurons have a constant output of 1, they are not connected to the previous layer. The value of 1, called the bias activation, can be set to values other than 1. However, 1 is the most common bias activation. Not all neural networks have bias neurons. Figure 3.BIAS shows a single-hidden-layer neural network with bias neurons:\n",
    "\n",
    "**Figure 3.BIAS: Neural Network with Bias Neurons**\n",
    "![Neural Network with Bias Neurons](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann.png \"Neural Network with Bias Neurons\")\n",
    " \n",
    "The above network contains three bias neurons. Except for the output layer, every level includes a single bias neuron. Bias neurons allow the program to shift the output of an activation function. We will see precisely how this shifting occurs later in the module when discussing activation functions.  \n",
    "\n",
    "\n",
    "## Other Neuron Types\n",
    "\n",
    "The individual units that comprise a neural network are not always called neurons. Researchers will sometimes refer to these neurons as nodes, units, or summations. You will almost always construct neural networks of weighted connections between these units. \n",
    "\n",
    "## Why are Bias Neurons Needed?\n",
    "\n",
    "The activation functions from the previous section specify the output of a single neuron. Together, the weight and bias of a neuron shape the output of the activation to produce the desired output. To see how this process occurs, consider the following equation. It represents a single-input sigmoid activation neural network.\n",
    "\n",
    "$$ f(x,w,b) = \\frac{1}{1 + e^{-(wx+b)}} $$ \n",
    "\n",
    "The $x$ variable represents the single input to the neural network. The $w$ and $b$ variables specify the weight and bias of the neural network. The above equation combines the weighted sum of the inputs and the sigmoid activation function. For this section, we will consider the sigmoid function because it demonstrates a bias neuron's effect.\n",
    "\n",
    "The weights of the neuron allow you to adjust the slope or shape of the activation function. Figure 3.A-WEIGHT shows the effect on the output of the sigmoid activation function if the weight is varied:\n",
    "\n",
    "**Figure 3.A-WEIGHT: Neuron Weight Shifting**\n",
    "![Adjusting Weight](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_weight.png \"Neuron Weight Shifting\")\n",
    "\n",
    "The above diagram shows several sigmoid curves using the following parameters:\n",
    "\n",
    "$$ f(x,0.5,0.0) $$\n",
    "$$ f(x,1.0,0.0) $$\n",
    "$$ f(x,1.5,0.0) $$\n",
    "$$ f(x,2.0,0.0) $$\n",
    "\n",
    "We did not use bias to produce the curves, which is evident in the third parameter of 0 in each case. Using four weight values yields four different sigmoid curves in the above figure. No matter the weight, we always get the same value of 0.5 when *x* is 0 because all curves hit the same point when x is 0. We might need the neural network to produce other values when the input is near 0.5.  \n",
    "\n",
    "Bias does shift the sigmoid curve, which allows values other than 0.5 when *x* is near 0. Figure 3.A-BIAS shows the effect of using a weight of 1.0 with several different biases:\n",
    "\n",
    "**Figure 3.A-BIAS: Neuron Bias Shifting**\n",
    "![Adjusting Bias](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_value.png \"Neuron Bias Shifting\")\n",
    "\n",
    "The above diagram shows several sigmoid curves with the following parameters:\n",
    "\n",
    "$$ f(x,1.0,1.0) $$\n",
    "$$ f(x,1.0,0.5) $$\n",
    "$$ f(x,1.0,1.5) $$\n",
    "$$ f(x,1.0,2.0) $$\n",
    "\n",
    "We used a weight of 1.0 for these curves in all cases. When we utilized several different biases, sigmoid curves shifted to the left or right. Because all the curves merge at the top right or bottom left, it is not a complete shift.\n",
    "\n",
    "When we put bias and weights together, they produced a curve that created the necessary output. The above curves are the output from only one neuron. In a complete network, the output from many different neurons will combine to produce intricate output patterns.\n",
    "\n",
    "\n",
    "## Modern Activation Functions\n",
    "\n",
    "Activation functions, also known as transfer functions, are used to calculate the output of each layer of a neural network. Historically neural networks have used a hyperbolic tangent, sigmoid/logistic, or linear activation function. However, modern deep neural networks primarily make use of the following activation functions:\n",
    "\n",
    "* **Rectified Linear Unit (ReLU)** - Used for the output of hidden layers. [[Cite:glorot2011deep]](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf)\n",
    "* **Softmax** - Used for the output of classification neural networks.\n",
    "* **Linear** - Used for the output of regression neural networks (or 2-class classification).\n",
    "\n",
    "## Linear Activation Function\n",
    "The most basic activation function is the linear function because it does not change the neuron output. The following equation 1.2 shows how the program typically implements a linear activation function:\n",
    "\n",
    "$$ \\phi(x) = x $$\n",
    "\n",
    "As you can observe, this activation function simply returns the value that the neuron inputs passed to it.  Figure 3.LIN shows the graph for a linear activation function:\n",
    "\n",
    "**Figure 3.LIN: Linear Activation Function**\n",
    "![Linear Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-linear.png \"Linear Activation Function\")\n",
    "\n",
    " \n",
    "Regression neural networks, which learn to provide numeric values, will usually use a linear activation function on their output layer. Classification neural networks, which determine an appropriate class for their input, will often utilize a softmax activation function for their output layer.\n",
    "\n",
    "## Rectified Linear Units (ReLU)\n",
    "\n",
    "Since its introduction, researchers have rapidly adopted the rectified linear unit (ReLU). [[Cite:nair2010rectified]](https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf) Before the ReLU activation function, the programmers generally regarded the hyperbolic tangent as the activation function of choice. Most current research now recommends the ReLU due to superior training results. As a result, most neural networks should utilize the ReLU on hidden layers and either softmax or linear on the output layer. The following equation shows the straightforward ReLU function:\n",
    "\n",
    "$$ \\phi(x) = \\max(0, x) $$\n",
    "\n",
    "Figure 3.RELU shows the graph of the ReLU activation function:\n",
    "\n",
    "**Figure 3.RELU: Rectified Linear Units (ReLU)**\n",
    "![Rectified Linear Units (ReLU)](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-relu.png \"Rectified Linear Units (ReLU)\")\n",
    " \n",
    "Most current research states that the hidden layers of your neural network should use the ReLU activation.\n",
    "\n",
    "## Softmax Activation Function\n",
    "\n",
    "The final activation function that we will examine is the softmax activation function. Along with the linear activation function, you can usually find the softmax function in the output layer of a neural network. Classification neural networks typically employ the softmax function. The neuron with the highest value claims the input as a member of its class. Because it is a preferable method, the softmax activation function forces the neural network's output to represent the probability that the input falls into each of the classes. The neuron's outputs are numeric values without the softmax, with the highest indicating the winning class.\n",
    "\n",
    "To see how the program uses the softmax activation function, we will look at a typical neural network classification problem. The iris data set contains four measurements for 150 different iris flowers. Each of these flowers belongs to one of three species of iris. When you provide the measurements of a flower, the softmax function allows the neural network to give you the probability that these measurements belong to each of the three species. For example, the neural network might tell you that there is an 80% chance that the iris is setosa, a 15% probability that it is virginica, and only a 5% probability of versicolor. Because these are probabilities, they must add up to 100%. There could not be an 80% probability of setosa, a 75% probability of virginica, and a 20% probability of versicolor—this type of result would be nonsensical.\n",
    "\n",
    "To classify input data into one of three iris species, you will need one output neuron for each species. The output neurons do not inherently specify the probability of each of the three species. Therefore, it is desirable to provide probabilities that sum to 100%. The neural network will tell you the likelihood of a flower being each of the three species. To get the probability, use the softmax function in the following equation: \n",
    "\n",
    "$$ \\phi_i(x) = \\frac{exp(x_i)}{\\sum_{j}^{ }exp(x_j)} $$\n",
    "\n",
    "In the above equation, $i$ represents the index of the output neuron ($\\phi$) that the program is calculating, and $j$ represents the indexes of all neurons in the group/level. The variable $x$ designates the array of output neurons. It's important to note that the program calculates the softmax activation differently than the other activation functions in this module. When softmax is the activation function, the output of a single neuron is dependent on the other output neurons. \n",
    " \n",
    "To see the softmax function in operation, refer to this [Softmax example website](http://www.heatonresearch.com/aifh/vol3/softmax.html).\n",
    "\n",
    "Consider a trained neural network that classifies data into three categories: the three iris species. In this case, you would use one output neuron for each of the target classes. Consider if the neural network were to output the following:   \n",
    "\n",
    "* **Neuron 1**: setosa: 0.9\n",
    "* **Neuron 2**: versicolour: 0.2\n",
    "* **Neuron 3**: virginica: 0.4\n",
    "\n",
    "The above output shows that the neural network considers the data to represent a setosa iris. However, these numbers are not probabilities. The 0.9 value does not represent a 90% likelihood of the data representing a setosa. These values sum to 1.5. For the program to treat them as probabilities, they must sum to 1.0. The output vector for this neural network is the following:\n",
    "\n",
    "$$ [0.9,0.2,0.4] $$\n",
    "\n",
    "If you provide this vector to the softmax function it will return the following vector:\n",
    "\n",
    "$$ [0.47548495534876745 , 0.2361188410001125 , 0.28839620365112] $$\n",
    "\n",
    "The above three values do sum to 1.0 and can be treated as probabilities.  The likelihood of the data representing a setosa iris is 48% because the first value in the vector rounds to 0.48 (48%).  You can calculate this value in the following manner:\n",
    "\n",
    "$$ sum=\\exp(0.9)+\\exp(0.2)+\\exp(0.4)=5.17283056695839 $$\n",
    "$$ j_0= \\exp(0.9)/sum = 0.47548495534876745 $$\n",
    "$$ j_1= \\exp(0.2)/sum = 0.2361188410001125 $$\n",
    "$$ j_2= \\exp(0.4)/sum = 0.28839620365112 $$\n",
    "\n",
    "## Step Activation Function\n",
    "\n",
    "The step or threshold activation function is another simple activation function. Neural networks were initially called perceptrons. McCulloch & Pitts (1943) introduced the original perceptron and used a step activation function like the following equation:[[Cite:mcculloch1943logical]](https://link.springer.com/article/10.1007/BF02478259) The step activation is 1 if x>=0.5, and 0 otherwise.\n",
    "  \n",
    "This equation outputs a value of 1.0 for incoming values of 0.5 or higher and 0 for all other values. Step functions, also known as threshold functions, only return 1 (true) for values above the specified threshold, as seen in Figure 3.STEP.\n",
    "\n",
    "**Figure 3.STEP: Step Activation Function**\n",
    "![Step Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-step.png \"Step Activation Function\")\n",
    " \n",
    "## Sigmoid Activation Function\n",
    "\n",
    "The sigmoid or logistic activation function is a common choice for feedforward neural networks that need to output only positive numbers. Despite its widespread use, the hyperbolic tangent or the rectified linear unit (ReLU) activation function is usually a more suitable choice. We introduce the ReLU activation function later in this module. The following equation shows the sigmoid activation function:\n",
    "\n",
    "$$ \\phi(x) = \\frac{1}{1 + e^{-x}} $$\n",
    "\n",
    "Use the sigmoid function to ensure that values stay within a relatively small range, as seen in Figure 3.SIGMOID:\n",
    "\n",
    "**Figure 3.SIGMOID: Sigmoid Activation Function**\n",
    "![Sigmoid Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-sigmoid.png \"Sigmoid Activation Function\")\n",
    "\n",
    "As you can see from the above graph, we can force values to a range. Here, the function compressed values above or below 0 to the approximate range between 0 and 1.\n",
    "\n",
    "## Hyperbolic Tangent Activation Function\n",
    "\n",
    "The hyperbolic tangent function is also a prevalent activation function for neural networks that must output values between -1 and 1. This activation function is simply the hyperbolic tangent (tanh) function, as shown in the following equation:\n",
    "\n",
    "$$ \\phi(x) = \\tanh(x) $$\n",
    "\n",
    "The graph of the hyperbolic tangent function has a similar shape to the sigmoid activation function, as seen in Figure 3.HTAN.\n",
    "\n",
    "**Figure 3.HTAN: Hyperbolic Tangent Activation Function**\n",
    "![Hyperbolic Tangent Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-tanh.png \"Hyperbolic Tangent Activation Function\")\n",
    " \n",
    "The hyperbolic tangent function has several advantages over the sigmoid activation function.  \n",
    "\n",
    "## Why ReLU?\n",
    "\n",
    "Why is the ReLU activation function so popular? One of the critical improvements to neural networks makes deep learning work. [[Cite:nair2010rectified]](https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf) Before deep learning, the sigmoid activation function was prevalent.  We covered the sigmoid activation function earlier in this module. Frameworks like Keras often train neural networks with gradient descent. For the neural network to use gradient descent, it is necessary to take the derivative of the activation function. The program must derive partial derivatives of each of the weights for the error function. Figure 3.DERV shows a derivative, the instantaneous rate of change.\n",
    "\n",
    "**Figure 3.DERV: Derivative**\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "The derivative of the sigmoid function is given here:\n",
    "\n",
    "$$ \\phi'(x)=\\phi(x)(1-\\phi(x)) $$\n",
    "\n",
    "Textbooks often give this derivative in other forms. We use the above form for computational efficiency. To see how we determined this derivative, [refer to the following article](http://www.heatonresearch.com/aifh/vol3/deriv_sigmoid.html).\n",
    "\n",
    "We present the graph of the sigmoid derivative in Figure 3.SDERV.\n",
    "\n",
    "**Figure 3.SDERV: Sigmoid Derivative**\n",
    "![Sigmoid Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv_sigmoid.png \"Sigmoid Derivative\")\n",
    "\n",
    "The derivative quickly saturates to zero as $x$ moves from zero.  This is not a problem for the derivative of the ReLU, which is given here:\n",
    "\n",
    "$$ \\phi'(x) = \\begin{cases} 1 & x > 0 \\\\ 0 & x \\leq 0 \\end{cases} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 3](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}