19. Neural Networks - Activation Functions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "\n",
       "code {\n",
       "    font-size: 80%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks - Activation Functions\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "The role of an ***activation function*** is to ***introduce a non-linearity***. If all activations were ***linear***, the whole network could be replace by a ***single affine transformation*** (a linear transformation followed by a translation).\n",
    "\n",
    "### 1.1 Motivation\n",
    "\n",
    "As you have seen in previous lessons, what most algorithms do to perform ***classification*** of data points is to ***separate them linearly*** (in some hyper-plane).\n",
    "\n",
    "What do you do if the points are ***not*** linearly separable, like this?\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/800/1*3jGx6YoSYXsrSfgbxO2yzw.png)\n",
    "\n",
    "Support Vector Machines make use of the ***kernel trick*** to overcome this difficulty. In a way, they ***modify the feature space*** using a kernel.\n",
    "\n",
    "The way ***neural networks*** modify the feature space is by ***twisting and turning it*** using an ***activation function***, until it looks like this:\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/800/1*ZNPrD0PmXz7I6rhnbPuh1A.png)\n",
    "\n",
    "Success! Now the colored points are ***separated by a line***!\n",
    "\n",
    "But, if you look at the ***decision boundary in the original feature space***, they look like this:\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/800/1*BoWjJPEuXqtUJXFuOnu_gQ.png)\n",
    "\n",
    "Can you guess ***which*** activation function yielded each one of these boundaries?\n",
    "\n",
    "### 1.2 Functions\n",
    "\n",
    "Sigmoid | Tanh | ReLU\n",
    ":---:|:---:|:---:\n",
    "![sigmoid](https://cdn-images-1.medium.com/max/800/1*tXzS5GwC3BBqi7ppwcQWdw.png) | ![tanh](https://cdn-images-1.medium.com/max/800/1*SeCBB7lfA7KPJ-T1Mi7GRg.png) | ![](https://cdn-images-1.medium.com/max/800/1*piSRCnAIA2paTMd8kVDfKg.png)\n",
    "\n",
    "For a thorough explanation of the most common activations functions (sigmoid, tanh and ReLU), check my blog [post](https://towardsdatascience.com/hyper-parameters-in-action-a524bf5bf1c)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "You are feeding the two lines as inputs to a very simple neural network, with only ***two hidden units*** ($h_1$ and $h_2 \\ $) and a single ouput unit ($h_3 \\ $) to perform the classification. It looks like this:\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/800/1*Frni4L9WiHQCNvVFCIW2ZA.png)\n",
    "\n",
    "The controls below allow you to:\n",
    "- the first set of weights ($w_{11} \\ $, $w_{12} \\ $, $w_{21} \\ $ and $w_{22} \\ $) define a ***transformation matrix*** to apply to the ***feature space***\n",
    "- the biases ($b_{11} \\ $ and $b_{12} \\ $) perform the ***translation*** of the feature space by modifying its origin\n",
    "- the ***grid*** and the ***colored lines*** are subject to the result of both transformation and translation, resulting in z-values which are then fed to the ***activation function*** $\\sigma \\ $.\n",
    "- the ***activation values*** are then multiplied by the weights ($w_{13} \\ $ and $w_{23} \\ $) and have the bias added ($b_{13} \\ $) - whenever the result of this operation ***equals zero***, it defines a ***linear boundary*** which is used to ***separate*** the classes.\n",
    "\n",
    "Use the controls to play with different configurations and answer the ***questions*** below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.classification.Activations import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = load_data()\n",
    "myact = plotActivations(X, y)\n",
    "vb = VBox(build_figure(myact), layout={'align_items': 'center'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "56780bef49954b82baf8c22b307bb832",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'line': {'color': 'black', 'width': 1},\n",
       "              'mode': 'lin…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. Keep ***activation linear***:\n",
    "    - change w21 to 1.0 - what kind of transformation is that?\n",
    "    - now slide w11 slowly to -1.0 - what happened to the grid?\n",
    "    - change b11 to 1.0 and observe the scale on the x axis - what is this operation called?\n",
    "    - change w12 to 1.0\n",
    "        - what happened to the grid? Why? \n",
    "        - did anything else change? Why?\n",
    "    - change b23 to -1.0 - what happened?\n",
    "    - can you separate the colored points with these operations?\n",
    "    \n",
    "    \n",
    "2. Change ***activation to sigmoid***:\n",
    "    - what is the first thing you noticed? (hint: look at the scales)\n",
    "    - set the weights to the ***basis vectors*** (recall from the lesson about eigenvectors and values) and set biases to ***zero*** - what does the grid look like?\n",
    "    - change w11 and w22 to 3.0 (we are ***scaling*** the grid) - what is the ***effect*** of the sigmoid activation?\n",
    "    - make w21 equals 3.0 - what happened to the grid? Does the shape look familiar?\n",
    "    - from this set of weights, which kind of ***transformation*** does it represent?\n",
    "        - change ***activation to linear*** and look at the ***shape of the grid*** to confirm\n",
    "        - then set it back to sigmoid\n",
    "    - set b12 to 3.0 - what effect did the bias have on the grid?\n",
    "    \n",
    "    \n",
    "3. Change ***activation to tanh***:\n",
    "    - what is the first thing you noticed?\n",
    "    - change the value of w12 to both extremes and observe what happened to the grid space\n",
    "    \n",
    "    \n",
    "4. Change ***activation to ReLU***:\n",
    "    - what is the first thing you noticed?\n",
    "    - slowly change b11 from 0 to -3 - what happened to the grid?\n",
    "  \n",
    "Before next question, set:\n",
    "- activation to ***linear***\n",
    "- set the ***transformation matrix*** and ***biases*** to: $$\n",
    "\\begin{bmatrix}\n",
    "   -3 & 3 \\\\\n",
    "   -3 & -3\n",
    " \\end{bmatrix}\n",
    " \\\n",
    "\\begin{bmatrix}\n",
    "   1 \\\\\n",
    "   -1\n",
    " \\end{bmatrix} \n",
    "$$\n",
    "   \n",
    "\n",
    "5. Starting from the configuration above:\n",
    "    - what transformation is that?\n",
    "    - change ***activation to sigmoid*** - what happened to the grid and, especially, the colored points?\n",
    "    - change the last unit's weights and bias and try ***separating the points***? Did you manage to do it?\n",
    "    - change ***activation to tanh*** - what happened to the grid and, especially, the colored points?\n",
    "    - change the last unit's weights and bias and try ***separating the points***? Did you manage to do it? How do you compare it to the ***sigmoid***?\n",
    "\n",
    "Before next question, set:\n",
    "- activation to ***linear***\n",
    "- set the ***transformation matrix*** and ***biases*** to: $$\n",
    "\\begin{bmatrix}\n",
    "   -3 & 0.4 \\\\\n",
    "   -3 & 3\n",
    " \\end{bmatrix}\n",
    " \\\n",
    "\\begin{bmatrix}\n",
    "   0.1 \\\\\n",
    "   1.8\n",
    " \\end{bmatrix} \n",
    "$$\n",
    "\n",
    "\n",
    "6. Starting from the configuration above:\n",
    "    - change ***actiation to ReLU*** - what happened to the grid and the data points to the left?\n",
    "    - change the last unit's weights and bias and try ***separating the points***? Did you manage to do it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Keras\n",
    "\n",
    "This is the Keras' implementation of the simple network diagram shown before.\n",
    "\n",
    "```python\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import SGD\n",
    "from keras.initializers import glorot_normal, normal\n",
    "\n",
    "activation = 'sigmoid'\n",
    "\n",
    "# Uses Glorot initializer for hidden layer with a typical seed: 42\n",
    "glorot_initializer = glorot_normal(seed=42)\n",
    "# Uses Normal initializer for outputlayer with the same seed\n",
    "normal_initializer = normal(seed=42)\n",
    "\n",
    "# Uses Stochastic Gradient Descent with a learning rate of 0.05\n",
    "sgd = SGD(lr=0.05)\n",
    "\n",
    "# Uses Keras' Sequential API\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Dense(input_dim=2, # Input layer contains 2 units\n",
    "                units=2,     # Hidden layer contains 2 units\n",
    "                kernel_initializer=glorot_initializer, \n",
    "                activation=activation))\n",
    "\n",
    "# Output layer with sigmoid activation for binary classification\n",
    "model.add(Dense(units=1, \n",
    "                kernel_initializer=normal_initializer,\n",
    "                activation='sigmoid'))\n",
    "\n",
    "# Compiles model using binary crossentropy as loss\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=sgd, \n",
    "              metrics=['acc'])\n",
    "\n",
    "# Fits the model using a mini-batch size of 16 during 150 epochs\n",
    "model.fit(X, y, epochs=150, batch_size=16)\n",
    "\n",
    "print(model.get_weights())\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dvgodoy/Programs/anaconda/envs/py35/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning:\n",
      "\n",
      "Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/150\n",
      "2000/2000 [==============================] - 0s 52us/step - loss: 0.6802 - acc: 0.6605\n",
      "Epoch 2/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.6365 - acc: 0.7650\n",
      "Epoch 3/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.5692 - acc: 0.7545\n",
      "Epoch 4/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.5104 - acc: 0.7535\n",
      "Epoch 5/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.4760 - acc: 0.7645\n",
      "Epoch 6/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.4577 - acc: 0.7745\n",
      "Epoch 7/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.4480 - acc: 0.7720\n",
      "Epoch 8/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.4418 - acc: 0.7745\n",
      "Epoch 9/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.4369 - acc: 0.7860\n",
      "Epoch 10/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.4316 - acc: 0.7860\n",
      "Epoch 11/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.4303 - acc: 0.8015\n",
      "Epoch 12/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.4272 - acc: 0.8040\n",
      "Epoch 13/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.4249 - acc: 0.7970\n",
      "Epoch 14/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.4223 - acc: 0.8000\n",
      "Epoch 15/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.4203 - acc: 0.8265\n",
      "Epoch 16/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.4188 - acc: 0.8025\n",
      "Epoch 17/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.4169 - acc: 0.8165\n",
      "Epoch 18/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.4153 - acc: 0.8150\n",
      "Epoch 19/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.4136 - acc: 0.8255\n",
      "Epoch 20/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.4124 - acc: 0.8235\n",
      "Epoch 21/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.4107 - acc: 0.8325\n",
      "Epoch 22/150\n",
      "2000/2000 [==============================] - 0s 43us/step - loss: 0.4097 - acc: 0.8300\n",
      "Epoch 23/150\n",
      "2000/2000 [==============================] - 0s 49us/step - loss: 0.4086 - acc: 0.8360\n",
      "Epoch 24/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.4070 - acc: 0.8215\n",
      "Epoch 25/150\n",
      "2000/2000 [==============================] - 0s 39us/step - loss: 0.4063 - acc: 0.8335\n",
      "Epoch 26/150\n",
      "2000/2000 [==============================] - 0s 44us/step - loss: 0.4052 - acc: 0.8395\n",
      "Epoch 27/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.4042 - acc: 0.8345\n",
      "Epoch 28/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.4033 - acc: 0.8450\n",
      "Epoch 29/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.4022 - acc: 0.8415\n",
      "Epoch 30/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.4013 - acc: 0.8385\n",
      "Epoch 31/150\n",
      "2000/2000 [==============================] - 0s 44us/step - loss: 0.4006 - acc: 0.8460\n",
      "Epoch 32/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3995 - acc: 0.8395\n",
      "Epoch 33/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.3990 - acc: 0.8460\n",
      "Epoch 34/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3982 - acc: 0.8435\n",
      "Epoch 35/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.3973 - acc: 0.8445\n",
      "Epoch 36/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3965 - acc: 0.8445\n",
      "Epoch 37/150\n",
      "2000/2000 [==============================] - 0s 46us/step - loss: 0.3959 - acc: 0.8495\n",
      "Epoch 38/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3949 - acc: 0.8460\n",
      "Epoch 39/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3944 - acc: 0.8470\n",
      "Epoch 40/150\n",
      "2000/2000 [==============================] - 0s 42us/step - loss: 0.3937 - acc: 0.8450\n",
      "Epoch 41/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3929 - acc: 0.8475\n",
      "Epoch 42/150\n",
      "2000/2000 [==============================] - 0s 44us/step - loss: 0.3920 - acc: 0.8465\n",
      "Epoch 43/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3913 - acc: 0.8490\n",
      "Epoch 44/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.3905 - acc: 0.8460\n",
      "Epoch 45/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.3899 - acc: 0.8445\n",
      "Epoch 46/150\n",
      "2000/2000 [==============================] - 0s 39us/step - loss: 0.3885 - acc: 0.8475\n",
      "Epoch 47/150\n",
      "2000/2000 [==============================] - 0s 44us/step - loss: 0.3877 - acc: 0.8465\n",
      "Epoch 48/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3864 - acc: 0.8455\n",
      "Epoch 49/150\n",
      "2000/2000 [==============================] - 0s 39us/step - loss: 0.3852 - acc: 0.8395\n",
      "Epoch 50/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.3837 - acc: 0.8460\n",
      "Epoch 51/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.3815 - acc: 0.8475\n",
      "Epoch 52/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.3782 - acc: 0.8485\n",
      "Epoch 53/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.3723 - acc: 0.8415\n",
      "Epoch 54/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.3617 - acc: 0.8525\n",
      "Epoch 55/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.3398 - acc: 0.8555\n",
      "Epoch 56/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.3024 - acc: 0.8660\n",
      "Epoch 57/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.2587 - acc: 0.8820\n",
      "Epoch 58/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.2184 - acc: 0.9275\n",
      "Epoch 59/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.1846 - acc: 0.9685\n",
      "Epoch 60/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.1577 - acc: 0.9905\n",
      "Epoch 61/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.1367 - acc: 0.9955\n",
      "Epoch 62/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.1200 - acc: 0.9980\n",
      "Epoch 63/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.1067 - acc: 0.9985\n",
      "Epoch 64/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0959 - acc: 1.0000\n",
      "Epoch 65/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0871 - acc: 1.0000\n",
      "Epoch 66/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0798 - acc: 1.0000\n",
      "Epoch 67/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0735 - acc: 1.0000\n",
      "Epoch 68/150\n",
      "2000/2000 [==============================] - 0s 39us/step - loss: 0.0681 - acc: 1.0000\n",
      "Epoch 69/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.0634 - acc: 1.0000\n",
      "Epoch 70/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0593 - acc: 1.0000\n",
      "Epoch 71/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0556 - acc: 1.0000\n",
      "Epoch 72/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0523 - acc: 1.0000\n",
      "Epoch 73/150\n",
      "2000/2000 [==============================] - 0s 45us/step - loss: 0.0494 - acc: 1.0000\n",
      "Epoch 74/150\n",
      "2000/2000 [==============================] - 0s 49us/step - loss: 0.0467 - acc: 1.0000\n",
      "Epoch 75/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0444 - acc: 1.0000\n",
      "Epoch 76/150\n",
      "2000/2000 [==============================] - 0s 42us/step - loss: 0.0422 - acc: 1.0000\n",
      "Epoch 77/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.0402 - acc: 1.0000\n",
      "Epoch 78/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.0383 - acc: 1.0000\n",
      "Epoch 79/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0366 - acc: 1.0000\n",
      "Epoch 80/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0351 - acc: 1.0000\n",
      "Epoch 81/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0336 - acc: 1.0000\n",
      "Epoch 82/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0323 - acc: 1.0000\n",
      "Epoch 83/150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0311 - acc: 1.0000\n",
      "Epoch 84/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0299 - acc: 1.0000\n",
      "Epoch 85/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0289 - acc: 1.0000\n",
      "Epoch 86/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0279 - acc: 1.0000\n",
      "Epoch 87/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0269 - acc: 1.0000\n",
      "Epoch 88/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0260 - acc: 1.0000\n",
      "Epoch 89/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0252 - acc: 1.0000\n",
      "Epoch 90/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0244 - acc: 1.0000\n",
      "Epoch 91/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0236 - acc: 1.0000\n",
      "Epoch 92/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0229 - acc: 1.0000\n",
      "Epoch 93/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0223 - acc: 1.0000\n",
      "Epoch 94/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0216 - acc: 1.0000\n",
      "Epoch 95/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0210 - acc: 1.0000\n",
      "Epoch 96/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0205 - acc: 1.0000\n",
      "Epoch 97/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0199 - acc: 1.0000\n",
      "Epoch 98/150\n",
      "2000/2000 [==============================] - 0s 41us/step - loss: 0.0194 - acc: 1.0000\n",
      "Epoch 99/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.0189 - acc: 1.0000\n",
      "Epoch 100/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0184 - acc: 1.0000\n",
      "Epoch 101/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.0180 - acc: 1.0000\n",
      "Epoch 102/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0176 - acc: 1.0000\n",
      "Epoch 103/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0172 - acc: 1.0000\n",
      "Epoch 104/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0168 - acc: 1.0000\n",
      "Epoch 105/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0164 - acc: 1.0000\n",
      "Epoch 106/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0160 - acc: 1.0000\n",
      "Epoch 107/150\n",
      "2000/2000 [==============================] - 0s 40us/step - loss: 0.0157 - acc: 1.0000\n",
      "Epoch 108/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0154 - acc: 1.0000\n",
      "Epoch 109/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0150 - acc: 1.0000\n",
      "Epoch 110/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0147 - acc: 1.0000\n",
      "Epoch 111/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0144 - acc: 1.0000\n",
      "Epoch 112/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0142 - acc: 1.0000\n",
      "Epoch 113/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0139 - acc: 1.0000\n",
      "Epoch 114/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0136 - acc: 1.0000\n",
      "Epoch 115/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0134 - acc: 1.0000\n",
      "Epoch 116/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0131 - acc: 1.0000\n",
      "Epoch 117/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0129 - acc: 1.0000\n",
      "Epoch 118/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0127 - acc: 1.0000\n",
      "Epoch 119/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0124 - acc: 1.0000\n",
      "Epoch 120/150\n",
      "2000/2000 [==============================] - 0s 33us/step - loss: 0.0122 - acc: 1.0000\n",
      "Epoch 121/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0120 - acc: 1.0000\n",
      "Epoch 122/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0118 - acc: 1.0000\n",
      "Epoch 123/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0116 - acc: 1.0000\n",
      "Epoch 124/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0114 - acc: 1.0000\n",
      "Epoch 125/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0112 - acc: 1.0000\n",
      "Epoch 126/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0111 - acc: 1.0000\n",
      "Epoch 127/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0109 - acc: 1.0000\n",
      "Epoch 128/150\n",
      "2000/2000 [==============================] - 0s 38us/step - loss: 0.0107 - acc: 1.0000\n",
      "Epoch 129/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0106 - acc: 1.0000\n",
      "Epoch 130/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0104 - acc: 1.0000\n",
      "Epoch 131/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0102 - acc: 1.0000\n",
      "Epoch 132/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0101 - acc: 1.0000\n",
      "Epoch 133/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0099 - acc: 1.0000\n",
      "Epoch 134/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0098 - acc: 1.0000\n",
      "Epoch 135/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0097 - acc: 1.0000\n",
      "Epoch 136/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0095 - acc: 1.0000\n",
      "Epoch 137/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0094 - acc: 1.0000\n",
      "Epoch 138/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0093 - acc: 1.0000\n",
      "Epoch 139/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0091 - acc: 1.0000\n",
      "Epoch 140/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0090 - acc: 1.0000\n",
      "Epoch 141/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0089 - acc: 1.0000\n",
      "Epoch 142/150\n",
      "2000/2000 [==============================] - 0s 37us/step - loss: 0.0088 - acc: 1.0000\n",
      "Epoch 143/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0087 - acc: 1.0000\n",
      "Epoch 144/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0086 - acc: 1.0000\n",
      "Epoch 145/150\n",
      "2000/2000 [==============================] - 0s 36us/step - loss: 0.0085 - acc: 1.0000\n",
      "Epoch 146/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0084 - acc: 1.0000\n",
      "Epoch 147/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0083 - acc: 1.0000\n",
      "Epoch 148/150\n",
      "2000/2000 [==============================] - 0s 39us/step - loss: 0.0082 - acc: 1.0000\n",
      "Epoch 149/150\n",
      "2000/2000 [==============================] - 0s 34us/step - loss: 0.0081 - acc: 1.0000\n",
      "Epoch 150/150\n",
      "2000/2000 [==============================] - 0s 35us/step - loss: 0.0080 - acc: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f9c6a00bdd8>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import SGD\n",
    "from keras.initializers import glorot_normal, normal\n",
    "\n",
    "activation = 'tanh'\n",
    "\n",
    "# Uses Glorot initializer for hidden layer with a typical seed: 42\n",
    "glorot_initializer = glorot_normal(seed=42)\n",
    "# Uses Normal initializer for outputlayer with the same seed\n",
    "normal_initializer = normal(seed=42)\n",
    "\n",
    "# Uses Stochastic Gradient Descent with a learning rate of 0.05\n",
    "sgd = SGD(lr=0.05)\n",
    "\n",
    "# Uses Keras' Sequential API\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Dense(input_dim=2, # Input layer contains 2 units\n",
    "                units=2,     # Hidden layer contains 2 units\n",
    "                kernel_initializer=glorot_initializer, \n",
    "                activation=activation))\n",
    "\n",
    "# Output layer with sigmoid activation for binary classification\n",
    "model.add(Dense(units=1, \n",
    "                kernel_initializer=normal_initializer,\n",
    "                activation='sigmoid'))\n",
    "\n",
    "# Compiles model using binary crossentropy as loss\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=sgd, \n",
    "              metrics=['acc'])\n",
    "\n",
    "# Fits the model using a mini-batch size of 16 during 150 epochs\n",
    "model.fit(X, y, epochs=150, batch_size=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[-4.7761188, -4.3531065],\n",
      "       [-4.5376463,  4.010479 ]], dtype=float32), array([-1.7944971,  1.6984158], dtype=float32), array([[ 6.3906374],\n",
      "       [-6.5252028]], dtype=float32), array([5.300107], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "print(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[Hyper-parameters in Action! Part I — Activation Functions](https://towardsdatascience.com/hyper-parameters-in-action-a524bf5bf1c)\n",
    "\n",
    "[Hyper-parameters in Action! Introducing DeepReplay](https://towardsdatascience.com/hyper-parameters-in-action-introducing-deepreplay-31132a7b9631)\n",
    "\n",
    "[Hyper-parameters in Action! Part II — Weight Initializers](https://towardsdatascience.com/hyper-parameters-in-action-part-ii-weight-initializers-35aee1a28404)\n",
    "\n",
    "[Neural Networks Series](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)\n",
    "\n",
    "[A Visual and Interactive Guide to the Basics of Neural Networks](http://jalammar.github.io/visual-interactive-guide-basics-neural-networks/)\n",
    "\n",
    "[A Visual And Interactive Look at Basic Neural Network Math](https://jalammar.github.io/feedforward-neural-networks-visual-interactive/)\n",
    "\n",
    "[A visual proof that neural nets can compute any function](http://neuralnetworksanddeeplearning.com/chap4.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}