Machine Learning Reading Group.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unfolding Black Box Learning\n",
    "## Seeing the World from the Perspective of a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last time, Harman demonstrated the fundamentals of a neural network. We were introduced to concepts like \"activation function\", \"gradient descent\", and \"backpropagation\" and saw how it came together to recognize hand-written digits. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we are looking at what is happening between the input layer and the output layer of the neural network: we are going to understand what exactly those hidden layers are doing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/NeuralNetwork.png\" width=\"500px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's use the Sci-Kit Learn Library to Program this Example!\n",
    "# Code from: https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets, metrics\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from scripts import *\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "images, labels, n_samples, data = preprocess(plt, digits.images, digits.target)\n",
    "\n",
    "# MLP = Multi-Layer Perceptron = Neural Network\n",
    "classifier = MLPClassifier(hidden_layer_sizes=(300, 100))\n",
    "\n",
    "# We learn the digits on the first half of the digits\n",
    "classifier.fit(data[:n_samples // 2], labels[:n_samples // 2])\n",
    "\n",
    "# Now predict the value of the digit on the second half:\n",
    "expected = labels[n_samples // 2:]\n",
    "predicted = classifier.predict(data[n_samples // 2:])\n",
    "\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (classifier, metrics.classification_report(expected, predicted)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at the confusion matrix:\n",
    "print(\"Confusion matrix:\\n%s\" % metrics.confusion_matrix(expected, predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/ConfusionMatrix.png\" width=\"500px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the intuition behind these hidden layers? Why do we need them at all?\n",
    "\n",
    "Consider what we see when we see a number. We don't see the individual pixels - that kind of information is too dense for us to do anything useful with. Instead, we see higher-order patterns: lines and edges.\n",
    "So, what if instead of traing a network to recognize digits, we train it to recognizes lines and edges first.\n",
    "Then we use the lines and edges to predict numbers.\n",
    "\n",
    "This is exactly what the neural network is doing!\n",
    "\n",
    "<img src=\"images/HiddenLayers.gif\" width=\"500px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see this effect in <a target=\"_blank\" href=\"http://scs.ryerson.ca/~aharley/vis/\">action</a>!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my view, \"hidden layer\" is a bad term - it doesn't get acrosss any of the intuition as to what hidden layers are doing, or why they work! From this point on we will call these layers **latent factor models**, because the job of each layer to to build a model that represents the higher-order components (\"latent factors\") for the next layer (whether the next layer is building another latent factor model, or if the next layer is the output layer)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, when we visualize these hidden layers, we find a latent-factor model that is *meaningful* to us, like in the convolutional neural network case in the visualization example. But sometimes, the latent-factor model computed is only vaguely familar to the object it is trying to recognize, like in image classification:\n",
    "\n",
    "<img src=\"images/CIFAR-10-LFM.png\" width=\"750px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these models are reasonable: an airplane is usually pictured with a blue sky above, a ship is pictured with a blue sea below, a frog is very green in the center, etc. But for the most part, these latent factor models are fairly opaque to us. Even still, they definitely aren't \"hidden\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other reason the term \"hidden layer\" is a poor description of what is going on is that it assumes that these layers are meaningless to us - that all we care about are input and output layers. This couldn't be further from the truth. In fact, the latent factor models generated from a neural network unlock the ability to use machine learning beyond the usual classification, regression, or clustering tasks. Using these latent factor models, we can generate new outputs using only these latent factor representation via **autoencoders**.\n",
    "\n",
    "<img src=\"images/Autoencoder.png\" width=\"475px\" style=\"float: left\" />\n",
    "<img src=\"images/MNIST-Autoencoder.png\" width=\"500px\" style=\"position: relative; float: right; top: 50px\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An autoencoder using the latent factor representation to generate new outputs. To visualize this, imagine taking the MNIST neural network example and running it backwards - that is, given a number from 0 to 9, draw it on a two dimensional plane. This sounds difficult, so maybe instead of representing from the output space you represent it using one the latent factor models, manually setting each of the nodes until an interesting image comes out on the other end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with anything in machine learning, the best way to explain it is <a href=\"http://nvidia-research-mingyuliu.com/gaugan/\" target=\"_blank\">by example</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens when we use an auto-encoder on MNIST hand-written digits?\n",
    "\n",
    "First we to understand the architecture:\n",
    "(from https://gertjanvandenburg.com/blog/autoencoder/)\n",
    "\n",
    "<img src=\"images/MNIST-Autoencoder-Nodes.png\" width=\"500px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each input image (green) is sent through two \"hidden layers\" (AKA the encoder, in blue). The two yellow nodes represent the \"Compressed Feature Vector\" - that is, the latent-factor model we will use to reconstruct/generate hand-written digits.\n",
    "\n",
    "Then, when we reconstruct each image by using the Compressed Feature Vector to pass through two extra hidden layers (AKA the decoder, in blue on the right) to get to the reconstructed image (AKA the output, in red)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The meaning of this is mind-blowing: we can represent using **two** real numbers! Using these numbers, we can graph the input space like so:\n",
    "\n",
    "<img src=\"images/MNIST-Latent-Factor-Space.png\" width=\"500px\" style=\"float: left\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Not only can we represent hand-written digits we have already seen using these numbers, \n",
    "# but we can also generate new digits based on these numbers:\n",
    "from IPython.display import Video\n",
    "\n",
    "Video(\"videos/MNIST-Latent-Circle.mp4\", width=750)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for this session! To recap, we covered:\n",
    "- An intuition for hidden layers in a neural network\n",
    "- Why \"hidden layer\" is a terrible term, and why \"latent factor model\" is better\n",
    "- How to visualize latent factor models\n",
    "- How to use latent factor models to generate output using auto-encoders"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}