deep-neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* * *\n",
    "![Alt text](./images/jmsl-7.png \"Neural Network\")\n",
    "* * *\n",
    "Image Source: https://machprinciple.wordpress.com/2013/11/09/neural-network-controlled-spacecraft-control-logic-exists/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "A [Deep Neural Network](https://en.wikipedia.org/wiki/Deep_learning#Deep_neural_network_architectures) (DNN) is an [artificial neural network](https://en.wikipedia.org/wiki/Artificial_neural_network) (ANN) with multiple hidden layers of units between the input and output layers.  Similar to shallow ANNs, DNNs can model complex non-linear relationships. DNN architectures (e.g. for object detection and parsing) generate compositional models where the object is expressed as a layered composition of image primitives. The extra layers enable composition of features from lower layers, giving the potential of modeling complex data with fewer units than a similarly performing shallow network.\n",
    "\n",
    "DNNs are typically designed as [feedforward](https://en.wikipedia.org/wiki/Feedforward_neural_network) networks, but research has very successfully applied recurrent neural networks, especially LSTM, for applications such as language modeling. Convolutional deep neural networks (CNNs) are used in computer vision where their success is well-documented. CNNs also have been applied to acoustic modeling for automatic speech recognition, where they have shown success over previous models.\n",
    "\n",
    "This tutorial will cover DNNs, however, we will introduce the concept of a \"shallow\" neural network as a starting point.\n",
    "\n",
    "## History\n",
    "\n",
    "An interesting fact about neural networks is that the first artificial neural network was implemented in hardware, not software.  In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a paper on how neurons might work. In order to describe how neurons in the brain might work, they modeled a simple neural network using electrical circuits. [1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation\n",
    "\n",
    "[Backpropagation](https://en.wikipedia.org/wiki/Backpropagation), an abbreviation for \"backward propagation of errors\", is a common method of training artificial neural networks used in conjunction with an optimization method such as [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent). The method calculates the gradient of a loss function with respect to all the weights in the network. The gradient is fed to the optimization method which in turn uses it to update the weights, in an attempt to minimize the loss function.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Architectures \n",
    "\n",
    "### Multilayer Perceptron (MLP) \n",
    "\n",
    "A [multilayer perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron) (MLP) is a [feed-forward](https://en.wikipedia.org/wiki/Feedforward_neural_network) artificial neural network model that maps sets of input data onto a set of appropriate outputs. This is also called a fully-connected fFeed-forward ANN.  An MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the next one. Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training the network.  MLP is a modification of the standard linear perceptron and can distinguish data that are not linearly separable.\n",
    "\n",
    "![Alt text](./images/mlp_network.png \"Multilayer Perceptron (MLP)\")\n",
    "Image Souce: neuralnetworksanddeeplearning.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurrent \n",
    "\n",
    "A [recurrent neural network](https://en.wikipedia.org/wiki/Recurrent_neural_network) (RNN) is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. Unlike feed-forward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. This makes them applicable to tasks such as unsegmented connected handwriting recognition or speech recognition.  \n",
    "\n",
    "There are a number of Twitter bots that are created using LSTMs.  Some fun examples are [@DeepDrumpf](https://twitter.com/DeepDrumpf) and [@DeepLearnBern](https://twitter.com/DeepLearnBern) by [Brad Hayes](https://twitter.com/hayesbh) from MIT.\n",
    "\n",
    "![Alt text](./images/rnn.jpg \"Recurrent Neural Network (RNN)\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional \n",
    "\n",
    "A [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN, or ConvNet) is a type of [feed-forward](https://en.wikipedia.org/wiki/Feedforward_neural_network) artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. They have wide applications in image and video recognition, recommender systems and natural language processing.\n",
    "\n",
    "Some creative application of CNNs are [DeepDream](https://github.com/ledell/grayarea-deepdream) images and [Neural Style Transfer](https://github.com/fzliu/style-transfer).\n",
    "\n",
    "![Alt text](./images/cnn.png \"Convolutional Neural Network (CNN)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Neural Nets\n",
    "\n",
    "![Alt text](./images/tensorflow_playground.png \"Tensorflow Playground\")\n",
    "\n",
    "\n",
    "[http://playground.tensorflow.org](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=xor&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.21753&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false) is a website where you can tweak and visualize neural networks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Software in R\n",
    "\n",
    "For the purposes of this tutorial, we will review CPU-based deep learning packages in R that support numeric, tabular data (data frames).  There is deep learning software that supports image data directly, but we will not cover those features in this tutorial.  The demos below will train a fully-connected, feed-forward ANN (aka mutilayer perceptron) using the [mxnet](http://mxnet.readthedocs.io/en/latest/how_to/build.html#r-package-installation) and [h2o](https://cran.r-project.org/web/packages/h2o/index.html) R packages.  Also worth noting are two other Deep Learning packages in R, [deepnet](https://cran.r-project.org/web/packages/deepnet/index.html) and [darch](https://cran.r-project.org/web/packages/darch/index.html) (which I don't have much experience with)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MXNet\n",
    "\n",
    "Authors: [Tianqi Chen](http://homes.cs.washington.edu/~tqchen/), [Qiang Kou (KK)](https://github.com/thirdwing), et. al.\n",
    "\n",
    "Backend: C++\n",
    "\n",
    "[MXNet](http://mxnet.readthedocs.io/en/latest/) is deep neural net implementation by the same authors as [xgboost](https://xgboost.readthedocs.io/en/latest/), and a part of the same umbrella project called [DMLC](http://dmlc.ml/) (Distributed Machine Learning Community). The [MXNet R package](http://mxnet.readthedocs.io/en/latest/packages/r/index.html) brings flexible and efficient GPU computing and state-of-art deep learning to R.\n",
    "\n",
    "Features:\n",
    "- It enables you to write seamless tensor/matrix computation with multiple GPUs in R.\n",
    "- Distributed computation.\n",
    "- Read image files directly.\n",
    "- Offers both a simplified and complex interface for architecting networks.\n",
    "- Supports mulit-layer feed-forward ANNs (aka. [multi-layer perceptrons (MLP)](https://en.wikipedia.org/wiki/Multilayer_perceptron)).\n",
    "- Supports [Convolutional Neural Networks (CNN)](https://en.wikipedia.org/wiki/Convolutional_neural_network) -- good for image recognition.\n",
    "- Supports [Long Short-term memory (LSTM)](https://en.wikipedia.org/wiki/Long_short-term_memory), a type of [Recurrent Neural Network (RNN)](https://en.wikipedia.org/wiki/Recurrent_neural_network) -- good for sequence learning, (e.g. speech, text).\n",
    "- License: BSD\n",
    "\n",
    "MXNet Example code below modified from: http://www.r-bloggers.com/deep-learning-with-mxnetr/\n",
    "The example below contains the canonical 60k/10k MNIST train/test files as opposed to the Kaggle version in the blog post. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The downloaded binary packages are in\n",
      "\t/var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T//RtmpJVnB5G/downloaded_packages\n",
      "\n",
      "The downloaded binary packages are in\n",
      "\t/var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T//RtmpJVnB5G/downloaded_packages\n"
     ]
    }
   ],
   "source": [
    "# Install pre-build CPU-based mxnet package (no GPU support)\n",
    "#install.packages(\"drat\")\n",
    "#drat:::addRepo(\"dmlc\")\n",
    "#install.packages(\"mxnet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(mxnet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "train_y\n",
       "   0    1    2    3    4    5    6    7    8    9 \n",
       "5923 6742 5957 6131 5842 5421 5918 6265 5851 5949 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Data load & preprocessing\n",
    "\n",
    "train <- read.csv(\"data/mnist_train.csv\", header = TRUE)\n",
    "test <- read.csv(\"data/mnist_test.csv\", header = TRUE)\n",
    "train <- data.matrix(train)\n",
    "test <- data.matrix(test)\n",
    "\n",
    "train <- data.matrix(train)\n",
    "test <- data.matrix(test)\n",
    "\n",
    "train_x <- train[,-785]\n",
    "train_y <- train[,785]\n",
    "test_x <- test[,-785]\n",
    "test_y <- test[,785]\n",
    "\n",
    "# lineraly transform it into [0,1]\n",
    "# transpose input matrix to npixel x nexamples (format expected by mxnet)\n",
    "train_x <- t(train_x/255)\n",
    "test_x <- t(test_x/255)\n",
    "\n",
    "#see that the number of each digit is fairly even\n",
    "table(train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Configure the structure of the network\n",
    "\n",
    "#in mxnet use its own data type symbol to configure the network\n",
    "data <- mx.symbol.Variable(\"data\")\n",
    "\n",
    "#set the first hidden layer where data is the input, name and number of hidden neurons\n",
    "fc1 <- mx.symbol.FullyConnected(data, name = \"fc1\", num_hidden = 128)\n",
    "\n",
    "#set the activation which takes the output from the first hidden layer fc1\n",
    "act1 <- mx.symbol.Activation(fc1, name = \"relu1\", act_type = \"relu\")\n",
    "\n",
    "#second hidden layer takes the result from act1 as the input, name and number of hidden neurons\n",
    "fc2 <- mx.symbol.FullyConnected(act1, name = \"fc2\", num_hidden = 64)\n",
    "\n",
    "#second activation which takes the output from the second hidden layer fc2\n",
    "act2 <- mx.symbol.Activation(fc2, name = \"relu2\", act_type = \"relu\")\n",
    "\n",
    "#this is the output layer where number of nuerons is set to 10 because there's only 10 digits\n",
    "fc3 <- mx.symbol.FullyConnected(act2, name = \"fc3\", num_hidden = 10)\n",
    "\n",
    "#finally set the activation to softmax to get a probabilistic prediction\n",
    "softmax <- mx.symbol.SoftmaxOutput(fc3, name = \"sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message:\n",
      "In mx.model.select.layout.train(X, y): Auto detect layout input matrix, use colmajor..\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training with 1 devices\n",
      "[1] Train-accuracy=0.88949916527546\n",
      "[2] Train-accuracy=0.96528333333334\n",
      "[3] Train-accuracy=0.975250000000006\n",
      "[4] Train-accuracy=0.98128333333334\n",
      "[5] Train-accuracy=0.984483333333339\n",
      "[6] Train-accuracy=0.987766666666672\n",
      "[7] Train-accuracy=0.989250000000005\n",
      "[8] Train-accuracy=0.990266666666671\n",
      "[9] Train-accuracy=0.991733333333338\n",
      "[10] Train-accuracy=0.99181666666667\n"
     ]
    }
   ],
   "source": [
    "#set which device to use before we start the computation\n",
    "#assign cpu to mxnet\n",
    "devices <- mx.cpu()\n",
    "\n",
    "#set seed to control the random process in mxnet\n",
    "mx.set.seed(0)\n",
    "\n",
    "#train the nueral network\n",
    "model <- mx.model.FeedForward.create(softmax, \n",
    "                                     X = train_x, \n",
    "                                     y = train_y,\n",
    "                                     ctx = devices, \n",
    "                                     num.round = 10, \n",
    "                                     array.batch.size = 100,\n",
    "                                     learning.rate = 0.07, \n",
    "                                     momentum = 0.9,\n",
    "                                     eval.metric = mx.metric.accuracy,\n",
    "                                     initializer = mx.init.uniform(0.07),\n",
    "                                     epoch.end.callback = mx.callback.log.train.metric(100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message:\n",
      "In mx.model.select.layout.predict(X, model): Auto detect layout input matrix, use colmajor..\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>10</li>\n",
       "\t<li>9999</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 10\n",
       "\\item 9999\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 10\n",
       "2. 9999\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1]   10 9999"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "pred_label\n",
       "   0    1    2    3    4    5    6    7    8    9 \n",
       " 990 1153 1013 1036  987  903  935 1006  969 1007 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] 0.9765977\n"
     ]
    }
   ],
   "source": [
    "#make a prediction\n",
    "preds <- predict(model, test_x)\n",
    "dim(preds)\n",
    "# [1]    10 10000\n",
    "\n",
    "#matrix with 10000 rows and 10 columns containing the classification probabilities from the output layer\n",
    "#use max.col to extract the maximum label for each row\n",
    "pred_label <- max.col(t(preds)) - 1\n",
    "table(pred_label)\n",
    "\n",
    "# Compute accuracy\n",
    "acc <- sum(test_y == pred_label)/length(test_y)\n",
    "print(acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MXNet also has a simplified api, [mx.mlp()](https://github.com/dmlc/mxnet/blob/master/R-package/R/mlp.R), which provides a more generic interface compared to the network construction procedure above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## h2o\n",
    "\n",
    "Authors: [Arno Candel](https://www.linkedin.com/in/candel), H2O.ai contributors\n",
    "\n",
    "Backend: Java\n",
    "\n",
    "[H2O Deep Learning](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/DeepLearningBooklet.pdf) builds a feed-forward multilayer artificial neural network on an distributed H2O data frame.\n",
    "\n",
    "Features:\n",
    "\n",
    "- Distributed and parallelized computation on either a single node or a multi-node cluster.\n",
    "- Data-distributed, which means the entire dataset does not need to fit into memory on a single node.\n",
    "- Uses [HOGWILD!](http://pages.cs.wisc.edu/~brecht/papers/hogwildTR.pdf) for fast computation.\n",
    "- Simplified network building interface.\n",
    "- Fully connected, feed-forward ANNs only (CNNs and LSTMs in development).\n",
    "- CPU only (GPU interface in development).\n",
    "- Automatic early stopping based on convergence of user-specied metrics to user-specied relative tolerance.\n",
    "- Support  for  exponential  families  (Poisson,  Gamma,  Tweedie)  and  loss functions in addition to binomial (Bernoulli), Gaussian and multinomial distributions, such as Quantile regression (including Laplace).\n",
    "- Grid search for hyperparameter optimization and model selection.\n",
    "- Apache 2.0 Licensed.\n",
    "- Model export in plain Java code for deployment in production environments.\n",
    "- GUI for training & model eval/viz (H2O Flow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading required package: statmod\n",
      "\n",
      "----------------------------------------------------------------------\n",
      "\n",
      "Your next step is to start H2O:\n",
      "    > h2o.init()\n",
      "\n",
      "For H2O package documentation, ask for help:\n",
      "    > ??h2o\n",
      "\n",
      "After starting H2O, you can use the Web UI at http://localhost:54321\n",
      "For more information visit http://docs.h2o.ai\n",
      "\n",
      "----------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Attaching package: ‘h2o’\n",
      "\n",
      "The following objects are masked from ‘package:stats’:\n",
      "\n",
      "    sd, var\n",
      "\n",
      "The following objects are masked from ‘package:base’:\n",
      "\n",
      "    &&, %*%, %in%, ||, apply, as.factor, as.numeric, colnames,\n",
      "    colnames<-, ifelse, is.character, is.factor, is.numeric, log,\n",
      "    log10, log1p, log2, round, signif, trunc\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Connection successful!\n",
      "\n",
      "R is connected to the H2O cluster: \n",
      "    H2O cluster uptime:         2 hours 14 minutes \n",
      "    H2O cluster version:        3.8.2.6 \n",
      "    H2O cluster name:           H2O_started_from_R_me_vrr258 \n",
      "    H2O cluster total nodes:    1 \n",
      "    H2O cluster total memory:   3.18 GB \n",
      "    H2O cluster total cores:    8 \n",
      "    H2O cluster allowed cores:  8 \n",
      "    H2O cluster healthy:        TRUE \n",
      "    H2O Connection ip:          localhost \n",
      "    H2O Connection port:        54321 \n",
      "    H2O Connection proxy:       NA \n",
      "    R Version:                  R version 3.3.0 (2016-05-03) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "library(h2o)\n",
    "h2o.init(nthreads = -1)  # This means nthreads = num available cores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# Data load & preprocessing\n",
    "\n",
    "train <- h2o.importFile(\"data/mnist_train.csv\")\n",
    "test <- h2o.importFile(\"data/mnist_test.csv\")\n",
    "\n",
    "# Specify the response and predictor columns\n",
    "y <- \"C785\"\n",
    "x <- setdiff(names(train), y)\n",
    "\n",
    "# We encode the response column as categorical for multinomial classification\n",
    "train[,y] <- as.factor(train[,y])\n",
    "test[,y] <- as.factor(test[,y])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message:\n",
      "In .h2o.startModelJob(algo, params, h2oRestApiVersion): Dropping constant columns: [C86, C85, C729, C728, C646, C645, C169, C760, C561, C53, C11, C55, C10, C54, C57, C12, C56, C58, C17, C19, C18, C731, C730, C20, C22, C21, C24, C23, C26, C25, C28, C27, C702, C701, C29, C700, C1, C2, C784, C3, C783, C4, C782, C5, C781, C6, C142, C7, C141, C8, C9, C31, C30, C32, C759, C758, C757, C756, C755, C477, C113, C674, C112, C673, C672, C84, C83].\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  10%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  20%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  30%\r",
      "  |                                                                            \r",
      "  |============================                                          |  40%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  50%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  60%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  70%\r",
      "  |                                                                            \r",
      "  |========================================================              |  80%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  90%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# Train an H2O Deep Learning model\n",
    "\n",
    "model <- h2o.deeplearning(x = x, \n",
    "                          y = y, \n",
    "                          training_frame = train,\n",
    "                          sparse = TRUE,  #speed-up on sparse data (like MNIST)\n",
    "                          distribution = \"multinomial\",\n",
    "                          activation = \"Rectifier\", \n",
    "                          hidden = c(128,64),\n",
    "                          epochs = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "[1] 0.9728\n"
     ]
    }
   ],
   "source": [
    "# Get model performance on a test set\n",
    "perf <- h2o.performance(model, test)\n",
    "\n",
    "# Or get the preds directly and compute classification accuracy\n",
    "preds <- h2o.predict(model, newdata = test)\n",
    "acc <- sum(test[,y] == preds$predict)/nrow(test)\n",
    "print(acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References \n",
    "\n",
    "[1] [http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/History/history1.html](http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/History/history1.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}