6.2-understanding-recurrent-neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding recurrent neural networks\n",
    "\n",
    "In this example, we will try out recurrent neural networks (RNNs). Recall that the idea of recurrent layers is that their output is fed back into the input. This enables them to learn about sequential patterns in the data, e.g. temporal ones.\n",
    "\n",
    "## A first recurrent layer in Keras\n",
    "\n",
    "Keras contains a straightforward recurrent layer implementation &mdash; the `SimpleRNN` layer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import SimpleRNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SimpleRNN` processes batches of sequences, like all other Keras layers, not just a single sequence. This means that it takes inputs of shape `(batch_size, timesteps, input_features)`, rather than `(timesteps, \n",
    "input_features)`.\n",
    "\n",
    "Like all recurrent layers in Keras, `SimpleRNN` can be run in two different modes: it can return either the full sequences of successive \n",
    "outputs for each timestep (a 3D tensor of shape `(batch_size, timesteps, output_features)`), or it can return only the last output for each \n",
    "input sequence (a 2D tensor of shape `(batch_size, output_features)`). These two modes are controlled by the `return_sequences` constructor \n",
    "argument. Let's take a look at an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, SimpleRNN\n",
    "\n",
    "model = Sequential()\n",
    "# Don't worry about this embedding layer here; it serves as a better input representation for text data\n",
    "model.add(Embedding(10000, 32))\n",
    "model.add(SimpleRNN(32))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(10000, 32))\n",
    "model.add(SimpleRNN(32, return_sequences=True))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is sometimes useful to stack several recurrent layers one after the other in order to increase the representational power of a network. \n",
    "In such a setup, you have to get all intermediate layers to return full sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(10000, 32))\n",
    "model.add(SimpleRNN(32, return_sequences=True))\n",
    "model.add(SimpleRNN(32, return_sequences=True))\n",
    "model.add(SimpleRNN(32, return_sequences=True))\n",
    "model.add(SimpleRNN(32))  # This last layer only returns the last outputs.\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to use such a model on the IMDB movie review classification problem. First, let's preprocess the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "max_features = 10000  # number of words to consider as features\n",
    "maxlen = 500  # cut texts after this number of words (among top max_features most common words)\n",
    "batch_size = 32\n",
    "\n",
    "print('Loading data...')\n",
    "(input_train, y_train), (input_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(input_train), 'train sequences')\n",
    "print(len(input_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "input_train = sequence.pad_sequences(input_train, maxlen=maxlen)\n",
    "input_test = sequence.pad_sequences(input_test, maxlen=maxlen)\n",
    "print('input_train shape:', input_train.shape)\n",
    "print('input_test shape:', input_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a simple recurrent network using an `Embedding` layer (don't worry about this) and a `SimpleRNN` layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 32))\n",
    "model.add(SimpleRNN(32))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "history = model.fit(input_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=128,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display the training and validation loss and accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a reminder, our very first naive approach to this very dataset got us to 88% test accuracy. Unfortunately, our small \n",
    "recurrent network doesn't perform very well at all compared to this baseline (only up to 85% validation accuracy). Part of the problem is \n",
    "that our inputs only consider the first 500 words rather the full sequences -- \n",
    "hence our RNN has access to less information than our earlier baseline model. The remainder of the problem is simply that `SimpleRNN` isn't very good at processing long sequences, like text. Other types of recurrent layers perform much better. Let's take a look at some \n",
    "more advanced layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## A concrete LSTM example in Keras\n",
    "\n",
    "Now let's switch to more practical concerns: we will set up a model using a LSTM layer and train it on the IMDB data. Here's the network, \n",
    "similar to the one with `SimpleRNN` that we just presented. We only specify the output dimensionality of the LSTM layer, and leave every \n",
    "other argument (there are lots) to the Keras defaults. Keras has good defaults, and things will almost always \"just work\" without you \n",
    "having to spend time tuning parameters by hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import LSTM\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 32))\n",
    "model.add(LSTM(32))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(input_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=128,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"border:3px; border-style:solid; border-color:blue; padding: 1em;\"><b>TASK:</b> In both models, try adding or removing units; try adding more recurrent layers. &#8679;</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}