Modules and MLPs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modules and MLPs\n",
    "\n",
    "We've seen how the internals of simple linear classifier work. However, we still had to set a lot of things manually. It's much better to have a higher-level API that encapsulates the classifier.\n",
    "\n",
    "We are going to see that now, with pytorch Module objects. Then, it will allow us to build more complex models, like a multilayer perceptron."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin by loading the data again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import mnist\n",
    "from matplotlib import pyplot as pl\n",
    "\n",
    "train_x = mnist.train_images()\n",
    "train_y = mnist.train_labels()\n",
    "test_x = mnist.test_images()\n",
    "test_y = mnist.test_labels()\n",
    "\n",
    "num_features = 28 * 28\n",
    "num_classes = len(np.unique(train_y))\n",
    "new_shape = [-1, num_features]\n",
    "train_x_vectors = train_x.reshape(new_shape)\n",
    "test_x_vectors = test_x.reshape(new_shape)\n",
    "\n",
    "# shorten the names\n",
    "train_x = train_x_vectors / 255\n",
    "test_x = test_x_vectors / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sequential\n",
    "\n",
    "Let's create a model similar to the one in the previous notebook, but now with a more organized structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_layer = torch.nn.Linear(num_features, num_classes)\n",
    "linear_model = torch.nn.Sequential(linear_layer)\n",
    "loss_function = torch.nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be called as function to compute an output. Let's see how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 8\n",
    "batch = torch.tensor(train_x[:batch_size], dtype=torch.float)\n",
    "labels = torch.tensor(train_y[:batch_size], dtype=torch.long)\n",
    "\n",
    "answers = linear_model(batch)\n",
    "answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer\n",
    "\n",
    "The answers and loss are pretty much in the same way as in our last notebook. Now let's define an optimizer that will update weights more efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-2\n",
    "\n",
    "# the optimizer needs to be told which are the parameters to optimize\n",
    "optimizer = torch.optim.SGD(linear_model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training loop\n",
    "\n",
    "Now we write the main training loop. This is the basic skeleton for training pytorch models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, train_x, train_y, num_epochs, batch_size, optimizer):\n",
    "    losses = []\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        print('Starting epoch %d' % epoch)\n",
    "        batch_index = 0\n",
    "        total_loss = 0\n",
    "        hits = 0\n",
    "\n",
    "        while batch_index < len(train_x):\n",
    "            # get the data for this batch\n",
    "            next_index = batch_index + batch_size\n",
    "            batch_x = torch.tensor(train_x[batch_index:next_index], dtype=torch.float)\n",
    "            batch_y = torch.tensor(train_y[batch_index:next_index], dtype=torch.long)\n",
    "            batch_index = next_index\n",
    "\n",
    "            # forward pass\n",
    "            logits = model(batch_x)\n",
    "\n",
    "            # compute the loss\n",
    "            loss = loss_function(logits, batch_y)\n",
    "            loss_value = loss.item()\n",
    "            total_loss += loss_value\n",
    "            losses.append(loss_value)\n",
    "            \n",
    "            y_pred = logits.argmax(dim=1)\n",
    "            \n",
    "            hits += torch.sum(y_pred == batch_y).item()\n",
    "\n",
    "            # important: zero the gradients before recomputing them again\n",
    "            model.zero_grad()\n",
    "            loss.backward()\n",
    "\n",
    "            # after determining the gradients, take a step toward their direction\n",
    "            optimizer.step()\n",
    "\n",
    "        avg_loss = total_loss / len(train_x)\n",
    "        print('Epoch loss: %.4f' % avg_loss)\n",
    "        acc = hits / len(train_x)\n",
    "        print('Epoch accuracy: %.4f' % acc)\n",
    "    \n",
    "    return np.array(losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = train_model(linear_model, train_x, train_y, 5, 8, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing the loss decreases is good, but in classification problems, we usually want to know other metrics such as accuracy or F1.\n",
    "\n",
    "**Exercise:** Include accuracy report!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphics are good to understand the performance of a model. Let's plot the loss curve by batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "pl.rcParams['figure.figsize'] = [10, 5]\n",
    "pl.plot(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That might be too dense, although we can see that the loss doesn't decrease smoothly. Let's downsample the array, picking only every 10th value, remove the lines and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pl.plot(losses[::10], 'b.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is clearer to see that the bulk of the batches have a lower loss. Interestingly, some patterns of hard examples to classify are repeated every epoch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron\n",
    "\n",
    "We can now proceed to a more sofisticated classifier: a multilayer perceptron. Let's build one using the Sequential API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 200\n",
    "learning_rate = 1e-2\n",
    "\n",
    "linear_layer1 = torch.nn.Linear(num_features, hidden_size)\n",
    "linear_layer2 = torch.nn.Linear(hidden_size, num_classes)\n",
    "mlp = torch.nn.Sequential(linear_layer1, \n",
    "                          torch.nn.ReLU(), \n",
    "                          linear_layer2)\n",
    "\n",
    "optimizer = torch.optim.SGD(mlp.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the model. How do the loss and accuracy compare with the linear model?\n",
    "\n",
    "You probably also noticed a difference in running time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = train_model(mlp, train_x, train_y, 3, 8, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the different concentration of dots in the MLP and Linear graphics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pl.plot(losses[::10], 'b.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation data\n",
    "\n",
    "Evaluating the performance on training data is important to understand if the model is actually learning, but if we want to know if our model has any usefulness, we should evaluate its performance on validation or test data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_model(model, test_x, test_y):\n",
    "    test_x = torch.tensor(test_x, dtype=torch.float)\n",
    "    test_y = torch.tensor(test_y, dtype=torch.long)\n",
    "    loss_function = torch.nn.CrossEntropyLoss()\n",
    "    logits = model(test_x)\n",
    "    loss = loss_function(logits, test_y)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, test_x, test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(linear_model, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation loss is way higher than training loss: that's plain overfitting.\n",
    "\n",
    "How can we remedy that? There are two things to be done:\n",
    "\n",
    "1. **Generalize better**. This include either finding some better feature representation or regularizing, i.e., add some kind of penalty to the model weights that encourages it to find a more general solution. Examples: L2-norm weight regularization, dropout.\n",
    "1. **Early stop**. Evaluate the model on validation data after each epoch or some number of batches; only save it when validation performance increases. This means detecting when the model achieved its performance peak."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropout\n",
    "\n",
    "We could try dropout. It effectivelly deactivates some neural connections at random, forcing the network to avoid depending on specific inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 200\n",
    "learning_rate = 1e-2\n",
    "\n",
    "linear_layer1 = torch.nn.Linear(num_features, hidden_size)\n",
    "linear_layer2 = torch.nn.Linear(hidden_size, num_classes)\n",
    "mlp_dropout = torch.nn.Sequential(linear_layer1, \n",
    "                          torch.nn.ReLU(), \n",
    "                          torch.nn.Dropout(0.25),  # drop 25% of the connections\n",
    "                          linear_layer2)\n",
    "\n",
    "optimizer_dropout = torch.optim.SGD(mlp_dropout.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = train_model(mlp_dropout, train_x, train_y, 3, 8, optimizer_dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training loss is a bit worse, as expected. After all, we are obstructing some connections.\n",
    "\n",
    "Now let's check validation performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp_dropout, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No improvement. Ideally, we should retrain our model with different hyperparamters (learning rates, layer sizes, number of layers, dropout rate) as well as some changes in the structure (different optimizers, activation functions, losses).\n",
    "\n",
    "However, data representation plays a key role. Do you think representing the input as independent pixels is a good idea for recognizing digits?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving\n",
    "\n",
    "Persisting the model after training is obviously important to reuse it later.\n",
    "\n",
    "In Pytorch, we can save the model calling `save()` and passing  the model's `state_dict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(mlp.state_dict(), 'mlp.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, recreate the model and load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp2 = torch.nn.Sequential(linear_layer1, \n",
    "                          torch.nn.ReLU(), \n",
    "                          linear_layer2)\n",
    "mlp2.load_state_dict(torch.load('mlp.model'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the performance to see if it's the same!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** implement early stopping, so the model is saved whenever it reaches a lower loss best."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}