lab_day2/day2_code00_keras_MNIST_detailed.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUy9M38H59ta"
      },
      "source": [
        "# A deep-learning neural network for image recognition\n",
        "We present here a `Python Keras` implementation of a deep learning neural network for image recognition.\n",
        "\n",
        "This is a detailed implementation with the code blocks from the previous implementation that are thoroughlly described and explained.\n",
        "\n",
        "The publicly available `MNIST` dataset is used.\n",
        "\n",
        "`Keras` is a popular `Python` library for deep learning models:\n",
        "- wrapper for `TensorFlow`\n",
        "- minimalistic\n",
        "- modular\n",
        "- easy to implement\n",
        "\n",
        "The `MNIST` database (Modified National Institute of Standards and Technology database) is a large database of hand-written digits (details and data [here](http://yann.lecun.com/exdb/mnist/)):\n",
        "\n",
        "![mnist](https://drive.google.com/uc?id=1KNK3-8qahQixvL-StpDAs6GoOUAHKSDy)\n",
        "\n",
        "Deep learning consists of neural networks with multiple hidden layers that learn increasingly abstract and complex representations of the input data.\n",
        "For instance, if we train a deep learning model to recognize hand-written digits (images):\n",
        "\n",
        "- the first hidden layers might only learn local edge patterns;\n",
        "- subsequent layers learns more complex representations of the data;\n",
        "- the last layer will classify the image as one of ten digits.\n",
        "\n",
        "For image recognition we use a specific deep learning architecture: **convolutional neural networks** (*CNN*), which assume that input data are images, thereby greatly reducing the number of model parameters to be tuned (more on *CNN's* later in the course).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EG18PhUOLMFk"
      },
      "source": [
        "## 1. SET UP\n",
        "\n",
        "- **Yesterday**: we loaded an external `python` script (`support_code.py`) to import libraries and set the seed 'behind the scenes'\n",
        "- **Today**: we import libraries and set the seed <u>manually</u>, \"in front of the public\"\n",
        "\n",
        "#### Importing libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqG72eSnLJYC"
      },
      "source": [
        "**Now that you learnt the basic syntax for Python and Keras you should be able to recognise (some of) this!**\n",
        "\n",
        "We import the necessary libraries to build a DL NN for image recognition:\n",
        "\n",
        "- import the Sequential model type from Keras: linear stack of neural network layers, to be used to build a feed-forward CNN\n",
        "-  import the 'core' layers from Keras: layers that are used in almost any neural network\n",
        "- import the CNN layers from Keras: convolutional layers to train the model on image data\n",
        "- load the MNIST dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bh-2gj8A5WAM"
      },
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow.keras.utils\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Activation, Flatten\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "from keras import backend as K # needed for image_data_format()\n",
        "from keras.datasets import mnist\n",
        "from tensorflow.keras.utils import to_categorical"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We also import the following libraries:\n",
        "\n",
        "- `numpy`\n",
        "- `matplotlib`\n",
        "- `sklearn`"
      ],
      "metadata": {
        "id": "ha1hnfNI72jj"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_QNtsrFaVgR7"
      },
      "source": [
        "#libraries\n",
        "import numpy as np\n",
        "\n",
        "#general random seed\n",
        "from numpy.random import seed\n",
        "\n",
        "## matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "## scikit-learn\n",
        "import sklearn.metrics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Setting the seed(s)\n",
        "\n",
        "We set the seed for the different libraries that make use of some random operations (e.g. data split, batches etc.)"
      ],
      "metadata": {
        "id": "BhmtOcT38T1B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## numpy seed\n",
        "np.random.seed(123)\n",
        "\n",
        "## tensorflow-specific seed\n",
        "tf.keras.utils.set_random_seed(10) ## python, numpy and tensorflow random seeds (all in one function call)\n",
        "tf.config.experimental.enable_op_determinism()"
      ],
      "metadata": {
        "id": "Nt-QJsS27wvZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rG7RUXrA52gj"
      },
      "source": [
        "## 2. LOAD THE DATA\n",
        "\n",
        "- **Yesterday**: we loaded data using the `load_data()` function\n",
        "- **Today**: we load the MNIST data and assign part to training and part to testing, manually\n",
        "\n",
        "We load the data from the MNIST dataset, and assign them to the training and testing sets.\n",
        "\n",
        "Image data is generally harder to work with than flat relational data. The MNIST dataset is a beginner-friendly intoduction to working with image data: it contains $70\\,000$ labeled images of handwritten digits. These are grey-scale images, 28 x 28 pixels.\n",
        "\n",
        "The MNIST dataset comprises $60\\,000$ training observations and $10\\,000$ test observations: the function `load_data()` automatically assigns these to the training and testing sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uaLJe1Vn0lXI"
      },
      "source": [
        "# the data, split between train and test sets\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
        "\n",
        "ntrain = 20000\n",
        "ntest = 5000\n",
        "\n",
        "X_train = X_train[0:ntrain,]\n",
        "y_train = y_train[0:ntrain]\n",
        "X_test = X_test[0:ntest,]\n",
        "y_test = y_test[0:ntest]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "A little sanity check:"
      ],
      "metadata": {
        "id": "U5xvoHyC-TJg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Size of the training set\")\n",
        "print(X_train.shape)\n",
        "print(\"Size of the test set\")\n",
        "print(X_test.shape)"
      ],
      "metadata": {
        "id": "MP3ZkZi4-PLx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgVK5wI4GetE"
      },
      "source": [
        "Data have been split into a **training** and a **testing set**, and within these into a **three-dimensional array** $X$ of **features** (samples x pixels x pixels) and a vector $y$ of labels (0-9 digits).\n",
        "\n",
        "Each record in the 3-D array $X$ is a 28 x 28 matrix of grayscale intensities (1 byte = 8 bits = 0 - 255 values). Grayscale (black-n-white) images only use one color channel. Colour images use three channels (e.g. RGB) and each image (record) is therefore a 3-D matrix (pixels x pixels x 3)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2jt6MZLCb5M6"
      },
      "source": [
        "## 3. CONFIGURE THE PARAMETERS\n",
        "\n",
        "- **Yesterday**: we set parameters 'behind the scenes' with the `set_parameters()` function\n",
        "- **Today**: we specify the parameters manually\n",
        "\n",
        "Define model parameters:\n",
        "\n",
        "- input shape\n",
        "- n. of classes: n. of classes to predict (10 digits, in the MNIST problem)\n",
        "- batch size: DL models typically do not process the entire dataset at once, rather break it in **batches**\n",
        "- n. of epochs: n. of **iterations** over the entire dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wljzYJtfbxji"
      },
      "source": [
        "img_rows = 28 #pixels\n",
        "img_cols = 28 #pixels\n",
        "num_classes = 10\n",
        "batch_size = 64\n",
        "num_epochs = 20"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Your turn! QUESTION: do you remember why we specify 10 classes? (`num_classes`)**"
      ],
      "metadata": {
        "id": "hrPBNJKOfwl_"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1i4lpSr9IbD"
      },
      "source": [
        "## 4. DATA PREPROCESSING\n",
        "\n",
        "- **Yesterday**: we processed data 'behind the scenes' with the `preprocess()` function\n",
        "- **Today**: we preprocess data <u>manually</u> (shape, range, labels)\n",
        "\n",
        "#### Get the data size right\n",
        "\n",
        "First, we need to explicitly declare the depth of the image representation array: in the case of grayscale images there is only one channel, and this dimension is 1.\n",
        "\n",
        "We use the utility function [image_data_format()](https://keras.io/api/utils/backend_utils#imagedataformat-function) from keras [backend utilities](https://keras.io/api/utils/backend_utils/) to discover the convention ('channels_first' or 'channels_last') of our current system.\n",
        "\n",
        "Depending on the backend (Theano or TensorFlow), the depth dimension is either the first or the last to be declared:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y0iC2KdoCyGv"
      },
      "source": [
        "if K.image_data_format() == 'channels_first':\n",
        "    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n",
        "    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)\n",
        "    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "print(\"Modified array dimensions:\")\n",
        "print(X_train.shape)\n",
        "print(input_shape)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uvOBH3WoLy6p"
      },
      "source": [
        "#### Normalization\n",
        "\n",
        "We then convert the input data type to `float32` and normalize the data values to the range $[0, 1]$.\n",
        "These are operational modifications necessary to speed up and optimize the calculations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ER8vhNco50si"
      },
      "source": [
        "X_train = X_train.astype('float32')\n",
        "X_test = X_test.astype('float32')\n",
        "X_train /= 255 #max value of pixel intensity\n",
        "X_test /= 255 #max value of pixel intensity\n",
        "print('X_train shape:', X_train.shape)\n",
        "print(X_train.shape[0], 'train samples')\n",
        "print(X_test.shape[0], 'test samples')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's have a look at the normalised data:"
      ],
      "metadata": {
        "id": "a9xFDdXmmsjw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = 5\n",
        "print(y_train[i])\n",
        "temp = np.copy(X_train)\n",
        "temp = temp.reshape(ntrain, img_rows, img_cols)\n",
        "plt.imshow(temp[i, :, :,], cmap='gray')\n",
        "print(np.round(X_train[i,3:18,0:12,0],3))"
      ],
      "metadata": {
        "id": "Dskeo97Tgida"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Labels format\n",
        "\n",
        "Finally, label vectors are converted to binary class matrices. This serves to convert a vector of numerical digits to a matrix of ten classes per observation, which is a better suited representation for a classification problem."
      ],
      "metadata": {
        "id": "Ncmdj2fg-2g-"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8pg9tUjtQX2d"
      },
      "source": [
        "# convert class vectors to binary class matrices (also known as OHE - One Hot Encoding)\n",
        "y_train = tf.keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = tf.keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sanity check:"
      ],
      "metadata": {
        "id": "8mPFcH8rm5dM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(y_test[0:6,])"
      ],
      "metadata": {
        "id": "XSpLiHfNm7R2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KlUJzLw46FvK"
      },
      "source": [
        "## 5. BUILD THE MODEL\n",
        "\n",
        "- **Yesterday**: we built the model 'behind the scenes' with the function `build_model()`\n",
        "- **Today**: we build the model <u>manually</u>, step-by-step, using what we learnt on **Keras/Tensorflow**\n",
        "\n",
        "We now define our deep-learning **neural network architecture**, and start building our model for image recognition.\n",
        "\n",
        "First, we declare a [sequential model](https://keras.io/guides/sequential_model/), that is a sequence of layers each with one input tensor and one output tensor.\n",
        "Then we add a first convolutional layer ([Conv2D](https://keras.io/api/layers/convolution_layers/convolution2d/)) to our model. We see that there's a bunch of additional parameters:\n",
        "\n",
        "- number of units (nodes)\n",
        "- size of the kernel (filter: much more on this later!)\n",
        "- type of activation function\n",
        "- shape of the input array\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "udCTMi2JWaY3"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(\n",
        "          Conv2D(32, kernel_size=(3, 3),\n",
        "          activation='relu',\n",
        "          input_shape=input_shape))\n",
        "\n",
        "print(model.input_shape) ## convolutional \"padding\" (28-2 x 28-2) + 32 kernels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y68JuKReaLOg"
      },
      "source": [
        "The input shape is (None, 28, 28, 1): 28 x 28 pixels, times 1 channel (grayscale), per 60,000 training samples.\n",
        "The convolutional output shape is:\n",
        "\n",
        "- None: not yet any samples trained (to be added later)\n",
        "- 28 pixel x 28 pixel greyscale image\n",
        "\n",
        "**Your turn! QUESTION: do you remember what `ReLU` is?**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then we can add more layers to the deep-learning model:"
      ],
      "metadata": {
        "id": "ehVjYVBOwZLc"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_wPciJ0ambwV"
      },
      "source": [
        "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(128, activation='relu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(num_classes, activation='softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yC8qRgETmd00"
      },
      "source": [
        "You'll encounter these new layers later on in the course: do not worry if you don't understand everything for the moment!\n",
        "\n",
        "A couple of things, though, you **already know**:\n",
        "- [Dense](https://keras.io/api/layers/core_layers/dense/) layers (see image below)\n",
        "- the [softmax](https://keras.io/api/layers/activations/#softmax-function) activation function (the multiclass analog of the logistic function) which returns a probability for each class, e.g. 10% of chance of the sample belonging to class 1, 15% for class 2 and so forth. The sum of all probabilities adds to 100%\n",
        "\n",
        "![dnn](https://drive.google.com/uc?id=1XD6xqrN3xQdaCSyQhWOiMlbxKqualeKo)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF9yHLD4sl4T"
      },
      "source": [
        "## 6. COMPILE THE MODEL\n",
        "\n",
        "- **Yesterday**: we compiled the model 'behind the scenes' using the function `compile_model()`\n",
        "- **Today**: we compile the model <u>manually</u>, looking at the specific 'compilation ingredients'\n",
        "\n",
        "When compiling the model we specify the **loss function** (here: [categorical_crossentropy](https://keras.io/api/losses/probabilistic_losses/#categoricalcrossentropy-class)), the **optimizer** (here: [Adadelta](https://keras.io/api/optimizers/adadelta/)) and the **metrics**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9COW8p3Z6IBQ"
      },
      "source": [
        "model.compile(loss=tensorflow.keras.losses.categorical_crossentropy,\n",
        "              optimizer=tf.keras.optimizers.Adadelta(),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FKeXuJK86QCJ"
      },
      "source": [
        "## TRAIN THE MODEL\n",
        "\n",
        "- **Yesterday**: we trained the model 'behind the scenes', using the function `train_model`\n",
        "- **Today**: we train the model <u>manually</u>, looking at the specific 'training ingredients'\n",
        "\n",
        "We then fit the model on the training data, specifying:\n",
        "\n",
        "- the batch size\n",
        "- the number of epochs to train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tv_dGPGu6RWx"
      },
      "source": [
        "model.fit(X_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=num_epochs,\n",
        "          verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G7n-6rZQ6TWk"
      },
      "source": [
        "## 8. TEST THE MODEL\n",
        "\n",
        "- **Yesterday**: we tested the model 'behind the scenes', using the function `evaluate_model()`\n",
        "- **Today**: we test the model <u>manually</u>, by caclulating predictions and using metrics to measure model performance\n",
        "\n",
        "#### Accuracy\n",
        "\n",
        "We can now measure the performance (in terms of prediction accuracy) of the trained deep-learning model for image recognition.\n",
        "To measure the performance, we applied our trained model to independent test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e0ZyOafK6UnN"
      },
      "source": [
        "score = model.evaluate(X_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3Rx5ww3ZfsO"
      },
      "source": [
        "#### Confusion matrix\n",
        "\n",
        "A [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) is another way to express the accuracy of your predictions. It's a square matrix, with as many rows (and columns) as your classes. Rows represent *true values* and columns represent *predicted values*. On the main diagonal are thus reported the correct predictions, while off-diagonal elements represent errors.\n",
        "\n",
        "We'll use the [confusion_matrix()](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) function part of [scikit-learn library](https://scikit-learn.org/stable/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bHY9mr6OXvDh"
      },
      "source": [
        "#asking our model to return its predictions for the test set\n",
        "predictions = model.predict(X_test)\n",
        "\n",
        "#confusion_matrix function requires actual classes labels (expressed as int)\n",
        "#and not probabilities as we handled so far\n",
        "predicted_classes = predictions.argmax(axis=1)\n",
        "true_classes = y_test.argmax(axis=1)\n",
        "\n",
        "#rows are true values, columns are predicted values, numbering starts from zero\n",
        "sklearn.metrics.confusion_matrix(true_classes, predicted_classes)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}