examples/gctf_horses_v_humans.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "gctf-horses-v-humans.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/main/examples/gctf_horses_v_humans.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IieqPex4O3eJ"
      },
      "source": [
        "# GCTF Horses vs Humans\r\n",
        "\r\n",
        "This notebook shows the the process of using the [`gradient-centralization-tf`](https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow) Python package to train on the [Horses vs Humans](http://www.laurencemoroney.com/horses-or-humans-dataset/) dataset by [Laurence Moroney](https://twitter.com/lmoroney). Gradient Centralization is a simple and effective optimization technique for Deep Neural Networks as suggested by Yong et al. in the paper \r\n",
        "[Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://arxiv.org/abs/2004.01461). It can both speedup training \r\n",
        " process and improve the final generalization performance of DNNs.\r\n",
        "\r\n",
        "If you find this useful please consider giving a ⭐ to [the repo](https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/).\r\n",
        "\r\n",
        "## A bit about GC\r\n",
        "\r\n",
        "Gradient Centralization operates directly on gradients by centralizing the gradient vectors to have zero mean. It can both speedup training process and improve the final generalization performance of DNNs. Here is an Illustration of the GC operation on gradient matrix/tensor of weights in the fully-connected layer (left) and convolutional layer (right). GC computes the column/slice mean of gradient matrix/tensor and centralizes each column/slice to have zero mean.\r\n",
        "\r\n",
        "![](https://i.imgur.com/KitoO8J.png)\r\n",
        "\r\n",
        "GC can be viewed as a projected gradient descent method with a constrained loss function. The geometrical interpretation of GC. The gradient is projected on a hyperplane $e^T(w-w^t)=0$, where the projected gradient is used to update the weight.\r\n",
        "\r\n",
        "![](https://i.imgur.com/ekHhQv0.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WcpkfjkwSjmv"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XjTQqCEaTSSG"
      },
      "source": [
        "import tensorflow as tf\r\n",
        "from time import time"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32NuQrDdSpP3"
      },
      "source": [
        "### Install the package\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qce8MP-42vkt",
        "outputId": "571100f0-5735-42eb-d9c5-1868db53e755"
      },
      "source": [
        "!pip install gradient-centralization-tf"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting gradient-centralization-tf\n",
            "  Downloading https://files.pythonhosted.org/packages/58/4c/6253587b8f6ccdf03fd4830de2574cbda48a1a84bc660d5dd8978d0f94fb/gradient_centralization_tf-0.0.2-py3-none-any.whl\n",
            "Requirement already satisfied: tensorflow~=2.4.0 in /usr/local/lib/python3.6/dist-packages (from gradient-centralization-tf) (2.4.1)\n",
            "Requirement already satisfied: keras~=2.4.0 in /usr/local/lib/python3.6/dist-packages (from gradient-centralization-tf) (2.4.3)\n",
            "Requirement already satisfied: h5py~=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (2.10.0)\n",
            "Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.1.2)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (0.3.3)\n",
            "Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (0.2.0)\n",
            "Requirement already satisfied: grpcio~=1.32.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.32.0)\n",
            "Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (3.7.4.3)\n",
            "Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (0.10.0)\n",
            "Requirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.19.5)\n",
            "Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.15.0)\n",
            "Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.1.0)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (3.12.4)\n",
            "Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (0.36.2)\n",
            "Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (2.4.0)\n",
            "Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.12.1)\n",
            "Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.12)\n",
            "Requirement already satisfied: tensorboard~=2.4 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (2.4.1)\n",
            "Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow~=2.4.0->gradient-centralization-tf) (1.6.3)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras~=2.4.0->gradient-centralization-tf) (1.4.1)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras~=2.4.0->gradient-centralization-tf) (3.13)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.9.2->tensorflow~=2.4.0->gradient-centralization-tf) (53.0.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (1.8.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (0.4.2)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (3.3.3)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (2.23.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (1.0.1)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (1.25.0)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (3.4.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (2020.12.5)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (2.10)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (4.2.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (4.7)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (3.1.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (3.4.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow~=2.4.0->gradient-centralization-tf) (0.4.8)\n",
            "Installing collected packages: gradient-centralization-tf\n",
            "Successfully installed gradient-centralization-tf-0.0.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-lVvc6uKHDYI"
      },
      "source": [
        "## Get the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5lEoHGCV61Bv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "87e66434-eaba-4892-d2b1-99972e63305a"
      },
      "source": [
        "!wget --no-check-certificate \\\r\n",
        "    https://storage.googleapis.com/laurencemoroney-blog.appspot.com/horse-or-human.zip \\\r\n",
        "    -O /tmp/horse-or-human.zip\r\n",
        "\r\n",
        "!wget --no-check-certificate \\\r\n",
        "    https://storage.googleapis.com/laurencemoroney-blog.appspot.com/validation-horse-or-human.zip \\\r\n",
        "    -O /tmp/validation-horse-or-human.zip\r\n",
        "  \r\n",
        "import os\r\n",
        "import zipfile\r\n",
        "\r\n",
        "local_zip = '/tmp/horse-or-human.zip'\r\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\r\n",
        "zip_ref.extractall('/tmp/horse-or-human')\r\n",
        "local_zip = '/tmp/validation-horse-or-human.zip'\r\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\r\n",
        "zip_ref.extractall('/tmp/validation-horse-or-human')\r\n",
        "zip_ref.close()\r\n",
        "# Directory with our training horse pictures\r\n",
        "train_horse_dir = os.path.join('/tmp/horse-or-human/horses')\r\n",
        "\r\n",
        "# Directory with our training human pictures\r\n",
        "train_human_dir = os.path.join('/tmp/horse-or-human/humans')\r\n",
        "\r\n",
        "# Directory with our training horse pictures\r\n",
        "validation_horse_dir = os.path.join('/tmp/validation-horse-or-human/horses')\r\n",
        "\r\n",
        "# Directory with our training human pictures\r\n",
        "validation_human_dir = os.path.join('/tmp/validation-horse-or-human/humans')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-02-21 12:08:31--  https://storage.googleapis.com/laurencemoroney-blog.appspot.com/horse-or-human.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.115.128, 172.253.122.128, 142.250.31.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.115.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 149574867 (143M) [application/zip]\n",
            "Saving to: ‘/tmp/horse-or-human.zip’\n",
            "\n",
            "/tmp/horse-or-human 100%[===================>] 142.65M   262MB/s    in 0.5s    \n",
            "\n",
            "2021-02-21 12:08:31 (262 MB/s) - ‘/tmp/horse-or-human.zip’ saved [149574867/149574867]\n",
            "\n",
            "--2021-02-21 12:08:32--  https://storage.googleapis.com/laurencemoroney-blog.appspot.com/validation-horse-or-human.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.164.144, 142.250.73.208, 172.253.62.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.164.144|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 11480187 (11M) [application/zip]\n",
            "Saving to: ‘/tmp/validation-horse-or-human.zip’\n",
            "\n",
            "/tmp/validation-hor 100%[===================>]  10.95M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2021-02-21 12:08:32 (93.0 MB/s) - ‘/tmp/validation-horse-or-human.zip’ saved [11480187/11480187]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UsLyqydwHPOZ"
      },
      "source": [
        "## Image Augmentation\r\n",
        "\r\n",
        "We will perform a couple of augmentations on the image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JUagrOo6wHJP",
        "outputId": "873ec40e-58a0-4aa6-91d0-4d80cd810947"
      },
      "source": [
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\r\n",
        "\r\n",
        "# All images will be rescaled by 1./255\r\n",
        "train_datagen = ImageDataGenerator(\r\n",
        "      rescale=1./255,\r\n",
        "      rotation_range=40,\r\n",
        "      width_shift_range=0.2,\r\n",
        "      height_shift_range=0.2,\r\n",
        "      shear_range=0.2,\r\n",
        "      zoom_range=0.2,\r\n",
        "      horizontal_flip=True,\r\n",
        "      fill_mode='nearest')\r\n",
        "\r\n",
        "validation_datagen = ImageDataGenerator(rescale=1/255)\r\n",
        "\r\n",
        "# Flow training images in batches of 128 using train_datagen generator\r\n",
        "train_generator = train_datagen.flow_from_directory(\r\n",
        "        '/tmp/horse-or-human/',  # This is the source directory for training images\r\n",
        "        target_size=(300, 300),  # All images will be resized to 150x150\r\n",
        "        batch_size=128,\r\n",
        "        # Since we use binary_crossentropy loss, we need binary labels\r\n",
        "        class_mode='binary')\r\n",
        "\r\n",
        "# Flow training images in batches of 128 using train_datagen generator\r\n",
        "validation_generator = validation_datagen.flow_from_directory(\r\n",
        "        '/tmp/validation-horse-or-human/',  # This is the source directory for training images\r\n",
        "        target_size=(300, 300),  # All images will be resized to 150x150\r\n",
        "        batch_size=32,\r\n",
        "        # Since we use binary_crossentropy loss, we need binary labels\r\n",
        "        class_mode='binary')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 1027 images belonging to 2 classes.\n",
            "Found 256 images belonging to 2 classes.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JCcBZ1UTHVc3"
      },
      "source": [
        "## Training the model\r\n",
        "\r\n",
        "Here we have built a very simple model with 5 Convolutional for this example. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LuW4o9yyvlg_"
      },
      "source": [
        "model = tf.keras.models.Sequential([\r\n",
        "    # Note the input shape is the desired size of the image 300x300 with 3 bytes color\r\n",
        "    # This is the first convolution\r\n",
        "    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(300, 300, 3)),\r\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\r\n",
        "    # The second convolution\r\n",
        "    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\r\n",
        "    tf.keras.layers.Dropout(0.5),\r\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\r\n",
        "    # The third convolution\r\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\r\n",
        "    tf.keras.layers.Dropout(0.5),\r\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\r\n",
        "    # The fourth convolution\r\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\r\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\r\n",
        "    # The fifth convolution\r\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\r\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\r\n",
        "    # Flatten the results to feed into a DNN\r\n",
        "    \r\n",
        "    tf.keras.layers.Flatten(),\r\n",
        "    tf.keras.layers.Dropout(0.5),\r\n",
        "    # 512 neuron hidden layer\r\n",
        "    tf.keras.layers.Dense(512, activation='relu'),\r\n",
        "    # Only 1 output neuron. It will contain a value from 0-1 where 0 for 1 class ('horses') and 1 for the other ('humans')\r\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\r\n",
        "])"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMBGO9JbHgWG"
      },
      "source": [
        "On the same note since we are interested in comparing results we will create a callback which allows us to compute the training time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pfk5BFgewUfq"
      },
      "source": [
        "class TimeHistory(tf.keras.callbacks.Callback):\r\n",
        "    def on_train_begin(self, logs={}):\r\n",
        "        self.times = []\r\n",
        "\r\n",
        "    def on_epoch_begin(self, batch, logs={}):\r\n",
        "        self.epoch_time_start = time()\r\n",
        "\r\n",
        "    def on_epoch_end(self, batch, logs={}):\r\n",
        "        self.times.append(time() - self.epoch_time_start)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xf1NPKe3HqVf"
      },
      "source": [
        "### Train a model without [`gctf`](https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m-_AtY2fwCDZ",
        "outputId": "38f19bc7-c7b1-4d9b-e543-4d31cbc51105",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from tensorflow.keras.optimizers import RMSprop\r\n",
        "\r\n",
        "time_callback_no_gctf = TimeHistory()\r\n",
        "model.compile(loss='binary_crossentropy',\r\n",
        "              optimizer=RMSprop(lr=1e-4),\r\n",
        "              metrics=['accuracy'])\r\n",
        "\r\n",
        "history_no_gctf = model.fit(\r\n",
        "      train_generator,\r\n",
        "      steps_per_epoch=8,  \r\n",
        "      epochs=10,\r\n",
        "      verbose=1,\r\n",
        "      validation_data = validation_generator,\r\n",
        "      validation_steps=8,\r\n",
        "      callbacks = [time_callback_no_gctf])"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "8/8 [==============================] - 27s 2s/step - loss: 0.7229 - accuracy: 0.4852 - val_loss: 0.6898 - val_accuracy: 0.5000\n",
            "Epoch 2/10\n",
            "8/8 [==============================] - 22s 3s/step - loss: 0.6871 - accuracy: 0.5560 - val_loss: 0.6858 - val_accuracy: 0.5234\n",
            "Epoch 3/10\n",
            "8/8 [==============================] - 22s 3s/step - loss: 0.6732 - accuracy: 0.6040 - val_loss: 0.6801 - val_accuracy: 0.5508\n",
            "Epoch 4/10\n",
            "8/8 [==============================] - 22s 3s/step - loss: 0.6343 - accuracy: 0.6694 - val_loss: 0.6916 - val_accuracy: 0.5000\n",
            "Epoch 5/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.6548 - accuracy: 0.6131 - val_loss: 0.6718 - val_accuracy: 0.8281\n",
            "Epoch 6/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5896 - accuracy: 0.6966 - val_loss: 0.6733 - val_accuracy: 0.5000\n",
            "Epoch 7/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5870 - accuracy: 0.7058 - val_loss: 0.6604 - val_accuracy: 0.6094\n",
            "Epoch 8/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5534 - accuracy: 0.7235 - val_loss: 0.6887 - val_accuracy: 0.5000\n",
            "Epoch 9/10\n",
            "8/8 [==============================] - 23s 3s/step - loss: 0.5626 - accuracy: 0.7112 - val_loss: 0.6570 - val_accuracy: 0.5586\n",
            "Epoch 10/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5607 - accuracy: 0.7258 - val_loss: 0.6463 - val_accuracy: 0.6016\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96YKj6_GH5Yf"
      },
      "source": [
        "### Train a model with [`gctf`](https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRNC9FT0wjY0",
        "outputId": "e74de2a8-4ceb-4713-f35c-a7f88b84704b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import gctf #import gctf\r\n",
        "\r\n",
        "time_callback_gctf = TimeHistory()\r\n",
        "model.compile(loss='binary_crossentropy',\r\n",
        "              optimizer=gctf.optimizers.rmsprop(learning_rate = 1e-4),\r\n",
        "              metrics=['accuracy'])\r\n",
        "\r\n",
        "history_gctf = model.fit(\r\n",
        "      train_generator,\r\n",
        "      steps_per_epoch=8,  \r\n",
        "      epochs=10,\r\n",
        "      verbose=1,\r\n",
        "      validation_data = validation_generator,\r\n",
        "      validation_steps=8,\r\n",
        "      callbacks = [time_callback_gctf])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "8/8 [==============================] - 24s 3s/step - loss: 0.6394 - accuracy: 0.6779 - val_loss: 0.6885 - val_accuracy: 0.5000\n",
            "Epoch 2/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5504 - accuracy: 0.7124 - val_loss: 0.6450 - val_accuracy: 0.5625\n",
            "Epoch 3/10\n",
            "8/8 [==============================] - 22s 3s/step - loss: 0.5050 - accuracy: 0.7673 - val_loss: 0.6163 - val_accuracy: 0.6094\n",
            "Epoch 4/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5206 - accuracy: 0.7589 - val_loss: 0.5969 - val_accuracy: 0.6797\n",
            "Epoch 5/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5175 - accuracy: 0.7506 - val_loss: 0.7745 - val_accuracy: 0.5000\n",
            "Epoch 6/10\n",
            "8/8 [==============================] - 23s 3s/step - loss: 0.6449 - accuracy: 0.6996 - val_loss: 0.6114 - val_accuracy: 0.5820\n",
            "Epoch 7/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.5059 - accuracy: 0.7551 - val_loss: 0.5494 - val_accuracy: 0.7461\n",
            "Epoch 8/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.4751 - accuracy: 0.7774 - val_loss: 0.5426 - val_accuracy: 0.7461\n",
            "Epoch 9/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.4755 - accuracy: 0.7816 - val_loss: 0.5948 - val_accuracy: 0.6172\n",
            "Epoch 10/10\n",
            "8/8 [==============================] - 21s 3s/step - loss: 0.4431 - accuracy: 0.7922 - val_loss: 0.7306 - val_accuracy: 0.5273\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P6_aG1L_H-Ko"
      },
      "source": [
        "## Compare results\r\n",
        "\r\n",
        "In this example we are further interested in also comparing the results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zOg1wLwrfpqd",
        "outputId": "2a00e0f6-c9b0-44bb-ce8d-e4f373777b38",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from tabulate import tabulate\r\n",
        "\r\n",
        "data = [[\"Model without gctf:\",sum(time_callback_no_gctf.times),history_no_gctf.history['accuracy'][-1],history_no_gctf.history['loss'][-1]],\r\n",
        "        [\"Model with gctf\",sum(time_callback_gctf.times),history_gctf.history['accuracy'][-1],history_gctf.history['loss'][-1]]] \r\n",
        "\r\n",
        "print(tabulate(data, headers=[\"Type\",\"Execution time\", \"Accuracy\", \"Loss\"]))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type                   Execution time    Accuracy      Loss\n",
            "-------------------  ----------------  ----------  --------\n",
            "Model without gctf:           221.626    0.690768  0.625912\n",
            "Model with gctf               216.744    0.805339  0.426568\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Y5WxRlBf45e"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}