t81_558_class_03_3_save_load.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wDNsTn6vLDUJ"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fb-QhI-oLDUL"
      },
      "source": [
        "# T81-558: Applications of Deep Neural Networks\n",
        "**Module 3: Introduction to TensorFlow**\n",
        "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
        "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idZkAn99LDUL"
      },
      "source": [
        "# Module 3 Material\n",
        "\n",
        "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb)\n",
        "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb)\n",
        "* **Part 3.3: Saving and Loading a Keras Neural Network** [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb)\n",
        "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_4_early_stop.ipynb)\n",
        "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_5_weights.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKQylnEiLDUM"
      },
      "source": [
        "# Google CoLab Instructions\n",
        "\n",
        "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n",
        "  Running the following code will map your GDrive to ```/content/drive```."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "seXFCYH4LDUM",
        "outputId": "c05015aa-871e-4779-9265-5ad07e8bf617",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n",
            "Note: using Google CoLab\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    from google.colab import drive\n",
        "    drive.mount('/content/drive', force_remount=True)\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "    %tensorflow_version 2.x\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihLGqmbXLDUN"
      },
      "source": [
        "# Part 3.3: Saving and Loading a Keras Neural Network\n",
        "\n",
        "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that you can reload them later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n",
        "\n",
        "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n",
        "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n",
        "\n",
        "Usually, you will want to save in HDF5."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "RYGcgGF_LDUN",
        "outputId": "69112820-bdb1-4842-c2e3-4d8e3bc50835",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/100\n",
            "13/13 - 1s - loss: 223035.4531 - 729ms/epoch - 56ms/step\n",
            "Epoch 2/100\n",
            "13/13 - 0s - loss: 94454.9609 - 26ms/epoch - 2ms/step\n",
            "Epoch 3/100\n",
            "13/13 - 0s - loss: 31163.9199 - 26ms/epoch - 2ms/step\n",
            "Epoch 4/100\n",
            "13/13 - 0s - loss: 6590.2344 - 24ms/epoch - 2ms/step\n",
            "Epoch 5/100\n",
            "13/13 - 0s - loss: 692.0208 - 23ms/epoch - 2ms/step\n",
            "Epoch 6/100\n",
            "13/13 - 0s - loss: 110.7149 - 23ms/epoch - 2ms/step\n",
            "Epoch 7/100\n",
            "13/13 - 0s - loss: 154.5042 - 22ms/epoch - 2ms/step\n",
            "Epoch 8/100\n",
            "13/13 - 0s - loss: 118.5529 - 29ms/epoch - 2ms/step\n",
            "Epoch 9/100\n",
            "13/13 - 0s - loss: 91.5691 - 27ms/epoch - 2ms/step\n",
            "Epoch 10/100\n",
            "13/13 - 0s - loss: 85.7397 - 24ms/epoch - 2ms/step\n",
            "Epoch 11/100\n",
            "13/13 - 0s - loss: 85.6981 - 23ms/epoch - 2ms/step\n",
            "Epoch 12/100\n",
            "13/13 - 0s - loss: 85.2837 - 24ms/epoch - 2ms/step\n",
            "Epoch 13/100\n",
            "13/13 - 0s - loss: 84.9037 - 23ms/epoch - 2ms/step\n",
            "Epoch 14/100\n",
            "13/13 - 0s - loss: 84.6506 - 30ms/epoch - 2ms/step\n",
            "Epoch 15/100\n",
            "13/13 - 0s - loss: 84.4048 - 26ms/epoch - 2ms/step\n",
            "Epoch 16/100\n",
            "13/13 - 0s - loss: 84.1072 - 24ms/epoch - 2ms/step\n",
            "Epoch 17/100\n",
            "13/13 - 0s - loss: 83.9168 - 23ms/epoch - 2ms/step\n",
            "Epoch 18/100\n",
            "13/13 - 0s - loss: 83.7391 - 24ms/epoch - 2ms/step\n",
            "Epoch 19/100\n",
            "13/13 - 0s - loss: 83.1922 - 21ms/epoch - 2ms/step\n",
            "Epoch 20/100\n",
            "13/13 - 0s - loss: 82.9178 - 27ms/epoch - 2ms/step\n",
            "Epoch 21/100\n",
            "13/13 - 0s - loss: 82.5835 - 28ms/epoch - 2ms/step\n",
            "Epoch 22/100\n",
            "13/13 - 0s - loss: 82.2728 - 24ms/epoch - 2ms/step\n",
            "Epoch 23/100\n",
            "13/13 - 0s - loss: 81.9899 - 24ms/epoch - 2ms/step\n",
            "Epoch 24/100\n",
            "13/13 - 0s - loss: 81.7262 - 23ms/epoch - 2ms/step\n",
            "Epoch 25/100\n",
            "13/13 - 0s - loss: 81.2958 - 26ms/epoch - 2ms/step\n",
            "Epoch 26/100\n",
            "13/13 - 0s - loss: 80.9488 - 30ms/epoch - 2ms/step\n",
            "Epoch 27/100\n",
            "13/13 - 0s - loss: 80.5811 - 33ms/epoch - 3ms/step\n",
            "Epoch 28/100\n",
            "13/13 - 0s - loss: 80.3213 - 25ms/epoch - 2ms/step\n",
            "Epoch 29/100\n",
            "13/13 - 0s - loss: 79.8659 - 27ms/epoch - 2ms/step\n",
            "Epoch 30/100\n",
            "13/13 - 0s - loss: 79.5628 - 24ms/epoch - 2ms/step\n",
            "Epoch 31/100\n",
            "13/13 - 0s - loss: 79.2613 - 24ms/epoch - 2ms/step\n",
            "Epoch 32/100\n",
            "13/13 - 0s - loss: 78.8549 - 23ms/epoch - 2ms/step\n",
            "Epoch 33/100\n",
            "13/13 - 0s - loss: 78.3649 - 23ms/epoch - 2ms/step\n",
            "Epoch 34/100\n",
            "13/13 - 0s - loss: 78.0478 - 23ms/epoch - 2ms/step\n",
            "Epoch 35/100\n",
            "13/13 - 0s - loss: 77.6581 - 30ms/epoch - 2ms/step\n",
            "Epoch 36/100\n",
            "13/13 - 0s - loss: 77.1970 - 24ms/epoch - 2ms/step\n",
            "Epoch 37/100\n",
            "13/13 - 0s - loss: 76.8659 - 23ms/epoch - 2ms/step\n",
            "Epoch 38/100\n",
            "13/13 - 0s - loss: 76.6319 - 23ms/epoch - 2ms/step\n",
            "Epoch 39/100\n",
            "13/13 - 0s - loss: 76.0007 - 24ms/epoch - 2ms/step\n",
            "Epoch 40/100\n",
            "13/13 - 0s - loss: 75.5929 - 25ms/epoch - 2ms/step\n",
            "Epoch 41/100\n",
            "13/13 - 0s - loss: 75.2667 - 26ms/epoch - 2ms/step\n",
            "Epoch 42/100\n",
            "13/13 - 0s - loss: 75.3607 - 24ms/epoch - 2ms/step\n",
            "Epoch 43/100\n",
            "13/13 - 0s - loss: 74.5779 - 27ms/epoch - 2ms/step\n",
            "Epoch 44/100\n",
            "13/13 - 0s - loss: 73.9867 - 21ms/epoch - 2ms/step\n",
            "Epoch 45/100\n",
            "13/13 - 0s - loss: 73.7650 - 25ms/epoch - 2ms/step\n",
            "Epoch 46/100\n",
            "13/13 - 0s - loss: 73.0263 - 24ms/epoch - 2ms/step\n",
            "Epoch 47/100\n",
            "13/13 - 0s - loss: 72.7102 - 23ms/epoch - 2ms/step\n",
            "Epoch 48/100\n",
            "13/13 - 0s - loss: 72.2177 - 28ms/epoch - 2ms/step\n",
            "Epoch 49/100\n",
            "13/13 - 0s - loss: 71.8469 - 22ms/epoch - 2ms/step\n",
            "Epoch 50/100\n",
            "13/13 - 0s - loss: 71.4904 - 28ms/epoch - 2ms/step\n",
            "Epoch 51/100\n",
            "13/13 - 0s - loss: 71.1223 - 25ms/epoch - 2ms/step\n",
            "Epoch 52/100\n",
            "13/13 - 0s - loss: 70.5943 - 25ms/epoch - 2ms/step\n",
            "Epoch 53/100\n",
            "13/13 - 0s - loss: 70.1748 - 21ms/epoch - 2ms/step\n",
            "Epoch 54/100\n",
            "13/13 - 0s - loss: 69.8101 - 25ms/epoch - 2ms/step\n",
            "Epoch 55/100\n",
            "13/13 - 0s - loss: 69.3219 - 23ms/epoch - 2ms/step\n",
            "Epoch 56/100\n",
            "13/13 - 0s - loss: 68.7525 - 22ms/epoch - 2ms/step\n",
            "Epoch 57/100\n",
            "13/13 - 0s - loss: 68.4256 - 22ms/epoch - 2ms/step\n",
            "Epoch 58/100\n",
            "13/13 - 0s - loss: 67.8394 - 23ms/epoch - 2ms/step\n",
            "Epoch 59/100\n",
            "13/13 - 0s - loss: 67.4138 - 22ms/epoch - 2ms/step\n",
            "Epoch 60/100\n",
            "13/13 - 0s - loss: 66.9941 - 33ms/epoch - 3ms/step\n",
            "Epoch 61/100\n",
            "13/13 - 0s - loss: 66.6573 - 29ms/epoch - 2ms/step\n",
            "Epoch 62/100\n",
            "13/13 - 0s - loss: 66.1712 - 22ms/epoch - 2ms/step\n",
            "Epoch 63/100\n",
            "13/13 - 0s - loss: 65.8375 - 29ms/epoch - 2ms/step\n",
            "Epoch 64/100\n",
            "13/13 - 0s - loss: 65.3441 - 23ms/epoch - 2ms/step\n",
            "Epoch 65/100\n",
            "13/13 - 0s - loss: 64.9143 - 22ms/epoch - 2ms/step\n",
            "Epoch 66/100\n",
            "13/13 - 0s - loss: 64.7354 - 24ms/epoch - 2ms/step\n",
            "Epoch 67/100\n",
            "13/13 - 0s - loss: 63.8731 - 30ms/epoch - 2ms/step\n",
            "Epoch 68/100\n",
            "13/13 - 0s - loss: 63.5211 - 26ms/epoch - 2ms/step\n",
            "Epoch 69/100\n",
            "13/13 - 0s - loss: 62.9679 - 22ms/epoch - 2ms/step\n",
            "Epoch 70/100\n",
            "13/13 - 0s - loss: 62.6917 - 21ms/epoch - 2ms/step\n",
            "Epoch 71/100\n",
            "13/13 - 0s - loss: 62.1212 - 22ms/epoch - 2ms/step\n",
            "Epoch 72/100\n",
            "13/13 - 0s - loss: 62.1577 - 32ms/epoch - 2ms/step\n",
            "Epoch 73/100\n",
            "13/13 - 0s - loss: 61.1758 - 22ms/epoch - 2ms/step\n",
            "Epoch 74/100\n",
            "13/13 - 0s - loss: 61.0303 - 24ms/epoch - 2ms/step\n",
            "Epoch 75/100\n",
            "13/13 - 0s - loss: 60.5673 - 23ms/epoch - 2ms/step\n",
            "Epoch 76/100\n",
            "13/13 - 0s - loss: 60.0197 - 24ms/epoch - 2ms/step\n",
            "Epoch 77/100\n",
            "13/13 - 0s - loss: 59.7046 - 24ms/epoch - 2ms/step\n",
            "Epoch 78/100\n",
            "13/13 - 0s - loss: 59.0460 - 25ms/epoch - 2ms/step\n",
            "Epoch 79/100\n",
            "13/13 - 0s - loss: 58.6879 - 27ms/epoch - 2ms/step\n",
            "Epoch 80/100\n",
            "13/13 - 0s - loss: 58.2086 - 28ms/epoch - 2ms/step\n",
            "Epoch 81/100\n",
            "13/13 - 0s - loss: 58.1870 - 40ms/epoch - 3ms/step\n",
            "Epoch 82/100\n",
            "13/13 - 0s - loss: 57.3580 - 35ms/epoch - 3ms/step\n",
            "Epoch 83/100\n",
            "13/13 - 0s - loss: 57.0140 - 24ms/epoch - 2ms/step\n",
            "Epoch 84/100\n",
            "13/13 - 0s - loss: 56.5466 - 36ms/epoch - 3ms/step\n",
            "Epoch 85/100\n",
            "13/13 - 0s - loss: 56.2083 - 30ms/epoch - 2ms/step\n",
            "Epoch 86/100\n",
            "13/13 - 0s - loss: 55.7131 - 24ms/epoch - 2ms/step\n",
            "Epoch 87/100\n",
            "13/13 - 0s - loss: 55.2924 - 28ms/epoch - 2ms/step\n",
            "Epoch 88/100\n",
            "13/13 - 0s - loss: 54.9157 - 26ms/epoch - 2ms/step\n",
            "Epoch 89/100\n",
            "13/13 - 0s - loss: 54.8022 - 27ms/epoch - 2ms/step\n",
            "Epoch 90/100\n",
            "13/13 - 0s - loss: 53.9416 - 25ms/epoch - 2ms/step\n",
            "Epoch 91/100\n",
            "13/13 - 0s - loss: 53.6013 - 30ms/epoch - 2ms/step\n",
            "Epoch 92/100\n",
            "13/13 - 0s - loss: 53.3547 - 25ms/epoch - 2ms/step\n",
            "Epoch 93/100\n",
            "13/13 - 0s - loss: 52.7261 - 39ms/epoch - 3ms/step\n",
            "Epoch 94/100\n",
            "13/13 - 0s - loss: 52.3562 - 25ms/epoch - 2ms/step\n",
            "Epoch 95/100\n",
            "13/13 - 0s - loss: 51.9567 - 23ms/epoch - 2ms/step\n",
            "Epoch 96/100\n",
            "13/13 - 0s - loss: 51.4552 - 29ms/epoch - 2ms/step\n",
            "Epoch 97/100\n",
            "13/13 - 0s - loss: 51.4597 - 23ms/epoch - 2ms/step\n",
            "Epoch 98/100\n",
            "13/13 - 0s - loss: 50.6219 - 24ms/epoch - 2ms/step\n",
            "Epoch 99/100\n",
            "13/13 - 0s - loss: 50.2118 - 25ms/epoch - 2ms/step\n",
            "Epoch 100/100\n",
            "13/13 - 0s - loss: 49.8828 - 25ms/epoch - 2ms/step\n",
            "Before save score (RMSE): 7.044431690300903\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation\n",
        "import pandas as pd\n",
        "import io\n",
        "import os\n",
        "import requests\n",
        "import numpy as np\n",
        "from sklearn import metrics\n",
        "\n",
        "save_path = \".\"\n",
        "\n",
        "df = pd.read_csv(\n",
        "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
        "    na_values=['NA', '?'])\n",
        "\n",
        "cars = df['name']\n",
        "\n",
        "# Handle missing value\n",
        "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
        "\n",
        "# Pandas to Numpy\n",
        "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
        "       'acceleration', 'year', 'origin']].values\n",
        "y = df['mpg'].values # regression\n",
        "\n",
        "# Build the neural network\n",
        "model = Sequential()\n",
        "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
        "model.add(Dense(10, activation='relu')) # Hidden 2\n",
        "model.add(Dense(1)) # Output\n",
        "model.compile(loss='mean_squared_error', optimizer='adam')\n",
        "model.fit(x,y,verbose=2,epochs=100)\n",
        "\n",
        "# Predict\n",
        "pred = model.predict(x)\n",
        "\n",
        "# Measure RMSE error.  RMSE is common for regression.\n",
        "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
        "print(f\"Before save score (RMSE): {score}\")\n",
        "\n",
        "# save neural network structure to JSON (no weights)\n",
        "model_json = model.to_json()\n",
        "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
        "    json_file.write(model_json)\n",
        "\n",
        "# save entire network to HDF5 (save everything, suggested)\n",
        "model.save(os.path.join(save_path,\"network.h5\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lRH-rnocLDUO"
      },
      "source": [
        "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network. The code loads the weights from the previous fit. Now we reload the network and perform another prediction. The RMSE should match the previous one exactly if we saved and reloaded the neural network correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "XhbEI5jjLDUO",
        "outputId": "51404d9a-579d-4bc2-b636-68054cb5e0af",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "After load score (RMSE): 7.044431690300903\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.models import load_model\n",
        "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n",
        "pred = model2.predict(x)\n",
        "# Measure RMSE error.  RMSE is common for regression.\n",
        "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
        "print(f\"After load score (RMSE): {score}\")"
      ]
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3.9 (tensorflow)",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "name": "t81_558_class_03_3_save_load.ipynb",
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}