t81_558_class_09_1_keras_transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ht3sZclXu-GP"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_1_keras_transfer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q2BV1TL0u-GR"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Transfer Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nSeaIPICu-GS"
   },
   "source": [
    "# Module 9 Material\n",
    "\n",
    "* **Part 9.1: Introduction to Keras Transfer Learning** [[Video]](https://www.youtube.com/watch?v=AtoeoNwmd7w&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_1_keras_transfer.ipynb)\n",
    "* Part 9.2: Keras Transfer Learning for Computer Vision [[Video]](https://www.youtube.com/watch?v=nXcz0V5SfYw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_2_keras_xfer_cv.ipynb)\n",
    "* Part 9.3: Transfer Learning for NLP with Keras [[Video]](https://www.youtube.com/watch?v=PyRsjwLHgAU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_3_transfer_nlp.ipynb)\n",
    "* Part 9.4: Transfer Learning for Facial Feature Recognition [[Video]](https://www.youtube.com/watch?v=uUZg33DfCls&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_4_facial_points.ipynb)\n",
    "* Part 9.5: Transfer Learning for Style Transfer [[Video]](https://www.youtube.com/watch?v=pLWIaQwkJwU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_5_style_transfer.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gyyeB513u-GS"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Okmp94eLu-GS",
    "outputId": "7fbd44ba-256a-4645-9a46-7ffc3ace8f31"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "# Start CoLab\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NE0HusRwu-GT"
   },
   "source": [
    "# Part 9.1: Introduction to Keras Transfer Learning\n",
    "\n",
    "Human beings learn new skills throughout their entire lives. However, this learning is rarely from scratch. No matter what task a human learns, they are most likely drawing on experiences to learn this new skill early in life. In this way, humans learn much differently than most deep learning projects. \n",
    "\n",
    "A human being learns to tell the difference between a cat and a dog at some point. To teach a neural network, you would obtain many cat pictures and dog pictures. The neural network would iterate over all of these pictures and train on the differences. The human child that learned to distinguish between the two animals would probably need to see a few examples when parents told them the name of each type of animal. The human child would use previous knowledge of looking at different living and non-living objects to help make this classification. The child would already know the physical appearance of sub-objects, such as fur, eyes, ears, noses, tails, and teeth.\n",
    "\n",
    "Transfer learning attempts to teach a neural network by similar means. Rather than training your neural network from scratch, you begin training with a preloaded set of weights. Usually, you will remove the topmost layers of the pretrained neural network and retrain it with new uppermost layers. The layers from the previous neural network will be locked so that training does not change these weights. Only the newly added layers will be trained.  \n",
    "\n",
    "It can take much computing power to train a neural network for a large image dataset. Google, Facebook, Microsoft, and other tech companies have utilized GPU arrays for training high-quality neural networks for various applications. Transferring these weights into your neural network can save considerable effort and compute time. It is unlikely that a pretrained model will exactly fit the application that you seek to implement. Finding the closest pretrained model and using transfer learning is essential for a deep learning engineer.\n",
    "\n",
    "## Transfer Learning Example\n",
    "\n",
    "Let's look at a simple example of using transfer learning to build upon an imagenet neural network. We will begin by training a neural network for Fisher's Iris Dataset. This network takes four measurements and classifies each observation into three iris species. However, what if later we received a data set that included the four measurements, plus a cost as the target? This dataset does not contain the species; as a result, it uses the same four inputs as the base model we just trained.\n",
    "\n",
    "We can take our previously trained iris network and transfer the weights to a new neural network that will learn to predict the cost through transfer learning. Also of note, the original neural network was a classification network, yet we now use it to build a regression neural network. Such a transformation is common for transfer learning. As a reference point, I randomly created this iris cost dataset.\n",
    "\n",
    "The first step is to train our neural network for the regular Iris Dataset. The code presented here is the same as we saw in Module 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gU7hrCGku-GU",
    "outputId": "e7510576-dbdf-41eb-9088-9b78dafc522f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "5/5 - 1s - loss: 1.7638 - 634ms/epoch - 127ms/step\n",
      "Epoch 2/100\n",
      "5/5 - 0s - loss: 1.2951 - 9ms/epoch - 2ms/step\n",
      "Epoch 3/100\n",
      "5/5 - 0s - loss: 1.0713 - 8ms/epoch - 2ms/step\n",
      "Epoch 4/100\n",
      "5/5 - 0s - loss: 1.0110 - 12ms/epoch - 2ms/step\n",
      "Epoch 5/100\n",
      "5/5 - 0s - loss: 0.9364 - 9ms/epoch - 2ms/step\n",
      "Epoch 6/100\n",
      "5/5 - 0s - loss: 0.8444 - 8ms/epoch - 2ms/step\n",
      "Epoch 7/100\n",
      "5/5 - 0s - loss: 0.7800 - 12ms/epoch - 2ms/step\n",
      "Epoch 8/100\n",
      "5/5 - 0s - loss: 0.7321 - 13ms/epoch - 3ms/step\n",
      "Epoch 9/100\n",
      "5/5 - 0s - loss: 0.6806 - 13ms/epoch - 3ms/step\n",
      "Epoch 10/100\n",
      "5/5 - 0s - loss: 0.6377 - 12ms/epoch - 2ms/step\n",
      "Epoch 11/100\n",
      "5/5 - 0s - loss: 0.6021 - 13ms/epoch - 3ms/step\n",
      "Epoch 12/100\n",
      "5/5 - 0s - loss: 0.5693 - 10ms/epoch - 2ms/step\n",
      "Epoch 13/100\n",
      "5/5 - 0s - loss: 0.5470 - 11ms/epoch - 2ms/step\n",
      "Epoch 14/100\n",
      "5/5 - 0s - loss: 0.5219 - 11ms/epoch - 2ms/step\n",
      "Epoch 15/100\n",
      "5/5 - 0s - loss: 0.4992 - 24ms/epoch - 5ms/step\n",
      "Epoch 16/100\n",
      "5/5 - 0s - loss: 0.4757 - 15ms/epoch - 3ms/step\n",
      "Epoch 17/100\n",
      "5/5 - 0s - loss: 0.4576 - 11ms/epoch - 2ms/step\n",
      "Epoch 18/100\n",
      "5/5 - 0s - loss: 0.4422 - 11ms/epoch - 2ms/step\n",
      "Epoch 19/100\n",
      "5/5 - 0s - loss: 0.4267 - 13ms/epoch - 3ms/step\n",
      "Epoch 20/100\n",
      "5/5 - 0s - loss: 0.4119 - 17ms/epoch - 3ms/step\n",
      "Epoch 21/100\n",
      "5/5 - 0s - loss: 0.3994 - 12ms/epoch - 2ms/step\n",
      "Epoch 22/100\n",
      "5/5 - 0s - loss: 0.3877 - 16ms/epoch - 3ms/step\n",
      "Epoch 23/100\n",
      "5/5 - 0s - loss: 0.3783 - 16ms/epoch - 3ms/step\n",
      "Epoch 24/100\n",
      "5/5 - 0s - loss: 0.3640 - 16ms/epoch - 3ms/step\n",
      "Epoch 25/100\n",
      "5/5 - 0s - loss: 0.3537 - 16ms/epoch - 3ms/step\n",
      "Epoch 26/100\n",
      "5/5 - 0s - loss: 0.3443 - 9ms/epoch - 2ms/step\n",
      "Epoch 27/100\n",
      "5/5 - 0s - loss: 0.3375 - 11ms/epoch - 2ms/step\n",
      "Epoch 28/100\n",
      "5/5 - 0s - loss: 0.3245 - 14ms/epoch - 3ms/step\n",
      "Epoch 29/100\n",
      "5/5 - 0s - loss: 0.3152 - 16ms/epoch - 3ms/step\n",
      "Epoch 30/100\n",
      "5/5 - 0s - loss: 0.3050 - 15ms/epoch - 3ms/step\n",
      "Epoch 31/100\n",
      "5/5 - 0s - loss: 0.2934 - 16ms/epoch - 3ms/step\n",
      "Epoch 32/100\n",
      "5/5 - 0s - loss: 0.2830 - 14ms/epoch - 3ms/step\n",
      "Epoch 33/100\n",
      "5/5 - 0s - loss: 0.2738 - 13ms/epoch - 3ms/step\n",
      "Epoch 34/100\n",
      "5/5 - 0s - loss: 0.2651 - 17ms/epoch - 3ms/step\n",
      "Epoch 35/100\n",
      "5/5 - 0s - loss: 0.2609 - 19ms/epoch - 4ms/step\n",
      "Epoch 36/100\n",
      "5/5 - 0s - loss: 0.2499 - 13ms/epoch - 3ms/step\n",
      "Epoch 37/100\n",
      "5/5 - 0s - loss: 0.2436 - 12ms/epoch - 2ms/step\n",
      "Epoch 38/100\n",
      "5/5 - 0s - loss: 0.2353 - 13ms/epoch - 3ms/step\n",
      "Epoch 39/100\n",
      "5/5 - 0s - loss: 0.2291 - 12ms/epoch - 2ms/step\n",
      "Epoch 40/100\n",
      "5/5 - 0s - loss: 0.2226 - 14ms/epoch - 3ms/step\n",
      "Epoch 41/100\n",
      "5/5 - 0s - loss: 0.2171 - 16ms/epoch - 3ms/step\n",
      "Epoch 42/100\n",
      "5/5 - 0s - loss: 0.2140 - 16ms/epoch - 3ms/step\n",
      "Epoch 43/100\n",
      "5/5 - 0s - loss: 0.2037 - 17ms/epoch - 3ms/step\n",
      "Epoch 44/100\n",
      "5/5 - 0s - loss: 0.2039 - 17ms/epoch - 3ms/step\n",
      "Epoch 45/100\n",
      "5/5 - 0s - loss: 0.1982 - 14ms/epoch - 3ms/step\n",
      "Epoch 46/100\n",
      "5/5 - 0s - loss: 0.1897 - 9ms/epoch - 2ms/step\n",
      "Epoch 47/100\n",
      "5/5 - 0s - loss: 0.1862 - 18ms/epoch - 4ms/step\n",
      "Epoch 48/100\n",
      "5/5 - 0s - loss: 0.1812 - 13ms/epoch - 3ms/step\n",
      "Epoch 49/100\n",
      "5/5 - 0s - loss: 0.1798 - 15ms/epoch - 3ms/step\n",
      "Epoch 50/100\n",
      "5/5 - 0s - loss: 0.1726 - 19ms/epoch - 4ms/step\n",
      "Epoch 51/100\n",
      "5/5 - 0s - loss: 0.1685 - 14ms/epoch - 3ms/step\n",
      "Epoch 52/100\n",
      "5/5 - 0s - loss: 0.1674 - 15ms/epoch - 3ms/step\n",
      "Epoch 53/100\n",
      "5/5 - 0s - loss: 0.1651 - 16ms/epoch - 3ms/step\n",
      "Epoch 54/100\n",
      "5/5 - 0s - loss: 0.1576 - 10ms/epoch - 2ms/step\n",
      "Epoch 55/100\n",
      "5/5 - 0s - loss: 0.1550 - 14ms/epoch - 3ms/step\n",
      "Epoch 56/100\n",
      "5/5 - 0s - loss: 0.1515 - 16ms/epoch - 3ms/step\n",
      "Epoch 57/100\n",
      "5/5 - 0s - loss: 0.1526 - 16ms/epoch - 3ms/step\n",
      "Epoch 58/100\n",
      "5/5 - 0s - loss: 0.1487 - 14ms/epoch - 3ms/step\n",
      "Epoch 59/100\n",
      "5/5 - 0s - loss: 0.1458 - 17ms/epoch - 3ms/step\n",
      "Epoch 60/100\n",
      "5/5 - 0s - loss: 0.1411 - 14ms/epoch - 3ms/step\n",
      "Epoch 61/100\n",
      "5/5 - 0s - loss: 0.1401 - 9ms/epoch - 2ms/step\n",
      "Epoch 62/100\n",
      "5/5 - 0s - loss: 0.1355 - 14ms/epoch - 3ms/step\n",
      "Epoch 63/100\n",
      "5/5 - 0s - loss: 0.1344 - 18ms/epoch - 4ms/step\n",
      "Epoch 64/100\n",
      "5/5 - 0s - loss: 0.1321 - 15ms/epoch - 3ms/step\n",
      "Epoch 65/100\n",
      "5/5 - 0s - loss: 0.1295 - 12ms/epoch - 2ms/step\n",
      "Epoch 66/100\n",
      "5/5 - 0s - loss: 0.1278 - 12ms/epoch - 2ms/step\n",
      "Epoch 67/100\n",
      "5/5 - 0s - loss: 0.1261 - 10ms/epoch - 2ms/step\n",
      "Epoch 68/100\n",
      "5/5 - 0s - loss: 0.1255 - 15ms/epoch - 3ms/step\n",
      "Epoch 69/100\n",
      "5/5 - 0s - loss: 0.1237 - 9ms/epoch - 2ms/step\n",
      "Epoch 70/100\n",
      "5/5 - 0s - loss: 0.1210 - 16ms/epoch - 3ms/step\n",
      "Epoch 71/100\n",
      "5/5 - 0s - loss: 0.1184 - 15ms/epoch - 3ms/step\n",
      "Epoch 72/100\n",
      "5/5 - 0s - loss: 0.1168 - 15ms/epoch - 3ms/step\n",
      "Epoch 73/100\n",
      "5/5 - 0s - loss: 0.1146 - 12ms/epoch - 2ms/step\n",
      "Epoch 74/100\n",
      "5/5 - 0s - loss: 0.1169 - 15ms/epoch - 3ms/step\n",
      "Epoch 75/100\n",
      "5/5 - 0s - loss: 0.1123 - 10ms/epoch - 2ms/step\n",
      "Epoch 76/100\n",
      "5/5 - 0s - loss: 0.1109 - 15ms/epoch - 3ms/step\n",
      "Epoch 77/100\n",
      "5/5 - 0s - loss: 0.1089 - 15ms/epoch - 3ms/step\n",
      "Epoch 78/100\n",
      "5/5 - 0s - loss: 0.1079 - 15ms/epoch - 3ms/step\n",
      "Epoch 79/100\n",
      "5/5 - 0s - loss: 0.1074 - 12ms/epoch - 2ms/step\n",
      "Epoch 80/100\n",
      "5/5 - 0s - loss: 0.1060 - 10ms/epoch - 2ms/step\n",
      "Epoch 81/100\n",
      "5/5 - 0s - loss: 0.1054 - 10ms/epoch - 2ms/step\n",
      "Epoch 82/100\n",
      "5/5 - 0s - loss: 0.1030 - 18ms/epoch - 4ms/step\n",
      "Epoch 83/100\n",
      "5/5 - 0s - loss: 0.1037 - 15ms/epoch - 3ms/step\n",
      "Epoch 84/100\n",
      "5/5 - 0s - loss: 0.1014 - 15ms/epoch - 3ms/step\n",
      "Epoch 85/100\n",
      "5/5 - 0s - loss: 0.0989 - 17ms/epoch - 3ms/step\n",
      "Epoch 86/100\n",
      "5/5 - 0s - loss: 0.0989 - 9ms/epoch - 2ms/step\n",
      "Epoch 87/100\n",
      "5/5 - 0s - loss: 0.1000 - 13ms/epoch - 3ms/step\n",
      "Epoch 88/100\n",
      "5/5 - 0s - loss: 0.0953 - 8ms/epoch - 2ms/step\n",
      "Epoch 89/100\n",
      "5/5 - 0s - loss: 0.0994 - 11ms/epoch - 2ms/step\n",
      "Epoch 90/100\n",
      "5/5 - 0s - loss: 0.0941 - 12ms/epoch - 2ms/step\n",
      "Epoch 91/100\n",
      "5/5 - 0s - loss: 0.0947 - 15ms/epoch - 3ms/step\n",
      "Epoch 92/100\n",
      "5/5 - 0s - loss: 0.0963 - 18ms/epoch - 4ms/step\n",
      "Epoch 93/100\n",
      "5/5 - 0s - loss: 0.0913 - 14ms/epoch - 3ms/step\n",
      "Epoch 94/100\n",
      "5/5 - 0s - loss: 0.0922 - 20ms/epoch - 4ms/step\n",
      "Epoch 95/100\n",
      "5/5 - 0s - loss: 0.0916 - 13ms/epoch - 3ms/step\n",
      "Epoch 96/100\n",
      "5/5 - 0s - loss: 0.0905 - 12ms/epoch - 2ms/step\n",
      "Epoch 97/100\n",
      "5/5 - 0s - loss: 0.0886 - 9ms/epoch - 2ms/step\n",
      "Epoch 98/100\n",
      "5/5 - 0s - loss: 0.0900 - 15ms/epoch - 3ms/step\n",
      "Epoch 99/100\n",
      "5/5 - 0s - loss: 0.0868 - 15ms/epoch - 3ms/step\n",
      "Epoch 100/100\n",
      "5/5 - 0s - loss: 0.0892 - 8ms/epoch - 2ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fea3fb1ef50>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s8TRnJOUu-GV"
   },
   "source": [
    "To keep this example simple, we are not setting aside a validation set.  The goal of this example is to show how to create a multi-layer neural network, where we transfer the weights to another network.  We begin by evaluating the accuracy of the network on the training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ssZrMUaeu-GV",
    "outputId": "11fb4ba3-548c-444b-dff3-ea4747e9485c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4oElXastu-GW"
   },
   "source": [
    "Viewing the model summary is as expected; we can see the three layers previously defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tc8-F-1lu-GW",
    "outputId": "2e073593-29d3-41df-fa89-5b292ac5d7ea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense (Dense)               (None, 50)                250       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 25)                1275      \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 3)                 78        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RXZbvEbbu-GW"
   },
   "source": [
    "## Create a New Iris Network\n",
    "\n",
    "Now that we've trained a neural network on the iris dataset, we can transfer the knowledge of this neural network to other neural networks. It is possible to create a new neural network from some or all of the layers of this neural network. We will create a new neural network that is essentially a clone of the first neural network to demonstrate the technique. We now transfer all of the layers from the original neural network into the new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "M79VumhTu-GW",
    "outputId": "a85f4f49-c15b-4595-9e93-3f078e9c8272"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense (Dense)               (None, 50)                250       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 25)                1275      \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 3)                 78        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2 = Sequential()\n",
    "for layer in model.layers:\n",
    "    model2.add(layer)\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-dQ7BeJ-u-GW"
   },
   "source": [
    "As a sanity check, we would like to calculate the accuracy of the newly created model.  The in-sample accuracy should be the same as the previous model that the new model transferred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lzkErX1Ku-GX",
    "outputId": "11bb8875-0b8b-47f7-f840-496edacc8f74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model2.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Y-0VOYvu-GX"
   },
   "source": [
    "The in-sample accuracy of the newly created neural network is the same as the first neural network. We've successfully transferred all of the layers from the original neural network.\n",
    "\n",
    "## Transfering to a Regression Network\n",
    "\n",
    "The Iris Cost Dataset has measurements for samples of these flowers that conform to the predictors contained in the original iris dataset: sepal width, sepal length, petal width, and petal length. We present the cost dataset here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "6ZohkVGx5ukW",
    "outputId": "e4e92550-2ebb-4426-9c82-26cf27b10a51"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-6039cb75-c735-47f7-b76b-604ad8bb228a\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_l</th>\n",
       "      <th>sepal_w</th>\n",
       "      <th>petal_l</th>\n",
       "      <th>petal_w</th>\n",
       "      <th>cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.8</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.2</td>\n",
       "      <td>1.7</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6.9</td>\n",
       "      <td>2.6</td>\n",
       "      <td>3.7</td>\n",
       "      <td>1.4</td>\n",
       "      <td>4.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.9</td>\n",
       "      <td>2.2</td>\n",
       "      <td>3.7</td>\n",
       "      <td>2.4</td>\n",
       "      <td>6.558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.9</td>\n",
       "      <td>6.8</td>\n",
       "      <td>0.7</td>\n",
       "      <td>7.395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245</th>\n",
       "      <td>4.7</td>\n",
       "      <td>2.1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>5.721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.3</td>\n",
       "      <td>1.1</td>\n",
       "      <td>5.266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>6.6</td>\n",
       "      <td>3.4</td>\n",
       "      <td>4.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>5.776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>5.7</td>\n",
       "      <td>3.7</td>\n",
       "      <td>3.1</td>\n",
       "      <td>0.4</td>\n",
       "      <td>2.233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>7.6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.4</td>\n",
       "      <td>7.508</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>250 rows × 5 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6039cb75-c735-47f7-b76b-604ad8bb228a')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-6039cb75-c735-47f7-b76b-604ad8bb228a button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-6039cb75-c735-47f7-b76b-604ad8bb228a');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "     sepal_l  sepal_w  petal_l  petal_w    cost\n",
       "0        7.8      3.0      6.2      2.0  10.740\n",
       "1        5.0      2.2      1.7      1.5   2.710\n",
       "2        6.9      2.6      3.7      1.4   4.624\n",
       "3        5.9      2.2      3.7      2.4   6.558\n",
       "4        5.1      3.9      6.8      0.7   7.395\n",
       "..       ...      ...      ...      ...     ...\n",
       "245      4.7      2.1      4.0      2.3   5.721\n",
       "246      7.2      3.0      4.3      1.1   5.266\n",
       "247      6.6      3.4      4.6      1.4   5.776\n",
       "248      5.7      3.7      3.1      0.4   2.233\n",
       "249      7.6      4.0      5.1      1.4   7.508\n",
       "\n",
       "[250 rows x 5 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_cost = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris_cost.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "df_cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "31Zz7cyU6DGT"
   },
   "source": [
    "For transfer learning to be effective, the input for the newly trained neural network most closely conforms to the first neural network we transfer.\n",
    "\n",
    "We will strip away the last output layer that contains the softmax activation function that performs this final classification. We will create a new output layer that will output the cost prediction. We will only train the weights in this new layer. We will mark the first two layers as non-trainable. The hope is that the first few layers have learned to abstract the raw input data in a way that is also helpful to the new neural network.\n",
    "This process is accomplished by looping over the first few layers and copying them to the new neural network. We output a summary of the new neural network to verify that Keras stripped the previous output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EE2mBp98u-GX",
    "outputId": "611f86a6-a890-4cfc-f6f2-c8a19597d9ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense (Dense)               (None, 50)                250       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 25)                1275      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,525\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3 = Sequential()\n",
    "for i in range(2):\n",
    "    layer = model.layers[i]\n",
    "    layer.trainable = False\n",
    "    model3.add(layer)\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "38iQTGH0u-GX"
   },
   "source": [
    "We add a final regression output layer to complete the new neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pcWB19aPu-GX",
    "outputId": "ff8b7124-eef1-4221-b23c-a871ccb8d33c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense (Dense)               (None, 50)                250       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 25)                1275      \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 1)                 26        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,551\n",
      "Trainable params: 26\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3.add(Dense(1)) # Output\n",
    "\n",
    "model3.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0by_NO-9u-GX"
   },
   "source": [
    "Now we train just the output layer to predict the cost. The cost in the made-up dataset is dependent on the species, so the previous learning should be helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sVV5IXYxu-GY",
    "outputId": "93cc7fdb-a574-48e9-e70e-dc1fc0af93d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "8/8 - 0s - loss: 14.0400 - 379ms/epoch - 47ms/step\n",
      "Epoch 2/100\n",
      "8/8 - 0s - loss: 12.6133 - 10ms/epoch - 1ms/step\n",
      "Epoch 3/100\n",
      "8/8 - 0s - loss: 11.3224 - 12ms/epoch - 1ms/step\n",
      "Epoch 4/100\n",
      "8/8 - 0s - loss: 10.1006 - 19ms/epoch - 2ms/step\n",
      "Epoch 5/100\n",
      "8/8 - 0s - loss: 9.0898 - 19ms/epoch - 2ms/step\n",
      "Epoch 6/100\n",
      "8/8 - 0s - loss: 8.1514 - 13ms/epoch - 2ms/step\n",
      "Epoch 7/100\n",
      "8/8 - 0s - loss: 7.3497 - 11ms/epoch - 1ms/step\n",
      "Epoch 8/100\n",
      "8/8 - 0s - loss: 6.6789 - 14ms/epoch - 2ms/step\n",
      "Epoch 9/100\n",
      "8/8 - 0s - loss: 6.0785 - 11ms/epoch - 1ms/step\n",
      "Epoch 10/100\n",
      "8/8 - 0s - loss: 5.5620 - 11ms/epoch - 1ms/step\n",
      "Epoch 11/100\n",
      "8/8 - 0s - loss: 5.1035 - 11ms/epoch - 1ms/step\n",
      "Epoch 12/100\n",
      "8/8 - 0s - loss: 4.7415 - 12ms/epoch - 2ms/step\n",
      "Epoch 13/100\n",
      "8/8 - 0s - loss: 4.4169 - 13ms/epoch - 2ms/step\n",
      "Epoch 14/100\n",
      "8/8 - 0s - loss: 4.1181 - 19ms/epoch - 2ms/step\n",
      "Epoch 15/100\n",
      "8/8 - 0s - loss: 3.8847 - 20ms/epoch - 3ms/step\n",
      "Epoch 16/100\n",
      "8/8 - 0s - loss: 3.6586 - 13ms/epoch - 2ms/step\n",
      "Epoch 17/100\n",
      "8/8 - 0s - loss: 3.4690 - 17ms/epoch - 2ms/step\n",
      "Epoch 18/100\n",
      "8/8 - 0s - loss: 3.3085 - 16ms/epoch - 2ms/step\n",
      "Epoch 19/100\n",
      "8/8 - 0s - loss: 3.1461 - 17ms/epoch - 2ms/step\n",
      "Epoch 20/100\n",
      "8/8 - 0s - loss: 3.0206 - 20ms/epoch - 2ms/step\n",
      "Epoch 21/100\n",
      "8/8 - 0s - loss: 2.8936 - 14ms/epoch - 2ms/step\n",
      "Epoch 22/100\n",
      "8/8 - 0s - loss: 2.7855 - 11ms/epoch - 1ms/step\n",
      "Epoch 23/100\n",
      "8/8 - 0s - loss: 2.6949 - 10ms/epoch - 1ms/step\n",
      "Epoch 24/100\n",
      "8/8 - 0s - loss: 2.6056 - 13ms/epoch - 2ms/step\n",
      "Epoch 25/100\n",
      "8/8 - 0s - loss: 2.5355 - 24ms/epoch - 3ms/step\n",
      "Epoch 26/100\n",
      "8/8 - 0s - loss: 2.4600 - 10ms/epoch - 1ms/step\n",
      "Epoch 27/100\n",
      "8/8 - 0s - loss: 2.4041 - 13ms/epoch - 2ms/step\n",
      "Epoch 28/100\n",
      "8/8 - 0s - loss: 2.3449 - 14ms/epoch - 2ms/step\n",
      "Epoch 29/100\n",
      "8/8 - 0s - loss: 2.2991 - 17ms/epoch - 2ms/step\n",
      "Epoch 30/100\n",
      "8/8 - 0s - loss: 2.2528 - 15ms/epoch - 2ms/step\n",
      "Epoch 31/100\n",
      "8/8 - 0s - loss: 2.2143 - 15ms/epoch - 2ms/step\n",
      "Epoch 32/100\n",
      "8/8 - 0s - loss: 2.1818 - 17ms/epoch - 2ms/step\n",
      "Epoch 33/100\n",
      "8/8 - 0s - loss: 2.1527 - 15ms/epoch - 2ms/step\n",
      "Epoch 34/100\n",
      "8/8 - 0s - loss: 2.1262 - 17ms/epoch - 2ms/step\n",
      "Epoch 35/100\n",
      "8/8 - 0s - loss: 2.1046 - 13ms/epoch - 2ms/step\n",
      "Epoch 36/100\n",
      "8/8 - 0s - loss: 2.0811 - 13ms/epoch - 2ms/step\n",
      "Epoch 37/100\n",
      "8/8 - 0s - loss: 2.0657 - 10ms/epoch - 1ms/step\n",
      "Epoch 38/100\n",
      "8/8 - 0s - loss: 2.0487 - 15ms/epoch - 2ms/step\n",
      "Epoch 39/100\n",
      "8/8 - 0s - loss: 2.0368 - 17ms/epoch - 2ms/step\n",
      "Epoch 40/100\n",
      "8/8 - 0s - loss: 2.0257 - 21ms/epoch - 3ms/step\n",
      "Epoch 41/100\n",
      "8/8 - 0s - loss: 2.0131 - 15ms/epoch - 2ms/step\n",
      "Epoch 42/100\n",
      "8/8 - 0s - loss: 2.0053 - 24ms/epoch - 3ms/step\n",
      "Epoch 43/100\n",
      "8/8 - 0s - loss: 1.9972 - 22ms/epoch - 3ms/step\n",
      "Epoch 44/100\n",
      "8/8 - 0s - loss: 1.9898 - 17ms/epoch - 2ms/step\n",
      "Epoch 45/100\n",
      "8/8 - 0s - loss: 1.9838 - 14ms/epoch - 2ms/step\n",
      "Epoch 46/100\n",
      "8/8 - 0s - loss: 1.9788 - 12ms/epoch - 1ms/step\n",
      "Epoch 47/100\n",
      "8/8 - 0s - loss: 1.9743 - 14ms/epoch - 2ms/step\n",
      "Epoch 48/100\n",
      "8/8 - 0s - loss: 1.9702 - 19ms/epoch - 2ms/step\n",
      "Epoch 49/100\n",
      "8/8 - 0s - loss: 1.9662 - 15ms/epoch - 2ms/step\n",
      "Epoch 50/100\n",
      "8/8 - 0s - loss: 1.9646 - 14ms/epoch - 2ms/step\n",
      "Epoch 51/100\n",
      "8/8 - 0s - loss: 1.9604 - 15ms/epoch - 2ms/step\n",
      "Epoch 52/100\n",
      "8/8 - 0s - loss: 1.9578 - 16ms/epoch - 2ms/step\n",
      "Epoch 53/100\n",
      "8/8 - 0s - loss: 1.9552 - 14ms/epoch - 2ms/step\n",
      "Epoch 54/100\n",
      "8/8 - 0s - loss: 1.9528 - 11ms/epoch - 1ms/step\n",
      "Epoch 55/100\n",
      "8/8 - 0s - loss: 1.9511 - 19ms/epoch - 2ms/step\n",
      "Epoch 56/100\n",
      "8/8 - 0s - loss: 1.9489 - 18ms/epoch - 2ms/step\n",
      "Epoch 57/100\n",
      "8/8 - 0s - loss: 1.9468 - 17ms/epoch - 2ms/step\n",
      "Epoch 58/100\n",
      "8/8 - 0s - loss: 1.9462 - 19ms/epoch - 2ms/step\n",
      "Epoch 59/100\n",
      "8/8 - 0s - loss: 1.9435 - 14ms/epoch - 2ms/step\n",
      "Epoch 60/100\n",
      "8/8 - 0s - loss: 1.9421 - 11ms/epoch - 1ms/step\n",
      "Epoch 61/100\n",
      "8/8 - 0s - loss: 1.9406 - 17ms/epoch - 2ms/step\n",
      "Epoch 62/100\n",
      "8/8 - 0s - loss: 1.9389 - 14ms/epoch - 2ms/step\n",
      "Epoch 63/100\n",
      "8/8 - 0s - loss: 1.9387 - 15ms/epoch - 2ms/step\n",
      "Epoch 64/100\n",
      "8/8 - 0s - loss: 1.9358 - 19ms/epoch - 2ms/step\n",
      "Epoch 65/100\n",
      "8/8 - 0s - loss: 1.9347 - 17ms/epoch - 2ms/step\n",
      "Epoch 66/100\n",
      "8/8 - 0s - loss: 1.9332 - 20ms/epoch - 3ms/step\n",
      "Epoch 67/100\n",
      "8/8 - 0s - loss: 1.9329 - 16ms/epoch - 2ms/step\n",
      "Epoch 68/100\n",
      "8/8 - 0s - loss: 1.9300 - 17ms/epoch - 2ms/step\n",
      "Epoch 69/100\n",
      "8/8 - 0s - loss: 1.9289 - 22ms/epoch - 3ms/step\n",
      "Epoch 70/100\n",
      "8/8 - 0s - loss: 1.9286 - 17ms/epoch - 2ms/step\n",
      "Epoch 71/100\n",
      "8/8 - 0s - loss: 1.9259 - 13ms/epoch - 2ms/step\n",
      "Epoch 72/100\n",
      "8/8 - 0s - loss: 1.9250 - 16ms/epoch - 2ms/step\n",
      "Epoch 73/100\n",
      "8/8 - 0s - loss: 1.9231 - 13ms/epoch - 2ms/step\n",
      "Epoch 74/100\n",
      "8/8 - 0s - loss: 1.9217 - 14ms/epoch - 2ms/step\n",
      "Epoch 75/100\n",
      "8/8 - 0s - loss: 1.9199 - 17ms/epoch - 2ms/step\n",
      "Epoch 76/100\n",
      "8/8 - 0s - loss: 1.9189 - 21ms/epoch - 3ms/step\n",
      "Epoch 77/100\n",
      "8/8 - 0s - loss: 1.9176 - 15ms/epoch - 2ms/step\n",
      "Epoch 78/100\n",
      "8/8 - 0s - loss: 1.9163 - 16ms/epoch - 2ms/step\n",
      "Epoch 79/100\n",
      "8/8 - 0s - loss: 1.9146 - 14ms/epoch - 2ms/step\n",
      "Epoch 80/100\n",
      "8/8 - 0s - loss: 1.9132 - 15ms/epoch - 2ms/step\n",
      "Epoch 81/100\n",
      "8/8 - 0s - loss: 1.9118 - 14ms/epoch - 2ms/step\n",
      "Epoch 82/100\n",
      "8/8 - 0s - loss: 1.9100 - 15ms/epoch - 2ms/step\n",
      "Epoch 83/100\n",
      "8/8 - 0s - loss: 1.9083 - 15ms/epoch - 2ms/step\n",
      "Epoch 84/100\n",
      "8/8 - 0s - loss: 1.9070 - 25ms/epoch - 3ms/step\n",
      "Epoch 85/100\n",
      "8/8 - 0s - loss: 1.9074 - 12ms/epoch - 2ms/step\n",
      "Epoch 86/100\n",
      "8/8 - 0s - loss: 1.9039 - 18ms/epoch - 2ms/step\n",
      "Epoch 87/100\n",
      "8/8 - 0s - loss: 1.9027 - 13ms/epoch - 2ms/step\n",
      "Epoch 88/100\n",
      "8/8 - 0s - loss: 1.9010 - 15ms/epoch - 2ms/step\n",
      "Epoch 89/100\n",
      "8/8 - 0s - loss: 1.8994 - 16ms/epoch - 2ms/step\n",
      "Epoch 90/100\n",
      "8/8 - 0s - loss: 1.8978 - 15ms/epoch - 2ms/step\n",
      "Epoch 91/100\n",
      "8/8 - 0s - loss: 1.8968 - 14ms/epoch - 2ms/step\n",
      "Epoch 92/100\n",
      "8/8 - 0s - loss: 1.8952 - 15ms/epoch - 2ms/step\n",
      "Epoch 93/100\n",
      "8/8 - 0s - loss: 1.8956 - 20ms/epoch - 2ms/step\n",
      "Epoch 94/100\n",
      "8/8 - 0s - loss: 1.8925 - 16ms/epoch - 2ms/step\n",
      "Epoch 95/100\n",
      "8/8 - 0s - loss: 1.8906 - 16ms/epoch - 2ms/step\n",
      "Epoch 96/100\n",
      "8/8 - 0s - loss: 1.8891 - 13ms/epoch - 2ms/step\n",
      "Epoch 97/100\n",
      "8/8 - 0s - loss: 1.8877 - 11ms/epoch - 1ms/step\n",
      "Epoch 98/100\n",
      "8/8 - 0s - loss: 1.8860 - 14ms/epoch - 2ms/step\n",
      "Epoch 99/100\n",
      "8/8 - 0s - loss: 1.8851 - 17ms/epoch - 2ms/step\n",
      "Epoch 100/100\n",
      "8/8 - 0s - loss: 1.8838 - 9ms/epoch - 1ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fea3f9bc890>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert to numpy - Classification\n",
    "x = df_cost[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "y = df_cost.cost.values\n",
    "\n",
    "# Train the last layer of the network\n",
    "model3.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DAXFzJIQu-GY"
   },
   "source": [
    "We can evaluate the in-sample RMSE for the new model containing transferred layers from the previous model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4XU98v2Du-GY",
    "outputId": "bee6db78-5e5e-4d2c-be58-f0cf6649a05b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 1.3716589625823072\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model3.predict(x)\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hOvFqmoou-GY"
   },
   "source": [
    "# Module 9 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 9](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class9.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "z5u3j5G4u-GY"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "new_t81_558_class_09_1_keras_transfer.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}