t81_558_class_08_2_keras_ensembles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XjaM_R6LeV5S"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_2_keras_ensembles.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fnpz-3gAeV5T"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 8: Kaggle Data Sets**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DpUhdx0FeV5T"
   },
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=v4lJBhdCuCU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_1_kaggle_intro.ipynb)\n",
    "* **Part 8.2: Building Ensembles with Scikit-Learn and Keras** [[Video]](https://www.youtube.com/watch?v=LQ-9ZRBLasw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_2_keras_ensembles.ipynb)\n",
    "* Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters [[Video]](https://www.youtube.com/watch?v=1q9klwSoUQw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_3_keras_hyperparameters.ipynb)\n",
    "* Part 8.4: Bayesian Hyperparameter Optimization for Keras [[Video]](https://www.youtube.com/watch?v=sXdxyUCCm8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]](https://www.youtube.com/watch?v=PHQt0aUasRg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_5_kaggle_project.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DaG7LEHQeV5U"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n",
    "  Running the following code will map your GDrive to ```/content/drive```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wmJ4sdveeV5U",
    "outputId": "ccade7c3-5c27-46e5-ea44-cb4ca00934ac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n",
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FHx0cbrxeV5U"
   },
   "source": [
    "# Part 8.2: Building Ensembles with Scikit-Learn and Keras\n",
    "\n",
    "### Evaluating Feature Importance\n",
    "\n",
    "Feature importance tells us how important each feature (from the feature/import vector) is to predicting a neural network or another model. There are many different ways to evaluate the feature importance of neural networks. The following paper presents an excellent (and readable) overview of the various means of assessing the significance of neural network inputs/features.\n",
    "\n",
    "* An accurate comparison of methods for quantifying variable importance in artificial neural networks using simulated data [[Cite:olden2004accurate]](http://depts.washington.edu/oldenlab/wordpress/wp-content/uploads/2013/03/EcologicalModelling_2004.pdf). *Ecological Modelling*, 178(3), 389-397.\n",
    "\n",
    "In summary, the following methods are available to neural networks:\n",
    "\n",
    "* Connection Weights Algorithm\n",
    "* Partial Derivatives\n",
    "* Input Perturbation\n",
    "* Sensitivity Analysis\n",
    "* Forward Stepwise Addition \n",
    "* Improved Stepwise Selection 1\n",
    "* Backward Stepwise Elimination\n",
    "* Improved Stepwise Selection\n",
    "\n",
    "For this chapter, we will use the input Perturbation feature ranking algorithm. This algorithm will work with any regression or classification network. In the next section, I provide an implementation of the input perturbation algorithm for scikit-learn. This code implements a function below that will work with any scikit-learn model.\n",
    "\n",
    "[Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman) provided this algorithm in his seminal paper on random forests. [[Citebreiman2001random:]](https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf)  Although he presented this algorithm in conjunction with random forests, it is model-independent and appropriate for any supervised learning model.  This algorithm, known as the input perturbation algorithm, works by evaluating a trained model’s accuracy with each input individually shuffled from a data set. Shuffling an input causes it to become useless—effectively removing it from the model. More important inputs will produce a less accurate score when they are removed by shuffling them. This process makes sense because important features will contribute to the model's accuracy. I first presented the TensorFlow implementation of this algorithm in the following paper.\n",
    "\n",
    "* Early stabilizing feature importance for TensorFlow deep neural networks[[Cite:heaton2017early]](https://www.heatonresearch.com/dload/phd/IJCNN%202017-v2-final.pdf)\n",
    "\n",
    "This algorithm will use log loss to evaluate a classification problem and RMSE for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EjMqkygLeV5a"
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "import scipy as sp\n",
    "import numpy as np\n",
    "import math\n",
    "from sklearn import metrics\n",
    "\n",
    "def perturbation_rank(model, x, y, names, regression):\n",
    "    errors = []\n",
    "\n",
    "    for i in range(x.shape[1]):\n",
    "        hold = np.array(x[:, i])\n",
    "        np.random.shuffle(x[:, i])\n",
    "        \n",
    "        if regression:\n",
    "            pred = model.predict(x)\n",
    "            error = metrics.mean_squared_error(y, pred)\n",
    "        else:\n",
    "            pred = model.predict(x)\n",
    "            error = metrics.log_loss(y, pred)\n",
    "            \n",
    "        errors.append(error)\n",
    "        x[:, i] = hold\n",
    "        \n",
    "    max_error = np.max(errors)\n",
    "    importance = [e/max_error for e in errors]\n",
    "\n",
    "    data = {'name':names,'error':errors,'importance':importance}\n",
    "    result = pd.DataFrame(data, columns = ['name','error','importance'])\n",
    "    result.sort_values(by=['importance'], ascending=[0], inplace=True)\n",
    "    result.reset_index(inplace=True, drop=True)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0dnNa09QeV5a"
   },
   "source": [
    "## Classification and Input Perturbation Ranking\n",
    "\n",
    "We now look at the code to perform perturbation ranking for a classification neural network.  The implementation technique is slightly different for classification vs. regression, so I must provide two different implementations.  The primary difference between classification and regression is how we evaluate the accuracy of the neural network in each of these two network types.  We will use the Root Mean Square (RMSE) error calculation, whereas we will use log loss for classification.\n",
    "\n",
    "The code presented below creates a classification neural network that will predict the classic iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lEY1hZigeV5a",
    "outputId": "80673c5c-3264-4d54-e19b-02742b563e12"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "4/4 - 1s - loss: 2.0814 - 1s/epoch - 292ms/step\n",
      "Epoch 2/100\n",
      "4/4 - 0s - loss: 1.6125 - 14ms/epoch - 4ms/step\n",
      "Epoch 3/100\n",
      "4/4 - 0s - loss: 1.3316 - 26ms/epoch - 7ms/step\n",
      "Epoch 4/100\n",
      "4/4 - 0s - loss: 1.2246 - 13ms/epoch - 3ms/step\n",
      "Epoch 5/100\n",
      "4/4 - 0s - loss: 1.1989 - 13ms/epoch - 3ms/step\n",
      "Epoch 6/100\n",
      "4/4 - 0s - loss: 1.1349 - 14ms/epoch - 4ms/step\n",
      "Epoch 7/100\n",
      "4/4 - 0s - loss: 1.0543 - 21ms/epoch - 5ms/step\n",
      "Epoch 8/100\n",
      "4/4 - 0s - loss: 0.9987 - 25ms/epoch - 6ms/step\n",
      "Epoch 9/100\n",
      "4/4 - 0s - loss: 0.9449 - 20ms/epoch - 5ms/step\n",
      "Epoch 10/100\n",
      "4/4 - 0s - loss: 0.9032 - 16ms/epoch - 4ms/step\n",
      "Epoch 11/100\n",
      "4/4 - 0s - loss: 0.8623 - 20ms/epoch - 5ms/step\n",
      "Epoch 12/100\n",
      "4/4 - 0s - loss: 0.8274 - 12ms/epoch - 3ms/step\n",
      "Epoch 13/100\n",
      "4/4 - 0s - loss: 0.8013 - 18ms/epoch - 4ms/step\n",
      "Epoch 14/100\n",
      "4/4 - 0s - loss: 0.7718 - 18ms/epoch - 5ms/step\n",
      "Epoch 15/100\n",
      "4/4 - 0s - loss: 0.7426 - 19ms/epoch - 5ms/step\n",
      "Epoch 16/100\n",
      "4/4 - 0s - loss: 0.7163 - 13ms/epoch - 3ms/step\n",
      "Epoch 17/100\n",
      "4/4 - 0s - loss: 0.6933 - 13ms/epoch - 3ms/step\n",
      "Epoch 18/100\n",
      "4/4 - 0s - loss: 0.6689 - 14ms/epoch - 3ms/step\n",
      "Epoch 19/100\n",
      "4/4 - 0s - loss: 0.6488 - 11ms/epoch - 3ms/step\n",
      "Epoch 20/100\n",
      "4/4 - 0s - loss: 0.6294 - 11ms/epoch - 3ms/step\n",
      "Epoch 21/100\n",
      "4/4 - 0s - loss: 0.6094 - 20ms/epoch - 5ms/step\n",
      "Epoch 22/100\n",
      "4/4 - 0s - loss: 0.5911 - 18ms/epoch - 4ms/step\n",
      "Epoch 23/100\n",
      "4/4 - 0s - loss: 0.5725 - 16ms/epoch - 4ms/step\n",
      "Epoch 24/100\n",
      "4/4 - 0s - loss: 0.5550 - 13ms/epoch - 3ms/step\n",
      "Epoch 25/100\n",
      "4/4 - 0s - loss: 0.5389 - 14ms/epoch - 3ms/step\n",
      "Epoch 26/100\n",
      "4/4 - 0s - loss: 0.5207 - 15ms/epoch - 4ms/step\n",
      "Epoch 27/100\n",
      "4/4 - 0s - loss: 0.5041 - 14ms/epoch - 4ms/step\n",
      "Epoch 28/100\n",
      "4/4 - 0s - loss: 0.4901 - 14ms/epoch - 3ms/step\n",
      "Epoch 29/100\n",
      "4/4 - 0s - loss: 0.4765 - 14ms/epoch - 4ms/step\n",
      "Epoch 30/100\n",
      "4/4 - 0s - loss: 0.4619 - 16ms/epoch - 4ms/step\n",
      "Epoch 31/100\n",
      "4/4 - 0s - loss: 0.4489 - 16ms/epoch - 4ms/step\n",
      "Epoch 32/100\n",
      "4/4 - 0s - loss: 0.4366 - 13ms/epoch - 3ms/step\n",
      "Epoch 33/100\n",
      "4/4 - 0s - loss: 0.4243 - 13ms/epoch - 3ms/step\n",
      "Epoch 34/100\n",
      "4/4 - 0s - loss: 0.4124 - 14ms/epoch - 3ms/step\n",
      "Epoch 35/100\n",
      "4/4 - 0s - loss: 0.4015 - 14ms/epoch - 3ms/step\n",
      "Epoch 36/100\n",
      "4/4 - 0s - loss: 0.3917 - 21ms/epoch - 5ms/step\n",
      "Epoch 37/100\n",
      "4/4 - 0s - loss: 0.3826 - 30ms/epoch - 7ms/step\n",
      "Epoch 38/100\n",
      "4/4 - 0s - loss: 0.3713 - 18ms/epoch - 4ms/step\n",
      "Epoch 39/100\n",
      "4/4 - 0s - loss: 0.3621 - 16ms/epoch - 4ms/step\n",
      "Epoch 40/100\n",
      "4/4 - 0s - loss: 0.3543 - 14ms/epoch - 4ms/step\n",
      "Epoch 41/100\n",
      "4/4 - 0s - loss: 0.3460 - 14ms/epoch - 4ms/step\n",
      "Epoch 42/100\n",
      "4/4 - 0s - loss: 0.3385 - 28ms/epoch - 7ms/step\n",
      "Epoch 43/100\n",
      "4/4 - 0s - loss: 0.3280 - 31ms/epoch - 8ms/step\n",
      "Epoch 44/100\n",
      "4/4 - 0s - loss: 0.3211 - 15ms/epoch - 4ms/step\n",
      "Epoch 45/100\n",
      "4/4 - 0s - loss: 0.3144 - 14ms/epoch - 3ms/step\n",
      "Epoch 46/100\n",
      "4/4 - 0s - loss: 0.3068 - 15ms/epoch - 4ms/step\n",
      "Epoch 47/100\n",
      "4/4 - 0s - loss: 0.2992 - 19ms/epoch - 5ms/step\n",
      "Epoch 48/100\n",
      "4/4 - 0s - loss: 0.2922 - 18ms/epoch - 4ms/step\n",
      "Epoch 49/100\n",
      "4/4 - 0s - loss: 0.2847 - 37ms/epoch - 9ms/step\n",
      "Epoch 50/100\n",
      "4/4 - 0s - loss: 0.2803 - 13ms/epoch - 3ms/step\n",
      "Epoch 51/100\n",
      "4/4 - 0s - loss: 0.2756 - 13ms/epoch - 3ms/step\n",
      "Epoch 52/100\n",
      "4/4 - 0s - loss: 0.2665 - 18ms/epoch - 4ms/step\n",
      "Epoch 53/100\n",
      "4/4 - 0s - loss: 0.2632 - 16ms/epoch - 4ms/step\n",
      "Epoch 54/100\n",
      "4/4 - 0s - loss: 0.2571 - 20ms/epoch - 5ms/step\n",
      "Epoch 55/100\n",
      "4/4 - 0s - loss: 0.2499 - 17ms/epoch - 4ms/step\n",
      "Epoch 56/100\n",
      "4/4 - 0s - loss: 0.2458 - 17ms/epoch - 4ms/step\n",
      "Epoch 57/100\n",
      "4/4 - 0s - loss: 0.2399 - 23ms/epoch - 6ms/step\n",
      "Epoch 58/100\n",
      "4/4 - 0s - loss: 0.2340 - 16ms/epoch - 4ms/step\n",
      "Epoch 59/100\n",
      "4/4 - 0s - loss: 0.2318 - 16ms/epoch - 4ms/step\n",
      "Epoch 60/100\n",
      "4/4 - 0s - loss: 0.2225 - 12ms/epoch - 3ms/step\n",
      "Epoch 61/100\n",
      "4/4 - 0s - loss: 0.2266 - 15ms/epoch - 4ms/step\n",
      "Epoch 62/100\n",
      "4/4 - 0s - loss: 0.2178 - 12ms/epoch - 3ms/step\n",
      "Epoch 63/100\n",
      "4/4 - 0s - loss: 0.2116 - 15ms/epoch - 4ms/step\n",
      "Epoch 64/100\n",
      "4/4 - 0s - loss: 0.2137 - 21ms/epoch - 5ms/step\n",
      "Epoch 65/100\n",
      "4/4 - 0s - loss: 0.2030 - 17ms/epoch - 4ms/step\n",
      "Epoch 66/100\n",
      "4/4 - 0s - loss: 0.2041 - 14ms/epoch - 3ms/step\n",
      "Epoch 67/100\n",
      "4/4 - 0s - loss: 0.2001 - 15ms/epoch - 4ms/step\n",
      "Epoch 68/100\n",
      "4/4 - 0s - loss: 0.1919 - 25ms/epoch - 6ms/step\n",
      "Epoch 69/100\n",
      "4/4 - 0s - loss: 0.1894 - 23ms/epoch - 6ms/step\n",
      "Epoch 70/100\n",
      "4/4 - 0s - loss: 0.1863 - 17ms/epoch - 4ms/step\n",
      "Epoch 71/100\n",
      "4/4 - 0s - loss: 0.1823 - 16ms/epoch - 4ms/step\n",
      "Epoch 72/100\n",
      "4/4 - 0s - loss: 0.1790 - 24ms/epoch - 6ms/step\n",
      "Epoch 73/100\n",
      "4/4 - 0s - loss: 0.1780 - 16ms/epoch - 4ms/step\n",
      "Epoch 74/100\n",
      "4/4 - 0s - loss: 0.1755 - 15ms/epoch - 4ms/step\n",
      "Epoch 75/100\n",
      "4/4 - 0s - loss: 0.1719 - 31ms/epoch - 8ms/step\n",
      "Epoch 76/100\n",
      "4/4 - 0s - loss: 0.1767 - 21ms/epoch - 5ms/step\n",
      "Epoch 77/100\n",
      "4/4 - 0s - loss: 0.1694 - 17ms/epoch - 4ms/step\n",
      "Epoch 78/100\n",
      "4/4 - 0s - loss: 0.1655 - 27ms/epoch - 7ms/step\n",
      "Epoch 79/100\n",
      "4/4 - 0s - loss: 0.1634 - 23ms/epoch - 6ms/step\n",
      "Epoch 80/100\n",
      "4/4 - 0s - loss: 0.1566 - 17ms/epoch - 4ms/step\n",
      "Epoch 81/100\n",
      "4/4 - 0s - loss: 0.1563 - 17ms/epoch - 4ms/step\n",
      "Epoch 82/100\n",
      "4/4 - 0s - loss: 0.1536 - 15ms/epoch - 4ms/step\n",
      "Epoch 83/100\n",
      "4/4 - 0s - loss: 0.1504 - 18ms/epoch - 4ms/step\n",
      "Epoch 84/100\n",
      "4/4 - 0s - loss: 0.1502 - 16ms/epoch - 4ms/step\n",
      "Epoch 85/100\n",
      "4/4 - 0s - loss: 0.1469 - 17ms/epoch - 4ms/step\n",
      "Epoch 86/100\n",
      "4/4 - 0s - loss: 0.1448 - 28ms/epoch - 7ms/step\n",
      "Epoch 87/100\n",
      "4/4 - 0s - loss: 0.1424 - 23ms/epoch - 6ms/step\n",
      "Epoch 88/100\n",
      "4/4 - 0s - loss: 0.1401 - 25ms/epoch - 6ms/step\n",
      "Epoch 89/100\n",
      "4/4 - 0s - loss: 0.1386 - 47ms/epoch - 12ms/step\n",
      "Epoch 90/100\n",
      "4/4 - 0s - loss: 0.1365 - 30ms/epoch - 7ms/step\n",
      "Epoch 91/100\n",
      "4/4 - 0s - loss: 0.1383 - 41ms/epoch - 10ms/step\n",
      "Epoch 92/100\n",
      "4/4 - 0s - loss: 0.1332 - 12ms/epoch - 3ms/step\n",
      "Epoch 93/100\n",
      "4/4 - 0s - loss: 0.1311 - 20ms/epoch - 5ms/step\n",
      "Epoch 94/100\n",
      "4/4 - 0s - loss: 0.1320 - 20ms/epoch - 5ms/step\n",
      "Epoch 95/100\n",
      "4/4 - 0s - loss: 0.1302 - 17ms/epoch - 4ms/step\n",
      "Epoch 96/100\n",
      "4/4 - 0s - loss: 0.1311 - 19ms/epoch - 5ms/step\n",
      "Epoch 97/100\n",
      "4/4 - 0s - loss: 0.1248 - 14ms/epoch - 3ms/step\n",
      "Epoch 98/100\n",
      "4/4 - 0s - loss: 0.1254 - 12ms/epoch - 3ms/step\n",
      "Epoch 99/100\n",
      "4/4 - 0s - loss: 0.1275 - 13ms/epoch - 3ms/step\n",
      "Epoch 100/100\n",
      "4/4 - 0s - loss: 0.1225 - 41ms/epoch - 10ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fc869fd2950>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x_train,y_train,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vd-Y5j9MeV5b"
   },
   "source": [
    "Next, we evaluate the accuracy of the trained model.  Here we see that the neural network performs great, with an accuracy of 1.0.  We might fear overfitting with such high accuracy for a more complex dataset.  However, for this example, we are more interested in determining the importance of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2-FOcsWieV5b",
    "outputId": "b5380db0-cf09-4973-c0e5-91a2041892d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "pred = model.predict(x_test)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y_test,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IhMMwAhzeV5b"
   },
   "source": [
    "We are now ready to call the input perturbation algorithm.  First, we extract the column names and remove the target column.  The target column is not important, as it is the objective, not one of the inputs.  In supervised learning, the target is of the utmost importance.\n",
    "\n",
    "We can see the importance displayed in the following table.  The most important column is always 1.0, and lessor columns will continue in a downward trend.  The least important column will have the lowest rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "OTUe2xOZeV5b",
    "outputId": "0d9610d9-1fa2-4438-ed3b-77e44029aa84"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-63e004b9-1db0-4b5a-b6b8-2e067ed32965\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>petal_l</td>\n",
       "      <td>2.609378</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>petal_w</td>\n",
       "      <td>0.480387</td>\n",
       "      <td>0.184100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sepal_l</td>\n",
       "      <td>0.223239</td>\n",
       "      <td>0.085553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sepal_w</td>\n",
       "      <td>0.128518</td>\n",
       "      <td>0.049252</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-63e004b9-1db0-4b5a-b6b8-2e067ed32965')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-63e004b9-1db0-4b5a-b6b8-2e067ed32965 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-63e004b9-1db0-4b5a-b6b8-2e067ed32965');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "      name     error  importance\n",
       "0  petal_l  2.609378    1.000000\n",
       "1  petal_w  0.480387    0.184100\n",
       "2  sepal_l  0.223239    0.085553\n",
       "3  sepal_w  0.128518    0.049252"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df.columns) # x+y column names\n",
    "names.remove(\"species\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, False)\n",
    "display(rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5YUQdraleV5b"
   },
   "source": [
    "## Regression and Input Perturbation Ranking\n",
    "\n",
    "We now see how to use input perturbation ranking for a regression neural network.  We will use the MPG dataset as a demonstration.  The code below loads the MPG dataset and creates a regression neural network for this dataset.  The code trains the neural network and calculates an RMSE evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kB0AMVAneV5b",
    "outputId": "94e06bc1-6028-4a0f-b76e-76df20d7925d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "10/10 - 1s - loss: 328433.8125 - 898ms/epoch - 90ms/step\n",
      "Epoch 2/100\n",
      "10/10 - 0s - loss: 78914.6406 - 26ms/epoch - 3ms/step\n",
      "Epoch 3/100\n",
      "10/10 - 0s - loss: 5371.1025 - 50ms/epoch - 5ms/step\n",
      "Epoch 4/100\n",
      "10/10 - 0s - loss: 4021.9753 - 34ms/epoch - 3ms/step\n",
      "Epoch 5/100\n",
      "10/10 - 0s - loss: 4438.5728 - 33ms/epoch - 3ms/step\n",
      "Epoch 6/100\n",
      "10/10 - 0s - loss: 1030.3115 - 34ms/epoch - 3ms/step\n",
      "Epoch 7/100\n",
      "10/10 - 0s - loss: 594.9177 - 31ms/epoch - 3ms/step\n",
      "Epoch 8/100\n",
      "10/10 - 0s - loss: 655.3908 - 31ms/epoch - 3ms/step\n",
      "Epoch 9/100\n",
      "10/10 - 0s - loss: 465.0457 - 25ms/epoch - 2ms/step\n",
      "Epoch 10/100\n",
      "10/10 - 0s - loss: 458.7520 - 30ms/epoch - 3ms/step\n",
      "Epoch 11/100\n",
      "10/10 - 0s - loss: 452.4102 - 22ms/epoch - 2ms/step\n",
      "Epoch 12/100\n",
      "10/10 - 0s - loss: 439.8730 - 25ms/epoch - 3ms/step\n",
      "Epoch 13/100\n",
      "10/10 - 0s - loss: 434.8245 - 27ms/epoch - 3ms/step\n",
      "Epoch 14/100\n",
      "10/10 - 0s - loss: 433.7303 - 25ms/epoch - 3ms/step\n",
      "Epoch 15/100\n",
      "10/10 - 0s - loss: 427.2859 - 46ms/epoch - 5ms/step\n",
      "Epoch 16/100\n",
      "10/10 - 0s - loss: 424.1164 - 50ms/epoch - 5ms/step\n",
      "Epoch 17/100\n",
      "10/10 - 0s - loss: 422.3007 - 42ms/epoch - 4ms/step\n",
      "Epoch 18/100\n",
      "10/10 - 0s - loss: 418.4877 - 31ms/epoch - 3ms/step\n",
      "Epoch 19/100\n",
      "10/10 - 0s - loss: 414.2283 - 23ms/epoch - 2ms/step\n",
      "Epoch 20/100\n",
      "10/10 - 0s - loss: 410.2691 - 34ms/epoch - 3ms/step\n",
      "Epoch 21/100\n",
      "10/10 - 0s - loss: 407.0490 - 29ms/epoch - 3ms/step\n",
      "Epoch 22/100\n",
      "10/10 - 0s - loss: 406.2433 - 46ms/epoch - 5ms/step\n",
      "Epoch 23/100\n",
      "10/10 - 0s - loss: 399.7404 - 37ms/epoch - 4ms/step\n",
      "Epoch 24/100\n",
      "10/10 - 0s - loss: 396.3280 - 66ms/epoch - 7ms/step\n",
      "Epoch 25/100\n",
      "10/10 - 0s - loss: 391.0629 - 28ms/epoch - 3ms/step\n",
      "Epoch 26/100\n",
      "10/10 - 0s - loss: 387.3203 - 26ms/epoch - 3ms/step\n",
      "Epoch 27/100\n",
      "10/10 - 0s - loss: 382.7670 - 54ms/epoch - 5ms/step\n",
      "Epoch 28/100\n",
      "10/10 - 0s - loss: 380.6316 - 21ms/epoch - 2ms/step\n",
      "Epoch 29/100\n",
      "10/10 - 0s - loss: 375.9518 - 30ms/epoch - 3ms/step\n",
      "Epoch 30/100\n",
      "10/10 - 0s - loss: 372.7001 - 24ms/epoch - 2ms/step\n",
      "Epoch 31/100\n",
      "10/10 - 0s - loss: 366.7871 - 24ms/epoch - 2ms/step\n",
      "Epoch 32/100\n",
      "10/10 - 0s - loss: 363.4180 - 42ms/epoch - 4ms/step\n",
      "Epoch 33/100\n",
      "10/10 - 0s - loss: 359.6006 - 47ms/epoch - 5ms/step\n",
      "Epoch 34/100\n",
      "10/10 - 0s - loss: 359.4055 - 46ms/epoch - 5ms/step\n",
      "Epoch 35/100\n",
      "10/10 - 0s - loss: 350.7181 - 29ms/epoch - 3ms/step\n",
      "Epoch 36/100\n",
      "10/10 - 0s - loss: 348.6260 - 42ms/epoch - 4ms/step\n",
      "Epoch 37/100\n",
      "10/10 - 0s - loss: 343.6122 - 28ms/epoch - 3ms/step\n",
      "Epoch 38/100\n",
      "10/10 - 0s - loss: 339.6165 - 32ms/epoch - 3ms/step\n",
      "Epoch 39/100\n",
      "10/10 - 0s - loss: 334.5634 - 32ms/epoch - 3ms/step\n",
      "Epoch 40/100\n",
      "10/10 - 0s - loss: 332.6061 - 34ms/epoch - 3ms/step\n",
      "Epoch 41/100\n",
      "10/10 - 0s - loss: 326.7434 - 22ms/epoch - 2ms/step\n",
      "Epoch 42/100\n",
      "10/10 - 0s - loss: 323.8063 - 40ms/epoch - 4ms/step\n",
      "Epoch 43/100\n",
      "10/10 - 0s - loss: 320.2585 - 29ms/epoch - 3ms/step\n",
      "Epoch 44/100\n",
      "10/10 - 0s - loss: 315.3609 - 23ms/epoch - 2ms/step\n",
      "Epoch 45/100\n",
      "10/10 - 0s - loss: 311.4920 - 23ms/epoch - 2ms/step\n",
      "Epoch 46/100\n",
      "10/10 - 0s - loss: 308.9212 - 29ms/epoch - 3ms/step\n",
      "Epoch 47/100\n",
      "10/10 - 0s - loss: 303.1410 - 24ms/epoch - 2ms/step\n",
      "Epoch 48/100\n",
      "10/10 - 0s - loss: 299.9317 - 24ms/epoch - 2ms/step\n",
      "Epoch 49/100\n",
      "10/10 - 0s - loss: 294.4305 - 23ms/epoch - 2ms/step\n",
      "Epoch 50/100\n",
      "10/10 - 0s - loss: 291.4469 - 24ms/epoch - 2ms/step\n",
      "Epoch 51/100\n",
      "10/10 - 0s - loss: 287.3263 - 41ms/epoch - 4ms/step\n",
      "Epoch 52/100\n",
      "10/10 - 0s - loss: 284.3096 - 49ms/epoch - 5ms/step\n",
      "Epoch 53/100\n",
      "10/10 - 0s - loss: 280.5522 - 30ms/epoch - 3ms/step\n",
      "Epoch 54/100\n",
      "10/10 - 0s - loss: 276.1487 - 26ms/epoch - 3ms/step\n",
      "Epoch 55/100\n",
      "10/10 - 0s - loss: 271.3444 - 42ms/epoch - 4ms/step\n",
      "Epoch 56/100\n",
      "10/10 - 0s - loss: 280.0936 - 33ms/epoch - 3ms/step\n",
      "Epoch 57/100\n",
      "10/10 - 0s - loss: 263.7166 - 40ms/epoch - 4ms/step\n",
      "Epoch 58/100\n",
      "10/10 - 0s - loss: 261.6750 - 56ms/epoch - 6ms/step\n",
      "Epoch 59/100\n",
      "10/10 - 0s - loss: 258.5714 - 45ms/epoch - 4ms/step\n",
      "Epoch 60/100\n",
      "10/10 - 0s - loss: 252.6791 - 31ms/epoch - 3ms/step\n",
      "Epoch 61/100\n",
      "10/10 - 0s - loss: 250.1348 - 53ms/epoch - 5ms/step\n",
      "Epoch 62/100\n",
      "10/10 - 0s - loss: 246.4157 - 72ms/epoch - 7ms/step\n",
      "Epoch 63/100\n",
      "10/10 - 0s - loss: 242.3768 - 46ms/epoch - 5ms/step\n",
      "Epoch 64/100\n",
      "10/10 - 0s - loss: 238.7874 - 28ms/epoch - 3ms/step\n",
      "Epoch 65/100\n",
      "10/10 - 0s - loss: 235.8578 - 42ms/epoch - 4ms/step\n",
      "Epoch 66/100\n",
      "10/10 - 0s - loss: 233.7492 - 24ms/epoch - 2ms/step\n",
      "Epoch 67/100\n",
      "10/10 - 0s - loss: 229.0066 - 26ms/epoch - 3ms/step\n",
      "Epoch 68/100\n",
      "10/10 - 0s - loss: 225.7449 - 25ms/epoch - 3ms/step\n",
      "Epoch 69/100\n",
      "10/10 - 0s - loss: 223.5038 - 25ms/epoch - 2ms/step\n",
      "Epoch 70/100\n",
      "10/10 - 0s - loss: 219.9561 - 39ms/epoch - 4ms/step\n",
      "Epoch 71/100\n",
      "10/10 - 0s - loss: 215.1055 - 58ms/epoch - 6ms/step\n",
      "Epoch 72/100\n",
      "10/10 - 0s - loss: 211.9364 - 39ms/epoch - 4ms/step\n",
      "Epoch 73/100\n",
      "10/10 - 0s - loss: 208.1019 - 55ms/epoch - 5ms/step\n",
      "Epoch 74/100\n",
      "10/10 - 0s - loss: 207.4119 - 34ms/epoch - 3ms/step\n",
      "Epoch 75/100\n",
      "10/10 - 0s - loss: 206.8693 - 40ms/epoch - 4ms/step\n",
      "Epoch 76/100\n",
      "10/10 - 0s - loss: 197.9749 - 49ms/epoch - 5ms/step\n",
      "Epoch 77/100\n",
      "10/10 - 0s - loss: 196.9090 - 34ms/epoch - 3ms/step\n",
      "Epoch 78/100\n",
      "10/10 - 0s - loss: 192.6349 - 45ms/epoch - 4ms/step\n",
      "Epoch 79/100\n",
      "10/10 - 0s - loss: 189.6783 - 31ms/epoch - 3ms/step\n",
      "Epoch 80/100\n",
      "10/10 - 0s - loss: 186.6584 - 25ms/epoch - 2ms/step\n",
      "Epoch 81/100\n",
      "10/10 - 0s - loss: 186.1920 - 29ms/epoch - 3ms/step\n",
      "Epoch 82/100\n",
      "10/10 - 0s - loss: 181.1735 - 31ms/epoch - 3ms/step\n",
      "Epoch 83/100\n",
      "10/10 - 0s - loss: 177.9338 - 51ms/epoch - 5ms/step\n",
      "Epoch 84/100\n",
      "10/10 - 0s - loss: 174.6662 - 87ms/epoch - 9ms/step\n",
      "Epoch 85/100\n",
      "10/10 - 0s - loss: 172.9421 - 90ms/epoch - 9ms/step\n",
      "Epoch 86/100\n",
      "10/10 - 0s - loss: 169.1906 - 58ms/epoch - 6ms/step\n",
      "Epoch 87/100\n",
      "10/10 - 0s - loss: 166.4181 - 57ms/epoch - 6ms/step\n",
      "Epoch 88/100\n",
      "10/10 - 0s - loss: 163.7466 - 36ms/epoch - 4ms/step\n",
      "Epoch 89/100\n",
      "10/10 - 0s - loss: 161.4653 - 29ms/epoch - 3ms/step\n",
      "Epoch 90/100\n",
      "10/10 - 0s - loss: 158.6274 - 30ms/epoch - 3ms/step\n",
      "Epoch 91/100\n",
      "10/10 - 0s - loss: 159.4237 - 32ms/epoch - 3ms/step\n",
      "Epoch 92/100\n",
      "10/10 - 0s - loss: 159.2035 - 31ms/epoch - 3ms/step\n",
      "Epoch 93/100\n",
      "10/10 - 0s - loss: 150.2793 - 38ms/epoch - 4ms/step\n",
      "Epoch 94/100\n",
      "10/10 - 0s - loss: 148.9276 - 36ms/epoch - 4ms/step\n",
      "Epoch 95/100\n",
      "10/10 - 0s - loss: 146.7706 - 34ms/epoch - 3ms/step\n",
      "Epoch 96/100\n",
      "10/10 - 0s - loss: 144.4946 - 29ms/epoch - 3ms/step\n",
      "Epoch 97/100\n",
      "10/10 - 0s - loss: 141.5782 - 28ms/epoch - 3ms/step\n",
      "Epoch 98/100\n",
      "10/10 - 0s - loss: 139.3355 - 27ms/epoch - 3ms/step\n",
      "Epoch 99/100\n",
      "10/10 - 0s - loss: 136.9762 - 56ms/epoch - 6ms/step\n",
      "Epoch 100/100\n",
      "10/10 - 0s - loss: 135.6660 - 23ms/epoch - 2ms/step\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "save_path = \".\"\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x_train,y_train,verbose=2,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9ynzp9RTeV5c"
   },
   "source": [
    "Just as before, we extract the column names and discard the target.  We can now create a ranking of the importance of each of the input features.  The feature with a ranking of 1.0 is the most important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "nm3PeQckeV5c",
    "outputId": "47179baf-9747-4ef9-9174-706c346cfe07"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-4b9eb361-66f3-4f65-8f5a-485d5a411b41\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>displacement</td>\n",
       "      <td>139.657598</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>acceleration</td>\n",
       "      <td>139.261508</td>\n",
       "      <td>0.997164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>origin</td>\n",
       "      <td>134.637690</td>\n",
       "      <td>0.964056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>year</td>\n",
       "      <td>134.177126</td>\n",
       "      <td>0.960758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cylinders</td>\n",
       "      <td>132.747246</td>\n",
       "      <td>0.950519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>horsepower</td>\n",
       "      <td>121.501102</td>\n",
       "      <td>0.869993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>weight</td>\n",
       "      <td>75.244610</td>\n",
       "      <td>0.538779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4b9eb361-66f3-4f65-8f5a-485d5a411b41')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-4b9eb361-66f3-4f65-8f5a-485d5a411b41 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-4b9eb361-66f3-4f65-8f5a-485d5a411b41');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "           name       error  importance\n",
       "0  displacement  139.657598    1.000000\n",
       "1  acceleration  139.261508    0.997164\n",
       "2        origin  134.637690    0.964056\n",
       "3          year  134.177126    0.960758\n",
       "4     cylinders  132.747246    0.950519\n",
       "5    horsepower  121.501102    0.869993\n",
       "6        weight   75.244610    0.538779"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df.columns) # x+y column names\n",
    "names.remove(\"name\")\n",
    "names.remove(\"mpg\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, True)\n",
    "display(rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o581oa2ceV5c"
   },
   "source": [
    "## Biological Response with Neural Network\n",
    "\n",
    "The following sections will demonstrate how to use feature importance ranking and ensembling with a more complex dataset. Ensembling is the process where you combine multiple models for greater accuracy. Kaggle competition winners frequently make use of ensembling for high-ranking solutions.\n",
    "\n",
    "We will use the biological response dataset, a Kaggle dataset, where there is an unusually high number of columns. Because of the large number of columns, it is essential to use feature ranking to determine the importance of these columns. We begin by loading the dataset and preprocessing. This Kaggle dataset is a binary classification problem. You must predict if certain conditions will cause a biological response.\n",
    "\n",
    "* [Predicting a Biological Response](https://www.kaggle.com/c/bioresponse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "O9qJ0tqueV5c"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "URL = \"https://data.heatonresearch.com/data/t81-558/kaggle/\"\n",
    "\n",
    "df_train = pd.read_csv(\n",
    "    URL+\"bio_train.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "df_test = pd.read_csv(\n",
    "    URL+\"bio_test.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "activity_classes = df_train['Activity']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7cbGqaUIeV5c"
   },
   "source": [
    "A large number of columns is evident when we display the shape of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xkVUrKiVeV5c",
    "outputId": "14cb3834-33f0-4393-dd69-c7dd3aeef9ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3751, 1777)\n"
     ]
    }
   ],
   "source": [
    "print(df_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tk-HF1CieV5c"
   },
   "source": [
    "The following code constructs a classification neural network and trains it for the biological response dataset.  Once trained, the accuracy is measured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J_miQaHneV5c",
    "outputId": "6bebb475-b6da-4991-b077-ac2a76f96138"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting/Training...\n",
      "Epoch 7: early stopping\n",
      "Fitting done...\n",
      "Validation logloss: 0.5564708781752792\n",
      "Validation accuracy score: 0.7515991471215352\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import numpy as np\n",
    "import sklearn\n",
    "\n",
    "# Encode feature vector\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df_train.columns.drop('Activity')\n",
    "x = df_train[x_columns].values\n",
    "y = df_train['Activity'].values # Classification\n",
    "x_submit = df_test[x_columns].values.astype(np.float32)\n",
    "\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42) \n",
    "\n",
    "print(\"Fitting/Training...\")\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(10))\n",
    "model.add(Dense(1,activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "                        patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=0,epochs=1000)\n",
    "print(\"Fitting done...\")\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x_test).flatten()\n",
    "\n",
    "\n",
    "# Clip so that min is never exactly 0, max never 1\n",
    "pred = np.clip(pred,a_min=1e-6,a_max=(1-1e-6)) \n",
    "print(\"Validation logloss: {}\".format(\n",
    "    sklearn.metrics.log_loss(y_test,pred)))\n",
    "\n",
    "# Evaluate success using accuracy\n",
    "pred = pred>0.5 # If greater than 0.5 probability, then true\n",
    "score = metrics.accuracy_score(y_test, pred)\n",
    "print(\"Validation accuracy score: {}\".format(score))\n",
    "\n",
    "# Build real submit file\n",
    "pred_submit = model.predict(x_submit)\n",
    "\n",
    "# Clip so that min is never exactly 0, max never 1 (would be a NaN score)\n",
    "pred = np.clip(pred,a_min=1e-6,a_max=(1-1e-6)) \n",
    "submit_df = pd.DataFrame({'MoleculeId':[x+1 for x \\\n",
    "        in range(len(pred_submit))],'PredictedProbability':\\\n",
    "                          pred_submit.flatten()})\n",
    "submit_df.to_csv(\"submit.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wluUEn10eV5d"
   },
   "source": [
    "## What Features/Columns are Important\n",
    "The following uses perturbation ranking to evaluate the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "204BmljWeV5d",
    "outputId": "619dd1de-7b67-4cb8-82a7-ccb1dcf9deaa"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-8ac5f362-c9fa-46fd-93e6-b1d63b9f218f\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>D27</td>\n",
       "      <td>0.603974</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>D1049</td>\n",
       "      <td>0.565997</td>\n",
       "      <td>0.937122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>D51</td>\n",
       "      <td>0.565883</td>\n",
       "      <td>0.936934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>D998</td>\n",
       "      <td>0.563872</td>\n",
       "      <td>0.933604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>D1059</td>\n",
       "      <td>0.563745</td>\n",
       "      <td>0.933394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>D961</td>\n",
       "      <td>0.563723</td>\n",
       "      <td>0.933357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>D1407</td>\n",
       "      <td>0.563532</td>\n",
       "      <td>0.933041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>D1309</td>\n",
       "      <td>0.562244</td>\n",
       "      <td>0.930908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>D1100</td>\n",
       "      <td>0.561902</td>\n",
       "      <td>0.930341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>D1275</td>\n",
       "      <td>0.561659</td>\n",
       "      <td>0.929940</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8ac5f362-c9fa-46fd-93e6-b1d63b9f218f')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-8ac5f362-c9fa-46fd-93e6-b1d63b9f218f button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-8ac5f362-c9fa-46fd-93e6-b1d63b9f218f');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "    name     error  importance\n",
       "0    D27  0.603974    1.000000\n",
       "1  D1049  0.565997    0.937122\n",
       "2    D51  0.565883    0.936934\n",
       "3   D998  0.563872    0.933604\n",
       "4  D1059  0.563745    0.933394\n",
       "5   D961  0.563723    0.933357\n",
       "6  D1407  0.563532    0.933041\n",
       "7  D1309  0.562244    0.930908\n",
       "8  D1100  0.561902    0.930341\n",
       "9  D1275  0.561659    0.929940"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df_train.columns) # x+y column names\n",
    "names.remove(\"Activity\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, False)\n",
    "display(rank[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_EejOUAleV5d"
   },
   "source": [
    "## Neural Network Ensemble\n",
    "\n",
    "A neural network ensemble combines neural network predictions with other models. The program determines the exact blend of these models by logistic regression. The following code performs this blend for a classification.  If you present the final predictions from the ensemble to Kaggle, you will see that the result is very accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LaXLD3yneV5d",
    "outputId": "2ba6092b-ae16-4b63-fa39-7d3a2a0251bb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:44: DeprecationWarning: KerasClassifier is deprecated, use Sci-Keras (https://github.com/adriangb/scikeras) instead. See https://www.adriangb.com/scikeras/stable/migration.html for help migrating.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 : <keras.wrappers.scikit_learn.KerasClassifier object at 0x7fc869809610>\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.6048\n",
      "Fold #0: loss=0.5544745638322883\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.6046\n",
      "Fold #1: loss=0.5684765604955473\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5943\n",
      "Fold #2: loss=0.5214491621944897\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.6301\n",
      "Fold #3: loss=0.5264746750391351\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5905\n",
      "Fold #4: loss=0.5327822461352748\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5993\n",
      "Fold #5: loss=0.5800157462831582\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5877\n",
      "Fold #6: loss=0.5189563830365144\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.6038\n",
      "Fold #7: loss=0.5625417655617023\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5935\n",
      "Fold #8: loss=0.5238374326475557\n",
      "106/106 [==============================] - 1s 2ms/step - loss: 0.5991\n",
      "Fold #9: loss=0.5322226787930878\n",
      "KerasClassifier: Mean loss=0.5421231214018752\n",
      "Model: 1 : KNeighborsClassifier(n_neighbors=3)\n",
      "Fold #0: loss=3.606678388314123\n",
      "Fold #1: loss=2.2256421551487593\n",
      "Fold #2: loss=3.6815437059542186\n",
      "Fold #3: loss=2.416161292225968\n",
      "Fold #4: loss=4.442472310149748\n",
      "Fold #5: loss=4.321350530738247\n",
      "Fold #6: loss=3.400455469543658\n",
      "Fold #7: loss=3.1724147110842513\n",
      "Fold #8: loss=2.117356283193681\n",
      "Fold #9: loss=3.0532135963322586\n",
      "KNeighborsClassifier: Mean loss=3.243728844268491\n",
      "Model: 2 : RandomForestClassifier(n_jobs=-1)\n",
      "Fold #0: loss=0.4657177982691548\n",
      "Fold #1: loss=0.4346825805694879\n",
      "Fold #2: loss=0.4593868993445528\n",
      "Fold #3: loss=0.41674899522216713\n",
      "Fold #4: loss=0.4851849131056564\n",
      "Fold #5: loss=0.48473291073937\n",
      "Fold #6: loss=0.41274608628217674\n",
      "Fold #7: loss=0.47405291219252377\n",
      "Fold #8: loss=0.44974230059938286\n",
      "Fold #9: loss=0.46340159258241087\n",
      "RandomForestClassifier: Mean loss=0.45463969889068834\n",
      "Model: 3 : RandomForestClassifier(criterion='entropy', n_jobs=-1)\n",
      "Fold #0: loss=0.4511847247326708\n",
      "Fold #1: loss=0.42707704254926593\n",
      "Fold #2: loss=0.5550335199035183\n",
      "Fold #3: loss=0.42186970733328516\n",
      "Fold #4: loss=0.4794331756190797\n",
      "Fold #5: loss=0.4730559509802762\n",
      "Fold #6: loss=0.41116235817215196\n",
      "Fold #7: loss=0.46835919493314265\n",
      "Fold #8: loss=0.4496144890690015\n",
      "Fold #9: loss=0.4625902934553457\n",
      "RandomForestClassifier: Mean loss=0.4599380456747738\n",
      "Model: 4 : ExtraTreesClassifier(n_jobs=-1)\n",
      "Fold #0: loss=0.45496751079363495\n",
      "Fold #1: loss=0.5013051157905043\n",
      "Fold #2: loss=0.5886179891724027\n",
      "Fold #3: loss=0.41646902160044674\n",
      "Fold #4: loss=0.4957910697444236\n",
      "Fold #5: loss=0.4773401028797005\n",
      "Fold #6: loss=0.41935061504547827\n",
      "Fold #7: loss=0.5757908399174205\n",
      "Fold #8: loss=0.4585195863412778\n",
      "Fold #9: loss=0.6210675972963805\n",
      "ExtraTreesClassifier: Mean loss=0.500921944858167\n",
      "Model: 5 : ExtraTreesClassifier(criterion='entropy', n_jobs=-1)\n",
      "Fold #0: loss=0.44825346440152214\n",
      "Fold #1: loss=0.40764412171784686\n",
      "Fold #2: loss=0.5819367378417363\n",
      "Fold #3: loss=0.4140589874942631\n",
      "Fold #4: loss=0.4923489720481471\n",
      "Fold #5: loss=0.5744429921555051\n",
      "Fold #6: loss=0.42334390524742155\n",
      "Fold #7: loss=0.6409291880353659\n",
      "Fold #8: loss=0.45627884947155956\n",
      "Fold #9: loss=0.466653395317917\n",
      "ExtraTreesClassifier: Mean loss=0.49058906137312847\n",
      "Model: 6 : GradientBoostingClassifier(learning_rate=0.05, max_depth=6, n_estimators=50,\n",
      "                           subsample=0.5)\n",
      "Fold #0: loss=0.4789324034433162\n",
      "Fold #1: loss=0.4565636914381977\n",
      "Fold #2: loss=0.47057741836357014\n",
      "Fold #3: loss=0.4436328438944843\n",
      "Fold #4: loss=0.4883293501002484\n",
      "Fold #5: loss=0.4843521206311074\n",
      "Fold #6: loss=0.4436043855503229\n",
      "Fold #7: loss=0.45977393398397765\n",
      "Fold #8: loss=0.46632256794136323\n",
      "Fold #9: loss=0.4703354072414907\n",
      "GradientBoostingClassifier: Mean loss=0.4662424122588078\n",
      "\n",
      "Blending models.\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import math\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier \n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "SHUFFLE = False\n",
    "FOLDS = 10\n",
    "\n",
    "def build_ann(input_size,classes,neurons):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_dim=input_size, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Dense(classes,activation='softmax'))\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "    return model\n",
    "\n",
    "def mlogloss(y_test, preds):\n",
    "    epsilon = 1e-15\n",
    "    sum = 0\n",
    "    for row in zip(preds,y_test):\n",
    "        x = row[0][row[1]]\n",
    "        x = max(epsilon,x)\n",
    "        x = min(1-epsilon,x)\n",
    "        sum+=math.log(x)\n",
    "    return( (-1/len(preds))*sum)\n",
    "\n",
    "def stretch(y):\n",
    "    return (y - y.min()) / (y.max() - y.min())\n",
    "\n",
    "\n",
    "def blend_ensemble(x, y, x_submit):\n",
    "    kf = StratifiedKFold(FOLDS)\n",
    "    folds = list(kf.split(x,y))\n",
    "\n",
    "    models = [\n",
    "        KerasClassifier(build_fn=build_ann,neurons=20,\n",
    "                    input_size=x.shape[1],classes=2),\n",
    "        KNeighborsClassifier(n_neighbors=3),\n",
    "        RandomForestClassifier(n_estimators=100, n_jobs=-1, \n",
    "                               criterion='gini'),\n",
    "        RandomForestClassifier(n_estimators=100, n_jobs=-1, \n",
    "                               criterion='entropy'),\n",
    "        ExtraTreesClassifier(n_estimators=100, n_jobs=-1, \n",
    "                             criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=100, n_jobs=-1, \n",
    "                             criterion='entropy'),\n",
    "        GradientBoostingClassifier(learning_rate=0.05, \n",
    "                subsample=0.5, max_depth=6, n_estimators=50)]\n",
    "\n",
    "    dataset_blend_train = np.zeros((x.shape[0], len(models)))\n",
    "    dataset_blend_test = np.zeros((x_submit.shape[0], len(models)))\n",
    "\n",
    "    for j, model in enumerate(models):\n",
    "        print(\"Model: {} : {}\".format(j, model) )\n",
    "        fold_sums = np.zeros((x_submit.shape[0], len(folds)))\n",
    "        total_loss = 0\n",
    "        for i, (train, test) in enumerate(folds):\n",
    "            x_train = x[train]\n",
    "            y_train = y[train]\n",
    "            x_test = x[test]\n",
    "            y_test = y[test]\n",
    "            model.fit(x_train, y_train)\n",
    "            pred = np.array(model.predict_proba(x_test))\n",
    "            dataset_blend_train[test, j] = pred[:, 1]\n",
    "            pred2 = np.array(model.predict_proba(x_submit))\n",
    "            fold_sums[:, i] = pred2[:, 1]\n",
    "            loss = mlogloss(y_test, pred)\n",
    "            total_loss+=loss\n",
    "            print(\"Fold #{}: loss={}\".format(i,loss))\n",
    "        print(\"{}: Mean loss={}\".format(model.__class__.__name__,\n",
    "                                        total_loss/len(folds)))\n",
    "        dataset_blend_test[:, j] = fold_sums.mean(1)\n",
    "\n",
    "    print()\n",
    "    print(\"Blending models.\")\n",
    "    blend = LogisticRegression(solver='lbfgs')\n",
    "    blend.fit(dataset_blend_train, y)\n",
    "    return blend.predict_proba(dataset_blend_test)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    np.random.seed(42)  # seed to shuffle the train set\n",
    "\n",
    "    print(\"Loading data...\")\n",
    "    URL = \"https://data.heatonresearch.com/data/t81-558/kaggle/\"\n",
    "\n",
    "    df_train = pd.read_csv(\n",
    "        URL+\"bio_train.csv\", \n",
    "        na_values=['NA', '?'])\n",
    "\n",
    "    df_submit = pd.read_csv(\n",
    "        URL+\"bio_test.csv\", \n",
    "        na_values=['NA', '?'])\n",
    "\n",
    "    predictors = list(df_train.columns.values)\n",
    "    predictors.remove('Activity')\n",
    "    x = df_train[predictors].values\n",
    "    y = df_train['Activity']\n",
    "    x_submit = df_submit.values\n",
    "\n",
    "    if SHUFFLE:\n",
    "        idx = np.random.permutation(y.size)\n",
    "        x = x[idx]\n",
    "        y = y[idx]\n",
    "\n",
    "    submit_data = blend_ensemble(x, y, x_submit)\n",
    "    submit_data = stretch(submit_data)\n",
    "\n",
    "    ####################\n",
    "    # Build submit file\n",
    "    ####################\n",
    "    ids = [id+1 for id in range(submit_data.shape[0])]\n",
    "    submit_df = pd.DataFrame({'MoleculeId': ids, \n",
    "                              'PredictedProbability': \n",
    "                              submit_data[:, 1]},\n",
    "                             columns=['MoleculeId',\n",
    "                            'PredictedProbability'])\n",
    "    submit_df.to_csv(\"submit.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "dBfgUuateV5d"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_08_2_keras_ensembles.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}