t81_558_class_14_03_anomaly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UhLdicqoAMGl"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_03_anomaly.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BGssM4-jAMGn"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64qCM6NsAMGn"
   },
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=1mB_5iurqzw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_02_auto_encode.ipynb)\n",
    "* **Part 14.3: Training an Intrusion Detection System with KDD99** [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* Part 14.5: The Deep Learning Technologies I am Excited About [[Video]]() [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k9pmbHghAMGo"
   },
   "source": [
    "# Part 14.3: Anomaly Detection in Keras\n",
    "\n",
    "Anomaly detection is an unsupervised training technique that analyzes the degree to which incoming data differs from the data you used to train the neural network. Traditionally, cybersecurity experts have used anomaly detection to ensure network security. However, you can use anomalies in data science to detect input for which you have not trained your neural network.  \n",
    "\n",
    "There are several data sets that many commonly use to demonstrate anomaly detection. In this part, we will look at the KDD-99 dataset.\n",
    "\n",
    "\n",
    "* [Stratosphere IPS Dataset](https://www.stratosphereips.org/category/dataset.html)\n",
    "* [The ADFA Intrusion Detection Datasets (2013) - for HIDS](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-IDS-Datasets/)\n",
    "* [ITOC CDX (2009)](https://westpoint.edu/centers-and-research/cyber-research-center/data-sets)\n",
    "* [KDD-99 Dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html)\n",
    "\n",
    "## Read in KDD99 Data Set\n",
    "\n",
    "Although the KDD99 dataset is over 20 years old, it is still widely used to demonstrate Intrusion Detection Systems (IDS) and Anomaly detection. KDD99 is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, held in conjunction with KDD-99, The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between \"bad\" connections, called intrusions or attacks, and \"good\" normal connections. This database contains a standard set of data to be audited, including various intrusions simulated in a military network environment.\n",
    "\n",
    "The following code reads the KDD99 CSV dataset into a Pandas data frame. The standard format of KDD99 does not include column names. Because of that, the program adds them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 325
    },
    "id": "Z7Nc6xKjAMGo",
    "outputId": "854a3977-8e15-4d42-a9be-917ea70cb108"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://github.com/jeffheaton/jheaton-ds2/raw/main/kdd-with-columns.csv\n",
      "68132864/68132668 [==============================] - 1s 0us/step\n",
      "68141056/68132668 [==============================] - 1s 0us/step\n",
      "/root/.keras/datasets/kdd-with-columns.csv\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-366e16f8-0305-452d-945d-e44ad3c0f275\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>494019</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>494020</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>494021 rows × 42 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-366e16f8-0305-452d-945d-e44ad3c0f275')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-366e16f8-0305-452d-945d-e44ad3c0f275 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-366e16f8-0305-452d-945d-e44ad3c0f275');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "        duration protocol_type  ... dst_host_srv_rerror_rate  outcome\n",
       "0              0           tcp  ...                      0.0  normal.\n",
       "1              0           tcp  ...                      0.0  normal.\n",
       "...          ...           ...  ...                      ...      ...\n",
       "494019         0           tcp  ...                      0.0  normal.\n",
       "494020         0           tcp  ...                      0.0  normal.\n",
       "\n",
       "[494021 rows x 42 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "try:\n",
    "    path = get_file('kdd-with-columns.csv', origin=\\\n",
    "    'https://github.com/jeffheaton/jheaton-ds2/raw/main/'\\\n",
    "    'kdd-with-columns.csv',archive_format=None)\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# Origional file: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to \n",
    "# sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) \n",
    "# For now, just drop NA's (rows with missing values)\n",
    "\n",
    "\n",
    "# display 5 rows\n",
    "pd.set_option('display.max_columns', 5)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qiiHziMXAMGp"
   },
   "source": [
    "The KDD99 dataset contains many columns that define the network state over time intervals during which a cyber attack might have taken place.  The \" outcome \" column specifies either \"normal,\" indicating no attack, or the type of attack performed.  The following code displays the counts for each type of attack and \"normal\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cxgbeLBpAMGq",
    "outputId": "11827d16-bbaf-4edc-c5fb-d25c787aac07"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "back.               2203\n",
       "buffer_overflow.      30\n",
       "                    ... \n",
       "warezclient.        1020\n",
       "warezmaster.          20\n",
       "Name: outcome, Length: 23, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('outcome')['outcome'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Y1YqytVAMGq"
   },
   "source": [
    "## Preprocessing \n",
    "\n",
    "We must perform some preprocessing before we can feed the KDD99 data into the neural network. We provide the following two functions to assist with preprocessing. The first function converts numeric columns into Z-Scores. The second function replaces categorical values with dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "xRR6q2kOAMGq"
   },
   "outputs": [],
   "source": [
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "    \n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] \n",
    "# for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DuEFJKt_AMGr"
   },
   "source": [
    "This code converts all numeric columns to Z-Scores and all textual columns to dummy variables. We now use these functions to preprocess each of the columns. Once the program preprocesses the data, we display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "id": "stntDxM9AMGr",
    "outputId": "ac43723d-a105-4105-fe1a-d45153e3294f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-cc155a21-199b-449f-bbce-0391702b42a2\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>...</th>\n",
       "      <th>is_host_login-0</th>\n",
       "      <th>is_guest_login-0</th>\n",
       "      <th>is_guest_login-1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002879</td>\n",
       "      <td>0.138664</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002820</td>\n",
       "      <td>-0.011578</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002824</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002840</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002842</td>\n",
       "      <td>0.035214</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cc155a21-199b-449f-bbce-0391702b42a2')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-cc155a21-199b-449f-bbce-0391702b42a2 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-cc155a21-199b-449f-bbce-0391702b42a2');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   duration  src_bytes  dst_bytes  ...  is_host_login-0  is_guest_login-0  \\\n",
       "0 -0.067792  -0.002879   0.138664  ...                1                 1   \n",
       "1 -0.067792  -0.002820  -0.011578  ...                1                 1   \n",
       "2 -0.067792  -0.002824   0.014179  ...                1                 1   \n",
       "3 -0.067792  -0.002840   0.014179  ...                1                 1   \n",
       "4 -0.067792  -0.002842   0.035214  ...                1                 1   \n",
       "\n",
       "   is_guest_login-1  \n",
       "0                 0  \n",
       "1                 0  \n",
       "2                 0  \n",
       "3                 0  \n",
       "4                 0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "for name in df.columns:\n",
    "  if name == 'outcome':\n",
    "    pass\n",
    "  elif name in ['protocol_type','service','flag','land','logged_in',\n",
    "                'is_host_login','is_guest_login']:\n",
    "    encode_text_dummy(df,name)\n",
    "  else:\n",
    "    encode_numeric_zscore(df,name)    \n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fyHlHDwGAMGr"
   },
   "source": [
    "We divide the data into two groups, \"normal\" and the various attacks to perform anomaly detection. The following code divides the data into two data frames and displays each of these two groups' sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vZhVyry_AMGr",
    "outputId": "4809f4ec-49d3-419d-a82b-25dd9c52f3c9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal count: 97278\n",
      "Attack count: 396743\n"
     ]
    }
   ],
   "source": [
    "normal_mask = df['outcome']=='normal.'\n",
    "attack_mask = df['outcome']!='normal.'\n",
    "\n",
    "df.drop('outcome',axis=1,inplace=True)\n",
    "\n",
    "df_normal = df[normal_mask]\n",
    "df_attack = df[attack_mask]\n",
    "\n",
    "print(f\"Normal count: {len(df_normal)}\")\n",
    "print(f\"Attack count: {len(df_attack)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QnihkQx_AMGr"
   },
   "source": [
    "Next, we convert these two data frames into Numpy arrays. Keras requires this format for data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Vr_C7KEUAMGs"
   },
   "outputs": [],
   "source": [
    "# This is the numeric feature vector, as it goes to the neural net\n",
    "x_normal = df_normal.values\n",
    "x_attack = df_attack.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YH_QLaPvAMGs"
   },
   "source": [
    "## Training the Autoencoder\n",
    "\n",
    "It is important to note that we are not using the outcome column as a label to predict. We will train an autoencoder on the normal data and see how well it can detect that the data not flagged as \"normal\" represents an anomaly. This anomaly detection is unsupervised; there is no target (y) value to predict. \n",
    "\n",
    "Next, we split the normal data into a 25% test set and a 75% train set. The program will use the test data to facilitate early stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "MX6PCz-RAMGs"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_normal_train, x_normal_test = train_test_split(\n",
    "    x_normal, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ozvx3sr5AMGs"
   },
   "source": [
    "We display the size of the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CuH9Bxg6AMGs",
    "outputId": "1e8aa872-3c95-4299-aa30-896577f2db84"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal train count: 72958\n",
      "Normal test count: 24320\n"
     ]
    }
   ],
   "source": [
    "print(f\"Normal train count: {len(x_normal_train)}\")\n",
    "print(f\"Normal test count: {len(x_normal_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06DNCbB4AMGs"
   },
   "source": [
    "We are now ready to train the autoencoder on the normal data. The autoencoder will learn to compress the data to a vector of just three numbers. The autoencoder should be able to also decompress with reasonable accuracy. As is typical for autoencoders, we are merely training the neural network to produce the same output values as were fed to the input layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fpXiVRTuAMGs",
    "outputId": "d5280198-5f5c-4c3c-b0be-f0aaa68bcb75"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.3912\n",
      "Epoch 2/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.2831\n",
      "Epoch 3/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.2448\n",
      "Epoch 4/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.2268\n",
      "Epoch 5/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1952\n",
      "Epoch 6/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1939\n",
      "Epoch 7/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1830\n",
      "Epoch 8/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1894\n",
      "Epoch 9/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1863\n",
      "Epoch 10/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1740\n",
      "Epoch 11/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1774\n",
      "Epoch 12/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1657\n",
      "Epoch 13/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1657\n",
      "Epoch 14/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1596\n",
      "Epoch 15/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1593\n",
      "Epoch 16/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1561\n",
      "Epoch 17/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1516\n",
      "Epoch 18/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1539\n",
      "Epoch 19/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1506\n",
      "Epoch 20/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.1503\n",
      "Epoch 21/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1553\n",
      "Epoch 22/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1498\n",
      "Epoch 23/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.1491\n",
      "Epoch 24/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1465\n",
      "Epoch 25/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1474\n",
      "Epoch 26/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1476\n",
      "Epoch 27/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1445\n",
      "Epoch 28/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1416\n",
      "Epoch 29/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1410\n",
      "Epoch 30/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1404\n",
      "Epoch 31/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1363\n",
      "Epoch 32/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1354\n",
      "Epoch 33/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1352\n",
      "Epoch 34/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1322\n",
      "Epoch 35/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1313\n",
      "Epoch 36/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1254\n",
      "Epoch 37/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1273\n",
      "Epoch 38/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1197\n",
      "Epoch 39/100\n",
      "2280/2280 [==============================] - 10s 4ms/step - loss: 0.1220\n",
      "Epoch 40/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1163\n",
      "Epoch 41/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1156\n",
      "Epoch 42/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1084\n",
      "Epoch 43/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1109\n",
      "Epoch 44/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1054\n",
      "Epoch 45/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.1024\n",
      "Epoch 46/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1058\n",
      "Epoch 47/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0980\n",
      "Epoch 48/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.1034\n",
      "Epoch 49/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0995\n",
      "Epoch 50/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0920\n",
      "Epoch 51/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0877\n",
      "Epoch 52/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0967\n",
      "Epoch 53/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0842\n",
      "Epoch 54/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0835\n",
      "Epoch 55/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0829\n",
      "Epoch 56/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0834\n",
      "Epoch 57/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0781\n",
      "Epoch 58/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0774\n",
      "Epoch 59/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0703\n",
      "Epoch 60/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0751\n",
      "Epoch 61/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0787\n",
      "Epoch 62/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0766\n",
      "Epoch 63/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0665\n",
      "Epoch 64/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0690\n",
      "Epoch 65/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0708\n",
      "Epoch 66/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0689\n",
      "Epoch 67/100\n",
      "2280/2280 [==============================] - 8s 3ms/step - loss: 0.0651\n",
      "Epoch 68/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0640\n",
      "Epoch 69/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0664\n",
      "Epoch 70/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0617\n",
      "Epoch 71/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0640\n",
      "Epoch 72/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0626\n",
      "Epoch 73/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0595\n",
      "Epoch 74/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0646\n",
      "Epoch 75/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0646\n",
      "Epoch 76/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0606\n",
      "Epoch 77/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0592\n",
      "Epoch 78/100\n",
      "2280/2280 [==============================] - 7s 3ms/step - loss: 0.0604\n",
      "Epoch 79/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0636\n",
      "Epoch 80/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0630\n",
      "Epoch 81/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0552\n",
      "Epoch 82/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0586\n",
      "Epoch 83/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0629\n",
      "Epoch 84/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0582\n",
      "Epoch 85/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0618\n",
      "Epoch 86/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0560\n",
      "Epoch 87/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0582\n",
      "Epoch 88/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0572\n",
      "Epoch 89/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0585\n",
      "Epoch 90/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0593\n",
      "Epoch 91/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0552\n",
      "Epoch 92/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0537\n",
      "Epoch 93/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0625\n",
      "Epoch 94/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0561\n",
      "Epoch 95/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0536\n",
      "Epoch 96/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0580\n",
      "Epoch 97/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0546\n",
      "Epoch 98/100\n",
      "2280/2280 [==============================] - 6s 2ms/step - loss: 0.0548\n",
      "Epoch 99/100\n",
      "2280/2280 [==============================] - 6s 3ms/step - loss: 0.0512\n",
      "Epoch 100/100\n",
      "2280/2280 [==============================] - 5s 2ms/step - loss: 0.0562\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb3db41d590>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x_normal.shape[1], activation='relu'))\n",
    "model.add(Dense(3, activation='relu')) # size to compress to\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(x_normal.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x_normal_train,x_normal_train,verbose=1,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mw9KbY1LAMGs"
   },
   "source": [
    "## Detecting an Anomaly\n",
    "\n",
    "We are now ready to see if the abnormal data is an anomaly. The first two scores show the in-sample and out of sample RMSE errors. These two scores are relatively low at around 0.33 because they resulted from normal data. The much higher 0.76 error occurred from the abnormal data. The autoencoder is not as capable of encoding data that represents an attack. This higher error indicates an anomaly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NWFVmgy-AMGs",
    "outputId": "6e5d1893-dac3-48df-b9b8-cce31db3e3f3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out of Sample Normal Score (RMSE): 0.27485267641044275\n",
      "Insample Normal Score (RMSE): 0.24613762509093587\n",
      "Attack Underway Score (RMSE): 0.6398492471974858\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x_normal_test)\n",
    "score1 = np.sqrt(metrics.mean_squared_error(pred,x_normal_test))\n",
    "pred = model.predict(x_normal)\n",
    "score2 = np.sqrt(metrics.mean_squared_error(pred,x_normal))\n",
    "pred = model.predict(x_attack)\n",
    "score3 = np.sqrt(metrics.mean_squared_error(pred,x_attack))\n",
    "print(f\"Out of Sample Normal Score (RMSE): {score1}\")\n",
    "print(f\"Insample Normal Score (RMSE): {score2}\")\n",
    "print(f\"Attack Underway Score (RMSE): {score3}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_14_03_anomaly.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}