t81_558_class_14_04_ids_kdd99.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KzGsP2__Fl6L"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_04_ids_kdd99.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zw-8u9pdFl6M"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aiG4cu4xFl6M"
   },
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=1mB_5iurqzw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_02_auto_encode.ipynb)\n",
    "* Part 14.3: Training an Intrusion Detection System with KDD99 [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_03_anomaly.ipynb)\n",
    "* **Part 14.4: Anomaly Detection in Keras** [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* Part 14.5: The Deep Learning Technologies I am Excited About [[Video]]() [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xR5gRHDiFl6M"
   },
   "source": [
    "# Part 14.4: Training an Intrusion Detection System with KDD99\n",
    "\n",
    "The [KDD-99 dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) is very famous in the security field and almost a \"hello world\" of Intrusion Detection Systems (IDS) in machine learning. An intrusion detection system (IDS) is a program that monitors computers and network systems for malicious activity or policy violations. Any intrusion activity or violation is typically reported to an administrator or collected centrally. IDS types range in scope from single computers to large networks. Although the KDD99 dataset is over 20 years old, it is still widely used to demonstrate Intrusion Detection Systems (IDS). KDD99 is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, which was held in conjunction with KDD-99, The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between \"bad\" connections, called intrusions or attacks, and \"good\" normal connections. This database contains a standard set of data to be audited, including various intrusions simulated in a military network environment.\n",
    "\n",
    "\n",
    "## Read in Raw KDD-99 Dataset\n",
    "\n",
    "The following code reads the KDD99 CSV dataset into a Pandas data frame. The standard format of KDD99 does not include column names. Because of that, the program adds them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 325
    },
    "id": "pY9Jy3ZbFl6N",
    "outputId": "3a66550a-267e-4ccd-93c0-bda362b05893"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://github.com/jeffheaton/jheaton-ds2/raw/main/kdd-with-columns.csv\n",
      "68132864/68132668 [==============================] - 1s 0us/step\n",
      "68141056/68132668 [==============================] - 1s 0us/step\n",
      "/root/.keras/datasets/kdd-with-columns.csv\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-46feac53-0e1f-4579-9b8d-576426cfe19b\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>494019</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>494020</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>494021 rows × 42 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-46feac53-0e1f-4579-9b8d-576426cfe19b')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-46feac53-0e1f-4579-9b8d-576426cfe19b button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-46feac53-0e1f-4579-9b8d-576426cfe19b');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "        duration protocol_type  ... dst_host_srv_rerror_rate  outcome\n",
       "0              0           tcp  ...                      0.0  normal.\n",
       "1              0           tcp  ...                      0.0  normal.\n",
       "...          ...           ...  ...                      ...      ...\n",
       "494019         0           tcp  ...                      0.0  normal.\n",
       "494020         0           tcp  ...                      0.0  normal.\n",
       "\n",
       "[494021 rows x 42 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "try:\n",
    "    path = get_file('kdd-with-columns.csv', origin=\\\n",
    "    'https://github.com/jeffheaton/jheaton-ds2/raw/main/'\\\n",
    "    'kdd-with-columns.csv',archive_format=None)\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# Origional file: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to \n",
    "# sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) \n",
    "# For now, just drop NA's (rows with missing values)\n",
    "\n",
    "\n",
    "# display 5 rows\n",
    "pd.set_option('display.max_columns', 5)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MQasim3FFl6O"
   },
   "source": [
    "## Analyzing a Dataset\n",
    "\n",
    "Before we preprocess the KDD99 dataset, let's look at the individual columns and distributions.  You can use the following script to give a high-level overview of how a dataset appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "BBfsfvwoFl6O"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "def expand_categories(values):\n",
    "    result = []\n",
    "    s = values.value_counts()\n",
    "    t = float(len(values))\n",
    "    for v in s.index:\n",
    "        result.append(\"{}:{}%\".format(v,round(100*(s[v]/t),2)))\n",
    "    return \"[{}]\".format(\",\".join(result))\n",
    "        \n",
    "def analyze(df):\n",
    "    print()\n",
    "    cols = df.columns.values\n",
    "    total = float(len(df))\n",
    "\n",
    "    print(\"{} rows\".format(int(total)))\n",
    "    for col in cols:\n",
    "        uniques = df[col].unique()\n",
    "        unique_count = len(uniques)\n",
    "        if unique_count>100:\n",
    "            print(\"** {}:{} ({}%)\".format(col,unique_count,\\\n",
    "                int(((unique_count)/total)*100)))\n",
    "        else:\n",
    "            print(\"** {}:{}\".format(col,expand_categories(df[col])))\n",
    "            expand_categories(df[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h1xLBxINFl6P"
   },
   "source": [
    "The analysis looks at how many unique values are present.  For example, duration, a numeric value, has 2495 unique values, and there is a 0% overlap.  A text/categorical value such as protocol_type only has a few unique values, and the program shows the percentages of each.  Columns with many unique values do not have their item counts shown to save display space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PBj7r_wbFl6P",
    "outputId": "aca8d7e8-20c4-43fd-9919-15242ec04598"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "494021 rows\n",
      "** duration:2495 (0%)\n",
      "** protocol_type:[icmp:57.41%,tcp:38.47%,udp:4.12%]\n",
      "** service:[ecr_i:56.96%,private:22.45%,http:13.01%,smtp:1.97%,other:1.46%,domain_u:1.19%,ftp_data:0.96%,eco_i:0.33%,ftp:0.16%,finger:0.14%,urp_i:0.11%,telnet:0.1%,ntp_u:0.08%,auth:0.07%,pop_3:0.04%,time:0.03%,csnet_ns:0.03%,remote_job:0.02%,gopher:0.02%,imap4:0.02%,discard:0.02%,domain:0.02%,iso_tsap:0.02%,systat:0.02%,shell:0.02%,echo:0.02%,rje:0.02%,whois:0.02%,sql_net:0.02%,printer:0.02%,nntp:0.02%,courier:0.02%,sunrpc:0.02%,netbios_ssn:0.02%,mtp:0.02%,vmnet:0.02%,uucp_path:0.02%,uucp:0.02%,klogin:0.02%,bgp:0.02%,ssh:0.02%,supdup:0.02%,nnsp:0.02%,login:0.02%,hostnames:0.02%,efs:0.02%,daytime:0.02%,link:0.02%,netbios_ns:0.02%,pop_2:0.02%,ldap:0.02%,netbios_dgm:0.02%,exec:0.02%,http_443:0.02%,kshell:0.02%,name:0.02%,ctf:0.02%,netstat:0.02%,Z39_50:0.02%,IRC:0.01%,urh_i:0.0%,X11:0.0%,tim_i:0.0%,pm_dump:0.0%,tftp_u:0.0%,red_i:0.0%]\n",
      "** flag:[SF:76.6%,S0:17.61%,REJ:5.44%,RSTR:0.18%,RSTO:0.12%,SH:0.02%,S1:0.01%,S2:0.0%,RSTOS0:0.0%,S3:0.0%,OTH:0.0%]\n",
      "** src_bytes:3300 (0%)\n",
      "** dst_bytes:10725 (2%)\n",
      "** land:[0:100.0%,1:0.0%]\n",
      "** wrong_fragment:[0:99.75%,3:0.2%,1:0.05%]\n",
      "** urgent:[0:100.0%,1:0.0%,2:0.0%,3:0.0%]\n",
      "** hot:[0:99.35%,2:0.44%,28:0.06%,1:0.05%,4:0.02%,6:0.02%,5:0.01%,3:0.01%,14:0.01%,30:0.01%,22:0.01%,19:0.0%,24:0.0%,18:0.0%,20:0.0%,7:0.0%,17:0.0%,12:0.0%,16:0.0%,10:0.0%,15:0.0%,9:0.0%]\n",
      "** num_failed_logins:[0:99.99%,1:0.01%,2:0.0%,5:0.0%,4:0.0%,3:0.0%]\n",
      "** logged_in:[0:85.18%,1:14.82%]\n",
      "** num_compromised:[0:99.55%,1:0.44%,2:0.0%,4:0.0%,3:0.0%,6:0.0%,5:0.0%,7:0.0%,767:0.0%,12:0.0%,9:0.0%,884:0.0%,13:0.0%,38:0.0%,18:0.0%,11:0.0%,275:0.0%,281:0.0%,16:0.0%,238:0.0%,21:0.0%,22:0.0%,102:0.0%]\n",
      "** root_shell:[0:99.99%,1:0.01%]\n",
      "** su_attempted:[0:100.0%,1:0.0%,2:0.0%]\n",
      "** num_root:[0:99.88%,1:0.05%,9:0.03%,6:0.03%,2:0.0%,5:0.0%,4:0.0%,3:0.0%,7:0.0%,993:0.0%,54:0.0%,306:0.0%,14:0.0%,39:0.0%,278:0.0%,268:0.0%,12:0.0%,857:0.0%,16:0.0%,119:0.0%]\n",
      "** num_file_creations:[0:99.95%,1:0.04%,2:0.01%,4:0.0%,16:0.0%,5:0.0%,22:0.0%,25:0.0%,12:0.0%,8:0.0%,7:0.0%,21:0.0%,14:0.0%,10:0.0%,28:0.0%,9:0.0%,15:0.0%,20:0.0%]\n",
      "** num_shells:[0:99.99%,1:0.01%,2:0.0%]\n",
      "** num_access_files:[0:99.91%,1:0.09%,2:0.01%,3:0.0%,4:0.0%,6:0.0%,8:0.0%]\n",
      "** num_outbound_cmds:[0:100.0%]\n",
      "** is_host_login:[0:100.0%]\n",
      "** is_guest_login:[0:99.86%,1:0.14%]\n",
      "** count:490 (0%)\n",
      "** srv_count:470 (0%)\n",
      "** serror_rate:[0.0:81.94%,1.0:17.52%,0.99:0.06%,0.08:0.03%,0.05:0.03%,0.06:0.03%,0.07:0.03%,0.14:0.02%,0.04:0.02%,0.01:0.02%,0.09:0.02%,0.1:0.02%,0.03:0.02%,0.11:0.02%,0.13:0.02%,0.5:0.02%,0.12:0.02%,0.2:0.01%,0.25:0.01%,0.02:0.01%,0.17:0.01%,0.33:0.01%,0.15:0.01%,0.22:0.01%,0.18:0.01%,0.23:0.01%,0.16:0.01%,0.21:0.01%,0.19:0.0%,0.27:0.0%,0.98:0.0%,0.29:0.0%,0.44:0.0%,0.24:0.0%,0.97:0.0%,0.31:0.0%,0.96:0.0%,0.26:0.0%,0.79:0.0%,0.28:0.0%,0.36:0.0%,0.94:0.0%,0.95:0.0%,0.65:0.0%,0.67:0.0%,0.85:0.0%,0.64:0.0%,0.62:0.0%,0.6:0.0%,0.53:0.0%,0.81:0.0%,0.88:0.0%,0.93:0.0%,0.57:0.0%,0.55:0.0%,0.58:0.0%,0.56:0.0%,0.52:0.0%,0.51:0.0%,0.66:0.0%,0.75:0.0%,0.3:0.0%,0.83:0.0%,0.71:0.0%,0.78:0.0%,0.63:0.0%,0.68:0.0%,0.86:0.0%,0.54:0.0%,0.61:0.0%,0.74:0.0%,0.72:0.0%,0.69:0.0%,0.59:0.0%,0.38:0.0%,0.84:0.0%,0.35:0.0%,0.76:0.0%,0.42:0.0%,0.82:0.0%,0.77:0.0%,0.32:0.0%,0.7:0.0%,0.4:0.0%,0.73:0.0%,0.91:0.0%,0.92:0.0%,0.87:0.0%,0.8:0.0%,0.9:0.0%,0.34:0.0%,0.89:0.0%]\n",
      "** srv_serror_rate:[0.0:82.12%,1.0:17.62%,0.03:0.03%,0.04:0.02%,0.05:0.02%,0.06:0.02%,0.02:0.02%,0.5:0.02%,0.08:0.01%,0.07:0.01%,0.25:0.01%,0.33:0.01%,0.17:0.01%,0.09:0.01%,0.1:0.01%,0.2:0.01%,0.12:0.01%,0.11:0.01%,0.14:0.01%,0.01:0.0%,0.67:0.0%,0.18:0.0%,0.92:0.0%,0.95:0.0%,0.94:0.0%,0.88:0.0%,0.19:0.0%,0.58:0.0%,0.75:0.0%,0.83:0.0%,0.76:0.0%,0.15:0.0%,0.91:0.0%,0.4:0.0%,0.85:0.0%,0.27:0.0%,0.22:0.0%,0.93:0.0%,0.16:0.0%,0.38:0.0%,0.36:0.0%,0.35:0.0%,0.45:0.0%,0.21:0.0%,0.44:0.0%,0.23:0.0%,0.51:0.0%,0.86:0.0%,0.9:0.0%,0.8:0.0%,0.37:0.0%]\n",
      "** rerror_rate:[0.0:94.12%,1.0:5.46%,0.86:0.02%,0.87:0.02%,0.92:0.02%,0.25:0.02%,0.9:0.02%,0.95:0.02%,0.5:0.02%,0.91:0.02%,0.88:0.01%,0.96:0.01%,0.33:0.01%,0.2:0.01%,0.93:0.01%,0.94:0.01%,0.01:0.01%,0.89:0.01%,0.85:0.01%,0.99:0.01%,0.82:0.01%,0.77:0.01%,0.17:0.01%,0.97:0.01%,0.02:0.01%,0.98:0.01%,0.03:0.01%,0.78:0.01%,0.8:0.01%,0.76:0.01%,0.79:0.0%,0.84:0.0%,0.75:0.0%,0.14:0.0%,0.05:0.0%,0.73:0.0%,0.81:0.0%,0.83:0.0%,0.71:0.0%,0.06:0.0%,0.67:0.0%,0.56:0.0%,0.08:0.0%,0.04:0.0%,0.1:0.0%,0.12:0.0%,0.09:0.0%,0.07:0.0%,0.11:0.0%,0.69:0.0%,0.4:0.0%,0.64:0.0%,0.7:0.0%,0.72:0.0%,0.74:0.0%,0.6:0.0%,0.29:0.0%,0.62:0.0%,0.65:0.0%,0.21:0.0%,0.22:0.0%,0.37:0.0%,0.58:0.0%,0.68:0.0%,0.19:0.0%,0.43:0.0%,0.35:0.0%,0.36:0.0%,0.23:0.0%,0.26:0.0%,0.27:0.0%,0.28:0.0%,0.66:0.0%,0.31:0.0%,0.32:0.0%,0.34:0.0%,0.24:0.0%]\n",
      "** srv_rerror_rate:[0.0:93.99%,1.0:5.69%,0.33:0.05%,0.5:0.04%,0.25:0.04%,0.2:0.03%,0.17:0.03%,0.14:0.01%,0.04:0.01%,0.03:0.01%,0.12:0.01%,0.06:0.01%,0.02:0.01%,0.05:0.01%,0.07:0.01%,0.4:0.01%,0.67:0.01%,0.08:0.01%,0.11:0.01%,0.29:0.01%,0.09:0.0%,0.1:0.0%,0.75:0.0%,0.6:0.0%,0.01:0.0%,0.71:0.0%,0.22:0.0%,0.83:0.0%,0.86:0.0%,0.18:0.0%,0.96:0.0%,0.79:0.0%,0.43:0.0%,0.92:0.0%,0.81:0.0%,0.88:0.0%,0.73:0.0%,0.69:0.0%,0.94:0.0%,0.62:0.0%,0.8:0.0%,0.85:0.0%,0.93:0.0%,0.82:0.0%,0.27:0.0%,0.37:0.0%,0.21:0.0%,0.38:0.0%,0.87:0.0%,0.95:0.0%,0.13:0.0%]\n",
      "** same_srv_rate:[1.0:77.34%,0.06:2.27%,0.05:2.14%,0.04:2.06%,0.07:2.03%,0.03:1.93%,0.02:1.9%,0.01:1.77%,0.08:1.48%,0.09:1.01%,0.1:0.8%,0.0:0.73%,0.12:0.73%,0.11:0.67%,0.13:0.66%,0.14:0.51%,0.15:0.35%,0.5:0.29%,0.16:0.25%,0.17:0.17%,0.33:0.12%,0.18:0.1%,0.2:0.08%,0.19:0.07%,0.67:0.05%,0.25:0.04%,0.21:0.04%,0.99:0.03%,0.22:0.03%,0.24:0.02%,0.23:0.02%,0.4:0.02%,0.98:0.02%,0.75:0.02%,0.27:0.02%,0.26:0.01%,0.8:0.01%,0.29:0.01%,0.38:0.01%,0.86:0.01%,0.3:0.01%,0.31:0.01%,0.44:0.01%,0.36:0.01%,0.83:0.01%,0.28:0.01%,0.43:0.01%,0.42:0.01%,0.6:0.01%,0.97:0.01%,0.32:0.01%,0.35:0.01%,0.45:0.01%,0.47:0.01%,0.88:0.0%,0.48:0.0%,0.39:0.0%,0.46:0.0%,0.52:0.0%,0.37:0.0%,0.41:0.0%,0.89:0.0%,0.34:0.0%,0.92:0.0%,0.54:0.0%,0.53:0.0%,0.95:0.0%,0.94:0.0%,0.57:0.0%,0.56:0.0%,0.96:0.0%,0.64:0.0%,0.71:0.0%,0.62:0.0%,0.78:0.0%,0.9:0.0%,0.49:0.0%,0.55:0.0%,0.91:0.0%,0.65:0.0%,0.73:0.0%,0.58:0.0%,0.93:0.0%,0.59:0.0%,0.82:0.0%,0.51:0.0%,0.81:0.0%,0.76:0.0%,0.77:0.0%,0.79:0.0%,0.74:0.0%,0.85:0.0%,0.72:0.0%,0.7:0.0%,0.68:0.0%,0.69:0.0%,0.87:0.0%,0.63:0.0%,0.61:0.0%]\n",
      "** diff_srv_rate:[0.0:77.33%,0.06:10.69%,0.07:5.83%,0.05:3.89%,0.08:0.66%,1.0:0.48%,0.04:0.19%,0.67:0.13%,0.5:0.12%,0.09:0.08%,0.6:0.06%,0.12:0.05%,0.1:0.04%,0.11:0.04%,0.14:0.03%,0.4:0.02%,0.13:0.02%,0.29:0.02%,0.01:0.02%,0.15:0.02%,0.03:0.02%,0.33:0.02%,0.25:0.02%,0.17:0.02%,0.75:0.01%,0.2:0.01%,0.18:0.01%,0.16:0.01%,0.19:0.01%,0.02:0.01%,0.22:0.01%,0.21:0.01%,0.27:0.01%,0.96:0.01%,0.31:0.01%,0.38:0.01%,0.24:0.01%,0.23:0.01%,0.43:0.0%,0.52:0.0%,0.44:0.0%,0.95:0.0%,0.36:0.0%,0.8:0.0%,0.53:0.0%,0.57:0.0%,0.42:0.0%,0.3:0.0%,0.26:0.0%,0.28:0.0%,0.56:0.0%,0.99:0.0%,0.54:0.0%,0.62:0.0%,0.37:0.0%,0.41:0.0%,0.35:0.0%,0.55:0.0%,0.47:0.0%,0.32:0.0%,0.46:0.0%,0.39:0.0%,0.58:0.0%,0.71:0.0%,0.89:0.0%,0.51:0.0%,0.45:0.0%,0.97:0.0%,0.73:0.0%,0.69:0.0%,0.78:0.0%,0.7:0.0%,0.74:0.0%,0.82:0.0%,0.86:0.0%,0.64:0.0%,0.83:0.0%,0.88:0.0%]\n",
      "** srv_diff_host_rate:[0.0:92.99%,1.0:1.64%,0.12:0.31%,0.5:0.29%,0.67:0.29%,0.33:0.25%,0.11:0.24%,0.25:0.23%,0.1:0.22%,0.14:0.21%,0.17:0.21%,0.08:0.2%,0.15:0.2%,0.18:0.19%,0.2:0.19%,0.09:0.19%,0.4:0.19%,0.07:0.17%,0.29:0.17%,0.13:0.16%,0.22:0.16%,0.06:0.14%,0.02:0.1%,0.05:0.1%,0.01:0.08%,0.21:0.08%,0.19:0.08%,0.16:0.07%,0.75:0.07%,0.27:0.06%,0.04:0.06%,0.6:0.06%,0.3:0.06%,0.38:0.05%,0.43:0.05%,0.23:0.05%,0.03:0.03%,0.24:0.02%,0.36:0.02%,0.31:0.02%,0.8:0.02%,0.57:0.01%,0.44:0.01%,0.28:0.01%,0.26:0.01%,0.42:0.0%,0.45:0.0%,0.62:0.0%,0.83:0.0%,0.71:0.0%,0.56:0.0%,0.35:0.0%,0.32:0.0%,0.37:0.0%,0.47:0.0%,0.41:0.0%,0.86:0.0%,0.55:0.0%,0.64:0.0%,0.54:0.0%,0.46:0.0%,0.88:0.0%,0.7:0.0%,0.77:0.0%]\n",
      "** dst_host_count:256 (0%)\n",
      "** dst_host_srv_count:256 (0%)\n",
      "** dst_host_same_srv_rate:101 (0%)\n",
      "** dst_host_diff_srv_rate:101 (0%)\n",
      "** dst_host_same_src_port_rate:101 (0%)\n",
      "** dst_host_srv_diff_host_rate:[0.0:89.45%,0.02:2.38%,0.01:2.13%,0.04:1.35%,0.03:1.34%,0.05:0.94%,0.06:0.39%,0.07:0.31%,0.5:0.15%,0.08:0.14%,0.09:0.13%,0.15:0.09%,0.11:0.09%,0.16:0.08%,0.13:0.08%,0.1:0.08%,0.14:0.07%,1.0:0.07%,0.17:0.07%,0.2:0.07%,0.12:0.07%,0.18:0.07%,0.25:0.05%,0.22:0.05%,0.19:0.05%,0.21:0.05%,0.24:0.03%,0.23:0.02%,0.26:0.02%,0.27:0.02%,0.33:0.02%,0.29:0.02%,0.51:0.02%,0.4:0.01%,0.28:0.01%,0.3:0.01%,0.67:0.01%,0.52:0.01%,0.31:0.01%,0.32:0.01%,0.38:0.01%,0.53:0.0%,0.43:0.0%,0.44:0.0%,0.34:0.0%,0.6:0.0%,0.36:0.0%,0.57:0.0%,0.35:0.0%,0.54:0.0%,0.37:0.0%,0.56:0.0%,0.55:0.0%,0.42:0.0%,0.46:0.0%,0.39:0.0%,0.45:0.0%,0.41:0.0%,0.48:0.0%,0.62:0.0%,0.8:0.0%,0.58:0.0%,0.75:0.0%,0.7:0.0%,0.47:0.0%]\n",
      "** dst_host_serror_rate:[0.0:80.93%,1.0:17.56%,0.01:0.74%,0.02:0.2%,0.03:0.09%,0.09:0.05%,0.04:0.04%,0.05:0.04%,0.07:0.03%,0.08:0.03%,0.06:0.02%,0.14:0.02%,0.15:0.02%,0.11:0.02%,0.13:0.02%,0.16:0.02%,0.1:0.02%,0.12:0.01%,0.18:0.01%,0.25:0.01%,0.2:0.01%,0.17:0.01%,0.33:0.01%,0.99:0.01%,0.19:0.01%,0.31:0.01%,0.27:0.01%,0.5:0.0%,0.22:0.0%,0.98:0.0%,0.35:0.0%,0.28:0.0%,0.24:0.0%,0.53:0.0%,0.96:0.0%,0.3:0.0%,0.94:0.0%,0.29:0.0%,0.26:0.0%,0.97:0.0%,0.42:0.0%,0.32:0.0%,0.6:0.0%,0.95:0.0%,0.56:0.0%,0.55:0.0%,0.23:0.0%,0.85:0.0%,0.93:0.0%,0.34:0.0%,0.89:0.0%,0.58:0.0%,0.21:0.0%,0.92:0.0%,0.57:0.0%,0.91:0.0%,0.9:0.0%,0.43:0.0%,0.82:0.0%,0.49:0.0%,0.36:0.0%,0.76:0.0%,0.47:0.0%,0.46:0.0%,0.62:0.0%,0.38:0.0%,0.45:0.0%,0.87:0.0%,0.61:0.0%,0.65:0.0%,0.41:0.0%,0.39:0.0%,0.44:0.0%,0.48:0.0%,0.52:0.0%,0.81:0.0%,0.77:0.0%,0.79:0.0%,0.73:0.0%,0.88:0.0%,0.69:0.0%,0.67:0.0%,0.54:0.0%,0.72:0.0%,0.68:0.0%,0.4:0.0%,0.64:0.0%,0.51:0.0%,0.84:0.0%,0.59:0.0%,0.7:0.0%,0.75:0.0%,0.8:0.0%,0.71:0.0%,0.83:0.0%,0.66:0.0%,0.74:0.0%,0.78:0.0%,0.86:0.0%,0.37:0.0%]\n",
      "** dst_host_srv_serror_rate:[0.0:81.16%,1.0:17.61%,0.01:0.99%,0.02:0.14%,0.03:0.03%,0.04:0.02%,0.05:0.01%,0.06:0.01%,0.08:0.0%,0.5:0.0%,0.07:0.0%,0.1:0.0%,0.09:0.0%,0.11:0.0%,0.17:0.0%,0.96:0.0%,0.33:0.0%,0.14:0.0%,0.12:0.0%,0.67:0.0%,0.97:0.0%,0.25:0.0%,0.98:0.0%,0.48:0.0%,0.75:0.0%,0.83:0.0%,0.4:0.0%,0.69:0.0%,0.8:0.0%,0.2:0.0%,0.91:0.0%,0.93:0.0%,0.78:0.0%,0.95:0.0%,0.16:0.0%,0.57:0.0%,0.94:0.0%,0.31:0.0%,0.92:0.0%,0.62:0.0%,0.88:0.0%,0.63:0.0%,0.29:0.0%,0.56:0.0%,0.3:0.0%,0.38:0.0%,0.32:0.0%,0.85:0.0%,0.68:0.0%,0.23:0.0%,0.15:0.0%,0.47:0.0%,0.52:0.0%,0.6:0.0%,0.24:0.0%,0.79:0.0%,0.74:0.0%,0.82:0.0%,0.64:0.0%,0.18:0.0%,0.13:0.0%,0.45:0.0%,0.66:0.0%,0.9:0.0%,0.42:0.0%,0.46:0.0%,0.86:0.0%,0.87:0.0%,0.84:0.0%,0.55:0.0%,0.81:0.0%,0.53:0.0%]\n",
      "** dst_host_rerror_rate:101 (0%)\n",
      "** dst_host_srv_rerror_rate:101 (0%)\n",
      "** outcome:[smurf.:56.84%,neptune.:21.7%,normal.:19.69%,back.:0.45%,satan.:0.32%,ipsweep.:0.25%,portsweep.:0.21%,warezclient.:0.21%,teardrop.:0.2%,pod.:0.05%,nmap.:0.05%,guess_passwd.:0.01%,buffer_overflow.:0.01%,land.:0.0%,warezmaster.:0.0%,imap.:0.0%,rootkit.:0.0%,loadmodule.:0.0%,ftp_write.:0.0%,multihop.:0.0%,phf.:0.0%,perl.:0.0%,spy.:0.0%]\n"
     ]
    }
   ],
   "source": [
    "# Analyze KDD-99\n",
    "analyze(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aSWky8HQFl6P"
   },
   "source": [
    "## Encode the feature vector\n",
    "\n",
    "We use the same two functions provided earlier to preprocess the data. The first encodes Z-Scores, and the second creates dummy variables from categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "HjZvCwoKFl6Q"
   },
   "outputs": [],
   "source": [
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "    \n",
    "# Encode text values to dummy variables(i.e. [1,0,0],\n",
    "# [0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gbmM2bqFFl6Q"
   },
   "source": [
    "Again, just as we did for anomaly detection, we preprocess the data set.  We convert all numeric values to Z-Score and translate all categorical to dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "-m4DK8DfFl6Q"
   },
   "outputs": [],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "for name in df.columns:\n",
    "  if name == 'outcome':\n",
    "    pass\n",
    "  elif name in ['protocol_type','service','flag','land','logged_in',\n",
    "                'is_host_login','is_guest_login']:\n",
    "    encode_text_dummy(df,name)\n",
    "  else:\n",
    "    encode_numeric_zscore(df,name)    \n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]\n",
    "\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('outcome')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['outcome']) # Classification\n",
    "outcomes = dummies.columns\n",
    "num_classes = len(outcomes)\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KMANNuZBFl6Q"
   },
   "source": [
    "We will attempt to predict what type of attack is underway.  The outcome column specifies the attack type.  A value of normal indicates that there is no attack underway.  We display the outcomes; some attack types are much rarer than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SUVJiVF-Fl6R",
    "outputId": "07b7e52c-57bf-437a-b2f0-cc6f34c3f54b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "back.               2203\n",
       "buffer_overflow.      30\n",
       "                    ... \n",
       "warezclient.        1020\n",
       "warezmaster.          20\n",
       "Name: outcome, Length: 23, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('outcome')['outcome'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8NFnJKhjFl6R"
   },
   "source": [
    "## Train the Neural Network\n",
    "\n",
    "We now train the neural network to classify the different KDD99 outcomes.  The code provided here implements a relatively simple neural with two hidden layers.  We train it with the provided KDD99 data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HYRkjqbcFl6R",
    "outputId": "7a3b64b1-8191-4249-f100-524ec1f6cbc2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000\n",
      "11579/11579 - 25s - loss: 0.1851 - val_loss: 0.0362 - 25s/epoch - 2ms/step\n",
      "Epoch 2/1000\n",
      "11579/11579 - 23s - loss: 0.0360 - val_loss: 0.0275 - 23s/epoch - 2ms/step\n",
      "Epoch 3/1000\n",
      "11579/11579 - 22s - loss: 0.0271 - val_loss: 0.0270 - 22s/epoch - 2ms/step\n",
      "Epoch 4/1000\n",
      "11579/11579 - 22s - loss: 0.0244 - val_loss: 0.0221 - 22s/epoch - 2ms/step\n",
      "Epoch 5/1000\n",
      "11579/11579 - 23s - loss: 0.0235 - val_loss: 0.0228 - 23s/epoch - 2ms/step\n",
      "Epoch 6/1000\n",
      "11579/11579 - 22s - loss: 0.1370 - val_loss: 0.0210 - 22s/epoch - 2ms/step\n",
      "Epoch 7/1000\n",
      "11579/11579 - 22s - loss: 0.0859 - val_loss: 0.0218 - 22s/epoch - 2ms/step\n",
      "Epoch 8/1000\n",
      "11579/11579 - 22s - loss: 0.0507 - val_loss: 0.0192 - 22s/epoch - 2ms/step\n",
      "Epoch 9/1000\n",
      "11579/11579 - 33s - loss: 0.0203 - val_loss: 0.0185 - 33s/epoch - 3ms/step\n",
      "Epoch 10/1000\n",
      "11579/11579 - 21s - loss: 0.0180 - val_loss: 0.0185 - 21s/epoch - 2ms/step\n",
      "Epoch 11/1000\n",
      "11579/11579 - 22s - loss: 0.0180 - val_loss: 0.0208 - 22s/epoch - 2ms/step\n",
      "Epoch 12/1000\n",
      "11579/11579 - 22s - loss: 0.0166 - val_loss: 0.0165 - 22s/epoch - 2ms/step\n",
      "Epoch 13/1000\n",
      "11579/11579 - 23s - loss: 0.0165 - val_loss: 0.0166 - 23s/epoch - 2ms/step\n",
      "Epoch 14/1000\n",
      "11579/11579 - 24s - loss: 0.0158 - val_loss: 0.0152 - 24s/epoch - 2ms/step\n",
      "Epoch 15/1000\n",
      "11579/11579 - 21s - loss: 0.0152 - val_loss: 0.0182 - 21s/epoch - 2ms/step\n",
      "Epoch 16/1000\n",
      "11579/11579 - 21s - loss: 0.0160 - val_loss: 0.0149 - 21s/epoch - 2ms/step\n",
      "Epoch 17/1000\n",
      "11579/11579 - 22s - loss: 0.0142 - val_loss: 0.0143 - 22s/epoch - 2ms/step\n",
      "Epoch 18/1000\n",
      "11579/11579 - 22s - loss: 0.0139 - val_loss: 0.0153 - 22s/epoch - 2ms/step\n",
      "Epoch 19/1000\n",
      "Restoring model weights from the end of the best epoch: 14.\n",
      "11579/11579 - 23s - loss: 0.0141 - val_loss: 0.0152 - 23s/epoch - 2ms/step\n",
      "Epoch 19: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3974599a50>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "# Create a test/train split.  25% test\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create neural net\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(10, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(1, kernel_initializer='normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "                        patience=5, verbose=1, mode='auto',\n",
    "                           restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=2,epochs=1000)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fZbIjoeMFl6R"
   },
   "source": [
    "We can now evaluate the neural network.  As you can see, the neural network achieves a 99% accuracy rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-RgMZeFlFl6R",
    "outputId": "9bd0141c-a98e-46d6-e9b1-10d4e9e4c460"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation score: 0.9977005165740935\n"
     ]
    }
   ],
   "source": [
    "# Measure accuracy\n",
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "y_eval = np.argmax(y_test,axis=1)\n",
    "score = metrics.accuracy_score(y_eval, pred)\n",
    "print(\"Validation score: {}\".format(score))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_14_04_ids_kdd99.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}