t81_558_class_13_2_anomaly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UhLdicqoAMGl"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_13_2_anomaly.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BGssM4-jAMGn"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64qCM6NsAMGn"
   },
   "source": [
    "# Module 13 Video Material\n",
    "\n",
    "* Part 13.1: Using Denoising AutoEncoders [[Video]](https://www.youtube.com/watch?v=BBrRD89sTk8&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_13_1_auto_encode.ipynb)\n",
    "* **Part 13.2: Anomaly Detection** [[Video]](https://www.youtube.com/watch?v=wubZ516TkI8&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_13_2_anomaly.ipynb)\n",
    "* Part 13.3: Model Drift and Retraining [[Video]](https://www.youtube.com/watch?v=F4395B1ySpg&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_13_3_retrain.ipynb)\n",
    "* Part 13.4: Tensor Processing Units (TPUs) [[Video]](https://www.youtube.com/watch?v=Cp3xOyxOZNo&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_13_4_tpu.ipynb)\n",
    "* Part 13.5: Future Directions in Artificial Intelligence [[Video]](https://www.youtube.com/watch?v=RjxvEZh73Yc&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_13_5_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code checks that Google CoLab is and sets up the correct hardware settings for PyTorch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k9pmbHghAMGo"
   },
   "source": [
    "# Part 13.2: Anomaly Detection\n",
    "\n",
    "Anomaly detection is an unsupervised training technique that analyzes the degree to which incoming data differs from the data you used to train the neural network. Traditionally, cybersecurity experts have used anomaly detection to ensure network security. However, you can use anomalies in data science to detect input for which you have not trained your neural network.  \n",
    "\n",
    "There are several data sets that many commonly use to demonstrate anomaly detection. In this part, we will look at the KDD-99 dataset.\n",
    "\n",
    "\n",
    "* [Stratosphere IPS Dataset](https://www.stratosphereips.org/category/dataset.html)\n",
    "* [The ADFA Intrusion Detection Datasets (2013) - for HIDS](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-IDS-Datasets/)\n",
    "* [ITOC CDX (2009)](https://westpoint.edu/centers-and-research/cyber-research-center/data-sets)\n",
    "* [KDD-99 Dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html)\n",
    "\n",
    "## Read in KDD99 Data Set\n",
    "\n",
    "Although the KDD99 dataset is over 20 years old, it is still widely used to demonstrate Intrusion Detection Systems (IDS) and Anomaly detection. KDD99 is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, held in conjunction with KDD-99, The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between \"bad\" connections, called intrusions or attacks, and \"good\" normal connections. This database contains a standard set of data to be audited, including various intrusions simulated in a military network environment.\n",
    "\n",
    "The following code reads the KDD99 CSV dataset into a Pandas data frame. The standard format of KDD99 does not include column names. Because of that, the program adds them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 325
    },
    "id": "Z7Nc6xKjAMGo",
    "outputId": "854a3977-8e15-4d42-a9be-917ea70cb108"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kdd-with-columns.csv\n",
      "Read 494021 rows.\n",
      "        duration protocol_type  ... dst_host_srv_rerror_rate  outcome\n",
      "0              0           tcp  ...                      0.0  normal.\n",
      "1              0           tcp  ...                      0.0  normal.\n",
      "...          ...           ...  ...                      ...      ...\n",
      "494019         0           tcp  ...                      0.0  normal.\n",
      "494020         0           tcp  ...                      0.0  normal.\n",
      "\n",
      "[494021 rows x 42 columns]\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import urllib.request\n",
    "import os\n",
    "\n",
    "# Set Pandas display options\n",
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "# Download the file using urllib\n",
    "url = 'https://github.com/jeffheaton/jheaton-ds2/raw/main/kdd-with-columns.csv'\n",
    "filename = 'kdd-with-columns.csv'\n",
    "\n",
    "if not os.path.isfile(filename):\n",
    "    try:\n",
    "        urllib.request.urlretrieve(url, filename)\n",
    "    except:\n",
    "        print('Error downloading')\n",
    "        raise\n",
    "\n",
    "print(filename)\n",
    "\n",
    "# Original file: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(filename)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to sample only 10% of the dataset\n",
    "df.dropna(inplace=True, axis=1) \n",
    "# For now, just drop NA's (rows with missing values)\n",
    "\n",
    "# Display 5 rows\n",
    "pd.set_option('display.max_columns', 5)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qiiHziMXAMGp"
   },
   "source": [
    "The KDD99 dataset contains many columns that define the network state over time intervals during which a cyber attack might have taken place.  The \" outcome \" column specifies either \"normal,\" indicating no attack, or the type of attack performed.  The following code displays the counts for each type of attack and \"normal\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "cxgbeLBpAMGq",
    "outputId": "11827d16-bbaf-4edc-c5fb-d25c787aac07"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "back.               2203\n",
       "buffer_overflow.      30\n",
       "                    ... \n",
       "warezclient.        1020\n",
       "warezmaster.          20\n",
       "Name: outcome, Length: 23, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('outcome')['outcome'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Y1YqytVAMGq"
   },
   "source": [
    "## Preprocessing \n",
    "\n",
    "We must perform some preprocessing before we can feed the KDD99 data into the neural network. We provide the following two functions to assist with preprocessing. The first function converts numeric columns into Z-Scores. The second function replaces categorical values with dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "xRR6q2kOAMGq"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def encode_numeric_zscore(df, name):\n",
    "    \"\"\"\n",
    "    Apply z-score normalization to a specified numeric column.\n",
    "\n",
    "    Parameters:\n",
    "    df (DataFrame): The pandas DataFrame containing the column.\n",
    "    name (str): The name of the column to normalize.\n",
    "    \"\"\"\n",
    "    mean = df[name].mean()\n",
    "    sd = df[name].std()\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "def encode_text_dummy(df, name):\n",
    "    \"\"\"\n",
    "    Convert a categorical column to dummy variables.\n",
    "\n",
    "    Parameters:\n",
    "    df (DataFrame): The pandas DataFrame containing the column.\n",
    "    name (str): The name of the categorical column.\n",
    "    \"\"\"\n",
    "    dummies = pd.get_dummies(df[name], prefix=name, dtype=float)\n",
    "    df = pd.concat([df, dummies], axis=1)\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "    return df\n",
    "\n",
    "def process_dataframe(df):\n",
    "    \"\"\"\n",
    "    Process a DataFrame by encoding its features.\n",
    "\n",
    "    Parameters:\n",
    "    df (DataFrame): The pandas DataFrame to process.\n",
    "    \"\"\"\n",
    "    for name in df.columns:\n",
    "        if name == 'outcome':\n",
    "            continue\n",
    "        #elif df[name].dtype == bool:\n",
    "        #    print(\"**\", name)\n",
    "        #    df[name] = df[name].astype(float)\n",
    "        elif name in ['protocol_type', 'service', 'flag', 'land', 'logged_in',\n",
    "                      'is_host_login', 'is_guest_login']:\n",
    "            df = encode_text_dummy(df, name)\n",
    "        else:\n",
    "            encode_numeric_zscore(df, name)\n",
    "    return df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DuEFJKt_AMGr"
   },
   "source": [
    "This code converts all numeric columns to Z-Scores and all textual columns to dummy variables. We now use these functions to preprocess each of the columns. Once the program preprocesses the data, we display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "id": "stntDxM9AMGr",
    "outputId": "ac43723d-a105-4105-fe1a-d45153e3294f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   duration  src_bytes  dst_bytes  ...  is_host_login_0  is_guest_login_0  \\\n",
      "0 -0.067792  -0.002879   0.138664  ...              1.0               1.0   \n",
      "1 -0.067792  -0.002820  -0.011578  ...              1.0               1.0   \n",
      "2 -0.067792  -0.002824   0.014179  ...              1.0               1.0   \n",
      "3 -0.067792  -0.002840   0.014179  ...              1.0               1.0   \n",
      "4 -0.067792  -0.002842   0.035214  ...              1.0               1.0   \n",
      "\n",
      "   is_guest_login_1  \n",
      "0               0.0  \n",
      "1               0.0  \n",
      "2               0.0  \n",
      "3               0.0  \n",
      "4               0.0  \n",
      "\n",
      "[5 rows x 121 columns]\n"
     ]
    }
   ],
   "source": [
    "pd.set_option('display.max_columns', 6)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "df = process_dataframe(df)\n",
    "df.dropna(inplace=True, axis=1)\n",
    "print(df.head())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fyHlHDwGAMGr"
   },
   "source": [
    "We divide the data into two groups, \"normal\" and the various attacks to perform anomaly detection. The following code divides the data into two data frames and displays each of these two groups' sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vZhVyry_AMGr",
    "outputId": "4809f4ec-49d3-419d-a82b-25dd9c52f3c9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal count: 97278\n",
      "Attack count: 396743\n"
     ]
    }
   ],
   "source": [
    "normal_mask = df['outcome']=='normal.'\n",
    "attack_mask = df['outcome']!='normal.'\n",
    "\n",
    "df.drop('outcome',axis=1,inplace=True)\n",
    "\n",
    "df_normal = df[normal_mask]\n",
    "df_attack = df[attack_mask]\n",
    "\n",
    "print(f\"Normal count: {len(df_normal)}\")\n",
    "print(f\"Attack count: {len(df_attack)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QnihkQx_AMGr"
   },
   "source": [
    "Next, we convert these two data frames into Numpy arrays. Keras requires this format for data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "Vr_C7KEUAMGs"
   },
   "outputs": [],
   "source": [
    "# This is the numeric feature vector, as it goes to the neural net\n",
    "x_normal = df_normal.values\n",
    "x_attack = df_attack.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YH_QLaPvAMGs"
   },
   "source": [
    "## Training the Autoencoder\n",
    "\n",
    "It is important to note that we are not using the outcome column as a label to predict. We will train an autoencoder on the normal data and see how well it can detect that the data not flagged as \"normal\" represents an anomaly. This anomaly detection is unsupervised; there is no target (y) value to predict. \n",
    "\n",
    "Next, we split the normal data into a 25% test set and a 75% train set. The program will use the test data to facilitate early stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "MX6PCz-RAMGs"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_normal_train, x_normal_test = train_test_split(\n",
    "    x_normal, test_size=0.25, random_state=42)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/10], Loss: 0.32732871251074563\n",
      "Epoch [2/10], Loss: 0.2666971355484668\n",
      "Epoch [3/10], Loss: 0.25296189310355927\n",
      "Epoch [4/10], Loss: 0.2429972947542474\n",
      "Epoch [5/10], Loss: 0.22758804882525285\n",
      "Epoch [6/10], Loss: 0.2041895669215081\n",
      "Epoch [7/10], Loss: 0.18810490698149232\n",
      "Epoch [8/10], Loss: 0.1733971669941144\n",
      "Epoch [9/10], Loss: 0.16838515430255874\n",
      "Epoch [10/10], Loss: 0.1590980681635668\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# Convert numpy arrays to PyTorch tensors and move them to the appropriate device\n",
    "x_normal_train_tensor = torch.tensor(x_normal_train).float().to(device)\n",
    "x_normal_tensor = torch.tensor(x_normal).float().to(device)\n",
    "x_attack_tensor = torch.tensor(x_attack).float().to(device)\n",
    "\n",
    "# Create DataLoader for batch processing\n",
    "train_data = TensorDataset(x_normal_train_tensor, x_normal_train_tensor)\n",
    "train_loader = DataLoader(train_data, batch_size=32, shuffle=True)\n",
    "\n",
    "# Define the model using Sequential\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(x_normal.shape[1], 25),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(25, 3),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(3, 25),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(25, x_normal.shape[1])\n",
    ").to(device)\n",
    "\n",
    "# Loss and optimizer\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "num_epochs = 10\n",
    "# Training loop\n",
    "for epoch in range(num_epochs):\n",
    "    running_loss = 0\n",
    "    den = 0\n",
    "    for data in train_loader:\n",
    "        inputs, targets = data\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs, targets)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss +=loss.item()\n",
    "        den+=1\n",
    "\n",
    "    print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {running_loss/den}')\n",
    "    running_loss = 0.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ozvx3sr5AMGs"
   },
   "source": [
    "We display the size of the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CuH9Bxg6AMGs",
    "outputId": "1e8aa872-3c95-4299-aa30-896577f2db84"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal train count: 72958\n",
      "Normal test count: 24320\n"
     ]
    }
   ],
   "source": [
    "print(f\"Normal train count: {len(x_normal_train)}\")\n",
    "print(f\"Normal test count: {len(x_normal_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06DNCbB4AMGs"
   },
   "source": [
    "We are now ready to train the autoencoder on the normal data. The autoencoder will learn to compress the data to a vector of just three numbers. The autoencoder should be able to also decompress with reasonable accuracy. As is typical for autoencoders, we are merely training the neural network to produce the same output values as were fed to the input layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mw9KbY1LAMGs"
   },
   "source": [
    "## Detecting an Anomaly\n",
    "\n",
    "We are now ready to see if the abnormal data is an anomaly. The first two scores show the in-sample and out of sample RMSE errors. These two scores are relatively low at around 0.33 because they resulted from normal data. The much higher 0.76 error occurred from the abnormal data. The autoencoder is not as capable of encoding data that represents an attack. This higher error indicates an anomaly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out of Sample Normal Score (RMSE): 0.391261488199234\n",
      "Insample Normal Score (RMSE): 0.3845501244068146\n",
      "Attack Underway Score (RMSE): 0.5126159191131592\n"
     ]
    }
   ],
   "source": [
    "model.eval()  # Set the model to evaluation mode\n",
    "\n",
    "# Function to calculate RMSE\n",
    "def calculate_rmse(model, data):\n",
    "    with torch.no_grad():\n",
    "        predictions = model(data)\n",
    "        mse_loss = nn.MSELoss()(predictions, data)\n",
    "    return torch.sqrt(mse_loss).item()\n",
    "\n",
    "# Evaluating the model\n",
    "score1 = calculate_rmse(model, torch.tensor(x_normal_test).float().to(device))\n",
    "score2 = calculate_rmse(model, x_normal_tensor)\n",
    "score3 = calculate_rmse(model, x_attack_tensor)\n",
    "\n",
    "print(f\"Out of Sample Normal Score (RMSE): {score1}\")\n",
    "print(f\"Insample Normal Score (RMSE): {score2}\")\n",
    "print(f\"Attack Underway Score (RMSE): {score3}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_14_03_anomaly.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}