AutoML-based_IDS_Batch_Learning_Dataset_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Study: Automated Machine Learning (AutoML) for Autonomous Intrusion Detection System Development \n",
    "This is the code for the paper entitled \"**[Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis](https://ieeexplore.ieee.org/document/10472316)**\" published in *IEEE Transactions on Network and Service Management* (IF:5.3).<br>\n",
    "Authors: Li Yang (liyanghart@gmail.com), Mirna El Rajab, Abdallah Shami, and Sami Muhaidat<br>\n",
    "\n",
    "L. Yang, M. E. Rajab, A. Shami, and S. Muhaidat, \"Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis,\" IEEE Transactions on Network and Service Management, pp. 1-28, 2024, doi: https://doi.org/10.1109/TNSM.2024.3376631."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Part 1: Automated Offline/Static/Batch Learning\n",
    "Batch learning: Batch learning methods analyze static data in batches and often need access to the entire dataset prior to model training. Traditional ML algorithms can effectively solve batch learning tasks. Although batch learning models often achieve high performance due to their ability to learn diverse data patterns, it is often difficult to update these models once created. Therefore, batch learning faces two significant challenges: model degradation and data unavailability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 2: 5G-NIDD\n",
    "A subset of the network traffic data randomly sampled from the [5G-NIDD dataset](https://ieee-dataport.org/documents/5g-nidd-comprehensive-network-intrusion-detection-dataset-generated-over-5g-wireless).  \n",
    "\n",
    "The 5G-NIDD dataset, created in December 2022, is a fully labeled resource constructed on a functional 5G test network for researchers and practitioners evaluating AI/ML solutions in the context of 5G/6G security [87]. 5G-NIDD encompasses data extracted from a 5G testbed connected to the 5G Test Network (5GTN) at the University of Oulu, Finland. The dataset is derived from two base stations, each featuring an attacker node and multiple benign 5G users. The attacker nodes target a server deployed within the 5GTN MEC environment. The attack scenarios captured in the dataset primarily include DoS attacks and port scans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.feature_selection import RFE\n",
    "from scipy.stats import shapiro\n",
    "from imblearn.over_sampling import SMOTE, ADASYN\n",
    "from hyperopt import fmin, tpe, hp\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled CICIDS2017 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Data/5gnidd_0.01_pre-processed.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>Dur</th>\n",
       "      <th>RunTime</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Sum</th>\n",
       "      <th>Min</th>\n",
       "      <th>Max</th>\n",
       "      <th>Proto</th>\n",
       "      <th>sTos</th>\n",
       "      <th>dTos</th>\n",
       "      <th>...</th>\n",
       "      <th>SrcWin</th>\n",
       "      <th>DstWin</th>\n",
       "      <th>sVid</th>\n",
       "      <th>dVid</th>\n",
       "      <th>SrcTCPBase</th>\n",
       "      <th>DstTCPBase</th>\n",
       "      <th>TcpRtt</th>\n",
       "      <th>SynAck</th>\n",
       "      <th>AckDat</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000496</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002442</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003295</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003317</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003375</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388669</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12154</th>\n",
       "      <td>0.003688</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12155</th>\n",
       "      <td>0.004104</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.00025</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.292755</td>\n",
       "      <td>0.030151</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12156</th>\n",
       "      <td>0.005234</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12157</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.830357</td>\n",
       "      <td>0.215054</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12158</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.830357</td>\n",
       "      <td>0.215054</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12159 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Seq           Dur       RunTime          Mean           Sum  \\\n",
       "0      0.000496  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "1      0.002442  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "2      0.003295  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "3      0.003317  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "4      0.003375  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "...         ...           ...           ...           ...           ...   \n",
       "12154  0.003688  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "12155  0.004104  2.000005e-07  2.000005e-07  2.000005e-07  2.000005e-07   \n",
       "12156  0.005234  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "12157  0.000015  4.079091e-02  4.079091e-02  4.079091e-02  4.079091e-02   \n",
       "12158  0.000015  4.088011e-02  4.088011e-02  4.088011e-02  4.088011e-02   \n",
       "\n",
       "                Min           Max  Proto      sTos      dTos  ...    SrcWin  \\\n",
       "0      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000002   \n",
       "1      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "2      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "3      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "4      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "...             ...           ...    ...       ...       ...  ...       ...   \n",
       "12154  0.000000e+00  0.000000e+00   1.00  0.000000  0.000000  ...  0.000000   \n",
       "12155  2.000005e-07  2.000005e-07   0.75  0.000000  0.000000  ...  0.000008   \n",
       "12156  0.000000e+00  0.000000e+00   1.00  0.000000  0.000000  ...  0.000000   \n",
       "12157  4.079091e-02  4.079091e-02   0.50  0.830357  0.215054  ...  0.000000   \n",
       "12158  4.088011e-02  4.088011e-02   0.50  0.830357  0.215054  ...  0.000000   \n",
       "\n",
       "        DstWin  sVid  dVid  SrcTCPBase  DstTCPBase  TcpRtt  SynAck  AckDat  \\\n",
       "0      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "1      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "2      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "3      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "4      0.00000   0.0   0.0    0.388669    0.000000     0.0     0.0     0.0   \n",
       "...        ...   ...   ...         ...         ...     ...     ...     ...   \n",
       "12154  0.00000   1.0   0.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12155  0.00025   0.0   0.0    0.292755    0.030151     0.0     0.0     0.0   \n",
       "12156  0.00000   1.0   0.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12157  0.00000   0.0   1.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12158  0.00000   0.0   1.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "\n",
       "       Label  \n",
       "0          1  \n",
       "1          1  \n",
       "2          1  \n",
       "3          1  \n",
       "4          1  \n",
       "...      ...  \n",
       "12154      0  \n",
       "12155      0  \n",
       "12156      0  \n",
       "12157      0  \n",
       "12158      0  \n",
       "\n",
       "[12159 rows x 49 columns]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Automated Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Transformation/Encoding\n",
    "Automatically identify and transform string/text features into numerical features by the label encoding method to make the data more readable by ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data encoding function\n",
    "def Auto_Encoding(df):\n",
    "    cat_features=[x for x in df.columns if df[x].dtype==\"object\"] ## Find string/text features\n",
    "    le=LabelEncoder()\n",
    "    for col in cat_features:\n",
    "        if col in df.columns:\n",
    "            i = df.columns.get_loc(col)\n",
    "            # Transform to numerical features\n",
    "            df.iloc[:,i] = df.apply(lambda i:le.fit_transform(i.astype(str)), axis=0, result_type='expand')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Encoding(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated normalization\n",
    "Normalize the range of features to a similar scale to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def Auto_Normalization(df):\n",
    "    stat, p = shapiro(df)\n",
    "    print('Statistics=%.3f, p=%.3f' % (stat, p))\n",
    "    # interpret\n",
    "    alpha = 0.05\n",
    "    numeric_features = df.drop(['Label'],axis = 1).dtypes[df.dtypes != 'object'].index\n",
    "    \n",
    "    # check if the data distribution follows a Gaussian/normal distribution\n",
    "    # If so, select the Z-score normalization method; otherwise, select the min-max normalization\n",
    "    # Details are in the paper\n",
    "    if p > alpha:\n",
    "        print('Sample looks Gaussian (fail to reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.mean()) / (x.std()))\n",
    "        print('Z-score normalization is automatically chosen and used')\n",
    "    else:\n",
    "        print('Sample does not look Gaussian (reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.min()) / (x.max()-x.min()))\n",
    "        print('Min-max normalization is automatically chosen and used')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistics=0.563, p=0.000\n",
      "Sample does not look Gaussian (reject H0)\n",
      "Min-max normalization is automatically chosen and used\n"
     ]
    }
   ],
   "source": [
    "df=Auto_Normalization(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Imputation\n",
    "Detect and impute missing values to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data imputation function\n",
    "def Auto_Imputation(df):\n",
    "    # Replace infinities with NaN to unify the treatment of missing data\n",
    "    df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "    \n",
    "    # Identify numeric columns\n",
    "    numeric_cols = df.select_dtypes(include=[np.number]).columns\n",
    "    \n",
    "    # Apply imputation only if necessary\n",
    "    if df[numeric_cols].isnull().values.any():\n",
    "        for col in numeric_cols:\n",
    "            # Check if the entire column is NaN\n",
    "            if df[col].isna().all():\n",
    "                # Optional: fill such columns with a placeholder value or drop them\n",
    "                # Here, we choose to fill with 0 as an example\n",
    "                df[col].fillna(0, inplace=True)\n",
    "            else:\n",
    "                # Apply median imputation\n",
    "                df[col].fillna(df[col].median(), inplace=True)\n",
    "    \n",
    "    # Final check for NaN or infinite values in numeric columns\n",
    "    if df[numeric_cols].isnull().any().any() or np.isinf(df[numeric_cols].values).any():\n",
    "        raise ValueError(\"Numeric data still contains NaN, infinity or a value too large after imputation.\")\n",
    "    \n",
    "    return df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train-test split\n",
    "Split the dataset into the training and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "\n",
    "# Here we used the 80%/20% split, it can be changed based on specific tasks\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated data balancing\n",
    "Generate minority class samples to solve class-imbalance and improve data quality.  \n",
    "Synthetic Minority Over-sampling Technique (SMOTE) method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    5927\n",
       "0    3800\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For binary data (can be modified for multi-class data with the same logic)\n",
    "def Auto_Balancing(X_train, y_train):\n",
    "    number0 = pd.Series(y_train).value_counts().iloc[0]\n",
    "    number1 = pd.Series(y_train).value_counts().iloc[1]\n",
    "    \n",
    "    if number0 > number1:\n",
    "        nlarge = number0\n",
    "    else:\n",
    "        nlarge = number1\n",
    "    \n",
    "    # evaluate whether the incoming dataset is imbalanced (the abnormal/normal ratio is smaller than a threshold (e.g., 50%)) \n",
    "    if (number1/number0 > 1.5) or (number0/number1 > 1.5):\n",
    "        balanced=ADASYN(n_jobs=-1,sampling_strategy={0:nlarge, 1:nlarge})\n",
    "\n",
    "        X_train, y_train = balanced.fit_resample(X_train, y_train)\n",
    "        \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    5927\n",
       "0    5912\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Time: 2.05038\n",
      "Wall time: 183 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.91799999999999%\n",
      "Precision: 99.933%\n",
      "Recall: 99.933%\n",
      "F1-score: 99.933%\n",
      "Time: 9.42224\n",
      "Wall time: 770 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.383%\n",
      "Precision: 99.59599999999999%\n",
      "Recall: 99.396%\n",
      "F1-score: 99.496%\n",
      "Time: 205.88003\n",
      "Wall time: 507 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = knn.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=32,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    K.clear_session()\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.507%\n",
      "Precision: 99.59700000000001%\n",
      "Recall: 99.59700000000001%\n",
      "F1-score: 99.59700000000001%\n",
      "Time: 205.88003\n",
      "Wall time: 5.01 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ann = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "ann.fit(X_train,y_train)\n",
    "predictions = ann.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Automated Feature Engineering\n",
    "Feature selection method 1: **Recursive Feature Elimination (RFE)**, used to remove irrelevant features to improve model efficiency  \n",
    "Feature selection method 2: **Pearson Correlation**, used to remove redundant features to improve model efficiency and accuracy  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Feature_Importance_RFE(data, n_features_to_select=20):\n",
    "    features = data.drop(['Label'], axis=1).values  # \"Label\" should be changed to the target class variable name if different\n",
    "    labels = data['Label'].values\n",
    "\n",
    "    # Extract feature names\n",
    "    feature_names = list(data.drop(['Label'], axis=1).columns)\n",
    "\n",
    "    # Create a base estimator\n",
    "    model = lgb.LGBMRegressor(verbose = -1)\n",
    "\n",
    "    # Create the RFE object and rank each feature\n",
    "    rfe = RFE(estimator=model, n_features_to_select=n_features_to_select, step=1)\n",
    "    rfe.fit(features, labels)\n",
    "\n",
    "    # Get the feature ranking\n",
    "    feature_ranking = rfe.ranking_\n",
    "\n",
    "    # Create a DataFrame for feature importances\n",
    "    feature_importances = pd.DataFrame({'feature': feature_names, 'ranking': feature_ranking})\n",
    "\n",
    "    # Sort features according to their ranking\n",
    "    feature_importances = feature_importances.sort_values('ranking', ascending=True).reset_index(drop=True)\n",
    "\n",
    "    # Get the features to drop\n",
    "    to_drop = list(feature_importances[feature_importances['ranking'] > 1]['feature'])\n",
    "\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove redundant features\n",
    "def Feature_Redundancy_Pearson(data):\n",
    "    correlation_threshold=0.90 # Only remove features with the redundancy>90%. It can be changed\n",
    "    features = data.drop(['Label'],axis=1)\n",
    "    corr_matrix = features.corr()\n",
    "\n",
    "    # Extract the upper triangle of the correlation matrix\n",
    "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k = 1).astype(np.bool))\n",
    "\n",
    "    # Select the features with correlations above the threshold\n",
    "    # Need to use the absolute value\n",
    "    to_drop = [column for column in upper.columns if any(upper[column].abs() > correlation_threshold)]\n",
    "\n",
    "    # Dataframe to hold correlated pairs\n",
    "    record_collinear = pd.DataFrame(columns = ['drop_feature', 'corr_feature', 'corr_value'])\n",
    "\n",
    "    # Iterate through the columns to drop\n",
    "    for column in to_drop:\n",
    "\n",
    "        # Find the correlated features\n",
    "        corr_features = list(upper.index[upper[column].abs() > correlation_threshold])\n",
    "\n",
    "        # Find the correlated values\n",
    "        corr_values = list(upper[column][upper[column].abs() > correlation_threshold])\n",
    "        drop_features = [column for _ in range(len(corr_features))]    \n",
    "\n",
    "        # Record the information (need a temp df for now)\n",
    "        temp_df = pd.DataFrame.from_dict({'drop_feature': drop_features,\n",
    "                                         'corr_feature': corr_features,\n",
    "                                         'corr_value': corr_values})\n",
    "        record_collinear = record_collinear.append(temp_df, ignore_index = True)\n",
    "#     print(record_collinear)\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Auto_Feature_Engineering(df):\n",
    "    drop1 = Feature_Importance_RFE(df)\n",
    "    dfh1 = df.drop(columns = drop1)\n",
    "    \n",
    "    drop2 = Feature_Redundancy_Pearson(dfh1)\n",
    "    dfh2 = dfh1.drop(columns = drop2)\n",
    "    \n",
    "    return dfh2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>Dur</th>\n",
       "      <th>Proto</th>\n",
       "      <th>sTtl</th>\n",
       "      <th>dTtl</th>\n",
       "      <th>sHops</th>\n",
       "      <th>TotBytes</th>\n",
       "      <th>SrcBytes</th>\n",
       "      <th>Offset</th>\n",
       "      <th>sMeanPktSz</th>\n",
       "      <th>...</th>\n",
       "      <th>Load</th>\n",
       "      <th>pLoss</th>\n",
       "      <th>Rate</th>\n",
       "      <th>SrcWin</th>\n",
       "      <th>DstWin</th>\n",
       "      <th>SrcTCPBase</th>\n",
       "      <th>TcpRtt</th>\n",
       "      <th>SynAck</th>\n",
       "      <th>AckDat</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000496</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.000201</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002442</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.145098</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.964286</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003295</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.149020</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001133</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003317</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001140</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003375</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001160</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388669</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12154</th>\n",
       "      <td>0.003688</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.976471</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.000117</td>\n",
       "      <td>0.003075</td>\n",
       "      <td>0.053852</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12155</th>\n",
       "      <td>0.004104</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.203922</td>\n",
       "      <td>0.250980</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.000158</td>\n",
       "      <td>0.000257</td>\n",
       "      <td>0.003462</td>\n",
       "      <td>0.039298</td>\n",
       "      <td>...</td>\n",
       "      <td>0.557175</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.00025</td>\n",
       "      <td>0.292755</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12156</th>\n",
       "      <td>0.005234</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.976471</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.000117</td>\n",
       "      <td>0.004298</td>\n",
       "      <td>0.053852</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12157</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.980392</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>0.000304</td>\n",
       "      <td>0.004546</td>\n",
       "      <td>0.069862</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12158</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.980392</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>0.000304</td>\n",
       "      <td>0.004757</td>\n",
       "      <td>0.069862</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12159 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Seq           Dur  Proto      sTtl      dTtl     sHops  TotBytes  \\\n",
       "0      0.000496  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "1      0.002442  0.000000e+00   0.75  0.145098  0.000000  0.964286  0.000007   \n",
       "2      0.003295  0.000000e+00   0.75  0.149020  0.000000  0.928571  0.000007   \n",
       "3      0.003317  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "4      0.003375  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "...         ...           ...    ...       ...       ...       ...       ...   \n",
       "12154  0.003688  0.000000e+00   1.00  0.976471  0.000000  0.250000  0.000014   \n",
       "12155  0.004104  2.000005e-07   0.75  0.203922  0.250980  0.428571  0.000158   \n",
       "12156  0.005234  0.000000e+00   1.00  0.976471  0.000000  0.250000  0.000014   \n",
       "12157  0.000015  4.079091e-02   0.50  1.000000  0.980392  0.035714  0.000145   \n",
       "12158  0.000015  4.088011e-02   0.50  1.000000  0.980392  0.035714  0.000145   \n",
       "\n",
       "       SrcBytes    Offset  sMeanPktSz  ...      Load  pLoss      Rate  \\\n",
       "0      0.000092  0.000201    0.042208  ...  0.000000    0.0  0.000000   \n",
       "1      0.000092  0.000850    0.042208  ...  0.000000    0.0  0.000000   \n",
       "2      0.000092  0.001133    0.042208  ...  0.000000    0.0  0.000000   \n",
       "3      0.000092  0.001140    0.042208  ...  0.000000    0.0  0.000000   \n",
       "4      0.000092  0.001160    0.042208  ...  0.000000    0.0  0.000000   \n",
       "...         ...       ...         ...  ...       ...    ...       ...   \n",
       "12154  0.000117  0.003075    0.053852  ...  0.000000    0.0  0.000000   \n",
       "12155  0.000257  0.003462    0.039298  ...  0.557175    0.0  1.000000   \n",
       "12156  0.000117  0.004298    0.053852  ...  0.000000    0.0  0.000000   \n",
       "12157  0.000304  0.004546    0.069862  ...  0.000002    0.0  0.000003   \n",
       "12158  0.000304  0.004757    0.069862  ...  0.000002    0.0  0.000003   \n",
       "\n",
       "         SrcWin   DstWin  SrcTCPBase  TcpRtt  SynAck  AckDat  Label  \n",
       "0      0.000002  0.00000    0.388654     0.0     0.0     0.0      1  \n",
       "1      0.000000  0.00000    0.388654     0.0     0.0     0.0      1  \n",
       "2      0.000000  0.00000    0.388654     0.0     0.0     0.0      1  \n",
       "3      0.000000  0.00000    0.388654     0.0     0.0     0.0      1  \n",
       "4      0.000000  0.00000    0.388669     0.0     0.0     0.0      1  \n",
       "...         ...      ...         ...     ...     ...     ...    ...  \n",
       "12154  0.000000  0.00000    0.000000     0.0     0.0     0.0      0  \n",
       "12155  0.000008  0.00025    0.292755     0.0     0.0     0.0      0  \n",
       "12156  0.000000  0.00000    0.000000     0.0     0.0     0.0      0  \n",
       "12157  0.000000  0.00000    0.000000     0.0     0.0     0.0      0  \n",
       "12158  0.000000  0.00000    0.000000     0.0     0.0     0.0      0  \n",
       "\n",
       "[12159 rows x 21 columns]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfh2 = Auto_Feature_Engineering(df)\n",
    "dfh2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Split & Balancing (After Feature Engineering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dfh2.drop(['Label'],axis=1)\n",
    "y = dfh2['Label']\n",
    "\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Automated Model Selection\n",
    "Select the best-performing model among five common machine learning models (Naive Bayes, KNN, random forest, LightGBM, and ANN/MLP) by evaluating their learning performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pipeline\n",
    "pipe = Pipeline([('classifier', KNeighborsClassifier())])\n",
    "\n",
    "# Create space of candidate learning algorithms and their hyperparameters\n",
    "search_space = [\n",
    "                {'classifier': [KNeighborsClassifier()]},\n",
    "                {'classifier': [RandomForestClassifier()]},\n",
    "                {'classifier': [lgb.LGBMClassifier(verbose = -1)]},\n",
    "                {'classifier': [KerasClassifier(build_fn=ANN, verbose=0)]},\n",
    "                 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = GridSearchCV(pipe, search_space, cv=5, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('classifier', KNeighborsClassifier())]),\n",
       "             param_grid=[{'classifier': [KNeighborsClassifier()]},\n",
       "                         {'classifier': [RandomForestClassifier()]},\n",
       "                         {'classifier': [LGBMClassifier(verbose=-1)]},\n",
       "                         {'classifier': [<keras.wrappers.scikit_learn.KerasClassifier object at 0x000001658BACF3C8>]}])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model:{'classifier': RandomForestClassifier()}\n",
      "Accuracy:0.9961346316222477\n"
     ]
    }
   ],
   "source": [
    "print(\"Best Model:\"+ str(clf.best_params_))\n",
    "print(\"Accuracy:\"+ str(clf.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([2.59265900e-03, 4.29303312e-01, 1.42621040e-01, 5.53982763e+00]),\n",
       " 'std_fit_time': array([4.88111152e-04, 1.78230136e-02, 9.09718641e-03, 6.47998989e-01]),\n",
       " 'mean_score_time': array([0.38965359, 0.01775336, 0.00498633, 0.1201952 ]),\n",
       " 'std_score_time': array([0.01900864, 0.00039945, 0.00089132, 0.01313867]),\n",
       " 'param_classifier': masked_array(data=[KNeighborsClassifier(), RandomForestClassifier(),\n",
       "                    LGBMClassifier(verbose=-1),\n",
       "                    <keras.wrappers.scikit_learn.KerasClassifier object at 0x000001658BACF3C8>],\n",
       "              mask=[False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'classifier': KNeighborsClassifier()},\n",
       "  {'classifier': RandomForestClassifier()},\n",
       "  {'classifier': LGBMClassifier(verbose=-1)},\n",
       "  {'classifier': <keras.wrappers.scikit_learn.KerasClassifier at 0x1658bacf3c8>}],\n",
       " 'split0_test_score': array([0.92269737, 0.99917763, 0.99465461,        nan]),\n",
       " 'split1_test_score': array([0.96833882, 0.98848684, 0.98889803,        nan]),\n",
       " 'split2_test_score': array([0.99917763, 0.99958882, 0.99917763,        nan]),\n",
       " 'split3_test_score': array([0.96669408, 0.99629934, 0.99917763,        nan]),\n",
       " 'split4_test_score': array([0.9691485 , 0.99712053, 0.99465241,        nan]),\n",
       " 'mean_test_score': array([0.96521128, 0.99613463, 0.99531206,        nan]),\n",
       " 'std_test_score': array([0.02444843, 0.00401694, 0.0037919 ,        nan]),\n",
       " 'rank_test_score': array([3, 1, 2, 4])}"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest model is the best performing machine learning model, and the best cross-validation accuracy is 99.613%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.91799999999999%\n",
      "Precision: 99.933%\n",
      "Recall: 99.933%\n",
      "F1-score: 99.933%\n",
      "Time: 2.46016\n",
      "Wall time: 326 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.866%\n",
      "Recall: 99.933%\n",
      "F1-score: 99.899%\n",
      "Time: 9.02354\n",
      "Wall time: 726 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.465%\n",
      "Precision: 99.664%\n",
      "Recall: 99.46300000000001%\n",
      "F1-score: 99.563%\n",
      "Time: 271.69228\n",
      "Wall time: 672 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = knn.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=32,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    K.clear_session()\n",
    "    inputs=Input(shape=(X_train.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X_train, pd.get_dummies(y_train).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.15%\n",
      "Precision: 97.5%\n",
      "Recall: 99.53%\n",
      "F1-score: 98.504%\n",
      "Time: 271.69228\n",
      "Wall time: 6.51 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ann = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "ann.fit(X_train,y_train)\n",
    "predictions = ann.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Hyperparameter Optimization\n",
    "Optimize the best performing machine learning model (lightGBM) by tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold-out validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 192.3828125, 'max_depth': 41.38671875, 'learning_rate': 0.74609375, 'num_leaves': 122.265625, 'min_child_samples': 49.53125}\n",
      "Accuracy:0.9995888157894737\n"
     ]
    }
   ],
   "source": [
    "#Particle Swarm Optimization\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "search = {\n",
    "    'n_estimators': [50, 500],\n",
    "    'max_depth': [5, 50],\n",
    "    'learning_rate': (0, 1),\n",
    "    \"num_leaves\":[100, 2000],\n",
    "    \"min_child_samples\":[10, 50],\n",
    "         }\n",
    "# Define the objective function\n",
    "def performance(n_estimators=None, max_depth=None,learning_rate=None,num_leaves=None,min_child_samples=None):\n",
    "    clf = lgb.LGBMClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   learning_rate=float(learning_rate),\n",
    "                                   num_leaves=int(num_leaves),\n",
    "                                   min_child_samples=int(min_child_samples),\n",
    "                                  )\n",
    "    clf.fit(X_train,y_train)\n",
    "    prediction = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,prediction)\n",
    "    return score\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.959%\n",
      "Precision: 99.933%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.966%\n",
      "Wall time: 244 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=41, learning_rate=  0.74609375, n_estimators = 192, \n",
    "                         num_leaves = 122, min_child_samples = 49)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "After hyperparameter optimization, the hold-out accuracy has been improved from 99.918% to 99.959%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 396.5087890625, 'max_depth': 42.19970703125, 'min_samples_split': 3.05029296875, 'min_samples_leaf': 2.4794921875, 'criterion_index': 0.62646484375}\n",
      "Accuracy:0.9991776315789473\n"
     ]
    }
   ],
   "source": [
    "import optunity\n",
    "import optunity.metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Define the hyperparameter configuration space for RandomForestClassifier\n",
    "search = {\n",
    "    'n_estimators': [50, 500],  # Number of trees in the forest\n",
    "    'max_depth': [5, 50],  # Maximum depth of the tree\n",
    "    'min_samples_split': [2, 11],  # Minimum number of samples required to split an internal node\n",
    "    'min_samples_leaf': [1, 11],  # Minimum number of samples required to be at a leaf node\n",
    "    'criterion_index': [0, 1],  # Index to select criterion, 0 for \"gini\", 1 for \"entropy\"\n",
    "}\n",
    "\n",
    "# Define the objective function for RandomForestClassifier\n",
    "def performance(n_estimators=None, max_depth=None, min_samples_split=None, min_samples_leaf=None, criterion_index=None):\n",
    "    # Convert criterion_index to actual criterion string\n",
    "    criterion = [\"gini\", \"entropy\"][int(criterion_index)]\n",
    "    \n",
    "    # Define and fit the model\n",
    "    clf = RandomForestClassifier(\n",
    "        n_estimators=int(n_estimators),\n",
    "        max_depth=int(max_depth),\n",
    "        min_samples_split=int(min_samples_split),\n",
    "        min_samples_leaf=int(min_samples_leaf),\n",
    "        criterion=criterion\n",
    "    )\n",
    "    clf.fit(X_train, y_train)\n",
    "    prediction = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test, prediction)\n",
    "    return score\n",
    "\n",
    "# Detect the optimal hyperparameter values using PSO\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                   solver_name='particle swarm',\n",
    "                                                   num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\" + str(info.optimum))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.91799999999999%\n",
      "Precision: 99.933%\n",
      "Recall: 99.933%\n",
      "F1-score: 99.933%\n",
      "Wall time: 2.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = RandomForestClassifier(max_depth=42, n_estimators = 396, min_samples_split = 3,\n",
    "                         min_samples_leaf = 2, criterion = 'gini')\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Combined Algorithm Selection and Hyperparameter tuning (CASH)\n",
    "CASH is the process of combining the two AutoML procedures: model selection and hyperparameter optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Method: Particle Swarm Optimization (PSO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'algorithm': 'lightgbm', 'epochs': None, 'neurons': None, 'patience': None, 'n_neighbors': None, 'learning_rate': 0.84814453125, 'max_depth': 30.29052734375, 'min_child_samples': 21.73828125, 'n_estimators': 427.2705078125, 'num_leaves': 1191.943359375, 'max_features': None, 'min_samples_leaf': None, 'min_samples_split': None}\n",
      "0.9995888157894737\n"
     ]
    }
   ],
   "source": [
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "search = {'algorithm': {'k-nn': {'n_neighbors': [3, 10]},\n",
    "                        'random-forest': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_features': [5, 12],\n",
    "                                'max_depth': [5,50],\n",
    "                                \"min_samples_split\":[2,11],\n",
    "                                \"min_samples_leaf\":[1,11]},\n",
    "                        'lightgbm': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_depth': [5, 50],\n",
    "                                'learning_rate': (0, 1),\n",
    "                                \"num_leaves\":[100, 2000],\n",
    "                                \"min_child_samples\":[10, 50],\n",
    "                                    },\n",
    "                        'ann': {\n",
    "                                'neurons': [10, 100],\n",
    "                                'epochs': [20, 50],\n",
    "                                'patience': [3, 20],\n",
    "                                }\n",
    "                        }\n",
    "          \n",
    "         }\n",
    "def performance(\n",
    "                algorithm, n_neighbors=None, \n",
    "    n_estimators=None, max_features=None,max_depth=None,min_samples_split=None,min_samples_leaf=None,\n",
    "    learning_rate=None,num_leaves=None,min_child_samples=None,\n",
    "    neurons=None,epochs=None,patience=None\n",
    "):\n",
    "    # fit the model\n",
    "    if algorithm == 'k-nn':\n",
    "        model = KNeighborsClassifier(n_neighbors=int(n_neighbors))\n",
    "    elif algorithm == 'random-forest':\n",
    "        model = RandomForestClassifier(n_estimators=int(n_estimators),\n",
    "                                       max_features=int(max_features),\n",
    "                                       max_depth=int(max_depth),\n",
    "                                       min_samples_split=int(min_samples_split),\n",
    "                                       min_samples_leaf=int(min_samples_leaf))\n",
    "    elif algorithm == 'lightgbm':\n",
    "        model = lgb.LGBMClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   learning_rate=float(learning_rate),\n",
    "                                   num_leaves=int(num_leaves),\n",
    "                                   min_child_samples=int(min_child_samples),\n",
    "                                  )\n",
    "    elif algorithm == 'ann':\n",
    "        model = KerasClassifier(build_fn=ANN, verbose=0,\n",
    "                               neurons=int(neurons),\n",
    "                                epochs=int(epochs),\n",
    "                                patience=int(patience)\n",
    "                               )\n",
    "    else:\n",
    "        raise ArgumentError('Unknown algorithm: %s' % algorithm)\n",
    "# predict the test set\n",
    "    model.fit(X_train,y_train)\n",
    "    prediction = model.predict(X_test)\n",
    "    score = accuracy_score(y_test,prediction)\n",
    "    return score\n",
    "\n",
    "# Run the CASH process\n",
    "optimal_configuration, info, _ = optunity.maximize_structured(performance, \n",
    "                                                              search_space=search, \n",
    "                                                              num_evals=10)\n",
    "print(optimal_configuration)\n",
    "print(info.optimum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.959%\n",
      "Precision: 99.933%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.966%\n",
      "Wall time: 317 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=30, learning_rate= 0.84814453125, n_estimators = 427, \n",
    "                         num_leaves = 1191, min_child_samples = 21)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM with the above hyperparameter values is identified as the optimal model"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}