LCCDE_IDS_GlobeCom22.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles\n",
    "This is the code for the paper entitled \"**LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles**\" accepted in 2022 IEEE Global Communications Conference (GLOBECOM).  \n",
    "Authors: Li Yang (lyang339@uwo.ca), Abdallah Shami (Abdallah.Shami@uwo.ca), Gary Stevens, and Stephen de Rusett  \n",
    "Organization: The Optimized Computing and Communications (OC2) Lab, ECE Department, Western University, Ontario, Canada; S2E Technologies, St. Jacobs, Ontario, Canada  \n",
    "\n",
    "If you find this repository useful in your research, please cite:  \n",
    "L. Yang, A. Shami, G. Stevens, and S. DeRusett, “LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles,\" in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "import lightgbm as lgb\n",
    "import catboost as cbt\n",
    "import xgboost as xgb\n",
    "import time\n",
    "from river import stream\n",
    "from statistics import mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled CICIDS2017 dataset\n",
    "The CICIDS2017 dataset is publicly available at: https://www.unb.ca/cic/datasets/ids-2017.html  \n",
    "Due to the large size of this dataset, the sampled subsets of CICIDS2017 is used. The subsets are in the \"data\" folder.  \n",
    "If you want to use this code on other datasets (e.g., CAN-intrusion dataset), just change the dataset name and follow the same steps. The models in this code are generic models that can be used in any intrusion detection/network traffic datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"./data/CICIDS2017_sample_km.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18225\n",
       "3     3042\n",
       "6     2180\n",
       "1     1966\n",
       "5     1255\n",
       "2       96\n",
       "4       36\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Corresponding Attack Types:**  \n",
    "0 BENIGN &emsp; 18225  \n",
    "3 DoS        &emsp;   &emsp;   3042  \n",
    "6 WebAttack    &emsp;      2180  \n",
    "1 Bot        &emsp;  &emsp;      1966    \n",
    "5 PortScan  &emsp;       1255  \n",
    "2 BruteForce  &emsp;      96  \n",
    "4 Infiltration  &emsp;       36  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split train set and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, random_state = 0) #shuffle=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## SMOTE to solve class-imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14569\n",
       "3     2430\n",
       "6     1728\n",
       "1     1579\n",
       "5     1024\n",
       "2       82\n",
       "4       28\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "smote=SMOTE(n_jobs=-1,sampling_strategy={2:1000,4:1000})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = smote.fit_resample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14569\n",
       "3     2430\n",
       "6     1728\n",
       "1     1579\n",
       "5     1024\n",
       "4     1000\n",
       "2     1000\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning (ML) model training\n",
    "### Training three base learners: LightGBM, XGBoost, CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3656\n",
      "           1       0.99      0.99      0.99       387\n",
      "           2       1.00      1.00      1.00        14\n",
      "           3       1.00      1.00      1.00       612\n",
      "           4       1.00      0.75      0.86         8\n",
      "           5       0.99      1.00      0.99       231\n",
      "           6       1.00      1.00      1.00       452\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       1.00      0.96      0.98      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n",
      "Accuracy of LightGBM: 0.9970149253731343\n",
      "Precision of LightGBM: 0.9970231077536348\n",
      "Recall of LightGBM: 0.9970149253731343\n",
      "Average F1 of LightGBM: 0.9969877746104384\n",
      "F1 of LightGBM for each type of attack: [0.99795054 0.99092088 1.         0.997543   0.85714286 0.99354839\n",
      " 0.99778271]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw+ElEQVR4nO3dd3wUdf7H8ddnkyBSpAgCSVAQsBdAilgQW8CCcBZURNEfJxZUUE/ELiqn3lk5PRUFARvEghRRQAQRTiFIJ/SipFCUooBiyuf3x05gxWySXWZ2l93P08c82Hx3dt4zi/nwnfYdUVWMMcb8lS/aK2CMMbHKCqQxxgRhBdIYY4KwAmmMMUFYgTTGmCCsQBpjTBDJ0V6BMonYNUjGRIOqhPOxgp/Whvw7m1Ln6LCyIiGmC2TBljVRy06p24TklNSo5RcW5CVsfmFBHoDlRznfxHiBNMYcZIqLor0GrrICaYxxjxZHew1cZQXSGOOeYiuQxhhTKrUepDHGBGE9SGOMCSLOepB2obgxxj3FRaFP5RCRyiIyR0QWishSERnotA8XkXUissCZmjvtIiKDRWS1iCwSkZYBy+opIqucqWd52daDNMa4x5se5B7gPFXdKSIpwEwR+dx57z5V/Wi/+S8CmjlTW+A1oK2I1AYeA1oBCnwvIuNUdVuwYOtBGmPcU1wc+lQO9dvp/JjiTGXdsdMFGOl87jugpog0ADoCU1R1q1MUpwCdysq2AmmMcY1qcciTiPQWkbkBU+/9lysiSSKyANiMv8jNdt4a5OxGvygihzhtacCGgI/nOG3B2oOyXWxjjHvCOIutqkOAIeXMUwQ0F5GawBgROQl4ANgIVHI+fz/wRMgrUAbrQRpj3KPFoU+hLF51OzAN6KSq+c5u9B7gbaCNM1su0DDgY+lOW7D2oKxAGmPc481Z7LpOzxERORS4EFjuHFdERAToCixxPjIOuME5m306sENV84FJQIaI1BKRWkCG0xbUQVUg9+z5g2v+3pfLe95Ol+tu4ZW33gFAVXn5jeFccs3f6dy9N+9+OBaAOfMWcXrGFVzRsw9X9OzDa8PeAyB/0xZuuuN+LruuN12uu4V3Mj91bR3fHPI8eTkLWTB/qmvLDFXHjA4sXTKD5dkz6X9fH8uPoEMOOYRvZ03g+7lTWLjgKx579N6I5kf7u/eoB9kAmCYii4As/McgJwDvichiYDFQB3jKmX8isBZYDbwJ3A6gqluBJ51lZAFPOG1BSSw/9nX/seVUld9++50qVQ6loLCQG277BwP63sLaHzYwZ94iBj10Dz6fj5+3befwWjWZM28Rwz/4mP/+e+Cflrvlp61s+XkrJxzblF27dtOt110MfvoRmjQ+au884Q53dvZZbdm5cxdvv/0yzVucH+aWhz/cmM/nY9nSb+h08bXk5OTz3bcT6XH97SxbtuqgyT+Q4b6inQ9QtWoVdu3aTXJyMjOmj+Huex5j9px5nue7+Xcf7niQe5ZNC7mgHHL8uTE7HqRnPUgROU5E7ncu2BzsvD7+AJdJlSqHAlBYWEhhYSEiwugxn3HbTd3x+fybc3itmmUup26d2pxwbFPA/z/z0Uc1ZNOWnw9k1fb6ZuZstm7b7sqywtGmdQvWrFnPunU/UlBQQGbmWC7r3NHyI2jXrt0ApKQkk5ySQqQ6IbGw7V5c5hNNnhRIEbkfGAUIMMeZBPhARAYcyLKLioq4omcf2l96Le1at+CUE49jQ24+n0/9mm7/dxe33vsIP2zYd9x14ZJlXN7zdm699xFWr/3hL8vLzd/EslVrOOXEYw9ktWJGalp9NuTsG/A0Jzef1NT6lh9BPp+PuVmTyc9dxNSpM5iTNT8iubGw7V6fpIk0r3qQvYDWqvqMqr7rTM/gP8vU60AWnJSUxMcjXmXqmHdYnL2SVWvX80dBAYdUqkTmsMFc0bkTj/zzRQBOOLYJUz4ewScj/kv3Kzpz1wN/vgJg9+7fuPuhp7j/rluoVrXqgayWMXsVFxfTqnUGRzVuRetWLTgxTv7xrRDrQVZIMVDaAZQGzntBBV40+tbID4LOd1j1arRpeQozv5tL/bp1uOCcMwG44JwzWLlmHQDVqlbdu0ve/ow2FBYWsm37DgAKCgvp99BTXJJxLhd2ODP0LYxRebkbaZi+76tPT2tAXt5Gy4+CHTt+YfrXs+iY0SEiebGw7apFIU+xzKsC2Q+YKiKfi8gQZ/oCmAr0LeuDqjpEVVupaqu/33Dtn97bum07v/zqv+Po9z17+DZrPo2Pash57dsxZ95CALLmL+aohv6L43/6eeve4z+Ls1dQrErNGoehqjz69EscfVRDel5zuasbHm1ZcxfQtGljGjVqSEpKCt26dWH8hMmWHyF16tSmRo3DAKhcuTIXnN+eFSsi82ylaG87EHe72J7cSaOqX4jIMfh3qUtu5ckFsvQA/snY8vM2HnrqOYqKi9FipeN5Z9PhzLa0POVE7h/4L94Z/SlVDq3MwAH9AJg8bSajx3xGUnISlStV4t8DByAizFu4hPFfTKVZk0Zc0dN/KUTfW3rS/ow2ZaRXzLvvvMo57dtRp05t1q+dy8AnnuPt4aMOeLkVVVRURN9+DzPxs/dJ8vkYPmI02dkrLT9CGjSox7ChL5GU5MPn8/HRR+P5bOKXEcmO9rYDMb/LHKqD6jKfSLKnGtpTDRM6P8zLfH7//tOQf2crn9Y1Zi/zsXuxjTHusacaGmNMEDF+TDFUViCNMe6Js2OQViCNMe6xHqQxxgRhPUhjjAnCCqQxxpQu1u+MCZUVSGOMe6wHaYwxQcTZSZqDakRxY4yJJOtBGmPcY7vYxhgTRJztYluBNMa4x3qQxhgThPUgIyelbpOo5pcMO2X5lp+I+WGxHqQxxgRhBTJyoj1g7HF1W0Utf/mWuVHf/oQeMNbyw2O72MYYE4T1II0xJgjrQRpjTBDWgzTGmCDirAdp92IbY9xTXBz6VA4RqSwic0RkoYgsFZGBTntjEZktIqtFZLSIVHLaD3F+Xu283yhgWQ847StEpGN52VYgjTHu8aBAAnuA81T1VKA50ElETgeeBV5U1abANqCXM38vYJvT/qIzHyJyAnANcCLQCfiviCSVFWwF0hjjHtXQp3IXqaqqO50fU5xJgfOAj5z2EUBX53UX52ec988XEXHaR6nqHlVdB6wG2pSVbQXSGOMeb3qQiEiSiCwANgNTgDXAdlUtdGbJAdKc12nABgDn/R3A4YHtpXymVFYgjTHuCaNAikhvEZkbMPXef7GqWqSqzYF0/L2+4yKxOXYW2xjjnjDOYqvqEGBIBefdLiLTgHZATRFJdnqJ6UCuM1su0BDIEZFkoAbwc0B7icDPlMp6kMaYmCYidUWkpvP6UOBCYBkwDbjSma0nMNZ5Pc75Gef9r1RVnfZrnLPcjYFmwJyysq0HaYxxjzcXijcARjhnnH1ApqpOEJFsYJSIPAXMB4Y68w8F3hGR1cBW/GeuUdWlIpIJZAOFQB8t5zGMcdmDXL3yO+bP+5K5WZP57tuJnmRUOqQSmV8M59Np7zF+xmju7O8/bHL62a35+Mt3GPPVe7w3/k2ObJwOwI23dmfCN6MZO/193v7ov6Sm1/dkvQA6ZnRg6ZIZLM+eSf/7+niWY/mxlR0L+R6dxV6kqi1U9RRVPUlVn3Da16pqG1VtqqpXqeoep/135+emzvtrA5Y1SFWbqOqxqvp5edlxWSABLrjwKlq1zuD0dhd7svw/9vzBjVfcRtdzr+Nv53XnrHPbceppJ/H4v+7nvtse4W/nXceETyZx293+S7OWLV7BlRk30KVDdyZNmMo/Hr3Lk/Xy+XwMfnkQl3buwcmnnsvVV3fl+OObeZJl+bGTHQv5gGdnsaMlbgtkJOze9RsAySnJJKcko6qoQrXqVQGoXr0amzduAWD2rO/5/bc9ACycu5j6qUd4sk5tWrdgzZr1rFv3IwUFBWRmjuWyzuXeMGD5B3l2LOQDViAPBqrK5xM/YPZ3n/P3Xtd5luPz+Rjz1XvMyp7M/76ezaJ5S3n47qcY8v5LTF8wgcuuuoghg0f85XNXXteFGVP/58k6pabVZ0POvvH8cnLzSU31bnfe8mMjOxbyAf9Z7FCnGBaVAikiN3m5/HPO/Rtt2nbi0s49uO22Gzn7rLae5BQXF/O3866jw6mXcEqLE2l2XBN63tqd3t370aH5pXwyajwDnuj3p890vvIiTjz1eIa++o4n62RMNGmxhjzFsmj1IAcGeyPwotHi4l1hLTwvbyMAW7b8zNixn9O6dfOwllNRv/6yk9mzvufs89tx3InNWDRvKQCffzqFFq1P2Ttfu/ZtuLXfTdx+w70U/FHgybrk5W6kYfq+kajT0xrs/T4iIZHzE3nb97Jd7IoRkUVBpsVAvWCfU9UhqtpKVVv5fFVDzq1S5VCqVau69/WFF5zD0qUrwt6OYGodXpPqh1UD4JDKh3DGOW1Yu3I91atXo9HRRwJwxjltWbtqPQDHn3QMA597gNuvv5etP21zfX1KZM1dQNOmjWnUqCEpKSl069aF8RMme5Zn+bGRHQv5QNztYnt5HWQ9oCP+UTYCCeDNATigXr26fPSh/3Ko5OQkRo36lEmTp7ueU7deHZ75z+MkJfkQ8fHFuC+ZPmUmj9w7iMHDnqVYi/ll+6882O9JAO57vC9Vqh7KS0OfASA/ZyO333Cv6+tVVFRE334PM/Gz90ny+Rg+YjTZ2Stdz7H82MqOhXwAYnyXOVSiFbgOKawFiwwF3lbVmaW8976qdi9vGcmV0qL2bdtDu+yhXQmdryrhfHb3f24P+Xe2yp3/DSsrEjzrQapqrzLeK7c4GmMOQjF+TDFUdquhMcY9Hu2RRosVSGOMe6wHaYwxQcTZSRorkMYY98T4ZTuhsgJpjHFPnPUg4/JebGOMcYP1II0xrlE7SWOMMUHE2S62FUhjjHvsJI0xxgRhPUhjjAnCjkEaY0wQ1oM0xpgg7BikMcYEYT3IyCkZFy9alm+ZG9X8aG+/5Sd2fjjsOsgIStQBYxM9PyYGjLX88FgP0hhjgrACaYwxQdhJGmOMCSLOepA2mo8xxjVarCFP5RGRhiIyTUSyRWSpiPR12h8XkVwRWeBMFwd85gERWS0iK0SkY0B7J6dttYgMKC/bepDGGPd404MsBO5V1XkiUh34XkSmOO+9qKrPBc4sIicA1wAnAqnAlyJyjPP2q8CFQA6QJSLjVDU7WLAVSGOMezy4zEdV84F85/WvIrIMSCvjI12AUaq6B1gnIquBNs57q1V1LYCIjHLmDVogbRfbGHPQEJFGQAtgttN0h4gsEpFhIlLLaUsDNgR8LMdpC9YelBVIY4x7ijXkSUR6i8jcgKl3aYsWkWrAx0A/Vf0FeA1oAjTH38N83u3NsV1sY4x7wjgGqapDgCFlzSMiKfiL43uq+onzuU0B778JTHB+zAUaBnw83WmjjPZSWQ/SGOMaVQ15Ko+ICDAUWKaqLwS0NwiY7W/AEuf1OOAaETlERBoDzYA5QBbQTEQai0gl/CdyxpWVbT1IY4x7vDmLfSZwPbBYRBY4bQ8C14pIc0CB9cAtAKq6VEQy8Z98KQT6qGoRgIjcAUwCkoBhqrq0rGArkMYY93hQIFV1JiClvDWxjM8MAgaV0j6xrM/tzwqkMcY1Fbnw+2ASl8cgO2Z0YOmSGSzPnkn/+/rEff6bQ54nL2chC+ZP/ct7d/e7hcI/cjn88FqlfNIbifb9x0p2LOSHcxY7lsVdgfT5fAx+eRCXdu7Byaeey9VXd+X445vFdf7IkZlccul1f2lPT0/lwgva88MPOZ7mB0rE7z8WsmMhH4DiMKYYFncFsk3rFqxZs551636koKCAzMyxXNa5Y/kfPIjzv5k5m63btv+l/fnnHmfAg4MqdKbQLYn4/cdCdizkgzf3YkeTZwVSRI4TkfOdizsD2zt5lQmQmlafDTn7BvzMyc0nNbW+l5ExlV+ic+cMcnPzWbQo6F1Unoj29kczP5G3fS/bxS6fiNwFjAXuBJaISJeAt//pRabZ59BDK/PA/Xfy+MDnyp/ZGDfF2S62V2exbwZOU9Wdzr2TH4lII1V9mdJP1+/l3GbUG0CSauDzVQ0pOC93Iw3T9w1Vn57WgLy8jSGufviinQ/QpEkjGjU6knlz/QOepKc3IGv2JNqdeQmbNm3xNDva2x/N/ETe9hKxvsscKq92sX2quhNAVdcDHYCLROQFyimQqjpEVVupaqtQiyNA1twFNG3amEaNGpKSkkK3bl0YP2Fy6FsQpmjnAyxZspzU9FNpeszpND3mdHJy8mndtqPnxRGiv/3RzE/kbd/LepAVsklEmqvqAgCnJ3kpMAw42aNMAIqKiujb72EmfvY+ST4fw0eMJjt7pZeRUc9/951XOad9O+rUqc36tXMZ+MRzvD18lKeZwSTi9x8L2bGQD/HXgxQvznCKSDpQqKp/6d+LyJmqOqsiy0mulBa1bzuRnyoY7fxYeapfQuerlrmnF8zWLueE/Dtbe+zXYWVFgic9SFUNeuFdRYujMebgE2fP7Iq/6yCNMcYtdi+2McY9cdaDtAJpjHFNvO1iW4E0xrjHCqQxxpTOepDGGBOEFUhjjAnCCqQxxgQT3vXlMcsKpDHGNdaDNMaYILTYepDGGFMq60EaY0wQYY5xEbOsQBpjXGM9SGOMCcKOQUZQybh4lm/5ln9wiOADNCMipgtkog4YGyv5HdLOj0r29NypQIIPWBsD+eGwHqQxxgQRbwXSBsw1xsQ0EWkoItNEJFtElopIX6e9tohMEZFVzp+1nHYRkcEislpEFolIy4Bl9XTmXyUiPcvLtgJpjHGNauhTBRQC96rqCcDpQB8ROQEYAExV1WbAVOdngIuAZs7UG3gN/AUVeAxoC7QBHispqsFYgTTGuEaLJeSp3GWq5qvqPOf1r8AyIA3oAoxwZhsBdHVedwFGqt93QE0RaQB0BKao6lZV3QZMATqVlW3HII0xrvH6QnERaQS0AGYD9VQ133lrI1DPeZ0GbAj4WI7TFqw9qHJ7kCJST0SGisjnzs8niEiv8jfFGJNotDj0SUR6i8jcgKl3acsWkWrAx0A/Vf3lT7n+51e7fpFRRXaxhwOTgJJrDlYC/dxeEWPMwa9YJeRJVYeoaquAacj+yxWRFPzF8T1V/cRp3uTsOuP8udlpzwUaBnw83WkL1h5URQpkHVXNxHnahKoWAkUV+JwxJsGoSshTeUREgKHAMlV9IeCtcUDJmeiewNiA9hucs9mnAzucXfFJQIaI1HJOzmQ4bUFV5BjkLhE5HKf7WhJYgc8ZYxKMR9dBnglcDywWkQVO24PAM0Cmc8jvB6Cb895E4GJgNbAbuAlAVbeKyJNAljPfE6q6tazgihTIe/BX5CYiMguoC1xZse0yxiQSL241VNWZQLDK+5fbvZzjkX2CLGsYMKyi2eUWSFWdJyLnAMc6K7lCVQsqGmCMSRzxdidNuQVSRG7Yr6mliKCqIz1aJ2PMQao4AceDbB3wujL+Lu08wAqkMeZP4m3A3HLPYqvqnQHTzUBLoJr3qxaeQw45hG9nTeD7uVNYuOArHnv03oivQ8eMDixdMoPl2TPpf1+ph0IO+vxqh1Vl4BuPMnL6MEZMG8oJLY+nes3qPPf+s7z7zXCee/9ZqtXw/29SrUY1nnzrcYZOGcJrE16h8bGNPFmnEtH8/hPh774sHt1qGDXh3Gq4C2js9oq4Zc+ePVyQ0Y3TWl3Iaa0y6JjRgbZtWpb/QZf4fD4GvzyISzv34ORTz+Xqq7ty/PHN4i7/joF9mDM9ixs6/B+9Mm7hx9U/0r3PNcybNZ8eZ9/IvFnz6d7nGgB63Nmd1UvX0OvC3jzd91nuGHi76+tTIprff6L83ZclnOsgY1lF7qQZLyLjnGkCsAIYU4HPtRGR1s7rE0TkHhG5+MBXuXy7du0GICUlmeSUFDSC/0y1ad2CNWvWs27djxQUFJCZOZbLOneMq/yq1atyatuT+eyDzwEoLChk5y+7ODPjDL74cDIAX3w4mbM6ngnAUc2OYt6s+QD8uGYD9dPrU6tOTVfXqUQ0v/9E+LsvjxfXQUZTRXqQzwHPO9PTQHtVHVDWB0TkMWAw8JqIPA28AlQFBojIQwe2yuXz+XzMzZpMfu4ipk6dwZys+V5H7pWaVp8NOfsGHM3JzSc1tX5c5TdoWJ/tW3cw4IX7ePOL17nv3/dQ+dDK1K5Ti62b/ZeVbd28ldp1/AOlrMleQ/uLzgbguObHUj+9HnUb1HV1nUpE8/tPhL/78iTULraIJAGPq+rXzjRLVXMqsNwr8V/c2R7/9UhdVfVJ/KNpXH2gK12e4uJiWrXO4KjGrWjdqgUnnnis15EJJSk5iWNOasbYd8Zzc6db+W3373t3pwOV9Nzff3UU1Q6ryluTXufym7qyaslqiovi7OlOBoi/Xewyz2KrapGIFItIDVUN5e6ZQlUtAnaLyJqSG8tV9TcRKfM3w7lRvTeAJNXA56saQuyf7djxC9O/nuU/cL10RdjLCUVe7kYapu8bKj89rQF5eRsjkh2p/C35W9iSv4Vl85cD8PVnM+je51q2/rSN2kfU9vcej6jNtp+3A7B7526evfe5vZ8f9e275P2YX9qiD1g0v/9E+LsvT6zvMoeqIrvYO/Hf4jPUGaV3sIgMLuczf4hIFef1aSWNIlID557uYAJvXA+nONapU5saNQ4DoHLlylxwfntWrFgT8nLClTV3AU2bNqZRo4akpKTQrVsXxk+YHFf5W7dsY3PeFhoenQ7AaWe15IdVP/C/Kd/S6aoMADpdlcGsyf8D/Ge8k1P8/xZf0v1iFs5ezO6du11dpxLR/P4T4e8+0VTkOshPnClQeUcO2qvqHgDVPz0pN4V9N5d7okGDegwb+hJJST58Ph8ffTSezyZ+6WXknxQVFdG338NM/Ox9knw+ho8YTXb2yrjLH/zIKzz8nwdIrpRC/g/5PHPvv/GJj8def5iLr+nEppzNPH7bkwAc2fRIHnjpflSV9SvX869/PO/6+pSI5vefKH/3ZYn1XeZQSXlneEWkr6q+XF6bF5IrpUXtEG4sPFUw2vn2VMMEzg9zX/m71MtD/p09Pe+TmK2qFdnFLq3Hd6PL62GMiQMJc5JGRK4FugONRWRcwFvVgTKHCDLGJKZ4O0lT1jHI/wH5QB3810CW+BVY5OVKGWMOTvF28VbQAqmqP+AfhLJdWQsQkW9Vtcx5jDGJQYMO23hwcuOphpVdWIYxJg4Ux/idMaFyo0DG2VdijAlXsfUgjTGmdPG2i12R0XzudJ4AFnQWF9fHGHMQKw5jimUVuQ6yHpAlIpki0sl5BGOg6z1YL2PMQUiRkKdYVpERxR8GmuF/Lu2NwCoR+aeINHHeX+LpGhpjDhqJ2IMseYziRmcqBGoBH4nIvzxcN2PMQSbeCmRFnmrYF7gB+Al4C7hPVQtExAesAvp7u4rGmINFrO8yh6oiZ7FrA5c7F47vparFInKpN6tljDkYxdljscsvkKr6WBnvLXN3dYwxB7N4uw4ynKcaGmNMQojpC8VLxsWz/OgoGZcxWqK9/YmeH454u60upgtktAeMtfwEHjDW8sPixVlpERkGXApsVtWTnLbHgZuBLc5sD6rqROe9B4BeQBFwl6pOcto7AS8DScBbqvpMedkxXSCNMQeX4r/cR+KK4fgfHT1yv/YXVfW5wAYROQG4BjgRSAW+FJFjnLdfBS4EcvDf/DJOVbPLCrYCaYxxjRe72Ko6Q0QaVXD2LsAo55lY60RkNdDGeW+1qq4FEJFRzrxlFkg7SWOMcU2ELxS/Q0QWiciwgPEi0oANAfPkOG3B2stkBdIY45piCX0Skd4iMjdg6l2BqNeAJkBz/E8+8ORRmbaLbYxxTTjXQarqEGBIiJ/ZVPJaRN4EJjg/5gINA2ZNd9oooz0o60EaY1yjYUzhEJEGAT/+DSgZNGcccI2IHCIijfEPtDMHyAKaiUhjEamE/0RO4MMIS2U9SGOMa7y41VBEPgA6AHVEJAd4DOggIs3x19j1wC0AqrpURDLxn3wpBPqoapGznDuASfgv8xmmqkvLy7YCaYxxjRfXQarqtaU0Dy1j/kHAoFLaJwITQ8m2AmmMcY3dSWOMMUEk3Gg+xhhTUbE+AG6orEAaY1xjBdIYY4LQONvFjrvrIN8c8jx5OQtZMD86Q3VFOx+gY0YHli6ZwfLsmfS/r0/C5deocRijRw1hyeKvWbxoOqe3PS1i2dHe9mjnx9szaeKuQI4cmckll16XsPk+n4/BLw/i0s49OPnUc7n66q4cf3yzhMkHePGFJ5g0aRonnXwOLU+7kGXLV0UkN9rbHu38eBSxAiki+w9V5IlvZs5m67btkYiKyfw2rVuwZs161q37kYKCAjIzx3JZ544Jk3/YYdU5+6y2DHv7AwAKCgrYseOXiGRHe9ujnQ/Wg6wQERm33zQeuLzkZy8yjV9qWn025Owb8DQnN5/U1PoJk9+48ZH89NPPDH3rRbLmTOKN1/9NlSqHRiQ72tse7XyI3K2GkeJVDzId+AV4Af8oG88Dvwa8NsYTyUlJtGhxMm+8MZLWbTqya9du7u9/R7RXK2GEM5pPLPOqQLYCvgceAnao6nTgN1X9WlW/LuuDgUMfFRfv8mj14lde7kYapu8bqj89rQF5eRsTJj8nN5+cnHzmZM0H4JNPPqNF85Mjkh3tbY92PtgudoWoarGqvgjcBDwkIq9QwUuKVHWIqrZS1VY+X1UvVi+uZc1dQNOmjWnUqCEpKSl069aF8RMmJ0z+pk1byMnJ45hjmgBw3nlnsWzZyohkR3vbo50P8VcgPb0OUlVzgKtE5BL8u9yee/edVzmnfTvq1KnN+rVzGfjEc7w9fFQkomMiv6ioiL79HmbiZ++T5PMxfMRosrMjUyBiIR+g792PMHLEf6hUKYV1636k19/viUhutLc92vkQ+8cUQyWqsbtJyZXSorZyifxUwWjnx8pT/RI6X8O75PtfR/UI+Xe2/w/vxuyRSLuTxhjjmljfZQ6VFUhjjGtid380PFYgjTGuKY6zEmkF0hjjGtvFNsaYIOKr/2gF0hjjIutBGmNMELF+62CorEAaY1xjJ2mMMSaI+CqPcThgrjHGuMV6kMYY19hJGmOMCcKOQRpjTBDxVR6tQBpjXGS72BFUMuyT5Vu+5R8c4m0X285iG2Nc48VDu0RkmIhsFpElAW21RWSKiKxy/qzltIuIDBaR1SKySERaBnympzP/KhHpWZHtiekeZKIOGJvo+TExYCzQoMbxUcnP37EMiP72h8OjXezhwCtA4KOjBwBTVfUZERng/Hw/cBHQzJnaAq8BbUWkNvAY/udlKfC9iIxT1W1lBVsP0hjjGg3jv3KXqToD2LpfcxdghPN6BNA1oH2k+n0H1BSRBkBHYIqqbnWK4hSgU3nZViCNMa4J56FdgU8ydabeFYiqp6r5zuuNQD3ndRqwIWC+HKctWHuZYnoX2xhzcAnnJI2qDgGGhJupqioinpwdsh6kMcY1XpykCWKTs+uM8+dmpz0XaBgwX7rTFqy9TFYgjTGuKUZDnsI0Dig5E90TGBvQfoNzNvt0YIezKz4JyBCRWs4Z7wynrUy2i22McY0XZ7FF5AOgA1BHRHLwn41+BsgUkV7AD0A3Z/aJwMXAamA3cBOAqm4VkSeBLGe+J1R1/xM/f2EF0hjjmoqclQ55marXBnnr/FLmVaBPkOUMA4aFkm0F0hjjGrvV0BhjgvCiBxlNdpLGGGOCsB6kMcY1tottjDFBFGt87WJbgTTGuCa+ymOcHoPsmNGBpUtmsDx7Jv3vK/WMv+VbfthS0+rz0fi3+fq78Uz/dhx/v7UHAP0fupOps8Yw5ZtPGPXJm9SrXxeAps0aM37y+6zftIBb77jJ9fUJFO3vPoIXikeEaAx3iZMrpYW8cj6fj2VLv6HTxdeSk5PPd99OpMf1t7Ns2aqQlhPucF+Wf+D5BzLcmZv5wYY7O6JeHerVr8vihcuoWq0Kk6Z/xP9ddyd5eRvZ+esuAHrd0oNjjm3C/fcM5PA6tUlvmMpFl5zP9u2/8Porb5eZH+5wZ27+3aMqIX3Ice1RXUP+nf3gh0/DyoqEiPQgReQsEblHRDK8zmrTugVr1qxn3bofKSgoIDNzLJd17uh1rOUnUP7mTT+xeKG/iO3auZtVK9dSv8ERe4sjQJUqh1LS+fj5p60snL+EgsJCV9djf9H+7iG80XximScFUkTmBLy+Gf9gl9WBx5zBLT2TmlafDTn7BvzMyc0nNbW+l5GWn8D56UemcvLJxzPv+0UADHi4L3OXTOXyqy7l3//8j2e5pYn2dw/xt4vtVQ8yJeB1b+BCVR2I/wbx6zzKNCaiqlStwtCRL/Pog0/v7T0+89TLtDrpfD75cAI39U68/9W9GDA3mrwqkD5n1IzD8R/n3AKgqruAMvczAgfPLC7eVdaspcrL3UjD9H3HbtLTGpCXtzHk5YTL8hMjPzk5maEjX+KTDycwcfyXf3n/kw8ncEnnC13PLUu0v3uwXeyKqgF8D8wFageM21YNKPOArKoOUdVWqtrK56sacnDW3AU0bdqYRo0akpKSQrduXRg/YXIYmxAey0+M/BdeeZJVK9fyxqsj9rY1Pvqova87Xnweq1etdT23LNH+7gFUNeQplnlyHaSqNgryVjHwNy8ySxQVFdG338NM/Ox9knw+ho8YTXb2Si8jLT/B8tuc3pKrrulC9tIVTPnmEwCefuIlul9/OU2aNqZYi8nZkMf9dw8EoO4RdfhiWibVq1ejWIu5+bbrOef0zn86qeOGaH/3EH+PfY27y3zckshPFYx2vj3VMAaeahjmZT6dj7w05N/Z8T9OiNnLfOxOGmOMa2L9pEuorEAaY1wTb7vYViCNMa6J5UN24bACaYxxTaxfthMqK5DGGNfE2zHIuBzNxxhj3GA9SGOMa+wkjTHGBGEnaYwxJgjrQRpjTBDxdpLGCqQxxjX20C5jjAkivsqjFUhjjIvi7RikXQdpjHGNV49cEJH1IrJYRBaIyFynrbaITBGRVc6ftZx2EZHBIrJaRBaJSMtwtyeme5Alw05ZvuVHQ8mwY9ES7e0Ph8eX+Zyrqj8F/DwAmKqqzzjPuhoA3A9cBDRzprbAa86fIbMepDHGNRF+aFcXoGRI9xFA14D2ker3HVCz5KkGoYrpHmSiDhib6PmxMmButPNPrdcuKvkLN30b9mc9vMxHgckiosAbqjoEqKeq+c77G4F6zus0YEPAZ3OctnxCFNMF0hhzcAlnF1tEeuN/+mmJIU4BDHSWquaKyBHAFBFZvl+uOsXTVVYgjTGuCWeX2SmG+xfE/efJdf7cLCJjgDbAJhFpoKr5zi70Zmf2XKBhwMfTnbaQ2TFIY4xrvHiqoYhUFZHqJa+BDGAJMA7o6czWExjrvB4H3OCczT4d2BGwKx4S60EaY1zj0XWQ9YAxIgL+mvW+qn4hIllApoj0An4AujnzTwQuBlYDu4Gbwg22AmmMcY0XJ2lUdS1waintPwPnl9KuQB83sm0X2xhjgrAepDHGNTZYhTHGBGHDnRljTBDWgzTGmCCsB2mMMUFYD9IYY4KwHqQxxgQRbz3IuLsOMj09lS8nf8iihdNYuOAr7ryjV8TXoWNGB5YumcHy7Jn0v8+V61Ut/yDJj2S2z+dj9JTh/OedfwPwxMsPMXHOR4z+cjijvxzOsSc22zvv/U/dzfhvM/nwq5Ecd/Ixnq2ThvFfLIu7HmRhYSH39R/I/AVLqFatKnNmf8GXU2ewbNmqiOT7fD4GvzyIThdfS05OPt99O5HxEyZbfgLkRzr7upu7sXbVeqpVr7q37YUnXuXLCdP+NN9Z57fjyKPT6dyuGye3PJGHn72PHhff7Mk6qRZ7stxo8aQHKSJtReQw5/WhIjJQRMaLyLMiUsOLzBIbN25m/oIlAOzcuYvly1eRllrfy8g/adO6BWvWrGfduh8pKCggM3Msl3XuaPkJkB/J7CMa1OXsC85gzHvjy5333I5nMz7zCwAWz1tK9cOqUeeIwz1ZrwgPmOs5r3axh+G/SRzgZaAG8KzT9rZHmX9x1FHpND/1JGbPmR+pSFLT6rMhZ99Q+Tm5+aRGsEBbfvTyI5nd/8l+vPjkqxTv12O7c0BvPvxqJP8YeBcplVIAfzHdlLdp7zyb8rdwRIO6nqyXF6P5RJNXBdKnqoXO61aq2k9VZ6rqQODosj4oIr1FZK6IzC0u3hX2ClStWoXM0W9yzz8e49dfd4a9HGNiTfsLz2DrT9tYtmjFn9oHD3qdLmddS/dOvahR6zD+744eEV8360FWzBIRKRliaKGItAIQkWOAgrI+qKpDVLWVqrby+aqWNWtQycnJfDj6TT74YAyffvp5WMsIV17uRhqm7xuqPz2tAXl5Gy0/AfIjld289Sl0yDiLiVkf8+zrT9D6zNP45yuP8dPmnwEo+KOAsaM+46QWJwCwOX8L9VLr7f18vQZ12Zy/xfX1AutBVtTfgXNEZA1wAvCtiKwF3nTe89SbQ55n2fLVvPRymYMUeyJr7gKaNm1Mo0YNSUlJoVu3LoyfMNnyEyA/UtmD//k6GS27cnHrK7j/1kfJmvU9D94x8E/HFc/t1J7Vy9cCMH3yTDp36wTAyS1PZOevu/YWU7cVq4Y8xTJPzmKr6g7gRudETWMnJ0dVN5X9yQN35hmtub7HlSxanM3cLP//nI888gyff/GV19EAFBUV0bffw0z87H2SfD6GjxhNdvbKiGRbfnTzo73tT//3cWodXhMRYcWSVTzZ/18AfPPl/zjr/HZM+O5Dfv/tdx7tN8izdYj1y3ZCJbHcxU2ulBa1lUvkpwpGOz9WnioY7fyoPtVQVcL5bL0ax4X8O7tpx/KwsiIh7i4UN8YYt8TdheLGmOiJ9bPSobICaYxxTSwfsguHFUhjjGti/ax0qKxAGmNcYz1IY4wJwo5BGmNMENaDNMaYIOwYpDHGBBFvd9JYgTTGuMZ6kMYYE0S8HYO0Ww2NMa7x6pk0ItJJRFaIyGoRGeDxZuxlPUhjjGu86EGKSBLwKnAhkANkicg4Vc12PWw/1oM0xrjGowFz2wCrVXWtqv4BjAK6eLohjpjuQZYM+2T5lp+I+Qs3fRvV/HB4dAQyDdgQ8HMO0NabqD+L6QIZ7ph0JUSkt6pGfljxKGdbvuVHK7/wj9yQf2dFpDfQO6BpSDS/u0Dxvovdu/xZ4jLb8i0/2vkVFvgcKmfavzjmAg0Dfk532jwX7wXSGHPwywKaiUhjEakEXAOMi0RwbO9iG2MSnqoWisgdwCQgCRimqksjkR3vBTKaxzGifQzF8i0/bqjqRGBipHNj+qFdxhgTTXYM0hhjgojLAhmt25Kc7GEisllElkQyNyC/oYhME5FsEVkqIn0jnF9ZROaIyEInf2Ak8511SBKR+SIyIdLZTv56EVksIgtEZG6Es2uKyEcislxElolIdJ4dGyfibhfbuS1pJQG3JQHXRuK2JCe/PbATGKmqJ0Uic7/8BkADVZ0nItWB74GuEdx+Aaqq6k4RSQFmAn1V9btI5DvrcA/QCjhMVS+NVG5A/nqglar+FIXsEcA3qvqWc8a3iqpuj/R6xIt47EFG7bYkAFWdAWyNVF4p+fmqOs95/SuwDP+dCJHKV1Xd6fyY4kwR+1dYRNKBS4C3IpUZK0SkBtAeGAqgqn9YcTww8VggS7stKWIFIpaISCOgBTA7wrlJIrIA2AxMUdVI5r8E9AeKI5i5PwUmi8j3zl0ikdIY2AK87RxieEtEqkYwP+7EY4E0gIhUAz4G+qnqL5HMVtUiVW2O/46HNiISkUMNInIpsFlVv49EXhnOUtWWwEVAH+ewSyQkAy2B11S1BbALiOgx+HgTjwUyarclxQrn2N/HwHuq+km01sPZvZsGdIpQ5JnAZc4xwFHAeSLyboSy91LVXOfPzcAY/Id9IiEHyAnosX+Ev2CaMMVjgYzabUmxwDlJMhRYpqovRCG/rojUdF4fiv9k2fJIZKvqA6qarqqN8P+9f6WqPSKRXUJEqjonx3B2bzOAiFzRoKobgQ0icqzTdD4QkZNz8Sru7qSJ5m1JACLyAdABqCMiOcBjqjo0Uvn4e1HXA4ud44AADzp3IkRCA2CEczWBD8hU1ahcbhMl9YAx/n+nSAbeV9UvIph/J/Ce0zlYC9wUwey4E3eX+RhjjFvicRfbGGNcYQXSGGOCsAJpjDFBWIE0xpggrEAaY0wQViCNMSYIK5AmpojIjSLySrTXwxiwAmkixLlw3JiDihVIUyoReUJE+gX8PKi0wXdFpIOIzBCRz5xBil8XEZ/z3k4ReV5EFgLtRKSHM5juAhF5o6RoishNIrJSRObgvxPImJhgBdIEMwy4AcApeNcAwQZ+aIP/FrcTgCbA5U57VWC2qp4K/AxcDZzpjPRTBFznDPA7EH9hPMtZhjExIe7uxTbuUNX1IvKziLTAf3/xfFX9Ocjsc1R1Ley9F/0s/CPJFOEfVQj8AyecBmQ59ykfin+8yLbAdFXd4nx+NHCMN1tlTGisQJqyvAXcCNTH36MMZv8b+kt+/l1Vi5zXAoxQ1QcCZxSRrge+msZ4w3axTVnG4B/LsTX+0ZGCaeMML+fDvxs9s5R5pgJXisgRACJSW0SOwj/a+TkicrgzjuVVrm6BMQfAepAmKFX9Q0SmAdsDeoKlyQJeAZriHyB3TCnLyhaRh/E/isAHFAB9VPU7EXkc+BbYDixwdSOMOQA23JkJyilk84CrVHVVkHk6AP+IxtMDjfGa7WKbUonICcBqYGqw4mhMvLMepKkQETkZeGe/5j2q2jYa62NMJFiBNMaYIGwX2xhjgrACaYwxQViBNMaYIKxAGmNMEFYgjTEmiP8HmUspzJEWQkgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Train the LightGBM algorithm\n",
    "import lightgbm as lgb\n",
    "lg = lgb.LGBMClassifier()\n",
    "lg.fit(X_train, y_train)\n",
    "y_pred = lg.predict(X_test)\n",
    "print(classification_report(y_test,y_pred))\n",
    "print(\"Accuracy of LightGBM: \"+ str(accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision of LightGBM: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Recall of LightGBM: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Average F1 of LightGBM: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n",
    "print(\"F1 of LightGBM for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n",
    "lg_f1=f1_score(y_test, y_pred, average=None)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm=confusion_matrix(y_test,y_pred)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3656\n",
      "           1       1.00      0.99      0.99       387\n",
      "           2       1.00      1.00      1.00        14\n",
      "           3       1.00      1.00      1.00       612\n",
      "           4       1.00      0.75      0.86         8\n",
      "           5       0.99      1.00      0.99       231\n",
      "           6       1.00      1.00      1.00       452\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       1.00      0.96      0.98      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n",
      "Accuracy of XGBoost: 0.9975746268656717\n",
      "Precision of XGBoost: 0.9975807247413017\n",
      "Recall of XGBoost: 0.9975746268656717\n",
      "Average F1 of XGBoost: 0.9975482770384609\n",
      "F1 of XGBoost for each type of attack: [0.99836021 0.99351492 1.         0.99836334 0.85714286 0.99137931\n",
      " 0.99889258]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvd0lEQVR4nO3dfZyVc/7H8dfnNBNKIqWmmSgVQlK6QbYSKpTCKtKKRbs2FPuTkKVsu3bXzcrdilK5q1DbLUkiWTTpvindp7nphgrlbmbO5/fHuaaOzDUz57iuc86c83nu43p0zvdc1/W+zmn7+F5330tUFWOMMb8UiPcGGGNMorICaYwxLqxAGmOMCyuQxhjjwgqkMca4sAJpjDEu0uK9AWUSsWuQjIkHVYlmscIvN0X8bza99olRZcVCQhfIwl0b45adXqcxaen145ZfVJifsvlFhfkAlh/nfJPgBdIYU8kEi+O9BZ6yAmmM8Y4G470FnrICaYzxTtAKpDHGlEqtB2mMMS6sB2mMMS6SrAdpF4obY7wTLI58KoeIHC4ii0RkuYisFpHhTvs4EdksIsuc6UynXURklIhsEJEVItIqbF39RWS9M/UvL9t6kMYY7/jTg/wR6Kyq+0QkHVgoIm85n92lqm8cMv/FQFNnagc8C7QTkVrAA0BrQIHPRGS6qu5xC7YepDHGO8Fg5FM5NGSf8zbdmcq6Y6cnMMFZ7hPgaBHJALoCc1V1t1MU5wLdysq2AmmM8YxqMOJJRAaIyOKwacCh6xWRKiKyDNhJqMh96nw00tmNflxEDnPaMoFtYYvnOm1u7a5sF9sY450ozmKr6mhgdDnzFANnisjRwFQROR24B9gOVHWWvxsYEfEGlMF6kMYY72gw8imS1avuBeYD3VS1wNmN/hF4EWjrzJYHNAhbLMtpc2t3ZQXSGOMdf85i13F6jojIEcBFwFrnuCIiIkAvYJWzyHTgOuds9tnA16paAMwBuojIMSJyDNDFaXNVqQrkjz/+xNU3DeKK/n+i57V/4KkXXgJAVXniuXFcevVN9Og7gJdfnwbAoiUrOLvLlVzZfyBX9h/Is2NfKXM9XunapROrVy1gbc5Chtw10NN1W35i5x922GF8/NFMPls8l+XL3uOBv/w5pvnx/u196kFmAPNFZAWQTegY5EzgFRFZCawEagN/deafDWwCNgDPA38CUNXdwEPOOrKBEU6bK0nkx74eOracqvL99z9QrdoRFBYVcd0t/8fQQX9g09ZtLFqygpH33UkgEOCrPXs59pijWbRkBeNee5Nn/jWciqynxenNDswT7XBngUCANas/pNsl15CbW8AnH8+m3+/+xJo16yNaT7TDjSVD/q8Z7ive+QDVq1dj//7vSEtLY8H7U7njzgf4dNES3/O9/LuPdjzIH9fMj7igHNbs/IQdD9K3HqSInCIidzsXbI5yXjcrf8ky10m1akcAUFRURFFRESLCpKmzuOWGvgQCoa9z7DFHR7UeL7Rt05KNG7ewefMXFBYWMnnyNC7r0dWTdVt+4ucD7N//HQDp6WmkpacTq05IInx3Py7ziSdfCqSI3A1MBARY5EwCvCYiQ3/NuouLi7my/0A6dL+Gc9q05IzTTmFbXgFvzfuA3r+/nT/++X62bjt43HX5qjVc0f9P/PHP97Nh09Yy1+OF+pn12JZ7cMDR3LwC6tev58m6LT/x8yHUk1uc/Q4FeSuYN28Bi7KXxiQ3Eb673ydpYs2vHuSNQBtVfVhVX3amhwmdZbrx16y4SpUqvDn+aeZNfYmVOetYv2kLPxUWcljVqkweO4ore3Tj/r89DsCpJzdm7pvjmTL+Gfpe2YPb7xlR5nqM8UIwGKR1my6c0Kg1bVq35LTTTo73JsWO9SArJAiUdgAlw/nMVfhFoy9MeM11vqNqHEnbVmew8JPF1KtTmws7tgfgwo7nsm7jZgCOrF79wK50h3PbUlRUxJ69X7uuxwv5edtpkHXwq2dlZpCfv92TdVt+4ueH+/rrb3j/g4/o2qVTTPIS4burFkc8JTK/CuRgYJ6IvCUio53pbWAeMKisBVV1tKq2VtXWN113zc8+271nL998G7rj6Icff+Tj7KU0OqEBnTucw6IlywHIXrqSExqELo7/8qvdB47/rMz5nKAqR9c8ynU9XshevIwmTRrRsGED0tPT6d27JzNmvuPJui0/8fNr165FzZpHAXD44Ydz4QUd+Pzz2DxbKd7fHUi6XWxf7qRR1bdF5CRCu9Qlt/LkAdn6K/6TseurPdz310coDgbRoNK182/o1L4drc44jbuH/5OXJv2XakcczvChgwF4Z/5CJk2dRZW0KhxetSr/Gj4UEXFdjxeKi4sZNHgYs2e9SpVAgHHjJ5GTs86TdVt+4udnZNRl7Jh/U6VKgEAgwBtvzGDW7Hdjkh3v7w4k/C5zpCrVZT6xZE81tKcapnR+lJf5/PDZfyP+N3v4Wb0S9jIfuxfbGOMde6qhMca4SPBjipGyAmmM8U6SHYO0AmmM8Y71II0xxoX1II0xxoUVSGOMKV2i3xkTKSuQxhjvWA/SGGNcJNlJmko1orgxxsSS9SCNMd6xXWxjjHGRZLvYViCNMd6xHqQxxriwHmTspNdpHNf8kmGnLN/yUzE/KtaDNMYYF1YgYyfeA8aeUqd13PLX7loc9++f0gPGWn50bBfbGGNcWA/SGGNcWA/SGGNcWA/SGGNcJFkP0u7FNsZ4JxiMfCqHiBwuIotEZLmIrBaR4U57IxH5VEQ2iMgkEanqtB/mvN/gfN4wbF33OO2fi0jX8rKtQBpjvONDgQR+BDqragvgTKCbiJwN/AN4XFWbAHuAG535bwT2OO2PO/MhIqcCVwOnAd2AZ0SkSlnBViCNMd5RjXwqd5WqqrrPeZvuTAp0Bt5w2scDvZzXPZ33OJ9fICLitE9U1R9VdTOwAWhbVrYVSGOMd/zpQSIiVURkGbATmAtsBPaqapEzSy6Q6bzOBLYBOJ9/DRwb3l7KMqWyAmmM8U4UBVJEBojI4rBpwKGrVdViVT0TyCLU6zslFl/HzmIbY7wTxVlsVR0NjK7gvHtFZD5wDnC0iKQ5vcQsIM+ZLQ9oAOSKSBpQE/gqrL1E+DKlsh6kMSahiUgdETnaeX0EcBGwBpgP/NaZrT8wzXk93XmP8/l7qqpO+9XOWe5GQFNgUVnZ1oM0xnjHnwvFM4DxzhnnADBZVWeKSA4wUUT+CiwFxjjzjwFeEpENwG5CZ65R1dUiMhnIAYqAgVrOYxiTtgcZCATIXjSHaVPHlz9zFKoeVpXJb4/jv/NfYcaCSdw2JHTY5OzftOHNd19i6nuv8MqM5zm+URYAffpfwfT3XzvQ3vikRr5sF0DXLp1YvWoBa3MWMuSugb7lWH5iZSdCvk9nsVeoaktVPUNVT1fVEU77JlVtq6pNVPUqVf3Raf/Bed/E+XxT2LpGqmpjVT1ZVd8qLztpC+Ttt93E2rXrfVv/Tz/+xPVX3kKv86/l8s59Oe/8c2hx1uk8+M+7ueuW+7m887XMnDKHW+4IXZo18805XNbpGi7vfC0vPPUSQ0fc4ct2BQIBRj0xku49+tG8xfn06dOLZs2a+pJl+YmTnQj5gG9nseMlKQtkZmYGl1x8AWPHvuZrznf7vwcgLT2NtPQ0VBVVOLJGdQBq1DiSndt3AbB/3/4Dy1Wrdjhagf9yRqNtm5Zs3LiFzZu/oLCwkMmTp3FZj3JvGLD8Sp6dCPlA0hXIpDwG+dijwxl6z1+pUeNIX3MCgQBvvvsSxzfK4tWxr7NiyWqG3fFXRr/6b3744Uf2fbufPhf//sD8fX9/Fdf/sS/p6elcf8UtvmxT/cx6bMs9OJ5fbl4Bbdu09CXL8hMnOxHyAbsX2wsicoNf6770kgvZufNLlixd6VfEAcFgkMs7X0unFpdyRsvTaHpKY/r/sS8D+g6m05ndmTJxBkNHDD4w/6tjX6dL28t59KEnueXO37uv2JhKSoMa8ZTI4rWLPdztg/CLRoPB/W6zuTr33Nb06N6FDes+4ZWXn+H889szftyoX7Wx5fn2m318+tFn/OaCczjltKasWLIagLf+O5eWbc74xfyzpr7DBRd38mVb8vO20yDr4EjUWZkZ5Odv9yXL8hMnOxHygaTbxfatQIrICpdpJVDXbTlVHa2qrVW1dSBQPeLc+4Y9TMMTW9PkpLO5tt+fmD//I/pff/uv+SqlOubYo6lxVGgX/rDDD+Pcjm3ZtG4LNWocScMTjwfg3I7t2LR+CwAnNDp4fWqni85j66YvPN8mgOzFy2jSpBENGzYgPT2d3r17MmPmO75kWX7iZCdCPhDaxY50SmB+HoOsC3QlNMpGOAH+52NuTNSpW5uHn3yQKlUCiAR4e/q7vD93Iff/eSSjxv6DoAb5Zu+33Dv4IQCuvbE353RoS1FREd/s/Yaht7l2on+V4uJiBg0exuxZr1IlEGDc+Enk5KzzJcvyEyc7EfIBSPBd5kiJX2dTRWQM8KKqLizls1dVtW9560irmhm3X9se2mUP7UrpfFWJZtnvnvxTxP9mq932TFRZseBbD1JVbyzjs3KLozGmEkrwY4qRSsrLfIwxceLTHmm8WIE0xnjHepDGGOMiyU7SWIE0xngnwS/biZQVSGOMd5KsB5mUg1UYY4wXrAdpjPGM2kkaY4xxkWS72FYgjTHesZM0xhjjwnqQxhjjwo5BGmOMC+tBGmOMCzsGaYwxLqwHGTsl4+LFy9pdi+OaH+/vb/mpnR8Nuw4yhlJ1wNhUz0+IAWMtPzrWgzTGGBdWII0xxoWdpDHGGBdJ1oO00XyMMZ7RoEY8lUdEGojIfBHJEZHVIjLIaX9QRPJEZJkzXRK2zD0iskFEPheRrmHt3Zy2DSIytLxs60EaY7zjTw+yCPizqi4RkRrAZyIy1/nscVV9JHxmETkVuBo4DagPvCsiJzkfPw1cBOQC2SIyXVVz3IKtQBpjvOPDZT6qWgAUOK+/FZE1QGYZi/QEJqrqj8BmEdkAtHU+26CqmwBEZKIzr2uBtF1sY0ylISINgZbAp07TrSKyQkTGisgxTlsmsC1ssVynza3dlRVIY4x3ghrxJCIDRGRx2DSgtFWLyJHAm8BgVf0GeBZoDJxJqIf5qNdfx3axjTHeieIYpKqOBkaXNY+IpBMqjq+o6hRnuR1hnz8PzHTe5gENwhbPctooo71U1oM0xnhGVSOeyiMiAowB1qjqY2HtGWGzXQ6scl5PB64WkcNEpBHQFFgEZANNRaSRiFQldCJnelnZ1oM0xnjHn7PY7YHfAStFZJnTdi9wjYicCSiwBfgDgKquFpHJhE6+FAEDVbUYQERuBeYAVYCxqrq6rGArkMYY7/hQIFV1ISClfDS7jGVGAiNLaZ9d1nKHsgJpjPFMRS78rkyS8hhk1y6dWL1qAWtzFjLkroFJn//86EfJz13OsqXzfvHZHYP/QNFPeRx77DGlLOmPVPv9EyU7EfKjOYudyJKuQAYCAUY9MZLuPfrRvMX59OnTi2bNmiZ1/oQJk7m0+7W/aM/Kqs9FF3Zg69ZcX/PDpeLvnwjZiZAPQDCKKYElXYFs26YlGzduYfPmLygsLGTy5Glc1qNr+QtW4vwPF37K7j17f9H+6CMPMvTekRU6U+iVVPz9EyE7EfLBn3ux48m3Aikip4jIBc7FneHt3fzKBKifWY9tuQcH/MzNK6B+/Xp+RiZUfokePbqQl1fAihWud1H5It7fP575qfzdD7Bd7PKJyO3ANOA2YJWI9Az7+G9+ZJqDjjjicO65+zYeHP5I+TMb46Uk28X26yz2zcBZqrrPuXfyDRFpqKpPUPrp+gOc24wGAEiVmgQC1SMKzs/bToOsg0PVZ2VmkJ+/PcLNj1688wEaN25Iw4bHs2RxaMCTrKwMsj+dwzntL2XHjl2+Zsf7+8czP5W/e4lE32WOlF+72AFV3QegqluATsDFIvIY5RRIVR2tqq1VtXWkxREge/EymjRpRMOGDUhPT6d3757MmPlO5N8gSvHOB1i1ai31s1rQ5KSzaXLS2eTmFtCmXVffiyPE//vHMz+Vv/sB1oOskB0icqaqLgNwepLdgbFAc58yASguLmbQ4GHMnvUqVQIBxo2fRE7OOj8j457/8ktP07HDOdSuXYstmxYzfMQjvDhuoq+ZblLx90+E7ETIh+TrQYofZzhFJAsoUtVf9O9FpL2qflSR9aRVzYzbr53KTxWMd36iPNUvpfNVy9zTc7O7Z8eI/83WmvZBVFmx4EsPUlVdL7yraHE0xlQ+SfbMruS7DtIYY7xi92IbY7yTZD1IK5DGGM8k2y62FUhjjHesQBpjTOmsB2mMMS6sQBpjjAsrkMYY4ya668sTlhVIY4xnrAdpjDEuNGg9SGOMKZX1II0xxkWUY1wkLCuQxhjPWA/SGGNc2DHIGCoZF8/yLd/yK4cYPkAzJhK6QKbqgLGJkt8p84K4ZL+fNw9I8QFrEyA/GtaDNMYYF8lWIG3AXGNMQhORBiIyX0RyRGS1iAxy2muJyFwRWe/8eYzTLiIySkQ2iMgKEWkVtq7+zvzrRaR/edlWII0xnlGNfKqAIuDPqnoqcDYwUEROBYYC81S1KTDPeQ9wMdDUmQYAz0KooAIPAO2AtsADJUXVjRVIY4xnNCgRT+WuU7VAVZc4r78F1gCZQE9gvDPbeKCX87onMEFDPgGOFpEMoCswV1V3q+oeYC7QraxsOwZpjPGM3xeKi0hDoCXwKVBXVQucj7YDdZ3XmcC2sMVynTa3dlfl9iBFpK6IjBGRt5z3p4rIjeV/FWNMqtFg5JOIDBCRxWHTgNLWLSJHAm8Cg1X1m5/lhp5f7flFRhXZxR4HzAFKrjlYBwz2ekOMMZVfUCXiSVVHq2rrsGn0oesVkXRCxfEVVZ3iNO9wdp1x/tzptOcBDcIWz3La3NpdVaRA1lbVyThPm1DVIqC4AssZY1KMqkQ8lUdEBBgDrFHVx8I+mg6UnInuD0wLa7/OOZt9NvC1sys+B+giIsc4J2e6OG2uKnIMcr+IHIvTfS0JrMByxpgU49N1kO2B3wErRWSZ03Yv8DAw2TnktxXo7Xw2G7gE2AB8B9wAoKq7ReQhINuZb4Sq7i4ruCIF8k5CFbmxiHwE1AF+W7HvZYxJJX7caqiqCwG3yvuL272c45EDXdY1Fhhb0exyC6SqLhGRjsDJzkZ+rqqFFQ0wxqSOZLuTptwCKSLXHdLUSkRQ1Qk+bZMxppIKpuB4kG3CXh9OqEu7BLACaYz5mWQbMLfcs9iqelvYdDPQCjjS/02LTlZWfd5953VWLJ/P8mXvcdutsb9ks2uXTqxetYC1OQsZcleph0Iqff6RR1Vn+HN/YcL7Yxk/fwyntmpGx0s78OK8F3jvi3c4+YyTfjZ/34HX8MrC8Uz44EXadGztyzaViOfvnwp/92Xx6VbDuInmVsP9QCOvN8QrRUVF3DVkOGe0OJ/25/Xglluup1mzpjHLDwQCjHpiJN179KN5i/Pp06dXUubfOnwgi97P5rpOv+fGLn/giw1fsPnzLfzl5gdZ8enKn817QtPj6dyzE9d3vokh/e5h8MjbCQT8ucs1nr9/qvzdlyWa6yATWUXupJkhItOdaSbwOTC1Asu1FZE2zutTReROEbnk129y2bZv38nSZasA2LdvP2vXriezfj2/Yw9o26YlGzduYfPmLygsLGTy5Glc1qNrUuVXr1GdFu2aM+u1twAoKixi3zf7+WLDF2zblPuL+dt3ac97096n8KdCtm/bTt6WfE4582RPt6lEPH//VPi7L48f10HGU0WOQT4S9roI2Kqqv/xXEEZEHiA0okaaiMwlNHrGfGCoiLRU1ZHRbnAkTjghizNbnM6ni5bGIg6A+pn12JZ7cMDR3LwC2rZpmVT5GQ3qsXf31wx97C4an9qYdSvX8eRfnuGH738odf46GceSs2TNgfe7tu+iTkZtT7epRDx//1T4uy9Pou8yR6rMHqSIVAEeVNUPnOmj8oqj47eELu7sQOh6pF6q+hCh0TT6/NqNrojq1asxedLz3Pl/D/Dtt/tiEZkyqqRV4aTTmzLtpRnc3O2PfP/dD/QdeHW8N8skgJTaxVbVYiAoIjUjXG+Rqhar6nfAxpIby1X1e5xbFt2E37geDO6PMDYkLS2N1yc9z2uvTeW//30rqnVEKz9vOw2yDg6Vn5WZQX7+9qTK31Wwi10Fu1izdC0AH8xaQNPm7se6dhV8RZ2M4w68r1OvDrsKvvR0m0rE8/dPhb/78iTbLnZFjpTvI3SLzxhnlN5RIjKqnGV+EpFqzuuzShqdQltmgQy/cT0QqF6Bzful50c/ypq1G/j3E7+459132YuX0aRJIxo2bEB6ejq9e/dkxsx3kip/96497MzfRYMTswA467xWbF2/1XX+/839H517diK9ajr1GtQjq1Ema5d97uk2lYjn758Kf/eppiLHIKc4U7jyjjR0UNUfAVR/9qTcdA7eXO6L9ue24Xf9fsuKlTkszg79n+P++x/mrbff8zP2gOLiYgYNHsbsWa9SJRBg3PhJ5OSsi0l2LPNH3f8Uw568h7Sq6RRsLeDhP/+L87q1Z9BDt1KzVk3+Pn4kG1ZvZEi/oWxZt5X3Z3zAuPfGUFxczL+HjSIY9OcByvH8/VPl774sib7LHCnRco6qisggVX2ivDY/pFXNjNsh30R4qmC88+2phimcH+W+7yf1r4j43+zZ+VMStqpWZBe7tB7f9R5vhzEmCSTbSRrXXWwRuQboCzQSkelhH9UAyhwiyBiTmhL9pEukyjoG+T+gAKgNPBrW/i2wws+NMsZUTv4cWY4f1wKpqlsJDUJ5TlkrEJGPVbXMeYwxqUFdh22snLx4quHhHqzDGJMEgkl2J40XBTLJfhJjTLSC1oM0xpjSJdsudkVG87nNeQKY6ywebo8xphILRjElsopcB1kXyBaRySLSzXkEY7jf+bBdxphKSJGIp0RWkRHFhwFNCT2X9npgvYj8TUQaO5+v8nULjTGVRir2IEseo7jdmYqAY4A3ROSfPm6bMaaSSbYCWZGnGg4CrgO+BF4A7lLVQhEJAOuBIf5uojGmskj0XeZIVeQsdi3gCufC8QNUNSgi3f3ZLGNMZZRkj8Uuv0Cq6gNlfLbG7TNjTOpJtusg/Xm0nDHGJIGEvlC8ZFw8y4+PknEZ4yXe3z/V86ORbLfVJXSBjPeAsZafwgPGWn5U/DgrLSJjge7ATlU93Wl7ELgZ2OXMdq+qznY+uwe4ESgGblfVOU57N+AJoArwgqo+XF52QhdIY0zlEvzFfSSeGAc8BUw4pP1xVQ1/LDUicipwNXAaUB94V0ROcj5+GrgIyCV088t0Vc0pK9gKpDHGM37sYqvqAhFpWMHZewITnWdibRaRDUBb57MNqroJQEQmOvOWWSDtJI0xxjMxvlD8VhFZISJjw8aLyAS2hc2T67S5tZfJCqQxxjNBiXwSkQEisjhsGlCBqGeBxsCZhJ588GiZc0fJdrGNMZ6J5jpIVR0NRPQQe1XdUfJaRJ4HZjpv84AGYbNmOW2U0e7KepDGGM9oFFM0RCQj7O3lQMmgOdOBq0XkMBFpRGignUVANtBURBqJSFVCJ3LCH0ZYKutBGmM848ethiLyGtAJqC0iucADQCcROZNQjd0C/AFAVVeLyGRCJ1+KgIGqWuys51ZgDqHLfMaq6urysq1AGmM848d1kKp6TSnNY8qYfyQwspT22cDsSLKtQBpjPGN30hhjjIuUG83HGGMqKtEHwI2UFUhjjGesQBpjjAtNsl3spLsO8vnRj5Kfu5xlS+MzVFe88wG6dunE6lULWJuzkCF3DUy5/Jo1j2LSxNGsWvkBK1e8z9ntzopZdry/e7zzk+2ZNElXICdMmMyl3a9N2fxAIMCoJ0bSvUc/mrc4nz59etGsWdOUyQd4/LERzJkzn9Obd6TVWRexZu36mOTG+7vHOz8ZxaxAisihQxX54sOFn7J7z95YRCVkfts2Ldm4cQubN39BYWEhkydP47IeXVMm/6ijavCb89ox9sXXACgsLOTrr7+JSXa8v3u888F6kBUiItMPmWYAV5S89yPThNTPrMe23IMDnubmFVC/fr2UyW/U6Hi+/PIrxrzwONmL5vDcf/5FtWpHxCQ73t893vkQu1sNY8WvHmQW8A3wGKFRNh4Fvg17bYwv0qpUoWXL5jz33ATatO3K/v3fcfeQW+O9WSkjmtF8EplfBbI18BlwH/C1qr4PfK+qH6jqB2UtGD70UTC436fNS175edtpkHVwqP6szAzy87enTH5uXgG5uQUsyl4KwJQps2h5ZvOYZMf7u8c7H2wXu0JUNaiqjwM3APeJyFNU8JIiVR2tqq1VtXUgUN2PzUtq2YuX0aRJIxo2bEB6ejq9e/dkxsx3UiZ/x45d5Obmc9JJjQHo3Pk81qxZF5PseH/3eOdD8hVIX6+DVNVc4CoRuZTQLrfvXn7paTp2OIfatWuxZdNiho94hBfHTYxFdELkFxcXM2jwMGbPepUqgQDjxk8iJyc2BSIR8gEG3XE/E8Y/SdWq6Wze/AU33nRnTHLj/d3jnQ+Jf0wxUqKauF8prWpm3DYulZ8qGO/8RHmqX0rna3SXfP/zhH4R/5sdsvXlhD0SaXfSGGM8k+i7zJGyAmmM8Uzi7o9GxwqkMcYzwSQrkVYgjTGesV1sY4xxkVz9RyuQxhgPWQ/SGGNcJPqtg5GyAmmM8YydpDHGGBfJVR6TcMBcY4zxivUgjTGesZM0xhjjwo5BGmOMi+Qqj1YgjTEesl3sGCoZ9snyLd/yK4dk28W2s9jGGM/48dAuERkrIjtFZFVYWy0RmSsi650/j3HaRURGicgGEVkhIq3ClunvzL9eRPpX5PskdA8yVQeMTfX8hBgwFsio2Swu+QVfrwHi//2j4dMu9jjgKSD80dFDgXmq+rCIDHXe3w1cDDR1pnbAs0A7EakFPEDoeVkKfCYi01V1T1nB1oM0xnhGo/hfuetUXQDsPqS5JzDeeT0e6BXWPkFDPgGOFpEMoCswV1V3O0VxLtCtvGwrkMYYz0Tz0K7wJ5k604AKRNVV1QLn9XagrvM6E9gWNl+u0+bWXqaE3sU2xlQu0ZykUdXRwOhoM1VVRcSXs0PWgzTGeMaPkzQudji7zjh/7nTa84AGYfNlOW1u7WWyAmmM8UwQjXiK0nSg5Ex0f2BaWPt1ztnss4GvnV3xOUAXETnGOePdxWkrk+1iG2M848dZbBF5DegE1BaRXEJnox8GJovIjcBWoLcz+2zgEmAD8B1wA4Cq7haRh4BsZ74RqnroiZ9fsAJpjPFMRc5KR7xO1WtcPrqglHkVGOiynrHA2EiyrUAaYzxjtxoaY4wLP3qQ8WQnaYwxxoX1II0xnrFdbGOMcRHU5NrFtgJpjPFMcpXHJDwG+fzoR8nPXc6ypfPitg1du3Ri9aoFrM1ZyJC7Sr3iwPIrcX79zHq8MeNFPvhkBu9/PJ2b/tgPgCH33ca8j6Yy98MpTJzyPHXr1TmwzEP/uJf/LXmbeR9NpXkL/0YJivdvH8MLxWMi6QrkhAmTubT7tXHLDwQCjHpiJN179KN5i/Pp06cXzZo1tfwkyi8qKmL4sH/S8eweXHrR1Vx/U19OOrkxz4waywXtL+ei31zB3DkfcOeQPwHQ+aIOnHjiCZzbqht3DXqAhx99wNPtKRHv3x78Gc0nnmJSIEXkPBG5U0S6+J314cJP2b1nr98xrtq2acnGjVvYvPkLCgsLmTx5Gpf16Gr5SZS/c8eXrFweGrNx/77vWL9uE/UyjmPft/sPzFOt2hGoczyu2yWdeX1i6E64JYtXcFTNGhxXt7an2wTx/+0hutF8EpkvBVJEFoW9vpnQYJc1gAecwS2TVv3MemzLPTjgaG5eAfXr17P8JM3POr4+zZs3Y8lnKwAYOmwQi1fN44qruvOvvz0JQL2M48jP235gmYL8HWRk1C11fb9GvH97sF3sikoPez0AuEhVhxO6QTx++7/GeKha9WqMmfAEf7n37wd6jw//9Qlan34BU16fyQ0DUu//6raLXcH1OqNmHAuIqu4CUNX9QFFZC4YPnhkM7i9r1oSUn7edBlkHh8rPyswgP397GUtYfmXMT0tLY8yEfzPl9ZnMnvHuLz6f8vpMLu1xEQDbC3ZSP/NgTy6jfl0KCnZ4vk3x/u3BdrErqibwGbAYqBU2btuRgJS1oKqOVtXWqto6EKju0+b5J3vxMpo0aUTDhg1IT0+nd++ezJj5juUnWf5jTz3E+nWbeO7p8QfaGp14woHXXS/pzIb1mwCY89Z7XHV1TwBatT6Db7/5lp07vvR8m+L92wOoasRTIvPlOkhVbejyURC43I/MEi+/9DQdO5xD7dq12LJpMcNHPMKL4yb6GfkzxcXFDBo8jNmzXqVKIMC48ZPIyVln+UmU3/bsVlx1dU9yVn/O3A+nAPD3Ef+m7++uoHGTRgQ1SO62fO6+YzgA895ZwAUXdeDjpW/z/Xc/cMfA+zzdnhLx/u0h+R77KolcwdOqZsZt41L5qYLxzrenGibAUw1Vy9zTc9Pj+O4R/5ud8cXMqLJiwe6kMcZ4JtFPukTKCqQxxjPJtottBdIY45lEPmQXDSuQxhjPJPplO5GyAmmM8UyyHYNMusEqjDHGK9aDNMZ4xk7SGGOMCztJY4wxLqwHaYwxLpLtJI0VSGOMZ+yhXcYY4yK5yqMVSGOMh5LtGKRdB2mM8Yxfj1wQkS0islJElonIYqetlojMFZH1zp/HOO0iIqNEZIOIrBCRVtF+n4TuQZYMO2X5lh8PJcOOxUu8v380fL7M53xVDR9peCgwT1Ufdp51NRS4G7gYaOpM7YBnnT8jZj1IY4xnYvzQrp5AyZDu44FeYe0TNOQT4OiSpxpEKqF7kKk6YGyq5yfKgLnxzm9R95y45C/f8XHUy/p4mY8C74iIAs+p6migrqoWOJ9vB0oeFZkJbAtbNtdpKyBCCV0gjTGVSzS72CIygNDTT0uMdgpguPNUNU9EjgPmisjaQ3LVKZ6esgJpjPFMNLvMTjE8tCAeOk+e8+dOEZkKtAV2iEiGqhY4u9A7ndnzgAZhi2c5bRGzY5DGGM/48VRDEakuIjVKXgNdgFXAdKC/M1t/YJrzejpwnXM2+2zg67Bd8YhYD9IY4xmfroOsC0wVEQjVrFdV9W0RyQYmi8iNwFagtzP/bOASYAPwHXBDtMFWII0xnvHjJI2qbgJalNL+FXBBKe0KDPQi23axjTHGhfUgjTGescEqjDHGhQ13ZowxLqwHaYwxLqwHaYwxLqwHaYwxLqwHaYwxLpKtB5l010E+P/pR8nOXs2zpvLhtQ9cunVi9agFrcxYy5C5Prle1/EqSH8vsQCDApLnjePKlfwEw4on7mL3oDSa9O45J747j5NOaAtCwyQlMmDma7K3vc90t1/i6TRrF/xJZ0hXICRMmc2n3a+OWHwgEGPXESLr36EfzFufTp08vmjVravkpkB/r7Gtv7s2m9Vt+1vbYiKfpc+H19Lnwej5fvR6Ab/Z+wz+GPc74Z1/zbVtKqAYjnhKZLwVSRNqJyFHO6yNEZLiIzBCRf4hITT8yS3y48FN279nrZ0SZ2rZpycaNW9i8+QsKCwuZPHkal/XoavkpkB/L7OMy6vCbC89l6iszyp1395d7WL1sDUVFRb5sS7gYD5jrO796kGMJ3SQO8ARQE/iH0/aiT5kJoX5mPbblHhwqPzevgPr161l+CuTHMnvIQ4N5/KGnCR7SA7tt6ABef28C/zf8dtKrpvuSXRY/RvOJJ78KZEBVS/5z1VpVB6vqQlUdDpxY1oIiMkBEFovI4mBwv0+bZ0zl1eGic9n95R7WrPj8Z+2jRv6HnuddQ99uN1LzmKP4/a39Yr5t1oOsmFUiUjLE0HIRaQ0gIicBhWUtqKqjVbW1qrYOBKr7tHn+yc/bToOsg0P1Z2VmkJ+/3fJTID9W2We2OYNOXc5jdvab/OM/I2jT/iz+9tQDfLnzKwAKfypk2sRZnN7yVM+zy2M9yIq5CegoIhuBU4GPRWQT8LzzWdLKXryMJk0a0bBhA9LT0+nduyczZr5j+SmQH6vsUX/7D11a9eKSNldy9x//QvZHn3HvrcOpfdyxB+Y5v1sHNqzd5Hl2eYKqEU+JzJfrIFX1a+B650RNIycnV1V3+JEX7uWXnqZjh3OoXbsWWzYtZviIR3hx3ES/Yw8oLi5m0OBhzJ71KlUCAcaNn0ROzjrLT4H8eH/3vz/zIMccezQiwuer1vPQkH8CcGydWrw2ZyzVa1QnGAzS7+Y+XN6hL/v3fVfOGiOX6JftREoSuYubVjUzbhuXyk8VjHd+ojxVMN75cX2qoapEs2zdmqdE/G92x9dro8qKhaS7DtIYY7xitxoaYzyT6GelI2UF0hjjmUQ+ZBcNK5DGGM8k+lnpSFmBNMZ4xnqQxhjjwo5BGmOMC+tBGmOMCzsGaYwxLpLtThorkMYYz1gP0hhjXCTbMUi71dAY4xm/nkkjIt1E5HMR2SAiQ33+GgdYD9IY4xk/epAiUgV4GrgIyAWyRWS6quZ4HnYI60EaYzzj04C5bYENqrpJVX8CJgI9ff0ijoTuQZYM+2T5lp+K+ct3fBzX/Gj4dAQyE9gW9j4XaOdP1M8ldIGMdky6EiIyQFVHe7U5lSXb8i0/XvlFP+VF/G9WRAYAA8KaRsfztwuX7LvYA8qfJSmzLd/y451fYeHPoXKmQ4tjHtAg7H2W0+a7ZC+QxpjKLxtoKiKNRKQqcDUwPRbBib2LbYxJeapaJCK3AnOAKsBYVV0di+xkL5DxPI4R72Molm/5SUNVZwOzY52b0A/tMsaYeLJjkMYY4yIpC2S8bktysseKyE4RWRXL3LD8BiIyX0RyRGS1iAyKcf7hIrJIRJY7+cNjme9sQxURWSoiM2Od7eRvEZGVIrJMRBbHOPtoEXlDRNaKyBoRic+zY5NE0u1iO7clrSPstiTgmljcluTkdwD2ARNU9fRYZB6SnwFkqOoSEakBfAb0iuH3F6C6qu4TkXRgITBIVT+JRb6zDXcCrYGjVLV7rHLD8rcArVX1yzhkjwc+VNUXnDO+1VR1b6y3I1kkYw8ybrclAajqAmB3rPJKyS9Q1SXO62+BNYTuRIhVvqrqPudtujPF7L/CIpIFXAq8EKvMRCEiNYEOwBgAVf3JiuOvk4wFsrTbkmJWIBKJiDQEWgKfxji3iogsA3YCc1U1lvn/BoYAwRhmHkqBd0TkM+cukVhpBOwCXnQOMbwgItVjmJ90krFAGkBEjgTeBAar6jexzFbVYlU9k9AdD21FJCaHGkSkO7BTVT+LRV4ZzlPVVsDFwEDnsEsspAGtgGdVtSWwH4jpMfhkk4wFMm63JSUK59jfm8ArqjolXtvh7N7NB7rFKLI9cJlzDHAi0FlEXo5R9gGqmuf8uROYSuiwTyzkArlhPfY3CBVME6VkLJBxuy0pETgnScYAa1T1sTjk1xGRo53XRxA6WbY2Ftmqeo+qZqlqQ0J/7++par9YZJcQkerOyTGc3dsuQEyuaFDV7cA2ETnZaboAiMnJuWSVdHfSxPO2JAAReQ3oBNQWkVzgAVUdE6t8Qr2o3wErneOAAPc6dyLEQgYw3rmaIABMVtW4XG4TJ3WBqaH/TpEGvKqqb8cw/zbgFadzsAm4IYbZSSfpLvMxxhivJOMutjHGeMIKpDHGuLACaYwxLqxAGmOMCyuQxhjjwgqkMca4sAJpEoqIXC8iT8V7O4wBK5AmRpwLx42pVKxAmlKJyAgRGRz2fmRpg++KSCcRWSAis5xBiv8jIgHns30i8qiILAfOEZF+zmC6y0TkuZKiKSI3iMg6EVlE6E4gYxKCFUjjZixwHYBT8K4G3AZ+aEvoFrdTgcbAFU57deBTVW0BfAX0Ado7I/0UA9c6A/wOJ1QYz3PWYUxCSLp7sY03VHWLiHwlIi0J3V+8VFW/cpl9kapuggP3op9HaCSZYkKjCkFo4ISzgGznPuUjCI0X2Q54X1V3OctPAk7y51sZExkrkKYsLwDXA/UI9SjdHHpDf8n7H1S12HktwHhVvSd8RhHp9es30xh/2C62KctUQmM5tiE0OpKbts7wcgFCu9ELS5lnHvBbETkOQERqicgJhEY77ygixzrjWF7l6Tcw5lewHqRxpao/ich8YG9YT7A02cBTQBNCA+ROLWVdOSIyjNCjCAJAITBQVT8RkQeBj4G9wDJPv4Qxv4INd2ZcOYVsCXCVqq53macT8H/xeHqgMX6zXWxTKhE5FdgAzHMrjsYkO+tBmgoRkebAS4c0/6iq7eKxPcbEghVIY4xxYbvYxhjjwgqkMca4sAJpjDEurEAaY4wLK5DGGOPi/wHZC7AwKOoHIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 7.91 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Train the XGBoost algorithm\n",
    "import xgboost as xgb\n",
    "xg = xgb.XGBClassifier()\n",
    "\n",
    "X_train_x = X_train.values\n",
    "X_test_x = X_test.values\n",
    "\n",
    "xg.fit(X_train_x, y_train)\n",
    "\n",
    "y_pred = xg.predict(X_test_x)\n",
    "print(classification_report(y_test,y_pred))\n",
    "print(\"Accuracy of XGBoost: \"+ str(accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision of XGBoost: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Recall of XGBoost: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Average F1 of XGBoost: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n",
    "print(\"F1 of XGBoost for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n",
    "xg_f1=f1_score(y_test, y_pred, average=None)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm=confusion_matrix(y_test,y_pred)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3656\n",
      "           1       0.99      0.99      0.99       387\n",
      "           2       1.00      1.00      1.00        14\n",
      "           3       1.00      1.00      1.00       612\n",
      "           4       1.00      0.75      0.86         8\n",
      "           5       0.99      1.00      0.99       231\n",
      "           6       1.00      0.99      0.99       452\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       1.00      0.96      0.98      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n",
      "Accuracy of CatBoost: 0.996455223880597\n",
      "Precision of CatBoost: 0.9964590935743635\n",
      "Recall of CatBoost: 0.996455223880597\n",
      "Average F1 of CatBoost: 0.9964290021228678\n",
      "F1 of CatBoost for each type of attack: [0.99781241 0.99222798 1.         0.99591837 0.85714286 0.99137931\n",
      " 0.9944629 ]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxzElEQVR4nO3deXxU9fX/8dcZElQWUYRCSFAoYN2QRRYRBYQKqCBoK7igaG2pFi1oC6LVIlr6pa2oULU/QVY3CAqyiAIiilgFAoQtILuShUVRBKyY5fz+mJswQibJDPfODDPn6eM+mHzm3nnfO5jD526fK6qKMcaYE/mivQLGGBOrrEAaY0wQViCNMSYIK5DGGBOEFUhjjAnCCqQxxgSRFO0VKJOIXYNkTDSoSjiL5X+1I+Tf2eRaPw8rKxJiukDm798etezk2o1ISq4XtfyC/NyEzS/IzwWw/CjnmxgvkMaYU0xRYbTXwFVWII0x7tGiaK+Bq6xAGmPcU2QF0hhjSqXWgzTGmCDirAdp10EaY9yjRaFP5RCR00VkhYisFZGNIjLCaZ8sIjtFJNOZmjvtIiJjRWSbiKwTkZYBn9VfRLY6U//ysq0HaYxxjzdnsY8CnVX1sIgkA8tE5F3nvSGq+uZx818LNHGmtsB/gLYiUhMYDrQCFFglInNU9ZtgwdaDNMa4x4MepPoddn5MdqayLkjvBUx1lvsMOEtEUoBuwCJVPeAUxUVA97KyrUAaY9xTVBT6VAEiUklEMoF9+Ivccuetkc5u9LMicprTlgrsDlg822kL1h6UFUhjjGtUi0KeRGSAiGQETANO/FwtVNXmQBrQRkQuAR4BLgBaAzWBh93eHiuQxhj3hNGDVNVxqtoqYBoX7ONV9VtgCdBdVfOc3eijwCSgjTNbDlA/YLE0py1Ye1BWII0x7vHmLHZtETnLeX0GcA2w2TmuiIgI0BvY4CwyB7jTOZt9OXBQVfOABUBXETlbRM4GujptQdlZbGOMe7w5i50CTBGRSvg7demqOk9EPhCR2oAAmcC9zvzzgeuAbcD3wN0AqnpARJ4CVjrzPamqB8oKPqV6kEeP/sgtvx3ETf3/QK/bf8/zL78CgKoy5qXJXH/Lb+l52wBenTH7J8ut3/Q5zTpcz8IlH5e0PfPiBHr3u5fe/e7l3fc/cm0dx48bTW72WjLXLHbtM0PVrWsnNm5YyuasZQwdMjCi2Ym+/aeddhqffjKPVRmLWJv5AcP/+qeIZael1eP9hTNYt3YJazM/4IH774lYdglvzmKvU9UWqnqpql6iqk867Z1VtanT1q/4TLez2z1QVRs572cEfNZEVW3sTJPKyz6lepCVKyczcewoqlQ5g/yCAu68789cdXkrdnyxmz37vmLu6+Pw+Xx8/c23JcsUFhby7IuTuKJ1ybWifPTfFWR9vp03J7/Aj/n53H3/UK5q14pqVaue9DpOnZrOiy9OYtKkMSf9WeHw+XyMHTOS7tfdSnZ2Hp99Op+58xayadPWiOQn+vYfPXqUX3btw5Ej35OUlMTSD2fx3ntLWL5itefZBQUFDBk6gjWZG6hWrSorlr/H+4uXRmzb45FnPUgRuUBEHnauaB/rvL7wJD+TKlXOAPz/MxQUFCAiTJ/1DvfdfRs+n39zzjn7rJJlXn9zDtd0ak/NgLbtO7+kVfNLSEqqRJUzTuf8xg1Z9tmqk1m1Eh8vW86BgAIdaW1at2D79l3s3Pkl+fn5pKfP5oae3SKWn+jbD3DkyPcAJCcnkZScTKSePb9nzz7WZPoPwx0+fITNm7eSWq9uRLJLeHSZT7R4UiBF5GFgGv5jAyucSYA3RGTYyXx2YWEhv+o/kA49bqVd6xZcevEF7M7J493FH9HnN3/k3j89zhe7/Sem9u7/isVL/0vfG6//yWf8onFDli1fxf9++IFvvj3IytXr2LNv/8msVsyol1qX3dnHBjzNzsmjXqR/SaIoFrbf5/ORsXIheTnrWLx4KStWroloPsB556XRvNklLF8R4WwPdrGjyatd7HuAi1U1P7BRRJ4BNgKjwv3gSpUq8daUF/ju0GEGPfIUW3fs4sf8fE6rXJn0iWNZ9OEnPP73Z5n6n6f5x5iXePC+35T0LIu1b3sZGzZvod/v/8TZZ9Wg2cUXUMl3Sh2ONTGsqKiIVq27UqPGmbw1YwIXX/wLNm78PGL5VatWIX36eB7683AOHTpc/gJuivEeYai8qgpFQGnjxac47wUVeNHoy1PfCDrfmdWr0ablpSz7LIO6tWvxy47tAfhlxyvYsn0nABs3b2XI8FF0/VV/Fn64jL89/QKLl/4XgN/3v5W3przAy2P+jgLn1S/zgvpTRm7OHuqnHfvq01JTyM3dE8U1iqxY2v6DB7/jw48+oVvXThHLTEpKYsb08bzxxizefvvd8hdwmWphyFMs86oHORhYLCJbOXZrz7lAY+D+shZ0LhIdByc+AOjAN9+SlJTEmdWr8cPRo3y6cg2/6XcznTu0Y8XqtaTVq8vKNetLit2CNyeXLPuXv42mY/s2dOlwBYWFhRw6fISzapzJ59t2smXbTq547M/ubHmUrczIpHHjhjRoUJ+cnD306dOLO+6M7JncaIr29teqVZP8/AIOHvyO008/nV926cC/nn4xYvnjx41m0+ZtPDcm6LXW3orxXeZQeVIgVfU9ETkf/5XtxV2zHGClnsQ/Gfu//oa//O1pCouK0CKlW+er6NS+LS0vvZiHR/yTV6a/TZUzTmfEsMFlfk5BQSF3/sFfEKtVqcKovw4hKalSuKv1E6++8gIdO7SjVq2a7NqRwYgnn2bS5GmufHZFFBYWMmjwY8x/53Uq+XxMnjKdrKwtEctP9O1PSanDxAnPUamSD5/Px5tvzuWd+e9HJLv9Fa25o9+vWbc+i4yVCwF4/PFRvPveBxHJB+JuF1sidYYtHOE8QtIt9lRDe6phQueH+djXH1a9HfLv7OmX9bbHvhpjEoA91dAYY4KwY5DGGBNEnB2DtAJpjHGP9SCNMSYI60EaY0wQViCNMaZ0sX5nTKisQBpj3GM9SGOMCSLOTtLYEDbGGBOE9SCNMe6xXWxjjAkiznaxrUAaY9xjPUhjjAnCepCRk1y7UVTzi4edsnzLT8T8sFgP0hhjgrACGTnRHjD2gtqtopa/eX9G1Lc/oQeMtfzw2C62McYEYT1IY4wJwnqQxhgThPUgjTEmiDjrQdq92MYY9xQVhT6VQ0ROF5EVIrJWRDaKyAinvaGILBeRbSIyXUQqO+2nOT9vc95vEPBZjzjtn4tIt/KyrUAaY9zjQYEEjgKdVbUZ0BzoLiKXA/8AnlXVxsA3wD3O/PcA3zjtzzrzISIXAbcAFwPdgRdFpFJZwVYgjTHuUQ19KvcjVVX1sPNjsjMp0Bl402mfAvR2XvdyfsZ5v4uIiNM+TVWPqupOYBvQpqxsK5DGGPeE0YMUkQEikhEwDTj+Y0WkkohkAvuARcB24FtVLXBmyQZSndepwG4A5/2DwDmB7aUsUyo7SWOMcU8YZ7FVdRwwrpx5CoHmInIWMAu4IJzVC5X1II0x7tGi0KdQPl71W2AJ0A44S0SKO3lpQI7zOgeoD+C8XwP4OrC9lGVKZQXSGBPTRKS203NERM4ArgE24S+Uv3Zm6w/Mdl7PcX7Gef8DVVWn/RbnLHdDoAmwoqxs28U2xrjHmwvFU4ApzhlnH5CuqvNEJAuYJiJ/A9YAE5z5JwCviMg24AD+M9eo6kYRSQeygAJgoJbzGMa47UH6fD5WrljA7FlTyp85DJVPq0z6e5N5e8lrzF06nQeG+o8rX35Va956/xVmffAar80dz7kN0wDo2/8m5nz4Rkl7o/MberJeAN26dmLjhqVszlrG0CEDPcux/NjKjoV8j85ir1PVFqp6qapeoqpPOu07VLWNqjZW1ZtV9ajT/oPzc2Pn/R0BnzVSVRup6i9U9d3ysuO2QP7xgd+yefNWzz7/x6M/ctev7qP31bdzY+fbuPLqdjS77BKe+OfDDLnvcW7sfDvzZi7gvgf9l2bNe2sBN3S6lRs7387Lz7/CsCcf9GS9fD4fY8eMpEfPfjRtdjV9+/bmwgubeJJl+bGTHQv5gFfXQUZNXBbI1NQUrru2CxMnvuFpzvdH/gdAUnISSclJqCqqUK16VQCqV6/Gvj37AThy+EjJclWqnI5W4F/OcLRp3YLt23exc+eX5Ofnk54+mxt6lnvDgOWf4tmxkA/EXYGMy2OQz4wewbBH/kb16tU8zfH5fLz1/iuc2zCN1yfOYN3qjTz24N8Y9/pz/PDDUQ4fOkLfa39TMv9tv7mZu+69jeTkZO666T5P1qleal12Zx8bzy87J482rVt4kmX5sZMdC/mA3YvtBhG526vPvv66X7Jv31esXrPeq4gSRUVF3Nj5djo1u55LW1xMkwsa0f/e2xhw22A6Ne/BzGlzGfbk4JL5X584g65tbmT0U//mvod+E/yDjTlFaZGGPMWyaO1ijwj2RuBV9UVFR4LNFtQVV7SiZ4+ubNvyGa+9+iJXX92eKZPHntTKlufQd4dZ/skqrurSjgsubsK61RsBePftRbRofekJ878zayFdru3kybrk5uyhftqxkajTUlPIzd3jSZblx052LOQDcbeL7VmBFJF1Qab1QJ1gy6nqOFVtpaqtfL6qIef+5bFRNPh5Kxqffzm39/sDS5Z8Qv+7/ngym1Kqs885i+pn+nfhTzv9NK7o2IYdW3ZRvXo1Gvz8XACu6NiWHVt3AXBew2PXp3a65kq+2PGl6+sEsDIjk8aNG9KgQX2Sk5Pp06cXc+ct9CTL8mMnOxbyAc8vFI80L49B1gG64R9lI5AA//UwNyJq16nFqH8/QaVKPkR8vDfnfT5ctIzH/zSSsRP/QZEW8d23h3h08FMA3H5PH9p1aENBQQHfffsdwx4I2ok+KYWFhQwa/Bjz33mdSj4fk6dMJytriydZlh872bGQD0CM7zKHSrw6myoiE4BJqrqslPdeV9XbyvuMpMqpUfu27aFd9tCuhM5XlXCW/f7ffwj5d7bKAy+GlRUJnvUgVfWeMt4rtzgaY05BMX5MMVRxeZmPMSZKPNojjRYrkMYY91gP0hhjgoizkzRWII0x7onxy3ZCZQXSGOOeOOtBxuVgFcYY4wbrQRpjXKN2ksYYY4KIs11sK5DGGPfYSRpjjAnCepDGGBOEHYM0xpggrAdpjDFB2DFIY4wJwnqQkVM8Ll60bN6fEdX8aG+/5Sd2fjjsOsgIStQBYxM9PyYGjLX88FgP0hhjgrACaYwxQdhJGmOMCSLOepA2mo8xxjVapCFP5RGR+iKyRESyRGSjiAxy2p8QkRwRyXSm6wKWeUREtonI5yLSLaC9u9O2TUSGlZdtPUhjjHu86UEWAH9S1dUiUh1YJSKLnPeeVdWnA2cWkYuAW4CLgXrA+yJyvvP2C8A1QDawUkTmqGpWsGArkMYY93hwmY+q5gF5zutDIrIJSC1jkV7ANFU9CuwUkW1AG+e9baq6A0BEpjnzBi2QtottjDlliEgDoAWw3Gm6X0TWichEETnbaUsFdgcslu20BWsPygqkMcY9RRryJCIDRCQjYBpQ2keLSDXgLWCwqn4H/AdoBDTH38Mc7fbm2C62McY9YRyDVNVxwLiy5hGRZPzF8TVVnekstzfg/fHAPOfHHKB+wOJpThtltJfKepDGGNeoashTeUREgAnAJlV9JqA9JWC2G4ENzus5wC0icpqINASaACuAlUATEWkoIpXxn8iZU1a29SCNMe7x5ix2e+AOYL2IZDptjwK3ikhzQIFdwO8BVHWjiKTjP/lSAAxU1UIAEbkfWABUAiaq6saygq1AGmPc40GBVNVlgJTy1vwylhkJjCylfX5Zyx3PCqQxxjUVufD7VBKXxyC7de3Exg1L2Zy1jKFDBsZ9/vhxo8nNXkvmmsUnvPfg4N9T8GMO55xzdilLeiPRvv9YyY6F/HDOYseyuCuQPp+PsWNG0qNnP5o2u5q+fXtz4YVN4jp/6tR0ru9x+wntaWn1uOaXHfjii2xP8wMl4vcfC9mxkA9AURhTDIu7AtmmdQu2b9/Fzp1fkp+fT3r6bG7o2a38BU/h/I+XLefAN9+e0D766ScY9ujICp0pdEsifv+xkB0L+eDNvdjR5FmBFJELRKSLc3FnYHt3rzIB6qXWZXf2sQE/s3PyqFevrpeRMZVfrGfPruTk5LFuXdC7qDwR7e2PZn4ib3sJ28Uun4j8EZgNPABsEJFeAW//3YtMc8wZZ5zOIw8/wBMjni5/ZmPcFGe72F6dxf4dcJmqHnbunXxTRBqo6hhKP11fwrnNaACAVKqBz1c1pODcnD3UTzs2VH1aagq5uXtCXP3wRTsfoFGjBjRocC6rM/wDnqSlpbBy+QLatb+evXv3e5od7e2PZn4ib3uxWN9lDpVXu9g+VT0MoKq7gE7AtSLyDOUUSFUdp6qtVLVVqMURYGVGJo0bN6RBg/okJyfTp08v5s5bGPoWhCna+QAbNmymXlozGp9/OY3Pv5zs7Dxat+3meXGE6G9/NPMTedtLWA+yQvaKSHNVzQRwepI9gIlAU48yASgsLGTQ4MeY/87rVPL5mDxlOllZW7yMjHr+q6+8QMcO7ahVqya7dmQw4smnmTR5mqeZwSTi9x8L2bGQD/HXgxQvznCKSBpQoKon9O9FpL2qflKRz0mqnBq1bzuRnyoY7fxYeapfQuerlrmnF8yBXh1D/p2tOfujsLIiwZMepKoGvfCuosXRGHPqibNndsXfdZDGGOMWuxfbGOOeOOtBWoE0xrgm3naxrUAaY9xjBdIYY0pnPUhjjAnCCqQxxgRhBdIYY4IJ7/rymGUF0hjjGutBGmNMEFpkPUhjjCmV9SCNMSaIMMe4iFlWII0xrrEepDHGBGHHICOoeFw8y7d8yz81RPABmhER0wUyUQeMjZX8TqldopL9Yc5iIMEHrI2B/HBYD9IYY4KItwJpA+YaY2KaiNQXkSUikiUiG0VkkNNeU0QWichW58+znXYRkbEisk1E1olIy4DP6u/Mv1VE+peXbQXSGOMa1dCnCigA/qSqFwGXAwNF5CJgGLBYVZsAi52fAa4FmjjTAOA/4C+owHCgLdAGGF5cVIOxAmmMcY0WSchTuZ+pmqeqq53Xh4BNQCrQC5jizDYF6O287gVMVb/PgLNEJAXoBixS1QOq+g2wCOheVrYdgzTGuMbrC8VFpAHQAlgO1FHVPOetPUAd53UqsDtgsWynLVh7UOX2IEWkjohMEJF3nZ8vEpF7yt8UY0yi0aLQJxEZICIZAdOA0j5bRKoBbwGDVfW7n+T6n1/t+kVGFdnFngwsAIqvOdgCDHZ7RYwxp74ilZAnVR2nqq0CpnHHf66IJOMvjq+p6kynea+z64zz5z6nPQeoH7B4mtMWrD2oihTIWqqajvO0CVUtAAorsJwxJsGoSshTeUREgAnAJlV9JuCtOUDxmej+wOyA9juds9mXAwedXfEFQFcROds5OdPVaQuqIscgj4jIOTjd1+LACixnjEkwHl0H2R64A1gvIplO26PAKCDdOeT3BdDHeW8+cB2wDfgeuBtAVQ+IyFPASme+J1X1QFnBFSmQD+GvyI1E5BOgNvDrim2XMSaReHGroaouA4JV3hNu93KORw4M8lkTgYkVzS63QKrqahHpCPzCWcnPVTW/ogHGmMQRb3fSlFsgReTO45paigiqOtWjdTLGnKKKEnA8yNYBr0/H36VdDViBNMb8RLwNmFvuWWxVfSBg+h3QEqjm/aqFJy2tHu8vnMG6tUtYm/kBD9wf+Us2u3XtxMYNS9mctYyhQ0o9FHLK51c7syojXvorUz+cyJQlE7io5YV0vL4Dkxa/zAdfLuQXl57/k/lvG3grry2bwtSPJtG6YytP1qlYNL//RPi7L4tHtxpGTTi3Gh4BGrq9Im4pKChgyNARXNrsatpf2ZP77ruLCy9sErF8n8/H2DEj6dGzH02bXU3fvr3jMv/+EQNZ8eFK7uz0G+7p+nu+3PYlOz/fxV9/9wTrlq//ybznNTmXzr06cVfn3zK03yMMHvlHfD5v7nKN5vefKH/3ZQnnOshYVpE7aeaKyBxnmgd8DsyqwHJtRKS18/oiEXlIRK47+VUu2549+1iTuQGAw4ePsHnzVlLr1fU6tkSb1i3Yvn0XO3d+SX5+Punps7mhZ7e4yq9avSrN2jblnTfeBaAgv4DD3x3hy21fsntH9gnzt+/ang9mf0j+j/ns2b2HnF25XND8F66uU7Fofv+J8HdfHi+ug4ymihyDfDrgdQHwhaqe+FsQQESG4x9RI0lEFuEfPWMJMExEWqjqyHBXOBTnnZdG82aXsHzFmkjEAVAvtS67s48NOJqdk0eb1i3iKj+lfl2+PXCQYc8ModFFjdiyfgv//uuL/PC/H0qdv3bKOWSt3lTy8/49+6mdUsvVdSoWze8/Ef7uyxPru8yhKrMHKSKVgCdU9SNn+qS84uj4Nf6LOzvgvx6pt6o+hX80jb4nu9IVUbVqFdKnj+ehPw/n0KHDkYhMGJWSKnH+JU2Y/cpcftf9Xv73/Q/cNvCWaK+WiQEJtYutqoVAkYjUCPFzC1S1UFW/B7YX31iuqv/DuWUxmMAb14uKjoQY65eUlMSM6eN5441ZvP32u2F9Rrhyc/ZQP+3YUPlpqSnk5u6Jq/z9efvZn7efTWs2A/DRO0tp0jT4sa79eV9TO+VnJT/Xrlub/XlfubpOxaL5/SfC33154m0XuyJHyg/jv8VngjNK71gRGVvOMj+KSBXn9WXFjU6hLbNABt647vNVrcDqnWj8uNFs2ryN58accM+751ZmZNK4cUMaNKhPcnIyffr0Yu68hXGVf2D/N+zL3U/9n6cBcNmVLfli6xdB5//vov/SuVcnkisnU7d+XdIaprI583NX16lYNL//RPi7TzQVOQY505kClXekoYOqHgVQ/cmTcpM5dnO5J9pf0Zo7+v2adeuzyFjp/5/j8cdH8e57H3gZW6KwsJBBgx9j/juvU8nnY/KU6WRlbYlIdiTzxz7+PI/9+xGSKieT90Ueo/70L67s3p5BT91PjZo1+L8pI9m2cTtD+w1j15Yv+HDuR0z+YAKFhYU899hYioq8eYByNL//RPm7L0us7zKHSrSco6oiMkhVx5TX5oWkyqlRO+QbC08VjHa+PdUwgfPD3Pf9rN5NIf/OXp47M2arakV2sUvr8d3l8noYY+JAvJ2kCbqLLSK3ArcBDUVkTsBb1YEyhwgyxiSmWD/pEqqyjkH+F8gDagGjA9oPAeu8XCljzKnJmyPL0RO0QKrqF/gHoWxX1geIyKeqWuY8xpjEoEGHbTw1ufFUw9Nd+AxjTBwoirM7adwokHH2lRhjwlVkPUhjjCldvO1iV2Q0nwecJ4AFncXF9THGnMKKwphiWUWug6wDrBSRdBHp7jyCMdAdHqyXMeYUpEjIUyyryIjijwFN8D+X9i5gq4j8XUQaOe9v8HQNjTGnjETsQRY/RnGPMxUAZwNvisg/PVw3Y8wpJt4KZEWeajgIuBP4CngZGKKq+SLiA7YCQ71dRWPMqSLWd5lDVZGz2DWBm5wLx0uoapGI9PBmtYwxp6I4eyx2+QVSVYeX8d6mYO8ZYxJPvF0H6c2j5YwxJg7E9IXixePiWX50FI/LGC3R3v5Ezw9HvN1WF9MFMtoDxlp+Ag8Ya/lh8eKstIhMBHoA+1T1EqftCeB3wH5ntkdVdb7z3iPAPUAh8EdVXeC0dwfGAJWAl1V1VHnZMV0gjTGnlqIT7iNxxWTgeWDqce3PqmrgY6kRkYuAW4CLgXrA+yJyvvP2C8A1QDb+m1/mqGpWWcFWII0xrvFiF1tVl4pIgwrO3guY5jwTa6eIbAPaOO9tU9UdACIyzZm3zAJpJ2mMMa6J8IXi94vIOhGZGDBeRCqwO2CebKctWHuZrEAaY1xTJKFPIjJARDICpgEViPoP0Ahojv/JB6PLnDtMtottjHFNONdBquo4IKSH2Kvq3uLXIjIemOf8mAPUD5g1zWmjjPagrAdpjHGNhjGFQ0RSAn68ESgeNGcOcIuInCYiDfEPtLMCWAk0EZGGIlIZ/4mcwIcRlsp6kMYY13hxq6GIvAF0AmqJSDYwHOgkIs3x19hdwO8BVHWjiKTjP/lSAAxU1ULnc+4HFuC/zGeiqm4sL9sKpDHGNV5cB6mqt5bSPKGM+UcCI0tpnw/MDyXbCqQxxjV2J40xxgSRcKP5GGNMRcX6ALihsgJpjHGNFUhjjAlC42wXO+6ugxw/bjS52WvJXBOdobqinQ/QrWsnNm5YyuasZQwdMjDh8mvUOJPp08axYf1HrF/3IZe3vSxi2dHe9mjnx9szaeKuQE6dms71PW5P2Hyfz8fYMSPp0bMfTZtdTd++vbnwwiYJkw/w7DNPsmDBEi5p2pGWl13Dps1bI5Ib7W2Pdn48iliBFJHjhyryxMfLlnPgm28jERWT+W1at2D79l3s3Pkl+fn5pKfP5oae3RIm/8wzq3PVlW2ZOOkNAPLz8zl48LuIZEd726OdD9aDrBARmXPcNBe4qfhnLzKNX73UuuzOPjbgaXZOHvXq1U2Y/IYNz+Wrr75mwsvPsnLFAl76f/+iSpUzIpId7W2Pdj5E7lbDSPGqB5kGfAc8g3+UjdHAoYDXxngiqVIlWrRoyksvTaV1m24cOfI9Dw+9P9qrlTDCGc0nlnlVIFsBq4C/AAdV9UPgf6r6kap+VNaCgUMfFRUd8Wj14lduzh7qpx0bqj8tNYXc3D0Jk5+dk0d2dh4rVq4BYObMd2jRvGlEsqO97dHOB9vFrhBVLVLVZ4G7gb+IyPNU8JIiVR2nqq1UtZXPV9WL1YtrKzMyady4IQ0a1Cc5OZk+fXoxd97ChMnfu3c/2dm5nH9+IwA6d76STZu2RCQ72tse7XyIvwLp6XWQqpoN3Cwi1+Pf5fbcq6+8QMcO7ahVqya7dmQw4smnmTR5WiSiYyK/sLCQQYMfY/47r1PJ52PylOlkZUWmQMRCPsCgBx9n6pR/U7lyMjt3fsk9v30oIrnR3vZo50PsH1MMlajG7iYlVU6N2sol8lMFo50fK0/1S+h8De+S73+e1y/k39mhX7was0ci7U4aY4xrYn2XOVRWII0xrond/dHwWIE0xrimKM5KpBVIY4xrbBfbGGOCiK/+oxVIY4yLrAdpjDFBxPqtg6GyAmmMcY2dpDHGmCDiqzzG4YC5xhjjFutBGmNcYydpjDEmCDsGaYwxQcRXebQCaYxxke1iR1DxsE+Wb/mWf2qIt11sO4ttjHGNFw/tEpGJIrJPRDYEtNUUkUUistX582ynXURkrIhsE5F1ItIyYJn+zvxbRaR/RbYnpnuQiTpgbKLnx8SAsUBKjQujkp93cBMQ/e0Ph0e72JOB54HAR0cPAxar6igRGeb8/DBwLdDEmdoC/wHaikhNYDj+52UpsEpE5qjqN2UFWw/SGOMaDeO/cj9TdSlw4LjmXsAU5/UUoHdA+1T1+ww4S0RSgG7AIlU94BTFRUD38rKtQBpjXBPOQ7sCn2TqTAMqEFVHVfOc13uAOs7rVGB3wHzZTluw9jLF9C62MebUEs5JGlUdB4wLN1NVVUQ8OTtkPUhjjGu8OEkTxF5n1xnnz31Oew5QP2C+NKctWHuZrEAaY1xThIY8hWkOUHwmuj8wO6D9Tuds9uXAQWdXfAHQVUTOds54d3XaymS72MYY13hxFltE3gA6AbVEJBv/2ehRQLqI3AN8AfRxZp8PXAdsA74H7gZQ1QMi8hSw0pnvSVU9/sTPCaxAGmNcU5Gz0iF/puqtQd7qUsq8CgwM8jkTgYmhZFuBNMa4xm41NMaYILzoQUaTnaQxxpggrAdpjHGN7WIbY0wQRRpfu9hWII0xromv8hiHxyDHjxtNbvZaMtcsjto6dOvaiY0blrI5axlDh5R6xYHln8L59VLr8ubcSXz02Vw+/HQOv723HwBD//IAiz+ZxaKPZzJt5njq1K1dssxT/3iU/65+j8WfzKJpM+9GCYr2dx/BC8UjIu4K5NSp6Vzf4/ao5ft8PsaOGUmPnv1o2uxq+vbtzYUXNrH8OMovKChgxGP/pOPlPbn+mlu467e3cf4vGvHi2Il0aX8j11x1E4sWfMRDQ/8AQOdrOvDzn5/HFS27M2TQcEaNHu7q+hSL9ncP3ozmE00RKZAicqWIPCQiXb3O+njZcg58863XMUG1ad2C7dt3sXPnl+Tn55OePpsbenaz/DjK37f3K9av9Y/ZeOTw92zdsoO6KT/j8KEjJfNUqXIG6hyP635dZ2ZM898JtzpjHWfWqM7P6tRydZ0g+t89hDeaTyzzpECKyIqA17/DP9hldWC4M7hl3KqXWpfd2ccGHM3OyaNevbqWH6f5aefWo2nTC1m9ah0Awx4bRMaGxdx0cw/+9fd/A1A35Wfk5uwpWSYvdy8pKXVK/byTEe3vHmwXu6KSA14PAK5R1RH4bxCP3v6vMS6qUrUKE6aO4a+P/l9J73HU38bQ6pIuzJwxj7sHJN7/6raLXcHPdUbNOAcQVd0PoKpHgIKyFgwcPLOo6EhZs8ak3Jw91E87NlR+WmoKubl7yljC8k/F/KSkJCZMfY6ZM+Yxf+77J7w/c8Y8ru95DQB78vZRL/VYTy6lXh3y8va6vk7R/u7BdrErqgawCsgAagaM21YNkLIWVNVxqtpKVVv5fFU9Wj3vrMzIpHHjhjRoUJ/k5GT69OnF3HkLLT/O8p95/im2btnBSy9MKWlr+PPzSl53u64z27buAGDBux9w8y29AGjZ6lIOfXeIfXu/cn2dov3dA6hqyFMs8+Q6SFVtEOStIuBGLzKLvfrKC3Ts0I5atWqya0cGI558mkmTp3kZ+ROFhYUMGvwY8995nUo+H5OnTCcra4vlx1F+m8tbcvMtvcja+DmLPp4JwP89+Ry33XETjRo3pEiLyN6dy8MPjgBg8cKldLmmA5+ueY//ff8DDw78i6vrUyza3z3E32NfJZYreFLl1KitXCI/VTDa+fZUwxh4qqFqmXt6wfQ8t0fIv7Nzv5wXVlYk2J00xhjXxPpJl1BZgTTGuCbedrGtQBpjXBPLh+zCYQXSGOOaWL9sJ1RWII0xrom3Y5BxN1iFMca4xXqQxhjX2EkaY4wJwk7SGGNMENaDNMaYIOLtJI0VSGOMa+yhXcYYE0R8lUcrkMYYF8XbMUi7DtIY4xqvHrkgIrtEZL2IZIpIhtNWU0QWichW58+znXYRkbEisk1E1olIy3C3J6Z7kMXDTlm+5UdD8bBj0RLt7Q+Hx5f5XK2qgSMNDwMWq+oo51lXw4CHgWuBJs7UFviP82fIrAdpjHFNhB/a1QsoHtJ9CtA7oH2q+n0GnFX8VINQxXQPMlEHjE30/FgZMDfa+c3qtItK/tq9n4a9rIeX+SiwUEQUeElVxwF1VDXPeX8PUPyoyFRgd8Cy2U5bHiGK6QJpjDm1hLOLLSID8D/9tNg4pwAGulJVc0TkZ8AiEdl8XK46xdNVViCNMa4JZ5fZKYbHF8Tj58lx/twnIrOANsBeEUlR1TxnF3qfM3sOUD9g8TSnLWR2DNIY4xovnmooIlVFpHrxa6ArsAGYA/R3ZusPzHZezwHudM5mXw4cDNgVD4n1II0xrvHoOsg6wCwRAX/Nel1V3xORlUC6iNwDfAH0ceafD1wHbAO+B+4ON9gKpDHGNV6cpFHVHUCzUtq/BrqU0q7AQDeybRfbGGOCsB6kMcY1NliFMcYEYcOdGWNMENaDNMaYIKwHaYwxQVgP0hhjgrAepDHGBBFvPci4uw5y/LjR5GavJXPN4qitQ7eundi4YSmbs5YxdIgr16ueMvlpafV4f+EM1q1dwtrMD3jg/nsimg/R3f5IZvt8PqYvmsy/X/nXT9of/tuDfLr9/ZKfU9LqMm7GWGZ8MJWXZz7Pz1Jqe7ZOGsZ/sSzuCuTUqelc3+P2qOX7fD7GjhlJj579aNrsavr27c2FFzZJmPyCggKGDB3Bpc2upv2VPbnvvrsSZvsjnX377/qwY+uun7Rd1OwCzqxR/SdtDw2/n7kz3uXmzncybvQkBj16n2frpFoU8hTLPCmQItJWRM50Xp8hIiNEZK6I/ENEaniRWezjZcs58M23XkaUqU3rFmzfvoudO78kPz+f9PTZ3NCzW8Lk79mzjzWZGwA4fPgImzdvJbVe3YjlR3P7I5n9s5TaXPXLK5j12tySNp/Px0N/HcizT73wk3kbnd+AFctWAbDik1V06n6VJ+sEER8w13Ne9SAn4r9JHGAMUAP4h9M2yaPMmFAvtS67s48NlZ+dk0e9CBaIaOcHOu+8NJo3u4TlK9ZELDOa2x/J7KFPDebZp16gKKAHdstvfs2HC5bx1b6vfzLv5xu30eW6TgB0ua4j1apXpcbZZ3qyXl6M5hNNXhVIn6oWOK9bqepgVV2mqiOAn5e1oIgMEJEMEckoKjri0eoZr1WtWoX06eN56M/DOXTocLRXJ650uOYKDnz1DZvWfV7SVrtOLbr2vJo3Jrx5wvzPjHieVu2aM33RZC5r14K9ufsoKvRm1zbeepBencXeICJ3q+okYK2ItFLVDBE5H8gva8HAwTOTKqfG9rdXitycPdRPOzZUf1pqCrm5exImHyApKYkZ08fzxhuzePvtdyOaHc3tj1R289aX0qnrlVzZpR2nnVaZqtWqMvOjV/nxx3zmfpYOwOlnnM7cT9Pp2a4P+/d+xUP3PArAGVXO4JfXd+LQd978oxXrPcJQedWD/C3QUUS2AxcBn4rIDmC8817cWpmRSePGDWnQoD7Jycn06dOLufMWJkw++K8k2LR5G8+NKXOQaE9Ec/sjlT327/+Pri17c13rX/HwvX9l5SeruOqC7nS5tCfXtf4V17X+FT/87wd6tvMPj3hWzRo4Yylyzx/v5O1p81xfp2JFqiFPscyTHqSqHgTuck7UNHRyslV1rxd5gV595QU6dmhHrVo12bUjgxFPPs2kydO8ji1RWFjIoMGPMf+d16nk8zF5ynSysrYkTH77K1pzR79fs259Fhkr/cXh8cdH8e57H0QkP5rbH+3vPphWV7Tkj4/eC6qs+iyTvz8y2rOsWL9sJ1QSy13iaO5iJ/JTBaOdHytPFYx2flSfaqgq4Sxbp8YFIf/O7j24OaysSIi76yCNMcYtdquhMcY1sX5WOlRWII0xronlQ3bhsAJpjHFNrJ+VDpUVSGOMa6wHaYwxQdgxSGOMCcJ6kMYYE4QdgzTGmCDi7U4aK5DGGNdYD9IYY4KIt2OQdquhMcY1Xj2TRkS6i8jnIrJNRIZ5vBklrAdpjHGNFz1IEakEvABcA2QDK0VkjqpmuR52HOtBGmNc49EjF9oA21R1h6r+CEwDenm6IY6Y7kEWD/tk+ZafiPlr934a1fxweHQEMhXYHfBzNtDWm6ifiukCGe6YdMVEZIDzCIeIi2a25Vt+tPILfswJ+XdWRAYAAwKaxkXzuwsU77vYA8qfJS6zLd/yo51fYao6TlVbBUzHF8ccoH7Az2lOm+fivUAaY059K4EmItJQRCoDtwBzIhEc27vYxpiEp6oFInI/sACoBExU1Y2RyI73AhnN4xjRPoZi+ZYfN1R1PjA/0rkx/dAuY4yJJjsGaYwxQcRlgYzWbUlO9kQR2SciGyKZG5BfX0SWiEiWiGwUkUERzj9dRFaIyFonf0Qk8511qCQia0RkXqSznfxdIrJeRDJFJCPC2WeJyJsisllENolIdJ4dGyfibhfbuS1pCwG3JQG3RuK2JCe/A3AYmKqql0Qi87j8FCBFVVeLSHVgFdA7gtsvQFVVPSwiycAyYJCqfhaJfGcdHgJaAWeqao9I5Qbk7wJaqepXUcieAnysqi87Z3yrqOq3kV6PeBGPPcio3ZYEoKpLgQORyislP09VVzuvDwGb8N+JEKl8VdXDzo/JzhSxf4VFJA24Hng5UpmxQkRqAB2ACQCq+qMVx5MTjwWytNuSIlYgYomINABaAMsjnFtJRDKBfcAiVY1k/nPAUKAogpnHU2ChiKxy7hKJlIbAfmCSc4jhZRGpGsH8uBOPBdIAIlINeAsYrKrfRTJbVQtVtTn+Ox7aiEhEDjWISA9gn6quikReGa5U1ZbAtcBA57BLJCQBLYH/qGoL4AgQ0WPw8SYeC2TUbkuKFc6xv7eA11R1ZrTWw9m9WwJ0j1Bke+AG5xjgNKCziLwaoewSqprj/LkPmIX/sE8kZAPZAT32N/EXTBOmeCyQUbstKRY4J0kmAJtU9Zko5NcWkbOc12fgP1m2ORLZqvqIqqapagP8f+8fqGq/SGQXE5GqzskxnN3brkBErmhQ1T3AbhH5hdPUBYjIybl4FXd30kTztiQAEXkD6ATUEpFsYLiqTohUPv5e1B3Aeuc4IMCjzp0IkZACTHGuJvAB6aoalcttoqQOMMv/7xRJwOuq+l4E8x8AXnM6BzuAuyOYHXfi7jIfY4xxSzzuYhtjjCusQBpjTBBWII0xJggrkMYYE4QVSGOMCcIKpDHGBGEF0sQUEblLRJ6P9noYA1YgTYQ4F44bc0qxAmlKJSJPisjggJ9Hljb4roh0EpGlIvKOM0jx/xMRn/PeYREZLSJrgXYi0s8ZTDdTRF4qLpoicreIbBGRFfjvBDImJliBNMFMBO4EcAreLUCwgR/a4L/F7SKgEXCT014VWK6qzYCvgb5Ae2ekn0LgdmeA3xH4C+OVzmcYExPi7l5s4w5V3SUiX4tIC/z3F69R1a+DzL5CVXdAyb3oV+IfSaYQ/6hC4B844TJgpXOf8hn4x4tsC3yoqvud5acD53uzVcaExgqkKcvLwF1AXfw9ymCOv6G/+OcfVLXQeS3AFFV9JHBGEel98qtpjDdsF9uUZRb+sRxb4x8dKZg2zvByPvy70ctKmWcx8GsR+RmAiNQUkfPwj3beUUTOccaxvNnVLTDmJFgP0gSlqj+KyBLg24CeYGlWAs8DjfEPkDurlM/KEpHH8D+KwAfkAwNV9TMReQL4FPgWyHR1I4w5CTbcmQnKKWSrgZtVdWuQeToBf47G0wON8ZrtYptSichFwDZgcbDiaEy8sx6kqRARaQq8clzzUVVtG431MSYSrEAaY0wQtottjDFBWIE0xpggrEAaY0wQViCNMSYIK5DGGBPE/weHNF6Zh5rd/wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 49.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Train the CatBoost algorithm\n",
    "import catboost as cbt\n",
    "cb = cbt.CatBoostClassifier(verbose=0,boosting_type='Plain')\n",
    "#cb = cbt.CatBoostClassifier()\n",
    "\n",
    "cb.fit(X_train, y_train)\n",
    "y_pred = cb.predict(X_test)\n",
    "print(classification_report(y_test,y_pred))\n",
    "print(\"Accuracy of CatBoost: \"+ str(accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision of CatBoost: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Recall of CatBoost: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n",
    "print(\"Average F1 of CatBoost: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n",
    "print(\"F1 of CatBoost for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n",
    "cb_f1=f1_score(y_test, y_pred, average=None)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm=confusion_matrix(y_test,y_pred)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proposed ensemble model: Leader Class and Confidence Decision Ensemble (LCCDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LCCDE aims to achieve optimal model performance by identifying the best-performing base ML model with the highest prediction confidence for each class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the best-performing (leading) model for each type of attack among the three ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Leading model list for each class\n",
    "model=[]\n",
    "for i in range(len(lg_f1)):\n",
    "    if max(lg_f1[i],xg_f1[i],cb_f1[i]) == lg_f1[i]:\n",
    "        model.append(lg)\n",
    "    elif max(lg_f1[i],xg_f1[i],cb_f1[i]) == xg_f1[i]:\n",
    "        model.append(xg)\n",
    "    else:\n",
    "        model.append(cb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "               colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "               importance_type='gain', interaction_constraints='',\n",
       "               learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "               min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "               n_estimators=100, n_jobs=0, num_parallel_tree=1,\n",
       "               objective='multi:softprob', random_state=0, reg_alpha=0,\n",
       "               reg_lambda=1, scale_pos_weight=None, subsample=1,\n",
       "               tree_method='exact', validate_parameters=1, verbosity=None),\n",
       " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "               colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "               importance_type='gain', interaction_constraints='',\n",
       "               learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "               min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "               n_estimators=100, n_jobs=0, num_parallel_tree=1,\n",
       "               objective='multi:softprob', random_state=0, reg_alpha=0,\n",
       "               reg_lambda=1, scale_pos_weight=None, subsample=1,\n",
       "               tree_method='exact', validate_parameters=1, verbosity=None),\n",
       " LGBMClassifier(),\n",
       " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "               colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "               importance_type='gain', interaction_constraints='',\n",
       "               learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "               min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "               n_estimators=100, n_jobs=0, num_parallel_tree=1,\n",
       "               objective='multi:softprob', random_state=0, reg_alpha=0,\n",
       "               reg_lambda=1, scale_pos_weight=None, subsample=1,\n",
       "               tree_method='exact', validate_parameters=1, verbosity=None),\n",
       " LGBMClassifier(),\n",
       " LGBMClassifier(),\n",
       " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "               colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "               importance_type='gain', interaction_constraints='',\n",
       "               learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "               min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "               n_estimators=100, n_jobs=0, num_parallel_tree=1,\n",
       "               objective='multi:softprob', random_state=0, reg_alpha=0,\n",
       "               reg_lambda=1, scale_pos_weight=None, subsample=1,\n",
       "               tree_method='exact', validate_parameters=1, verbosity=None)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Leading Model for Each Type of Attack:**  \n",
    "0 BENIGN: &emsp; XGBClassifier  \n",
    "1 Bot:        &emsp;  &emsp;      XGBClassifier   \n",
    "2 BruteForce:  &emsp;      LGBMClassifier  \n",
    "3 DoS:        &emsp;   &emsp;   XGBClassifier  \n",
    "4 Infiltration:  &emsp;       LGBMClassifier  \n",
    "5 PortScan:  &emsp;       LGBMClassifier  \n",
    "6 WebAttack:    &emsp;      XGBClassifier  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## LCCDE Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LCCDE(X_test, y_test, m1, m2, m3):\n",
    "    i = 0\n",
    "    t = []\n",
    "    m = []\n",
    "    yt = []\n",
    "    yp = []\n",
    "    l = []\n",
    "    pred_l = []\n",
    "    pro_l = []\n",
    "\n",
    "    # For each class (normal or a type of attack), find the leader model\n",
    "    for xi, yi in stream.iter_pandas(X_test, y_test):\n",
    "\n",
    "        xi2=np.array(list(xi.values()))\n",
    "        y_pred1 = m1.predict(xi2.reshape(1, -1))      # model 1 (LightGBM) makes a prediction on text sample xi\n",
    "        y_pred1 = int(y_pred1[0])\n",
    "        y_pred2 = m2.predict(xi2.reshape(1, -1))      # model 2 (XGBoost) makes a prediction on text sample xi\n",
    "        y_pred2 = int(y_pred2[0])\n",
    "        y_pred3 = m3.predict(xi2.reshape(1, -1))      # model 3 (Catboost) makes a prediction on text sample xi\n",
    "        y_pred3 = int(y_pred3[0])\n",
    "\n",
    "        p1 = m1.predict_proba(xi2.reshape(1, -1))     # The prediction probability (confidence) list of model 1 \n",
    "        p2 = m2.predict_proba(xi2.reshape(1, -1))     # The prediction probability (confidence) list of model 2  \n",
    "        p3 = m3.predict_proba(xi2.reshape(1, -1))     # The prediction probability (confidence) list of model 3  \n",
    "\n",
    "        # Find the highest prediction probability among all classes for each ML model\n",
    "        y_pred_p1 = np.max(p1)\n",
    "        y_pred_p2 = np.max(p2)\n",
    "        y_pred_p3 = np.max(p3)\n",
    "\n",
    "        if y_pred1 == y_pred2 == y_pred3: # If the predicted classes of all the three models are the same\n",
    "            y_pred = y_pred1 # Use this predicted class as the final predicted class\n",
    "\n",
    "        elif y_pred1 != y_pred2 != y_pred3: # If the predicted classes of all the three models are different\n",
    "            # For each prediction model, check if the predicted class’s original ML model is the same as its leader model\n",
    "            if model[y_pred1]==m1: # If they are the same and the leading model is model 1 (LightGBM)\n",
    "                l.append(m1)\n",
    "                pred_l.append(y_pred1) # Save the predicted class\n",
    "                pro_l.append(y_pred_p1) # Save the confidence\n",
    "\n",
    "            if model[y_pred2]==m2: # If they are the same and the leading model is model 2 (XGBoost)\n",
    "                l.append(m2)\n",
    "                pred_l.append(y_pred2)\n",
    "                pro_l.append(y_pred_p2)\n",
    "\n",
    "            if model[y_pred3]==m3: # If they are the same and the leading model is model 3 (CatBoost)\n",
    "                l.append(m3)\n",
    "                pred_l.append(y_pred3)\n",
    "                pro_l.append(y_pred_p3)\n",
    "\n",
    "            if len(l)==0: # Avoid empty probability list\n",
    "                pro_l=[y_pred_p1,y_pred_p2,y_pred_p3]\n",
    "\n",
    "            elif len(l)==1: # If only one pair of the original model and the leader model for each predicted class is the same\n",
    "                y_pred=pred_l[0] # Use the predicted class of the leader model as the final prediction class\n",
    "\n",
    "            else: # If no pair or multiple pairs of the original prediction model and the leader model for each predicted class are the same\n",
    "                max_p = max(pro_l) # Find the highest confidence\n",
    "                \n",
    "                # Use the predicted class with the highest confidence as the final prediction class\n",
    "                if max_p == y_pred_p1:\n",
    "                    y_pred = y_pred1\n",
    "                elif max_p == y_pred_p2:\n",
    "                    y_pred = y_pred2\n",
    "                else:\n",
    "                    y_pred = y_pred3  \n",
    "        \n",
    "        else: # If two predicted classes are the same and the other one is different\n",
    "            n = mode([y_pred1,y_pred2,y_pred3]) # Find the predicted class with the majority vote\n",
    "            y_pred = model[n].predict(xi2.reshape(1, -1)) # Use the predicted class of the leader model as the final prediction class\n",
    "            y_pred = int(y_pred[0]) \n",
    "\n",
    "        yt.append(yi)\n",
    "        yp.append(y_pred) # Save the predicted classes for all tested samples\n",
    "    return yt, yp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 59.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Implementing LCCDE\n",
    "yt, yp = LCCDE(X_test, y_test, m1 = lg, m2 = xg, m3 = cb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of LCCDE: 0.9975746268656717\n",
      "Precision of LCCDE: 0.9975807247413017\n",
      "Recall of LCCDE: 0.9975746268656717\n",
      "Average F1 of LCCDE: 0.9975482770384609\n",
      "F1 of LCCDE for each type of attack: [0.99836021 0.99351492 1.         0.99836334 0.85714286 0.99137931\n",
      " 0.99889258]\n"
     ]
    }
   ],
   "source": [
    "# The performance of the proposed lCCDE model\n",
    "print(\"Accuracy of LCCDE: \"+ str(accuracy_score(yt, yp)))\n",
    "print(\"Precision of LCCDE: \"+ str(precision_score(yt, yp, average='weighted')))\n",
    "print(\"Recall of LCCDE: \"+ str(recall_score(yt, yp, average='weighted')))\n",
    "print(\"Average F1 of LCCDE: \"+ str(f1_score(yt, yp, average='weighted')))\n",
    "print(\"F1 of LCCDE for each type of attack: \"+ str(f1_score(yt, yp, average=None)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 of LightGBM for each type of attack: [0.99795054 0.99092088 1.         0.997543   0.85714286 0.99354839\n",
      " 0.99778271]\n",
      "F1 of XGBoost for each type of attack: [0.99836021 0.99351492 1.         0.99836334 0.85714286 0.99137931\n",
      " 0.99889258]\n",
      "F1 of CatBoost for each type of attack: [0.99781241 0.99222798 1.         0.99591837 0.85714286 0.99137931\n",
      " 0.9944629 ]\n"
     ]
    }
   ],
   "source": [
    "# Comparison: The F1-scores for each base model\n",
    "print(\"F1 of LightGBM for each type of attack: \"+ str(lg_f1))\n",
    "print(\"F1 of XGBoost for each type of attack: \"+ str(xg_f1))\n",
    "print(\"F1 of CatBoost for each type of attack: \"+ str(cb_f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Conclusion**: The performance (F1-score) of the proposed LCCDE ensemble model on each type of attack detection is higher than any base ML model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}