IV. Machine Learning and Other Topics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In This Notebook:\n",
    "\n",
    "This notebook gives a quick overview and pointer to some of the common python data analysis libraries.  All of these and other packages are built on top of the libraries covered in the first three lessons (numpy, pandas, and matplotlib).  \n",
    "\n",
    "- [Machine Learning with scikit-learn](#Machine-Learning-with-scikit-learn)\n",
    "    - [Scikit-Learn Overview Using Linear Regression](#Scikit-Learn-Overview-Using-Linear-Regression)\n",
    "    - [Data Prep](#Data-Prep)\n",
    "        - [Data Reduction with PCA](#Data-Reduction-with-PCA)\n",
    "        - [Train-Test Splits](#Train-Test-Splits)\n",
    "    - [Classification](#Classification)\n",
    "        - [Multi-Layer Perceptron Neural Nets (MLPs)](#Multi-Layer-Perceptron-Neural-Nets)\n",
    "        - [Logistic Regression](#Logistic-Regression)\n",
    "        - [Random Forest](#Random-Forest)\n",
    "        - [Decision Trees](#Decision-Trees)\n",
    "        - [K Nearest Neighbors](#K-Nearest-Neighbors)\n",
    "    - [Cross Validation with GridSearch](#Cross-Validation-with-GridSearch)\n",
    "    - [Unsupervised Learning / Clustering](#Unsupervised-Learning-/-Clustering)\n",
    "- [Other Data Analysis Tools](#Other-Data-Analysis-Tools)\n",
    "    - [Natural Lanugage Processing with nltk](#Natural-Language-Processing-with-nltk)\n",
    "    - [Network Analysis with networkx](#Network-Analysis-with-networkx)\n",
    "    - [Statistical Modeling and Time Series Analysis with statsmodel](#Statistical-Modeling-and-Time-Series-Analysis-with-statsmodel)\n",
    "    - [Optimization](#Optimization)\n",
    "    - [Simulation](#Simulation)\n",
    "    - [Association Analysis](#Association-Analysis)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run to load packages before jumping to other parts of the notebook\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since its inception in 2010, [Scikit-learn](https://scikit-learn.org/stable/index.html) has become the premier python toolkit for machine learning.  It includes built-in functions and models for all types of machine learning tasks, including regression, classification, clustering, dimension reduction, parameter tuning, model selection, feature extraction, and more.  \n",
    "\n",
    "We give an overview of the main mechanics and some examples in this notebook.  For additiona reference I'd recommend the Coursera [Applied Machine Learning in Python](https://www.coursera.org/search?query=applied%20machine%20learning%20python&) course, or [Introduction to Machine Learning with Python](https://learning.oreilly.com/library/view/introduction-to-machine/9781449369880/) by Andreas C. Müller and Sarah Guidomon on O'Reily is also a great resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scikit-Learn Overview Using Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scki-kit makes it extremely easy to fit machine learning models.  Lets start with a simple linear regression.  Using the conventional notation, we assume we have $n$ data points, each with $p$ features $(x_1, x_2, ..., x_p)$ and an output value $y$.  In some cases $y$ only takes on discrete non-numerical values, in which case we have a **Classification** problem, or when $y$ is numerical we have a **Regression** problem.  Both classifiation and regression are types of **Supervised Learning** problems where we have known output values, but we can also have **Unsupervised Learning** where there are no outputs, only input values of the features from which we can learn patterns or identify clusters and anomalies in the data.  \n",
    "\n",
    "For now lets look at regression, and in particular linear regression (aka Oridinary Least Squares) where we assume a linear model mapping inputs to outputs.  That is, for data point $i$, we have\n",
    "\n",
    "$$y_i = \\beta_0 + \\beta_1 x_{i1} + \\beta_2 x_{i2} + ... + \\beta_p x_{ip}$$\n",
    "\n",
    "In matrix equation form it then looks like this:\n",
    "\n",
    "$$\n",
    "y = \\begin{bmatrix}y_1 \\\\ y_2 \\\\ \\vdots \\\\ \\\\ y_n \\end{bmatrix}_{n \\times 1}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "1 & x_{1,1} & x_{1,2} & \\ldots & x_{1,p} \\\\\n",
    "1 & x_{2,1} & x_{2,2} & \\ldots & x_{2,p} \\\\\n",
    "\\vdots  & \\ddots \\\\\n",
    "\\vdots \\\\\n",
    "1 & x_{n,1} & x_{n,2} & \\cdots & x_{n,p}\n",
    "\\end{bmatrix}_{n \\times (p+1)}\n",
    "\\begin{bmatrix}\\beta_0 \\\\ \\beta_1 \\\\ \\vdots \\\\ \\\\ \\beta_p \\end{bmatrix}_{(p+1) \\times 1}\n",
    "=X\\beta\n",
    "$$\n",
    "\n",
    "As with convention, we'll use y to denote any 1-dimensional vector of output values or labels, and X to denote any matrix of input features.  Lets look at a simple linear regression model, with only one feature (p = 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5QcdZ338fe3u2cmmQkQcoFkgDCIyJ2g5CKsF0RUgkHWdX14XAQCKCuP7lnPwyqKrCAosveL6wOLAgHJgp5FRQZRiNxELrm4BEKCEXAIkJCEBEwmCTPT07/nj6qe9PRUVVdP37s/r3NyMt3V3fWr6upvVX1/N3POISIijSdR6wKIiMj4KICLiDQoBXARkQalAC4i0qAUwEVEGpQCuIhIg1IAL4KZXWlmt9W6HNVgZu81s9/VuhyFFPpOzKzPzE6tZpmimNn1Zva3Ecvr8hgzs2fN7ORal6PczOxsM7sv5mvr7lhriABuZu8xs8fM7I9mts3MfmNmc2tdrmbmnPu1c+7wWpej2TjnPuecuxrAzE42s1dqXaY4nHNHO+ceqvR6zGyxmX2z0uvJcs4tcc59uFrrK7e6D+BmtjfQC3wHmAIcAHwDGKhluUphZql6/rx60szbJrXVDMdW3Qdw4B0AzrnbnXPDzrndzrn7nHNPA5jZoWb2gJltNbPXzWyJmU3Ovtm/rfmSmT1tZjvN7EYz29/M7jWzHWa21Mz29V/bY2bOzC4ysw1mttHMLgkrmJm9278zeNPMVkXdYvrluNTMngZ2mlkq6v1mdoiZPZJTxu9mb99yynmhma0HHvCfv8DM1prZG2b2SzM72H/ezOxfzGyzfxfztJkd4y873czW+Ot51cz+xn9+1NWhmR1pZg/5ZX3WzD6Ws2yxX757/M950swODdkPkfvYv039bzO7zcy2A4vMrNvMfubffT1vZp/N+9gJZvZDf92/NbPZIetOmNlXzOwF/3j5kZlNySvX+Wb2sr8PP2dmc/399aaZ/UfI504ws91mNs1/fLmZpf2LD8zsm2b2rzn76ptm1gXcC3SbWb//r9v/yHYzu9XfnmfNbE7Qev3PO8nMlvvf63IzOyln2UNmdrV5d6w7zOy+bBn95cUev6fmfEc/Ciuj/9qv+sfVG2Z2s5lN8JctMrNH8z7bmdnbzewi4Gzgy/7+uDugHNeb2T/mPXeXmf1f/+/s97vDX//Hc163yN8X/2Jm24Ar88tjZv/mf//bzWylmb03rwglH2tl5Zyr63/A3sBW4BZgAbBv3vK3Ax8COoDpwCPAv+Ys7wOeAPbHu3rfDPwWeKf/ngeAK/zX9gAOuB3oAo4FtgCn+suvBG7z/z7AL9fpeCfCD/mPp4dsRx/wFHAQMLHQ+4HHgX8E2oH3ANtz1p0t561+OScCfwo8DxwJpIDLgcf8138EWAlMBsx/zUx/2Ubgvf7f+wLv8v8+GXjF/7vN/+zL/PKcAuwADveXLwa2AfP8dS8B7gjZD3H28ZC/PQl/2x4G/h8wATjef/0H817/5345/wb4A9CWs9+zn/1FvGPhQLzv/j+B2/PKdb2/ng8DbwE/BfZjz7Hz/pDtegT4hP/3fcALwIKcZR/P2VffzN/HOZ9zpb/e04Ek8G3giZB1TgHeAM7x9/un/MdT/eUP+eV4h78fHwKuLeH4PTVOGf3XrsY71qcAv8nZ5kXAo3mf7YC35++fkHK8D3gZsJxjdjfQ7T/+JNDtb9NZwE72HOuLgDTwV/7+mphfHuDTwFR/+SXAa8CEch5rZY2PtQjKRRfSCziLgVf8L+BnwP4hr/1T4H/yDqazcx7fCVyX8/ivgJ/m/YiPyFn+98CNOV9gNoheCvwgb92/BM6L+AFckPM49P3ALH87O3OW3cbYAP62nOX3AhfmPE4Au4CD8QLuOuDdQCJvneuBvwT2znv+ZPYE8Pf6B3IiZ/ntwJU5P7rv5yw7HXguZD/E2ceP5Cw7CBgG9sp57tvA4pzXP5G33bknpT72/KjW4gd+//FMvB9kKqdcB+Qs3wqclXfsfDFku64G/t3/rNeAvwauxTsZ7Aam5eyrQgF8ac7jo4DdIes8B1iW99zjwCL/74eAy3OW/R/gFyUcv6fGKaP/2s/lHQ8v+H8vorQAbnjH7Pv8x58FHoh4/VPAmTnrXp+3fEx58pa/Acwu57EWtq7x/GuEFArOubXOuUXOuQOBY/DOsNlb0v3M7A7zbv+34wW6aXkfsSnn790Bjyflvf7lnL9f8teX72Dgk/7t55tm9ibelfLMiE3J/dyo93cD25xzu0LeG/Z5/5bzWdvwDvYDnHMPAP8BfBfYZGY3ZG/vgU/g/cBeMrOHzezEgPV0Ay875zI5z72EdxWX9VrO37sYu0+jyp6/j3OXZffFjoh1j7zeL+MrhH9nP8nZR2vxTg7757ym2GMl62G8gPwu4BngfuD9eCfN551zr4e8L0j+vpxgwfnabrx9kSvu9zKe47eYMsb5DRXNedHwDry7DYC/wLvjA8DMzjWzp3K26RhGx4Og39EIM7vEvDTkH/337xP2/jIcayVriACeyzn3HN5Z+hj/qW/jncGPc87tjXcLZCWu5qCcv2cBGwJe8zLeFczknH9dzrlro4of8/0bgSlm1hlSprDP+8u8z5vonHsMwDn37865E4Cj8W6pv+Q/v9w5dyZemuCnwI8C1rMBOMjMco+XWcCrEdtaSNQ+zt2uDXj7Yq+IdY98ll/GAwn/zhbk7aMJzrlStiPrMeBw4OPAw865NX45P4oX3IO4kOfj2oAXKHLF/V7Gc/wWI+z73QmMHNdmNiPvfXH2ye3An5tXxzMf784I//H3gC/gpZEm46VycuNB6Of7+e5Lgf+Fl6qdDPwx7/31cKyNqPsAbmZH+GfFA/3HB+GdfZ/wX7IX0A+8aWYH4AemEv2tmXWa2dHA+cAPA15zG3CGmX3EzJLmVWSdnC1nDKHvd869BKzAq2Rp96+KzyjwedcDX/XLjJntY2af9P+ea2bzzawN7wf0FjDsf/bZZraPc24IL88+HPDZT/rv+7KZtfmVXWfgXQmNV5x9jHPuZbzg+G1/Hx0HXEjOVRdwgpn9mX8F+EW8FkpPjP00rge+ZXsqd6eb2ZklbENuOXfh1TN8nj0B+zG89FRYAN8ETDWzfca52p8D7zCzvzCvUvwsvHRGb4z3lnr8FvJ5MzvQr7i7jD3f7yrgaDM73ryKzSvz3rcJeFvUBzvn/gevHuT7wC+dc2/6i7rwAvQWADM7nz0XenHshZe63AKkzOzreHVwuWp+rOWq+wCOV1k2H3jSzHbi7azVeBUM4DUpfBfemfIe4MdlWOfDeJV2vwL+0Tk3pqG/H1jOxDs4t+Cdcb9EzH0a4/1nAyfi5WG/ifcDCG066Zz7CfB3wB1+Kmk1XqUveAfh9/DyeS/5n5mtyT8H6PPf8zm8O5j8zx4EPuZ/3ut4FYrn+ndD41VwH+f4FF6OegPwE7xK5/tzlt+FV2GVrdD7M/+ElO/f8OpP7jOzHXjH0vwStiHfw3iVW8tyHu+FV4k5hr//bgde9G+1i0ozOOe2AgvxfgtbgS8DC+Oka0o9fmP4L7zK3Bf9f9/017sOuApYCvweeDTvfTcCR/n746cRn387cKq/HvzPXgP8E149wCa8CvLfFFHmX+LVJa3D+528xdiUS70ca8CemlzBa0rGnlrldG1LM5qZ/RCvYvCKWpelFPW8j6U8zKwP+Ixzbmmty9LsGuEKvCX5aY9D/fakp+FdLUVdkYhIi2n4nkhNbAZeOmgqXk33xX7uT0QEUApFRKRhKYUiItKgqppCmTZtmuvp6anmKkVEGt7KlStfd85Nz3++qgG8p6eHFStWVHOVIiINz8zye9wCSqGIiDQsBXARkQalAC4i0qAUwEVEGpQ68oiIVFD/QJreVRvo27qTnqldLJzdzaSO8oReBXARkQpZ3reNRTcvwznYNThMZ3uSq+9Zw+Lz5zG3p/QZ1pRCERGpgP6BNItuXsbOgWF2DXqjNO8aHGbnwLD/fOljuSmAi4hUQO+qDYSNVOIc9D4dNA9EcRTARUQqoG/rzpEr73y7Bofpe31X4LJiKICLiFRAz9QuOtuTgcs625P0TOsMXFYMBXARkQpYOLsbC5md1wwWHlf6PM8K4CIiFTCpI8Xi8+fR1ZEcuRLvbE/S1ZH0ny+9EaCaEYqIVMjcniksu+xUep/eQN/ru+iZ1snC47rLErxBAVxEpKK6OlKcNXdWRT5bKRQRkQalAC4i0qAUwEVEGpQCuIhIg1IAFxFpUArgIiINSgFcRKRBKYCLiDQoBXARkQalAC4i0qAUwEVEGpQCuIhIg1IAFxFpUArgIiINqmAAN7ODzOxBM1trZs+a2V/7z08xs/vN7Pf+//tWvrgiIpIV5wo8DVzinDsSeDfweTM7CvgK8Cvn3GHAr/zHIiJSJQUndHDObQQ2+n/vMLO1wAHAmcDJ/stuAR4CLq1IKUVE6lj/QJreVRvo27qTnqldLJzdzaQyzboTpag1mFkP8E7gSWB/P7jjnNtoZvuFvOci4CKAWbMqMyuFiEitLO/bxqKbl+Ec7BocprM9ydX3rGHx+fOY2zOlouuOXYlpZpOAO4EvOue2x32fc+4G59wc59yc6dOnj6eMIiJ1qX8gzaKbl7FzYJhdg8OAF8R3Dgz7z6cruv5YAdzM2vCC9xLn3I/9pzeZ2Ux/+Uxgc2WKKCJSn3pXbcC54GXOQe/TGyq6/jitUAy4EVjrnPvnnEU/A87z/z4PuKv8xRMRqV99W3eOXHnn2zU4TN/ruyq6/jg58D8BzgGeMbOn/OcuA64FfmRmFwLrgU9WpogiIvWpZ2oXne3JwCDe2Z6kZ1pnRdcfpxXKo4CFLP5geYsjItI4Fs7u5up71gQuM4OFx3VXdP3qiSkiMk6TOlIsPn8eXR1JOtuTgHfl3dWR9J+vbFPCyjdUFBFpYnN7prDsslPpfXoDfa/vomdaJwuP66548AYFcBGRknV1pDhrbvX7uSiFIiLSoBTARUQalAK4iEiDUgAXEWlQCuAiIg1KAVxEpEEpgIuINCgFcBGRBqUALiLSoBTARUQalLrSi0jLqtVcluXSOCUVESmjWs5lWS5KoYhIy6n1XJblogAuIi2n1nNZlosCuIi0nEJzWa7b1F/lEo2PAriItJzsXJZhfvBEH8v7tlWxROOjAC4iLWfh7G4sbKZfYDDtGiIXrgAuIi3pnPkHk4yI4o2QC1czQhFpKbnNB4fDajLxcuF9r++qYsmKpwAuIi0jt/lgIZ3tSXqmdVahVOOnFIqItIyo5oP5zGDhcd2VLVCJdAUuIi0jqvlgVmd7EjNYfP48uuq8W319l05EpIyyzQeDgnh70jjp0GksOHYGC4/rrvvgDUqhiEgLiWo+2JZK8N2z38VZc2c1RPAGBXARaSGTOlJ+aiQ50pGnsz1JV0eyIVIm+RqrtCIiJZrbM4Vll51K79Mb6Ht9Fz3TOhsmZZKv8UosIlKiro4UZ82dVetilEwpFBGRBqUrcBGpmUafEafWtKdEpCaaYUacWlMKRUSqrllmxKk1BXARqbpmmRGn1hTARaTqCs2IU++jANYLBXARqbqoGXEaYRTAeqEALiJVF9Wl3aj/UQDrRcEAbmY3mdlmM1ud89yVZvaqmT3l/zu9ssUUkWaS7dI+oW1sCBp2jjUbt9egVI0nzhX4YuC0gOf/xTl3vP/v5+Utlog0uyNn7k0i4Cr8raGMWqLEVDCAO+ceAep/emYRaSi9qzbgJUzGUkuUeErJgX/BzJ72Uyz7hr3IzC4ysxVmtmLLli0lrE5EmolaopRuvAH8OuBQ4HhgI/BPYS90zt3gnJvjnJszffr0ca5ORJpNnJYo/QNp7li2nmvvXcsdy9bTr7TKKOPqSu+c25T928y+B/SWrUQi0hIWzu7m6nvWBC4zg5mTJzL/mqXqah9hXFfgZjYz5+HHgdVhrxURCRI1ucJ1Z5/AxbetVFf7AgpegZvZ7cDJwDQzewW4AjjZzI4HHNAH/GUFyygiTSpscoW7Y3S1b4bxvEtVMIA75z4V8PSNFSiLiDSw8Q4NGzS5gio449FwsiJSsnIPDRs1e7y62u+hrvQiUpJKDA0b2dXe1NU+S1fgIlK03HTJ5u0DZDLBCevx5quzFZz5V/VmNOTs8ZWivSAiRclPl6QSRjokgJeSr26m2eMrRXtCRGLLTZdkhQVvKD1fHVTBqXk092jNrRaRcYmaSSdIufPVmkdzNFViikhsUc37AFJ+RMl2yClnvlrzaI6lK3ARiS2qed/EtiSnHzuD/faaUJF8dZx5NFutc4+uwEUktqjmfRnnmNzZxsFTO/loBSob1blnLAVwEYktaPySjtSeMHLjo31c1buG+dcsZXlfeacR0DyaYymAi0hRss37rjjjKC58zyE4vLzGQDoDVC4vrc49YymAi0jRss37DttvEqlEcBgp96w6UaMXtmrnntbbYhEpm2Ly0uVov63OPaO15laLSFnEHXSqnO23gzr3tCqlUERk3OLkpdV+u3IUwEVkjLhzUcbJS8dpvy3joxSKiIxSbLqjUF5a7bcrRwFcREYEDVaVDb6Lbl7GsstODawwjMpLa3KGylEKRURGVCLdofbblaMALiIjKpHuKLb9dtz8uyiFIiI5KpXuiNt+W8PFFkdX4CIyopLpjmye/NIFR3DW3FmBV95qblgcBXARGVHL7upqblg8pVBEWlhQ9/ZadVdXc8PiKYCLtKhC+eY43dXLOT+lmhsWz1wxE9yVaM6cOW7FihVVW5+IBOsfSDP/mqWj2ntndXUkR7X3DgvSQScAM8Zd4VhMmVqNma10zs3Jf145cJEWFDffvLxvG/OvWcpVvWu4/uEXRyZreGTdlrJXOGq42OJpj4i0oDj55qhemZ+9dQWJkNYqpcxPqeFii6O9ItKC4uSbo67ShzOOgUzwwlIrHDVcbHxKoYi0oDjtvaOu0tMZRyokeqjCsXoUwEVaUJx8c9QkwhPbkiRCciga36R6lEIRaVGF8s0LZ3dz9T1rAt+bcY4PHbk/96/dRNKM3UOZUa1QlLOuDjUjFJER+U0GuydP5OIlK0eaCnakEgykMyP/T2xLknGOBcfM4MRDp6rCsULCmhFqT4sIENyxxwyu+/QJbHxzN+s29fODJ/oAGEhnANg95OXI71+7iW99/FgF7ypTDlxEIgeSuvi2lSw8rpvD9ptEKhEcMjRWSW3odCnSAMrZZT1InI49Gquk/iiAi9S5aoyRHSc4R7UdB3BUrz5NPEqhiNSxao2RHdVkMNuue+HsbkKajgNw6+N9GrO7ygoGcDO7ycw2m9nqnOemmNn9ZvZ7//99K1tMkdYUldrIZODrd60OnHqs2GnJ4nTsmdSR4px3HxzxKaY8eJXFSaEsBv4DuDXnua8Av3LOXWtmX/EfX1r+4om0tqjUxu6hYe566lXSGUalVYCiUy7Zjj1howuOtC6JuARXHrz6CgZw59wjZtaT9/SZwMn+37cAD6EALlJ2hfLOfmu+keXn3fQkhrFzcOwAVItuXhY5JGucgaQ0Znd9GW8OfH/n3EYA///9ylckEcmKSm0ESQ87hoYzgcviNPUrNG9lJefMlOJVvBLTzC4ysxVmtmLLli2VXp1IUwkasyTK4LBjcLgyowSGlUdjdtfOePf2JjOb6ZzbaGYzgc1hL3TO3QDcAF5X+nGuT6RlZVMbd658mavvWctQSIAGaE96l8dBQbxcKQ6N2V0/xrvHfwacB1zr/39X2UokImN0daRoTyVpSyYYGg7OhwOkkoZhDAa8ppwpDo3ZXR8KBnAzux2vwnKamb0CXIEXuH9kZhcC64FPVrKQIq2gUG/LqBYpAKkE3HLBfGBsKxSNEtic4rRC+VTIog+WuSwiLStOb8uoFiDtSePyjx418lqlOFqDemKK1Fjc3pZRLUDaUgk+ccKBI48LtSaR5qAALlJjcWeIVwsQyadvXKTGihnlTy1AJJe+dZEaK9S7ccY+E7hj2XrWbdrBm7uGmNzZxjv234vPn/L2sg4pK41HU6qJ1Fj/QJr51yxl58DYAJ4wwIElILeDZUcqQSppZR1SVupX2JRqyoGL1FhUb8uMgwyjgzd4U5qVe0jZMP0DaW75zR8476YnOe+mJ1n8WF/B0Q2lOnQFLlIndg6k+e+Vr3B177Mjg1QVkjT403cewDfOPKYi6ZTlfds458YneWtodIEmtCX4wYXzdfVfJboCF6lzXR0pOlIJEkWMXjXs4Ce/fZX51yxled+2spanfyDNeTeNDd4Abw1lWHRT5a/+JZoCuEgd6du6M3QwqjAZqEg6pXfVBtIRZRkazmgChxpTFbZIlQR1lQdGPTcYcLUbl3Nw58qXaU8lyzL5caGTyeCw0wQONaYALlIFQV3lr7z7WQASZuwaHGZiW4LdJQTwXYPDfOPuNbSnvM8pdfLjnqldtCctNIi3J00TONSYUigiFRbWVf6toQxvDWVGnisleGcNuz2fU+rkxwtnd5NKhufj25IJTeBQYwrgIhUW1VW+GsJm4ik08fGkjhS3XDCfCW1jw8SEtgSLL1D3/VrT3hepsELDwFZa0Ew8cUY/BK/r/srLP8Sdv32ZB9Z6M2qdcsR+fOKEAxW864C+AZEKKzQxcaXlz8STm9LJipr4uKsjxbknHsK5Jx5SvUJLLEqhiFRYsRMTl1v+TDxxRz+U+qcALhKhUJ44jrBhYCe0JZjQlhh5rj2iwrAjlaA9ZaPe39meCMxPZ01sSwQONVvM6IdS35RCEQkRN0+cFTUlWv4wsDP2mQA4Xtq6iz/uHmLyxHae37KDh9e9HliWgXSGz7znEA7bf9KoYWRXvvQGn711BcMZRzrjmNiWIONgwTEzOPHQqYFDzRYa/VBNAxuHArhIgGLzxHGCfXaWnKDXmsE57z6Y5X1vhAbWw/afNGoi4eV927h4yUoSBgMZRyoBw87x/fPm8r7Dpodu28LZ3Vx9z5rAZeWc+FgqTykUkQC9qzaQyQQnijMZx9fvWj2SVtm0/a1YU6JlR/X7i+89EfjaWx/vIyyJkh9Yc08w2Xbf6QwMph0X37Yyst23ZvZpHvqmRAI8/uLW0I41u4cy3PXUBtIZR2d7kq//bDUWEnqzlYIz95nIZ25dTnrYEXJeAIxzTjyYHzzxUsEZ5eNUROZerefTzD7NQd+WSJ7+gTS/WP1a5GvSfhTek+4Ijqa7Bof59e9fp/fpjQXXu2twGMNiBdZyVERmUzrSuBTARXzZSsh7V28kU6aukxPbkgVPBlnZCsQ4gVUVkQIK4CLA2ErIcsk4R9xJU4qpQFRFpIAqMUUCB5sqVkcqQUcqMaZS8LRjZlBoeO/2pBVdgZhbETnRbwueSkB7yrju7BOUy24R+pal5ZVrsKlffvG9PPmHbaNy13ev2sB9z25i91DwiSFhcPlHjxrX2CJze6Zw3dkn8NlbV5BKmNcOPJng4iUrNdlxi9AVuLS8YgebaksaHamxP52PfudR3jZ9EpcuOIKz5s6iqyPFwtndJCJ+ZYsvmMe5J/WM64q5fyDNxUtWMpDOjFSq7h4qbQhZaSwK4NLyshWCcaQS8OWPHI7La3USNkv86FRH0v8M7wRw6wXzIjvcFKIxTUQpFKk7UV3So5aNV1SFYL5EwrzQXUQb7Eq1udaYJqIALnUlqks6UNTYJHFlr5IX3byMTMZFzozzpY8czt/d+xzpkJeEBc5ytbnOPYFt3j7AxLZkYH5dTQlbg8Vt4lQOc+bMcStWrKja+qSx9A+kmX/N0lHjj2R1tSdxOHYNjo2cXR3JMWOTjMfOgTS9T2/g8Re2cu/q10iY1+syO0jgETP2Zu1r2yN6UnqB84ozjio6WMe5s8g/uUXNoVmufSL1wcxWOufm5D+vb1eqKipQReV0h4bDr4rjdB2PI3uVfNbcWXwrJ5jf88xGhoYdz27cXvAzxtMGO85AWEGDa+UG7+yVeFjXe2lO+oalYvKDdffkiVy8ZGVooIrK6YbNjA6Vyfd2daT46HHdXHX3GoYKNeT2tSUtduDM7pt1m3aw5Mn1DOTkZIJGPYw6uU1sS3D6sTPYb68JGtOkxehbloqIc7ufH6iiuodnJzsICuSVyvf2rtoQeeWfK2kw/5ApvLC5nyNn7h1ZsRq312d62I3cWUSd3HYPZdhvrwlcuuCIWGWV5qFmhFJ2QT0boyoGsymQqKnH2pIJUiEz1lSq63jf1p2RV/65hh08+vxWrupdw/xrlrK8b1vg64rp9TmQzvD7Tf1AdFNHVVi2LgVwKbtiezZmUyCR41RfMI9bLphf1TGse6Z2RU5zFrYtUR1pit03b+waBKLn1dTYJ61LKRQpu2J7NuZeQRZqM13NMawXzu7m6t41DA4XPz5KWMXquk07ito3+3a2A6ObOhYaK1xah751KbuoXHaQ/CvIqDbT1RzDelJHisUXzOOcG5/krYgUUJCgitXlfdu47cmXYn9GRyrBYftPGnmsSRgkX0nfvJn1ATuAYSAd1E5RWk+hno2N1ORtbs8UVl7+Ie5c+Qrf+/WLvPzG7ljvSyVg84636B9IM6kjNZL7HkzHz5+kkjYmNaJJGCRXOX41H3DOBU+l3eQq0a27GUTd7l/36RPY+ObuhrqC7OpIce5JPbSnElzVuybWnUU6Az9/5jV+8exrLD5/Hi9s7i+Y++5IJRhIZxrixCb1oaSemP4V+Jy4AbyZemKGzSzeyMN4lvuElO3Z2EjBOkpUT9Eone0JPjVvFjc+2hf6mvNOPJijuvdumn0l5RXWE7PUAP4H4A28oX3+0zl3Q9TrmyWAR3b5btAuzKWckFrpTiRwPwFzevblN89vHRnWNd/73zGN5X1vhE6BNp7u99I6KtWV/k+ccxvMbD/gfjN7zjn3SN6KLwIuApg1qzkO0FJnBK83Qd20g3oDBonTDbyZhFUkfueB3/PwuvAb0cde2EpblduxS/MrKYA75zb4/282s58A84BH8l5zA3ADeFfgpayvXjTbMJ7jPSGVEvjLLe5dQDnuFoIqErNtxsM6/hhw7ok9/OCJl9QMUMpm3EeNmXUBCefcDv/vDwNXla1kdQl+uIEAAAknSURBVKzZZgQf7wmpXu5E4t4FVPJuYeHsbi7/6TOhyweHHYapGaCUVSk9MfcHHjWzVcAy4B7n3C/KU6z61mi94voH0tyxbD3X3ruWO5atpz+vh+B4u2nXw51IUNf0oN6QcV83XpM6UnzmvW8LXZ7dj9mr99xp10TGa9wB3Dn3onNutv/vaOfct8pZsHoW2eW7zm6Hl/dtY/41S7mqdw3XP/xi4Fgd4z0h1cP4HHGnFavG9GNfOOUwukL2Rz2e2KXxaSyUccpWZl1xxlFc/P5DueKMo1h22al1VXEX96pzvCekWt+J9A+kuXf1xlh3AdW4W8j23GyEE7s0Bx1RJaj3XnHF5KjH0027luNzZPPZQ2FzmzH6LqBa9Rbq7i7VpKOqiRV71TmeE1ItAlZQ65cguXcBUd37y323UO8ndmkeCuBNrFpXneUMWHGa+RUakrU9abSlEqPuAjSanzQjHbVNrJpXneUQt5lfoeFqTzp0Gt89+11jgrLSG9JsVInZxBqptUwxzfx6pnbRkQo+dDtSCRYcOyN029SMT5qJjt4m1yhXnb2rNpAJqY/Mr3D9wBH78ZUfB3eaGUhn+MDh+42rDK00pos0Bx2dLaARKtUef3Eru4fiVbg++NzmkaFX83WkEjz4u81Fb2+rjekizUEpFKm5/oE09zyzMXT5xLbRFa59W3cGBm/wrsCLbdNd6V6aIpWiAC7jUqh7fjH+e+UrpCNmfx92blSFazl7gPYPpPn6T1czEHL1PzCU4et3rS5p+0Qqpe5TKMpL1p9ypxsefG5T5PKjZu49KmdfrtY12e0YGMoQ1h8onXHc9dSrIzPrKJ0i9aSur8DjjOMh1VWLdMPkiW2jHpejdU3udoRNwpCVzqB0itSlug3gykvWp0oMCnVKgVYjpxw5fcxzc3um8OAlJ7PgmBkcf9BkFhwzgwcvOTnW7EF3LFvP55esjOyGH6Rcg16JlEvd5iLqZaxpGa0Sg0KdduxMrrg7fBb7046eOea5R9Zt4TO3LieTcaQz8LvXthdMc+SnforViJN1SHOr2yvwehhrWsaqxBCy2WaBQbLNAnM9sm4L5960jMG0G8ld7x7KRN6dBd3RFasRJ+uQ5la3AbwexpqWsSoxhGwxzQL7B9J89tbwibEzGReY5ig0fkoc9Tj8gLS2ug3gtR5rOkw5m881okp0zy/mZN27agPDEZWOu4eC24EXGj8lu66ujiRXnXl0Qww/IFK3R2M9jh6n3nqecnfPL6ZZYN/WnZGtRlIJRgJ+bhPUzdsHmNiWDOzt2Z40Tjp0GguOnTGyHZ9414F1P/yAiLlS7yuLMGfOHLdiRfjtb5CdA+m6+CH1D6SZf83SwDGouzqSVZ2BvVqq2QY/6OSYPVnnnhzvWLaeb9y9JrTbfXvK+J+//TBrNm4f9XkT2xLsHgpO0zTr9yfNw8xWOufm5D9f90dsvYzj0WqtYqp9txH3qj7qah3g++fNxcGYCR/CgveEtoRSI9KwdNTG1EqtYoJmvMlu+6Kbl41crZb7Cj3OyTo3tZbJwO6hYVIJI5kwvnfuHN532HTuWLY+doVlwryeniKNSAE8pmrNblMP4txtvG36pJrVBxS6Wo9TYbmHNd3dk7SOum2FUm/qtVVMJRS621j3Wn/Ne8lGTcwQ1aolX7PdPUlrUQCPqZFmtylVoRlv3tw9WPbu9PlKaa4ZdbLN12x3T9JamifqVEGjzG5TqkIz3nSFpJJgfFe0+bn07skTuXjJynGnZ4KaoIZptrsnaS3NFXmqoF5axVRSoRlvdvpBtRz1AfmtXYKa+wVVoBaSf7J1OG59vA+wuuhTIFIOOnJljEJd2ydPbC9LfUBQa5ew5n5QfHPN/JPtX51yWNPfPUlr0dErYxRqcfOOGZPK0ku22PFJSq1wbIW7J2ktCuAyRpyu7V0dqZLrA4pr7qcKR5F8CuAyRtxxaEq9oo260g+iCkeR0RTAJVA1WtwU6hafHXxKFY4iwfRrkFCVzhlHXelf9+kT2PjmblU4ikTQL0JqqlXa1otUgn4lUnNqHSIyPupKLyLSoBTARUQalAK4iEiDUgAXEWlQVZ0T08y2AC9VbYVjTQNer+H6K6EZtwmac7uacZugOber3rbpYOfc9PwnqxrAa83MVgRNDNrImnGboDm3qxm3CZpzuxplm5RCERFpUArgIiINqtUC+A21LkAFNOM2QXNuVzNuEzTndjXENrVUDlxEpJm02hW4iEjTUAAXEWlQLRfAzexqM3vazJ4ys/vMrOFnCDCzfzCz5/zt+omZTa51mcrBzD5pZs+aWcbM6r5JVxQzO83Mfmdmz5vZV2pdnnIws5vMbLOZra51WcrFzA4yswfNbK1/7P11rcsUpeUCOPAPzrnjnHPHA73A12tdoDK4HzjGOXccsA74ao3LUy6rgT8DHql1QUphZkngu8AC4CjgU2Z2VG1LVRaLgdNqXYgySwOXOOeOBN4NfL6ev6uWC+DOue05D7uAhq/Fdc7d55xL+w+fAA6sZXnKxTm31jn3u1qXowzmAc875150zg0CdwBn1rhMJXPOPQJsq3U5ysk5t9E591v/7x3AWuCA2pYqXEuOB25m3wLOBf4IfKDGxSm3C4Af1roQMsoBwMs5j18B5teoLBKTmfUA7wSerG1JwjVlADezpcCMgEVfc87d5Zz7GvA1M/sq8AXgiqoWcBwKbZP/mq/h3QIuqWbZShFnu5qABTzX8Hd+zczMJgF3Al/Mu2uvK00ZwJ1zp8Z86X8B99AAAbzQNpnZecBC4IOugRr3F/FdNbJXgINyHh8IbKhRWaQAM2vDC95LnHM/rnV5orRcDtzMDst5+DHguVqVpVzM7DTgUuBjzrldtS6PjLEcOMzMDjGzduB/Az+rcZkkgJkZcCOw1jn3z7UuTyEt1xPTzO4EDgcyeEPbfs4592ptS1UaM3se6AC2+k894Zz7XA2LVBZm9nHgO8B04E3gKefcR2pbqvExs9OBfwWSwE3OuW/VuEglM7PbgZPxhl7dBFzhnLuxpoUqkZm9B/g18AxejAC4zDn389qVKlzLBXARkWbRcikUEZFmoQAuItKgFMBFRBqUAriISINSABcRaVAK4CIiDUoBXESkQf1/eBAojcIkj6sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Create some fake data\n",
    "X = np.random.randn(100)\n",
    "y = (X+2)**2 + 5*np.random.random(len(X))\n",
    "plt.figure()\n",
    "plt.title('Sample regression problem with one input variable')\n",
    "plt.scatter(X,y, marker= 'o', s=50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x20a74cb08b0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUZf7A8c83mwIkgFKkt1OkiBI0iCIoiHeHiOJZDjmOYtef9UQFsSEoIurZzwoCyoF6KkLsSBMF6R0FhAgI0lsIJNnk+f0xs2Gz2d1sspts+75fL17s7szOPDM7+c4zTxVjDEoppaJPQrgToJRSqnw0gCulVJTSAK6UUlFKA7hSSkUpDeBKKRWlNIArpVSU0gBeBiIyQkTeD3c6KoOIdBWRX8KdjtKU9puISJaIXFKZafJHRN4QkUf9LI/Ia0xE1opIt3CnI9REpL+IfBPguhF3rUVFABeRLiLyo4gcEpH9IvKDiHQMd7pimTHme2NMq3CnI9YYY24zxowCEJFuIrI93GkKhDHmDGPMnIrej4hMEJEnK3o/LsaYycaYv1TW/kIt4gO4iNQAMoFXgFpAI+AJIDec6QqGiCRG8vYiSSwfmwqvWLi2Ij6AA6cDGGOmGGMKjDHHjDHfGGNWAYjIqSIyS0T2icheEZksIie5vmw/1jwgIqtE5KiIjBOReiLypYgcEZGZInKyvW5zETEicouI7BCRnSIyxFfCROQ8+8ngoIis9PeIaadjqIisAo6KSKK/74tICxGZ55bG11yPb27pvFFEtgKz7M9vEJH1InJARL4WkWb25yIiL4jIbvspZpWItLOX9RKRdfZ+fheR++3Pi+UORaSNiMyx07pWRK5wWzbBTt/n9nZ+EpFTfZwHv+fYfkz9n4i8LyKHgcEi0lBEpttPX5tE5GaPzVYRkQ/sfS8TkfY+9p0gIsNE5Ff7evlQRGp5pOt6Edlmn8PbRKSjfb4OisirPrZbRUSOiUgd+/0jIuK0Mx+IyJMi8qLbuXpSRFKBL4GGIpJt/2tobzJZRCbZx7NWRDK87dfeXmcRWWz/rotFpLPbsjkiMkqsJ9YjIvKNK4328rJev5e4/UYf+kqjve5D9nV1QETeFZEq9rLBIjLfY9tGRE4TkVuA/sCD9vmY4SUdb4jIcx6ffSYi99mvXb/vEXv/f3Nbb7B9Ll4Qkf3ACM/0iMhL9u9/WESWikhXjyQEfa2FlDEmov8BNYB9wETgUuBkj+WnAX8GUoC6wDzgRbflWcBCoB5W7n03sAzoYH9nFvC4vW5zwABTgFTgTGAPcIm9fATwvv26kZ2uXlg3wj/b7+v6OI4sYAXQBKha2veBBcBzQDLQBTjstm9XOifZ6awKXAlsAtoAicAjwI/2+n8FlgInAWKv08BethPoar8+GTjbft0N2G6/TrK3PdxOz8XAEaCVvXwCsB841973ZGCqj/MQyDnOt48nwT62ucB/gCpAur1+D4/1r7HTeT+wBUhyO++ubd+LdS00xvrt3wSmeKTrDXs/fwGOA9OAUzhx7Vzk47jmAVfbr78BfgUudVv2N7dz9aTnOXbbzgh7v70AB/A0sNDHPmsBB4AB9nnvZ7+vbS+fY6fjdPs8zgHGBHH9XhJIGu1112Bd67WAH9yOeTAw32PbBjjN8/z4SMeFwDZA3K7ZY0BD+/21QEP7mPoCRzlxrQ8GnMBd9vmq6pke4J9AbXv5EOAPoEoor7WQxsdwBOUyJ9IKOBOA7fYPMB2o52PdK4HlHhdTf7f3HwOvu72/C5jm8Ufc2m35WGCc2w/oCqJDgfc89v01MMjPH8ANbu99fh9oah9nNbdl71MygP/JbfmXwI1u7xOAHKAZVsDdAJwHJHjscytwK1DD4/NunAjgXe0LOcFt+RRghNsf3Ttuy3oBP/s4D4Gc43luy5oABUB1t8+eBia4rb/Q47jdb0pZnPijWo8d+O33DbD+IBPd0tXIbfk+oK/HtXOvj+MaBbxsb+sP4B5gDNbN4BhQx+1clRbAZ7q9bwsc87HPAcAij88WAIPt13OAR9yW/R/wVRDX7yWBpNFe9zaP6+FX+/VgggvggnXNXmi/vxmY5Wf9FUAft31v9VheIj0eyw8A7UN5rfnaV3n+RUMRCsaY9caYwcaYxkA7rDus65H0FBGZKtbj/2GsQFfHYxO73F4f8/I+zWP9bW6vf7P356kZcK39+HlQRA5i5ZQb+DkU9+36+35DYL8xJsfHd31t7yW3be3HutgbGWNmAa8CrwG7ROQt1+M9cDXWH9hvIjJXRM73sp+GwDZjTKHbZ79h5eJc/nB7nUPJc+ov7Z7n2H2Z61wc8bPvovXtNG7H92/2qds5Wo91c6jntk5ZrxWXuVgB+WxgNfAtcBHWTXOTMWavj+9543kuq4j38tqGWOfCXaC/S3mu37KkMZC/oTIzVjScivW0AfAPrCc+AERkoIiscDumdhSPB97+joqIyBCxiiEP2d+v6ev7IbjWghYVAdydMeZnrLt0O/ujp7Hu4GcZY2pgPQJJkLtp4va6KbDDyzrbsHIwJ7n9SzXGjPGX/AC/vxOoJSLVfKTJ1/Zu9dheVWPMjwDGmJeNMecAZ2A9Uj9gf77YGNMHq5hgGvChl/3sAJqIiPv10hT43c+xlsbfOXY/rh1Y56K6n30XbctOY2N8/2aXepyjKsaYYI7D5UegFfA3YK4xZp2dzsuwgrs3xsfngdqBFSjcBfq7lOf6LQtfv+9RoOi6FpH6Ht8L5JxMAa4Rq46nE9aTEfb7t4E7sYqRTsIqynGPBz63b5d3DwX+jlVUexJwyOP7kXCtFYn4AC4ire27YmP7fROsu+9Ce5XqQDZwUEQaYQemID0qItVE5AzgeuADL+u8D1wuIn8VEYdYFVndXOkMgM/vG2N+A5ZgVbIk27niy0vZ3hvAQ3aaEZGaInKt/bqjiHQSkSSsP6DjQIG97f4iUtMYk49Vzl7gZds/2d97UESS7Mquy7FyQuUVyDnGGLMNKzg+bZ+js4Abcct1AeeIyFV2DvBerBZKC0tujTeAp+RE5W5dEekTxDG4pzMHq57hDk4E7B+xiqd8BfBdQG0RqVnO3X4BnC4i/xCrUrwvVnFGZgDfDfb6Lc0dItLYrrgbzonfdyVwhoiki1WxOcLje7uAP/nbsDFmOVY9yDvA18aYg/aiVKwAvQdARK7nREYvENWxii73AIki8hhWHZy7sF9r7iI+gGNVlnUCfhKRo1gnaw1WBQNYTQrPxrpTfg58EoJ9zsWqtPsOeM4YU6Khvx1Y+mBdnHuw7rgPEOA5DeD7/YHzscphn8T6A/DZdNIY8ynwDDDVLkpag1XpC9ZF+DZWed5v9jZdNfkDgCz7O7dhPcF4bjsPuMLe3l6sCsWB9tNQeZV6jt30wyqj3gF8ilXp/K3b8s+wKqxcFXpX2TckTy9h1Z98IyJHsK6lTkEcg6e5WJVbi9zeV8eqxCzBPn9TgM32o3aZihmMMfuA3lh/C/uAB4HegRTXBHv9BuC/WJW5m+1/T9r73QCMBGYCG4H5Ht8bB7S1z8c0P9ufAlxi7wd72+uA57HqAXZhVZD/UIY0f41Vl7QB6+/kOCWLXCLlWgNO1OQqrKZknKhVdoY3NcWJyAdYFYOPhzstwYjkc6xCQ0SygJuMMTPDnZZYFw058LhkF3ucarcn7YmVW/KXI1FKxZmo74kUw+pjFQfVxqrpvt0u+1NKKUCLUJRSKmppEYpSSkWpSi1CqVOnjmnevHll7lIppaLe0qVL9xpj6np+XqkBvHnz5ixZsqQyd6mUUlFPRDx73AJahKKUUlFLA7hSSkUpDeBKKRWlwt4OPD8/n+3bt3P8+PFwJ0W5qVKlCo0bNyYpKSncSVFK+RD2AL59+3aqV69O8+bNEQl2EEEVCsYY9u3bx/bt22nRokW4k6NUVMrOdZK5cgdZ+47SvHYqvds3JC0ltCE37AH8+PHjGrwjjIhQu3Zt9uzZE+6kKBWVFmftZ/C7izAGcvIKqJbsYNTn65hw/bl0bB66mdUiogxcg3fk0d9EqfLJznUy+N1FHM0tICfPGp05J6+Ao7kF9uehG8MtIgK4UkrFisyVO/A1QokxkLnK2/wP5aMBHKscvk+fPrRs2ZJTTz2Ve+65h7y8PADmzJlD7969S3wnMzOTDh060L59e9q2bcubb75ZYp0JEyZQt25d0tPTSU9PZ+DAgUyfPp0xY6xJT6ZNm8a6desq9uCUUpUqa9/Ropy3p5y8ArL25nhdVh5xH8CNMVx11VVceeWVbNy4kQ0bNpCdnc3DDz/s8zv5+fnccsstzJgxg5UrV7J8+XK6devmdd2+ffuyYsUKVqxYwaRJk7jiiisYNmwYoAFcqVjUvHYq1ZIdXpdVS3bQvE41r8vKI+4D+KxZs6hSpQrXX389AA6HgxdeeIHx48eTk+P9TnnkyBGcTie1a9cGICUlhVatWgW0vwkTJnDnnXfy448/Mn36dB544AHS09P59ddfQ3NASqmw6t2+Ib6qkESg91khmd8ZiIBWKMXcey+sWBHabaanw4sv+ly8du1azjnnnGKf1ahRg6ZNm7Jp0yav36lVqxZXXHEFzZo1o0ePHvTu3Zt+/fqRkFDyfvjBBx8wf741a9Q999xTVDnYuXNnrrjiCnr37s0111xT3qNTSkWYtJREJlx/bolWKCIw4fpzSQ1hU8LICuBhYIzx2uLC1+cu77zzDqtXr2bmzJk899xzfPvtt0yYMKHEen379uXVV18teu9tHaVUbOnYvBaLhl9C5qodZO3NoXmdavQ+q2FIgzdEWgD3k1OuKGeccQYff/xxsc8OHz7Mtm3bOPXUU9m3b5/P75555pmceeaZDBgwgBYtWmhwVkoVSU1JpG/HphW6j7gvA+/Rowc5OTlMmjQJgIKCAoYMGcLgwYOpVs17ZUN2djZz5swper9ixQqaNWtW5n1Xr16dI0eOlCvdSikV9wFcRPj000/56KOPaNmyJaeffjpVqlRh9OjRRet89913NG7cuOjf8uXLGTt2LK1atSI9PZ3HH3+8XLnv6667jmeffZYOHTpoJaZSqswqdU7MjIwM4zmhw/r162nTpk2lpUEFTn8bpSKDiCw1xmR4fh73OXCllIpWGsCVUipKaQBXSqkopQFcKaWilAZwpZSKUhrAlVIqSsV9AN+3b1/RcK/169enUaNGRe9dQ8r6smTJEu6+++5S99G5c+eQpHXOnDnUrFmTDh060KpVKy688EIyMzMD+t6PP/4YkjQopSJHZHWlD0Co55mrXbs2K+wBtEaMGEFaWhr3339/0XKn00liovftZ2RkkJFRomlmCaEMnl27di0K2itWrODKK6+katWq9OjRw+d35syZQ1paWshuJEqpyBBVOfDFWfvpNHomIzPX8cbczYzMXEen0TNZnLU/pPsZPHgw9913H927d2fo0KEsWrSIzp0706FDBzp37swvv/wCFJ/sYcSIEdxwww1069aNP/3pT7z88stF20tLSytav1u3blxzzTW0bt2a/v374+pI9cUXX9C6dWu6dOnC3Xff7XUSCU/p6ek89thjRYNlzZgxg06dOtGhQwcuueQSdu3aRVZWFm+88QYvvPAC6enpfP/9917XU0pFn1KzriLSBJgE1AcKgbeMMS+JSC3gA6A5kAX83RhzoKIS6j7PnItr1ovB7y5i0fBLQjrS14YNG5g5cyYOh4PDhw8zb948EhMTmTlzJsOHDy8xABbAzz//zOzZszly5AitWrXi9ttvJykpqdg6y5cvZ+3atTRs2JALLriAH374gYyMDG699VbmzZtHixYt6NevX8DpPPvss3n22WcB6NKlCwsXLkREeOeddxg7dizPP/88t912W7EniwMHDnhdTykVXQKJeE5giDFmmYhUB5aKyLfAYOA7Y8wYERkGDAOGVlRCA5lnLpQjf1177bU4HNasGocOHWLQoEFs3LgRESE/P9/rdy677DJSUlJISUnhlFNOYdeuXTRu3LjYOueee27RZ+np6WRlZZGWlsaf/vQnWrRoAUC/fv146623Akqn+1AI27dvp2/fvuzcuZO8vLyi7XkKdD2lVOlCXaxbFqUWoRhjdhpjltmvjwDrgUZAH2CivdpE4MqKSiRU7jxzAKmpqUWvH330Ubp3786aNWuYMWMGx48f9/qdlJSUotcOhwOns+Ts097WCWY8muXLlxeNV3LXXXdx5513snr1at58802f6Qx0PaWUf5VVrOtLmcrARaQ50AH4CahnjNkJVpAHTvHxnVtEZImILNmzZ0+5E1qZ88x5OnToEI0aNQIqZkKG1q1bs3nzZrKysgBrFp9ArFq1ilGjRnHHHXeUSOfEiROL1vMcttbXekqpwLkX67oylzl5BRzNLbA/L5mBC7WAA7iIpAEfA/caYw4H+j1jzFvGmAxjTEbdunXLk0agcueZ8/Tggw/y0EMPccEFF1BQ4P0pIBhVq1blP//5Dz179qRLly7Uq1ePmjVrel33+++/L2pGeMcdd/Dyyy8XtUAZMWIE1157LV27dqVOnTpF37n88sv59NNPiyoxfa2nlApcIMW6FS2g4WRFJAnIBL42xvzb/uwXoJsxZqeINADmGGP8zuwb7HCyi7P2+5xnrmPzWgFtI1JlZ2eTlpaGMYY77riDli1b8q9//SusadLhZJXybcyX63lj7mafy2+/6FSGXto6JPvyNZxsIK1QBBgHrHcFb9t0YBAwxv7/s5Ck1I/KmmcuHN5++20mTpxIXl4eHTp04NZbbw13kpRSfriKdb3VzVV0sa5LqTlwEekCfA+sxmpGCDAcqxz8Q6ApsBW41hjjt+ReJ3SILvrbKOVbdq6TTqNnFmva7JKa4ghp0+Zy58CNMfMBX9Oz++7+VwalzQCvKl9lztSkVDRKS0lkwvXn+izWrYySgbCXPVSpUoV9+/ZRu3ZtDeIRwhjDvn37qFKlSriTolREC3exbtgDeOPGjdm+fTvBNDFUoVelSpUSnZCUUiWlpiSGtBNhWYQ9gCclJWlPQKWUKoeoGsxKKaXUCRrAlVIqSmkAV0qpKKUBXCmlopQGcKWUilIawJVSKkppAFdKqSgV9nbgSikVDuGcSSdUoiu1SikVAt6Gph71+bqoG5pai1CUUnElEmbSCRUN4EqpuOJvJp18ZyH/W7q9chMUBA3gSqm44m+C9LwCw6jMtZU2KXGwNIArpeKKvwnSAZyFRE1RigZwpVTcyM51kptfQH5Bod/1KmtS4mBpKxSlVFxwb3mSX+B/xqmcvAKy9uZUUsrKTwO4Uirmubc8CURlTUocLC1CUUrFPH8tT7wRgd5nNay4BIWI5sCVUjHPX8sTgMQEq/KysiclDlbkp1AppYLkanniLYhXTXLQ68z6nFK9SqVPShys6EilUkoFoXf7hoz6fJ3XZQkJMLJPu6gJ2u60DFwpFfPSUhLtYhFHURvwaskOUlMclVNcUhBY5WlZRd8tRymlyqFj81osGn4Jmat2kLU3p3KKS4yBqVPh0Udhxgxo0yakm9cArpSKG6kpifTt2LRydvbDD3DffbBoEaSnw7FjId+FBnClVFjEwnjcXv36KwwbBv/7HzRsCO++CwMGgMN39/3yioGzpZSKNrEyHncxBw7AU0/Byy9DUhI88QQMGQKpqRW2S63EVEpVqlgajxuAvDx46SU47TT497+t3PbGjfDYYxUavEEDuFKqkvnrFRktg0gBVmI//RTatYN774Wzz4bly2HcOKvopBJoAFdKVSp/vSKjZRApliyBbt3gqqsgMRE+/xy++Qbat6/UZGgAV0pVqtLG4zaUYdCSyrZtm1VE0rEjrF8Pr78Oq1ZBr17WACqVrNQALiLjRWS3iKxx+2yEiPwuIivsf70qNplKqVjRu31D/IW6SQuyIq8c/MgRePhhOP10+OgjeOgh2LQJbrvNyoGHSSA58AlATy+fv2CMSbf/fRHaZCmlYlVaSiIDzmvmZw2JnHJwpxPeegtatoTRo+Hqq2HDBut1jRrhTl3pzQiNMfNEpHnFJ0UpFTf8ZMEjphz8q6/g/vth7Vro0gWmT4dzzw13qooJpgz8ThFZZRexnOxrJRG5RUSWiMiSPXv2BLE7pVSs8FcO7ppMITvXydRFWxnz5XqmLtpKdmUVq6xeDX/9K1x6KRw/Dh9/DPPmRVzwhvIH8NeBU4F0YCfwvK8VjTFvGWMyjDEZdevWLefulFKxpHf7hj7r/ESgwUlV6TR6JiMz1/HG3M2MzFxHp9EzK3a2+D/+gFtusbq9L14ML7wA69ZZLU3CUEEZiHIFcGPMLmNMgTGmEHgbiLxbk1IqYvkbHfD1/udw+/tLK6+jT04OPPmk1RFnwgS4+26rgvLeeyE5ObT7CrFyVZ+KSANjzE777d+ANf7WV0opT75GB5wRQEefkAxIVVgI778Pw4fD779bOe1nnrECeZQoNYCLyBSgG1BHRLYDjwPdRCQdMEAWcGsFplEpFeHKOzCVt9EBK6Wjz5w51jgly5ZZbbqnTIGuXYPfbiULpBVKPy8fj6uAtCilolCoB6byN/1Z0LPF//ILPPig1aKkSRMrB96vnzUtTxSKzlQrpcLK1UJk5Iy19H9nYUjLq0ur4CzXbPF791pl2+3awezZVjvuX36B/v2jNniDDierlCojzxy3L+Utr3ZVcHrm6ss1W3xuLrzyilVJeeSI1cpkxAioV69MaYpUGsCVUgFzHwq2NMGUV/uq4DTA1EVbSy9rN8bq8j5sGGzZYrXpfvZZOOOMcqUnUmkAV0oFzN9QsJ6CLa/2rOAMuKx9wQKrgnLBAjjzTGuUwD//udzpiGTRW/ijlKp0/lqIeCp3ebUXAU0CsWULXHcddO5svX7nHWt87hgN3qA5cKVUGfhrIeJSNSmBhAQpe3m1H/5y/mnHstl60520+fBda97JRx+1WpqkpYVk35FMA7hSKmC92zdk1OfrfC53CBQaeLP/OSGd29Jbzj+xwMk/VnzJvT9M4eTjR2DgQKuysnHjkO030mkRilIqYO5d4KsmlRyMqsBArrOQ2ycvDWmX92KDXxlDj00/8fX4Oxk580021GvBN5MyrW7wcRS8QQO4UqqMXC1Eep1Zn0QfESTUc1u62oafsetX/jv1YcZ9PAqAG69+lBsHPE2Xa/8Ssn1FEy1CUUqVWWpKInWrp+As9L7cswlhebvau6Tt3cXctROp9fEHHKxWnUf/fBufdbyMgsTEkJa1R5v4PGqlVNAC7fIeVFf77Gyr/fZzz1HH6SRvyBDmXDaItOOJPGy3DY/X4A0awJVS5eSvQtPVhNBbxx9XwB/87iIWDb/EewAuKLDKtB95xBqnu29fePppklu04KqKOJgopWXgSqliAp0Jx9+Y3q5iDX/N/3yWk3/7LZx9Ntx0E7RoYXXImTrVeq2K0Ry4UqpIWYs7fHV5d+WqyzQ07Nq18MAD8OWXVrD+8EO45pqInQ0nEmgAV0oB3sc5CaS4w9uY3i4BlZPv2gWPPw5vvw3Vq1tl3nfdBSkpITqy2KVFKEopwH9vx/I2C/Q3NGwVZy5/++o9aNkSxo2DO+4ge+3PTL3w74yZtblyJzKOUpoDV0oBFTMTjrehYVOThMvWzmHUT1NI3rEdrrgCxo5lcUpdBr8Vuokh4oEGcKUUUHEz4biXkzvnzKPnhOeovW6lVVE5+T3o1s0qvhk9s+ytVeKcFqEopYAKmgnHlroti75j/kX/BwdQ+9BemDgRFi+Gbt2Aiim+iQcawJWKU57NBYFSmwWW2f798K9/Qdu28PXXMGoUbNhgDTzlNpVZpUxkHIP0mUSpOOSvuaC/ZoHu/HaPz8uD116zAvahQ3DDDTByJDRo4DU9FTqRcQwTE+j0GiGQkZFhlixZUmn7U0qVlJ3rpJNHebNLaoqjWHmzryDt7QYgAhMGd6TjsjkwdCj8+qs1mcLzz1sz44QoTfFIRJYaYzI8P4/fM6JUnAqkvLlvx6Y+c+mv9z/HHi62eIVj+x2/kHDR/bBtrVVk8sUX0LNnQB1xQjqRcRzRs6JUnAmkvNlfp56bJy0hwS0mNzq0mwfmTeTKdXPZm3oSS4aNJmPUA5BYtvBSWq9OVZKeGaXiTCDlzf5y6QWFhtxCQ1puDrcv/IibFk/DiPDq+X/njU7XMODCM8koY/B28derU5WkAVypOBPIKIKvzNroM5dunE4GrPqae76fTJ2cQ3xyRneeu3AAO2qcohWOlUwDuFJxxmd5MzCgUzNembWR3YdzqZrk4Fi+WxA3hm6bl/DwnHdpuXcrPzVpx/XXjGB1g5ZFq4RyJnpVOm2FolScOprrLCpvNhgmLcgChJy8AqomJXAs/8R0O613b2H47PFcmLWcLSc34OluN/BNy/OKKijdKxy123voaSsUpVQxrvJmVxO+nLwTAdsVvOtm72foD5O5asU3HKqSxhM9bub9Dr3IdyQVrZvkEIZd2oqrz26iFY6VTM+2UlEg2Dkl/fFWYVk17zg3L/6UW3/6mJRCJ1//pR+PnnEFe5PTSnw/yZFASqJDg3cY6BlXKsIFNadkANybFYop5Oo1s7h/3iTqZ+/ni9M7s23o4xxo0IS9czd7/b52dQ8fDeBKRbByzylZBq5mhe03LuORWeM4Y/dmVjQ4nTv7DGXdn87i8TNbU9Pgs+khgKHy6tLUCaX+8iIyHugN7DbGtLM/qwV8ADQHsoC/G2MOVFwylYpP/tpj5zsL+b/JS7m0XYNiRSplLW65vMph6k99nG4bF7G9xincffkDzGjTFSMJpNqtSgwwKtN700OASQuyuOvillqMUskCGY1wAtDT47NhwHfGmJbAd/Z7pVSI+es1mVdgmLthLyMz19Fp9EwWZ+1ncdZ+Oo2eycjMdbwxd3OxZSXs2QN33knq2el03bmOf/e4nt53vM30thdRNSWp2CiEaSmJDDivmZ+Uig75Ggal3i6NMfNEpLnHx32AbvbricAcYGgI06WUwn+vSRfXskHjf0IQjuaVUtxy/Di8/DI89RQcPQq33IJjxAhurVmLRv66sfsZ0kTLwcOjvM879YwxOwGMMTtF5JQQpkkpZfPXa9KTs8CAj7JoYyBz5e/03bwAHtVYdE0AABtySURBVHoIsrLgsstg7Fhr4CkgFfx2Y9chXyNPhU/oICK3iMgSEVmyZ8+eit6dUjHF1WsyNcVBSqL/P9e8AkNegfcA3nrzGroM6gP9+kHNmjBzJmRmFgXvQFTkjD2qfMobwHeJSAMA+//dvlY0xrxljMkwxmTUrVu3nLtTKn51bF6L2UO6lbpeskNIdhSPsE0O/sGr08bwyeQHqLV3J4wfD0uXQo8eZU6H+80kZDP2qKCU94xPBwYBY+z/PwtZipSKU/5aj8z6eTeOBP/jaic6BEHIKyigxvFs7vzxAwYtm0FBgoPXLuzP4E9ehdonBZVGHfI1sgTSjHAKVoVlHRHZDjyOFbg/FJEbga3AtRWZSKViXWmddfy1RgFITICJN3RC8vP55q4R/N+8ydQ4ls209pfwSveBjL2nF6lBBm8XHfI1cgTSCqWfj0VlfwZTSpUQSGcdfxWIyQ7hkV5t6LhiHjz4IBkbN7IrozMTB91PtU7nkKk55Jilv6pSYRbIFGf+WqO03/0r/R96Br6fB61bw4wZ1LvsMq4PYCozFd00gCsVZoFMcQYw4LxmvPP9ZhJEyCswtDi+n3tnT6TPqu+gTh1rFvibb4akJK/bUrFHA7hSYeaveMQhMH3F77w1bzMJCeAshJOdOdyz8GNuWTKNRIw1A/xDD1nNA1Vc0QCuVJj5Kx4pMPD7oeMAmPwCrlv1LUPmv0/dowf5vF03uv/vbaq1Oq1C01eRQ9mq4OivoFSYFU1xNn5RsW7w7rpuWcbDs8bReu9vLG7UlpuvepQ1jVtzwQ/7uPRQcoUF1Xkb9nDzpCUUFBqchYaqSaEdylYFR6dUUypCTPxxC099vr5Yb8rT92QxfPa7dNuylN9Oqs+YiwbzZasLcO8SWVHTmc3bsIeB4xd5XZaa4gjJULYqMDqlmlIRbueh40XBu272Af41/336rvqW7OSqjOp+I++d3Zu8xJIVlO6DWQ3t2Yadh44FXdSRnevkpkmLfS4vLLRax2h78PDSAK5UJfEsS+7e+hRm/7y76H1efiEp+bnctHgat//0P1KceUw8uzcvX3AdB6vWKHX7OXmFPPX5OvIKTNCz9mSu3EFhoe+n82P5OvpgJNAArlQl8OxpmZKYwLBPVpOSmECus5BqifDXFbOYPW8SDY/s5avTz2fMRYPJqtWoTPtx5eCDnbUna99RnIW+lycmiI4+GAE0gCtVwbz1tMy1o2Ous5BOW1fz8OxxnPXHJlbWb8m9l9/PoibtQrJvV0cgz6KO0lqWNK+dStWkhKLZ6T05EkRHH4wAGsCVqmC+elo23/87D815l79uXMjv1etyb+8hfNb2IoyEbpRnbxMtBDJJcmnjkL89MEMrMCOA/gJKVTDPnpYnHTvMPT9M4Z/LvyA3MZmxFw5kXEYfcpNSQr5vz4kWAp0kuahp47uLKCw0HMsvJDEBEhKEdwZ15MKWOjR0JNAArlQFc/W0dOYcZ+CyGdz14wek5R1javu/8EKX/uxNPbnC9u050UIg4664ilt06NjIp7+EUj6Eqgdi77Ma8NOzb3LvzPE0O/gHc1qcw+ju17OhbvPQJ9qW7BCSEhNKTLQQ6LgrLjp0bGTTAK6UF4GUE3vyGvBXLCVtyBBe+OEHNpzSnJv+8SQzm6STmCBQaEi0xzepluwgz1lQSsuPE+uCodDAcS+VjEkO4ZHebbj67CYlcss6r2Vs0QCulIfsXCeDxv9ETt6J4FhaszzPgH9azl5q3PAuvdbOhXr14K23aPSPAZw2bwtzvt+Ma3KdBBGSHDDw/GY0qFmVZ7762WtwrZrkoNeZ9TmlepWiooyPl23nsc/Wllj30d5tGXh+c6/H5q9yUue1jD4awJXy8Op3G4sFb3eFhYbHPltD3eopRblsoKhisHruUYYt+JDrl0ynUBJ4o2s/BnzyGqZ6dT5eso1x87fYuWyrINrVbvu9hb8xe0g3xn79s9f9JiTAyD7tim4c2blOnvnK+7rPfPUzV5/d2GtZtXvlpPvThasrvpZvRxf9tZRyk53rZNwPW3wuP5ZfyGcrduAsPNHbccB5zXA4nQxY9jn3zv8vtY8d5uN2F/Ns14EcrlOPPQt38t7CBTgLrGIPb4yB2b/sDji4lqUy0pNWTsYO/cWUcpO5cgelzWPjLHTr7WgMv46fyiezxnPa/u0saHomT3a/kbX17SFe8woYN9/3DcHFVYHYt2PTgIJrWSsjPWnlZGzQAK4UJyogpy7eWmw0QH/a7N7Mw7PG0eW3lWyu1YibrnqUmaedW2ykwEC5VyAGEly1MlKBBnClSlRAluaUI/u4//v3uGb1dxysWp3HLrmVj87pxTEc5U5DWSsQtTJSgQZwFee89Uz0pWrecW5d9DG3LPoER2EBb5/7N147/+/kVKvOBafVZknWAQwnyq7zCwrJLyU376u9dmnSUhIZ2rO111YoQ3u21vLsOKG/sopr/ioDXRIKC7h6zSzu//496mXvJ7N1V565aBDbTqpvrVBoWLRlPyIw8PzmCNZIfcfzCxjz5S8cy/d+c0gQeOSytlx9jvcWI/6UtxWKii36C6u45q8yEOCCrBU8PHscbXdvYVnDVtze5yFWNW1bVJHp4hq1772FvxW1E8/OdTL26198bnvCDeeWe0yRYFqhqNihAVxFHH9d2EM9wa6vysBT925j+Jzx9Ph1Mdtq1uPOKx4ks3VXHAlCo5OqsG3/MbzFT/fgWXxAKGsShMQEwZEgvD0wI6gBoYJthaJigwZwFVH8dWEHyty9vTSelYG1jx7k3h/+S78VX5GTXJXR3a5n4jmXk5uYDFizxPsK3lAyeAbS5jrQm5L7ersP51I1yeG1eEZbocQPndRYVZrSAlV2rpNOo2d6rVBMTXZgMF57SAY7we7irP3c8vZ8Biycxs3zp1I1P5f3O/TipQv6caBazTJtq1qyg8cvbxtw8YW3G5a3CYo91/M32YJOOBx7dFJjVencAzbGKh92b6XhmXv2V66bX+B7lKegynyNoeOCr1n63jAStm1l43kXM/uGIby0LcHqNemjAtKXQJrwuc7Lhl1HmPzT1qLZecD7mCveWsq4B29XTly7xMcf/ZVVhSitbbW3QOWvXNdf55pyl/n+8APcdx8sWkRChw4wcQItu3enJdA/18nHS7fxxIz1FATwlCoCyY4EXu9/jt/gGWibc2eBKbop+buxVU1KKDHIlQbv+BG6uZuUsrnnGEvrGOPKPcOJCkVvkh1CssN7D8cyl/n++itccw106QK//w4TJsCSJdC9e9EqqSmJJCc6Au5UaYzVLPD2yUtZnLXf6zplOS+5zkI27soG/FdYHssv5JTqVRh6aWv6dmyqwTvOaABXIRdI22oX99xz7/YNfQbMJEcCiT4CeMA9Dw8cgCFDoE0b+OorGDkSNmyAQYOs4f48WDOzB15HdCy/kKO5BXaQdpZYXpbzAnAgJw/wf2PTCsv4pgFchVxpbavduQcgV7O71BRHUcCqluwgNcXBhBvOZeINnbwvK63MNy8PXnoJTjsNXngBBg6EjRvh0Uehmu/gZ83MXvbu8e5PFe427DoS8HkBOLma1fLF341Nu83HN33eUiHnb6AlT54BqLRmd2UaBtUYmDYNHnwQNm2CSy6B556D9u0DOo7SZmb3JSevgC9W/8FlZ51oZbM4az/v//RbwNtISUygZb00QMfwVr4F1YxQRLKAI0AB4PTWzMVdrDUjDHWnkljhrzmgi6/mciGzZIlVXDJvnlVk8txzcOmlZR4p0FXpmJtfWKbiFPcxTto0qFHq+fDkrSng0VynjuEdp3w1IwxFAM8wxuwNZP1YCuCBtt+NFqG+GXk7P2CKjRVSIQFo61YYPhwmT4a6da1y7ptugsTy7+dorpPHPlvDZyt+9ztnpTfJicL9fzmdF2du8vtEkpKYQK6zMOqvI1UxNICHkN8OJ1HYiSLYm5Gv4F+pOcbDh2HMGKuM2xireeCwYVCjRkg2H8hThS8OEb9NEQed34y2DWtozlr5VFEBfAtwAGuCvzeNMW95WecW4BaApk2bnvPbb4GXA0aqqYu2MjJznc/B9MvSEy/cgr0Zhf1JxOmEcePgscdg927o3x9Gj4amoT//Xo8VaFkvjRXbDpVrm9F2vajwqKiemBcYY3aIyCnAtyLyszFmnvsKdlB/C6wceJD7iwixNJBQMKPaeeshWNrs7SFjjNUU8P77Yd06dqefyxfDX6HK+efRu15D0nykN5hiIl8VrDNW7mDDLu839NJoKxIVjKD+uowxO+z/d4vIp8C5wDz/34p+sTSdVTA3o7ANabp6tVVB+e23HG/WgqHXPsK3rTqTs7OQapnrvA5w5W+QrLI8KXib7izQ1iqJCZCc6NBWJCpkyn3liEgqkGCMOWK//gswMmQpi2DRNp2Vv5xnMDejSn8S+eMPq+32+PFQsya5zz7HeQdP56AzAeyxQQIdSySUTwquZn7931lIntP7Ha1asoNhPVuTkpSgZd0qZILpyFMPmC8iK4FFwOfGmK9Ck6zI5rfDSYTlqBZn7afT6JmMzFzHG3M3MzJzHZ1Gzyzq7h1MJ5FK6yGYkwNPPml1xJk4Ee65h+y1P/NQkx5kF3rfv3tnmkCeFILVsXkt5j94MSmJ3v+kRODqcxrTt2NT7fauQqbcV5AxZjMQWI+IGBTIOM/hFkjOM5hOIhX+JFJYCO+/bzUL/P13uOoqeOYZFifWYvAbi8jNL/DZrM/9CaCynhROqVGF92/qpB1uVKXRKyoI3spDI0mgZdTlvRlVaA/BOXOscu5ly6BjR5gyBbp2tW5KATTnc38CqMw6i2i4savYoVdVDCtLzrO8N6NQBqzsXCdzp3/P6f8eRcuFsyls0pSEyZPhuuuKBpsKdEAo9yeAyq6ziPQbu4odGsBjWGXlPEMRsJYv28i624bw96VfcDwxmX9ffD3vd+rDm5270tFtpMDSBspKTBBSkhKKPQHoWCIqVumVG8OiorVMbi65/36RU58YxZl5x5iS3pMXL/gH+1JPgsKSrURKGyjrgtNq8x8vkypo0YaKRXr1xrCIznkaAx99BMOGkbJlCwtO7ciT3a5nU53iOXn3mWnAvill+m5z7WsyBfD/pKADk6lopFdojIvInOeCBVYF5YIFcNZZvDfybR492sDrqu4z04B1UxpwXjPemLfZx8alzB2IQtXJR6nKpgE8DlREpVq5cqxbtlgDTH34IdSvb41hMmgQy/+3Gpb/7vNrrplpivgZEbYszQKzc538b+l2nsxcW6w5oqt45p/v/MQ/z2tKy1Oqa45cRaSIvyL10TbylDnHevAgPPUUvPwyOBzWwFMPPABp1oglqT46A7l4/t6hqJx1HUO+s9BnW/JcZyHj5mdpjlxFrIieUq20XoSq8nmbmDcnr8D7XJD5+fDKK1YPyuefh379rKnMnniiKHgDHC1lEKhsj/klg51izP0Y/M127+Lz+JQKs4gN4GUKFKrSBNQt3RiYPh3atYO774azzoKlS63Z3xs1KvG9407/Adw1N6RLWkoir/c/h+REwdVzvWpSQkBDGWTnOnls2hpy88s4MwOh63avVKhEbACvjPErVNmV1jno2MLF0KMH9OljZYenT4fvvoMOHbx+JzvXyXfrd/vcX7JDiuaGdJm3YQ83TVpMQYHBWQgOgUIDr/c/x28Rh+uJbvrK38s0PZr78UXTUMEq9kVsAI+lMbdjia8BrOof3suLX77AoHuutYZ7ffVV6//LL/c7D2Xmyh0k+Jun0qNIZN6GPQwcv4g8p8FV+lFgrPLq2ycv9flk5v5EV9Zp0VyibahgFfsiNoBX2kh3ZZSd62Tqoq2M+XI9UxdtLVE+G+s8y5+r5R3jX9+/z+y3b6XX2rnk3zfEmgH+jjsgKanU7WXtO8qxfN9FKL3aNSgqEsnOdXLzJN9T8hUWGp9PZoF2wU9yCEk+6lQjpvOTUraIbc4Rib0Itb3wic5BN4xbwJUrvuWu2ZM45egBvjjjQhq9/iLtu3ovKvHFX4uSqkkOzj+1dtH7zJU7KPBT9HEsv7DYk5l7C6b1Ow8H3AUfiMzOT0p5iNirMdJ6EYZ1+rAw8Nd8s+OGJaycNoyENWv4vU06M+8fz0X9e5fr+P3dqBMSit+os/Yd9Vt2nZhA0ZOZ5802yeG7mCYxAfqkN2Rkn3ZFxxBxnZ+U8iKir8hI6kUYtunDwsDXk8bUC2pw5guj4KuvSGjRAj78kEbXXEMjf2XYpSjLjbp57VSqJjl8FrkkJAi9z2ro9Wab76e5YEqSo1jwBh1RUEWHiA7gEDl/SPFSqeot+FU7sJd/zZ9M21HfYGpUR559luxbbifz531kffVz0B2sAr1Rlzb35DuDOpKaksjURVsDKu8GqxORFo2oaKVXbYBiaSJjf9yfNFLyc7lxyWf838KPSHHmMSWjN6lPPUHjlk0Z/Pz3Ia0LCORG7Z5bLyyEY/kFJCYIjgTh7YEZXNiyLlD6kLMuyQ5h6KWt4qb+QsUeDeABisRK1YqQte8ox3LzuXLdXB6YO4lGR/bwTcvzeLrb9Wyp1YgbjyfzSCXUBfgqgw8kt17akLMueQWGnQdzg06rUuGiATxAkVapWlE6bl3LZe8N58ydG1ld71SG9P4XC5ueVbR8054jIa8L8AzWDU+qyu2Tl/rM4ZeWWy+tqMUllp6cVHwSE2hhYQhkZGSYJUt8t+ONBkdznRFRqRpyGzfC0KHw6afsrFGHsV0HMu2Mbhgp3lUgMQG/HWFuv+hUhl7aOuDdelaYVk1K4JiPbu6pKY6Ac/iu7bqKWoLdnlLhJCJLjTEZnp/rlVtGkVKpGjL798OoUfDaa5CcDE8+yeQ2l/Lpop1eV08QIdmB10Ggypqj9VZh6it4Q9ly+O5FLQt+3ccXa3biEOFYfmFMPjmp+KRXb7zKy7OC9qhRcOgQ3HgjjBwJ9evj/HK9768VGJ9tqstaFxBo70iXsrb2cd1s+3ZsylO5Z8bmk5OKa3oFxxtj4JNPrOKSX3+Fv/zFGuq1XbuiVUprcTPw/Ga8t/C3oOsCAm0t4r7v8pZZx9yTk1JoAI8vixZZU5nNnw9nnAFffgk9e5ZYrbQWN3dd3JK7Lm4ZdI420NYi7vuOldY+SoVCxA5mpULot9/gH/+ATp1gwwZ4801YscJr8IYTLW5SUxxFA4pVS3YUG2/blaMdemlr+nZsWu5u9P46cVZN8r5vpZRFW6HEskOH4Omn4cUXrezrffdZc1JWrx7Q1yujxY23bvsi8Po/z2HnwWNaZq0UvluhaACPRU4nvP02PP447NkD//wnjB4NTZqEO2VexWzTTKVCRJsRxgNj4IsvrAmD16+HCy+03meU+N0jilYwKlU+WgYeK1auhD//GXr3tnLgn34Kc+ZEfPBWSpWfBvBot2OH1Ya7QwdYvhxeegnWrIErr/Q7lZlSKvppEUq0OnoUnnsOxo6F/HyrgvLhh+Hkk8OdMqVUJQkqBy4iPUXkFxHZJCLDQpUo5UdBAbz7Lpx+OowYAb16WeXdzz2nwVupOFPuAC4iDuA14FKgLdBPRNqGKmHKi1mzrDLtG26wWpTMnw8ffQSnnhrulCmlwiCYHPi5wCZjzGZjTB4wFegTmmSpYn7+GS6/HHr0gAMHYMoUWLAALrgg3ClTSoVRMAG8EbDN7f12+7NiROQWEVkiIkv27NkTxO7i0J49cMcd1jgl8+bBmDFWML/uOq2gVEoFFcC9RZASvYKMMW8ZYzKMMRl169YNYndx5Phxq3LytNOsbu+33gqbNlkDUFWpEu7UKaUiRDCtULYD7l37GgM7gktOnDMGPvgAHnoIsrKsNt1jx0KbNuFOmVIqAgWTA18MtBSRFiKSDFwHTA9NsuLQjz/C+edDv35QsyZ89x3MmKHBWynlU7kDuDHGCdwJfA2sBz40xqwNVcLixubN8Pe/WxWSW7daTQSXLoWLLw53ypRSES6ojjzGmC+AL0KUlvhy4AA89RS88gokJlptuu+/H1JTw50ypVSU0J6YlS0/H15/HZ54wgrigwfDk09CQ52oQClVNjoWSmUxBqZNs2bCuecea+ySZctg/HgN3kqpctEAXhmWLoXu3eFvf7OKSzIz4dtvIT093ClTSkUxDeAVads2GDjQ6v6+dq01C/yqVXDZZdoRRykVNC0DrwhHjsAzz1izvRtjTWM2bJjVPFAppUJEA3iozZtnNQvctctq0z16NDRvHu5UKaVikBahhFrLltC+PSxcCP/9rwZvpVSF0Rx4qDVoAF9/He5UKKXigObAlVIqSmkAV0qpKKUBXCmlopQGcKWUilIawJVSKkppAFdKqSilAVwppaKUBnCllIpSYkyJeYgrbmcie4DfKm2HJdUB9oZx/xVBjyl6xOJx6TFVjmbGmBKzwldqAA83EVlijMkIdzpCSY8pesTicekxhZcWoSilVJTSAK6UUlEq3gL4W+FOQAXQY4oesXhcekxhFFdl4EopFUviLQeulFIxQwO4UkpFqbgL4CIySkRWicgKEflGRBqGO03BEpFnReRn+7g+FZGTwp2mYInItSKyVkQKRSQqmnT5IiI9ReQXEdkkIsPCnZ5QEJHxIrJbRNaEOy2hIiJNRGS2iKy3r717wp2m0sRdAAeeNcacZYxJBzKBx8KdoBD4FmhnjDkL2AA8FOb0hMIa4CpgXrgTEgwRcQCvAZcCbYF+ItI2vKkKiQlAz3AnIsScwBBjTBvgPOCOSP+t4i6AG2MOu71NBaK+FtcY840xxmm/XQg0Dmd6QsEYs94Y80u40xEC5wKbjDGbjTF5wFSgT5jTFDRjzDxgf7jTEUrGmJ3GmGX26yPAeqBReFPlX1zOiSkiTwEDgUNA9zAnJ9RuAD4IdyJUkUbANrf324FOYUqLCpCINAc6AD+FNyX+xWQAF5GZQH0vix42xnxmjHkYeFhEHgLuBB6v1ASWQ2nHZK/zMNZj4OTKTFt5BXJMMUC8fBb1T32xTETSgI+Bez2e2CNOTAZwY8wlAa76X+BzoiCAl3ZMIjII6A30MFHSuL8Mv1M02w40cXvfGNgRprSoUohIElbwnmyM+STc6SlN3JWBi0hLt7dXAD+HKy2hIiI9gaHAFcaYnHCnRxWzGGgpIi1EJBm4Dpge5jQpL0REgHHAemPMv8OdnkDEXU9MEfkYaAUUYg1te5sx5vfwpio4IrIJSAH22R8tNMbcFsYkBU1E/ga8AtQFDgIrjDF/DW+qykdEegEvAg5gvDHmqTAnKWgiMgXohjX06i7gcWPMuLAmKkgi0gX4HliNFR8AhhtjvghfqvyLuwCulFKxIu6KUJRSKlZoAFdKqSilAVwppaKUBnCllIpSGsCVUipKaQBXSqkopQFcKaWi1P8DbrOzG5DDgWAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Training a scikit learn model follows these simple steps.\n",
    "\n",
    "#Import the classifier class we want\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "#Initialize a classifier object\n",
    "#This will initiate it with defautl parameters \n",
    "cls = LinearRegression()\n",
    "\n",
    "#Fit the model based on the training data\n",
    "#First re-sahape X to be column rather than a flat array\n",
    "X = X.reshape(-1,1)\n",
    "cls.fit(X,y)\n",
    "\n",
    "#And that's it! \n",
    "#We can now use the fitted model to make predictions\n",
    "x_range = np.arange(-2.5,2.5,.1).reshape(-1,1)\n",
    "y_pred = cls.predict(x_range)\n",
    "\n",
    "#And we can plot this to see how well it matches\n",
    "plt.figure()\n",
    "plt.title('Sample regression problem with one input variable')\n",
    "plt.scatter(X,y, marker= 'o', s=50,label='Training Data')\n",
    "plt.plot(x_range,y_pred,'r',label='OLS Fit')\n",
    "plt.legend()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "linear model coeff (w): [3.59574108]\n",
      "linear model intercept (b): 7.598\n",
      "R-squared score (training): 0.750\n",
      "R-squared score (test): 0.750\n"
     ]
    }
   ],
   "source": [
    "#We can also get the parameters of the fit\n",
    "print('linear model coeff (w): {}'.format(cls.coef_))\n",
    "print('linear model intercept (b): {:.3f}'.format(cls.intercept_))\n",
    "print('R-squared score (training): {:.3f}'.format(cls.score(X, y)))\n",
    "print('R-squared score (test): {:.3f}'.format(cls.score(X, y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use linear regression to fit higher order polynomials by mapping a single feature to higher order featuers.  That is, $y = \\beta_0 + \\beta_1 x$ becomes $y = \\beta_0 + \\beta_1 x + \\beta_2 x^2$ where $x$ and $x^2$ are now treated as two unique features by the ordinary least squares model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x20a74ceec40>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gU5fbA8e9JJwkdBEILAtJDLyooIHYQFL2IqDQbol6vXhuigojt6vWqP7vSUayoFBsCotIRRKQKhiIIGIqEhCSbvL8/ZnfZbHY3m2ST3U3O53nyJLszO/PO7OTMzJm3iDEGpZRS4Sci2AVQSilVPBrAlVIqTGkAV0qpMKUBXCmlwpQGcKWUClMawJVSKkxpAC8CEZkgIrOCXY6yICK9RGRbsMtRmMK+ExFJFZF+ZVkmX0TkdRF5xMf0kDzGRORXEekd7HIEmogME5Gv/Zw35I61sAjgItJTRJaLyHEROSIiP4pI12CXqzwzxnxvjGkR7HKUN8aY24wxkwBEpLeI7At2mfxhjGljjFla2usRkWki8kRpr8fBGDPbGHNRWa0v0EI+gItIFWA+8DJQA6gPTASyglmukhCRqFBeXigpz9umgqs8HFshH8CBswCMMe8ZY3KNMZnGmK+NMRsBRKSpiCwWkTQR+UtEZotINceH7bc194nIRhE5KSLviEgdEflCRE6IyCIRqW6fN1lEjIjcIiL7ReSAiNzrrWAi0sN+Z3BMRH72dYtpL8cDIrIROCkiUb4+LyJNRGSZSxlfcdy+uZRztIjsARbb3x8lIltE5KiIfCUije3vi4i8ICKH7HcxG0WkrX3aZSKy2b6eP0Tk3/b3810dikgrEVlqL+uvInKFy7Rp9vItsC9nlYg09bIffO5j+23qRyIyS0T+BkaISJKIfG6/+/pNRG52W2yciLxvX/dPItLey7ojRORBEdlpP14+EJEabuUaKSJ77fvwNhHpat9fx0Tk/7wsN05EMkWklv31eBGx2S8+EJEnROR/LvvqCRFJAL4AkkQk3f6TZF9kjIjMsG/PryLSxdN67cs7R0TW2L/XNSJyjsu0pSIySaw71hMi8rWjjPbpRT1++7l8Rx94K6N93ofsx9VREZkqInH2aSNE5Ae3ZRsRaSYitwDDgPvt+2Oeh3K8LiLPub33mYjcY//b8f2esK//Spf5Rtj3xQsicgSY4F4eEXnR/v3/LSLrRKSXWxFKfKwFlDEmpH+AKkAaMB24FKjuNr0ZcCEQC9QGlgH/c5meCqwE6mBdvR8CfgI62j+zGHjMPm8yYID3gASgHXAY6GefPgGYZf+7vr1cl2GdCC+0v67tZTtSgQ1AQ6BSYZ8HVgDPATFAT+Bvl3U7yjnDXs5KwCDgN6AVEAWMB5bb578YWAdUA8Q+Tz37tANAL/vf1YFO9r97A/vsf0fblz3OXp6+wAmghX36NOAI0M2+7tnAHC/7wZ99nGPfngj7tn0HvArEAR3s81/gNv/V9nL+G/gdiHbZ745l3411LDTA+u7fAN5zK9fr9vVcBJwCPgXO4PSxc76X7VoGDLb//TWwE7jUZdqVLvvqCfd97LKcCfb1XgZEAk8BK72sswZwFLjBvt+H2l/XtE9fai/HWfb9uBR4ugTHbz9/ymifdxPWsV4D+NFlm0cAP7gt2wDN3PePl3KcB+wFxOWYzQSS7K+vAZLs2zQEOMnpY30EYAPutO+vSu7lAa4Hatqn3wv8CcQF8lgLaHwMRlAuciGtgDMN2Gf/Aj4H6niZdxCw3u1gGuby+mPgNZfXdwKfuv0Tt3SZ/izwjssX6AiiDwAz3db9FTDcxz/AKJfXXj8PNLJvZ7zLtFkUDOBnukz/Ahjt8joCyAAaYwXc7UAPIMJtnXuAW4Eqbu/35nQA72U/kCNcpr8HTHD5p3vbZdplwFYv+8GffbzMZVpDIBeo7PLeU8A0l/lXum2360kpldP/VFuwB37763pY/5BRLuWq7zI9DRjiduzc7WW7JgEv2Zf1J/BP4Gmsk0EmUMtlXxUWwBe5vG4NZHpZ5w3Aarf3VgAj7H8vBca7TLsd+LIEx28/f8pon/c2t+Nhp/3vEZQsgAvWMXue/fXNwGIf828ABrqse4/b9ALlcZt+FGgfyGPN27qK8xMOKRSMMVuMMSOMMQ2AtlhnWMct6RkiMkes2/+/sQJdLbdFHHT5O9PD60S3+fe6/L3bvj53jYFr7Lefx0TkGNaVcj0fm+K6XF+fTwKOGGMyvHzW2/JedFnWEayDvb4xZjHwf8ArwEERedNxew8MxvoH2y0i34nI2R7WkwTsNcbkuby3G+sqzuFPl78zKLhPfZXdfR+7TnPsixM+1u2c317GfXj/zua67KMtWCeHOi7zFPVYcfgOKyB3An4BvgHOxzpp/maM+cvL5zxx35dx4jlfm4S1L1z5+70U5/gtShn9+R8qMmNFwzlYdxsA12Hd8QEgIjeKyAaXbWpL/njg6f/ISUTuFSsNedz++arePh+AY63EwiKAuzLGbMU6S7e1v/UU1hk8xRhTBesWSEq4moYufzcC9nuYZy/WFUw1l58EY8zTvorv5+cPADVEJN5Lmbwt71a35VUyxiwHMMa8ZIzpDLTBuqW+z/7+GmPMQKw0wafABx7Wsx9oKCKux0sj4A8f21oYX/vYdbv2Y+2Lyj7W7VyWvYwN8P6dXeq2j+KMMSXZDoflQAvgSuA7Y8xmezkvxwrunhgv7/trP1agcOXv91Kc47covH2/JwHncS0idd0+588+eQ+4WqxnPN2x7oywv34LuAMrjVQNK5XjGg+8Lt+e734A+AdWqrYacNzt86FwrDmFfAAXkZb2s2ID++uGWGfflfZZKgPpwDERqY89MJXQIyISLyJtgJHA+x7mmQUMEJGLRSRSrAdZvR3l9IPXzxtjdgNrsR6yxNivigcUsrzXgYfsZUZEqorINfa/u4pIdxGJxvoHOgXk2pc9TESqGmNysPLsuR6Wvcr+uftFJNr+sGsA1pVQcfmzjzHG7MUKjk/Z91EKMBqXqy6gs4hcZb8CvBurhtLKgkvjdWCynH64W1tEBpZgG1zLmYH1nGEspwP2cqz0lLcAfhCoKSJVi7nahcBZInKdWA/Fh2ClM+b78dmSHr+FGSsiDewP7sZx+vv9GWgjIh3EerA5we1zB4EzfS3YGLMe6znI28BXxphj9kkJWAH6MICIjOT0hZ4/KmOlLg8DUSLyKNYzOFdBP9ZchXwAx3pY1h1YJSInsXbWJqwHDGBVKeyEdaZcAHwSgHV+h/XQ7lvgOWNMgYr+9sAyEOvgPIx1xr0PP/epH58fBpyNlYd9AusfwGvVSWPMXOAZYI49lbQJ66EvWAfhW1j5vN32ZTqe5N8ApNo/cxvWHYz7srOBK+zL+wvrgeKN9ruh4ip0H7sYipWj3g/MxXro/I3L9M+wHlg5HuhdZT8huXsR6/nJ1yJyAutY6l6CbXD3HdbDrdUurytjPcQswL7/3gN22W+1i5RmMMakAf2x/hfSgPuB/v6ka0p6/PrhXayHubvsP0/Y17sdeBxYBOwAfnD73DtAa/v++NTH8t8D+tnXg33Zm4HnsZ4DHMR6QP5jEcr8FdazpO1Y/yenKJhyCZVjDTj9JFdhVSXj9FNlW3BLk5+IvI/1YPCxYJelJEJ5H6vAEJFU4CZjzKJgl6W8C4cr8ArJnvZoaq9PegnW1ZKvKxKlVAUT9i2RyrG6WOmgmlhPusfYc39KKQVoCkUppcKWplCUUipMlWkKpVatWiY5ObksV6mUUmFv3bp1fxljaru/X6YBPDk5mbVr15blKpVSKuyJiHuLW0BTKEopFbY0gCulVJjSAK6UUmEq6PXAc3Jy2LdvH6dOnQp2UVQFEhcXR4MGDYiOjg52UZQqtqAH8H379lG5cmWSk5MRKWkngkoVzhhDWloa+/bto0mTJsEujiqn0rNszP95P6lpJ0mumUD/9kkkxgY25AY9gJ86dUqDtypTIkLNmjU5fPhwsIuiyqk1qUcYMXU1xkBGdi7xMZFMWrCZaSO70TU5cCOrhUQOXIO3Kmt6zKnSkp5lY8TU1ZzMyiUj2+qdOSM7l5NZufb3A9eHW0gEcKWUKi/m/7wfbz2UGAPzN3oa/6F4KnwA37t3L3369KFVq1a0adOGF198scjL6N27t8cGStnZ2dx99900bdqU5s2bM3DgQPbt2+dhCQVNmDCB5557rvAZvVi6dClVq1alY8eOtGrViokTJ/qcf8SIEXz00UfFXl9JXHbZZRw7dsznPMnJyfz1V8FurpOTk2nXrh0dOnSgQ4cOLF++3Lm8Y8eO8eqrr5ZWsZXyKDXtpPPK211Gdi6pf2V4nFYcFT6AR0VF8fzzz7NlyxZWrlzJK6+8wubNmwOy7HHjxnHixAm2b9/Ojh07GDRoEFdddRXuHYgZY8jLy/OyFP/YbAVvy3r16sX69etZu3Yts2bNYt26dSVaR2lZuHAh1apVK/bnlyxZwoYNG9iwYQPnnHOOc3kawFUwJNdMID4m0uO0+JhIkmvFe5xWHBU+gNerV49OnToBULlyZVq1asUff1jD1vXu3ZsHHniAbt26cdZZZ/H9998DkJmZybXXXktKSgpDhgwhMzOzwHIzMjKYOnUqL7zwApGR1pc5cuRIYmNjWbx4MampqbRq1Yrbb7+dTp06sXfvXiZPnkyLFi3o168f27Ztcy5r586dXHLJJXTu3JlevXqxdas1EM6IESO455576NOnDw888IDXbUxISKBz587s3LmTDRs20KNHD1JSUrjyyis5evRovnm//fZbrrzySufrb775hquuugqAxMREHn74Ydq3b0+PHj04eNAa73f37t1ccMEFpKSkcMEFF7Bnzx5n+caMGUOfPn0488wz+e677xg1ahStWrVixIgRznW4Xl0PGjSIzp0706ZNG958883Cvj6PHMt78MEH2blzJx06dOC++wIx0p5ShevfPglvj1hEoH9KQMZ3BkKgFko+d98NGzYEdpkdOsD//ufXrKmpqaxfv57u3U+PfGSz2Vi9ejULFy5k4sSJLFq0iNdee434+Hg2btzIxo0bnScAV7/99huNGjWiSpX8Q+p16dKFX3/9laZNm7Jt2zamTp3Kq6++yrp165gzZw7r16/HZrPRqVMnOnfuDMAtt9zC66+/TvPmzVm1ahW33347ixcvBmD79u0sWrTIeZLwJC0tjZUrV/LII48wdOhQXn75Zc4//3weffRRJk6cyP9c9k/fvn0ZO3Yshw8fpnbt2kydOpWRI0cCcPLkSXr06MHkyZO5//77eeuttxg/fjx33HEHN954I8OHD2fKlCncddddfPqpNfbE0aNHWbx4MZ9//jkDBgzgxx9/5O2336Zr165s2LCBDh065CvrlClTqFGjBpmZmXTt2pXBgwdTs2ZNn99bnz59iIyMJDY2llWrVjnff/rpp9m0aRMbAn1MKeVDYmwU00Z2K1ALRQSmjexGQgCrEoZWAA+i9PR0Bg8ezP/+9798Qddx9dm5c2dSU1MBWLZsGXfddRcAKSkppKSkFFieMcZjTQfX9xs3bkyPHj0A+P7777nyyiuJj7dur6644gpnuZYvX84111zjXEZW1umhMa+55hqvwfv777+nY8eORERE8OCDD9KgQQOOHTvG+eefD8Dw4cPzLRes2hk33HADs2bNYuTIkaxYsYIZM2YAEBMTQ//+/Z3745tvrGEpV6xYwSefWEOR3nDDDdx///3O5Q0YMAARoV27dtSpU4d27doB0KZNG1JTUwsE8Jdeeom5c+cC1vOJHTt2FBrAlyxZQq1atXzOo1RZ6ppcg9Xj+jF/435S/8oguVY8/VOSAhq8IdQCuJ9XyoGWk5PD4MGDGTZsmDNgO8TGxgIQGRmZL89cWDW0Zs2asXv3bk6cOEHlypWd7//0008MGGANMJ+QkJDvM56WmZeXR7Vq1bxeRbovw1WvXr2YP//0AOXHjx/3WWaHkSNHMmDAAOLi4rjmmmuIirIOk+joaGcZ3feHt+1w7L+IiAjn347X7p9funQpixYtYsWKFcTHx9O7d29toavCVkJsFEO6NirVdVT4HLgxhtGjR9OqVSvuuecevz5z3nnnMXv2bAA2bdrExo0bC8yTkJDA8OHDueeee8jNtZ5Iz5gxg4yMDPr27etxmXPnziUzM5MTJ04wb948AKpUqUKTJk348MMPneX9+eefi7WtVatWpXr16s5c/syZM51X466SkpJISkriiSeeyJer9uacc85hzpw5AMyePZuePXsWq3zHjx+nevXqxMfHs3XrVlauXFms5ThUrlyZEydOlGgZSoWyCh/Af/zxR2bOnMnixYudVdEWLlzo8zNjxowhPT2dlJQUnn32Wbp16+Zxvqeeeoq4uDjOOussmjdvzocffsjcuXM9Xml36tSJIUOG0KFDBwYPHkyvXr2c02bPns0777xD+/btadOmDZ999lmxt3f69Oncd999pKSksGHDBh599FGP8w0bNoyGDRvSunXrQpf50ksvMXXqVFJSUpg5c2axqmICXHLJJdhsNlJSUnjkkUec6aXiqlmzJueeey5t27bVh5gquNaXznC2ZTomZpcuXYx7fektW7bQqlWrMiuD8s8dd9xBx44dGT16dLCLUmr02FNlYsEC6N8fPvgA3J45+UtE1hljuri/H1o5cBUSOnfuTEJCAs8//3ywi6JUeDt6FG6+Gdq2BXvFhEDSAK4KCNUGP0qFnX/+Ew4dgvnzweUhfqBU+By4UkqVis8/h5kzYdw48NBWJBA0gCulVKAdOQK33gopKTB+fKmtRlMoSikVaHfdBX/9BV98ATExpbYavQJXSqlA+vRTmD3buvJ2a2kcaBU+gKelpTnrf9etW5f69es7X2dnZ/v87Nq1a51N6n0555xzAlJW1y5iW7RowXnnnZevpaWvzy1fvjwgZVBK+ZCWBrfdZgXuceNKfXVhl0IJ9DhzNWvWdDZTnzBhAomJifz73/92TrfZbM6m5O66dOlCly4FqmYWEMjg6do8fsOGDQwaNIhKlSpxwQUXeP3M0qVLSUxMDNiJRCnlxR13WEH8q6+gDAbMDqsr8DWpR+j+5CIen7+Z17/bxePzN9P9yUWsST0S0PW4d9O6evVqzjnnHDp27Mg555zj7Op16dKlzs6dJkyYwKhRo+jduzdnnnkmL730knN5iYmJzvl79+7N1VdfTcuWLRk2bJizb/CFCxfSsmVLevbsyV133eVcri8dOnTg0Ucf5f/+7/8AmDdvHt27d6djx47069ePgwcPkpqayuuvv84LL7xAhw4d+P777z3Op5QqofffhzlzYMIEaN++TFYZNlfgruPMOThGvRgxdTWrx/ULaE9frt20/v333yxbtoyoqCgWLVrEuHHj+Pjjjwt8ZuvWrSxZsoQTJ07QokULxowZQ7TbWXj9+vX8+uuvJCUlce655/Ljjz/SpUsXbr31VpYtW0aTJk0YOnSo3+Xs1KkT//nPfwDo2bMnK1euRER4++23efbZZ3n++ee57bbb8t1ZHD161ON8SqliOnAAbr8duncHH33zB1rYBHB/xpkLZM9frt20Hj9+nOHDh7Njxw5EhJycHI+fufzyy4mNjSU2NpYzzjiDgwcP0qBBg3zzdOvWzflehw4dSE1NJTExkTPPPJMmTZoAMHToUL8HM3DtCmHfvn0MGTKEAwcOkJ2d7VyeO3/nU0r5wRi46SbIzITp08FLyrU0FJpCEZGGIrJERLaIyK8i8k/7+zVE5BsR2WH/Xb00C1qW48xB/m5aH3nkEfr06cOmTZuYN2+e1y5OXbtL9dbdqqd5StIfzfr16539edx5553ccccd/PLLL7zxxhtey+nvfEopP7z9NixcCM88Ay1alOmq/cmB24B7jTGtgB7AWBFpDTwIfGuMaQ58a39daspynDl3x48fp379+gBMmzYt4Mtv2bIlu3btcg4Y8f777/v1uY0bNzJp0iTGjh1boJzTp093zuferaq3+ZRSRfT773DPPdC3L9j/D8tSoQHcGHPAGPOT/e8TwBagPjAQcPz3TwcGlVYhoWzHmXN3//3389BDD3Huuec6+/YOpEqVKvHqq69yySWX0LNnT+rUqUPVqlU9zusYZadFixaMHTuWl156yVkDZcKECVxzzTX06tUr3wg1AwYMYO7cuc6HmN7mU0oVQW4uDB+OiYjg838+wdNfbWPO6j2kZ3ke6KQ0FKk7WRFJBpYBbYE9xphqLtOOGmMKpFFE5BbgFoBGjRp13r17d77pRenSc03qEa/jzHVNruH3doSi9PR0EhMTMcYwduxYmjdvzr/+9a9gF6tc0+5kVYk8/zz8+9+MG3gPn6ZcWKoxqcTdyYpIIvAxcLcx5u/ChhRzMMa8CbwJVn/g/q7Pk7IaZy4Y3nrrLaZPn052djYdO3bk1ltvDXaRlFLe/PIL5uGHWdzibN5t0Qfsz+dKs2acJ34tXUSisYL3bGPMJ/a3D4pIPWPMARGpBxwqrUK6Kotx5oLhX//6l15xKxUOsrJg2DBOJVTh0cvvwlNutzRqxnniTy0UAd4Bthhj/usy6XNguP3v4UCxx/kqy1GBlAI95lQJjB8Pv/zC/Lse54/oyh5nKY2acZ74UwvlXOAGoK+IbLD/XAY8DVwoIjuAC+2viywuLo60tDT9h1JlxhhDWloacXFxwS6KCjeLF1u57zFjyL30sqDVjHMoNIVijPkB8Jbw9t4Bh58aNGjAvn37OHz4cEkXpZTf4uLiCjSyUsqno0dh+HBo3hyee47+kTFMWrDZ46ylXTPOIehP/6Kjo7UloFIq9N1+O/z5J6xYAfHxJGLVNvFWM64sKlcEPYArpVTIe/ddq6OqSZPApQfSYNeMK1I98JLq0qWLWbt2bZmtTymlSmz3bqt3wTZt4LvvyrSvEwdv9cDDqjtZpZQqUzYbXH895OVZAxQHIXj7ElqlUUqpUPLUU/DDD1bwPvPMYJemAL0CV0opT5Yvh4kTYdgw6yo8BGkAV0opd8ePW4G7USN45ZVgl8YrTaEopZQrY2DMGNi7F77/Hrz0DBoKNIArpZSrWbPgvfesKoNnnx3s0vikKRSllHLYudMamKFXL3jooWCXplAawJVSCiA7G4YOhchI6yo80nM/J6FEUyhKKQUwbhysWQMff2w9vAwDegWulFILFli9DI4dC1ddFezS+E2vwJVSFVJ6lo35P+/nr227uOmuG4lOSSHyueeCXawi0QCulKpwHOPrSm4ub818iLyTGVzR804m/JlB1+Tw6SdeUyhKqQolPcvGiKmrOZmVy+il73L2nl945MIx/Fq5nv39shtVvqQ0gCulKpT5P+/HGOi+5xfuWj6Hj9v04eN21tg0jrEsw4UGcKVUhZKadpK4Y0d4cd5/2F2tLo9eOMY5LSM7l+0H04NYuqLRAK6UqlCaVIvjpQXPUz3zBHcMfJCTsfnHrpy5MpU1qUeCVLqi0QCulKpQBn05g567fmJCv1vZXKdgF7HZNhM2uXAN4EqpCiPzq0VEP/E4a8+9lA87XuJ1vnDJhWsAV0pVCOvXbOHk4H/we/X63NhtFL4Gk8zIziX1r4wyK1txaT1wpVS5l56RRfY/hpKQlcF1QyaREVMJXxE8PiaS5Frx3mcIEXoFrpQq91LvvJ/uqT/zyEVj2F47udD5RaB/SlLpF6yE9ApcKVW+ffUVbaa+zIdt+/FRu34eZ4mKAFuedeUtAtNGdiMhNvTDY+iXUCmliis1Fa67juNNW/BU/7Ee0yaVoiO5rF1dzqgcR3KtePqnJIVF8AYN4Eqp8urUKbj6arDZiP70E7I+2A1ZuQVmi4iAxwe2DZug7Upz4Eqp8umf/4R162DGDBLatLKnRSKJj7EGaoiPiSQhNjJs0iWehGeplVLKl2nT4M034cEHYeBAALom12D1uH7M37if1L8ywi5d4kn4llwppTzZsMEaVb5PH2tgYhcJsVEM6Roeo+34Q1MoSqny4+hRGDwYataEOXMgqnxfo5bvrVNKhSzHiDipaSdJrplA//ZJJJYknZGXBzfcAHv2wLJlcMYZgStsiNIArpQqc44RcYyxmq3Hx0QyacFmpo3sRtfkGsVb6MSJ1tiWr7wCZ58d2AKHKE2hKKXKlOuIOBnZVrW+jOxcTmblFr8XwM8+g8cfh5Ejrfx3BaEBXClVphwj4nhSrF4At261UiddusCrr1rt4CsIDeBKqTKVmnbSeeXtrsi9AP79NwwaBHFx8Mkn1u8KRHPgSqkylVwzgfiYSI9BvEi9AOblwY03wm+/wbffQsOGAS5p6NMrcKVUmerfPslrlkMoQi+Akydbue///hfOPz9g5QsnhQZwEZkiIodEZJPLexNE5A8R2WD/uax0i6mUKi8SY6OYNrIbcdEFw0+uMWw+8HfhC/nsM3j0Ubj+erjzzlIoZXjw5wp8GuBp7KEXjDEd7D8LA1sspVR51qpeFSI8XIWfyskrvCbKL79YgbtrV6u5fAV6aOmu0ABujFkGhMcQzUqpsDD/5/1YCZOCfNZE+esvq2+TypVh7lyoVKn0ChkGSpIDv0NENtpTLNW9zSQit4jIWhFZe/jw4RKsTilVXhSrJkpODvzjH7B/vxW869cv5VKGvuIG8NeApkAH4ADwvLcZjTFvGmO6GGO61K5du5irU0qVJ46aKJ44aqKkZ9mYs3oPT3+xhTmr95B91z9hyRIrbdK9exmXODQVqxqhMeag428ReQuYH7ASKaXKvf7tk5i0YLPHaSJQr1oluj+5yNnUfvimr4hZ8BoHbr6DejfeWMalDV3FugIXkXouL68ENnmbVyml3DlqongaYOG1YZ0ZM2uds6l99z2/8PAXr7G0SWcuqnNJ8Zral1OFXoGLyHtAb6CWiOwDHgN6i0gHrBHmUoFbS7GMSqlyyNsAC/Ncmto3Prqf1+c+yZ5qdbnrivvIlUjmb9xfrvr0LolCA7gxZqiHt98phbIopcJUcbuG9TTAguMBZ5VT6Uz56HGMCKOufoy/4xKhqE3tyzltSq+UKpFAdw2bXDOBKpGGVz99iobH/uT6a59gT3Ura1ukpvYVgDalV0oVW2l0Dds/pTZnNB0AACAASURBVB4Pf/UaPXf/zLhL7mB1w7bOaSJFaGpfAWgAV0oVmaOK39jZ68ix5Xmcp1hdwwKJb7zKkHULefuca1jY+WKgfIwgXxp0TyilisQ9ZeJNkbuGBWtEnXvugUGDGPruLCpv+rPcjCBfGnRvKKX85poyKUyR89Xr18OQIdC+PcyaRUKlmAIPOAM+jmaYq7hbrpQqMl+j6bgrUr567164/HKoUQPmz4eEhAKzlMo4mmFOc+BKKb/56sPEocj56uPH4bLL4ORJK4WSVDDol8o4muWAXoErpfzmazSdmEjhnKa1uLRdXf/z1Tk5cM011riWX3wB7dp5nM2fcTQrYuMevQJXSvnN12g6BujTsjaX+xu8jbFGkP/mG3jjDejXz+usAR1HsxzRAK6U8punPkwccnINz3y5je5PLmJNqh9DCDz1FLzzDowfD6NG+ZzVn94LKyIN4EqpInH0YfLAJS2Jcosgfuelp0+Hhx+GYcPg8ccLXafPcTQrcOMeDeBKqSJLiI0iNiqCmCjPV8U+G/F8+SWMHm2lTKZM8WtINF+9F1bkxj0Vc6uVUiVWlLy0o/72qRWrGPbQCCLatiXy448hJsbv9XnrvbCiBm/QAK6UKiZfNVJc89KO+tsN0vYza+q9HIytzLC+9/PcERtdqxRtnZ56L6zINIWilCoWf/LSjvrbcUfSeGP2eCJNHjdePZHdMVUrdP3tQNEArpTKx30synQvQdafvPT8n/dTKSuTdz6eSJ30I4we/Ci7ajYAit/ZlTpNUyhKKaeiNlcvLC+998+j/Pf9SbT9cye3XvUw6+u3dH62ItffDhQN4EopwHNHVY789oipq1k9rp/HB4Ze89K5uQx56WEapa7n3sv+xbfN8o8kX5HrbweKplCUUoB/zdX9ZgyMHUujbxfw7IU383G7CwrMUpHrbweKBnClFBDg5uqPPGI1j3/oIfq8+bTf9bf9zb8ri6ZQlFKA/9UCC/XCCzB5Mtx8M0yeTFcRv+pva3exRSfG3859A6BLly5m7dq1ZbY+pZT/0rNsdH9ykcfBGhJiI73mwPOZPh1GjIDBg+H99yHSc0vNUll3OSYi64wxXdzf1xSKUgoIQHP1jz6yOqXq1w9mz/Y7eEOA8+8VSMU9pSlVwXkanqzYzdUXLIChQ+Hss+HTTyE2tkhl0e5ii0cDuFIVUGH5Zn+aqztOAHmLFzPksdsgJYXIBQs8DodWmIDl3ysYTaEoVcEUZXgyb7VC1qQeofuTi/js9Y8Z+OgYdlWty3l972fN0cIHO/ZEu4stHg3gSlUw/uabHUH68fmbef27XTw+fzPdn1zEsu2HGTF1NY337OCNdx/hUGJ1rvvHJP6ITCx2/ybaXWzx6F5RqoLxJ9/sq1XmzTPW0uJwKlPfH8+JmHiuHzKZw4lWNb+SjE+p3cUWne4ZpSoYf/LNvq7SmxzazZTZD5ETGcWwa5/gj6pnOKeV9IGjdhdbNJpCUaqC8Sff7O0qvWnaXma++xB5EsF11z5Jao36+abrA8eypQFcqQrGn3yzp0GEk4/8wbtzHkaA4dc/6ewW1pU+cCxbmkJRqgIqLN/cv30SkxZsds7f6OgB3ntvHFG5Nm684RnO7NWVnVsOEilCZk4e8TGRiKAPHMuYNqVXSgEFG/YkVavEmNnrqH/kAFNmPEBCzimGXvskvyc1JcuWR6XoSPKM4dK2dTm7aU194FiKvDWl172tlPLYsEcEpvasTtvrR4HJZuh1k9laqwnY8gDIzLFy5N9sOcjkK9tp8A4CzYErVcF5a9hTZ/9uGg++nEp5OSx7dQ47kpp7/Lz2VRI8espUKgx46rckMUBXvJ6qDDb7aw/vznmYCAxfvfwBG6o2ICN7l8fPa18lwaMBXKkQV9r9ZLtXGTzrcCrvznmYPIlgyLVPcVHVhiTXjPdadxzAUHbP0tRpmkJRKoQVpd+S4nKtMtjq0C7ee28ctohIrh36FAeSkq0aKu2T8FJ1HIAZK1IDUhZVNIUGcBGZIiKHRGSTy3s1ROQbEdlh/129dIupVMXkq0Vkji2P22evKzD0WFGHJXM07On4x1bmvPsQWVExDLnuaXbVbOCs150YG8UNPRr7WIpoHjwI/EmhTAP+D5jh8t6DwLfGmKdF5EH76wcCXzylKjZf/ZZk5xq+2/4Xa1KPOlMqQJHTLYmxUXzcMotGT43nr8TqXDfkCY7UTiLBvV63j0twzYMHR6EB3BizTESS3d4eCPS2/z0dWIoGcKUCzle/JQ6OacOnrEIQTmYX7IBqxNTV3oclW7CAlqOuJa95U9Y/N5UBVPbYkZT22R16ipsDr2OMOQBg/31GIfMrpYrBV78l7my5hpzcPI/TvFb1+/BDGDQI2rQh4rulDLy0Kw9c2pIhXRsVCPbaZ3foKfWHmCJyi4isFZG1hw8fLu3VKVWuuPZbEhvl+981O9eQnes5Ye4xxTFtGlx7LfToAd9+C7Vq+V0W7bM7NBR3jx8UkXrGmAMiUg845G1GY8ybwJtgNaUv5vqUqrC6Jtdgyb296fXsEp/zxURal8eegniBFMdzz8F998FFF8Enn/g9DJr22R1airvXPweGA0/bf38WsBIppQpYvPUQkRG+cylRkYIgZOcWzFE7UxzGwAMPwH/+A0OGwPTpRR6AWPvsDh2FBnAReQ/rgWUtEdkHPIYVuD8QkdHAHuCa0iykUhWBr9aWvmqjAERFwPRR3YGCtVCcvQRGAqNGWamTsWPhxRchMtLrMlXo86cWylAvky4IcFmUqrAKa23pqwZITKQw/vLWzmqCHlMcudlw1VUwbx5MnAiPPILfT0dVyNLuZJUKsvQsG92fXJRv/EmHhNhIVo/rh4FC5/Gahz5yBAYOhB9/hFdegTFjArwFqrR5605Wm9IrFWT+jBJf7Bogu3dDz56wejXMmaPBu5zRR8dKBZk/o8RDMWqA/PQTXH45nDoFX38N559fWpuggkQDuFJB5iu/XSk6kkMnTvH4vF85lpFDtfhozqpTmbF9m/nuTvbLL+Hqq6FmTauOd+vWpbgFKlg0B65UkPnKgYOV53RtXxkbFUFUpHjv32TKFLjlFmjXDhYsgCRtIRnuNAeuVIgqrLWle+P4LFue5+5k8/Ks2iWjR8MFF8CyZSUO3ulZNqb/+DvDp6xi+JRVTFueWmjvhqrs6BW4UiHi0N+n6PnsYrJt/v1PRkUI5zaryaVt69H/rOok3noTfPCBVdf79dchOrpE5VmTeoQb3lnFqZz8p5C46Ahmju4ekMEklH/0ClypELd46yGiIvz/l7TlWd3JvjrnB3a16Yr58EN49ll4++0SB+/0LBvDpxQM3gCncvIYMSUwg0moktEArlSIKKy1pSctD/3Ou+/cTbPDqdx1zXhO3vWvgDTQmf/zfmxeOsYCyMnN0wEcQoDWQlGqjLg3le/T8gyWbD3kfJ3t4WrXlz471/Dy58+SHlOJa657hl0Nz+LRzzZRu3JsiQc+Tk076bVnQ7A6zNIBHIJPA7hSZcC9qXxsVAQPfvILsVERZNnyqBQdQaa/AdwYblv1Mfd/N53Ndc5k9OBHOFi5FuTk8en6P8g1lHjg4+SaCcREitcgHhMpOoBDCNAUilKlzNPAxFm2vHy//Q3ecTmneGnef3jwu2ksaNmTq4c9YwVvO0e8LenAx/3bJxEV6T0VEx0ZoQM4hAAN4EqVMl9N5Ysi6e9DfDT7Afpv+Z5nzh/OnVfcz6noOJ+f8TYST2EDHyfGRjF9VHfioguGiLjoCKaN0gEcQoF+A0qVsuI8nHTXde8mXvv0KWJsOYy++lGWNO3q1+c8jcRTWM+HznUm12Dd+Av5+Ke9LN5ijabVt+UZDO7cQIN3iNBvQalS5s/AxF4Zw/Cf5jN+8dvsqVaXW64az86aDf3+uPtIPK7pHAdfAx8nxEZx49lNuPHsJkUvuyp1mkJRqpQVZWBiV5WyT/HivOeYuOgNlp7ZmStveL5IwRsKDjbsT8+HKnzoFbhSXvgaIacoHE3l3WuhZNnynL/jYyKx5eWBsaroNTnyB6/PnUyztH08e96NvNbjaqKjIiHX2LuTNeQZPDa0AagUHUFEhBToatbfng9VeNAArpQH/uaJXfkK+K5dwW4/mM6xjGxnWqVapRjOqpvItj9PMOXHVC7evpznFrxAdmQ0N/7jcX5M7gDAOU1r0apeFWc3spsP/M2IKavJyc0jO9cQHSmICJe1rcvZTWt67GrWVzqnwMDHKuRpAFfKTVHzxOBfwE+IjeLM2ok8Pn+zxzErJTubx76bwsiVn7Ch3lmMGfQQB6rUBqzgemm7ugUGEzaczocI1tiYw3o09nqS6d8+iUkLNnuc5p5uUaFPO7NSys2c1XuYOO9Xj3WzK0VHcFm7evlaO0Lhw50Z4OO1e3li4RZyPDSOOSvjMAuWv0L0urVM73Q5k/vcRHZUdIHlOE4c/gzD5q2miKeTjeMkoh1UhSZvnVnpFbhSblbsSvPasCYzJ4/PNuzHlmecV9k39Gjs88Hgf7/ZzsyVqdhyrby1u4u3Lec/X7yIiYrgt1em8uzBukQZyHYfVd4lIPvzMNL9at2hyCP7qJCl35hSLtKzbHy56U+f89jsUdiRVnn7+13YvDSkzMjO5Z0ffvc4LcaWw7gl7zDip/n8XLc5Kya/wm2jLmR1lq3Q4FrSh5EJsVFeA7wKHxrAleL0A8gvNh0gr4hpxQgRYiLx2fmTu6Z/7eXF+c/R9uBO3ukykJcuGs24di0A/4KrPoxUoAFcqQI54aJy1ADxizEM2/AF4xe/Q0Z0LDdd9QiLmncnISqySA8Q9WGkAm3Ioyo4Tx1NFVV8TCSjezYhITbSXkfbes89qNc8eYy3PpnE5K9fZU2D1lwy6v9Y1rIHCbGRBXLchXHULY+PiSDGvp6YSCE+JqLIy1LhS79lVaEFoqOpnNw86lWNY8m9vVmy7ZAzd30qJ5env9hGZk4u5+9ax3MLX6DKqZNMvOBmpnUegEREMOHy1iXqW0QQyFeVsOSDOajwoQFcVWjF6WgqKkKcDzIBcnINz3y5jWe/2sa0kd2c+ev0LBv/9/l6xn/1FsM2fMnWWo25fsgTbKudDMD0Ud04r3ntYpXbeefgUvbsXEN2bq7Xuuqq/NEUiqrQHA8D/RUpcHm7urgPHu+p/+3EFT/ww+x/MXTDV7zTYzADh7/AzjpNiI2KYEYJgjdonybKoqdoFXJ8NUkPVP8kDr4eBnoSGSEYBNxSFw7GwBerd3L1R6/CSy8R26wZmYuXkJjYhFEBrHOtfZoo0ACuQoyvJulAkfsnKYynjqZ8EYGFv+z3Wu+79a6N9L32Nti/G+68E556ikoJCQwpVunycz15Hfo7i0rRkWTmaDXCikyb0quQ4bN5eEwkBkNGdsHIWVjTcX+cdGk8YzDMXLGbPAOZObk4KpPkGU/X3JbErAwe+G4aN6xfSHq9BiTOmg59+/q1bn/uKtxPbL7G0AzE/lChRZvSq6ArLFD5yuvm5HofM7KwpuP+cG88c2ff5szfuJ8VO9NY8MsBj/2XOPTbsYpJX7/KGSePMr37lVy9YArUrObXev3pBMtT51quwdtxJe6t2b0qv/RbVqXGNWBjYObK3Ri8BypfeV1frRxLI+ebEBvF5SlJPD5vs9fgXevkUSYsepP+W79nW+3G/GvII9w7/gYSCgnejv2y/eAJZq/a4xzY2LEtkL/XQ18nNqtzrbqcUTlO+zSpgPSbVqWisNaNngKVr+bhjsYqngJ5aeV85/+83+OVf0ReLtdt+JL7l80g1pbF872u59sBwxlyblNa1avic5n+tvq05RrnXYWvE1tmTh5nVI7jgUtbFm3jVLmg1QhVwBWldaNrlTdfQ49FR0YQ5aW5emk1HU9NO1nghJFyYDufzryXJ755jV/qNuWykS/z8jnXsjkti2e+3Eb3JxexJvWIx+UVZb9k2fLYcTAd8F3VUR9YVmwawFXAFaV1o2v6w1EjxL1JekJsJNNGdWP6qO6ep5VSzje5ZoLzyr/KqXQmff0qn864lzrpR7hzwH0MGzI53xiVnuqCuypqq8+jGdmA7xOb9ntSsWkKRQVcUVo3ul9BFtZXdVn2Y92/fRKTP/+FwRu+5N7vZ1I98wRTu1zBCz2HkR7r/arX20PV7QdPFKnVZ/X4GMBzVUd9YKlAA7gqBb5y2e48XUH66k61LPuxTlz+Pas+foD4rb+yqkEbJlx4K1vOOLPQz3l6qLom9QizVu32e92xURE0r5PofK2DMChPSvTti0gqcALIBWye6imqisef1o0hfQW5axf8+98wdy7xjRtz6t05vJzeiC07Pee23UVFwKETp0jPspEYG+XMfWfb/M+fREVKkU5sqmIKxH9OH2PMXwFYTtgJdLPu8sLbLT8Ybjw7GUFC8woyLQ2eegpefhmio+GJJ+Cee4irVIn+q/fw097jft1V2PJg4S9/8uWvfzJtZDd2HkovNPcdGxVBli0vtE9sKuSUqCWm/Qq8i78BvDy1xCxvA8OWxsnopB9Dg4WEjAx46SV4+mn4+28YPhwmT4ak01fAvlqJ+hITJQzp0pCZK/d4nWf42Y1pnVQl9PeTChpvLTFLGsB/B45itTB+wxjzpq/5y0sAL8mI4KGopCejsL0Tsdlg+nR47DH44w/o39+6Am/b1uPsHvcT0CW5Oj/+lpavi1lXURFCZITka7DjEB8TyWMDWmtqRPlUWk3pzzXG7BeRM4BvRGSrMWaZ24pvAW4BaNSofBykJRkRPNR4aqbtqZGNN/40BQ85ubnwwQcwcSJs2wY9esC778J55/n8mLcHiS8v3sF3273fhNryjNfgrtUAVUmUKIAbY/bbfx8SkblAN2CZ2zxvAm+CdQVekvWFivLUlWdJTkYlDf6B4u8dQHpmNj//9y1avPFfau3dRW6bNkR+8gkMGoTXitZuPD1I9KfWTWxUBAZDVESEVgNUAVPsI0dEEoAIY8wJ+98XAY8HrGQhrDyNCF6Sk1Eo3In4dQeQl8dvr88gb+JEzj2UyvaajZg8eBzftD6XqR170NXP4O2NP7Vusmx53NSzCc3rJGquWwVMSVpi1gF+EJGfgdXAAmPMl4EpVmgLt5Zx6Vk25qzew9NfbGHO6j2ku7QSLEkz7WDfiXhqmp6vNeSJDJg6lbxWrWk2diQRNht3DriPS0a9zNxm55Cebby2miwKR62bmCjvJ4L4mEia10lkSNdGPHBpS4Z0baTBW5VYsY8gY8wuoH0AyxI2wqllXGFXqL6uHgs7GQX7TsTbHUCl7FNcv/5rpNkoOHSA42e1ZtJVD/Jp07PJi8h/sgrUnULX5Br8cH9fej27xOPDylA8savwFzqRJsyEQ8s4f3LUJTkZlST4l1R6lo0vNh3Id/KonX6EG35awPUbvqBG5t/saduFRtOn8EZeIz5Z9rvH5QTyTuGMKnHMuql7WJzYVfmgR1QJhHrLOH9z1MU9GQXrTsRxV5Fjv9Jt++dvjFr7Gf23fE9UXi7fNuvG1J7/YODtV9OoayOSV+8pszuFcDixq/JDj6pyrCg56uKejMo6YDmbpWec4sIdqxi+bh7d9/1KekwlZnW8jGmdB7Cnej0SYiN5y34HUNZ3CqF+YlflhwbwcqysctSBClj+VAdcsnAldy+ayqANX1M74xh7q9ZhUt+b+CDlQk7EWt2/JkRF5LsDCKdnFkoVhQ5qXI6FU4tRn61BkxJh/nx4803M11+Th7C4aVdmd7iEZU065Xsw2fus2rwyrJPH7Qqbpv1KudFBjSugcLny9PiwNctGyp872P6PV+myazmSlgZJSfw6+p/cFd+JXZU8t/JsWa+y1+3S1IYqb0LjP1iVmnB4qOb6sLX+8UMM2LKMwZu+pXnaXrIio9nb52Ia3X0bXHwxyblwcPIi8JLbn7EilTv7Ni/y9oVtfy6qQtMjtAII9SvPI1t2MPSHj+i/9Qc6HtgGwNr6rXjo4jtY0LInwy5u7xy0NzEKbujRmNeX7fKyNClyve6w7M9FKTSAq2DZtg3mzYNPPuH2FSsA2FSnKc+cP5wFLXqyp3o956z1qsbl/6yPlu9FrdcdKv25KFUcemSqYilyysFmgx9+sIL2vHmwY4f1focOrLr5Xu6XFuyu7rk6X5Ytf7okULVr0rNsPPrpJrJyPKdjcmx53D57HZe2racpFRWSQr4WiuYmQ49f/YcbA7/9BosWwTffwOLFcPw4xMRA795wxRVW/9uNGzN8yiqf3bH2Pqs200Z1c74ORO0axzZk5eR57erVIdwH61DhLyxroWhuMvR4TTkYw/j/fsbn7SF2xY9W0N5jH4WmUSMYPBguuwwuuggqVy5RGUpau8bTNviiKRUVqkL2SNTcZGhy1BiJzcmi1eFUOu7fSpd9m+nyxxbqpNsH/a1WDfr0gQcfhH79oFkzn/1tN62V6PMKvG+r2gXea1WvCvdf3JLFWw8CwgUtazO4c0O/greVNinY4VRhwm2wDlX+hWwEDIW+ppXdsWOweTP89BOtP1/MxxvW0/yvPUQZKwjuq3IGyxulsK5Ba5oMvJjRt/WHCP96Kk7PsvH+2r1ep8dGCYM7Ncz33rLth7lpxhry8gy2PKgUHcHa3UdolVTV553Z6bRJLh46DCxUuA3Wocq/kA3gwe5ruti2b7dGM2/WzPpp3tz6Xb16sEvmW04O7N0Lv/9u5a63bIFff7UC9/79ztnOql6TNTWasKhZdzbVbcrPdc/izyq1APv4jl1b+x28wX6i9jF95LlN8l1VL9t+mBunrM43T6b9atrXnZm/aZMIsSq55HooVLgN1qHKv5AN4MHua7rYDhyApUth5sz879eoAY0bQ/360KDB6d9160LNmqd/qlTxe3gvv2RnWw8Pjx6FQ4es8rn+OIL2nj3WWJEOCQnQqpWVAmnTxvq7Qwdstety21PfegyExekYyteJGkBc6gymZ9m4eYb3h+B5ecbrnZmvOzpXlaIjMRgysrVPbxX6QjaAB7OvaV8KrRVz/vlWMMzMPH01u2OH9XvvXutnxQpIS/O8gqgo62o9IQHi4/P/REVZGx8RYf0Wgbw8yMqyfk6dsn5nZlpB+9gxyPBypxIVZZ08GjSwBvW97jo480xo0sT63bChxyvpRAho8/yinKjn/7yfXB81RjJz8rzemRV2ooiKEGKjrU6wgJDvfkApCOEAHor9eBSpVkylStC6tfXjSWamlZr4808rmB85Yv1OS7OuljMyrJ+TJ63faWlWXWpjrJ+8POt3RATExlo/CQnWlX5cnPUg0fWnalWoU8cK2vXqWVf7RUh1uApk8/yinKhT0076rPIXFUG+gO96sj30dxaVoiPJ9FDnOyoCBnZI4vGBbZ3bEOrdDygFYVAPPFR6kAunnv0CoSzr3/tVrxyYs3oPE+dt9hiEAWKihPWPXERCbFSBZcZFRXDKy5PL8vj9qfIlLOuBQ+j041GRasWUdf17f6/oCxv9/e3hXUmIjfL4wNJT8I6NiiAqUjQ1osKWHrV+CttaMUXkb/37QF+h+3Oidk2r5eVBZk4uURFCZITw1o1dOK+5VV/c3weWBsOSe/twRpW4wmdWKgRpAPdT2NaKKSJ/7jTOrJ0YtBay/lytF/bA0iEqIoIl2w6VmzsnVfEU7ylWBdS/fZLX2n3lqXpZYXca2/9Md16hO+bLyM7lZFau/X1bqZfRcbX+wKUtGdK1UYH0h+NkW5jydOekKiYN4H5y3L4nxEY6g0N8TCQJsZHlKodaWPD77fCJQq/QAyE9y8ac1Xt4+ostzFm9h/QinBh8nWxdlac7J1UxhXwtlFATKrViSkt6lo3ukxdx0stVeFQEPpuhjzm/qXPwhaKs0zWfnlStEmNmryu0Voovjgexjly5J1r7RIWLsK2FEmpCpVZMaUmMjfI54k2ECDGRkO2hrXlxrmjda7xUio5wNo13KE4nZq658hU701i46QCRImTm5AW9PYFSgaJHryrIR/ohO9cQHel5hqI+C/BU48U9eLsqanVNx8l2SNdGTM5qV67vnFTFpEewKqCwGjc3nt2YmSt3l7iFrL/V/RxK8tCxvN85qYpJA7gqoLDm7Xf2bc6dfZuX+IrW3+p+DvrQUan8NICrAvzth6akV7S+rvQ9KU/VNZUKBA3gyqNAdljlTWFN4x2dT+lDR6U80/8G5VVp5419Xem/dn1nDhzL1IeOSvmg/xEqqMriSl+p8kr/S1TQaQ0RpYpHm9IrpVSY0gCulFJhSgO4UkqFKQ3gSikVpsq0N0IROQzsLrMVFlQL+CuI6y8Nuk3hozxul25T2WhsjKnt/maZBvBgE5G1nrpkDGe6TeGjPG6XblNwaQpFKaXClAZwpZQKUxUtgL8Z7AKUAt2m8FEet0u3KYgqVA5cKaXKk4p2Ba6UUuWGBnCllApTFS6Ai8gkEdkoIhtE5GsRCfsRAkTkPyKy1b5dc0WkWrDLVFIico2I/CoieSISFlW6vBGRS0Rkm4j8JiIPBrs8gSAiU0TkkIhsCnZZAkVEGorIEhHZYj/2/hnsMhWmwgVw4D/GmBRjTAdgPvBosAsUAN8AbY0xKcB24KEglycQNgFXAcuCXZCSEJFI4BXgUqA1MFREWge3VAExDbgk2IUIMBtwrzGmFdADGBvq31WFC+DGmL9dXiYAYf8U1xjztTHGZn+5EmgQzPIEgjFmizFmW7DLEQDdgN+MMbuMMdnAHGBgkMtUYsaYZcCRYJcjkIwxB4wxP9n/PgFsAeoHt1S+Vcj+wEVkMnAjcBzoE+TiBNoo4P1gF0I51Qf2urzeB3QPUlmUn0QkGegIrApuSXwrlwFcRBYBdT1MetgY85kx5mHgYRF5CLgDeKxMC1gMhW2TfZ6HsW4DZ5dl2YrLn20qB8TDe2F/11eeiUgi8DFwt9sde8gplwHcGNPPz1nfBRYQBgG8sG0SkeFA3JpUHgAAAQpJREFUf+ACEyaV+4vwPYWzfUBDl9cNgP1BKosqhIhEYwXv2caYT4JdnsJUuBy4iDR3eXkFsDVYZQkUEbkEeAC4whiTEezyqHzWAM1FpImIxADXAp8HuUzKAxER4B1gizHmv8Eujz8qXEtMEfkYaAHkYXVte5sx5o/glqpkROQ3IBZIs7+10hhzWxCLVGIiciXwMlAbOAZsMMZcHNxSFY+IXAb8D4gEphhjJge5SCUmIu8BvbG6Xj0IPGaMeSeohSohEekJfA/8ghUfAMYZYxYGr1S+VbgArpRS5UWFS6EopVR5oQFcKaXClAZwpZQKUxrAlVIqTGkAV0qpMKUBXCmlwpQGcKWUClP/Dw983oOKztdSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Fit higher order polynomial features\n",
    "from sklearn.preprocessing import PolynomialFeatures  \n",
    "poly = PolynomialFeatures(degree = 2)\n",
    "\n",
    "#Transform \n",
    "X_poly = poly.fit_transform(X) \n",
    "\n",
    "#Fit new model\n",
    "lin2 = LinearRegression()\n",
    "lin2.fit(X_poly,y)\n",
    "y_pred2 = lin2.predict(poly.fit_transform(x_range))\n",
    "\n",
    "#See if that matches data\n",
    "plt.figure()\n",
    "plt.title('Sample regression problem with one input variable')\n",
    "plt.scatter(X,y, marker= 'o', s=50,label='Training Data')\n",
    "plt.plot(x_range,y_pred2,'r',label='2nd Order Polynomial Fit')\n",
    "plt.legend()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "linear model coeff (w): [0.         3.92597696 0.96481572]\n",
      "linear model intercept (b): 6.597\n",
      "R-squared score (training): 0.893\n",
      "R-squared score (test): 0.893\n"
     ]
    }
   ],
   "source": [
    "#And find the fit parameters\n",
    "print('linear model coeff (w): {}'.format(lin2.coef_))\n",
    "print('linear model intercept (b): {:.3f}'.format(lin2.intercept_))\n",
    "print('R-squared score (training): {:.3f}'.format(lin2.score(poly.fit_transform(X), y)))\n",
    "print('R-squared score (test): {:.3f}'.format(lin2.score(poly.fit_transform(X), y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The quadratic fit performed much better, which makes sense since the data was generated from a quadratic model to begin with. \n",
    "\n",
    "For these models we just trained and any scikit-learn models, it also can be useful to save a version of the trained model so we can re-use it elsewhere without having to reload the data and re-train the model.  This can be accomplished with they `pickle` module as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save a copy of the trained polynomial model with pickle\n",
    "import pickle\n",
    "\n",
    "# Dump the trained model with Pickle\n",
    "pkl_filename = 'Data/trained_poly_model.pkl'\n",
    "# Open the file to save as pkl file\n",
    "model_pkl = open(pkl_filename, 'wb')\n",
    "pickle.dump(lin2, model_pkl)\n",
    "# Close the pickle instances\n",
    "model_pkl.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded model ::  LinearRegression()\n"
     ]
    }
   ],
   "source": [
    "#Now wherever we have access to the pickeld file we can read back in the trained scikit model\n",
    "my_model_pkl = open(pkl_filename, 'rb')\n",
    "my_model = pickle.load(my_model_pkl)\n",
    "print (\"Loaded model :: \", my_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See https://dataaspirant.com/2017/02/13/save-scikit-learn-models-with-python-pickle/ for more of a primer on using pickle, which is useful for saving and storing many types of python objects, not just ML models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Prep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Reduction with PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than use iris or mtcars data, lets look at an interesting sample data set of network traffic.  This data is from the Detection of [IoT Botnet Attacks dataset](https://archive.ics.uci.edu/ml/datasets/detection_of_IoT_botnet_attacks_N_BaIoT).  There is data for 9 IoT devices.  We will focus on one of these, SimpleHome XCS7 1003 WHT Security Camera, chosen because it has the most recent data (07-Jun-2018).\n",
    "\n",
    "For this SimpleHome Camera, we have about 18,000 examples of normal traffic.  We also have sample attack data for two types of botnets: Mirai and BASHLITE.  Each of these attack types consists of 5 specific classes of attacks:\n",
    "\n",
    "- <u>Mirai botnet attack types:</u> ack, scan, syn, udp, udpplain\n",
    "- <u>BASHLITE botnet attack types:</u> combo, junk, scan, tcp, udp\n",
    "\n",
    "There are 115 total features related to the flow of network traffic.  See link for full description of what these features represent.\n",
    "\n",
    "One other note on the data.  Each attack class is a 30-100MB csv file, containing from 20k-100k data points.  To keep the data more managable, we've already narrowed the attack data to a csv file containing 2000 data points from each attack class (e.g. 2000 for Mirai ack, 2000 for Mirai scan, etc).  Hence we have 10k total Mirai attacks, and 10k total Bashlite attacks, or 20k total attack records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal traffic has 19528 data points and 115 features\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MI_dir_L5_weight</th>\n",
       "      <th>MI_dir_L5_mean</th>\n",
       "      <th>MI_dir_L5_variance</th>\n",
       "      <th>MI_dir_L3_weight</th>\n",
       "      <th>MI_dir_L3_mean</th>\n",
       "      <th>MI_dir_L3_variance</th>\n",
       "      <th>MI_dir_L1_weight</th>\n",
       "      <th>MI_dir_L1_mean</th>\n",
       "      <th>MI_dir_L1_variance</th>\n",
       "      <th>MI_dir_L0.1_weight</th>\n",
       "      <th>...</th>\n",
       "      <th>HpHp_L0.1_pcc</th>\n",
       "      <th>HpHp_L0.01_weight</th>\n",
       "      <th>HpHp_L0.01_mean</th>\n",
       "      <th>HpHp_L0.01_std</th>\n",
       "      <th>HpHp_L0.01_magnitude</th>\n",
       "      <th>HpHp_L0.01_radius</th>\n",
       "      <th>HpHp_L0.01_covariance</th>\n",
       "      <th>HpHp_L0.01_pcc</th>\n",
       "      <th>label</th>\n",
       "      <th>attack_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.061357</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9.540000e-07</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9.090000e-13</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 117 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MI_dir_L5_weight  MI_dir_L5_mean  MI_dir_L5_variance  MI_dir_L3_weight  \\\n",
       "0               1.0            60.0                 0.0               1.0   \n",
       "1               1.0            60.0                 0.0               1.0   \n",
       "2               1.0            60.0                 0.0               1.0   \n",
       "\n",
       "   MI_dir_L3_mean  MI_dir_L3_variance  MI_dir_L1_weight  MI_dir_L1_mean  \\\n",
       "0            60.0                 0.0               1.0            60.0   \n",
       "1            60.0                 0.0               1.0            60.0   \n",
       "2            60.0                 0.0               1.0            60.0   \n",
       "\n",
       "   MI_dir_L1_variance  MI_dir_L0.1_weight  ...  HpHp_L0.1_pcc  \\\n",
       "0                 0.0                 1.0  ...            0.0   \n",
       "1                 0.0                 1.0  ...            0.0   \n",
       "2                 0.0                 1.0  ...            0.0   \n",
       "\n",
       "   HpHp_L0.01_weight  HpHp_L0.01_mean  HpHp_L0.01_std  HpHp_L0.01_magnitude  \\\n",
       "0           1.000000             60.0    0.000000e+00                  60.0   \n",
       "1           1.061357             60.0    9.540000e-07                  60.0   \n",
       "2           1.000000             60.0    0.000000e+00                  60.0   \n",
       "\n",
       "   HpHp_L0.01_radius  HpHp_L0.01_covariance  HpHp_L0.01_pcc   label  \\\n",
       "0       0.000000e+00                    0.0             0.0  normal   \n",
       "1       9.090000e-13                    0.0             0.0  normal   \n",
       "2       0.000000e+00                    0.0             0.0  normal   \n",
       "\n",
       "   attack_type  \n",
       "0       normal  \n",
       "1       normal  \n",
       "2       normal  \n",
       "\n",
       "[3 rows x 117 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Read in normal traffic\n",
    "df_normal = pd.read_csv('Data/normal_traffic.csv')\n",
    "print('Normal traffic has {} data points and {} features'.format(len(df_normal),\n",
    "                                                                df_normal.shape[1]))\n",
    "#Add label columns\n",
    "df_normal['label'] = 'normal'\n",
    "df_normal['attack_type'] = 'normal'\n",
    "df_normal.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000 mirai attacks data points\n",
      "10000 bashlite attacks data points\n"
     ]
    }
   ],
   "source": [
    "#Read in attack data\n",
    "df1 = pd.read_csv('Data/all_mirai_attacks.csv')\n",
    "df1['label'] = 'attack'\n",
    "df1['attack_type'] = 'mirai attack'\n",
    "print('{} mirai attacks data points'.format(len(df1)))\n",
    "df2 = pd.read_csv('Data/all_bashlite_attacks.csv')\n",
    "df2['label'] = 'attack'\n",
    "df2['attack_type'] = 'bashlite attack'\n",
    "print('{} bashlite attacks data points'.format(len(df2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have n = 39528 records and p = 115 features.\n",
      "\n",
      "Breakdown by classes:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "attack    20000\n",
       "normal    19528\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Combine into a single dataframe\n",
    "df_all = pd.concat([df_normal,df1,df2])\n",
    "print(\"We have n = {} records and p = {} features.\".format(df_all.shape[0],df_all.shape[1]-2))\n",
    "print('\\nBreakdown by classes:')\n",
    "df_all['label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets view the first two principle components with PCA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 2 components accounts for 43.3257618915204% of variance.\n",
      "Explained Variance Ratios of first two principle components: [0.27187671 0.16138091]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\matthew.d.smith\\anaconda3\\lib\\site-packages\\seaborn\\axisgrid.py:243: UserWarning: The `size` parameter has been renamed to `height`; please update your code.\n",
      "  warnings.warn(msg, UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x20a74dda820>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg8AAAGoCAYAAADB1AZpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df5xcdX3v8fcnm4UsAXehIglLMEgxvUpCAnv5IWpBlCARDBSDSFtafUh9WC4/7m1qUC5uASWa2hh6WytcuaaKSKAYwShBI1oLhrrmxyYppiimhBAgCAkaNmSz+7l/nDPZ2ck5s+fs/DhnZl/Px2Mfu3PmzMwnk92Z93x/mrsLAAAgqXFZFwAAABoL4QEAAKRCeAAAAKkQHgAAQCqEBwAAkMr4rAuolvPOO88feuihrMsAAAxnWReA6mualocXX3wx6xIAABgTmiY8AACA+iA8AACAVAgPAAAgFcIDAABIhfAAAABSITwAAIBUCA8AACCVzMODmd1pZi+Y2caiY91mts3M1oVf52dZIwAAGJJ5eJD0VUnnRRxf7O4zw6/v1rkmAAAQI/Pw4O7/KumlrOsAAADJZB4eyrjKzHrDbo3Do04wsyvNrMfMenbs2FHv+gAAGJPyGh6+JOl4STMlbZf0haiT3P12d+9y964jjzyynvUBADBm5TI8uPvz7j7g7oOS7pB0atY1AQCAQC7Dg5lNLrp4kaSNcecCQE31LpMWnyh1dwTfe5dlXRGQufFZF2Bmd0s6S9LrzewZSZ+WdJaZzZTkkrZI+ovMCgQwdvUukx68WurvCy7v2hpclqQZ87KrC8iYuXvWNVRFV1eX9/T0ZF0GgGay+MQgMJRqnyJdR4NoQpZ1Aai+XHZbAEAu7Hom3XFgjCA8AECc9mPSHQfGCMIDAMQ550aptW34sda24DgwhhEeACDOjHnSBbcFYxxkwfcLbmOwJMa8zGdbAECuzZhHWABK0PIAAABSITwAAIBUCA8AACAVwgOQZyyNDCCHGDAJ5BVLIwPIKVoegLxaddNQcCjo7wuOA0CGCA9AXrE0MoCcIjwAecXSyAByivAA5BVLIwPIKcIDkFcsjQwgp5htAeQZSyMDyCFaHgAAQCqEBwDNhYW1gJqj2wJA82BhLaAuaHkA0DxYWAuoC8IDgObBwlpAXRAeADQPFtYC6oLwAKB5sLAWUBeEBwDNg4W1gLpgtgWA5sLCWkDN0fIAAABSITwAAIBUCA8AACAVwgOAfGKZaSC3GDAJIH9YZhrINVoeAOQPy0wDuUZ4AJA/LDMN5BrhAUD+sMw0kGuEBwD5wzLTQK4RHgDkD8tMA7nGbAsA+cQy00Bu0fIAAABSITwAAIBUCA8AACAVwgMAAEiF8AAAAFIhPAAAgFQIDwAAIBXCAwAASIXwAAAAUiE8AACAVAgPAAAgFcIDMFq9y6TFJ0rdHcH33mVZVwQAdcHGWMBo9C6THrxa6u8LLu/aGlyW2MwJQNOj5QEYjVU3DQWHgv6+4DgANDnCAzAau55JdxwAmgjhARiN9mPSHQeAJkJ4QL7ldVDiOTdKrW3Dj7W2BccBoMkxYBL5ledBiYXHX3VT0FXRfkwQHLKuCwDqwNw96xqqoqury3t6erIuA9W0+MQgMJRqnyJdt7H+9QAYDcu6AFQf3RbILwYlAkAuER6QXwxKBIBcIjwgvxiUCAC5RHhAfs2YJ11wWzDGQRZ8v+A2BiUCQMaYbYF8mzGPsAAAOUPLAwAASIXwAAAAUsk8PJjZnWb2gpltLDp2hJl938yeDL8fnmWNAABgSObhQdJXJZ1XcmyBpFXufoKkVeFlAACQA5mHB3f/V0kvlRx+v6Sl4c9LJc2ta1EAACBW5uEhxlHuvl2Swu9viDrJzK40sx4z69mxY0ddCwQAYKzKa3hIxN1vd/cud+868sgjsy4HeZLX3TgBoAnkdZ2H581ssrtvN7PJkl7IuiA0kDzvxgkATSCvLQ8PSLoi/PkKSd/OsBY0mlU3DQWHgv6+4DgAoGKZhwczu1vSTyVNM7NnzOwjkhZKeo+ZPSnpPeFlIBl24wSAmsq828LdL4u56py6FoLm0X5M0FURdRwAULHMWx6AqmM3TgCoKcIDmg+7cQJATWXebQHUBLtxAkDN0PIAAABSITwAAIBUCA8AACAVwgMAAEiF8AAAAFIhPADIFpuYAQ2HqZoAssMmZkBDouUBjYtPrI2PTcyAhkTLAxoTn1ibQz02MetdFoSRXc8E+5uccyO/I0CFaHlAY4r7xPq9T2RTD0YnbrOyam1iVgiZu7ZK8qGQSSsVUBHCAxpT3CfTvpd4Y2gktd7EjG4RoCYID8he2rELvcskK/OryxtD46j1Jmb16BYBxiDGPCBbacYu9C4LuiX6Xip/n7wxNJZabmLWfkzYZRFxHMCo0fKAbCVtVi6EjJGCg8QbA4bUulsEGKMID8hW0mblqJARhTcGFKt1twgwRtFtgWwlbVZO1BVh0kkf4o0Bw9WyWwQYo2h5QLaimpVl0gnnDj+UqCvCpScfrlZlAIAYhAdka8a8oLVAVnTQpfXfGD7rIjJkRGCwJADUHOEB2XvyYUk+/FjpoMnSvmtrib4vG8c6DwBQY4x5QPaSDpos7rsuneJZ4AMsUw0ANUbLA7I3miWKCy0RUS0QrCAIADVFeEA2ileV3LtbGtc6/PokUy5nzJN8MPq6qBkcAICqIDyg/ko3K+p7STKT2o5Q6rn45VonPncc4x8AoAYY84D6i1rwaWCvdNBE6RO/Tndf59wo3X+lDhhwKQWhhPEPAFB1tDyg/qq5WdGMeYoMDgWMfwCAqiM8oP5GM0Cy7P1NKX894x8AoKoID6i/am9WNOICUsbYBwCoIsID6q/amxUV7q/tiJgTnK4LAKgicy/TX9xAurq6vKenJ+sykLXu9pgrTOreWddSAEgavvY8mgQtD2guca0PbYfXtw4AaGKEBwAAkArhAc2l7+V0xwEAqREe0FyqPQ0UAHAAwgOaS7WngQIADkB4QHOp9jRQAMAB2NsCzWfGPMICANQQLQ8AACAVwgMAAEiF8AAAAFIhPAAAgFQIDwAAIBXCAwAASIXwAAAAUiE8AACAVAgPAAAgFcIDAABIhfAAAABSITwAAIBUCA8AACAVwgMAAEiF8AAAAFIhPAAAgFQIDwAAIBXCAwCU07tMWnyi1N0RfO9dlnVFQObGZ10AAORW7zLpwaul/r7g8q6twWVJmjEvu7qAjNHyAABxVt00FBwK+vuC48AYRngAgDi7nkl3HBgjCA8AEKf9mHTHgTGC8AAAcc65UWptG36stS04DoxhuR4waWZbJP1W0oCkfe7elW1FAMaUwqDIVTcFXRXtxwTBgcGSGONyHR5CZ7v7i1kXAWCMmjGPsACUoNsC6THvHQDGtLyHB5f0sJn93MyuLL3SzK40sx4z69mxY0cG5Y1BhXnvu7ZK8qF57wQIABgzzN2zriGWmR3t7s+a2RskfV/S/3D3f406t6ury3t6eupb4Fi0+MQwOJRonyJdt7H+9QCSlq/dpkUrN+vZnX06uqNN82dP09xZnVmXhYBlXQCqL9ctD+7+bPj9BUnfknRqthWNYYWuiqjgIDHvHZlZvnabrr9/g7bt7JNL2razT9ffv0HL127LujSgaeU2PJjZRDM7rPCzpHMl8dE2C8O6KmIw7x0ZWbRys/r6B4Yd6+sf0KKVmzOqCGh+eZ5tcZSkb5mZFNT5DXd/KNuSxqioJXqLMe8dGXp2Z/TvZtxxAJXLbXhw96cknZR1HVD5Lon2Kcx7R6aO7mjTtoigcHRHW8TZAKoht90WyJHYJXrDQZIEB2Ro/uxpamttGXasrbVF82dPy6gioPkRHhBv2CDJkgHTdFXUBmtopDZ3VqduvXi6OjvaZJI6O9p068XTmW0B1FCup2qmwVTNKisMkhw21sEkOV0VtRL1nLe2SRfcxnONRsZUzSZEywOiRQ6SdLoqainqOe/vC44DQI4QHhAtbpAk6znUTuwaGmWmyAJABggPiBY7SJL1HGrGWtIdB4CMEB4Q7Zwbg/72YgySrC0fSHccADJCeEC0GfOCgXrtUyRZ8J2Be7XVPiXdcQDISG4XiUIOzJhHWKinc26Mnm1Baw+AnKHlAcgLWnsANAhaHoA8obUHQAOg5QGopZFWjGRFSQANiJYHoFZKV4zctTW4XPCda6W9u4cu79oqLf948DOtDwByjOWpgVrZvy9IibYjpNd2SYMxUzDbjpA+8eva1gbUD8tTNyFaHoBaiVuNs++l8rcb6XoAyBhjHoBaYTVOAE2K8ADUStQqnZbgT67tiNrUAwBVQngAaqV03YaDJko+OPLt3vu5mpcGAJUgPIxFTA+svcJzfP+VweWLbx8+syJO60RmWgDIPcLDWFOYPrhrqyQfmj5IgKieqOd4+ceS3faCL9a0NGAsM7NPFv3cYWYfr+C+vmpmlyQ891ozO2S0j5VHhIexZtVNw/dOkILLq27Kpp5mFPUcx03LLNb1EVodgNr6ZNHPHZJGHR5SulYS4aHAzN5TrUJQJ3HTB+OOI73RPpfv+7vq1gGMYWa23Mx+bmabzOxKM1soqc3M1pnZXZIWSjo+vLzIzA41s1VmtsbMNpjZ+4vu60/NrNfM1pvZ1yIe6+awJeKA91Qzu1rS0ZIeMbNHzOwjZra46PqPmtnfmdlUM/uFmS0NH+u+QmuFmZ1iZj8O/z0rzWxyDZ6yVCpd5+Erko6tRiGok/ZjohcuYlph9cQ9xwDq6cPu/pKZtUn6maQ/lHSVu8+UJDObKunEosvjJV3k7q+Y2eslrTazByS9RdKnJJ3p7i+a2bDpUGb2eUntkv7cI1ZddPfbzOx/Sjo7vP1ESb1m9tfu3i/pzyX9RXj6NEkfcfdHzexOSR83syWS/l7S+919h5ldKukzkj5cvacqvRHDQ/jkRV4l6feqWw5qjm2fay/qOR7XkqzrAkC1XG1mF4U/T5F0wgjnm6TPmtk7JQ1K6pR0lKR3SbrP3V+UJHcvXsXtf0t63N2vTFqUu+82sx9Kep+ZPSGp1d03hGFmq7s/Gp76dUlXS3pI0omSvm9mktQiaXvSx6uVJC0P75D0x5J+V3LcJJ1a9YpQW4U+9VU3Bc3r7ccEb3b0tVdP3HN8/0fjb8PaDkDVmNlZkt4t6Qx3f9XMfiRpwgg3u1zSkZJOcfd+M9sS3sYkxe3j8DNJp5jZESWhYiT/V8H4i19I+n9Fx0sfx8PH3+TuZ6S4/5pLEh5WS3rV3X9ceoWZba5+Sag5tn2uvajnuHQjrGKs7QBUU7ukl8Pg8AeSTg+P95tZa9hd8FtJh5Xc5oUwOJwt6Y3h8VWSvmVmi939NyVB4SFJKyWtMLNz3f23MfUUHqvQevG4mU2RdLKkGUXnHWtmZ7j7TyVdJunfJG2WdGThuJm1Snqzu28a7ZNTDSOGB3d/b5nr3lndcoAm07ssbIEoMwaCWRZAtT0k6WNm1qvgzXd1ePx2BeMN1rj75Wb2qJltlPQ9SZ+T9KCZ9Uhap6BVQO6+ycw+I+nHZjYgaa2kPys8kLvfa2aHSXrAzM5395KpVvsf93tmtt3dzw6PLZM0091fLjrvCUlXmNmXJT0p6UvuvjecEnqbmbUreN/+oqRMw0PVdtU0s59m2azCrprIndItuaOwgyaaH7tqRjCz70ha7O6rwstTJX3H3U/Msq6kqrnOw0j9SagVVozMp6j1Hkr1vVz+egBNJVyc6j8l9RWCQyOq5pbc1WnCQDq9y6TlH5cG+4PLu7YGlyWawrOWZL0HpsgCTcPMviXpuJLDn3D3lYUL7r5T0ptLb+vuWxTMqmgI1QwPqLfeZeHeCSW5bbBf+t4nCA9ZG2m9B6bIAk3F3S8a+azmUM3w0LD9WtOXTpeKx36YacMVG7IrKIlCf3pcg09fmllDqImo9R4Ks77apzBFFkDDSrJI1O9LOqpo4YrC8XdIetbdfxUe+pMa1Fdz+4ODFWUfd01fOj1ZgNg/mr6CNRNGcx9J+tORLdbUANCkkrQ8fFHDNxMp6Auvu0CS3H1jFeuqn9LgIAWXk8xCKR1NX9ihUkr+BhF3H0+vlp58eOhN54RzpfXflPoTbOsssehQXrCmBoAmlGS2xVR37y096O49kqZWvaIGsOKpFTr3vnM1Y81NOveow7ViYtFmaWl2qOxdJn3rY9G7XPZ8ZfiWzj1fSR4cWg5i0SEAGCUzu9DMFqS8TZeZ3Zbi/Klm9qGiyzPN7Pw0j1lyf1vCPTnqIknLQ7kpmG3VKiSPVjy1QkvWLNFzu5/TpImTdM3J10iSuh/r1p6BPZKZtreOV/frg0/5c3a/GtwwySj7QouDV3m/g7YjguDAp10AGBV3f0DSAfs6mdl4d98Xc5seSWkWG5oq6UOSvhFenimpS9J3UxWbkSTh4Wdm9lF3v6P4oJl9RNLPa1NWHRW6KErGPMhsKCRI2r57u7of69aE8RP2HyvYM26clhzeMRQekky/q+qYBaM/HUDTmbpgxYckfVbB7s1PS/rkloVzvlH+VvHChZgeUrDs8+mS1ivYW+JvJL1B0uXu/u9m9meSutz9KjP7qqSXJM2StMbM7lHQZd+moPv+z919c7ifxl+5+/siHvNrkiaGh65y98cUbAn+38xsnaS7Jf2lgi3D3y7pVkm/jnmcFgWrYc5WMGL+Dnf/+6LHa5P0LUn/Uvq+XU1JwsO1Ctb1vlxDYaFL0kGSGn5ayoYrNkTOtpg8cbK27x6+cdmegT0HBIeC58a3BD8knX6XpHUiqYtvJzQAaCphcLhDUqFf+I2S7pi6YIUqCRCSfl/SByRdqWBjqw9JerukCxWM75sbcZs3S3q3uw+Y2eskvdPd95nZuxWEmz8q83gvSHqPu+8xsxMUBIUuSQtUFDbM7HmFgSW8HPc4VypYS2JWeF3xALdDJX1T0j+7+z+nfmZSSLK3xfOS3hZuFFJYwGKFu/+wloXVU9SsihlLZ0ScGc8kzZg6RZMO6tA1h07UnJFuMNIaAGkUxlgwqh9A8/ishoJDwSHh8UrCw6/dfYMkmdkmSavc3c1sg+LH8d3rvr+PuV3S0jAIuKTWER6vVdL/MbOZkgYUsUBUjLjHebekfyp0n5Ts5vltSZ9397sSPsaojThg0swmmNm1ChLPXgUbdTRNcIgzaeKkyOPtB7VrQsuBw0AGzeRm2t6/S92PdWvFUyvKP8A5NwatFNVQmKFRPMDy238pfe44lqwG0KiOTXk8qdeKfh4sujyo+A/UxaPVb5b0SLgHxQUaeWuG6yQ9L+kkDbXaJxH3OOW2CH9U0nvNSqcQVl+S2RZLFfyDN0h6r6S/rWlFOXHNydccEBImtEzQ9addr+63dWvyxMmxt90zsEdL1iwp/wAz5kkX3BYsFlQpazlw/MTA3nChKB8KFwQIAI3j6ZTH66Vd0rbw5z9LeP52dx9UsB5S2Md9wJbgUVuERz3Owwp2DB0vSSXdFjdK+o2kf0xQV0WShIe3uPsfu/uXJV0iaUxswz3nTXOGhYRxNm5YKHj4kofLBojndj838oPMmCddt1Hq3jX8q+sjyQttbUs2YyPNFFIAyN4nJb1acuxVRa87VE+fl3SrmT2qoSBQzj8q2GZ7tYIui0IrRq+kfWa23syuk/SIpLeY2Tozu7TM4/xfBQGq18zWKxizUexaSRPM7POj/PclMuKW3OG+5yfHXc6LWm3JveKpFcNmXUhBC0T327p1/U+ul8e0Hk2eOFkPX/Lw6B84ajvn1jbppA8NXzzqnBvDsQ5Jxk+Y1L1z9DUBQHqjbkKv9mwLVE+S8DCgoaRkCqaNvBr+7O7+uppWmFCtwsPb7367du3ddcDxQqtD6YyMgoXvWKg5bxpx2GR5SZetjgoaUdqnBC0dAFA/DbvvEeIlmW2RpFmmKa14akVkcJDiQ4MkXTrt0sqDg5R8aePSPRTaDpde++3QNt0SOzgCAKqGLbnLuPXxW1PfxrIK2aVBoxobdgEAEIHwUEZcq0M5Ltc9m++RJN1w+g3VLik5NmQCANRIktkWGIV7//PerEsAAKAmaHmQdMPyDbr78a0acFeLmS47bYpumTtdrdaqfu8f+Q4iDPpglasEACAfxnx4uGH5Bn199dCaIwPu+y/vi948LZFxRqMOAGA4M9uiYA+LF7OupRJj/h3urtXRi5Xdtfrp2DUcpJHDwalHnVpRXQCAfCms6ghaHmLjgSsICHHdDyN1S/zXb/+rssIAYKzrbj9gkSh176pokahwi+zvKdiW+20KloB+v6Rpkv5JweZbv5L0YXd/2cx+JOkxSWdKesDMLpC0VtIpko6U9KeSrpc0XdI97n5D+DjLJU1RsCfFEne/vZK682bMtzyUU0nrQbl1IAAAIwiCwx0KtuK28Psd4fFKnSDpH9z9rZJ2Ktj48Z8lfcLdZyjYy+nTRed3uPsfuvsXwst73f2dCsLGtyX9pYJdp//MzH4vPOfD7n6Kgr2hri463hQID2U88dITo74tYx4AoCLltuSu1K/dfV34888lHa8gIPw4PLZUw/dxuqfk9g+E3zdI2uTu2939NUlPKWhtkILAsF7S6vDYCVWoOzfGfLdFOaNZ56GA2RYAUJFabcktDd+We0BSxwjn7y65XLyNd+kW3+PN7CxJ75Z0hru/GnZ9jLR1d0Ph43GNdBw80u8iAKCMem7JvUvSy2b2jvDyn0j6cZnzR9Iu6eUwOPyBpNMrLTBvCA81MtKGYwCAsuq9JfcVkhaZWa+kmZJuquC+HlLQAtEr6WYFXRdNhW6LMjoO7tDO10a3hfUre1+pcjUAMIZ07/qGutulKs+2cPctCgY3Fi7/bdHVB7QQuPtZcZfd/UeSfhRz7ntjHn9qinJzi/BQxoJTF2jBTxaM6raTJk6qcjUAMMYEQaGisIDaoNuijJ/2jn5czjuPeefIJwEA0IAID2V84/HRj8u5/8n7teKpFVWsBgCAfCA8lDHo0iHjS6cZJ9M/2K+F/76wyhUBAJC93IYHMzvPzDab2S/NbHQDDxLo7Ggre/2NZ9yoFmsZ1X2PdrAlAAB5lsvwYGYtkv5BwWjVt0i6zMzeUovHmj972ojnHNp66P6f21ra1DqutRalAADQEHIZHiSdKumX7v6Uu++V9E0FG5dU3dxZnbHXjX/dWt3wbzcMW2myb6BP/YP9ie67/aD2iusDAFSHmU01s41VuJ9uM/urmOt+F34/2szuC3+eaWbnV/q44X11mNnHiy5PNbNR7/dhZj8ys660t8treOiUtLXo8jPhsWHM7Eoz6zGznh07dlS9iIOPekD7fN+objvexuv6066vckUAgEbg7s+6+yXhxZmSqhIeFCyl/fGiy1MlVWOzsFTyGh4s4tgBSza6++3u3uXuXUceeWT1i2jpS3xux8EdmjxxskymyRMn65a336I5b5pT9ZoAYKyYvnT6h6Yvnb5l+tLpg+H3arxJjjezpWbWa2b3mdkhkmRmN5rZz8xso5ndbmYWHr/azP4jPP+bRffzlvBT+1NmdnXpgxRaOczsIAWrVV5qZuvM7FIzm2hmd4aPt9bMDmhZN7NDzWyVma0xsw1F5yyUdHx4X4vCy+8IL18XPu5PwtutMbO3Fd3nX4f3td7MFpY83rjwebkl0ZOY5KQMPKOhnckk6RhJz2ZUy4gmtEzQglMXEBYAoErCoHCHhnbWfKOkO6Yvna4NV2yoZOGoaZI+4u6PmtmdCj7F/62k/+PuN0mSmX1N0vskPShpgaTj3P01MyvetOgPJJ0t6TBJm83sS+5+QJ+2u+81sxsldbn7VeH9f1bSD939w+F9/ruZ/cDdizfg2iPpInd/xcxeL2m1mT0Q1nOiu88M7+ssSX/l7u8LLx8i6T3uvsfMTpB0t6QuM3uvpLmSTgv33Dii6LHGS7pL0kZ3/0ySJzGvLQ8/k3SCmR0XprYPamgL1Lrxgehpmm0tbcNaGbrf1k1wAIDqqtWW3Fvd/dHw569Lenv489lm9riZbZD0LklvDY/3SrrLzP5YUnE/9gp3f83dX5T0gqSjUtRwrqQFZrZOwfLWE3TgbqEm6bPh/hg/UNB1n+QxWiXdEf477lUw6UAKdvn8f+7+qiS5+0tFt/myUgQHKactD+6+z8yukrRSUoukO919U73reO35C/S6KfcPGyDZOq5Vn37bpwkLAFBbtdqSu7QL3M1sgqR/VNA6sNXMujW0hfYcSe+UdKGk/21mhVBRuq13mvdTk/RH7r65zDmXSzpS0inu3m9mW5RsW+/rJD0v6SQFDQR7ih4zbsfGxxSEpy+4+56Yc4bJa8uD3P277v5mdz8+TRqqpn2vzNLNZ948rJXh5jNvJjgAQO3VakvuY83sjPDnyyT9m4belF80s0MlXSIF4wAkTXH3RyT9tYLBiocqvd8q6N4oWCnpfxSNq5gVcZt2SS+EweFsBd02UfdVerld0nZ3H1SwtXhhoaKHJX24aIxHcbfFVyR9V9K9ZpYoBOWy5SFP5rxpDmEBAOrvkxo+5kGqzpbcT0i6wsy+LOlJSV8KxwDcIWmDpC0Kus6l4I3362bWruCT+2J33xm+56fxiIa6KW5VsE33FyX1hgFii4IxFsXukvSgmfVIWifpF5Lk7r8xs0fDKaffU/B87DOz9ZK+qqAF5V/M7APh4+4Ob/eQmc2U1GNmexWEhf3Ppbv/Xfjv/JqZXR6Gj1jmHteK0Vi6urq8p6dnVLeduiB+D4otCwkOAFCB1O+0BeGgyWFbclc4WBJVQssDACCXwqBAWMih3I55qKeOtujlpuOOAwAwlhEeJHVf+Fa1jhvestY6ztR94VtjbgEAwNhFt4WG9rdYtHKznt3Zp6M72jR/9rSy+14AADBWER5Cc2d1EhYAAEiAbgsAAJAK4QEAAKRCeAAAAKkQHgAAQCqEBwAAkArhAQAApEJ4AAAAqbDOAwA0seVrt7EAHqqO8AAATWr52m26/v4N6usfkCRt29mn6+/fIEkECFSE8ICmxycvNLu43/FFKzfvDw4Fff0DWrRyM38DqAjhAU0t6pPX/PvWq/uBTdrV10+YQMMr17rw7M6+yNvEHQeSYsAkmlrUJ6/+AdfOvn65hl5ol+5/cMkAABV9SURBVK/dlk2BQIXKtS4c3dEWeZu440BShAc0tSSfsAovtEAjKte6MH/2NLW1tgw73tbaovmzp9WjNDQxwgOaWschrYnOoxkXjapc68LcWZ269eLp6uxok0nq7GjTrRdPp5sOFWPMA5qae7LzaMZFo5o/e9qwMQ/S8NaFubM6CQuoOsIDmtquvv4Rz6EZF42sEAyYUYR6IjygqR3d0aZtEV0SLWYadOeFFk2B1gXUG2Me0NSiBoxJ0mETxmvxpTP16IJ38aILACnR8oCmVggGf/PgJr386lAXxs6+flbaA4BRouUBTW/urE4dctCBOZkpmgAwOoQHjAlR4x4kpmgCwGjQbYGmt3ztNpmkqFmbTNFEM2I/F9Qa4QFNb9HKzZHBwSSmaKLpsJMm6sE86So6OdfV1eU9PT1Zl4EcWb52m7of2KSdZdZ62LJwTh0rwlhQz0/9UY+1aOXmyG66ww9p1SEHjc+iNcLq8SCoL8IDmtLytds0/9716h+M//1uMdMX5p3EpzFUTemnfilYhKwWS0LHPVbpJllxalVXBMJDE6LbAk2h9BPY7tf2lQ0OkjTgTnMuqqrcDpfV/h2Le6wWMw0k+FBYq7owNjDbAg2v8Als286+/dtsl+uqKMZ0TVRTuR0u6/VYA+6RC6OluQ9gJIQHNLyoT2Bp8AKKaim3w2W9Hquwc2Yl9wGMhPCAhlfpmz8voKiWqOXQa7XxWrnHmjurU50j/F6zIRwqwZiHGMyTbhxxm18l0TrOeAFF1dR6h8vS16U/OqVTj/xiR+RjRW3VXWxCK58dMXrMtohQzxHTqFzU/1fcolClDmkdp/+4+b01qw2olrjf88tPP1a3zJ0+7LxCwGhva5WZ9PKr/RpnUtQY4okHtegzF9X0tY3ZFk2I6Bmh3Ihp5M/cWZ269eLp6uxokyno800aiV/tH9TytdtqWR7GuOVrt+nMhT/UcQtW6MyFPxz171vU65JLumv10/vvs3Tw8M6+fv1uzz61tlhkcJCk3XsHNP++9fwdIBXCQ4R6jphGdcyd1alHF7xLv144R48ueNeI/b3FCIWolaiZQNffv2FUb9Rxrz+uod/hqIDRP+jqHygfp/sHnL8DpEJ4iNBxSGuq48iX5Wu3afdr+xKfTyhErVSzFbPcwN7C73Alv8vbdvbR+oDECA8R4oaBNMnwkKZW+KSXdJ0HKfjkNnXBCh1//Xd1w/INtSsOY041WzHnz54WO3igECwqnTk02lYRjD2Ehwhxbzxp3pCQjUrWfBhw19dXP02AQNVUc92HubM6dfnpxx4QIIqnXEZN32wdZ2ptSTZmkbFdSIrwUKJc6jbGDOdeNbog7n58axUqAaq/7sMtc6dr8aUzhw0OLp4FFjV4eNEHTtKiS07af+zwQ1pVbpbmaKc9Y2xhnYcS5VI33Rb5F7fmQ+s4qX8w2X0k2RcASKIW6z7MndVZ9vZx1xcfO3PhD2NDQgufkpAA4aEEg+caW9TCOIU1Oq5bti5RAOTFE9U00pt9Fsq9zhGekQTdFiVYqrixRTXbFpp1k74mXnbalJrWCGSt3OtcmmnOGLtoeSgxf/Y0XXvPusjrOtqYqtkIKv2k1/XGI6pYDZA/82dP0/z71h+w/gPLtSMpWh5KzJ3VqTOPj37zeN9Jk+tcDbLAaHM0u7mzOrXokpN0eNHaNR1trVr0gZNy18WCfKLlIcKW30T3Bz7yix11rgRZYLQ5xoI8jsVA4yA8RGB56ubUmXD3TQZMYixg52BUgm6LCNVc2AX5cfYfHJnoPEabI41qbXxVT9XccwNjE+EhQrUXdkE+JO12Opw9TJBQ3t+E44JN3J4b/2vZ+oYKQcgO3RYRarGwC7KXdCwDDQ9IqtzGV1GvF/XsKigEm0J9hWAjxXfBFlrdis/ldQ9RCA8xGEzUfFrMEnVJ7GIPEyQU9ya8bWefzlz4Qz27s0/tba3qHxjU7r0DB5xTyzfocsEmbiXWqHN5HUQUui0wZiQdy8DYFiRV7nel0JWxs6//gOBQ0Nc/oL95cFNNais38DuqazbNfQCEB4wJy9duSzSLgkVykEbSN+FyXn61vybjC8oN/C5diTXub4MgjTiEBzS9Qt9vkpYHFslBGknfhEdSi4XJRhr4PXdWpx5d8C4tvnSmDhofXXfSGUoYe8ybZHRYV1eX9/T0ZF0GcqjcDoLFvnjpTIIDKnLcghUa7SvqloVzqlqLNPIAzdJBlVE6Kx/YycIpTYgBk2h6I/XbmqTLTz+W4ICKJRmIGMWk/QMsqzkLY6SB31GDKktt29mn6+5Zp57/ekm3zJ1ecU1oDnRboOmNtIPg4ktn8qKIqhjtGIjCGhH1Xisi6YBIl3TX6qdZ+wH7ER7Q9OL6fr946Uw9uuBdtDigagpjICpdaKwwTbLW2lPsFOxi0zgMyWV4MLNuM9tmZuvCr/OzrgmNq3RQW2dHm269eDqhATVzyEFBj3Al+6TUY5pk2vKYuomCPI95WOzuf5t1EWgOLPqFeigdgDjgrrbWFh08fpx2plx8rB7TJHe+mr+a0Bhy2fIAAI0oblXHvfvKD0osVa+9dNKEAfb3QbE8h4erzKzXzO40s8OjTjCzK82sx8x6duxItukRANRKXLP+q/2Die/DJP3RKfVpKUsywJOuPkTJrNvCzH4gaVLEVZ+S9CVJNysYo3OzpC9I+nDpie5+u6TbpWCdh5oVCwAJjHaqZjFX8h1gK1UIA/9r2frIRdQ6O9r06IJ31aUWNJbMWh7c/d3ufmLE17fd/Xl3H3D3QUl3SDo1qzoBIKm4mT1pZ19s29lXt2mRc2d16gvzTiq7GiVQKpfdFmY2uejiRZI2ZlULACQVN7Pn0xe8NfUyi9fes043LN9QizIPwIwkpJXL5anN7GuSZipowdsi6S/cfXu527A8NYA8u2H5Bt21+ulhy1e3tpj6B8q/BjfBsuksT92Ectny4O5/4u7T3X2Gu184UnAAgLy7Ze50Lb505rBP94suOWnE29Vqy26gEnle5wEAmsbytdv0Nw9u0svh2grP7urTtfes0ziTBss0Prycci0GoB4IDwBQY8vXbtP8+9YP66Io9BiXCw5AXuWy2wIAmsmilZtHHNsQpyPF/hNAvRAeAKDGKtkTovvCt1axEqA6CA8AUGOV7AnR4DMt0KQIDwBQYyy2hGZDeACAGhtt68E4VkhAThEeAKDGlq/dNqqVkj502rFVrwWoBsIDANRQYZpm2rkWZx5/hG6ZO70mNQGVIjxkZMVTK3TufedqxtIZOve+c7XiqRVZlwSgBkY7TXPLbyrbnROoJRaJysCKp1ao+7Fu7RnYI0navnu7uh/rliTNedOcDCsDUG2jnaZZyfROoNZoecjAkjVL9geHgj0De7RkzZKMKgJQK6OdpjnOrG7bcgNpER4y8Nzu51IdB9C4RjtNc8Bd8+9dT4BALhEeMjBp4qRUxwE0t7gX4v5BV/cD7KqJ/CE8ZOCNh70x1XEAjWvRys1lrzdJg2Wu39nHrprIH8JDBh5/7vFUxwE0rnIDH9taW1JP4QTygPCQAY95uYg7DqBxxQ2YbDHTrRdPV2cF+14AWSE8AEANzZ89TW2tLcOOtbW26AvzTtLcWZ2R1xdjS27kEeEhZ25ZfUvWJQCoormzOve3MJikzo423Xrx9P37Xcyd1amTj22PvG3rOGNLbuSSuTdHU3lXV5f39PRkXUYi05eWX3J2wxUb6lQJgKzdsHyDvr766QOOH9I6Tp+9eEYzbMnN9l5NiJaHDHQc3JF1CQBy4u7Ht0Yef22fN0NwQJMiPGSgWVp7AFRuIOb1IO44kAeEhwy8sveVrEsAkBMtFt2qH3ccyAPCQwZe13pk1iUAyInLTpuS6jiQB+yqWWfL127Ty7v7ZK0SHywA3DI3GEB99+NbNeCuFjNddtqU/ceBPCI81NmilZtlk35LcACw3y1zpxMW0FDotqizckvVAgDQCAgPdRa3VC0AAI2C8FBn82dP0+BrbxCzsAAAjYrwUGf39jytV3/9P7MuAwCAUSM81Nmjv3op6xIAAKgI4QEAAKRCeAAAAKkQHurszOOPyLoEAAAqQnios7s+eoZaedYBAA2Mt7EM7BvMugIAAEaP8JCB9rbWrEsAAGDUCA8Z6B8o3/Sw4qkVdaoEAID0CA8Z2L13oOz1S9YsqVMlAACkR3jIoe27t2ddAgAAsQgPdXbD8g1ZlwAAQEUID3V21+qnJUne35FxJQAAjA7hoc4Km2m+tmM2O2sCABoS4SEj+16ZlXUJAACMCuGhziYe1DJ0wS3ynHH8twAAcox3qTr7zEXThy5YdL/FoFiCEgCQX4SHOps7qzPrEgAAqAjhIQPjonsrAABoCISHDAwyywIA0MAIDwAAIBXCAwAASIXwkCH36Ke/1diyGwCQX4SHDFnMVM19vq/OlQAAkBzhIVPR4cFjjgMAkAeEBwAAkArhAQAApEJ4yFDcttyTJ06ucyUAACRHeMhAYYXJ13bMlg+2DLuudVyrrjn5mgyqAgAgGcJDBoavMDl8cKQ7gyUBAPlGeMhAZ0ebJOngI1fKxg3fQXOf79OSNUuyKAsAgEQIDxmYP3ua2lpbZK07I69/bvdzda4IAIDkxmddwFhU2Jb7xjWHy8e/fMD1kyZOqndJAAAkllnLg5l9wMw2mdmgmXWVXHe9mf3SzDab2eysaqylubM6devZn9CElgnDjk9omcCASQBArmXZ8rBR0sWSvlx80MzeIumDkt4q6WhJPzCzN7v7QP1LrK05b5ojSVqyZome2/2cJk2cpGtOvmb/cQAA8iiz8ODuT0iSmZVe9X5J33T31yT92sx+KelUST+tb4X1MedNcwgLAICGkscBk52SthZdfiY8dgAzu9LMesysZ8eOHXUpDgCAsa6mLQ9m9gNJUaP/PuXu3467WcSxyMUP3P12SbdLUldXFwskAABQBzUND+7+7lHc7BlJU4ouHyPp2epUBAAAKpXHbosHJH3QzA42s+MknSDp3zOuCQAAhLKcqnmRmT0j6QxJK8xspSS5+yZJyyT9h6SHJP1lM860AACgUVmz7KXQ1dXlPT09WZcBABguahwbGlweuy0AAECOER4AAEAqhAcAAJAK4QEAAKRCeAAAAKkQHgAAQCpNM1XTzHZI+q8MS3i9pBczfPwkqLE68l5j3uuTqLFa8l7j6yX9wt3Py7oQVFfThIesmVmPu3dlXUc51Fgdea8x7/VJ1Fgtea8x7/Vh9Oi2AAAAqRAeAABAKoSH6rk96wISoMbqyHuNea9PosZqyXuNea8Po8SYBwAAkAotDwAAIBXCAwAASIXwMEpm1m1m28xsXfh1fsx555nZZjP7pZktqHONi8zsF2bWa2bfMrOOmPO2mNmG8N9R833NR3pOLHBbeH2vmZ1c65pKHn+KmT1iZk+Y2SYzuybinLPMbFfR//+N9awxrKHs/1sOnsdpRc/POjN7xcyuLTmn7s+jmd1pZi+Y2caiY0eY2ffN7Mnw++Ext63533NMfbn6W46pMfeviagid+drFF+SuiX91QjntEj6laQ3STpI0npJb6ljjedKGh/+/DlJn4s5b4uk19epphGfE0nnS/qeJJN0uqTH6/x/O1nSyeHPh0n6z4gaz5L0nax+/5L8v2X9PEb8vz8n6Y1ZP4+S3inpZEkbi459XtKC8OcFUX8r9fp7jqkvV3/LMTXm/jWRr+p90fJQW6dK+qW7P+XueyV9U9L76/Xg7v6wu+8LL66WdEy9HruMJM/J+yX9swdWS+ows8n1KtDdt7v7mvDn30p6QlJnvR6/ijJ9HkucI+lX7p7lKrCSJHf/V0kvlRx+v6Sl4c9LJc2NuGld/p6j6svb33LMc5hEpq+JqB7CQ2WuCpsR74xp5uyUtLXo8jPK7k3owwo+hUZxSQ+b2c/N7Moa15HkOcnN82ZmUyXNkvR4xNVnmNl6M/uemb21roUFRvp/y83zKOmDku6OuS7r51GSjnL37VIQHiW9IeKcvDyfeflbjtJIr4mowPisC8gzM/uBpEkRV31K0pck3azgj/VmSV9Q8Ec97C4iblvVubHlanT3b4fnfErSPkl3xdzNme7+rJm9QdL3zewX4SeLWkjynNT8eUvCzA6V9C+SrnX3V0quXqOgCf53Yd/uckkn1LnEkf7f8vI8HiTpQknXR1ydh+cxqcyfz5z9LZfKxWsi6oPwUIa7vzvJeWZ2h6TvRFz1jKQpRZePkfRsFUrbb6QazewKSe+TdI67R/6Ruvuz4fcXzOxbCpoWa/WCk+Q5qfnzNhIza1UQHO5y9/tLry8OE+7+XTP7RzN7vbvXbZOiBP9vmT+PofdKWuPuz5dekYfnMfS8mU129+1h184LEedk+nzm8G+59LH3//9m+ZqI+qDbYpRK+o4vkrQx4rSfSTrBzI4LP319UNID9ahPCkY1S/qEpAvd/dWYcyaa2WGFnxUMzIr6t1RLkufkAUl/Gs4WOF3SrkKTcj2YmUn6iqQn3P3vYs6ZFJ4nMztVwd/Sb+pYY5L/t0yfxyKXKabLIuvnscgDkq4If75C0rcjzsns7zmnf8ulj5/710RUUdYjNhv1S9LXJG2Q1Kvgl39yePxoSd8tOu98BaP1f6WgK6GeNf5SQf/iuvDrn0prVDDqeX34takeNUY9J5I+Julj4c8m6R/C6zdI6qrz8/Z2BU2pvUXP3fklNV4VPl/rFQxge1uda4z8f8vT8xjWcIiCMNBedCzT51FBkNkuqV/BJ+GPSPo9SaskPRl+PyI8t+5/zzH15epvOabG3L8m8lW9L5anBgAAqdBtAQAAUiE8AACAVAgPAAAgFcIDAABIhfAAAABSITwAAIBUCA9AnZjZQLhV8UYzu9fMDgmPTzKzb5rZr8zsP8zsu2b25jL3c4UFW0c/Ga46CAB1xToPQJ2Y2e/c/dDw57sk/VzSYkmPSVrq7v8UXjdT0mHu/pOI+zhCUo+kLgULWf1c0inu/nJ9/hUAQMsDkJWfSPp9SWdL6i8EB0ly93VRwSE0W9L33f2lMDB8X9J5Na8WAIoQHoA6M7PxCjaL2iDpRAWtB0mxpTGAzBEegPppM7N1Crodnlaw+VZabGkMIHNsyQ3UT5+7zyw+YGabJF2S4j6ekXRW0eVjJP2o4soAIAVaHoBs/VDSwWb20cIBM/vvZvaHMeevlHSumR1uZocr2HZ5ZR3qBID9CA9AhjyY7nSRpPeEUzU3SeqW9GzM+S9JulnSz8Kvm8JjAFA3TNUEAACp0PIAAABSYcAkkENmNl3S10oOv+bup2VRDwAUo9sCAACkQrcFAABIhfAAAABSITwAAIBUCA8AACCV/w/9f7lj87HauQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 542.125x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "\n",
    "#Reduce features to 2 PCA components\n",
    "#First, get just the feature values\n",
    "X_labeled = df_all.drop(['label','attack_type'],axis=1)\n",
    "#Scale all the features so they have mean of 0 and standard deviation of 1\n",
    "X_labeled_scaled = preprocessing.StandardScaler().fit_transform(X_labeled)\n",
    "#Convert this to a dataframe to facilitate plotting\n",
    "X_labeled_scaled = pd.DataFrame(X_labeled_scaled,columns=X_labeled.columns)\n",
    "#Now apply PCA\n",
    "n_components = 2\n",
    "pca = PCA(n_components=n_components)\n",
    "X_labeled_reduced = pca.fit_transform(X_labeled_scaled)\n",
    "X_labeled_reduced = pd.DataFrame(X_labeled_reduced)\n",
    "X_labeled_reduced = X_labeled_reduced.add_prefix('PC_')\n",
    "print('Using {} components accounts for {}% of variance.'.format(n_components,\n",
    "                                                                 100*sum(pca.explained_variance_ratio_)))\n",
    "print('Explained Variance Ratios of first two principle components:',pca.explained_variance_ratio_)\n",
    "\n",
    "#add labels back in\n",
    "df_labeled_reduced = X_labeled_reduced.copy()\n",
    "df_labeled_reduced['label'] = df_all['label'].values\n",
    "df_labeled_reduced['attack_type'] = df_all['attack_type'].values\n",
    "\n",
    "#reduce to managable sample\n",
    "df_labeled_sample = df_labeled_reduced.sample(frac=.02,random_state=0)\n",
    "\n",
    "#Now plot\n",
    "sns.FacetGrid(df_labeled_sample,hue=\"attack_type\", size=6).map(plt.scatter, \"PC_0\", \"PC_1\").add_legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-Test Splits\n",
    "\n",
    "Scikit has built in tools to split data into training and test sets for testing and validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Split raw data into features (X) and target labels (y)\n",
    "X = df_all.drop(['label','attack_type'],axis=1)\n",
    "y = df_all['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29646 samples in training set (75%), 9882 samples in test set (25%)\n",
      "\n",
      "Fraction of each class in the test set:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "attack    5017\n",
       "normal    4865\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Here is the process to create a train-test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0) \n",
    "#Don't need to include random_state, but useful to ensure we get the same results every time we run it\n",
    "print('{} samples in training set ({:.0%}), {} samples in test set ({:.0%})'.format(len(y_train),len(y_train)/len(y),\n",
    "                                                                                    len(y_test),len(y_test)/len(y)))\n",
    "\n",
    "#By default, train_test_split will shuffle the data to ensure train and test sets each has same portion of each class\n",
    "print('\\nFraction of each class in the test set:')\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, the default is to split 25% into the test set.  We can adjust this fraction , e.g. `train_test_split(X,y,test_size=0.1)` to get 10% into test set.\n",
    "\n",
    "For high dimension data like this, it also can be very helpful to reduce the data to a smaller number of features as a pre-processing step before running machine learning models.  As we'll see, this can greatly enhance the performance of many ML techniques, and also reduce the computational resources.  Lets also create a set of training and test data for reduced features.\n",
    "\n",
    "It turns out, when we want a training set of reduced data, we FIRST have to split the full data into training and test sets, and THEN reduce the data using PCA or some other feature reduction technique.  If we scaled and reduced the data and then did a train-test split, then we will have **data leakage** of test data into training data.  A big no-no. So lets apply PCA feature reduction to the training-test split we've already created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#As prep for PCA, scale the data so all features have mean 0 and variance of 1\n",
    "#NOTE: The scaling transform should be LEARNED from the training data, then applied as is to the test data.  \n",
    "#We don't re-fit the scaler on the test data.  \n",
    "scaler = preprocessing.StandardScaler().fit(X_train)\n",
    "X_train_scaled = scaler.transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: Using 5 PCA components accounts for 69.3% of variance.\n"
     ]
    }
   ],
   "source": [
    "#Now pick number of principle components to reduce down to.  5 is a good number.\n",
    "n_components = 5\n",
    "pca = PCA(n_components=n_components)\n",
    "pca.fit(X_train_scaled)\n",
    "X_train_reduced = pca.transform(X_train_scaled)\n",
    "X_train_reduced = pd.DataFrame(X_train_reduced)\n",
    "X_train_reduced = X_train_reduced.add_prefix('PC_')\n",
    "\n",
    "X_test_reduced = pca.transform(X_test_scaled)\n",
    "X_test_reduced = pd.DataFrame(X_test_reduced)\n",
    "X_test_reduced = X_test_reduced.add_prefix('PC_')\n",
    "\n",
    "print('Note: Using {} PCA components accounts for {:.1%} of variance.'.format(n_components,\n",
    "                                                                 sum(pca.explained_variance_ratio_)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a training and test data, lets run various classification algorithms and see how they do.  For the first example (Multi-Layer Perceptrons), we'll step through the modeling in detail including basic model training, evaluation criteria, and parameter tuning with grid search validation.  Then we'll quickly run through several other classification algorithms and see which one does best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Multi-Layer Perceptron Neural Nets\n",
    "\n",
    "Lets start with a type of neural network classifier, multi-layer perceptrons (go big or go home).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of trained MLP model on Test Set = 78.9%\n"
     ]
    }
   ],
   "source": [
    "#The basics of training an MLP model\n",
    "\n",
    "#Load the package\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "#Initialize a MLP Classifier object using default parameter values, call it cls\n",
    "cls = MLPClassifier()\n",
    "\n",
    "#Now fit the model on the training data\n",
    "cls.fit(X_train,y_train)\n",
    "\n",
    "#Now cls is a trained/fitted MLPClassifier object\n",
    "#We can use to to make predictions off of the test data (X_test) and see how well it matches \n",
    "y_predict = cls.predict(X_test)\n",
    "\n",
    "#Lets see how often the predictions match the actual labels\n",
    "acc = sum(pd.Series(y_predict).reset_index(drop=True) == pd.Series(y_test).reset_index(drop=True)) / len(y_test)\n",
    "print('Accuracy of trained MLP model on Test Set = {:.1%}'.format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "So far so good.  Scikit also has several built in tools to help evaluate the performance of classifiers.  In addition to accuracy, there are many other measure we can use, including:\n",
    "\n",
    "- Accuracy: What percentage of predictions match the test set\n",
    "- ROC AUC: Area under the ROC curve, useful way to think about a binary classifier\n",
    "- precision: Pr(Attack|\"Attack\"), where \"Attack\" means record is classified as an attack, and Attack means the record actually is an attack\n",
    "- recall: Pr(\"Attack\"|Attack), aka True Positive Rate (if attack is defined as positive)\n",
    "- F1 score: harmonic mean of precision and recall, $F1 = 2\\frac{precision \\bullet recall}{precision + recall}$\n",
    "- beta-f1 score: weighted harmonic mean of precision and recall.  This turns out to be a very useful metric in classification settings because the beta parameter roughly means that recall is $\\beta$ times as important as precision.  This can be applicable to network security applications where a low recall (meaning we fail to catch actual attacks, aka false negative) can be much worse than a low precision (meaning we mis-classify normal traffic as an attack, aka false positive).  See [F-Score description](https://en.wikipedia.org/wiki/F1_score).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#Load the various classifier scoring functions\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import fbeta_score\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.788606</td>\n",
       "      <td>0.79111</td>\n",
       "      <td>0.933649</td>\n",
       "      <td>0.628264</td>\n",
       "      <td>0.751102</td>\n",
       "      <td>0.636268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accuracy  ROC AUC  precision    recall        F1        F5\n",
       "0  0.788606  0.79111   0.933649  0.628264  0.751102  0.636268"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Turns out for some of the scoring metrics, the outputs must be binary (0 or 1)\n",
    "y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "\n",
    "#Now compute the various evaluation metrics\n",
    "s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "s5 = f1_score(y_test,y_predict,pos_label = 'attack')\n",
    "s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "score_names = ['Accuracy','ROC AUC','precision','recall','F1','F5']\n",
    "df_scores = pd.DataFrame(columns=score_names)\n",
    "df_scores.loc[0] = [s1,s2,s3,s4,s5,s6]\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4641  224]\n",
      " [1865 3152]]\n",
      "\n",
      "Out of 5017 attacks in test set, classifer failed to detect 1865\n"
     ]
    }
   ],
   "source": [
    "#We can also look at the confustion matrix, which shows following raw numbers:\n",
    "#  [[TP  FP]\n",
    "#    FN  TN]]\n",
    "cm = confusion_matrix(y_test,y_predict,labels=['normal','attack'])\n",
    "print(cm)\n",
    "print('\\nOut of {} attacks in test set, classifer failed to detect {}'.format(cm[1,1]+cm[1,0],cm[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "      attack    0.93365   0.62826   0.75110      5017\n",
      "      normal    0.71334   0.95396   0.81629      4865\n",
      "\n",
      "    accuracy                        0.78861      9882\n",
      "   macro avg    0.82350   0.79111   0.78369      9882\n",
      "weighted avg    0.82519   0.78861   0.78319      9882\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#The classification report is also useful\n",
    "print(classification_report(y_test,y_predict,digits=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "So far we have been using the default parameters of the MLP classifier.  We can also tune the classifer parameters to try and achieve better performance.  The main parameter for MLP classifiers is the hidden layer sizes.  Lets define some possible values for this parameter then see which one does best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1 of 4\n",
      "Training model # 2 of 4\n",
      "Training model # 3 of 4\n",
      "Training model # 4 of 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden Layer Sizes</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(10, 10)</td>\n",
       "      <td>0.993119</td>\n",
       "      <td>0.993011</td>\n",
       "      <td>0.986627</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.993269</td>\n",
       "      <td>0.999479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>0.883323</td>\n",
       "      <td>0.884596</td>\n",
       "      <td>0.961980</td>\n",
       "      <td>0.801874</td>\n",
       "      <td>0.874660</td>\n",
       "      <td>0.807040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(20, 20)</td>\n",
       "      <td>0.851953</td>\n",
       "      <td>0.854046</td>\n",
       "      <td>0.986849</td>\n",
       "      <td>0.717959</td>\n",
       "      <td>0.831199</td>\n",
       "      <td>0.725563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>0.744384</td>\n",
       "      <td>0.747723</td>\n",
       "      <td>0.939640</td>\n",
       "      <td>0.530596</td>\n",
       "      <td>0.678217</td>\n",
       "      <td>0.539631</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Hidden Layer Sizes  Accuracy   ROC AUC  precision    recall        F1  \\\n",
       "1           (10, 10)  0.993119  0.993011   0.986627  1.000000  0.993269   \n",
       "2                 20  0.883323  0.884596   0.961980  0.801874  0.874660   \n",
       "3           (20, 20)  0.851953  0.854046   0.986849  0.717959  0.831199   \n",
       "0                 10  0.744384  0.747723   0.939640  0.530596  0.678217   \n",
       "\n",
       "         F5  \n",
       "1  0.999479  \n",
       "2  0.807040  \n",
       "3  0.725563  \n",
       "0  0.539631  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#This may take a couple minutes, get up and stretch\n",
    "\n",
    "hidden_layer_sizes_vals = [(10),(10,10),(20),(20,20)]\n",
    "#Length of tuple is number of layers; each value is number of neurons in that layer\n",
    "#So (10,10) means two layers with 10 neurons each\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['Hidden Layer Sizes','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "for i,h in enumerate(hidden_layer_sizes_vals):\n",
    "    print('Training model # {} of {}'.format(i+1,len(hidden_layer_sizes_vals)))\n",
    "    nn = MLPClassifier(hidden_layer_sizes=h)\n",
    "    nn.fit(X_train,y_train)\n",
    "    y_predict = nn.predict(X_test)\n",
    "    \n",
    "    #find binary output vectors, which are needed for roc_auc_score\n",
    "    y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "    y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "    \n",
    "    #find scores\n",
    "    s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "    s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "    s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "    s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "    #Record Results\n",
    "    df_scores.loc[i] = [h,s1,s2,s3,s4,s5,s6]\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "So far we trained on the full data set with all 115 features.  We can also try to train the models on the reduced data which contains only the 5 principle components.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1 of 4\n",
      "Training model # 2 of 4\n",
      "Training model # 3 of 4\n",
      "Training model # 4 of 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden Layer Sizes</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(20, 20)</td>\n",
       "      <td>0.998179</td>\n",
       "      <td>0.998166</td>\n",
       "      <td>0.997413</td>\n",
       "      <td>0.999003</td>\n",
       "      <td>0.998208</td>\n",
       "      <td>0.998942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>0.998179</td>\n",
       "      <td>0.998172</td>\n",
       "      <td>0.997809</td>\n",
       "      <td>0.998605</td>\n",
       "      <td>0.998207</td>\n",
       "      <td>0.998574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>0.997875</td>\n",
       "      <td>0.997867</td>\n",
       "      <td>0.997411</td>\n",
       "      <td>0.998405</td>\n",
       "      <td>0.997908</td>\n",
       "      <td>0.998367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(10, 10)</td>\n",
       "      <td>0.997673</td>\n",
       "      <td>0.997667</td>\n",
       "      <td>0.997410</td>\n",
       "      <td>0.998007</td>\n",
       "      <td>0.997708</td>\n",
       "      <td>0.997984</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Hidden Layer Sizes  Accuracy   ROC AUC  precision    recall        F1  \\\n",
       "3           (20, 20)  0.998179  0.998166   0.997413  0.999003  0.998208   \n",
       "0                 10  0.998179  0.998172   0.997809  0.998605  0.998207   \n",
       "2                 20  0.997875  0.997867   0.997411  0.998405  0.997908   \n",
       "1           (10, 10)  0.997673  0.997667   0.997410  0.998007  0.997708   \n",
       "\n",
       "         F5  \n",
       "3  0.998942  \n",
       "0  0.998574  \n",
       "2  0.998367  \n",
       "1  0.997984  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Train on PCA-REDUCED data\n",
    "\n",
    "hidden_layer_sizes_vals = [(10),(10,10),(20),(20,20)]\n",
    "#Length of tuple is number of layers; each value is number of neurons in that layer\n",
    "#So (10,10) means two layers with 10 neurons each\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['Hidden Layer Sizes','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "for i,h in enumerate(hidden_layer_sizes_vals):\n",
    "    print('Training model # {} of {}'.format(i+1,len(hidden_layer_sizes_vals)))\n",
    "    nn = MLPClassifier(hidden_layer_sizes=h)\n",
    "    nn.fit(X_train_reduced,y_train)\n",
    "    y_predict = nn.predict(X_test_reduced)\n",
    "    \n",
    "    #find binary output vectors, which are needed for roc_auc_score\n",
    "    y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "    y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "    \n",
    "    #find scores\n",
    "    s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "    s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "    s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "    s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "    #Record Results\n",
    "    df_scores.loc[i] = [h,s1,s2,s3,s4,s5,s6]\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Turns out that the model performed better when trained on reduced data.  It always strikes me as odd and counterintuitive that training on LESS INFORMATION (recall the first 5 components only accounted for 68% of the variance in the features) led to BETTER results.  By allowing the model to train on the most relevant information, turns out we can actually improved performance.  Moving forward we will train all subsequent models on the PCA-reduced data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MLP': [0.9981785063752276,\n",
       "  0.9981656202416408,\n",
       "  0.997412935323383,\n",
       "  0.9990033884791708,\n",
       "  0.9982075283808006,\n",
       "  0.9989421234189344]}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Before moving on, grab the best performance (the one that does best on F5 score, F score with beta = 5)\n",
    "best_scores = df_scores[score_names].iloc[0].tolist()\n",
    "best_scores_dict = {'MLP':best_scores}\n",
    "best_scores_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Logistic Regression\n",
    "\n",
    "Parameter Tuning:\n",
    "- C = regularization parameter.  Small C (.1) for more shrinkage/regularization; large C (100) for less.  More regularization would probably be useful when dealing with all 115 features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1 of 5\n",
      "Training model # 2 of 5\n",
      "Training model # 3 of 5\n",
      "Training model # 4 of 5\n",
      "Training model # 5 of 5\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100.00</td>\n",
       "      <td>0.994637</td>\n",
       "      <td>0.994621</td>\n",
       "      <td>0.993832</td>\n",
       "      <td>0.995615</td>\n",
       "      <td>0.994723</td>\n",
       "      <td>0.995546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.00</td>\n",
       "      <td>0.994333</td>\n",
       "      <td>0.994319</td>\n",
       "      <td>0.993632</td>\n",
       "      <td>0.995216</td>\n",
       "      <td>0.994423</td>\n",
       "      <td>0.995155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.993422</td>\n",
       "      <td>0.993404</td>\n",
       "      <td>0.992442</td>\n",
       "      <td>0.994618</td>\n",
       "      <td>0.993529</td>\n",
       "      <td>0.994534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.10</td>\n",
       "      <td>0.991500</td>\n",
       "      <td>0.991491</td>\n",
       "      <td>0.991237</td>\n",
       "      <td>0.992027</td>\n",
       "      <td>0.991632</td>\n",
       "      <td>0.991997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.988767</td>\n",
       "      <td>0.988782</td>\n",
       "      <td>0.990012</td>\n",
       "      <td>0.987841</td>\n",
       "      <td>0.988925</td>\n",
       "      <td>0.987925</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        C  Accuracy   ROC AUC  precision    recall        F1        F5\n",
       "4  100.00  0.994637  0.994621   0.993832  0.995615  0.994723  0.995546\n",
       "3   10.00  0.994333  0.994319   0.993632  0.995216  0.994423  0.995155\n",
       "2    1.00  0.993422  0.993404   0.992442  0.994618  0.993529  0.994534\n",
       "1    0.10  0.991500  0.991491   0.991237  0.992027  0.991632  0.991997\n",
       "0    0.01  0.988767  0.988782   0.990012  0.987841  0.988925  0.987925"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Parameter values to evalutate\n",
    "C_vals = [.01,.1,1,10,100]\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['C','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "for i,c in enumerate(C_vals):\n",
    "    print('Training model # {} of {}'.format(i+1,len(C_vals)))\n",
    "    cls = LogisticRegression(C=c)\n",
    "    cls.fit(X_train_reduced,y_train)\n",
    "    y_predict = cls.predict(X_test_reduced)\n",
    "    \n",
    "    #find binary output vectors, which are needed for roc_auc_score\n",
    "    y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "    y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "    \n",
    "    #find scores\n",
    "    s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "    s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "    s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "    s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "    #Record Results\n",
    "    df_scores.loc[i] = [c,s1,s2,s3,s4,s5,s6]\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#Record results\n",
    "best_scores = df_scores[score_names].iloc[0].tolist()\n",
    "best_scores_dict['Logistic Regression'] = best_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest\n",
    "\n",
    "Parameter tuning:\n",
    "- max_depth: how many splits to conduct per tree\n",
    "- criterion = ['gini','entropy], criteria to evaluate quality of a split at distinguishing between classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1\n",
      "Training model # 2\n",
      "Training model # 3\n",
      "Training model # 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>criterion</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>entropy</td>\n",
       "      <td>5</td>\n",
       "      <td>0.998482</td>\n",
       "      <td>0.998480</td>\n",
       "      <td>0.998406</td>\n",
       "      <td>0.998605</td>\n",
       "      <td>0.998505</td>\n",
       "      <td>0.998597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gini</td>\n",
       "      <td>5</td>\n",
       "      <td>0.998583</td>\n",
       "      <td>0.998586</td>\n",
       "      <td>0.998804</td>\n",
       "      <td>0.998405</td>\n",
       "      <td>0.998604</td>\n",
       "      <td>0.998421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gini</td>\n",
       "      <td>3</td>\n",
       "      <td>0.996762</td>\n",
       "      <td>0.996745</td>\n",
       "      <td>0.995823</td>\n",
       "      <td>0.997807</td>\n",
       "      <td>0.996814</td>\n",
       "      <td>0.997731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>entropy</td>\n",
       "      <td>3</td>\n",
       "      <td>0.997268</td>\n",
       "      <td>0.997262</td>\n",
       "      <td>0.997012</td>\n",
       "      <td>0.997608</td>\n",
       "      <td>0.997310</td>\n",
       "      <td>0.997585</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  criterion max_depth  Accuracy   ROC AUC  precision    recall        F1  \\\n",
       "3   entropy         5  0.998482  0.998480   0.998406  0.998605  0.998505   \n",
       "1      gini         5  0.998583  0.998586   0.998804  0.998405  0.998604   \n",
       "0      gini         3  0.996762  0.996745   0.995823  0.997807  0.996814   \n",
       "2   entropy         3  0.997268  0.997262   0.997012  0.997608  0.997310   \n",
       "\n",
       "         F5  \n",
       "3  0.998597  \n",
       "1  0.998421  \n",
       "0  0.997731  \n",
       "2  0.997585  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Parameter values to evalutate\n",
    "criterion_vals = ['gini','entropy']\n",
    "max_depth_vals = [3,5]\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['criterion','max_depth','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "i = 0\n",
    "for c in criterion_vals:\n",
    "    for d in max_depth_vals:\n",
    "        print('Training model # {}'.format(i+1))\n",
    "        cls = RandomForestClassifier(criterion=c,max_depth=d)\n",
    "        cls.fit(X_train_reduced,y_train)\n",
    "        y_predict = cls.predict(X_test_reduced)\n",
    "\n",
    "        #find binary output vectors, which are needed for roc_auc_score\n",
    "        y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "        y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "\n",
    "        #find scores\n",
    "        s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "        s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "        s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "        s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "        s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "        s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "        #Record Results\n",
    "        df_scores.loc[i] = [c,d,s1,s2,s3,s4,s5,s6]\n",
    "        i += 1\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Record results\n",
    "best_scores = df_scores[score_names].iloc[0].tolist()\n",
    "best_scores_dict['Random Forest'] = best_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PC_1    0.399132\n",
       "PC_0    0.278114\n",
       "PC_3    0.133638\n",
       "PC_2    0.107996\n",
       "PC_4    0.081121\n",
       "dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Another good thing about random forests is that we get the feature importance\n",
    "features = X_train_reduced.columns\n",
    "feature_importances = cls.feature_importances_\n",
    "feature_ser = pd.Series(feature_importances,index=features)\n",
    "feature_ser.sort_values(ascending=False).iloc[:15]\n",
    "\n",
    "#Here it's just showing that the 1st principle compoment is most important, as expected\n",
    "#If we trained on real feature names this would be more useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1\n",
      "Training model # 2\n",
      "Training model # 3\n",
      "Training model # 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>criterion</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gini</td>\n",
       "      <td>5</td>\n",
       "      <td>0.997976</td>\n",
       "      <td>0.997963</td>\n",
       "      <td>0.997214</td>\n",
       "      <td>0.998804</td>\n",
       "      <td>0.998008</td>\n",
       "      <td>0.998743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gini</td>\n",
       "      <td>3</td>\n",
       "      <td>0.996458</td>\n",
       "      <td>0.996443</td>\n",
       "      <td>0.995623</td>\n",
       "      <td>0.997409</td>\n",
       "      <td>0.996515</td>\n",
       "      <td>0.997340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>entropy</td>\n",
       "      <td>3</td>\n",
       "      <td>0.996863</td>\n",
       "      <td>0.996864</td>\n",
       "      <td>0.997010</td>\n",
       "      <td>0.996811</td>\n",
       "      <td>0.996910</td>\n",
       "      <td>0.996818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>entropy</td>\n",
       "      <td>5</td>\n",
       "      <td>0.998179</td>\n",
       "      <td>0.998203</td>\n",
       "      <td>0.999800</td>\n",
       "      <td>0.996612</td>\n",
       "      <td>0.998203</td>\n",
       "      <td>0.996734</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  criterion max_depth  Accuracy   ROC AUC  precision    recall        F1  \\\n",
       "1      gini         5  0.997976  0.997963   0.997214  0.998804  0.998008   \n",
       "0      gini         3  0.996458  0.996443   0.995623  0.997409  0.996515   \n",
       "2   entropy         3  0.996863  0.996864   0.997010  0.996811  0.996910   \n",
       "3   entropy         5  0.998179  0.998203   0.999800  0.996612  0.998203   \n",
       "\n",
       "         F5  \n",
       "1  0.998743  \n",
       "0  0.997340  \n",
       "2  0.996818  \n",
       "3  0.996734  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Parameter values to evalutate\n",
    "criterion_vals = ['gini','entropy']\n",
    "max_depth_vals = [3,5]\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['criterion','max_depth','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "i = 0\n",
    "for c in criterion_vals:\n",
    "    for d in max_depth_vals:\n",
    "        print('Training model # {}'.format(i+1))\n",
    "        cls = DecisionTreeClassifier(criterion=c,max_depth=d)\n",
    "        cls.fit(X_train_reduced,y_train)\n",
    "        y_predict = cls.predict(X_test_reduced)\n",
    "\n",
    "        #find binary output vectors, which are needed for roc_auc_score\n",
    "        y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "        y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "\n",
    "        #find scores\n",
    "        s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "        s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "        s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "        s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "        s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "        s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "        #Record Results\n",
    "        df_scores.loc[i] = [c,d,s1,s2,s3,s4,s5,s6]\n",
    "        i += 1\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Record results\n",
    "best_scores = df_scores[score_names].iloc[0].tolist()\n",
    "best_scores_dict['Decision Tree'] = best_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### K Nearest Neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model # 1\n",
      "Training model # 2\n",
      "Training model # 3\n",
      "Training model # 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>k</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.999899</td>\n",
       "      <td>0.999900</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999801</td>\n",
       "      <td>0.999900</td>\n",
       "      <td>0.999808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.999089</td>\n",
       "      <td>0.999084</td>\n",
       "      <td>0.998805</td>\n",
       "      <td>0.999402</td>\n",
       "      <td>0.999103</td>\n",
       "      <td>0.999379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>0.998583</td>\n",
       "      <td>0.998574</td>\n",
       "      <td>0.998009</td>\n",
       "      <td>0.999203</td>\n",
       "      <td>0.998606</td>\n",
       "      <td>0.999157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15.0</td>\n",
       "      <td>0.998381</td>\n",
       "      <td>0.998371</td>\n",
       "      <td>0.997810</td>\n",
       "      <td>0.999003</td>\n",
       "      <td>0.998406</td>\n",
       "      <td>0.998957</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      k  Accuracy   ROC AUC  precision    recall        F1        F5\n",
       "0   3.0  0.999899  0.999900   1.000000  0.999801  0.999900  0.999808\n",
       "1   5.0  0.999089  0.999084   0.998805  0.999402  0.999103  0.999379\n",
       "2  10.0  0.998583  0.998574   0.998009  0.999203  0.998606  0.999157\n",
       "3  15.0  0.998381  0.998371   0.997810  0.999003  0.998406  0.998957"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Parameter values to evalutate: number of neighbors used to assess most common class among neighbors\n",
    "k_vals = [3,5,10,15]\n",
    "\n",
    "#Initiate dataframe to hold all scores\n",
    "df_scores = pd.DataFrame(columns=['k','Accuracy','ROC AUC','precision','recall','F1','F5'])\n",
    "\n",
    "#Now see how they all do!\n",
    "i = 0\n",
    "for k in k_vals:\n",
    "    print('Training model # {}'.format(i+1))\n",
    "    cls = KNeighborsClassifier(n_neighbors=k)\n",
    "    cls.fit(X_train_reduced,y_train)\n",
    "    y_predict = cls.predict(X_test_reduced)\n",
    "\n",
    "    #find binary output vectors, which are needed for roc_auc_score\n",
    "    y_test_bin = (y_test == 'attack').replace({True:1,False:0})\n",
    "    y_predict_bin = (pd.Series(y_predict) == 'attack').replace({True:1,False:0})\n",
    "\n",
    "    #find scores\n",
    "    s1 = accuracy_score(y_test_bin,y_predict_bin)\n",
    "    s2 = roc_auc_score(y_test_bin,y_predict_bin)\n",
    "    s3 = precision_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s4 = recall_score(y_test,y_predict,pos_label = 'attack')\n",
    "    s5 = fbeta_score(y_test,y_predict,beta=1,pos_label = 'attack')\n",
    "    s6 = fbeta_score(y_test,y_predict,beta=5,pos_label = 'attack')\n",
    "\n",
    "    #Record Results\n",
    "    df_scores.loc[i] = [k,s1,s2,s3,s4,s5,s6]\n",
    "    i += 1\n",
    "\n",
    "df_scores.sort_values('F5',ascending=False,inplace=True)\n",
    "df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#Record results\n",
    "best_scores = df_scores[score_names].iloc[0].tolist()\n",
    "best_scores_dict['KNN'] = best_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### Check out classifier results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['MLP', 'Logistic Regression', 'Random Forest', 'Decision Tree', 'KNN'])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_scores_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>F5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>KNN</th>\n",
       "      <td>0.999899</td>\n",
       "      <td>0.999900</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999801</td>\n",
       "      <td>0.999900</td>\n",
       "      <td>0.999808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random Forest</th>\n",
       "      <td>0.998482</td>\n",
       "      <td>0.998480</td>\n",
       "      <td>0.998406</td>\n",
       "      <td>0.998605</td>\n",
       "      <td>0.998505</td>\n",
       "      <td>0.998597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MLP</th>\n",
       "      <td>0.998179</td>\n",
       "      <td>0.998166</td>\n",
       "      <td>0.997413</td>\n",
       "      <td>0.999003</td>\n",
       "      <td>0.998208</td>\n",
       "      <td>0.998942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Decision Tree</th>\n",
       "      <td>0.997976</td>\n",
       "      <td>0.997963</td>\n",
       "      <td>0.997214</td>\n",
       "      <td>0.998804</td>\n",
       "      <td>0.998008</td>\n",
       "      <td>0.998743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Logistic Regression</th>\n",
       "      <td>0.994637</td>\n",
       "      <td>0.994621</td>\n",
       "      <td>0.993832</td>\n",
       "      <td>0.995615</td>\n",
       "      <td>0.994723</td>\n",
       "      <td>0.995546</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Accuracy   ROC AUC  precision    recall        F1  \\\n",
       "KNN                  0.999899  0.999900   1.000000  0.999801  0.999900   \n",
       "Random Forest        0.998482  0.998480   0.998406  0.998605  0.998505   \n",
       "MLP                  0.998179  0.998166   0.997413  0.999003  0.998208   \n",
       "Decision Tree        0.997976  0.997963   0.997214  0.998804  0.998008   \n",
       "Logistic Regression  0.994637  0.994621   0.993832  0.995615  0.994723   \n",
       "\n",
       "                           F5  \n",
       "KNN                  0.999808  \n",
       "Random Forest        0.998597  \n",
       "MLP                  0.998942  \n",
       "Decision Tree        0.998743  \n",
       "Logistic Regression  0.995546  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all_scores = pd.DataFrame(data=best_scores_dict,index=score_names).transpose()\n",
    "df_all_scores.sort_values(by='Accuracy',ascending=False,inplace=True)\n",
    "df_all_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Turns out KNN performed the best (in terms of F5 score, or F-beta score with beta = 5), though all classifiers did very well.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Validation with GridSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, we manually searched through different parameter values.  Scitkit has many tools that automate this task for us.  One very useful one is GridSearchCV which can tune the parameters of any classifier using cross validation.\n",
    "\n",
    "Here's how it would have worked for the Random Forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since GridSearchCV performs its own cross validation, we can feed it the entire data set instead of just the training set, since each fold of the data acts as its own training-test split.  However one caution with GridSearch is that it doesn't automatically shuffle the data, so you'll want to shuffle it before feeding it in to make sure the training sets always contain a mix of all classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "print(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Mix the data up\n",
    "df_all = df_all.sample(frac=1)\n",
    "#Now grab feature data and target vector\n",
    "X = df_all.drop(['label','attack_type'],axis=1)\n",
    "y = df_all.label\n",
    "\n",
    "#Optionally scale and reduce the data back to 5 PCA dimensions\n",
    "# X_scaled = preprocessing.StandardScaler().fit_transform(X)\n",
    "# #Now apply pca\n",
    "# n_components = 5\n",
    "# pca = PCA(n_components=n_components)\n",
    "# X_reduced = pca.fit_transform(X_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'criterion': 'entropy', 'max_depth': 5}\n",
      "Best score: 0.9999241044323012\n"
     ]
    }
   ],
   "source": [
    "#Initialize classifier object\n",
    "cls = RandomForestClassifier(n_estimators=10)\n",
    "\n",
    "#Define parameters to cycle through\n",
    "criterion_vals = ['gini','entropy']\n",
    "max_depth_vals = [3,5]\n",
    "\n",
    "#Set up GridSearchCV object\n",
    "num_folds = 4 #number of cross validation folds\n",
    "grid_values = {'criterion':criterion_vals,'max_depth':max_depth_vals}\n",
    "grid_clf = GridSearchCV(cls,param_grid=grid_values,cv=num_folds)\n",
    "\n",
    "#Now fit it based on the full data set\n",
    "#GridSearch will automatically split the data set into folds\n",
    "grid_clf.fit(X,y)\n",
    "print('Best parameters:',grid_clf.best_params_)\n",
    "print('Best score:',grid_clf.best_score_)\n",
    "\n",
    "#Takes a minute or two to run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was an easy way to evaluate many parameters and find the best ones.  By default GridSearchCV uses accuracy to determine the best one, but we could also adjust that using scoring argument, which can take following options: ['f1-weighted','accuracy' (default),'roc_auc','recall','recall_micro'].  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(criterion='entropy', max_depth=5, n_estimators=10)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Note that we can also retrieve the best classifier object as follows:\n",
    "best_cls = grid_clf.best_estimator_\n",
    "best_cls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised Learning / Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even when we don't have output values (y), and we only have inputs/features (X), simply clustering the data can be an insightful way to explore the data, identify patterns, or detect anomalies.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x20a024d3640>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxcdZnv8c/T1d1JSAygRAKBhDAsE8wIF9sAAeYiAVkicB1nQZhhcWZy8UVQ7zjewTCKOAPOy20GA4oZVGQui3MHEGQLSy4jEAgkDkogIjEhCwkSFAJZSG/P/aOrQqX6nFOnllPndJ3v+/XKK91Vp6qfpE+d5/yW5/czd0dERPKrI+0AREQkXUoEIiI5p0QgIpJzSgQiIjmnRCAiknOdaQdQj7322ssPOOCAtMMQERlRli1b9pq7T6h8fEQmggMOOIClS5emHYaIyIhiZmuCHlfXkIhIzikRiIjknBKBiEjOKRGISMv1DQymHYKUSXSw2MwOBX5U9tCBwBfd/V/KjjkBuBNYXXzodnf/cpJxiUjrbd7ex7WLVnLzU2vZuqOfsaM6OWfGZObOOojxo7vSDi/XEk0E7v4CcASAmRWAl4E7Ag591N0/kmQsIpKezdv7OPOax9j4xnZ6B4YWutyyo58bFq9m4fOvcNfc49h9jJJBWlrZNTQL+LW7B05fEpH2de2ilbskgZLeAWfjG9u5dtHKlCITaG0iOBu4JeS5Y8zs52Z2n5m9r4UxiUgL3PzU2mFJoKR3wLn5qbUtjkjKtSQRmFk3cCbwfwOe/hkwxd0PB+YDPw55jzlmttTMlm7atCm5YEWkqfoGBtna2x95zNYd/fRrADk1rWoRnAb8zN1/U/mEu7/p7luKX98LdJnZXgHHLXD3HnfvmTBhWIW0iGRUV6GDsd3Rw5FjR3XSWdAkxrS06n/+44R0C5nZRDOz4tczijH9tkVxiUgLnDNjMt0FC3yuu2CcM2NyiyOScoknAjPbDTgZuL3ssYvM7KLit38MLDeznwPfAs527Z8p0lbmzjqIffYYMywZdBeMffYYw9xZB6USl+oZhthIvOb29PS4Fp0TGVmyUkeQlTjSYGbL3L1n2ONKBCLSan0Dg3SlMCYQVM8A77RM2r2eISwRaHQmI2ppoqo5KyNdGkkAVM8QZkTuR9Auammi5rk5K9IsceoZ5s2e1uKo0qdEkJJaSu5Vni/SuFrqGfI2lTVf/9oMqaWJquasSONUzxAuf//ijKil5F7l+SLNoXqGYEoEKailiVpPeb4GkyUtWT/3slrPkDaNEVSRxDS3UhN1y47wC3x5EzXOsVt7B/jWw7/kR0+v02CytFQSExmSml46fnQXd809Lna8aU1zbTXVEQRIcoZO6cS6/M7l/PCJ8BW5zz9mClecNR2Aq+5ZwQ2LVwd2D3UVjEPeO47nN75F5bMdBpP2GMPdnzo+dDA5Lye6JKOZ8/LTmBkXdP638ww9FZTFlETBSdCJNeU9Y3huw1uhrzl/5hSuOHMoEbz5dh9nzB8eU1fBcIf+wejfYfl7hcXTLie6tFbUTUp3wbhg5tRY0zGzUuiVlTiSooKymJo9Q6d0Yt2weDVbdvTjDE39jEoCALcte3nn16Xm7AUzpzJuVCcGjBvVybSJ47Fh7YDhbl7yzmByWDw3LF7NGfMfY/P2vpr+fTIyJNV336yJDFmZGZeVOFpNiaBCs2fohJ1Y1VQOAO8+pot5s6ex/IpT+NWVp7H8ilNY9dpW+mJ8vvsGfOd75fVEz6PN2/u46p4VTL98IYdcdh/TL1/IVfes4M23o5N93KTRyD4DlT8jKzPjshJHq2mwuEwzC05KfY9RJ1aUqPnMXYWOWLEGUWVlPtRahFhPd2Gtkx7CfsZFJxyYiUKvPBecKRGUqfXErlR5ou/WXWBr70DNccSZzxwn1p3HdhidMZNHs050DUKnK07Lr5TwG6lcP2fG5NAxAoAdfQNcdc8Kzps5hXOvXxL6M3briv6stKLQq9HP/0jWfv+iBtVbcBLU915vEog7nzkq1nLnHjUFSL6yst6uCGm+Wro4GukuDJuXX9I36NyweDWnXf0oGyJ+xu9NGFfT5y6pMY+8FpwpEVSot+CknrGAroLx/km77zIAfMHMqfzkkuNizd4pxdoV8ls0YP89x/A3pxyy87FqyePAvcbWdeHWIHR21Np3X0+/eOlCXD6RoSvkvOodcN56u5++iJ+x6rWtVT93rbjRyGvBmaaPBqi1v3Tz9j56/uFB+qpM4yxXOrFKF/16ulL6BgbZ1jvA/Ide5Jan1+7SAhkXEnNpKuqG17cFDjR3FYx965gm16xphNIc0y9fGNnFMW5UJ8uvOIW+gUEO+fv7iLoMGPDilaextXcg8nNR7WdGMWDpF07iu4+sCnx/d1o2rbOdp1erjqBO1S7Qm7cPXVjX/m5b1fcaN6qz4ROrdJLetGTNsAv/n31wfz590sGM6SpUjfnPr3+SZ19+M/D5ei7ccS880hq1JOY4v7vHLz0x8kJ8+ydn0nPlQ5EJJUrl+VH5uUvrRqPdxrpSqyMws5fM7Fkze8bMhl29bci3zGylmf3CzI5MOqZaVDsJrl20klc2b6/+Ph1DTc1G0m6p++UHj68aNv6wZUc/Nz7xEmfMf4xtIWMTpab1sf+0KDQJQO3T5BqZRijJqKWLI06/eNQ4wobXt/Hd/1xVdfwpTFDfe+XnLq1pne2UBKK06l/5IXc/IigTAacBBxf/zAG+06KYmiLu9NAB9513XfX2nZc+jGG1A30Rg3uVffjV1HLh1vK+2VA+gBpWhBg0BhUnaUSd532DsODRVRy419jQ8aquDhg/urOuvnfdaCQvC9NHzwJu9KE+qifNbA8z28fdN6YdWDW1zOWvHD4ImsZXTZykE1YLUOtgdq0X7qhphO082yJtUf3ZpSLEebOnRXZxVFuIbUxXIdZ5vuKVN3GMTnP6y06D0sX+pr86ihsXr6m57z3P0zpbpRWJwIEHzMyB77r7gornJwHryr5fX3xsl0RgZnMYajEweXL6F5XSByvuXP4gtRRw9Q0Mxv45QbUAtRS21XPhnjvrIBY+/0poH3K7zrZIUy3z/6t1cVRLGnHO86BZQZ0G58yYwt+ccgjjR8dLTEF0o5GsVqTQY939SIa6gC42sz+seD6oc3LYb9vdF7h7j7v3TJgwIYk4qwqavnbgXmNDp83FmeNf3qSNmhvdVejYOc5QTeXdUa1VyHuPH13zhbuWrghpjqSWCwm6QMetWanU7/DwL38zbBC51r73vE7rbJXEWwTuvqH496tmdgcwA/hp2SHrgf3Lvt8P2JB0XLUKu/tasXEzjtFVsF3uiEon6Gtv7ahaMfnV+1+I11yO8TkMG3iL23LpAGb9/t51XbjjdkVIc7RyuZBSiy9s2nGU9a/X1gUapNZ9BKQ2iX5SzWysmb2r9DXwYWB5xWF3AecVZw8dDWzO4vhA2N1X3yAYzrSJ4wPvhM89akronVRXwTAjVhFW38Ag/TG6dibuHnw3H/eObhD4j5+tr3pcNUoCyWr1AGrpQnzhsQfW/FoHbloSvvdGXOULLz735VNYfsUpzJs9TUmgCZJuEewN3GFmpZ91s7vfb2YXAbj7dcC9wOnASmAbcGHCMdWl2qyJVa9t3VmgU34RjOo7H91VYHtv/7A7rKCB5K5CB2NHRd/VdxWMuz91fOAHo5Y7unZdWKudJDGAWq0VV7oQA5HrCwXZ2jvQ8DnVzoVeaUv0k+7uq9z98OKf97n7lcXHrysmAXzIxe7+e+7+B+6eaKVYPWuU1HL3VflBiuo7Hxj00Ity0NzoavO9L5w5NfQD4Q4nHDoBrHqrQDMwRoZmrItTz7INpf76WtWz9lZ5nFrCJDm5+LQ3ukZJo/Pkg/YS+Nyph7KtL/qDUdm0r3fArPQhumXJ2tD1XsrfSzMwRoZGB1DrvbiOH93F7Z+cWXO88x9+sebXlGgfjWS1fSJo1p1Es1YlLLUY6kku9c7MiVtDoBkYI0ujM7Uaubi+Z9woxo2qrWf51qfXVT8oRF43jGmVLBSUJaqWddmjJDFPvp650fXMzIlTQxC2SJ1kWyMztRqddVRtL4JK9Y495XnDmFZp+/+1Zt1JJDFPvtGmfZwPfdwagme+eLJmYIxwtSSBZsw6qrYXQaV6x560hEny2rpF0Ow7iWbPk2/F3Og4s0vG6UOUO82YdVR5/ka9V6NjT6osTlZbJ4Ik1yhp1jz5VhRh6UMkQZpxXpSfv7/duoM/+vbiRJYZOW/mFG55ei29A7t+lrs60LhWE7T9beBI2nouqSIsledLkGafF+8ZOyqRZUY2b+/j3OuX8HbALLsx3Z3c9FdHqUuzQW2/MU1pR66wu5RmroOT5WUVVIwjQZI8L5r1edDud82T6x3KkjzZR+IFNssJS9KT1fNCu981T1giaOsxgpKk+uFrWQY4S7L4YZf0JXle1Pu509TR1shFIijXzJO9WTUKIu0oqLV8ds/+fOrkg2tqLWtTmuTlLhE0UyuXARYZScJay9c/vprvLV7NecdM4bMfPjQwIVQmkM4Oo8OG7/IH2ZvwMVIpEdRJTVaRcFHLmrjDvz2xhkde2DSs+zQogfQFZQA0662ZdIWqk6odRYYrLfC44NFVkUtPDDqB6xlFJZAOint4oN3vmk0tggaoUEvkHaW7+Q2vb4t1fFD3aVR36yCwW2eB5798siY8NJn+NxugQi2Rd5Tu5mvZyrJyz+443a2175ws1SgRNEAbtou8I84qt5XKu0/V3ZqeRLuGzGx/4EZgIkMtuwXufnXFMScAdwKriw/d7u5fTjKuZtKG7SLxV7ktF9R9qu7WdCR91eoHPuvu04CjgYvN7LCA4x519yOKf0ZMEqg0UpJAPdt1ikSJczdfLqz7VN2t6Ui0ReDuG4GNxa/fMrMVwCTg+SR/rgw3EpfCkJEl6m7egM6C0T/gkedeK5Zml+FattaQmR0A/BSY7u5vlj1+AnAbsB7YAPytuz8X8Po5wByAyZMnf2DNmjXJB90mguZmwzt3WVldCkNGljgLPI7pKtS8gc5IaWmPBGFrDbXkf9jMxjF0sf9MeRIo+hkwxd0PB+YDPw56D3df4O497t4zYcKEZANuM9r4W1ohzuSJWi/qSgKtkXiLwMy6gLuBhe7+zRjHvwT0uPtrYcfUuvpo3mn1RkmD7uazJ5UWgZkZ8D1gRVgSMLOJxeMwsxnFmH6bZFx50oy9aUXqoSQwciRdWXws8BfAs2b2TPGxecBkAHe/Dvhj4JNm1g9sB872kbhJQkYluV2niLSHpGcNPQbRhYDufg1wTZJx5J3mZotIFN0G5oDmZotIFCWCHNBSGCISJRd7FsuuNJtDJJ9SrSOQbFESEJFyuiKIiOScEoGISM4pEYiI5JwSgYhIzikRiIjknBKBiEjOKRGIiOScEoGISM4pEYiI5JwSgYhIzikRiIjknBKBiEjOKRGIiORc4onAzE41sxfMbKWZXRrwvJnZt4rP/8LMjkw6JhEReUfSm9cXgGuB04DDgI+b2WEVh50GHFz8Mwf4TpIxiYjIrpJuEcwAVrr7KnfvBW4Fzqo45izgRh/yJLCHme2TcFwiIlKUdCKYBKwr+3598bFajxERkYQknQgs4LHKvTHjHIOZzTGzpWa2dNOmTU0JTkREkk8E64H9y77fD9hQxzG4+wJ373H3ngkTJjQ9UBGRvEo6ETwNHGxmU82sGzgbuKvimLuA84qzh44GNrv7xoTjEhGRos4k39zd+81sLrAQKADfd/fnzOyi4vPXAfcCpwMrgW3AhUnGJCIiu0o0EQC4+70MXezLH7uu7GsHLk46DhERCabKYhGRnFMiEBHJOSUCEZGcUyIQEck5JQIRkZxTIhARyTklAhGRnFMiEBHJOSUCEZGcUyIQEck5JQIRkZxTIhARyTklAhGRnFMiEBHJOSUCEZGcUyIQEck5JQIRkZxTIhARybnEtqo0s68BZwC9wK+BC939jYDjXgLeAgaAfnfvSSomEREZLskWwYPAdHd/P/Ar4PMRx37I3Y9QEhARab3EEoG7P+Du/cVvnwT2S+pniYhI/Vo1RvAJ4L6Q5xx4wMyWmdmcsDcwszlmttTMlm7atCmRIEVE8qihMQIzewiYGPDUZe5+Z/GYy4B+4KaQtznW3TeY2XuBB83sl+7+08qD3H0BsACgp6fHG4lbJI/6BgbpKmh+iAzXUCJw95Oinjez84GPALPcPfDi7e4bin+/amZ3ADOAYYlARGq3eXsf1y5ayc1PrWXrjn7GjurknBmTmTvrIMaP7ko7PMmIxG4PzOxU4O+AM919W8gxY83sXaWvgQ8Dy5OKSSRPNm/v48xrHuOGxavZsqMfB7bs6OeGxas5Y/5jbN7el3aIkhFJthOvAd7FUHfPM2Z2HYCZ7Wtm9xaP2Rt4zMx+DjwF3OPu9ycYk0huXLtoJRvf2E7vwK6N8d4BZ+Mb27l20cqUIpOsSayOwN0PCnl8A3B68etVwOFJxSCSZzc/tXZYEijpHXBufmot82ZPa3FUkkUaORJpQ30Dg2zt7Y88ZuuOfvoHBlsUkWSZEoFIG+oqdDC2O7rBP3ZUJ52aRSQoEYi0rXNmTKa7YIHPdReMc2ZMbnFEklVKBCJtau6sg9hnjzHDkkF3wdhnjzHMnRU4jCc5pEQgknF9dfbjjx/dxV1zj+OCmVMZN6oTA8aN6uSCmVP5ySXHqY5AdrKQOq9M6+np8aVLl6YdhkhikigEa6SyWFXJ7cHMlgUt7pnY9FERqU+pEKy8BqBUCLbw+Ve4a+5x7D6m9mRQ64VcVcn5oRQvkjFZKARTVXK+KBGIZEycQrCkZSEZSesoEYhkSFYKwbKQjKR1lAhEMiQLhWBZSUbSOkoEIhnzsSMnNfR8o7KQjKS19JsUaaFYNQHBxcDxn28CVSXnixKBSMI2b+/jqntWMP3yhRxy2X1Mv3whV92zgjffDp55c9uylyPfr9rzzaCq5HxRIhBJUK3TMNPsny9vragqOV9UUCaSoDjTMMv3BCj1z2/ZEZ4MGu2fL68Sjioa231MF/NmT2Pe7GmqLG5zSgQiCapnc5hzZkzmhsWrA19Xb/980AX/Yx+YxKIVr/KbN9+uWsGsJNDektyz+Etm9nJxm8pnzOz0kONONbMXzGylmV2aVDwirVLqYqm3myesfx6GksdNS9ZEjjFUCuue+rcn1rDudRWNSfItgn9296+HPWlmBeBa4GRgPfC0md3l7s8nHJdIU4V1sdTTzVPqny+9X+Xrt/YO1LTuUFj31GDEepPayjJf0m7vzQBWuvsqd+8FbgXOSjkmkZpEDQibQWfIpyyqm6fUP3/OjMl0Bby+2l17+cBvVPdUFBWN5UfSLYK5ZnYesBT4rLu/XvH8JGBd2ffrgaOC3sjM5gBzACZP1hxmyY6oAeHegeDWQNxpmDc/tZa+kGtx5V17UKvk7J792RrRIomiorH8aCgRmNlDwMSApy4DvgP8A+DFv78BfKLyLQJeG3jr4u4LgAUwtB9BnSGLNF09d9wOnHDohMhjahlj2No7ELh09Y1PvjT0KavxE6OisXxpKBG4+0lxjjOzfwXuDnhqPbB/2ff7ARsaiUmkleJcrINf59yyZC2PvLAptJ+/lqmk1y56IbRVUmshsorG8ifJWUP7lH37UWB5wGFPAweb2VQz6wbOBu5KKiaRekQtCxFnXZ4wcWbnxF3qIapVErcxoKKx/EpyjOCrZnYEQ+fhS8D/BDCzfYHr3f10d+83s7nAQqAAfN/dn0swJpFYatmdK2refzXVZufMnXUQC59/Zdjdfvlde72tknJzjj+Qz516qOoFckp7FotUCNoqEoYuvhN3H81PLjl+l66cN9/u44z5w4+Py4AXrzwtdGA2TlKafvnCyC6kav7rCyex59hRdb9eRgbtWSwSU9QsoLW/207PPz7IhTOn7rwQV5v3X03U7Jy+gcFYSz000ioZ211QEsg5tQhEKsS5uy51zQQN9P5u6w4++u3FNbUQxpXd5btT86bxYa2S7oIxuqvA9t7+wGmo3QXjgplTVTiWE2EtAnUIipSJ298eNdD77rGjdq7cGVepAG321Y/ykW89WvOm8VGrhd73mePZd8/dEl1SOtY+C5JZahGIVKilv33cqE6WX3FK4HN9A4Mc8vf3UctHrAPAgpd/CLp7D+sqqny8lsHvuJJ4T0lWWItAiUCkwlX3rIjd315toLfRQdxK40Z18vilJzZ0AW7GktJRA+phXWaSPnUNicQUtfpnpWrLMETVAdRj645+zphfe9dRuWZMEY2zz4KMHEoEIhXK+9u7OsIv4nGWYaglqcTRWTBe2fx26hfgOPssyMihRCASoDRlc9kXT2byu+sfaC0llY8fNZmuGMmgAwjLPd0FAyf1C3Ca22lKMpQIRCKMH93FTy6pb+/e8pk0j7ywqeqaP90FY9KeY5gUsmn8xN3H0Be1iQCtuQDHWVZDK5eOLCooE6milr17g2bSHLjXWDa8vi10OWmIX0cw8yuLEt3POK4kttOU9CgRiNSgWhIIWgr6Fy9vjnzPoCmoYYknKxfgOGsgycihtptIk4TNpKkmqjunMvGEDT63+gIcVcCmlUtHHtURiDRJvTUDYUVpYd1QWSzkakZtQjvI+v+DFp0TSVAjS0GXd+fEucjXMmbRKlmIIS1ZTMy1UiIQaYI4u4mF+eQJBwLhYww3LF7NwudfCazWzfMFOAvq+Z1lkc4ikSapp4q4fAloVeuOPO3yO1MiEGmSWquIuwvGuUdN2fm9qnVHnnb5nSW5Z/GPzOyZ4p+XzOyZkONeMrNni8dpBFhGrLCZNO8a3UlXxSetcpaPqnVHnnb6nSU2RuDuf1b62sy+AURNpv6Qu7+WVCwirRI0kBtnMDHOGIOqdbOlnX5niQ8Wm5kBfwqcmPTPEsmS0kBu3Fk+WSkWk/ja5XfWilR1PPAbd38x5HkHHjCzZWY2pwXxiKQmapZPVorFJL4kfmdp7PbWUEGZmT0ETAx46jJ3v7N4zHeAle7+jZD32NfdN5jZe4EHgUvc/acBx80B5gBMnjz5A2vWrKk7bpGsaoc56XnTjN9Zq37vqexQZmadwMvAB9x9fYzjvwRscfevRx2nymLJg6wUi0l89fzOWrnbW1o7lJ0E/DIsCZjZWDN7V+lr4MPA8oRjEhkRlARGhvKunHp+Z1moRUh6sPhs4JbyB8xsX+B6dz8d2Bu4Y2g8mU7gZne/P+GYREQa0syunDi1CPNmT2tG2KESTQTufkHAYxuA04tfrwIOTzIGEZFmqmVZiWpdRbXUIiQ5DVVrDYmI1KBaV843H3iBUZ2FWK2FrNQiqBNSRHKjGVMzq3Xl3PjEGm5YvJotO/px3mktnDH/MTZv7xv2mqg1qlpVi6AWgYi0tWb258fpynGIHPit7O/Pwm5vahGISNsq9efXcocepdSVU4+wReiysNubWgQi0rbiTM0sv0OPUwcQtaxENWEDv2lvNqREICJtK87UzItPPCiy66jywhzVldM36ETV6MYZ+E2jfkSJQETaUpz+/C07+jlj/mO8snnXqaA/WLyaW59ey8Cgs613YFhyuGvucYHJY8fAALcsCU4+WV6ETpvXi0jbmn75wsipmV0dhtnwwd0gYUs+lFoMfQODbO8b4Iz54ctFtKrPP4w2rxeR3Km2THTQDJ8wQeMKQTOSPnbkJDC4bdnLI2bhQCUCEWlbUf35E3cfzbrfba/p/UrjCp879VC29Q4EVhjf8tRa9tljDI9feiK7dReqVhZnYU0pdQ2JSNvqGxhkW+9A6GDwzK8siuw6CmNAZ4cx4M5gwCW0u2BcMHNq4BpBtdY1NDNZqGtIRHIh7EK7+PMnMqZr1zv0eqeCOtAXlAGKwhaLi7tOUav3pUi/TSIi0iTVCsi29Q7scnzYDmPNsGVHP7/dumOXx+LUNTS7CC4OJQIRaRu1ru1fXtU7trvQ9Hi++8gq4J01juLUNaSxP4HGCESkbVSbLjpuVCfLrzgFCO57r/b6WnV1GKO6Cju7d6q9t0HV48r/DbXSGIGItLW4a/v/493Pc+vT6wL73htZPiIwpkGnr3hRj5NgdusupLI/gbqGRKQtxFoQzuDGJ14K7XsPGzPoaP4QwjDdBePco6ZU/TcksT+BEoGItI2otf07DMyjl4gOWwn03KOTXRqifMnpNPYnaCgRmNmfmNlzZjZoZj0Vz33ezFaa2QtmFtihZWbvNrMHzezF4t97NhKPiORb1B39oEPYtjTlS0SXVgJ9/NIT+ctjpwLwf54Yvnx0s1QuOR32b0hyf4JGWwTLgT8Cflr+oJkdxtDG9e8DTgW+bWZBQ/KXAg+7+8HAw8XvRUTqEnRHb0bkiqAlpb53eGca6o1PvtONlITSwO+82dN21geksT9BQ4nA3Ve4+wsBT50F3OruO9x9NbASmBFy3A+LX/8Q+B+NxCMiUrqjX37FKfzlsVPp6rBYF/LyvvewKZzNVp58SiqLyXbrLiS+VlFSYwSTgHVl368vPlZpb3ffCFD8+71hb2hmc8xsqZkt3bRpU1ODFZH2dOvSdbFXFi3ve4+a799MlQO/QcVkW3sHEi0mgxiJwMweMrPlAX/OinpZwGMN/a+6+wJ373H3ngkTJjTyViKSA3Gmk8Lwvve4r2tU0MBvGsVkEKOOwN1PquN91wP7l32/H7Ah4LjfmNk+7r7RzPYBXq3jZ4mIDFOaTlpt/v4FM6fu0u0S93WNCBv4jVN5/LlTD236iqVJdQ3dBZxtZqPMbCpwMPBUyHHnF78+H7gzoXhEJIeipmJ2dcCc4w/cZaA2zuu6C8b7J+0ePk0V6CrYzkHe82dO4YJjDqg68Bt3R7VDLruP6Zcv5Kp7VvDm283pKmpoiQkz+ygwH5gAvAE84+6nFJ+7DPgE0A98xt3vKz5+PXCduy81s/cA/w5MBtYCf+Luv6v2c7XEhIjE8ebbfXXtGFbtdTf/9VGc869LIt+3cqVTqL6kdC1LXITtmBYlbIkJrTUkIm2t3iWdq70uiaWir7pnRU1LXETtexBEiUBEcq/eTV6qva5Zm8eEtUSi1LIIXVgi0BITIv8Q4V0AAAUMSURBVJIb9V6sq72uWYO3lcVkcQTVItRKiUBEJEPKC+JevPK0qgmhGYvQKRGISCb1NXiX2w66Ch0tWYRO+xGISGa0eq/ekWDurINY+PwroTOUmrEInVoEIpIJaezVOxK0YhE6tQhEJBPiLK8Qd5pkuymNG8ybPa1pM5TKqUUgIpkQZ3kFad4MpXJKBCKSurj7DTc6TVKCKRGISOri7DecxF69MkT/qyKSCWns1StDlAhEJBPS2KtXhigRiEgmpLFXrwzRonMikklJTJPMOy06JyIjipJA6+h/WkQk55QIRERyTolARCTnRuRgsZltAtYk8NZ7Aa8l8L7Novgao/jql+XYQPHFNcXdJ1Q+OCITQVLMbGnQiHpWKL7GKL76ZTk2UHyNUteQiEjOKRGIiOScEsGuFqQdQBWKrzGKr35Zjg0UX0M0RiAiknNqEYiI5JwSgYhIzikRVDCzI8zsSTN7xsyWmtmMtGMqZ2aXmNkLZvacmX017XiCmNnfmpmb2V5px1LOzL5mZr80s1+Y2R1mtkfaMQGY2anF3+lKM7s07XjKmdn+Zvb/zGxF8Zz7dNoxBTGzgpn9l5ndnXYslcxsDzP7j+K5t8LMjkk7pkpKBMN9FbjC3Y8Avlj8PhPM7EPAWcD73f19wNdTDmkYM9sfOBnI4gazDwLT3f39wK+Az6ccD2ZWAK4FTgMOAz5uZoelG9Uu+oHPuvs04Gjg4ozFV/JpYEXaQYS4Grjf3X8fOJwMxqlEMJwD44tf7w5sSDGWSp8E/snddwC4+6spxxPkn4H/zdD/Y6a4+wPuXtoY90lgvzTjKZoBrHT3Ve7eC9zKULLPBHff6O4/K379FkMXsUnpRrUrM9sPmA1cn3YslcxsPPCHwPcA3L3X3d9IN6rhlAiG+wzwNTNbx9Add+p3jWUOAY43syVm9p9m9sG0AypnZmcCL7v7z9OOJYZPAPelHQRDF9V1Zd+vJ2MX2hIzOwD4b8CSdCMZ5l8YuvnI4s72BwKbgB8Uu66uN7OxaQdVKXq36DZlZg8BEwOeugyYBfwvd7/NzP6UoUx+UkZi6wT2ZKiJ/kHg383sQG/hHOAq8c0DPtyqWIJExefudxaPuYyhLo+bWhlbiKBNejPXmjKzccBtwGfc/c204ykxs48Ar7r7MjM7Ie14AnQCRwKXuPsSM7sauBT4Qrph7Up1BBXMbDOwh7u7mRmw2d3HV3tdK5jZ/Qx1DT1S/P7XwNHuvinVwIZi+QPgYWBb8aH9GOpWm+Hur6QWWAUzOx+4CJjl7tuqHd+CeI4BvuTupxS//zyAu38l1cDKmFkXcDew0N2/mXY85czsK8BfMJTYRzPUrXu7u/95qoEVmdlE4El3P6D4/fHApe4+O9XAKqhraLgNwH8vfn0i8GKKsVT6MUMxYWaHAN1kY0VD3P1Zd3+vux9QPOnXA0dmLAmcCvwdcGYWkkDR08DBZjbVzLqBs4G7Uo5pp+LN0PeAFVlLAgDu/nl33694zp0NLMpKEgAonv/rzOzQ4kOzgOdTDClQLruGqvhr4Goz6wTeBuakHE+57wPfN7PlQC9wfiu7hdrANcAo4MGh6xtPuvtFaQbk7v1mNhdYCBSA77v7c2nGVOFYhu64nzWzZ4qPzXP3e1OMaaS5BLipmOhXARemHM8w6hoSEck5dQ2JiOScEoGISM4pEYiI5JwSgYhIzikRiIjknBKBiEjOKRGIiOTc/weJH1I8/rfaggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Create some fake data\n",
    "from sklearn.datasets import make_blobs\n",
    "\n",
    "X, y = make_blobs(random_state = 10,centers=4)\n",
    "plt.scatter(X[:, 0], X[:, 1], s=65)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start with K-Means clustering, a common clutering approach where points are initially assigned to some initial random clusters, then points are repeatedly assigned to whichever cluster a majority of their neighbors belong to until no points change cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\matthew.d.smith\\anaconda3\\lib\\site-packages\\seaborn\\axisgrid.py:243: UserWarning: The `size` parameter has been renamed to `height`; please update your code.\n",
      "  warnings.warn(msg, UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x20a0250b550>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAGoCAYAAAAdGw+vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7BdZX3v8c+Hk9DEaBM7BnMSoEIvUhDCD49UZKqSYAAjBS2XYue2WO+YoVfawFjbgJVJ8VaptAL+aL1YaekMSikCJQYhCCqFFuXwM0CkYPxBkqOEtokUEyHhe//Yayfn7LN/r7X3Wnvt92smc85Ze+3NlxbzOc+znuf7OCIEAAC6t0/eBQAAMOgIUwAAUiJMAQBIiTAFACAlwhQAgJRm5F1AP51yyilx22235V0GAGAv511AFoZqZPrcc8/lXQIAoISGKkwBAOgFwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUipkmNo+1PbDk/781Pb5Nfe83fb2SfdcnFe9AIDhVshTYyLiSUlHS5LtEUmbJd1U59Z/iYh39bM2AABqFXJkWmOppO9FxA/zLgTA4Fq7ca2W3bBMi69ZrGU3LNPajWvzLgklMghherakLzd47Xjbj9j+mu039LMoAINj7ca1Wv2vqzXxwoRCoYkXJrT6X1cTqMhMocPU9r6SfkPSP9V5+UFJvxwRR0n6jKSbG3zGCtvjtse3bt3au2IBFNaVD16pnbt3Trm2c/dOXfnglTlVhLIpdJhKOlXSgxHxk9oXIuKnEfHfyfe3Sppp+zV17rsqIsYiYmz+/Pm9rxhA4fz4hR93dB3oVNHD9L1qMMVre4FtJ98fp8q/y3/0sTYAA2LBnAUdXQc6Vdgwtf0KSe+QdOOka+faPjf58UxJj9l+RNKnJZ0dEdH/SgEU3cpjV2rWyKwp12aNzNLKY1fmVBHKxsOUP2NjYzE+Pp53GQBysHbjWl354JX68Qs/1oI5C7Ty2JVafvDyvMuC5LwLyAJhCgDIUynCtLDTvAAADArCFACAlAhTAABSIkwBAEiJMAUAICXCFAByQOP9cinkEWwAUGbVxvvVfsHVxvuS2Ps6oBiZAkCf0Xi/fAhTAOgzGu+XD2EKAH1G4/3yIUxL4uaHNuuES+/SQavW6oRL79LND23OuyQADdB4v3xYgFQCNz+0WRfeuF47XtotSdq8bYcuvHG9JOmMYxbVvf+y25/Ulm07tHDebH345EPr3gegN6qLjGi8Xx40ui+BEy69S5u37Zh2fdG82bp31ZIp12qDV5JmzxzRJ95zJIEKIA80ukcxbKkTpI2uX3b7k1OCVJJ2vLRbl93+ZE9qA4BhQJiWwMJ5s9u+3knwAgDaQ5iWwIdPPlSzZ45MuTZ75og+fPKh0+7tJHgBAO0hTEvgjGMW6RPvOVKL5s2WVXlW2ugZaCfBCwBoD6t5S+KMYxa1tYCoeg+reQEgO4RpSXSy3aXd4AUAtIcwLYFO95m2+5mMXlF0azeuZa8mCoFnpiWQ9XaXajhv3rZDob3hTFclFEn15JWJFyYUij0nr6Q5yqxox6IVrR40Rpj2SD/b+2W53eXmhzbrQ9c/wl5UFF7WJ6/0IpzTKFo9aI4w7YF+j+yy2u5SrXt3g65YzcKZ3sDot6xPXinasWhFqwfNEaY90KsuQ40CK6vtLvXqnmzeK2Y2rItpYfRb1ievFO1YtKLVg+YI0x7oRZehZoHVyT7TbuquatTGmRaFyEPWJ68U7Vi0otWD5gjTHuhFl6FWgXXGMYt076ol+v6ly3XvqiVdrbxtVd/2HS/VvU6LQuRh+cHLtfotqzU6Z1SWNTpnVKvfsrrr1bxFOxataPWgObbG9MCHTz607sksaboM9SOw6tU9WbNfEuqdWkOLwuHT760qyw9entnnF+1YtKLVg+YI0x7oRZehfgRWtb7VtzyubTWj0Ga/DPTilwcMnurq0+qimerqU0kDEwBZhnMWilYPGuM80wHR73NIO23aQJMHLLthmSZemJh2fXTOqNadua7h+2i8MPRKcZ4pYTpACCwU2eJrFis0/e8Ty3r0nEfrvqd2NCtVngumefbZCKFdWKUIU6Z5C6ZZYNJTF0W2YM6CuiPTZqtPm+2l7DTomoVlGaagUWys5i0Q9ms2RlOI4utm9WlWeylbdQuiAQJ6jTAtEPZr1scvGYOhm60qWe2lbBWWWYY2vXJRD9O8BcJ+zfqa/ZLBtHexdLr6dOWxK+s+M+10L2WrsOxmCroWU8VohpFpgfSi2UO/9WI6ll8yyiurxgutRrhZNEBgqhjNMDItkBN/db6uve9HU9ZDDtJ+zV6cqyrRFKLs2hnNtlqJ22qEm0UDBHrlopnChqntH0h6XtJuSbsiYqzmdUu6UtI7Jf1M0vsi4sF+15mVmx/arK88sHlKkFrSb76xtyt4q6uHN2/boRFbuyO0qMttN72ajqUpxHBrZ3q1nbBM2wAhi6lilFdhwzRxYkQ81+C1UyUdkvz5NUl/k3wdSPWCKCR947tb9/yc9T7T2pFk9ei1bkeUvZqO7UVHKQyOdrfP9LpbUFbPd1FORQ/TZk6X9A9R6Tpxn+15tkcjYvqvjgOgVRD1Ygq12ZFr7Y4oJwf8PsnItlYW07HssR1eRZlepVcumilymIakdbZD0v+LiKtqXl8k6ZlJP29Krg1kmLZ6LtiLKdRWI8ZWrzca2U7GdCzSajS9Ggotu2FZXwONXrlopMireU+IiGNVmc79oO231rxerwXVtL/Nba+wPW57fOvWrXXeUgytDvjuxRRqqxFjq9cbjWxH7FTnqgKT1VuJW1XbnAHIS2HDNCK2JF+flXSTpONqbtkk6YBJP+8vaUudz7kqIsYiYmz+/Pm9Kje1Vgd892LbTL0Ar2pnRNkoyF+OSHWuKoqt340LJm+fqYftKSiCQk7z2p4jaZ+IeD75fpmkS2puu0XSebavU2Xh0fZBfV5a1ey5YC9WtJ5xzCKN//A/9eVvPzNlirbd1bxsWRk+eTUuqE6vNmqmz/YU5K2oI9PXSrrH9iOSviNpbUTcZvtc2+cm99wqaaOkpyV9QdL/yafU/mg1cu1GdTvO5CCtBnQ7n1tvZGtVFkfRP7ec8m5ckFX7QSBrHME2xE649K66I8tF82br3lVL2vqMyftULU1rOMEz03Lp5pi1LNU7sm3mPjM1wzO0Y3flv+W5+87Vhb92IQuFBkcpjmAr6si0tIp0+kkWi5rOOGaR7l21RIvmzZ72VyxN+ssn75FhbfvBeb8wT7tf3r0nSCVp+4vb9af3/CmLktBXhGkfFe30kywXNdE/dzhk0eM2reUHL9e6M9ftGQm/rJen3bMrdrEoCX1FmPZRr49Y63TU22o7TifK0KQfrWXVmD4Lazeu1bafb2v4er29qUCvFHI1b1n1cvTWTYekLNv00T93eBSlcUE7I8+1G9cWolaUH2HaR73cStJth6Ss2vTRPxf91s52mNr+vUCvEKZ91MvRWxGeWdI/F/3UqM3gZL3Yf9rqODgMJ56Z9lEv9opW5fHMskgrkzF8mrUZrMp6lXF1a87ECxMKBe0MsQcj0z7r1eit388se3UQOIZDFqO7yae4NBqhvnX/2pbe6bR7HByGDyPTkujlqLeeXq9MRnllObqrbpNp1Lf37k13py13iqIcB4fiYWRaIv18ZlmEZ7QYTL0Y3fUr5Bo9p6WdIRiZtoFng9OxrxTd6kXw9aszUxGaVqCYCNMWita1qCiybPiA4dKL4OtXyBWpaQWKhWneFrrdv1l27CtFt1Yeu3Jas/q0wTd5MVKvt6wUpWkFioVTY1o4aNXaOmdkVI45+P6l/A8K6AZ7NTFJKU6NYWTaAgdgA9ljdIey4ZlpCzwbBAC0wsi0BZ4NAgBaIUzbQM9ZAEAzTPMCQBvWblyrZTcs0+JrFmvZDcvox4spGJkCQAvVFojV7TzVFoiSWEgFSYxMAaClZi0QAYkwBYCWaHCPVghTAGihX71/MbgI04KgmT5QXDS4RyssQCoADtoGiq2fvX8xmOjNWwAnXHpX3ZaFi+bN1r2rluRQEQD0TSl68zLNWwActA0Ag40wLQAO2gayR5MF9BNhWgA00weyVW2yMPHChEKxp8lCu4FKEKNTLEAqAJrpA9lq1GThonsuktS8axHdjtANFiABKJ3F1yxWqP7fbbNGZmn1W1Y3DMZlNyzTxAsT066PzhnVujPXZVonJLEACQCKY/LUrN347+dWbQDpdoRuEKYABl7tM9KX4+Wm9zcLRrodoRuEKYCBV+8ZaTPNgpFuR+gGC5AADLxmI81ZI7OmBG2rYKTbEbpBmAIYeAvmLGi4aGjlsSs7DsblBy8nPNERwhTAwFt57Mop21mkvSNQghH9UMhnprYPsP0N2xtsP2572pyM7bfb3m774eTPxXnUCiB/yw9ertVvWa3ROaOyrNE5o023vwBZK+rIdJekD0XEg7ZfJekB23dExBM19/1LRLwrh/oAFAwjUOSpkCPTiJiIiAeT75+XtEES7YAAAIVUyDCdzPbrJB0j6dt1Xj7e9iO2v2b7DQ3ev8L2uO3xrVu39rBSAMCwKnSY2n6lpK9IOj8iflrz8oOSfjkijpL0GUk31/uMiLgqIsYiYmz+/Pm9LRgAMJSK+sxUtmeqEqTXRsSNta9PDteIuNX2X9t+TUQ81886h9XND22mMT8AJAoZpq401vyipA0R8akG9yyQ9JOICNvHqTLK/o8+ljm0bn5osy68cb12vLRbkrR52w5deON6SSJQAQylQoappBMk/Y6k9bYfTq5dJOlASYqIz0s6U9Lv294laYeks2OYjsDJ0WW3P7knSKt2vLRbl93+JGEKYCgVMkwj4h61OJYnIj4r6bP9qQiTbdm2o6PrAFB2hV6AhGJaOG92R9cBoOwIU3TswycfqtkzR6Zcmz1zRB8++dCcKgKAfBVymhfFVn0uympeAKggTNGVM45ZRHgCQIJpXgAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUiJMAQBIiTAFACAlwhQAgJQIUwAAUipsmNo+xfaTtp+2varO67b96eT1R20fm0edAAAUMkxtj0j6nKRTJR0u6b22D6+57VRJhyR/Vkj6m74WCQBAopBhKuk4SU9HxMaIeFHSdZJOr7nndEn/EBX3SZpne7TfhQIAUNQwXSTpmUk/b0qudXqPbK+wPW57fOvWrZkXCgBAUcPUda5FF/coIq6KiLGIGJs/f34mxQEAMFlRw3STpAMm/by/pC1d3AMAQM8VNUzvl3SI7YNs7yvpbEm31Nxzi6TfTVb1vlnS9oiY6HehAADMyLuAeiJil+3zJN0uaUTS1RHxuO1zk9c/L+lWSe+U9LSkn0n6vbzqBQAMN0dMe8xYWmNjYzE+Pp53GQCAveqtfxk4RZ3mBQBgYBCmAACkRJgCAJASYQoAQEqEKQAAKRGmAACkRJgCAJASYQoAQEqEKQAAKRGmAACkRJgCAJASYQoAKCzbC2xfZ/t7tp+wfavt19t+rMvPe5/thVnXSZgCAArJtiXdJOmbEfErEXG4pIskvTbFx75PUkdharvlCWuEKQCgqE6U9FJy7KYkKSIelvRM9edkpPnZST9/1fbbbY/Y/nvbj9leb/sC22dKGpN0re2Hbc+2/Ubb37L9gO3bbY8mn/NN2x+3/S1JK1sVWsjzTAEAkHSEpAe6fO/RkhZFxBGSZHteRGxLzsr+o4gYtz1T0mcknR4RW23/lqQ/l/T+5DPmRcTb2vmHEaYAgDLaKOlg25+RtFbSujr3HKpKYN9RmVHWiKSJSa//Y7v/MMIUAFBUj0s6s8U9uzT1keUsSYqI/7J9lKSTJX1Q0lnaO+KssqTHI+L4Bp/9QruF8swUAFBUd0n6BdsfqF6w/SZJvzzpnh9IOtr2PrYPkHRcct9rJO0TEV+R9FFJxyb3Py/pVcn3T0qab/v45D0zbb+hm0IZmQIACikiwva7JV1he5WknaqE5/mTbrtX0vclrZf0mKQHk+uLJP2d7eqg8cLk699L+rztHZKOV2Xk+2nbc1XJxCtUGRF3xBHR6XsG1tjYWIyPj+ddBgBgL+ddQBaY5gUAICXCFACAlAhTAABSIkwBAEiJMAUAICXCFACAlAhTAAAStk+x/aTtp5O9rW0hTAEAkGR7RNLnJJ0q6XBJ77V9eDvvpQMSAGAgvW7V2t+W9HFJB0r6kaSLfnDp8i+l+MjjJD0dERslyfZ1kk6X9ESrN7Ycmdo+z/arUxQHAECmkiD9gip9ep18/UJyvVuLNOmsVEmbkmsttTPNu0DS/bavT+aSS9H6CQAw0D4u6RU1116RXO9WvXxrq+duyzCNiD+VdIikL0p6n6SnktPHf6WTCgEAyNCBHV5vxyZJB0z6eX9JW9p5Y1sLkKLSDf/HyZ9dkl4t6Qbbn+ysTgAAMvGjDq+3435Jh9g+yPa+ks6WdEs7b2znmekf2n5A0idVOermyIj4fUlvlPSb3dcMAEDXLpL0s5prP0uudyUidkk6T9LtkjZIuj4i2jqOrZ2R6WskvSciTo6If4qIl5J/6MuS3tVlzQAAdC1ZtfsBST9U5bnmDyV9IOVqXkXErRHx+oj4lYj483bfx3mmAIA8lWJRK00bAABIqXBNG2xfJuk0SS9K+p6k34uIbXXu+4Gk5yXtlrQrIsb6WScAAFVFHJneIemIiFgs6d8lXdjk3hMj4miCFACQp8KFaUSsS1ZUSdJ9quzzAQCgsAoXpjXeL+lrDV4LSetsP2B7RaMPsL3C9rjt8a1bt/akSADAcMslTG1/3fZjdf6cPumej6jSIOLaBh9zQkQcq0p3/w/afmu9myLiqogYi4ix+fPnZ/7vAgAoD9tX237W9mOdvC+XBUgRcVKz122fo8oe1qXRYO9ORGxJvj5r+yZVuv3fnXWtAICh8veSPivpHzp5UxFX854i6U8kvS0iartbVO+ZI2mfiHg++X6ZpEv6WCYAIG+r5047gk2rt6dt2nC37dd1+r4iPjP9rKRXSbrD9sO2Py9JthfavjW557WS7rH9iKTvSFobEbflUy4AoO8qQTrtCLbket8VbmQaEf+jwfUtkt6ZfL9R0lH9rAsAUCjNjmBLNTrtRhFHpgAAtNKLI9i6RpgCyNaj10uXHyGtnlf5+uj1eVeEcurFEWxdI0wBZOfR66U1fyhtf0ZSVL6u+UMCFb2Q+RFskmT7y5L+TdKhtjfZ/t/tvI8wBZCdOy+RXtox9dpLOyrXgSxVVu1OO4Itg9W8742I0YiYGRH7R8QX23lf4RYgARhg2zd1dh1IoxKcfV9sVA8jUwDZmduglXaj60BJEKYAsrP0Ymnm7KnXZs6uXAdKjDAFkJ3FZ0mnfVqae4AkV76e9unKdaDEeGYKIFuLzyI8MXQYmQIAkBJhCgBAwvYBtr9he4Ptx22vbOd9TPMCALDXLkkfiogHbb9K0gO274iIJ5q9iTAFAAykI685ctoRbOvPWZ+2acOEpInk++dtb5C0SFLTMGWaFwAwcJIgnXYEW3I9E8m5psdI+narewlTAMAganYEW2q2XynpK5LOj4iftrqfMAUADKKeHcFme6YqQXptRNzYznsIUwDAIOrJEWy2LemLkjZExKfafR9hCqCCc0gxWHpyBJukEyT9jqQlth9O/ryz1ZtYzQtg7zmk1ePTqueQSnQzQiGtP2f9l4685kgp+9W896iyoKkjjog0/9yBMjY2FuPj43mXARTP5UckB3rXmHuAdMFj/a8Hw6Tj4CoipnkBFOscUqabMYAIUwDFOYe0Ot28/RlJsXe6mUBFwRGmAIpzDumdl+x9blv10o7KdaDACFMAxTmHtEjTzUAHWM0LoKII55DO3b/BQqg+TzcDHWJkCqA4ijLdjKFle5bt79h+JDmC7c/aeR9hCqA4K2iLMt2MYfZzSUsi4ihJR0s6xfabW72JaV5g2BWtYUMRppsxEDb86mHTjmA77Lsb0jZtCEn/nfw4M/nTsiEDI1Ng2LGCFgMoCdJpR7Al11OxPWL7YUnPSrojIjiCDUALrKDFYOrZEWwRsTsijpa0v6TjbB/R6j2EKTDsitKwAehMz45gq4qIbZK+KemUVvcSpkCZdLOQiBW0GEy9OoJtvu15yfezJZ0k6but3keYAmXRbSs+VtBiMPXqCLZRSd+w/aik+1V5ZvrVVm/i1BigLDj5BYOp61NjerGat1tsjQHKYpAWEj16fWW18PZNlWezSy9mJIyOJcGZS3jWYpoXKItBWUjEyTAoIcIUKItBWUjEvlaUUOHC1PZq25ttP5z8eWeD+06x/aTtp22v6nedQOEUbSFRo5XFgzQdDbSpqM9ML4+Iv2z0ou0RSZ+T9A5JmyTdb/uWiHiiXwUChVSUVnzNWhRyMgxKqHAj0zYdJ+npiNgYES9Kuk7S6TnXBJRH2sb3zaZyB2U6GuhAUcP0PNuP2r7a9qvrvL5I0uRfbTcl16axvcL2uO3xrVu39qJWoFyyWCDUcCr3meJNRwMZyGWa1/bXJS2o89JHJP2NpI+p0qX/Y5L+StL7az+iznvrbpiNiKskXSVV9pl2WTJQHL3eVtJsVNnuP6fRVK5cqb8o09FARnIJ04g4qZ37bH9BUr3OE5skHTDp5/0lbcmgNKDY+nFcWhYLhJZeLN24QtN/x43OQhkYEIWb5rU9OunHd0uq17rlfkmH2D7I9r6SzpZ0Sz/qA3LVj20lne5Xrfd8dfFZangEJKt2UUKFC1NJn7S9PumLeKKkCyTJ9kLbt0pSROySdJ6k2yVtkHR9RDyeV8FA3zR7FtntYqFanSwQavZ8de4B0++XWLWLUqI3LzBIGvXfnWzmbOmo35aeWtf9c9V2n8s26we89OKpU9LV2lhshKm67s1bJEXdZwqgnnoBVeulHdL41dozzdrNc9V2Fwi1er46Y/beWmf/knTqXxCkKKUiTvMCaKR2W0lDNTNOvWrX12jKdvarKwG+4z/3XtvV5BcAYMARpsCgWXxW5Ui11dsaP5espxcLfxo9X5Xov4uhQpgCg6xemDUasfZi4U+jBgw7/qv+/azkRUnxzBQYZNXnj5MXCx2yTHrkS9MX/vSqXV+956t3XkL/XQwVwhTIW9qORvXC7MA353v4dqOVvPTfRUkRpkCeetXRKO92ffVGzP0OdKCP2GcK5KnZPs0L6jX/aqHXfXuB7JVinykLkIA8Neto1GknoyxOewHQFaZ5gTw1PF1FnU/3turb22jEymgWSI1pXiBPtc9Ma3Uy3bt6nho2l585u35bP4mWf8hbKaZ5GZkCeaoG1o0fqP96J/syG41yPdJ8xJr27FIAPDMFcrf4rGxOWGnUjSh2179/+6Zszi4FQJgChdDJsWeNNOpG1CyoOz27FEBdTPMCRZDVvsxG+0ubNVAYxOYKLJpCwRCmQFH0qtFCO0HdKpiKFF69anQBpMBqXgDN1VtxnOeK36wbXSBvpVjNyzNTAM212r/abyyaQgERpgCaK1p4sWgKBUSYAmiuaOGVxcpnIGOEKYDmihZejbYAsfgIOWI1L1BGWa6+LeJxankfMQfUIEyBsunF1hHCC2iKaV6gbIq2+hYYAoQpUDZFW30LDAHCFBhEj15faV6wel7l6+QDwIu2+hYYAoQpMGiqz0S3PyMp9j4TrQbqIcs0rakMW0eAnmIBEjBoWj0TfeRLmnpIuKWjfpsFREAPMTIFBk2zZ6L1glYhPbWu52UBw4wwBQZNs2eiLD4CckGYAoOmWUeibhYfNVvMBKAtPDMFBk2rjkSdHPbN2aBAJjjPFCibTloJcjZo6Wxfs0bPXn6Fdk1MaMboqPa74HzNPe20vMtqphTnmTIyBcqmk9Z/PGMtle1r1mjioxcrdu6UJO3askUTH63MShQ8UAcez0yBYUaDh1J59vIr9gRpVezcqWcvvyKnioYHYQoMs6Idr4ZUdk1MdHQd2SFMgWHG2aClMmN0tKPryE7hnpna/kdJhyY/zpO0LSKOrnPfDyQ9L2m3pF0RMda3IoEy4Xi10tjvgvOnPDOVJM+apf0uOD/HqoZD4cI0In6r+r3tv5K0vcntJ0bEc72vCgCKr7rIaMBW85ZC4cK0yrYlnSVpSd61AMCgmHvaaYRnDor8zPTXJf0kIp5q8HpIWmf7AdsrGn2I7RW2x22Pb926tSeFAgCGWy4jU9tfl7SgzksfiYh/Tr5/r6QvN/mYEyJii+39JN1h+7sRcXftTRFxlaSrpErThpSlAwAwTS5hGhEnNXvd9gxJ75H0xiafsSX5+qztmyQdJ2lamAIA0GtFneY9SdJ3I6JuGxbbc2y/qvq9pGWS6H0GAMhFUcP0bNVM8dpeaPvW5MfXSrrH9iOSviNpbUTc1ucaAQCQVNDVvBHxvjrXtkh6Z/L9RklH9bksABgaA9gwP1eFDFMAQH5omN+5ok7zAgByQsP8zhGmAIApaJjfOcIUADAFDfM7R5gCwADbvmaNnlqyVBsOO1xPLVmq7WvWpP7M/S44X541a8o1GuY3xwIkABhQvVooRMP8zjlieDrsjY2Nxfj4eN5lAEAmnlqyVLu2bJl2fcbChTrkrjtzqKgrzruALDDNCwADqp2FQr2YBsZ0hCkADKhWC4Wq08C7tmyRIvZMAxOo2SNMAWBAtVooxH7R/mEBEgAMqFYLhdgv2j+EKQD0Qa963c497bSGnzNjdLT+AiX2i2aOaV4A6LG8nl2yX7R/CFMA6LG8nl3OPe00jX7sEs1YuFCyNWPhQo1+7BL2i/YA07wA0GN5PrtsNg2M7DAyBYAeo9dt+RGmANBjPLssP8IUAHqs3WeXdCsaXPTmBYACqG1aL1VGr6Mfu0RSqZvOl6I3L2EKAAXQqGn9yLx5ennnzrohW5JALUWYMs0LAAXQaGXv7ids22YAAAooSURBVG3baAk4AAhTACiATlf21gtfnrnmhzAFgAJotOLX8+bVvb82fDkhJl+EKQAUQKMVv6MfuaitbTWcEJMvOiABQEHUdiuqNsePnTulkRFp927NWLhwT5A+tWTpnhW+9RYvSZwQ0y+EKQAU0LStMrt3TxmRTn6tUZBKdFnqF6Z5AaCAmk3b1ntNkuSpu0za6bLU7qIlFjc1R5gCQA+kDZ9mzfEbTt1GdHRCTLuLlljc1BphCgAZyyJ8mjXHb/ha8jx1xuiodk1M6NnLr2j6z2x30RKLm1ojTAEgY1mET7Pm+I1ee+Xb3tpRiLd7NFyeR8gNCsIUADKWRfg0a47f6LX//tbdHYV4u0fDcYRca6zmBYCMNdqq0mn4NDvYu95rW/74T+re2yjE97vg/LrN9WsXLbV73zBjZAoAGcvr/NJmI8h6C6LaPRqu3fuGGafGAEAPVBsu1Ds2rdlraT5fUt0R5Nx3n6HtN91c1JNnSnFqDGEKAH3U7NzSdoOt07NPn738ivrTzgsX6pC77kz5b5QaYTpoCFMAeWt0bmknwdbpZ2w47HCp3t/1tg7b8ERb/8weKkWY8swUAPooi5W+7X5G9Tlp3SCVpH32ofFCRnIJU9v/0/bjtl+2PVbz2oW2n7b9pO2TG7z/l2zfYfup5Our+1M5AKSTxTaTdj5jSuOIRnbvppNRRvIamT4m6T2S7p580fbhks6W9AZJp0j6a9sjdd6/StKdEXGIpDuTnwGg8LJY6VvvM2Rr15Yte1bqNuzfW4NORtnIZZ9pRGyQJHvaVPnpkq6LiJ9L+r7tpyUdJ+nf6tz39uT7ayR9U1L9DVYAUACTV9967lyNzJql3du3d7Wat3rvlIVFyVRutetRO0FaRSej9IrWtGGRpPsm/bwpuVbrtRExIUkRMWF7v0YfaHuFpBWSdOCBB2ZYKgC0p3b1bWzbppdnzdLCT/5FWyHaaCvN3NNOq7sYafL5p+2gk1F6PZvmtf1124/V+XN6s7fVuZZquXFEXBURYxExNn/+/DQfBQBdSdOrt1XT/IajyuT808k8c6Y0Y+oYik5G2ehZmEbESRFxRJ0//9zkbZskHTDp5/0l1Xt6/hPbo5KUfH02u8oBIFtpVvC2CuJmJ8hM61r08T/Xwk98nE5GPVC0ad5bJH3J9qckLZR0iKTvNLjvHEmXJl+bBTQA5CpNr95WQdysb26j3r5ZdFsigKfKa2vMu21vknS8pLW2b5ekiHhc0vWSnpB0m6QPRsTu5D1/O2kbzaWS3mH7KUnvSH4GgEJKs4K31TaYXvfN5WDw9tABCQD6oNvRXRbtB9PIomNTC6XogFS0aV4AKKVmx6m1ep80vd9uv6ZZORi8PYQpABRct0GchazOZi07evMCABrK62zWQcPIFADQUKNpZil5nsoKX0ksQAIAdCjjRVGlWIDENC8AoCNpOjqVFWEKAOgIK3ynI0wBAB3J4kzWsiFMAQAdYYXvdKzmBQB0JO9GEkXEal4AQJ5YzQsAAAhTAABSI0wBAEiJMAUAICXCFACAlAhTAJhk+5o1emrJUm047HA9tWSptq9Zk3dJGADsMwWARG0D911btmjioxdL0lDvoURrjEwBIEEDd3SLMAWABA3c0S3CFAASNHBHtwhTAEjQwB3dYgESACRo4I5u0egeAJAnGt0DAADCFACA1AhTAABSIkwBAEiJMAUAICXCFACAlAhTAABSIkwBAEiJMAUAICXCFACAlAhTAABSIkwBAEhpqBrd294q6Yc9+vjXSHquR5+dNWrtjUGpdVDqlKi1V4pU63MRcUreRaQ1VGHaS7bHI2Is7zraQa29MSi1DkqdErX2yiDVOiiY5gUAICXCFACAlAjT7FyVdwEdoNbeGJRaB6VOiVp7ZZBqHQg8MwUAICVGpgAApESYAgCQEmGaIdtH277P9sO2x20fl3dNzdj+A9tP2n7c9ifzrqcV239kO2y/Ju9a6rF9me3v2n7U9k225+VdUy3bpyT/P3/a9qq862nE9gG2v2F7Q/Lf58q8a2rG9ojth2x/Ne9amrE9z/YNyX+nG2wfn3dNZUGYZuuTkv4sIo6WdHHycyHZPlHS6ZIWR8QbJP1lziU1ZfsASe+Q9KO8a2niDklHRMRiSf8u6cKc65nC9oikz0k6VdLhkt5r+/B8q2pol6QPRcRhkt4s6YMFrlWSVkrakHcRbbhS0m0R8auSjtJg1DwQCNNshaRfTL6fK2lLjrW08vuSLo2In0tSRDybcz2tXC7pj1X5v3EhRcS6iNiV/HifpP3zrKeO4yQ9HREbI+JFSdep8gtV4UTEREQ8mHz/vCp/6S/Kt6r6bO8vabmkv827lmZs/6Kkt0r6oiRFxIsRsS3fqsqDMM3W+ZIus/2MKiO9Qo1Marxe0q/b/rbtb9l+U94FNWL7NyRtjohH8q6lA++X9LW8i6ixSNIzk37epIIG1GS2XyfpGEnfzreShq5Q5Re9l/MupIWDJW2V9HfJlPTf2p6Td1FlMSPvAgaN7a9LWlDnpY9IWirpgoj4iu2zVPkN8KR+1jdZi1pnSHq1KlNob5J0ve2DI6e9Ui1qvUjSsv5WVF+zOiPin5N7PqLKNOW1/aytDa5zrbAjfUmy/UpJX5F0fkT8NO96atl+l6RnI+IB22/Pu54WZkg6VtIfRMS3bV8paZWkj+ZbVjmwzzRDtrdLmhcRYduStkfEL7Z6Xx5s36bKNO83k5+/J+nNEbE118Jq2D5S0p2SfpZc2l+V6fPjIuLHuRXWgO1zJJ0raWlE/KzV/f2ULDZZHREnJz9fKEkR8YlcC2vA9kxJX5V0e0R8Ku966rH9CUm/o8ovT7NUecxzY0T8r1wLq8P2Akn3RcTrkp9/XdKqiFiea2ElwTRvtrZIelvy/RJJT+VYSys3q1KjbL9e0r4qzikSe0TE+ojYLyJel/wlsEnSsQUN0lMk/Ymk3yhakCbul3SI7YNs7yvpbEm35FxTXckvo1+UtKGoQSpJEXFhROyf/Ld5tqS7ihikkpT8b+YZ24cml5ZKeiLHkkqFad5sfUDSlbZnSNopaUXO9TRztaSrbT8m6UVJ5+Q1xVsin5X0C5LuqGSB7ouIc/Mtaa+I2GX7PEm3SxqRdHVEPJ5zWY2coMqIb73th5NrF0XErTnWVAZ/IOna5JepjZJ+L+d6SoNpXgAAUmKaFwCAlAhTAABSIkwBAEiJMAUAICXCFACAlAhTAABSIkwBAEiJMAUGiO03JeelzrI9Jznr84i86wKGHU0bgAFj+/+q0gd2tqRNRe2tCwwTwhQYMEkruPtVaVn5lojYnXNJwNBjmhcYPL8k6ZWSXqXKCBVAzhiZAgPG9i2SrpN0kKTRiDgv55KAocepMcAAsf27knZFxJdsj0j6V9tLIuKuvGsDhhkjUwAAUuKZKQAAKRGmAACkRJgCAJASYQoAQEqEKQAAKRGmAACkRJgCAJDS/wctVN4uTMpc3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 475.5x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Import KMeans class\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "#Load and fit a model\n",
    "kmeans = KMeans(n_clusters = 4)\n",
    "kmeans.fit(X)\n",
    "#Get the labels for each point\n",
    "labels = kmeans.labels_\n",
    "\n",
    "#And that's it!  See the reulting clusters\n",
    "df = pd.DataFrame({'x':X[:,0],'y':X[:,1],'Cluster':labels})\n",
    "sns.FacetGrid(df,hue=\"Cluster\",size=6).map(plt.scatter, \"x\", \"y\").add_legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, we don't always know ahead of time how many clusters we are going to have.  A common approach is to try increasing the number of clusters in KMeans until you no longer see a significant decrease in the within cluster sum of squares (WCSS), a measure of the overall quality of the resulting cluster assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAFNCAYAAAD4uuNYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxcVZ338c+3u9OdpJslId0sWQgUkVVBiYgibqBEUeAZRXFGQUVxFFFcB3xmHh2VEcdtdFQUUYFRloiMIArCZAAFEQj7LiGBJATIwhYS6KS7f88f93RSaao7XZ3qvrV8369Xv+rWuUude+tWfevec/peRQRmZmblaMq7AmZmVnscHmZmVjaHh5mZlc3hYWZmZXN4mJlZ2RweZmZWtlEND0n3SHrDaL5G0WudLelrY/A6H5B0XdHzkLTbaL/uWJH0ZUm/rNCyNtlW1aCS61ctxmrfH+S1JekXkp6SdFOZ8z4s6dDRqlulSZoh6TlJzXnXpRpsNjxKfTkO9wMYEXtHxDVbUL/B6lR1X0q1SNIbJC3N6bVnpn2rJY/X3xJ5brcq9FrgzcC0iDhgrF98LH8MRMTiiOiIiN5y5x24z0hqlXSxpOslbZ3WIyR9csB8J6fyL1dgFSrKp63MbIMR/KreGXg4ItaMRn1GWx4/XiS1ARcD2wJviYhn06i/AccNmPzYVF51tjg8JE2RdJmkpyU9KenPkprSuA2HpSlZ50o6V9LqdEprdtFyXiHptjTu15IuLHUoLmlP4MfAq9Mh5NNFoydJ+n1axo2SCkXz7SHpqlTHByS9e4h12kbSzyQ9JulRSV/bzIfqbZIWSlop6ZtF698k6Z8lPSJpeVr3bdK4cyR9Ng1PTb8uPp6e75bqqRJ1+0D6tfLdtM0XSnpNKl+SXue4ounbJH1L0mJJT0j6saQJktqBy4Gd0nZ8TtJOabbWId6nPSVdk177HklHFI3bTtKlkp5NpzA2bP8S/pQen06v/eqhtleJ7TDUfreTpN9IWiFp0cBfcwOWc6Ckv6Tl3KGi06ySJis7JbNM2WmZ325muxUv92xJPyy1P6rEUVfaph9Ow2W9x8mUtH+vlnStpJ2Llj3ovp/qeYakP0haA7yxxLrslN7XJyUtkPSRVH48cBYbP4v/Osg2/oik+1Ld7pX0ikG219eKng/8pf5Pyj6Lq9M6HCJpDvBF4D3p9e9I0w76+R2wbZ8Evqzs83atpGeUfYYvHGQ9Nnnf0nv21bS81ZKulDSl1LxFy5gI/A4YBxw+IHRvBiZK2jtNuzcwIZUXL+Ptkm5P+8ZfJL2saNwpkh4q2tb/p2jcoOuZ1usfJT2Y9vUfSi/+/tlERAz5BwSw24CyLwO/TMNfJ/syH5f+DgaUxj0MHFo0zwvA24DmNN9f07hW4BHgU2kZfwesA742SJ0+AFw3oOxs4EngAKAF+BVwQRrXDiwBPpjGvQJYCew9yPJ/C/wkzdcF3AR8tNRrp+1zNTAZmEH2K+HDadyHgAXArkAH2a+N/yoa97s0/PfAQ8CFReMuGWLde9K6NANfAxYDPwTagLcAq4GONP1/AJem+m1FtuN+PY17A7C0xHs72Ps0Lq3PF9N79qb0Wrun8RcAc9N22wd4dOD7VPQ6M9O2aykqG3R7lZi/5H5H9oPoFuD/pTruCiwEDiux704FVqV1bSI7/bIK6Ezjfw9cCExKr/H6wbZbifqdzeD7Y6l1v4aN+0257/HZ6fnr0vjv9W93NrPvp3mfAQ5K22B8iXW5FvgRMB7YD1gBHDLYZ3HAvEen/eCV6f3ZDdi5xPfD2RR93ou3MbB7WoedirZfYeD7Wcbntwc4KW2PCcD5wP/tX3/gtcPZZ9N79hDwkrSca4DTB5n3DWm7XUv2eWwr9Z1K9tn6Rir7d+DUVP7lVPYKYDnwqrRvHJe2Y1vR9t4prct7gDXAjmncoOuZ1usysqOhGamuc4bcx4caWbTQocLjK8AlA6cpsXN8GfifonF7Ac+n4deR7WAqGn8d5YfHWUXP3wbcn4bfA/x5wPQ/Ab5UYtnbA93AhKKy9wJXl3rttH3mFD3/ODAvDc8DPl40bndgPdlOWwCeTm/kj4GPsvHDcg7wmSHW/cGi5y9Nddi+qGwV2YdcaecpFI17NbBo4Ad0wHs72Pt0MPA40FQ0/vw0T3Natz2Kxv3bwPdpsA/i5rZXiflL7ndkH6rFA8pOBX5RYt/9JwaEE/BHsg/kjkAfMGmQL4LhhMdg+2Opdb+GTcNjWO9x0WtdUDSuA+gFprOZfT/Ne+4Q6zE9LWurorKvA2cP9lkssT0/Nci4hxleeOxG9oV5KDCuxP76y6Lnw/n8Dtw/zgXOJGu3Geo93eR9S+/ZPw/47F8xyLxvIPtRtg54Z4nxXyYLiRlkPxTGpcfpbBoeZwBfHTDvA6QfNiWWeztw5ObWM61XcZjMBU4ZansM57RVb1qRYuPIPtQA3yT7tXhlOrw+ZYhlPV40vBYYnw4BdwIejVTrZMkw6ra55Xek4Z2BV6XDvKeVner6B2CHEsvYmWz9Hiua9idkv2AGU1zXR8jWh/T4yIBxLWRfAg8Bz5F9yR9MlvrLJO0OvJ7sF8pgnigafh4gIgaWdQCdwETglqJ1uSKVD2Wo92lJRPQNWKepaZktvHhblGPQ7VVi2sH2u53JTikVv9dfHGQZOwNHD5j2tWTBMR14MiKeKnMdig22Pw7HcN/jfhu2e0Q8R3bUsxPD2/eH+qztRLYdVheV9b/nwzGd7Nf5iEXEAuBksi/Y5ZIuUIlThclwPr8D1/cLZD+0blJ2KvZDZVSvnPd4JXAMcI6kw0pNEBGLyfbrfyP7ATGwrjsDnx3wfk4nfedIOrbolNbTZGcA+k+lbW49y9pfh9NYtJgsce8rKtuF1IiTdqrPphXaG7ha0s0RMW8Yy+73GDBVkooCZKidLgYpH8wS4NqIePMwp+0GpkREzzCXPx24Jw3PAJal4WVkbzZF43rY+MVwLfAuoDUiHpV0LVkD2SSyXwxbaiXZl8zeEfFoifHlbsdlwHRJTUUB0n+qbgXZuk0H7i8aN5hSr7257bVx5kH2O7L3b1FEzBrG+iwhO/L4yMARknYEJkvaNiKeHjC63O02UP957olAf2NpqR8y5ZjePyCpg+w05TKGt+8PtT7LyLbDVkUBMoPsTMFwLGHotq9+a8i2R79NtkdEnAecJ2lrsjD4BvD+EnUfzud3k3ki4nGgvx3ntcD/SPpTCq2KioiLlTWYXyTpiIi4usRk5wI/JzvVONAS4LSIOG3gCGXtXD8FDgFuiIheSbeTBUbF13M4Rx4XAv8saZqyBs1DgXcAF6VKvD01xIjsg9Cb/spxQ5rnE5JaJB1Jdq54ME8A0yS1DnP5lwEvkfR+SePS3yuVNb5vIiIeA64Evq2sC12TpIKk1w+x/M9LmiRpOlm7TX9D1PnApyXtkj7Q/0bWrtG/U18LfIKNjcfXkJ2LvS5G0B2wxLr0ke1M35XUBRsa5/t/9TwBbKdBGqVLuJHsQ/6FtA3fQLYvXJDqezFZA+RESXvx4p4jxVaQnRbatahsc9trgyH2u5uAZ5U1sE6Q1CxpH0mvLFGHXwLvkHRYmm68sobaaWk/uBz4UXpvx0l6XZqv3O22iYhYQfbl+770uh9ieF+wQ3mbpNemz8RXgRvTr9Zh7/uD1HUJ8Bfg62n7vAw4nqwNZzjOAj4naX9ldlNRY36R29M6TJa0A9mRBgCSdpf0pvSl+wLZD6L+z8cTwEylzhIj+fxKOlrStPT0KbJw2eLP32Ai4nyyz/0lkg4qMcmFZO1ac0uM+ynwj5JelbZnu6TDJW1F1sYTZJ8tJH2Q7MiD9Lyi6zmc8PgK2c5zXXrBfwf+ISLuTuNnAf9DdgrmBuBHUeb/dkTEOrJG8uPJ2gHeR7bTdw8yy/+S/dJ/XNLKYSx/NdmbcQzZL6nHyX65tA0yy7Fkja33kq3zRWSnMgZzCVkj7e1kjaw/S+U/B/6LLBwWke34JxXNdy1ZI3Z/eFxH9uvrT1TOP5EdBv9V0rNk79XuABFxP9kX9sJ0mDvYqQDS9OuAI4C3kh3V/Ag4Ni0Hsg9EB9n2PRv4xRDLWgucBlyfXvtANr+9ipXc71KIvYPsdOCiVM+zgBd90acvxiPJTmutIPtV93k2fi7eT3Z69n6yc+4np/nK2m6D+Eh6rVXA3mSfsS1xHvAlstNV+5OdmhrJvl/Ke8nOPiwD/pusveSq4cwYEb8me5/PI2vU/y3ZUdFA/wXcQdYOciUbf4CR6no62Xv5ONkpqC+mcb9Oj6sk3ZqGy/38vhK4UdJzZI3Zn4qIRcNZv5GKiHPIjpx/L+mAAeOej4j/iYjnS8w3n2zf+QHZui0ga8chIu4Fvk32eXiCrK3s+qLZK7qe/b2iqo6kG4EfR8SgX0BmZpaPqvknQUmvl7RDOm11HPAyssZdMzOrMtV0aYjdyc7xdZA1lL8rnb80M7MqU7WnrczMrHpVzWkrMzOrHQ4PMzMrWzW1eYyJKVOmxMyZM/OuhpnVmVtuuWVlRGzu6g11o+HCY+bMmcyfPz/vaphZnZFU7uV4appPW5mZWdkcHmZmVjaHh5mZlc3hYWZmZXN4mJlZ2RweZmZWNoeHmZmVzeFhZmZlyy08JG0r6SJJ90u6T9Kr013ErpL0YHqcVDT9qZIWSHqg6E54pDuU3ZXGfT/dWc7MzEZRnkce3wOuiIg9gH3J7pF+CjAv3X96XnpOuqXpMWR3XJtDdmvQ5rScM4ATyO4sNyuNr5ibH36S/75taSUXaWZW83IJD2U3sX8d6XatEbEuIp4muyXoOWmyc4Cj0vCRZPfJ7k63TVwAHCBpR2DriLghsmvLn1s0T0VcNH8pX7vsvkou0sys5uV15LEr2T2jfyHpNklnSWoHtu+/AVR67ErTTyW7v3S/palsahoeWF4xha52Vq1Zx1Nr1lVysWZmNS2v8GgBXgGcEREvB9aQTlENolQ7RgxRvunM0gmS5kuav2LFirIqWujsAGDhyufKms/MrJ7lFR5LgaURcWN6fhFZmDyRTkWRHpcXTT+9aP5pwLJUPq1E+SYi4syImB0Rszs7y7ticn94PLR8TVnzmZnVs1zCIyIeB5ZI2j0VHQLcC1wKHJfKjgMuScOXAsdIapO0C1nD+E3p1NZqSQemXlbHFs1TEdMmTaC1uYmHVvjIw8ysX5738zgJ+JWkVmAh8EGyMJsr6XhgMXA0QETcI2kuWcD0ACdGRG9azseAs4EJwOXpr2JampuYOWWiw8PMrEhu4RERtwOzS4w6ZJDpTwNOK1E+H9insrXbVKGzg/sfXz2aL2FmVlP8H+bDUOjsYPGTa+nu6d38xGZmDcDhMQyFrnZ6+4LFq9bmXRUzs6rg8BiGDT2u3O5hZgY4PIZl1w3h4e66Zmbg8BiWjrYWdth6PA8t95GHmRk4PIat0NXu01ZmZonDY5gKnR08tGIN2fUXzcwam8NjmAqdHTzX3cPy1d15V8XMLHcOj2HaeI0rn7oyM3N4DFOhqx1wd10zM3B4DNsOW4+nvbXZ3XXNzHB4DJskCl0dPvIwM8PhUZZCZ4fbPMzMcHiUpdDZzrJnXmBNd0/eVTEzy5XDowz9Pa4WrXS7h5k1NodHGQpdvkCimRk4PMqy83YTaZL/18PMzOFRhraWZmZMnujuumbW8BweZcquceUjDzNrbA6PMhW6Oli4cg29fb5Aopk1LodHmQqd7azr6ePRp57PuypmZrlxeJTJt6Q1M3N4lM3hYWbm8CjbpPZWJre3OjzMrKE5PEag0NnOQ8vdXdfMGpfDYwTcXdfMGp3DYwQKnR2sWrOOp9asy7sqZma5cHiMQP9dBReu9NGHmTUmh8cIbLyfuds9zKwxOTxGYNqkibQ2N7ndw8walsNjBJqbxC5T2h0eZtawHB4jVOhq99V1zaxh5RYekh6WdJek2yXNT2WTJV0l6cH0OKlo+lMlLZD0gKTDisr3T8tZIOn7kjQW9S90drD4ybV09/SOxcuZmVWVvI883hgR+0XE7PT8FGBeRMwC5qXnSNoLOAbYG5gD/EhSc5rnDOAEYFb6mzMWFS90dtDbFyxetXYsXs7MrKrkHR4DHQmck4bPAY4qKr8gIrojYhGwADhA0o7A1hFxQ0QEcG7RPKPK17gys0aWZ3gEcKWkWySdkMq2j4jHANJjVyqfCiwpmndpKpuahgeWj7pdO7P/9XC7h5k1opYcX/ugiFgmqQu4StL9Q0xbqh0jhijfdOYsnE4AmDFjxkjq+iLtbS3suM1438/czBpSbkceEbEsPS4H/hs4AHginYoiPS5Pky8FphfNPg1YlsqnlSgf+FpnRsTsiJjd2dlZsXXwNa7MrFHlEh6S2iVt1T8MvAW4G7gUOC5NdhxwSRq+FDhGUpukXcgaxm9Kp7ZWSzow9bI6tmieUVfozLrrZs0tZmaNI6/TVtsD/5161bYA50XEFZJuBuZKOh5YDBwNEBH3SJoL3Av0ACdGRH8f2Y8BZwMTgMvT35godHXwXHcPy1d3s/3W48fqZc3McpdLeETEQmDfEuWrgEMGmec04LQS5fOBfSpdx+HYeI2r5xweZtZQqq2rbk1xd10za1QOjy2w/dZttLc2u7uumTUch8cWkEShyz2uzKzxODy2UKGzw//rYWYNx+GxhQqd7Sx75gXWdPfkXRUzszHj8NhC/Y3mi1a63cPMGofDYwsVutzjyswaj8NjC+283USahNs9zKyhODy2UFtLMzMmT3R3XTNrKA6PCvAFEs2s0Tg8KqDQ1cHClWvo7fMFEs2sMTg8KqDQ2c66nj4efer5vKtiZjYmHB4V4GtcmVmjcXhUgMPDzBqNw6MCJrW3Mrm91eFhZg3D4VEhhc52Hlru7rpm1hgcHhXi7rpm1kgcHhVS6Oxg1Zp1PLVmXd5VMTMbdQ6PCil0tQOwcKWPPsys/jk8KmTj/czd7mFm9c/hUSHTJk2ktbnJ7R5m1hAcHhXS3CR2mdLu8DCzhuDwqKBCV7uvrmtmDcHhUUGFzg4WP7mW7p7evKtiZjaqHB4VVOjsoLcvWLxqbd5VMTMbVQ6PCvI1rsysUTg8KmjXzux/PdzuYWb1zuFRQe1tLey4zXjfz9zM6p7Do8J8jSszawQOjwordGbddSN8S1ozq18OjwordHXwXHcPy1d3510VM7NR4/CosI3XuPKpKzOrXw6PCnN3XTNrBLmFh6RmSbdJuiw9nyzpKkkPpsdJRdOeKmmBpAckHVZUvr+ku9K470tSHutSbPut22hvbXZ3XTOra3keeXwKuK/o+SnAvIiYBcxLz5G0F3AMsDcwB/iRpOY0zxnACcCs9DdnbKo+OEkUutzjyszqWy7hIWkacDhwVlHxkcA5afgc4Kii8gsiojsiFgELgAMk7QhsHRE3RNa16dyieXJV6Oxwm4eZ1bW8jjz+A/gC0FdUtn1EPAaQHrtS+VRgSdF0S1PZ1DQ8sDx3hc52lj3zAmu6e/KuipnZqBjz8JD0dmB5RNwy3FlKlMUQ5aVe8wRJ8yXNX7FixTBfduT6G80XrXS7h5nVpzyOPA4CjpD0MHAB8CZJvwSeSKeiSI/L0/RLgelF808DlqXyaSXKXyQizoyI2RExu7Ozs5LrUlKhyz2uzKy+jXl4RMSpETEtImaSNYT/b0S8D7gUOC5NdhxwSRq+FDhGUpukXcgaxm9Kp7ZWSzow9bI6tmieXO283USa5P/1MLP61ZJ3BYqcDsyVdDywGDgaICLukTQXuBfoAU6MiP67LX0MOBuYAFye/nLX1tLMjMkT3V3XzOpWruEREdcA16ThVcAhg0x3GnBaifL5wD6jV8OR8wUSzaye+T/MR0mhq4OFK9fQ2+cLJJpZ/XF4jJJCZzvrevp49Knn866KmVnFOTxGia9xZWb1zOExShweZlbPHB6jZFJ7K5PbWx0eZlaXHB6jqNDZzkPL3V3XzOqPw2MUubuumdUrh8coKnR2sGrNOp5asy7vqpiZVZTDYxQVutoBWLjSRx9mVl8cHqNo4/3M3e5hZvXF4TGKpk2aSGtLk9s9zKzuODxGUXOT2HVKu8PDzOqOw2OUZT2ufNrKzOqLw2OUFTrbWfzkWrp7ejc/sZlZjXB4jLJCVwe9fcHiVWvzroqZWcU4PEaZr3FlZvXI4THKdpmS/a+H2z3MrJ44PEZZe1sLO20z3vczN7O64vAYA4UuX+PKzOqLw2MM9HfXjfAtac2sPjg8xkChs53nuntYvro776qYmVWEw2MMbLzGlU9dmVl9GFF4SHqlpB2Knh8r6RJJ35c0uXLVqw+FLnfXNbP6MtIjj58A6wAkvQ44HTgXeAY4szJVqx9dW7XR0dbi7rpmVjdaRjhfc0Q8mYbfA5wZEb8BfiPp9spUrX5Iym5J6yMPM6sTIz3yaJbUHzyHAP9bNG6kgVTXCp0dbvMws7ox0vA4H7hW0iXA88CfASTtRnbqygYodHWw7JkXWNPdk3dVzMy22IiOEiLiNEnzgB2BK2PjPzA0ASdVqnL1pNCZXaZk0co17DN1m5xrY2a2ZUYUHpImArdExPr0fHfgbcAjEXFxBetXN4ovkOjwMLNaN9LTVlcAM2HDqaobgF2BEyV9vTJVqy8ztptIc5Pc7mFmdWGk4TEpIh5Mw8cB50fEScBbgbdXpGZ1pq2lmRmTJ7q7rpnVhZGGR/FFmt4EXAUQEeuAvi2tVL1yd10zqxcjDY87JX1L0qeB3YArASRtO5yZJY2XdJOkOyTdI+lfU/lkSVdJejA9Tiqa51RJCyQ9IOmwovL9Jd2Vxn1fkka4TqOu0NnBwpVr6O3zBRLNrLaNNDw+Aqwka/d4S0T032N1L+Bbw5i/G3hTROwL7AfMkXQgcAowLyJmAfPScyTtBRwD7A3MAX4kqTkt6wzgBGBW+pszwnUadYXODtb19PHoU8/nXRUzsy0y0vDoAH4XEZ+KiDuKyp8la0wfUmT6z9+MS38BHAmck8rPAY5Kw0cCF0REd0QsAhYAB0jaEdg6Im5I3YXPLZqn6hS6+u8q6FNXZlbbRhoe/wlMKVE+FfjecBYgqTldymQ5cFVE3AhsHxGPAaTHrqLlLimafWkqm5qGB5ZXpV2n+AKJZlYfRhoeL42IawcWRsQfgZcNZwER0RsR+wHTyI4i9hli8lLtGDFE+aYzSydImi9p/ooVK4ZTvVExqb2V7dpbHR5mVvNGGh7jRjjuRSLiaeAasraKJ9KpKNLj8jTZUmB60WzTgGWpfFqJ8oGvcWZEzI6I2Z2dneVUr+Kya1y5u66Z1baRhseDkt42sFDSW4GFm5tZUmd/zyxJE4BDgfuBS8n+b4T0eEkavhQ4RlKbpF3IGsZvSqe2Vks6MPWyOrZonqpU6HJ3XTOrfSO9Au7JwO8lvRu4JZXNBl7N8P5JcEfgnNRjqgmYGxGXSboBmCvpeGAxcDRARNwjaS5wL9ADnBgRvWlZHwPOBiYAl6e/qlXo7GDVmiU8tWYdk9pb866OmdmIjDQ8DgeOB/YAdk9l1wIfjYgXNjdzRNwJvLxE+SqyS7yXmuc04LQS5fOBodpLqkr/Na4WrnyO/dt900Uzq00jPW01DfgG8O9kRxzrgCeAiRWqV93aeD9zt3uYWe0a6SXZPwcgqZUsPF4DfAj4qaSnI2KvylWxvkydNIHWlia3e5hZTdvSu/5NALYGtkl/y4C7trRS9ay5Sew6xY3mZlbbRno/jzPJLhWyGrgR+AvwnYh4qoJ1q1uFzg7ufezZvKthZjZiI23zmAG0AY8Dj5L9v8XTlapUvSt0trP4ybV09/RufmIzsyo0ovCIiDnAK9l4EcTPAjdLurL/Crk2uEJXB719weJVazc/sZlZFRrpkUf/xQ3vBv5A9r8V1wMF4FMVqlvdKr4lrZlZLRppm8cnyXpYHQSsJwuOG4Cf4wbzzdplSv/Vdd1d18xq00h7W80ELgI+3X8VXBu+9rYWdtpmvO9nbmY1a6T/5/GZSlek0RS6Onzaysxq1ojbPGzLFDo7eGjFGrJ7WJmZ1RaHR04Kne08193D8tXdeVfFzKxsDo+cbLzGlU9dmVntcXjkpNDl7rpmVrscHjnp2qqNjrYWd9c1s5rk8MiJJAqdvkCimdUmh0eOsvuZOzzMrPY4PHJU6Opg2TMvsKa7J++qmJmVxeGRo0JndpmSRSvd7mFmtcXhkSNfINHMapXDI0cztptIc5Pc7mFmNcfhkaO2lmZmTJ7o7rpmVnMcHjlzd10zq0UOj5wVOjtYuHINvX2+QKKZ1Q6HR84KnR2s6+nj0aeez7sqZmbD5vDIWaGr/66CPnVlZrXD4ZGzXae4u66Z1R6HR84mtbeyXXurw8PMaorDowpk17hyd10zqx0OjypQ6HJ3XTOrLQ6PKlDo7GDVmnU8tWZd3lUxMxsWh0cV6L/G1cKVPvows9rg8KgCG+9n7nYPM6sNuYSHpOmSrpZ0n6R7JH0qlU+WdJWkB9PjpKJ5TpW0QNIDkg4rKt9f0l1p3PclKY912hJTJ02gtaXJ7R5mVjPyOvLoAT4bEXsCBwInStoLOAWYFxGzgHnpOWncMcDewBzgR5Ka07LOAE4AZqW/OWO5IpXQ3CR2neJGczOrHbmER0Q8FhG3puHVwH3AVOBI4Jw02TnAUWn4SOCCiOiOiEXAAuAASTsCW0fEDRERwLlF89SUQmeHr65rZjUj9zYPSTOBlwM3AttHxGOQBQzQlSabCiwpmm1pKpuahgeW15xCZzuPrFpDd09v3lUxM9usXMNDUgfwG+DkiHh2qElLlMUQ5QNf5wRJ8yXNX7FixcgqO8oKXR30BTyyam3eVTEz26zcwkPSOLLg+FVEXJyKn0inokiPy1P5UmB60ezTgGWpfFqJ8k1ExJkRMTsiZnd2dlZ2RSpkY48rt3uYWfXLq7eVgJ8B90XEd4pGXQocl4aPAy4pKj9GUpukXcgaxm9Kp7ZWSzowLfPYonlqyi5TfHVdM6sdLTm97kHA+4G7JN2eyr4InA7MlXQ8sBg4GiAi7pE0F7iXrKfWiRHR3zjwMeBsYAJwefqrOe1tLey0zXg3mptZTcglPCLiOkq3VwAcMsg8pwGnlf7Q00sAABEaSURBVCifD+xTudrlp9DV4SMPM6sJufe2so2yq+s+R9br2Mysejk8qkihs50163p54tnuvKtiZjYkh0cV2dDjyqeuzKzKOTyqSKHL4WFmtcHhUUW6tmqjo63F/+thZlXP4VFFJFHobHd3XTOreg6PKpNdINFHHmZW3RweVabQ1cFjz7zAopU++jCz6uXwqDLvfMU0tpkwjs/MvZ2e3r68q2NmVpLDo8rssM14vnbUPty2+GnOuOahvKtjZlaSw6MKvWPfnThi35343rwHuXPp03lXx8zsRRweVeqrR+7DlI42Pn3h7byw3jeIMrPq4vCoUttMHMe3jt6Xh1as4fTL78+7OmZmm3B4VLHXzprCBw+aydl/eZg/P1idd0A0s8bk8Khy/zRnD3br6uDzv76TZ9auz7s6ZmaAw6PqjR/XzHffvR8rn+vmXy65O+/qmJkBDo+a8NJp23DyobO49I5lXHL7o3lXx8zM4VEr/vH1BV4+Y1v+5bd389gzz+ddHTNrcA6PGtHS3MR3370f63uDz//6Tvr6fLdBM8uPw6OGzJzSzr+8fS+uW7CSc294OO/qmFkDc3jUmPceMJ037dHF1y+/nwXLV+ddHTNrUA6PGiOJ09/5Uia2NvPpC+9gvS+eaGY5cHjUoK6txvP1v3spdz36DP8578G8q2NmDcjhUaPm7LMj73zFNH5w9QJuXfxU3tUxswbj8KhhXzpiL3bcZgKfufB21q7rybs6ZtZAHB41bOvx4/j2u/flkSfXctrv78u7OmbWQBweNe7AXbfjhIN35Vc3Lubq+5fnXR0zaxAOjzrwmbe8hD122IrPX3QnT65Zl3d1zKwBODzqQFtLM999z348+/x6vnjxXUT4v8/NbHQ5POrEnjtuzWff8hKuuOdxLr7VF080s9Hl8KgjHz54Vw6YOZkvXXoPS55cm3d1zKyOOTzqSHOT+Pa79wXgc7++wxdPNLNR4/CoM9MnT+RL79iLGxc9yc+uW5R3dcysTuUSHpJ+Lmm5pLuLyiZLukrSg+lxUtG4UyUtkPSApMOKyveXdFca931JGut1qUbv2n8ab9lre775xwe4//Fn866OmdWhvI48zgbmDCg7BZgXEbOAeek5kvYCjgH2TvP8SFJzmucM4ARgVvobuMyGJImv/91L2XpCCydfcDvdPb15V8nM6kwu4RERfwKeHFB8JHBOGj4HOKqo/IKI6I6IRcAC4ABJOwJbR8QNkfVNPbdonoa3XUcb33jny7j/8dV856q/5V0dM6sz1dTmsX1EPAaQHrtS+VRgSdF0S1PZ1DQ8sPxFJJ0gab6k+StWrKh4xavVIXtuz3sPmMGZf1rITYsGZrWZ2chVU3gMplQ7RgxR/uLCiDMjYnZEzO7s7Kxo5ardPx++JzMmT+Qzc29n9Qvr866OmdWJagqPJ9KpKNJj/4WalgLTi6abBixL5dNKlFuR9rYWvvPufVn29PN85Xf35l0dM6sT1RQelwLHpeHjgEuKyo+R1CZpF7KG8ZvSqa3Vkg5MvayOLZrHiuy/82Q+/obd+PUtS/njPY/nXR0zqwN5ddU9H7gB2F3SUknHA6cDb5b0IPDm9JyIuAeYC9wLXAGcGBH93Yc+BpxF1oj+EHD5mK5IDfnkIbPYe6etOfXiu1ixujvv6phZjVOjXURv9uzZMX/+/LyrkYsHn1jN4f95HQfvNoWzjpuN/y3GrHIk3RIRs/Oux1ipptNWNspmbb8Vp8zZg3n3L+eCm5dsfgYzs0E4PBrMB14zk4N2246vXnYvj6xak3d1zKxGOTwaTFOT+Oa79qW5SXz6wtvp6e3Lu0pmVoMcHg1op20n8LWj9uHWxU/zkz8tzLs6ZlaDHB4N6oh9d+Lwl+3Id6/6G3c/+kze1TGzGuPwaFCSOO2ofZjc3srJF97Oc909eVfJzGqIw6OBbTuxlW8dvS8LVzzHkT+4jgXLV+ddJTOrEQ6PBve6l3Tyyw+/imeeX8+RP7ie39/5WN5VMrMa4PAwXlOYwmUnHczuO2zFiefdylcvu5f17oVlZkNweBgAO2wzngtOeDUfeM1MfnbdIv7+p39l+bMv5F0tM6tSDg/boLWliS8fsTffO2Y/7n70WQ7/z+t8HxAzK8nhYS9y5H5T+e2JB7FVWwvv/elfOevPC2m0a6CZ2dAcHlbS7jtsxSWfOIhD9+zia7+/j0+cd5u785rZBg4PG9RW48fx4/ftz6lv3YPL737M3XnNbAOHhw1JEh99fYFfffhAnnl+PUf84Houu9M3bDRrdA4PG5ZXF7bjspMOZs8dt+YT593GV37n7rxmjczhYcO2wzbjOf8jB/KB18zk59e7O69ZI3N4WFncndfMwOFhI+TuvGaNzeFhI9bfnffNe27v7rxmDcbhYVtkq/HjOON9r3B3XrMG4/CwLebuvGaNx+FhFePuvGaNw+FhFeXuvGaNweFhFVeqO++NC1flXS0zqyCHh42aI/ebyiWfyLrz/v1ZN/LTP7k7r1m9cHjYqHrJ9hu78572h/s48bxb3Z3XrA605F0Bq3/93Xl/+ueFfOOKB7j/8es4bO8dGN/SzPhxTYwf10xbS/Y4flwTbeOaNxm3oby/rKWZpiblvVpmDc3hYWNCEie8rsDLpm3LFy66k7P+vJD1vSM/hdXa3ETbuI2BM76lOXve0lwyhNqKHtvGNdHWkoZbmtLzovFFZdl0/ctqoqXZB+tm4PCwMXbgrtvxpy+8EYDevqC7p5cX1vfxwvre9NfHCz3ZcHcq7+4pGr9huH/cpvN39/Sxdl0PT67JllO8jO6e3i0KLIDmJm0MlAFBVBw+relvXHMabu5/LlqbmxnXog1lrc0bpxvXnC1vXHOadkBZqWU2+yjMcuDwsNw0N4mJrS1MbB271+ztC9alIHlhffbY3dNHd/Fwf+ikx+KyDcM9RaFUNO/z63t5+vl1dK/vY31vH+t6+ljXG6xLwbWut4/evsp2GmgSRWGyMbz6w2dDSLU0Z0dsLU0DyjcNrk3KB0zTVhRgLc2ipSkLr5YmZY/N6XFgeXqUHHT1wuFhDaW5SUxobWZCa3NudejtiyxYUrisL3rs7unLQqanOHyGMV0a191TPE9vGp9N98zz67NxPb0bpl9XNP2WHpUNR3MKkXEbwqZ0yGwIn+ZBytO4ocKrpUk0N/ePayrxGtnrbzrvxueFznZ269pq1LdJraqL8JA0B/ge0AycFRGn51wls0FlX6BZ20w16euLFCKbhtaGUBoQOL0R9PYFPX1Bb18fPb3Fz4vK+4Ke3k2f9/aWnm7D894Xz98bwfPre+npHTBtX9DT17fJMtenI7zi6cr1iTfuxucO230UtnR9qPnwkNQM/BB4M7AUuFnSpRFxb741M6stTU1ifBWGWiVEBH1BFjJ9wfreotDpD6cBYbRde1ve1a5qNR8ewAHAgohYCCDpAuBIwOFhZkDW269Z0NxUf8GYl3rodzgVWFL0fGkqMzOzUVIP4VGq+8YmJzglnSBpvqT5K1asGKNqmZnVr3oIj6XA9KLn04BNbiYREWdGxOyImN3Z2TmmlTMzq0f1EB43A7Mk7SKpFTgGuDTnOpmZ1bWabzCPiB5JnwD+SNZV9+cRcU/O1TIzq2s1Hx4AEfEH4A9518PMrFHUw2krMzMbYw4PMzMrm8PDzMzK5vAwM7OyqdHuKS1pBfBI3vXYjCnAyrwrMcrqfR29frWv3HXcOSIa5h/JGi48aoGk+RExO+96jKZ6X0evX+1rhHXcEj5tZWZmZXN4mJlZ2Rwe1enMvCswBup9Hb1+ta8R1nHE3OZhZmZl85GHmZmVzeFRRSRNl3S1pPsk3SPpU3nXaTRIapZ0m6TL8q7LaJC0raSLJN2f3stX512nSpL06bR/3i3pfEnj867TlpL0c0nLJd1dVDZZ0lWSHkyPk/KsY7VxeFSXHuCzEbEncCBwoqS9cq7TaPgUcF/elRhF3wOuiIg9gH2po3WVNBX4JDA7IvYhu5L1MfnWqiLOBuYMKDsFmBcRs4B56bklDo8qEhGPRcStaXg12ZdOXd1SV9I04HDgrLzrMhokbQ28DvgZQESsi4in861VxbUAEyS1ABMZcPO1WhQRfwKeHFB8JHBOGj4HOGpMK1XlHB5VStJM4OXAjfnWpOL+A/gC0Jd3RUbJrsAK4Bfp1NxZktrzrlSlRMSjwLeAxcBjwDMRcWW+tRo120fEY5D9sAO6cq5PVXF4VCFJHcBvgJMj4tm861Mpkt4OLI+IW/KuyyhqAV4BnBERLwfWUEenO9J5/yOBXYCdgHZJ78u3VpYHh0eVkTSOLDh+FREX512fCjsIOELSw8AFwJsk/TLfKlXcUmBpRPQfMV5EFib14lBgUUSsiIj1wMXAa3Ku02h5QtKOAOlxec71qSoOjyoiSWTnyu+LiO/kXZ9Ki4hTI2JaRMwka2T934ioq1+tEfE4sETS7qnoEODeHKtUaYuBAyVNTPvrIdRRh4ABLgWOS8PHAZfkWJeqUxe3oa0jBwHvB+6SdHsq+2K6za7VjpOAX0lqBRYCH8y5PhUTETdKugi4lax34G3UwX9iSzofeAMwRdJS4EvA6cBcSceThebR+dWw+vg/zM3MrGw+bWVmZmVzeJiZWdkcHmZmVjaHh5mZlc3hYWZmZXN42KiTFJK+XfT8c5K+XKFlny3pXZVY1mZe5+h0hdyrR7NekmZK+vvya2g2thweNha6gb+TNCXvihST1FzG5McDH4+IN45WfZKZQFnhUeZ6mFWEw8PGQg/ZP5J9euCIgb/QJT2XHt8g6VpJcyX9TdLpkv5B0k2S7pJUKFrMoZL+nKZ7e5q/WdI3Jd0s6U5JHy1a7tWSzgPuKlGf96bl3y3pG6ns/wGvBX4s6Zsl5vlCmucOSaeXGP9wf3BKmi3pmjT8ekm3p7/bJG1F9o9pB6eyTw93PSS1S/p9qsPdkt4znDfGbKT8H+Y2Vn4I3Cnp38uYZ19gT7JLZS8EzoqIA9JNsk4CTk7TzQReDxSAqyXtBhxLdsXXV0pqA66X1H/11wOAfSJiUfGLSdoJ+AawP/AUcKWkoyLiK5LeBHwuIuYPmOetZJfqflVErJU0uYz1+xxwYkRcny6G+QLZRRQ/FxH9IXjCcNZD0juBZRFxeJpvmzLqYVY2H3nYmEhXBz6X7EZCw3VzusdJN/AQ0P+leRdZYPSbGxF9EfEgWcjsAbwFODZd5uVGYDtgVpr+poHBkbwSuCZd9K8H+BXZvTmGcijwi4hYm9Zz4D0hhnI98B1JnwS2Ta850HDX4y6yI7BvSDo4Ip4pox5mZXN42Fj6D7K2g+L7W/SQ9sN0ob3WonHdRcN9Rc/72PSoeeA1dgIQcFJE7Jf+dim678SaQeqn4a7IgHk2d42fDesIbLhla0ScDnwYmAD8VdIegyx/s+sREX8jO2K6C/h6OtVmNmocHjZm0q/yuWQB0u9hsi89yO4TMW4Eiz5aUlNqB9kVeAD4I/CxdIl7JL1kGDdluhF4vaQpqRH6vcC1m5nnSuBDkiam1yl12uphNq7jO/sLJRUi4q6I+AYwn+yIaTWwVdG8w1qPdMptbUT8kuxmTfV0GXirQm7zsLH2beATRc9/Clwi6Say+0QPdlQwlAfIvuS3B/4xIl6QdBbZqa1b0xHNCjZzG9GIeEzSqcDVZL/4/xARQ16GOyKukLQfMF/SOuAPwBcHTPavwM8kfZFN7wx5sqQ3Ar1kl22/nOyoqkfSHWT31f7eMNfjpcA3JfUB64GPDVVvsy3lq+qamVnZfNrKzMzK5vAwM7OyOTzMzKxsDg8zMyubw8PMzMrm8DAzs7I5PMzMrGwODzMzK9v/B4XS3SD5r040AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Find the optimum number of clusters for k-means classification\n",
    "n_cluster_vals = np.arange(1,11) # user specified, you can experiment...\n",
    "wcss = [] #Will hold within cluster sum of squares for each n_clusters\n",
    "\n",
    "for i in n_cluster_vals:\n",
    "    kmeans = KMeans(n_clusters = i, init = 'k-means++', max_iter = 300, \n",
    "                    n_init = 10, random_state = 0)\n",
    "    kmeans.fit(X)\n",
    "    wcss.append(kmeans.inertia_)\n",
    "    \n",
    "#Plotting the results onto a line graph, allowing us to observe 'The elbow'\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(n_cluster_vals, wcss)\n",
    "plt.title('Using the elbow method to select number of clusters in KMeasn')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('WCSS') #within cluster sum of squares\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach is called the \"elbow method\" because we see an elbow at the point where the WCSS measure of quality of fit no longer decreases with each additional cluster.\n",
    "\n",
    "Another common clustering method is Agglomerative Clustering, where every point starts as its own cluster, then points that are most similar are grouped together until some stopping criteria is me. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x20a02706a00>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUZfbA8e+5U1NJIAm9ixQBESkqgqiIipW1gQXLuqxtdfdnd3ctu7pr37WtimXXLnZREbAiKkiTXqT3TkidTH1/f0yIKTOTOklIzud58pDce+fOGQj33PuW84oxBqWUUs2X1dABKKWUaliaCJRSqpnTRKCUUs2cJgKllGrmNBEopVQzZ2/oAGoiIyPDdOnSpaHDUEqpQ8qCBQv2GmMyy28/JBNBly5dmD9/fkOHoZRShxQR2RRpuzYNKaVUM6eJQCmlmjlNBEop1cwdkn0ESqlDUyAYYtnGHRT5AvRon0Gr1KSGDkkR50QgIj2ByaU2dQPuNsb8u9QxI4GPgQ3Fmz4wxvwtnnEpperfh98v5YkPvycYCiECPn+QYX27cu+E0aQkuBo6vGYtronAGLMaGAAgIjZgG/BhhENnGWPOjGcsSqmG896sJTz+3kyKfIEy279ftp7fPjqZN+66BIfN1kDRqfrsIzgZWGeMiTh8SSnVNPkDQZ78cFaFJBDeF2L7vly+XbSuASJTB9VnIhgHvBVl37EislhEPheRI+oxJqVUnC1cu41Y1e4LvX4++nFZ/QWkKqiXRCAiTuBs4N0IuxcCnY0xRwJPAR9FOcdEEZkvIvP37NkTv2CVUnXK4/VXekyBx1cPkaho6uuJ4HRgoTFmV/kdxphcY0x+8fdTAYeIZEQ4bpIxZpAxZlBmZoUZ0kqpRqpH+wz8wWDU/Q6bRf9ubesxIlVefSWC8URpFhKRNiIixd8PKY5pXz3FpZSKs/YZLejftS02SyLutyyLi0YOqOeoVGlxn0cgIonAKcDvS227BsAY8xxwPnCtiAQADzDO6PqZSjUp//ztGC5/+G2y8zx4fOGmIrvNwmZZ3H3pKbTPaFFvsWzcuZ/3vlvChl37adcqlfOG96dXx6x6e//GSA7Fa+6gQYOMFp1T6tDi8fmZPm81H/+4jEKvnwGHteeSk46iU1Z6vcUw6bM5/Hf6PIKhEIFgCMsSnDYbZwztzV0Xn0xx40STJSILjDGDym/XmcVKqXqR4HRw7rC+nDusb4O8/3dL1/O/GfPw+n8dxhoKGYpCAT6bu5KeHbM4f0T/BomtoWkiaECb8vfwwrqvmLlrJQETpFtyFld1P5GT2/SrcGzIhJi3bx0bC/bQwpHI8KzeJNl1NqZSVfXCZ3MizmUAKPIFePHznzhveL8m/1QQiSaCBrIiZyvXzn0Rb9BPiHDz3Jq8ndy39D2WH9jKjb1OL3PsLQtfozDgI2CC2MXigWUfcsPhp3JRl+Ma6iModUhZvTX2sPP9eYXke7ykJLrrKaLGQ6uPNgBjDHctegtP0FeSBA4qCvp5d/McVuduB2CHJ5vr5r7IXm8ehUEvvlCAwqAPb8jPM79MZ9r2nxviIyh1yLHbYl/ujDHY7c2zzIUmggawImcr2b6CqPv9oQDvbpoDwJsbv8cXijwGuyjk55lfZnAodvgrVd9G9OuGFaPZp2+XNiQ4HfUYUeOhiaABbPdkYxH9FzKEYVNB+DH2653LCZjok3FyfIVs8+yv8xiVamomnnEMLkfk1nCXw86NY4fXc0SNh/YRNIBMdyqG6HfxgtA2ITykLmhCMc8lIgRCQQ74Cvls20LW5O2glSuZM9oPpFty6zqNW6lI1u/Yx+RvF7Fm214y05I5f3h/Bh3eodF1unZr24pnbhzLHS9OJd/jxRiwLMFus7jv8lM56rD2DR1ig9FEEEVhwMumgr24bHa6JmXV6S91/7ROMffbLYvzOw0FYGDLrny9c1mFvoSDbGLx9sYf+HDrfChOLxbCO5tmM7rtkfy571gs0Qc/FR8vT5vLC1N/IhAMEgwZBPh+2QYGH96BR39/dqXt8qWt37GPFz6bw8yl6wmGDL06ZjFxzFCG9e1aZ/EO6N6eqQ9czc9rt7F9Xw4ZLZIZ3LNjSZzGGKbPX83L0+exeVc2Loed04f04qrThpCVllxncTQ2OqGsHG/Qz79XTeXTbQuxi0WQECn2BG7qdTqj2x5Zo3OGTIif9q5lcfZGHDY7vVPbc+vC1/FHafJxiI1vR92Dw2Znde52rp7zPN5QxcJdLsuBXYSCYOSCXQ7Lxu+6n8wV3UeW2Z7n9/DptgV8t3sVFsKoNv04rd0AEuzOGn0+1Tz9tGozf3r244hDMt0OO5ePHsTvzzy2SudatG4b1z35AT5/kFCpa5Lbaefq04dy1WlD6izuaIwx/O31L5g+f3WZz2S3WSS6nLx6+3g6ZaXFPY54ijahTBNBKSET4rp5L7H8wBa8obK/3C7Lwa19zuLsDhX+DmPaVrif6+a9xAFfAZ6gDxsCIjGbfBJtLv4xYBzHZfYEYPr2Rdy/7AMAvKEAFoLDspNkd7Hflx/z/V2Wg29G3Y3dCo+GWJ27nWvnvkggFKSoOLkk2Jwk2By8MPQaOia1qtbnU83XxH+9y/xftkbdn5zg4qtHfl/pgjOhkOG0OyexN7cw4n6nw8Y7f5kQ94vwT6s283/PTikpgVGaJUK/rm34763j4hpDvEVLBNpmUMrcfetYmbOtQhIA8IbCTwr+CPui8YUC/O6n59npOYCn+K49iKm03T9oQuzwHCj5+dR2A5gy8jau7n4yo9v056LOx/HQUReTHyiqNAZvyM+avJ0l8dww72XyA0UlSQDAE/SR7Svkxvn/1RFITcjKzbu486WpnPHnF/nNfa/w3+nzyC2o/HemqlZt2R1zfyAYZM+B6KPjDpr/yxYKY5SqDoUM789aUu34quuNrxZETAIAIWNYtWUP2/flxD2OhqB9BKV8vGVeyQU7EmMM8/et59jMw6t0vq93LqMg4I3ZMRyJTSyy3KlltqU7k7m8+wklP3+zcxkOy4avColp/r519G7Rnm92Lo+ayAyG/b585u9fz+BW3asVr2p83pu1hMffnYkv8GtTy6TPZvPGVwt59fbxtGuVGvF12/bmsCs7jzYtU2jXKnYhOGclY+6DIYM7wiid/bmF5Bf5yEpLxu20s31/bpnmoPICwRAbd8Z/ZNzm3bEv8g67jR378ir9ezkUaSIo5YC/8ruXvIAn6j5fKMCs3SvZ6TlApjuVr3Yui5lYorFEODYjdrJx26rfnr84eyOFMeLxBQOsyNlaJ4nAGNPoRo00F5t3Z/PYuzPL1NQB8PqD+IMebn/xM167fXyZfWu37eXe12awbvs+HHYLfyBEj/YZ3DthNN3aRm4uPHVQL979bjGBYOQn3PTkhDJ32Es37OCRd75l9dY92G0WxhjOOqYPQ3p1wmZFb5ywRGgTJXHVpay0JDbvzo66PxAM0jI1Me5xNARNBKX0Sm3P4uzNUcftB00o6pDMWbtXcvfidzAYfKEgTstGUbDylZnKc1sO7ul3fkmbfjQDW3atUjOOQ2x0KG73T7C7ECTqE4rNsnBbNZ9Qc8BXwH/Xfcsn2xaQHyiilTOZizofx/guw3DZmudEnYbw9reLCIYiX5xDIcPabXvZsHM/Xdu0BGDz7gNc+ehkCot8GOBgK83yjTu5/OG3eeuuS+iQWbF9fsIpR/PJnBUUeLwRf6Oy8wo5775XOL5vVy4aOYCb/vNRSSesPxD+P/bxj8tZsmEHse4ZHHYb5x1fsf5WXbto5ACWb9oVdUW19q1alPydNTXaR1DK+Z2OwRblN1IQOiZlcFhKmwr7lh/Ywp8XvU1B0EthMFwPqDBC+Yjy7GLRzp2OhWAhHJXehScHX8kJrftUGqvL5uD3PU7BZcXO5W6bg+GZvQAY1aZfzOONMYxo3bvS945kvzefS354inc3zynpu9jny+eldd9w7dwXq9SEperGik27ot6lQ3gUzPodv6799J8pP+Dx+iv8thrCy0w+9+nsiOdpnZ7C/269iC5tWuJ22im/7ow/GMIXCPLD8g1RRxf5AkE27zrAWcccEXGyl9tp56xj+nB4h0y27DnAP978ilPvmMToOyZxzyvTy3yO2hp5ZHf6dW0TNY57Joyus/dqbDQRlNI+sSW39jkbl+UoM/PXZdlp4UjgoaMujvi6SWu/LNP5WhUuy87wrN58NPJWZp5yL9+P/hvPD53IgPQuVT7H+C7DuOHw03BKxacHIfx08c+jLi55uujdoj1HpneJmgyCJsSLa7+uUid0eU+snsp+X36FpylvyM/avJ18uHlutc+paiY9OaHSY1ISwpVrjTF8vWht1Db6kDF8sXBNydOnMeEnikXrtpGdV0i3tq14/57LufWCkdijjA7y+oMxO4M9Pj9L1m/nqRvOpW+XNtgsC4fNIistmT/9ZgR3jj+JBWu2Mu7+1/nwh6XsySlgb04BU+eu5NIH32TW0vWVft6qsFkWT10/lstHDyI10YXTbsNusxjaqxP/veUi+napeBPYVGjTUDlndxhEr9R2vLZhFouzN+Gy2RnT7ijGdhxKmjNy++CcvWsqPW+izUlh8fBRu2VnQHoX7ut/AUC1mk2WZG/mf+u/ZXH2RgCy3C04sfURuO0ufsndzsaCPdjEYlhmT67oNpLuKWWbsh4ZeCkPLPuAr3Yuq3DRDmGYtn0Ry3O28L9jr8ddxbi8QT9f7VwWdTRUUcjP25t+0Eqp9WTs8f2Ytzr6SBybJQzs0QGAQCgUtRnpoEBxh/OcFZv4x1tfcSDPg80m+PxBjjuiC3dfegprt+8tae6piex8D4MO78irt4+nsMhHIBgiJdGFiOAPBLn5uYrDOoMhQ9AX4PYXP+OLh35Pkrv282AcdhvXnHksvxszlNxCL26nvVnUH9JEEMHhqe34+5EXVenYWbtXVWlMkMtyFDcXwYC0ztzS+8xqd/h+sPkn/r1qapmnj7z8Itbl78JtOUhzJvHWsJtolxh5xadAKMj3e1bhDwVwWDYCERYU95sg2z3ZTNu+iHM7Dq5SXDn+QiRG7SSAvd7Y8x1U3Rl2RBcOa5/Bqi278fnL/hu7nXZuveDEkpm0DpuNzBbJ7D4Q/d+ndXoKc1Zu5pbnP6nQAT1r6XomPPwWg3p0rObYuLJKt70nlrugz1yyLmZTlyBMn7+a39RhP4LNsqr0ZNVUxL1pSEQ2ishSEVkkIhVmgUnYkyKyVkSWiMjAeMdUl/7zy/RKjxEgu3hEksEwd/86Jsx+pqTUdFXs8hzg8VWfRW2CKgr52V2UwzVzX4h4Z763KJfzZz3O/Uvf5+tdy2OOZioK+nlv85wqx5bqSCRE7LvKdGdSlc+nasdmWTx703mcMaQ3LoedJLeTBKeDrLRk7ptwKmccU7Yf6LJRR+N2Rr4ndDvtTDjlaB586+sKSQDCd+Vb9+SwcO3WqOcAsISolT8TnA4uHXV01Ndu3JkddXw/hJuW1m7bG3W/qlx9PRGcaIyJ9i91OtCj+Gso8Gzxn41evr+opEpoLBU74QyeoI97Fr/D28P/WKX3+nDL3ErnI4Qw5PoL+XHPaoZnlf3P/n8LX2WX5wDBKt635fojz/KMxG1zcEJWH77ZuSzi+V2WnQs7V63UgKqeH1ds5MWpP7Fy827sNouRR3bn6tOH0rl1On+99BT+7/wT2LRrPy6HnW5tW0Uc0jvuxAHMW72Feb9sKTNiJsHl4JhenTj68A489dH3MePYvPtA1JE/Nkvo0iad3AIvOQVF+Eo1ISU47Zx9bB+G9OwY9dwtkty4HPaoq4s5bBYtU5rmsM760hiahs4BXjXh3qg5IpImIm2NMTsaOrDKBE2o0iaRWLZ7slmfv6tKVULX5O3EH2VdgtIKgz5+2ru2TCJYlbONjfl7qpwEALokZVX5WIA/9hrDgv3ryfV7yjyROC07nZIySoroqbrzv+nzmDT11+UXvX6YNm8VXy9ay7M3nkf/bm1Jcjvp0zl2J6fNsnj8mrP5bul63vrmZ3buz6Vtq1TGn3gUw/t2Y9nGnTHH+ZdmWYIFBELh37VEl4PMFsk8e+P5OOw2Jn+7iI9+WEaB10e3Ni25YvRgRvTvFnPOyaiBPXjsvZlR94sIpw/pVaX4VGT1kQgMMENEDPC8MWZSuf3tgS2lft5avK1MIhCRicBEgE6dYlfvjJd93jymbJ3PurxdZLhTOLPd0WS6U9nuiT4JJRa7ZWN3UW6VEkGGKxWh4tNFeUJ4ZnJpy3K2VGt2s9vm4JKux1f5eAh3Wr9+3B94fs2XTNuxiKAJkWBzcl7HoVzV/cQaTYBT0W3Zc4DnP5tTobkmGDJ4vH5ue+FTPv/H1VWe1GdZwsgjuzPyyIqTCTtlpZW5i4/GmPCoooO3AQIM79eVv19xWsmIoolnHMPEM46pUkwHpackctVpQ/jfjHkVngrcTjvnHteX9hlNb7ZvfaqPRDDMGLNdRLKAL0RklTHmu1L7I/2mVrhqFSeQSRAuOhefUKP7bNsCHlz+MRAu/GZDeH/zXPq0aM9+b37EtnsLiTmXIBAKkuVOpSDgZdbuleT4C+malMWgVt0qlI4+p+MgPto6j8pSgdvmZES5ZiG72Kr15DI8szdDWh1W5eMPynSn8pd+v+GuvufiCwVwWQ6dXRwn7323JOZon3yPl4VrtnH04R1q/V5pyQmM6NeNmUvW4Y/RaVueAb5YuIYju7dj3MijahXD78YMpVVqIs9+MptCrw8RwWm3cdVpQ7j05EOqW7FRinsiMMZsL/5zt4h8CAwBSieCrUDpBsIOQNV7UevBipytPLR8SplidEEMwZCflbnb6JfWicUHNmGMwW+CuCw7Brig41A+2Bq9flHbhHR+2ruW//wyA5sIARPCLhZJdjePDryM3i1+XSijR0obKn8egA6JLRnYsmz99mGZPXl05SdV+qxy8KsWF3BLLH0CiLP1O/bFHEljgK17c+okEQD85ZJRrNm2l817DlSrMGEoZHjyg1lcMOLIKjcvRSIinDe8P2OH9WPr3gOEQoYOmWnVWu9ARRfXRCAiSYBljMkr/n408Ldyh00BbhCRtwl3Euc0tv6B/677JmJFUgiPsPklbweTj/8j07YvYldRDp2TMjij/UBaOBLZ7c1j1u6VZZ4YBMFtc3BK2748+8uMMmsN+Ai3818790UmH38TrRPCU/v9oSCWWJVWLn1mcMXmgEx3Kqe3G8C07YsjrmtQmgGW50QvLawahzYtU7AsIRSKfFG2RMiow7o4qUlu3vrzpTzyzjd89OMyqlOktsgf5Juf1zLq6KoVa4ymsMjHW9/+zDszF5OTX0RGiyQuPvEozh/RH2eUJShV1cT7b6818GHxhckOvGmMmSYi1wAYY54DpgJjgLVAIXBlnGOqtp+zN8ZsYz94cf3tYSdV2Pf3Iy/kgy1zeXX9d+wqysESYURWb35/2ChumP9y1OGg/lCAtzf9yE29xgDhNQNaOVPY7Y1eIbFHSpuok96GZfRk5q4VlSaC8Hs1/Qk0h7rfHN+Pz35aGXUkjWUJQ3pX3pe2eXc2r8yYz8wl6wkZw+CeHbnq1MH07FhxsIDbaeeui09m+75cFq3bhtdf9QlkH89eXqtEkO/xcvnDb7N9X07J+27fl8vTH//AjIW/MOmP52syqIW4/s0ZY9YDFZb1Kk4AB783wPXxjKP4fdjuycYXCtA+sSXOSmr0lFa+87W8kDFRi8RZYnF+p2M4v9MxBEJBbGIhImwq2BuzlIPfBPly59KSRCAiXNH9BJ5c9XnE5OG2Obiq+4kRz/W/dd/y8rpvqlQGw205OKuai++o+te7U2tOPbon0xesrpAMXA4791w2utIFYRas2cofnv4QfyC8zCTAVwvX8N2S9fz9ilMZNbDihdtmWTx5w7k8Mvkb3p+1tMpDEJZv2lXFIyN77tPZbN2bU2H2cpE/wC9b9jB55iIuG6W/tzXVLBrYZu1eyXnfPca475/gytn/YfRX9/P06mkEqjAcE2Bk1hExk0GWK5UsV+Vlcu2WraTZJhAKlqlnFEn5ZqDzOg5lVNv+uG2Oks5fC8FlOTiv41BOat23wjm2F2bz0rqvq5QEbGKR6kjgrA7RJ/eoxuPuy07hD+ccT0ZqUkldnD6dWvPEdedw0oDYnf3+YJCbn/uEIl+gJAlA+KbG6w/w1/9NJ68w8o2Kw2bj8tGDq3UHnldYVONFj4KhEB/9sCxqCYsif4A3vvq5RudWYU3+WSq8zOOHFZpEJm+azdq8nfzr6Msr7Ri9rNtwpu1YFLHT12U5uP7wU6vdudopKfLknoMshEEtyw7lExH+2vc3jO04mPc3/8R2TzadEjO4oPMx9ExtF/E8H2+dF3PRDwg/BYQwHNGiA38/8iKS7e5qfRbVMESE8ScdxUUjB5BT4MFut5UUk6vM90s3RCwx8uu5YercVVw0ckDE/e0zWtC9bStWbt5VpacCQdiwc3/UtQ1iKSzyVTp8dW9u5WuJqOiadCIIhII8tGJKxHZxb8jPz9kbWbh/A0e36hbzPB0SW/HUoKu47efXKQr6CZggdrERMiFu6jWGk9tWv8aJw7IzvvMwXt8wK+LdutOyc1m34RW2iwj90jrRL61qcym2FO7DH2V9BQivVzCm3VFc1m0E7RObZq31ps6yhPRqzqzdtCs7YsmIg4p8AdZUUrbh3gmjueLRtynyBiq92XA6bDHfL5YElzNqeYqDqpoAVWRNOhHM27cu5uNoUdDHh1vnVZoIAPqnd+KzE+9g7t61bC7cS7ozieGZvUmw13yY5G8PO4mthfv5Zld4CckQpqRE9N39zqNHStsan/ugdgnp2MUiEGW0kcOyc0zm4ZoEmpm05AScdnvUGj52m0WrFrGTy2HtM3jzzkt4ZsqPfLnwl5gjiULG0KV1zX7H7DaLU44+nGnzVpVpxjrIWU8L1zRlTToR5PgLY472McB+b16Vz2cTi2MzD+dYajcMrvT5/nbkhazJ28GnWxeyz5fH4SntOLvD0aTVUZG2czoM5u1NP0KURCACw6q4BrNqOk4acBgPTf466n6bZXHm0MoXSOqUlc5DV5/Bis2D+O0jk/FGaMJxOeycc9wRJLhqPhrtD+cez6yl68nzeMskHLvNolVqIhNO0Y7i2mjSncVdkjJjzux1iK1O7rprq0dKW/7U+wzuP3IcE7qNqLMkANAxqRWXdDk+4toCLsvBPf0uwFGNEVSqaUhNcnPtWcdFrBjqdto5d9gRdIywPGU0fTq15vZxJ+Fy2MqMVkpwOTiyW1v+9JsRNY51X24Bd740lSJfoMyMe5slnDGkN2/ceQmpSdqvVRtN+grQq0V72rjT2FSwJ2I6sES4oHP16p4ciq49fDTdklvzwtqv2FoYXtqvf3onru0xmqPKzUJWzceEUwbRKiWJp6d8z4H8IkTCJaGvOm0IF59U/ZIQ5w7ry5BeHXn3uyWs2LSL9OQExh7fjyE9O9Z4prrPH+DKRyezc39ehZnUDruNDplppDWjdQPiRWo6pKshDRo0yMyfX2Fpg4g25O/m6jnPUxTylVTvFASXZecPPU/jgjoojxwyIZYd2MI+Xz7tE1pyeGrDP2VE4w36sYkVdd6Dan6MMezYn0coFKJtq9RalYKoa5/PXcX9b34ZdUH5BJeDrx6+JuZaCOpXIrLAGFOhHa3J/+11Tc5i8vE3MXnTbKbtWIQvFKBfWicmdD2B/um1r2L609413Lf0PQoDXkSEoAnR1p3OAwPGRVzovqFVZ1lM1TyICO1aVT4PpiF8+tOKqEkAwk/1C9ZsZdgRXeovqCaoyScCgAx3Ktf3PJXre55ap+ddnL2JWxa+XmF46oaC3fzup+d5c9iNtE2IvGykUs2BMYbNuw9QUOSjY1ZatYd5xkoCB9V0WKr6VbNIBPHyxKqpUWv3FAX9vLJ+JncccW49R6VU4/Djio089PY37DmQj2WFF6Ef0L09t100ku7tMqK+zhjDwrXbeO+7xWzfn4slEnWegj8QpE+nytfzULFpIqihfH8Rq3K3Rd0fNCG+2LFUE4Fqlr5ftoFbJ31a4W593i9buPDvrzG4Zyf++dvTK0yEC4ZC3PnSVH5YtpEinz/mrGWHzWJwz460aZkSh0/QvDSeXqFDTFHIX2HxmPL8UUpXK9UUGWOYuXgdv31sMn/8z8dRm2wMMP+XLUx46K0KE9pe/3IB3y/bgKeSJJDoctCpdToPXHl63X2AZkyfCGoo3ZlEgs2JL8bFvjF2FisVD8YY7n5lOl//vDbqbOXSQsawP6+QaXNXMbZ4VnAoZHj1iwVRS2sDJLmdDO7ZkbOO6cPwft10YZo6on+LNWQTi0u6DMNtRR6F47Y5uDJKWWilmpoZC36pchI4yOML8NGPy0p+zinwUFAUeTW/g/yBII9fczYnDjhMk0Ad0r/JWris2wkck9GDhFLLMh4sCz2+8zCGZ/VqwOiUqj+vzJhfrSRwUOkLv9Nhr7x4nV3nv8SDNg3Vgk0sHjrqEhbsX897m39id1EOXZIzuajzcVHLQivVFG3enV3t19gsoX/XX/+fJLmdHNG5NYvXR16p1mZJxMVyVO3Fe83ijsCrQBsgBEwyxjxR7piRwMfAhuJNHxhjyq9r3GiJCINadWdQq+6VH6xUE5XkdlJYhTH/pdltNi45uWwpiz/+ZgTXPPF+xI5ml8POVacNrlWcKrJ4Nw0FgJuNMb2BY4DrRSRSScNZxpgBxV+HTBI41KzL28knWxfwxY4l5Pk9DR2OakLOPa5vlZttbJbgcti57cKK8wmO7N6Ox685m4zUJBJdDpLcThKcdjplpfHSzRfSoRqF8FTVxXvN4h3AjuLv80RkJdAeWBHP91Vl7SnK5dafX2dd3i4sCS9yGTAhLu4yjGt7jK5xQTClDrr45IF8PHs5+3ILKqwZ4Hba+d2YocxesYl8j5d+Xdsy/sSj6NIm8voEx/bpzLR//o7F67ezP7eQdhmp9OqYpb+ncVRvRedEpAvwHdDXGJNbavtI4H1gK7AduMUYszzC6ycCEwE6dep09KZNm+IfdBPgDfq5YNa/2OPNrbAGstvm4NIuw5nYY1QDRS0Xn+0AACAASURBVKeakr05Bdz/5pfMWbEJh92GPxikR7sM7rr4ZHrr7N9GIVrRuXpJBCKSDMwEHjDGfFBuXyoQMsbki8gY4AljTI9Y56tO9dHm7rNtC3l4xZSI6y1DeL3i6Sf9uVYrrSlV2oF8Dzuz80hLStBZv41MtEQQ9+GjIuIgfMf/RvkkAGCMyTXG5Bd/PxVwiEj0QiSqWj7fvihqEoDwSlQLszdE3a9UdaUlJ9CrY5YmgUNIXBOBhBv1XgJWGmMej3JMm+LjEJEhxTHti2dczUmsmc8AmCoco5Rq0uI9j2AYcBmwVEQWFW+7C+gEYIx5DjgfuFZEAoAHGGcOxdVyGqkhrbqzMmcr3igXe78JckSLDvUclVKqMYn3qKHvgZhd/caYp4Gn4xlHcza24xBe2zAr4j6H2DgmowdZ7hb1HJVSqjHREhNNXCtXCo8NvIwEm7NMXaQEm5PuKa25t/8FDRidUqox0BITzcCgVt2ZcsJtfLJtAT/v30Ci3cWYdkcxNOOwSktpK6WaPk0EzUQLZyKXdh3OpV2HN3QoSqlGRm8HlVKqmdNEoJRSzZwmAqWUauY0ESilVDOniUAppZo5TQRKKdXMaSJQSqlmThOBUko1c5oIlFKqmdNEoJRSzZwmAqWUauY0ESilVDOniUAppZo5TQRKKdXM1cfi9aeJyGoRWSsid0TYLyLyZPH+JSIyMN4xKaWU+lW8F6+3Ac8ApwN9gPEi0qfcYacDPYq/JgLPxjMmpZRSZcX7iWAIsNYYs94Y4wPeBs4pd8w5wKsmbA6QJiJt4xyXUkqpYvFOBO2BLaV+3lq8rbrHKKWUipN4JwKJsM3U4BhEZKKIzBeR+Xv27KmT4JRSSsU/EWwFOpb6uQOwvQbHYIyZZIwZZIwZlJmZWeeBKqVUcxXvRDAP6CEiXUXECYwDppQ7ZgowoXj00DFAjjFmR5zjUkopVcwez5MbYwIicgMwHbABLxtjlovINcX7nwOmAmOAtUAhcGU8Y1JKKVVWXBMBgDFmKuGLfeltz5X63gDXxzsOpZRSkenMYqWUauY0ESilVDOniUAppZo5TQRKKdXMaSJQSqlmThOBUko1c5oIlFKqmdNEoJRSzZwmAqWUauY0ESilVDOniUAppZo5TQRKKdXMaSJQSqlmThOBUko1c5oIlFKqmdNEoJRSzZwmAqWUauY0ESilVDMXt6UqReQR4CzAB6wDrjTGHIhw3EYgDwgCAWPMoHjFpJRSqqJ4PhF8AfQ1xvQHfgHujHHsicaYAZoElFKq/sUtERhjZhhjAsU/zgE6xOu9lFJK1Vx99RFcBXweZZ8BZojIAhGZGO0EIjJRROaLyPw9e/bEJUillGqOatVHICJfAm0i7PqzMebj4mP+DASAN6KcZpgxZruIZAFfiMgqY8x35Q8yxkwCJgEMGjTI1CZupZoDYwzfvTeHdx75mG1rd5CSlsyZ15zCWdeeSmJKQkOHpxoRMSZ+11QRuRy4BjjZGFNYhePvBfKNMY/GOm7QoEFm/vz5dROkUk2QMYYHL3uSHz+eR1GBt2S7M8FJq7bpPD33n6S2TGnACFVDEJEFkfpi49Y0JCKnAbcDZ0dLAiKSJCIpB78HRgPL4hWTUs3FrPfnVEgCAD6Pjz1b9vLMjf9toMhUYxTPPoKngRTCzT2LROQ5ABFpJyJTi49pDXwvIouBucBnxphpcYxJqWbh3UenVEgCBwX8Qb7/YA6FeZ56jko1VnGbR2CMOSzK9u3AmOLv1wNHxisGpZqr7et2xtxvc9jYu20/nXq1r6eIVGOmM4uVaoKS05Nj7g/4gqS0jH2Maj40ESjVBJ11zWhcCc6o+3sO7k56Vot6jEg1ZpoIlGqCzvj9KWR2ysDhrNj6605284enr26AqFRjpYlAqSYoIcnNU7P/wUmXDMeZ4MSd7MbutNP/hD488f39dOvfuaFDVI1IXOcRxIvOI1BNndfj5ZNnZ/DR059zYFcOaa1bcO4Np3P2dafidEdv8onEU1DE/h3ZpKQnk9pK5w40Z9HmEWgiUKqRKSr08n8j/srmVdvwFvpKtrsSnHTq3Z5/zfo7rgRXvcSSf6CAPVv3kZaZSnrrtHp5TxU/0RJB3IaPKqVqZvLDH7FpxVZ8Rf4y270eH5tWbGPywx8z4Z4L4xpD9q4DPHn9i/z02ULsTjsBX4DDj+7GTc/+jq79tFmpqdE+AqUamSnPTKuQBA7yFfn4+Olo9RvrRl52PtcNvp3ZU+bj9/rx5Hnwe/0s/3E1Nw37CxuXb4nr+6v6p4lAqUYk4A+Qtz8/5jF5+/MJ+AMxj6mND574jJw9eQQDwQr7PPlFPHfzK3F7b9UwNBEo1YjY7DYclXQGOxNc2Oy2uMXw+Qtf4fdGfiIBWPztcgpyK60hqQ4hmgiUakSMMbRsE32il81uceoVIxGRuMVQ2UXeZrdRmKt1ipoSTQRKNSJfvvYd2btyou63O+xc+tfz4xpD226tY+4XgRaZqXGNQdUvTQRKxdmO9bt46oYXGdfx91zUfiIPX/F01A7Xdx+dUmbIaCRJaUnxCLPEhbeegzsp8vBUh8vBqVeeiNPliGsMqn5pIlAqjpZ9v5KJR97CZy98yb5t+9m/I5uv3pjFDUPvYPYnFefC7NiwO/YJBXL25MYp2rCTLj6eoWcMrJAMXIkuOvZsx2//cXFc31/VP51HoFScBPwB7hn7MEUFRWW2h4IhvIU+Hhj/byZvn0RSamLJvuS0RLyFkdcRAAgGQiS1SIy6v7pWz1vLq/e9w6Kvl2GM4Yhhvbjs7gu4680/8sOHc3n3sU/YsX4nLTJSOef60xh9xch6m8ym6o8mAqXi5KfPFuL3RR/mKQJfvT6Ls687tWTbmN+N4u0HP4o4akcEjjqpX43XGw74A+zbno0zwUl6Vgt++Ggu/7zkCXxFPg4WGFj09TJWzv6FPzxzNadecSLDzzumRu+lDi2aCJSKky3lSkSUV1TgZcOyTWW2/eamM5j+32/YtyOboP/Xcfwi4fWGjztnMNm7DlSr3EPAH+D1v7/HR099TtAfJBAI0rFnO7au2YE/wsQ1r8fHk9e9wLFnD9J1jZuJeK5ZfK+IbCtepnKRiIyJctxpIrJaRNaKyB3xikepeAsGg6xZuJ5lP6wi/0ABqa1ScLqjd6raHDbSs8pe0JPTknhm3oOMOO8YHG4HCcluLJuFWBYiwot3vM4lXa7jbxc+VqWlJo0x3DP2Ed577BMKcgopKvQS8AXYsHRzxCRwkFjC129+X/UPrw5p8X4i+Jcx5tFoO0XEBjwDnAJsBeaJyBRjzIo4x6VUnZrx6re8cNvreD1eLMvC7/Vz7DmDCQZDUV9js1mMumxEhe1pmS24680/Upjn4dX73uGTZ2fg8/jKrEE855MF3HLSvTw15x/YbNEnl/381VKWzFyO1xN7JFJ53kIfW3/ZXq3XqENXQzcNDQHWFq9djIi8DZwDaCJQh4ypL37Jf/74vwqdvD9+NBdXkgsTNBVKQrgTXZwy4QTadW8T9bwi8GlxEijP7/WzdfV25k9fzNAxA4Hw3f+S71bw2aQv2b8jm279O7N51baoi9jH4nA5yOyYUe3XqUNTvBPBDSIyAZgP3GyMyS63vz1QekD1VmBopBOJyERgIkCnTp3iEKpS1ef3+Xn+llcjjvTxewP4vQEse3ELrISfAiybjePOHcw1/7oi5rkXfLEEm8MGUVqAPPlFfPHKtwwdM5BgIMi95z3Koq+X4i30Ygws+2EVoUD0J5LKjLq04tOKappq1UcgIl+KyLIIX+cAzwLdgQHADuCxSKeIsC3iAgnGmEnGmEHGmEGZmZm1CVupOrPom+WVHlNyMTbh4Z9+r5/ZU+YzvsPv+WXBuqivKyrwYkKx1ws5WA7itb+9y89fLgm/pvglQX+Qmqw34kp0cdnd59OqbXq1X6sOTbV6IjDGjKrKcSLyAvBphF1bgY6lfu4AaMOkOmQU5nqi3LrE5skvwpNfxG2j/sYra56iRUbFkg09BnYlFKOPwel2cMSwXgT8AT566vNq9wOUZrNbgNC2WxZX/G0cJ1x4XI3PpQ49cWsaEpG2xpgdxT+OBZZFOGwe0ENEugLbgHGATltUjcKuTXuY9vLX7Nywm3Y92nDqFSeSVa7dvEvfjhHLNVeV3xfg8xe/YtwdYyvs69ynI136dWLtwvUEIzTxiGVxxsRT2L15b61i6NynA0/O/gfGmDKT21TzEc8SEw+LyFIRWQKcCPwJQETaichUAGNMALgBmA6sBN4xxlT+rK1UnL3+93e5qvdNTH7oI758/Tve+seHXNnzRt55dEqZ4zr37kCXvh2xbDX7r+Tz+Pj+o7lR99/7/i20bJteptyDw2XHlejir5P/RHpWC5xuR8wnh8qcde2pJKYkaBJoxuL2RGCMuSzK9u3AmFI/TwWmxisOparru/dm8/ZDH5dZJezgTN8Xb3+d7J3Z/Pafl2B3hP/73P3uzdxwzF0U5hZWWjAuklglpTPat+LllU/wzVvf88WrM/EWejnq5H6cff1pJU8nGe1b0bpLFltWbav2ezvdDnoM7Frt16mmRRevV6qcq/v+iU0rtkbdL5bQ9/jePDTjLzic4QljBTkFTHv5a2a8MpPt63bh83gJVdLRe1BCspsTLjyWC289hw6Ht+O79+bw5gPvs2nFFuxOB8PGDuGyuy+gQ4+2Uc8x9/Of+dv5j1a7n8DpdvDWludJbaUziJuDaIvXayJQqhS/z88ZiZdUOlrHmeBk/B3nculfL6iwz5Pv4Y7THmDNgnX4vVVbUtKyWThcDgafNoD50xeVGftv2SxcCU4e/eZeDj+6e9RzzHx3Nk9c8zzBQAgDmFCI1FYpZO8+gL+oYhwOp51jzh7E3e/cXKUY1aFPE4FSVRAMBDndPb7SRACQ2iqZ93a/HLFpxxjDFT1vZPvanXUWW7vD2vC/1U/GbEoKBoIs/nY5OXvz6HB4W3oM7MbbD33I639/D1+Rv+RzuZNcZHRoxZM/PkBKenKNYwoGg+zetBexhNadM+O6cpqqvWiJoKFnFivVqNjsNvoc25PlP6yq9NiCHA+e/KKI1UBFBOr4Jmv/jmzWLFxPt/6dmT1lPt+9N4eAP8DQMQMZOW4Y7sTwWsYDR/Uv87pxt4/lqJP68e5jn7D25w0kpyVyxsRTOOni42tcUtoYw8fPTOON+9/Hk18ExpDSKpkr7x/P6Akj6+DTqvqkTwRKlbPkuxXcNeaBSjt+7Q4bnxa8EXUh+QfG/4uZ786u0tNFVSSmJjDxkQm8/eCH5OzJDV+AAXeyG6fbwaNf30vXvvUz6/65m//Hp89/WWFGtSvRyaV/vYBxt59bL3Go6on2RKArlClVTv8RfbjtfzeEyztEIZZw7DmDoyYBgPNvPjtm9dHqMiHDOw9/zJ4te0uSAEBRfhG5e/O47eT78EVYx6Cu7diwiynPzohYVsNb6OO1+94hd19e3ONQdUcTgVIRjDj/WF5e8W8Skt0V2r1FwiN9Kluyseeg7lx+30W4Epw1nmdQmivByb4d+yNOLoPwOgLfvz+n1u9Tma/emIWJMW/BslnMqoc4VN3RRKBUFO26t+H5xY9y1Mn9cLgdJKYm4HA76HNcL56c/Q/aHxZ9OOdBF9x8Nv/+/n6OPXsQSS0SidWX6kxw0rlPB1yJFdvtXYlORpx/bMyJY578In7+JtIE/rq1f+cBAv7oM5l9RX5y9uoTwaFEO4uViqFt19Y8NOOvZO86wJ6t+2jZNp2Mdi1jvsYYg98XwOG0IyIkpSWyZOYKPPlFFfqPxRKcbgfuJDeX3X0+Z14zmin/mc7khz7iwJ5cTMjQ/4Q+XP3PS1g9bx1iRb93EwFXHTZFRdP1iI64k1xRy1u7Ep10OLzyJKkaD+0sVqqOFOQU8Nrf3+PzF76iqKAIV6KL0357EhuXbwkvDh+h0zg8H2Es4+8aW2aBGWMMhXkeHC4HTlf44r5r0x6u7H1T1JXF3Mlu7v/kDo484Yj4fMBiBTkFXNR+YtTO9OS0JN7Z+ULJZDvVeGhnsVJxVJBTwHWD72DKM9MozPMQChk8+UV88p/p/Pzl0qgjh3weH99O/qHCKmMiQlJqYkkSAGjdOZPjxw7FleCscB6H007n3h3oP6JP3X6wCJJaJHHHazfiSnRiWb+2dVk2C1eii3vev0WTwCFGE4FSdeCtBz9kz5Z9FWYSx2pLP2jfjvLrNUV363+vY8QFx+JwOXAnu3EnuXC6HfQ/oQ8PzfhLvU3oOn7sUJ744QFGXHgcqRkppGWlMvryE3h2wUMMOLFvvcSg6o42DSlVB8a2vIL8AwU1em2Xvh15YcnjABQVepn6wpd8/Mw0cvfl0aZLFhfcfBYjxw3DKtU/sG9HNgu/WEIwGKLf8F5V6rhW8WVMERgvSGqjnWGtM4uVipNQKFTjJCCWcN6fzgLCq43ddNyf2blxd0n7+9rsDfzr988z893Z3PP+LSXJoFXbdE6ZcELdfABVK8a3GJP3CPgXAgJWKibxd0jS5YhEn2fSmGjTkFK1ZFkWyelJNXptYkoCoy4dDsBLd77B9nU7K3TCFhV4WfjlEr54dWatY1V1y3jnYPZfBv65QADwQ2gf5D+BOfCnGi0V2hA0EShVB866ZjQOV/U6SG0OG1fcPx67w47P62fGKzOjVistKvDybrlFcVTDMsZgcm4DiiLs9YDvO/BFX3SoMdFEoFQdGHfHWNp0yayQDCybBRGaiy1LSG2VwqmXh5t3cvbkUtnixzs37qmrcFVd8C8Gkxt9v/FgCt+ov3hqIW6JQEQmi8ii4q+NIrIoynEbi5e0XCQi2gOsDkmJKQk8PfdBzv+/M0uaiZLSEjnvT2dy7g2nl4zycSY4cSe56NirPU/++AAJyeHKpUktEqOWjjgoOU2XkmxUQnuIfQk1ENwRY3/jEc+lKi86+L2IPAbkxDj8RGPM3njFolR9SExJ4KoHLuaqBy4mFAqVGeVz2d0X8NNnCykqKOKwgd3oNeSwMiNLElMSOOrkfsyftihiu7LD7eCMiafUy+dQVWRrD8QaHmyB/dBYBjTuo4Yk/Nt+IXBSvN9LqcbCKlcKIrVVSqWjfK59/HJu+GEVnjxPmVIUNoeN9KwWjL1xTPQXq3onjj4Yqy0E10c5wokkRVy6vdGpjz6C4cAuY8yaKPsNMENEFojIxHqIR6lGqWPP9jw5+x8MOKkfDqedhOJ1BkacfwzPzHuQ5LSajUxS8SNpj4MkAeWHiSZA4njE0a/K5zKB9YTyHieU81dCBW9gQvl1GmsstZpQJiJfAm0i7PqzMebj4mOeBdYaYx6Lco52xpjtIpIFfAH8wRjzXYTjJgITATp16nT0pk2bahy3Uo1d7v48cvfl07JNWsQV0FTjYQKbMQXPQtE0MD6wH4YkXweu0VWaWGZMCJP7F/B8SngIagAI/5tL2r8R94l1FmuDrFksInZgG3C0MWZrFY6/F8g3xjwa6zidWayUaipC+c9A/iTAE2FvApLxIWLvVifv1VBF50YBq6IlARFJEpGUg98Do4H4F1RXSqlaMKEDhPJfIrT/SkLZv8d4PsOY2EubRjyP8UHBy0ROAgA+TMFLtYq1KuLdWTwOeKv0BhFpB7xojBkDtAY+LH58sgNvGmOmxTkmpZSqMeObj8n+HZggByeTGe9PkP84tHwbsWVW/WSBDUCsYcNB8M6qTbhVEtdEYIy5IsK27cCY4u/XA0fGMwallKorJpRXnATK15YqhKAPc+B6aPEYxvNO+CJv64AkXhi9aUdsVDaRsD7G9GjROaWUqiLj+YDod/AB8C/F7D2d8MXdD9gxhW9gEi9BUm6v2Hls6wq4gcIo53SA+7S6CT4GLTGhlGrSjG8eoezrCe0ZTWjfeIznU4yJvMpbpXw/gYnWng/hCWY+wkkAwiOAvFD4FsbzUYWjRWyQfCNIlJFh4kSSJtQs1mrQJwKlVJMVyv0neN4GU0S45MNGTM5KKHwFWr6GiLt6J4x2wa6UBwqegcSxFU+ZOB5j9kP+c+GmIuMHcQFOJH0SYmtXw/esOk0ESqkmyXi/LU4C5e/gC8G/CpP3CJL612qdUxLOwni/AhOtKSeG4DZMqBCxytaMEhEk+QZM4iVQNANCuWDvDq4RhEfgx58mAqVUk2Tyn4/RjOMFz3uYlNsQcWGCu4pH5/jBMRBx9Iz8MudwsHWCwDp+bf6pckQQ48IuVjokXhR1fzxpIlBKNU2BaFVtDhJMYBumcFJ4Vq/YOFjkyTh6QdqTiH8pBHeBrR24hiPigJavYQ78CXzzAHu4zLgJgNUaQluIOgrIMQARZx1+wLqjiUAp1TRJYiXrBQQg/2nwfgn4yl6//UtgzwkYEoBg8Z28DdIeR1wjkJYvYwJbwmsSiAOcx0FwC2bfeCJPDnMjKTfX5aerUzpqSCnVNCWcB8S4A7d1Be8MIq8wFiKcGQoBb3jegMnFZN+A8YeLH4i9I5JwJti6YPL+jcl/AlwngKQWF6Jzh/+UFCTtUcQ5uK4/YZ3RJwKlVJMkSZdhPJMhlE3FdQPc4D4pPHqoWqUhvJi8JyD9eUAwefdD4buE+wuCgAMQcJ+OOI8CK6u407dxNgkdpE8ESqkmSayWSKv3wTEIcIGkAAlg64y0fBGxtYdqF9004JuJ2dULs+toKHyL8BPFwUTjB3zh0T9WW8Q9KmISMMaPCe6qtNS0MSGMib1yXV3QJwKlVJNjQnnhUUCmCEn9C1ipENwCkh4uEy2CsVpSeXmHWGJdxD2Y/KcrlJA2xovJfzKcQEwACGKcRyMpdyKO3r8e550dbmry/xz+2dEfSb4JcR1fi3ij00SglGoyjDHhC2jBS8UdvAZMKHzxT38WsbUuOVbsh2Ec/YovtoG6Dyawslxsfsz+y8G/HPD+usM3B7N/PLR8FXH0J1T4IeTeQ5m+C/9iTPZ1mJS7sJLG1Xmo2jSklGoyTMEzUPBffu3gLQSKILASs+8ijCnbMSzpT4O9W3iEUZ0LlV1/umg6+FdRJgmUBF6IyflLuKmofBL49QSQ9wAmlF3nkWoiUEo1CSZUCPkvEHn4ZhDMASiaWmarWOlIqylI2pPgGEjFJSdrGZP3a4znQ0zh25iC54leXA4IbMQUvk54YkI0UrySWd3SpiGlVNPgn188KSzKflOIKXwXE9gEvjkgCUjC2PAIH9cIkBaY/ROIvkhMdYXgwPUY3OHvIz0JlCYOCGyq5P2LMMHtMVNFTWgiUEo1DVUZBur/GfxLCVcIBeNfBPlPQsvJ4OgPtrYQ3EjsxWKqI0TMp4DSjC9cY4gEoicDV1yK0GnTkFKqaXD0q0IyCHEwCQDhPoTgDsyBG8PF39KfA2lBeI2AkhMTc2JanbDAeRSSeBGVjmRKODMe766UUoc+sbUG1wiqf9EuXlAmsBGxd0EyZ4TXCLD3AltncI0G9wWEE0I8OEDSkBYPIlYKpN5D2UR0kBtS7gwXp6tjtUoEInKBiCwXkZCIDCq3704RWSsiq0Xk1CivbykiX4jImuI/6/4TKqWaDWnxMNgPr8EoIHvxsE4Qq0W4dITVBoI7wDcTit6i+tVGqyIVki5HMj4tafKxEs9D0p8FxwDCHccC9r5I+tNYSRfHIYba9xEsA34DPF96o4j0Ibxw/RFAO+BLETncGFN+nvcdwFfGmAdF5I7in2+vZUxKqWZKrGRo9R74fsAUfgTer6hSG71IyaIzJrgPs29scWmKUDVLUFSHBYm/wUq5reIu50Ak5bbw8FNHHywrKU4xhNUqERhjVgIV1+GEc4C3jTFeYIOIrAWGALMjHDey+PtXgG/RRKCUqgURC1zDwWqB8X5ZtReZILiODX9b8Hx4cZg66zCOJlShOqoxQUze4+B5nV+HsoYIJV6FJN8Q/mxxEK8+gvbAllI/by3eVl5rY8wOgOI/s6KdUEQmish8EZm/Z8+eOg1WKdUE+Stbj+CgBEi+Djm4DKXnA+LTDBThfe1Hltlicu6EwtfCC+qY/OKvQih4CZP7t7hFUmkiEJEvRWRZhK9zYr0swrbaFPXAGDPJGDPIGDMoMzOzNqdSSjUHVjJUegdtg+QbkKSJv26qyTKUNSGCJJz169sGNkDR50SeVewJr6gW3BmXUCptGjLGjKrBebcCHUv93AHYHuG4XSLS1hizQ0TaArtr8F5KKVWRa0S4zlD0AyD9Wazyhdxs7SC4OcbrnITva6M9NdgoqUYqCWC8hEccHZxQ5gSxI2nPhfs0ihnPVCqWyy4tgMm+HuM6AUk8L1w9tY7Eq2loCjBORFwi0hXoAcyNctzlxd9fDnwcp3iUUs2MSAKk/InwBK3yXOA8GnEOq7gr8eooryl+XeJ4sGVRcTipDUiH5JvBdRq4z0Va/Bsyf4Dkm8DeB+w9IekKJGMa4jqm7MtNLrGL34UgsBQKnsfsOY1Q/tMxjq2eWnUWi8hY4CkgE/hMRBYZY041xiwXkXeAFYQ/2fUHRwyJyIvAc8aY+cCDwDsi8ltgM3BBbeJRSqnSrKQrCEkC5D1OeCKZhMs/J/wGSb0r0kAXJPECjO9b8M0Ot9WX7EgAe4/wkpPJ12HyHgXPJ4SfDgy4RyMptyG2NmXPB5B8dfgrBnEcgZHEKjRNFT+J5L+AsXVDEsZUcnzlxFR7YYaGN2jQIDN//vyGDkMpdYgwJgiB1cVlHA4r0yQT+fgQFH2OKfwvBLeBlYEkToCEc8osNGOMLzzCyEpBxFXLGL2Y3ceByav6i2xdsTKnV/lwEVlgjBlUfrvWGlJKNXkiNnD0qcbxFiScgSScUclxTrBl1Da84nO5IH0SJvu3xYvWVGH+QnAzxhQhEmkmctVpiQmllGokxHk0kjEdkq4CW48qvMJQF5dxTQRKKdWIiK01Vsr/YWV+Bs4RsQ929Iu4JnJ1adOQUqpRMYH1GM/7ENwZ7pxNOB+po+aXYmF0EgAABaBJREFUQ42k3IjZN5fIcwvcSPL/1cn7aCJQSjUKxhhM3v1Q+A7h8fQBwIXJfwaTejdWYvMbVCiO/pD2BCbnZsLrL/tAiucxpN6PFJfFqC1NBEqpRsEUvgmF71F2Ja/i73P/jrF3Q5xHN0RoDUrcJ4JrDni/KR7BlAXuUbXuIC5NE4FSqsEZE4KC/xB9Za4iTP5/kJYv1WdYjYaIE9wRq/nXCe0sVko1vNDe4oqfMfgX1k8szZAmAqVUwxM7lZd9tlWyX9WUJgKlVIMTqyXYOsU4wgLXSfUWT3OjiUAp1ShIyi1EXqsXwIUkT4yyT9WWJgKlVKMg7pMh9a9AAkgS4XLNSSDpSMsXEPthDR1ik6WjhpRSjYaVeAHGfQZ4v4bQfrB1BNdwRPRSFU/6t6uUalTESoSEMxs6jGZFm4aUUqqZ00SglFLNnCYCpZRq5jQRKKVUM3dILlUpInuATXE4dQawNw7nrSsaX+1ofDXXmGMDja+qOhtjMstvPCQTQbyIyPxI63k2Fhpf7Wh8NdeYYwONr7a0aUgppZo5TQRKKdXMaSIoa1JDB1AJja92NL6aa8yxgcZXK9pHoJRSzZw+ESilVDOniUAppZo5TQTliMgAEZkjIotEZL6IDGnomEoTkT+IyGoRWS4iDzd0PJGIyC0iYkQko6FjKU1EHhGRVSKyREQ+FJG0ho4JQEROK/43XSsidzR0PKWJSEcR+UZEVhb/zt3U0DFFIiI2EflZRD5t6Fj+v727CbGqDuM4/v2hWVBMtgmhCSYhKysrQVHC3qZXDNuFQSEFhVGSEZSjFLUyKqqB2qmthIiMiijNitrNEElhNdCLi8YsdJMFUsPQr8X5T9w599q0us+p83xW53/mDnwZDuc598/M3DpJCyW9Xq69CUmro5vqchB0ewZ4yvblwBNl3QiSrgVuA5bZvhh4Ljipi6RzgRuAH6JbetgPXGJ7GfANMBLcg6R5wMvALcBS4A5JS2OrZpkGHrF9EbAKeKBhfTMeAiaiI05iFNhr+0LgMhrYmYOgm4GBcnwmcCSwpe5+4GnbfwDYPhrc08sLwKNUP8dGsf2+7emyHAMGI3uKlcB3tg/ZngJepRr2jWD7J9sHyvFvVDexc2KrZpM0CKwFdkS31EkaAK4CdgLYnrL9S2xVtxwE3TYDz0qapHriDn9q7LAEWCNpXNInklZEB3WStA740fYX0S3/wj3Ae9ERVDfVyY71YRp2o50haQi4AhiPLenyItXDx5/RIT0sBo4Br5Stqx2STo+OqmvlB9NI+gBY1ONL24Bh4GHbeyTdTjXJr29I23zgLKq36CuA1yQtdh9/B3iOvq3Ajf1q6eWf+my/VV6zjWrLY3c/205CPc417t2UpDOAPcBm279G98yQdCtw1PZnkq6J7ulhPrAc2GR7XNIosAV4PDZrtvw7ghpJx4GFti1JwHHbA3N9Xz9I2ku1NfRxWX8PrLJ9LDSsarkU+BA4UU4NUm2rrbT9c1hYjaQNwEZg2PaJuV7fh57VwJO2byrrEQDb20PDOkg6BXgH2Gf7+eieTpK2A3dRDfbTqLZ137B9Z2hYIWkRMGZ7qKzXAFtsrw0Nq8mtoW5HgKvL8XXAt4EtdW9SNSFpCbCAZvxHQ2wftH227aFy0R8GljdsCNwMPAasa8IQKD4Fzpd0nqQFwHrg7eCmv5WHoZ3ARNOGAIDtEduD5ZpbD3zUlCEAUK7/SUkXlFPDwNeBST21cmtoDvcCo6o+Lft34L7gnk67gF2SvgSmgA393Bb6H3gJOBXYX93fGLO9MTLI9rSkB4F9wDxgl+2vIptqrqR64j4o6fNybqvtdwOb/ms2AbvLoD8E3B3c0yW3hlJKqeVyayillFouB0FKKbVcDoKUUmq5HAQppdRyOQhSSqnlchCklFLL5SBIKaWW+wupnqSyj/ZBggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "cls = AgglomerativeClustering(n_clusters = 4)\n",
    "cls_assignment = cls.fit_predict(X)\n",
    "plt.scatter(X[:, 0], X[:, 1], s=65,c=cls_assignment)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agglomerative clustering are also useful for viewing the dendorgram tree-like structures showing how individual objects/data points on the bottom are combined/agglomerated into clusters as we move up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD1CAYAAACm0cXeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAakklEQVR4nO3cfZRdVXnH8e+TN0iCvAwOIUQwBGMClSA2EjCIAiIUpNAuUt9NozZ21Qp2iRHRigVb6cuiWrUoIGkElUJwNVFSJI3FFrHU4UVGSAI2SgSSMGHkZVhDCMzTP/Y+uTsn921m7szNDr/PWllzZ+49+zx7n3N+Z59z7425OyIikqcx7S5ARESGTiEuIpIxhbiISMYU4iIiGVOIi4hkTCEuIpKxcaO5sle+8pU+ffr00VyliEj27r777q3u3lntuVEN8enTp9PV1TWaqxQRyZ6ZPVLrOd1OERHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYyN6ufEpXnfuWsjK+57rN1liABwzuun8Z55h7W7DKlCM/Hd1Ir7HuPBTc+0uwwRHtz0jCYUuzHNxHdjR03dl3/9yAntLkNe5t75jZ+2uwSpQzNxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkY02FuJn9hZk9YGa/MLPvmtneZtZhZqvN7OH484CRLlZERHbWMMTNbBpwPjDX3V8HjAXeBVwErHH3mcCa+LuIiIyiZm+njAMmmtk4YBLwOHAOsCw+vww4t/XliYhIPQ1D3N0fA/4B2AhsAp5299uAKe6+Kb5mE3DQSBYqIiK7auZ2ygGEWffhwCHAZDN7X7MrMLPFZtZlZl09PT1Dr1RERHbRzO2UtwG/cvced98OfA94E7DFzKYCxJ9PVFvY3a9y97nuPrezs7NVdYuICM2F+EbgeDObZGYGnAqsBVYCC+NrFgIrRqZEERGpZVyjF7j7XWa2HLgHeBG4F7gK2Ae40cw+RAj6BSNZqIiI7KphiAO4+yXAJaU/byPMykVEpE30jU0RkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJWFMhbmb7m9lyM1tnZmvN7AQz6zCz1Wb2cPx5wEgXKyIiO2t2Jv5l4FZ3nw0cA6wFLgLWuPtMYE38XURERlHDEDezfYGTgG8CuPsL7v4UcA6wLL5sGXDuSBUpIiLVNTMTnwH0AEvN7F4zu8bMJgNT3H0TQPx50AjWKSIiVTQT4uOANwBXuvuxwHMM4taJmS02sy4z6+rp6RlimSIiUk0zIf4o8Ki73xV/X04I9S1mNhUg/nyi2sLufpW7z3X3uZ2dna2oWUREooYh7u6bgd+Y2az4p1OBB4GVwML4t4XAihGpUEREahrX5Os+BnzbzCYAG4BFhBPAjWb2IWAjsGBkShQRkVqaCnF3vw+YW+WpU1tbjoiIDIa+sSkikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikjGFuIhIxhTiIiIZU4iLiGRMIS4ikrFx7S5A5OXqpoduYtWGVe0uo6H1vW8BYNGtV7W5ksbOnHEmC167oN1ljCqFuEibrNqwivW965nVMavdpdR17LE/bncJTVnfux5AIS4io2dWxyyWnrG03WXsERbduqjdJbSF7omLiGRMIS4ikjGFuIhIxhTiIiIZazrEzWysmd1rZj+Iv3eY2Wozezj+PGDkyhQRkWoGMxO/AFib/H4RsMbdZwJr4u8iIjKKmgpxM3sVcBZwTfLnc4Bl8fEy4NzWliYiIo00OxP/ErAEGEj+NsXdNwHEnwe1uDYREWmgYYib2TuAJ9z97qGswMwWm1mXmXX19PQMpQkREamhmZn4fOD3zezXwA3AKWZ2PbDFzKYCxJ9PVFvY3a9y97nuPrezs7NFZYuICDQR4u7+aXd/lbtPB94F/Mjd3wesBBbGly0EVoxYlSIiUtVwPid+OXCamT0MnBZ/FxGRUTSo/wDL3W8Hbo+PnwRObX1JIiLSLH1jU0QkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMKcRFRDKmEBcRyZhCXEQkYwpxEZGMDepr91npWgrdy9tdxdBtPif8XPqF9tYxVEefB3MXtbsKkT3enhvi3cthczccfHS7KxmSfz0s4/8UcnN3+KkQFxlxe26IQwjwRbe0u4qXn6VntbsCkZcN3RMXEcmYQlxEJGMKcRGRjCnERUQyphAXEcmYQlxEJGMKcRGRjCnERUQyphAXEcmYQlxEJGMKcRGRjCnERUQyphAXEcmYQlxEJGMKcRGRjCnERUQyphAXEcmYQlxEJGMKcRGRjCnERUQyphAXEcmYQlxEJGMNQ9zMDjWz/zSztWb2gJldEP/eYWarzezh+POAkS9XRERSzczEXwQ+4e5HAscDHzWzo4CLgDXuPhNYE38XEZFR1DDE3X2Tu98THz8LrAWmAecAy+LLlgHnjlSRIiJS3aDuiZvZdOBY4C5girtvghD0wEGtLk5EROprOsTNbB/gZuDj7v7MIJZbbGZdZtbV09MzlBpFRKSGpkLczMYTAvzb7v69+OctZjY1Pj8VeKLasu5+lbvPdfe5nZ2drahZRESiZj6dYsA3gbXufkXy1EpgYXy8EFjR+vJERKSecU28Zj7wfqDbzO6Lf7sYuBy40cw+BGwEFoxMiSIiUkvDEHf3OwCr8fSprS1HREQGQ9/YFBHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYwpxEVEMqYQFxHJmEJcRCRjCnERkYyNa3cBspvoWgrdy1vT1ub7w8+lZ7WmPYCjz4O5i1rXnsgeQiEuQfdy2NwNBx89/LYOnjP8NlKbu8NPhfge46aHbmLVhlUtbXNd7zoAFt3a2v3kzBlnsuC1C1raZispxKXi4KNh0S3trmJXrZzRt0grQqhVobO7h0w1qzasYn3vemZ1zGpZm7M7ZresrcL63vUAu/X4KsRFhqAVIdSK0MkhZGqZ1TGLpWcsbXcZdbV6Vj8SFOIiQ7Q7hFAOIdNuw7lqGu7V0mhcJbU3xFv5ZlrZSLy5ltIbbTJCBhM6QwmZHG+/DMdwrpqGc7U0WldJ7Q3xVr6ZVtbsm2vPboLnegbX9rZnwklisCcgBb80YTChM9iQyfn2y3C046pptK6S2n87pd1vpi09C57bOjInkpQ+YSGDMFKho9sve572h/juoNUnkqq3iTzM3qvd3mn1DH0ot6mGcvtptOquV5uubnZR73ZMvdsvI3GbpVYttep4ud3qaQV9Y3MkFLeJUgfPqX6LZ3N3698XqLb+RmrVV8to1j2aY7cHKG7HVDO7Y3bVWzDre9e3/HPb9WqpVsdI1bCn27Nm4rvTDLTZ2f1IvfE60repdoe6R/Dz443eXGz0hmK7Z5S1bsfU6pfjrOtdNyL9Kdcy2BraPZapkXzTeaj93LNCfChvlA7224Uvl3vbjU6IzZz8mjnZpeup1ma5jcG+fogavblY7w3FRm8eDvcEAUM/4Gv1azj9GckaBrvuamPbyls3I/Wm83DGeM8KcRiZGehOgVbl3vZgwypVKwwHE0bltlsRbo1OiI1Ofs2e7NL1lNus1sZgXz8MQ3lz8aaHbmJd77odM8pqQTGcEwQMP1Sb6VcahuUZcitmxtVqqBbA1Wbn9dZfbWxbcXJoVPtQtGqMhxXiZnYG8GVgLHCNu1/ecKE0cB7rgpe2wxcP3fmArBc49WaIaXjV+uhgtfBpFHCtCI5aoVi0l9Zb/gjjYOpL2xxsjWWDOSHusl1KJ7t6fSivZ0dbsY2upTsvW7y+0ToLTZ7A0oOqe2s32we2c8J3TgB2DoJGIWIYszpm1Q2KwQbBcA74dNl0VtpsGKZ9H0z4FSe0ZtZXXmeh2n3zRuuvN7bFWDQzhs3O6ltxVVT0s6e/h0eeeYR1vetYtWFVU20POcTNbCzwNeA04FHgZ2a20t0frLtgGjhjJ4QQTzUKnHozxDS8nuuBF56DCZPrd6TZgKsVaI3uyxZBU5xgCuVgqfVRx6HWV+/qoVao1btVMdQTybOboPf/Kielop3yuCw9q/LcjrbmVN68rLbuaut8dtPOYz2IE2J6UI0fM54XB17c8VxPfw9P9j9J3/a+HQcYVD+AixBJD/RaQVqrjXq1DfaAr7bsYMJwqCeQZk9oRfvFuBTKbTdzldNIMRazO2bT09/Dut51O7Zpub5WnVTKyv1N+7Ho1kX09vc2HLPUcGbixwG/dPcNAGZ2A3AOUD/EoRI4RUDsNKtKAqd8wENl9r75/kpQVHvdSy8k66v3um2w8ac7Xw3UCpp0RljvuVQaSEXApMGShkrVGaaHPjdTX9UgPBomd4aTWrHux7pg9ed2ba98xZHWWyxTjGdad9fSnYMzfS49OaUnpHRcYNeTVXkfqaU8Zs/1hHr32jf0u1E/SvWWA7iY0RUH19wpcxse/NWUgzRto3trN1d0XQFUQqI4sFt1wJcD2XG6t3bvcqXR6PZPo9rLy1c7odUam0bBWu+kkI5T+QqqXk3FNi0+GZO2l55Uytuj8MLAC9zzxD0c861jABg/Zjzjx4znwIkH8mT/k1XHNu1veb3NjllqOB8xnAb8Jvn90fi3oeleDo/cUfk42SN3VIKseA7C7N1fqiw3Gq87eE4IgPTSvd5zZUXQvGJqeO2rT9y1j9XGAsLrxk4Iy9Wrr1p75fUePCesu1Z76TLleust07280n61PhVt7nIrKVnXcL9slY7Fq08M9bxiav1+VOtLHcXB1Tmxk77tfcydMpe+7X1Nf1qhWL7cxvgx4+nb3rfjdV1buna0uWrDKrq2dDG7Y3bVdRVtNvuV8rS9eusdSu31lm+k2bGt1d9a/WpUU732+rb3MbtjdtXtURg/ZjwDPgDAgA+wfSDcWSiu2grlOga73eoxdx/agmYLgNPd/cPx9/cDx7n7x0qvWwwsjr/OAqp/gFVERGp5tbt3VntiOLdTHgUOTX5/FfB4+UXufhVw1TDWIyIiNQzndsrPgJlmdriZTQDeBaxsTVkiItKMIc/E3f1FM/tz4IeEjxhe6+4PtKwyERFpaMj3xEVEpP30H2CJiGRMIS4ikjGFuIhIxtryH2CZ2R/Fh/OBn7j7jW2o4bNAH7APsM3d/360axgqM/sw4RuzACvcvep/cGJmbyBs43OBte5+3SiVOKLM7Mz4cB7wtLtf0cK2vwGsAH7onn4LrPXM7I+Bg+KvW9x9WQvbfjfhY79XAie4++pWtS1DE79LY8B+QL+7X1Pjdae7+w+bbXfUQtzMlsSH84HxhP935an43BnASYQd+cvJMtcB3XGZ/YA7gOeBk4H/i39bCRwI7A30A78D/FvRHuDxOQjBdxdhII8FHnP3883sEjP7GNABXA1cSvg26tXAPwO/iOudFOsZAI4CzgP+CbjZ3bcmdS+I69oWlzsJWB3XexTw7/GnAVuT2n8PuD32d3pc983uvtXMzgf2isu82d3PNrOLgFlmNpvwRar94voeiO2dBvzM3S82s8+Vtsf1wP1xXZtKY/ZE0sd5wJoatf8o1n5SXP9dcb2nAnfGxxPjWBbtzQduo7IfFP39ibv/XVLfijjWzwOT3P3iZGxfAzwNfBG4OFnmklhD0Y+BZJteBqyLdffHbensum8dDjwJXGxm5u6XJu1fG9voB44n7HvFWDxd2gYbaqz3xDhuNwNT3P1vY9tfN7MtSe3jgdfG9g4D/iUuc1oylgY8k7xuMnBfXP/bgTOBJcAEMzumxutOAW4ttgHwv8k2nQasStbVE7dnP/C2OGYGnBCfK47Hx5I2zgC+EWv/46SGlXHbFNvnJuBbVD+WplLZ9/vd/SvxuW/FPlRrLz1uTwL+g8r+2Enl2JwNXF+jv8/FMt4EvJpwQiyOxxuA/4z9eg+VY/hU4N5YwwnAHyXb+2DgQHe/yMxuMbNpyevOi8sZ8F4zO754zt1vpo7RnIlvJITF8+7+T2b2ZkKIHw0Qg+YzZraKcGADHAlcQxjwyYSZxYWEsPs4cAFhJjMGGHD3r5jZ7cD8oj3CwXAZYXBWA1fEx9cDq+M3Sn8V276MsNPPIXzLdAlwCOFLTJ8E/pIQIE/F5ecTdp53m9kFVL7UNJ8QJkW9ZyTr/WGs+TrgT2J9Re1vjstti+2siG13Eg7coh+vj6F8M2EbnpeMx/FJe2fEvgF0JWNrhJ336riuV5bGbN/Yx+sIO2W12j8cnyu23apkvWcmj8tj9vqkj1viPrHN3b9cqm8mIQw+CdxvZk8lY/tp4A8JIX68mfXFZd4DjEv6UYzZEuCYuE0N+Azhm8ObCAfyPnFbfRI4yt3vAu4yszGl/fE44MHYr3NK23H/ZBucSAisT1VZ753FdgWOjJObAUIIpNtgr6S9Cckyfwh8L47lh+O2uiC+9kRCiF0IHOvufWb2+TgG+5bq64j9fWPcHsVxeVmyTVfHPhbrmpBs1z9I9ov/AN5L5XicnbQxh3iMAGcTjt3idXsn2wcqx9L5hH2z2N73Utn3HzSzifG5eYTJWNHexKS94ri9kDDpS/fH3ybb+7Zk3Mv9fTC20R/XXdTXCfw06dc7gNti+79PmAR+ijAhuDN53ZGxXQjHQ1Hr4cDX3f3HAGZ2SOm5uiE+ah8xjJf2e8WClrj79fHvnyAM/kTCAQXwLPE2B/ADwpnqzYTZ9/uBewiztCOAh4DTgX8ghMxfAw8TZlr7EWaVX6USQmsIJ4SdbqGY2Qfiw/2Aw9z9k/HvVwDfIYTKM0AXsIAQvicTNviJhMC5nfCfgPUTDkwDPkCYGd9N2GluIWzYvwE+H+t8mHC2PyS+blxscw3h7P8mwo53KuFEcrm7X5bUPo+ws98AfB1YTjhot7n71cnrbgO+FMfhOiqz5U8Bf0GYkc2LdZ9MmOXeGV+/N2HH/03sx9/GfvW5+xoz+35c93HARHdfEtd5VqxlJmHmeyjh27698fGv4lj2E2ZIL8Z2fhZr3Bj7/aO4bZ4v9ekYd/95fHwNIZyvjP3pB/6LcDtpHuFg2hfYTDg4ZwAXAT+P224TsDm99WBmFxJmj8Tt8+PY/leAhbFP5xNmtm8hnLA6CPsdwFuBZYR9eyXhS3E79u9iHzSzd8ZxeCPh5DY21r93rH1TbO9QwjExicos8K+AbxJmrJMJs8bz09ts8cTghGNmGmFCM4mwTX8NfBZYRLiaeJ5wrB4bx/3AOF7rgO2Efbw3Pgb407itllHZd5+OfXkf8DHCPvQZwi3MyXEbTCUcU/sQju9vE7bxybE/x8XnJ8V1fRr4H+Bawv4+M9b3XcLJ4VnCcXciIU+WEWb3/83OE6M3xD5OjDX+DuFEdDhhfymumn5LmEm/FfhEcgVwOuEKeixh35oV17EPIajvI+zD7ybs0/dSuvVnZu+Itc4jHENpFj1EOOHeBnwwfuu9ptF8Y/MtwNvc/RDCwVMYIJxMPkMYvD5ghrt/gRCAZxMGYh5hMG4g7HQ/J4TBdwkb5klC6BwLzIwDPhbodPcJwOcIAXF4bHtSqb4phFnYV4AzzGyJmX2KsBFPIVzSLiaE4CTCrOQ1hEuu3vi3t8fLfov96iXM6veNj6fEfwOEM/7+saY+wpn5QeCQWN8EQlgsies/ERjv7pNj+6mPEwLg/YSD9Ch3/yohdFK3JOO0PyEoDwF+SdgpLyZcJh8Xt8kkwj5yZWx/VtwGXyKcvN4F/G4cpyMI4fFF4Mxk/BYTDrYrYhtHxscHEQ6g6XHMp8XnryScJB4lHFjTCQd7sW2mlvp0UbKu+cmYTQZeIvz/PhfHcZ4aT36HxO3yUqx7EuHyeybx6iKxl7svJtweGZe0P8PdxxDCb684ZmPd/YPAPskyD8R+TCIc1On+ne6DhxNCo7hi6SNMJr5AOCkW7T1PmEjsTQhId/ex8bmx7n4KYRZb/p+VBmLbH6VyXFxGOBbvJ0yC/ieu99C43n3i40vj9nkjlX38A7GGvePYrgE+SDixHRb/fZEwQSn2ufnAQck2eDYZC6hs41dQ2Zf2IwR4N2G/uyfW2Es4Xq4k7GNHEPaV4nh8HDjZ3WcA05Lxm5X08dI4tjPic8/Fcbk0bpsp7v524BJgUbKf/Rnh9kuxb01K+jGVyj48I4779NiX/ZPtcXTy91eUttXX4ph9mnALpq7RvJ0yQHJpX+fvewO/NLOPACuL+6Rm9u+EDfqUu/9jcTvG3Tea2f3JJfB8wqUOhHtiL8ZbD8sJB8Avk1soqZ8QdmKAG5P1PuHuS+PjOcAByWVPUdPTsR9PJv34NZVL/msIO1n58Y3u/tPk1tIjwPbY95+XLodrjR/sfCn2OGFGU/Q/lY7TKTUejyHMnIp13VGq45lk3NP1rgdeF/v1NXe/Mhmzw5M2ZiePr6VyT/sXwIvJc9+qsU3L3wpOa+ivM2a3Af+YrGstle2zqrRc6hfJrau0/RuS/Wo8YbZWrOvOZJmtpbE9l8r+vWMfdPfLS/v0I8nruuu0d3ZaX+k2W2pF0t+bCPshhBljuk2L9S4m3Gb5fDJmr0pqvj45Rso1zU7W9f06x6YnfUzX9QPC1cwux0hy7D8N3Jtut2QMy8fjnaVxOSLpYzq229g5H55Ontvi7tfG/s4hXC0UY3F37N9idj1u03FPP8BR6++w83HacKKtb2yKiGRMnxMXEcmYQlxEJGMKcRGRjCnERUQyphAXEcnY/wMyaj5UwBW01wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from scipy.cluster.hierarchy import ward, dendrogram\n",
    "plt.figure()\n",
    "dendrogram(ward(X))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Density based clustering, where points either classified as part of a cluster or outliers depending on the density of points in their vicinity, is another popular clustering technique.  It is used widely in anomaly detection.  A popular method for density-based scanning is DBSCAN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x20a02871610>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUZdbA8d+5U1MJkNAJRZqAgFIEERcEEQt2xe5aFvva17K+li32smsXsYu9F1aKWLCgFJEivfcWElJmMu15/5gQU2YmdZKQnK8fPiT33rn3DCb3zH3KecQYg1JKqabLqu8AlFJK1S9NBEop1cRpIlBKqSZOE4FSSjVxmgiUUqqJs9d3ANWRnp5uOnfuXN9hKKXUAWX+/Pm7jTEZZbcfkImgc+fOzJs3r77DUEqpA4qIbIi0XZuGlFKqidNEoJRSTZwmAqWUauIOyD4CpdSBKRAMsWT9Nry+AN3bp9MyNam+Q1LEORGISE/gnRKbugJ3GWP+U+KYkcAnwLqiTR8aY/4Rz7iUUnXvo+8X89+PvicYCiECPn+Q4X27cM+FY0lJcNV3eE1aXBOBMWYFMABARGzAFuCjCIfONsacGM9YlFL15/3Zi3js/W/x+gKltn+/ZC2XPvIOU+44D4fNVk/RqbrsIxgNrDHGRBy+pJRqnPyBIE98NLtcEgjvC7F1zz6+WbimHiJT+9VlIjgbeCvKvmEi8puI/E9E+tRhTEqpOFuweguxqt0XFPr5+McldReQKqdOEoGIOIGTgPci7F4AdDLG9AeeBD6Oco6JIjJPRObt2rUrfsEqpWqVp9Bf4TH5Hl8dRKKiqasnguOABcaYHWV3GGP2GWPyir6eCjhEJD3CcZOMMYOMMYMyMsrNkFZKNVDd26fjDwaj7nfYLPp1bVuHEamy6ioRnEOUZiERaSMiUvT1kKKY9tRRXEqpOGuf3ox+XdpisyTifsuymDByQB1HpUqK+zwCEUkEjgEuL7HtCgBjzHPAGcCVIhIAPMDZRtfPVKpRuf/S47noobfZm+vB4ws3FdltFjbL4q7zj6F9erM6i2X99ize/24R63Zk0a5lKqeP6Eevjq3q7PoNkRyI99xBgwYZLTqn1IHF4/Mzbe4KPvlxCQWFfgZ0a895Rx9KZqvmdRbDpC/m8PK0uQRDIQLBEJYlOG02Tjj8YO44dzRFjRONlojMN8YMKrtdZxYrpepEgtPBKcP7csrwvvVy/e8Wr+WV6XMp9P8xjDUUMnhDAb74ZRk9O7bijKP61Uts9U1rDdWjTSu2cN+5/+HEpPMY55zA5QNu5rv3f4p4bCgUYv6M3/j4yf8x683ZFOR66jhapQ5sL3wxJ+JcBgCvL8Dk//3MgdhCUhv0iaCerJi7mptH34uvoJBQKPzDt3bRBh6++GmW/7KaiQ9dUOrYu055EE+el6A/iM1hI/SX57jsgfM45drj6+stKHVAWbE59rDzrNwC8jyFpCS66yiihkOfCOqBMYZ/nf043jxvcRLYz5tfyKfPfMnqX8Oll3Zs2MUtY+4la1s2nlwvPq8fT66XQo+PybdP4as3Z9fHW1DqgGO3xb7dGWOw25tmmQtNBPVgxdzVZO/aF3W/vzDAJ09/CcD7j32G3xv5cbawwMeLt09pso+zSlXFUYd0xYrRGdy3cxsSnI46jKjh0ERQD3as34UVZUw1QCgYYvPKrQB8/+HPBPyREwHAvj25bFtbbp6eUqqMiScMxeWI3Bructj566kj6jiihkP7COpBy3bNY36KFxFadwrPng74o8/IhPBknIA/yL49uUx/9RvWLNpAi9ZpjL3oT3Tq3bFW41YqkrXb9vDONwtZtWU3GWnJnDGiH4N6dGhwQzG7tm3J0389ldsmTyXPU4gxYFmC3WZx70XHcmi39vUdYr3RRBCFJ8/DphVbcSU4yTy4dn+oex/RM+Z+u8vO+CuPBaD/yD7Mfv+ncn0J+1k2i4/++wVfvDATTLid07JZfPLU/xh19nBueOEKLEsf/FR8vPTlL7ww9WcCwSDBkEGA75esY3CPDjxy+UkVtsuXtHbbHl74Yg7fLl5LMGTo1bEVE48/nOF9u9RavAMOas/Uf1/Gr6u3sHVPDunNkhncs2NxnMYYps1bwUvT5rJxx15cDjvHDenFJeOG0CotudbiaGh0QlkZPq+P5256lemvfBMenRMMkZyWxMRHLmTUhOHVOmcoFGL+9N9Y8sMKnC4H3Qd24e5THyYQZSibw2Xn032vY3fYWf3rOq4/8k4KIxTlciU4sTlsFOyLPJTU4bJzwV1ncs7tp5Xanpedz7RXvuanz+ZhWRZ/OnMYR583goSkpjdaQlXfz8s3csOzn0Qckul22Llo7CAuP3FYpc61cM0WrnriQ3z+IKES9yS3085lxx3OJeOG1Frc0Rhj+McbM5g2b0Wp92S3WSS6nLx26zlktkqLexzxFG1CmSaCEkKhEH8bfS/Lfl6Fz1u6YqIr0ck1T17KuIuPrtI5t63dwS1j7mXf7lw8eV4sm4WIEAwGIco/fUJKAne+fQNDjjsUgFlvzebRy54DwOfxYdksHE47iamJ7N2RHfP6rgQnH2e/ir2obXT1r+u4+eh7CPiDFBYUAuBOduNOdPGf7/9J+25a/EtVzsTH32Peys1R9ycnuPjq4csrXHAmFDKMu30Su/cVRNzvdNh4984L434T/nn5Rm589tPiEhglWSIc0qUNL99ydlxjiLdoiUDbDEpYMHMxK+avLZcEIDxC57kbX8Uf4YckGl+hn+tH/B87N+7Gk+cFwh3BwUD0JAAQCgTZseGPMc9HnzOCKeuf4YK7zmTU2cM55drjuOv9m8jPya8whkKPj7WLNxbHc+vYf5KfU1CcBAC8eV5ydu/j9nH/1hFIjciyjTu4/cWpnPD3yZx276u8PG0u+/K9tXb+5Zt2xtwfCAbZlV3xz+i8lZsoiFGqOhQyfDB7UZXjq6opX82PmAQAQsawfNMutu7JiXsc9UH7CEr43+SZePOi/6IYY1g4awmDxx1aqfPNfn8OnlwPJkr7fjQ2u4309i1KbUvLaMbZt57yx7k//Bm70x4xaZW1cNYSehzWle8//DlqIjMhQ/bOHBZ+vYRDjz6kSvGqhuf92Yt47L1v8QX+aGqZ9MVPTPlqAa/deg7tWqZGfN2W3Tns2JtLmxYptGsZuxCcs4Ix98GQwR1hlE7WvgLyvD5apSXjdtrZmrWvVHNQWYFgiPXbs2JeqzZs3Bn7Ju+w29i2J7fCf5cDkSaCEnJ258Y+wEBeduTHVwh/4p7z2Tx2bNhNevsWzP5gTvGTQFWITRg8LnZZXndSFRb7LvolW/LDcjy50ePxeX2smLumVhKBMabBjRppKjbu3Muj731bqqYOQKE/iD/o4dbJX/D6reeU2rd6y27ueX06a7buwWG38AdCdG+fzj0XjqVr25YRr3PsoF68991vBIKhiPubJyeU+oS9eN02Hn73G1Zs3oXdZmGMYfzQ3gzplYktxoAGS4Q2URJXbWqVlsTGnXuj7g8Eg7RITYx7HPVBE0EJ3Qd2ZemPK6J24gaDQTr36RBx30+fzeOBC57AGIPf68fhckTs4K2IK9HJ316+prhNP5r+f+pdqScNh8tOu25tAEhIdiOWRH2dzW7DleCscsz75ezex1v3f8iXL31N/r4CWrRO45Rrj+P0G07E6a7+eVXVvP3NQoKhyDfnUMiwestu1m3Pokub8FPnxp3ZXPzIOxR4fRhgfyvN0vXbueiht3nrjvPokFG+ff7CYwby2ZzfyfcURmzp3JtbwOn3vsqRfbswYeQArnvm4+JOWH8gPCz6kx+XsmjdNmJ9ZnDYbZx+ZPyfUieMHMDSDTuirqjWvmWz4n+zxkb7CEo46apjsaIMdxNLaN+9LV0O6VRu3/JfVvHvcx6nYJ8HT66XgD+IJ89LKMonpf1sDhutu7TCsllYlsUhIw7mgS/v5IiTB1cYq9Pt5M//nICzghu3K8HF0BMHAvCnM4fFvCGHjKnUtSPZuzOHKw69hU+enkZ+TgEYyNqezZR/fcBNo+7BV4nlClXt+H3Djqif0iE8Cmbttj/Wfnrm0x/wFPrL3cwN4WUmn/s8ciHE1s1TeOWWCXRu0wK3007ZOZL+YAhfIMgPS9dFHV3kCwTZuCOb8UP7RJzs5XbaGT+0Nz06ZLBpVzb3vfkVx942ibG3TeLuV6eVeh81NbL/QRzSpU3UOO6+cGytXauh0URQQtsurbnmyUtwJThLzfx1uh2ktkjm7vdvjvi61+5+l8KCqn36d7odDBs/iDfWPM1nua/zhWcKj337D/oeeXClz3HadSdy2QPn4XCXnxYvluBKdPF/791U/HTRY+BB9D2yF84IxwOEAiFe/8d75EcZvRHL8ze/xt4dOeWepgo9PtYt3sAXk2ZU+ZyqeponJ1R4TEpCuGnRGMOshaujttGHjGHGglXFgwiMCT9RLFyzhb25BXRt25IP7r6IW84ciT3K6KBCfzBmZ7DH52fR2q08ec0p9O3cBptl4bBZtEpL5obTjuL2c45m/qrNnP2vN/joh8Xsyslnd04+U39ZxvkPvMnsxWsrfL+VYbMsnrz6VC4aO4jURBdOuw27zeLwXpm8fPME+nZuUyvXaYh0+GgEqxeu471HPmXJ98txJTgZc+GfOHHiMaS2TIl4/Fj7WRU20yQku4uHjzpcdvoeeTD3fHgL7sQqtPUDv/+0grfu/4glPywHY0jvkE6XQzqSkORm1a/r2LRiKzabxeEnHMY5t59G5z6lZxcXegp57C/P8e37PxH0lZ+17HA5aN+tDU/9cj+uhMrF5vP6OKX5n/HH+GVv27UVr61+ukrvVVXPd4vXcseLU6PefFMTXcx86ArsNgt/MMjQa58g1m1AgF+evo45yzZy31tfkZ3rwWYTfP4gR/TpzF3nH8MLU+fw9tcLYw2Gi6ldy1Q+/9elABR4fQSCIVISXYgI/kCQY259nn0lRrqV5HbamfHg5STVYvNjMBRiX0Ehbqe9UdUf0oVpqqDbgC7c/sZ1lTp2zufzK9VW70p04snzYoyhz/BeXP3fi6ucBD6fNIPnbny11NDPvOyNrF+yEVeii2bpKbyw6FHadI687F7AH2DO5wvwF/pxOOwRE4G/0M/29Tv5asr3HH/Z6ErFtW9PbszaSQBZ22LPd1C1Z3ifznRrn87yTTvxlSlR4nbaueXMUcUzaR02GxnNktmZnRf1fK2bpzBn2UZufv6zch3Qsxev5cKH3mJQ947VTgJAqbb3xDI39G8XrYnZ1CUI0+at4LRa7EewWValnqwai7g3DYnIehFZLCILRaTcx3gJe0JEVovIIhE5LN4x1aYX75hS4TEiQvbOcLVREzL8+tVirh58W3Gp6crYuWk3z17/cqkkUFJhQSG7N+/hplF3hyerlbFn214u7nUdj172DLM/+BlvfuTzwB+lsCsrpUUyoSidk/ultWp8Q+4aKptl8ex1p3PCkINxOewkuZ0kOB20Skvm3guP5YShpZsfLxgzELcz8mdCt9POhccM5IG3ZpVLAhAeIrp5Vw4LVm+Oeg4AS4ha+TPB6eD8MQOjvnb99r1Rx/dDuGlp9ZbdUferitXVE8EoY0y0/1PHAd2L/hwOPFv0d4OXn5PPphVbKzyubPObCRk8eV4euOAJJi95vFLXmjppRoWTvUIhQ25WHr9M/ZVh40s//f3f+PvZuXF3hR3Y++VmRf+EWJYrwcURJw9h9gdzIp7fmeDkpKvHVfp8qvJ+/H09k6f+zLKNO7HbLEb2P4jLjjucTq2b83/nH8ONZ/yJDTuycDnsdG3bMuKQ3rNHDWDuik3MXbmp1IiZBJeDob0yGdijA09+/H3MODbuzI468sdmCZ3bNGdffiE5+V58gT8+qCQ47Zw0rDdDekYvkNgsyY3LYY+6upjDZtEipXEO66wrDaFp6GTgNRO+y80RkTQRaWuM2VbfgVUkGAzVaKz89vU72fD7pkpVCV27aCP+wujlqPfz5HqZP2NRqUSwasFaNi7fUukkAJB5cNUqMV7x6IX89s1ScrPywjOnizjdDjp0b8tJVx1bpfOpir0ybS6Tpv6x/GKhH76cu5xZC1fz7F9Pp1/XtiS5nfTuFLuT02ZZPHbFSXy3eC1vff0r27P20bZlKueMOpQRfbuyZP32mOP8S7IswQICRc2liS4HGc2SefavZ+Cw23jnm4V8/MMS8gt9dG3Tgj+PHcxR/brG/D0ac1h3Hn3/26j7RYTjhvSqVHwqsrpIBAaYLiIGeN4YM6nM/vbAphLfby7aVioRiMhEYCJAZmZm/KKNYe+ObL58aRbrFm+kRbvmHPvnUaS3b8H2dbGn2kdjd9jZtTmrUomgRdvmiBCzUw9ABGz20r+0y+aswlQ+B+BOcnHGjeMr/wIgvX1Lnl3wEK/e/Q6zpswmGAjiTnIz/sqxnPv306vcH6Ji27Qrm+e/mFOuuSYYMngK/fzthc/5332XVfqDimUJI/sfxMj+B5Xbl9kqrdSn+GhMUfXb/T9qAow4pAv//PO44hFFE08YysQThlYqpv2apyRyybghvDJ9brmnArfTzilH9KV9ujY91kRdJILhxpitItIKmCEiy40x35XYH+kntdztriiBTILwqKH4hBrd9Ne+4b9XvgDG4PP6sWwWnz87nZ5DurF3R07EtnvLZsX8FB7wB8jo0IKCXA8/fTqP3Kw8Mg9uz4Cj+5YrHX3cZaOZOnlmhZnAleTmiJNKzwWwO+0VduaWNPTEgRw2pl+lj98vvV0LbnrhSm54/nJ8Xj+uBKfOLo6T979bFHXSGECep5AFq7YwsEfkCZBVkZacwFGHdOXbRWvwV+Gp0gAzFqyi/0HtOHtk5cqyRPOX4w+nZWoiz372EwWFPkQEp93GJeOGcP7oA6pbsUGKeyIwxmwt+nuniHwEDAFKJoLNQMmPxB2Aihve69CKuat54qoX8JWYKRwKhij0hEsy9BnWgyU/LMeEDH5fIDzJyxhOuupYPn9+RtSO2dadWjF/xm+8dMdbWDaLYCCIzW4jMTWRf3zyN3oM/OPTWdd+mZUqCNfuoNb0+1PvUtuGHH8oT137YqXeq1gS/lODG7hlWfoEEGdrt+2JOZLGAJt359RKIgC487wxrNqym427sqtUmDAUMjzx4WzOPKp/pZuXIhERTh/Rj1OHH8Lm3dmEQoYOGWlVWu9ARRfXRCAiSYBljMkt+nos8I8yh30KXCMibxPuJM5paP0Db973IT5P5FELhQWFrP5tPZOXPs6sKbPZuWkPHXu245gL/0RqyxR2b93LT5/OK/XEsH+y18gJR/DS398uU4rCjyfPy81H38vkJY/RqmM6AAFfAJvNIhiI/Yns4Zl3l7uJp7drwZjzRzDrre8rnPhmQoblP6+OeYyqf21apGBZEn3BIhHSa7EuTmqSm7f+fj4Pv/s1H/+4pMImypK8/iBf/7qaMQN71CiGAq+Pt775lXe//Y2cPC/pzZI4d9ShnHFUP5wVlGRRscX7X6818FHRjckOvGmM+VJErgAwxjwHTAWOB1YDBcDFcY6pyhbPXhbzU9D+m+t5d55Rbt/tb/yVz5+fwTsPfsyuTXuwbBZHnDyIC++dwG3H/DPqcFB/oZ8P//sFVzxyEQDuJDfN2zRn9+boU+q79usUddLb4ScM5IdP5lZqBnSVCtqpenHakYfwxc/Loo6ksSxhyMEV96Vt3LmXV6fP49tFawkZw+CeHbnk2MH07Fh+LorbaeeOc0ezdc8+Fq7ZQmEFy6iW9MlPS2uUCPI8hVz00Nts3ZNTfN2te/bx1Cc/MH3BSiZdf4YmgxqI63OVMWatMaZ/0Z8+xph/F21/rigJYMKuNsYcZIw5xBgTlynDxhi2rdvBhmWbq1z3xlbB46cJhaIWibMsi5OuPJYp659lqvdN/lf4Fne9dzN2uy1mKYeAL8B3780p/l5EOOf2U3FFaXJxJ7k4787TI+5764GPuP/8J9hXUXVVwhPfxl1StcV3VN07OLM1xw7sGXHsvsth5+4Lxla4IMz8VZs5+99v8OlPS8nKLSA7z8NXC1bx54ffYeaClRFfY7MsnrjmFMYP7R2xcy+apRt2VOHo8p77/Cc2784pl3y8/gArN+3inW8X1uj8TV2TaGD76bN5/LnHtfyl741cO/R2zsi4hMm3TyEQYYJMJMNPPRxbjNrrLdu3KLd+QCR2h7242SbgDyIVdOAGy4zUGH/FWEZOOAJXoqv4tZbNwpXg5MQrxjLi9PKjMbav38kb/3gv6pNHSTa7RUrzZI69eFSFx6r6d9cFx3DtyUeSnppUXBend2Zr/nvVyRw9oFvM1/qDQW567jO8vgDBEs1LIWMo9Af4v1emkVsQuWS5w2bjorGDq/QJPLfAW+1Fj4KhEB//sKS4YmlZXn+AKV/9Wq1zq7BG/yw1663ZPHbZc+VKQn/8xFTWLtrAvz+/vcKO0bNuOYmv3pxNMK/8D6Ir0cll959X5c7V9t3bxHyNZbMYcHTfUttEhJsmX8nxfxnDZ89NZ8e6nbTv3paTrx5Ht0MjL/D9v8lfRW1H/uM9uDChED0Hd+OON68jqZHWXG9sRIRzjj6UCSMHkJPvwW63FReTq8j3i9cRiDAD/Y9zw9RfljNhZOR1MdqnN+Ogti1ZtnFHpUpLCMK67VlR1zaIpcDrq3D46u59Fa+EpqJr1Ikg4A/wxNWTI64LUOjxsfi731n07e/0H9kn5nnaHdSGB6fdyT2nPYy3oJCgP1i0sL3hikcv5KgzKrdAd0kOp4PTbziRdx/+JGK7vcNlZ8ItJ5fbLiL0HtqD3kMr1966ZdW2qOsrQHho6ZgLjmLCLSfTtmvryr8B1WBYltC8ijNrN+zYG7FkxH5eX4BVFZRtuOfCsfz5kbfxFgZirjAG4XWHY10vlgSXM2p5iv0qmwBVZI06Efw6a0nMgnCFBYV88cLMChMBQO9hPXlr8/MsmLmYLSu30SwjlaHjB5KQ5K52fOfdeTpbVm/nh49+xl8YIBQMFa8vcMtLV9O1X/m1D6qqTddW2B02AlE69hwuB4OPHaBJoIlJS07AabdHreFjt1m0bBY7uXRrn86bt5/H05/+yMwFK2OOJAoZQ+fW1VvUxW6zOGZgD76cu7xUM9Z+zjpauKYxa9SJIHdPbsx2SWNg7/bKV8W02WwMPnYAg4+NvYxkVc53++t/Ze2iDUx75Wv27sim24AuHHvxKJql187SfMddOpqP/jsVoiQCkfA8A9W0HD2gGw++MyvqfptlceLhvaPu3y+zVXMevOwEft84iEsffofCCE04Loedk4/oQ4Kr+uWcrz3lSGYvXkuup7BUwrHbLFqmJnLhMeUqK6sqaNSdxR17tY/5RGB32DloQM0/dddU136duPKxP3PHlOs565aTay0JALTv1pbTbzwx4gQvV4KTv71yDY5GVG9dVU5qkpsrxx8RcdSR22nnlOF96Bhhecpoeme25tazj8blsJUarZTgctC/a1tuOO2oase6Z18+t784Fa8vgCV/3LJslnDCkIOZcvt5pNbgyVw18ieC7od1pVVmOpuWb4n42GrZhJOuavxVMS/517l07pPJ6/e+y9bV2wGh9/AeXPKvczlkROVXRFONy4XHDKJlShJPffo92XleRMIloS8ZN4Rzj676U+Ipw/sypFdH3vtuEb9v2EHz5AROPfIQhvTsWO2Z6j5/gIsfeYftWbnlZlI77DY6ZKSR1oTWDYiXRr9C2YZlm7n+yDspLCgsrt4pIjgTHPzlwfM5+erjahxPKBRi2ZxV7N2RTduurTmof+canzNefF4fls2KOu9BNT3GGLZl5RIKhWjbMrVGpSBq2/9+Wc6/3pwZdUH5BJeDrx66IuZaCOoPTXaFsk4Hd2Dyksf5+MmpfDXle/yFPg4e2oOzbz2F3sN61vj882f8xsMXP01BrgcRIRQM0Sozgzvfvj7iQvf1Ldbi9appEhHatay95sja9PnPv0dNAhAupTF/1WaG9+lcd0E1Qo0+EQC0bNucS+87j0vvO69Wz7vkh+XcfcpD5Yanbly2metH/B+TfnuU1p0yavWaSh1IjDFs3JlNvtdHx1ZpVR7mGSsJ7FfdYanqD00iEcTLpFteizhHAcL1h95+4COue3ZiHUelVMPw4+/refDtr9mVnYdlhRehH3BQe/42YSQHtUuP+jpjDAtWb+H9735ja9Y+LJGo8xT8gSC9M3Xoc01pIqim/Jx8Vs1fG3V/MBDkm3d/1ESgmqTvl6zjlkmfl/u0PnflJs765+sM7pnJ/ZceV24iXDAU4vYXp/LDkvV4ff6Ys5YdNovBPTvSpkXkQouq8hpOr9ABptAT7nSNxV/F4nZKHciMMXz72xouffQdrn/mk6hNNgaYt3ITFz74VrkJbW/MnM/3S9bhqSAJJLocZLZuzr8vrvlgD6VPBNXWLCMVd5Ibnzf6zb42ZgYrdSAwxnDXq9OY9evqqLOVSwoZQ1ZuAV/+spxTi2YFh0KG12bMj1paGyDJ7WRwz46MH9qbEYd01YVpaon+K1aTzWbj9BtPxJUYeRSOO8nFuXdELgutVGMzff7KSieB/Ty+AB//uKT4+5x8D/ne2Otl+ANBHrviJEYN6KZJoBbpv2QNTPjbyQwaO6DUQi77y0Kfet3xDD1xYD1Gp1TdeXX6vColgf1K3vidDnvFxetilINX1adNQzVgs9m4+4Ob+e2bpXz6zDR2b9lDZq/2nHLt8VHLQivVGG3cubfKr7FZQr8u7Yq/T3I76dOpNb+tjbxSrc0SxhxWs+UuVWTxXrO4I/Aa0AYIAZOMMf8tc8xI4BNgXdGmD40xZdc1brBEhAGj+jJgVN+KD1aqkUpyOymo4uAIu83GeaNLl7K4/rSjuOK/H0TsaHY57FwybnCN4lSRxbtpKADcZIw5GBgKXC0ikUoazjbGDCj6c8AkgQPNuiUb+fLlr/nmnR/Iy9aFPFTtOeWIvpVutrFZgsth529nlZ9P0P+gdjx2xUmkpyaR6HKQ5HaS4LST2SqNF286iw5VKISnKi+uTwTGmG3AtqKvc0VkGdAe+D2e11Wl7d6axT2nPsT6pZsQEcQSgv4gp0E+k9UAACAASURBVN9wIhf/65xqFwRTar9zRx/GJz8tZc++/HJrBriddv5y/OH89PsG8jyFHNKlLeeMOpTObSKvTzCsdye+vP8v/LZ2K1n7CmiXnkqvjq305zSO6qzonIh0Br4D+hpj9pXYPhL4ANgMbAVuNsYsjfD6icBEgMzMzIEbNmyIf9CNgM/r4+Je17FnaxbBQOnqje5EF2fcfBIX3XNWPUWnGpPdOfn8682ZzPl9Aw67DX8wSPd26dxx7mgO1tm/DUK0onN1kghEJBn4Fvi3MebDMvtSgZAxJk9Ejgf+a4zpHut8Vak+2tTNeO1bnrj6Bbz5kRevdyW6eG/H5BqttKZUSdl5HrbvzSUtKUFn/TYw0RJB3IePioiD8Cf+KWWTAIAxZp8xJq/o66mAQ0SiFyJRVTLzje+iJgEAm91i0bfaUqdqT1pyAr06ttIkcACJayKQcKPei8AyY8xjUY5pU3QcIjKkKKY98YyrKfFVYiSHlsJQqmmL9zyC4cAFwGIRWVi07Q4gE8AY8xxwBnCliAQAD3C2ORBXy2mgBo45hJVzV0ctheEvDNBrSLc6jkop1ZDEe9TQ90DMrn5jzFPAU/GMoyk7YeIxvPPQJxH3OVx2Bh3bn/T2Les4KqVUQ6IlJhq55q3T+Oent+FOdpeqi5SQ7KZzn47c+tq19RidUqoh0BITTcCAUX2Zsv4Zpr38NYu++53ElATGnH8UA8f2x2pA69MqpepHo1+8XimlVFi9DR9VSinVsGkiUEqpJk4TgVJKNXGaCJRSqonTRKCUUk2cJgKllGriNBEopVQTp4lAKaWaOE0ESinVxGkiUEqpJk4TgVJKNXGaCJRSqonTRKCUUk2cJgKllGri6mLx+nEiskJEVovIbRH2i4g8UbR/kYgcFu+YlFJK/SHei9fbgKeB44DewDki0rvMYccB3Yv+TASejWdMSimlSov3E8EQYLUxZq0xxge8DZxc5piTgddM2BwgTUTaxjkupZRSReKdCNoDm0p8v7loW1WPUUopFSfxTgQSYVvZtTErcwwiMlFE5onIvF27dtVKcEoppeKfCDYDHUt83wHYWo1jMMZMMsYMMsYMysjIqPVAlVKqqYp3IpgLdBeRLiLiBM4GPi1zzKfAhUWjh4YCOcaYbXGOSymlVBF7PE9ujAmIyDXANMAGvGSMWSoiVxTtfw6YChwPrAYKgIvjGZNSSqnS4poIAIwxUwnf7Etue67E1wa4Ot5xKKWUikxnFiulVBOniUAppZo4TQRKKdXEaSJQSqkmThOBUko1cZoIlFKqidNEoJRSTZwmAqWUauI0ESilVBOniUAppZo4TQRKKdXEaSJQSqkmThOBUko1cZoIlFKqidNEoJRSTZwmAqWUauI0ESilVBOniUAppZq4uC1VKSIPA+MBH7AGuNgYkx3huPVALhAEAsaYQfGKSSmlVHnxfCKYAfQ1xvQDVgK3xzh2lDFmgCYBpZSqe3FLBMaY6caYQNG3c4AO8bqWUkqp6qurPoJLgP9F2WeA6SIyX0QmRjuBiEwUkXkiMm/Xrl1xCVIppZqiGvURiMhMoE2EXX83xnxSdMzfgQAwJcpphhtjtopIK2CGiCw3xnxX9iBjzCRgEsCgQYNMTeJWqikwxkDhl5i8yRBcD1YzSDgbSTwXsZLrOzzVgNQoERhjxsTaLyIXAScCo40xEW/expitRX/vFJGPgCFAuUSglKo8Ywwm52bwzgQ84Y3BXMh7EuN5B1p+gFhp9Rqjajji1jQkIuOAW4GTjDEFUY5JEpGU/V8DY4El8YpJqSajcFrpJPDHDghux+z7Z31EpRqoePYRPAWkEG7uWSgizwGISDsRmVp0TGvgexH5DfgF+MIY82UcY1KqSTD5kymfBPbzg3c6JpRXlyGpBixu8wiMMd2ibN8KHF/09Vqgf7xiUKrJCmyMvV/sENoB2leg0JnFSjVOVrPY+40fRPsIVJgmAqUao4RzAHf0/Y5+iK1lnYWjGjZNBEo1QpJ4NtjaAs5IO5HUu+o8JtVwaSJQqhESKxFp+R4kjAdcIImAAxyDkRZvI45e9R2iakDi1lmslKo+Y7yY/DfB8zoE94CtJSRcgCSdh4irUucQKxVpdj8m5f8gtAusVMRqHufI1YFIE4FSDYwxHsyecyGwBvCGNwa3QN5/MN7PoOVbiMRo/y9DrESwOlUvltA+CG4HqwViS6/WOVTDp01DSjUwJu8FCKymOAkU80JgdXh/vGMI7ia09xrMziMwWRMwu0YS2nM2xr8i7tdWdU8TgVINTcEbQGGUnYVQ8HpcL29COZg9p0HhLMAHJj/8t39BOCn4V8X1+qruaSJQqgExxg8mp4KDcsLHxSuG/FchlEW4VmTZnQWY3Pvjdm1VPzQRKNWg2IGKOoPdxLV7z/Mu4YUFo/D9rOUpGhlNBEo1KAasWJ2yNkg4DRGJYwi5FRxgA6OJoDHRRKBUQ+L9GEK7YxxgR5Kvjm8MtszY+0XAahHfGFSd0kSgVJyZwEZCOfcS2jmC0M7hhLJvjdrhavJfpPxooZIErNS4xFl8haTLQBKi7HWC+wxEIsxYVgcsTQRKxZHxzcPsOQk874SrfYZ2gfdTzJ7TMd5Z5V8Q2FzxSUNZtR9oSe7x4BwJJJbZkQD2LkjKjfG9vqpzOqFMqTgxxo/ZexWUW5cpCAQx2TdAqx9KLxtppUAo2joCRa+V2isdbfyLMLlPgm8OYMB5GJJ8LZL2OBTOCD+hBDeCtIDE85HE06o0mU0dGDQRKBUvhd8AMYZ5ChjPJ0jSeX9sS5gA+c8TedSOgHNYtdcbNsYPoZ2AG7G1xHhnYLJvIjxnoWglWd8cTNZCSL0bK/F0xH1sta6lDiyaCJSKl8BaMDHa+40HAitLbZKkizCeD4pu2CXH8QvgAvdoTHB3lco9GOPH5D1dNBEtACaAsXeFwHoiT1zzwr57MO7Ruq5xExHPNYvvEZEtRctULhSR46McN05EVojIahG5LV7xKBVvxgQx/qUY3/xwjR4rDWJ2qtrBKr0mgFipSPqH4B7HH1VDbYQTgQW5D4fLPey9tlJj+Y0x4eap/JfCw0KNB/BDYAXRZy8DCMbzWYXnV41DvJ8IHjfGPBJtp4jYgKeBY4DNwFwR+dQY83uc41KqVoUKPoK8h4qeACwwPnCPBhOM8SobknBKua1itUDSHsOE8jB5T0LBW4RHEhUUt+BQ+DUm60Jo+R7hX6MofD+Cfy6xRyJF4oXAuiq+Rh2o6rtpaAiwumjtYkTkbeBkQBOBOmCECt6Bff+m3M3WOxPEDcZQvq8gARJOQeyxxuxLiSRQlg+Ca8E3G1wjgfCnf/xzMQVvQ3AXOHoWNU+V7ayuDGfRwjaqKYh3IrhGRC4E5gE3GWP2ltnfHthU4vvNwOGRTiQiE4GJAJmZFUx4UaqOGOOD3AeJerM2PsJNO1DcvIMNXGMg5Y7YJ/f9EF5k3kTZbwowBR8hrpEYE8BkXxMe/WM8gAH/fMIjlKpHEk6u9mvVgaVGfQQiMlNElkT4czLwLHAQMADYBjwa6RQRtkX8sTfGTDLGDDLGDMrIyKhJ2ErVHt/PlTho/83YFH3tA98s2DUC418S/WXGA4Rin7qoHITJewoKfyz69L//VyhA9CwSSwIkX4PYWlXjtepAVKMnAmPMmMocJyIvAJ9H2LUZ6Fji+w7A1prEpFSdMnlU62Zr8sN/ZV0EGTOQSCUbHH3AxEoELnAODA8LLXiNqvcDlFTUIW3rgCRfjyREHNuhGqm4NQ2JSFtjzLaib08FIn30mQt0F5EuwBbgbODceMWkVFWY4BZMwfsQ3Ay2Tkji6UjZdnN79wo6hCu6iB9T8B6SfHm5XWLvhrH3gMBSIjbxiIUknA3BbVT45BCLrRvS8t2iU9beZDV14IhniYmHRGSxiCwCRgE3AIhIOxGZCmCMCQDXANOAZcC7xpilcYxJqUoJ5T2N2TUO8l8A7yeQ/xxm11hCeS+WOk7s3cLJoNq/Sl7wzoi6V5o/BVYGULL2jxNwI83+g9hahoeo1iQZJZ6LWMmaBJqwuD0RGGMuiLJ9K3B8ie+nAlPjFYdSVWW8X0LeJEqPsy+a6Zv3EKHQTiTlZkQcAEjzJzF7zoBQHtVrnoleUlpsbSBjGng+x3g+Cg9PdQ1DEs8vfjoRWxuMrQME11Tj2i7E0acar1ONSX0PH1WqwTG5TwDR6v0YKHgl3Mnb4mVEnIitHaRPw3jeB8+HENhIOIlUsrkmsJpQzh3hqp+2LlD4JSbvWQisAhzgHoskX4OVeGbUU0jqbZi911KtRGTvXPXXqEZFq48qVYIxRePzYx8F/kWY/EnFW8RKwUq6GCv9M6TVj+DoT7gJpzLywfMRZvepmOxrMTm3QWA54X4BL3g/x+w5JeYII3H9CZo9CNIMJKnoTwJY7WLE4QDXSMRqXsk4VWMlxlRneFn9GjRokJk3b159h6EaIWMCmB19qdSneUlDWv0ccbUwYwxm9zHhyp21xdYJSZ8ec3UyYwLhIa2hveGS0Y4+hPImQd5ThJu39r+vBLC1QVq+i1jNqh2SMUEIbgEssLWP78ppqsZEZL4xZlDZ7do0pFQJInaM49CiyVgVMHnhYaARykKLSLVG8McU3AWBpRh7TyicFe7LMH7ENRISTkAkARE7uIaXepmVPBHjGorJeyk8AslKDY82Shhf7ZLSxhhMwRuQ/0x47oIxYKVhkm/ASjy1Ft6sqkuaCJQqQ1JuxGRdSsXt7RIuIRGNo2946GlNhnaWupyF8S+F7L+GF6cpKh1hCr+H3IehxeuIo0fklzr6Ic3/UztxACb3ASh4m1J9KaHtsO9uQqFdWMkTa+1aKv60j0CpMsQ5GJo9ROzPSRa4Roc/gUc7T9KlVL6foDJCkD85PG+gVP2gAjB7MVkXhvs44swENkHBFCJ3qHsh70lMqGw1GdWQaSJQKgIrYRykf1lUBrpsu7eAJCIpN8U8hzgOgZTrADe186vmhuAOotcPKgTvtFq4TmzG8ykxZ1OLVSdxqNqjiUCpKCx7JtLyM3AOA5xFfQFOcBwW7mS1d6r4HEmXIi3fBtdokBRizRkAN9i6UXryWIl97nHEbGYy+RjfnApjqrHQLmKuvGYKw53V6oChfQRKxSD2jkiLVzDB3eE2cCsDsbWO+RpTXHbaER5FIyng+6VMQbjiKwAusBIh6RpIOAc8UyB/UtEi9Qacg5GUm8G/GOOJ9dlNqN2mqChXcfTAeBOKiuJFOiC8yL06cGgiUKoSxJYOFSwPaUK54SUhPe+Eb5LixiScCf6VYPYRuTnFCUmXI8lX/LHATNKFmMQLikYkOZGiVc6M1RK4P0aQCUjCcdV6f1XiHg/7HohxgA1cR8c/DlVrtGlIqVpgQrmYPaeF1wU2+UAo/ARQMAX8PxG9Tb0QvF+UW2VMRML1f0osdSm29uAeS7jPoSxnuFnJMbiW3lF0YqVAs0cp3/dhAxKQtKdKxa0aPk0EStUCk/98URXQsm3ngUiHlxbaVenrSLMHwH0c4T6LRML9CS5wDkFavFxnE7qshGOQlu+EY5Hm4bWXE05F0j9CXEPrJAZVe7RpSKnaUPA2xYXpqqpEn4MxHkzBO+EniVB2eLZu0qXgPgERK1zbKO1BTPCm8ApmJgTOQZXquK5t4jgYSXu8zq/bUHmDfnyhACl29wE3w1oTgVI1ZEyoqA+gOgRJvDh8nlAeZs9ZRZPQiiazBXIw++4E7/8g7SlEwg/xYmsFCTqDtyFYmr2JJ1d8yW/ZG7AQUhwJXNBlBGd3Ho5NDoxGlwMjSqUaMBErXOytWi9OgoSTADC5j0BwA+VmNBsPFP4Ano9rFqiqdfP2rOHKXyazYO86giaE3wTJ8uXx/KqZ3Pnb2xwotdw0EShVGxLPoepDN+2QdCMijvCMYM9HRB+f78HkT65ZjKpWGWO4Z/H7eEPl/595Q35+3LWSBVnr6iGyqtNEoFQtkKSJYGtP+WRgEXkSmQVWGrK/QNv+OQOxBLfUOE5Ve5bkbCLPH23dCvAGfby/sQ4m+NWCeK5Z/A7Qs+jbNCDbGDMgwnHrgVzC8+YDkUqkKtXQiZUMLT8Ir1FQ8BaYnPBEsoSzwPjCcwuwE54ZLOFO4OaTECup6ATJRC8dUcRKie+bUFWypzAPK0YfgAG2e3PqLqAaiOdSlRP2fy0ijwKx/kVGGWN2xysWpeqCWMlIyo2QciPGhIo7dgFMyjVQ+E24vd/eBxz9So0sESsZ4xwGvtlEnXiWMCHCdlVf2iWkEYixVrSF0Ckp9iTEhiLuo4Yk/NN+FqBTDVWTIWU+KYrVvMJRPpJ6R3jtY5NP6WRgBysdSbqo9gNV1dYjtR2t3WlsyI88D8Rp2Tmr07A6jqp66qKPYASwwxizKsp+A0wXkfkiokXMVZMl9q5Iy3fBORRwFE0Yc4F7HJL+IWKl1neIqox/959Aos2FrUw/kNvm4NTMIfRu1qHS59qQt4tnV07nviUf8f6GOeQFqrH+dDXVaKlKEZkJtImw6+/GmE+KjnkWWG2MeTTKOdoZY7aKSCtgBnCtMea7CMdNBCYCZGZmDtywYUO141aqoTOh7PCEMis93P+gGqwtBVm8tOZrvtq+GH8oSNfkVlx80ChGte5TqYllIRPiviUfM23bQgImRNCEcFsOELiv/zkc2apXrcUabanKuK5ZLOFVO7YAA40xmytx/D1AnjHmkVjH6ZrFSqnGYvLqWby27lu8wfLDUN2Wg9ePuIZOyRm1cq1oiSDeTUNjgOXRkoCIJIlIyv6vgbHAkjjHpJRSNZLjK+CNdbO5Zu5L3DT/NWZsW4Q/VIm6UmX4QwHeXD87YhII7w/y+rrZNQ23QvHuLD4beKvkBhFpB0w2xhwPtAY+Knp8sgNvGmO+jHNMSilVbQv3ruf6ea8QNIbCoslk87PW8szK6bww9HLSXZUf5rshfzehGI0yQULM2ROte7X2xDURGGP+HGHbVuD4oq/XAv3jGYNSStWWPL+XG+a9SkGwdIHBgqCPQm82f1vwBv/sP4GPN81lY8Fu2iU055QOg6M27VgimAomEloxV7WrHVp0TimlKunzLfMJRlkuNGhCLMvZwlnfP44xEDBBbGLx/sY5nJE5lL/2PK5c53GnpAzcNgeeYOTKtXaxMbpN31p/H+WuE/crKKVUPfo1ax1vrf+BtXk7ae5M4ozMoYxu0xe7Zav4xWXMz1oXtT0fwk05wRJ5Ilg0CuiDTT/TLaUNJ7Q/rNTxNrH4S7fRPLHifxHP67RsTOh0RJXjrCpNBEqpRus/y7/gw02/UBj0Y4CNBbtZmbuNdzb8yDNDLsNtc1TpfFU9fj9v0M+Lq2eVSwQAp3c8nGxfPq+s/RZLhEAoiNOy47QcPDbwQtokpFXrmlWhiUAp1Sj9sGsFH276pdwnbU/Qx8rcbTy18ktuPnh8lc45rt0AZu9cVq6PoDK2ebPxBHwk2EsXJhQRLus2mjMyh/HNjqXs83vokpzBsPQe1XpqqQ5NBEqpRunVtd9EbcbxhQJ8tnke1/YYh8vmYJd3Hz/tXok/FKR/8050S4k0TxaGpnenfWIL1uftwh+jzlAkxhhsVvQR+2nORE7pGP81pyPRRKCUapTW5O6o4Ahhm2cvr637junbFmGJVbyQTPeUNtx/6Lksy9nCLm8ObRLSij+hPzvkL9z529ssyFqHXSwEIWCCZLhT2VKwN+oooL5pHXFaDfOW2zCjUkqpGkqwu8iNUa8nYIJMXj2L73Yuw1dmMtjvOZs56ZuHcFsOQoSwiQ2bWPyr/wSGZfTgiUEXs6Ugi6U5m3CIjcEtu7HVk8Vlc56PuFCNy3JwVY9ja/091hZdmEYp1SiNb38YTonext4pMYNvdv4e8cYdwmAweEI+CkMBCoKF5AY83PrrFJblhBcIap/YgrFt+9MxKZ3nVs3guVUzGZ7RkxS7m0SbC5flINHmItnu5h/9z+KwFl3i9l5rSp8IlFKN0oROR/DRprnk+AsImtJj/92WgxGtevH2hh+rdM7CkJ/nV83ksYEXIAiPLvucTzbPJRAKEsRgFwsLYXSbQzikeSYZrlSOyOiBo4E2Ce3XsKNTSqlqSnMm8eqwq7hn0XssztmE07KH2/Jdqfy976lszN9T4azesgzw4+4VDJ12J0k2J4WhYKnFaQJFCefrnUsZ07YfI6JUDg2Eguz15ZNgd5Jsd0e9XqjofLFWQqsNmgiUUo1Ont/LT7tXUhj0c1Pv8aTY3WzxZNHMkUTX5FaICM2dydSk+nJ+jCGk4XkDX5VLBIVBPy+s/ooPNv0cfoowhgHNO3F9rxPokdq2+Li5e9bw/KoZLM7eBECfZh2Y2H0MQ9O7VzveWDQRKKUaDWMMz6+ayZT1s7GJDYMhZAxdk1vxyGEXkOH+Y3GfLsmt6N2sA4uzN8VccrK6VuRuK/V9IBTk6rkvsmLfVgpLdE7Py1rLX35+nmeHXEbvZh34Yst8Hlz6aam+iyU5m/jbgje4vtfxnJZ5eK3Hqp3FSqlG48U1s3hz/ffFHbyeoI/CkJ+Vudu4dM5z5eYVPHjo+XRKSifR5oxyxuozxpR64vhq+xJW5W4vlQT28wR9/HvJh+QFvDxQJgns5w35eWz5F2T7Cmo9Vk0ESqlGwRPw8dra7yLeRIMmRI6/gJnbF5fanuZMZMrwa7l/wLn0S+tU65U+Z+9cxhdbFvDhxp95de23UYvLAWzM38N7G37CihGCANO3/VarMYI2DSmlGomFe9dji9Gp6gn6+GTTXDbl72Z+1lpcNgcntj+M0W0OYVhGD1IdCVw198WYN+uqCGH4269TcNkcmBJrF0TjsGxszs/CE6OoXWEowDbP3lqJryRNBEqpRsEXClDRB/pF2Rv4fd9m/KFwn8CS7E1MWvUVk4deTu9mHWjtbsaG/N1VHk0UTQhT6cTiCwXoktwKt+WI+FQD4YlpbROa10psJWnTkFKqUejdrEPxDT4aA6WO8QR97PBmc/vCtxARHjnsApo5EnCVGPdvFyvupSEshH5pnTil4+CYKchgGNu29tfy0kSglGoUMtypDEvvjiPGbOJIAibE7zmb2Zi/m8ykdN4/6iYmdhtD95Q2dExsyajWfTmp/SDscRrL7xAbzZyJ3HXI6SQ73NzSezwuq3y5a7fl4Ppex5PmTKz1GGr0zkTkTBFZKiIhERlUZt/tIrJaRFaISMQiGyLSQkRmiMiqor9r/5lHKdVk3N3vTA5KaU1CFUcB2cVixb6tAKQ6Ehjbrj+t3M3Y7s3mx10r+HDTz8WTxWpTit3N2Z2P4M3hfy1ed+CkDoN49LAL6NusI1L038Gp7Xnw0PM4I3NorccANe8jWAKcBjxfcqOI9Ca8cH0foB0wU0R6GFNusO5twFfGmAdE5Lai72+tYUxKqSYq2e7mlWFX8fPu1Uzd+ivf7VxWqTZ6QYoXnckqzOOiH58m25dPCIOf2p9jsP+aJ3YYyLU9jyu3r1/zTP7a6ziMMfRIbUeS3RWXGParUSIwxiwDyq3DCZwMvG2MKQTWichqYAjwU4TjRhZ9/SrwDZoIlFI1YInFsIweNHMm8u3O3yv1mqAJMbjlQQC8uvZbcv0eQrXUYRyNwZDr95SL45mV03lv40/FI6BCxnBe5yO5rNvRcSs1Ea8ekPbAnBLfby7aVlZrY8w2AGPMNhFpFe2EIjIRmAiQmZlZi6EqpRqjNbk7kErcy92Wg0sOGoW7qDnp8y0LqrzoTHUk2Jz0adax1LZ/Ln6fWduXlhs19Ma62ez15XNrn5PjEkuF6UVEZorIkgh/YkUUaRBXjdKrMWaSMWaQMWZQRkZGTU6llGoCku0upIJP0DaxuKzb0VzU9U/F2zzBwniHVmxcuwHFX2/I381X25dEnVX82Zb57PDmxCWOCp8IjDFjqnHezUDJVNcB2BrhuB0i0rboaaAtsLMa11JKqXKGZfQgRPQOXqdl55HDzmdoeo9S29skpLG5ICvq6xxiwxhDIMq5bWIVl7122xz4ggHsYsNnwqUlHJYNu9h49LALSrX9z9j2W8wO6UAoyK0L3uCIjJ6M7zCwVucTxGv46KfA2SLiEpEuQHfglyjHXVT09UXAJ3GKRynVxLhtTq7sPhZ3hKGYLstO/+adOLxl+Wqe53ceUdxxXJbTsnN65uFkJKSWG05qQ0hzJHJ192MZ3bovx7cbwH39z2HqqNu5vPsYeqa2o1tyG87pNJx3R9zAoKI+if3yAt5y6yaUFMLw+74tvLr2W86c/TiTV39VmX+GSqlRH4GInAo8CWQAX4jIQmPMscaYpSLyLvA7EACu3j9iSEQmA88ZY+YBDwDvisilwEbgzJrEo5RSJZ3TeTgJNgfPrJyOLxRARAiEQpzY/jBuOPiESANdOLnjYH7YtYJ5WWtKlXtw2xwclNyaq3ocyyUHHc3TK7/ky60Li9u8R7buzV97Hkcrd7Ny57yg61Fc0PWomLH2Sm1Pos1JQQWjnPwmCAZeW/sdnZIyOKZtv4r/ISogNanHXV8GDRpk5s2bV99hKKUOEEETYnXudnyhAF2TW1c4HDNkQny1fQlvrv+ebZ5sWriSmdDpCI5vN6DUamP+UIB9fg/JdjeuKE8RlVUY9HPc1/eTF2Od5bI6JaXz3ogbK328iMw3xgwqu11rDSmlGj2bWPRMbVfp4y2xOKZtvwo/bTssOy1dKTUNDwCXzcHjAy/iunkv4zfBCstlAGwq2IM36I/alFVZWmJCKaUaiP7NO/HeiBs5r/ORdE1uXfELDFgRmreqShOBUko1IBnuVK7qcSxvH3kdw8qMaCqrd7MOtVIQT5uGlFINyoa8XXy2ZT47vTl0SW7FSR0G1Vrzy4Hm8u5jWJC1LuJaBi7LwZU9xtbKdTQRKKUaBGMMjy77nI83zyVoQgRNCJdl58U1X3NL7/Gc3GFwfYdY53o368B9A87hrkXvuxQ/0AAABZNJREFUYEx4zQKnZcdguKPPqcVlMWpKE4FSqkF4f9PPfLp5XniBmSL71/d95PfP6ZzUiv7NO9VXePVmRKteTDv673y/cznbPNlkuFM4qlXvGncQl6SJQClV70ImxMtrvo66MldhyM9La2bx30EX13FkDYPTsnN0m75xO792Fiul6l1WYR77ylTiLOu3vRvrKJqmRxOBUqre2SyLiia32mphmKSKTBOBUqreNXcm0z6xRdT9FsKIVr3qMKKmRROBUqpBuKbnuIhr9QI4bXYu6jqyTuNpSjQRKKUahKNaHczNvU/EbXOQaHPhsGwk2lw0cyTyn4EX0SU56rpVqoZ01JBSqsE4ucNgxrbpz+xdy9hbmE/7xBYMTe+O3bLVd2iNmiYCpVSDkmB3MrZt//oOo0nRpiGllGriNBEopVQTp4lAKaWaOE0ESinVxB2QS1WKyC5gQxxOnQ7sjsN5a4vGVzMaX/U15NhA46usTsaYjLIbD8hEEC8iMi/Sep4NhcZXMxpf9TXk2EDjqyltGlJKqSZOE4FSSjVxmghKm1TfAVRA46sZja/6GnJsoPHViPYRKKVUE6dPBEop1cRpIlBKqSZOE0EZIjJAROaIyEIRmSciQ+o7ppJE5FoRWSEiS0XkofqOJxIRuVlEjIik13csJYnIwyKyXEQWichHIpJW3zEBiMi4ov+nq0XktvqOpyQR6SgiX4vIsqKfuf9v725CrKrDOI5/f8xkQTHZJoQmmISsrKwERQl7m14xbBcGhRQURolGUI5S1MqoqAZqp9NKiMiokNKsqN0MkRVWA724GbPQTRZIDUO/Fuc/cefca9PqPqfu81md/5k78GU4nOfcPzNzN0c3dSKpT9LnkvZGt9RJWijpjXLtTUpaHd1Ul4Og3bPA07avBJ4s60aQdD1wB7DM9qXA88FJbSSdD9wENPEDZg8Al9leBnwLjAT3IKkPeAW4DVgK3CVpaWzVHDPAo7YvAVYBDzWsb9ZmYDI64hRGgX22LwauoIGdOQjaGRgox2cDRwNb6h4EnrH9B4DtY8E9nbwIPEb1c2wU2+/bninLcWAwsqdYCXxv+7DtaeA1qmHfCLZ/sn2wHP9GdRM7L7ZqLkmDwFpgZ3RLnaQB4BpgF4Dtadu/xFa1y0HQbgvwnKQpqifu8KfGFkuANZImJH0iaUV0UCtJ64AfbX8Z3fIv3Ae8Fx1BdVOdalkfoWE32lmShoCrgInYkjYvUT18/Bkd0sFi4Djwatm62inpzOioup78YBpJHwCLOnxpOzAMPGJ7j6Q7qSb5jQ1p6wfOoXqLvgJ4XdJid/F3gOfp2wbc3K2WTv6pz/bb5TXbqbY8dnez7RTU4Vzj3k1JOgvYA2yx/Wt0zyxJtwPHbH8m6brong76geXAJtsTkkaBrcATsVlz5d8R1Eg6ASy0bUkCTtgemO/7ukHSPqqtoY/L+gdgle3joWFVy+XAh8DJcmqQalttpe2fw8JqJG0ANgLDtk/O9/ou9KwGnrJ9S1mPANjeERrWQtJpwF5gv+0XontaSdoB3EM12M+g2tZ90/bdoWGFpEXAuO2hsl4DbLW9NjSsJreG2h0Fri3HNwDfBbbUvUXVhKQlwAKa8R8NsX3I9rm2h8pFfwRY3rAhcCvwOLCuCUOg+BS4UNIFkhYA64F3gpv+Vh6GdgGTTRsCALZHbA+Wa2498FFThgBAuf6nJF1UTg0D3wQmddSTW0PzuB8YldQP/A48ENzTagwYk/QVMA1s6Oa20P/Ay8DpwIHq/sa47Y2RQbZnJD0M7Af6gDHbX0c21VxN9cR9SNIX5dw22+8GNv3XbAJ2l0F/GLg3uKdNbg2llFKPy62hlFLqcTkIUkqpx+UgSCmlHpeDIKWUelwOgpRS6nE5CFJKqcflIEgppR73F5H+kTbqO59KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "\n",
    "dbscan = DBSCAN(eps = 2, min_samples = 2)\n",
    "cls_assignment = dbscan.fit_predict(X)\n",
    "plt.scatter(X[:, 0], X[:, 1], s=65,c=cls_assignment)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course this just scratches the surface of clustering techniques.  For a great overview of the theory behind these and other clustering techniques, see the two Clustering chapters in [Data Mining: Concepts and Techniques, 3rd Edition](https://learning.oreilly.com/library/view/data-mining-concepts/9780123814791/).  For a guide and examples of how to implement the algorithms, scitkit has many great [clustering examples](https://scikit-learn.org/stable/auto_examples/index.html#cluster-examples) on its website.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Data Analysis Tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Natural Language Processing with nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[NLTK](https://www.nltk.org/) (natural language toolkit) has been the most common python library for natural language processing and text analysis, although other packages such as gensim and SpaCy are emerging as well.  To learn more, I'd recommend the Coursera course [Applied Text Mining in Python](https://www.coursera.org/learn/python-text-mining?specialization=data-science-python).  There are also many good online books on O'Reilly.  I'd recommend [Natural Language Process: Python and NLTK](https://learning.oreilly.com/library/view/natural-language-processing/9781787285101/), by Perkins, Mathur, Joshi, Hardeniya, Chopra as a good starting point to the main functionalities of NLTK.  [Applied Text Analysis with Python](https://learning.oreilly.com/library/view/applied-text-analysis/9781491963036/) by Bengfort, Bilbro, and Ojeda is also very good and goes beyond NLTK to cover the broader landscape of python text analytics tools (such as gensim for topic modeling and word embedding models), but requires more advanced python background from the reader.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Analysis with networkx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Newtorkx has many powerful tools for the analysis, visualization, and any types of networks.  For a good primer I'd again recommend the coursera course [Applied Social Network Analysis in Python](https://www.coursera.org/learn/python-text-mining?specialization=data-science-python), or see the [documentation](https://networkx.github.io/documentation/stable/tutorial.html) for a good tutorial.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Modeling and Time Series Analysis with statsmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [statsmodel](https://www.statsmodels.org/stable/index.html) package has a wealth of statistical analysis tools, including regression models, Analysis of Variance (ANOVA), time series analysis (AR, ARIMA, etc), Kernel density estimation, and more.  \n",
    "\n",
    "Here is a good example using statsmodel for ARIMA time series analysis: https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [scipy](https://docs.scipy.org/doc/scipy/reference/optimize.html) libary has many built in methods for general optimization problems.  I also like the [cvxpy](https://www.cvxpy.org/) package since it has some added functionality.  For example, cvxpy makes it easier to conduct integer or binary programming where decision variables must be integers (or must be 0 or 1). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, most typical simulation in python is done with for loops.  However, there are some packages for more intense simulation work, such as [SimPy](https://realpython.com/simpy-simulating-with-python/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Association Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Association analysis, which aims to find correlations and linkages in large datasets, is arguably an under-utilized data mining tool since it tends to fall outside the bounds of machine learning, which is all anyone cares about these days.  Here's a good article on implementing association analysis with teh mlextend library: https://pbpython.com/market-basket-analysis.html.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}