5. Scaling and Outliers.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scaling and Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "***Scaling***, ***standardizing*** and ***normalizing*** are used somewhat interchangeably, even though they are ***not the same thing***.\n",
    "\n",
    "True, ***standardizing*** and ***normalizing*** can be seen as particular ways of ***scaling***. Let's sort the differences between all them before proceeding.\n",
    "\n",
    "### 1.1 Scaling vs Standardizing vs Normalizing\n",
    "\n",
    "According to the implementations in Scikit-Learn library:\n",
    "\n",
    "1. ***Scaling (MinMaxScaler)***:\n",
    "\n",
    "    Transforms features by scaling each feature to a given range.\n",
    "\n",
    "    This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one.\n",
    "\n",
    "\n",
    "2. ***Standardizing (StandardScaler)***:\n",
    "\n",
    "    Standardize features by removing the mean and ***scaling to unit variance\n",
    "\n",
    "    The standard score of a sample x is calculated as:\n",
    "\n",
    "        z = (x - u) / s\n",
    "\n",
    "    where u is the mean of the training samples or zero if with_mean=False, and s is the standard deviation of the training samples or one if with_std=False.\n",
    "        \n",
    "        \n",
    "3. ***Normalizing (Normalizer)***:\n",
    "\n",
    "    Normalize samples individually to unit norm.\n",
    "\n",
    "    Each sample (i.e. each row of the data matrix) with at least one non zero component is rescaled independently of other samples so that its norm (l1 or l2) equals one.\n",
    "        \n",
    "The main difference is: ***scaling*** and ***standardizing*** operate on ***features*** (columns of your dataset), while ***normalizing*** operates on ***individual samples*** (rows on your dataset).\n",
    "\n",
    "Even though it is very common for people to say *\"you need to normalize on your features\"*, they usually mean ***standardize your features***.\n",
    "\n",
    "Why ***standardize*** instead of simply ***scaling***? It turns out, ***scaling*** is fine when you have features with a ***limited range***, like ***age*** (0-120) or ***pixel values*** (0-255), but it is likely a bad choice whenever your features may have values ***very far apart***, like ***salaries*** (maybe not yours and mine, but think of some CEOs...).\n",
    "\n",
    "### 1.2 Why Scaling?\n",
    "\n",
    "Most Machine Learning algorithms use ***gradient descent*** to learn the weights of the model. You'll see in the next lesson that ***scaling*** the features makes a ***BIG*** difference in performance.\n",
    "\n",
    "It is also important for other techniques, like ***Principal Component Analysis (PCA)*** for dimensionality reduction, and for identifying ***outliers***.\n",
    "\n",
    "![](https://imgs.xkcd.com/comics/log_scale.png)\n",
    "<center>Source: <a href=\"https://xkcd.com/1162/\">XKCD</a></center>\n",
    "\n",
    "### 1.3 Outliers\n",
    "\n",
    "What is an ***outlier***? It could be defined in several ways:\n",
    "\n",
    "- a point that is distant from the others (again, think of salaries, everyone is between USD 30k and 100k per year, and a CEO is making 50M!)\n",
    "- a point that is distinct from the others (think of black sheep in a flock of white sheep)\n",
    "- an error of measurement (think of someone listed as being 450 years old)\n",
    "- an anomaly / fraud (think of finding the purchase of a USD 10k Rolex on your credit card bill)\n",
    "\n",
    "The last case, anomalies / frauds, is a special case where ***your goal is to detect the outlier***.\n",
    "\n",
    "For now, let's focus on the other cases: in all of them, the ***presence of an outlier*** may ***hurt your model***, impacting its training and making its predictions less useful.\n",
    "\n",
    "So, how do you ***detect*** and ***remove or fence outliers***?\n",
    "\n",
    "#### 1.3.1 Tukey's Fences\n",
    "\n",
    "This is a very straightforward way of detecting ***possible*** outliers based on the ***InterQuartile Range (IQR)***.\n",
    "\n",
    "It defines a ***lower*** and an ***upper fence***, which are given by:\n",
    "\n",
    "$$\n",
    "IQR = Q_3 - Q_1\n",
    "\\\\\n",
    "lower fence = Q1 - k * IQR\n",
    "\\\\\n",
    "upper fence = Q3 + k * IQR\n",
    "$$\n",
    "\n",
    "Typical values for ***k*** are 1.5 (outlier) and 3.0 (far out).\n",
    "\n",
    "The plot below illustrates this:\n",
    "\n",
    "![](http://www.physics.csbsju.edu/stats/complex.box.defs.gif)\n",
    "\n",
    "<center>Source: <a href=\"http://www.physics.csbsju.edu/stats/box2.html\">Box Plot: Display of Distribution</a></center>\n",
    "\n",
    "Although easy to compute, ***Tukey's Fences*** only consider the distribution of a ***single feature*** to assess values as outliers or not.\n",
    "\n",
    "What if we wanted to check if a value can be considered an outlier, ***given its many features***?\n",
    "\n",
    "#### 1.3.2 Mahalanobis Distance\n",
    "\n",
    "In a single dimension, we can easily compute how far (in standard deviations) a point is from the mean. This is what ***standardization*** does for a single feature (refer to the previous lesson for more details).\n",
    "\n",
    "Mahalanobis Distance is the generalization of the same idea in multiple dimensions.\n",
    "\n",
    "The ***Mahalanobis Distance*** is given by:\n",
    "\n",
    "$$\n",
    "\\large\\sqrt{(x_1 - x_2)\\ S^{-1}\\ (x_1 - x_2)}\n",
    "$$\n",
    "\n",
    "where ***S*** is the covariance matrix.\n",
    "\n",
    "If we ***standardize*** all features, the ***Mahalanobis Distance*** corresponds to the distance from the origin:\n",
    "\n",
    "$$\n",
    "\\large\\sqrt{x\\ S^{-1}\\ x}\n",
    "$$\n",
    "\n",
    "Knowing the distance is not enough, though. To determine if a given point is an ***outlier*** or not, we need to compare its computed distance to the ***cumulative chi-squared distribution*** using the ***number of features*** as ***degrees of freedom***. If it falls ***above a threshold***, like 99.9%, it is considered an ***outlier***.\n",
    "\n",
    "```python\n",
    "from scipy.stats import chi2\n",
    "\n",
    "chi2.cdf(mahalanobis_distance, df=n_features)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "There are 200 data points (in blue).\n",
    "\n",
    "The controls below allow you:\n",
    "\n",
    "- change the ***scaling method***\n",
    "    - obs.: MinMaxScaling is configured to scale features in [-5, 5] range\n",
    "- include ***ONE outlier*** (single red point)\n",
    "- plot ***Tukey's fences*** on horizontal and vertical axes (k = 1.5)\n",
    "- plot the ***Chi-Squared Probabilities*** contour plot for ***Mahalanobis Distance*** (under StandardScaling only!)\n",
    "- choose a ***threshold for Chi-Sq.Prob.*** between 99.1% and 99.9% for (under StandardScaling only!)\n",
    "\n",
    "Use the controls to play with different configurations and answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.feature.Scaling import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data()\n",
    "mysc = plotScaling(X, outlier=(-9, 6))\n",
    "vb = VBox(build_figure(mysc), layout={'align_items': 'center'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2be22d1a67d240c5a405775f9d9bf6d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'mode': 'markers',\n",
       "              'showlegend': False,\n",
       "            …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. Using ***no scaling***, turn ***Tukey's fences*** on:\n",
    "    - how many ***outlier candidates*** you found on the horizontal axis (X1 feature)?\n",
    "    - how many ***outlier candidates*** you found on the vertical axis (X2 feature)?\n",
    "    - is there any point that is an ***outlier candidate*** on both features? If so, would you consider it an outlier or not? Why?\n",
    "    - how do you like Tukey's method for outlier detection?\n",
    "\n",
    "\n",
    "2. Include the ***outlier*** to the same configuration above:\n",
    "    - is the ***red point*** an outlier according to ***Tukey's method***?\n",
    "    - how do you compare the ***red point*** to any outliers you found in question 1?\n",
    "\n",
    "\n",
    "3. Using ***MinMax Scaling***, make all boxes ***unchecked***:\n",
    "    - take note of the general position of the blue points\n",
    "    - include the ***outlier*** - what happens to the blue points? Why?\n",
    "    - add ***Tukey's Fences*** - do you see any differences?\n",
    "\n",
    "\n",
    "4. Using ***Standard Scaling***, make all boxes ***unchecked***:\n",
    "    - take note of the general position of the blue points\n",
    "    - include the ***outlier*** - what happens to the blue points? Why?\n",
    "    - how is this different from what happened using ***MinMax Scaling***? Why?\n",
    "    - add ***Tukey's Fences*** - do you see any differences?\n",
    "\n",
    "\n",
    "5. Using ***Standard Scaling*** and check ONLY ***ChiSq.Prob. for L2 Norm***:\n",
    "    - are there any points outside the 90% probability circle (dashed circle)?\n",
    "    - include the ***outlier*** - is it outside the 90% probability circle?\n",
    "    - change the ***probability threshold*** to different values and observe how the circle grows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scikit-Learn\n",
    "\n",
    "[MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)\n",
    "\n",
    "[StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
    "\n",
    "[Normalizer](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Normalizer.html)\n",
    "\n",
    "[Comparison of the effect of different scalers on data with outliers](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[About Feature Scaling and Normalization](https://sebastianraschka.com/Articles/2014_about_feature_scaling.html)\n",
    "\n",
    "[Outlier Detection with Isolation Forest](https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}