Outliers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dealing with Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes outliers can mess up an analysis; you usually don't want a handful of data points to skew the overall results. Let's revisit our example of income data, with Donald Trump thrown in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAENCAYAAAD6/JlzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEwpJREFUeJzt3X+s3XV9x/HnCypFpGuqC2W26HBafhjUNbO6uWV3swNx\nGxTdCGosCDNLYNNsi0qXLZQlS2U/MjQbJGwKxWhI1QzIRECGV6MT7ZhQpFDqNpB2comiiEvYCnnv\nj/MtO17vp72cc3rPaXk+kpt8z/t8vt/7/uTee17n+/2e7/emqpAkaS6HjbsBSdLkMiQkSU2GhCSp\nyZCQJDUZEpKkJkNCktS035BI8pEkM0m29dWWJbk1yY4ktyRZ2vfchiQ7k9yX5NS++uok25I8kOTy\nvvoRSa7r1vlKkpeMcoKSpMHNZ0/iauC0WbWLgduq6gTgdmADQJKTgbOBk4DTgSuSpFvnSuCCqloF\nrEqyd5sXAI9V1SuAy4G/GGI+kqQR2m9IVNWXgO/NKp8JbO6WNwPruuUzgOuq6qmqehDYCaxJciyw\npKq2duOu7Vunf1ufAt44wDwkSQfAoOckjqmqGYCqegQ4pquvAB7uG7e7q60AdvXVd3W1H1mnqp4G\nvp/khQP2JUkaoVGduB7lvT2y/yGSpIWwaMD1ZpIsr6qZ7lDSo119N3Bc37iVXa1V71/nv5IcDvxE\nVT021zdN4o2mJGkAVTXQG/D57kmEH32HfyNwXrd8LnBDX/2c7hNLxwMvB77WHZJ6PMma7kT2+lnr\nnNst/za9E+FNVXVIfq1b9056p2mq7+sHLF589Nh7G9XXJZdcMvYenJvzey7Obxj73ZNI8glgCnhR\nkm8BlwAfBD6Z5HzgIXqfaKKqtifZAmwH9gAX1v93eBFwDXAkcFNV3dzVPwJ8LMlO4LvAOUPNSJI0\nMvsNiap6e+OptY3xm4BNc9TvBE6Zo/4/dCEjSZosXnE9UU4cdwMH1NTU1LhbOGAO5bmB83suy7DH\nqxZSkjqY+n02zjprPddfv5be6Zq9nmDx4hfz5JNPjKstSYeAJNQBPnEtSXoOMiQkSU2GhCSpyZCQ\nJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlS\nkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZ\nEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNQ4VEkg1J7k2yLcnHkxyRZFmSW5PsSHJLkqWzxu9M\ncl+SU/vqq7ttPJDk8mF6kiSNzsAhkeSlwLuBn62qVwGLgLcBFwO3VdUJwO3Ahm78ycDZwEnA6cAV\nSdJt7krggqpaBaxKctqgfUmSRmeYPYkfAP8LvCDJIuD5wG7gTGBzN2YzsK5bPgO4rqqeqqoHgZ3A\nmiTHAkuqams37tq+dSRJYzRwSFTV94C/Br5FLxwer6rbgOVVNdONeQQ4pltlBfBw3yZ2d7UVwK6+\n+q6uJkkas0WDrpjkZcAfAC8FHgc+meQdQM0aOvvxUDZu3PjM8tTUFFNTU6PcvCQd9Kanp5menh7J\ntgYOCeDngC9X1WMASf4R+AVgJsnyqprpDiU92o3fDRzXt/7Krtaqz6k/JCRJP272G+hLL7104G0N\nc05iB/D6JEd2J6DfCGwHbgTO68acC9zQLd8InNN9Aup44OXA17pDUo8nWdNtZ33fOpKkMRp4T6Kq\n7k5yLXAn8DTwdeAqYAmwJcn5wEP0PtFEVW1PsoVekOwBLqyqvYeiLgKuAY4EbqqqmwftS5I0OsMc\nbqKq/hL4y1nlx4C1jfGbgE1z1O8EThmmF0nS6HnFtSSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKT\nISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNRkS\nkqQmQ0KS1GRISJKaDAlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJ\najIkJElNhoQkqcmQkCQ1GRKSpKahQiLJ0iSfTHJfknuTvC7JsiS3JtmR5JYkS/vGb0iysxt/al99\ndZJtSR5IcvkwPUmSRmfYPYkPATdV1UnAq4H7gYuB26rqBOB2YANAkpOBs4GTgNOBK5Kk286VwAVV\ntQpYleS0IfuSJI3AwCGR5CeAX6qqqwGq6qmqehw4E9jcDdsMrOuWzwCu68Y9COwE1iQ5FlhSVVu7\ncdf2rSNJGqNh9iSOB76T5Ook/5bkqiRHAcuragagqh4BjunGrwAe7lt/d1dbAezqq+/qapKkMRsm\nJBYBq4G/q6rVwH/TO9RUs8bNfixJOkgsGmLdXcDDVfWv3eNP0wuJmSTLq2qmO5T0aPf8buC4vvVX\ndrVWfU4bN258ZnlqaoqpqakhpiBJh57p6Wmmp6dHsq1UDf5GP8kXgHdX1QNJLgGO6p56rKouS/IB\nYFlVXdyduP448Dp6h5M+B7yiqirJHcB7gK3AZ4APV9XNc3y/GqbfSXbWWeu5/vq1wPq+6hMsXvxi\nnnzyiXG1JekQkISqyv5H/rhh9iSg98L+8STPA/4DeBdwOLAlyfnAQ/Q+0URVbU+yBdgO7AEu7HvF\nvwi4BjiS3qelfiwgJEkLb6iQqKq7gdfO8dTaxvhNwKY56ncCpwzTiyRp9LziWpLUZEhIkpoMCUlS\nkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZ\nEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEh\nSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqGjokkhyW5N+S3Ng9Xpbk1iQ7\nktySZGnf2A1Jdia5L8mpffXVSbYleSDJ5cP2JEkajVHsSbwX2N73+GLgtqo6Abgd2ACQ5GTgbOAk\n4HTgiiTp1rkSuKCqVgGrkpw2gr4kSUMaKiSSrATeDPxDX/lMYHO3vBlY1y2fAVxXVU9V1YPATmBN\nkmOBJVW1tRt3bd86kqQxGnZP4m+A9wHVV1teVTMAVfUIcExXXwE83Ddud1dbAezqq+/qapKkMRs4\nJJL8OjBTVXcB2cfQ2sdzkqQJtmiIdd8AnJHkzcDzgSVJPgY8kmR5Vc10h5Ie7cbvBo7rW39lV2vV\n57Rx48ZnlqemppiamhpiCpJ06JmenmZ6enok20rV8G/0k/wy8EdVdUaSvwC+W1WXJfkAsKyqLu5O\nXH8ceB29w0mfA15RVZXkDuA9wFbgM8CHq+rmOb5PjaLfSXTWWeu5/vq1wPq+6hMsXvxinnzyiXG1\nJekQkISq2tcRn6Zh9iRaPghsSXI+8BC9TzRRVduTbKH3Sag9wIV9r/gXAdcARwI3zRUQkqSFN5KQ\nqKovAF/olh8D1jbGbQI2zVG/EzhlFL1IkkbHK64lSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJ\nSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAk\nNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKT\nISFJajIkJElNhoQkqcmQkCQ1GRKSpKaBQyLJyiS3J7k3yT1J3tPVlyW5NcmOJLckWdq3zoYkO5Pc\nl+TUvvrqJNuSPJDk8uGmJEkalWH2JJ4C/rCqXgn8PHBRkhOBi4HbquoE4HZgA0CSk4GzgZOA04Er\nkqTb1pXABVW1CliV5LQh+pIkjcjAIVFVj1TVXd3yD4H7gJXAmcDmbthmYF23fAZwXVU9VVUPAjuB\nNUmOBZZU1dZu3LV960iSxmgk5ySS/DTwGuAOYHlVzUAvSIBjumErgIf7Vtvd1VYAu/rqu7qaJGnM\nFg27gSRHA58C3ltVP0xSs4bMfjyUjRs3PrM8NTXF1NTUKDcvSQe96elppqenR7KtoUIiySJ6AfGx\nqrqhK88kWV5VM92hpEe7+m7guL7VV3a1Vn1O/SEhSfpxs99AX3rppQNva9jDTR8FtlfVh/pqNwLn\ndcvnAjf01c9JckSS44GXA1/rDkk9nmRNdyJ7fd86kqQxGnhPIskbgHcA9yT5Or3DSn8MXAZsSXI+\n8BC9TzRRVduTbAG2A3uAC6tq76Goi4BrgCOBm6rq5kH7kiSNzsAhUVVfBg5vPL22sc4mYNMc9TuB\nUwbtRZJ0YHjFtSSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKS\npCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlq\nMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpKaJ\nCYkkb0pyf5IHknxg3P1IkiYkJJIcBvwtcBrwSuBtSU4cb1fjcN+4Gzigpqenx93CAXMozw2c33PZ\nRIQEsAbYWVUPVdUe4DrgzDH3NAb3j7uBA+pQ/kM8lOcGzu+5bFJCYgXwcN/jXV1NkjRGi8bdgHoW\nL34eixZ9haOO+s2+6lM8/fTzxtaTJKWqxt0DSV4PbKyqN3WPLwaqqi6bNW78zUrSQaiqMsh6kxIS\nhwM7gDcC3wa+Brytqg7tM7mSNOEm4nBTVT2d5PeAW+mdJ/mIASFJ4zcRexKSpMk0KZ9umlOSZUlu\nTbIjyS1Jls4xZmWS25Pcm+SeJO8ZR6/PxnwuHEzy4SQ7k9yV5DUL3eOg9je3JG9Pcnf39aUkp4yj\nz0HN96LPJK9NsifJWxayv2HN83dzKsnXk3wjyecXusdhzOP380VJPtv93d2T5LwxtDmQJB9JMpNk\n2z7GPPvXlaqa2C/gMuD93fIHgA/OMeZY4DXd8tH0zm2cOO7e9zGnw4BvAi8FngfcNbtf4HTgM93y\n64A7xt33COf2emBpt/ymg2Vu851f37h/Bv4JeMu4+x7xz28pcC+wonv8k+Pue8TzuwTYtHduwHeB\nRePufZ7z+0XgNcC2xvMDva5M9J4EvQvqNnfLm4F1swdU1SNVdVe3/EN6ly1P8jUW87lw8EzgWoCq\n+iqwNMnyhW1zIPudW1XdUVWPdw/vYLJ/VrPN96LP3wc+BTy6kM2NwHzm93bg01W1G6CqvrPAPQ5j\nPvN7BFjSLS8BvltVTy1gjwOrqi8B39vHkIFeVyY9JI6pqhnohQFwzL4GJ/lpekn61QPe2eDmc+Hg\n7DG75xgziZ7tRZG/A3z2gHY0WvudX5IXA+uq6kpgoI8cjtF8fn6rgBcm+XySrUneuWDdDW8+8/t7\n4JVJ/gu4G3jvAvW2EAZ6XRn7p5uSfA7oT7MABfzJHMObZ9mTHE3v3dt7uz0KTbAkvwK8i94u8qHk\ncnqHRvc62IJifxYBq4FfBV4AfCXJV6rqm+Nta2Q2AHdX1a8k+Rngc0le9Vx+TRl7SFTVr7We607C\nLK+qmSTH0th9T7KIXkB8rKpuOECtjspu4CV9j1d2tdljjtvPmEk0n7mR5FXAVcCbqmpfu8eTZj7z\n+znguiShd0z79CR7qurGBepxGPOZ3y7gO1X1JPBkki8Cr6Z3rH/SzWd+bwD+HKCq/j3JfwInAv+6\nIB0eWAO9rkz64aYbgfO65XOBVgB8FNheVR9aiKaGtBV4eZKXJjkCOIfePPvdCKyHZ65G//7ew24T\nbr9zS/IS4NPAO6vq38fQ4zD2O7+qeln3dTy9Ny4XHiQBAfP73bwB+MUkhyc5it4J0IPlmqb5zO8+\nYC1Ad7x+FfAfC9rlcEJ773Wg15Wx70nsx2XAliTnAw8BZwMk+Sng76vqN5K8AXgHcE+Sr9M7JPXH\nVXXzuJrel2pcOJjkd3tP11VVdVOSNyf5JvDf9A7LTLz5zA34U+CFwBXdu+09VbVmfF3P3zzn9yOr\nLHiTQ5jn7+b9SW4BtgFPA1dV1fYxtj1v8/z5bQKuTnI3vRfb91fVY+Prev6SfAKYAl6U5Fv0Pql1\nBEO+rngxnSSpadIPN0mSxsiQkCQ1GRKSpCZDQpLUZEhI0oSaz037+sa+JMlt3c0zb++u/h+aISFJ\nk+tq4LR5jv0r4JqqejXwZ8AHR9GAISFJE2qum/YleVl3O/OtSb6QZFX31MnA57v1ppn75pPPmiEh\nSQeXq4Dfq6rXAu8DruzqdwFvAej+j8nRSZYN+80m/YprSVInyQuAXwA+2d2xAHr/GwN6gfG33T9K\n+iK9+zI9Pez3NCQk6eBxGPC9qlo9+4mq+jbwVngmTN5aVT8YxTeUJE2uZ27aV1VPAP+Z5LeeebJ3\nV+W9/3p1797FBno3Ph2aISFJE6q7ad+/AKuSfCvJu+jd0PSC7v9UfwM4oxs+BexIcj+9f9D25yPp\nwRv8SZJa3JOQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqen/APZqLdK1DKDGAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa797da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "\n",
    "incomes = np.random.normal(27000, 15000, 10000)\n",
    "incomes = np.append(incomes, [1000000000])\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.hist(incomes, 50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not very helpful to look at. One billionaire ended up squeezing everybody else into a single line in my histogram. Plus it skewed my mean income significantly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "127148.50796177129"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "incomes.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to dig into what is causing your outliers, and understand where they are coming from. You also need to think about whether removing them is a valid thing to do, given the spirit of what it is you're trying to analyze. If I know I want to understand more about the incomes of \"typical Americans\", filtering out billionaires seems like a legitimate thing to do.\n",
    "\n",
    "Here's something a little more robust than filtering out billionaires - it filters out anything beyond two standard deviations of the median value in the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEACAYAAABYq7oeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFghJREFUeJzt3W2MXOd53vH/RdGULTtm6DTkGqIjypFfqLap7cZUUMXo\nNC+05ACSPsk02kayEhSoHNgoClek04LrfIhNAa2dolCAIK7MunJYKokruqglSpUmRYtYiiMpEkSa\nYe2QpplwXDuKAMOFIFl3P8yhNNwzFGeXszszu/8fMOCZh8/MuWd3Z645zznPOakqJEkatG7SBUiS\npo/hIElqMRwkSS2GgySpxXCQJLUYDpKklguGQ5K3J3kiyePNv88l+WiSTUkOJzmW5IEkGwcesyfJ\n8SRHk+xc3pcgSRq3LGaeQ5J1wLeBa4BfA75XVXcmuQPYVFW7k1wN3AO8F9gKPAS8rZxQIUkzY7HD\nSr8AfKOqTgE3Avub9v3ATc3yDcCBqnqxqk4Ax4EdY6hVkrRCFhsOHwS+2CxvqaoeQFWdATY37ZcD\npwYec7ppkyTNiJHDIclr6G8V3Ns0LRwmcthIklaJ9Yvoez3wp1X13eZ+L8mWquolmQO+07SfBt4y\n8LitTds5khgmkrQEVZXlXsdihpU+BPzewP1DwK3N8i3AfQPtu5JsSHIlcBXw2LAnrKqZve3du3fi\nNVj/5OtYi/XPcu2rof6VMtKWQ5LL6O+M/mcDzfuAg0luA04CNwNU1ZEkB4EjwAvA7bWSr0iSdNFG\nCoeq+gHw4wva/pp+YAzr/yngUxddnSRpIpwhvUSdTmfSJVwU65+sWa5/lmuH2a9/pSxqEtxYV5w4\n2iRJi5SEmrId0pKkNcJwkCS1GA6SpBbDQZLUYjhIkloMB0lSi+EgSWoxHCRJLYaDJKnFcJAktRgO\nkqQWw0Ezb25uG0nOuc3NbZt0WdJM88R7mnlJaF+lNit6YRRppXjiPUnSxBgOkqQWw0GS1GI4SJJa\nDAdJUovhIElqMRwkSS2GgySpZaRwSLIxyb1JjiZ5Jsk1STYlOZzkWJIHkmwc6L8nyfGm/87lK1+6\neMNmWDvLWmvdSDOkk3we+KOqujvJeuD1wCeA71XVnUnuADZV1e4kVwP3AO8FtgIPAW9bOB3aGdIa\nl4udIT388Yt7DmmlTM0M6SRvBN5XVXcDVNWLVfUccCOwv+m2H7ipWb4BOND0OwEcB3aMu3BJ0vIZ\nZVjpSuC7Se5O8niS30lyGbClqnoAVXUG2Nz0vxw4NfD4002bNHHDhpAkta0fsc97gI9U1deSfAbY\nTXs7fNHb3/Pz8y8vdzodOp3OYp9CWpRe7yTDhqCkadXtdul2uyu+3gvuc0iyBfjjqnprc/9n6YfD\nTwKdquolmQMeqartSXYDVVX7mv73A3ur6tEFz+s+B43FYvY5nK+v+xw0K6Zmn0MzdHQqydubpp8H\nngEOAbc2bbcA9zXLh4BdSTYkuRK4CnhsnEVLkpbXKMNKAB8F7knyGuCbwIeBS4CDSW4DTgI3A1TV\nkSQHgSPAC8DtbiJI0mzxYj+aecOHil4LPH+eRzispNm1UsNKo245SDPmec73gS/pwjx9hnRelzpr\nWmuW4SCd19mtj1duvd4ZT7WhNcF9Dpp5iz08dbn6+veslTA1h7JKktYew0GS1GI4SJJaDAdJUovh\nIElqMRwkSS2GgySpxXCQJLUYDpKkFsNBktRiOEiSWgwHSVKL4SBJajEcNDPm5rYNPV22pPHzlN2a\nGcNPzQ3LeRpuT9mtaeMpuyVJE2M4SJJaDAdJUovhIElqGSkckpxI8mdJnkjyWNO2KcnhJMeSPJBk\n40D/PUmOJzmaZOdyFS9JWh6jbjm8BHSq6t1VtaNp2w08VFXvAB4G9gAkuRq4GdgOXA/cFY83lKSZ\nMmo4ZEjfG4H9zfJ+4KZm+QbgQFW9WFUngOPADqRV7dKhczDm5rZNujBpSUYNhwIeTPInSX61adtS\nVT2AqjoDbG7aLwdODTz2dNMmrWLP03+bnHvr9U5OtCppqdaP2O/aqvqrJD8OHE5yjPZMoEXPAJqf\nn395udPp0Ol0FvsUkrSqdbtdut3uiq930TOkk+wFvg/8Kv39EL0kc8AjVbU9yW6gqmpf0/9+YG9V\nPbrgeZwhrUWZ9hnSzpzWSpiaGdJJLkvyhmb59cBO4GngEHBr0+0W4L5m+RCwK8mGJFcCVwGPjblu\nSdIyGmVYaQvwpSTV9L+nqg4n+RpwMMltwEn6RyhRVUeSHASOAC8At7uJIEmzxRPvaWY4rCRN0bCS\nJGntMRwkSS2Gg6bSsAv7SFo57nPQVBq+f2Ea9iO4z0GT5T4HSdLEGA6SpBbDQZLUYjhIkloMB0lS\ni+EgSWoxHKRl1b4IkBcA0ixwnoOm0mqa5zDsOfzb11I5z0GSNDGGgySpxXCQJLUYDpKkFsNBktRi\nOEiSWgwHSVKL4aAVMeziPUm45JLXD22XNFlOgtOKGD6pDaZjspqT4DQ7nAQnSZoYw0GS1DJyOCRZ\nl+TxJIea+5uSHE5yLMkDSTYO9N2T5HiSo0l2LkfhkqTls5gth48BRwbu7wYeqqp3AA8DewCSXA3c\nDGwHrgfuinsYJWmmjBQOSbYCHwB+d6D5RmB/s7wfuKlZvgE4UFUvVtUJ4DiwYyzVSpJWxKhbDp8B\nPs65h11sqaoeQFWdATY37ZcDpwb6nW7aJAHDrvHgdR40bdZfqEOSXwJ6VfVkks6rdF30sXnz8/Mv\nL3c6HTqdV3t6abV4nmFvl17P0Ve1dbtdut3uiq/3gvMckvwm8E+AF4HXAT8CfAn4aaBTVb0kc8Aj\nVbU9yW6gqmpf8/j7gb1V9eiC53WewxriPIfR+vqe0IVMzTyHqvpEVf1EVb0V2AU8XFX/FPgycGvT\n7Rbgvmb5ELAryYYkVwJXAY+NvXJJ0rK54LDSq/g0cDDJbcBJ+kcoUVVHkhykf2TTC8DtbiJI0mzx\n9BlaEQ4rjdbX94QuZGqGlSRJa4/hIElqMRwkSS2GgySpxXCQJLUYDpKkFsNBktRiOEiSWgwHSVKL\n4SBJajEcJEkthoPGbm5uW+tCNpJmiyfe09gNP8meJ94bpa/vCV2IJ96TJE2M4SBJajEcpKlxaWtf\nzdzctkkXpTXKfQ4aO/c5jHd9vk80yH0OkqSJMRwkSS2GgySpxXCQJLUYDpKkFsNBktRywXBIcmmS\nR5M8keSZJL/ZtG9KcjjJsSQPJNk48Jg9SY4nOZpk53K+AEnS+F0wHKrqeeAfVdW7gZ8Cfi7JtcBu\n4KGqegfwMLAHIMnVwM3AduB64K545rVVadgJ9vxVS6vDSMNKVfWDZvHS5jHPAjcC+5v2/cBNzfIN\nwIGqerGqTgDHgR3jKljTo9c7SX/S1sKbpFk3UjgkWZfkCeAM0K2qI8CWquoBVNUZYHPT/XLg1MDD\nTzdtkqQZsX6UTlX1EvDuJG8EHkjSof0VcdFfGefn519e7nQ6dDqdxT6FJK1q3W6Xbre74utd9LmV\nkvwb4P8BvwJ0qqqXZA54pKq2J9kNVFXta/rfD+ytqkcXPI/nVppxw8+hBNNyTqLJ9x3P+nyfaNDU\nnFspyd86eyRSktcBvwg8ARwCbm263QLc1ywfAnYl2ZDkSuAq4LEx1y2tEe0ztXq2Vq2EUYaV3gzs\nb444Wgd8oar+R7MP4mCS24CT9I9QoqqOJDkIHAFeAG53E0FaqucZtpXR63lUmJaXp+zWkjmsNNn1\n+f5Zm6ZmWEmStPYYDpKkFsNBktRiOEiSWgwHSVKL4SBJajEcJEkthoMkqcVwkCS1GA6SpBbDQZLU\nYjhIkloMB0lSi+EgSWoxHCRJLYaDJKnFcJAktRgOGsnc3LbWdYw1Se1rS3tdaY2TlwnVSIZfEnQ2\nL6+5ml+H76nVz8uESpImxnCQJLUYDpKkFsNBktRywXBIsjXJw0meSfJ0ko827ZuSHE5yLMkDSTYO\nPGZPkuNJjibZuZwvQJI0fhc8WinJHDBXVU8meQPwp8CNwIeB71XVnUnuADZV1e4kVwP3AO8FtgIP\nAW9beGiSRyvNFo9WWkrfla/N99TqNzVHK1XVmap6sln+PnCU/of+jcD+ptt+4KZm+QbgQFW9WFUn\ngOPAjjHXLUlaRova55BkG/Au4KvAlqrqQT9AgM1Nt8uBUwMPO920SZJmxPpROzZDSr8PfKyqvp9k\n4fbrordn5+fnX17udDp0Op3FPoUkrWrdbpdut7vi6x1phnSS9cB/A75SVb/VtB0FOlXVa/ZLPFJV\n25PsBqqq9jX97gf2VtWjC57TfQ4zxH0OS+m70ut7LfB8q+eWLVdw5syJIc+hWTQ1+xwa/xE4cjYY\nGoeAW5vlW4D7Btp3JdmQ5ErgKuCxMdQq6VU9Tz8wzr31eicnWpVm0yhHK10L/E/gaV75i/sE/Q/8\ng8BbgJPAzVX1N81j9gC/ArxAfxjq8JDndcthhrjlsJS+01Ob77XVY6W2HDzxns4xN7ftVb5prr0P\n1dXyOnyvrR6GgyZi+BYCrNUP1dXyOnyvrR7Tts9BkrSGGA6SpBbDQZLUYjhIkloMB0lSi+EgSWox\nHCRJLYaDJKnFcJAktRgO0qp3KUnOuc3NbZt0UZpyI1/PQdKsOnu21lf0est+9gXNOLccJEkthoMk\nqcVwkCS1GA6SpBbDQZLUYjisYXNz21qHOEoSeCW4NW1lrws9m1dQW82vw/ffbPJKcJKkiTEcJEkt\nhoMkqcVwkCS1XDAcknwuSS/JUwNtm5IcTnIsyQNJNg78354kx5McTbJzuQqXJC2fUbYc7gbev6Bt\nN/BQVb0DeBjYA5DkauBmYDtwPXBXPD5SmkLtM7V6tlYNumA4VNX/Ap5d0HwjsL9Z3g/c1CzfAByo\nqher6gRwHNgxnlK1VMPmM5jZa93ZM7Wee+v1Tk60Kk2Ppe5z2FxVPYCqOgNsbtovB04N9DvdtGmC\n+m/49geBJJ3PuK7nsKRPmvn5+ZeXO50OnU5nTOVI0urQ7Xbpdrsrvt6RZkgnuQL4clX9VHP/KNCp\nql6SOeCRqtqeZDdQVbWv6Xc/sLeqHh3ynM6QXiHDZ0LD9M7enebaVv/r8H053aZthnSa21mHgFub\n5VuA+wbadyXZkORK4CrgsTHUKUlaQRccVkryRaAD/FiSbwF7gU8D9ya5DThJ/wglqupIkoPAEeAF\n4HY3DyRp9njivTXAYaVJ9Z3m2hxWmlXTNqwkSVpDDAdJUovhIElqMRxWGa/uJmkcDIdVZvhsaGlU\nnnNJfeOaIS1pVTh7zqVz9Xpuga41bjlIGkF7i8KtidXNLQdJI2hvUbg1sbq55SBJajEcZpTXaJC0\nnBxWmlGvHJW0kAEh6eK55SBJajEcJC2RcyJWM4eVJC2RcyJWM7ccJEkthoMkqcVwkCS1GA5TzvkM\nkibBHdJTzvkMkibBLYcp4rUYtDq0D3G95JLXe9jrjMmkLiaepLyQ+bn6YbD2Lmg/+fX5Oia5Pj8H\nFicJVbXs3xzdcpgA9yNIZ3kq8Gm1bOGQ5LokX0/y50nuWK71TLthQTD8am1+e9JadHYi3Su3/vtD\nk7Ys4ZBkHfAfgPcDfxv4UJJ3Lse6JqXb7Y7Ub3ov29mddAEXqTvpAi5Sd9IFXITupAu4KKO+d9e6\n5dpy2AEcr6qTVfUCcAC4cZnWNREL/8Bmb6ioO+kCLlJ30gVcpO6kC7gI3UkXcFEMh9Es16GslwOn\nBu5/m35gTNSzzz7Lgw8+2Gp/85vfzPve976Lem4POZXG5dKhX6y2bLmCM2dOrHw5a9Samufw2c/+\ne37jN+aH/t+WLVe0xjrXrbuMl176Qavv2fZPfvKTy1GmtMad74R+r22FxoXeo+drH3zvLiZ05ua2\nDd0nshqDa1kOZU3yM8B8VV3X3N8NVFXtG+gzLYPvkjRTVuJQ1uUKh0uAY8DPA38FPAZ8qKqOjn1l\nkqSxW5Zhpar6YZJfAw7T3+n9OYNBkmbHxGZIS5Km19gPZU3yL5O8lORNA217khxPcjTJzoH29yR5\nqpko99mB9g1JDjSP+eMkPzHwf7c0/Y8l+eUx1n1nU9+TSf4gyRtnqf5RTdPkxCRbkzyc5JkkTyf5\naNO+Kcnh5mf0QJKNA48Z2+9iTK9hXZLHkxyawdo3Jrm3qeeZJNfMWP17mrqfSnJPs76prT/J55L0\nkjw10LYi9S7pc6eqxnYDtgL3A38BvKlp2w48QX8Iaxvwf3hli+VR4L3N8n8H3t8s/3Pgrmb5g8CB\nZnkT8A1gI/CjZ5fHVPsvAOua5U8Dn2qWr56F+kd8jeua+q8AXgM8CbxzpdY/pJ454F3N8hvo76d6\nJ7AP+FdN+x3Ap8f9uxjja/gXwH8GDjX3Z6n2zwMfbpbXN3+XM1F/8zf8TWBDc/+/ALdMc/3AzwLv\nAp4aaFv2elni58643+z3An+Xc8NhN3DHQJ+vANfQ/2A4MtC+C/jtZvl+4Jpm+RLgOwv7NPd/G/jg\nOF9D87w3AV+Y1fpf5XX9DPCVgfvnvLZJ34D/Sj+kvw5sadrmgK+P8Xfxf8dY71bgQaDDK+EwK7W/\nEfjGkPZZqX9TU+sm+h+gh2bhb4d+qA2Gw3LWe1GfO2MbVkpyA3Cqqp5e8F8LJ8Sdbtoupz857qxv\nN23nPKaqfgg8l/4w1fmea9xuo5/I59SyYJ3TXP/5DJucuJLrP68k2+h/q/oq/TdLD6CqzgCbm27j\n+F38TQaGPC/SZ4CPc+5B+bNS+5XAd5Pc3QyL/U6Sy2al/qp6Fvi3wLeaWp6rqodmpf4Bm5ex3ov6\n3FnU0UpJHgS2DDbRf2P8a+ATwC8u5vkWs+qxPMn56//1qvpy0+fXgReq6vfGsc6B9eg8krwB+H3g\nY1X1/bTnwIzzqIlx/S39EtCrqieTdF6l69TV3lgPvAf4SFV9Lcln6H9bnfqfPUCSt9If0rsCeA64\nN8k/ZkbqfxVTU++iwqGqhn74J/k79MfF/ixJ6G9uP55kB/2UGtyRs7VpOw28ZUg7A//3l+nPmXhj\nVf11ktP0N+EHH/PIxdY/8DpuBT4A/NxA8/nqXPH6x+B8v4uJSbKefjB8oarua5p7SbZUVS/JHPCd\npn1sv4sxlH4tcEOSDwCvA34kyReAMzNQO/S/cZ6qqq819/+AfjjMws8e4KeB/332+ZJ8CfgHM1T/\nWcte75I/d8Y1BrhgXO0vgE3N8tkdKxvob8oO7lj5Kv1zLoX+MM51TfvtvLJjZRfDd6ycXf7RMdV8\nHfAM8GML2mei/hFf4yW8skN6A/0d0ttXav3nqek/Af9uQds+mvFWhu+ku+jfxZhfwz/klX0Od85K\n7cAfAW9vlvc2P/eZ+NkDfw94Gnhts97PAx+Z9vrpf4l+eiX/1lni585yveG/SbNDurm/p3lxR4Gd\nA+1/v/kFHwd+a6D9UuBg0/5VYNvA/93atP858MtjrPk4cBJ4vLndNUv1L+J1Xkf/qKDjwO6VXv+C\nWq4Ffkg/pJ5ofu7XAW8CHmrqPDz4hzzO38UYX8dgOMxM7fQ/YP+k+fn/YfPhMUv1f5z+F7qngP30\nj8Cb2vqBLwJ/Sf/kUd8CPkz/w3rZ62UJnztOgpMktXiZUElSi+EgSWoxHCRJLYaDJKnFcJAktRgO\nkqQWw0GS1GI4SJJa/j86q2TRi3PRWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa929940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def reject_outliers(data):\n",
    "    u = np.median(data)\n",
    "    s = np.std(data)\n",
    "    filtered = [e for e in data if (u - 2 * s < e < u + 2 * s)]\n",
    "    return filtered\n",
    "\n",
    "filtered = reject_outliers(incomes)\n",
    "\n",
    "plt.hist(filtered, 50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks better. And, our mean is more, well, meangingful now as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27161.222812567459"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(filtered)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of a single outlier, add several randomly-generated outliers to the data. Experiment with different values of the multiple of the standard deviation to identify outliers, and see what effect it has on the final results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}