Module_2c_resampling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem\n",
    "The number of shoes sold by an e-commerce company during the first three months(12 weeks) of the year were:\n",
    "<br>\n",
    "23 21 19 24 35 17 18 24 33 27 21 23\n",
    "\n",
    "Meanwhile, the company developed some dynamic price optimization algorithms and the sales for the next 12 weeks were:\n",
    "<br>\n",
    "31 28 19 24 32 27 16 41 23 32 29 33\n",
    "\n",
    "Did the dynamic price optimization algorithm deliver superior results? Can it be trusted?\n",
    "\n",
    "### Solution\n",
    "\n",
    "Before we get onto different approaches, let's quickly get a feel for the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "sns.set(color_codes=True)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load the data\n",
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23.75"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "before_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27.916666666666668"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "after_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "observed_difference = after_opt.mean() - before_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference between the means is: 4.16666666667\n"
     ]
    }
   ],
   "source": [
    "print(\"Difference between the means is:\", observed_difference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, the sales after optimization is more than the sales before optimization. But is the difference legit? Could it be due to chance?\n",
    "\n",
    "**Classical Method** : We could cover this method later on. This entails doing a *t-test* \n",
    "\n",
    "**Hacker's Method** : Let's see if we can provide a hacker's perspective to this problem, similar to what we did in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Step 1: Create the dataset. Let's give Label 0 to before_opt and Label 1 to after_opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Learn about the following three functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.append"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),\n",
    "np.append(before_opt, after_opt)], dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (2, 24)\n",
      "Data: \n",
      " [[ 0  0  0  0  0  0  0  0  0  0  0  0  1  1  1  1  1  1  1  1  1  1  1  1]\n",
      " [23 21 19 24 35 17 18 24 33 27 21 23 31 28 19 24 32 27 16 41 23 32 29 33]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape:\", shoe_sales.shape)\n",
    "print(\"Data:\", \"\\n\", shoe_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (24, 2)\n",
      "Data: \n",
      " [[ 0 23]\n",
      " [ 0 21]\n",
      " [ 0 19]\n",
      " [ 0 24]\n",
      " [ 0 35]\n",
      " [ 0 17]\n",
      " [ 0 18]\n",
      " [ 0 24]\n",
      " [ 0 33]\n",
      " [ 0 27]\n",
      " [ 0 21]\n",
      " [ 0 23]\n",
      " [ 1 31]\n",
      " [ 1 28]\n",
      " [ 1 19]\n",
      " [ 1 24]\n",
      " [ 1 32]\n",
      " [ 1 27]\n",
      " [ 1 16]\n",
      " [ 1 41]\n",
      " [ 1 23]\n",
      " [ 1 32]\n",
      " [ 1 29]\n",
      " [ 1 33]]\n"
     ]
    }
   ],
   "source": [
    "shoe_sales = shoe_sales.T\n",
    "print(\"Shape:\",shoe_sales.shape)\n",
    "print(\"Data:\", \"\\n\", shoe_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This is the approach we are going to take\n",
    "#We are going to randomly shuffle the labels. Then compute the mean between the two groups. \n",
    "#Find the % of times when the difference between the means computed is greater than what we observed above\n",
    "#If the % of times is less than 5%, we would make the call that the improvements are real"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.shuffle(shoe_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0, 33],\n",
       "       [ 0, 35],\n",
       "       [ 1, 24],\n",
       "       [ 0, 19],\n",
       "       [ 1, 29],\n",
       "       [ 1, 33],\n",
       "       [ 1, 28],\n",
       "       [ 0, 24],\n",
       "       [ 0, 23],\n",
       "       [ 0, 24],\n",
       "       [ 1, 32],\n",
       "       [ 1, 41],\n",
       "       [ 0, 17],\n",
       "       [ 1, 19],\n",
       "       [ 1, 16],\n",
       "       [ 0, 27],\n",
       "       [ 1, 32],\n",
       "       [ 0, 21],\n",
       "       [ 1, 31],\n",
       "       [ 1, 23],\n",
       "       [ 0, 23],\n",
       "       [ 0, 21],\n",
       "       [ 0, 18],\n",
       "       [ 1, 27]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shoe_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "experiment_label = np.random.randint(0,2,shoe_sales.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0,\n",
       "       0])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0 33]\n",
      " [ 0 35]\n",
      " [ 0 24]\n",
      " [ 1 19]\n",
      " [ 1 29]\n",
      " [ 0 33]\n",
      " [ 0 28]\n",
      " [ 0 24]\n",
      " [ 0 23]\n",
      " [ 0 24]\n",
      " [ 0 32]\n",
      " [ 0 41]\n",
      " [ 0 17]\n",
      " [ 0 19]\n",
      " [ 1 16]\n",
      " [ 0 27]\n",
      " [ 0 32]\n",
      " [ 1 21]\n",
      " [ 1 31]\n",
      " [ 1 23]\n",
      " [ 1 23]\n",
      " [ 1 21]\n",
      " [ 0 18]\n",
      " [ 0 27]]\n"
     ]
    }
   ],
   "source": [
    "experiment_data = np.array([experiment_label, shoe_sales[:,1]])\n",
    "experiment_data = experiment_data.T\n",
    "print(experiment_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_diff_mean =  experiment_data[experiment_data[:,0]==1].mean() \\\n",
    "                        - experiment_data[experiment_data[:,0]==0].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.71875"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_diff_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Like the previous notebook, let's repeat this experiment 100 and then 100000 times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def shuffle_experiment(number_of_times):\n",
    "    experiment_diff_mean = np.empty([number_of_times,1])\n",
    "    for times in np.arange(number_of_times):\n",
    "        experiment_label = np.random.randint(0,2,shoe_sales.shape[0])\n",
    "        experiment_data = np.array([experiment_label, shoe_sales[:,1]]).T\n",
    "        experiment_diff_mean[times] =  experiment_data[experiment_data[:,0]==1].mean() \\\n",
    "                        - experiment_data[experiment_data[:,0]==0].mean()\n",
    "    return experiment_diff_mean    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_diff_mean = shuffle_experiment(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.81428571],\n",
       "       [ 1.38571429],\n",
       "       [-1.01428571],\n",
       "       [ 1.09375   ],\n",
       "       [ 0.41666667],\n",
       "       [ 1.35314685],\n",
       "       [ 2.10839161],\n",
       "       [ 2.66666667],\n",
       "       [ 3.2394958 ],\n",
       "       [ 0.5979021 ]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_diff_mean[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10db0ba20>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAFoCAYAAAAlwpNtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAGUhJREFUeJzt3X+MbGd93/H3nNm9uzu2l73Xbm2DW8WN5CfXUUSAxMaB\nFDslYKRgqEX5GQXkqNCAcYJTiaA6uAkVwuZHAk6pC0kLbhokIsCGpEpLnApwjWU71CTI5usqDQqx\njcHG43t9Z/feu3umf8zs7fj+WO/snJkzs8/7JV3NnTOz53yfZ2bOfM5zfkyj2+0iSZLyVNRdgCRJ\nqo9BQJKkjBkEJEnKmEFAkqSMGQQkScqYQUCSpIwZBCRJyphBQJKkjBkEJEnK2NwwT04pPRv4GHAZ\n0AE+C7wnIo6klD4KvBPoAo3+7Tsj4uPVlixJkqoyVBAAPgc8DrwIOBP4z8A68G5gf//20wPPP1BB\njZIkaUy2HQRSSgm4CDg7Ih7rT3sv8EH+fxC4MSK+P45CJUlS9YY5RuB7wOWbIaCvATwrpXQG8Bzg\nwSqLkyRJ49XY6a8PppQawB3Ao8AHgK8DfwC8gt7ug49ExC0V1SlJksZglLMGPgj8JHAd8GNACdxP\nLwj8PvCJlNKrRq5QkiSNzbAHCwKQUroBuAZ4bUTcD9yfUvpiRLT7T/lWSukC4FeA27Y732632200\nGjspSZKk3O3oC3ToIJBSugl4G/CmiLh1c/pACNj0AL3TDLet0Whw4MAqGxvlsGVlq9ksWF5est+G\nYJ/tjP02PPtsZ+y34W322U4Mex2B64G3Aq+LiC8MTP8t4Gci4ucHnv484NvDFrSxUbK+7gs/LPtt\nePbZzthvw7PPdsZ+m4xhTh/cT+94gPcDd6aUzh54+EvAb6SUrgVuBV4O/CJwaXWlSpKkqg1zsOAV\n/edfBzzc//cI8HBE3Au8Bvgl4K+Bq4E3RMTd1ZYrSZKqtO0RgYi4Abhhi8e/RG9kQJIkzQh/dEiS\npIwZBCRJyphBQJKkjBkEJEnKmEFAkqSMGQQkScqYQUCSpIwZBCRJyphBQJKkjBkEJEnKmEFAkqSM\nGQQkScqYQUCSpIwZBCRJyphBQJKkjBkEJEnKmEFAkqSMGQQkScqYQUCSpIwZBCRJyphBQJKkjBkE\nJEnKmEFAkqSMGQQkScqYQUCSpIwZBCRJyphBQJKkjBkEJEnKmEFAkqSMGQQkScrYXN0FaDhlWdJu\nt4/dn5srKMs12u0O6+tljZVVY2VlhaIwn0rSpBgEZky73ebP736QVusMAIqiQWtpD53VI5Rlt+bq\nRtPpHOSlF13Avn376i5FkrJhEJhBrdYZnL68AkCzaNBqLdCcP8zGjAcBSdLkOQYrSVLGDAKSJGXM\nICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAg\nSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxuaGeXJK6dnAx4DLgA7wWeA9EXEkpfQjwCeBS4DvAO+K\niC9XWq0kSarUsCMCnwMWgRcBrwdeCbyv/9htwMPAC4A/BL6QUjqvojolSdIYbHtEIKWUgIuAsyPi\nsf609wIfTCn9GXA+cHFErAEfSCn9M+Aq4LerL1uSJFVhmBGB7wGXb4aAAc8CXgh8ox8CNt1BbzeB\nJEmaUtseEYiIJ4Fj+/xTSg3gauB24Fx6uwUGPQq4a0CSpCk21MGCx/kg8Dzgp4FrgcPHPX4YWBh2\nps2mJzJsZW6uoCgaNIsGAEVRDNyWNVY2uqJoMDdXMDc33vfA5nts0u+1sixpt9sTXWaVms0GZbnG\nwYOrbGx0T3h8ZWXl2PtRPXW912ad/Ta8UfpqR0EgpXQDcA3w2oi4P6W0Buw77mkL9M4sGMry8tJO\nSspGWa7RWtpDq/X0jLW4OF9TRdXZOLqHlZUWe/eeNpHlTfq99vjjj/O1+/6W0047Y6LLnYRDhw7y\nykt/nL17z6y7lKnkem1n7LfJGDoIpJRuAt4GvCkibu1Pfgi48LinngM8Muz8DxxYZWNjtrdsx6nd\n7tBZPUJzvjcAUxQFi4vzrK0dpSxnu986q0dotzsUxeJYl9NsFiwvL038vdZud2gUCzTnWxNbZpW2\neq81ism8drOmrvfarLPfhrfZZzsx7HUErgfeCrwuIr4w8NBdwLtTSgsRsbmL4MXA14YtaGOjZH3d\nF/5U1tdLyrLLRrk5NNvrq7IsB6bNprLssr4+udd/0u+1E1+7WXPq99qkX7tZ43ptZ+y3yRjm9MH9\nwHXA+4E7U0pnDzz8FeC7wKdSSu8DrqB37MBbqitVkiRVbZijC67oP/86emcIPExv6P/hiCiBV9Pb\nHXAv8Ebg1RHx99WWK0mSqjTM6YM3ADds8fjf0Lv0sCRJmhGemyFJUsYMApIkZcwgIElSxgwCkiRl\nzCAgSVLGDAKSJGVslB8dkirV+1GeJ8a+nLm5grJco93uTPSqZe32E3Rn9qqCknYrg4Cmxuqhg9xx\n32PsO2ttrMspigatpT10Vo9QTvCL+Qfff4jTTt/LGeyd2DIl6ZkYBDRVFlunc/ryyliX0SwatFoL\nNOcPT/S6/4cOPjmxZUnSdnmMgCRJGTMISJKUMYOAJEkZMwhIkpQxg4AkSRkzCEiSlDGDgCRJGTMI\nSJKUMYOAJEkZMwhIkpQxg4AkSRkzCEiSlDGDgCRJGTMISJKUMYOAJEkZMwhIkpQxg4AkSRkzCEiS\nlDGDgCRJGTMISJKUMYOAJEkZMwhIkpQxg4AkSRkzCEiSlDGDgCRJGTMISJKUMYOAJEkZMwhIkpQx\ng4AkSRkzCEiSlDGDgCRJGTMISJKUMYOAJEkZMwhIkpSxuboLkDT7yrKk3X6i7jLGZmVlhaJwu0m7\nk0FA0shWDx3kjvseY99Za3WXUrlO5yAvvegC9u3bV3cp0lgYBCRVYrF1Oqcvr9RdhqQhOdYlSVLG\nDAKSJGXMICBJUsYMApIkZWzHBwumlBaAe4F3RMRX+9M+CrwT6AKN/u07I+LjFdQqSZIqtqMg0A8B\nnwEuPO6h/cC7gU8PTDuws9IkSdK4DR0EUkr7gT86xcP7gRsj4vsjVSVJkiZiJ8cIvAS4HbiE3vA/\nACmlM4DnAA9WU5okSRq3oUcEIuLmzf+nlAYf2k/vmIDrUkqvAB4HPhIRt4xapCRJGo8qryz4Y0AJ\n3A98DLgU+ERK6cmIuG27M2k2PZFhK3NzBUXRoFn0BmM2r3/euy1rrGx0jaJBURTH2jYudfXZpNo3\nLlv126y3bStF0WBurmBubvh10+b6zPXacOy34Y3SV5UFgYi4JaX0xYho9yd9K6V0AfArwLaDwPLy\nUlUl7UpluUZraQ+t1sLTpi8uztdUUXVaS3toNOdPaNu4TLrPJt2+cTlZv+2Wtp3MxtE9rKy02Lv3\ntB3Pw/Xazthvk1Hpbw0MhIBNDwCXDTOPAwdW2diY7S3bcWq3O3RWj9CcPwz0ts4WF+dZWztKWc52\nv3VWj1A0odM5PNbl1NVnk2rfuGzVb7Petq10Vo/QbncoisWh/7bZLFheXnK9NiT7bXibfbYTlQWB\nlNJvAT8TET8/MPl5wLeHmc/GRsn6ui/8qayvl5Rll42y25/S66uyLAemzaZu2aVsTKId9fTZ5No3\nLqfut9lv26mVZZf19dHWS67XdsZ+m4wqRwS+BPxGSula4Fbg5cAv0jtWQJIkTaFRj8Q4Fv8j4l7g\nNcAvAX8NXA28ISLuHnEZkiRpTEYaEYiI5nH3v0RvZECSJM0Az82QJCljBgFJkjJmEJAkKWMGAUmS\nMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJm\nEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQ\nJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQp\nYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMG\nAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjI2t9M/TCktAPcC74iIr/an/QjwSeAS4DvAuyLiy6OX\nKUmSxmFHIwL9EPAZ4MLjHroVeBh4AfCHwBdSSueNVKEkSRqboYNASmk/cBdw/nHTfw74J8DboucD\nwNeBq6ooVJIkVW8nIwIvAW6nN/zfGJh+MfCNiFgbmHZH/3mSJGkKDX2MQETcvPn/lNLgQ+fS2y0w\n6FHAXQOSJE2pHR8seBIt4PBx0w4DC8PMpNn0RIatzM0VFEWDZtEbjCmKYuC2rLGy0TWKBkVRHGvb\nuNTVZ5Nq37hs1W+z3ratdXnqqSeZmxt+3dRsNijLNQ4eXGVjozuG2qqxsrJy7PWdBpvfA34fbN8o\nfVVlEFgD9h03bQHoDDOT5eWlygrajcpyjdbSHlqtp+erxcX5miqqTmtpD43m/AltG5dJ99mk2zcu\nJ+u33dK2k3nyh4e5+4EDnHXWRt2ljMWhQwd55aU/zt69Z9Zdygn8PpiMKoPAQ5x4FsE5wCPDzOTA\ngVU2NmZ7y3ac2u0OndUjNOd7gy9FUbC4OM/a2lHKcrb7rbN6hKIJnc7xA0vVqqvPJtW+cdmq32a9\nbVvptW2B5nxr6L+dhc9nozhCu92hKBbrLuWYZrNgeXnJ74MhbPbZTlQZBO4C3p1SWoiIzbXBi4Gv\nDTOTjY2S9XVf+FNZXy8pyy4b5eYwY6+vyrIcmDabumWXsjGJdtTTZ5Nr37icut9mv22nNlrbpv/z\nWZZd1tenc73r98FkVBkEvgJ8F/hUSul9wBXATwNvqXAZkiSpQqMeiXEs4kZECbyK3u6Ae4E3Aq+O\niL8fcRmSJGlMRhoRiIjmcff/L3DZSBVJkqSJ8dwMSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQ\nkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAk\nKWMGAUmSMmYQkCQpYwYBSZIyZhCQJCljBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyZhCQJClj\nBgFJkjJmEJAkKWMGAUmSMmYQkCQpYwYBSZIyNld3AeNwx11/yXp3d2acQwefgMWz6y5DkrRL7Mog\n8NRayeLKOXWXMRaHHvshrcW6q5Ak7Ra7c7NZkiRti0FAkqSMGQQkScqYQUCSpIwZBCRJyphBQJKk\njBkEJEnKmEFAkqSMGQQkScqYQUCSpIztyksMS5K2pyxL2u0n6i7jaebmCspyjXa7w/p6OdK8VlZW\nKAq3ebdiEJCkjK0eOsgd9z3GvrPW6i7lmKJo0FraQ2f1CGXZ3fF8Op2DvPSiC9i3b1+F1e0+BgFJ\nytxi63ROX16pu4xjmkWDVmuB5vxhNkYIAtoex0skScqYQUCSpIwZBCRJyphBQJKkjFV6sGBK6dXA\n54Eu0Ojffi4iXlvlciRJUjWqPmvgQuCLwL+kFwQApuecFEmS9DRVB4H9wLci4gcVz1eSJI1B1ccI\nXAg8WPE8JUnSmFQ9IpCAy1NK/wZoAn8MvDcijla8HEmSVIHKgkBK6R8DS8Aq8C+A84GbgEXgXdud\nT7M5+iBF0SxoFo1nfuIMahYFRdE41r7Na2j3bke7JnfdGkWDohj/a1dXn02qfeOyVb/Netu2Mkrb\nZuHzOY2vXVX9VhQN5uYK5uZ2/wlyo3x3VhYEIuLvUkpnRkS7P+mvUkpN4L+klK6NiG1dJ3J5eWnk\nWlqtBZZaCyPPZxotteZpLe2hdVz7Fhfna6qoOq2lPTSa8ye0bVwm3WeTbt+4nKzfdkvbTqaKtk3z\n53OaX7tR+23j6B5WVlrs3XtaRRXtTpXuGhgIAZseoDcisA94fDvzOHBglY2N0ZJzp3OY7vzhkeYx\nrVY7R2H+CM1++4qiYHFxnrW1o5TldG5xbFdn9QhFs/f6jVNdfTap9o3LVv02623byihtm4XP5zS+\ndlX1W2f1CO12h6JYrLC66dRsFjvekK5y18DLgD8CzouIzVMGnwc8HhHbCgEAGxvlyD87WW6Uu/aH\nKjbKkrLsDrSv11dlOftt7pZdysYk2lFPn02ufeNy6n6b/bad2mhtm/7P53S+dtX0W1l2WV8f/Ttl\nt6tyROBOoAP8fkrpt4EfBW4EbqhwGZIkqUKVHUEREU8BLwf+AXAP8Eng5oj4cFXLkCRJ1ar6GIEH\n6IUBSZI0A3b/ORWSJOmUDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJ\nUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLG\nDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwC\nkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIk\nZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRlzCAgSVLGDAKSJGVs\nrsqZpZQWgI8DVwId4MMR8ZEqlyFJkqpT9YjAh4DnA5cCbweuTyldWfEyJElSRSoLAimlFvDLwDUR\n8c2IuA24Ebi6qmVIkqRqVTki8Fx6uxq+PjDtDuDiCpchSZIqVGUQOBd4LCLWB6Y9CiymlM6scDmS\nJKkiVR4s2AIOHzdt8/7CdmfSbI6eTYpmQbNojDyfadQsCg6vPkXRb19RFGwcnWPt8DplWdZc3WgO\nrx2i0dxD56knx7qcuvpsUu0bl636bdbbtpVR2jYLn89pfO2q6rfDq08xN1cwN7f7T5Ab5buzyiCw\nxolf+Jv3O9ucR2N5eWnkQt505aUjz2N6pboLkCTtIlXGpIeAs1JKg/M8B1iNiHaFy5EkSRWpMgjc\nBxwFXjgw7WeBeypchiRJqlCj2+1WNrOU0n8AXgRcBZwHfAp4c/9UQkmSNGUqvbIgcC29Kwv+BfAk\n8JuGAEmSplelIwKSJGm27P5zKiRJ0ikZBCRJyphBQJKkjBkEJEnKmEFAkqSMVX364MhSSs8CPgz8\nAr2g8qfAr0XE9FwIe8qllP478F8j4pa6a5k2KaUFeqe4Xknv0tcfjoiP1FvVbOj33b3AOyLiq3XX\nM+1SSs8GPgZcRu+99lngPRFxpNbCplhK6UeBf0/vejSPA78XER+qt6rZkVL6U+DRiLhqmL+bxhGB\n/wj8BHA58DJgP/CJWiuaESmlRkrpJuClddcyxT4EPB+4FHg7cH1K6cpaK5oB/RDwGeDCumuZIZ8D\nFul9qb0eeCXwvlormmIppQa9Db9HgZ8E/hVwXUrp9bUWNiP6/fSKnfztVAWBlFKL3pbaOyLivoi4\nD/g14J+nlPbUW91062993E5vJMXfdjiJ/vvrl4FrIuKb/Ytd3QhcXW9l0y2ltB+4Czi/7lpmRUop\nARcBb4mIb0fE/wLeC7yx3sqm2tnA/wbeHhF/ExF/Rm+d9uJ6y5p+KaW99NZld+/k76cqCAAlvS+y\nbw5MawBN4PRaKpodzwf+DngBcKDmWqbVc+ntDvv6wLQ7gIvrKWdmvITeCvkSep9HPbPvAZdHxGMD\n0xrAs2qqZ+pFxPci4g0RcQggpfQi4J8C/7PeymbCh4BbgAd28sdTdYxARKwB/+O4yb8K/FVE/LCG\nkmZGRPwJ8CcAvY0RncS5wGMRsT4w7VFgMaV0ZkQ8XlNdUy0ibt78v++t7ekf0/Tlzfv9Ye+rgT+v\nragZklL6DvCP6K3TPl9rMVMupfRz9H7g7yeAm5/h6Sc18SCQUloEnnOKhx+JiM7Ac68GXgO8fBK1\nTbNh+k2n1AIOHzdt8/7ChGtRXj5Ib7/3T9VdyIy4kt7P2N8M/C69DUIdp3/szs30dqcc3mlQr2PX\nwMXA/wEePMm/Ywe5pZTeDnyU3hkDt9dQ57TZVr9pS2uc+IW/ed8gpbFIKd0AXAO8KSJ2NHSbm4j4\nRkT8N+BdwFtTSlM1ej1F/i1wT0SMNNI08c6NiK/wDAEkpfSv6R348OsR8XsTKWzKbaff9IweAs5K\nKRURUfannQOsRoQHWKpy/bN43kYvBNxadz3TLKX0D4FLjvvF2vuBPcAy4O7hE70OODuldLB/fwEg\npfSaiFje7kym7oslpfRm4AbgVyPid+quR7vKfcBR4IUD034WuKeecrSbpZSuB94KvC4i/rjuembA\n+cDnU0rnDkz7KeAHHiN2Si+hd2zAc/v/vgjc1v//tk3VcEv/FIibgE8Dn00pnT3w8A8GtuKkoUXE\nakrpFuDmlNJVwHnArwNvrrcy7Tb9Uy6vA94P3Dm4LouIR2srbLrdQ++CVf8ppXQtvWBwI/Dvaq1q\nikXEdwfv90cGuhHxt8PMZ9pGBF4GnEZvxfxw/98j/dvzaqxr1nTrLmCKXQv8JfAX9ELnbx43FKmt\n+d7anivorV+v48R1mU6iv6H3KuAQcCe9C8n9rruHx6/R7fq5liQpV9M2IiBJkibIICBJUsYMApIk\nZcwgIElSxgwCkiRlzCAgSVLGDAKSJGXMICBJUsYMApIkZcwgIElSxgwCkiRl7P8BKgMxV9ux3JMA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10daf2a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(experiment_diff_mean, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data: Difference in mean greater than observed: []\n",
      "Number of times diff in mean greater than observed: 0\n",
      "% of times diff in mean greater than observed: 0.0\n"
     ]
    }
   ],
   "source": [
    "#Finding % of times difference of means is greater than observed\n",
    "print(\"Data: Difference in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference])\n",
    "\n",
    "print(\"Number of times diff in mean greater than observed:\", \\\n",
    "            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0])\n",
    "print(\"% of times diff in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Repeat the above for 100,000 runs and report the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Is the result by chance? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the justification for shuffling the labels? \n",
    "\n",
    ">Thought process is this: If price optimization had no real effect, then, the sales before optimization would often give more sales than sales after optimization. By shuffling, we are simulating the situation where that happens -  sales before optimization is greater than sales after optimization. If many such trials provide improvements, then, the price optimization has no effect. In statistical terms, *the observed difference could have occurred by chance*. \n",
    "\n",
    "Now, to show that the same difference in mean might lead to a different conclusion, let's try the same experiment with a different dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])\n",
    "after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean sales before price optimization: 237.5\n",
      "Mean sales after price optimization: 241.666666667\n",
      "Difference in mean sales: 4.16666666667\n"
     ]
    }
   ],
   "source": [
    "print(\"Mean sales before price optimization:\", np.mean(before_opt))\n",
    "print(\"Mean sales after price optimization:\", np.mean(after_opt))\n",
    "print(\"Difference in mean sales:\", np.mean(after_opt) - np.mean(before_opt)) #Same as above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),\n",
    "np.append(before_opt, after_opt)], dtype=int)\n",
    "shoe_sales = shoe_sales.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10dbf34e0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAFoCAYAAADgsAn8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X+YZXV94Pn3PVXdVXW7q6xqmNjM4Ipkxi+S3WUNIqKo\ngTDReZ4NEkzir2cmiNthHpc1mmFgzA+YlZmoxAyuSQhoRJIxM+u4CZpHx0UwO0JPaBtEyI6aD+sg\njjS/m77Q9q3qpurU/nHOtS/Vdavq3jp1q27X+/U8PF33fL7n1Pd8OLfqU9/zPd9bm5+fR5IkabWy\n9e6AJEk6PlhUSJKkSlhUSJKkSlhUSJKkSlhUSJKkSlhUSJKkSlhUSJKkSlhUSJKkSlhUSJKkSgx3\n0zil9CvAZ4B5oNb2bx4RwymllwGfBM4BHgY+EBG3t+1/AXA9cCpwN7ArIr7fFn8/cAUwDnweuDwi\nZno+O0mS1DfdjlT8n8BO4KTy35cC3wM+Xsa/ADwKnAl8Frg1pXQyQErpJcCtwKeBVwFPl+0p428F\nrgZ2AecDrwGu6+WkJElS/9VW89kfKaUPAu8Gfgp4PUWR8BOt0YWU0u3AXRHxoZTSh4BzI+L8MjYG\nPA78fETcmVL6OnBHRFxbxl8HfBU4wdEKSZI2vp7nVKSUpoArgasi4nngbOC+BQXAbopbIZTxO1uB\niJgG7gPOSSllwFnAXW377gG2Amf02kdJktQ/q5mo+V5gX0TcWr4+ieLWR7sngJNXEJ8ERtvjETEH\n7G/bX5IkbWCrKSreA3yi7XUdOLygzWFgZAXxetvrTvtLkqQNrKunP1pSSmcBfw/4XNvmGWDHgqYj\nQLMtvrBAGAEOlDE6xJus0Pz8/HytVltpc0mSdNSqf4H2VFQAbwLujIhn27btA05f0G4n8FhbfOci\n8W9R3OaYKV8/CJBSGgJOaNt/WbVajeeem2ZuLl/pLpva0FDGxMSYOeuSeeueOeuNeeueOetNK2+r\n1WtRcTbwnxds2wNclVIaiYjWbYxzOTr5ck/5GoCUUh14JXB1RMynlO4p463JnK8FjgAPdNOxubmc\n2VkvpG6Ys96Yt+6Zs96Yt+6Zs/XRa1Hx3wP/dsG2rwM/BG5JKV0LXEjxRMclZfxm4IqU0pXAl4Br\ngIciolVE3ADcmFL6NsWEzRuAT/o4qSRJg6HXiZo/QTEX4sciIgfeQnEL417gncBFEfFIGf8BcDFw\nKbCX4omPi9r2/xzwYeAm4DaKFTev6rF/kiSpz1a1+NUGNH/gwCGHvFZoeDhjamob5qw75q175qw3\n5q175qw3Zd5WPVHTDxSTJEmVsKiQJEmVsKiQJEmVsKiQJEmV6PWRUkk6Rp7nNBqNY7YPD2fk+QyN\nRpPt2yfIMv+ekY5HFhWSKtNoNLhj74PU6+Mv2J5lNepjW9n/9H7OO+sfsGPHwhX9JR0PLCokVape\nH2f7xOQLtg1lNer1EZrTR9apV5L6wTFISZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJU\nCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsK\nSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUieH17oCkzSPPcxqNA8u2m5ycJMv8m0caNBYVkvqm\neeggu+9/ih0nznRu0zzIBa9+OTt27OhjzyRVwaJCUl+N1rezfWJyvbshaQ04vihJkiphUSFJkirR\n9e2PlNJW4HrgHcBh4OaI+M0ydgrwKeAc4GHgAxFxe9u+F5T7ngrcDeyKiO+3xd8PXAGMA58HLo+I\nzjdfJVWmmETZWLKNEyglLaWXORWfAH4G+IfABPC5lNLDEfEp4IvA/cCZwC8At6aUTouIR1JKLwFu\nBX4buA24BvgCcAZASumtwNXAu4AngT8BrgPe1/PZSVqxRqPBHXsfpF4fXzTuBEpJy+mqqEgpTQGX\nAudHxDfLbR8Dzk4pfQ94GXB2ObrwkZTSz5btPwTsAu6JiI+X+70beDyl9IaIuJOieLg+Ir5Sxi8D\nvppSutLRCqk/6vVxJ1FK6lm3IxXnAo2I2N3aEBHXAaSUPgjct6AA2E1xKwTgbODOtv2mU0r3Aeek\nlHYDZ1GMXrTsAbZSjGR8o8t+SjpOeZtG2ri6LSpOBR5OKf1j4Dcoful/BvjXwEnAowvaPwGcXH69\nVHwSGG2PR8RcSml/GbeokAR4m0bayLotKrYDLwd+FbiEolC4CWgCdYqJm+0OAyPl10vF622vO+0v\nSYC3aaSNqtuiYpbiyYx3RMQjACmllwLvBb4KnLCg/QhFwQEww7EFwghwoIzRId6kC0NDDnmuVCtX\n5qw7x2vehoczsqzGUFZbNJ5lNYaHM4aHO593p2O0bkVkWQa1rOP3WMn3qaKfg+J4vdbWkjnrTVX5\n6raoeAyYaRUUpaC4RbEP+KkF7XeW+1DGdy4S/xawn6Kw2Ak8CJBSGqIoUh6jCxMTY900F+asV8db\n3vJ8hvrYVur1xQcH557fyuRknampbT0fY3RkmNrQlo7xlXyfKvo5aI63a60fzNn66Lao2AOMppT+\nfkR8r9x2OsWaFHuAD6aURiKidRvjXOCutn3PbR0opVQHXglcHRHzKaV7ynhrMudrgSPAA9108Lnn\nppmby7s8rc1paChjYmLMnHXpeM1bo9GkOX2EoS0L70IWmtNHaDSaZNlo18fIsozR0S3MHJ6FGjSb\ni3+PlXyfKvo5KI7Xa20tmbPetPK2Wl0VFRHxYErpy8AtKaX3UsypuIrikdE7gR+WsWuBCyme6Lik\n3P1m4IqU0pXAlyie9HiofJwU4AbgxpTStykmbN4AfLLbx0nn5nJmZ72QumHOenO85W12NifP55nL\n5xeN5/k8s7NLn3PnY+TlMXKo5R2/x0q+TxX9HDTH27XWD+ZsffRyE+VdwPcoRiBuAT4REX8YETlF\nIbETuBd4J3BR61ZJRPwAuJhi3Yq9FE98XNQ6aER8DvgwxcTP2yhW3Lyqp7OSJEl91/WKmhFxkGL0\n4ZJFYg8B5y2x723AaUvEr6NYRVOSJA0Yp8dKkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRI\nkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRK\nWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKWFRIkqRKDK93BySp3/I8p9FoLNlmcnKSLPPv\nLqkbFhWSNp1Go8Edex+kXh9fNN5sHuSCV7+cHTt29Lln0mCzqJC0oRSjCAc6xhuNA8zn86v+PvX6\nONsnJld9HElHWVRI2lCmDx1k9/1Ps+PEmUXjTz25j23bpxhnqs89k7QciwpJG85ofXvHUYRDB5/t\nc28krZSzkCRJUiUsKiRJUiUsKiRJUiUsKiRJUiUsKiRJUiUsKiRJUiUsKiRJUiW6XqcipXQR8BfA\nPFAr//3ziPjllNIpwKeAc4CHgQ9ExO1t+14AXA+cCtwN7IqI77fF3w9cAYwDnwcuj4jFV8CRtGIr\n+ayLqlaqlLR59bL41enAXwK7KIoKgNYv/i8C9wNnAr8A3JpSOi0iHkkpvQS4Ffht4DbgGuALwBkA\nKaW3AlcD7wKeBP4EuA54Xw99lNRmuc+6gOVXqlxu+ezi+1iYSJtZL0XFK4D/EhFPtW9MKZ0PvAw4\nuxxd+EhK6WeBS4EPURQh90TEx8v27wYeTym9ISLupCgero+Ir5Txy4CvppSudLRCWr3lPutiuZUq\nl1s+G1xCW9rseplTcTrw4CLbzwbuW1AA7Ka4FdKK39kKRMQ0cB9wTkopA84C7mrbdw+wlXIkQ9L6\nay2f3em/+tj29e6ipHXUy0hFAt6cUvpNYIhi7sPVwEnAowvaPgGcXH69VHwSGG2PR8RcSml/Gf9G\nD/2UJEl91FVRkVL674AxYBr4JYrbHZ8ot9WBwwt2OQyMlF8vFa+3ve60/4oMDflAy0q1cmXOujOI\neRsezsiyGkNZrWObWlYjy7KObZaLL9Umy7Kj/9Z6O8ZK41lWY3g4Y3i48/+f5fKxkmP0wyBea+vN\nnPWmqnx1VVRExH9LKZ0QEa1p5H+TUhoCPgt8Bo65kToCNMuvZzi2QBgBDnB0oudi8SZdmJgY66a5\nMGe9GqS85fkM9bGt1Ouda/T62FZqQ1s6tlkuvpI2oyPDqz7GcvG557cyOVlnampbx++xXD5Wcox+\nGqRrbaMwZ+uj69sfbQVFy3cpbl08TjGJs91O4LHy633l64XxbwH7KQqLnZTzNcpi5YS2/Vfkueem\nmZvLu9ll0xoaypiYGDNnXRrEvDUaTZrTRxjasnAw8Kjm9BGyIWg2F2+zXHypNlmWMTq6hZnDs1Dr\n7RjdxBuNJlk22vF7LJePlRyjHwbxWltv5qw3rbytVre3P34O+HfAyW0TMl8JPE0xyfKKlNJIRLTe\nqedydPLlnvJ161j1ct+rI2I+pXRPGW9N5nwtcAR4oJs+zs3lzM56IXXDnPVmkPI2O5uT5/PMLfG4\n53w+T17LO7ZZLr50myJPeZ5Dz8dYWTzP55mdXfr/zXL5WMkx+mmQrrWNwpytj25HKv6a4nbEH6eU\nPgT8JMVaEh+lKAZ+CNySUroWuJDiiY5Lyn1vpig6rgS+RLFOxUPl46QANwA3ppS+TTFh8wbgkz5O\nKknSYOhqZkZE/Ah4E/B3gHsoVs+8MSJ+LyJyikJiJ3Av8E7gooh4pNz3B8DFFOtW7KV44uOitmN/\nDvgwcBPF4lh3A1et5uQkSVL/9DKn4rsUhcVisYeA85bY9zbgtCXi11GMfEiSpAHjMzeSJKkSFhWS\nJKkSFhWSJKkSFhWSJKkSFhWSJKkSFhWSJKkSFhWSJKkSvXz0uaQNJs9zGo2FH8tzVKNxgPkllsaW\npCpYVEjHgUajwR17H6ReH180/tST+9i2fYrxYz5IWJKqY1EhHSfq9XG2T0wuGjt08Nk+90bSZmRR\nIUkLFLeTDizbbnJykixzaprUYlEhSQtMHzrI7vufZseJnT8kudk8yAWvfjk7duzoY8+kjc2iQtJx\nZSWjDCuZuDpa397xdpKkxVlUSBvcck92gE93tFvJKIMTV6W1YVEhbXDLPdkB/pJcaLlRBieuSmvD\nokIaAEs92QH+kpS0MThtWZIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKi\nQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVWK41x1T\nSl8GnoiIS8vXpwCfAs4BHgY+EBG3t7W/ALgeOBW4G9gVEd9vi78fuAIYBz4PXB4RM732T5Ik9VdP\nIxUppbcD/2jB5i8AjwJnAp8Fbk0pnVy2fwlwK/Bp4FXA02X71vHeClwN7ALOB14DXNdL3yRJ0vro\nuqhIKU1R/MLf27btfIoRiMui8BGK0YhLyya7gHsi4uMR8V3g3cApKaU3lPH3AddHxFci4pvAZcB7\nUkqjvZ6YJEnqr15GKj4G/Cnw3bZtZwP3LbhdsZviVkgrfmcrEBHTwH3AOSmlDDgLuKtt3z3AVuCM\nHvonSZLWQVdFRTki8Xrg2gWhkyhufbR7Ajh5BfFJYLQ9HhFzwP62/SVJ0ga34omaKaUR4EbgvRFx\nOKXUHq4DhxfschgYWUG83va60/4rNjTkAy0r1cqVOetOv/M2PJyRZTWGslrHNrWsRpZlHdssF1/r\nY2RZdvTf2tr2o1/nmmU1hoczhofX7jrwPdo9c9abqvLVzdMf/5JiXsQdi8RmgB0Lto0Azbb4wgJh\nBDhQxugQb9KliYmxbnfZ9MxZb/qVtzyfoT62lXq9c41dH9tKbWhLxzbLxft1jNGR4TXvR7/Ode75\nrUxO1pma2taxTVV8j3bPnK2PboqKtwEvTikdLF+PAKSUfhH4HeD0Be13Ao+VX+8rXy+Mf4viNsdM\n+frB8phDwAlt+6/Yc89NMzeXd7vbpjQ0lDExMWbOutTvvDUaTZrTRxjasnAw76jm9BGyIWg2F2+z\nXHytj5FlGaOjW5g5PAu1te1HP8+10WiSZWs3n9z3aPfMWW9aeVutboqKNwJb2l5fB8wDVwKnAP8i\npTQSEa134bkcnXy5p3wNQEqpDrwSuDoi5lNK95Tx1mTO1wJHgAe6Ohtgbi5ndtYLqRvmrDf9ytvs\nbE6ezzOXz3dsM5/Pk9fyjm2Wi6/9MYo85XkOa9yPfp1rns8zO9ufa8D3aPfM2fpYcVERET9sf12O\nWMxHxPdTSj8AfgjcklK6FriQ4omOS8rmNwNXpJSuBL4EXAM8FBGtIuIG4MaU0rcpJmzeAHzSxa8k\nSRoclczMiIgceAvFLYx7gXcCF0XEI2X8B8DFFOtW7KV44uOitv0/B3wYuAm4jWKNi6uq6JskSeqP\nnpfpjoh3L3j9EHDeEu1vA05bIn4drqIpSdLA8pkbSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJU\nCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUCYsKSZJUiZ4/+0NSNfI8p9FodIw3GgeYX+Ij\nuCVpo7CokNZZo9Hgjr0PUq+PLxp/6sl9bNs+xThTfe6ZJHXHokLaAOr1cbZPTC4aO3Tw2T73RpJ6\nY1EhSWtgudtaLZOTk2SZ09t0fLCokNbQSn6xOGfi+LTcbS2AZvMgF7z65ezYsaOPPZPWjkWFtIZW\n8ovFORPHr6Vua0nHI4sKaY0t94vFOROSjhfeyJMkSZWwqJAkSZXw9ock9aCYhHugY9wJuNqMLCok\nqQfThw6y+/6n2XHizKJxJ+BqM7KokKQejda3u2iZ1MY5FZIkqRIWFZIkqRIWFZIkqRIWFZIkqRIW\nFZIkqRIWFZIkqRIWFZIkqRIWFZIkqRIWFZIkqRJdr6iZUvpJ4A+B1wH7gT+IiI+VsVOATwHnAA8D\nH4iI29v2vQC4HjgVuBvYFRHfb4u/H7gCGAc+D1weEYuvgStJkjaUrkYqUko14MvAE8D/BPxT4LdS\nSm8vm3wReBQ4E/gscGtK6eRy35cAtwKfBl4FPA18oe3YbwWuBnYB5wOvAa7r9cQkSVJ/dXv748XA\nt4D3RsR/jYj/G/gacG5K6TzgZcBlUfgIxWjEpeW+u4B7IuLjEfFd4N3AKSmlN5Tx9wHXR8RXIuKb\nwGXAe1JKo6s6Q0mS1BddFRUR8XhEvCMiDgGklF4HvB74TxQjC/ctuF2xm+JWCMDZwJ1tx5oG7gPO\nSSllwFnAXW377gG2Amd000dJkrQ+ep6omVJ6mKJIuBv4C+Akilsf7Z4ATi6/Xio+CYy2xyNijmLO\nxslIkqQNbzUffX4xsBP4I4rJl3Xg8II2h4GR8uul4vW21532X5GhIR9oWalWrsxZd7rJ2/BwRpbV\nGMpqHdvUshpZlnVss1x8EI6RZdnRf2tr24/1PtdujpFlNYaHM4aHF7+WfI92z5z1pqp89VxURMR9\nACmlXwf+jGIC5tSCZiNAs/x6hmMLhBHgQBmjQ7xJFyYmxrppLsxZr1aStzyfoT62lXq9c21cH9tK\nbWhLxzbLxQfpGKMjw2vej41yris5xtzzW5mcrDM1ta1jG/A92gtztj66KipSSj8BnBMRX2zb/B2K\nuQ+PAa9YsMvOcjvAvvL1wvi3KG5zzJSvHyy/1xBwQtv+K/Lcc9PMzeXd7LJpDQ1lTEyMmbMudZO3\nRqNJc/oIQ1sWDsId1Zw+QjYEzebibZaLD8IxsixjdHQLM4dnoba2/Vjvc+32GI1GkyxbfD6679Hu\nmbPetPK2Wt2OVLwM+IuU0skR0fpl/yrgSYpJmf88pTQSEa130bkcnXy5p3wNQEqpDrwSuDoi5lNK\n95Tx1mTO1wJHgAe66eDcXM7srBdSN8xZb1aSt9nZnDyfZy6f79hmPp8nr+Ud2ywXH4xjFHnK8xzW\nuB/rf64rP0aezzM7u/x15Hu0e+ZsfXRbVNwD3AvcXN72eBnFWhL/iqIY+CFwS0rpWuBCiic6Lin3\nvRm4IqV0JfAl4BrgoYhoFRE3ADemlL5NMWHzBuCTLn4lSdJg6PaR0hx4C3AI+Gvgk8DHI+IPytiF\nFLcw7gXeCVwUEY+U+/6AYnLnpcBeiic+Lmo79ueADwM3AbdRPFVy1WpOTpIk9U/XEzUj4nHgFzvE\nHgLOW2Lf24DTlohfh6toSpI0kHzmRpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKi\nQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVcKiQpIkVaLrDxSTdFSe5zQajY7xRuMA8/l8H3uk\nQVJcPwc6xoeHM170orE+9khaHYsKaRUajQZ37H2Qen180fhTT+5j2/Ypxpnqc880CKYPHWT3/U+z\n48SZReOHp3/EL03WybLRPvdM6o1FhbRK9fo42ycmF40dOvhsn3ujQTNa397x+smyWp97I62Ocyok\nSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIl\nLCokSVIlLCokSVIlLCokSVIl/JRSSdqg8jznmWeeIctGmZ3NO7abnJwky/wbUevPokLqIM9zGo3G\nMduHhzPyfIZGo0mjcYD5fH4deqfNoHnoIHd84znq2ybJO1xnzeZBLnj1y9mxY0efeycdy6JC6qDR\naHDH3gep18dfsD3LatTHttKcPsITjz/Ctu1TjDO1Tr3U8W6svp3xiUnmLF41ACwqpCXU6+Nsn5h8\nwbahrEa9PsLQlsMcfPbYkQxJ2qy6KipSSn8X+ARwHtAE/gPwwYg4klI6BfgUcA7wMPCBiLi9bd8L\ngOuBU4G7gV0R8f22+PuBK4Bx4PPA5REx0/OZSZKkvup2Zs+fA6PA64C3Az8PXFvGvgg8CpwJfBa4\nNaV0MkBK6SXArcCngVcBTwNfaB00pfRW4GpgF3A+8Brgup7OSJIkrYsVFxUppQS8GrgkIv42Iv4z\nRSHwzpTSecDLgMui8BGK0YhLy913AfdExMcj4rvAu4FTUkpvKOPvA66PiK9ExDeBy4D3pJRGqzhJ\nSZK09roZqXgceHNEPL1g+4soRhbuW3C7YjfFrRCAs4E7W4GImAbuA85JKWXAWcBdbfvuAbYCZ3TR\nP0mStI5WXFRExLML5kjUgMuBrwEnUdz6aPcEcHL59VLxSYpbKj+OR8QcsL9tf0mStMGtZrWU3wVe\nCfwmUAcOL4gfBkbKr5eK19ted9pfkiRtcD09UppS+ijFPIhfjojvpJRmgIUrr4xQPCECMMOxBcII\ncKCM0SHepEtDQ64qt1KtXJmzxQ0PZ2RZjaGs9oLtrZULsyyjltXIsuyYNi3LxVfS5ng4RnvOqK1t\nP9b7XKs8xgvyxuIramZZjeHhjOFh38fgz7VeVZWvrouKlNLvU0ykfFdEtJ7g2AecvqDpTuCxtvjO\nReLforjNMVO+frD8HkPACW37r9jExFi3u2x65mxxeT5DfWwr9friA2ajo1uoj22lNrSlY5vl4itp\nczwdY3RkeM37sVHOtap8QXGtdTL3/FYmJ+tMTW3r2GYz8ufa+uh2nYprgF8F3hYRt7aF9gBXpZRG\nIqJ1G+Ncjk6+3FO+bh2nTnHr5OqImE8p3VPGW5M5XwscAR7o8nx47rlp5uY6r5Gvo4aGMiYmxsxZ\nB41Gk+b0EYa2vPDOXJZljI5uYWbmeZrTR8iGoNlcePeusFx8JW2Oh2P8OGeHZ6G2tv1Y73Ot8hgz\nh2cZqxfXWp4v/h5tTh+h0WiSZT4sB/5c61Urb6u14qIipfQK4LeA3wH+OqX04rbw14EfAreklK4F\nLqR4ouOSMn4zcEVK6UrgS8A1wEMR0SoibgBuTCl9m2LC5g3AJ3tZ/GpuLl/yg3d0LHO2uNnZnDyf\nX2R55CJXeZ4zn8+T1/KOSygvF19Jm+PjGEdzxhr3Y/3PtbpjtAqJPF+qzTyzs76HF/Ln2vro5ibK\nhWX736L4xf8oxe2JRyMiBy6iuIVxL/BO4KKIeAQgIn4AXEyxbsVeiic+LmodOCI+B3wYuAm4jWKN\ni6tWc2KSJKm/VjxSEREfBT66RPy/Uizf3Sl+G3DaEvHrcBVNSZIGlh8oJkkDLM9zGo0DS7aZnJz8\n8ZMk0lqyqJCkATZ96CC773+aHScuPgWt2TzIBa9+OTt2LHzqX6qeRYUkDbjR+na2T0yudzckiwpt\nXsWwcaNjvNE4wPwSM/clSS9kUaFNq9FocMfeB6nXxxeNP/XkPrZtn2KcqT73TJIGk0WFNrV6fbzj\nsPGhg8/2uTeSNNicDixJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJ\nkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiphUSFJkiph\nUSFJkiphUSFJkiphUSFJkiphUSFJkioxvN4dkNZCnuc0Go0l2zQaB5jP5/vUI2l9FO+FA8u2m5yc\nJMv8O1OrY1Gh41Kj0eCOvQ9Sr493bPPUk/vYtn2Kcab62DOpv6YPHWT3/U+z48SZjm2azYNc8OqX\ns2PHjj72TMcjiwodt+r1cbZPTHaMHzr4bB97I62f0fr2Jd8LUlUc65IkSZWwqJAkSZWwqJAkSZXo\neU5FSmkEuBf4XyPiznLbKcCngHOAh4EPRMTtbftcAFwPnArcDeyKiO+3xd8PXAGMA58HLo+IzrOL\nJEnShtHTSEVZUPx74PQFoS8AjwJnAp8Fbk0pnVzu8xLgVuDTwKuAp8v2rWO+Fbga2AWcD7wGuK6X\n/kmSpP7ruqhIKb0C2AO8bMH28ylGIC6LwkcoRiMuLZvsAu6JiI9HxHeBdwOnpJTeUMbfB1wfEV+J\niG8ClwHvSSmN9nJikiSpv3oZqXgj8DWKWxy1tu1nA/ctuF2xu2zXit/ZCkTENHAfcE5KKQPOAu5q\n23cPsBUiyjuVAAANfUlEQVQ4o4c+SpKkPut6TkVE3Nj6OqXUHjqJ4tZHuyeAk1cQnwRG2+MRMZdS\n2l/Gv9FtPyVJUn9VufhVHTi8YNthYGQF8Xrb6077r8jQkA+0rFQrV4OYs+WW4f7Rj56lVoOhrNax\nTS2rkWVZxzad4q2ljLMs6/kYVfRjkI7RnjNqa9uP9T7XKo/xgryRr1k/sqzG8HDG8PDg/SxYaJB/\nrq2nqvJVZVExAyxc43UEaLbFFxYII8CBMkaHeJMuTEyMddNcDGbO9u/fz133f59t2xZfhvvJx/ex\nbWKKer1zTVof20ptaEvHNsvFR0e3rPoYVfRjkI4xOjK85v3YKOdaVb6guNbWsh9zz29lcrLO1NS2\njm0GzSD+XDseVFlU7OPYp0F2Ao+1xXcuEv8WsJ+isNgJPAiQUhoCTmjbf0Wee26aubnFK3q90NBQ\nxsTE2EDmrNFoUstGGNpSX7xBtpXp6edpNhcOfh3VnD5CNkTHNp3iWZYxOrqFmZnnez5GFf0YpGP8\nOGeHZ6G2tv1Y73Ot8hgzh2cZqxfXWp4v/h6toh8/OjTDww/vo9Ho/DfcoHzg2CD/XFtPrbytVpVF\nxR7gqpTSSES0rt5zOTr5ck/5GoCUUh14JXB1RMynlO4p463JnK8FjgAPdNOJubmc2VkvpG4MYs5m\nZ3PyfJ65Dp8yOp/Pk9fyjvGVtOkcL3KV5/kqjlFFPwbpGEdzxhr3Y/3PtbpjtAqJPF/bfBw6+Bz/\n6d6n2HHi4kXFIH7g2CD+XDseVFlUfB34IXBLSula4EKKJzouKeM3A1eklK4EvgRcAzzUWjgLuAG4\nMaX0bYoJmzcAn3TxK0lae37omKqw2rGsH5e+EZEDb6G4hXEv8E7gooh4pIz/ALiYYt2KvRRPfFzU\ntv/ngA8DNwG3UaxxcdUq+ydJkvpkVSMVETG04PVDwHlLtL8NOG2J+HW4iqYkSQNp48+6kSRJA8Gi\nQpIkVcKiQpIkVaLKpz+kSiy3WiZAo3GA+SUekZMk9Z9FhTacRqPBHXsfpF5ffLVMgKee3Me27VOM\nM9XHnkmbU1HoH1i23aAskKW1Y1GhDaleH1/ymflDB5/tY2+kzW360EF23/80O07svGzQIC6QpepZ\nVEiSluXiWFoJx6kkSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIlLCokSVIl\nLCokSVIlLCokSVIlXKZbkrRqK/nQMT9w7PhnUSFJWrXlPnTMDxzbHCwqJEmV8EPH5DiUJEmqhCMV\n6rvi3mujY7zROMB8Pt/HHkmSqmBRoUotVzBAUTTc87dPsW37+KLxp57cx7btU4wztRZdlCStEYsK\nVarRaHDH3gep1xcvGOBo0dDp3uuhg8+uVfckSWvIokKVq9fHl5ysZdEgbT4reeQUfOx00FlUSJLW\n3HKPnIKPnR4PLCokSX3hI6fHP8eYJElSJSwqJElSJbz9IUnaEPz8kMFnUaGuuHCVpLXi54cMPosK\ndWW5dShcuErSajiZc7BtqKIipTQC3ABcDDSB34uIf7O+vdJCS61D4RoUktbKSm6PDA9nvOhFY33q\nkRbaUEUF8DHgp4GfAU4B/jSl9HBE/MV6dmqzWOkS297ekLQeVrLWxeHpH/FLk3WybLSPPVPLhikq\nUkp14D3AmyLiAeCBlNJ1wOWARUUFFhYNw8MZeT5Do9FkdjZf9jM5wNsbktbXcrdHsqzWx95ooQ1T\nVABnUPTn7rZtu4HfWJ/uHH8WzofIshr1sa00p4+Q5/PLfiYHeHtD0saW5znPPPMMWTbK7Gy+aBuf\nIFk7G6moOAl4OiJm27Y9AYymlE6IiP3r1K+BsNJbF2OjR6v8oaxGvT7C0JbDzOXzFgySBl7z0EHu\n+MZz1LdNki9yq/bQj57lrNNezOTk0qOtFh692UhFRR04vGBb6/XISg8yNFT9RfDMM89UfsyqNRoH\n+Po3H2JktPMEpQP7n6S+fYqh4SJHWZYx9/wwM4dnyfOcwzOHqA1tpfmjzsXFcm02wzHa8zbo59Kv\nY7RydnimCbXhNe3Hep9rtcdokmXzDA1tJc8X/6t7cM6lX8doMlYfKwuCY3N2ePoQd+z5WyanOhcV\nh2emeeOZpy5bePRDvx6frep350YqKmY4tnhovW6u8Bi1iYnqZ/1OTW2r/JjVewlnnvk/rncnNqGz\n1rsDA+aV690BSWtoI43t7ANOTCm192knMB0RS4/rS5KkdbeRior7geeB17Rtez1wz/p0R5IkdaM2\nP79x1hxIKf0R8DrgUuBk4BbgVyLii+vZL0mStLyNNKcC4NcpVtT8K+BZ4LctKCRJGgwbaqRCkiQN\nro00p0KSJA0wiwpJklQJiwpJklQJiwpJklQJiwpJklSJjfZIaU9SSv87cBnF+fw58L9FxJEytgP4\nFPAPgaeAqyPiz9arrxtNSukPgdMj4ry2beZsESmlFwG/B/zPFAX5l4H3R8SzZdy8LSKlNELxqPjF\nFEvu/15E/Jv17dXGklL6u8AngPMocvQfgA9GxJGU0ikU19U5wMPAByLi9nXq6oaVUvoy8EREXFq+\nPgXzdoyU0lbgeuAdFJ+vdXNE/GYZO4VV5mzgRypSSv8C+KfA24A3A+cD17Q1+RNgHDgb+NfAH6eU\nXtXvfm5EKaXXUuRu4XPF5mxxNwH/A8V19nPAKyjegC3mbXEfA34a+BngvcA1KaWL17VHG8+fA6MU\ni/+9Hfh54Noy9kXgUeBM4LPArSmlk9ejkxtVSuntwD9asPkLmLfFfAL4WYo/ft4J7Eop7Spjq77W\nBnqdivJzQh4D/llEfLbc9g6KVTjfnFL6SeD/A14aET8s458ChlrV7GaVUtoC3Ac0gOcj4vxy+6nA\n9zBnL5BSqlPk6rURcW+57TXAncB24CV4rR2jzNvTwJsi4q5y228CP9u65ja7lFICvgO8OCKeLre9\nHfhd4J9Q/KD/iYiYKWO3A3dFxIfWqcsbSkppCniA4pfhdyLi0pTS+RRFhXlrU+bqCeD8iNhdbrsS\neDnwZ1RwrQ36SMVPASdQJAKAiPj3EfHm8uWrgf/W+iFf2k0xtLPZfZDijXjHgu1nY84Wk1Pc9nig\nbVsNGKIoKrzWFncGxW3Ju9u27aa4zlR4HHhzq6Bo8yKKz0K6r/VDvuR19UIfA/4U+G7btrMxb4s5\nF2i0CgqAiLguIv4XKrrWBn1OxanAM8DrUkq/A5xIMYx4VTmn4iSK6rXdExSfK7JppZROo7jtcQbF\ncHQ7c7aI8o321QWbfw34m4h4JqVk3hZ3EvB0RMy2bXsCGE0pnRAR+9epXxtGOSfnx/etU0o14HLg\na/h+XFI5IvF6ituSN7aFzNviTgUeTin9Y+A3gK3AZyhu11aSsw1fVKSURoG/1yH8ImAb8GHg/RTn\ncxPFCMyvAXWKiSjtDgMja9LZDWKZnD1GkaOrI+KpYuT1BTZlzmD5vEVEs63t5cAvAm8qN23avC2j\nU17A3HTyu8ArgbMoPg/J62oR5QTgG4H3RsThBT/LfD8ubjvFrY5fBS6hKCRuopgcXEnONnxRQTGM\n9f9w7GRCKCaZjFE87dG6P/TPgH9HUVTMcGxCRigSeDxbKmcfBLKI+OMO+27WnMHSefsF4C8BUkrv\nBf4P4Nci4mtlfDPnbSmd8gLm5hgppY8C7wN+OSK+k1KaAXYsaOZ1VfiXwD0RsfAWLhTXnXk71izF\nZPJ3RMQjACmll1KMWH+VYjpBu65ztuGLioj4Oh3mfqSU3kDxCyDad6EYWv07wD5g54LddlL8tX7c\nWiZnfwW8KqV0sNy0FRhKKT0HnM4mzRksnbeWlNIVwHUUk4P/oC20afO2jH3AiSmlLCLycttOYDoi\nGuvYrw0npfT7FI/GvysivlBu3kfxvmzndVV4G/Ditp9lIwAppV8EfgfztpjHgJlWQVEKilsc+yjm\nKbbrOmeDPlHzW8ARirkBLacDB4H9wB7gpeUz4C3nlts3q3dRXDhnlP/dCNxTfv0o5qyjlNKvAB+l\nGKG4fkHYvC3ufuB5iklgLa+nuOZUSildQzEk/baI+HxbaA/w0+VQf4vXVeGNFHMpWj/L/pJi0v4Z\nwDcwb4vZQ/FH999v23Y6xZoUe4AzV5uzgX6kFH5c3V9AcX8oo1gr4IsR8c/L+H+keP771yhm6H8C\neENEfHNdOrzBlD/M3tj+eJ85O1b5KNYPgP+L4hZSuycjYt68LS6l9EcU6y9cSvEX0S0Uj31/can9\nNouU0iuAv6H46/qGBeGnKJ44+i8U61ZcSHH9/dSCvzY3vZTSZ4D58pHSDPO2qJTSX1LcGnovxZyK\nPwU+BPwRxXX4/7KKnA36SAXAB4CvAP8R+FL572+0xf8J8BxFtfVB4N2b/Yf8CpizY/0cxaTgX6EY\n0XmUYljwUYo1Kihj5u1Yvw58E/gr4PeB37ageIELKX4W/xYLrq3yltFFFMPQ91LMI7tos/9iXE6Z\nt7dg3hbzLoq1iO6iKPA/ERF/WObsQlaZs4EfqZAkSRvD8TBSIUmSNgCLCkmSVAmLCkmSVAmLCkmS\nVAmLCkmSVAmLCkmSVAmLCkmSVAmLCkmSVAmLCkmSVAmLCkmSVAmLCkmSVIn/H18t/TPwlt+oAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c330358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment_diff_mean = shuffle_experiment(100000)\n",
    "sns.distplot(experiment_diff_mean, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of times diff in mean greater than observed: 40498\n",
      "% of times diff in mean greater than observed: 40.498\n"
     ]
    }
   ],
   "source": [
    "#Finding % of times difference of means is greater than observed\n",
    "print(\"Number of times diff in mean greater than observed:\", \\\n",
    "            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0])\n",
    "print(\"% of times diff in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Did the conclusion change now? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effect Size\n",
    "\n",
    "> **Because you can't argue with all the fools in the world. It's easier to let them have their way, then trick them when they're not paying attention**  - Christopher Paolini\n",
    "\n",
    "In the first case, how much did the price optimization increase the sales on average?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The % increase of sales in the first case: 17.5438596491 %\n"
     ]
    }
   ],
   "source": [
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])\n",
    "\n",
    "print(\"The % increase of sales in the first case:\", \\\n",
    "(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,\"%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The % increase of sales in the second case: 1.75438596491 %\n"
     ]
    }
   ],
   "source": [
    "before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])\n",
    "after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])\n",
    "\n",
    "print(\"The % increase of sales in the second case:\", \\\n",
    "(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Would business feel comfortable spending millions of dollars if the increase is going to be just 1.75%. Does it make sense? Maybe yes - if margins are thin and any increase is considered good. But if the returns from the price optimization module does not let the company break even, it makes no sense to take that path.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Someone tells you the result is statistically significant. The first question you should ask?\n",
    "\n",
    "# How large is the effect?\n",
    "\n",
    "To answer such a question, we will make use of the concept **confidence interval**\n",
    "\n",
    "In plain english, *confidence interval* is the range of values the measurement metric is going to take. \n",
    "\n",
    "An example would be: 90% of the times, the increase in average sales (before and after price optimization) would be within the bucket `3.4 and 6.7` (These numbers are illustrative. We will derive those numbers below)\n",
    "\n",
    "What is the *hacker's way* of doing it? We will do the following steps:\n",
    "\n",
    "1. From actual sales data, we sample the data with repetition (separately for before and after) - sample size will be the same as the original\n",
    "2. Find the differences between the mean of the two samples.\n",
    "3. Repeat steps 1 and 2 , say 100,000 times.\n",
    "4. Sort the differences. For getting 90% interval, take the 5% and 95% number. That range gives you the 90% confidence interval on the mean.\n",
    "5. This process of generating the samples is called **bootstrapping**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load the data\n",
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#generate a uniform random sample\n",
    "random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sample before optimization: [23 21 19 24 35 17 18 24 33 27 21 23]\n",
      "Bootstrapped sample before optimization:  [17 33 24 33 21 23 23 23 23 19 21 24]\n"
     ]
    }
   ],
   "source": [
    "print(\"Actual sample before optimization:\", before_opt)\n",
    "print(\"Bootstrapped sample before optimization: \", random_before_opt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean for actual sample: 23.75\n",
      "Mean for bootstrapped sample: 23.6666666667\n"
     ]
    }
   ],
   "source": [
    "print(\"Mean for actual sample:\", np.mean(before_opt))\n",
    "print(\"Mean for bootstrapped sample:\", np.mean(random_before_opt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sample after optimization: [31 28 19 24 32 27 16 41 23 32 29 33]\n",
      "Bootstrapped sample after optimization:  [41 29 23 31 32 32 31 32 31 33 24 24]\n",
      "Mean for actual sample: 27.9166666667\n",
      "Mean for bootstrapped sample: 30.25\n"
     ]
    }
   ],
   "source": [
    "random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)\n",
    "print(\"Actual sample after optimization:\", after_opt)\n",
    "print(\"Bootstrapped sample after optimization: \", random_after_opt)\n",
    "print(\"Mean for actual sample:\", np.mean(after_opt))\n",
    "print(\"Mean for bootstrapped sample:\", np.mean(random_after_opt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference in means of actual samples: 4.16666666667\n",
      "Difference in means of bootstrapped samples: 6.58333333333\n"
     ]
    }
   ],
   "source": [
    "print(\"Difference in means of actual samples:\", np.mean(after_opt) - np.mean(before_opt))\n",
    "print(\"Difference in means of bootstrapped samples:\", np.mean(random_after_opt) - np.mean(random_before_opt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Like always, we will repeat this experiment 100,000 times. \n",
    "\n",
    "def bootstrap_experiment(number_of_times):\n",
    "    mean_difference = np.empty([number_of_times,1])\n",
    "    for times in np.arange(number_of_times):\n",
    "        random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)\n",
    "        random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)\n",
    "        mean_difference[times] = np.mean(random_after_opt) - np.mean(random_before_opt)\n",
    "    return mean_difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1105ec080>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAFoCAYAAADgsAn8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X2UXOV94Plv32qpW4XU6RYkEYkcA8nhB86cZTDm1cQJ\nmIkzZzdYwZ68kJ0NL9Ew62G8xsvAOHFg12xiR5sE4mQIBpsQj5NZjyeRnSHrIZCZGJSgSCAgG4c8\nTAw4IDCvKiSruiW6b+8f9xYqWtVNV/dVdXXr+zlHp6vu7z63n3r6qvrXz1sNTE9PI0mStFjZUldA\nkiStDCYVkiSpEiYVkiSpEiYVkiSpEiYVkiSpEiYVkiSpEiYVkiSpEiYVkiSpEiYVkiSpEoPdFoiI\n7wR+B3gv8BLwyyml3ytjJwB3AOcCTwPXpJTubSt7EXAzcBLwILA5pfRUW/wjwLXAOuBLwNUppYmF\nvDBJktRbC+mp+DLwPcAPAx8BfiMiNpWxrwDPAWcAXwC2RsRGgIh4G7AV+BzwLuDl8lqU8Q8ANwCb\ngQuBc4AtC6ifJElaAgPdfPZHRJwB7ABOSil9szx2HbAJ+EWKpOK7Wr0LEXEv8EBK6RMR8Qng/JTS\nhWVsDfAt4MdTSvdHxNeA+1JKN5XxdwN/Chxrb4UkSf2v256Kk4CXWglF6a8peh5+CNg1IwHYRjEU\nAnA2cH8rkFIaB3YB50ZEBpwJPNBWdjuwGjityzpKkqQl0G1S8QIwGhHDbce+j2JuxndTDH3MPH9j\n+fj4OeKjwHB7PKU0BbzSVl6SJPWxbidq/hXwPPDbEfFhirkV1wDTFEnBgRnnHwCGysf1OeL1tuez\nlZckSX2sq6QipXQgIj4I/EdgL0VPwxaKFR05sGZGkSGgWT6e4PAEYQjYU8aYJd5knqanp6cHBgbm\ne7okSTpk0b9Au15SmlJ6GPj+iPguihUc76NYWvoN4EdnnL6BomcDYHf5fGb8EYphjony+RMAEVED\njm0r/5YGBgbYu3ecqam8m5ekBarVMkZG1tjmPWSb955t3nu2ee+12nyxukoqImIM+GPg4pTSi+Wx\n/wn4c4qhkY9FxFBKqTWMcT6HJl9uL5+3rlUHTgduSClNR8TOMt6azHkecBB4rJs6Tk3lTE56E/aS\nbd57tnnv2ea9Z5svP90Of+yJiGOALRHxKxQbYF1GsfLjUeAfgLsi4ibgYooVHZeVxe8Eri2XoN4N\n3Ag8mVJqJRG3ArdFxNcpJmzeCtzuclJJkpaHhWx+9VPAD1AsJf0w8MGU0q6UUg68n2II4yHgUmBT\nSulZgHIZ6iXAFRR7XYxS7G9BGf8i8EngM8A9FDtuXr+wlyVJknqtq82vloHpPXv2213WI4ODGWNj\nx2Cb945t3nu2ee/Z5r1XtvmiJ2r6gWKSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkS\nJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWS\nJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSJhWSJKkSg90WiIiNwO8A7wFeAX4zpfSb\nZewE4A7gXOBp4JqU0r1tZS8CbgZOAh4ENqeUnmqLfwS4FlgHfAm4OqU0sZAXJunolec5jUZjznNG\nR0fJMv+ukqrUdVJB8cv+KeCdwA8CfxART6eUvgJ8BXgUOAP4CWBrRJySUno2It4GbAV+CbgHuBH4\nMnAaQER8ALgB+FngReD3gC3Ahxf+8iQdjRqNBvfteIJ6fV3HeLO5j4vOOpn169f3uGbSytZVUhER\no8DZwJUppW8A34iI/wK8NyL2AicCZ5e9C5+KiPcCVwCfADYDO1NKt5TXuhz4VkS8J6V0P0XycHNK\n6atl/CrgTyPiOnsrJHWrXl/H2pHRpa6GdFTptu9vHNgPXB4RgxERwLuBR4BzgF0zEoBtFEMhUCQj\n97cCKaVxYBdwbkRkwJnAA21ltwOrKXsyJElSf+sqqUgpHQCuBv4lRYLxOPD/ppR+FzgeeG5GkReA\njeXjueKjwHB7PKU0RTFnYyOSJKnvLWSW0qnAHwNnAZcBH4yIS4E6cGDGuQeAofLxXPF62/PZykuS\npD7W7ZyK9wJXAhvLXotHytUgHwf+DDh2RpEhoFk+nuDwBGEI2FPGmCXepAu1mrO5e6XV1rZ579jm\n8zM4mJFlA9SygY7xLBtgcDBjcPCt29E27z3bvPeqautuV3+8E/jvZULR8gjwC8BuitUg7TYAz5eP\nd5fPZ8YfoRjmmCifPwEQETWKJOV5ujAysqab01UB27z3bPO55fkE9TWrqdc7d3ROvb6a0dE6Y2PH\nzPuatnnv2ebLT7dJxXPAD0TEYEppsjx2KsUS0+3AxyJiqC3pOJ9Dky+3l88BiIg6cDpwQ0ppOiJ2\nlvHWZM7zgIPAY91UcO/ecaam8i5flhaiVssYGVljm/eQbT4/jUaT5vhBaqtmjqgWmuMHaTSaZNnw\nW17LNu8927z3Wm2+WN0mFf+ZYu+Iz0bELwOnAB8r/90PPAPcFRE3ARdTrOi4rCx7J3BtRFwH3E2x\nT8WT5XJSgFuB2yLi6xTJy63A7d0uJ52aypmc9CbsJdu892zzuU1O5uT5NFP5dMd4nk8zOdldG9rm\nvWebLz/drv7YC7yXYiXHDuDXgU+klD6bUsopEokNwEPApcCmlNKzZdlvApdQ7Fuxg2LFx6a2a38R\n+CTwGYrNsR4Erl/Mi5MkSb0zMD3dOZNfpqb37NlvZtsjg4MZY2PHYJv3jm0+P6+++ip/+TfPz7r5\n1bf3NjjvHx0/rx01bfPes817r2zzzjObu+DUWkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmT\nCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmS\nVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVInBbk6OiJ8D\nfheYBgbavuYppcGIOBG4HTgXeBq4JqV0b1v5i4CbgZOAB4HNKaWn2uIfAa4F1gFfAq5OKU0s+NVJ\nkqSe6ban4v8BNgDHl1/fDvw9cEsZ/zLwHHAG8AVga0RsBIiItwFbgc8B7wJeLs+njH8AuAHYDFwI\nnANsWciLkiRJvddVUpFSOpBSerH1D/jnZehjEXEhcCJwVSp8iqI34orynM3AzpTSLSmlx4HLgRMi\n4j1l/MPAzSmlr6aUHgauAq6MiOHFvURJktQLC55TERFjwHXA9Sml14GzgV0zhiu2UQyFUMbvbwVS\nSuPALuDciMiAM4EH2spuB1YDpy20jpIkqXcWM1HzQ8DulNLW8vnxFEMf7V4ANs4jPgoMt8dTSlPA\nK23lJUlSH+tqouYMVwKfanteBw7MOOcAMDSPeL3t+Wzl56VWc0FLr7Ta2jbvHdt8fgYHM7JsgFo2\n0DGeZQMMDmYMDr51O9rmvWeb915Vbb2gpCIizgS+F/hi2+EJYP2MU4eAZlt8ZoIwBOwpY8wSb9KF\nkZE13ZyuCtjmvWebzy3PJ6ivWU293vlvkqnXVzM6Wmds7Jh5X9M27z3bfPlZaE/F+4D7U0qvtR3b\nDbxjxnkbgOfb4hs6xB+hGOaYKJ8/ARARNeDYtvLzsnfvOFNTeTdFtEC1WsbIyBrbvIds8/lpNJo0\nxw9SWzWz87PQHD9Io9Eky956Hrht3nu2ee+12nyxFppUnA38xYxj24HrI2IopdT6n3w+hyZfbi+f\nAxARdeB04IaU0nRE7Czjrcmc5wEHgce6qdjUVM7kpDdhL9nmvWebz21yMifPp5nKpzvG83yaycnu\n2tA27z3bfPlZaFLxj4B/P+PY14BngLsi4ibgYooVHZeV8TuBayPiOuBu4EbgyZRSK4m4FbgtIr5O\nMWHzVuB2N7+SJGl5WOjMjO+imAvxhpRSDryfYgjjIeBSYFNK6dky/k3gEop9K3ZQrPjY1Fb+i8An\ngc8A91DscXH9AusnSZJ6bEE9FSmljrObUkpPAhfMUe4e4JQ54ltwF01JkpalxSwplaRlKc9zGo09\nc54zOjpKlrmkUeqGSYWko874/n1se/Rl1h/XecpWs7mPi846mfXrZ66SlzQXkwpJy07R09CYNd5o\n7GF6lpUfLcP1tawdGa26atJRzaRC0rLTaDS4b8cT1OvrOsZfenE3x6wdYx1jPa6ZdHQzqZC0LNXr\n62btadi/77WOxyUdWc5CkiRJlbCnQpJmaF8dMjiYkecTNBrNN+3u6OoQ6XAmFZI0Q/vqkCwboL5m\nNc3xg+Tl5E9Xh0idmVRIUget1SG1bIB6fYjaqgOzfpaIpIJ9d5IkqRImFZIkqRImFZIkqRLOqZDU\nd6rYMVNS75lUSOo77pgpLU8mFZL6kjtmSsuPcyokSVIlTCokSVIlTCokSVIlTCokSVIlTCokSVIl\nTCokSVIlTCokSVIlut6nIiJWAzcDPwMcAO5MKf1iGTsBuAM4F3gauCaldG9b2YvKsicBDwKbU0pP\ntcU/AlwLrAO+BFydUppYyAuTJEm9tZCeik8D7wX+CXApsDkiNpexrwDPAWcAXwC2RsRGgIh4G7AV\n+BzwLuBl4Muti0bEB4AbgM3AhcA5wJYF1E+SJC2BrpKKiBgDrgB+PqX0cErpvwG/BpwdERcAJwJX\npcKnKHojriiLbwZ2ppRuSSk9DlwOnBAR7ynjHwZuTil9NaX0MHAVcGVEDC/2RUqSpCOv256K84FG\nSmlb60BKaUtK6ecpehZ2zRiu2EYxFAJwNnB/W7lxYBdwbkRkwJnAA21ltwOrgdO6rKMkSVoC3c6p\nOAl4OiL+OfALFL/0fxf4ZeB4iqGPdi8AG8vHc8VHgeH2eEppKiJeKeN/1WU9JUlSj3WbVKwFTgb+\nBXAZRaLwGaAJ1CkmbrY7AAyVj+eK19uez1ZekiT1sW6TikmKlRk/k1J6FiAi3g58CPhT4NgZ5w9R\nJBwAExyeIAwBe8oYs8SbdKFWc5Vsr7Ta2jbvnaOlzQcHM7JsgFo20DE+kA2QZVlP4llWtHXxNS8f\nDzA4mDE4uLJ/DkvlaLnP+0lVbd1tUvE8MNFKKEqJYohiN/CDM87fUJahjG/oEH8EeIUisdgAPAEQ\nETWKJOV5ujAysqab01UB27z3Vnqb5/kE9TWrqdc7d1TW16xmoLaqp/Hh4VVvPJ56fTWjo3XGxo6Z\n70vSAqz0+3wl6jap2A4MR8QPpJT+vjz2Doo9KbYDH4uIoZRSaxjjfA5NvtxePgcgIurA6cANKaXp\niNhZxluTOc8DDgKPdVPBvXvHmZrKu3xZWohaLWNkZI1t3kNHS5s3Gk2a4weprZo5Ilpojh8kq0Gz\neeTjWZYxPLyKiYnXyfP8jXij0STLXJx2JBwt93k/abX5YnWVVKSUnoiIPwHuiogPUcypuB74BEUy\n8EwZuwm4mGJFx2Vl8TuBayPiOuBu4EbgyZRSK4m4FbgtIr5OMWHzVuD2bje/mprKmZz0Juwl27z3\nVnqbT07m5Pk0U/l0x/h0Pk0+kPcoXrRznh86P8+nmZxc2T+DfrDS7/OVqOsdNYGfBX6LogeiCXw6\npfTvACLiYorNrR4C/h7Y1BoqSSl9MyIuAX6TYpOrvwA2tS6aUvpiOT/jMxSrSv4TRcIiaYXJ85xG\nozFrvNHYw/Qsv/Al9a+uk4qU0j6K3ofLOsSeBC6Yo+w9wClzxLfgLprSitdoNLhvxxPU6+s6xl96\ncTfHrB1jHWM9rtn8FEnRnjnPGR0dfWOSp3S0WEhPhSQtWr2+jrUjox1j+/e91uPadGd8/z62Pfoy\n64/rPDrbbO7jorNOZv369T2umbS0TCokaQGG62tnTYqko5V9c5IkqRImFZIkqRImFZIkqRImFZIk\nqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRImFZIkqRJ+\nSqkkVSzPcxqNPXOeMzo6Spb5d51WFpMKSarY+P59bHv0ZdYfN9Ex3mzu46KzTmb9+vU9rpl0ZJlU\nSKpc8Zd6Y9Z4o7GH6Xy6hzXqveH6WtaOjC51NaSeMqmQVLlGo8F9O56gXl/XMf7Si7s5Zu0Y6xjr\ncc0kHUkmFZKOiHp93ax/qe/f91qPayOpF5wlJEmSKmFSIUmSKmFSIUmSKtH1nIqI2AT8ETANDJRf\n/zCl9JMRcQJwB3Au8DRwTUrp3rayFwE3AycBDwKbU0pPtcU/AlwLrAO+BFydUuq8JkuSJPWVhfRU\nvAP4Y2BD+e944OfL2FeA54AzgC8AWyNiI0BEvA3YCnwOeBfwMvDl1kUj4gPADcBm4ELgHGDLAuon\nSZKWwEJWf5wK/E1K6aX2gxFxIXAicHbZu/CpiHgvcAXwCYpkYWdK6Zby/MuBb0XEe1JK9wMfBm5O\nKX21jF8F/GlEXGdvhSRJ/W+hPRVPdDh+NrBrRgKwjWIopBW/vxVIKY0Du4BzIyIDzgQeaCu7HVgN\nnLaAOkqSpB5bSE9FAD8WEb8I1CjmPtxAMQzy3IxzXwA2lo/nio8Cw+3xlNJURLxSxv9qAfWUJEk9\n1FVSERHfB6wBxoF/RjHc8enyWB04MKPIAWCofDxXvN72fLby81KruaClV1ptbZv3znJp88HBjCwb\noJYNdIwPZANkWbYs4q0P/Sq+5pVcP8sGGBzMGBzs75/jUlku9/lKUlVbd5VUpJT+ISKOTSm1NvX/\n64ioUUzK/F04bM/dIaBZPp7g8ARhCNhTxpgl3qQLIyNrujldFbDNe6/f2zzPJ6ivWU293vlvgvqa\n1QzUVi2r+PDwqsquP/X6akZH64yNHdMxrkK/3+c6XNfDH20JRcvjFEMX36KYxNluA/B8+Xh3+Xxm\n/BHgFYrEYgPlfI0yWTm2rfy87N07ztRU3k0RLVCtljEyssY276Hl0uaNRpPm+EFqq2Z2Phaa4wfJ\natBs9n88yzKGh1cxMfE6eZ5Xdv1Go0mWDXeMH+2Wy32+krTafLG6Hf74UeAPgI1tEzJPp1ge+gBw\nbUQMpZRa/5PO59Dky+3l89a16mXZG1JK0xGxs4y3JnOeBxwEHuumjlNTOZOT3oS9ZJv3Xr+3+eRk\nTp5PMzXLJ5FO59PkA/kyiRftnOeHzl/s9fN8msnJ/v4Z9oN+v891uG57Kv6SYjjisxHxCeD7KfaS\n+FWKZOAZ4K6IuAm4mGJFx2Vl2Tspko7rgLuBG4Eny+WkALcCt0XE1ykmbN4K3O5yUkmSloeuZmak\nlL4NvA/4TmAnxe6Zt6WUfj2llFMkEhuAh4BLgU0ppWfLst8ELqHYt2IHxYqPTW3X/iLwSeAzwD0U\nO25ev5gXJ0mSemchcyoep0gsOsWeBC6Yo+w9wClzxLfgLpqSVrg8z2k09sx5zujo6BsrT6TlYiH7\nVEiSFmF8/z62Pfoy64/rPLrbbO7jorNOZv369T2umbQ4JhWStASG62tZOzK61NWQKmXfmiRJqoRJ\nhSRJqoRJhSRJqoRJhSRJqoRJhSRJqoRJhSRJqoRLSiV1rdi8aeZnCx7SaOxhepbPvZC0cplUSOpa\no9Hgvh1PUK+v6xh/6cXdHLN2jHWM9bhmkpaSSYWkBanX1826edP+fa/1uDaS+oFzKiRJUiVMKiRJ\nUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiUW\n/NkfEfEnwAsppSvK5ycAdwDnAk8D16SU7m07/yLgZuAk4EFgc0rpqbb4R4BrgXXAl4CrU0oTC62f\nJEnqrQX1VETETwP/dMbhLwPPAWcAXwC2RsTG8vy3AVuBzwHvAl4uz29d7wPADcBm4ELgHGDLQuom\nSZKWRtdJRUSMUfzC39F27EKKHoirUuFTFL0RV5SnbAZ2ppRuSSk9DlwOnBAR7ynjHwZuTil9NaX0\nMHAVcGVEDC/0hUmSpN5aSE/FrwGfBx5vO3Y2sGvGcMU2iqGQVvz+ViClNA7sAs6NiAw4E3igrex2\nYDVw2gLqJ0mSlkBXSUXZI/FDwE0zQsdTDH20ewHYOI/4KDDcHk8pTQGvtJWXJEl9bt5JRUQMAbcB\nH0opHZgRrgMzjx0AhuYRr7c9n628JEnqc92s/vg/KOZF3NchNgGsn3FsCGi2xWcmCEPAnjLGLPEm\nXarVXCXbK622ts17p1/afHAwI8sGqGUDHeMD2QBZlq2IeJYVbV18zXvy/bNsgMHBjMHBo/P/Vr/c\n50eTqtq6m6Tip4Dvjoh95fMhgIj4IPArwDtmnL8BeL58vLt8PjP+CMUwx0T5/InymjXg2Lby8zYy\nsqbbIlok27z3lrrN83yC+prV1OudOxPra1YzUFu1ouLDw6t69v2nXl/N6GidsbFjOsaPFkt9n6t7\n3SQVPwysanu+BZgGrgNOAP5tRAy1DY2cz6HJl9vL5wBERB04HbghpTQdETvLeGsy53nAQeCxrl4N\nsHfvOFNTebfFtAC1WsbIyBrbvIf6pc0bjSbN8YPUVs0ctSw0xw+S1aDZXP7xLMsYHl7FxMTr5Hne\nk+//7f0TPP30bhqN2TtrR0dH3+hFWWn65T4/mrTafLHmnVSklJ5pf172WEynlJ6KiG8CzwB3RcRN\nwMUUKzouK0+/E7g2Iq4D7gZuBJ5MKbWSiFuB2yLi6xQTNm8Fbl/I5ldTUzmTk96EvWSb914v2jzP\ncxqNRsdYo7GHqcmcqXy6Y3w6nyYfWCnxop3z/ND5R/r779+3lz9/6CXWH9c5qWg293HRWSezfv3M\nUeeVxfeW5WfBO2q2SynlEfF+is2tHgL+HtiUUnq2jH8zIi4BfpNik6u/ADa1lf9iRLwd+AzFUtL/\nBFxfRd0kLUyj0eC+HU9Qr687LPbSi7s5Zu0Y6xhbgpodHYbra1k7MrrU1ZC6suCkIqV0+YznTwIX\nzHH+PcApc8S34C6aUl+p19d1/MW2f99rS1AbSf1uZQ7ISZKknjOpkCRJlTCpkCRJlTCpkCRJlTCp\nkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJlTCpkCRJ\nlTCpkCRJlTCpkCRJlRhc6gpIkrqT5zmNxp45zxkdHSXL/LtRvWVSIUnLzPj+fWx79GXWHzfRMd5s\n7uOis05m/fr1Pa6ZjnYmFZK0DA3X17J2ZHSpqyG9iUmFdJQqutAbs8YbjT1M59M9rJGk5c6kQjpK\nNRoN7tvxBPX6uo7xl17czTFrx1jHWI9rJmm5MqmQjmL1+rpZu9D373utx7WRtNw5NViSJFWi656K\niPh+4N8B7wZeAX47pfRrZewE4A7gXOBp4JqU0r1tZS8CbgZOAh4ENqeUnmqLfwS4FlgHfAm4OqXU\neXqzJEnqK131VETEAPAnwAvAPwb+JfDxiPjp8pSvAM8BZwBfALZGxMay7NuArcDngHcBLwNfbrv2\nB4AbgM3AhcA5wJaFvjBJktRb3Q5/fDfwCPChlNI3Ukr/Bfgz4PyIuAA4EbgqFT5F0RtxRVl2M7Az\npXRLSulx4HLghIh4Txn/MHBzSumrKaWHgauAKyNieFGvUJIk9URXSUVK6VsppZ9JKe0HiIh3Az8E\n/DlFz8KuGcMV2yiGQgDOBu5vu9Y4sAs4NyIy4Ezggbay24HVwGnd1FGSJC2NBU/UjIinKZKEB4E/\nAo6nGPpo9wKwsXw8V3wUGG6Pp5SmKOZsbESSJPW9xSwpvQTYAPwOxeTLOnBgxjkHgKHy8Vzxetvz\n2crPS63mgpZeabW1bd47Vbb54GBGlg1QywY6xgeyAbIs6xifK7bS4q3Pzyi+5n1Xv86m+fa3X2Nw\ncPb7pJ8/G8T3lt6rqq0XnFSklHYBRMRHgd+nmIA5c5ecIaBZPp7g8ARhCNhTxpgl3qQLIyNrujld\nFbDNe6+KNs/zCeprVlOvd87b62tWM1Bb1TE+V2ylxoeHV/V1/dq99uoBdjy+l+OOm+oY379/Hz/+\nIz/I2NixHeP9wveW5aerpCIivgs4N6X0lbbDf0sx9+F54NQZRTaUxwF2l89nxh+hGOaYKJ8/UX6v\nGnBsW/l52bt3nKmpvJsiWqBaLWNkZI1t3kNVtnmj0aQ5fpDaqpkdhIXm+EGyGjSbh8fniq20eJZl\nDA+vYmLidfI877v6zR4foraq3jE+kB2k0WiSZf05D973lt5rtfliddtTcSLwRxGxMaXU+mX/LuBF\nikmZ/yYihlJKrTv9fA5NvtxePgcgIurA6cANKaXpiNhZxluTOc8DDgKPdVPBqamcyUlvwl6yzXuv\nijafnMzJ82mmZvl8j+l8mnwg7xifK7by4kU75/mh8/urft3H83yaycn+/3/re8vy021SsRN4CLiz\nHPY4kWIvif+LIhl4BrgrIm4CLqZY0XFZWfZO4NqIuA64G7gReDKl1EoibgVui4ivU0zYvBW43c2v\nJElaHrpdUpoD7wf2A38J3A7cklL67TJ2McUQxkPApcCmlNKzZdlvUkzuvALYQbHiY1Pbtb8IfBL4\nDHAPxaqS6xfz4iRJUu90PVEzpfQt4IOzxJ4ELpij7D3AKXPEt+AumpIkLUuu15EkSZUwqZAkSZUw\nqZAkSZUwqZAkSZVYzDbdkvpYnuc0Go1Z443GHqZn2cdAkhbCpEJaoRqNBvfteIJ6fV3H+Esv7uaY\ntWOsO2x3fUlaGJMKaQWr19exdmS0Y2z/vtd6XBtJK51zKiRJUiVMKiRJUiVMKiRJUiVMKiRJUiVM\nKiRJUiVc/SEtU+5DIanfmFRIy5T7UEjqNyYV0jLmPhSS+olzKiRJUiVMKiRJUiVMKiRJUiVMKiRJ\nUiVMKiRJUiVMKiRJUiVMKiRJUiW62qciIr4H+DRwAdAE/iPwsZTSwYg4AbgDOBd4GrgmpXRvW9mL\ngJuBk4AHgc0ppafa4h8BrgXWAV8Crk4pTSz4lUmSpJ7qtqfiD4Fh4N3ATwM/DtxUxr4CPAecAXwB\n2BoRGwFvJbyRAAAQqElEQVQi4m3AVuBzwLuAl4Evty4aER8AbgA2AxcC5wBbFvSKJEnSkph3UhER\nAZwFXJZS+ruU0l9QJAKXRsQFwInAVanwKYreiCvK4puBnSmlW1JKjwOXAydExHvK+IeBm1NKX00p\nPQxcBVwZEcNVvEhJknTkddNT8S3gx1JKL884/h0UPQu7ZgxXbKMYCgE4G7i/FUgpjQO7gHMjIgPO\nBB5oK7sdWA2c1kX9JEnSEpp3UpFSem3GHIkB4Grgz4DjKYY+2r0AbCwfzxUfpRhSeSOeUpoCXmkr\nL0mS+txiPlDs/wZOp+hl+ChwYEb8ADBUPq7PEa+3PZ+t/LzVai5o6ZVWW9vmvdPe5oODGVk2QC0b\n6HjuQDZAlmVHJH4kr91v8Swr2rz4mvdd/RYSz7IBBgeLe6gf+d7Se1W19YKSioj4VYp5ED+ZUvrb\niJgA1s84bYhihQjABIcnCEPAnjLGLPEmXRoZWdNtES2Sbd57IyNreP31OvU1q6nXO+fe9TWrGait\nOiLxI3ntfo0PD6/q6/p1E596fTWjo3XGxo7pGO8XvrcsP10nFRHxWxQTKX82pdRawbEbeMeMUzcA\nz7fFN3SIP0IxzDFRPn+i/B414Ni28vO2d+84U1N5t8W0ALVaxsjIGtu8h9rbvNFo0hw/SG3VzE6+\nQnP8IFkNms3q40fy2v0Wz7KM4eFVTEy8Tp7nfVe/hcYbjSZZ1p9z4X1v6b1Wmy9Wt/tU3Aj8C+Cn\nUkpb20LbgesjYiil1LqLz+fQ5Mvt5fPWdeoUQyc3pJSmI2JnGW9N5jwPOAg81uXrYWoqZ3LSm7CX\nbPPea7V5nk8zlU93PGc6nyYfyI9I/Eheu//ixb2d54fO76/6dR/P82kmJ/v//63vLcvPvJOKiDgV\n+DjwK8BfRsR3t4W/BjwD3BURNwEXU8y1uKyM3wlcGxHXAXcDNwJPppRaScStwG0R8XWKCZu3Are7\n+ZUkSctHNzMzLi7P/zjFL/7nKIYnnksp5cAmiiGMh4BLgU0ppWcBUkrfBC6h2LdiB8WKj02tC6eU\nvgh8EvgMcA/FHhfXL+aFSZKk3pp3T0VK6VeBX50j/g2K7btni98DnDJHfAvuoilJ0rLleh1JklSJ\nxexTIUlahvI8p9HYM2t8dHT0jf05pG6YVEjSUWZ8/z62Pfoy6487fC58s7mPi846mfXrZ249JL01\nkwpJOgoN19eydmR0qauhFcb+LUmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmSVAmTCkmS\nVAn3qZD6VLHrYeNNxwYHM/J8gkajSaOxh+lZPtpakpaCSYXUpxqNBvfteIJ6fd0bx7JsgPqa1TTH\nD/LCt57lmLVjrGNsCWspSYeYVEh9rF5f96ZdD2vZAPX6ELVVB9j3WmOOkpLUe86pkCRJlTCpkCRJ\nlTCpkCRJlXBOhSTpDcWqoz1znjM6OkqW+TepDmdSIUl6w/j+fWx79GXWHzfRMd5s7uOis05m/fr1\nPa6ZlgOTCknSmwzX175p1ZE0X/ZfSZKkSphUSJKkSphUSJKkSix4TkVEDAEPAf8qpXR/eewE4A7g\nXOBp4JqU0r1tZS4CbgZOAh4ENqeUnmqLfwS4FlgHfAm4OqXUebaQJEnqKwvqqSgTiv8AvGNG6MvA\nc8AZwBeArRGxsSzzNmAr8DngXcDL5fmta34AuAHYDFwInANsWUj9JElS73WdVETEqcB24MQZxy+k\n6IG4KhU+RdEbcUV5ymZgZ0rplpTS48DlwAkR8Z4y/mHg5pTSV1NKDwNXAVdGxPBCXpjU7/I859VX\nX531n59CKmm5Wcjwxw8DfwZ8HGi2HT8b2DVjuGIbxVBIK35/K5BSGo+IXcC5EbENOBO4sa3sdmA1\ncBrwVwuop9TXOn0KabuXXtztp5BKWla6TipSSre1HkdEe+h4iqGPdi8AG+cRHwWG2+MppamIeKWM\nm1RoRZr5KaTt9u97rce1kaTFqXLzqzpwYMaxA8DQPOL1tuezlZ+XWs0FLb3SamvbfGEGBzOybIBa\nNtAxPpANkGXZm+KtrZGzLOsYf6vyVcWX8nv3Ot7e5pD3Xf2qjr9VWZjm299+jcHB2f/fL3Ybb99b\neq+qtq4yqZgAZu7bOsShIZIJDk8QhoA9ZYxZ4k26MDKyppvTVQHbfGHyfIL6mtXU653z5vqa1QzU\nVnWMDw+vmjP+VuUXG1/K771U8eHhVX1dv6rib1X2tVcPsOPxvRx33FTH+P79+/jxH/lBxsaO7Rjv\nhu8ty0+VScVuDl8NsgF4vi2+oUP8EeAVisRiA/AEQETUgGPbys/L3r3jTE3lXVVcC1OrZYyMrLHN\nF6jRaNIcP0ht1cwOukJz/CBZDZrNQ/EsyxgeXsXExOsd429Vvqr4Un7vXsfb2zzP876rX9Xx+ZUd\noraq3jE+kB2k0WiSZQufY+97S++12nyxqkwqtgPXR8RQSql1N54PPNAWP791ckTUgdOBG1JK0xGx\ns4y3JnOeBxwEHuumElNTOZOT3oS9ZJsvzORkTp5PMzXLCo/pfJp8IJ8RL9o5z/NZ4m9Vvpr4Un7v\n3scPtXnr/P6qX7XxxV47z6eZnKzmPcH3luWnyqTia8AzwF0RcRNwMcWKjsvK+J3AtRFxHXA3xUqP\nJ1sbZwG3ArdFxNcpJmzeCtzu5leSJC0Pi52Z8UaqmlLKgfdTDGE8BFwKbEopPVvGvwlcQrFvxQ6K\nFR+b2sp/Efgk8BngHoo9Lq5fZP0kSVKPLKqnIqVUm/H8SeCCOc6/BzhljvgW3EVTkqRlyfU6kiSp\nElXOqZDUJs9zGo3GrHG34Za00phUSEeI23BLOtqYVEhHkNtwSzqamFRIkipTDPvtmfOcxW7jrf5l\nUiFJqsz4/n1se/Rl1h/XeYuhZnMfF511MuvXz/xUB60EJhWSpEoN19fOOuynlc3+J0mSVAmTCkmS\nVAmHP6QFch8KSXozkwppgdyHQpLezKRCWgT3oZCkQ0wqJEk9M599LI47zuWmy5VJhSSpZ+azj8WP\nnXcKxx7beVhR/c2kQpLUU+5jsXKZVEizcHWHJHXHpEKahas7JKk7JhXSHFzdIfVWayLnK6/UaTSa\nTE7mh53jB5L1L5MKHbUc3pD6z/j+fXxt16s8+dIUzfGD5DP+D/qBZP3NpEJHLYc3pP60pr6Wdd8x\nRm3VAaZM7JcVkwod1RzekKTqmFRIkpaN+Wye5ZyLpdNXSUVEDAG3ApcATeDXU0q/sbS10nLlnAlp\n5ZnP5lnOuVg6fZVUAL8GvBP4EeAE4PMR8XRK6Y+WslJanpwzIa1Mc22eZU/G0uqbpCIi6sCVwPtS\nSo8Bj0XEFuBqwKRCh5lPT8Sa4dnffJwzIa089mQsrb5JKoDTKOrzYNuxbcAvLE11tNTmkzTs/LuX\nOGatPRGSDrEnY+n0U1JxPPBySmmy7dgLwHBEHJtSemWJ6qUjpKqkwZ4ISfP1Vj0Z+7/9Gmee8t2M\njs7+x4hJx+z6KamoAwdmHGs9H5rvRWo1f9Azvfrqq0fkurXaAHk+wb5940xNdT/hsdHYw9cefpKh\n4TUd43teeZH62jHWZSOdv//AAAcmmjS/3Tl5ODCxn4Ha6hUVz7KMqdcHmTgwuaT168e2OVLx9jbP\n87zv6ld1vB/qVpt6nX2v7XlTm1f9/bNsoHN8fD/3bf87Rsc6JxUHJsb54TNOmjPp6Hedhn6q+t3Z\nT0nFBIcnD63nzXleY2BkpPMvqKPZ2NgxR/T6xx670JJv44wz/ocqq3KUOXOpKyBJb9JPf9bvBo6L\niPY6bQDGU0qz95FLkqS+0E9JxaPA68A5bcd+CNi5NNWRJEndGJie7p/NfyLid4B3A1cAG4G7gJ9L\nKX1lKeslSZLeWj/NqQD4KMWOmv8VeA34JRMKSZKWh77qqZAkSctXP82pkCRJy5hJhSRJqoRJhSRJ\nqoRJhSRJqoRJhSRJqkS/LSldlIi4B/j9lNLn246tB+4A/gnwEnBDSun3l6iKK1JE/GNgFzANtDbU\nfyildNbS1WpliYghiuXWl1BsW//rKaXfWNparWwRsQn4Iw7d19PAH6aUfnJJK7ZClff4Q8C/Sind\nXx47geL9+1zgaeCalNK9S1XHlWaWNv9N4F/z5vv+X6eUbp3PNVdET0VEDETEbwEXdQj/HrAOOBv4\nZeCzEfGuXtbvKPAO4BGKbdVb/963pDVaeX4NeCfwI8CHgBsj4pIlrdHK9w7gjzl0Tx8P/PyS1miF\nKn+5/QeKNm/3ZeA54AzgC8DWiNjY4+qtSHO0+anA9RT3e+u+v3O+1132PRUR8T0UN9uJQGNG7CTg\nfwTenlJ6Bng8Is6leFO+otd1XcFOBR5PKb201BVZiSKiDlwJvC+l9BjwWERsAa6m+EtaR8apwN94\nXx9ZEXEq8Acdjl8InASck1KaAD4VEe+leO/+RG9rubLM1ualU4EtKaUXF3LtldBT8U7gHygy2b0z\nYmcD/1AmFC3bKLrSVJ13AE8sdSVWsNMo/gB4sO3YNor7W0eO93Vv/DDwZxTvy+2fR342sKtMKFp8\n/65GxzaPiHXA97KI+37Z91SklO4G7gaIiJnh4ym6ztq9QPG5IqrOqUAWEX8NfAfwVeDfpJT2LW21\nVozjgZdTSpNtx14AhiPi2JTSK0tUr5UugB+LiF8EasCXKOZkvb601VpZUkq3tR7PeA/3/fsImaPN\nT6WYQ/HxiPinwCvAb7TPU3wrfZ9URMQwRebUyfMppeYcxevAgRnHDgBDVdTtaPEWP4OXgO8HvgFc\nBowBtwCfB36iF/U7Csx2H4P38hEREd8HrAHGgX9GMbz6W8AwcM0SVu1o4vt3750C5MDfAp+mmMN1\ne0S8Nt/P4er7pIKiC+y/UWRPM/0ExUSq2Uxw+A04RDF7XvP3Vj+DY4HxlNIUQET8HPBQRGxIKX2r\nd9VcsWa7j8F7+YhIKf1D2QvUmqf11xFRA/59RHw0peSHJh15E8D6Gcd8/z6CUkqfj4g/brvv/yYi\nTgb+V2BlJBUppa+x8Lkfuylmr7bbADy/qEodZRbwM3i8/Pq9gEnF4u0GjouILKWUl8c2UCRyjTnK\naRE6tO3jFD0V6ym6hXVk7ebwlQm+fx9hs9z3F8y3/EqYqDmX7cDbyxUiLeeXx1WBiDg1IvZGxNvb\nDp8OvA78/RJVa6V5lKI9z2k79kPAzqWpzsoXET8aES+XQ38tpwOvOIelZ7YD7yyXPrb4/n0ERcT/\nGREz9wE5Hfi7+V6j73sqFiOl9FS5IdYXIuJ/A84CfgZ4z9LWbEX5O+C/A3dExDUUcypuA25PKb22\npDVbIVJK4xHxeeC2iLiCYqLa/w783NLWbEX7S4pu9s9GxCco5g1tAX51SWt1dPka8AxwV0TcBFwM\nnEkxd0tHxn8G/m1EfJRij5D3Af8zxdyKeVlpPRWdxjn/F4qlptuBjwGXp5Qe7mmtVrBybPliija+\nH9gK3At8dCnrtQJ9FHgY+K8UEwZ/ab4Tp9S9lNK3Kd5Qv5OiR+gO4LaU0q8vacVWvjfew8uhvvdT\nDHk8BFwKbEopPbtEdVup2tv8IeCDFL83/z+KvXB+JqW0Y74XG5iedr6RJElavJXWUyFJkpaISYUk\nSaqESYUkSaqESYUkSaqESYUkSaqESYUkSaqESYUkSaqESYUkSaqESYUkSaqESYUkSaqESYUkSarE\n/w9G8Qnjr1AzJwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11012dc18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean_difference = bootstrap_experiment(100000)\n",
    "sns.distplot(mean_difference, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_difference = np.sort(mean_difference, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -6.66666667],\n",
       "       [ -6.41666667],\n",
       "       [ -5.91666667],\n",
       "       ..., \n",
       "       [ 13.91666667],\n",
       "       [ 14.        ],\n",
       "       [ 14.66666667]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_difference #Sorted difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.16666667,  8.08333333])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(mean_difference, [5,95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reiterating what this means: 90% of the times, the mean difference is between the limits as shown above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise: Find the 95% percentile for confidence intevals**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where do we go from here? \n",
    "\n",
    "First of all there are two points to be made.\n",
    "\n",
    "1. Whey do we need signficance testing if confidence intervals can provide us more information?\n",
    "2. How does it relate to the traditional statistical procedure of finding confidence intervals\n",
    "\n",
    "For the first one:\n",
    "\n",
    "What if sales in the first month after price changes was 80 and the month before price changes was 40. The difference is 40. And confidence interval,as explained above, using replacements, would always generate 40. But if we do the significance testing, as detailed above - where the labels are shuffled, the prices are equally likely to occur in both the groups. And so, significance testing would answer that there was no difference. But don't we all know that the data is **too small** to make meaningful inferences?\n",
    "\n",
    "For the second one:\n",
    "\n",
    "Traditional statistics derivation assumes normal distribution. But what if the underlying distribution isn't normal? Also, people relate to resampling much better :-) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}