Module_2g_HackerStats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A shoe company launches a premium shoe model SC-X01. The sales for the first month (in USD) are given below:\n",
    "\n",
    "[54, 69, 50, 57, 70, 69, 52, 46, 60, 42, 57, 64, 58, 60, 40, 63, 60, 70, 57, 71, 66, 67, 68, 42, 46, 50, 48, 65, 45, 45]\n",
    "\n",
    "Ignoring all other effects (eg: holidays, seasonality, new-ness factor, etc), can you provide an estimate of the average sales the company should expect? \n",
    "\n",
    "What is the variability around that average estimate that you are providing?\n",
    "\n",
    "The marketing manager has heard about the term `confidence interval`. Can you provide that for the estimate and explain what it means to the marketing manager?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "sns.set(color_codes=True)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the data into a numpy array\n",
    "sales_1 = np.array([54, 69, 50, 57, 70, 69, 52, 46, 60, 42, 57, 64, 58, \n",
    "                    60, 40, 63, 60, 70, 57, 71, 66, 67, 68, 42, 46, 50, \n",
    "                    48, 65, 45, 45])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57.033333333333331"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find mean of sales_1\n",
    "\n",
    "np.mean(sales_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the above is based on one sample. How do I get an estimate for the population?\n",
    "\n",
    "**Resampling/Bootstrapping**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's create a bootstrap sample - we will use np.random.choice\n",
    "\n",
    "?np.random.choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#create a bootstrap sample from sales_1\n",
    "sample_1 = np.random.choice(sales_1, 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([64, 40, 71, 46, 68, 69, 50, 42, 52, 58, 42, 68, 50, 42, 42, 67, 71,\n",
       "       45, 60, 45, 46, 70, 70, 46, 66, 50, 54, 50, 71, 67])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56.06666666666667"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#mean of sample_1\n",
    "np.mean(sample_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now, let's write a function that returns mean of n bootstrap samples\n",
    "\n",
    "def bootstrap_sample_mean(input, num_samples):\n",
    "    \"\"\"\n",
    "      Function that generates bootstrap sample, computes its mean and return the array\n",
    "      \n",
    "      Arguments:\n",
    "         input : the input array for which bootstrap sample has to be generated\n",
    "         num_samples: num of times the sample has to be generated\n",
    "         \n",
    "      output:\n",
    "         bootstrap_mean : This returns the mean of each of the bootstrap sample as a numpy array\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    bootstrap_mean = np.zeros([num_samples, 1])\n",
    "    \n",
    "    for i in np.arange(num_samples):\n",
    "        sample = np.random.choice(input, input.shape[0])\n",
    "        bootstrap_mean[i] = np.mean(sample)\n",
    "        \n",
    "    return bootstrap_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bootstrap_mean = bootstrap_sample_mean(sales_1, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f947fe46be0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD9hJREFUeJzt3X+MZXV5x/H3sLMw3c4IUxiQLKtrCjwNIQGkXSBWC1jM\nQgibRijQxoCaYhBsoVDTkiotf4G/yCYQiAEqay1IEHFrqdZEK5ooRQgoiE+7VdLdhZZxGWEm60KW\nTP+4Z3EY5/7aOXfu3O+8X8kk58f33vM8zN3PPfO951yGZmdnkSQNvgP6XYAkqR4GuiQVwkCXpEIY\n6JJUCANdkgox3K8DT05OzwKMj69hamp3v8pYMvZZjpXQI9jncjUxMTbUbF/fz9CHh1f1u4QlYZ/l\nWAk9gn0Oor4HuiSpHga6JBXCQJekQhjoklQIA12SCmGgS1Ih2l6HHhEjwMPAQdX4+zPz+nljDgK2\nACcDu4ALM/PZ2quVJDXVyRn6K8CZmXkCcCKwMSJOnTfmg8BUZh4N3AzcVG+ZkqR22gZ6Zs5m5ky1\nurr6mf8l6puAu6vl+4F3R0TTu5kkSfXr6Nb/iFgFPAYcDdyamY/MG7IW2A6QmXsj4iXgUODnzZ5z\nfHzN63doTUyMdVX01773bFfjl4Vtu/pdwdIopM+Np61vub/b1+ygss/B0lGgZ+ZrwIkRcQjw5Yg4\nPjOfWsyB9313wsTEGJOT0109dnpmz2IO3RdjoyMDWXe3Sumz1Wtyf16zg8g+l6dWbz5dXeWSmb8A\nvgVsnLdrJ7AOICKGgYNpfDgqSVoibQM9IiaqM3Mi4jeAs4CfzBu2FbikWj4f+GZm+j8rlaQl1MmU\ny5HA3dU8+gHAfZn51Yi4AfhBZm4F7gQ+HxHbgBeBi3pWsSRpQW0DPTN/CJy0wPaPz1neA1xQb2mS\npG54p6gkFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12S\nCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQ\nBrokFWK43YCIWAdsAY4AZoHPZubmeWNOB74C/Kza9EBm3lBvqZKkVtoGOrAXuCYzH4+IMeCxiPhG\nZv543rjvZOa59ZcoSepE2ymXzHw+Mx+vlqeBZ4C1vS5MktSdTs7QXxcR64GTgEcW2H1aRDwJPAdc\nm5lPt3qu8fE1DA+vAmBiYqybMhgbHelq/HIxqHV3q4Q+270mu33NDir7HCwdB3pEjAJfAq7KzJfn\n7X4ceGtmzkTEOcCDwDGtnm9qajfQ+A85OTndVdHTM3u6Gr8cjI2ODGTd3Sqlz1avyf15zQ4i+1ye\nWr35dHSVS0SsphHmX8jMB+bvz8yXM3OmWn4IWB0Rh+1fuZKk/dE20CNiCLgTeCYzP9NkzJurcUTE\nhup5d9VZqCSptU6mXN4BvA/4UUQ8UW27DngLQGbeDpwPXB4Re4FfAhdl5mwP6pUkNdE20DPzu8BQ\nmzG3ALfUVZQkqXveKSpJhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWp\nEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgph\noEtSIQx0SSqEgS5JhTDQJakQw+0GRMQ6YAtwBDALfDYzN88bMwRsBs4BdgOXZubj9ZcrSWqmkzP0\nvcA1mXkccCpwRUQcN2/M2cAx1c9lwG21VilJaqttoGfm8/vOtjNzGngGWDtv2CZgS2bOZub3gUMi\n4sjaq5UkNdV2ymWuiFgPnAQ8Mm/XWmD7nPUd1bbnmz3X+PgahodXATAxMdZNGYyNjnQ1frkY1Lq7\nVUKfj23b1Xxnq30lsc+e2Xja+p48b8eBHhGjwJeAqzLz5cUeeGpqN9AI88nJ6a4eOz2zZ7GHX3Jj\noyMDWXe3VkKfK6FHsM9e6jbz5mp1AtzRVS4RsZpGmH8hMx9YYMhOYN2c9aOqbZKkJdLJVS5DwJ3A\nM5n5mSbDtgJXRsS9wCnAS5nZdLpFklS/TqZc3gG8D/hRRDxRbbsOeAtAZt4OPETjksVtNC5bfH/9\npUqSWmkb6Jn5XWCozZhZ4Iq6ipIkdc87RSWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSB\nLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiS\nVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEMPtBkTEXcC5wAuZefwC+08HvgL8rNr0QGbeUGeRkqT2\n2gY68DngFmBLizHfycxza6lIkrRf2k65ZObDwItLUIskaRE6OUPvxGkR8STwHHBtZj7d7gHj42sY\nHl4FwMTEWFcHGxsd2Z8a+25Q6+7WSuhzJfQI9tkr3WZep+oI9MeBt2bmTEScAzwIHNPuQVNTu4FG\nY5OT010dcHpmz36U2V9joyMDWXe3VkKfK6FHsM9e6jbz5mr1ZrDoq1wy8+XMnKmWHwJWR8Rhi31e\nSVJ3Fh3oEfHmiBiqljdUz7lrsc8rSepOJ5ct3gOcDhwWETuA64HVAJl5O3A+cHlE7AV+CVyUmbM9\nq1iStKC2gZ6ZF7fZfwuNyxolSX3knaKSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJek\nQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqE\ngS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKMdxuQETcBZwLvJCZxy+wfwjYDJwD7AYuzczH6y5U\nktRaJ2fonwM2tth/NnBM9XMZcNviy5IkdattoGfmw8CLLYZsArZk5mxmfh84JCKOrKtASVJn2k65\ndGAtsH3O+o5q2/OtHjQ+vobh4VUATEyMdXXAsdGR7ipcJga17m6thD5XQo9gn73SbeZ1qo5A3y9T\nU7uBRmOTk9NdPXZ6Zk8vSuqpsdGRgay7Wyuhz5XQI9hnL3WbeXO1ejOo4yqXncC6OetHVdskSUuo\njjP0rcCVEXEvcArwUma2nG6RJNWvk8sW7wFOBw6LiB3A9cBqgMy8HXiIxiWL22hctvj+XhUrSWqu\nbaBn5sVt9s8CV9RWkSRpv3inqCQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrok\nFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1Ih\nDHRJKoSBLkmFMNAlqRAGuiQVYriTQRGxEdgMrALuyMwb5+2/FPgksLPadEtm3lFjnZKkNtoGekSs\nAm4FzgJ2AI9GxNbM/PG8oV/MzCt7UKMkqQOdTLlsALZl5k8z81XgXmBTb8uSJHWrkymXtcD2Oes7\ngFMWGPfeiHgX8J/A1Zm5fYExrxsfX8Pw8CoAJibGOqu2MjY60tX45WJQ6+7WSuhzJfQI9tkr3WZe\npzqaQ+/APwP3ZOYrEfEh4G7gzFYPmJraDTQam5yc7upg0zN79rPM/hkbHRnIuru1EvpcCT2CffZS\nt5k3V6s3g04CfSewbs76Ufzqw08AMnPXnNU7gE90UZ8kqQadzKE/ChwTEW+LiAOBi4CtcwdExJFz\nVs8DnqmvRElSJ9qeoWfm3oi4Evg6jcsW78rMpyPiBuAHmbkV+POIOA/YC7wIXNrDmiVJCxianZ3t\ny4EnJ6dnYf/m0P/9iZ3tBy0zzkeWYyX0CPbZS6efuHa/HzsxMTbUbJ93ikpSIQx0SSqEgS5JhTDQ\nJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12S\nCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSrEcCeDImIjsBlY\nBdyRmTfO238QsAU4GdgFXJiZz9ZbqiSplbZn6BGxCrgVOBs4Drg4Io6bN+yDwFRmHg3cDNxUd6GS\npNY6mXLZAGzLzJ9m5qvAvcCmeWM2AXdXy/cD746IofrKlCS108mUy1pg+5z1HcApzcZk5t6IeAk4\nFPh5syedmBgbmrPcab0AXHDW73Q1XpJWAj8UlaRCdBLoO4F1c9aPqrYtOCYihoGDaXw4KklaIp1M\nuTwKHBMRb6MR3BcBfzJvzFbgEuB7wPnANzNzts5CJUmttT1Dz8y9wJXA14FngPsy8+mIuCEizquG\n3QkcGhHbgL8E/rpXBUuSFjY0O+uJtCSVwA9FJakQBrokFaKjW//rEhHPAtPAa8DezPzdOfuuAT4F\nTGRm0+vXB8FCfUbE3wF/BkxWw67LzIf6UmBNmv0+I+IjwBXV9n/JzI/2q8Y6NPl9fhGIasghwC8y\n88T+VFiPJn2eCNwOjAB7gQ9n5n/0rchFatLjCTR6HAWeBf40M1/uV42LsaSBXjljfmBHxDrgPcD/\n9KGeXvm1PoGbM/NTfammd97QZ0ScQePO4RMy85WIOLx/pdXqDX1m5oX7liPi08BLfamqfvNft58A\n/j4z/zUizqnWT+9LZfWZ3+MdwLWZ+e2I+ADwV8DH+lPa4iyXKZebgY8CfkI7+C4HbszMVwAy84U+\n19NT1Vdc/DFwT79r6ZFZ4E3V8sHAc32spVeOBR6ulr8BvLePtSzKUgf6LPBvEfFYRFwGEBGbgJ2Z\n+eQS19JLv9Zn5cqI+GFE3BUR4/0qrkYL9Xks8M6IeCQivh0Rv9fH+urS7PcJ8E7g/zLzv/pQV90W\n6vMq4JMRsZ3GlOjf9K26eizU49P86vupLuCNN1IOlKUO9N/PzLfT+ObGKyLiXcB1wMeXuI5eW6jP\n24DfBk4Engc+3cf66rJQn8PAbwGn0vjT9b4CvqhtoT73uZhyzs4X6vNy4OrMXAdcTeOek0G2UI8f\nAD4cEY8BY8Cr/SxwMfp2HXr1IeFrwEeA3dXmo2j8SbchM/+3L4XVrOpzZu7ceUSsB76amcf3q666\n7esT+EPgpsz8VrX9v4FTM3OyxcMHxtzfZ/U1FzuBkzNzR38rq9ec3+fHgEMyc7Z6Y34pM9/U8sED\nosm/zWOBf8zMDX0rbBGW7Aw9In4zIsb2LdP4EPTRzDw8M9dn5noa3+T49kEO8yZ9PhURR84Z9kfA\nU/2ory7N+gQeBM6oth8LHEiLb91c7lr0CY03r5+UEOYt+nwO+INq2JnAwE4ttfi3eXi17QDgb2lc\n8TKQlvIqlyOAL0fEvuP+U2Z+bQmPv1QW7DMiPl9dAjZL49KoD/WvxFo06/NA4K6IeIrGn66XDPj3\n+rR63V5EOdMtzX6fM8Dm6q+RPcD8zxAGSbMe/yIirqjGPAD8Q78KXCxv/ZekQiyXyxYlSYtkoEtS\nIQx0SSqEgS5JhTDQJakQBrokFcJAl6RC/D8x6YnlKv29fAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f947fddc748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(bootstrap_mean, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#What is the variability around this estimate?\n",
    "# this is referred to as standard error of the estimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.9794050733602864"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.std(bootstrap_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Of course, this is small sample. So, generate it for a number of samples (say: 100k). \n",
    "# Report the estimate and standard error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Mean\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Standard Error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.7885223679161515"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bonus - this is how the standard error is obtained using scipy\n",
    "\n",
    "from scipy import stats\n",
    "stats.sem(sales_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### A/B Testing\n",
    "\n",
    "The shoe company, after a month of sales, got endorsed by the current MVP of NBA. \n",
    "\n",
    "The sales for the second month after lunch, and the first month after endorsement is as follows:\n",
    "\n",
    "[64, 60, 52, 51, 49, 71, 71, 68, 56, 43, 71, 61, 43, 47, 40, 42, 42, 42, 54, 45, 59, 44, 64, 63, 59, 40, 55, 61, 57, 63]\n",
    "\n",
    "Did the endorsement impact the sales positively? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load 2nd month sales data\n",
    "sales_2 = np.array([64, 60, 52, 51, 69, 71, 71, 68, 56, 63, 71, 61, 63, 67, 60, 62, 72,\n",
    "       42, 54, 45, 59, 44, 64, 63, 59, 40, 55, 61, 57, 63])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "59.56666666666667"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find mean of sales_2\n",
    "np.mean(sales_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.5333333333333385"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The difference between the two means:\n",
    "shoe_sales_diff = np.mean(sales_2) - np.mean(sales_1)\n",
    "shoe_sales_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# So, sales improved, on average, by 2.53 shoes. But is it statistically significant?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the approach we are going to take:\n",
    "\n",
    "- We will assign label to sales before endorsement as 1 and sales after endorsement as 2\n",
    "- The observed difference is given in the above cell\n",
    "- We will then randomly shuffle the labels between the two groups \n",
    "- Compute the mean between the two groups. \n",
    "- Find the % of times when the difference between the means computed is greater than what we observed above\n",
    "- If the % of times is less than x%, we would make the call that the improvements are real  (generally x is taken as 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_before = np.array([sales_1, np.repeat(1, sales_1.shape[0])]).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_after = np.array([sales_2, np.repeat(2, sales_2.shape[0])]).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine them into one ndarray\n",
    "\n",
    "shoe_sales = np.concatenate((sales_before, sales_after), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "shoe_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "#randomly shuffle labels. We will do this by introducing a new variable experiment_label\n",
    "experiment_label = np.random.randint(1, 3, shoe_sales.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create the experiment data combining actual sales with the shuffled label\n",
    "\n",
    "experiment_data = np.transpose(np.array([shoe_sales[:, 0], experiment_label]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Compute mean between the two groups now.\n",
    "# Again - take a minute to think what these two groups mean?\n",
    "\n",
    "experiment_diff_mean =  experiment_data[experiment_data[:,1]==2].mean() \\\n",
    "                        - experiment_data[experiment_data[:,1]==1].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.38392857142857295"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_diff_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's repeat this process many times. Let's write a function to do this\n",
    "\n",
    "def mean_diff_shuffle_labels(input, num_times):\n",
    "    \"\"\"\n",
    "      Function that shuffles the labels, computes the difference between the groups and return the mean difference array\n",
    "      \n",
    "      arguments:\n",
    "         input: input array that has two columns: the data in the first column and the label in the second column\n",
    "         num_times: number of times the label \n",
    "    \n",
    "      output:\n",
    "        mean_diff_shuffle_mean : This returns the mean differene of each of the shuffled sample as a numpy array\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    mean_diff_shuffle_mean = np.zeros([num_times, 1])\n",
    "    \n",
    "    for i in np.arange(num_times):\n",
    "    \n",
    "        experiment_label = np.random.randint(1, 3, input.shape[0])\n",
    "        experiment_data = np.transpose(np.array([input[:, 0], experiment_label]))\n",
    "        experiment_diff_mean =  experiment_data[experiment_data[:,1]==2].mean() \\\n",
    "                        - experiment_data[experiment_data[:,1]==1].mean()\n",
    "        \n",
    "        mean_diff_shuffle_mean[i] = experiment_diff_mean\n",
    "        \n",
    "    return mean_diff_shuffle_mean\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "shoe_mean_diff_shuffle = mean_diff_shuffle_labels(shoe_sales, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's plot this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f947a161d30>"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFZtJREFUeJzt3X2QXXd93/H3dtegEdoJC7oYVxK2mWq+xXETe+KRoTBF\nToIQHiPlqY2cB2wC48SDmyn0KSlM3RF06sYzTTyxwWSExnYnyFCCQZkxttUYxmmDW9kurrHNl1mE\ng7V10UZaYFX5YeRu/7hH9fWyu/fo7n04+vn9mvHsnt/5nXs/urv+3LvnnnvO2MLCApKkcv2tUQeQ\nJA2WRS9JhbPoJalwFr0kFc6il6TCTYw6wFJmZ+d7PhRoamotc3Mn+hmnr5qeD5qfsen5oPkZm54P\nmp+xaflarcmx5dYV94p+YmJ81BFW1PR80PyMTc8Hzc/Y9HzQ/IxNz9epuKKXJL2cRS9JhbPoJalw\nFr0kFc6il6TCWfSSVLiux9FHxCbgDuBsYAH4k8y8adGcMeAm4HLgBHB1Zj5SrbsK+Fg19ROZeXv/\n4kuSuqnziv4k8E8z8wLgrcCHIuKCRXPeA2yu/rsG+BRARLwOuB64FNgCXB8RU33KLkmqoWvRZ+Yz\np16dZ+Y88CSwYdG0ncAdmbmQmQ8Cr42Ic4B3Awcy81hmzgEHgO19/RdIklZ0WqdAiIjzgIuB/7Zo\n1Qbg6Y7lw9XYcuMrmppau6pPnbVakz1vOwxNzwfNzziqfPd8/al6E6ePDjLG6g0p3/a3nbeq7f09\n7I/aRR8R64A/A/5JZv5ocJFY1fkjWq1JZmfn+5imv5qeD5qfcZT55o8/V2ve5Lo1teeOwrDyrebn\n5O/h6VnpSafWUTcRcRbtkv/TzPziElNmgE0dyxurseXGJUlD0rXoqyNqPgM8mZn/YZlp+4H3RcRY\nRLwV+GFmPgPcC2yLiKnqTdht1ZgkaUjq7Lp5O/CbwGMR8Y1q7F8BbwLIzFuBu2kfWjlN+/DK91fr\njkXEx4GD1Xa7M/NY/+JLkrrpWvSZ+V+AZc9zXM1ZAD60zLq9wN6e0kmSVs1PxkpS4Sx6SSqcRS9J\nhbPoJalwFr0kFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqnEUvSYWz6CWpcBa9JBXOopekwln0klQ4\ni16SCtf1wiMRsRe4AjiSmRcusf6fA7/ecXtvAVrV1aWeAuaBF4GTmXlJn3JLkmqqcynB24CbgTuW\nWpmZNwI3AkTEe4EPL7pc4GWZ+TerzClJ6lHXXTeZ+QBQ9zqvVwL7VpVIktRXfdtHHxFrge3An3UM\nLwD3RcTDEXFNv+5LklRfnV03db0X+K+Ldtu8IzNnIuINwIGI+Fb1F8KKpqbWMjEx3nOQVmuy522H\noen5oPkZR5Vvct2agcwdhWHkW+3Pyd/D/uhn0e9i0W6bzJypvh6JiLuALUDXop+bO9FziFZrktnZ\n+Z63H7Sm54PmZxxlvvnjz9WaN7luTe25ozCsfKv5Ofl7eHpWetLpy66biPgJ4J3AlzvGXhMRk6e+\nB7YB3+zH/UmS6qtzeOU+YCuwPiIOA9cDZwFk5q3VtF8E7svM/9Ox6dnAXRFx6n4+m5n39C+6JKmO\nrkWfmVfWmHMb7cMwO8cOAT/dazBJUn/4yVhJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn\n0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqXJ1LCe4FrgCOZOaF\nS6zfSvtasd+thr6YmburdduBm4BxYE9m3tCn3JKkmroWPe1LBN4M3LHCnL/MzCs6ByJiHLgFeBdw\nGDgYEfsz84kes0qSetB1101mPgAc6+G2twDTmXkoM18A7gR29nA7kqRVqPOKvo63RcSjwP8C/llm\nPg5sAJ7umHMYuLTOjU1NrWViYrznMK3WZM/bDkPT80HzM44q3+S6NQOZOwrDyLfan5O/h/3Rj6J/\nBDg3M49HxOXAl4DNq7nBubkTPW/bak0yOzu/mrsfqKbng+ZnHGW++ePP1Zo3uW5N7bmjMKx8q/k5\n+Xt4elZ60ln1UTeZ+aPMPF59fzdwVkSsB2aATR1TN1ZjkqQhWvUr+oh4I/D9zFyIiC20nzyOAj8A\nNkfE+bQLfhfwa6u9P0nS6alzeOU+YCuwPiIOA9cDZwFk5q3ArwDXRsRJ4FlgV2YuACcj4jrgXtqH\nV+6t9t1Lkoaoa9Fn5pVd1t9M+/DLpdbdDdzdWzRJUj/4yVhJKpxFL0mFs+glqXAWvSQVzqKXpMJZ\n9JJUOItekgpn0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqXJ1L\nCe4FrgCOZOaFS6z/deBfAmPAPHBtZj5arXuqGnsROJmZl/QtuSSpljqv6G8Dtq+w/rvAOzPz7wEf\nB/5k0frLMvMiS16SRqPONWMfiIjzVlj/Vx2LDwIb+5BLktQnXYv+NH0A+ErH8gJwX0QsAJ/OzMWv\n9pc0NbWWiYnxnkO0WpM9bzsMTc8Hzc84qnyT69YMZO4oDCPfan9O/h72R9+KPiIuo1307+gYfkdm\nzkTEG4ADEfGtzHyg223NzZ3oOUerNcns7HzP2w9a0/NB8zOOMt/88edqzZtct6b23FEYVr7V/Jz8\nPTw9Kz3p9OWom4j4KWAPsDMzj54az8yZ6usR4C5gSz/uT5JU36qLPiLeBHwR+M3M/HbH+GsiYvLU\n98A24JurvT9J0umpc3jlPmArsD4iDgPXA2cBZOatwL8GXg98MiLgpcMozwbuqsYmgM9m5j0D+DdI\nklZQ56ibK7us/yDwwSXGDwE/3Xs0SVI/+MlYSSqcRS9JhbPoJalwFr0kFc6il6TCWfSSVDiLXpIK\nZ9FLUuEsekkqnEUvSYWz6CWpcBa9JBXOopekwln0klQ4i16SCmfRS1LhLHpJKlzXK0wBRMRe4Arg\nSGZeuMT6MeAm4HLgBHB1Zj5SrbsK+Fg19ROZeXs/gkuS6qn7iv42YPsK698DbK7+uwb4FEBEvI72\nNWYvBbYA10fEVK9hJUmnr1bRZ+YDwLEVpuwE7sjMhcx8EHhtRJwDvBs4kJnHMnMOOMDKTxiSpD6r\nteumhg3A0x3Lh6ux5cZXNDW1lomJ8Z6C3PP1p3rabmimj446QXdNzzjCfJPr1gxk7igMI9/Dq/lZ\nvQJ/D7e/7by+3yb0r+j7am7uxKq2nz/+XJ+S9N/kujWNzgfNz9j0fND8jE3PB83POIh8s7PzPW/b\nak0uu65fR93MAJs6ljdWY8uNS5KGpF9Fvx94X0SMRcRbgR9m5jPAvcC2iJiq3oTdVo1Jkoak7uGV\n+4CtwPqIOEz7SJqzADLzVuBu2odWTtM+vPL91bpjEfFx4GB1U7szc6U3dSVJfVar6DPzyi7rF4AP\nLbNuL7D39KNJkvrBT8ZKUuEsekkqnEUvSYWz6CWpcBa9JBXOopekwln0klQ4i16SCmfRS1LhLHpJ\nKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn0UtS4epeSnA7cBMwDuzJzBsWrf9D4LJqcS3w\nhsx8bbXuReCxat33MnNHP4JLkurpWvQRMQ7cArwLOAwcjIj9mfnEqTmZ+eGO+f8YuLjjJp7NzIv6\nF1mSdDrq7LrZAkxn5qHMfAG4E9i5wvwrgX39CCdJWr06u242AE93LB8GLl1qYkScC5wP3N8xvCYi\nHgJOAjdk5pe63eHU1FomJsZrRFvC9FEm163pbdshaXo+aH7GpueD5mdsej5ofsZ+52u1Jvt6e6fU\n2kd/GnYBX8jMFzvGzs3MmYh4M3B/RDyWmd9Z6Ubm5k6sKsT88edWtf0gTa5b0+h80PyMTc8Hzc/Y\n9HzQ/IyDyDc7O9/ztis9SdTZdTMDbOpY3liNLWUXi3bbZOZM9fUQ8DVevv9ekjRgdYr+ILA5Is6P\niFfRLvP9iydFxN8FpoCvd4xNRcSrq+/XA28Hnli8rSRpcLoWfWaeBK4D7gWeBD6fmY9HxO6I6DxU\nchdwZ2YudIy9BXgoIh4Fvkp7H71FL0lDNLawsNB91pDNzs73HOrh6aOvuP16/db0jE3PB83P2PR8\n0PyMg8i39aINPW/bak2OLbfOT8ZKUuEsekkqnEUvSYWz6CWpcBa9JBXOopekwln0klQ4i16SCmfR\nS1LhLHpJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn0UtS4SbqTIqI7cBNwDiwJzNvWLT+\nauBGXrpo+M2ZuadadxXwsWr8E5l5ex9yS5Jq6lr0ETEO3AK8CzgMHIyI/Utc+/VzmXndom1fB1wP\nXAIsAA9X2871Jb0kqas6u262ANOZeSgzXwDuBHbWvP13Awcy81hV7geA7b1FlST1os6umw3A0x3L\nh4FLl5j3yxHxD4BvAx/OzKeX2bbr1W+nptYyMTFeI9oSpo8yuW5Nb9sOSdPzQfMzNj0fND9j0/NB\n8zP2O1+rNdnX2zul1j76Gv4c2JeZz0fEbwO3Az/b643NzZ1YVZhX2pXj+63pGZueD5qfsen5oPkZ\nB5Fvdna+521XepKos+tmBtjUsbyRl950BSAzj2bm89XiHuBn6m4rSRqsOkV/ENgcEedHxKuAXcD+\nzgkRcU7H4g7gyer7e4FtETEVEVPAtmpMkjQkXXfdZObJiLiOdkGPA3sz8/GI2A08lJn7gd+NiB3A\nSeAYcHW17bGI+DjtJwuA3Zl5bAD/DknSMsYWFhZGneHHzM7O9xzq4emjr7j9ev3W9IxNzwfNz9j0\nfND8jIPIt/WirseqLKvVmhxbbp2fjJWkwln0klQ4i16SCmfRS1LhLHpJKpxFL0mFs+glqXAWvSQV\nzqKXpMJZ9JJUOItekgpn0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TCdb2UIEBEbAduon0pwT2Z\necOi9R8BPkj7UoKzwG9l5l9X614EHqumfi8zd/QpuySphq5FHxHjwC3Au4DDwMGI2J+ZT3RM+x/A\nJZl5IiKuBf4A+NVq3bOZeVGfc0uSaqrzin4LMJ2ZhwAi4k5gJ/D/iz4zv9ox/0HgN/oZUpLUuzpF\nvwF4umP5MHDpCvM/AHylY3lNRDxEe7fODZn5pW53ODW1lomJ8RrRljB9lMl1a3rbdkiang+an7Hp\n+aD5GZueD5qfsd/5Wq3Jvt7eKbX20dcVEb8BXAK8s2P43MyciYg3A/dHxGOZ+Z2Vbmdu7sSqcrzS\nrhzfb03P2PR80PyMTc8Hzc84iHyzs/M9b7vSk0Sdo25mgE0dyxursZeJiJ8HPgrsyMznT41n5kz1\n9RDwNeDiOqElSf1R5xX9QWBzRJxPu+B3Ab/WOSEiLgY+DWzPzCMd41PAicx8PiLWA2+n/UatJGlI\nur6iz8yTwHXAvcCTwOcz8/GI2B0Rpw6VvBFYB/yniPhGROyvxt8CPBQRjwJfpb2P/gkkSUMztrCw\nMOoMP2Z2dr7nUA9PH33F7dfrt6ZnbHo+aH7GpueD5mccRL6tF23oedtWa3JsuXV+MlaSCmfRS1Lh\nLHpJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6i\nl6TCWfSSVDiLXpIKV+easUTEduAmYBzYk5k3LFr/auAO4GeAo8CvZuZT1brfBz4AvAj8bmbe27f0\nkqSuur6ij4hx4BbgPcAFwJURccGiaR8A5jLz7wB/CPz7atsLaF9M/CeB7cAnq9uTJA1JnV03W4Dp\nzDyUmS8AdwI7F83ZCdxeff8F4OciYqwavzMzn8/M7wLT1e1Jkoakzq6bDcDTHcuHgUuXm5OZJyPi\nh8Drq/EHF23b9eq3K13ktpvtrcleN5WkIvlmrCQVrk7RzwCbOpY3VmNLzomICeAnaL8pW2dbSdIA\n1Sn6g8DmiDg/Il5F+83V/Yvm7Aeuqr7/FeD+zFyoxndFxKsj4nxgM/Df+xNdklRH16LPzJPAdcC9\nwJPA5zPz8YjYHRE7qmmfAV4fEdPAR4Dfq7Z9HPg88ARwD/ChzHyx//8MSdJyxhYWFkadQZI0QL4Z\nK0mFs+glqXC1ToHQZBFxI/Be4AXgO8D7M/MHS8xb8TQOA8z3D4F/A7wF2JKZDy0z7ylgnvapIk5m\n5iXDyFfdd92Mo3oMXwd8DjgPeAr4R5k5t8S8F4HHqsXvZeaOxXP6nKvnU4MMS42MVwM38tLRcDdn\n5p4h5tsLXAEcycwLl1g/Rjv/5cAJ4OrMfKRB+bYCXwa+Ww19MTN3DytfXSW8oj8AXJiZPwV8G/j9\nxRNqnsZhUL4J/BLwQI25l2XmRcMs+UrXjCN+DH8P+IvM3Az8RbW8lGerx++iIZR8z6cGGZbT+Jl9\nruNxG1rJV26jfXqU5byH9tF6m4FrgE8NIVOn21g5H8Bfdjx+jSt5KKDoM/O+6sggaH8Kd+MS0+qc\nxmFQ+Z7MzBzGffWqZsaRPYa8/BQbtwO/MKT7XclqTg3SpIwjlZkPAMdWmLITuCMzFzLzQeC1EXHO\ncNLVyndGOOOLfpHfAr6yxPhSp3HoeiqGIVsA7ouIhyPimlGHWcIoH8OzM/OZ6vv/DZy9zLw1EfFQ\nRDwYEYN+MqjzeLzs1CDAqVODDEvdn9kvR8T/jIgvRMSmJdaP0pnw/+7bIuLRiPhKRPzkqMMs5YzY\nRx8R/xl44xKrPpqZX67mfBQ4CfzpMLNV9901Xw3vyMyZiHgDcCAivlW9mmhSxoFZKV/nQmYuRMRy\nxwSfWz2Gbwbuj4jHMvM7/c5amD8H9mXm8xHx27T/AvnZEWc6kzxC+/fueERcDnyJ9m6mRjkjij4z\nf36l9dUbSlcAP1d9InexgZ6KoVu+mrcxU309EhF30f6zu29F34eMI3sMI+L7EXFOZj5T/dl+ZJnb\nOPUYHoqIrwEX036DfhBO59QghxedGmRYumbMzM48e4A/GEKu09Ho06hk5o86vr87Ij4ZEesz829G\nmWuxM37XTXVUwb8AdmTmiWWm1TmNw8hExGsiYvLU98A22m+QNskoH8POU2xcRfsoh5eJiKnqKBci\nYj3wdtqfyB6U1ZwaZFi6Zly0v3sH7U+/N8l+4H0RMRYRbwV+2LEbb+Qi4o2n3neJiC20O3WYT+a1\nnPGfjK1Ou/BqXnpwH8zM34mIv037cLLLq3mXA39E+zCzvZn5b4eU7xeBPwZawA+Ab2TmuzvzVbsa\n7qo2mQA+O6x8dTNW80b1GL6e9qk03gT8Ne3DK49FxCXA72TmByPi7wOfBv4v7f/Z/igzPzPgXD/2\neETEbuChzNwfEWuA/0j7L4tjwK7MPDTITD1k/He0C/5klfHazPzWEPPtA7YC64HvA9cDZwFk5q1V\nid5M+8iXE7QPn17y8N8R5bsOuJb24/cs8JHM/Kth5avrjC96SdLKzvhdN5KklVn0klQ4i16SCmfR\nS1LhLHpJKpxFL0mFs+glqXD/DwUMme31yuCjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f947a0ca588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(shoe_mean_diff_shuffle, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# find % of times difference of means is greater than observed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data: Difference in mean greater than observed: []\n"
     ]
    }
   ],
   "source": [
    "#Finding % of times difference of means is greater than observed\n",
    "print(\"Data: Difference in mean greater than observed:\", \\\n",
    "        shoe_mean_diff_shuffle[shoe_mean_diff_shuffle>shoe_sales_diff])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of times diff in mean greater than observed: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of times diff in mean greater than observed:\", \\\n",
    "            shoe_mean_diff_shuffle[shoe_mean_diff_shuffle>shoe_sales_diff].shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of times diff in mean greater than observed: 0.0\n"
     ]
    }
   ],
   "source": [
    "print(\"% of times diff in mean greater than observed:\", \\\n",
    "        shoe_mean_diff_shuffle[shoe_mean_diff_shuffle>shoe_sales_diff].shape[0]/float(shoe_mean_diff_shuffle.shape[0])*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What does the above mean?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise: Repeat the above for 100,000 runs and report the results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "shoe_mean_diff_shuffle = mean_diff_shuffle_labels(shoe_sales, 100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What do you infer?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Effect Size**\n",
    "\n",
    "Find % increase in sales. Does it make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.4418468731735921"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(np.mean(sales_2) - np.mean(sales_1) )/np.mean(sales_1)*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Does 4.4% make sense for the business?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confidence interval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confidence interval is the range of values the measurement metric is going to take.\n",
    "\n",
    "An example would be: 90% of the times, the increase in average sales (before and after endorsement) would be within the bucket  3.4 and 6.7 (These numbers are illustrative. We will derive those numbers below)\n",
    "\n",
    "What is the hacker's way of doing it? We will do the following steps:\n",
    "\n",
    "- From actual sales data, we sample the data with repetition (separately for before and after) - sample size will be the same as the original\n",
    "- Find the differences between the mean of the two samples.\n",
    "- Repeat the above two steps, say 100,000 times.\n",
    "- Sort the differences. For getting 90% interval, take the 5% and 95% number. That range gives you the 90% confidence interval on the mean.\n",
    "\n",
    "**Does this process sound familiar** ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function to generate mean difference for many bootstrap samples\n",
    "\n",
    "def mean_diff_bootstrap(before_sales, after_sales, num_times):\n",
    "    \"\"\"\n",
    "      Function that computes the difference between the groups on a bootstrap sample and return the mean difference array\n",
    "      \n",
    "      arguments:\n",
    "         before_sales: sales before the campaign\n",
    "         after_sales : sales after the campaign\n",
    "         num_times: number of times to run the bootstrap sample\n",
    "    \n",
    "      output:\n",
    "        mean_diff : This returns the mean differene of each of the bootstrap groups as a numpy array\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    mean_diff = np.zeros([num_times, 1])\n",
    "    \n",
    "    for i in np.arange(num_times):\n",
    "        \n",
    "        bootstrap_before = np.random.choice(before_sales, before_sales.shape[0])\n",
    "        bootstrap_after = np.random.choice(after_sales, after_sales.shape[0])\n",
    "        bootstrap_mean_diff = np.mean(bootstrap_after) - np.mean(bootstrap_before)\n",
    "        \n",
    "        mean_diff[i] = bootstrap_mean_diff\n",
    "    \n",
    "        \n",
    "    return mean_diff\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_diff_bootstrap_experiment = mean_diff_bootstrap(sales_2, sales_1, 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Let's plot this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f947a3a54a8>"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADR5JREFUeJzt3X+MZeVdx/H3dG7bdbNj3YYrNkCymLTfBtcWDa0gUWi3\nmjWl4j+Som344Y+0IkKDIQWixD9MiJjqJpoYAqQlJa20ENukLYKNKzGBrS3ZBnXzrbauZbcgYzqk\ngyuug9c/7m0Cw+zcufecuWe/M+/XX/c8c8853yf33s8888w5z50bDAZIkup5TdcFSJKmY4BLUlEG\nuCQVZYBLUlEGuCQV1ZvlyRYXlwcAu3fvZGnpxCxP3Tn7vD1stz5vt/5CN33u9xfm1mrvZATe6813\ncdpO2eftYbv1ebv1F06vPjuFIklFGeCSVJQBLklFGeCSVJQBLklFGeCSVJQBLklFGeCSVJQBLklF\nzfRWeqkNBw8f77qEU1rYtYPlF17suoypXXr+WV2XoAk4ApekogxwSSrKAJekogxwSSrKAJekogxw\nSSrKAJekogxwSSrKAJekogxwSSrKAJekosauhRIR9wKXAc9l5t5R253A+4CTwDeBazLz+c0sVJL0\nShsZgX8c2L+q7VFgb2a+DfgGcEvLdUmSxhgb4Jn5GPDdVW2PZObKaPMJ4OxNqE2StI425sCvBb7U\nwnEkSRNotB54RNwGrAD3b+T5u3fvpNebB6DfX2hy6pLsczsWdu1o/ZhtOt3rW880r5fv6+5MHeAR\ncTXDf27uy8zBRvZZWjoBDDu/uLg87alLss/tOZ2/MKH6FzpM+nr5vp7dOdcyVYBHxH7gZuCSzDzR\noC5J0pQ2chnhp4BLgTMi4hhwO8OrTl4PPBoRAE9k5oc2sU5J0ipjAzwzr1yj+Z5NqEWSNAHvxJSk\nogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekogxw\nSSrKAJekogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekogxwSSrKAJekonrjnhAR9wKXAc9l5t5R\n2xuBvwT2AEeBKzJzafPKlCSttpER+MeB/avaPgp8OTPfDHx5tC1JmqGxAZ6ZjwHfXdV8OfCJ0eNP\nAL/Ucl2SpDGmnQM/MzOfGT1+FjizpXokSRs0dg58nMwcRMRgI8/dvXsnvd48AP3+QtNTl2Of27Gw\na0frx2zT6V7feqZ5vXxfd2faAP+PiHhTZj4TEW8CntvITktLJ4Bh5xcXl6c8dU32uT3LL7zY+jHb\nsrBrx2ld3ziTvl6+r2d3zrVMO4XyeeCq0eOrgM9NeRxJ0pQ2chnhp4BLgTMi4hhwO3AH8EBE/Brw\n78AVm1mkJOnVxgZ4Zl55ih/ta7kWSdIEvBNTkooywCWpKANckooywCWpKANckooywCWpKANckooy\nwCWpKANckooywCWpqMbLyaq+g4ePb8pxq6/MJ53uHIFLUlEGuCQVZYBLUlEGuCQVZYBLUlEGuCQV\nZYBLUlEGuCQVZYBLUlEGuCQVZYBLUlEGuCQV1Wgxq4j4CPDrwAB4CrgmM129SJJmYOoReEScBfwO\ncEFm7gXmgfe3VZgkaX1Np1B6wA9ERA/YCXyneUmSpI2YGwwGU+8cETcAfwj8N/BIZv7qes9fWXlp\n0OvNT32+Sh5+/GjXJUgT23/Rnq5L0Nrm1mqceg48InYDlwPnAs8Dn4mID2TmJ0+1z9LSCQD6/QUW\nF5enPXUJq7/IYDt+uYF9rmfSz+V2+Cyv1kWf+/2FNdubTKG8B/i3zFzMzP8FHgJ+usHxJEkTaHIV\nyreBCyNiJ8MplH3AV1upSpI01tQj8Mw8BHwWeJLhJYSvAe5qqS5J0hiNrgPPzNuB21uqRZI0Ae/E\nlKSiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6Si\nDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6SiDHBJKsoAl6Siek12jogfAu4G9gID\n4NrMfLyNwiRJ62s6Aj8APJyZbwXeDhxpXpIkaSOmHoFHxBuAnwWuBsjMk8DJdsqSJI0zNxgMptox\nIs4H7gL+meHo+2vADZn5X6faZ2XlpUGvNz/V+ap5+PGjXZcgTWz/RXu6LkFrm1ursckceA/4SeD6\nzDwUEQeAjwK/d6odlpZOANDvL7C4uNzg1Ke/5RdefMX2wq4dr2rb6uxzPZN+LrfDZ3m1Lvrc7y+s\n2d5kDvwYcCwzD422P8sw0CVJMzB1gGfms8DTERGjpn0Mp1MkSTPQ6DJC4Hrg/oh4HfAt4JrmJUmS\nNqJRgGfmYeCClmqRJE3AOzElqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgD\nXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKKMsAlqSgDXJKK\nMsAlqahe0wNExDzwVeB4Zl7WvCRJ0ka0MQK/ATjSwnEkSRNoFOARcTbwXuDudsqRJG1U0ymUPwVu\nBhY28uTdu3fS680D0O9vaJeyFnbt2FDbVmefa5nmc7nVP8trOV36PHWAR8RlwHOZ+bWIuHQj+ywt\nnQCGnV9cXJ721CUsv/DiK7YXdu14VdtWZ5/rmfRzuR0+y6t10edT/cJoMoVyMfCLEXEU+DTw7oj4\nZIPjSZImMPUIPDNvAW4BGI3AfzczP9BSXZKkMbwOXJKKanwdOEBmHgQOtnEsSdLGOAKXpKIMcEkq\nygCXpKIMcEkqygCXpKIMcEkqygCXpKIMcEkqygCXpKIMcEkqqpVb6SVp1g4ePt7JeaddMvjS889q\nvRZH4JJUlAEuSUUZ4JJUlAEuSUUZ4JJUlAEuSUUZ4JJUlAEuSUUZ4JJUlAEuSUUZ4JJUlAEuSUVN\nvZhVRJwD3AecCQyAuzLzQFuFSZLW12QEvgLclJnnARcC10XEee2UJUkaZ+oAz8xnMvPJ0eNl4AjQ\n/nqJkqQ1zQ0Gg8YHiYg9wGPA3sz83qmet7Ly0qDXm5/qHA8/fnSq/STpdLD/oj1Ndp9bq7HxFzpE\nxC7gQeDG9cIbYGnpBAD9/gKLi8sTnWeaBdRPJ9MuAl+Zfd76tlt/Yfo+T5p5L9fvL6zZ3ugqlIh4\nLcPwvj8zH2pyLEnSZKYO8IiYA+4BjmTmx9orSZK0EU2mUC4GPgg8FRGHR223ZuYXm5clSRpn6gDP\nzL/nFBPrkqTN552YklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklSUAS5JRRng\nklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklSUAS5JRRngklRU\nr8nOEbEfOADMA3dn5h2tVCVJGmvqEXhEzAN/DvwCcB5wZUSc11ZhkqT1NZlCeSfwr5n5rcw8CXwa\nuLydsiRJ4zSZQjkLePpl28eAn1pvh35/Ye5ljyc62S//3Fsner4kbXX+E1OSimoS4MeBc162ffao\nTZI0A02mUP4BeHNEnMswuN8P/EorVUmSxpp6BJ6ZK8BvA38NHAEeyMx/aqswSdL65gaDQdc1SJKm\n4D8xJakoA1ySimp0K30TEXE+8BfADmAF+K3M/EpX9cxKRFwPXAe8BHwhM2/uuKSZiIibgD8G+pn5\nn13Xs1ki4k7gfcBJ4JvANZn5fLdVbY7ttpRGRJwD3AecCQyAuzLzQJc1dTkC/yPgDzLzfOD3R9tb\nWkS8i+Hdqm/PzB9jGGhb3uiN//PAt7uuZQYeBfZm5tuAbwC3dFzPptimS2msADdl5nnAhcB1Xfe5\nywAfAD84evwG4Dsd1jIrHwbuyMz/AcjM5zquZ1b+BLiZ4Wu+pWXmI6MrtACeYHh/xFa07ZbSyMxn\nMvPJ0eNlhlffndVlTV0G+I3AnRHxNMOR6JYcqazyFuBnIuJQRPxdRLyj64I2W0RcDhzPzK93XUsH\nrgW+1HURm2StpTQ6DbNZiog9wE8Ah7qsY1PnwCPib4AfWeNHtwH7gI9k5oMRcQVwD/CezaxnFsb0\nuQe8keGfX+8AHoiIH83M0iPTMX2+leH0yZaxXn8z83Oj59zG8E/u+2dZmzZfROwCHgRuzMzvdVnL\npgZ4Zp4ykCPiPuCG0eZngLs3s5ZZGdPnDwMPjQL7KxHxf8AZwOKs6tsMp+pzRPw4cC7w9YiA4XTC\nkxHxzsx8doYltmq91xggIq4GLgP2Vf/lvI5tuZRGRLyWYXjfn5kPdV1PZ1ehMJzzvgQ4CLwb+JcO\na5mVvwLeBfxtRLwFeB2wZa/IyMyngB/+/nZEHAUu2OJXoexnON9/SWae6LqeTbTtltKIiDmGMwVH\nMvNjXdcD3Qb4bwAHIqIHvAj8Zoe1zMq9wL0R8Y8MLzO7aguP0LarPwNeDzw6+qvjicz8ULcltS8z\nVyLi+0tpzAP3boOlNC4GPgg8FRGHR223ZuYXuyrIW+klqSjvxJSkogxwSSrKAJekogxwSSrKAJek\nogxwSSrKAJekov4f3uwqfqDYu5IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f947a44c908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(mean_diff_bootstrap_experiment, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_diff_bootstrap_experiment_sorted = np.sort(mean_diff_bootstrap_experiment, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.62166667,  1.175     ])"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(mean_diff_bootstrap_experiment_sorted, [5, 95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run the above for 500k times and report the interval**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}