resampling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teaching statistical inference with resampling\n",
    "\n",
    "Copyright 2018 Allen Downey\n",
    "\n",
    "License: http://creativecommons.org/licenses/by/4.0/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure Jupyter so figures appear in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# Configure Jupyter to display the assigned value after an assignment\n",
    "%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "np.random.seed(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Euro problem\n",
    "\n",
    "In *Information Theory, Inference, and Learning Algorithms*, David MacKay writes, \"A statistical statement appeared in *The Guardian* on Friday January 4, 2002:\n",
    "\n",
    "*When spun on edge 250 times, a Belgian one-euro coin came\n",
    "up heads 140 times and tails 110. ‘It looks very suspicious\n",
    "to me’, said Barry Blight, a statistics lecturer at the London\n",
    "School of Economics. ‘If the coin were unbiased the chance of\n",
    "getting a result as extreme as that would be less than 7%’.*\n",
    "\n",
    "But do these data give evidence that the coin is biased rather than fair?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we answer MacKay's question, let's unpack what Dr. Blight said:\n",
    "\n",
    "\"If the coin were unbiased the chance of getting a result as extreme as that would be less than 7%\".\n",
    "\n",
    "To see where that comes from, I'll simulate the result of spinning an \"unbiased\" coin, meaning that the probability of heads is 50%.\n",
    "\n",
    "Here's an example with 10 spins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False,  True, False,  True,  True,\n",
       "        True])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spins = np.random.random(10) < 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.random.random` returns numbers between 0 and 1, uniformly distributed.  So the probability of being less than 0.5 is 50%.  \n",
    "\n",
    "The sum of the array is the number of `True` elements, that is, the number of heads:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(spins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can wrap that in a function that simulates `n` spins with probability `p`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spin(n, p):\n",
    "    return np.sum(np.random.random(n) < p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example with the actual sample size (250) and hypothetical probability (50%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "250"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heads, tails = 140, 110\n",
    "sample_size = heads + tails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "118"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hypo_prob = 0.5\n",
    "spin(sample_size, hypo_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each time we run this simulated experiment, we get a different outcome.\n",
    "\n",
    "Here's a function that runs the experiment (250 spins) many times, and collects the outcomes (number of heads) in an array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_experiments(n, p, iters):\n",
    "    t = [spin(n, p) for i in range(iters)]\n",
    "    return np.array(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "outcomes = run_experiments(sample_size, hypo_prob, 10000);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is an array of 10000 integers, each representing the number of heads in a simulated experiment.  The mean of `outcomes` is about 125:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "125.0893"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(outcomes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which makes sense.  On average, the expected number of heads is the product of the hypothetical probability and the sample size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "125.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expected = hypo_prob * sample_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how much the values in `outcomes` differ from the expected value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "diffs = outcomes - expected;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function plots a histogram of a sequence of values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_hist(values, low=None, high=None):\n",
    "    options = dict(alpha=0.5, color='C0')\n",
    "    xs, ys, patches = plt.hist(values,\n",
    "                               density=True,\n",
    "                               histtype='step', \n",
    "                               linewidth=3,\n",
    "                               **options)\n",
    "    \n",
    "    \n",
    "    plt.ylabel('Density')\n",
    "    plt.tight_layout()\n",
    "    return patches[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the deviations from the expected value look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xm8HFWZ//HP1wQCCIQhBJTNIAQ1OAgaYBhx1B+LhJ8QHGAIoiwyRkYRBVFZFAOugGNGBQUGkBCRZUCZCwQhgGwCMUHCEtYYgoQgBgNhDRh45o9zLima7txOciv33Hu/79frvlJ96lTVc7o7/fQ5VX1KEYGZmVlp3tLTAZiZmTXjBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKOvVJI2T9Mu8vLGk5yUNWAHHPUjSLZXHz0t6Zzft+1hJZ+XlYZJC0sBu2netz5GkCyTtWce+u5uk9STdL2lQT8dizTlB2TKRtIOkWyUtkDRf0u8lbdOTMUXEnyNi9Yh4tQeOvXpEzFpSHUkfkTSnjX19LyL+vTvikjRb0k6Vfdf2HEnaEngf8L/dvN8fSnpY0nOSHpB0QMP6kPRCTrzPdyb3vE6STpL0t/x3siQBRMSTwO+Asd0Zr3WfbvlWZv2LpDWBK4D/AC4GVgY+BLzck3H1BZIGRsSino5jGX0OOD+6/9f/LwC7Aw8B2wC/lTQzIm6t1HlfRMxssu1YYE9S4gxgMjALOD2vPx84A/hpN8ds3cA9KFsWmwNExAUR8WpEvBQR10TE3QCSNpV0ff7G+pSk8yWt1blx/lb/VUl352++Z+fhlqvyt+RrJf1Drts5xDVW0lxJT0j6SrOgGofDJN0g6du5d/ecpGskrVOpf4CkR3Oc32zsbTTse4ikDknPSvoDsGnD+pC0WV7eTdJ9+ZiPSzpK0luBq4D1K9/0189DlJdI+qWkZ4GDqsOWFZ9p1n5J50r6TuXx6700SROBjYHL8/G+1uQ5Wj+3a76kmZI+W9nXOEkXSzovt2WGpJFN3xHJKODGyvYHSbol94CelvSIpFFL2L6piPhWRDwQEa9FxBTgZmD7Njc/EPjPiJgTEY8D/wkcVFk/BXinpHcsbVxWPycoWxYPAa9KmiBpVGcyqRDwfWB94D3ARsC4hjp7ATuTkt3upA/vY4F1SO/LwxvqfxQYDuwCHN0qkTTxSeBgYF1ST+8oAEkjgJ8B+wNvBwYDGyxhP6cBC3Pdz+S/Vs4GPhcRawDvBa6PiBdIH+Bz8xDb6hExN9cfDVwCrEX6Rt/MUrc/Ij4N/BnYPR/v5CbVLgDmkF6rvYHvSdqxsn4P4MIcWwdwarNj5QS8CfBgw6rtctk6wMnA2Z1DbJJ+JumZFn93tzjOqqRe1IyGVTdJ+oukX0saVinfArir8viuXAZA7q3OJPWwrDBOULbUIuJZYAfSkMl/A/Pyt/D18vqZETE5Il6OiHnAj4APN+zmpxHxZP5WezMwJSLujIiXgd8AWzfUPyEiXoiIe4BfAPu1Ge4vIuKhiHiJNBy5VS7fG7g8Im6JiFeA43N73kTpgoK9gONzDPcCE5ZwzL8DIyStGRFPR8Qfu4jxtoi4LPcQXmpRZ1nb35KkjUiv49cjYmFETAfOAj5dqXZLREzK56wm0vqDvLOH/FxD+aMR8d95+wmkBN/5Pvl8RKzV4m/LFsc5nZRkrq6UfRgYBrwbmAtcocUXlawOLKjUXQCs3pkkKzGvhRXHCcqWSUTcHxEHRcSGpF7C+sB/AUhaV9KFeXjrWeCXpG/QVU9Wll9q8nj1hvqPVZYfzcdrx18qyy9W9rt+dZ8R8SLwtxb7GEo6X9sYQyt7AbsBj0q6UVJXw1GPdbG+sc7StH9J1gfmR0Q1qTzKG3uSjc/fKmp+ReEz+d81Gspf3z4/x/Dm17Ytkk4hvdf+rXqeKyJuiohXIuIZ4Eukntx78urngTUru1kTeL7hPNkalfitIE5Qttwi4gHgXNKHB6ThvQC2jIg1gU+Rhv2Wx0aV5Y1J35SXxxPAhp0P8tDRkBZ15wGLmsTQVERMjYjRpGHFy0g9N2jRQ1tCeVWr9r8ArFZZ97al2PdcYG1J1aSyMfB4G/G88SBpCPNP5POT7ZB0euV8XOPfjIa6J5CGSHfJPfglhsPi99sM3tjrex+V4cGcbDfjjcOAVggnKFtqkt4t6SuSNsyPNyINOd2eq6xB+ub6jKQNgK92w2G/KWk1SVuQzildtJz7uwTYXdI/S1oZOIEWSTQPT/0aGJdjGEE6+f4mklaWtL+kwRHxd+BZoPOS7ieBIZIGL0O8rdo/HdhN0tqS3gZ8uWG7J4Gmv8+KiMeAW4HvS1pF6TLxQ2h9Hqwrk3jzUG5LEXFo5Xxc49/r54kkHUM6l7hzRLyhlytpC0lbSRogaXXSRRCPA/fnKucBR0raQNL6wFdIX6Y6bQvMjogl9YithzhB2bJ4jnTye4qkF0iJ6V7Sf35IH/bvJ433X0n6cF9eN5JOZl8H/DAirlmenUXEDOCLpAsAniC16a+0vlT+MNLQ1F9IH3C/WMLuPw3MzsObh5J6kJ09zQuAWflCgKUZpmvV/omkb/+zgWt4c+L+PvCNfLyjmux3P9L5m7mkc3/fiojJSxFX1ZnA/g3nd7rD90g9u4crPaxj87r1SG1+lnT5+DDg4/nLAaRLyC8H7iG9R6/MZZ32Z/El51YY+YaFVrJ8RdYjwEp1/j4of/t+BhgeEY/UdZy+TtKvgIsj4rKejqUrktYlJf6tI2JhT8djb+YEZUWrM0FJ2p3UIxFpaGg74P01/NDUzJaBh/isPxtNGtqaS/qN0RgnJ7NyuAdlZmZFcg/KzMyK5ARlZmZF6jOzma+zzjoxbNiwng7DzMy6cMcddzwVEUO7qtdnEtSwYcOYNm1aT4dhZmZdkNTWD6M9xGdmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmReozM0mY9TXjJz/U0yG8wRE7b97TIVg/4x6UmZkVyQnKzMyK5CE+s16gp4bXShtmtP6l1h6UpF0lPShppqSjm6wfJOmivH6KpGG5fJiklyRNz3+n1xmnmZmVp7YelKQBwGnAzsAcYKqkjoi4r1LtEODpiNhM0hjgJGDfvO5PEbFVXfGZmVnZ6uxBbQvMjIhZEfEKcCEwuqHOaGBCXr4E2FGSaozJzMx6iToT1AbAY5XHc3JZ0zoRsQhYAAzJ6zaRdKekGyV9qNkBJI2VNE3StHnz5nVv9GZm1qPqTFDNekLRZp0ngI0jYmvgSOBXktZ8U8WIMyNiZESMHDq0y7sHm5lZL1JngpoDbFR5vCEwt1UdSQOBwcD8iHg5Iv4GEBF3AH8C/CtBM7N+pM4ENRUYLmkTSSsDY4COhjodwIF5eW/g+ogISUPzRRZIeicwHJhVY6xmZlaY2q7ii4hFkg4DrgYGAOdExAxJJwLTIqIDOBuYKGkmMJ+UxAD+BThR0iLgVeDQiJhfV6xmZlaeWn+oGxGTgEkNZcdXlhcC+zTZ7lLg0jpjMzOzsnmqIzMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMr0sCeDsCsROMnP9TTIZj1e+5BmZlZkZygzMysSLUO8UnaFfgxMAA4KyJ+0LB+EHAe8AHgb8C+ETG7sn5j4D5gXET8sM5YzVo5YufNezoEs36pth6UpAHAacAoYASwn6QRDdUOAZ6OiM2A8cBJDevHA1fVFaOZmZWrziG+bYGZETErIl4BLgRGN9QZDUzIy5cAO0oSgKQ9gVnAjBpjNDOzQtWZoDYAHqs8npPLmtaJiEXAAmCIpLcCXwdOqDE+MzMrWJ0JSk3Kos06JwDjI+L5JR5AGitpmqRp8+bNW8YwzcysRHVeJDEH2KjyeENgbos6cyQNBAYD84HtgL0lnQysBbwmaWFEnFrdOCLOBM4EGDlyZGPyMzOzXqzOBDUVGC5pE+BxYAzwyYY6HcCBwG3A3sD1ERHAhzorSBoHPN+YnMzMrG+rLUFFxCJJhwFXky4zPyciZkg6EZgWER3A2cBESTNJPacxdcVjZma9S62/g4qIScCkhrLjK8sLgX262Me4WoIzM7OieSYJMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkQb2dABm1juMn/xQT4fAETtv3tMh2ArkHpSZmRXJCcrMzIrkIT4za6mEIbUShhatZ7gHZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmR2kpQki6V9P8lLVVCk7SrpAclzZR0dJP1gyRdlNdPkTQsl28raXr+u0vSJ5bmuGZm1vu1m3B+DnwSeFjSDyS9u6sNJA0ATgNGASOA/SSNaKh2CPB0RGwGjAdOyuX3AiMjYitgV+AMSZ7Y1sysH2krQUXEtRGxP/B+YDYwWdKtkg6WtFKLzbYFZkbErIh4BbgQGN1QZzQwIS9fAuwoSRHxYkQsyuWrANF+k8zMrC9oe8hO0hDgIODfgTuBH5MS1uQWm2wAPFZ5PCeXNa2TE9ICYEg+3naSZgD3AIdWElY1prGSpkmaNm/evHabYmZmvUC756B+DdwMrAbsHhF7RMRFEfFFYPVWmzUpa+wJtawTEVMiYgtgG+AYSau8qWLEmRExMiJGDh06tJ2mmJlZL9HueZ2zImJStUDSoIh4OSJGtthmDrBR5fGGwNwWdebkc0yDgfnVChFxv6QXgPcC09qM18zMerl2h/i+06Tsti62mQoMl7SJpJWBMUBHQ50O4MC8vDdwfURE3mYggKR3AO8infsyM7N+Yok9KElvI50nWlXS1iwekluTNNzXUkQsknQYcDUwADgnImZIOhGYFhEdwNnAREkzST2nMXnzHYCjJf0deA34fEQ8tUwtNDOzXqmrIb6PkS6M2BD4UaX8OeDYrnaehwUnNZQdX1leCOzTZLuJwMSu9m9mZn3XEhNUREwAJkjaKyIuXUExmZmZdTnE96mI+CUwTNKRjesj4kdNNjMzM1tuXQ3xvTX/2+pScjMzs1p0NcR3Rv73hBUTjpmZWdLuD3VPlrSmpJUkXSfpKUmfqjs4MzPrv9r9HdQuEfEs8HHSj2s3B75aW1RmZtbvtZugOieE3Q24ICLmL6mymZnZ8mp3qqPLJT0AvAR8XtJQYGF9YZmZWX/X7u02jga2J92j6e/AC7z51hlmZmbdZmluAvge0u+hqtuc183xmJmZAW0mKEkTgU2B6cCruThwgjIzs5q024MaCYyICN/Z1szMVoh2r+K7F3hbnYGYmZlVtduDWge4T9IfgJc7CyNij1qiMjOzfq/dBDWuziDMzMwatZWgIuLGfGfb4RFxraTVSDchNDMzq0W7c/F9FrgEOCMXbQBcVldQZmZm7V4k8QXgg8CzABHxMLBuXUGZmZm1m6BejohXOh/kH+v6knMzM6tNuwnqRknHAqtK2hn4H+Dy+sIyM7P+rt0EdTQwD7gH+BwwCfhGXUGZmZm1exXfa5IuAy6LiHk1x2RmZrbkHpSScZKeAh4AHpQ0T9LxKyY8MzPrr7oa4vsy6eq9bSJiSESsDWwHfFDSEbVHZ2Zm/VZXCeoAYL+IeKSzICJmAZ/K68zMzGrRVYJaKSKeaizM56FWalLfzMysW3SVoF5ZxnVmZmbLpaur+N4n6dkm5QJWqSEeMzMzoIsEFRGeENbMzHpEuz/UNTMzW6GcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRao1QUnaVdKDkmZKOrrJ+kGSLsrrp0galst3lnSHpHvyv/+vzjjNzKw8tSUoSQOA04BRwAhgP0kjGqodAjwdEZsB44GTcvlTwO4R8Y/AgcDEuuI0M7My1dmD2haYGRGzIuIV4EJgdEOd0cCEvHwJsKMkRcSdETE3l88AVpE0qMZYzcysMHUmqA2AxyqP5+SypnUiYhGwABjSUGcv4M6IeLmmOM3MrEBt3fJ9GalJWSxNHUlbkIb9dml6AGksMBZg4403XrYozcysSHUmqDnARpXHGwJzW9SZI2kgMBiYDyBpQ+A3wAER8admB4iIM4EzAUaOHNmY/Mysjxk/+aGeDoEjdt68p0PoN+oc4psKDJe0iaSVgTFAR0OdDtJFEAB7A9dHREhaC7gSOCYifl9jjGZmVqjaElQ+p3QYcDVwP3BxRMyQdKKkPXK1s4EhkmYCRwKdl6IfBmwGfFPS9Py3bl2xmplZeeoc4iMiJgGTGsqOrywvBPZpst13gO/UGZuZ9Q4lDKmVMLTYH3kmCTMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyKVOtcfGbLwvOemRm4B2VmZoVygjIzsyJ5iM+KVsKtFsysZ7gHZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK1KtCUrSrpIelDRT0tFN1g+SdFFeP0XSsFw+RNLvJD0v6dQ6YzQzszLVlqAkDQBOA0YBI4D9JI1oqHYI8HREbAaMB07K5QuBbwJH1RWfmZmVrc4e1LbAzIiYFRGvABcCoxvqjAYm5OVLgB0lKSJeiIhbSInKzMz6oToT1AbAY5XHc3JZ0zoRsQhYAAxp9wCSxkqaJmnavHnzljNcMzMrSZ0JSk3KYhnqtBQRZ0bEyIgYOXTo0KUKzszMylZngpoDbFR5vCEwt1UdSQOBwcD8GmMyM7Neos4ENRUYLmkTSSsDY4COhjodwIF5eW/g+ohouwdlZmZ918C6dhwRiyQdBlwNDADOiYgZkk4EpkVEB3A2MFHSTFLPaUzn9pJmA2sCK0vaE9glIu6rK14zMytLbQkKICImAZMayo6vLC8E9mmx7bA6YzMzs7J5JgkzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFanWqY6s9xk/+aGeDsHMDHAPyszMCuUEZWZmRfIQn7V0xM6b93QIZtaPuQdlZmZFcoIyM7MiOUGZmVmRfA7KzGwplPBTjP5yftg9KDMzK5ITlJmZFclDfGZmXShhSK2EocUVzT0oMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyJ5NvOC9MfZis3MWnEPyszMilRrD0rSrsCPgQHAWRHxg4b1g4DzgA8AfwP2jYjZed0xwCHAq8DhEXF1nbGamfUWpYy21H2frNoSlKQBwGnAzsAcYKqkjoi4r1LtEODpiNhM0hjgJGBfSSOAMcAWwPrAtZI2j4hX64q3NCXcIM3MrCfV2YPaFpgZEbMAJF0IjAaqCWo0MC4vXwKcKkm5/MKIeBl4RNLMvL/b6gq2lG8kZmaW1JmgNgAeqzyeA2zXqk5ELJK0ABiSy29v2HaD+kI1MytbKaMqK/LLfJ0JSk3Kos067WyLpLHA2PzweUkPLlWE7VsHeKqmfTd15Io82GIrvJ09xO3sW9zOHrIcn1PvaKdSnQlqDrBR5fGGwNwWdeZIGggMBua3uS0RcSZwZjfG3JSkaRExsu7j9DS3s29xO/uW/tLOqjovM58KDJe0iaSVSRc9dDTU6QAOzMt7A9dHROTyMZIGSdoEGA78ocZYzcysMLX1oPI5pcOAq0mXmZ8TETMknQhMi4gO4GxgYr4IYj4piZHrXUy6oGIR8IX+dAWfmZnV/DuoiJgETGooO76yvBDYp8W23wW+W2d8S6H2YcRCuJ19i9vZt/SXdr5OaUTNzMysLJ7qyMzMiuQE1YKkb0u6W9J0SddIWj+XS9JPJM3M69/f07EuD0mnSHogt+U3ktaqrDsmt/NBSR/ryTiXl6R9JM2Q9JqkkQ3r+kw7IU0xltsyU9LRPR1Pd5J0jqS/Srq3Ura2pMmSHs7//kNPxri8JG0k6XeS7s/v2S/l8j7VznY4QbV2SkRsGRFbAVcAnefORpGuKhxO+g3Wz3sovu4yGXhvRGwJPAQcA9Aw3dSuwM/y9FW91b3AvwI3VQv7WjsrU4yNAkYA++U29hXnkl6nqqOB6yJiOHBdftybLQK+EhHvAf4J+EJ+DftaO7vkBNVCRDxbefhWFv9QeDRwXiS3A2tJevsKD7CbRMQ1EbEoP7yd9JszqEw3FRGPAJ3TTfVKEXF/RDT7IXefaieVKcYi4hWgc4qxPiEibiJd8Vs1GpiQlycAe67QoLpZRDwREX/My88B95Nm0ulT7WyHE9QSSPqupMeA/Vncg2o2hVNfmYbpM8BVebkvt7Oqr7Wzr7WnHetFxBOQPtyBdXs4nm4jaRiwNTCFPtzOVvr1DQslXQu8rcmq4yLifyPiOOC4fOuPw4Bv0eY0TCXpqp25znGkoYXzOzdrUr/Xt7PZZk3Kim5nF/pae/otSasDlwJfjohn0zza/Uu/TlARsVObVX8FXElKUG1Nw1SSrtop6UDg48COsfh3B32unS30unZ2oa+1px1PSnp7RDyRh9v/2tMBLS9JK5GS0/kR8etc3Ofa2RUP8bUgaXjl4R7AA3m5AzggX833T8CCzm53b5RvKvl1YI+IeLGyqr9MN9XX2tnOFGN9TXXKtAOBVr3lXiHfcuhs4P6I+FFlVZ9qZzv8Q90WJF0KvAt4DXgUODQiHs9vnlNJVxK9CBwcEdN6LtLlk6eZGkS6ozHA7RFxaF53HOm81CLSMMNVzfdSPkmfAH4KDAWeAaZHxMfyuj7TTgBJuwH/xeIpxkqZkWW5SboA+AhpZu8nSaMalwEXAxsDfwb2iYjGCyl6DUk7ADcD95A+fwCOJZ2H6jPtbIcTlJmZFclDfGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKC6sckvZpna58h6S5JR0papveEpJGSftJFna3yJdCdj/fortm2JR2eZ38+v+vavY+kgzpn1F+KbYZVZ/1eESTNlrTOCjjOKfl9e0pD+ThJR9VwvI9IuqK792tL1q9nkjBeyrO1I2ld0owZg0m/LVkq+bdgXf0ebCtgJPkuyxHRQff9iPTzwKg84evrJA2sTIbbmx1EmpG9z84KsZSv1eeAoRHxcp0xWc9yD8oAiIi/km4fclieJWNA/pY6Nd8r6nMAki5q6AWdK2mv6jdMSdtKulXSnfnfd+VZDU4E9s29tn1zr+DUvM07JF2Xj3WdpI0r+/9J3s8sSXs3xi7pdOCdQIekI/K36DMlXQOcJ2kVSb+QdE+O6aN5u4MkXSbpckmPSDos9yLvlHS7pLWbHGuopEvz8zJV0gdz+U8kHZ+XPybpJklvyfGfLulmSQ9J+niu0/T5zeu+lmO9S9IPcptHAufn525VSR+QdKOkOyRdrTyjfi6/S9JtwBeavdb5tbpB0iVK9wI7X0oTvVV7QLlXfENeHidpgtK90WZL+ldJJ+c4f6s0NU+nr0r6Q/7brIvn7Q2vVUOcys/Rvfk4++byDtIdBqZ0ljUYkds3S9Lhlf19Ksc0XdIZyrdVkfRzSdOUemQnVOrvmp+fW0i3auks/3Dex/T8Xlmj2fNs3SAi/NdP/4Dnm5Q9DaxHSlbfyGWDSL2jTYBPABNy+cqkmbNXJf26/4pcviYwMC/vBFyalw8CTq0c6/XHwOXAgXn5M8Bleflc4H9IX6ZGkG4l0awts4F18vI44A5g1fz4K8Av8vK7Sb/CXyUffyawBmmGiQWkGUMAxpNmlWg8zq+AHfLyxqTpaABWA2YAHwUeBDatxP/bHP9w0lx5qyzh+R0F3Aqsltetnf+9ARiZl1fKdYbmx/uSZowAuBv4cF4+Bbi3SRs+ktu6YY7rtkqbqs/jSOCGynN6Sz72+0izqIzK634D7FnZ/ri8fACL3xOtnrc3vFYNce5Ful/ZANJ78s/A21u9dyv7uzU/p+uQZkhZCXgP6T22Uq73M+CAhud4QH6et8yv0WP5NRNpBofOtlwOfDAvr05+r/uv+/88xGeNOqdM3gXYstJjGUz6z3oV8BNJg0jTPd0UES/pjTMtDwYmKM1nGKQPiK5sz+JvqROBkyvrLouI14D7JK3XZjs6IuKlvLwDaZojIuIBSY8Cm+d1v4t0z53nJC0gffhAmmZmyyb73Yn0Db3z8ZqS1oiI5yR9lnRDxCMi4k+VbS7O8T8saRYpSbZ6fnciJdMXc7zNprJ5F/BeYHKOYwDwhKTBwFoRcWOuN5GU8Jr5Q0TMAZA0HRhGSkBLclVE/F3SPfmYv83l9+TtO11Q+Xd8Xm76vOXl6mtVtQNwQUS8Spoo9UZgG7oeFr4y0tDfy5L+SkpuOwIfAKbmGFZl8WSr/yZpLOmUx9tJX4TeAjwSEQ8DSPol6UsFwO+BHymd7/x15/No3c8Jyl4n6Z3Aq6T/uAK+GBFXN6l3A/Ax0jf3CxrXA98mffB/Qul+NjcsQzjVObiq5xnavefAC21uU933a5XHr9H8/8dbgO1bfKD+I+kbe+PFDI3ziQUtnl+lyXu7mn9MwIyI2L5h27Xa2LZTtd2vsriti1g89L9Ks20i4jVJf4/br4yBAAACa0lEQVTcheDNz1U0WW76vOVkUX2t3rC6iza00qxtIvX8j2k4/ibAUcA2EfG0pHNZ3O6mz2VE/EDSlcBuwO2SdoqIB5rVteXjc1AGpHMEwOmkIbcArgb+o/PcgqTNJb01V78QOBj4UK7XaDDweF4+qFL+HGk4rZlbSTNvQ7pBZFff5pfGTXmfSNqcNMTU7O667biGdG8w8v46LzJ5B2kocWtglKTtKtvso3Q+alPSubIHaf38XgN8RtJqubzzPFj1uXsQGCpp+1xnJUlbRMQzwAKlyUbpbPNSmk3qaUAaYlsW+1b+vS0vN33eunAT6ZzlgPz+/BeWfab564C9lS4GQtLa+TVbk5QgF+TeeWeP8wFgk/yaAexXiX3TiLgnIk4iDc2+exljsi44QfVvq+YTvTOAa0kfIp0nic8C7gP+qHSp8hks/pZ8DenD4tpItxVvdDLwfUm/Jw0FdfodaZhnepOT24cDB0u6G/g08KXlb97rfgYMyENTFwEHxbJf/XU4MFLpwob7gEOl12+PcFREzAUOAc6S1PlN/EHgRtLw6KERsZAWz29E/JY0hDUtD711XjJ9LnB6LhsA7A2cJOkuYDrwz7newcBpShdJNOvldeUE4MeSbib1PpbFIElTSK/hEbnsTc9bG/v5Demc2l3A9cDXIuIvyxJQRNwHfAO4Jr/HJpPOZ90F3Ek6f3gOafiO/BqNBa7MF0k8Wtndl/OFG3eRnuNePft9yTybuVmN8pDRFRFxSU/HYtbbuAdlZmZFcg/KzMyK5B6UmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxI/wegeeI3UFE5hQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf6205f080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_hist(diffs)\n",
    "\n",
    "plt.title('Sampling distribution (n=250)')\n",
    "plt.xlabel('Deviation from expected number of heads');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the \"sampling distribution\" of deviations.  It shows how much variation we should expect between experiments with this sample size (n = 250)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## P-values\n",
    "\n",
    "Getting get back to this line:\n",
    "\n",
    "\"If the coin were unbiased the chance of getting a result as extreme as that would be less than 7%\".\n",
    "\n",
    "Let's count how many times, in 10000 attempts, the outcome is \"as extreme as\" the observed outcome, 140 heads.\n",
    "\n",
    "The observed deviation is the difference between the observed and expected number of heads:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observed_diff = heads - expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how many times the simulated `diffs` exceed the observed deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0313"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(diffs >= observed_diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's around 3%.  But Dr. Blight said 7%.  Where did that come from?\n",
    "\n",
    "So far, we only counted the cases where the outcome is *more* heads than expected.  We might also want to count the cases where the outcome is *fewer* than expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the probability of falling below the expected number by 15 or more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0334"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(diffs <= -observed_diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the total probability of a result \"as extreme as that\", we can use the absolute value of the simulated differences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0647"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(abs(diffs) >= observed_diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that's consistent with what Dr. Blight reported.\n",
    "\n",
    "To show what that looks like graphically, I'll use the following function, which fills in the histogram between `low` and `high`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fill_hist(low, high, patch):\n",
    "    options = dict(alpha=0.5, color='C0')\n",
    "    fill = plt.axvspan(low, high, \n",
    "                       clip_path=patch,\n",
    "                       **options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following plot shows the sampling distribution of `diffs` with two regions shaded.  These regions represent the probability that an unbiased coin yields a deviation from the expected as extreme as 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X+8VVWd//HXOxD8CSaipWhoYoWNWZHWZFPzNUv6ptSoI06lljPklP2wrLQfhk5TaTMxNdqYoyaS+WO0nGtiipqa+SMxMcUfSIiJmKEg+ANU4DN/rHVlezyHe4C771333vfz8bgP9ll77b0/65zD+Zy19j5rKyIwMzMrzSt6OwAzM7NmnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGV9mqTJkn6al3eU9LSkQT1w3CMk3Vh5/LSknbtp31+VdGZeHi0pJA3upn3X+hxJOl/Sh+rYd3eTtK2keyUN7e1YrDknKFsvkvaWdJOkpZIWS/qtpLf1ZkwR8aeI2DwiVvXCsTePiHlrqyPpPZIWtLGvb0fEP3ZHXJLmS3pvZd+1PUeSdgfeBPxvN+/33yQ9IOkpSfdJOqxhfUh6JifepzuTe14nSSdLeiL/nSJJABHxGPBrYFJ3xmvdp1u+ldnAImkY8Evgn4GLgCHAu4DnejOu/kDS4IhY2dtxrKdPAudF9//6/xlgf2AO8DbgV5LmRsRNlTpvioi5TbadBHyIlDgDmAHMA07P688Dfgz8ZzfHbN3APShbH7sCRMT5EbEqIpZHxFUR8QcASa+VdG3+xvq4pPMkbdm5cf5W/yVJf8jffM/Kwy1X5G/JV0t6Za7bOcQ1SdJCSY9K+mKzoBqHwyRdJ+lfcu/uKUlXSdq6Uv8wSQ/lOL/R2Nto2PcISR2Slkn6HfDahvUhaZe8/AFJ9+RjPiLpWEmbAVcA21W+6W+XhygvlvRTScuAI6rDlhWfaNZ+SedI+lbl8Yu9NEnTgB2By/LxvtzkOdout2uxpLmS/qmyr8mSLpJ0bm7LbEnjmr4jkvHA9ZXtj5B0Y+4BLZH0oKTxa9m+qYj4ZkTcFxGrI+JW4DfAO9rc/HDg3yNiQUQ8Avw7cERl/a3AzpJes65xWf2coGx9zAFWSZoqaXxnMqkQ8B1gO+ANwA7A5IY6BwL7kpLd/qQP768CW5Pel59tqP+3wBjgfcBxrRJJE/8AfBzYhtTTOxZA0ljgR8BHgFcDw4Ht17Kf04AVue4n8l8rZwGfjIgtgDcC10bEM6QP8IV5iG3ziFiY608ALga2JH2jb2ad2x8RHwP+BOyfj3dKk2rnAwtIr9VBwLcl7VNZfwBwQY6tAzi12bFyAt4JuL9h1V65bGvgFOCsziE2ST+S9GSLvz+0OM4mpF7U7IZVN0j6s6SfSxpdKd8NuLPy+M5cBkDurc4l9bCsME5Qts4iYhmwN2nI5L+BRflb+LZ5/dyImBERz0XEIuD7wLsbdvOfEfFY/lb7G+DWiLgjIp4DfgG8uaH+iRHxTETcBfwEOLTNcH8SEXMiYjlpOHKPXH4QcFlE3BgRzwMn5Pa8jNIFBQcCJ+QY7gamruWYLwBjJQ2LiCUR8fsuYrw5Ii7NPYTlLeqsb/tbkrQD6XX8SkSsiIhZwJnAxyrVboyI6fmc1TRaf5B39pCfaih/KCL+O28/lZTgO98nn4qILVv87d7iOKeTksyVlbJ3A6OB1wMLgV9qzUUlmwNLK3WXApt3JslKzFtixXGCsvUSEfdGxBERMYrUS9gO+A8ASdtIuiAPby0Dfkr6Bl31WGV5eZPHmzfUf7iy/FA+Xjv+XFl+trLf7ar7jIhngSda7GMk6XxtYwytHAh8AHhI0vWSuhqOeriL9Y111qX9a7MdsDgiqknlIV7ak2x8/jZW8ysKn8z/btFQ/uL2+TmGl7+2bZH0PdJ77e+r57ki4oaIeD4ingQ+R+rJvSGvfhoYVtnNMODphvNkW1Tit4I4QdkGi4j7gHNIHx6QhvcC2D0ihgEfJQ37bYgdKss7kr4pb4hHgVGdD/LQ0YgWdRcBK5vE0FRE3BYRE0jDipeSem7Qooe2lvKqVu1/Bti0su5V67DvhcBWkqpJZUfgkTbieelB0hDmH8nnJ9sh6fTK+bjGv9kNdU8kDZG+L/fg1xoOa95vs3lpr+9NVIYHc7LdhZcOA1ohnKBsnUl6vaQvShqVH+9AGnK6JVfZgvTN9UlJ2wNf6obDfkPSppJ2I51TunAD93cxsL+kv5Y0BDiRFkk0D0/9HJicYxhLOvn+MpKGSPqIpOER8QKwDOi8pPsxYISk4esRb6v2zwI+IGkrSa8CPt+w3WNA099nRcTDwE3AdyRtrHSZ+JG0Pg/Wlem8fCi3pYg4qnI+rvHvxfNEko4nnUvcNyJe0suVtJukPSQNkrQ56SKIR4B7c5VzgS9I2l7SdsAXSV+mOu0JzI+ItfWIrZc4Qdn6eIp08vtWSc+QEtPdpP/8kD7s30Ia77+c9OG+oa4nncy+Bvi3iLhqQ3YWEbOBz5AuAHiU1Ka/0PpS+aNJQ1N/Jn3A/WQtu/8YMD8Pbx5F6kF29jTPB+blCwHWZZiuVfunkb79zweu4uWJ+zvA1/Pxjm2y30NJ528Wks79fTMiZqxDXFVnAB9pOL/THb5N6tk9UOlhfTWv25bU5mWky8dHAx/MXw4gXUJ+GXAX6T16eS7r9BHWXHJuhZFvWGgly1dkPQhsVOfvg/K37yeBMRHxYF3H6e8k/Qy4KCIu7e1YuiJpG1Lif3NErOjteOzlnKCsaHUmKEn7k3okIg0N7QW8pYYfmprZevAQnw1kE0hDWwtJvzGa6ORkVg73oMzMrEjuQZmZWZGcoMzMrEj9ZjbzrbfeOkaPHt3bYZiZWRduv/32xyNiZFf1+k2CGj16NDNnzuztMMzMrAuS2vphtIf4zMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxI/WYmCbP+ZsqMOb0dwkscs++uvR2CDTDuQZmZWZGcoMzMrEge4jPrA3preK20YUYbWGrtQUnaT9L9kuZKOq7J+qGSLszrb5U0OpePlrRc0qz8d3qdcZqZWXlq60FJGgScBuwLLABuk9QREfdUqh0JLImIXSRNBE4GDsnr/hgRe9QVn5mZla3OHtSewNyImBcRzwMXABMa6kwApubli4F9JKnGmMzMrI+oM0FtDzxcebwglzWtExErgaXAiLxuJ0l3SLpe0ruaHUDSJEkzJc1ctGhR90ZvZma9qs4E1awnFG3WeRTYMSLeDHwB+JmkYS+rGHFGRIyLiHEjR3Z592AzM+tD6kxQC4AdKo9HAQtb1ZE0GBgOLI6I5yLiCYCIuB34I+BfCZqZDSB1JqjbgDGSdpI0BJgIdDTU6QAOz8sHAddGREgamS+yQNLOwBhgXo2xmplZYWq7ii8iVko6GrgSGAScHRGzJZ0EzIyIDuAsYJqkucBiUhID+BvgJEkrgVXAURGxuK5YzcysPLX+UDcipgPTG8pOqCyvAA5ust0lwCV1xmZmZmXzVEdmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmVqTBvR2AWYmmzJjT2yGYDXjuQZmZWZGcoMzMrEi1DvFJ2g/4ATAIODMivtuwfihwLvBW4AngkIiYX1m/I3APMDki/q3OWM1aOWbfXXs7BLMBqbYelKRBwGnAeGAscKiksQ3VjgSWRMQuwBTg5Ib1U4Ar6orRzMzKVecQ357A3IiYFxHPAxcAExrqTACm5uWLgX0kCUDSh4B5wOwaYzQzs0LVmaC2Bx6uPF6Qy5rWiYiVwFJghKTNgK8AJ9YYn5mZFazOBKUmZdFmnROBKRHx9FoPIE2SNFPSzEWLFq1nmGZmVqI6L5JYAOxQeTwKWNiizgJJg4HhwGJgL+AgSacAWwKrJa2IiFOrG0fEGcAZAOPGjWtMfmZm1ofVmaBuA8ZI2gl4BJgI/ENDnQ7gcOBm4CDg2ogI4F2dFSRNBp5uTE5mZta/1ZagImKlpKOBK0mXmZ8dEbMlnQTMjIgO4CxgmqS5pJ7TxLriMTOzvqXW30FFxHRgekPZCZXlFcDBXexjci3BmZlZ0TyThJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEiDezsAM+sbpsyY09shcMy+u/Z2CNaD3IMyM7MiOUGZmVmRPMRnZi2VMKRWwtCi9Q73oMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrUlsJStIlkv6/pHVKaJL2k3S/pLmSjmuyfqikC/P6WyWNzuV7SpqV/+6U9OF1Oa6ZmfV97Sac/wL+AXhA0nclvb6rDSQNAk4DxgNjgUMljW2odiSwJCJ2AaYAJ+fyu4FxEbEHsB/wY0me2NbMbABpK0FFxNUR8RHgLcB8YIakmyR9XNJGLTbbE5gbEfMi4nngAmBCQ50JwNS8fDGwjyRFxLMRsTKXbwxE+00yM7P+oO0hO0kjgCOAfwTuAH5ASlgzWmyyPfBw5fGCXNa0Tk5IS4ER+Xh7SZoN3AUcVUlY1ZgmSZopaeaiRYvabYqZmfUB7Z6D+jnwG2BTYP+IOCAiLoyIzwCbt9qsSVljT6hlnYi4NSJ2A94GHC9p45dVjDgjIsZFxLiRI0e20xQzM+sj2j2vc2ZETK8WSBoaEc9FxLgW2ywAdqg8HgUsbFFnQT7HNBxYXK0QEfdKegZ4IzCzzXjNzKyPa3eI71tNym7uYpvbgDGSdpI0BJgIdDTU6QAOz8sHAddGRORtBgNIeg3wOtK5LzMzGyDW2oOS9CrSeaJNJL2ZNUNyw0jDfS1FxEpJRwNXAoOAsyNitqSTgJkR0QGcBUyTNJfUc5qYN98bOE7SC8Bq4FMR8fh6tdDMzPqkrob43k+6MGIU8P1K+VPAV7vaeR4WnN5QdkJleQVwcJPtpgHTutq/mZn1X2tNUBExFZgq6cCIuKSHYjIzM+tyiO+jEfFTYLSkLzSuj4jvN9nMzMxsg3U1xLdZ/rfVpeRmZma16GqI78f53xN7JhwzM7Ok3R/qniJpmKSNJF0j6XFJH607ODMzG7ja/R3U+yJiGfBB0o9rdwW+VFtUZmY24LWboDonhP0AcH5ELF5bZTMzsw3V7lRHl0m6D1gOfErSSGBFfWGZmdlA1+7tNo4D3kG6R9MLwDO8/NYZZmZm3WZdbgL4BtLvoarbnNvN8ZiZmQFtJihJ04DXArOAVbk4cIIyM7OatNuDGgeMjQjf2dbMzHpEu1fx3Q28qs5AzMzMqtrtQW0N3CPpd8BznYURcUAtUZmZ2YDXboKaXGcQZmZmjdpKUBFxfb6z7ZiIuFrSpqSbEJqZmdWi3bn4/gm4GPhxLtoeuLSuoMzMzNq9SOLTwDuBZQAR8QCwTV1BmZmZtZugnouI5zsf5B/r+pJzMzOrTbsJ6npJXwU2kbQv8D/AZfWFZWZmA127Ceo4YBFwF/BJYDrw9bqCMjMza/cqvtWSLgUujYhFNcdkZma29h6UksmSHgfuA+6XtEjSCT0TnpmZDVRdDfF9nnT13tsiYkREbAXsBbxT0jG1R2dmZgNWVwnqMODQiHiwsyAi5gEfzevMzMxq0VWC2igiHm8szOehNmpS38zMrFt0laCeX891ZmZmG6Srq/jeJGlZk3IBG9cQj5mZGdBFgooITwhrZma9ot0f6pqZmfUoJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZFqTVCS9pN0v6S5ko5rsn6opAvz+lsljc7l+0q6XdJd+d//V2ecZmZWntoSlKRBwGnAeGAscKiksQ3VjgSWRMQuwBTg5Fz+OLB/RPwVcDgwra44zcysTHX2oPYE5kbEvIh4HrgAmNBQZwIwNS9fDOwjSRFxR0QszOWzgY0lDa0xVjMzK0ydCWp74OHK4wW5rGmdiFgJLAVGNNQ5ELgjIp6rKU4zMytQW7d8X09qUhbrUkfSbqRhv/c1PYA0CZgEsOOOO65flGZmVqQ6E9QCYIfK41HAwhZ1FkgaDAwHFgNIGgX8AjgsIv7Y7AARcQZwBsC4ceMak5+Z9TNTZszp7RA4Zt9dezuEAaPOIb7bgDGSdpI0BJgIdDTU6SBdBAFwEHBtRISkLYHLgeMj4rc1xmhmZoWqLUHlc0pHA1cC9wIXRcRsSSdJOiBXOwsYIWku8AWg81L0o4FdgG9ImpX/tqkrVjMzK0+dQ3xExHRgekPZCZXlFcDBTbb7FvCtOmMzs76hhCG1EoYWByLPJGFmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkWqdi89sfXjeMzMD96DMzKxQTlBmZlYkD/FZ0Uq41YKZ9Q73oMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkWqNUFJ2k/S/ZLmSjquyfqhki7M62+VNDqXj5D0a0lPSzq1zhjNzKxMtSUoSYOA04DxwFjgUEljG6odCSyJiF2AKcDJuXwF8A3g2LriMzOzstXZg9oTmBsR8yLieeACYEJDnQnA1Lx8MbCPJEXEMxFxIylRmZnZAFRngtoeeLjyeEEua1onIlYCS4ER7R5A0iRJMyXNXLRo0QaGa2ZmJakzQalJWaxHnZYi4oyIGBcR40aOHLlOwZmZWdnqTFALgB0qj0cBC1vVkTQYGA4srjEmMzPrI+pMULcBYyTtJGkIMBHoaKjTARyelw8Cro2ItntQZmbWfw2ua8cRsVLS0cCVwCDg7IiYLekkYGZEdABnAdMkzSX1nCZ2bi9pPjAMGCLpQ8D7IuKeuuI1M7Oy1JagACJiOjC9oeyEyvIK4OAW246uMzYzMyubZ5IwM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZFqnerI+p4pM+bw2LKeu0/k8hdWMXrEZj12PDPrO5yg7GUefPyZHjvWsuUvOEGZWVMe4jMzsyK5B2UtvX3nEbUfY8GSZzlm311rP46Z9T3uQZmZWZGcoMzMrEhOUGZmViSfgzIzWwdTZszhiWeeY8ignv9+P2TwKxg6eNCAOW/rBGVmto5un7+EYZts1OPHHfXKTRj1yk17/Li9xUN8ZmZWJPegzMy60DiktmDJsz3ek7ll3hM9erwSuAdlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViTPZl6QKTPm9HYIZmbFcIIqyMpVq5n96LLeDsPMrAi1JihJ+wE/AAYBZ0bEdxvWDwXOBd4KPAEcEhHz87rjgSOBVcBnI+LKOmMtxfLnV/V2CGZWqBdWBcufX8W3L7+312IY9AoxZHA6O1T3redrS1CSBgGnAfsCC4DbJHVExD2VakcCSyJiF0kTgZOBQySNBSYCuwHbAVdL2jUiBsyn99t3HtHbIZhZYR5btoLHlq3o1RheNWxjRm+9WY8cq84e1J7A3IiYByDpAmACUE1QE4DJefli4FRJyuUXRMRzwIOS5ub93VxXsD7/Y2ZWljoT1PbAw5XHC4C9WtWJiJWSlgIjcvktDdtuX1+oyZw/P8VmQ3vvtNzqCDYbko6/zbChvRZHT3ph1ereDsFsnW07bOMe/z96wB7b9ejxWpn1pyd77Fh1fhqrSVm0WaedbZE0CZiUHz4t6f51irB9WwOP17Tvpn7Skwdbo8fbCfCVnj5gL7WzF7id/Utx7fzC+m/6mnYq1ZmgFgA7VB6PAha2qLNA0mBgOLC4zW2JiDOAM7ox5qYkzYyIcXUfp7e5nf2L29m/DJR2VtX5Q93bgDGSdpI0hHTRQ0dDnQ7g8Lx8EHBtREQunyhpqKSdgDHA72qM1czMClNbDyqfUzoauJJ0mfnZETFb0knAzIjoAM4CpuWLIBaTkhi53kWkCypWAp8eSFfwmZlZzb+DiojpwPSGshMqyyuAg1ts+6/Av9YZ3zqofRixEG5n/+J29i8DpZ0vUhpRMzMzK4snizUzsyI5QbUg6V8k/UHSLElXSdoul0vSDyXNzevf0tuxbghJ35N0X27LLyRtWVl3fG7n/ZLe35txbihJB0uaLWm1pHEN6/pNOyFNMZbbMlfScb0dT3eSdLakv0i6u1K2laQZkh7I/76yN2PcUJJ2kPRrSffm9+zncnm/amc7nKBa+15E7B4RewC/BDrPnY0nXVU4hvQbrP/qpfi6ywzgjRGxOzAHOB6gYbqp/YAf5emr+qq7gb8DbqgW9rd2VqYYGw+MBQ7NbewvziG9TlXHAddExBjgmvy4L1sJfDEi3gC8Hfh0fg37Wzu75ATVQkRUpxXfjDU/FJ4AnBvJLcCWkl7d4wF2k4i4KiJW5oe3kH5zBpXppiLiQaBzuqk+KSLujYhmP+TuV+2kMsVYRDwPdE4x1i9ExA2kK36rJgBT8/JU4EM9GlQ3i4hHI+L3efkp4F7STDr9qp3tcIJaC0n/Kulh4COs6UE1m8Kp9mmYesgngCvycn9uZ1V/a2d/a087to2IRyF9uAPb9HI83UbSaODNwK3043a2MqDvByXpauBVTVZ9LSL+NyK+Bnwt3/rjaOCbtDkNU0m6ameu8zXS0MJ5nZs1qd/n29lssyZlRbezC/2tPQOWpM2BS4DPR8SyNI/2wDKgE1REvLfNqj8DLiclqLamYSpJV+2UdDjwQWCfWPO7g37Xzhb6XDu70N/a047HJL06Ih7Nw+1/6e2ANpSkjUjJ6byI+Hku7nft7IqH+FqQNKby8ADgvrzcARyWr+Z7O7C0s9vdF+WbSn4FOCAinq2sGijTTfW3drYzxVh/U50y7XCgVW+5T8i3HDoLuDcivl9Z1a/a2Q7/ULcFSZcArwNWAw8BR0XEI/nNcyrpSqJngY9HxMzei3TD5GmmhpLuaAxwS0Qcldd9jXReaiVpmOGK5nspn6QPA/8JjASeBGZFxPvzun7TTgBJHwD+gzVTjJUyI8sGk3Q+8B7SzN6PkUY1LgUuAnYE/gQcHBGNF1L0GZL2Bn4D3EX6/AH4Kuk8VL9pZzucoMzMrEge4jMzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5IT1AAmaVWerX22pDslfUHSer0nJI2T9MMu6uyRL4HufHxAd822Lemzefbn87qu3fdIOqJzRv112GZ0ddbvniBpvqSte+A438vv2+81lE+WdGwNx3uPpF92935t7Qb0TBLG8jxbO5K2Ic2YMZz025J1kn8L1tXvwfYAxpHvshwRHXTfj0g/BYzPE76+SNLgymS4fdkRpBnZ++2sEOv4Wn0SGBkRz9UZk/Uu96AMgIj4C+n2IUfnWTIG5W+pt+V7RX0SQNKFDb2gcyQdWP2GKWlPSTdJuiP/+7o8q8FJwCG513ZI7hWcmrd5jaRr8rGukbRjZf8/zPuZJ+mgxtglnQ7sDHRIOiZ/iz5D0lXAuZI2lvQTSXflmP42b3eEpEslXSbpQUlH517kHZJukbRVk2ONlHRJfl5uk/TOXP5DSSfk5fdLukHSK3L8p0v6jaQ5kj6Y6zR9fvO6L+dY75T03dzmccB5+bnbRNJbJV0v6XZJVyrPqJ/L75R0M/DpZq91fq2uk3Sx0r3AzpPSRG/VHlDuFV+XlydLmqp0b7T5kv5O0ik5zl8pTc3T6UuSfpf/dunieXvJa9UQp/JzdHc+ziG5vIN0h4FbO8sajM3tmyfps5X9fTTHNEvSj5VvqyLpvyTNVOqRnVipv19+fm4k3aqls/zdeR+z8ntli2bPs3WDiPDfAP0Dnm5StgTYlpSsvp7LhpJ6RzsBHwam5vIhpJmzNyH9uv+XuXwYMDgvvxe4JC8fAZxaOdaLj4HLgMPz8ieAS/PyOcD/kL5MjSXdSqJZW+YDW+flycDtwCb58ReBn+Tl15N+hb9xPv5cYAvSDBNLSTOGAEwhzSrReJyfAXvn5R1J09EAbArMBv4WuB94bSX+X+X4x5Dmytt4Lc/veOAmYNO8bqv873XAuLy8Ua4zMj8+hDRjBMAfgHfn5e8Bdzdpw3tyW0fluG6utKn6PI4Drqs8pzfmY7+JNIvK+LzuF8CHKtt/LS8fxpr3RKvn7SWvVUOcB5LuVzaI9J78E/DqVu/dyv5uys/p1qQZUjYC3kB6j22U6/0IOKzhOR6Un+fd82v0cH7NRJrBobMtlwHvzMubk9/r/uv+Pw/xWaPOKZPfB+xe6bEMJ/1nvQL4oaShpOmeboiI5XrpTMvDgalK8xkG6QOiK+9gzbfUacAplXWXRsRq4B5J27bZjo6IWJ6X9yZNc0RE3CfpIWDXvO7Xke6585SkpaQPH0jTzOzeZL/vJX1D73w8TNIWEfGUpH8i3RDxmIj4Y2Wbi3L8D0iaR0qSrZ7f95KS6bM53mZT2bwOeCMwI8cxCHhU0nBgy4i4PtebRkp4zfwuIhYASJoFjCYloLW5IiJekHRXPuavcvldeftO51f+nZKXmz5vebntgoBZAAADOklEQVT6WlXtDZwfEatIE6VeD7yNroeFL4809PecpL+Qkts+wFuB23IMm7BmstW/lzSJdMrj1aQvQq8AHoyIBwAk/ZT0pQLgt8D3lc53/rzzebTu5wRlL5K0M7CK9B9XwGci4som9a4D3k/65n5+43rgX0gf/B9Wup/NdesRTnUOrup5hnbvOfBMm9tU97268ng1zf9/vAJ4R4sP1L8ifWNvvJihcT6xoMXzqzR5b1fzjwmYHRHvaNh2yza27VRt9yrWtHUla4b+N262TUSslvRC5C4EL3+uosly0+ctJ4vqa/WS1V20oZVmbROp5398w/F3Ao4F3hYRSySdw5p2N30uI+K7ki4HPgDcIum9EXFfs7q2YXwOyoB0jgA4nTTkFsCVwD93nluQtKukzXL1C4CPA+/K9RoNBx7Jy0dUyp8iDac1cxNp5m1IN4js6tv8urgh7xNJu5KGmJrdXbcdV5HuDUbeX+dFJq8hDSW+GRgvaa/KNgcrnY96Lelc2f20fn6vAj4hadNc3nkerPrc3Q+MlPSOXGcjSbtFxJPAUqXJRuls8zqaT+ppQBpiWx+HVP69OS83fd66cAPpnOWg/P78G9Z/pvlrgIOULgZC0lb5NRtGSpBLc++8s8d5H7BTfs0ADq3E/tqIuCsiTiYNzb5+PWOyLjhBDWyb5BO9s4GrSR8inSeJzwTuAX6vdKnyj1nzLfkq0ofF1ZFuK97oFOA7kn5LGgrq9GvSMM+sJie3Pwt8XNIfgI8Bn9vw5r3oR8CgPDR1IXBErP/VX58Fxild2HAPcJT04u0Rjo2IhcCRwJmSOr+J3w9cTxoePSoiVtDi+Y2IX5GGsGbmobfOS6bPAU7PZYOAg4CTJd0JzAL+Otf7OHCa0kUSzXp5XTkR+IGk35B6H+tjqKRbSa/hMbnsZc9bG/v5Bemc2p3AtcCXI+LP6xNQRNwDfB24Kr/HZpDOZ90J3EE6f3g2afiO/BpNAi7PF0k8VNnd5/OFG3eSnuM+Pft9yTybuVmN8pDRLyPi4t6OxayvcQ/KzMyK5B6UmZkVyT0oMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmR/g8eSBlHuWW8ggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf5a5639b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "patch = plot_hist(diffs)\n",
    "\n",
    "low = observed_diff\n",
    "high = diffs.max()\n",
    "fill_hist(low, high, patch)\n",
    "\n",
    "low = diffs.min()\n",
    "high = -observed_diff\n",
    "fill_hist(low, high, patch)\n",
    "\n",
    "plt.title('Sampling distribution (n=250)')\n",
    "plt.xlabel('Deviation from expected number of heads');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results show that there is a non-negligible chance of getting a result as extreme as 140 heads, even if the coin is actually fair.\n",
    "\n",
    "So even if the results are \"suspicious\" they don't provide compelling evidence that the coin is biased."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimation\n",
    "\n",
    "Suppose we want to estimate the average height of men in the U.S.\n",
    "\n",
    "We can use data from the [BRFSS](https://www.cdc.gov/brfss/index.html):\n",
    "\n",
    "\"The Behavioral Risk Factor Surveillance System (BRFSS) is the nation's premier system of health-related telephone surveys that collect state data about U.S. residents regarding their health-related risk behaviors, chronic health conditions, and use of preventive services.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function reads the most recent data from BRFSS (2016), selects just the columns we need, and saves the results in an HDF file.\n",
    "\n",
    "After we run this function, it is much faster to read the HDF file than the original data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def read_brfss():\n",
    "    \"\"\"Read the BRFSS dataset, select columns, and save as HDF.\n",
    "    \"\"\"\n",
    "    df = pd.read_sas('LLCP2016.XPT')\n",
    "    df.head()\n",
    "    columns = ['SEX', 'HTM4', 'WTKG3', '_LLCPWT']\n",
    "    selected = df[columns]\n",
    "    selected.to_hdf('LLCP2016.HDF', 'brfss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can read the HDF file.  The result is a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SEX</th>\n",
       "      <th>HTM4</th>\n",
       "      <th>WTKG3</th>\n",
       "      <th>_LLCPWT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>6123.0</td>\n",
       "      <td>767.844566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>6940.0</td>\n",
       "      <td>329.659884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>5443.0</td>\n",
       "      <td>290.749306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>9979.0</td>\n",
       "      <td>211.039206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>5670.0</td>\n",
       "      <td>1582.539834</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SEX   HTM4   WTKG3      _LLCPWT\n",
       "0  1.0  173.0  6123.0   767.844566\n",
       "1  2.0  160.0  6940.0   329.659884\n",
       "2  2.0  165.0  5443.0   290.749306\n",
       "3  1.0  185.0  9979.0   211.039206\n",
       "4  1.0  168.0  5670.0  1582.539834"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_hdf('LLCP2016.HDF')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `SEX` to select male respondents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "210606"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "males = df[df.SEX==1]\n",
    "len(males)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we select height data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.894979250353741"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = males.HTM4\n",
    "np.mean(np.isnan(data)) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 3% of the values are missing.\n",
    "\n",
    "Here are the mean and standard deviation, ignoring missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean male height in cm = 178.06121490985726\n",
      "Std male height in cm = 7.792878100892029\n"
     ]
    }
   ],
   "source": [
    "print('Mean male height in cm =', np.nanmean(data))\n",
    "print('Std male height in cm =', np.nanstd(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantifying precision\n",
    "\n",
    "At this point we have an estimate of the average adult male height.  We'd like to know how accurate this estimate is, and how precise.  In the context of estimation, these words have a [technical distinction](https://en.wikipedia.org/wiki/Accuracy_and_precision):\n",
    "\n",
    "*\"Given a set of data points from repeated measurements of the same quantity, the set can be said to be precise if the values are close to each other, while the set can be said to be accurate if their average is close to the true value of the quantity being measured.\"*\n",
    "\n",
    "Usually accuracy is what we really care about, but it's hard to measure accuracy unless you know the true value.  And if you know the true value, you don't have to estimate it.\n",
    "\n",
    "Quantifying precision is not as useful, but it is much easier.  Here's one way to do it:\n",
    "\n",
    "1.  Use the data you have to make a model of the population.\n",
    "\n",
    "2.  Use the model to simulate the data collection process.\n",
    "\n",
    "3.  Use the simulated data to compute an estimate.\n",
    "\n",
    "4.  Repeat steps 1-3 and collect the results.\n",
    "\n",
    "To model the population, I'll use **resampling**; that is, I will treat the observed measurements as if they were taken from the entire population, and I will draw random samples from them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a function that takes observed measurements and returns a new set of measurements with the same sample size.\n",
    "\n",
    "With `replace=True`, we sample with replacement, which means that some measurements might be chosen more than once, and some might not be chosen at all.\n",
    "\n",
    "(If we sample *without* replacement, the resampled data is always identical to the original, so that's no good.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resample(data):\n",
    "    size = len(data)\n",
    "    return np.random.choice(data, size, \n",
    "                            replace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simulate an experiment, we run `resample` to generate data and `nanmean` to compute the mean (ignoring missing data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178.084596120171"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resampled_data = resample(data)\n",
    "np.nanmean(resampled_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can simulate 1000 experiments and collect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.45 s, sys: 4 ms, total: 4.46 s\n",
      "Wall time: 4.45 s\n"
     ]
    }
   ],
   "source": [
    "%time sampling_dist_mean = \\\n",
    "    [np.nanmean(resample(data)) \\\n",
    "     for i in range(1000)];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the \"sampling distribution\", which shows how much the results of the experiment would vary if we ran it many times.  Here's what it looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEzCAYAAACVGMzNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHPNJREFUeJzt3XucZGV95/HPV0DuIJdBAYExyqwhRNDXRI2aDUZxI14wq0ZRFBIi0aib4CVr3ERxdSPJqiRZcxHRgGgQvIOLykhERAEZBQERgXBRZAKDyDWKXH7545yWmp7unpqerq5nuj/v16tec+qcU+f8nu6a/tbz1KmnUlVIktSSh4y7AEmSJjOcJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc0xnDQ2SY5O8tF+ec8kdyXZZB7Oe3iScwfu35Xkl+bo2G9Ncny/vDRJJdl0jo49bz+jSed9eJJzktyZ5L1DPua6JM8cdW1auAynRSjJ05J8I8ntSW5N8vUkvzbOmqrqB1W1TVXdP4Zzb1NV18y0T5IDktwwxLH+sqr+YC7qmvwHfow/oyOBW4DtquqNkzcmOSHJu+a5Ji1wc/KKThuPJNsBnwdeA5wKPBT4DeCecda1ECTZtKruG3cdI7AXcHn5iX3NI3tOi88ygKo6uarur6qfVtWZVXUJQJJHJ/nXJD9OckuSjyV52MSD+1fzb05ySZK7k3yoH/b5Qj/s8+UkO/T7TgxrHZnkxiSrkqz1ynvSvpv2989O8s6+V3dnkjOT7Dyw/yuTXN/X+RczDSMl2SnJaUnuSPJN4NGTtleSx/TLByW5vD/nj5K8KcnWwBeA3fphtbuS7NYPS34yyUeT3AEcPjhUOeD3p2r/5B7HYO8syUnAnsDp/fn+dIqf0W59u25NcnWSVw0c6+gkpyb5SN+W7yZZPuUzotv/KUku7HvTFyZ5ykSNwGHAn/Z1PHPS444EXj6w/fSBzfv3z5Pbk5ySZIuBxz03ycVJbut78Y+bobZK8kdJrurb8s7+eXpe/zs9NclDhzl2krck+bf+OJcn+Z2BbYcnOTfJe5L8JMm1SZ49XV0asarytohuwHbAj4ETgWcDO0za/hjgQGBzYAlwDvA3A9uvA84HHg7sDtwMfBt4fP+YfwXe3u+7FCjgZGBr4FeB1cAz++1HAx+dtO+m/f2zgX+jC9Mt+/vH9Nv2Ae4CnkbX83sPcO/Ecado88fpeolbA/sCPwLOHdhewGP65VXAb/TLOwBP6JcPAG6YdNyj+/O+gO6F3pbTtGm69p8AvGvgeGuco/9ZP3Pg/uSf0VeBfwC2APbvj/2Mgdp+BhwEbAK8Gzh/mp/PjsBPgFfQjaYc0t/faao6p3j8Wtv72r8J7NYf/3vAq/ttT6B73jypr+2wfv/Npzl+AafRPXd/ha6XfxbwS8D2wOXAYcMcG3hxX9NDgJcAdwO79tsO73+fr+of+xrgRiDj/n+7GG/2nBaZqrqD7o96AR8EVvevvh/eb7+6qlZU1T1VtRp4H/Cbkw7z/6rqpqr6EfA14IKquqiq7gE+QxdUg95RVXdX1aXAP9P98RvGP1fVlVX1U7pw2b9f/yLg9Ko6t6p+Drytb89a0l088ELgbX0Nl9EF83TuBfZJsl1V/aSqvr2OGs+rqs9W1QN9nVOZbfunlWQPut/j/6yqn1XVxcDxdAEz4dyqOqO696hOAvab5nDPAa6qqpOq6r6qOhm4AnjeBpb5d1V1Y1XdCpzOg7+/VwEfqKoLquu9n0gXOE+e4Vh/VVV3VNV3gcuAM6vqmqq6na5XO/Gcm/HYVfWJvqYHquoU4CrgiQPnub6qPtj/zE4EdqV7IaZ5ZjgtQlX1vao6vKoeSdeT2A34G4AkuyT5eD+kdQfwUWDnSYe4aWD5p1Pc32bS/j8cWL6+P98w/n1g+T8Gjrvb4DGr6j/oeoNTWULXG5hcw3ReSNfbuD7JV5P8+jpq/OE6tk/eZ33aP5PdgFur6s5Jx9594P7kn98WmfrKwd1Y+2cy+VizMd3vby/gjf2w221JbgP2YOafy7DPuRmP3Q8HXzywbV/WfH7/oub+eQVrP581DwynRa6qrqAbltm3X/Vuul7I46pqO+BQIBt4mj0GlvekGyrZEKuAR07cSbIlsNM0+64G7puihilV1YVVdTCwC/BZuh4bTNMzm2H9oOnafzew1cC2R6zHsW8Edkyy7aRj/2iIeqY61l6T1q3Psdb3QokfAv+nqh42cNuq77FtqGmPnWQvutGC19ENWT6Mrhe2oc9vjYDhtMgkeWySNyZ5ZH9/D7phpvP7Xbalez/ntiS7A2+eg9P+RZKtkvwK8HvAKRt4vE8Cz+vfxH8o8A6m+QPTD898Gji6r2Efuvch1pLkoUlenmT7qroXuAOYuGz7JmCnJNvPot7p2n8xcFCSHZM8AviTSY+7ie59lana9UPgG8C7k2zRv+l/BPCxWdR3BrAsycuSbJrkJXTv631+yMdPW+c0Pgi8OsmT0tk6yXMmBe1szXTsremCdDVAkt/jwRdlaozhtPjcSfdm8QVJ7qYLpcuAiavI3kH3pvLtwP+n+8O+ob4KXE33JvZ7qurMDTlY/77D6+kudFhF16abmf5y+NfRDc38O10v8Z9nOPwrgOv6Ic1X0/UcJ3qYJwPX9ENC6zM0N137TwK+Q/eG/ZmsHdrvBv68P9+bpjjuIXQXSdxI917f26tqxXrUBUBV/Rh4Lt1z4MfAnwLPrapbhjzEh+jep7styWeHON9KuveG3k934cXVdBcjbLCZjl1VlwPvBc6jC9RfBb4+F+fV3EuVH13QaCRZClwLbFYj/PxPkm2A24C9q+raUZ1H0vyx56SNUpLn9UNlW9NdSn4pXQ9E0gJgOGljdTDdcNaNwN7AS8thAGnBcFhPktQce06SpOYYTpKk5mwUs5LvvPPOtXTp0nGXIUnaQN/61rduqaol69pvowinpUuXsnLlynGXIUnaQElmmj7sFxzWkyQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNWejmCFCGpVjV1w57hLWctSBy8ZdgjR29pwkSc0xnCRJzXFYT+qNczitxeFFaZzsOUmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkpozsnBKskeSryT5XpLvJvnjfv2OSVYkuar/d4dR1SBJ2jiNsud0H/DGqvpl4MnAa5PsA7wFOKuq9gbO6u9LkvQLIwunqlpVVd/ul+8EvgfsDhwMnNjvdiLwglHVIEnaOM3Le05JlgKPBy4AHl5Vq6ALMGCX+ahBkrTxGHk4JdkG+BTwJ1V1x3o87sgkK5OsXL169egKlCQ1Z6ThlGQzumD6WFV9ul99U5Jd++27AjdP9diqOq6qllfV8iVLloyyTElSY0Z5tV6ADwHfq6r3DWw6DTisXz4M+NyoapAkbZw2HeGxnwq8Arg0ycX9urcCxwCnJjkC+AHw4hHWIEnaCI0snKrqXCDTbH7GqM4rSdr4jbLnJGkWjl1x5bhLAOCoA5eNuwQtYk5fJElqjuEkSWqOw3pSA1oZQmtlSFGy5yRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWrOpuMuQIvTsSuuHHcJkhpmz0mS1BzDSZLUHIf1NHZHHbhs3CVIaow9J0lSc0YWTkk+nOTmJJcNrDs6yY+SXNzfDhrV+SVJG69R9pxOAH57ivXHVtX+/e2MEZ5fkrSRGlk4VdU5wK2jOr4kaeEax3tOr0tyST/st8MYzi9Jatx8h9M/Ao8G9gdWAe+dbsckRyZZmWTl6tWr56s+SVID5jWcquqmqrq/qh4APgg8cYZ9j6uq5VW1fMmSJfNXpCRp7OY1nJLsOnD3d4DLpttXkrR4jexDuElOBg4Adk5yA/B24IAk+wMFXAf84ajOL0naeI0snKrqkClWf2hU55MkLRzOECFJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqjuEkSWrOyGaIkLRxO3bFleMuAYCjDlw27hI0BvacJEnNMZwkSc1xWE/SL7QyhNbKkKLGx56TJKk5hpMkqTmGkySpOUOFU5JPJXlOEsNMkjRyw4bNPwIvA65KckySx46wJknSIjdUOFXVl6vq5cATgOuAFUm+keT3kmw2ygIlSYvP0MN0SXYCDgf+ALgI+Fu6sFoxksokSYvWUJ9zSvJp4LHAScDzqmpVv+mUJCtHVZwkaXEa9kO4x1fVGYMrkmxeVfdU1fIR1CVJWsSGHdZ71xTrzpvLQiRJmjBjzynJI4DdgS2TPB5Iv2k7YKsR1yZJWqTWNaz33+gugngk8L6B9XcCbx1RTZKkRW7GcKqqE4ETk7ywqj41TzVJkha5dQ3rHVpVHwWWJnnD5O1V9b4pHiZJ0gZZ17De1v2/24y6EEmSJqxrWO8D/b/vmJ9yJEkafuLXv06yXZLNkpyV5JYkh466OEnS4jTs55yeVVV3AM8FbgCWAW8eWVWSpEVt2HCamNz1IODkqrp1RPVIkjT09EWnJ7kC+CnwR0mWAD8bXVmSpMVs2K/MeAvw68DyqroXuBs4eJSFSZIWr2F7TgC/TPd5p8HHfGSO65EkaeivzDgJeDRwMXB/v7ownCRJIzBsz2k5sE9V1SiLkSQJhr9a7zLgEaMsRJKkCcP2nHYGLk/yTeCeiZVV9fyRVCVJWtSGDaejR1mEJEmDhgqnqvpqkr2Avavqy0m2AjYZbWmSpMVq2Ln1XgV8EvhAv2p34LOjKkqStLgNe0HEa4GnAncAVNVVwC6jKkqStLgNG073VNXPJ+70H8T1snJJ0kgMG05fTfJWYMskBwKfAE4fXVmSpMVs2HB6C7AauBT4Q+AM4M9nekCSDye5OcllA+t2TLIiyVX9vzvMtnBJ0sI17MSvD9BdAPFHVfWiqvrgELNFnAD89qR1bwHOqqq9gbP6+5IkrWHGcErn6CS3AFcA30+yOsnb1nXgqjoHmPy9TwcDJ/bLJwIvmEXNkqQFbl09pz+hu0rv16pqp6raEXgS8NQkR83ifA+vqlUA/b/TXvGX5MgkK5OsXL169SxOJUnaWK0rnF4JHFJV106sqKprgEP7bSNTVcdV1fKqWr5kyZJRnkqS1Jh1hdNmVXXL5JVVtZoHv7p9fdyUZFeA/t+bZ3EMSdICt65w+vkst03nNOCwfvkw4HOzOIYkaYFb19x6+yW5Y4r1AbaY6YFJTgYOAHZOcgPwduAY4NQkRwA/AF683hVLkha8GcOpqmY9uWtVHTLNpmfM9piSpMVh2A/hSpI0bwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lSczYddwGaf8euuHLcJUjSjOw5SZKaYzhJkprjsN4id9SBy8ZdgiStxZ6TJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOYaTJKk5hpMkqTmGkySpOZuO46RJrgPuBO4H7quq5eOoQ5LUprGEU+/pVXXLGM8vSWqUw3qSpOaMK5wKODPJt5IcOaYaJEmNGtew3lOr6sYkuwArklxRVecM7tCH1pEAe+655zhqlNSAY1dcOe4SADjqwGXjLmFRGUvPqapu7P+9GfgM8MQp9jmuqpZX1fIlS5bMd4mSpDGa93BKsnWSbSeWgWcBl813HZKkdo1jWO/hwGeSTJz/X6rqi2OoQ1KjWhlCa2VIcTGa93CqqmuA/eb7vJKkjYeXkkuSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmjPOr8xYdPxAnyQNx56TJKk5hpMkqTkO641JK3OHSVKL7DlJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmrPpuAuQpI3BsSuuHHcJHHXgsnGXMG/sOUmSmmM4SZKa47CeJE2jhWG0FoYTx8GekySpOYaTJKk5hpMkqTmGkySpOYvigojF+oaiJG2s7DlJkppjOEmSmjOWYb0kvw38LbAJcHxVHTNf527hcwuSNButvEUxH39H573nlGQT4O+BZwP7AIck2We+65AktWscw3pPBK6uqmuq6ufAx4GDx1CHJKlR4xjW2x344cD9G4AnjaEOSWpeK29FzPeQ4jjCKVOsq7V2So4Ejuzv3pXk+3Nx8jfMxUFGY2fglnEXMY9s78Jmexew/u/obNu81zA7jSOcbgD2GLj/SODGyTtV1XHAcfNV1LglWVlVy8ddx3yxvQub7V34Rt3mcbzndCGwd5JHJXko8FLgtDHUIUlq1Lz3nKrqviSvA75Edyn5h6vqu/NdhySpXWP5nFNVnQGcMY5zN2zRDGH2bO/CZnsXvpG2OVVrXYsgSdJYOX2RJKk5htMcSPLhJDcnuWxg3SlJLu5v1yW5uF+/WZITk1ya5HtJ/myaYz4qyQVJruqP9dB+/RuSXJ7kkiRnJRnqssy5Np9tHtj+oiSVZN6viprv9ib53f73/N0k/zL6Fq5V23w+p/dM8pUkF/XP64Pmp5Vr1DaK9r4uydX9c3bngfVJ8nf9tkuSPGH0LVyrtvls78v7dl6S5BtJ9huqyKrytoE34L8CTwAum2b7e4G39csvAz7eL28FXAcsneIxpwIv7Zf/CXhNv/x0YKt++TXAKQu9zf39bYFzgPOB5Qu5vcDewEXADv39XRZ4e48bWN4HuG6BtPfxwNJ++84D6w8CvkD3mc8nAxcs8PY+ZeC5/Oxh22vPaQ5U1TnArVNtSxLgd4GTJ3YHtk6yKbAl8HPgjike81vAJ/tVJwIv6M/1lar6j379+XSfE5t389nm3juBvwZ+NkdNWC/z3N5XAX9fVT/pz33z3LVkOPPc3gK265e3Z4rPPY7aXLe3P+ZFVXXdFIc8GPhIdc4HHpZk1w1vxfDms71V9Y2J5zLr8TfLcBq93wBuqqqr+vufBO4GVgE/AN5TVZOfJDsBt1XVff39G+imfZrsCLpXYK2Z0zYneTywR1V9fuSVz85c/46XAcuSfD3J+elm8W/JXLf3aODQJDfQXcX7+hHWPhuzae9MpprCbar/3+My1+0dNPTfrEXxTbhjdggPvgKBbuLb+4HdgB2AryX5clVdM7DPOqd4SnIosBz4zbktd07MWZuTPAQ4Fjh8RLXOhbn+HW9KN7R3AN2rzK8l2beqbpvrwmdprtt7CHBCVb03ya8DJ/XtfWAEtc/GbNo7k6GmcBujuW4vAEmeThdOTxtmf3tOI9R3g/87cMrA6pcBX6yqe/vhmq/ThcygW+i6+hMvHtaY4inJM4H/BTy/qu4ZVf2zMYI2bwvsC5yd5Dq6MfrTMoaLIqYyot/xDcDn+sdfC3yfLqzGbkTtPYLu/Siq6jxgC7p528ZuA9o7k6GmcBuHEbWXJI8DjgcOrqofD/MYw2m0nglcUVU3DKz7AfBb/RU7W9P9sb1i8EHVvXP4FeBF/arDgM/BL4a4PkAXTPP+XsQQ5rTNVXV7Ve1cVUuraindmPXzq2rlqBsypDn/HQOfpbvwhf6qp2XAer1KHaFRtPcHwDMAkvwyXTitHlkL1s+s2rsOpwGv7B//ZOD2qlo1dyVvkDlvb5I9gU8Dr6iq4ac2n6urPxbzja4LvAq4l+5V0RH9+hOAV0/adxvgE8B3gcuBNw9sOwPYrV/+JeCbwNX9/pv3678M3ARc3N9OW+htnnSssxnP1Xrz+TsO8L7+sZfSX+G2gNu7D92r8e/0z+lnLZD2/o/+WPfR9YyOH/j9/j3wb/3vd6E8n6dr7/HAT3jwb9bKYWp0hghJUnMc1pMkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3DSvOlnKz5p4P6mSVYnaWJaoiR3DbHP2RMfAE7y1hHVsXRwtui53r9/zKuTvHId+xye5P3TbBuq7elmtx6cofqAqX7fSR6Sbqbuy9LNfn1hkkcNcw4tTIaT5tPdwL5JtuzvHwj8aIz1bKiRhNN8qKp/qqqPbMAh5rrtL6GbHudxVfWrwO8ArUzXpDEwnDTfvgA8p19eYw6vJFun+56ZC9N9t8/B/fqlSb6W5Nv97Sn9+gP6nswnk1yR5GP9jMprSPKq/pjfSfKpJFv16x+V5Lx+2zsH9l/j1X2S9yc5fNIxjwG2TPfdNx+b4px3JfmrJN9K8uUkT+xrvSbJ82dq16TjbJLk//Y1XpLkD6f5uW6S5IPpvv/pzIkXAEkeneSLfR1fS/LYfv3RSd7UL/9af+zz+nMN9sJ26x9/VZK/Hqbts7QrsKr6+fSq6oZ6cCZrLUbz/clkb4v3BtwFPI5uluMt6D4tfgDw+X77XwKH9ssPA64Etqb7Dpkt+vV703/CvH/s7XRzkz0EOA942hTn3Wlg+V3A6/vl04BX9suvBe4aOO7nBx7zfuDwfvls+k/0T+w/TVsLeHa//BngTGAzYD/g4n79dO1aSv89O8CRwJ/3y5sDK4FHTTrXUrpP5e/f3z914Od4FrB3v/wk4F/75aOBN/XLlwFP6ZePGTj34XTTJm3f/76up5sdfsa2T6rtOtb8bp81frYD6x/Z73sx3XcJPX7cz1dv4705K7nmVVVdkmQpXa/pjEmbnwU8f+IVPd0fxD3ppkJ5f5L96WZHXjbwmG9WPw9Yum/uXAqcO+m4+yZ5F13gbQN8qV//VOCF/fJJwF9tSNsm+TnwxX75UuCeqro3yaV9jdCF1XTtmvAs4HFJJuak254uyK6dtN+1VXVxv/wtYGmSbei+6O0TAx3KzQcflORhwLZV9Y1+1b8Azx3Y5ayqur3f93JgL9b8uod1mWoKmrXWVdUNSf4L3Xc+/RZwVpIXV9VZ63EuLSCGk8bhNOA9dK+idxpYH+CFVfX9wZ2THE03n+B+dD2kwS8cHJyV/X6mfk6fALygqr7TD88dMLBtqj+e97HmkPcW0zVkBvdW1cSxH5ios6oeyIMzcx/F9O2aELqe3pem2DZo8s9hy/6Yt1XV/jM8bqqvb5jpuOv7N+PHdF+zcEt/f8eB5TVUN8P+F4AvJLmJ7ssIDadFyvecNA4fBv53VV06af2XgNdPvG+UbgZ26HoLE+9HvALYZD3Pty2wKslmwMsH1n8deGm/PLj+emCfJJsn2Z5+xuwp3Nsfc7aGadeXgNdMnCfJsnQzQ69TVd0BXJvkxf1jk2S/Sfv8BLgz3ezY8ODPY13WaHuSs5JM9YV5Z9O1jSSbAIfSzU6+hiRPSLJbv/wQuuHf64esRQuQ4aR5V92b3X87xaZ30g11XdK/KT9xkcI/AIclOZ9u6Ovu9TzlXwAXACtYc6r/PwZem+RCuqCYqO+HdO/bXAJ8DLhomuMe19c624sChmnX8XQzQX+7/5l8gPXrvbwcOCLJd+hmlT54in2OAI5Lch5dT+r2IY77i7b3YfIYpv7a73cCj+nPfxHdjOQfBUiyPMnx/X67AKf3bbyErvf6/n6/49PI93dp/jgrubTIJdmmqu7ql98C7FpVf7wej98X+P2qesOoatTiYzhJi1ySlwB/Rtcju57uysRWvuxPi5ThJElqju85SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmvOfSbefVIyPqKkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf4ef1d9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_hist(sampling_dist_mean)\n",
    "plt.title('Sampling distribution of the mean')\n",
    "plt.xlabel('Mean adult male height, U.S.');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The width of this distribution shows how much the results vary from one experiment to the next.\n",
    "\n",
    "We can quantify this variability by computing the standard deviation of the sampling distribution, which is called \"standard error\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.01769828262861241"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "std_err = np.std(sampling_dist_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also summarize the sampling distribution with a \"confidence interval\", which is a range that contains a specified fraction, like 90%, of the values in `sampling_dist_mean`.\n",
    "\n",
    "The central 90% confidence interval is between the 5th and 95th percentiles of the sampling distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([178.02951276, 178.08952944])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ci_90 = np.percentile(sampling_dist_mean, [5, 95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function plots a histogram and shades the 90% confidence interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_sampling_dist(dist):\n",
    "    patch = plot_hist(dist)\n",
    "    low, high = np.percentile(dist, [5, 95])\n",
    "    fill_hist(low, high, patch)\n",
    "    print('Mean = ', np.mean(dist))\n",
    "    print('Std error = ', np.std(dist))\n",
    "    print('90% CI = ', (low, high))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what it looks like for the sampling distribution of mean adult height:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean =  178.06027748143575\n",
      "Std error =  0.01769828262861241\n",
      "90% CI =  (178.02951275886966, 178.08952943883128)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEmCAYAAADGA1/lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAF3ZJREFUeJzt3Xu4XXV95/H3h4Q7opgEixeMKKkyKugEi9LpoBbH2lZ0rFUUxSkV66ijOHVK7YzDqH9YW+XRR2tNqQNa73ecQW1kRKyCGpSbguAI0hQGgiAEQQjkO3+sdchOOJedcPbev5zzfj3Pec7e6/r9ZZ/sz/6ttfZvpaqQJKklu0y6AEmStmU4SZKaYzhJkppjOEmSmmM4SZKaYzhJkppjOEmSmmM4SZKaYzhJkpqzdNIFDGP58uW1cuXKSZchSbqfLrjgghurasVcy+0U4bRy5UrWrVs36TIkSfdTkp8Ns5yH9SRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc3ZKUaIkEbl1LVXcPPtd7Hp7s2TLoVl++zOkl3CSUevmnQp0sQZTlr0rv3FHWz81d2TLoP99t6NJWTSZUhN8LCeJKk59pyk3hEHLZvYvs//6c8ntm+pRfacJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNGVk4JXlEkq8nuSzJD5O8vp/+4CRrk1zZ/95vVDVIknZOo+w53Q3856p6HHAE8JokhwAnA2dX1cHA2f1zSZLuNbJwqqrrqur7/eONwGXAw4BjgDP6xc4AnjeqGiRJO6exnHNKshJ4EvAd4CFVdR10AQbsP8M6JyZZl2Tdhg0bxlGmJKkRIw+nJPsAnwXeUFW3DrteVa2pqtVVtXrFihWjK1CS1JyRhlOSXemC6aNV9bl+8vVJDujnHwDcMMoaJEk7n1FerRfg74HLqurdA7POBI7vHx8PfHFUNUiSdk5LR7jtI4GXAZckubCf9mbgHcCnkpwAXAO8cIQ1SJJ2QiMLp6r6JyAzzH7mqPYr7exOXXvFpEsA4KSjV026BC1io+w5SdoO19x0O7tkps9z47FL4JHL9p5oDRIYTlIzNmy8c9IlsHSXGE5qguEkNeCIg5ZNugQA1l1906RLkAAHfpUkNchwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDXHcJIkNcdwkiQ1x3CSJDVn6aQL0OJ06toruPWOTfxq0z2TLoVN99SkS5C0DcNJE3PjbXdyw8Y7J12GpAZ5WE+S1Bx7Tpq4Iw5aNukSJDXGnpMkqTkjC6ckH0pyQ5JLB6adkuRfklzY/zxnVPuXJO28RtlzOh149jTTT62qw/qfs0a4f0nSTmpk4VRV5wI3jWr7kqSFaxLnnF6b5OL+sN9+E9i/JKlx4w6nDwCPBg4DrgPeNdOCSU5Msi7Jug0bNoyrPklSA8YaTlV1fVXdU1Wbgb8DnjLLsmuqanVVrV6xYsX4ipQkTdxYwynJAQNPnw9cOtOykqTFa2Rfwk3yceAoYHmS9cB/B45KchhQwNXAq0a1f0nSzmtk4VRVx04z+e9HtT9J0sLhCBGSpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmjGyECEk7n80F1/7iDgDe9OmLJlbHbkt3Yfk+uwNw0tGrJlaHJsdwknSvzVVcc9Ptky6DffZYem84aXHysJ4kqTn2nCTd64iDlk26BAAuvfaWSZegCbPnJElqjuEkSWqO4SRJao7hJElqzlDhlOSzSX43iWEmSRq5YcPmA8BLgCuTvCPJY0dYkyRpkRsqnKrqa1X1UuDJwNXA2iTfTvIfkuw6ygIlSYvP0IfpkiwDXgH8MfAD4D10YbV2JJVJkhatob6Em+RzwGOBjwC/X1XX9bM+mWTdqIqTJC1Ow44QcVpVnTU4IcnuVXVnVa0eQV2SpEVs2MN6b59m2nnzWYgkSVNm7Tkl+TXgYcCeSZ4EpJ+1L7DXiGuTJC1Scx3W+3d0F0E8HHj3wPSNwJtHVJMkaZGbNZyq6gzgjCQvqKrPjqkmSdIiN9dhveOq6h+AlUneuO38qnr3NKtJknS/zHVYb+/+9z6jLkSSpClzHdb7YP/7f4ynHEmShh/49Z1J9k2ya5Kzk9yY5LhRFydJWpyG/Z7Ts6rqVuD3gPXAKuBNI6tKkrSoDRtOU4O7Pgf4eFXdNKJ6JEkaeviiLyW5HLgD+I9JVgC/Gl1ZkqTFbNhbZpwMPBVYXVWbgF8Cx4yyMEnS4jVszwngcXTfdxpc58PzXI8kSUPfMuMjwKOBC4F7+smF4SRJGoFhe06rgUOqqkZZjCRJMPzVepcCvzbKQiRJmjJsz2k58KMk3wXunJpYVc8dSVWSpEVt2HA6ZZRFSJI0aKhwqqpvJHkkcHBVfS3JXsCS0ZYmSVqshh1b75XAZ4AP9pMeBnxhVEVJkha3YS+IeA1wJHArQFVdCew/qqIkSYvbsOF0Z1XdNfWk/yKul5VLkkZi2HD6RpI3A3smORr4NPCl2VZI8qEkNyS5dGDag5OsTXJl/3u/HS9dkrRQDRtOJwMbgEuAVwFnAf91jnVOB549zXbOrqqDgbP755IkbWXYq/U2J/kC8IWq2jDkOucmWbnN5GOAo/rHZwDnAH82zPYkSYvHrD2ndE5JciNwOfDjJBuSvGUH9/eQqroOoP8940UVSU5Msi7Jug0bhspDSdICMddhvTfQXaV3eFUtq6oHA78BHJnkpFEWVlVrqmp1Va1esWLFKHclSWrMXOH0cuDYqrpqakJV/RQ4rp+3va5PcgBA//uGHdiGJGmBmyucdq2qG7ed2J932nWa5edyJnB8//h44Is7sA1J0gI3VzjdtYPzSPJx4Dzg15OsT3IC8A7g6CRXAkf3zyVJ2spcV+sdmuTWaaYH2GO2Favq2BlmPXOYwiRJi9es4VRVDu4qSRq7Yb+EK0nS2BhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTmGE6SpObMept2LUynrr2Cq278JXvttmSiddyx6Z6J7l9SuwynRerK6zey7567TroMSZqWh/UkSc2x57TIHXHQskmXIEn3Yc9JktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktQcw0mS1BzDSZLUHMNJktScpZPYaZKrgY3APcDdVbV6EnVIkto0kXDqPb2qbpzg/iVJjfKwniSpOZPqORXwj0kK+GBVrdl2gSQnAicCHHjggWMuT1IrTl17xaRLAOCko1dNuoRFZVLhdGRVXZtkf2Btksur6tzBBfrAWgOwevXqmkSRkibj9rvu4dJrb9mudTbesYmnPnr5iCrSuE0knKrq2v73DUk+DzwFOHf2tSQtFps3F7f96u7tWmfjdi6vto09nJLsDexSVRv7x88C3jruOiS164iDlm33Outvvn3eD721ckhxMZpEz+khwOeTTO3/Y1X1lQnUIUlq1NjDqap+Chw67v1KknYeXkouSWqO4SRJao7hJElqjuEkSWqO4SRJao7hJElqziRHJV90/EKfJA3HnpMkqTmGkySpOR7Wm5BJD7+//ubbefh+e020BkmaiT0nSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnMMJ0lScwwnSVJzDCdJUnOWTroASdoZnLr2ikmXwElHr5p0CWNjz0mS1BzDSZLUHA/rSdIMWjiM1sLhxEmw5yRJao7hJElqjuEkSWqO4SRJas6iuCBisZ5QlKSdlT0nSVJzDCdJUnMWxWG9QS18b0GSdkQrpyjG8T46kZ5Tkmcn+XGSnyQ5eRI1SJLaNfZwSrIEeD/wO8AhwLFJDhl3HZKkdk3isN5TgJ9U1U8BknwCOAb40QRqWbQesu8e7L/v7pMuQ5o3m+7ZPOkSRqKVUxHjPqQ4iXB6GPDPA8/XA7+x7UJJTgRO7J/eluTH87HzN87HRkZjOXDjpIsYI9u7sE2kvX827h1usahe3/59dEfb/MhhFppEOGWaaXWfCVVrgDWjL6cNSdZV1epJ1zEutndhs70L36jbPIkLItYDjxh4/nDg2gnUIUlq1CTC6XvAwUkelWQ34MXAmROoQ5LUqLEf1ququ5O8FvgqsAT4UFX9cNx1NGjRHMLs2d6FzfYufCNtc6ruc7pHkqSJcvgiSVJzDCdJUnMMp3mQ5ENJbkhy6cC0Tya5sP+5OsmF/fRdk5yR5JIklyX58xm2+agk30lyZb+t3frpb0zyoyQXJzk7yVDfGZhv42zzwPw/SFJJxn7J7rjbm+QP+9f5h0k+NvoW3qe2cf5NH5jk60l+0P9dP2c8rdyqtlG097XphmirJMsHpifJe/t5Fyd58uhbeJ/axtnel/btvDjJt5McOlSRVeXP/fwBfgt4MnDpDPPfBbylf/wS4BP9472Aq4GV06zzKeDF/eO/BV7dP346sFf/+NXAJxd6m/vnDwDOBc4HVi/k9gIHAz8A9uuf77/A27tm4PEhwNULpL1PAlb285cPTH8O8GW673weAXxngbf3aQN/y78zbHvtOc2DqjoXuGm6eUkC/CHw8anFgb2TLAX2BO4Cbp1mnWcAn+knnQE8r9/X16vq9n76+XTfExu7cba59zbgncCv5qkJ22XM7X0l8P6qurnf9w3z15LhjLm9BezbP34gE/je43y3t9/mD6rq6mk2eQzw4eqcDzwoyQH3vxXDG2d7q+rbU3/LbMd7luE0ev8GuL6qruyffwb4JXAdcA3w11W17R/JMuAXVXV3/3w93bBP2zqB7hNYa+a1zUmeBDyiqv7XyCvfMfP9Gq8CViX5VpLzkzx7tOVvt/lu7ynAcUnWA2cBrxth7TtiR9o7m+mGcJvu//ekzHd7Bw39nrXo7uc0Acey5RMIdAPf3gM8FNgP+GaSr1U/EG5vziGekhwHrAb+7fyWOy/mrc1JdgFOBV4xolrnw3y/xkvpDu0dRfcp85tJHl9Vv5jvwnfQfLf3WOD0qnpXkqcCH+nb28pIrjvS3tkMNYTbBM13ewFI8nS6cPrNYZa35zRCfTf43wOfHJj8EuArVbWpP1zzLbqQGXQjXVd/6sPDVkM8Jflt4C+A51bVnaOqf0eMoM0PAB4PnJPkarpj9GdmAhdFTGdEr/F64Iv9+lcBP6YLq4kbUXtPoDsfRVWdB+xBN6joxN2P9s6m2SHcRtRekjwROA04pqp+Psw6htNo/TZweVWtH5h2DfCM/oqdvenebC8fXKm6M4dfB/6gn3Q88EW49xDXB+mCaeznIoYwr22uqluqanlVrayqlXTHrJ9bVetG3ZAhzftrDHyB7sIX+queVgHb9Sl1hEbR3muAZwIkeRxdOG0YWQu2zw61dw5nAi/v1z8CuKWqrpu/ku+XeW9vkgOBzwEvq6rh77sxX1d/LOYfui7wdcAmuk9FJ/TTTwf+ZJtl9wE+DfyQ7h5WbxqYdxbw0P7xQcB3gZ/0y+/eT/8acD1wYf9z5kJv8zbbOofJXK03ztc4wLv7dS+hv8JtAbf3ELpP4xf1f9PPWiDt/U/9tu6m6xmdNvD6vh/4v/3ru1D+nmdq72nAzWx5z1o3TI0OXyRJao6H9SRJzTGcJEnNMZwkSc0xnCRJzTGcJEnNMZx0v/WjEH9k4PnSJBuSNDHcUJLbhljmnKkv9iZ584jqWDk4CvR8L9+v8ydJXj7HMq9I8r4Z5g3V9nSjVg+OPH3UdK93kl3SjcB9abpRrb+X5FFDbP8NU+1I8pfpRrT+8MD8lyV5/cDzJyQ5fZjatXMwnDQffgk8Psme/fOjgX+ZYD3310jCaRyq6m+r6sNzLzmj+W77i+iGvXliVT0BeD4w6zBM/SgFfwR8LMkDgadV1ROBJX0I7Uk3nNXfTK1TVZcAD++/8KkFwHDSfPky8Lv9463G5kqyd7r7x3wv3T17jumnr0zyzSTf73+e1k8/qu/JfCbJ5Uk+muQ+45EleWW/zYuSfDbJXv30RyU5r5/3toHlt/p0n+R9SV6xzTbfAeyZ7p42H51mn7f1n+QvSPK1JE/pa/1pkufO1q5ttrMkyV/1NV6c5FUz/LsuSfJ36e7r9I9THwCSPDrJV/o6vpnksf30U5L8af/48H7b5/X7GuyFPbRf/8ok7xym7TvoAOC66sfJq6r1tWWE6pk8A/h+dYPEbgZ261//Pem+NPom4L1VtWmb9b4EvHie6taEGU6aL58AXpxkD+CJwHcG5v0F8H+q6nC6YXn+qh8G5Qbg6Kp6Mt0n7PcOrPMk4A10owccBBw5zT4/V1WHV9WhwGV0Y7QBvAf4QL+//7c9jaiqk4E7quqwqnrpNIvsDZxTVf8a2Ai8na6n+Hzgrf0ys7Vrygl0w9YcDhwOvHKGw10H090+41/R9The0E9fA7yur+NPGehFDPifdN/2fyrdwJ2DDutrewLwoiSPGKLtO+JTwO/3gfeudMNvzeVI4AKAqtoIfJbu/lZXAbcAh1fVF6dZbx3diNpaAByVXPOiqi5OspKu13TWNrOfBTx36hM93dhpB9INcfK+JIfRvXmuGljnu9WP75XujpwrgX/aZruPT/J24EF0Q6x8tZ9+JFvexD8C/OX9ads27gK+0j++BLizqjYluaSvEWBXZm7XlGcBT0wyNdbcA+mC6Kptlruqqi7sH18ArEyyD90N3D490KHcfXClJA8CHlBV3+4nfQz4vYFFzq6qW/plfwQ8kq1v4zCX6YaWuc+0qlqf5NfpekPPAM5O8sKqOnuWbR9A92FjahvvpLuXF0lOA96S5I/p/g0vrqq394veQHcIUQuA4aT5dCbw13S3elg2MD3AC6rqx4MLJzmFbpzAQ+l68YM3Ehwcbf0epv9bPR14XlVd1B+eO2pg3nRvnnez9dGCPWZqyCw21ZYxvzZP1VlVm7NlxO2TmLldU0LX8/nqNPMGbfvvsGe/zV9U1WGzrDfdbRlm2+72vhf8nO72CTf2zx888Hgr1Y2c/2Xgy0mup7vJ4GzhdAfTvDYDva4rgPdU1W8l+USSg6u799Ae/bpaADysp/n0IeCt/cnpQV8FXjd13mjgTeaBbDkf8TJgyXbu7wHAdUl2BQYPQ32LLeceBqf/DDgkye79ifZnzrDdTf02d9Qw7foq8Oqp/SRZ1R/qnFNV3QpcleSF/bpJcug2y9wMbEw36jUMfy5mq7YnOTvJdDfCO4eubSRZAhxHN+r4VpI8OclD+8e70B3y/dkcNVwGPGaa6W8D3kLXM536N91Md+tw6Hqo23V1o9plOGne9Ce73zPNrLfRvaFc3J+Un7pI4W+A45OcT/fG8svt3OV/ozu3tZath/B/PfCaJN+jC4qp+v6Z7hzIxcBH6c5jTGdNX+uOXhQwTLtOoxvh+fv9v8kH2b7ey0uBE5JcRDda9DHTLHMCsCbJeXQ9qVuG2O69be/D5DFMfzvvtwGP6ff/A7qRxv8BIMnq/vAbwP7Al/o2XkzXe31fv9xpmf6+XF8GfmtwQpLnAd+rqmuru+nief2h1Kqqi/rFng787yHaqJ2Ao5JLC1SSfarqtv7xycABVfX6OVYbXP/xwB9V1RtHVeMs+/488F9qy63C51p+d+AbwG/WllvBaydmOEkLVJIXAX9O1yP7GfCKqmrlJn6z6i+ieEhVnTvk8gcDD6uqc0ZamMbGcJIkNcdzTpKk5hhOkqTmGE6SpOYYTpKk5hhOkqTm/H+ClFtBGuDXawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf5a471860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_sampling_dist(sampling_dist_mean)\n",
    "plt.xlabel('Mean adult male height, U.S. (%)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an experiment like this, we can compute the standard error analytically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def analytic_stderr(data):\n",
    "    size = len(data)\n",
    "    return np.std(data) / np.sqrt(size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is close to what we observed computationally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.01698097071589304, 0.01769828262861241)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analytic_stderr(data), std_err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other sample statistics\n",
    "\n",
    "One nice thing about using computaton is that it is easy to compute the sampling distribution for other statistics.\n",
    "\n",
    "For example, suppose we want to estimate the coefficient of variation for adult male height (standard deviation as a percentage of the mean).  We can define a function to compute it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def coef_var(data):\n",
    "    return np.nanstd(data) / np.nanmean(data) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And estimate the sampling distribution by running simulated experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "sampling_dist_cv = [coef_var(resample(data))\n",
    "                    for i in range(1000)];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the sampling distribution of CV looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean =  4.3765423194516915\n",
      "Std error =  0.0096268557742705\n",
      "90% CI =  (4.361073922387724, 4.391944430007911)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmYXFW97vHvSxJmwhAaDAQMIKg4EKRFEPUgCscBjQjXAyKKU9CrR0VFkXtUcAK8ajjHAY0yREAGGRwwIgEZ5KJoIgESg4AxyhBJh8EQCCHD7/6xVmNRqequdLKrVqXfz/P009V7r733ryqdfmutvWttRQRmZmal2aDTBZiZmTXigDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgrKtIOlnS+fnxzpKWSBrRhuMeK+mmmp+XSNp1He37JEk/yI/HSwpJI9fRvtv2GtUdd3tJN0p6TNLX23lsW384oKwlkl4h6WZJ/5T0sKT/J+mlnawpIv4eEZtHxMoOHHvziJg3UBtJB0q6r4V9fSUi3rcu6pI0X9Jra/bdqddoErAIGB0Rn2jUQNK+kqZJejT/Tv1e0rsl7ShphaTdGmxzhaSvNVi+paRf5X1dUBvIkr4v6bB1+eSsPRxQNihJo4ErgW8C2wA7AqcAyzpZ1/pgXfWUCvRs4E/RZCYASfsDvwZuAJ4DjAE+CLw+Iu4HrgWOqdtmG+ANwNQGuzwOuBXYHhgPHFZznLERccXaPyVrNweUtWIPgIi4MCJWRsTSiLg6Im4HkLSbpF9LekjSovwOdqv+jfO7+hMk3S7pcUln5SGgX+YhoGskbZ3b9g9xTZL0gKQFkpq9A3/GcJik6yV9MffuHpN0taRta9q/U9Lfcp2fre9t1O17jKSfSVos6ffAbnXrQ9Jz8uM3SPpTPub9kj4paTPgl8AOeYhtiaQd8hDlpZLOl7QYOLZ22LLGexo9f0nnSvpSzc9P99IknQfsDPw8H+9TDV6jHfLzeljSPZLeX7OvkyVdIumH+bnMkdTb8DcitX+5pD/kXvUfJL28v0bgXcCnch2NXuP/C0yNiNMjYlEkMyPibXn9VOoCCjgSmBMRdzTY3y7AdRGxDPgNsGvuRU0GPtrsOVjZHFDWiruAlZKmSnp9f5jUEHAqsAPwfGAn4OS6NocDB5PC7k2kP94nAduSfg8/Utf+1cDuwCHAic2CpIG3A+8GtgM2BD4JIGlP4DvA0cBYYEtST7CZbwNP5rbvyV/NnAUcFxFbAC8Efh0RjwOvBx7IQ2ybR8QDuf1E4FJgK+CCJvtc4+cfEccAfwfelI/31QbNLgTuI/1bHQF8RdJrata/Gbgo1/Yz4FuNjpV7M78A/ofU+/kG8AtJYyLi2Py8vprruKZu202B/UmvQTNXANtKekXNsmOAHzZpPxt4raRNgFcCc0i/U7+MiL8McBwrmAPKBhURi4FXAAF8H+jL78K3z+vviYjpEbEsIvpIf6z+rW4334yIB/PwzW+AWyLi1vyO9wpg77r2p0TE4/nd8jnAUS2We05E3BURS4FLgAl5+RHAzyPipoh4Cvhcfj6rye+8Dwc+l2uYTeNhpX7LgT0ljY6IRyLij4PU+NuI+ElErMp1NjLU59+UpJ1I/46fjognI2IW8AOe2VO5KSKm5XNW5wF7NdndG4G7I+K8iFgRERcCd5LefAxma9LfngXNGuTX5cfAO3PtuwP7AD9qsslZpDcdt5B+v27Lz+sMSWcqXbDxpSbbWqEcUNaSiJgbEcdGxDhSL2EH4AwASdtJuigPby0Gzif1jGo9WPN4aYOfN69rf2/N47/l47XiHzWPn6jZ7w61+4yIJ4CHmuyjBxjZoIZmDiedG/mbpBvyeY+B3DvI+vo2a/L8B7ID8HBEPFa379qeZP3rt3GT82Q7sPprUr+vZh4BVpF6pwOZCrxN0saksLkqIhY2apgDd1JEvDgiTiQN7Z1E6jGPIL1hepmk17VQnxXCAWVrLCLuBM4lBRWk4b0AXhwRo4F3kIb91sZONY93Bh5o1rBFC4Bx/T/koaAxTdr2ASsa1NBQRPwhIiaShhV/Quq5QZMe2gDLazV7/o8Dm9ase9Ya7PsBYBtJW9Tt+/4W6mm0r2fXLWtpX/nNwW9JwT5Qu9+Q3kRMJP1ONRvee4YcQoqIq4AXATPyxRozgBe3sg8rgwPKBiXpeZI+IWlc/nkn0pDT73KTLYAlwKOSdgROWAeH/aykTSW9gHRO6eK13N+lwJvyif0NSVchNgzRPLx1OXByrmFP0kn/1UjaUNLRkraMiOXAYqD/ku4HgTGSthxCvc2e/yzgDZK2kfQs4GN12z0INPx8VkTcC9wMnCppY0kvBt5L8/NgA5kG7CHp7ZJGSvoPYE/S1Z6t+BTpApETJI0BkLSXpIvq2v0QOJ10Tuzng+0097ZOA47Pi/4KHJj/zQ8ABvxogJXFAWWteAx4GXCLpMdJwTQb6L+67BTgJcA/SSfOL18Hx7wBuId0ufHXIuLqtdlZRMwB/pN0AcAC0nNaSPNL5T9MGh78B6m3eM4Auz8GmJ+HNz9Aerff39O8EJin9PmcNRmma/b8zyOdX5kPXM3qwX0q8F/5eJ9ssN+jSJdhP0A69/f5iJi+BnUBEBEPAYeSfgceIgXOoRGxqMXtbwYOyl/zJD0MTCEFX60fknpmF+fzlYM5CbgghzHA90jDzX2ki0N8uXkXkW9YaCWRNJ70rndURKyo8DibA48Cu0fEX6s6jpkNnXtQNmxIelMeNtsM+BpwB6knYmYFckDZcDKRNLT1AOkzRkc2m+nAzDrPQ3xmZlYk96DMzKxIDigzMytSV8ykvO2228b48eM7XYaZma0DM2fOXBQRPYO164qAGj9+PDNmzOh0GWZmtg5IGmjqsKd5iM/MzIrkgDIzsyJVHlCSRki6VdKV+eddJN0i6W5JF+c5sszMzJ6hHT2ojwJza34+HZgcEbuTpt1/bxtqMDOzLlNpQOXZr99IuikakkSaHLL/TppTgbdUWYOZmXWnqntQZ5BmOV6Vfx4DPFozCeh9tHaDMzMzG2YqCyhJhwILI2Jm7eIGTZvddnuSpBmSZvT19VVSo5mZlavKHtQBwJslzSfdg+cgUo9qq5pbSI+jyZ1SI2JKRPRGRG9Pz6Cf5zIzs/VMZQEVEZ+JiHERMR44Evh1RBwNXAcckZu9C/hpVTWYmVn36sRMEp8GLpL0JeBW4KwO1GAFOHXaXBYvXd7pMth41AhGbzIKgOMP3qPD1ZhZv7YEVERcD1yfH88D9m3Hca1sTzy1knmLHu90GWy3xUZPB5SZlcMzSZiZWZG6YrJYW//tt+uYjh17Xt+Sjh3bzJpzD8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MieaojsxqTp9/V6RIAz6puBg4oMx57ckUR8/FttemGbLPZhp0uw6wYDigb9pYuX8nS5Ss7XQYjR2zggDKr4YCyYa+TM6nX+vvDT3S6BLOi+CIJMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMytSZQElaWNJv5d0m6Q5kk7Jy8+V9FdJs/LXhKpqMDOz7lXlZebLgIMiYomkUcBNkn6Z150QEZdWeGwzM+tylQVURATQ//H8UfkrqjqemZmtXyo9ByVphKRZwEJgekTckld9WdLtkiZL2qjJtpMkzZA0o6+vr8oyzcysQJUGVESsjIgJwDhgX0kvBD4DPA94KbAN8Okm206JiN6I6O3p6amyTDMzK1BbruKLiEeB64HXRcSCSJYB5wD7tqMGMzPrLlVexdcjaav8eBPgtcCdksbmZQLeAsyuqgYzM+teVV7FNxaYKmkEKQgviYgrJf1aUg8gYBbwgQprMDOzLlXlVXy3A3s3WH5QVcc0M7P1h2eSMDOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIpUWUBJ2ljS7yXdJmmOpFPy8l0k3SLpbkkXS9qwqhrMzKx7VdmDWgYcFBF7AROA10naDzgdmBwRuwOPAO+tsAYzM+tSlQVUJEvyj6PyVwAHAZfm5VOBt1RVg5mZda9Kz0FJGiFpFrAQmA78BXg0IlbkJvcBO1ZZg5mZdadKAyoiVkbEBGAcsC/w/EbNGm0raZKkGZJm9PX1VVmmmZkVqC1X8UXEo8D1wH7AVpJG5lXjgAeabDMlInojorenp6cdZZqZWUGqvIqvR9JW+fEmwGuBucB1wBG52buAn1ZVg5mZda+RgzcZsrHAVEkjSEF4SURcKelPwEWSvgTcCpxVYQ1mZtalKguoiLgd2LvB8nmk81FmZmZNeSYJMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK1JlASVpJ0nXSZoraY6kj+blJ0u6X9Ks/PWGqmowM7PuNbLCfa8APhERf5S0BTBT0vS8bnJEfK3CY5uZWZerLKAiYgGwID9+TNJcYMeqjmdmZuuXtpyDkjQe2Bu4JS/6sKTbJZ0taet21GBmZt2l8oCStDlwGfCxiFgMnAnsBkwg9bC+3mS7SZJmSJrR19dXdZlmZlaYSgNK0ihSOF0QEZcDRMSDEbEyIlYB3wf2bbRtREyJiN6I6O3p6amyTDMzK1CVV/EJOAuYGxHfqFk+tqbZYcDsqmowM7PuVeVVfAcAxwB3SJqVl50EHCVpAhDAfOC4CmswM7MuVeVVfDcBarBqWlXHNDOz9YdnkjAzsyI5oMzMrEgOKDMzK1KVF0mY2RBNnn5Xp0vg+IP36HQJNsy1FFCSLgPOBn6ZP79kZuvYPxY/yaIlyzpdBjtutQnbj96402WYtdyDOhN4N/A/kn4MnBsRd1ZXltnws2pV8NSq6HQZrCygBjNoMaAi4hrgGklbAkcB0yXdS5oJ4vyIWF5hjWbDwn67jul0Cfxu3kOdLsHsaS1fJCFpDHAs8D7gVuC/gZcA0wfYzMzMbEhaPQd1OfA84DzgTflWGgAXS5pRVXFmZjZ8tXoO6gcR8YwZICRtFBHLIqK3grrMzGyYa3WI70sNlv12XRZiZmZWa8AelKRnke6Cu4mkvfnX3HqjgU0rrs3MzIaxwYb4/p10YcQ44Bs1yx8jzUxuZmZWiQEDKiKmAlMlHR4Rl7WpJjMzs0GH+N4REecD4yV9vH597Y0IzczM1qXBhvg2y983r7oQMzOzWoMN8X0vfz+lPeWYmZklLV1mLumrkkZLGiXpWkmLJL2j6uLMzGz4avVzUIdExGLgUOA+YA/ghMqqMjOzYa/VgBqVv78BuDAiHq6oHjMzM6D1qY5+LulOYCnwvyX1AE9WV5aZmQ13LfWgIuJEYH+gN99a43Fg4kDbSNpJ0nWS5kqaI+mjefk2kqZLujt/33ptn4SZma1/1uSW788nfR6qdpsfDtB+BfCJiPijpC2AmZKmk2amuDYiTpN0InAi8Ok1rNvMzNZzrd5u4zxgN2AWsDIvDgYIqHxLjgX58WOS5pLm9ZsIHJibTQWuxwFlZmZ1Wu1B9QJ7RsSQ7gUtaTywN3ALsH3//aQiYoGk7YayTzMzW7+1ehXfbOBZQzmApM2By4CP5UvVW91ukqQZkmb09fUN5dBmZtbFWu1BbQv8SdLvgWX9CyPizQNtJGkUKZwuiIjL8+IHJY3NvaexwMJG20bEFGAKQG9v75B6bmZm1r1aDaiT13THkgScBcytm1T2Z8C7gNPy95+u6b7NzGz911JARcQNkp4N7B4R10jaFBgxyGYHAMcAd0ialZedRAqmSyS9F/g78L+GVroN1enT5vLkilWdLoMnnlrR6RLMrGCtXsX3fmASsA3par4dge8Cr2m2TUTcxL/uwFuv6XZWvcVPLueevsc7XYaZ2YBavUjiQ6Qe0WKAiLgb8NV3ZmZWmVbPQS2LiKfSaSXIH9b1hQvrgf12HdPpEszMGmq1B3WDpJOATSQdDPwY+Hl1ZZmZ2XDXakCdCPQBdwDHAdOA/6qqKDMzs1av4lsl6SfATyLCn5o1M7PKDdiDUnKypEXAncCfJfVJ+lx7yjMzs+FqsCG+j5Gu3ntpRIyJiG2AlwEHSDq+8urMzGzYGiyg3gkcFRF/7V8QEfOAd+R1ZmZmlRgsoEZFxKL6hfk81KgG7c3MzNaJwQLqqSGuMzMzWyuDXcW3l6RGt8gQsHEF9ZiZmQGDBFREDDYhrJmZWSVa/aCumZlZWzmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyJVFlCSzpa0UNLsmmUnS7pf0qz89Yaqjm9mZt2tyh7UucDrGiyfHBET8te0Co9vZmZdrLKAiogbgYer2r+Zma3fOnEO6sOSbs9DgFs3ayRpkqQZkmb09fW1sz4zMytAuwPqTGA3YAKwAPh6s4YRMSUieiOit6enp131mZlZIdoaUBHxYESsjIhVwPeBfdt5fDMz6x5tDShJY2t+PAyY3aytmZkNb4PdUXfIJF0IHAhsK+k+4PPAgZImAAHMB46r6vhmZtbdKguoiDiqweKzqjqemZmtXzyThJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRWpsoCSdLakhZJm1yzbRtJ0SXfn71tXdXwzM+tuVfagzgVeV7fsRODaiNgduDb/bGZmtprKAioibgQerls8EZiaH08F3lLV8c3MrLuNbPPxto+IBQARsUDSds0aSpoETALYeeed21SemfWbPP2uTpcAwPEH79HpEqxD2h1QLYuIKcAUgN7e3uhwOWbDxqNPLGf5ylUDtlmybAUv2GHLNlVkw1W7A+pBSWNz72kssLDNxzezQSx+cjmLn1w+cJulyx1QVrl2B9TPgHcBp+XvP23z8c1sAPvtOqaldvc98kSlQ2+lDC9aZ1V5mfmFwG+B50q6T9J7ScF0sKS7gYPzz2ZmZquprAcVEUc1WfWaqo5pZmbrD88kYWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkYq95fv6yDdhMzNrnXtQw9CcBYs7XYKZ2aAcUMPQsuWrOl2CmdmgPMTXIccfvEfHjn3fI08wbutNO3Z8M7NWuAdlZmZFckCZmVmROjLEJ2k+8BiwElgREb2dqMPMzMrVyXNQr46IRR08vpmZFcxDfGZmVqROBVQAV0uaKWlSh2owM7OCdWqI74CIeEDSdsB0SXdGxI21DXJwTQLYeeedO1GjmZl1UEd6UBHxQP6+ELgC2LdBmykR0RsRvT09Pe0u0czMOqztASVpM0lb9D8GDgFmt7sOMzMrWyeG+LYHrpDUf/wfRcRVHajDzMwK1vaAioh5wF7tPq6ZmXUXX2ZuZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmROnXLdzOzlkyeflenSwDg+IP36HQJw457UGZmViQHlJmZFclDfGZWnFKG00oZXhyuhkVA+ZfMzKz7eIjPzMyK5IAyM7MidWSIT9LrgP8GRgA/iIjT2nXsUsa2zcxsYG3vQUkaAXwbeD2wJ3CUpD3bXYeZmZWtEz2ofYF7ImIegKSLgInAnzpQi5lZ8Uq80Ksdo1GdCKgdgXtrfr4PeFkH6hi2th+9MduN3qjTZVgXW75yVadLaLsSQ2J914mAUoNlsVojaRIwKf+4RNKf18XBPz60zbYFFq2L43eAa++Mbq295bo/XXEhQ9Ctrzl0Ye01f0uHUvuzW2nUiYC6D9ip5udxwAP1jSJiCjClXUUNRNKMiOjtdB1D4do7o1tr79a6wbV3SpW1d+Iy8z8Au0vaRdKGwJHAzzpQh5mZFaztPaiIWCHpw8CvSJeZnx0Rc9pdh5mZla0jn4OKiGnAtE4ce4iKGGocItfeGd1ae7fWDa69UyqrXRGrXZ9gZmbWcZ7qyMzMiuSAIs1uIelWSVc2WPcBSXdImiXppv5ZLySNl7Q0L58l6bvdUHde92JJv5U0J7fZuL2VD/k1P7rm9Z4laZWkCV1S+yhJU/O6uZI+0+66cx1DqX1DSefkdbdJOrDthTNw7TVtjpAUknprln1G0j2S/izp39tT7TNqWuO6JY2RdJ2kJZK+1b5qV6trKLUfLGlm/n2ZKemgIRcQEcP+i3RJ/4+AKxusG13z+M3AVfnxeGB2F9Y9Ergd2Cv/PAYY0Q2117V5ETCvi173twMX5cebAvOB8V1S+4eAc/Lj7YCZwAYl1Z7XbwHcCPwO6M3L9gRuAzYCdgH+0u7f9yHWvRnwCuADwLfa/VqvZe17Azvkxy8E7h/q8Yd9D0rSOOCNwA8arY+IxTU/bkaDDxV3wlrUfQhwe0Tclts9FBErq6y13jp6zY8CLlz31Q1sLWoPYDNJI4FNgKeAxbTRWtS+J3BtbrMQeBRo62d2Bqs9+yLwVeDJmmUTSW8MlkXEX4F7SNOttcVQ646IxyPiJp75XNpqLWq/NSL6P9s6B9hY0pCmrhn2AQWcAXwKaDp3i6QPSfoL6R/iIzWrdsnd3xskvbLiOusNte49gJD0K0l/lPSp6ktdzdq85v3+gw4EFEOv/VLgcWAB8HfgaxHxcMW11htq7bcBEyWNlLQLsA/P/LB9OwxYu6S9gZ0ion4oqtHUajtWUmFjQ627BOui9sOBWyNi2VAKGNYBJelQYGFEzByoXUR8OyJ2I83u8l958QJg54jYm9wNljS60oKztax7JGno4Oj8/TBJr6my3lprWXv/Pl4GPBERs6urdHVrWfu+wEpgB9JQ0yck7VplvbXWsvazSX/YZ5D+aN0MrKiw3GcYrHZJGwCTgU80Wt1gWVtGQday7o5aF7VLegFwOnDckAvp1NhmCV/AqaT/ePOBfwBPAOcP0H4D4J9N1l1PHoMtuW7SzB3n1qz7LHBCN73m+T/GSd30+0K6xcwxNevOBt7WDbU3WHczsGcptQNbkuaCm5+/niRNn9YLfAb4TE3bXwH7l153TZtj6cA5qLWtnTSF3V3AAWtVR7ufeKlfwIE0PnG8e83jNwEz8uMe8slWYFfgfmCbLqh7a+CPpBP1I4FrgDd2w2uef94g/8fZtct+Xz4NnEN6R78Z6fYyL+6S2jcFNsuPDwZuLO11r2tzfc0fyhfwzIsk5tGZi4LWqO6aZR0JqLV8zbfKr/nha3vsjswkUTpJXyD95/wZ8GFJrwWWA48A78rNXgV8QdIK0tDNB6L95xSeoZW6I+IRSd8gzYkYwLSI+EWnau7X4msO6XW/L/L9xErQYu3fJgXUbFJInRMRt3ei3lot1r4d8CtJq0hvxI7pSLF16mpvKCLmSLqE9IZgBfChaPNFQfVaqTu3mw+MBjaU9BbgkIjo6H3zWqz9w8BzgM9K+mxedkikC2zW7Hg58czMzIoyrC+SMDOzcjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaCsUpKeJekiSX+R9CdJ0yTtIemvkp5b1/aMoU69JOlkSZ8cpM2B/bMy58cvH8qxWqjlXElHVNU+b3NzC23mS9q2wfKWnrukY+tn0pZ0fe1M4TXL95N0i9JM6HMlndzC/jfJ04SNkPTcPPP1bZL2z+tHSrpG0qY121wkaffB9m3rBweUVUaSgCuA6yNit4jYEzgJ2B64iDSrRX/bDYAjgIvbVN6BQCUB1Q4RsTa1H8i6f+5TgUkRMYE0g/UlLWzzHuDy/Lmk44ATSb8D/W80PgicFxFP1GxzJml+OBsGHFBWpVcDyyPi6XtlRcSsiPgNaaLXI2vavgqYHxF/q92BpM0lXZsntr1D0sSadf9H6R4/1wDPrVn+9Lt8SdvmDzzW7nM86TYGx+d3/K+sW3+y0r2brs69kLdK+mo+/lWSRuV2n5P0B0mzJU3JgUzdvvbJvYSZShP0jm3yWr1K0s2S5tX2piSdkI9xu6RTapYvyd83kPQdpXt7XZl7qLW9sf+see2eN9hzXwvbkeanJCJWtviB0qOBn+bHy0mzvG8KLJe0FWk2ix/WbfMb4LVKs8Lbes4BZVV6IeneQavJsyiskrRXXnQkjWcnfxI4LCJeQgq8ryvZJ2+zN/BW4KWtFhUR84HvApMjYkIOzHq7kW41MBE4H7guIl4ELM3LIU1B89KIeCHpj+uhtTvIQfZN4IiI2Ic0/96Xm5Q1ljR576HAaXn7Q4DdSRPNTgD2kfSquu3eSro32YuA9wH7161flF+7M4FPtvjch2Iy8GdJV0g6ToPcBFPShqTpqubnRd8mTbr8XeArwOeAL0fdTAIRsYp0y4y9sPWeA8o66ULgyPxueCLw4wZtBHxF0u2keQN3JA0RvhK4IiKeiHQfowGnjRmCX0bEcuAOYARwVV5+BykQAF6dz7vcARxEmvet1nNJIT1d0izS7ODjmhzvJxGxKvc8ts/LDslft5LmT3weKbBqvQL4cd72H8B1desvz99n1tTdqmbTzKy2PCK+QJqc9WrSzRmvqm9TZ1vSfaX6t/97RBwYEfuTJibdAbhT0nmSLpa0R822C/N6W8+5m2xVmkM6p9DMhaQ/aDeQbqLYaK6uo0kT8+4TEcvzcF3/u/Nmf0BX8K83X0O9nf0ySO/YJS2veSe/ChiZewjfIU2QeW++KKD+WALm5D+6LR2vZrv+76dGxPcG2K7R7SQa7Xcla/7//SE07j+2AAACRklEQVTS5MK1tiHNYr2aiPgLcKak7wN9ksZExENN9r2U5v82XyaF+UeAC0izZX+e9LtA3m5pi8/Buph7UFalXwMbSXp//wJJL5X0b/D0H7SHSENazW4+uCXpvjTLJb0aeHZefiPpXlabSNqCdL6i33zSTfWgeUA+Rrpd9VD1/3FdJGnzJsf5M9BTc1XaKKV75LTqV8B78v6RtKOk7era3AQcns9FbU+6AGIwz3jukg6TdGqDdn8ADpD0rNyulzQr+L31DSW9seYc3O6kQHy0vl2/iHgEGFE/FJh/N+6PiLtJ56NW5X1tWtNsD9KbH1vPOaCsMrnXcRhwsNJl5nOAk0n3jel3IWno6oomu7kA6JU0g/QO+s687z+SrvibBVxGOnne72vAB5UuxV7tMuvs56SAG9KFAhHxKPB90pDfT0h/zOvbPEUKrtMl3ZZrbfnquYi4GvgR8Ns8jHgpq4fqZaTbj8wGvgfcAvxzkF3XP/fdaHD7+Yh4EPgoMC0PUZ4BHJXPAyHpB/rXJefHkM5BzQLOA46OiJWSeiU1u2X41aQhSvL+ROo5fTEvmkJ683IZ6d+UHMJLI2LBIM/R1gOezdysy0naPCKWSBoD/J50k7h/rMH25wPHR0RfZUU2Pu7ewMcjouXbd0g6HlgcEWdVV5mVwuegzLrflfmy7A2BL65JOAFExDuqKWvQ494q6TpJI9bgHk2PknpoNgy4B2VmZkXyOSgzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMivT/AYxmseHg1i5WAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf4dc17198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_sampling_dist(sampling_dist_cv)\n",
    "plt.title('Sampling distribution of CV %')\n",
    "plt.xlabel('CV adult male height, U.S. (%)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weighted resampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another nice thing about resampling is that we can extend it to handle the case where the data are weighted.  In fact, the BRFSS deliberately oversamples some groups, so each respondent has a weight that indicates how many people in the population they represent.\n",
    "\n",
    "The variable `_LLCPWT` contains these weight, which we can normalize so they add up to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_sampling_weights(df):\n",
    "    p = df._LLCPWT\n",
    "    p /= p.sum()\n",
    "    return p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "sampling_weights = compute_sampling_weights(males);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pass these weights to `np.random.choice`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resample_weighted(data, p):\n",
    "    size = len(df)\n",
    "    return np.random.choice(data, size, \n",
    "                            replace=True, p=p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 24s, sys: 3.6 s, total: 1min 28s\n",
      "Wall time: 1min 28s\n"
     ]
    }
   ],
   "source": [
    "%time sampling_dist_mean_weighted = \\\n",
    "    [np.nanmean(resample_weighted(data, sampling_weights)) \\\n",
    "     for i in range(1000)];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take the mean of the sampling distribution, we get an estimate of the average male height, taking account of the sampling weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "177.54278962889853"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xbar_weighted = np.mean(sampling_dist_mean_weighted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can compare to the unweighted version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178.06027748143575"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xbar = np.mean(sampling_dist_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this dataset, the weighted and unweighted estimates are not very different, which probably means there are no substantial differences in height between the groups that are oversampled or undersampled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Are first babies more likely to be late?\n",
    "\n",
    "When my wife and I were expecting our first baby, we heard that first babies are more likely to be late.  We also hear that first babies are more likely to be early.  Neither claim was supported by evidence.\n",
    "\n",
    "Fortunately, I am a data scientist!  Also fortunately, the CDC runs the National Survey of Family Growth (NSFG), which \"gathers information on family life, marriage and divorce, pregnancy, infertility, use of contraception, and men’s and women’s health.\"\n",
    "\n",
    "I got the data from their web page, https://www.cdc.gov/nchs/nsfg/index.htm, and wrote some code to get it into a Pandas Dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(13593, 244)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nsfg\n",
    "\n",
    "df = nsfg.ReadFemPreg()\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file contains 13,593 rows, one for each pregnancy reported by a one of the survey respondents, and 244, one for each variable.\n",
    "\n",
    "Here are the first few lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>caseid</th>\n",
       "      <th>pregordr</th>\n",
       "      <th>howpreg_n</th>\n",
       "      <th>howpreg_p</th>\n",
       "      <th>moscurrp</th>\n",
       "      <th>nowprgdk</th>\n",
       "      <th>pregend1</th>\n",
       "      <th>pregend2</th>\n",
       "      <th>nbrnaliv</th>\n",
       "      <th>multbrth</th>\n",
       "      <th>...</th>\n",
       "      <th>laborfor_i</th>\n",
       "      <th>religion_i</th>\n",
       "      <th>metro_i</th>\n",
       "      <th>basewgt</th>\n",
       "      <th>adj_mod_basewgt</th>\n",
       "      <th>finalwgt</th>\n",
       "      <th>secu_p</th>\n",
       "      <th>sest</th>\n",
       "      <th>cmintvw</th>\n",
       "      <th>totalwgt_lb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3410.389399</td>\n",
       "      <td>3869.349602</td>\n",
       "      <td>6448.271112</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.8125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3410.389399</td>\n",
       "      <td>3869.349602</td>\n",
       "      <td>6448.271112</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.8750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.1250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.1875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 244 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   caseid  pregordr  howpreg_n  howpreg_p  moscurrp  nowprgdk  pregend1  \\\n",
       "0       1         1        NaN        NaN       NaN       NaN       6.0   \n",
       "1       1         2        NaN        NaN       NaN       NaN       6.0   \n",
       "2       2         1        NaN        NaN       NaN       NaN       5.0   \n",
       "3       2         2        NaN        NaN       NaN       NaN       6.0   \n",
       "4       2         3        NaN        NaN       NaN       NaN       6.0   \n",
       "\n",
       "   pregend2  nbrnaliv  multbrth     ...       laborfor_i  religion_i  metro_i  \\\n",
       "0       NaN       1.0       NaN     ...                0           0        0   \n",
       "1       NaN       1.0       NaN     ...                0           0        0   \n",
       "2       NaN       3.0       5.0     ...                0           0        0   \n",
       "3       NaN       1.0       NaN     ...                0           0        0   \n",
       "4       NaN       1.0       NaN     ...                0           0        0   \n",
       "\n",
       "       basewgt  adj_mod_basewgt      finalwgt  secu_p  sest  cmintvw  \\\n",
       "0  3410.389399      3869.349602   6448.271112       2     9      NaN   \n",
       "1  3410.389399      3869.349602   6448.271112       2     9      NaN   \n",
       "2  7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "3  7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "4  7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "\n",
       "   totalwgt_lb  \n",
       "0       8.8125  \n",
       "1       7.8750  \n",
       "2       9.1250  \n",
       "3       7.0000  \n",
       "4       6.1875  \n",
       "\n",
       "[5 rows x 244 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables we need are `outcome`, which indicates whether the pregnancy ended in a live birth, `birthord`, which indicates birth order, and `prglength`, which is pregnancy length in weeks.\n",
    "\n",
    "From all live births, we can select first babies (`birthord==1`) and others:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4413, 4735)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "live = df[df.outcome == 1]\n",
    "firsts = live[live.birthord == 1]\n",
    "others = live[live.birthord != 1]\n",
    "\n",
    "len(firsts), len(others)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can get the list of pregnancy lengths for the two groups and compute their means:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(38.60095173351461, 38.52291446673706)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "group1 = firsts.prglngth\n",
    "group2 = others.prglngth\n",
    "\n",
    "np.mean(group1), np.mean(group2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference in means is small, about 0.78 weeks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07803726677754952"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff = group1.mean() - group2.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which is 13 about hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.11026081862832"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff * 7 * 24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So first babies are born about 13 hours later than other babies, on average. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis testing\n",
    "\n",
    "The size of this \"apparent effect\" is small, and we can't tell whether it is real or the result of random sampling.  After all, we did not survey the entire population; we only surveyed a random sample.\n",
    "\n",
    "There are two ways the sample might deviate from the population:\n",
    "\n",
    "*  Systematic errors: The pregnancies included in the survey might be different from other pregnancies in a way that biases the results.\n",
    "\n",
    "*  Sampling errors: The pregnancies lengths in one groups might be a little higher, or lower, than in the other group because of random variability.\n",
    "\n",
    "We can never rule out the possibility of systematic errors, but usually we can test whether an apparent effect could be explained by random sampling.\n",
    "\n",
    "Here's how:\n",
    "\n",
    "1.  Choose a \"test statistics\" that measures the size of the effect; in this case, the test statistic we started with is the difference in mean pregnancy length.\n",
    "\n",
    "2.  Use the data to make a model of the population under the assumption that there is actually no difference between the groups.  This assumption is called the \"null hypothesis\".\n",
    "\n",
    "3.  Use the model to simulate the data collection process.\n",
    "\n",
    "4.  Use the simulated data to compute the test statistic.\n",
    "\n",
    "4.  Repeat steps 2-4 and collect the results.\n",
    "\n",
    "5.  See how often the simulated test statistic exceeds the observed difference.\n",
    "\n",
    "The following function computes the test statistic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_stat(data):\n",
    "    group1, group2 = data\n",
    "    return group1.mean() - group2.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07803726677754952"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = group1, group2\n",
    "actual = test_stat(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need a model of the population under the assumption that these is actually no difference between the groups.\n",
    "\n",
    "Well, if there's no difference, we can put the two groups together and shuffle them, then divide them at random into two groups with the same sizes.\n",
    "\n",
    "That's what this function does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_model(data):\n",
    "    group1, group2 = data\n",
    "    pool = np.hstack((group1, group2))\n",
    "    np.random.shuffle(pool)\n",
    "    n = len(group1)\n",
    "    return np.split(pool, [n])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([42, 39, 38, ..., 39, 39, 39]), array([39, 38, 39, ..., 39, 39, 39])]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_model(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a list of two arrays, which we can pass to `test_stat`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.01127321097717271"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_stat(run_model(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's the result of one simulated experiment.\n",
    "\n",
    "We can run the experiment 1000 times and collect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0008537509532526997"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_stat_dist = np.array([test_stat(run_model(data)) \n",
    "                           for i in range(1000)])\n",
    "np.mean(test_stat_dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the \"sampling distribution of the test statistic under the null hypothesis\".\n",
    "\n",
    "The mean of this distribution is close to zero, which is not suprising because it is based on the assumption that there is actually no difference between the groups.\n",
    "\n",
    "Here's a function to plot the distribution of test stats:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_test_stats(test_stats):\n",
    "    plt.xlabel('Difference in mean (weeks)')\n",
    "    plt.title('Distribution of test stat under null hypothesis')\n",
    "    return plot_hist(test_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's what it looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAH01JREFUeJzt3XmcHFW99/HPlyQQICBLRhAQAkJQRGUZceHqRdmR5T5XroKAgHjj7hXcUAED4oP4KIjeRzEqu7IIioAoBLkBVBATdmQ1gISAmYBAQAgk/O4f5zRpmu6ZnszU9EnP9/169Wt6ajn1O1XV9es6VX1KEYGZmVlplut0AGZmZs04QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJqgCSTpZ05DCVtb6kpySNyf/PkPTh4Sg7l/cbSQcOV3mDWO6xkuZLemSkl90tJB0k6fedjgNA0lRJZ+X3kySFpLEtpr1f0g4jG2HTOF6MeRjLfMnn1V7KCapi+cP1jKQFkh6X9EdJH5X04rqPiI9GxNfaLKvfD2pE/C0iJkTE4mGI/WUfyIjYNSJOH2rZg4zj1cBngc0iYu0m47eTNGeYlrXUCX2w81ZxwLPhMZz7VH+G8/PajZygRsYeEbEKsAHwDeCLwE+GeyGtvoF2gQ2ARyNiXqcDGa38Dd86IiL8qvAF3A/s0DBsG+AFYPP8/2nAsfn9ROAS4HHgMeAa0heJM/M8zwBPAV8AJgEBHAL8Dbi6btjYXN4M4DjgeuAJ4FfAGnncdsCcZvECuwDPAc/n5d1cV96H8/vlgCOAB4B5wBnAK/K4WhwH5tjmA1/pZz29Is/fl8s7Ipe/Q67zCzmO0xrmW7lh/FPAOnnew4G/Ao8C59XVezxwVh7+OPBnYC3g68Bi4Nlczn83iXNQ8wInAQ8CTwKzgHfk4U3Xb5PlBbBx3f+nsWRf2Q6YQzq7nAc8DBxcN+2awEV52dcDXwN+Xzf+tcB00n52F/C+huX8ALgUeJqGfbhuX/ga8AdgAXA5MHGgfSu/nwqc1bCvjO3nM/Q54BbSPnwuMD6Pu430BbA27TjSvrZFXblTgLl5/Xy2btoVgO/kcXPz+xVovU9NJe1HZ+T63g701pW3DnABaR++D/h0w2d+Zt4WfwdOaFZ34CBgdi7/PmC/Th/DOvnqeADd/qJJgsrD/wZ8LL8/jSUHneOAk/MHbRzwDkDNyqrbuc/IH6oVm+zwM4CHgM3zNBfUHRjaPojUjZ/BkgT1IeBeYCNgAvAL4MyG2H6U43oTsBB4XYv1dAYpea6S570bOKRVnA3zNqvHZ4DrgPVIB50fAmfncR8BLgZWAsYAWwOrNtavxbIGNS+wPylRjCUlkkdYcnB92fptsryBEtQi4Ji8r+wG/BNYPY8/h3RAXTlv/4fICSoPexA4OMe2FenA/vq65TwBbEtK9uObxDaD9AVgct7GM4BvDHbfor0EdT0pAawB3AF8NI/7AnBu3bR7Abc2lHt2ru8bSMmjFsMxeR95JdAD/BH4Wj/xTyV9Adktb/vjgOvyuOVIX0COApYnfSZmAzvn8dcCB+T3E4C3NtY9x/gksGke96ra9hitLzfxdc5c0oet0fOkHXODiHg+Iq6JvLf2Y2pEPB0Rz7QYf2ZE3BYRTwNHAu8bpiab/UjfBGdHxFPAl4B9Gpoaj46IZyLiZuBmUqJ6iRzL+4EvRcSCiLgf+DZwwBBi+wjpjG1ORCwkHVz2zrE9T0oaG0fE4oiYFRFPtlnuoOaNiLMi4tGIWBQR3yYly02HUK9m8RyT95VLSd/2N83r9L3AUXnfuA2ov3a4O3B/RJyaY7uB9OVl77ppfhURf4iIFyLi2RbLPzUi7s773nmkM5cqfDci5kbEY6QvCLXlnAXsJmnV/P8BpNaGekfndXArcCqwbx6+H2ndzYuIPuBoBt7nfh8Rl0a6ZnQmS/bnNwM9EXFMRDwXEbNJX872yeOfBzaWNDEinoqI61qU/wKwuaQVI+LhiLh9gHi6mhNU56xLalpp9P9IZyWXS5ot6fA2ynpwEOMfIH3bnthWlP1bJ5dXX/ZYUpNXTf1dd/8kfXtsNJH0rbOxrHWHENsGwC/zjSmPk751L86xnQlcBpwjaa6kb0oa12a5g5pX0mcl3SHpiRzHKxiedV/zaEQsqvu/to57SNuicdvXbAC8pbZ+cmz7AfU3oQy0X0F723c4NF1ORMwlNTG+V9JqwK7ATxvmbVwH6+T3zfbfdehfYxzj85eeDYB1Gtbnl1nyWTiEdKZ5p6Q/S9q9seD8BfL9wEeBhyX9WtJrB4inqzlBdYCkN5MOvi+75TefQXw2IjYC9gAOk7R9bXSLIgc6w3p13fv1Sd/m5pOuLaxUF9cY0oGt3XLnkj6Y9WUvIrWxD8b8HFNjWQ+1OX+zOB8Edo2I1epe4yPioXy2cXREbAa8nXQ28cF+ylqyoEHMK+kdpBti3kdqdluN1GymdpaV/ZO6bcRLE0h/+kjbonHb1zwIXNWwfiZExMfqphnKow4G2reG0+mkptT/AK6NiMb9pnEdzM3vm+2/tXGDrfuDwH0N63OViNgNICLuiYh9Sc2JxwPnS1q5sZCIuCwidiS1otxJOgsbtZygRpCkVfM3p3NI7e+3Nplmd0kbSxKpPXpxfkE68G+0FIveX9JmklYitbufn5so7iZ9A3xPPgs4gtQEVfN3YFL9LfENzgYOlbShpAnA/yVdD1jUYvqmciznAV+XtIqkDYDDSM037fg7sKakV9QNOzmXtwGApB5Je+X375L0hnzQfJKUHNtax4OcdxVSkugDxko6Cli1bvxA6xfgJuADksZI2gX4136mfVFep78ApkpaSdJmpBtWai4BJks6QNK4/HqzpNe1U34bBtq3htOFpGto/0W6ltnoyLwOXk+65nZuHn42cETeNyaSrh/V9rlm+1R/rgeelPRFSSvm7bV5/jKKpP0l9UTEC6Sba2DJfkOeZi1Je+bEtZDUXDuqbz93ghoZF0taQPqW9RXgBNIHpZlNgCtIO+e1wPcjYkYedxzpA/W4pM8NYvlnki56P0K6C+3TABHxBPBx4Meks5WnSXeF1fw8/31U0g1Nyj0ll3016Y6jZ4FPDSKuep/Ky59NOrP8WS5/QBFxJ+lgMzuvm3VId89dRGoqXUC6GP6WPMvawPmkBHMHcBVLDkwnka5V/UPSd5ssbjDzXgb8hnSwfoC0fuqbmwZav5AOunuQDmr7kQ7G7fokqSnsEdL2P7U2IiIWADuRrpHMzdMczzAlkTb2rWGTr39dAGxISsqNriI1m/8O+FZEXJ6HH0u6s+4W4Fbghjys1T7VXwyLSdtpC9JnYT6p7rUEtwtwu6SnSPvJPk2u6y1HupFmLqn5/19J63DUqt0dZma2zMpnp5MjYv+6YZNIyWLcYM/qrQzd+sNOMxslJK1BuglhKHd9WoHcxGdmyyxJ/0lqNv1NRFzd6XhseLmJz8zMiuQzKDMzK1JR16AmTpwYkyZN6nQYZmZWoVmzZs2PiAF/F1dZgpK0KUt+bwDp9yFHRcR3Ws0zadIkZs6cWVVIZmZWAEkPDDxVhQkqIu4i95eVf9T4EPDLqpZnZmbdZaSuQW0P/DUi2sqaZmZmI5Wg9iH9KvtlJE2RNFPSzL6+vhEKx8zMSld5gpK0PLAnS7p1eYmImBYRvRHR29NTVV+SZma2rBmJM6hdgRsiYrA9XJuZ2Sg2EglqX1o075mZmbVSaYLKj3fYkeY9DJuZmbVU6Q91I+KfpMdjm5mZDUpRPUmYjYQTp9/d6RBe5tAdJ3c6BLPiuC8+MzMrks+gzApQ0lmdz+asFE5QNqp18mBcUlIyK5Gb+MzMrEg+gzLrkJKa0nw2ZyXyGZSZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrUqUJStJqks6XdKekOyS9rcrlmZlZ96j6ibonAb+NiL0lLQ+sVPHyzMysS1SWoCStCrwTOAggIp4DnqtqeWZm1l2qbOLbCOgDTpV0o6QfS1q5cSJJUyTNlDSzr6+vwnDMzGxZUmWCGgtsBfwgIrYEngYOb5woIqZFRG9E9Pb09FQYjpmZLUuqTFBzgDkR8af8//mkhGVmZjagyhJURDwCPChp0zxoe+AvVS3PzMy6S9V38X0K+Gm+g282cHDFyzMzsy5RaYKKiJuA3iqXYWZm3ck9SZiZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIo0tsrCJd0PLAAWA4siorfK5ZmZWfeoNEFl74qI+SOwHDMz6yIjkaDMbBly4vS7Ox0CAIfuOLnTIViHVX0NKoDLJc2SNKXZBJKmSJopaWZfX1/F4ZiZ2bKi6gS1bURsBewKfELSOxsniIhpEdEbEb09PT0Vh2NmZsuKSpv4ImJu/jtP0i+BbYCrq1ymmQ1eKc1ppTQvWhkqO4OStLKkVWrvgZ2A26panpmZdZcqz6DWAn4pqbacn0XEbytcnpmZdZHKElREzAbeVFX5ZmbW3dyThJmZFckJyszMiuQf6tqI8N1ZZjZYPoMyM7MiOUGZmVmR3MRnI66UH4WaWdl8BmVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK1FaCknSBpPdIGnRCkzRG0o2SLhl8eGZmNlq1m3B+AHwAuEfSNyS9dhDL+C/gjkFHZmZmo1pbCSoiroiI/YCtgPuB6ZL+KOlgSeNazSdpPeA9wI+HI1gzMxs92m6yk7QmcBDwYeBG4CRSwprez2zfAb4AvNBPuVMkzZQ0s6+vr91wzMysy7V7DeoXwDXASsAeEbFnRJwbEZ8CJrSYZ3dgXkTM6q/siJgWEb0R0dvT0zPI8M3MrFuNbXO6H0fEpfUDJK0QEQsjorfFPNsCe0raDRgPrCrprIjYfwjxmpnZKNFuE9+xTYZd298MEfGliFgvIiYB+wBXOjmZmVm7+j2DkrQ2sC6woqQtAeVRq5Ka+8zMzCoxUBPfzqQbI9YDTqgbvgD4crsLiYgZwIzBhWZmZqNZvwkqIk4HTpf03oi4YIRiMjMzG7CJb/+IOAuYJOmwxvERcUKT2czMzIZsoCa+lfPfpreSm5mZVWWgJr4f5r9Hj0w4ZmZmSbs/1P2mpFUljZP0O0nzJfmWcTMzq0y7v4PaKSKeBHYH5gCTgc9XFpWZmY167SaoWoewuwFnR8RjFcVjZmYGtN/V0cWS7gSeAT4uqQd4trqwzMxstGv3cRuHA28DeiPieeBpYK8qAzMzs9Gt3TMogNeRfg9VP88ZwxyPmZkZ0GaCknQm8BrgJmBxHhw4QZmZWUXaPYPqBTaLiKgyGDMzs5p27+K7DVi7ykDMzMzqtXsGNRH4i6TrgYW1gRGxZyVRmZnZqNdugppaZRBmZmaN2kpQEXGVpA2ATSLiCkkrAWOqDc3MzEazdvvi+0/gfOCHedC6wIVVBWVmZtbuTRKfALYFngSIiHuAV1YVlJmZWbsJamFEPFf7J/9Y17ecm5lZZdpNUFdJ+jKwoqQdgZ8DF1cXlpmZjXbtJqjDgT7gVuAjwKXAEVUFZWZm1u5dfC9IuhC4MCL62plH0njgamCFvJzzI+KrSx2pmZmNKv2eQSmZKmk+cCdwl6Q+SUe1UfZC4N0R8SZgC2AXSW8deshmZjYaDNTE9xnS3Xtvjog1I2IN4C3AtpIO7W/GSJ7K/47LL99YYWZmbRkoQX0Q2Dci7qsNiIjZwP55XL8kjZF0EzAPmB4RfxpKsGZmNnoMlKDGRcT8xoH5OtS4JtM3Trc4IrYA1gO2kbR54zSSpkiaKWlmX19bl7fMzGwUGChBPbeU414iIh4HZgC7NBk3LSJ6I6K3p6en3SLNzKzLDXQX35skPdlkuIDx/c0oqQd4PiIel7QisANw/NKFaWZmo02/CSoihtIh7KuA0yWNIZ2pnRcRlwyhPDMzG0XafdzGoEXELcCWVZVvZmbdrd2eJMzMzEaUE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMilRZX3xmZkNx4vS7Ox3Ciw7dcXKnQxiVfAZlZmZFcoIyM7MiuYnPzIpRUlNaSU2Mo5XPoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIlX2OyhJrwbOANYGXgCmRcRJVS3PmvNvOcxsWVXlD3UXAZ+NiBskrQLMkjQ9Iv5S4TLNzKxLVNbEFxEPR8QN+f0C4A5g3aqWZ2Zm3WVEujqSNAnYEvhTk3FTgCkA66+//kiEM2qV1I2MmdlAKr9JQtIE4ALgMxHxZOP4iJgWEb0R0dvT01N1OGZmtoyoNEFJGkdKTj+NiF9UuSwzM+sulSUoSQJ+AtwRESdUtRwzM+tOVZ5BbQscALxb0k35tVuFyzMzsy5S2U0SEfF7QFWVb2Zm3c09SZiZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIpUWYKSdIqkeZJuq2oZZmbWvao8gzoN2KXC8s3MrItVlqAi4mrgsarKNzOz7tbxa1CSpkiaKWlmX19fp8MxM7NCdDxBRcS0iOiNiN6enp5Oh2NmZoUY2+kAutGJ0+/udAhmZsu8jp9BmZmZNVPlbeZnA9cCm0qaI+mQqpZlZmbdp7ImvojYt6qylyWH7ji50yGYmS2T3MRnZmZF8k0SZmYDKOXGp9HWIuMzKDMzK5ITlJmZFclNfGZmTZTSnFZK82In+AzKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhd1Zv5aO7118ys2/gMyszMiuQEZWZmReqqJr56pTxszMxsuJR0GWMkjrGVnkFJ2kXSXZLulXR4lcsyM7PuUlmCkjQG+P/ArsBmwL6SNqtqeWZm1l2qbOLbBrg3ImYDSDoH2Av4S4XLNDPrKiVdrhjpJsYqE9S6wIN1/88B3tI4kaQpwJT871OS7hqOhR829CImAvOHXkwRuqku0F31cV3K5LoMYIjH2A3amajKBKUmw+JlAyKmAdMqjGOpSJoZEb2djmM4dFNdoLvq47qUyXUpQ5U3ScwBXl33/3rA3AqXZ2ZmXaTKBPVnYBNJG0paHtgHuKjC5ZmZWReprIkvIhZJ+iRwGTAGOCUibq9qeRUortlxCLqpLtBd9XFdyuS6FEARL7ssZGZm1nHu6sjMzIrkBGVmZkUa1QlK0hqSpku6J/9dvcV0v5X0uKRLGoafJuk+STfl1xYjE3nTGIdalw0l/SnPf26+saUjBlGXA/M090g6sG74jNzFVm27vHLkon8xhn67+ZK0Ql7P9+b1Pqlu3Jfy8Lsk7TyScbeytPWRNEnSM3Xb4uSRjr1RG3V5p6QbJC2StHfDuKb7XKcMsS6L67ZLmTewRcSofQHfBA7P7w8Hjm8x3fbAHsAlDcNPA/budD2GqS7nAfvk9ycDHyu5LsAawOz8d/X8fvU8bgbQ28H4xwB/BTYClgduBjZrmObjwMn5/T7Aufn9Znn6FYANczljOrxvDaU+k4DbOhn/UtRlEvBG4Iz6z3d/+9yyVpc87qlOb4+BXqP6DIrU9dLp+f3pwL81mygifgcsGKmgltJS10WSgHcD5w80/whppy47A9Mj4rGI+AcwHdhlhOIbyIvdfEXEc0Ctm6969XU8H9g+b4e9gHMiYmFE3Afcm8vrpKHUpzQD1iUi7o+IW4AXGuYtbZ8bSl2WCaM9Qa0VEQ8D5L9L0xT0dUm3SDpR0grDG96gDKUuawKPR8Si/P8cUldVndJOXZp1pVUf86m56eLIDhwoB4rtJdPk9f4EaTu0M+9IG0p9ADaUdKOkqyS9o+pgBzCU9VvathlqPOMlzZR0naROfiFtqWufB1Uj6Qpg7SajvjIMxX8JeIR0ej0N+CJwzDCU21SFdWmrW6rhNAx16S/m/SLiIUmrABcAB5CaOEZKO+uz1TQjvi3aMJT6PAysHxGPStoauFDS6yPiyeEOsk1DWb+lbZuhxrN+RMyVtBFwpaRbI+KvwxTbsOj6BBURO7QaJ+nvkl4VEQ9LehUwb5BlP5zfLpR0KvC5IYTazvKqqst8YDVJY/O338q7pRqGuswBtqv7fz3StSci4qH8d4Gkn5GaQkYyQbXTzVdtmjmSxgKvAB5rc96RttT1iXSxYyFARMyS9FdgMjCz8qibG8r6bbnPdciQ9pWImJv/zpY0A9iSdE2rGKO9ie8ioHYnzoHArwYzcz541q7h/Btw27BGNzhLXZd8EPkfoHaXz6DXxTBrpy6XATtJWj3f5bcTcJmksZImAkgaB+zOyG+Xdrr5qq/j3sCVeTtcBOyT74rbENgEuH6E4m5lqesjqUfp2XDkb+qbkG4u6JShdMHWdJ+rKM52LHVdch1WyO8nAttS4qOQOn2XRidfpDby3wH35L9r5OG9wI/rprsG6AOeIX1r2TkPvxK4lXQAPAuYsAzXZSPSgfBe4OfACstAXT6U470XODgPWxmYBdwC3A6cRAfuggN2A+4mfSP9Sh52DLBnfj8+r+d783rfqG7er+T57gJ27dR2GI76AO/N2+Fm4AZgj2WgLm/On42ngUeB2/vb55bFugBvz8eum/PfQzpdl2Yvd3VkZmZFGu1NfGZmVignKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCshFV14Py7ZJulnSYpOXyuF5J383vV5B0RZ72/ZLekee5SdKKna1Fc5KOkdTyB8ilUXKlpFUrKn+GpN42p/2WpHdXEYctu7q+JwkrzjMRsQWA0mMwfkbqdeCrETGTJT0MbAmMq5v2ZOBbEXFqOwvJP55WRIxYJ5kRcdRILWuY7AbcHJ3rdqje94AfkX5baAb4DMo6KCLmAVOAT+Zv89tJuiQnrrOALfIZ00eA9wFHSfopgKTPS/pz7qj36DxskqQ7JH2f9KPQV0vaSdK1Ss/E+bmkCXna+yUdnYffKum1efgESafmYbdIem8e3rScekrPB9u7v/Ibpj9I0oWSLlZ6rtgn8xnljUodeK6Rp3uN0nO8Zkm6pi7WPZSevXRjPttcKw+fKumUfAYzW9KnW2yC/ci9dEj6Qm06pY6Pr8zvt5d0Vn/rQNLWSh3BzpJ0mXIPK3X1XE7S6ZKOlTQmr6fb8no5NO8LDwBrSmrWP6ONUk5Q1lERMZu0H76ybtg84MPANRGxRUT8kNSFy+cjYj9JO5G6zNkG2ALYWtI78+ybAmdExJakX88fAewQEVuRzs4Oq1v8/Dz8ByzpR/FI4ImIeENEvJHUiebEAcpppVn5jTYHPpDr8nXgnzn2a4EP5mmmAZ+KiK1zOd/Pw38PvDVPfw7whbpyX0t6PMQ2wFeVun1qtC2p1w2Aq4FaT+O9wIQ8z78A17RaB3ma75GeNbQ1cEquR81Y4KfA3RFxBGl7rRsRm0fEG4D6M+IbckxmgJv4rAyDfRzGTvl1Y/5/Ailh/Q14ICKuy8PfSnoA4B9Six/Lkw78Nb/If2cB/57f70Dq0wyAiPiHpN0HKKeVZuU3+p+IWAAskPQEcHEefivwxnyW8nbg51ry1JDaY13WA87NZyzLA/fVlfvriFhI6sh4HrAWqcubemvkZddi3FqpB/iFpGTRS0pan6b1utyUlGSn5+FjSD2Y1/wQOC8iaklrNrCRpO8BvwYur5t2HrBOi/Vko5ATlHWUUgeii0kHp9e1OxtwXD6zqi9rEumsqX666RGxb4tyFua/i1nyWRDNHyXRXzmtNCu/1TSQHiq3sO79WNLZ5eO1a3ENvgecEBEXSdoOmNqi3FbLXyRpuYh4ISKel3Q/cDDwR1Jfhu8CXgPckf++bB1IegOpf7e3tajfH4F3Sfp2RDybE/6bSGd3nyA13X4oTzue1EekGeAmPusgST2kx8v/dwyuU8jLgA/VXQNZN1+3anQdsK2kjfN0K0maPEDZlwOfrItx9aUsZ1jkGxjuk/QfednKB3hIN5c8lN8f2Gz+AdxF6iS45mpSE+LVpE6FPwrclLdNq3VwF9Aj6W15+DhJr68r8yfApaQzwFpP88tFxAWk5tSt6qadTGefCGCFcYKykbai8m3mwBWkhHD0YAqIiMtJd/9dK+lW0iPGV2kyXR9wEHC2pFtIB9mX3azQ4Fhg9XwR/2bgXUtZznDaDzgkx3M7Sx7rPZV04L+G9Eyvwfo1L32+0TXAq4BrI+LvwLN5WMt1GelR43sDx+f4biI1Sb4oIk4gNRmeSXri6wxJNwGnkR76WXs0ysZ07jlRViD3Zm42SuVrV2dExI4FxPJ/gK0i4shOx2Ll8BmU2SgV6YnQP1JFP9QdpLHAtzsdhJXFZ1BmZlYkn0GZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXpfwFcaRtkRQPqHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf23e9d5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_test_stats(test_stat_dist);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compute the probability that the test statistic, under the null hypothesis, exceeds the observed differences in the means.\n",
    "\n",
    "This probability is called a \"p-value\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.075"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_value = np.mean(test_stat_dist >= actual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example the p-value is about 8%, which means that the difference we saw, 13 hours, could plausibly occur due to random sampling, even if there is actually no difference between the groups.\n",
    "\n",
    "The following figure shows the p-value as the shaded area of the distribution above the observed value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def annotate(text, x, y, length):\n",
    "    arrowprops = dict(width=1, headwidth=6, facecolor='black')\n",
    "    plt.annotate(text,\n",
    "                 xy=(x, y),\n",
    "                 xytext=(x, y+length),\n",
    "                 ha='center',\n",
    "                 arrowprops=arrowprops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcHVWd9/HPN3sgRJY0yN4ghkVACI0GMzCo7IY484gaBhxwmbgMMoKMgyMqII7LM8KI8yjiQgJojII6gAgEmBgQEJKQQFgSMAkkBEwCAmFLIP17/jinyeXmdvftdFffSvf3/XrdV9+u5dTvVNWt361TdU8pIjAzMyubAY0OwMzMrBYnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnqBKQdImkL/dQWbtIekHSwPz/DEmf6Imyc3m/l3RKT5XXheVeIGmVpKd6e9l9haRTJd3e6DgAJJ0r6cr8vllSSBrUzrRLJB3RuxHWjOP1mHuwzDd8Xu2NnKAKlj9cL0taLelZSXdI+pSk19d9RHwqIr5WZ1kdflAj4vGIGBER63og9g0+kBFxbERM6W7ZXYxjZ+DzwD4R8eYa4w+XtKyHlrXRCb2r8xZxwLOe0ZP7VEd68vPaFzlB9Y7jI2ILYFfgm8C/AT/p6YW09w20D9gVeDoiVjQ6kP7K3/CtISLCrwJfwBLgiKph7wBagX3z/5OBC/L7UcB1wLPAM8BtpC8SV+R5XgZeAL4ANAMBfBx4HJhZMWxQLm8G8A3gbuA54H+ArfO4w4FlteIFjgHWAq/m5c2rKO8T+f0A4BzgMWAFcDnwpjyuLY5TcmyrgC91sJ7elOdfmcs7J5d/RK5za45jctV8m1eNfwHYIc97NvBn4GnglxX1HgZcmYc/C9wDbAd8HVgHvJLL+e8acXZpXuC7wFLgeWA2cGgeXnP91lheAHtU/D+Z9fvK4cAy0tnlCuBJ4KMV024DXJOXfTfwNeD2ivF7AdNJ+9kC4ENVy/kBcD3wIlX7cMW+8DXgj8Bq4CZgVGf7Vn5/LnBl1b4yqIPP0FnAfaR9eBowLI+bT/oC2DbtYNK+dkBFuZOA5Xn9fL5i2qHAf+Vxy/P7obS/T51L2o8uz/V9AGipKG8H4GrSPrwYOL3qMz8rb4u/ABfWqjtwKrAol78YOKnRx7BGvhoeQF9/USNB5eGPA5/O7yez/qDzDeCS/EEbDBwKqFZZFTv35flDNbzGDj8DeALYN09zdcWBoe6DSMX4GaxPUB8DHgV2B0YAvwauqIrtRzmutwNrgL3bWU+Xk5LnFnnehcDH24uzat5a9fgccBewE+mg80Ngah73SeBaYDNgIHAQMLK6fu0sq0vzAieTEsUgUiJ5ivUH1w3Wb43ldZagXgPOz/vKccBLwFZ5/C9IB9TN8/Z/gpyg8rClwEdzbGNIB/a3VSznOWAcKdkPqxHbDNIXgNF5G88AvtnVfYv6EtTdpASwNfAQ8Kk87gvAtIpp3w/cX1Xu1Fzf/UjJoy2G8/M+si3QBNwBfK2D+M8lfQE5Lm/7bwB35XEDSF9AvgIMIX0mFgFH5/F3Ah/J70cAY6vrnmN8Htgzj9u+bXv015eb+BpnOenDVu1V0o65a0S8GhG3Rd5bO3BuRLwYES+3M/6KiJgfES8CXwY+1ENNNieRvgkuiogXgC8CE6uaGs+LiJcjYh4wj5So3iDH8mHgixGxOiKWAN8BPtKN2D5JOmNbFhFrSAeXE3Jsr5KSxh4RsS4iZkfE83WW26V5I+LKiHg6Il6LiO+QkuWe3ahXrXjOz/vK9aRv+3vmdfoB4Ct535gPVF47HA8siYjLcmxzSF9eTqiY5n8i4o8R0RoRr7Sz/MsiYmHe935JOnMpwsURsTwiniF9QWhbzpXAcZJG5v8/QmptqHReXgf3A5cBJ+bhJ5HW3YqIWAmcR+f73O0RcX2ka0ZXsH5/PhhoiojzI2JtRCwifTmbmMe/CuwhaVREvBARd7VTfiuwr6ThEfFkRDzQSTx9mhNU4+xIalqp9n9JZyU3SVok6ew6ylrahfGPkb5tj6oryo7tkMurLHsQqcmrTeVddy+Rvj1WG0X61lld1o7diG1X4Df5xpRnSd+61+XYrgBuBH4habmkb0saXGe5XZpX0uclPSTpuRzHm+iZdd/m6Yh4reL/tnXcRNoW1du+za7AO9vWT47tJKDyJpTO9iuob/v2hJrLiYjlpCbGD0jaEjgW+FnVvNXrYIf8vtb+uwMdq45jWP7SsyuwQ9X6/HfWfxY+TjrTfFjSPZLGVxecv0B+GPgU8KSk30naq5N4+jQnqAaQdDDp4LvBLb/5DOLzEbE7cDxwpqT3to1up8jOzrB2rni/C+nb3CrStYXNKuIaSDqw1VvuctIHs7Ls10ht7F2xKsdUXdYTdc5fK86lwLERsWXFa1hEPJHPNs6LiH2Ad5HOJv6xg7LWL6gL80o6lHRDzIdIzW5bkprNVM+yspeo2Ea8MYF0ZCVpW1Rv+zZLgT9UrZ8REfHpimm686iDzvatnjSF1JT6QeDOiKjeb6rXwfL8vtb+2zauq3VfCiyuWp9bRMRxABHxSEScSGpO/BZwlaTNqwuJiBsj4khSK8rDpLOwfssJqhdJGpm/Of2C1P5+f41pxkvaQ5JI7dHr8gvSgX/3jVj0yZL2kbQZqd39qtxEsZD0DfB9+SzgHFITVJu/AM2Vt8RXmQqcIWk3SSOA/yBdD3itnelryrH8Evi6pC0k7QqcSWq+qcdfgG0kvali2CW5vF0BJDVJen9+/25J++WD5vOk5FjXOu7ivFuQksRKYJCkrwAjK8Z3tn4B5gL/IGmgpGOAv+1g2tfldfpr4FxJm0nah3TDSpvrgNGSPiJpcH4dLGnvesqvQ2f7Vk/6Leka2r+QrmVW+3JeB28jXXOblodPBc7J+8Yo0vWjtn2u1j7VkbuB5yX9m6TheXvtm7+MIulkSU0R0Uq6uQbW7zfkabaTNCEnrjWk5tp+ffu5E1TvuFbSatK3rC8BF5I+KLW8FbiZtHPeCXw/Imbkcd8gfaCelXRWF5Z/Bemi91Oku9BOB4iI54DPAD8mna28SLorrM2v8t+nJc2pUe5Pc9kzSXccvQJ8tgtxVfpsXv4i0pnlz3P5nYqIh0kHm0V53exAunvuGlJT6WrSxfB35lneDFxFSjAPAX9g/YHpu6RrVX+VdHGNxXVl3huB35MO1o+R1k9lc1Nn6xfSQfd40kHtJNLBuF6nkZrCniJt/8vaRkTEauAo0jWS5Xmab9FDSaSOfavH5OtfVwO7kZJytT+Qms1vAf4zIm7Kwy8g3Vl3H3A/MCcPa2+f6iiGdaTtdADps7CKVPe2BHcM8ICkF0j7ycQa1/UGkG6kWU5q/v9b0jrst9ruDjMz22Tls9PREXFyxbBmUrIY3NWzeiuHvvrDTjPrJyRtTboJoTt3fVoJuYnPzDZZkv6J1Gz6+4iY2eh4rGe5ic/MzErJZ1BmZlZKpboGNWrUqGhubm50GGZmVqDZs2eviohOfxdXWIKStCfrf28A6fchX4mI/2pvnubmZmbNmlVUSGZmVgKSHut8qgITVEQsIPeXlX/U+ATwm6KWZ2ZmfUtvXYN6L/DniKgra5qZmfVWgppI+lX2BiRNkjRL0qyVK1f2UjhmZlZ2hScoSUOACazv1uUNIuLSiGiJiJampqL6kjQzs01Nb5xBHQvMiYiu9nBtZmb9WG8kqBNpp3nPzMysPYUmqPx4hyOp3cOwmZlZuwr9oW5EvER6PLaZmVmXlKonCbPecNH0hY0OYQNnHDm60SGYlY774jMzs1LyGZRZCZTprM5nc1YWTlDWrzXyYFympGRWRm7iMzOzUvIZlFmDlKkpzWdzVkY+gzIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1JygjIzs1IqNEFJ2lLSVZIelvSQpEOKXJ6ZmfUdRT9R97vADRFxgqQhwGYFL8/MzPqIwhKUpJHAYcCpABGxFlhb1PLMzKxvKbKJb3dgJXCZpHsl/VjS5tUTSZokaZakWStXriwwHDMz25QUmaAGAWOAH0TEgcCLwNnVE0XEpRHREhEtTU1NBYZjZmabkiIT1DJgWUT8Kf9/FSlhmZmZdaqwBBURTwFLJe2ZB70XeLCo5ZmZWd9S9F18nwV+lu/gWwR8tODlmZlZH1FogoqIuUBLkcswM7O+yT1JmJlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKQ0qsnBJS4DVwDrgtYhoKXJ5ZmbWdxSaoLJ3R8SqXliOmZn1Ib2RoMxsE3LR9IWNDgGAM44c3egQrMGKvgYVwE2SZkuaVGsCSZMkzZI0a+XKlQWHY2Zmm4qiE9S4iBgDHAv8s6TDqieIiEsjoiUiWpqamgoOx8zMNhWFNvFFxPL8d4Wk3wDvAGYWuUwz67qyNKeVpXnRyqGwMyhJm0vaou09cBQwv6jlmZlZ31LkGdR2wG8ktS3n5xFxQ4HLMzOzPqSwBBURi4C3F1W+mZn1be5JwszMSskJyszMSsk/1LVe4buzzKyrfAZlZmal5ARlZmal5CY+63Vl+VGomZWbz6DMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyU6kpQkq6W9D5JXU5okgZKulfSdV0Pz8zM+qt6E84PgH8AHpH0TUl7dWEZ/wI81OXIzMysX6srQUXEzRFxEjAGWAJMl3SHpI9KGtzefJJ2At4H/LgngjUzs/6j7iY7SdsApwKfAO4FvktKWNM7mO2/gC8ArR2UO0nSLEmzVq5cWW84ZmbWx9V7DerXwG3AZsDxETEhIqZFxGeBEe3MMx5YERGzOyo7Ii6NiJaIaGlqaupi+GZm1lcNqnO6H0fE9ZUDJA2NiDUR0dLOPOOACZKOA4YBIyVdGREndyNeMzPrJ+pt4rugxrA7O5ohIr4YETtFRDMwEbjVycnMzOrV4RmUpDcDOwLDJR0IKI8aSWruMzMzK0RnTXxHk26M2Am4sGL4auDf611IRMwAZnQtNDMz6886TFARMQWYIukDEXF1L8VkZmbWaRPfyRFxJdAs6czq8RFxYY3ZzMzMuq2zJr7N89+at5KbmZkVpbMmvh/mv+f1TjhmZmZJvT/U/bakkZIGS7pF0ipJvmXczMwKU+/voI6KiOeB8cAyYDTwr4VFZWZm/V69CaqtQ9jjgKkR8UxB8ZiZmQH1d3V0raSHgZeBz0hqAl4pLiwzM+vv6n3cxtnAIUBLRLwKvAi8v8jAzMysf6v3DApgb9LvoSrnubyH4zEzMwPqTFCSrgDeAswF1uXBgROUmZkVpN4zqBZgn4iIIoMxMzNrU+9dfPOBNxcZiJmZWaV6z6BGAQ9KuhtY0zYwIiYUEpWZmfV79Saoc4sMwszMrFpdCSoi/iBpV+CtEXGzpM2AgcWGZmZm/Vm9ffH9E3AV8MM8aEfgt0UFZWZmVu9NEv8MjAOeB4iIR4BtiwrKzKy3LVmyhH333bfRYViFehPUmohY2/ZP/rGubzk3M7PC1Jug/iDp34Hhko4EfgVcW1xYZmbtW7JkCXvttRennHIK+++/PyeccAIvvfTSG6b58Ic/zPXXX//6/6eeeipXX301S5Ys4dBDD2XMmDGMGTOGO+64Y4PyJ0+ezGmnnfb6/+PHj2fGjBkA3HTTTRxyyCGMGTOGD37wg7zwwgvFVNLqTlBnAyuB+4FPAtcD5xQVlJlZZxYsWMCkSZO47777GDlyJN///vffMH7ixIlMmzYNgLVr13LLLbdw3HHHse222zJ9+nTmzJnDtGnTOP300+te5qpVq7jgggu4+eabmTNnDi0tLVx44YU9Wi9br967+Fol/Rb4bUSsrGceScOAmcDQvJyrIuKrGx2pmVmFnXfemXHjxgFw8sknc/HFF3PWWWe9Pv7YY4/l9NNPZ82aNdxwww0cdthhDB8+nOeee47TTjuNuXPnMnDgQBYuXFj3Mu+66y4efPDB15e7du1aDjnkkJ6tmL2uwwQlScBXgdMA5UHrgO9FxPmdlL0GeE9EvCBpMHC7pN9HxF09EbiZ9W/p8LTec889xwEHHADA+eefz4QJEzj88MO58cYbmTZtGieeeCIAF110Edtttx3z5s2jtbWVYcOGbVD2oEGDaG1tff3/V15JTxeKCI488kimTp1aVLWsQmdNfJ8j3b13cERsExFbA+8Exkk6o6MZI2lrnB2cX76xwsx6xOOPP86dd94JwNSpUxk/fjxz585l7ty5TJiQOrmZOHEil112GbfddhtHH300kBLZ9ttvz4ABA7jiiitYt27dBmU3Nzczd+5cWltbWbp0KXfffTcAY8eO5Y9//COPPvooAC+99FKXzsCsazpLUP8InBgRi9sGRMQi4OQ8rkOSBkqaC6wApkfEn7oTrJlZm7333pspU6aw//7788wzz/DpT396g2mOOuooZs6cyRFHHMGQIUMA+MxnPsOUKVMYO3YsCxcuZPPNN99gvnHjxrHbbrux3377cdZZZzFmzBgAmpqamDx5MieeeCL7778/Y8eO5eGHHy62ov2YOuqgXNL8iKj5w4COxtWYdkvgN8BnI2J+1bhJwCSAXXbZ5aDHHnus3thtE3LR9PXfMs84cnQDI7Eyq3c/WbJkCePHj2f+/PntTmPlJWl2RLR0Nl1nZ1BrN3LcG0TEs8AM4Jga4y6NiJaIaGlqaqq3SDMz6+M6u4vv7ZKerzFcwIZXFisnkJqAVyPiWUnDgSOAb21cmGZm6zU3N/vsqR/o8AwqIgZGxMgary0iYnAnZW8P/K+k+4B7SNegruupwM3MNta1117LoEGD+NznPtfoUKwD9T5uo8si4j7gwKLKNzPbWKtWrWLdunU88cQTjQ7FOlBvTxJmZma9ygnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKqbC++MzMuqPy2VDtWdfausGj3+tx9/wnAVj4l9V856YF7U43IJftZ5g1hhOUmW2ybnloBSOHd/ZghQ0tXvUiAM+8uJa7Fz9Tc5rhgwfy9p237FZ81j1u4jMzs1LyGZSZlUZXm9KW/fUldtpqsy4vZ8AjI7gH2HrzIYzdfZua08xb+myXy7We5TMoMzMrJScoMzMrJScoMzMrJScoMzMrJScoMzMrJScoMzMrJScoMzMrpcJ+ByVpZ+By4M1AK3BpRHy3qOVZbfV0F2NmVkZF/lD3NeDzETFH0hbAbEnTI+LBApdpZmZ9RGFNfBHxZETMye9XAw8BOxa1PDMz61t6pasjSc3AgcCfaoybBEwC2GWXXXojnH7LPTKb2aak8JskJI0ArgY+FxHPV4+PiEsjoiUiWpqamooOx8zMNhGFJihJg0nJ6WcR8esil2VmZn1LYQlK6SliPwEeiogLi1qOmZn1TUWeQY0DPgK8R9Lc/DquwOWZmVkfUthNEhFxO9D1ZzGbmZnhniTMzKyknKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUnKDMzKyUeuWJumZmZTBv5g08/dRSli64H4CVyxZz6y9/xKDBQ/mbCScxYODABkdolZygzKzfuOHyi1mxbDHpcXXw1GOPcv1PLyIiGHvsBxkycHiDI7RKbuIzs35jr5ZDIaB13ToAorWV1tZ1bLfL7gwZ5uRUNk5QZtZvjD5oHEOHb/aGYRowgL3fcXhjArIOOUGZWb+x29vG8OraV94wbOiwzRg95l0Nisg64gRlZv3GsM1GsM32u7xh2KtrX6F5nwMbFJF1xAnKzPqVvQ8+DGn9oa9px+YNmv2sHJygzKxfqbwO5etP5eYEZWb9SuV1KF9/KjcnKDPrVyqvQ/n6U7kVlqAk/VTSCknzi1qGmdnG2PvgwwD5+lPJFXkGNRk4psDyzcw2yuiDxgHh608lV1iCioiZwDNFlW9mtrF2e9sYNGCArz+VXMOvQUmaJGmWpFkrV65sdDhm1g8M22wEE8/8D96y38GNDsU60PAEFRGXRkRLRLQ0NTU1Ohwz6ycOPurvGTRkSKPDsA64N/MCXDR9YaNDMDPb5DX8DMrMzKyWIm8znwrcCewpaZmkjxe1LDMz63sKa+KLiBOLKntTcsaRoxsdgpnZJslNfGZmVkq+ScLMrIYAIgKAC29a0NhgAEn9rkXGCcrMrIZXXl3HnxaXo6+BPbYdwagRQxsdRq9zE5+ZmZWSz6DMzGoYu/s2jQ4BgLsWPd3oEBrGZ1BmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKTlBmZlZKfao384umL2x0CGZm1kN8BmVmZqXkBGVmZqXUp5r4Kp1x5OhGh2Bm1qPKdBmjN46xhSYoSccA3wUGAj+OiG8WuTwzs75o0coXWbzqxUaHwbDBA9lvxzf12vIKS1CSBgL/DzgSWAbcI+maiHiwqGWamfVFrREQjY4C1rX2bhBFnkG9A3g0IhYBSPoF8H7ACcrMesR2I4ex7cihjQ6jUBMO2KHRIbzu9kdW9eryikxQOwJLK/5fBryzeiJJk4BJ+d8XJC3oiYWf2f0iRgG9uzWK05fqAn2rPq5LObkunejmMXbXeiYqMkGpxrANzg8j4lLg0gLj2CiSZkVES6Pj6Al9qS7Qt+rjupST61IORd5mvgzYueL/nYDlBS7PzMz6kCIT1D3AWyXtJmkIMBG4psDlmZlZH1JYE19EvCbpNOBG0m3mP42IB4paXgFK1+zYDX2pLtC36uO6lJPrUgKKKMG9i2ZmZlXc1ZGZmZWSE5SZmZVSv05QkraWNF3SI/nvVu1Md4OkZyVdVzV8sqTFkubm1wG9E3nNGLtbl90k/SnPPy3f2NIQXajLKXmaRySdUjF8hqQFFdtl296L/vUYjskxPCrp7Brjh+b1/Ghe780V476Yhy+QdHRvxt2eja2PpGZJL1dsi0t6O/ZqddTlMElzJL0m6YSqcTX3uUbpZl3WVWyXct7AFhH99gV8Gzg7vz8b+FY7070XOB64rmr4ZOCERtejh+ryS2Bifn8J8Oky1wXYGliU/26V32+Vx80AWhoY/0Dgz8DuwBBgHrBP1TSfAS7J7ycC0/L7ffL0Q4HdcjkDG7xvdac+zcD8Rsa/EXVpBvYHLq/8fHe0z21qdcnjXmj09ujs1a/PoEhdL03J76cAf1drooi4BVjdW0FtpI2uiyQB7wGu6mz+XlJPXY4GpkfEMxHxV2A6cEwvxdeZ17v5ioi1QFs3X5Uq63gV8N68Hd4P/CIi1kTEYuDRXF4jdac+ZdNpXSJiSUTcB7RWzVu2fa47ddkk9PcEtV1EPAmQ/25MU9DXJd0n6SJJjewUrDt12QZ4NiJey/8vI3VV1Sj11KVWV1qVMV+Wmy6+3IADZWexvWGavN6fI22Heubtbd2pD8Buku6V9AdJhxYdbCe6s37Ltm26G88wSbMk3SWpkV9I29VnnwfVRtLNwJtrjPpSDxT/ReAp0un1pcC/Aef3QLk1FViXurql6kk9UJeOYj4pIp6QtAVwNfARUhNHb6lnfbY3Ta9vizp0pz5PArtExNOSDgJ+K+ltEfF8TwdZp+6s37Jtm+7Gs0tELJe0O3CrpPsj4s89FFuP6PMJKiKOaG+cpL9I2j4inpS0PbCii2U/md+ukXQZcFY3Qq1neUXVZRWwpaRB+dtv4d1S9UBdlgGHV/y/E+naExHxRP67WtLPSU0hvZmg6unmq22aZZIGAW8Cnqlz3t620fWJdLFjDUBEzJb0Z2A0MKvwqGvrzvptd59rkG7tKxGxPP9dJGkGcCDpmlZp9PcmvmuAtjtxTgH+pysz54Nn2zWcvwPm92h0XbPRdckHkf8F2u7y6fK66GH11OVG4ChJW+W7/I4CbpQ0SNIoAEmDgfH0/napp5uvyjqeANyat8M1wMR8V9xuwFuBu3sp7vZsdH0kNSk9G478Tf2tpJsLGqU7XbDV3OcKirMeG12XXIeh+f0oYBxlfBRSo+/SaOSL1EZ+C/BI/rt1Ht5CegJw23S3ASuBl0nfWo7Ow28F7icdAK8ERmzCddmddCB8FPgVMHQTqMvHcryPAh/NwzYHZgP3AQ+Qn+jcgDocBywkfSP9Uh52PjAhvx+W1/Ojeb3vXjHvl/J8C4BjG7UdeqI+wAfydpgHzAGO3wTqcnD+bLwIPA080NE+tynWBXhXPnbNy38/3ui61Hq5qyMzMyul/t7EZ2ZmJeUEZWZmpeQEZWZmpeQEZWZmpeQEZWZmpeQEZb2qogflByTNk3SmpAF5XIuki/P7oZJuztN+WNKheZ65koY3tha1STpfUrs/QC4bJbdKGllQ+TMktdQ57X9Kek8Rcdimq8/3JGGl83JEHACg9BiMn5N6HfhqRMxifQ8DBwKDK6a9BPjPiLisnoXkH08rInqtk8yI+EpvLauHHAfMi8Z1O1Tpe8CPSL8tNAN8BmUNFBErgEnAafnb/OGSrsuJ60rggHzG9EngQ8BXJP0MQNK/Srond9R7Xh7WLOkhSd8n/Sh0Z0lHSbpT6Zk4v5I0Ik+7RNJ5efj9kvbKw0dIuiwPu0/SB/LwmuVUUno+2AkdlV81/amSfivpWqXnip2WzyjvVerAc+s83VuUnuM1W9JtFbEer/TspXvz2eZ2efi5kn6az2AWSTq9nU1wErmXDklfaJtOqePjW/P790q6sqN1IOkgpY6uRTj6AAADt0lEQVRgZ0u6UbmHlYp6DpA0RdIFkgbm9TQ/r5cz8r7wGLCNpFr9M1o/5QRlDRURi0j74bYVw1YAnwBui4gDIuKHpC5c/jUiTpJ0FKnLnHcABwAHSTosz74ncHlEHEj69fw5wBERMYZ0dnZmxeJX5eE/YH0/il8GnouI/SJif1InmqM6Kac9tcqvti/wD7kuXwdeyrHfCfxjnuZS4LMRcVAu5/t5+O3A2Dz9L4AvVJS7F+nxEO8AvqrU7VO1caReNwBmAm09jbcAI/I8fwPc1t46yNN8j/SsoYOAn+Z6tBkE/AxYGBHnkLbXjhGxb0TsB1SeEc/JMZkBbuKzcujq4zCOyq978/8jSAnrceCxiLgrDx9LegDgH1OLH0NIB/42v85/ZwP/J78/gtSnGQAR8VdJ4zsppz21yq/2vxGxGlgt6Tng2jz8fmD/fJbyLuBXWv/UkLbHuuwETMtnLEOAxRXl/i4i1pA6Ml4BbEfq8qbS1nnZbTEepNQD/BpSsmghJa3TaX9d7klKstPz8IGkHszb/BD4ZUS0Ja1FwO6Svgf8DripYtoVwA7trCfrh5ygrKGUOhBdRzo47V3vbMA38plVZVnNpLOmyummR8SJ7ZSzJv9dx/rPgqj9KImOymlPrfLbmwbSQ+XWVLwfRDq7fLbtWlyV7wEXRsQ1kg4Hzm2n3PaW/5qkARHRGhGvSloCfBS4g9SX4buBtwAP5b8brANJ+5H6dzuknfrdAbxb0nci4pWc8N9OOrv7Z1LT7cfytMNIfUSaAW7iswaS1ER6vPx/R9c6hbwR+FjFNZAd83WrancB4yTtkafbTNLoTsq+CTitIsatNrKcHpFvYFgs6YN52coHeEg3lzyR359Sa/5OLCB1EtxmJqkJcSapU+FPAXPztmlvHSwAmiQdkocPlvS2ijJ/AlxPOgNs62l+QERcTWpOHVMx7Wga+0QAKxknKOttw5VvMwduJiWE87pSQETcRLr7705J95MeMb5FjelWAqcCUyXdRzrIbnCzQpULgK3yRfx5wLs3spyedBLw8RzPA6x/rPe5pAP/baRnenXV73jj841uA7YH7oyIvwCv5GHtrstIjxo/AfhWjm8uqUnydRFxIanJ8ArSE19nSJoLTCY99LPt0Sh70LjnRFkJuTdzs34qX7u6PCKOLEEsfw+MiYgvNzoWKw+fQZn1U5GeCP0jFfRD3S4aBHyn0UFYufgMyszMSslnUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkr/H8p1qMku8TmDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf4db57cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "patch = plot_test_stats(test_stat_dist)\n",
    "low = actual\n",
    "high = np.max(test_stat_dist)\n",
    "fill_hist(low, high, patch)\n",
    "annotate('p-value', 1.2*actual, 1.0, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different test statistics\n",
    "\n",
    "What we computed in the previous section is the probability that first babies would be *later* than 13 hours, on average, under the null hypothesis.\n",
    "\n",
    "Depending on the context, we might also want to know the probability that first babies would be *earlier* on average.\n",
    "\n",
    "We can include both possibilities by defining a new test statistic, the *absolute* difference in means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_stat(data):\n",
    "    group1, group2 = data\n",
    "    return abs(group1.mean() - group2.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the observed difference with this test stat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07803726677754952"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actual = test_stat(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the actual difference is positive, its absolute value is the same.\n",
    "\n",
    "We can run the simulated experiments with this test statistic, and print the p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.156"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_stat_dist = np.array([test_stat(run_model(data)) \n",
    "                       for i in range(1000)])\n",
    "\n",
    "p_value = np.mean(test_stat_dist >= actual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the distribution looks like for this test statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcXFWd9/HPNzshhC0NErawyDaQgdhAkEFR2SfAPA8gyQCC4qAyyAzIKIyAgDjqjMKIPoq4ELYJUXBBRCHoBJBFDCFA2AKGyKpJQEJCSEKS3/PHOZ0Uleru6nRX3dvp7/v1qldX3eXcX50+Vb97z711riICMzOzsulXdABmZma1OEGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGZmVkpOUGVjKQrJV3QQ2VtI2mRpP759VRJH++JsnN5v5J0ck+V14XtXippvqQ/N3vb6wpJp0j6XdFxAEi6SNL1+fkoSSFpQDvLzpF0UHMjrBnHqph7sMx3fF7NCaqp8ofrLUkLJb0u6T5Jn5S06v8QEZ+MiC/WWVaHH9SIeD4ihkXEih6IfY0PZEQcHhHXdLfsLsaxNfAZYLeIeFeN+QdKerGHtrXWCb2r6zbiC896Rk+2qY705Od1XeEE1XxHRsQGwLbAV4DPAT/o6Y20twe6DtgWeDUi5hYdSF/lPXxrmojwo0kPYA5wUNW0fYCVwO759UTg0vx8BHAr8DrwGnAPaafiurzOW8Ai4LPAKCCAU4Hngbsrpg3I5U0Fvgw8CCwAfg5skucdCLxYK17gMGAZ8Hbe3iMV5X08P+8HnA/8CZgLXAtsmOe1xXFyjm0+8PkO6mnDvP68XN75ufyD8ntemeOYWLXe+lXzFwEj87rnAn8EXgV+VPG+hwDX5+mvA38ANge+BKwAluRyvlUjzi6tC3wDeAF4A3gIOCBPr1m/NbYXwI4Vryeyuq0cCLxIOrqcC7wCfLRi2U2BW/K2HwS+CPyuYv4uwBRSO3sa+HDVdr4D3Aa8SVUbrmgLXwTuBRYCdwAjOmtb+flFwPVVbWVAB5+hc4BHSW14MjAkz5tJ2gFsW3Ygqa3tWVHuacDLuX4+U7HsYOC/87yX8/PBtN+mLiK1o2vz+30caK0obyRwM6kNPwecWfWZn5b/F38BLqv13oFTgNm5/OeAE4r+Dmv2o/AA+tKDGgkqT38e+FR+PpHVXzpfBq7MH7SBwAGAapVV0bivzR+q9Wo0+KnAS8DueZmbK74Y6v4SqZg/ldUJ6mPAs8D2wDDgJ8B1VbF9L8f1t8BSYNd26ulaUvLcIK87Czi1vTir1q31Pv4VeADYivSl811gUp73CeAXwFCgP/AeYHj1+2tnW11aFziRlCgGkBLJn1n95bpG/dbYXmcJajlwSW4rRwCLgY3z/BtJX6jr5///S+QElae9AHw0xzaG9MX+NxXbWQDsT0r2Q2rENpW0A7BT/h9PBb7S1bZFfQnqQVIC2AR4EvhknvdZYHLFskcDj1WVOym/3z1IyaMthktyG9kMaAHuA77YQfwXkXZAjsj/+y8DD+R5/Ug7IBcCg0ifidnAoXn+/cBJ+fkwYGz1e88xvgHsnOdt0fb/6EsPd/GVw8ukD1u1t0kNc9uIeDsi7oncWjtwUUS8GRFvtTP/uoiYGRFvAhcAH+6hLpsTSHuCsyNiEXAeML6qq/HiiHgrIh4BHiElqnfIsRwPnBcRCyNiDvB14KRuxPYJ0hHbixGxlPTlcmyO7W1S0tgxIlZExEMR8Uad5XZp3Yi4PiJejYjlEfF1UrLcuRvvq1Y8l+S2chtpb3/nXKfHABfmtjETqDx3OA6YExFX59imk3Zejq1Y5ucRcW9ErIyIJe1s/+qImJXb3o9IRy6NcEVEvBwRr5F2ENq2cz1whKTh+fVJpN6GShfnOngMuBqYkKefQKq7uRExD7iYztvc7yLitkjnjK5jdXveG2iJiEsiYllEzCbtnI3P898GdpQ0IiIWRcQD7ZS/Ethd0noR8UpEPN5JPOscJ6hy2JLUtVLtv0hHJXdImi3p3DrKeqEL8/9E2tseUVeUHRuZy6ssewCpy6tN5VV3i0l7j9VGkPY6q8vashuxbQv8NF+Y8jppr3tFju064HbgRkkvS/pPSQPrLLdL60r6jKQnJS3IcWxIz9R9m1cjYnnF67Y6biH9L6r/9222BfZtq58c2wlA5UUonbUrqO//2xNqbiciXiZ1MR4jaSPgcOCGqnWr62Bkfl6r/Y6kY9VxDMk7PdsCI6vq899Z/Vk4lXSk+ZSkP0gaV11w3oE8Hvgk8IqkX0rapZN41jlOUAWTtDfpy3eNS37zEcRnImJ74EjgbEkfapvdTpGdHWFtXfF8G9Le3HzSuYWhFXH1J32x1Vvuy6QPZmXZy0l97F0xP8dUXdZLda5fK84XgMMjYqOKx5CIeCkfbVwcEbsB7yUdTXykg7JWb6gL60o6gHRBzIdJ3W4bkbrNVM+2ssVU/I94ZwLpyDzS/6L6f9/mBeCuqvoZFhGfqlimO7c96Kxt9aRrSF2pxwH3R0R1u6mug5fz81rtt21eV9/7C8BzVfW5QUQcARARz0TEBFJ34leBmyStX11IRNweEQeTelGeIh2F9SlOUAWRNDzvOd1I6n9/rMYy4yTtKEmk/ugV+QHpi3/7tdj0iZJ2kzSU1O9+U+6imEXaA/z7fBRwPqkLqs1fgFGVl8RXmQScJWk7ScOA/yCdD1jezvI15Vh+BHxJ0gaStgXOJnXf1OMvwKaSNqyYdmUub1sASS2Sjs7PPyBpj/yl+QYpOdZVx11cdwNSkpgHDJB0ITC8Yn5n9QswA/hHSf0lHQa8v4NlV8l1+hPgIklDJe1GumClza3ATpJOkjQwP/aWtGs95dehs7bVk35GOof2L6RzmdUuyHXwN6RzbpPz9EnA+bltjCCdP2prc7XaVEceBN6Q9DlJ6+X/1+55ZxRJJ0pqiYiVpItrYHW7IS+zuaSjcuJaSuqu7XOXnztBNd8vJC0k7WV9HriM9EGp5d3AnaTGeT/w7YiYmud9mfSBel3SOV3Y/nWkk95/Jl2FdiZARCwATge+TzpaeZN0VVibH+e/r0qaXqPcH+ay7yZdcbQE+HQX4qr06bz92aQjy//J5XcqIp4ifdnMznUzknT13C2krtKFpJPh++ZV3gXcREowTwJ3sfqL6Rukc1V/lXRFjc11Zd3bgV+Rvqz/RKqfyu6mzuoX0pfukaQvtRNIX8b1OoPUFfZn0v//6rYZEbEQOIR0juTlvMxX6aEkUkfb6jH5/NfNwHakpFztLlK3+W+Ar0XEHXn6paQr6x4FHgOm52nttamOYlhB+j/tSfoszCe997YEdxjwuKRFpHYyvsZ5vX6kC2leJnX/v59Uh31K2xVhZmbrhHx0ulNEnFgxbRQpWQzs6lG9FWdd/TGnmfVBkjYhXYTQnas+rSTcxWdm6wRJ/0TqNv1VRNxddDzWfe7iMzOzUvIRlJmZlVKvOAc1YsSIGDVqVNFhmJlZD3jooYfmR0Snv4XrFQlq1KhRTJs2regwzMysB0j6U+dLuYvPzMxKygnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKyQnKzMxKqVf8UHddcfmUWUWHsIazDt6p6BDMzGryEZSZmZWSE5SZmZWSu/gKUmTXWhm7Gs3MqvkIyszMSskJyszMSskJyszMSskJyszMSskJyszMSskJyszMSskJyszMSqlP/A7Kv/sxM+t9GnYEJemHkuZKmlkx7b8kPSXpUUk/lbRRo7ZvZma9WyO7+CYCh1VNmwLsHhGjgVnAeQ3cvpmZ9WIN6+KLiLsljaqadkfFyweAYxu1/fZ49G4zs96hyIskPgb8qsDtm5lZiRWSoCR9HlgO3NDBMqdJmiZp2rx585oXnJmZlULTE5Skk4FxwAkREe0tFxFXRURrRLS2tLQ0L0AzMyuFpl5mLukw4HPA+yNicTO3bWZmvUsjLzOfBNwP7CzpRUmnAt8CNgCmSJoh6cpGbd/MzHq3Rl7FN6HG5B80antmZrZu8VBHZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSgOKDsCKdfmUWUWHAMBZB+9UdAhmVjI+gjIzs1JqWIKS9ENJcyXNrJi2iaQpkp7Jfzdu1PbNzKx3a2QX30TgW8C1FdPOBX4TEV+RdG5+/bkGxmA1lKU7rSzdi2ZWTg07goqIu4HXqiYfDVyTn18D/EOjtm9mZr1bs89BbR4RrwDkv5u1t6Ck0yRNkzRt3rx5TQvQzMzKobQXSUTEVRHRGhGtLS0tRYdjZmZN1uwE9RdJWwDkv3ObvH0zM+slmp2gbgFOzs9PBn7e5O2bmVkv0cjLzCcB9wM7S3pR0qnAV4CDJT0DHJxfm5mZraFhl5lHxIR2Zn2oUds0M7N1R2kvkjAzs77NY/FZKZTlR7tl+RGzmfkIyszMSsoJyszMSsldfFaYsnSnlaV70czeyUdQZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSk5QZmZWSoUkKElnSXpc0kxJkyQNKSIOMzMrr6YnKElbAmcCrRGxO9AfGN/sOMzMrNyK6uIbAKwnaQAwFHi5oDjMzKykmp6gIuIl4GvA88ArwIKIuKN6OUmnSZomadq8efOaHaaZmRWsiC6+jYGjge2AkcD6kk6sXi4iroqI1ohobWlpaXaYZmZWsCK6+A4CnouIeRHxNvAT4L0FxGFmZiVWRIJ6HhgraagkAR8CniwgDjMzK7G6EpSkmyX9vaRuJ7SI+D1wEzAdeCzHcFV3yzUzs3VLvQnnO8A/As9I+oqkXbqz0Yj4QkTsEhG7R8RJEbG0O+WZmdm6p64EFRF3RsQJwBhgDjBF0n2SPippYCMDNDOzvqnuLjtJmwKnAB8HHga+QUpYUxoSmZmZ9WkD6llI0k+AXYDrgCMj4pU8a7KkaY0KzszM+q66EhTw/Yi4rXKCpMERsTQiWhsQl5mZ9XH1dvFdWmPa/T0ZiJmZWaUOj6AkvQvYkjRu3l6A8qzhpDH0zMzMGqKzLr5DSRdGbAVcVjF9IfDvDYrJzMys4wQVEdcA10g6JiJublJMZmZmnXbxnRgR1wOjJJ1dPT8iLquxmpmZWbd11sW3fv47rNGBmJmZVeqsi++7+e/FzQnHzMwsqXew2P+UNFzSQEm/kTS/1j2czMzMekq9v4M6JCLeAMYBLwI7Af/WsKjMzKzPqzdBtQ0IewQwKSJea1A8ZmZmQP1DHf1C0lPAW8DpklqAJY0Ly8zM+rp6b7dxLrAf0Jpv0/4mcHQjAzMzs76t3iMogF1Jv4eqXOfaHo7HzMwMqP92G9cBOwAzgBV5cuAEZWZmDVLvEVQrsFtERCODMTMza1PvVXwzgXc1MhAzM7NK9R5BjQCekPQgsLRtYkQc1ZCozMysz6s3QV3UyCDMzMyq1ZWgIuIuSdsC746IOyUNBfo3NjQzM+vL6h2L75+Am4Dv5klbAj9rVFBmZmb1XiTxz8D+wBsAEfEMsNnablTSRpJukvSUpCcl7be2ZZmZ2bqp3nNQSyNimSQA8o91u3PJ+TeAX0fEsZIGAUO7UZaZma2D6j2CukvSvwPrSToY+DHwi7XZoKThwPuAHwBExLKIeH1tyjIzs3VXvQnqXGAe8BjwCeA24Py13Ob2uayrJT0s6fuS1q9eSNJpkqZJmjZv3ry13JSZmfVW9Q4Wu5J0UcTpEXFsRHyvG6NKDADGAN+JiL1IA8+eW2ObV0VEa0S0trS0rOWmzNZtc+bMYffddy86DLOG6DBBKblI0nzgKeBpSfMkXdiNbb4IvBgRv8+vbyIlLDMzs1U6O4L6V9LVe3tHxKYRsQmwL7C/pLPWZoMR8WfgBUk750kfAp5Ym7LMepM5c+awyy67cPLJJzN69GiOPfZYFi9e/I5ljj/+eG677bZVr0855RRuvvlm5syZwwEHHMCYMWMYM2YM99133xrlT5w4kTPOOGPV63HjxjF16lQA7rjjDvbbbz/GjBnDcccdx6JFixrzJs16UGcJ6iPAhIh4rm1CRMwGTszz1tangRskPQrsCfxHN8oy6zWefvppTjvtNB599FGGDx/Ot7/97XfMHz9+PJMnTwZg2bJl/OY3v+GII45gs802Y8qUKUyfPp3Jkydz5pln1r3N+fPnc+mll3LnnXcyffp0Wltbueyyy3r0fZk1QmeXmQ+MiPnVEyNinqSBtVaoR0TMII2QbtanbL311uy///4AnHjiiVxxxRWcc845q+YffvjhnHnmmSxdupRf//rXvO9972O99dZjwYIFnHHGGcyYMYP+/fsza9asurf5wAMP8MQTT6za7rJly9hvP//00MqvswS1bC3nmVkNbb8lbLNgwQL23HNPAC655BKOOuooDjzwQG6//XYmT57MhAkTALj88svZfPPNeeSRR1i5ciVDhgxZo+wBAwawcuXKVa+XLFkCQERw8MEHM2nSpEa9LbOG6KyL728lvVHjsRDYoxkBmq1Lnn/+ee6//34AJk2axLhx45gxYwYzZszgqKPSzQHGjx/P1VdfzT333MOhhx4KpES2xRZb0K9fP6677jpWrFixRtmjRo1ixowZrFy5khdeeIEHH3wQgLFjx3Lvvffy7LPPArB48eIuHYGZFaXDBBUR/SNieI3HBhGx1l18Zn3VrrvuyjXXXMPo0aN57bXX+NSnPrXGMocccgh33303Bx10EIMGDQLg9NNP55prrmHs2LHMmjWL9ddf46eD7L///my33XbssccenHPOOYwZky6ObWlpYeLEiUyYMIHRo0czduxYnnrqqca+UbMeoN5wk9zW1taYNm3aWq9/+ZTVe4tnHbxTT4Rk65BmtY85c+Ywbtw4Zs6c2bBtmPUGkh6KiE6vQ6h3JAkzM7OmcoIya5JRo0Z16+hp4cKFHHnkkZx99tk9GJVZeTlBmfUSc+fO5dZbb+WGG24oOhSzpnCCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUnKCMjOzUupsLD6zPqXyR7tF8g/KzZygzNbaA7Pns8GQnhvxa8SwwWw+fM1BYM36Kicos7X0xlvLAXW6XL16MtmZrQucoKzPW9vutBf/upitNh7aY3E8/9rizhcy60N8kYSZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZWSE5SZmZVSYQlKUn9JD0u6tagYzMysvIo8gvoX4MkCt29mZiVWSIKStBXw98D3i9i+mZmVX1FHUP8NfBZY2d4Ckk6TNE3StHnz5jUvMjMzK4WmJyhJ44C5EfFQR8tFxFUR0RoRrS0tLU2KzszMyqKII6j9gaMkzQFuBD4o6foC4jAzsxJreoKKiPMiYquIGAWMB34bESc2Ow4zMys3327DrIRq3dl3/svPA7B42fKm3PnXd/W1ohWaoCJiKjC1yBjMymLhkrd5oYN7Qr3++lsArFgZHS7XXRsOHchw3zzRSsBHUGYlsXDJchYuWd7u/EULlwCwMuClnKwaoX8/OUFZKThBmZXE2O037XD+/CFvchswsL86XXZtPTD71YaUa7Y2PBafmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVksfiMyu5lStWsGzJWyx9600AIoIlby4CwZChwwqOzqxxnKDMSu5nV36Ze2+5gX79+9N/wECWvLmQC44by4oVy/nEl77Pzq1/V3SIZg3hBGVWciO335mBgwezbMk7b7ExYNBgNttm+4KiMms8n4MyK7kdRu9DRKwxfcjQYWy82cgCIjJrDicos5IbMXIbBgwctMb0Hffct4BozJrHCcqs5CSxw+i93zFt0HpD2W2f9xcUkVlzOEGZ9QK77fsBBg1Zb9XrlStWsMPofQqMyKzxnKDMeoHq81A+/2R9gROUWS9QfR7K55+sL3CCMusFKs9D+fyT9RVNT1CStpb0v5KelPS4pH9pdgxmvVHbeSiff7K+oogf6i4HPhMR0yVtADwkaUpEPFFALGa9xg6j9+HtZUtZf/jGPv9kfULTj6Ai4pWImJ6fLwSeBLZsdhxmvc2IkdswZOgwn3+yPqPQoY4kjQL2An5fY95pwGkA22yzTVPjMisjSYz54JHsPOa9RYdi1hSFJShJw4CbgX+NiDeq50fEVcBVAK2trWuO82LWBx1zxoVFh2DWNIVcxSdpICk53RARPykiBjMzK7ciruIT8APgyYi4rNnbNzOz3qGII6j9gZOAD0qakR9HFBCHmZmVWNPPQUXE7wA1e7tmZta7+IaFZlbT5VNmFR0CAGcdvFPRIVhBnKDM7B3+8sYSXl/8dtFhMHKjIWw0dM37YFnf4QRlZu+wdPlKli5fWXQYtKwYXHQIVjAnKDNbZez2mxYdAgAPzH616BCsBDyauZmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZITlJmZlZLH4jOzUlq2YiVvvb2C/7jtyULjGNS/H/37ybf9KIATlJmV0guvLeaF1xYXHQY7v2sDNvZtPwrhLj4zMyslH0GZWen4th8GTlBmZh2av2gpby5dzmd+NKPQOFo2GMzgAf371LkwJygzsw68umhZ0SEAsOF6Axk8oH/RYTSVE5SZWS9y+ZRZRYcA0JQjOScoM7N2lOlc2HPz36R/PxUdCtuNWJ+hg5qTOpygzMx6gcXLVhQdAgArV0bTtlVIgpJ0GPANoD/w/Yj4ShFxmJn1BmU6kmumpv8OSlJ/4P8BhwO7ARMk7dbsOMzMrNyKOILaB3g2ImYDSLoROBp4ooBYzNba5sOHsNnwwUWHYdY06w8agJp4GqyIBLUl8ELF6xeBfasXknQacFp+uUjS093c7ghg/tndLKRAI4D5RQexlnpz7OD4i+b4i1Uz/m5+l25bz0JFJKha+XeNs24RcRVwVY9tVJoWEa09VV6z9eb4e3Ps4PiL5viLVWT8RYzF9yKwdcXrrYCXC4jDzMxKrIgE9Qfg3ZK2kzQIGA/cUkAcZmZWYk3v4ouI5ZLOAG4nXWb+w4h4vAmb7rHuwoL05vh7c+zg+Ivm+ItVWPyKaN6PrszMzOrl+0GZmVkpOUGZmVkp9coEJekwSU9LelbSuTXmD5Y0Oc//vaRRFfPOy9OflnRovWWWIX5JB0t6SNJj+e8HK9aZmsuckR+blTD+UZLeqojxyop13pPf17OSrpAa93PAbsR/QkXsMyStlLRnnteU+q8j9vdJmi5puaRjq+adLOmZ/Di5YnqZ6r5m/JL2lHS/pMclPSrp+Ip5EyU9V1H3e5Yt/jxvRUWMt1RM3y63s2dyu2vY/eW7Uf8fqGr7SyT9Q57XuPqPiF71IF1Y8Udge2AQ8AiwW9UypwNX5ufjgcn5+W55+cHAdrmc/vWUWZL49wJG5ue7Ay9VrDMVaC15/Y8CZrZT7oPAfqTfyf0KOLxs8Vctswcwu5n1X2fso4DRwLXAsRXTNwFm578b5+cbl7Du24t/J+Dd+flI4BVgo/x6YuWyZaz/PG9RO+X+CBifn18JfKqM8Ve1pdeAoY2u/954BLVqqKSIWAa0DZVU6Wjgmvz8JuBDea/waODGiFgaEc8Bz+by6imz8Pgj4uGIaPvN2OPAEEnNHmunO/Vfk6QtgOERcX+kFn8t8A89HzrQc/FPACY1KMb2dBp7RMyJiEeBlVXrHgpMiYjXIuKvwBTgsLLVfXvxR8SsiHgmP38ZmAu0NCjO9nSn/mvK7eqDpHYGqd2Vrv6rHAv8KiIWNyjOVXpjgqo1VNKW7S0TEcuBBcCmHaxbT5k9pTvxVzoGeDgillZMuzofYl/QwG6a7sa/naSHJd0l6YCK5V/spMye0lP1fzxrJqhG13932mlHbb9Mdd8pSfuQjgD+WDH5S7nr7/IG7rR1N/4hkqZJeqCte4zUrl7P7WxtyuyKnvqeG8+abb8h9d8bE1Q9QyW1t0xXpzdCd+JPM6W/Ab4KfKJi/gkRsQdwQH6c1M0429Od+F8BtomIvUhDef2PpOF1ltlTeqL+9wUWR8TMivnNqP/u1FNvafsdF5CO+K4DPhoRbXv55wG7AHuTup8+150gO9p8jWldiX+bSEMG/SPw35J26IEyu6Kn6n8P0u9Y2zSs/ntjgqpnqKRVy0gaAGxI6jNtb91mDr/UnfiRtBXwU+AjEbFqDzIiXsp/FwL/QzqcL1X8uWv11RznQ6Q94J3y8lt1UmZP6Vb9Z2vsQTap/rvTTjtq+2Wq+3blnZlfAudHxANt0yPilUiWAldTbNtvV1v3fKQ7OUwlnVOeD2yU21mXy+yinvie+zDw04h4u21CI+u/NyaoeoZKugVou0rpWOC3uX/9FmC80lVa2wHvJp0gbubwS2sdv6SNSB/Q8yLi3raFJQ2QNCI/HwiMA2bSGN2Jv0XpfmBI2p5U/7Mj4hVgoaSxuWvsI8DPyxZ/jrsfcByp/548rVn13512ejtwiKSNJW0MHALcXsK6rykv/1Pg2oj4cdW8LfJfkc7fFNn2a8r1Pjg/HwHsDzyR29X/ktoZpHZXuvqvsMa514bWfyOuvGj0AzgCmEXaA/98nnYJcFR+PgT4MekiiAeB7SvW/Xxe72kqrlaqVWbZ4gfOB94EZlQ8NgPWBx4CHiVdPPENoH8J4z8mx/cIMB04sqLMVlLD/iPwLfIoJ2WKP887EHigqrym1X8dse9N2lN+E3gVeLxi3Y/l9/QsqYusjHVfM37gRODtqra/Z573W+Cx/B6uB4aVMP735hgfyX9PrShz+9zOns3tbnDZ4s/zRgEvAf2qymxY/XuoIzN/g2Z+AAAFK0lEQVQzK6Xe2MVnZmZ9gBOUmZmVkhOUmZmVkhOUmZmVkhOUmZmVkhOUFUarR3d+XNIjks7OvzNCUqukK/LzwZLuzMseL+mAvM4MSesV+y5qk3SJpIOKjqNeSn6bfwzbiPKnSmqtc9mvqWKkfuu7mn7Ld7MKb0VE2+0qNiONwLAh8IWImAZMy8vtBQysWPZK4GsRcXU9G8k/IFSsHhqn4SLiwmZtq4ccATwSEW8UHQjwTeB7pN/XWB/mIygrhYiYC5wGnJH35g+UdGtOXNcDe+Yjpk+Qhlu5UNINAJL+TdIf8mCVF+dpoyQ9KenbpB8Fby3pEKV7Ck2X9GNJw/KycyRdnKc/JmmXPH2YpKvztEclHZOn1yynktI9co7tqPyq5U+R9DNJv1C6t84Z+YjyYaXBRTfJy+0g6ddK9wO7pyLWI5XuKfRwPtrcPE+/SNIP8xHMbElntvMvOIE8goGkz7YtpzT452/z8w9Jur6jOlC6t9RdOb7b20YZqHif/SRdI+lSSf1zPc3M9XJWbgt/AjaV9K5Omo2t45ygrDQijVHWjzQ6Rtu0ucDHgXsiYs+I+C5peJZ/i4gTJB1CGjJpH2BP4D2S3pdX35k0NM5epF/Gnw8cFBFjSEdnZ1dsfn6e/h3gnDztAmBBROwREaOB3yoNU9NROe2pVX613UkDie4DfIk0IO1ewP2kIYgArgI+HRHvyeV8O0//HTA2L38j8NmKcnch3W5jH+ALSsMxVdufNBoGwN2kAW8hjTIxLK/zd8A97dVBXuabpHsDvQf4YX4fbQYANwCzIuJ80v9ry4jYPdJAu5VHxNNzTNaHuYvPyqart6k4JD8ezq+HkRLW88CfYvWgomNJN6y8N/X4MYj0xd/mJ/nvQ8D/zc8PIo1XBkBE/FXSuE7KaU+t8qv9b6TBZhdKWgD8Ik9/DBidj1LeC/xYq+/m0XZrg62AyfmIZRDwXEW5v4w0kOdSSXOBzXnnLTYANsnbbovxPZI2AJaSkkUrKWmdSft1uTMpyU7J0/uTRrBv813gRxHRlrRmA9tL+iZpjMk7KpadS7oxofVhTlBWGkoDyK4gfTntWu9qwJfzkVVlWaNIR02Vy02JiAntlNN2X60VrP5ciNq34uionPbUKr+9ZSDdMG5pxfMBpKPL19vOxVX5JnBZRNwi6UDgonbKbW/7yyX1i4iVEfG2pDnAR4H7SGMMfgDYAXgy/12jDiTtQRq7bb923t99wAckfT0iluSE/7eko7t/JnXdfiwvOwR4q51yrI9wF5+VgqQW0u2uvxVdGyDyduBjFedAtsznrao9AOwvace83FBJO3VS9h3AGRUxbryW5fSIfAHDc5KOy9tW/oKHdHHJS/n5ybXW78TTpEFL29xN6kK8G7gH+CQwI/9v2quDp4EWSfvl6QOV7l3W5gfAbaQjwLYR4PtFxM2k7tQxFcvuRONGJbdewgnKirSe8mXmwJ2khHBxVwqIiDtIV//dL+kx0q2zN6ix3DzgFGCSpEdJX7JrXKxQ5VJg43wS/xHgA2tZTk86ATg1x/M4q2/ZfRHpi/8e0j2GuuqXpJHa29wDbAHcHxF/AZbkae3WZaTbiB8LfDXHN4PUJblKRFxG6jK8jnQ316mSZgATSTe+a7tlyY6svorT+iiPZm5mbff0uTYiDi5BLP8HGBMRFxQdixXLR1BmRqQbF35PDfqhbhcNAL5edBBWPB9BmZlZKfkIyszMSskJyszMSskJyszMSskJyszMSskJyszMSun/A/tcoroInus/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf23d84198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "patch = plot_test_stats(test_stat_dist)\n",
    "low = actual\n",
    "high = np.max(test_stat_dist)\n",
    "fill_hist(low, high, patch)\n",
    "annotate('p-value', 1.2*actual, 2, 3.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this test statistic, the p-value is around 15%, which means is would not be uncommon to see an absolute difference as big as 13 hours, even if there is actually no difference between the groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An object-oriented framework\n",
    "\n",
    "We can organize the code from the previous sections in a class definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HypothesisTest(object):\n",
    "    \"\"\"Represents a hypothesis test.\"\"\"\n",
    "\n",
    "    def __init__(self, data):\n",
    "        \"\"\"Initializes.\n",
    "\n",
    "        data: data in whatever form is relevant\n",
    "        \"\"\"\n",
    "        self.data = data\n",
    "        self.actual = self.test_stat(data)\n",
    "        self.test_stats = np.array([self.test_stat(self.run_model())\n",
    "                                    for i in range(1000)])\n",
    "\n",
    "    def p_value(self):\n",
    "        \"\"\"Computes the p-value.\n",
    "\n",
    "        returns: float p-value\n",
    "        \"\"\"\n",
    "        return np.mean(self.test_stats >= self.actual)\n",
    "\n",
    "    def plot_test_stats(self):\n",
    "        \"\"\"Draws a Cdf with vertical lines at the observed test stat.\n",
    "        \"\"\"\n",
    "        patch = plot_hist(self.test_stats)\n",
    "        low = self.actual\n",
    "        high = np.max(self.test_stats)\n",
    "        fill_hist(low, high, patch)\n",
    "        plt.title('Distribution of test stat under null')\n",
    "        \n",
    "    def test_stat(self, data):\n",
    "        \"\"\"Computes the test statistic.\n",
    "\n",
    "        data: data in whatever form is relevant        \n",
    "        \"\"\"\n",
    "        raise UnimplementedMethodException()\n",
    "\n",
    "    def run_model(self):\n",
    "        \"\"\"Run the model of the null hypothesis.\n",
    "\n",
    "        returns: simulated data\n",
    "        \"\"\"\n",
    "        raise UnimplementedMethodException()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `HypothesisTest` provides implementations for some methods, because they are the same for all hypothesis tests.\n",
    "\n",
    "But it leaves some methods unimplemented.  We can write child classes to define the unimplemented methods.\n",
    "\n",
    "For example, `PermutationTest` implements `run_model` by combining the groups and shuffling them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PermutationTest(HypothesisTest):\n",
    "    \"\"\"Hypothesis test based on permutation.\"\"\"\n",
    "\n",
    "    def run_model(self):\n",
    "        \"\"\"Run the model of the null hypothesis.\n",
    "\n",
    "        returns: simulated data\n",
    "        \"\"\"\n",
    "        n = len(self.data[0])\n",
    "        pool = np.hstack(self.data)\n",
    "        np.random.shuffle(pool)\n",
    "        data = np.split(pool, [n])\n",
    "        return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DiffMeansPermute` extends `PermutationTest` and provides `TestStat`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DiffMeansPermute(PermutationTest):\n",
    "    \"\"\"Tests a difference in means by permutation.\"\"\"\n",
    "\n",
    "    def test_stat(self, data):\n",
    "        \"\"\"Computes the test statistic.\n",
    "\n",
    "        data: data in whatever form is relevant        \n",
    "        \"\"\"\n",
    "        group1, group2 = data\n",
    "        return abs(group1.mean() - group2.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can test the absolute difference in means by permutation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Diff means permute pregnancy length\n",
      "actual = 0.07803726677754952\n",
      "p-value = 0.163\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmYXFW97vHvSxIIIQQCaUAGCUECIihoqyhHDwhRVBDvhasgyOjFCT2CEwgiOFxnEEfMUWYFBVFQUQl6GFRQAgQBI4NhClM6jAEywu/+sVaFotJDdbqr9qr0+3meerp6D2v/snun3lq7dq2tiMDMzKw0q1VdgJmZWW8cUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeU9UnSaZI+O0xtvVjSU5JG5d+vkPS+4Wg7t/c7SQcPV3uD2O4XJc2X9FC7t72qkHSIpD9XXQeApBMlnZufT5YUkkZXXddI5YAaoSTdLWmhpAWSHpf0V0kfkLT8mIiID0TEF5psa/f+lomIeyNifEQ8Owy1L38RqWv/rRFx1lDbHmQdmwEfB7aNiI16mb+LpLnDtK2VDvTBrtvb/jWrggNqZNsrItYGNge+Anwa+PFwb2QVfge6OfBIRMyrupCRqtYjt1VURPgxAh/A3cDuDdNeAzwHbJd/PxP4Yn4+CfgN8DjwKHA16Q3OOXmdhcBTwKeAyUAAhwP3AlfVTRud27sC+DLwd+AJ4GJgvTxvF2Bub/UCewBLgKV5ezfVtfe+/Hw14HjgHmAecDawTp5Xq+PgXNt84Lh+9tM6ef2e3N7xuf3d87/5uVzHmQ3rrdUw/ylg47zuMcC/gUeAn9f9u8cC5+bpjwPXARsCXwKeBRbldr7bS52DWhc4FbgPeBK4HnhDnt7r/u1lewG8pO73M3n+WNkFmEvqXc4DHgQOrVt2feCSvO2/A18A/lw3fxtgBuk4uw14V8N2fgBcCjxNwzFcdyx8AfgLsAC4DJg00LGVn58InNtwrIyu+v/rSH24B2XLRcTfSS8sb+hl9sfzvC7SC99n0irxXtIL/V6RTuF9rW6d/wReCrylj00eBBxGeuFeBny7iRp/D/w/4Gd5e6/oZbFD8mNXYAowHvhuwzL/AWwN7AacIOmlfWzyO6SQmpL/PQeRXmwvB94KPJDrOKShzqcb5o+PiAeAjwLvzG1tDDwGfC+vdnDe1makF/EPAAsj4jjSG4IjcztH9lLnYNe9DtgBWA/4KXCBpLFN7t9mbJTr2YT0RuV7kibmed8jBeaLSH//w2orSVqLFE4/BTYA9ge+L+lldW2/hxS8awN9fXb1HuDQ3MbqwCdW8t9hFXJAWaMHSC9ajZaSXlA2j4ilEXF15LeZ/TgxIp6OiIV9zD8nIm7JL+afBd41TKdsDgBOjog5EfEUcCywX8OpxpMiYmFE3ATcBKzwQpxreTdwbEQsiIi7gW8C7x1Cbe8n9djmRsRi0jv2fXNtS0nh8pKIeDYiro+IJ5tsd1DrRsS5EfFIRCyLiG8Ca5ACe7gsBT6fj5VLSb2xrfM+3Qc4IR8btwD1nx3uCdwdEWfk2m4AfgHsW7fMxRHxl4h4LiIW9bH9MyLi9nzs/ZwUxtZhHFDWaBPSqZVGXwfuBC6TNEfSMU20dd8g5t8DjCGdShyqjXN79W2PJvX8auqvunuG1MtqNIn07ruxrU2GUNvmwC/zhSmPA7NJp+A2JJ0u/QNwvqQHJH1N0pgm2x3UupI+Lmm2pCdyHeswPPu+5pGIWFb3e20fd5H+Fo1/+5rNgdfW9k+u7QBSj6xmoOMKmvv7WuEcULacpFeTXnxXOG2SexAfj4gpwF7A0ZJ2q83uo8mBelib1T1/Meld93zSZwvj6uoaRXpha7bdB0gvdPVtLwMeHmC9RvNzTY1t3d/k+r3VeR/w1ohYt+4xNiLuz72NkyJiW+D1pN7EQf209fyGBrGupDeQLoh5FzAxItYlfQ6oZraVPUPd34gXBkh/ekh/i8a/fc19wJUN+2d8RHywbpmh3IJhoGPLCuKAMiRNkLQncD7pA+Kbe1lmT0kvkSTSh9vP5gekF/4pK7HpAyVtK2kc8HngwkiXod8OjJX09twLOJ50CqrmYWBy/SXxDc4DjpK0haTxPP+ZyrI+lu9VruXnwJckrS1pc+Bo0sUIzXgYWF/SOnXTTsvtbQ4gqUvS3vn5rpK2zy+aT5LCsal9PMh11yaFRA8wWtIJwISGuvvbvwCzgPdIGiVpD9JnagPK+/Qi4ERJ4yRtS/r8rOY3wFRJ75U0Jj9e3c9nhIM10LFlBXFAjWy/lrSA9K71OOBk0gfLvdkKuJz0WcI1wPcj4oo878vA8fmUzGA+jD6HdFXWQ6Sr0D4KEBFPAB8CfkTqrTxNukCj5oL88xFJN/TS7um57auAu0gfyH9kEHXV+0je/hxSz/Knuf0BRcS/SGE5J++bjUlXz11COlW6ALgWeG1eZSPgQlLAzAau5PkwPJX0WdVjknq7mGQw6/4B+B3pxfoe0v6pP2020P4F+C9ST7p2Cu5XA++R5Y4knXJ7iPT3P6M2IyIWAG8G9iP1hB8CvsowhUgTx5YVRAN/zm1mZtZ+7kGZmVmRHFBmZlYkB5SZmRXJAWVmZkVqWUBJOl3SPEm39DLvE3kY++H8YqCZma1CWjnK9Jmk8c/Orp+Yb1EwjTR+W1MmTZoUkydPHs7azMysItdff/38iBjwC9ItC6iIuErS5F5mnUIa8friZtuaPHkyM2fOHKbKzMysSpLuGXipNn8GJekdwP15gM6Blj1C0kxJM3t6etpQnZmZlaRtAZWHszkOOKGZ5SNiekR0R0R3V5eHyjIzG2na2YPaEtgCuEnS3cCmwA2Smh1k0szMRpC23Yo7D0C6Qe33HFLdETG/XTWYmVnnaOVl5ueRBhXdWtJcSYe3altmZrbqaeVVfPsPMH9yq7ZtZmadzyNJmJlZkRxQZmZWJAeUmZkVqW1X8VXplBm3V13CCo6aNrXqEszMiuYelJmZFckBZWZmRRoRp/jqVXlqrcRTjWZmpXIPyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMr0oi7YWEpSrl5YZU3cDQz6497UGZmViQHlJmZFallp/gknQ7sCcyLiO3ytK8DewFLgH8Dh0bE462qoTSlnE4r5fSimVl/WtmDOhPYo2HaDGC7iHg5cDtwbAu3b2ZmHaxlARURVwGPNky7LCKW5V+vBTZt1fbNzKyzVfkZ1GHA7/qaKekISTMlzezp6WljWWZmVoJKAkrSccAy4Cd9LRMR0yOiOyK6u7q62lecmZkVoe3fg5J0MOniid0iItq9fTMz6wxtDShJewCfBv4zIp5p57bNzKyztOwUn6TzgGuArSXNlXQ48F1gbWCGpFmSTmvV9s3MrLO1rAcVEfv3MvnHrdqemZmtWjyShJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFallASXpdEnzJN1SN209STMk3ZF/TmzV9s3MrLO1sgd1JrBHw7RjgD9GxFbAH/PvZmZmK2hZQEXEVcCjDZP3Bs7Kz88C3tmq7ZuZWWdr92dQG0bEgwD55wZt3r6ZmXWIYi+SkHSEpJmSZvb09FRdjpmZtVm7A+phSS8CyD/n9bVgREyPiO6I6O7q6mpbgWZmVoZ2B9QlwMH5+cHAxW3evpmZdYhWXmZ+HnANsLWkuZIOB74CTJN0BzAt/25mZraC0a1qOCL272PWbq3appmZrTqKvUjCzMxGNgeUmZkVyQFlZmZFckCZmVmRWnaRhHWGU2bcXnUJABw1bWrVJZhZYdyDMjOzIjmgzMysSD7FNwKVcjqtlNOLZlYm96DMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK1IlASXpKEm3SrpF0nmSxlZRh5mZlavtASVpE+CjQHdEbAeMAvZrdx1mZla2qk7xjQbWlDQaGAc8UFEdZmZWqKYCStIvJL1d0pADLSLuB74B3As8CDwREZcNtV0zM1u1NBs4PwDeA9wh6SuStlnZDUqaCOwNbAFsDKwl6cBeljtC0kxJM3t6elZ2c2Zm1qGaCqiIuDwiDgBeCdwNzJD0V0mHShozyG3uDtwVET0RsRS4CHh9L9ucHhHdEdHd1dU1yE2YmVmna/qUnaT1gUOA9wE3AqeSAmvGILd5L7CTpHGSBOwGzB5kG2Zmtoob3cxCki4CtgHOAfaKiAfzrJ9JmjmYDUbE3yRdCNwALCOF3fTBtGFmZqu+pgIK+FFEXFo/QdIaEbE4IroHu9GI+BzwucGuZ2ZmI0ezp/i+2Mu0a4azEDMzs3r99qAkbQRsQvrO0o6A8qwJpO8vmZmZtcRAp/jeQrowYlPg5LrpC4DPtKgmMzOz/gMqIs4CzpK0T0T8ok01mZmZDXiK78CIOBeYLOnoxvkRcXIvq5mZmQ3ZQKf41so/x7e6EDMzs3oDneL7Yf55UnvKMTMzS5odLPZrkiZIGiPpj5Lm9zZ+npmZ2XBp9ntQb46IJ4E9gbnAVOCTLavKzMxGvGZHkqgNCPs24LyIeDQNo2c2PE6ZcXvVJQBw1LSpVZdgZlmzAfVrSf8CFgIfktQFLGpdWWatdfWdPUwYm953rb/W6mw60d87NytNs7fbOAZ4Hek27UuBp0n3dDLrSAsXP8vCJemxZNlzVZdjZr1otgcF8FLS96Hq1zl7mOuxEaTK02lzH3tmea9pTs9TldVhZn1r9nYb5wBbArOAZ/PkwAFlZmYt0mwPqhvYNiKilcWYmZnVNHuZ+S3ARq0sxMzMrF6zPahJwD8l/R1YXJsYEe9oSVVmZjbiNRtQJ7ayCDMzs0ZNBVREXClpc2CriLhc0jhgVGtLMzOzkazZsfj+L3Ah8MM8aRPgV60qyszMrNmLJD4M7Aw8CRARdwAbtKooMzOzZgNqcUQsqf2Sv6zrS87NzKxlmg2oKyV9BlhT0jTgAuDXrSvLzMxGumYD6higB7gZeD9wKXB8q4oyMzNr9iq+5yT9CvhVRPS0uCYzM7P+e1BKTpQ0H/gXcJukHkkntKc8MzMbqQY6xfcx0tV7r46I9SNiPeC1wM6SjlrZjUpaV9KFkv4labak161sW2ZmtmoaKKAOAvaPiLtqEyJiDnBgnreyTgV+HxHbAK8AZg+hLTMzWwUN9BnUmIiY3zgxInokjelthYFImgC8ETgkt7UEWNLfOmZmNvIM1IPqLzhWNlSmkK4IPEPSjZJ+JGmtlWzLzMxWUQMF1CskPdnLYwGw/UpuczTwSuAHEbEj6fbxxzQuJOkISTMlzezp8YWDZmYjTb8BFRGjImJCL4+1I2KlTvEBc4G5EfG3/PuFpMBq3Pb0iOiOiO6urq6V3JSZmXWqZr+oO2wi4iHgPklb50m7Af9sdx1mZla2Zu8HNdw+AvxE0urAHODQiuowM7NCVRJQETEL6K5i22Zm1hnaforPzMysGQ4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrUmUBJWmUpBsl/aaqGszMrFxV9qD+C5hd4fbNzKxglQSUpE2BtwM/qmL7ZmZWvqp6UN8CPgU819cCko6QNFPSzJ6envZVZmZmRWh7QEnaE5gXEdf3t1xETI+I7ojo7urqalN1ZmZWiip6UDsD75B0N3A+8CZJ51ZQh5mZFWx0uzcYEccCxwJI2gX4REQc2O46zGoWLX2OngWLAfjMRTdXVse41Uex1hrpv+RR06ZWVodZKdoeUGaleXLRUp5ctLTqMth43TWXB5SZVRxQEXEFcEWVNZiZWZn8ds1GvJ2mrF91CQDc++gzVZdgVhQPdWRmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlak0VUXYGYrOmXG7VWXwFHTplZdgo1wbQ8oSZsBZwMbAc8B0yPi1HbXYVaaBYuWct+jz1RdBuuMG8OEsWOqLsOskh7UMuDjEXGDpLWB6yXNiIh/VlCLWTEWLFrGgkXLqi6DUavJAWVFaHtARcSDwIP5+QJJs4FNAAeUjWg7TVm/6hK4ds4jVZdgtlylF0lImgzsCPytyjrMzKw8lQWUpPHAL4CPRcSTvcw/QtJMSTN7enraX6CZmVWqkoCSNIYUTj+JiIt6WyYipkdEd0R0d3V1tbdAMzOrXNsDSpKAHwOzI+Lkdm/fzMw6QxU9qJ2B9wJvkjQrP95WQR1mZlawKq7i+zOgdm/XzMw6i4c6MjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEhV3FHXzAr29JJlzH9qMcf98uZK6xi/xmjGjhnFUdOmVlqHVccBZWYv8MhTS3jkqSVVl8GWXeMZO2ZU1WVYhXyKz8zMiuQelJktt9OU9asuAYBr5zxSdQlWAPegzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgeScLMivT0kmWMfkaccPEtldYxfo3RjBm1mgetrYADysyK9NATi3joiUVVl8HWG63NxHGrV13GiOSAMjNrwikzbq+6BIAR1ZOrJKAk7QGcCowCfhQRX6miDjMrU0mD1t7/2ELmPbm46lJ48frjWHOE3X6k7QElaRTwPWAaMBe4TtIlEfHPdtdiZjaQpxYvq7oEADZedyyMGTWienJV9KBeA9wZEXMAJJ0P7A04oMysKCX15GY/uACp6krgpRutzfixY9qyrSoCahPgvrrf5wKvraAOG8E2nDCWDSasUXUZZk1Za/VyLhdoZ0hW8a/u7Z8XKywkHQEckX99StJtQ9zuJGD+0UNspA0mAfOrLqJJnVJrp9QJnVNrp9QJnVNrp9QJMOnoodW6eTMLVRFQc4HN6n7fFHigcaGImA5MH66NSpoZEd3D1V6rdEqd0Dm1dkqd0Dm1dkqd0Dm1dkqd0L5aqxhJ4jpgK0lbSFod2A+4pII6zMysYG3vQUXEMklHAn8gXWZ+ekTc2u46zMysbJV88hYRlwKXtnmzw3a6sMU6pU7onFo7pU7onFo7pU7onFo7pU5oU62KWOH6BDMzs8p5NHMzMytSRwaUpD0k3SbpTknH9DJ/DUk/y/P/Jmly3bxj8/TbJL2l2TbbXaukaZKul3Rz/vmmunWuyG3Oyo8NKqxzsqSFdbWcVrfOq3L9d0r6tjQ836AYQq0H1NU5S9JzknbI86rYp2+UdIOkZZL2bZh3sKQ78uPguunDvk9Xtk5JO0i6RtKtkv4h6d11886UdFfd/txhqHUOpdY879m6ei6pm75FPk7uyMfNkEeGHcI+3bXhGF0k6Z15XlX79GhJ/8x/4z9K2rxuXmuP04joqAfpwop/A1OA1YGbgG0blvkQcFp+vh/ws/x827z8GsAWuZ1RzbRZQa07Ahvn59sB99etcwXQXcg+nQzc0ke7fwdeR/ru2++At1ZZa8My2wNzKt6nk4GXA2cD+9ZNXw+Yk39OzM8ntmKfDrHOqcBW+fnGwIPAuvn3M+uXrXqf5nlP9dHuz4H98vPTgA9WWWfDcfAoMK7ifbprXQ0f5Pn/+y0/TjuxB7V8qKSIWALUhkqqtzdwVn5+IbBbTvC9gfMjYnFE3AXcmdtrps221hoRN0ZE7fthtwJjJbVq6IOh7NNeSXoRMCEirol0xJ4NvLOgWvcHzhuGela6zoi4OyL+ATzXsO5bgBkR8WhEPAbMAPZo0T5d6Toj4vaIuCM/fwCYB3QNsZ6W1NqXfFy8iXScQDpuKtunDfYFfhcRzwyxnv40U+v/1NVwLem7q9CG47QTA6q3oZI26WuZiFgGPAGs38+6zbTZ7lrr7QPcGBH1Qyqfkbv5nx2G0zxDrXMLSTdKulLSG+qWnztAm1XUWvNuVgyodu/Twa7bin06LMe+pNeQ3oH/u27yl/JpoVOG6c3VUGsdK2mmpGtrp81Ix8Xj+ThZmTZbUWfNfqx4jFa9Tw8n9Yj6W3fYjtNODKhmhkrqa5nBTh+qodSaZkovA74KvL9u/gERsT3whvx4b4V1Pgi8OCJ2BI4GfippQpNtrozh2KevBZ6JiPpbtVaxTwe7biv26ZDbzO+YzwEOjYhaj+BYYBvg1aRTQJ8eSpG1TfUybTC1vjjS6AfvAb4lacthaLM3w7VPtyd9X7Sm0n0q6UCgG/j6AOsO2z7txIBqZqik5ctIGg2sQzqX29e6TQ2/1OZakbQp8EvgoIhY/s40Iu7PPxcAPyV10yupM58ufSTXcz3pHfTUvPymdesXsU+zFd6ZVrRPB7tuK/bpkI79/Gbkt8DxEXFtbXpEPBjJYuAMhr4/h1xr7ZR5pDspXEH6nHc+sG4+TgbdZivqzN4F/DIiltYmVLlPJe0OHAe8o+5MTuuP0+H8wK0dD9KXi+eQLnKofaj3soZlPswLPyT/eX7+Ml54kcQc0oeEA7ZZQa3r5uX36aXNSfn5GNK58w9UWGcXMCo/nwLcD6yXf78O2InnPyh9W5X7NP++Guk/0JSq92ndsmey4kUSd5E+eJ6Yn7dknw6xztWBPwIf62XZF+WfAr4FfKUdf/t+ap0IrJGfTwLuIF8MAFzACy+S+FBVddZNvxbYtYR9Sgryf5MviGnncTqkf1xVD+BtwO15px2Xp32elO4AY/NBdyfpapL6F6Pj8nq3UXdlSW9tVlkrcDzwNDCr7rEBsBZwPfAP0sUTp5IDoqI698l13ATcAOxV12Y3cEtu87vkL4ZX/PffBbi2ob2q9umrSWH5NPAIcGvduofl+u8knTpr2T5d2TqBA4GlDcfoDnnen4Cbc63nAuPb9Lfvq9bX53puyj8Pr2tzSj5O7szHzRoV/+0nk97ordbQZlX79HLg4bq/8SXtOk49koSZmRWpEz+DMjOzEcABZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckBZZepuf3CrpJvysP6r5Xndkr6dn68h6fK87LslvSGvM0vSmtX+K3on6fP52/cdQcmf8sgQrWj/CkndTS77DdXdXsZGrkpu+W6WLYyI2v2YNiANMbQO8LmImAnMzMvtCIypW/Y04BsRcUYzG8kDvyqeHyeu5SLihHZta5i8DbgpIp6suhDgO8B/k76YaiOYe1BWhIiYBxwBHJnfze8i6Tc5uM4Fdsg9pveTxik7QdJPACR9UtJ1eZTnk/K0yZJmS/o+aYSLzSS9WekGezdIukDS+Lzs3ZJOytNvlrRNnj5e0hl52j8k7ZOn99pOPaWby+3bX/sNyx8i6VeSfq10U7ojc4/yxjz69np5uS0l/V7pJpZX19W6l9JN927Mvc0N8/QTJZ2eezBzJH20jz/BAcDFeZ1P1ZbLo2b/KT/fTdK5/e0DpRvVXZnr+0Me9LT+37mapLMkfVHSqLyfbsn75ah8LNwDrC9powEOG1vFOaCsGJEG8VyNNKRTbdo84H3A1RGxQ0T8ELgE+GREHCDpzcBWpIEzdwBeJemNefWtgbMjjbT+NGn4qN0j4pWk3tnRdZufn6f/APhEnvZZ4ImI2D4iXg78SdKkAdrpS2/tN9qONNL2a4AvkUZc3xG4BjgoLzMd+EhEvCq38/08/c/ATnn584FP1bW7DenePa8BPidpTC/b3pk03BPAVaQR3SENWTM+r/MfwNV97YO8zHdIY8u9Cjg9/ztqRgM/AW6PiONJf69NImK7SCPJ1/eIb8g12QjmU3xWmsHeh+nN+XFj/n08KbDuBe6J50fY3ol0R+W/pDN+rE564a+5KP+8Hvjf+fnupMFmAYiIxyTtOUA7femt/Ub/E2k09QWSngB+naffDLw891JeD1yg529XVbsn0KbAz3KPZXXSwJ01v400AvViSfOADXnh/XogDfK5oK7GV0laG1hMCotuUmh9lL735dakkJ2Rp48i3Y6l5oekgXtroTUHmCLpO6QR0S+rW3Ye6S69NoI5oKwYkqYAz5JenF7a7GrAl3PPqr6tyaReU/1yMyJi/z7aqd1C4Fme/38her/XVH/t9KW39vtaBtKdVhfXPR9N6l0+XvssrsF3gJMj4hJJuwAn9tFuX9tfJmm1iHguIpZKuhs4FPgraRDdXYEtgdn55wr7QNL2pEFPX9fHv++vwK6SvhkRi3Lgv4LUu/sw6dTtYXnZscDCPtqxEcKn+KwIkrpItzr4bgxuBOM/AIfVfQaySf7cqtG1wM6SXpKXGydp6gBtXwYcWVfjxJVsZ1jkCxjukvR/8raVX+AhXVxyf35+8Eo0fxtpVO+aq0inEK8CrgY+AMzKf5u+9sFtQJek1+XpY5RuuFnzY+BSUg9wdD5VuFpE/IJ0OvWVdctOJY2GbSOYA8qqtKbyZeakIf0vA04aTAMRcRnp6r9rJN1MupfT2r0s1wMcApwn6R+kF9kVLlZo8EVgYv4Q/ybS/XlWpp3hdABweK7nVmDvPP1E0gv/1aSb8A3Wb0m3Iqm5GngRcE1EPAwsytP63JcRsQTYF/hqrm8W6ZTkchH0qaZOAAAAdElEQVRxMumU4Tmk24BfIWkW6b5Ix0IKNuAlPH8Vp41Qvt2GmdVuMX52REwroJb/BbwyIj5bdS1WLfegzIyIeBD4b7Xoi7qDNBr4ZtVFWPXcgzIzsyK5B2VmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRfr/M1JRHwyvlDEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf4dc2ef28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ht = DiffMeansPermute(data)\n",
    "p_value = ht.p_value()\n",
    "\n",
    "print('Diff means permute pregnancy length')\n",
    "print('actual =', ht.actual)\n",
    "print('p-value =', p_value)\n",
    "ht.plot_test_stats()\n",
    "plt.xlabel('Difference in mean (weeks)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different test statistic\n",
    "\n",
    "It's easy to run a test with a different test statistic.  For example, if we think that first babies might be more variable than others, we could use the difference in standard deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DiffStdPermute(PermutationTest):\n",
    "    \"\"\"Tests a difference in standard deviation by permutation.\"\"\"\n",
    "\n",
    "    def test_stat(self, data):\n",
    "        \"\"\"Computes the test statistic.\n",
    "\n",
    "        data: data in whatever form is relevant        \n",
    "        \"\"\"\n",
    "        group1, group2 = data\n",
    "        return abs(group1.std() - group2.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Diff std permute pregnancy length\n",
      "actual = 0.1760490642294399\n",
      "p-value = 0.157\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAH5RJREFUeJzt3XmYJWV59/Hvj5lhX4VxQ2Aggoomio5LYjTgvpu8crmB4vaicQ1q8mpQxCWLGjXGJYobKioqrlFUxii4gTIgqwjqiIKoDJssIsMM9/tHVQ+Hpnv69PRUn+ru7+e6zjV1anme+zynpu7zVFU/lapCkqS+2WzUAUiSNBETlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQuoUk70vy2k1U1u5Jrk2yqH1/YpLnbYqy2/K+luSQTVXeNOp9U5LLkvxutuueL5I8K8n3Rh0HQJIjkxzTTi9LUkkWjzoumaAWlCQXJrk+yTVJrkrygyQvSLJ+P6iqF1TVG4cs62EbWqeqfl1V21bVuk0Q+/qDyED5j66qj8607GnGsRvwCmDfqrr9BMv3T3LxJqproxP6dLedqH2lUTNBLTyPr6rtgD2Afwf+H/ChTV3JPP4FugdweVVdOupAFqqxHrkWgKrytUBewIXAw8bNux9wE3CP9v3RwJva6V2ArwBXAVcA36X5UfPxdpvrgWuBfwKWAQU8F/g18J2BeYvb8k4E/g34EfAH4EvAbdpl+wMXTxQv8ChgDXBjW9+ZA+U9r53eDHgN8CvgUuBjwA7tsrE4Dmljuww4fAPttEO7/eq2vNe05T+s/cw3tXEcPW67bcYtvxa4Y7vtq4BfAJcDnxn43FsCx7TzrwJOBW4H/AuwDvhTW867J4hzWtsC7wQuAq4GTgMe1M6fsH0nqK+AOw+8P5qb95X9gYtpepeXAr8Fnj2w7s7Al9u6fwS8EfjewPK7Aito9rPzgSePq+e/geOB6xi3Dw/sC28Evg9cA5wA7DLVvtVOHwkcM25fWTzq/6++yh7UQldVP6I5sDxogsWvaJctpTnw/XOzST2D5kD/+GpO4b1lYJu/Ae4GPHKSKp8JPIfmwL0W+K8hYvw68K/Ap9v67jnBas9qXwcAewHbAu8et85fA3cBHgockeRuk1T5LpoktVf7eZ5Jc7D9JvBo4JI2jmeNi/O6ccu3rapLgJcCf9uWdUfgSuA97WaHtHXtRnMQfwFwfVUdTvOD4MVtOS+eIM7pbnsqcC/gNsAngc8m2XLI9h3G7dt4dqX5ofKeJDu1y95DkzDvQPP9P2dsoyTb0CSnTwK3BZ4GvDfJ3QfKfjpN4t0OmOza1dOBZ7dlbA68ciM/h3rCBCWAS2gOWuPdSHNA2aOqbqyq71b7M3MDjqyq66rq+kmWf7yqzmkP5q8FnryJTtkcBLy9qlZV1bXAq4GnjjvV+Pqqur6qzgTOBG51IG5jeQrw6qq6pqouBN4GPGMGsT2fpsd2cVXdQPOL/cA2thtpksudq2pdVZ1WVVcPWe60tq2qY6rq8qpaW1VvA7agSdibyo3AG9p95Xia3thd2jZ9EnBEu2+cAwxeO3wccGFVfaSN7XTgc8CBA+t8qaq+X1U3VdWfJqn/I1V1QbvvfYYmGWsOM0EJml+8V0ww/63Az4ETkqxK8qohyrpoGst/BSyhOZU4U3dsyxssezFNz2/M4F13f6TpZY23C82v7/Fl7TqD2PYAvtDemHIVcB7NKbjb0Zwu/QZwbJJLkrwlyZIhy53WtklekeS8JH9o49iBTdP2Yy6vqrUD78faeCnNdzH+ux+zB3D/sfZpYzuIpkc2Zqr9Cob7fjWHmKAWuCT3pTn43uq0SduDeEVV7QU8Hnh5koeOLZ6kyKl6WLsNTO9O86v7MpprC1sPxLWI5sA2bLmX0BzoBsteC/x+iu3Gu6yNaXxZvxly+4nivAh4dFXtOPDasqp+0/Y2Xl9V+wJ/RdObeOYGyrq5omlsm+RBNDfEPBnYqap2pLkOmGHqav2Rge+IWyaQDVlN812M/+7HXAScNK59tq2qvx9YZyaPXZhq31JPmaAWqCTbJ3kccCzNBeKzJ1jncUnunCQ0F7fXtS9oDvx7bUTVByfZN8nWwBuA46q5Df0CYMskj217Aa+hOQU15vfAssFb4sf5FHBYkj2TbMvN11TWTrL+hNpYPgP8S5LtkuwBvJzmZoRh/B7YOckOA/Pe15a3B0CSpUme2E4fkOTP24Pm1TTJcag2nua229EkidXA4iRHANuPi3tD7QtwBvD0JIuSPIrmmtqU2jb9PHBkkq2T7Etz/WzMV4B9kjwjyZL2dd8NXCOcrqn2LfWUCWrh+Z8k19D8aj0ceDvNheWJ7A18k+ZawsnAe6vqxHbZvwGvaU/JTOdi9Mdp7sr6Hc1daC8FqKo/AC8EPkjTW7mO5gaNMZ9t/708yekTlPvhtuzvAL+kuSD/kmnENeglbf2raHqWn2zLn1JV/ZQmWa5q2+aONHfPfZnmVOk1wCnA/dtNbg8cR5NgzgNO4uZk+E6aa1VXJpnoZpLpbPsN4Gs0B+tf0bTP4GmzqdoX4GU0PemxU3BfnLpF1nsxzSm339F8/x8ZW1BV1wCPAJ5K0xP+HfBmNlESGWLfUk9l6mvekiTNPntQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqRe6tWI07vsskstW7Zs1GFIkjp02mmnXVZVU/6xdK8S1LJly1i5cuWow5AkdSjJr6Zey1N8kqSeMkFJknrJBCVJ6iUTlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqRe6tVIEvPFO1ZcMOoQJnTYw/cZdQiSNDR7UJKkXjJBSZJ6qdNTfEl2BD4I3AMo4DlVdXKXdfbNqE+r9fV0oyRNpetrUO8Evl5VBybZHNi64/okSfNEZwkqyfbAg4FnAVTVGmBNV/VJkuaXLq9B7QWsBj6S5MdJPphkmw7rkyTNI10mqMXAvYH/rqr9gOuAV41fKcmhSVYmWbl69eoOw5EkzSVdJqiLgYur6oft++NoEtYtVNVRVbW8qpYvXTrlE4AlSQtEZwmqqn4HXJTkLu2shwI/6ao+SdL80vVdfC8BPtHewbcKeHbH9UmS5olOE1RVnQEs77IOSdL85EgSkqReMkFJknrJBCVJ6iUTlCSpl0xQkqRemlcPLHTkbkmaP+xBSZJ6yQQlSeqleXWKb9CoHxQoSZoZe1CSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXFndZeJILgWuAdcDaqlreZX2SpPmj0wTVOqCqLpuFejSFd6y4YNQhAHDYw/cZdQiS5gBP8UmSeqnrBFXACUlOS3Jox3VJkuaRrk/xPbCqLklyW2BFkp9W1XcGV2gT16EAu+++e8fhLDx9OZ3Wl9OLkuaOTntQVXVJ+++lwBeA+02wzlFVtbyqli9durTLcCRJc0hnCSrJNkm2G5sGHgGc01V9kqT5pctTfLcDvpBkrJ5PVtXXO6xPkjSPdJagqmoVcM+uypckzW/eZi5J6iUTlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqReMkFJknrJBCVJ6iUTlCSpl0xQkqReMkFJknpp8agD0MLzjhUXjDqE9Q57+D6jDkHSJOxBSZJ6yQQlSeolT/FpVvTpVFqfTjFKmpw9KElSL5mgJEm9ZIKSJPWSCUqS1EsmKElSL3WeoJIsSvLjJF/pui5J0vwxGz2olwHnzUI9kqR5pNMEleROwGOBD3ZZjyRp/um6B/WfwD8BN3VcjyRpnuksQSV5HHBpVZ02xXqHJlmZZOXq1au7CkeSNMd02YN6IPCEJBcCxwIPSXLM+JWq6qiqWl5Vy5cuXdphOJKkuaSzBFVVr66qO1XVMuCpwLeq6uCu6pMkzS/+HZQkqZdmZTTzqjoROHE26pIkzQ/2oCRJvWSCkiT1kglKktRLJihJUi+ZoCRJvWSCkiT1kglKktRLQyWoJJ9L8tgkJjRJ0qwYNuH8N/B04GdJ/j3JXTuMSZKk4RJUVX2zqg4C7g1cCKxI8oMkz06ypMsAJUkL09Cn7JLsDDwLeB7wY+CdNAlrRSeRSZIWtKHG4kvyeeCuwMeBx1fVb9tFn06ysqvgJEkL17CDxX6wqo4fnJFki6q6oaqWdxCXJGmBG/YU35smmHfypgxEkqRBG+xBJbk9sCuwVZL9gLSLtge27jg2SdICNtUpvkfS3BhxJ+DtA/OvAf65o5gkSdpwgqqqjwIfTfKkqvrcLMUkSdKUp/gOrqpjgGVJXj5+eVW9fYLNJEmasalO8W3T/rtt14FIkjRoqlN872//ff3shCNJUmPYwWLfkmT7JEuS/G+Sy5Ic3HVwkqSFa9i/g3pEVV0NPA64GNgH+MfOopIkLXjDJqixAWEfA3yqqq7oKB5JkoDhhzr6nyQ/Ba4HXphkKfCn7sKSJC10wz5u41XAXwLLq+pG4DrgiV0GJkla2IbtQQHcjebvoQa3+dgmjkeSJGD4x218HPgz4AxgXTu7MEFJkjoybA9qObBvVVWXwUiSNGbYu/jOAW7fZSCSJA0atge1C/CTJD8CbhibWVVPmGyDJFsC3wG2aOs5rqpeN4NYJUkLyLAJ6siNKPsG4CFVdW2SJcD3knytqk7ZiLIkSQvMUAmqqk5Ksgewd1V9M8nWwKIpting2vbtkvblNSxJ0lCGHYvv/wLHAe9vZ+0KfHGI7RYlOQO4FFhRVT/c2EAlSQvLsDdJvAh4IHA1QFX9DLjtVBtV1bqquhfNE3nvl+Qe49dJcmiSlUlWrl69evjIJUnz2rAJ6oaqWjP2pv1j3aFP11XVVcCJwKMmWHZUVS2vquVLly4dtkhJ0jw3bII6Kck/A1sleTjwWeB/NrRBkqVJdmyntwIeBvx0JsFKkhaOYRPUq4DVwNnA84HjgddMsc0dgG8nOQs4leYa1Fc2NlBJ0sIy7F18NyX5IvDFqhrqQlFVnQXsN5PgJEkL1wZ7UGkcmeQymtNz5ydZneSI2QlPkrRQTXWK7x9o7t67b1XtXFW3Ae4PPDDJYZ1HJ0lasKZKUM8EnlZVvxybUVWrgIPbZZIkdWKqBLWkqi4bP7O9DrVkgvUlSdokpkpQazZymSRJMzLVXXz3THL1BPMDbNlBPJIkAVMkqKra4ICwkiR1Zdg/1JUkaVaZoCRJvWSCkiT1kglKktRLJihJUi+ZoCRJvWSCkiT1kglKktRLJihJUi8N9cBCab56x4oLZq2uK69bw5LFt/xNuOXizVi8aDMOe/g+sxaHNFeYoKRZcuqFV7D9Vrd8CMDet92WnbfdYkQRSf3mKT5JUi/Zg9KCM6rTaRdf+UfutNPW69+fsurykcQhzRX2oCRJvWSCkiT1kglKktRLJihJUi+ZoCRJvWSCkiT1kglKktRLJihJUi+ZoCRJvdRZgkqyW5JvJzkvyblJXtZVXZKk+afLoY7WAq+oqtOTbAeclmRFVf2kwzolSfNEZz2oqvptVZ3eTl8DnAfs2lV9kqT5ZVauQSVZBuwH/HA26pMkzX2dJ6gk2wKfA/6hqq6eYPmhSVYmWbl69equw5EkzRGdJqgkS2iS0yeq6vMTrVNVR1XV8qpavnTp0i7DkSTNIV3exRfgQ8B5VfX2ruqRJM1PXfagHgg8A3hIkjPa12M6rE+SNI90dpt5VX0PSFflS/PBdWvWsdkf13DEl84ZaRyLErbfagkwuicOS+P5yHdphC656vpRhwDAVksWcc/ddhx1GNItONSRJKmX7EFJI/KAvXYedQjrnXnRVaMOQboVe1CSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeqmzBJXkw0kuTXJOV3VIkuavLntQRwOP6rB8SdI8trirgqvqO0mWdVW+pG68Y8UFow4BgMMevs+oQ9CIdZagJM0df7pxHaf/+spRhwHAnjtvw07bbD7qMNQDI09QSQ4FDgXYfffdRxyNtDAVsGbtTaMOA4B1VaMOQT0x8gRVVUcBRwEsX77cPVMagQfstfOoQwDglFWXjzoE9Yi3mUuSeqnL28w/BZwM3CXJxUme21VdkqT5p8u7+J7WVdmSpPnPU3ySpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSeskEJUnqJROUJKmXTFCSpF4yQUmSemnko5lL0qA1a2/i+jXr+NevnjfSOBLYcskiwIcnjooJSlKv/PqKP/LrK/446jDYaski7rnbjqMOY0HzFJ8kqZfsQUnqjb48OBHgzIuuGnUIC549KElSL5mgJEm9ZIKSJPWSCUqS1EsmKElSL5mgJEm9ZIKSJPWSCUqS1EsmKElSLzmShCT12DtWXDDqECY0GwPomqAkaQqjThKrr7mB69asHWkMAEs224xdd9pq1uozQUnSBG5cd1MvRlUHuPK6NVx/47pRh8GWSxaZoCRp1NbeVFxy1fWjDuMWRj2Y7hmzPICuCUqSJjDqZCDv4pMk9VSnCSrJo5Kcn+TnSV7VZV2SpPmlswSVZBHwHuDRwL7A05Ls21V9kqT5pcse1P2An1fVqqpaAxwLPLHD+iRJ80iXN0nsClw08P5i4P4d1if12u2235Lbbr/FqMOQNtrWmy+a1fq6TFCZYF7daqXkUODQ9u21Sc6fYb27AJe9fIaFjMguwGWjDmIG5nL8czl2mNvxz+XYYYHGP8Nj7B7DrNRlgroY2G3g/Z2AS8avVFVHAUdtqkqTrKyq5ZuqvNk0l2OHuR3/XI4d5nb8czl2MP4udXkN6lRg7yR7JtkceCrw5Q7rkyTNI531oKpqbZIXA98AFgEfrqpzu6pPkjS/dDqSRFUdDxzfZR0T2GSnC0dgLscOczv+uRw7zO3453LsYPydSdWt7luQJGnkHOpIktRLcyZBTTVsUpItkny6Xf7DJMsGlr26nX9+kkfOZtwDMWxU/EmWJbk+yRnt6309jP3BSU5PsjbJgeOWHZLkZ+3rkNmL+hYxzCT+dQNtP+s3+QwR+8uT/CTJWUn+N8keA8vmQttvKP6Rtn0bw1TxvyDJ2W2M3xscLWfUx52Njb0Px5z1qqr3L5qbLH4B7AVsDpwJ7DtunRcC72unnwp8up3et11/C2DPtpxFcyj+ZcA5PW/7ZcBfAB8DDhyYfxtgVfvvTu30TnMl/nbZtT1v+wOArdvpvx/Yb+ZK208Y/6jbfhrxbz8w/QTg6+30SI87M4x9pMecwddc6UENM2zSE4GPttPHAQ9Nknb+sVV1Q1X9Evh5W95smkn8ozZl7FV1YVWdBdw0bttHAiuq6oqquhJYATxqNoIeMJP4R22Y2L9dVWNP1TuF5u8NYe60/WTx98Ew8V898HYbbh6MYNTHnZnE3htzJUFNNGzSrpOtU1VrgT8AOw+5bddmEj/Ankl+nOSkJA/qOtjJ4mpNp/3mSttvyJZJViY5JcnfbtrQpjTd2J8LfG0jt+3CTOKH0bY9DBl/khcl+QXwFuCl09m2QzOJHUZ7zFlvrjywcJhhkyZbZ6ghlzo2k/h/C+xeVZcnuQ/wxSR3H/frp0szab+50vYbsntVXZJkL+BbSc6uql9sotimMnTsSQ4GlgN/M91tOzST+GG0bQ9Dxl9V7wHek+TpwGuAQ4bdtkMziX3Ux5z15koPaphhk9avk2QxsANwxZDbdm2j429PEVwOUFWn0ZxX3qfziCeIqzWd9psrbT+pqrqk/XcVcCKw36YMbgpDxZ7kYcDhwBOq6obpbNuxmcQ/6raH6bfhscBYT2/U7b/RsffgmHOzUV8EG+ZF09NbRXOxceyC393HrfMibnmTwWfa6btzy4uVq5j9myRmEv/SsXhpLnj+BrhNn2IfWPdobn2TxC9pLtLv1E7PWuybIP6dgC3a6V2AnzHuQvOoY6c5aP8C2Hvc/DnR9huIf6RtP4349x6Yfjywsp0e6XFnhrGP9JhzixhHUelGNvhjgAvanfnwdt4baH51AWwJfJbmYuSPgL0Gtj283e584NFzKX7gScC57Q52OvD4HsZ+X5pfbNcBlwPnDmz7nPYz/Rx4dk/bfsL4gb8Czm7b/mzguT2M/ZvA74Ez2teX51jbTxh/H9p+yPjf2f7/PAP4NgNJYNTHnY2NvQ/HnLGXI0lIknpprlyDkiQtMCYoSVIvmaAkSb1kgpIk9ZIJSpLUSyYoSVIvmaAWkIHHF5yb5Mz2UQebtcuWJ/mvdnqLJN9s131Kkge125yRZKvRfoqJJXlDOyLBTMrYMckLN2FMy5KcM8MyjkzyyiHWu3Yjy79jkuOmWGdZOxTO2Pv1+8pMpfGtJNtvivImKP/EJMuHXPc/kjykizi0cebKWHzaNK6vqnsBJLkt8EmaIZVeV1UrgZXtevsBSwbWfR/wH1X1kWEqaUdhT1XN2ujgVXXEJihmR5rHnrx3E5Q1bUkWVdW62ayzmuGEDpxitWXA02n2F8btKzP1GODMGsE4bxN4F/AB4FujDkQNe1ALVFVdChwKvLj9Fbt/kq+0iesY4F5tj+n5wJOBI5J8AiDJPyY5Nc1D5l7fzluW5Lwk76X56/PdkjwiyclpHgb42STbtutemOT17fyzk9y1nb9tko+0885K8qR2/oTlDEpydNqHDU5W/rj1757kR+1nPCvJ3sC/A3/WzntrG8//DpTzxHGf9QNtz/KEsZ5lkvu0vdOTaYavYmCb77ZlnZ7kr9r5+yf5dpJP0oyYQJLD0zxo7pvAXSb6/pLs2bbJqUneOG7ZRN/Pmwd7h23P7BWDvbzJYmzb5UFtuxw2tq+029wmyRfbuk5J8hcD5X+47cGsSjI4Uvagg4Avtdv809h6Sd6R5Fvt9EOTHLOhfaFt95OSnJbkG0nuMK5NNkvy0SRvSrKo3V/Oab/XwwCq6lfAzkluP0msmm2jGsLC1+y/mOABcMCVwO2A/YGvtPPWT7fvj6Ydow54BHAUzWjJmwFfAR5M8yv7JuAB7Xq7AN8Btmnf/z/giHb6QuAl7fQLgQ+2028G/nOg3p02VM64zzEY44Tlj1v/XcBB7fTmwFaMe1AbzRmG7Qc+z8/bz70MWAvcq132GeDgdvos4G/a6beOlQdsDWzZTu/NzeOe7U8zxNKe7fv70CSqrYHt2zpfOUH8Xwae2U6/aOy73cD3sx9w0sD2PwF2H/zMU8Q4uD+sf9+24+va6YcAZ7TTRwI/oBmLbheaIaSWTPA5fgVs104/APhsO/1dmiG/lgCvA54/2b7QrvMDYGk7/ynAh9vpE9tyP8XNw/3ch+ZZWWMx7Dgw/QHgSaP+v+qreXmKT9N9KOIj2teP2/fb0hzMfg38qqpOaec/gOapot9P89zFzYGTB8r5fPvvacD/aacfRjNQLgBVdWWSx01RzmQmKn/QycDhSe4EfL6qfpZbPx8ywL8meTBN8t2VJpkD/LKqzhioY1mSHWgOdie18z8OPLqdXgK8O8m9gHXccnToH1XzUDuABwFfqPYhfpn8UecPpBkzbayeN7fTE34/VfWhJLdNckeawUCvrKpfJ1k2UOaGYpzMX4/FUVXfSrJz2w4AX61mdPIbklxK03YXj9v+NlV1TTt9GnCfJNsBN9D0xJe3bfJSJt+n7gLcA1jRzl9E88iIMe+nGXz5X9r3q4C9krwL+CpwwsC6lwJ3HOJzaxaYoBawNM/ZWUfzn/Juw24G/FtVvX9cWctoegKD662oqqdNUs7YYxXWcfN+GCZ+TtaGypnMROWvV1WfTPJD4LHAN5I8j+bANeggmoP5farqxiQX0gzqO1j+WB1bTRL/mMNoBkW9J03P5k8Dy64bt+6wA2ROtN6E30/rOJrrTbenebzCdGKczIaeOzS+jSY63qxNsllV3TTQxs+m6RGdRfNI+D8Dzmv/vdW+kOTPaQb4/ctJYvwBcECSt1XVn9ofPvekeerwi2hOYT+nXXdL4PoNfmLNGq9BLVBJlgLvA95d7bmNIX0DeM7Auf9d01y3Gu8U4IFJ7tyut3WSqX6RnwC8eCDGnTaynCm1yXlVVf0XzemyvwCuAbYbWG0H4NL2wHkAsMeGyqyqq4A/JPnrdtZB48r6bTU3jjyD5lf+RL4D/F2SrdqexOMnWe/73NzbHKxnQ9/Pse02B9Ikq/Emi3F8u4yP96C2rv2By2p6NzycT/NIh8HyXtn++13gBTSnDYvJ94XzgaVJ/rKdvyTJ3QfK/BBwPPDZJIuT7AJsVlWfA14L3Htg3X2AGd15qU3HBLWwbNVe6D6X5jEHJwCvn04BVXUCzd1cJyc5m+ZAd6uDV1WtBp4FfCrJWTQHl1vdrDDOm4Cd2ovXZwIHbGQ5w3gKcE6SM9ryPlbNQ9q+39b/VuATwPIkK2kOwj8dotxn0zyh9GRu+Uv8vcAhSU6hOQiO7zUBUFWnA5+meQTC52gO0hN5GfCiJKfSJJax7Sf9fqrq3Hb6N1X121sXOWmMZ9H0dM4cu6FgwJE0bXQWzc0Uh0wS72S+SnNNa8x3gTsAJ1fV72l6cd9t459wX6iqNTRJ983tfnMGzeM61quqt9OcMvw4zanaE9vv/mjg1dAkNuDObLo7FDVDPm5D0si0d9t9rKoe3oNY/g64d1W9dtSxqGEPStLItD25D6SjP9SdpsXA20YdhG5mD0qS1Ev2oCRJvWSCkiT1kglKktRLJihJUi+ZoCRJvfT/AWuFkbiBy2MkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf23d28e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ht = DiffStdPermute(data)\n",
    "p_value = ht.p_value()\n",
    "\n",
    "print('Diff std permute pregnancy length')\n",
    "print('actual =', ht.actual)\n",
    "print('p-value =', p_value)\n",
    "ht.plot_test_stats()\n",
    "plt.xlabel('Difference in standard deviation (weeks)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The observed difference in standard deviation is about 0.17 weeks.\n",
    "\n",
    "The p-value is around 17%, which indicates that, even if there is actually no difference between the groups, there is a non-negligible chance that we could see a difference in standard deviation as big as 0.17 weeks due to random sampling.\n",
    "\n",
    "## Different model\n",
    "\n",
    "It's also easy to run with a different model.  Instead of shuffling the observed data, we could use the data to choose the parameters of a normal distribution, and then draw random samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomNormalTest(HypothesisTest):\n",
    "    \"\"\"Tests by drawing values from a normal distribution.\"\"\"\n",
    "\n",
    "    def run_model(self):\n",
    "        \"\"\"Run the model of the null hypothesis.\n",
    "\n",
    "        returns: simulated data\n",
    "        \"\"\"\n",
    "        n = len(self.data[0])\n",
    "        pool = np.hstack(self.data)\n",
    "        mu, sigma = np.mean(pool), np.std(pool)\n",
    "        random = np.random.normal(mu, sigma, len(pool))\n",
    "        data = np.split(random, [n])\n",
    "        return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make a class that tests a difference in means by drawing values from a normal distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DiffMeansNormal(PermutationTest):\n",
    "    \"\"\"Tests a difference in means using a normal distribution.\"\"\"\n",
    "    \n",
    "    test_stat = DiffMeansPermute.test_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Diff means normal pregnancy length\n",
      "actual = 0.07803726677754952\n",
      "p-value = 0.17\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEzCAYAAABkE5dAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcXFWd9/HPlyQQIAkE0oAsEoIERVCWFlEGB4QgKojPI6MgyOqDjoM64AaCCC6PyygMrphBdgUFRVFRCCKLGsQACYtIwLCFxXRAIECISfjNH+dUKCq9VC9V93T6+369Kl11l3N+ffqmfnXOvXWuIgIzM7PSrFZ1AGZmZt1xgjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlPVI0pmSPjNEZb1c0jOSRuXX10p6/1CUncv7taTDhqq8ftT7BUkLJT3W7rpXFZIOl/T7quMAkHSKpAvz88mSQtLoquMaqZygRihJ90taLGmRpCcl/VHSByWtOCYi4oMR8fkmy9qrt20i4sGIGBcRy4cg9hVvInXlvzUizhts2f2MYzPgY8A2EbFRN+t3lzR/iOoacELv777dta9ZFZygRrb9ImI8sDnwZeBTwPeHupJV+BPo5sDjEbGg6kBGqlqP3FZREeHHCHwA9wN7NSzbGXgB2Da/Phf4Qn4+Cfgl8CTwBHAD6QPOBXmfxcAzwCeByUAARwEPAtfXLRudy7sW+BJwE/AU8HNgvbxud2B+d/EC+wD/BJbm+ubUlff+/Hw14CTgAWABcD6wTl5Xi+OwHNtC4MRe2mmdvH9XLu+kXP5e+Xd+IcdxbsN+azesfwbYOO97PPA34HHgx3W/91jgwrz8SeDPwIbAF4HlwPO5nG91E2e/9gXOAB4CngZuBnbLy7tt327qC+AVda/P5cVjZXdgPql3uQB4FDiibtv1gctz3TcBnwd+X7f+lcAM0nF2N/Duhnq+C1wBPEvDMVx3LHwe+AOwCLgKmNTXsZWfnwJc2HCsjK76/+tIfbgHZStExE2kN5bduln9sbyug/TG9+m0S7yP9Ea/X6QhvK/W7fOvwKuAt/RQ5aHAkaQ37mXAN5qI8TfA/wd+lOt7bTebHZ4fewBTgHHAtxq2+Rdga2BP4GRJr+qhym+SktSU/PscSnqzvRp4K/BIjuPwhjifbVg/LiIeAT4CvDOXtTHwD+DbebfDcl2bkd7EPwgsjogTSR8IjsnlHNNNnP3d98/A9sB6wA+BSySNbbJ9m7FRjmcT0geVb0uamNd9m5QwX0b6+x9Z20nS2qTk9ENgA+Ag4DuSXl1X9ntJiXc80NO5q/cCR+QyVgc+PsDfwyrkBGWNHiG9aTVaSnpD2TwilkbEDZE/ZvbilIh4NiIW97D+goi4I7+ZfwZ49xAN2RwMnBYR8yLiGeAE4MCGocZTI2JxRMwB5gArvRHnWN4DnBARiyLifuDrwPsGEdsHSD22+RGxhPSJ/YAc21JScnlFRCyPiJsj4ukmy+3XvhFxYUQ8HhHLIuLrwBqkhD1UlgKfy8fKFaTe2Na5Td8FnJyPjTuA+nOH+wL3R8Q5ObZbgJ8AB9Rt8/OI+ENEvBARz/dQ/zkRMTcfez8mJWMbZpygrNEmpKGVRv8F3AtcJWmepOObKOuhfqx/ABhDGkocrI1zefVljyb1/Grqr7p7jtTLajSJ9Om7saxNBhHb5sBl+cKUJ4G7SENwG5KGS68ELpb0iKSvShrTZLn92lfSxyTdJempHMc6DE3b1zweEcvqXtfauIP0t2j829dsDry+1j45toNJPbKavo4raO7va4VzgrIVJL2O9Oa70rBJ7kF8LCKmAPsBx0nas7a6hyL76mFtVvf85aRP3QtJ5xbWqotrFOmNrdlyHyG90dWXvQz4ex/7NVqYY2os6+Em9+8uzoeAt0bEunWPsRHxcO5tnBoR2wBvJPUmDu2lrBcr6se+knYjXRDzbmBiRKxLOg+oZurKnqPub8RLE0hvukh/i8a/fc1DwHUN7TMuIv69bpvB3IKhr2PLCuIEZUiaIGlf4GLSCeLbu9lmX0mvkCTSye3l+QHpjX/KAKo+RNI2ktYCPgdcGuky9LnAWElvz72Ak0hDUDV/BybXXxLf4CLgWElbSBrHi+dUlvWwfbdyLD8GvihpvKTNgeNIFyM04+/A+pLWqVt2Zi5vcwBJHZL2z8/3kLRdftN8mpQcm2rjfu47npQkuoDRkk4GJjTE3Vv7AswG3itplKR9SOfU+pTb9KfAKZLWkrQN6fxZzS+BqZLeJ2lMfryul3OE/dXXsWUFcYIa2X4haRHpU+uJwGmkE8vd2Qq4mnQuYSbwnYi4Nq/7EnBSHpLpz8noC0hXZT1GugrtIwAR8RTwIeAsUm/lWdIFGjWX5J+PS7qlm3LPzmVfD9xHOiH/4X7EVe/Duf55pJ7lD3P5fYqIv5KS5bzcNhuTrp67nDRUugi4EXh93mUj4FJSgrkLuI4Xk+EZpHNV/5DU3cUk/dn3SuDXpDfrB0jtUz9s1lf7AnyU1JOuDcH9rO8WWeEY0pDbY6S//zm1FRGxCNgbOJDUE34M+ApDlESaOLasIOr7PLeZmVn7uQdlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlakYTHL9KRJk2Ly5MlVh2FmZkPg5ptvXhgRfX5BelgkqMmTJzNr1qyqwzAzsyEg6YG+t/IQn5mZFcoJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWpJYlKElnS1og6Y5u1n1cUkgayltMm5nZKqSVPahzgX0aF0raDJgGPNjCus3MbJhr2UwSEXG9pMndrDod+CTw81bVvVKFM+a2q6qmHTttatUhmJkVra3noCS9A3g4IuY0se3RkmZJmtXV1dWG6MzMrCRtS1CS1gJOBE5uZvuImB4RnRHR2dHR55yCZma2imnnZLFbAlsAcyQBbArcImnniHisXUFUObRW4lCjmVmp2pagIuJ2YIPaa0n3A50RsbBdMZiZ2fDRysvMLwJmAltLmi/pqFbVZWZmq55WXsV3UB/rJ7eqbjMzG/48k4SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7Mija46gJHq9Blzqw4BgGOnTa06BDOzbrkHZWZmRXKCMjOzInmIr41KGU4rZXjRzKw3LetBSTpb0gJJd9Qt+y9Jf5V0m6TLJK3bqvrNzGx4a+UQ37nAPg3LZgDbRsRrgLnACS2s38zMhrGWJaiIuB54omHZVRGxLL+8Edi0VfWbmdnwVuVFEkcCv66wfjMzK1glCUrSicAy4Ae9bHO0pFmSZnV1dbUvODMzK0LbE5Skw4B9gYMjInraLiKmR0RnRHR2dHS0L0AzMytCWy8zl7QP8CngXyPiuXbWbWZmw0srLzO/CJgJbC1pvqSjgG8B44EZkmZLOrNV9ZuZ2fDWsh5URBzUzeLvt6o+MzNbtXiqIzMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmVqSWJShJZ0taIOmOumXrSZoh6Z78c2Kr6jczs+GtlT2oc4F9GpYdD/w2IrYCfptfm5mZraRlCSoirgeeaFi8P3Befn4e8M5W1W9mZsNbu89BbRgRjwLknxu0uX4zMxsmir1IQtLRkmZJmtXV1VV1OGZm1mbtTlB/l/QygPxzQU8bRsT0iOiMiM6Ojo62BWhmZmVod4K6HDgsPz8M+Hmb6zczs2GilZeZXwTMBLaWNF/SUcCXgWmS7gGm5ddmZmYrGd2qgiPioB5W7dmqOs3MbNVR7EUSZmY2sjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFatkXdW14OH3G3KpDAODYaVOrDsHMCuMelJmZFckJyszMiuQhvhGolOG0UoYXzaxM7kGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhNJShJP5H0dklOaGZm1hbNJpzvAu8F7pH0ZUmvbGFMZmZmzSWoiLg6Ig4GdgTuB2ZI+qOkIySNaWWAZmY2MjU9ZCdpfeBw4P3ArcAZpIQ1o7+VSjpW0p2S7pB0kaSx/S3DzMxWbc2eg/opcAOwFrBfRLwjIn4UER8GxvWnQkmbAB8BOiNiW2AUcGD/wjYzs1Vds/eDOisirqhfIGmNiFgSEZ0DrHdNSUtJSe+RAZRhZmarsGaH+L7QzbKZA6kwIh4GvgY8CDwKPBURVzVuJ+loSbMkzerq6hpIVWZmNoz1mqAkbSRpJ1JvZwdJO+bH7qSeT79JmgjsD2wBbAysLemQxu0iYnpEdEZEZ0dHx0CqMjOzYayvIb63kC6M2BQ4rW75IuDTA6xzL+C+iOiCFee33ghcOMDyzMxsFdRrgoqI84DzJL0rIn4yRHU+COwiaS1gMbAnMGuIyjYzs1VErwlK0iERcSEwWdJxjesj4rRudutVRPxJ0qXALcAy0iXr0/tbjpmZrdr6GuJbO//s16XkfYmIzwKfHcoyzcxs1dLXEN/38s9T2xOOmZlZ0uwXdb8qaYKkMZJ+K2lhd1femZmZDZVmvwe1d0Q8DewLzAemAp9oWVRmZjbiNZugahPCvg24KCKeaFE8ZmZmQPNTHf1C0l9Jl4V/SFIH8HzrwrKR5vQZc6sOAYBjp02tOgQzy5q93cbxwBtIE7wuBZ4lzQZhZmbWEs32oABeRfo+VP0+5w9xPGZmZkCTCUrSBcCWwGxgeV4cOEHZIJQynFbK8KKZvVSzPahOYJuIiFYGY2ZmVtPsVXx3ABu1MhAzM7N6zfagJgF/kXQTsKS2MCLe0ZKozMxsxGs2QZ3SyiDMzMwaNZWgIuI6SZsDW0XE1flWGaNaG5qZmY1kzc7F9/+AS4Hv5UWbAD9rVVBmZmbNXiTxH8CuwNMAEXEPsEGrgjIzM2s2QS2JiH/WXuQv6/qSczMza5lmE9R1kj4NrClpGnAJ8IvWhWVmZiNdswnqeKALuB34AHAFcFKrgjIzM2v2Kr4XJP0M+FlEdLU4JjMzs957UEpOkbQQ+Ctwt6QuSSe3JzwzMxup+hri+0/S1Xuvi4j1I2I94PXArpKObXl0ZmY2YvWVoA4FDoqI+2oLImIecEheZ2Zm1hJ9JagxEbGwcWE+DzWmm+3NzMyGRF8J6p8DXGdmZjYofV3F91pJT3ezXMDYgVYqaV3gLGBb0hd+j4yImQMtz8zMVj29JqiIaNWEsGcAv4mIAyStDqzVonrMzGyYavZ2G0NG0gTgTcDhAHkKJQ8XmpnZSzQ7k8RQmkKaleIcSbdKOkvS2hXEYWZmBasiQY0GdgS+GxE7AM+SplJ6CUlHS5olaVZXlyevMDMbaapIUPOB+RHxp/z6UlLCeomImB4RnRHR2dHR0dYAzcysem1PUBHxGPCQpK3zoj2Bv7Q7DjMzK1vbL5LIPgz8IF/BNw84oqI4zMysUJUkqIiYDXRWUbeZmQ0PVZyDMjMz65MTlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkar6HpRZkU6fMbfqEAA4dtrUqkMwq5wTlFmD3/11ARPWbP8NozeaMJZNJq7Z9nrNSuUEZdZg6fIXWLr8hbbXuzyi7XWalcwJyka8xuG0+f94jk0ntv8emg8+8Vzb6zQrmS+SMDOzIjlBmZlZkZygzMysSE5QZmZWJCcoMzMrkhOUmZkVyQnKzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRWpsgQlaZSkWyX9sqoYzMysXFX2oD4K3FVh/WZmVrBKEpSkTYG3A2dVUb+ZmZWvqh7UfwOfBNp/X20zMxsW2p6gJO0LLIiIm/vY7mhJsyTN6urqalN0ZmZWiip6ULsC75B0P3Ax8GZJFzZuFBHTI6IzIjo7OjraHaOZmVWs7QkqIk6IiE0jYjJwIHBNRBzS7jjMzKxs/h6UmZkVaXSVlUfEtcC1VcZgZmZlcg/KzMyK5ARlZmZFcoIyM7MiOUGZmVmRnKDMzKxITlBmZlYkJygzMyuSE5SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRXKCMjOzIjlBmZlZkUZXHYCZrez0GXOrDoFjp02tOgQb4ZygzArx9OKlPPD4s1WHwcS1VmfCmmOqDsPMCcqsFM8sWcYzS5ZVHQZjRq3mBGVFcIIyK8QuU9avOgRunPd41SGYreCLJMzMrEhtT1CSNpP0O0l3SbpT0kfbHYOZmZWviiG+ZcDHIuIWSeOBmyXNiIi/VBCLmZkVqu09qIh4NCJuyc8XAXcBm7Q7DjMzK1ul56AkTQZ2AP7UzbqjJc2SNKurq6vdoZmZWcUqS1CSxgE/Af4zIp5uXB8R0yOiMyI6Ozo62h+gmZlVqpIEJWkMKTn9ICJ+WkUMZmZWtiqu4hPwfeCuiDit3fWbmdnwUEUPalfgfcCbJc3Oj7dVEIeZmRWs7ZeZR8TvAbW7XjMzG148k4SZmRXJCcrMzIrkBGVmZkVygjIzsyI5QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyszMiuQEZWZmRarilu9mNgycPmNu1SEAcOy0qVWHYBVxgjKzl3jwied48Innqg6DLTvG0TF+jarDsAp5iM/MzIrkHpSZrbDLlPWrDgGAG+c9XnUIVgD3oMzMrEhOUGZmViQnKDMzK5LPQZlZ0Xy5+8jlBGVmRZrX9QzzFj5TdRhM3XA8E9daveowRiQnKDMrUqz4x0YqJygzK05Jl7s/vXgpy18ITrzs9kpjWWfNMYwZtdqIGmp0gjIz68WjTz1fdQgAvHrjCYwZNbKuaxtZv62ZmQ0blfSgJO0DnAGMAs6KiC9XEYeZWW9KGmocidqeoCSNAr4NTAPmA3+WdHlE/KXdsZiZDTcj6bL7KnpQOwP3RsQ8AEkXA/sDTlBmZj2469FFoKqjgG02Gs+4sWPaUlcVCWoT4KG61/OB11cQh1m3Npwwlg0m+DYPVo61Vy/neja1MUlW8Vt39+ut9G0HSUcDR+eXz0i6e5D1TgIWHjfIQtpoErCw6iD6yTG33nCLFxxzO7Q93kG+l27ezEZVJKj5wGZ1rzcFHmncKCKmA9OHqlJJsyKic6jKa7XhFi845nYYbvGCY26H4RZvs6q4zPzPwFaStpC0OnAgcHkFcZiZWcHa3oOKiGWSjgGuJF1mfnZE3NnuOMzMrGyVnHmLiCuAK9pc7ZANF7bJcIsXHHM7DLd4wTG3w3CLtymK8GyMZmZWHk91ZGZmRRqWCUrSPpLulnSvpOO7Wb+GpB/l9X+SNLlu3Ql5+d2S3tJsmVXFLGmapJsl3Z5/vrlun2tzmbPzY4MC4p0saXFdTGfW7bNT/j3ulfQNaWi/UTGImA+ui3e2pBckbZ/XtayNm4z5TZJukbRM0gEN6w6TdE9+HFa3vGXtPNB4JW0vaaakOyXdJuk9devOlXRfXRtvP1TxDibmvG55XVyX1y3fIh9D9+RjashuGDWINt6j4Th+XtI787qWtnHLRMSwepAurPgbMAVYHZgDbNOwzYeAM/PzA4Ef5efb5O3XALbI5YxqpswKY94B2Dg/3xZ4uG6fa4HOwtp4MnBHD+XeBLyB9F24XwNvLSHmhm22A+a1uo37EfNk4DXA+cABdcvXA+blnxPz84mtbOdBxjsV2Co/3xh4FFg3vz63fttS2jive6aHcn8MHJifnwn8ewnxNhwfTwBrtbqNW/kYjj2oFVMlRcQ/gdpUSfX2B87Lzy8F9syfIvcHLo6IJRFxH3BvLq+ZMiuJOSJujYja98TuBMZKavU0B4Np425JehkwISJmRvofcz7wzgJjPgi4aAjj6k2fMUfE/RFxG/BCw75vAWZExBMR8Q9gBrBPi9t5wPFGxNyIuCc/fwRYAHQMUVwtibkn+Zh5M+kYgnRMVd7GDQ4Afh0Rzw1RXJUYjgmqu6mSNulpm4hYBjwFrN/Lvs2UWVXM9d4F3BoRS+qWnZO77J8ZwqGcwca7haRbJV0nabe67ef3UWaVMde8h5UTVCva+CXxZP1pk96O5Va185D8P5G0M6l38Le6xV/MQ3+nD/EHsMHGPFbSLEk31obLSMfMk/kYGkiZvRmq96IDWfk4blUbt8xwTFDNTJXU0zb9XT5UBhNzWim9GvgK8IG69QdHxHbAbvnxvkHG2VQsfWzzKPDyiNiBNBvKDyVNaLLMwRiKNn498FxE3FG3vlVt3Gc8A9y3le086LJzD+8C4IiIqPUATgBeCbyONDT1qcEE2VhlN8v6E/PLI83Q8F7gvyVtOQRl9mao2ng70ndNa1rZxi0zHBNUM1MlrdhG0mhgHdJ4bE/7NjX9UkUxI2lT4DLg0IhY8akzIh7OPxcBPyQND1Qabx4+fTzHdTPpU/LUvP2mfZRZScx161f61NnCNm425v7u28p2HtT/k/xB5VfASRFxY215RDwayRLgHMpp49pwJJHuvnAt6ZzwQmDdfAz1u8w+DMV70buByyJiaW1Bi9u4ZYZjgmpmqqTLgdpVTQcA1+Tx+MuBA5Wu5toC2Ip0QrnV0y8NOGZJ65L+U58QEX+obSxptKRJ+fkYYF/gDobGYOLtULrnF5KmkNp4XkQ8CiyStEseJjsU+PkQxTuomHOsqwH/RhrzJy9rZRs3G3NPrgT2ljRR0kRgb+DKFrfzgOPN218GnB8RlzSse1n+KdK5nCLaOLftGvn5JGBX4C/5mPkd6RiCdExV3sZ1VjqP2uI2bp2qr9IYyAN4GzCX9On8xLzsc8A78vOxwCWkiyBuAqbU7Xti3u9u6q5u6q7MEmIGTgKeBWbXPTYA1gZuBm4jXTxxBjCqgHjfleOZA9wC7FdXZifpP8bfgG+Rvyhedcx53e7AjQ3ltbSNm4z5daRP1c8CjwN31u17ZP5d7iUNmbW8nQcaL3AIsLThON4+r7sGuD3HfCEwroQ2Bt6Y45qTfx5VV+aUfAzdm4+pNaqON6+bDDwMrNZQZkvbuFUPzyRhZmZFGo5DfGZmNgI4QZmZWZGcoMzMrEhOUGZmViQnKDMzK5ITlJmZFckJyiqjF29lcKekOZKOy1+YRVKnpG/k52tIujpv+x5Ju+V9Zktas9rfonuSPidpr6rjaJaSa/JsD60o/1pJnU1u+zXV3VbGRq5Kbvluli2OiNp9lzYgTSW0DvDZiJgFzMrb7QCMqdv2TOBrEXFOM5Xkb88rXpz7reUi4uR21TVE3gbMiYinqw4E+CbwP6Qvl9oI5h6UFSEiFgBHA8fkT/O7S/plTlwXAtvnHtMHSHONnSzpBwCSPiHpz3mm5lPzssmS7pL0HdKMFptJ2lvppnm3SLpE0ri87f2STs3Lb5f0yrx8nKRz8rLbJL0rL++2nHpKN4g7oLfyG7Y/XNLPJP1C6cZyx+Qe5a1KM2mvl7fbUtJvlG5eeUNdrPsp3UDv1tzb3DAvP0XS2bkHM0/SR3r4ExxMnq5H0idr2ynNfH1Nfr6npAt7awOlmyVel+O7sjbFTt3vuZqk8yR9QdKo3E535HY5Nh8LDwDrS9qoj8PGVnFOUFaMSBNyrkaayqm2bAHwfuCGiNg+Ir5HmpvsExFxsKS9SfP97QxsD+wk6U15961Jc7/tQJoW5iRgr4jYkdQ7O66u+oV5+XeBj+dlnwGeiojtIuI1wDVKc7L1Vk5Puiu/0bakWbN3Br5Imll9B2AmaU49gOnAhyNip1zOd/Ly3wO75O0vBj5ZV+4rSfeP2hn4rNK8go12JU3rBHA9aeZ2SNMmjcv7/AtwQ09tkLf5JunGeDsBZ+ffo2Y08ANgbkScRPp7bRIR20aaMb6+R3xLjslGMA/xWWn6e7+lvfPj1vx6HClhPQg8EC/Omr0L6Y7Kf0gjfqxOeuOv+Wn+eTPwf/PzvUiTdQIQEf+QtG8f5fSku/Ib/S7SrOmLJD0F/CIvvx14Te6lvBG4RC/elqp2X59NgR/lHsvqwH115f4q0izWSyQtADbkpfeMAlgv112LcSdJ44ElpGTRSUpaH6HnttyalGRn5OWjSLdfqfke8OOIqCWtecAUSd8kTYh8Vd22C0h33rURzAnKiqE0+/ly0pvTq5rdDfhS7lnVlzWZ1Guq325GRBzUQzm1m0Au58X/F6L7e0r1Vk5Puiu/p20g3S11Sd3z0aTe5ZO1c3ENvgmcFhGXS9odOKWHcnuqf5mk1SLihYhYKul+4Ajgj6TJcvcAtgTuyj9XagNJ25EmLn1DD7/fH4E9JH09Ip7PCf+1pN7df5CGbo/M244FFvdQjo0QHuKzIkjqAM4EvhX9m8H4SuDIunMgm+TzVo1uBHaV9Iq83VqSpvZR9lXAMXUxThxgOUMiX8Bwn6R/y3Urv8FDurjk4fz8sO7278PdpBm6a64nDSFeD9wAfBCYnf82PbXB3UCHpDfk5WOUbrRZ833gClIPsHYrk9Ui4iek4dQd67adynC5JYS1jBOUVWlN5cvMgatJCeHU/hQQEVeRrv6bKel24FJgfDfbdQGHAxdJuo30JrvSxQoNvgBMzCfx5wB7DLCcoXQwcFSO505g/7z8FNIb/w2kG+r1169ItxypuQF4GTAzIv4OPJ+X9diWEfFP0j2SvpLjm00aklwhIk4jDRleQLqV+bWSZgPnku76Wrv31it48SpOG6F8uw0zq93Q7vyImFZALP8H2DEiPlN1LFYt96DMjEh34v0fteiLuv00Gvh61UFY9dyDMjOzIrkHZWZmRXKCMjOzIjlBmZlZkZygzMysSE5QZmbdwLnHAAAACElEQVRWpP8FsoNSW8L94GUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcf23d12588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ht = DiffMeansNormal(data)\n",
    "p_value = ht.p_value()\n",
    "\n",
    "print('Diff means normal pregnancy length')\n",
    "print('actual =', ht.actual)\n",
    "print('p-value =', p_value)\n",
    "ht.plot_test_stats()\n",
    "plt.xlabel('Difference in mean (weeks)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results from this model are not substantially different, which suggests that they do not depend on arbitrary modeling decisions, and that's good.\n",
    "\n",
    "For this example, the permutation test is probably better because it does not make any assumptions about the distribution of pregnancy lengths.\n",
    "\n",
    "If the sample size were substantially smaller, the model that draws from the normal distribution might be better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}