ConfidenceIntervals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "executed-sword",
   "metadata": {},
   "source": [
    "# Confidence Intervals\n",
    "This notebook is designed to help you get familiar and comfortable with confidence intervals on proportion (percentage of total) metrics such as sensitivity.\n",
    "\n",
    "As you go through this notebook, you are encouraged to change the input variables from the defaults. This will help develop an intuition of how these inputs affect the results. After doing so, we suggest returning the values to the defaults and re-running so that the subsequent sections will have expected results (by using those defaults).\n",
    "\n",
    "To run a code block press `shift`+`enter`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tropical-behalf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "from time import sleep\n",
    "\n",
    "from helpers import run_many_proportion_trials, display_trials, run_single_proportion_trial, display_single_trial, within_confidence_interval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "likely-religious",
   "metadata": {},
   "source": [
    "# Variables\n",
    "\n",
    "| Variable | Description |\n",
    "|-|-|\n",
    "| true_sensitivity | The true sensitivity is the sensitivity one would get if they sampled the entire population. |\n",
    "| num_trials | The number of trials is how many independent trials are simulated. |\n",
    "| num_observations | The number of observations per trial. |\n",
    "| confidence | The percentage used to generate the confidence intervals. This percentage is the probability the true answer is within the bounds. |\n",
    "\n",
    "Run the code block below to initialize the default variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "annoying-scope",
   "metadata": {},
   "outputs": [],
   "source": [
    "default_true_sensitivity = 0.70\n",
    "default_num_trials = 50000\n",
    "default_num_observations = 200\n",
    "default_confidence = 0.95"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "automatic-bangkok",
   "metadata": {},
   "source": [
    "# Trial Distribution\n",
    "The code below runs `num_trials` trials each with `num_observations` observations. These are then plotted in a histogram where:\n",
    "* the x-axis is the sensitivity observed in the simulated trial\n",
    "* the y-axis is the count of simulated trials with that sensitivity\n",
    "\n",
    "Run the code below and then change the variables to see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "specific-facing",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variables to play with Note: percentages are done as ratios (e.g. 95% => 0.95)\n",
    "true_sensitivity = default_true_sensitivity\n",
    "num_trials = default_num_trials\n",
    "num_observations = default_num_observations\n",
    "percentile = default_confidence\n",
    "\n",
    "# Run the trials\n",
    "trials = run_many_proportion_trials(\n",
    "    true_sensitivity,\n",
    "    num_trials=num_trials,\n",
    "    num_observations=num_observations,\n",
    ")\n",
    "\n",
    "# Display trials\n",
    "display_trials(\n",
    "    trials,\n",
    "    true_value=true_sensitivity,\n",
    "    percentile_bounds=percentile,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prospective-division",
   "metadata": {},
   "source": [
    "## Questions\n",
    "* Are the red bars in the above graph confidence intervals?\n",
    "* Is the above graph a normal distribution? What if you change `true_sensitivity` to 0.99?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subject-subscription",
   "metadata": {},
   "source": [
    "# Run Trial With Observed Value\n",
    "The code below runs a single trial with a `single_trial_sensitivity` sensitivity and `single_trial_num_observations` observations. This single trial is displayed on top of the distribution from the previous section.\n",
    "\n",
    "Run the code below and then change the variables to see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "composite-adelaide",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variables to play with Note: percentages are done as ratios (e.g. 95% => 0.95)\n",
    "single_trial_true_sensitivity = default_true_sensitivity\n",
    "single_trial_num_observations = default_num_observations\n",
    "confidence = default_confidence\n",
    "\n",
    "# Run new trial\n",
    "single_trial = run_single_proportion_trial(\n",
    "    single_trial_true_sensitivity,\n",
    "    num_observations=single_trial_num_observations,\n",
    ")\n",
    "\n",
    "# Display new trial along with previously generated trials\n",
    "display_single_trial(\n",
    "    single_trial,\n",
    "    confidence=confidence,\n",
    "    true_value=true_sensitivity,\n",
    "    distribution=trials,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "russian-trash",
   "metadata": {},
   "source": [
    "## Questions\n",
    "* What happens when you rerun the new trial?\n",
    "* What happens when the single trial's sensitivity is set to a different number?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cardiac-rugby",
   "metadata": {},
   "source": [
    "# P-Hacking\n",
    "P-Hacking describes running many statistical tests in a row until one eventually succeeds (i.e., rejecting the null hypothesis). For the code below assume the `true_sensitivity` is the null hypothesis, and we want to show that our trial performs differently (can be higher or lower than null). The code below keeps running trials until one succeeds at rejecting the null-hypothesis. \n",
    "\n",
    "Run the code below and then change the variables to see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dressed-daisy",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variables to play with Note: percentages are done as ratios (e.g. 95% => 0.95)\n",
    "trial_sensitivity = default_true_sensitivity\n",
    "confidence = default_confidence\n",
    "\n",
    "# Initialize variables\n",
    "single_trial = (true_sensitivity, 0.1)\n",
    "count = 0\n",
    "# Keep running while observed sensitivity is within confidence intervals\n",
    "while within_confidence_interval(single_trial, true_sensitivity, confidence):\n",
    "    # Run new trial\n",
    "    single_trial = run_single_proportion_trial(\n",
    "        true_sensitivity,\n",
    "        num_observations=num_observations,\n",
    "    )\n",
    "    sleep(0.5)\n",
    "    clear_output()\n",
    "    print(f'trial {count+1}')\n",
    "    # Display trial\n",
    "    display_single_trial(\n",
    "        single_trial,\n",
    "        confidence=confidence,\n",
    "        true_value=true_sensitivity,\n",
    "        distribution=trials,\n",
    "    )\n",
    "    count +=1\n",
    "\n",
    "# Display last trial\n",
    "clear_output()\n",
    "print(f'Ran {count} trials')\n",
    "display_single_trial(\n",
    "    single_trial,\n",
    "    confidence=confidence,\n",
    "    true_value=true_sensitivity,\n",
    "    distribution=trials,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "supreme-storm",
   "metadata": {},
   "source": [
    "## Questions\n",
    "* Does the \"model\" we derived running above actually have a different sensitivity?\n",
    "* How might we see this occur in real life?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}